Data Analysis Using Python

Samatrix Consulting Pvt Ltd

Introduction to Python
High Level vs Low Level Programming
• High Level Programs
• Close to human language
• No particular knowledge of hardware is required
• Portable
• Far from machine code
• Example: Python, Java, C++

• Low Level Programs

• Uses instructions and object at machine level
• Related to specific architecture and hardware
• Example: Assembly Language and Machine Code
General Versus Targeted Language
• Whether the primitive operations of a programming language are
widely applicable or fine-tuned to one particular domain?
• Example: Adobe Flesh is used for graphics for webpage but cannot be
used for other purpose such as stock portfolio analysis
Interpreted versus Compiled Language
• Whether the sequence of instructions that is called source code is
executed directly by interpreter
• Whether source code is first converted into a sequence of machine
level primitive operations by compiler into a sequence of machine-
level primitive operations.

• Interpreted language are easy to debug

• Compiled languages run quickly and use less space
General Purpose Programming Language
• Python is a general-purpose programming language that can be used
effectively to build almost any kind of program that does not need
direct access to the computer’s hardware.
Advantages of Python
• Python is a relatively simple language that is easy to learn. Because Python
is interpreted language, it can provide runtime feedback that is especially
helpful to novice programmers.
• There are also a large number of freely available libraries that interface to
Python and provide useful extended functionality.
• Python is a living language. Since its introduction by Guido von Rossum in
1990, it has undergone many changes.
• Initially Python was a little known and little used language. It gained
popularity with the arrival of Python 2.0 in 2000.
• Python 3.0 was released at the end of 2008.
• This version of Python cleaned up many of the inconsistencies in the design
of the various releases of Python 2 (often referred to as Python 2.x).
Why Python for Data Analysis
• Python is a popular choice for data analysis and data visualization.
• Python can be compared with other open source and commercial
programming languages and tools, such as R, MATLAB, SAS, Stata, and
others.
• Due to its improved support for libraries (such as pandas and scikit-
learn),
• Python has become a popular choice for data analysis tasks.
• Due to its strength for general-purpose software engineering, Python
is an excellent option as a primary language for building data
applications.
Why not Python?
• Python is an interpreted programming language.
• Hence, most Python code will run substantially slower than code
written in a compiled language like Java or C++. As programmer time
is often more valuable than CPU time, many are happy to make this
trade-off.
• Python can be a challenging language for building highly concurrent,
multithreaded applications, particularly applications with many CPU-
bound threads
Essential Python Libraries
NumPy
• NumPy (Numerical Python) is used for numerical computing in Python.
• NumPy has the data structures, algorithms, and library glue that are
required for most scientific applications involving numerical data.
• Among other things, the important features of NumPy are as follows:
• A fast and efficient multidimensional array object ndarray
• Functions for performing element-wise computations with arrays or mathematical
operations between arrays
• Tools for reading and writing array-based datasets to disk
• Linear algebra operations, Fourier transform, and random number generation
• In addition to fast array-processing capabilities, the ability of passing data
as a container between algorithms and libraries makes NumPy an
important tool for data analysis.
pandas
• pandas helps working with structured and tabular data fast and easy.
• It has high level data structures and functions.
• It emerged in 2010. Since then, it has helped Python to be a powerful
data analysis environment.
• The primary objects of pandas are DataFrame and the series.
• pandas not only provides high-performance array computation but
also flexible data manipulation capabilities of spreadsheets and
relational databases (such as SQL).
• It makes several data analysis operations such as reshape, slice and
dice, aggregation, and select subsets of data, easy
matplotlib
• The most popular and the most widely used Python library for
producing plots and other two-dimensional data visualizations is
matplotlib.
• It was originally created by John D.
• Now the library is maintained by a large team of developers.
• The library can create the plots that are suitable for publication.
IPython and Jupyter
• In 2001 Fernando Pérez’s started the IPython project.
• Today, it has become one of the most important tools in the modern
Python data stack.
• IPython does not provide any computational or data analytical tools.
• However, it helps maximize your productivity in both interactive
computing and software development.
• Since much of data analysis coding involves exploration, trial and
error, and iteration, IPython can help you get the job done faster.
scipy
• SciPy is a collection of packages for scientific computing. Important SciPy
packages include
• scipy.integrate: Numerical integration routines and differential equation
solvers
• scipy.linalg: Linear algebra routines and matrix decompositions
extending beyond those provided in numpy.linalg
• scipy.optimize: Function optimizers (minimizers) and root finding
algorithms
• scipy.stats: Standard continuous and discrete probability distributions
(density functions, samplers, continuous distribution functions), various
statistical tests, and more descriptive statistics
scikit learn
• Since the project’s inception in 2010, scikit-learn has become the premier
general purpose machine learning toolkit for Python programmers. In just
seven years, it has had over 1,500 contributors from around the world. It
includes submodules for such models as:
• Classification: SVM, nearest neighbors, random forest, logistic regression, etc.
• Regression: Lasso, ridge regression, etc.
• Clustering: k-means, spectral clustering, etc.
• Dimensionality reduction: PCA, feature selection, matrix factorization, etc.
• Model selection: Grid search, cross-validation, metrics
• Preprocessing: Feature extraction, normalization
• Along with pandas, statsmodels, and IPython, scikit-learn has been critical
for enabling
• Python to be a productive data science programming language.
statsmodel
• Compared with scikit-learn, statsmodels contains algorithms for classical
(primarily frequentist) statistics and econometrics. This includes such
submodules as:
• Regression models: Linear regression, generalized linear models, robust linear
models, linear mixed effects models, etc.
• Analysis of variance (ANOVA)
• Time series analysis: AR, ARMA, ARIMA, VAR, and other models
• Nonparametric methods: Kernel density estimation, kernel regression
• Visualization of statistical model results
• statsmodels is more focused on statistical inference, providing uncertainty
estimates and p-values for parameters. scikit-learn, by contrast, is more
prediction-focused.
 Thanks
Samatrix Consulting Pvt Ltd