5/7/23, 8:15 AM Using pandas to perform data analysis - IBM Developer
IBM Developer
Round 2 of the Call for Code Global Challenge is now open! Register to grow your AI skills and win prizes.
Data science >
Analyze structured and unstructured data to extract knowledge and insights.
Site feedback
Using pandas to perform data analysis
Using pandas to perform data
analysis
Learn about pandas and how to use it to work with structured data
By Margriet Groenendijk, Yamini Rao, Samaya Madhavan
Published July 16, 2021
To save your progress, you
must be logged in to IBM
Developer
Log in now
Python overview
You might think that Python is only for developers and people with computer science degrees.
However, Python is great for beginners, even those with little coding experience because it's
Cookie Preferences
https://developer.ibm.com/learningpaths/get-started-data-science/data-analysis-in-python-using-pandas/ 1/15
5/7/23, 8:15 AM Using pandas to perform data analysis - IBM Developer
free, open source, and runs on any platform. The Python packages documentation is great,
and after an introductory course , you have a good foundation to build on.
Python is a general purpose and high-level programming language that is used for more than
working with data. For example, it's good for developing desktop GUI applications, websites,
and web applications. However, this tutorial focuses on the data and only goes through getting
started with data.
Unstructured versus structured data
Data is broadly classified into unstructured and structured data. Unstructured data refers to
data that is mostly free form with information that does not have consistent data types. Hand-
written notes by doctors and movie reviews collected from blogs are two examples of
unstructured data. On the other hand, structured data is information that is available in an
organized format so that it is easily readable. Examples of structured data are tables with
variables as columns and records as rows or key-value pairs in a noSQL database.
Introduction to pandas
pandas is an open source Python Library that provides high-performance data manipulation
and analysis. With the combination of Python and pandas, you can accomplish five typical
steps in the processing and analysis of data, regardless of the origin of data: load, prepare,
manipulate, model, and analyze.
There are many options when working with the data using pandas. The following list shows
some of the things that can be done using pandas.
Cleaning data by removing or replacing missing values
Converting data formats
Sorting rows
Deleting or adding rows and columns
Merging or joining DataFrames
Summarizing data by pivoting or reshaping
Cookie Preferences
https://developer.ibm.com/learningpaths/get-started-data-science/data-analysis-in-python-using-pandas/ 2/15
5/7/23, 8:15 AM Using pandas to perform data analysis - IBM Developer
Creating visualizations
This list is far from complete. See the pandas documentation for more of what you can do.
This tutorial walks you though some of the most interesting features of pandas using
structured data that contains information about the boroughs in London. You can download
the data used in the tutorial from data.gov.uk .
Getting started with Jupyter
Notebooks
Instead of writing code in a text file and then running the code with a Python command in the
terminal, you can do all of your data analysis in one place. Code, output, tables, and charts can
all be edited and viewed in one window in any web browser with Jupyter Notebooks . As the
name suggests, this is a notebook to keep all of your ideas and data explorations in one place.
In this tutorial, you use IBM Watson Studio to run a notebook. For this, you need a free IBM
Cloud account. The following steps show you how sign up and get started. When you have the
notebook up and running, you can go through the notebook.
Prerequisites
To complete this tutorial, you need:
An IBM Cloud
IBM Watson Studio
Steps
Set up Cookie Preferences
https://developer.ibm.com/learningpaths/get-started-data-science/data-analysis-in-python-using-pandas/ 3/15
5/7/23, 8:15 AM Using pandas to perform data analysis - IBM Developer
1. Sign up or log in to IBM Cloud.
2. Click Create Resource at the top of the Resources page. You'll see the resources under the
hamburger menu at the upper left.
3. Search for Watson Studio, and click that tile.
4. Select the Lite plan, and click Create.
5. Go back to the Resources list, click your Watson Studio service, and then click Get Started.
Cookie Preferences
https://developer.ibm.com/learningpaths/get-started-data-science/data-analysis-in-python-using-pandas/ 4/15
5/7/23, 8:15 AM Using pandas to perform data analysis - IBM Developer
You should now be in Watson Studio.
6. Click either Create a project or New project.
7. Select Create an empty project.
8. Give the project a name.
9. Choose an existing Object Storage service instance or create a new one. This is used to
store the notebooks and data. Note: Don't forget to click refresh when returning to the
Project page.
10. Click Create.
Load and run a notebook
1. Click Add to project, then click Notebook to add a new notebook.
2. Download the notebook .
3. Choose new notebook From File.
4. Select the downloaded notebook.
5. Select the default runtime.
6. Click Create Notebook. The notebook loads.
7. Run the notebook. In the open notebook, click Run to run the cells one at a time.
Notebook overview
Cookie Preferences
https://developer.ibm.com/learningpaths/get-started-data-science/data-analysis-in-python-using-pandas/ 5/15
5/7/23, 8:15 AM Using pandas to perform data analysis - IBM Developer
The following list shows some of the capabilities within pandas that are covered in this
tutorial.
Data exploration (loading data, Series, and DataFrames)
Data transformation (cleaning data, selecting data, merging data, and grouping data)
Data visualization
The notebook associated with this tutorial displays more elaborate functions of pandas.
Data exploration
Loading data
As long as the data is formatted consistently and has multiple records with numbers, text, or
dates, you can typically read the data with pandas. For example, a comma-separated text file
that is saved from a table in Excel can be loaded into a notebook with the following command.
import pandas as pd
df = pd.read_csv('data.csv')
Show more
You can load other formats of data files such as HTML, JSON, or Apache Parquet a similar
way.
Series and DataFrames
A Series is a one-dimensional labeled array that can contain data of any type (for example,
integer, string, float, or Python objects).
s = pd.Series([1, 3, 5, np.nan, 6, 8])
s
Show Preferences
Cookie more
https://developer.ibm.com/learningpaths/get-started-data-science/data-analysis-in-python-using-pandas/ 6/15
5/7/23, 8:15 AM Using pandas to perform data analysis - IBM Developer
A DataFrame is a two-dimensional data structure. The data consists of rows and columns that
you can create in various ways. For example, you can load a file or use a NumPy array and a
date for the index. NumPy is a Python library for working with multidimensional arrays and
matrices with a large collection of mathematical functions to operate on these arrays.
The following code is an example of a DataFrame df1 with dates as the index, a 6x4 array of
random numbers as values, and column names A, B, C, and D.
dates = pd.date_range('20200101', periods=6)
numbers = np.random.randn(6, 4)
df1 = pd.DataFrame(numbers, index=dates, columns=['A', 'B', 'C', 'D'])
df1
Show more
Running the previous code generates an output similar to the following image. The notebook
shows a few more ways of creating a DataFrame.
Selecting data
To select data, you access a single row or groups of rows and columns with labels using
.loc[]. (This only works for the column that was set to the index.) Or, you select data by a
position with .iloc[]. Cookie Preferences
https://developer.ibm.com/learningpaths/get-started-data-science/data-analysis-in-python-using-pandas/ 7/15
5/7/23, 8:15 AM Using pandas to perform data analysis - IBM Developer
boroughs = df.copy()
boroughs = boroughs.set_index(['Code'])
boroughs.loc['E09000001', 'Inland_Area_(Hectares)']
Show more
boroughs.iloc[0]
Show more
boroughs.iloc[:,1]
Show more
Cookie Preferences
https://developer.ibm.com/learningpaths/get-started-data-science/data-analysis-in-python-using-pandas/ 8/15
5/7/23, 8:15 AM Using pandas to perform data analysis - IBM Developer
Selecting rows based on a certain condition can be done with boolean indexing. This uses the
actual values of the data in the DataFrame as opposed to the row and column labels or index
positions. You can combine different columns using &, |, and == operators.
Data transformation
Cleaning data
When exploring data, there are always transformations needed to get it in the format that you
need for your analysis, visualization, or model. The best way to learn is to find a data set and
try to answer questions with the data. Some things to check when cleaning data are:
Is the data tidy, such as each variable forms a column, each observation forms a row, and
each type of observational unit forms a table?
Are all columns in the right data format?
Are there missing values?
Are there unrealistic outliers? Cookie Preferences
https://developer.ibm.com/learningpaths/get-started-data-science/data-analysis-in-python-using-pandas/ 9/15
5/7/23, 8:15 AM Using pandas to perform data analysis - IBM Developer
The following code shows how to add and drop a column that might not be required from the
data set using pandas.
boroughs['new'] = 1
boroughs.head()
Show more
boroughs = boroughs.drop(columns='new')
boroughs.head()
Show more
Merging data
pandas has several different options to combine or merge data. The documentation covers
these examples. In this notebook, you create two new DataFrames to explore how to merge
data. Then, you use append() to combine these DataFrames.
data = {'city': ['London','Manchester','Birmingham','Leeds','Glasgow'],
'population': [9787426, 2553379, 2440986, 1777934,1209143],
Cookie Preferences
https://developer.ibm.com/learningpaths/get-started-data-science/data-analysis-in-python-using-pandas/ 10/15
5/7/23, 8:15 AM Using pandas to perform data analysis - IBM Developer
'area': [1737.9, 630.3, 598.9, 487.8, 368.5 ]}
cities = pd.DataFrame(data)
data2 = {'city': ['Liverpool','Southampton'],
'population': [864122, 855569],
'area': [199.6, 192.0]}
cities2 = pd.DataFrame(data2)
cities = cities.append(cities2)
cities
Show more
Grouping data
Grouping data is a quick way to calculate values for classes in your DataFrame.
boroughs.groupby(['Inner/Outer']).mean()
Show more
Cookie Preferences
https://developer.ibm.com/learningpaths/get-started-data-science/data-analysis-in-python-using-pandas/ 11/15
5/7/23, 8:15 AM Using pandas to perform data analysis - IBM Developer
When you have multiple categorical variables, you can create a nested index.
boroughs.groupby(['Inner/Outer','Political control']).sum().head(8)
Show more
Data visualization
Visualization in pandas uses the Matplotlib library. This plotting library uses an object-
oriented API to embed plots into applications. Some of the examples are line plot, histograms,
scatter plot, image plot, and 3D plot. In this tutorial, we use matplotlib.pyplot, which is a
collection of command-style functions that make matplotlib work like MATLAB.
The following example shows a visualization of the employment rate through a histogram. You
can change the number of bins to get the wanted output on your histogram.
%matplotlib inline
import matplotlib.pyplot as plt
Cookie Preferences
https://developer.ibm.com/learningpaths/get-started-data-science/data-analysis-in-python-using-pandas/ 12/15
5/7/23, 8:15 AM Using pandas to perform data analysis - IBM Developer
boroughs = boroughs.reset_index()
boroughs['Employment_rate_(%)_(2015)'].plot.hist(bins=10);
Show more
Conclusion
This tutorial walked you through the steps to get IBM Cloud, Watson Studio, and a Jupyter
Notebook installed. It gave you an overview of the ways of analyzing data using pandas and a
notebook that you can run to try it yourself.
Legend
Categories
Analytics Data science Python
Cookie Preferences
https://developer.ibm.com/learningpaths/get-started-data-science/data-analysis-in-python-using-pandas/ 13/15
5/7/23, 8:15 AM Using pandas to perform data analysis - IBM Developer
Table of Contents
Previous
Data preprocessing in detail
Next
Explore other data science topics
Related
Learning Path
Get started with machine learning
Tutorial
Object-oriented programming in Python
Cookie Preferences
https://developer.ibm.com/learningpaths/get-started-data-science/data-analysis-in-python-using-pandas/ 14/15
5/7/23, 8:15 AM Using pandas to perform data analysis - IBM Developer
IBM Developer Follow Us
About Twitter
FAQ LinkedIn
Build
Third-party notice YouTube
Smart Explore
Build
Newsletters
Patterns
Secure APIs
Articles
Tutorials
Open source projects
Videos
Events
Community
Career Opportunities
Privacy
Terms of use
Accessibility
Cookie preferences
Sitemap
Cookie Preferences
https://developer.ibm.com/learningpaths/get-started-data-science/data-analysis-in-python-using-pandas/ 15/15
