Data Analysis Using Python: A Beginner’s Guide Featuring NYC Open Data

Part 2: Data Inspection, Cleaning and Wrangling in Python

Introduction
Part 2 demonstrates various ways to to inspect, clean, wrangle, and detect any outliers in your data. The project is located in my notebook 2-data-cleaning-wrangling.ipynb.

Key sections as follows:

3. Data Inspection

3.1 Previewing Data
3.2 Calcualting Summary Statistics
3.3 Identifying Null/NA Values

4. Data Cleaning/Wrangling

4.1 Previewing Column Values
4.2 Filling In Null/NA Values With Median
4.3 Plotting Histogram of Raw Construction Year Values
4.4 Identiyfying Values Not Between Years 1 and 2020
4.5 Replacing Values Not Between Years 1 and 2020 with the Median
4.6 Plotting Histogram of Construction Years Between 1 and 2020
4.7 Identifying Buildings Built Before 1652 - Possible Outliers
4.8 Replacing Values Less Than 1652 with the Median
4.9 Plotting Histogram of Construction Years with Cleaned Column

5. Detecting Outliers

5.1. Making a Box Plot
- 5.1.1 Calculating Interquartile Range (IQR)
- 5.1.2 Box Plot of Construction Year Values
- 5.1.3 Making Histograms of Values Above and Below Box Plot Whiskers
5.2 Z-Scores
- 5.2.1 Calculating Z-Scores
- 5.2.2 Identifying Possible Outliers from Z-Score
- 5.2.3 Making Histograms of Construction Year Raw and Normalized (Z-Score) Values

Feedback is very welcome!

Mark Bauer