Key Takeaways: Data Wrangling — Cleaning and Preparing Real Data

One-Sentence Summary

Data cleaning is the unglamorous foundation of all analysis — handling missing values, fixing inconsistencies, removing duplicates, transforming variables, and documenting every decision so your results are transparent and reproducible.

Core Concepts at a Glance

Decision Guide: Handling Missing Data

Missing data detected in a column
│
├── How much is missing?
│   ├── < 5% → Simple methods usually fine
│   ├── 5-20% → Think carefully about mechanism
│   └── > 20% → Consider dropping the column or using advanced methods
│
├── What TYPE of variable?
│   ├── Numerical + symmetric distribution → Mean imputation
│   ├── Numerical + skewed distribution → Median imputation
│   └── Categorical → Mode imputation
│
├── What's the MECHANISM?
│   ├── MCAR → Deletion or imputation both OK
│   ├── MAR → Group-based imputation preferred
│   └── MNAR → No simple fix; document limitation
│
└── ALWAYS: Create a missing-data flag + document your decision

Missing Data Mechanisms

Imputation Methods

Data Quality Checklist

Use this checklist every time you start a new data analysis:

• [ ] Run .info() — Check data types and non-null counts
• [ ] Run .describe() — Look for impossible min/max, suspicious means, large standard deviations
• [ ] Run .value_counts() on categorical columns — Check for inconsistent categories
• [ ] Check for duplicates — df.duplicated().sum()
• [ ] Count missing values — df.isna().sum() and percentages
• [ ] Check for placeholder values — 999, -1, -99, 0 in places where they shouldn't be
• [ ] Verify date columns are datetime type, not strings
• [ ] Check zip codes, phone numbers, IDs are strings, not numbers

The Data Cleaning Pipeline

Follow this order for every project:

1. ASSESS    → .info(), .describe(), .value_counts(), .duplicated()
      ↓
2. DEDUPLICATE → .drop_duplicates()
      ↓
3. FIX TYPES  → pd.to_datetime(), .astype()
      ↓
4. STANDARDIZE TEXT → .str.lower(), .str.strip(), .replace()
      ↓
5. FIX IMPOSSIBLE VALUES → Set out-of-range values to NaN
      ↓
6. HANDLE MISSING DATA → Flag first, then impute or drop
      ↓
7. CREATE NEW VARIABLES → Binning, calculations, recoding
      ↓
8. VERIFY    → Final .info(), .describe(), .isna().sum()
      ↓
9. DOCUMENT  → Write the cleaning log
      ↓
10. SAVE     → .to_csv('cleaned_data.csv')

Tidy Data Rules

Python Quick Reference

import pandas as pd
import numpy as np

# --- Assessment ---
df.info()                           # Types, non-null counts
df.describe()                       # Summary stats (spot impossibles)
df.isna().sum()                     # Missing count per column
df.isna().mean() * 100              # Missing percentage per column
df.duplicated().sum()               # Count duplicate rows
df['col'].value_counts()            # Category frequencies

# --- Cleaning ---
df.drop_duplicates()                # Remove duplicate rows
df.dropna()                         # Drop rows with any NaN
df.dropna(subset=['col'])           # Drop rows where 'col' is NaN
df['col'].fillna(value)             # Fill NaN with value
df['col'].fillna(df['col'].median()) # Fill with median
df['col'].str.lower().str.strip()   # Standardize text
df['col'].replace({'old': 'new'})   # Map values

# --- Transformation ---
pd.cut(df['col'], bins=..., labels=...)   # Equal-width bins
pd.qcut(df['col'], q=4, labels=...)       # Equal-frequency bins
pd.to_datetime(df['col'])                  # Convert to datetime
df.loc[condition, 'col'] = new_value       # Conditional assignment

# --- Reshaping ---
pd.melt(df, id_vars=..., value_vars=...)   # Wide → long
df.pivot(index=..., columns=..., values=...) # Long → wide

# REMEMBER: Most methods return a NEW object.
# Always assign the result back: df['col'] = df['col'].fillna(0)

Excel Quick Reference

Key Terms

Common Pandas Pitfalls

The One Thing to Remember

If you forget everything else from this chapter, remember this:

Every data cleaning decision is a judgment call that shapes your conclusions. There is no "neutral" cleaning — every choice to delete, impute, recode, or transform changes the data and the story it tells. The ethical move is to make your choices transparently, document them in a cleaning log, and test whether different choices would lead to different conclusions. Reproducibility isn't a luxury — it's the price of being trustworthy.