Tutorials › Python › Handling Missing Values (Nulls)

Handling Missing Values (Nulls)

Detect, fill, and drop NaN values — essential for accurate analysis

Missing values (NaN — Not a Number) are in every real-world dataset. Pandas represents missing data as NaN for numeric columns and None or NaN for object columns. Handling them correctly prevents silent errors — a SUM with NaN returns NaN, an average with NaN skews results, and a merge with NaN drops rows unexpectedly. The three strategies for missing values: drop (remove rows/columns with nulls), fill (replace with a value), or flag (add a boolean column marking where nulls were).

Example

Detecting and handling missing values
import pandas as pd

df = pd.read_csv("sales.csv")

# DETECT
df.isnull()              # True/False for each cell
df.isnull().sum()        # count nulls per column
df.isnull().sum() / len(df) * 100  # % null per column
df.notnull()             # inverse — True where NOT null

# DROP ROWS
df.dropna()                        # drop rows with ANY null
df.dropna(subset=["order_id"])     # drop only if key column is null
df.dropna(thresh=5)                # keep rows with at least 5 non-null values

# FILL — replace nulls with a value
df["Region"].fillna("Unknown", inplace=True)    # text column
df["Sales"].fillna(0, inplace=True)             # numeric — fill with 0
df["Sales"].fillna(df["Sales"].mean())          # fill with mean
df["Sales"].fillna(method="ffill")              # forward fill (prev value)
df["Sales"].fillna(method="bfill")              # backward fill (next value)

# FLAG — mark where nulls were (then fill)
df["Has_Region"] = df["Region"].notnull().astype(int)
df["Region"].fillna("Unknown", inplace=True)

# After handling
print(df.isnull().sum())   # should show zeros
💡 Never blindly fill all nulls with 0 or mean — think about what makes sense for each column. Missing Sales might mean no sale (0). Missing Region might mean "Unknown". Missing Date is often a data quality issue worth investigating.

Key Points

  • ✓df.isnull().sum() is the first thing to check after loading data
  • ✓dropna() removes rows — be careful about how many rows you lose
  • ✓fillna(value) fills with a constant; fillna(df["col"].median()) fills with statistics
  • ✓Forward fill (ffill) is useful for time series — carry the last known value forward
  • ✓inplace=True modifies the DataFrame directly — without it, you get a new DataFrame back

Practice Question

A "City" column has 200 null values. You want to fill them with the string "Unknown". Which is correct?

Related Topics

Data Cleaning with PandasRename columns, fix data types, remove duplicates, and standardise messy real-world dataPandas DataFramesThe core Pandas data structure — a 2D table with rows and columnsExploratory Data Analysis (EDA) WorkflowA systematic 6-step EDA process every data analyst should follow on any new dataset