Tutorials › Python › Outlier Detection in Python

Outlier Detection in Python

Identify and handle outliers using IQR, z-score and visualisation methods

Outliers are data points that are significantly different from the rest — they can represent genuine extreme events (a record-breaking sale) or data quality issues (a data entry error). Detecting them before analysis prevents skewed averages and incorrect conclusions. Two standard statistical methods: IQR (Interquartile Range) method and Z-score method. Both should be used alongside visualisation (boxplots, histograms) for context.

Example

IQR method and z-score method
import pandas as pd
import numpy as np

sales = df["Sales"]

# METHOD 1 — IQR (Interquartile Range)
Q1 = sales.quantile(0.25)
Q3 = sales.quantile(0.75)
IQR = Q3 - Q1

lower = Q1 - 1.5 * IQR
upper = Q3 + 1.5 * IQR

outliers_iqr = df[(df["Sales"] < lower) | (df["Sales"] > upper)]
clean_iqr    = df[(df["Sales"] >= lower) & (df["Sales"] <= upper)]

print(f"Outliers: {len(outliers_iqr)} rows ({len(outliers_iqr)/len(df)*100:.1f}%)")
print(f"Lower bound: {lower:,.0f}, Upper: {upper:,.0f}")

# METHOD 2 — Z-score (how many std deviations from mean)
df["zscore"] = (df["Sales"] - df["Sales"].mean()) / df["Sales"].std()
outliers_z = df[df["zscore"].abs() > 3]   # beyond 3 std devs

# VISUALISE — boxplot shows IQR and outliers
import matplotlib.pyplot as plt
plt.boxplot(sales.dropna())
plt.title("Sales Distribution — Boxplot")
plt.show()

# HANDLE outliers
# Option 1: Remove them
df_clean = df[df["zscore"].abs() <= 3]

# Option 2: Cap (winsorise)
df["Sales_capped"] = df["Sales"].clip(lower=lower, upper=upper)
💡 IQR method is more robust for skewed data. Z-score assumes normal distribution. Always investigate outliers before removing — they might be real and important.

Key Points

  • ✓IQR method: outliers are below Q1 − 1.5×IQR or above Q3 + 1.5×IQR
  • ✓Z-score > 3 (or < −3) usually indicates an outlier in normally distributed data
  • ✓Investigate before removing — an outlier might be a legitimate high-value transaction
  • ✓Capping (clip) is safer than dropping — you keep the row but limit the extreme value
  • ✓Boxplot visualises the IQR: the box is Q1 to Q3, the whiskers extend to 1.5×IQR

Practice Question

In the IQR method for outlier detection, the upper fence is defined as:

Related Topics

Descriptive Statistics in PythonCalculate mean, median, mode, variance, standard deviation and percentilesSeaborn for Statistical ChartsCreate beautiful distribution, correlation and categorical charts with SeabornData Cleaning with PandasRename columns, fix data types, remove duplicates, and standardise messy real-world data