Hypothesis Testing for Data Analysts

from scipy import stats
import pandas as pd

df = pd.read_csv('india_orders.csv')

# Business question: Do returning customers spend more than new customers?
# H₀: Mean order value for new and returning customers is the same
# H₁: Mean order value differs between new and returning customers

new_customers      = df[df['customer_type'] == 'new']['amount_inr']
returning_customers = df[df['customer_type'] == 'returning']['amount_inr']

# Print descriptive stats first — always check the raw numbers
print("=== Descriptive Stats ===")
print(f"New customers:       n={len(new_customers):,}  mean=₹{new_customers.mean():.0f}  median=₹{new_customers.median():.0f}")
print(f"Returning customers: n={len(returning_customers):,}  mean=₹{returning_customers.mean():.0f}  median=₹{returning_customers.median():.0f}")
print(f"Difference in means: ₹{returning_customers.mean() - new_customers.mean():.0f}")

# Run independent samples t-test
t_stat, p_value = stats.ttest_ind(new_customers, returning_customers, equal_var=False)
# equal_var=False → Welch's t-test (safer; does not assume equal variance in both groups)

print("\n=== t-Test Results ===")
print(f"t-statistic: {t_stat:.3f}")
print(f"p-value:     {p_value:.6f}")

alpha = 0.05
if p_value < alpha:
    print(f"\nResult: SIGNIFICANT (p={p_value:.4f} < α={alpha})")
    print("Reject H₀: returning customers have a statistically different order value")
else:
    print(f"\nResult: NOT SIGNIFICANT (p={p_value:.4f} ≥ α={alpha})")
    print("Fail to reject H₀: insufficient evidence of a difference")

# === PAIRED t-TEST (when the same customers appear in both conditions) ===
# Example: order values before vs after loyalty programme launch
before = df[df['period'] == 'pre_loyalty']['amount_inr']
after  = df[df['period'] == 'post_loyalty']['amount_inr']

t_paired, p_paired = stats.ttest_rel(before, after)
print(f"\nPaired t-test (pre vs post loyalty): t={t_paired:.3f}, p={p_paired:.4f}")

from scipy.stats import chi2_contingency
import pandas as pd

# Business question: Is return rate associated with product category?
# H₀: Return rate is the same across all product categories
# H₁: Return rate differs significantly across categories

# Build the contingency table (observed counts)
df['is_returned'] = (df['status'] == 'returned').astype(int)
contingency = pd.crosstab(df['category'], df['is_returned'],
                           margins=True)
print("Contingency Table (counts):")
print(contingency)

# Run chi-square test
chi2, p_value, dof, expected = chi2_contingency(
    pd.crosstab(df['category'], df['is_returned'])
)

print(f"\nChi-square statistic: {chi2:.2f}")
print(f"Degrees of freedom:   {dof}")
print(f"p-value:              {p_value:.6f}")

if p_value < 0.05:
    print("\nResult: SIGNIFICANT — return rate is NOT equal across categories")
else:
    print("\nResult: NOT SIGNIFICANT — no evidence of category-level return differences")

# Which categories have highest return rate?
return_by_cat = (df.groupby('category')['is_returned']
                   .agg(['sum', 'count'])
                   .assign(return_pct=lambda x: (x['sum'] / x['count'] * 100).round(1))
                   .sort_values('return_pct', ascending=False))
print("\nReturn rate by category:")
print(return_by_cat.rename(columns={'sum': 'returns', 'count': 'total'}))

from scipy.stats import norm
import numpy as np

# === SAMPLE SIZE CALCULATION (before running the test) ===
def required_sample_size(baseline_rate, mde, alpha=0.05, power=0.80):
    """MDE = minimum detectable effect (absolute), e.g. 0.02 = detect 2pp lift"""
    z_alpha = norm.ppf(1 - alpha / 2)  # 1.96 for 5% two-tailed
    z_beta  = norm.ppf(power)          # 0.84 for 80% power
    p1 = baseline_rate
    p2 = baseline_rate + mde
    p_pool = (p1 + p2) / 2
    n = (z_alpha * np.sqrt(2 * p_pool * (1 - p_pool)) +
         z_beta  * np.sqrt(p1 * (1 - p1) + p2 * (1 - p2))) ** 2 / (mde ** 2)
    return int(np.ceil(n))

baseline   = 0.032   # current conversion rate = 3.2%
mde        = 0.005   # we want to detect a 0.5 percentage point lift → 3.7%
n_per_arm  = required_sample_size(baseline, mde)
print(f"Required sample size per arm: {n_per_arm:,} users")
print(f"Total experiment size: {2 * n_per_arm:,} users")

# === AFTER THE TEST — ANALYSE RESULTS ===
control_visitors  = 48_320;  control_orders  = 1_544   # A: old checkout
variant_visitors  = 47_981;  variant_orders  = 1_703   # B: new checkout

control_cr = control_orders / control_visitors
variant_cr = variant_orders / variant_visitors
lift       = variant_cr - control_cr

print(f"\n=== A/B Test Results ===")
print(f"Control CVR:  {control_cr:.3%}")
print(f"Variant CVR:  {variant_cr:.3%}")
print(f"Absolute lift: {lift:.3%}")
print(f"Relative lift: {lift / control_cr:.1%}")

# Two-proportion z-test
p_pool = (control_orders + variant_orders) / (control_visitors + variant_visitors)
se     = np.sqrt(p_pool * (1 - p_pool) * (1/control_visitors + 1/variant_visitors))
z      = lift / se
p_val  = 2 * (1 - norm.cdf(abs(z)))  # two-tailed

# 95% confidence interval for the lift
ci_lo = lift - 1.96 * se
ci_hi = lift + 1.96 * se

print(f"\nz-statistic:   {z:.3f}")
print(f"p-value:        {p_val:.4f}")
print(f"95% CI:        [{ci_lo:.3%}, {ci_hi:.3%}]")

if p_val < 0.05 and ci_lo > 0:
    print("\n✅ SHIP IT: statistically significant positive lift. Entire 95% CI is above 0.")
elif p_val < 0.05 and lift < 0:
    print("\n❌ DO NOT SHIP: statistically significant NEGATIVE lift.")
else:
    print("\n⚠️  INCONCLUSIVE: insufficient evidence. Extend run or reconsider MDE.")

What is a p-value and what does p < 0.05 mean?

A p-value is the probability of observing results as extreme as (or more extreme than) your data, assuming the null hypothesis is true. If p = 0.03, it means: if there were truly no effect, there is only a 3% probability of seeing a difference this large or larger by random chance alone. The threshold p < 0.05 (called significance level or alpha) is a conventional boundary — if p falls below it, you "reject the null hypothesis" and conclude the effect is statistically significant. This does NOT mean: (1) a 95% chance that the alternative hypothesis is true; (2) the effect is practically important (a statistically significant difference may be too small to matter in business); or (3) the experiment proves causation. In A/B testing for an Indian e-commerce site, p < 0.05 means the observed conversion rate difference between control and variant is unlikely to be due to random fluctuation — but you still need to check whether the absolute difference (say 0.2%) justifies the engineering effort.

When should a data analyst use a t-test vs chi-square test?

Use a t-test when comparing means of a continuous variable between two groups — e.g. is average order value different for new vs returning customers? Is delivery time lower in Tier-1 vs Tier-2 cities? The variable being tested must be numeric (order value, delivery days, rating). Use chi-square test when testing whether a categorical outcome is associated with a categorical grouping — e.g. does return rate (returned vs not returned) differ across product categories? Is payment method (UPI vs card vs COD) associated with customer city tier? The variable being tested is a count or proportion split across categories. A quick rule: if you are comparing a number between groups → t-test. If you are comparing proportions or counts across categories → chi-square test.

What is statistical power and why does it matter for A/B tests?

Statistical power is the probability of correctly detecting a real effect when one exists — i.e. the probability that your test will produce p < 0.05 when the alternative hypothesis is actually true. Low-power tests frequently produce false negatives: the new variant really is better, but your test concludes no significant difference because the sample size was too small. Standard practice is to design tests with 80% power — meaning if a true effect exists, you have an 80% chance of detecting it. Power depends on: effect size (bigger differences are easier to detect), sample size (more users → more power), and significance level (stricter threshold → lower power). In India, where experiment traffic is concentrated in certain periods (festive season, salary dates), analysts sometimes under-power tests by running them during off-peak periods. Always calculate the required sample size before running an A/B test — running a test and stopping when it looks significant is a common but incorrect practice known as p-hacking.

What are the most common mistakes in hypothesis testing?

The five most common mistakes: (1) Stopping the test early once significance is reached — this is p-hacking and inflates false positives. Run the full planned duration. (2) Treating absence of significance as proof of no effect — a non-significant result means "not enough evidence," not "no effect exists." (3) Interpreting p < 0.05 as the effect is important — significance is not the same as practical significance. A 0.1% conversion rate improvement may be statistically significant with enough users but irrelevant for the business. (4) Running multiple comparisons without adjustment — if you test 20 variants, one will be significant by chance alone. Use Bonferroni correction or FDR adjustment. (5) Violating test assumptions — t-tests assume approximately normal distribution or large enough sample; using them on very small samples with skewed data produces unreliable results.