A/B Testing for Data Analysts — Complete Guide India

import numpy as np
from scipy.stats import norm

def sample_size(baseline, mde_abs, alpha=0.05, power=0.80):
    """Required sample size per arm for a two-tailed test."""
    za = norm.ppf(1 - alpha / 2)
    zb = norm.ppf(power)
    p1, p2 = baseline, baseline + mde_abs
    pp = (p1 + p2) / 2
    n = (za * np.sqrt(2 * pp * (1 - pp)) +
         zb * np.sqrt(p1*(1-p1) + p2*(1-p2)))**2 / mde_abs**2
    return int(np.ceil(n))

def days_needed(n_per_arm, daily_sessions, split=0.5):
    """Days to reach required sample at given traffic and split."""
    return int(np.ceil(n_per_arm / (daily_sessions * split)))

# ── Indian e-commerce checkout experiment ──────────────────
baseline_cvr   = 0.032   # current checkout CVR = 3.2 %
daily_sessions = 8_500   # daily unique sessions on checkout page

print("MDE vs Sample Size vs Days Needed (baseline CVR = 3.2 %, 8,500 sessions/day)")
print(f"{'MDE (pp)':>10}  {'New CVR':>10}  {'n per arm':>12}  {'Total n':>10}  {'Days':>6}")
print("-" * 58)
for mde in [0.005, 0.008, 0.010, 0.015, 0.020]:
    n = sample_size(baseline_cvr, mde)
    d = days_needed(n, daily_sessions)
    print(f"{mde*100:>9.1f}%  {(baseline_cvr+mde)*100:>9.1f}%  {n:>12,}  {2*n:>10,}  {d:>6}")

# Output:
#      MDE (pp)     New CVR     n per arm     Total n    Days
#          0.5%       3.7%        34,185      68,370       5
#          0.8%       4.0%        13,597      27,194       2
#          1.0%       4.2%         8,797      17,594       2
#          1.5%       4.7%         3,969       7,938       1
#          2.0%       5.2%         2,264       4,528       1

# LESSON: Detecting a 0.5pp lift requires 68,370 users — 8 days at this traffic.
# Detecting a 2pp lift only needs 4,528 users — done in < 1 day.
# Always ask: what is the MINIMUM lift that would change a business decision?
# That is your MDE. Don't design to detect 0.1pp when 1pp is the business threshold.

from scipy.stats import chi2_contingency, norm
import numpy as np
import pandas as pd

# ── STEP 1: Check for Sample Ratio Mismatch ───────────────
control_n   = 34_120
variant_n   = 33_887
expected_split = 0.5  # intended 50/50

total_n   = control_n + variant_n
expected_c = total_n * expected_split
expected_v = total_n * (1 - expected_split)

chi2_srm, p_srm = chi2_contingency(
    [[control_n, variant_n], [expected_c, expected_v]]
)[:2]

print(f"SRM Check: control={control_n:,}  variant={variant_n:,}")
print(f"Actual split: {control_n/total_n:.1%} / {variant_n/total_n:.1%}")
print(f"Chi-square p-value: {p_srm:.4f}")
if p_srm < 0.01:
    print("⚠️  SRM DETECTED — do not proceed with analysis")
    print("   Investigate: logging bug? Bot traffic? Redirect issue?")
else:
    print("✅ No SRM detected — split looks clean")

# ── STEP 2: Metric Analysis (run only if no SRM) ──────────
control_orders  = 1_093   # conversions in control arm
variant_orders  = 1_224   # conversions in variant arm

c_cvr = control_orders / control_n
v_cvr = variant_orders  / variant_n
lift  = v_cvr - c_cvr

# Two-proportion z-test
p_pool = (control_orders + variant_orders) / total_n
se     = np.sqrt(p_pool * (1 - p_pool) * (1/control_n + 1/variant_n))
z_stat = lift / se
p_val  = 2 * (1 - norm.cdf(abs(z_stat)))

ci_lo  = lift - 1.96 * se
ci_hi  = lift + 1.96 * se

print(f"\n=== Primary Metric: Checkout CVR ===")
print(f"Control: {c_cvr:.3%}  ({control_orders:,} / {control_n:,})")
print(f"Variant: {v_cvr:.3%}  ({variant_orders:,} / {variant_n:,})")
print(f"Absolute lift: {lift:+.3%}")
print(f"Relative lift: {lift/c_cvr:+.1%}")
print(f"95% CI: [{ci_lo:+.3%}, {ci_hi:+.3%}]")
print(f"p-value: {p_val:.4f}")

# ── STEP 3: Business impact translation ───────────────────
monthly_sessions = 8_500 * 30
avg_order_value  = 2_340   # ₹ AOV

monthly_extra_orders  = monthly_sessions * lift
monthly_extra_revenue = monthly_extra_orders * avg_order_value

print(f"\n=== Business Impact ===")
print(f"Monthly extra orders:  {monthly_extra_orders:,.0f}")
print(f"Monthly extra revenue: ₹{monthly_extra_revenue/1e5:.1f} lakh")
print(f"Annual extra revenue:  ₹{monthly_extra_revenue*12/1e7:.2f} crore")

# ── STEP 4: Segment breakdown ─────────────────────────────
# Always segment after aggregate analysis
segments = {
    'Mobile': {'c_n': 22_000, 'c_conv': 650, 'v_n': 21_900, 'v_conv': 780},
    'Desktop': {'c_n': 12_120, 'c_conv': 443, 'v_n': 11_987, 'v_conv': 444},
}
print("\n=== Segment Breakdown ===")
for seg, d in segments.items():
    c_r = d['c_conv'] / d['c_n']
    v_r = d['v_conv'] / d['v_n']
    print(f"  {seg:10s}: control={c_r:.2%}  variant={v_r:.2%}  Δ={v_r-c_r:+.2%}")

How long should an A/B test run in India?

An A/B test should run for at least 1–2 full business cycles — typically a minimum of 7 days and ideally 14 days. This captures weekday/weekend variation, which is significant in Indian consumer markets (weekend orders are typically 30–40% higher than weekdays). For monthly salary-driven spikes (1st–5th of the month), a 3-week run ensures you do not over-sample or under-sample salary-date traffic. Never stop a test the moment it becomes significant — run for the full planned duration. Two additional Indian-specific rules: (1) Do not run tests over Diwali, Navratri, or Holi unless you are specifically testing festive-season behaviour — these periods have abnormal traffic composition and the results will not generalise. (2) Be cautious running tests during IPL months (April–May) for sports or entertainment apps — traffic profile changes significantly. Plan experiment calendars around these events.

What is novelty effect and how does it affect A/B test results in India?

Novelty effect occurs when users in the variant group engage more with a new feature simply because it is new — not because it is better. This inflates variant performance in the first few days, gradually returning to the true steady-state effect as users habituate. This is particularly common in Indian mobile apps where users are highly engaged and tend to explore new UI elements. To mitigate: (1) Run tests for at least 2 weeks so the novelty wears off; (2) Analyse day-by-day metric trends — a genuine improvement maintains its lift over time, while novelty shows a declining trend; (3) If you have a returning user segment, compare the novelty-prone "first-week" cohort separately from "returning users who saw the variant multiple times" — the latter gives a more reliable signal. The opposite problem — change aversion — occurs when users initially resist a change but adopt it over time. Both effects mean short tests mislead.

What is the difference between A/B testing and multivariate testing?

A/B testing compares two versions of a single element: the control (A) vs one variant (B). It is simple, requires less traffic, and produces a clear go/no-go decision. Multivariate testing (MVT) simultaneously tests multiple elements and their combinations — e.g. headline colour (red vs blue) × CTA text ("Buy Now" vs "Add to Cart" vs "Shop Now") × banner image (3 options) = 18 combinations. MVT identifies interaction effects between elements (CTA text works better with a specific banner) but requires much more traffic because each combination needs statistical power. For most Indian startups and mid-size companies, A/B testing is the right tool — traffic is not high enough to power multivariate experiments. MVT is practical only at very high traffic volumes (millions of daily sessions). A sequential A/B approach — test headline first, then CTA on the winner — achieves similar learning with feasible sample sizes.

How do data analysts communicate A/B test results to non-technical stakeholders?

Lead with the business outcome, not the statistic. Instead of "p = 0.03, significant at α = 0.05," say: "The new checkout page increased order completion by 0.8 percentage points — from 3.2% to 4.0%. Over a month, this translates to approximately 2,400 additional orders and ₹34 lakh extra revenue." Structure every test readout as: (1) What we tested and why; (2) What we measured (primary metric); (3) What we found — absolute lift, relative lift, confidence interval; (4) Whether this is statistically reliable; (5) Business impact in INR or unit terms; (6) Recommendation with caveats. Always show confidence intervals rather than just point estimates — saying "the true lift is likely between 0.3% and 1.3%" is more honest than reporting "0.8% lift" as if it were exact. For inconclusive results, frame it as a decision about sample size or MDE, not as "the test failed." Quantify the cost of being wrong: if we ship this based on 80% confidence and it is a false positive, what does that cost?