Table of Contents

Module 5: A/B Testing & Experimentation

1. Designing a Valid Experiment — Control/Treatment, Sample Size, Randomization

The core design skeleton:

Component	What it means	Common mistake
Control	The existing experience (baseline)	Letting control quietly change mid-experiment (a “code freeze” violation) — invalidates comparison
Treatment	The new experience being tested	Bundling multiple changes into one treatment (“redesign the whole checkout”) — can’t attribute the lift to a specific change afterward
Randomization unit	What’s randomly assigned — user, session, device	Picking the wrong unit relative to the metric — e.g., randomizing by session when measuring user-level retention contaminates the analysis because the same user can land in both arms across sessions
Sample size	How many units you need before you can trust the result	Stopping early because the result “looks significant” (this is the peeking problem — covered in section 3)

Randomization — why it’s the load-bearing wall of causal inference: random assignment is what breaks the link between treatment and any confounder, which is exactly what makes an A/B test stronger evidence than the observational correlation analysis from the prior module. Without true randomization, you’re back to “correlation vs. causation” problems — e.g., letting users self-select into treatment (an opt-in beta) reintroduces selection bias.

Practical design checks senior analysts apply before launching:

SRM (Sample Ratio Mismatch) check: if you intended a 50/50 split and you’re observing 52/48 at meaningful scale, something is broken in the randomization or instrumentation before you even look at the outcome metric — this is a standard pre-flight sanity check, not optional.
A/A test: running two identical experiences against each other before trusting a new experimentation pipeline — if an A/A test shows a “significant” difference, your infrastructure (not your hypothesis) is broken.
Pre-registration of the primary metric: deciding before launch which single metric determines success, to avoid the multiple-comparisons trap of mining for some metric that moved (section 3).

Talking point: “Before I trust any experiment result, I check for sample ratio mismatch — if the actual traffic split doesn’t match the intended split, that’s usually evidence of a bug in assignment or logging, not a real effect. I’ve seen teams celebrate a ‘lift’ that was actually an instrumentation bug that happened to skew which users got logged into which bucket.”

2. Statistical Power, MDE (Minimum Detectable Effect)

Statistical power = 1 − β = probability of detecting a real effect, given that one exists. This is the practical, forward-looking question: “if the treatment really does improve conversion by 2%, how likely is my test to actually catch that?”

The four interlocking variables — and the genuinely important interview point is that you can only freely choose 3 of them, the 4th is determined:

Sample size  ↔  Statistical power  ↔  MDE (effect size)  ↔  Significance level (α)

Larger sample size → can detect smaller effects (smaller MDE) at the same power, or achieve higher power for the same effect.
Smaller MDE (you want to detect a tinier lift) → requires a much larger sample — and the relationship isn’t linear: halving the MDE roughly quadruples the required sample size, because sample size scales with 1/MDE². This is one of the most commonly tested quantitative facts in experimentation interviews.

MDE — the practical business framing: “What’s the smallest effect size that would actually be worth shipping, given the cost of building and maintaining this change?” This is a business question disguised as a stats question — a 0.1% conversion lift might be statistically detectable with enough traffic, but not worth the engineering cost to maintain. Senior analysts set MDE based on business materiality first, then calculate the required sample size — not the other way around.

Practical power calculation walkthrough (the kind of question asked directly in interviews):

Given:
  Baseline conversion rate: 10%
  MDE: want to detect a 1 percentage point lift (10% → 11%)
  α = 0.05, Power = 0.80 (conventional defaults)

→ Plug into a sample size formula (or use a calculator/library):
  n ≈ 16 × p(1-p) / MDE²     (rough two-proportion approximation)
  n ≈ 16 × 0.10 × 0.90 / 0.01²  ≈ 14,400 per arm

Talking point on why this matters before launch, not after: “If a stakeholder wants to detect a 0.1 percentage point lift on a metric that only gets 5,000 users a week, I’d tell them upfront that the experiment isn’t viable at that traffic level within a reasonable timeframe — rather than letting them run an underpowered test for a month, see a non-significant result, and wrongly conclude ‘the change didn’t work’ when really the test just never had the power to detect it.” This is a high-value point — it shows you prevent a very common real-world failure (mistaking “failed to detect” for “no effect”).

3. Common Pitfalls — Peeking, Novelty Effect, Simpson’s Paradox, Multiple Comparisons

Peeking (optional stopping) — checking results repeatedly and stopping as soon as you see significance:

Each “peek” at the data is effectively another chance for a false positive — checking daily over a 4-week test can inflate your true false-positive rate well above the nominal 5% α, even though each individual check “looks like” a standard 0.05 test.
Fix: commit to a pre-determined sample size/duration upfront (fixed-horizon testing), or explicitly use sequential testing methods (e.g., sequential probability ratio tests, or platforms with built-in “always-valid” p-values) that are designed to allow continuous monitoring without inflating error rates.

Talking point: “Peeking is dangerous precisely because it doesn’t feel like cheating — you’re just ‘checking in’ on a live experiment. But statistically, repeatedly testing the same data is equivalent to running many simultaneous hypothesis tests, which is the same multiple-comparisons problem, just spread across time instead of across metrics.” (Good explicit callback connecting this pitfall to the next one.)

Novelty effect (and its mirror, primacy effect):

Users react to a change because it’s new, not because it’s better — engagement spikes initially, then decays back toward baseline as the novelty wears off. The inverse — primacy effect — is when long-time users are temporarily worse with a new UI simply because they’re used to the old one, and recover with practice.
Fix: run the test long enough to observe post-novelty behavior (commonly 2+ weeks, depending on usage cycle), and/or segment results by new vs. returning users to separate “genuinely better” from “just different.”

Simpson’s Paradox — a trend that appears in aggregate reverses when the data is segmented, because a confounding variable is unevenly distributed across segments:

Classic real example: Treatment B looks worse than Treatment A overall, but when segmented by user type (new vs. returning), Treatment B is actually better within every single segment — because Treatment B happened to get a higher proportion of harder-to-convert new users in the overall mix, which dragged down its aggregate average.

Talking point: “This is exactly why I never trust an aggregate experiment result without checking it holds across the major known segments — device type, new vs. returning, geography. If a result flips sign when segmented, the segment mix itself — not the treatment — is driving the aggregate number, and you need to report the segment-level truth, not the misleading topline.”

Multiple comparisons problem — already foreshadowed in the Statistics module:

Testing many metrics (or many segments, or many variants) at α = 0.05 each means the overall probability of at least one false positive climbs fast — testing 20 independent metrics at α=0.05 gives roughly a 64% chance of at least one false “significant” result purely by chance (1 - 0.95²⁰).
Fixes: pre-register a single primary metric (the cleanest practical solution), or apply corrections — Bonferroni (conservative, divides α by number of comparisons) or False Discovery Rate / Benjamini-Hochberg (less conservative, more common in practice when testing many metrics simultaneously, e.g., in large-scale experimentation platforms).

Talking point tying all four pitfalls together: “Three of these four pitfalls — peeking, multiple comparisons, and to some extent novelty effects if you check too early — are really the same root issue: looking at the data more ways than your statistical framework accounts for. The fix in every case is the same discipline: decide your primary metric, sample size, and stopping rule before you look at any results, not after.”

4. Interpreting and Communicating Results to Non-Technical Stakeholders

This is the section that tests the Communication pillar from your orientation module’s competency map directly — and it’s frequently underweighted by technically strong candidates.

The translation hierarchy — lead with the decision, not the method:

What NOT to lead with	What to lead with instead
“We ran a two-proportion z-test with α=0.05…”	“The new checkout flow increased conversion by 2.3%, and we’re confident enough in this to roll it out.”
“The p-value was 0.03”	“This result is unlikely to be due to random chance — we’d recommend shipping it.”
“The 95% CI was [0.5%, 4.1%]”	“We’re confident the true improvement is somewhere between half a point and four points — even at the low end, this is worth shipping.”

Handling a non-significant or ambiguous result honestly — this is where a lot of analysts either oversell or undersell:

Avoid: “The test failed” or “there’s no effect” — neither is statistically accurate. The honest framing: “We didn’t detect a statistically significant effect at the sample size we tested — this could mean there’s truly no effect, or it could mean the effect is smaller than what we had power to detect. Here’s what it would take to find out for sure.” This directly applies the power/MDE reasoning from section 2 to a stakeholder conversation, which is exactly the kind of integration interviewers want to see.

Translating uncertainty without losing the audience:

“We’re not 100% certain, but the evidence strongly favors X” is usually more useful to a decision-maker than precise but opaque statistical language — confidence intervals are best communicated as a range of plausible outcomes for decision-making (“even in the worst case within our confidence interval, this is still positive”) rather than as the formal repeated-sampling definition from the Statistics module.

Talking point that synthesizes the whole module: “My job in communicating a test result isn’t to make the stakeholder understand p-values — it’s to make sure they walk away with an accurate sense of how confident they should be and what decision that confidence supports. If I’ve done my job on experiment design — clear hypothesis, adequate power, pre-registered metric — then the communication step should be straightforward, because I’m not trying to rescue a poorly-designed test with clever spin afterward.” This is a strong closing line for an interview — it reframes “communication skill” as downstream of design rigor, not a separate skill that compensates for sloppy analysis.