A/B testing interviews
and jobs in 2027

Experimentation is central to product companies: e-commerce, fintech, consumer apps, marketplaces, ad platforms and SaaS. The skill appears in job descriptions for product analysts, data scientists (product and experimentation), growth marketers, CRO specialists and product managers.

It is also a pay lever. On AmbitionBox in October 2026, product analysts in India typically earned ₹11.5–12.8 lakh and data scientists ₹15.2–16.8 lakh,⁶⁷ both well above general data analyst bands — and experiment design is one of the skills that most often separates those roles in interviews.

Revenue lift from one Bing headline experiment (HBR, 2017)

12%

Typical product analyst pay in India (AmbitionBox, Oct 2026)

₹11.5–12.8 lakh

Typical data scientist pay in India (AmbitionBox)

₹15.2–16.8 lakh

Year Microsoft researchers published CUPED

2013

Table: How A/B testing is tested in interviews (editorial summary)

1. Hypothesis — "Showing delivery time on the product page will increase add-to-cart because uncertainty causes hesitation."
2. Metrics — One primary metric, input metrics and guardrails (revenue per user, refunds, latency, crashes).
3. Unit of randomisation — User, session, device, store or city — chosen to avoid contamination.
4. Sample size and power — Based on baseline rate, minimum detectable effect, significance level and power.
5. Duration — Whole weeks to cover day-of-week patterns; long enough for novelty to fade.
6. Quality checks — A/A tests, SRM checks and logging validation before reading results.
7. Decision rule — Agreed in advance: what result means ship, iterate or stop.

The standard reference for all of these is Kohavi, Tang and Xu's book on trustworthy online controlled experiments,¹ and the ASA's statement on p-values explains why significance alone isn't a decision.⁵

"Design a test for adding 'Buy now, pay later' at checkout."

Hypothesis about conversion; primary metric checkout conversion; guardrails default rates, refunds, average order value; randomise by user; size for a realistic lift; run two to four full weeks; check SRM; agree the decision rule.

"How do you calculate the sample size?"

Use baseline conversion, minimum detectable effect, α (usually 0.05) and power (usually 80%). Smaller effects need many more users — roughly four times the sample to detect half the effect.

"The test shows +2% conversion but −3% revenue per user. Ship?"

Not without understanding why — maybe discounting moved users to cheaper items. Revenue is the business outcome; investigate segments and decide with the guardrail in mind.

"Control has 50.8% of users, variant 49.2%, with a million users. Problem?"

Yes — that's very unlikely by chance at that scale. Check an SRM test, then look for bot filtering, redirects, logging or eligibility bugs before reading any metric.

"The test is flat. What now?"

Check power (could it detect a meaningful effect?), segments for heterogeneous effects (pre-registered if possible), and the implementation. A well-powered flat result is useful: it saves building on a bad idea.

"What is CUPED and why use it?"

A variance-reduction method using pre-experiment behaviour as a covariate, so tests reach the same power with fewer users or less time.³

"How do you test in a two-sided marketplace?"

Interference between buyers and sellers biases user-level tests. Use cluster randomisation (by city or market), switchback designs over time, or model the spillover.

"When should you not A/B test?"

When traffic is too low to detect meaningful effects, the change is legally required or obviously necessary, ethics forbid withholding it, or the decision is easily reversible and cheap — use other evidence then.

Table: How experimentation hiring has changed (editorial summary)

• A written experiment design for a real product, with sample-size reasoning.
• An analysis write-up of a public or simulated experiment, including SRM and guardrails.
• A "flat result" memo — what you'd conclude and do next.
• A verified A/B testing score to link.

1. Week 1: Foundations — Hypotheses, metrics, randomisation; read the core chapters of Kohavi's book.
2. Week 2: Statistics — Power, sample size, intervals, p-values; the ASA statement.
3. Week 3: Pitfalls — Peeking, SRM, novelty, multiple testing, Twyman's law.
4. Week 4: Advanced — CUPED, sequential testing, interference and switchbacks.
5. Week 5: Practice — Five design cases and three interpretation cases; take the A/B Testing and Experimentation Certification.
6. Week 6: Portfolio — Publish one design and one analysis write-up.

Related tests: Statistics for Data Analysts, Product Analytics and Conversion Rate Optimization.

Baseline checkout conversion is 4%. The team wants to detect a relative lift of 5% (to 4.2%) with α = 0.05 and 80% power.

1. Effect — 0.2 percentage points.
2. Approximate users per group — 16 × p(1−p) / δ² = 16 × 0.04 × 0.96 / 0.002² ≈ 153,600.
3. Traffic check — With 25,000 eligible users a day split evenly, that's about 12 days per group's worth of traffic — round up to two full weeks to cover weekly patterns.
4. Trade-off — If two weeks is too long, either accept a larger minimum detectable effect, use CUPED to reduce variance, or test a bigger change.
5. Guardrails — Average order value, refunds and payment failures, checked with the same rigour.

The rule of sixteen (16 × variance / effect²) is a quick approximation interviewers accept; say you'd confirm with a proper power calculator.

1. Design five tests on paper — For apps you use, write hypothesis, primary and guardrail metrics, unit, sample size and duration.
2. Simulate peeking — In a notebook, simulate A/A tests and stop each at the first p < 0.05; count how many "wins" appear.
3. SRM check — Write a chi-square check for sample ratio mismatch and run it on simulated broken splits.
4. CUPED by hand — Simulate pre-period and test-period data and show how CUPED narrows the confidence interval.
5. Write the flat-result memo — Take a non-significant result and write what you'd conclude, using power and the interval.

Experiment design is one of the clearest signals for product and data science roles. Typical pay on AmbitionBox, October 2026:

Table: Roles that use experimentation and their typical pay in India (AmbitionBox, October 2026; indicative)

• Reading results before checking data quality.
• No guardrails in the design.
• Treating p < 0.05 as "ship".
• Ignoring power when a test is flat.
• Post-hoc segment fishing presented as discovery.

Experimentation guidance follows Kohavi, Tang and Xu's book, Microsoft Research's CUPED and SRM papers, Kohavi and Thomke's HBR article and the ASA's p-value statement. Pay figures come from AmbitionBox (read October 2026). Interview formats and the history table are editorial summaries. Science of Apps publishes the free certification linked here. Last reviewed October 2026.

• Control / variant — the existing experience and the changed one in an experiment.
• Minimum detectable effect — the smallest effect a test is designed to detect reliably.
• Guardrail metric — a metric that must not get worse, even if the primary metric improves.
• Sample ratio mismatch — an unexpected imbalance between groups, signalling a broken test.
• CUPED — a variance-reduction technique using pre-experiment data.
• Sequential testing — methods that allow valid monitoring of results as data arrives.