What is A/B testing?
A/B testing compares two variants using randomized assignment and a defined outcome. One variant is usually the current experience and the other contains the proposed change. Randomization helps create comparable groups, while an explicit measurement and analysis plan determines what comparison you can interpret.
Optimizely’s overview describes random allocation and statistical comparison as central parts of the method. Publishing one headline this week and another next week is a before-and-after change, not automatically a randomized A/B test. Different traffic sources, weekdays, or campaigns can change the audience between those periods.
For a fictional SaaS landing page, the question might be whether a task-focused explanation changes the probability that an eligible visitor starts a trial within seven days. That is a more useful test statement than asking which design people like. It connects a change with one observable outcome and a time limit.
Define the unit, outcome, and guardrails
Choose the assignment unit before calculating a sample. For an individual signup page, it might be an eligible person with consistent assignment across repeat visits. For a shared team workflow, account-level assignment may be more appropriate so teammates do not receive conflicting experiences. Sessions and repeated page views are not automatically independent people.
Write one primary binary outcome, such as whether an eligible assigned person starts a trial within seven days. Define eligibility, exposure, exclusions, duplicate handling, and how missing observations are treated. Check that assignment and event recording work before drawing conclusions about the product change.
Record guardrails such as failed submissions, important support problems, or a broken payment path. You can stop a harmful or broken experience operationally without claiming that the early conversion result established a statistical winner. Do not keep a test running merely to satisfy a sample plan when it is causing a material product failure.
Question: does a task-focused explanation change seven-day trial-start conversion? Unit: one eligible person, consistently assigned to A or B. Allocation: 1:1 random assignment. Primary outcome: at least one trial start within seven days of assignment. Planning baseline: 10%; target effect for planning: 12%. Guardrails: submission errors and reported inability to complete the task. Decision plan: fixed enrollment sample, mature outcome windows, then the prespecified analysis. Status: planning example only; no experiment has been run.
Understand baseline, minimum detectable effect, and power
The baseline is an estimate of the current outcome rate for a comparable eligible population and window. The minimum detectable effect is the difference the design is intended to detect with its stated power. Choose a difference that would matter to the decision, not merely a large number that makes the required sample look convenient.
Moving from 10% to 12% is a two-percentage-point absolute increase and a 20% relative increase. It is not a twenty-percentage-point increase. Keep both expressions visible when using a planning tool because the same numeric input can mean different things under absolute and relative settings.
In a prespecified frequentist design, a 5% significance level controls a long-run false-positive rate under the null and the method’s assumptions. It is not a 95% probability that a particular variant is better. An 80% power target describes the chance of detecting the planned effect if that effect is real under those assumptions; it does not promise that the experiment will succeed.
Hae-Young Kim’s statistical note explains how two-proportion planning depends on the test, significance level, power, allocation, and rate assumptions. Our local exercise uses a stated large-sample normal approximation; other valid methods and corrections can return different sample sizes.
Work through an original sample-size planning example
The downloadable lab uses a baseline of 0.10, target of 0.12, equal allocation, two-sided alpha 0.05, and power 0.80. It uses a normal approximation without continuity correction. Rates are fractions, not whole percentages, and the result is an enrollment plan rather than an analysis of experiment outcomes.
Let p1 = 0.10, p2 = 0.12, and pbar = (p1 + p2) ÷ 2. The lab calculates the per-variant sample as the ceiling of [1.959964 × sqrt(2 × pbar × (1 − pbar)) + 0.841621 × sqrt(p1 × (1 − p1) + p2 × (1 − p2))] squared, divided by (p2 − p1) squared. The file retains more precise constants.
That calculation gives 3,841 eligible independent units per variant, or 7,682 total. At a hypothetical 200 new eligible people per day across both variants, enrollment takes at least 39 days. Waiting seven more days for the last entrants’ outcome windows gives a 46-day arithmetic floor. At 40 per day, the same calculation gives 193 enrollment days plus seven, or 200 days.
These are invented traffic and rate assumptions, not ShowMeYourSaaS traffic or a forecast. A real schedule also needs stable eligibility, sufficient observations in each arm, business-cycle coverage, and review of missing data. The normal approximation is unsuitable for some small-cell or dependent-outcome settings; the lab rejects inputs outside its stated support.
| Planning item | Value |
|---|---|
| Baseline / target | 10% / 12% |
| Absolute / relative lift | 2 percentage points / 20% |
| Two-sided alpha / power | 0.05 / 0.80 |
| Sample per variant / total | 3,841 / 7,682 |
| 200 eligible people/day, plus 7-day outcome window | 39 + 7 = 46 days minimum |
| 40 eligible people/day, plus 7-day outcome window | 193 + 7 = 200 days minimum |
Avoid declaring a winner from repeated peeking
A conventional fixed-horizon test does not retain its stated false-positive control if you repeatedly check significance and stop whenever an attractive result appears. Evan Miller’s explanation shows why that stopping behavior creates more false positives than the original design allows.
You can inspect instrumentation, eligibility, allocation balance, and operational problems during a test. Keep those checks separate from a decision to declare the variant a conversion winner. If you need a sequential design, use a method with an appropriate stopping rule rather than improvising one around a fixed-horizon calculation.
At the planned analysis, report the population, sample by variant, outcome window, effect estimate, uncertainty, exclusions, and guardrails. A statistically detectable difference may still be too small to justify implementation or may coincide with worse product outcomes. An inconclusive result means the evidence does not settle the prespecified question.
What should a low-traffic SaaS do instead?
If the planned sample would take an impractical period, do not manufacture certainty by ending with a few conversions. Investigate whether the current question needs a randomized comparison at all. Watching relevant people attempt a task can reveal unclear instructions or a broken interaction before you test a subtle conversion effect.
The customer-interview guide supports problem investigation, and the landing-page guide helps make a useful first explanation. Those methods can reveal problems and guide changes, but they do not estimate a randomized causal conversion lift.
You can also change the decision being studied, choose a meaningfully larger effect if it is still relevant, or wait for enough eligible volume. Do not switch to a convenient upstream metric without checking that it still supports the business decision. The small-budget experiment guide helps keep those learning goals explicit.
Download the A/B test worksheet and planning lab
The original worksheet includes the complete test card, planning assumptions, normal-approximation formula, duration arithmetic, stopping rule, and a low-traffic decision checklist. Run node ab-test-planning-lab.mjs with Node.js 22 or newer to reproduce the planning numbers locally. It makes no network calls, changes no website, and declares no experiment winner. Sources were checked on October 27, 2026.