A/B TEST PLANNING WORKSHEET AND LOCAL LAB NOTES
Original ShowMeYourSaaS resource — 2026-10-27
Guide: https://showmeyoursaas.com/articles/ab-testing/
Companion code: ab-test-planning-lab.mjs

ORIGINAL FICTIONAL TEST CARD — NO EXPERIMENT HAS BEEN RUN
Question: does a task-focused explanation change seven-day trial-start conversion?
Unit: one eligible person, consistently randomized to A or B across visits.
Allocation: 1:1. Primary outcome: at least one trial start within seven days.
Planning baseline: 10%. Target effect for planning: 12%.
Two-sided alpha: 0.05. Power: 0.80. Fixed-horizon enrollment and analysis.
Guardrails: submission errors and inability to complete the intended task.
This scenario assumes independent units and a valid binary outcome.
Shared team workflows may require account-level assignment and different volume.

REPRODUCE THE PLAN
With Node.js 22 or newer: node ab-test-planning-lab.mjs
Inputs are fractional rates: baseline=0.10, target=0.12, dailyEligible=200
across BOTH variants, outcomeDays=7.
Method: normal approximation for two independent proportions, equal allocation,
two-sided alpha=0.05, power=0.80, no continuity correction.
pbar = (p1+p2)/2
n per variant = ceil((zAlpha*sqrt(2*pbar*(1-pbar)) +
zPower*sqrt(p1*(1-p1)+p2*(1-p2)))^2 / (p2-p1)^2)
zAlpha = 1.959963984540054; zPower = 0.8416212335729143.
Expected: 3,841 per variant; 7,682 total.
Absolute lift: 2 percentage points. Relative lift: 20%.
At 200 eligible people/day: ceil(7682/200)=39 enrollment days; +7=46 days.
At 40/day: ceil(7682/40)=193 enrollment days; +7=200 days.
These are arithmetic floors, not schedules or forecasts. Check business cycles,
balanced enrollment, changing eligibility, and complete outcome windows.
The lab does not analyze results, assign visitors, or declare a winner.
It makes no network calls and writes no files.

SUPPORT AND LIMITS
Use 0 < baseline < target < 1, integer dailyEligible from 1 to 1,000,000,000,
and integer outcomeDays from 0 to 365. The lab rejects unsupported sample sizes
above 1,000,000,000 per variant and expected cells below 5. These bounds do not
establish that every allowed scenario is appropriate for your experiment.
Different statistical methods/corrections can give different sample sizes.

PRESPECIFY YOUR OWN EXPERIMENT
Decision, hypothesis, and effect worth detecting:
Unit, eligibility, identity, assignment, and exposure rules:
Primary outcome and observation window:
Baseline source, date, and comparability:
Target effect (absolute AND relative):
Analysis method, alpha, power, allocation, and sample:
Duration assumptions and outcome-maturity allowance:
Exclusions, missing outcomes, and duplicate rules:
Instrumentation/assignment checks and guardrails:
Stopping rule and planned analysis date:
Owner and review:

INTERPRETATION
A 5% significance level is not a 95% probability that a variant is better.
80% power is not a promise of an 80% chance that this experiment succeeds.
Repeatedly checking fixed-horizon significance and stopping on a favorable
result does not preserve the original false-positive control.
Operationally stopping a harmful/broken experience is different from declaring
a statistical winner. Report effect size, uncertainty, exclusions, and guardrails.
If sample needs are impractical, consider task observation or interviews for
the current question; those do not estimate a randomized conversion lift.

Sources checked 2026-10-27:
https://www.optimizely.com/optimization-glossary/ab-testing
https://pmc.ncbi.nlm.nih.gov/articles/PMC4868880/
https://www.evanmiller.org/how-not-to-run-an-ab-test.html