Knowledge
Knowledge HubConcepts and strategic researchGlossaryDefinitions across six domainsFrameworksStructured strategy methodsTopicsConnected subject areasLibrary
PlaybooksPractical strategy guidesBooksBooks and long-form researchOperator VaultMember resources and toolkits
Explore Discover, the Knowledge Hub, playbooks, and the Operator Vault.
Glossary
A/B testing
A/B testing is a randomized comparison of two (or more) alternatives that estimates the causal effect of a change on predefined metrics by assigning units — such as users, sessions, stores, or other experimental units — to variants under a controlled design. Online controlled experiments are a major application, not the only setting in which A/B tests are run.
Why it matters
Opinion scales poorly; well-designed experiments localize the impact of product, pricing, messaging, or operations changes. Digital products made online A/B testing common, but the same logic applies offline when randomization and measurement are feasible (for example, store-level or sales-script tests).
Random assignment supports causal comparison; it does not automatically eliminate every threat to validity. Assignment-unit choice, interference between units (spillovers), measurement quality, peeking, underpowered tests, and novelty effects can still distort conclusions. Trustworthy online controlled experiments literature (Kohavi and coauthors) sets a practical bar for digital settings without claiming universal certainty.
Example (enterprise): A SaaS team tests a new onboarding checklist against the control with proper randomization and a pre-registered primary metric (activation within seven days). They also check for interference (shared workspaces) and resist shipping on a vanity uplift in pageviews that was not the decision metric.
Mechanisms
- Define the hypothesis, primary metric, and unit of randomization before launch.
- Assign variants randomly with adequate power and instrumentation; consider interference and segment effects.
- Estimate effects with uncertainty and decide with pre-agreed thresholds — then watch for long-run, novelty, and measurement risks.
Boundaries
- A/B testing ≠ online-only testing; online controlled experiments are a major application, not the definitional limit.
- A/B testing ≠ all experimentation; qualitative tests, holdouts, and quasi-experiments are related but not identical.
- Randomization ≠ automatic elimination of every validity threat (interference, bad metrics, peeking, underpowering).
- A/B testing ≠ proof of strategy; local metric wins can still harm long-term advantage.
Related topics
None linked yet.
Related concepts & frameworks
Further research
No related resources yet.