Figure 14 Sept 2026From “What is an A/B test? randomized controlled trials, statistical power, and governance”

The A/B test experimental architecture

Examine statistical power, significance thresholds, sample ratio mismatch, and MDE boundaries.

Figure 1The A/B test experimental architecture

Experimental Component	Statistical Function	Standard Operational Threshold	Primary Risk If Ignored
Statistical Power ($1 - \beta$)	Probability of detecting a true effect if one exists	80% to 90% ($\beta = 0.10$ to $0.20$)	Type II error: discarding real growth innovations
Significance Level ($\alpha$)	Probability of rejecting null hypothesis when it is true	5% ($\alpha = 0.05$, two-tailed, $Z \ge 1.96$)	Type I error: shipping ineffective or harmful code
Sample Ratio Mismatch (SRM)	Goodness-of-fit test for allocation integrity	$\chi^2$ test $p$-value $\ge 10^{-3}$	Experimental invalidation due to tracking or CDN bias
Minimum Detectable Effect	Smallest relative lift the test is sized to detect	Derived from baseline variance and sample size	Running underpowered tests with uninterpretable noise

Swipe or scroll horizontally if the table is wider than your screen.

Source: Author's experimental framework grounded in causal inference and split-testing literature from Kohavi et al. (2013), Kohavi et al. (2009), and Lewis and Rao (2015).

PNG4:3, 1600 × 1200, with the title, the source line and the attribution set in the image isoglu-2026-what-is-an-ab-test-fig01.png

From the essay

From the research bench

What is an A/B test? randomized controlled trials, statistical power, and governance

An A/B test is a randomized experiment comparing two concurrent variants to measure the causal impact of a single change on commercial performance.

4 Sept 2026

Read the essay

This exhibit belongs to an essay: the numbers live here, the argument lives there.

Cite or embed

APA 7

Isoglu, S. (2026). The A/B test experimental architecture [Figure]. isoglu.com. https://isoglu.com/insights/exhibits/isoglu-2026-what-is-an-ab-test-fig01/

BibTeX

@misc{isoglu-2026-what-is-an-ab-test-fig01,
  author = {Isoglu, Sinan},
  title = {{The A/B test experimental architecture}},
  year = {2026},
  howpublished = {Figure 1, isoglu.com},
  url = {https://isoglu.com/insights/exhibits/isoglu-2026-what-is-an-ab-test-fig01/},
  note = {From the essay {What is an A/B test? randomized controlled trials, statistical power, and governance}}
}

Embed this exhibit

Copy this HTML to use the PNG exhibit on your site with its source line and a link back to the exhibit page.

<figure>
  <a href="https://isoglu.com/insights/exhibits/isoglu-2026-what-is-an-ab-test-fig01/">
    <img src="https://isoglu.com/social/isoglu-2026-what-is-an-ab-test-fig01.png" alt="Figure 1: The A/B test experimental architecture" width="3200" height="2400">
  </a>
  <figcaption>Figure 1: The A/B test experimental architecture. Source: Author&#39;s experimental framework grounded in causal inference and split-testing literature from Kohavi et al. (2013), Kohavi et al. (2009), and Lewis and Rao (2015). <a href="https://isoglu.com/insights/exhibits/isoglu-2026-what-is-an-ab-test-fig01/">View exhibit</a>.</figcaption>
</figure>