Figure 1From “More advertising observations do not guarantee better decisions”
The advertising test-value screen
Release additional advertising data when it can inform a named action, not simply because the sample is larger.
Figure 1The advertising test-value screen
| Review field | Required input | Permitted interpretation | Stop signal |
|---|---|---|---|
| Decision | Action that could change | “This test informs this choice.” | The test has no stated decision. |
| Relevant effect | Minimum commercially meaningful difference | “This effect size matters at the stated threshold.” | Any statistically detectable difference is treated as useful. |
| Variance and power | Outcome noise and detection target | “The design has power for the effect that matters.” | Sample size is copied from another campaign. |
| Observation cost | Data, time, and follow-up burden | “More data is worth collecting under this cost.” | Data volume is treated as free. |
| Action threshold | Result that changes the recommendation | “This estimate crosses or does not cross the rule.” | A confidence interval is reported without a decision rule. |
| Claim boundary | Study, platform, horizon, and metric | “The evidence supports this bounded statement.” | One experiment becomes a universal sample-size or ROI rule. |
Swipe or scroll horizontally if the table is wider than your screen.
Source: Author's decision framework grounded in Johnson, Lewis and Reiley (2017) and Wernerfelt, Tuchman, Shapiro and Moakler (2025). The thresholds and example prompts are synthetic; the reported platform findings remain bounded to their studies.
Source and evidence
Each line names a claim related to this exhibit, its source, and how it was checked.
- A JLR17-C1 The design, in the authors' words: "We use a randomized field experiment on 3 million Yahoo! users who are also past customers of the retailer", and the paper is about what the data buy: "We show that control ads boost measurement precision by identifying and removing the half of in-campaign sales data that are unaffected by the ads"Johnson, Lewis & Reiley. (2017)
- A JLR17-C2 More data is not the lever the title implies: "Less data give us 31% more precision in our estimates", while "we only improve precision by 5% when we include additional covariate data to reduce the residual variance in our experimental regression"Johnson, Lewis & Reiley. (2017)
- A JLR17-C3 So the design question is power and decision value rather than observation count, which the authors demonstrate by the contrast between removing unaffected data and adding covariates: "Less data give us 31% more precision in our estimates"Johnson, Lewis & Reiley. (2017)
- A WTSM25-C1 The experiment, verbatim: "we conduct a large-scale, randomized experiment that includes more than 70,000 advertisers on Facebook and Instagram"Wernerfelt, Tuchman, Shapiro & Moakler. (2025)
- A WTSM25-C2 The two costs and the change between them, verbatim: "We find a median cost per incremental customer at baseline of $38.16 that under the median loss in effectiveness would rise to $49.93, a 31% increase"Wernerfelt, Tuchman, Shapiro & Moakler. (2025)
- A WTSM25-C3 And the distributional finding: "we find ads targeted using offsite data generate more long-term customers per dollar than those without, and losing offsite data disproportionately hurts small scale advertisers"Wernerfelt, Tuchman, Shapiro & Moakler. (2025)
Grades: A, verified against the printed page of the primary source · B, primary source, text layer only · C, authoritative secondary · D, reported.
From the essay
From the research bench
More advertising observations do not guarantee better decisions
More advertising data can improve precision, but decision value also depends on variance, effect size, observation cost, and action thresholds.
Read the essayThis exhibit belongs to an essay: the numbers live here, the argument lives there.
Cite or embed
Related exhibits
Table 1Table
Why must advertising measurement design begin with commercial decision thresholds?
More advertising observations do not guarantee better decisions
Figure 1Figure
Users needed for the same power, by correlation with the pre-experiment metric
What is CUPED?
Figure 1Figure
The A/B test experimental architecture
What is an A/B test? randomized controlled trials, statistical power, and governance
Contact
Bring the question you're working on.
Research, a guest lecture, or a commercial question.