On this page
An uplift number is not causal because it is large, precise, or produced by a model.
Before asking whether the number is plausible, ask what it is a number of: a difference in outcomes, an average treatment effect, a conditional effect for a subgroup, a predicted response, or an estimate of what would have happened without treatment.
The practical gate is simple to state and difficult to skip: who received what, compared with which counterfactual, for which estimand, over what time, with what interference and outcome measurement?
Why do published marketing uplift results systematically decay at commercial scale?
DellaVigna and Linos compare published nudge evidence with a census of 126 Nudge Unit trials covering more than 23 million participants. The average effect in their published sample was 8.7 percentage points, a 33.4% increase over control. The at-scale census averaged 1.4 percentage points, an 8.0% increase.
The contrast is not a reason to reject every published effect. Their model attributes about 70% of the gap to selective publication exacerbated by low statistical power, which brings the published estimate closer to the at-scale estimate. It is a reason to name the evidence object. A study estimate and a population of deployed trials answer related but different questions.
An uplift claim that does not name its sample, deployment context, and estimand can silently move from “the treated group in this study” to “the business will gain this much.” That is not a statistical detail. It changes the decision.
Why is unexposed audience behavior an invalid counterfactual in causal advertising?
Gordon and colleagues provide a worked Facebook example. The randomized benchmark was a 73% treatment lift, with a 95% interval from 49% to 103%. The naive exposed-versus-unexposed comparison estimated a 316% lift. Exact matching on age and gender alone still estimated 222%.
The comparison makes a common mistake visible. People who are exposed may differ from people who are not exposed before the treatment is delivered. They may be more active, more valuable, more likely to buy, or more likely to be targeted. The observed difference then contains selection, treatment, and measurement differences.
Matching can reduce some imbalance. It does not automatically recreate the counterfactual. The specification must say what makes the comparison credible, which variables are used, what remains unmeasured, and whether the estimand is individual, campaign, customer, or population level.
How does optimizing for intermediate engagement metrics mislead causal lift claims?
Randomization helps identify a treatment contrast. It does not decide whether the outcome is the one the business needs.
An intervention can lift clicks in the first week and reduce retained margin over a quarter. It can increase trial starts without increasing qualified adoption. It can change brand search without changing incremental purchases. The measurement window and outcome definition are part of the claim, not an appendix.
The eBay paid-search experiment by Blake, Nosko, and Tadelis illustrates the point. Brand-keyword advertising showed no measurable short-term benefit in the experiment. Frequent users accounted for most advertising expense and average returns were negative, while new and infrequent users were positively influenced by paid search. The experimental non-brand-search ROI estimate was -63%, with a 95% interval from -124% to -3%.
These results do not create a universal rule against brand search or for non-brand search. They show why treatment response, customer type, keyword class, outcome window, and cost definition need to travel with the uplift claim.
What documentation must commercial teams mandate in a causal-claim specification sheet?
The sheet has nine fields. For each one: the question it asks, what the number can no longer carry when the field is blank, and the language that stays safe.
- Unit. Person, account, order, campaign, market, or population? Blank: the denominator can change while the number stays precise. Safe language: the estimate is unresolved at the business level.
- Treatment. What exactly was delivered, to whom, and when? Blank: the exposure can contain several interventions. Safe language: describe the observed exposure only.
- Counterfactual. What would the treated unit have experienced without treatment? Blank: exposed and unexposed groups may have differed before treatment. Safe language: call the contrast observational.
- Estimand. Average, conditional, incremental, short-run, or long-run effect? Blank: different effects can be combined under one uplift label. Safe language: name the estimand or stop.
- Interference. Can one unit’s treatment change another unit’s outcome? Blank: the control can be contaminated or demand shifted. Safe language: state the interference boundary.
- Time window. Which outcome period matters? Blank: early lift can hide later cost or churn. Safe language: restrict the claim to the measured window.
- Outcome. Click, lead, adoption, revenue, margin, retention, or another object? Blank: a proxy can replace the decision outcome. Safe language: name the proxy and its limit.
- Measurement. How are costs, attribution, and missing outcomes defined? Blank: the effect can be a reporting change. Safe language: keep measurement and treatment separate.
- Decision. What action follows, and what would disconfirm it? Blank: a causal estimate can be decision-irrelevant. Safe language: record action, threshold, and follow-up.
Figure 1Causal claim specification sheet
One row per field, filled before the number is read. The nine fields and what each protects are spelled out above the sheet.
Source: Author's own worksheet; framework rows are the author's synthesis of DellaVigna and Linos (2022), Gordon, Zettelmeyer, Bhargava, and Chapsky (2019), and Blake, Nosko, and Tadelis (2015).
How does cross-channel market interference violate causal stable unit treatment assumptions?
Many commercial interventions do not affect one unit in isolation. A treated customer can tell an untreated colleague. A promotion can move demand across channels or periods. A sales message can change the workload received by a service team. A holdout can therefore be contaminated, or the treatment can shift outcomes rather than create them.
Interference does not make causal work impossible. It makes the boundary part of the specification. Is the estimand the effect on treated accounts, the effect on the whole market, or the effect of a campaign including spillovers? If the answer is unknown, the uplift language must stay provisional.
Why must experimental measurement design precede econometric model selection?
The temptation is to ask which uplift model, attribution platform, or matching algorithm to use first. The model is downstream of the object. A model can estimate a conditional response while leaving the counterfactual, cost, time window, or business outcome ambiguous.
Before choosing a method, write the specification in plain language, then identify the design that can support it. If the required counterfactual is not available, say what the data can show: an association, a descriptive difference, a prediction, or a bounded experiment result. The honest smaller claim is more useful than a larger causal word without an identification story.
What empirical release gates must commercial teams clear before scaling campaigns?
An uplift claim can enter a decision memo only when the sheet names:
- the unit and denominator;
- treatment and timing;
- counterfactual and assignment rule;
- estimand and interference boundary;
- outcome and time window;
- cost and measurement definition; and
- action, threshold, and disconfirmation observation.
If one field is missing, the number may still be useful as a descriptive input. It is not yet a causal uplift claim. The distinction protects both analysis and decision quality.
The specification belongs beside the counterfactual attribution model and the incrementality illusion, which keep observed lift separate from the outcome the decision actually needs. It also demonstrates why more advertising observations do not guarantee better decisions, since uncalibrated observational data inflates confidence without resolving selection bias.
References
- DellaVigna, S., & Linos, E. (2022). RCTs to scale: Comprehensive evidence from two nudge units. Econometrica, 90(1), 81–116. https://doi.org/10.3982/ECTA18709
- Gordon, B. R., Zettelmeyer, F., Bhargava, N., & Chapsky, D. (2019). A comparison of approaches to advertising measurement: Evidence from big field experiments at Facebook. Marketing Science, 38(2), 193–225. https://doi.org/10.1287/mksc.2018.1135
- Blake, T., Nosko, C., & Tadelis, S. (2015). Consumer heterogeneity and paid search effectiveness: A large-scale field experiment. Econometrica, 83(1), 155–174. https://doi.org/10.3982/ECTA12423
Updated 1 change
- The Econometrica article, 90(1), 81 to 116, is now held, so this pair returns to the published figures: 33.4% and 8.0% rather than the working paper's 33.5% and 8.1%, and the authors' model attributes about 70% of the gap to selective publication with low power rather than the full difference. The working paper had been cited since 19 August 2026 because it was the only version held. In the same revision the specification sheet moved from a filled table to a worksheet; its nine fields and their guidance now stand in the text above it.
