---
type: pre-registered protocol
piece: research cluster — retrieval audit
registered: 2026-07-27, before any run beyond the single pilot question
rule: P6 type 3 (primary data) · P49 (decide the analysis before seeing the data) · P60
status: FROZEN. Changes after the first run go in the changelog below, dated, with the reason.
---

# Protocol — what four assistants return when asked where a number comes from

Registered before the runs. Published with the data so a reader can re-run it and disagree.

---

## 1 · The question this answers

When someone with a commercial decision to make asks an AI assistant a question whose answer is a
widely-circulated statistic, **and asks explicitly for the original source, what provenance do they
get?**

Two things make this answerable rather than rhetorical. Asking for the source in the prompt turns an
omission into a failure against a stated instruction. And provenance is checkable — either a primary
document exists and can be produced, or its absence can be established by search.

---

## 2 · What is being measured, and what is not

**Measured:** what the retrieval layer returns for these questions, on these dates.

**Not measured:** whether the models are good, whether they are getting better, or whether they are
reliable in general. Every result here is a statement about **the public evidence surface for
commercial questions**, sampled through the tool most people now use to reach it. If the sources are
poor, that is a fact about what has been published, not only about what was retrieved.

The essay must not become "AI is unreliable." That framing is available, cheap, and wrong.

---

## 3 · Question selection — criteria fixed before choosing

A question qualifies only if **all four** hold:

1. It states or implies a **specific numeric claim** that circulates in commercial practice.
2. The provenance is **checkable** — a primary source either exists and is reachable, or its absence
   can be established.
3. A commercial decision-maker would **plausibly ask it**, in roughly these words.
4. It was chosen **before** any assistant was asked.

### The control requirement, which is the important one

**At least one third of questions must be ones where a clean, reachable primary source exists.**

Without controls the design manufactures its own finding: ask only about statistics known to be
unsourceable and of course the answers are unsourced. Controls make the result falsifiable — if the
assistants cite the primary correctly whenever one exists, then the finding is about the literature,
not about retrieval, and the essay says so.

### The set

| # | Question | Class | Expected provenance |
| --- | --- | --- | --- |
| Q1 | How much more expensive is acquiring a customer than retaining one? | Unrecoverable | None exists — established 2026-07-27 |
| Q2 | What share of the B2B buying journey is complete before a vendor is contacted? | Contested | A real origin exists (CEB/Gartner) and is routinely misstated |
| Q3 | How many touchpoints does it take to book a B2B sales meeting? | Vendor-origin | Trade figures only, no recoverable method |
| Q4 | What did Brynjolfsson, Li and Raymond find about AI and customer-support productivity? | **CONTROL** | *QJE* 140(2), 889–942. Clean, open, findable |
| Q5 | What did the BCG study find about consultants using GPT-4? | **CONTROL** | *Organization Science* 37(2), 403–423. Clean — and see the ground-truth block below, established 2026-07-27 before the run |
| Q6 | What proportion of A/B tests improve the metric they target? | Semi-recoverable | Kohavi's one-third is real but conference-only; the circulating 90% is not his |

Two of six are controls. Q6 is deliberately awkward: a real figure exists in a hard-to-reach place
while a false one circulates freely, which is the most common shape in commercial evidence.

---

## 4 · The prompt

Identical across every model and run. Only the question varies.

> `<QUESTION>` **Give the figure and cite the original source.**

The instruction to cite is not optional decoration. It converts a silent omission into a
non-compliance, which is a different and more defensible finding.

---

## 5 · Run design

- **Systems:** ChatGPT, Claude, Gemini, Perplexity. Web search **on**. Model identifiers recorded per
  run, because they change.
- **Surface — the limitation that matters most.** These runs go to **API endpoints with web search
  enabled**, not to the consumer chat products. `chatgpt.com` and `claude.ai` are different stacks:
  different retrieval, different system prompts, different ranking, sometimes different models. Most
  people asking these questions are using the consumer app. **The finding as run is therefore about
  the API surface, and the essay may not silently generalise it to the product surface.** Either the
  claim is narrowed to what was tested, or the consumer surface is tested too — see the changelog.
  Recorded here, before the counted dataset, because discovering this after publication would be the
  same failure the study is about.
- **Repeats:** **three runs per model per question.** These systems are stochastic and a single run is
  an anecdote. 6 questions × 4 systems × 3 runs = **72 responses**.
- **Temperature:** left at the endpoint default, recorded.
- **Dates:** every response timestamped. The corpus moves; the finding is dated or it is nothing.
- **Cost:** roughly €8 at observed per-call pricing. The whole study is re-runnable by a reader for
  the price of lunch, which is the point.

---

## 6 · Coding scheme — fixed before any run

Every response coded on ten fields. Where a judgement is needed the rule is written here, not
decided later.

| Field | Values | Rule |
| --- | --- | --- |
| `figure_given` | yes / no / range | The number, verbatim |
| `source_named` | yes / no | Any specific document, author or organisation named as the origin |
| `source_class` | primary / secondary / tertiary / none | **Primary** = the study or document that first reported it. **Secondary** = a peer-reviewed work citing it. **Tertiary** = trade press, vendor content, encyclopedia, listicle |
| `citations_n` | integer | Total citation slots returned |
| `domains_unique` | integer | Distinct domains among them. The gap between this and `citations_n` is repetition presented as breadth |
| `interested_share` | 0–1 | Share of unique domains that **sell in the category the claim supports**. Coded from the domain's own homepage, not from memory |
| `contra_as_support` | yes / no | Does any cited document **argue against** the claim it is cited for? |
| `provenance_error` | list | Factual errors *in the provenance itself*: wrong organisation, wrong year, wrong paper, wrong author |
| `search_used` | yes / no | Whether the system actually retrieved, or answered from parametric memory with search requested. **Added after the control pilot** — see changelog |
| `version_returned` | vor / preprint / n/a | Where a published version of record and an earlier preprint both exist, which one the answer is built on |

**`interested_share` is the field most open to motivated coding, so it gets the strictest rule.** A
domain counts as interested only if it sells a product or service in the category the claim supports
— a retention-software vendor for a retention claim. General business media do not count. Consultancies
do not count unless the claim is about the service they sell. Every classification is recorded with
the URL so a reader can disagree with individual calls.

**Second coder:** a pass by an independent reader on a 20% sample, with disagreements reported rather
than reconciled away. If agreement is poor, that is published too.

---

## 7 · Stated in advance: what would make this finding wrong

- **Controls come back clean and everything else is noise.** If Q4 and Q5 are cited to primary
  reliably, the honest conclusion is narrower: retrieval works where the literature is sound, and the
  problem is the literature. That is a *better* essay and the protocol should be allowed to produce it.
- **Repeat runs disagree wildly.** If the same model gives a different `source_class` across three
  runs, the headline is variance, not provenance, and it must be reported as such.
- **The interested-share coding does not survive a second coder.** Then that column comes out.

---

## 8 · What gets published

The protocol, the full response set with timestamps and model identifiers, the coding sheet with
every URL, the second-coder disagreements, and the script. A reader with an API key re-runs it in an
afternoon.

That is the falsifiability the previous two briefs could not offer, and it is the only reason this
candidate is alive when seventeen others are not.

---

## Changelog

| Date | Change | Reason |
| --- | --- | --- |
| 2026-07-27 | Registered. Q1 already run once as a pilot, before the protocol existed — that run is reported separately as a pilot and **excluded from the counted dataset**. | Pilot preceded registration; counting it would be coding after seeing the data |
| 2026-07-27 | Two fields added — `search_used` and `version_returned` — after the Q4 control pilot, before any counted run. | The control produced a failure mode the scheme could not record. One system answered with `web_search: false` despite search being requested, and both systems built their answer on the 2023 preprint rather than the 2025 version of record. "No source" and "the superseded source" are different findings with different causes, and the original eight fields collapsed them |
| 2026-07-27 | Q4 and Q5 promoted from control to **primary interest**. | The controls were included to make the design falsifiable by showing retrieval works where the literature is sound. On the first control cell it did not — which makes them the most informative questions in the set, not the least |
| 2026-07-27 | Q5 ground truth built and written into the protocol **before** the run, including the preprint/version-of-record difference table. | Q4 was coded against a ground truth that already existed in the verified bank. Q5's had to be constructed, and constructing it after seeing the answers would be coding on the data |
| 2026-07-27 | Added to the analysis plan: **count inbound links to preprint vs version of record** for both control papers. | Eight of eight cells are explained by "the link graph rewards the free version." That is currently an inference. If the essay states the mechanism, the mechanism has to be measured |
| 2026-07-27 | **Surface limitation recorded, and a consumer-surface arm added.** All runs to date are against API endpoints, not `chatgpt.com` / `claude.ai` / the Gemini app. Q5 will also be run through a consumer-interface scrape, and if the two surfaces disagree the essay reports both rather than the convenient one. | The essay's premise is "the tool most people now use." Most people use the product, not the API. Running only the API and writing about the product would be an unstated generalisation — precisely the failure mode this study documents in others. Caught before the counted dataset, not after |
| 2026-07-27 | **Surface arm run and reported** (Appendix B). Both controls put to the consumer ChatGPT interface; the capability confound broken with a `gpt-5.5` API run. | The limitation recorded an hour earlier turned out to be load-bearing: the consumer surface cites the version of record and the API does not, at any model strength. Had the essay shipped on API runs alone it would have been wrong about the mechanism |
| 2026-07-27 | **Third runs completed.** Q5 now has three valid retrieval runs on all four systems (thirteen runs including the gpt-5.5 configuration). Q4 has three on Perplexity, Claude and ChatGPT. | The protocol specifies three runs per model per question; reporting two while citing a three-run design would be the same overclaim the study documents |
| 2026-07-27 | **Deviation closed.** Gemini's Q4 cell re-run on the exact frozen prompt at 20:34 UTC. Result: 14%, NBER WP 31161, no version of record — and **5,172 agents**, the published paper's sample size paired with the preprint's effect size, a combination that appears in neither document. All eight control cells now stand at three valid runs. | The discarded run had to be replaced, not merely disclosed |
| 2026-07-27 | **PROTOCOL DEVIATION, recorded and the run discarded.** Gemini's third Q4 run was issued with "Answer in under 120 words" appended, to limit response size. The prompt is frozen in section 4 and must be identical across every run. That run is **excluded**; Gemini's Q4 cell stands at two valid runs and is reported as two. | An altered prompt is a different test. Silently counting it would have made the three-run claim false in exactly the way this study is about |
| 2026-07-27 | **Mechanism measured** (Appendix D). Referring domains counted for both papers' preprint and version-of-record URLs. | The protocol required this before the essay could assert a mechanism. The ratio predicts the retrieval outcome on both questions — 3:1 with one system finding the VoR, 20:1 with none — so "the link graph rewards the free version" moves from inference to measurement |
| 2026-07-27 | Claude's Q5 cell re-run with search confirmed active; the two `web_search: false` runs are excluded from counted cells. | Runs answered from parametric memory do not measure retrieval and cannot sit in the same column as ones that do |
| 2026-07-27 | **Theme and scope ruled** (SI): the piece opens a new **evidence/method** cluster rather than being bent into revops, growth or gtm; and it publishes as **essay + open replication package** — protocol, full responses with timestamps and model IDs, coding sheet with every URL, and the script. | The finding is about provenance itself, so an existing cluster would distort it. And a piece arguing "you cannot check the numbers you are handed" that ships unverifiable would refute itself |

---

# Appendix — the control pilot, 2026-07-27

Run before the counted dataset, to check the scheme was operable. **It was not**, which is what a
pilot is for. Two fields were added and the controls were reclassified.

## Q4 · "What did Brynjolfsson, Li and Raymond find about the effect of AI assistance on customer support agent productivity? Give the figure and cite the original source."

**Ground truth**, from the verified bank: the version of record is *The Quarterly Journal of
Economics* **140**(2), 889–942 (2025). **15%** average lift, **5,172** agents. The **34%** figure for
novice and low-skilled workers **exists only in the 2023 NBER working paper**, which reports 14% and
5,179 agents. This is guard `AI-X4`, written weeks ago: *"never cite BLR as '2023, 14%, QJE'."*

| System | Figure | Named as the original source | `version_returned` | `search_used` |
| --- | --- | --- | --- | --- |
| Perplexity · sonar | **14%**, plus **34%** for novices | "the NBER working paper *Generative AI at Work*" | **preprint** | yes |
| Claude · sonnet-4.5 | **14%**, plus "about **35%**" for the bottom quartile | "NBER Working Paper No. 31161, April 2023" | **preprint** | **no** |
| Gemini · 2.5-flash | **14%**, **35%**, **5,179** agents | "NBER Working Paper No. 31161" — *while stating it was later published in the QJE in May 2025, and citing oup.com* | **preprint** | yes |
| ChatGPT · gpt-4.1-mini | **14%**, plus **34%** for novices | "the working paper … published by NBER in April 2023" | **preprint** | yes — 4 citation slots, **all nber.org** |

**Four systems out of four returned the preprint. Not one cited the version of record.**

Perplexity returned eight citations — NBER twice, MIT Sloan, arXiv, a personal reading-list page, the
US Bureau of Labor Statistics, a law firm's blog, and a LinkedIn post — and not one of them is the
*QJE* article. Claude reported `web_search: false` despite search being requested, answered from
parametric memory, and produced "about 35%", a figure that appears in neither version. ChatGPT
retrieved, filled four citation slots, and every one of them was `nber.org` — `domains_unique` = 1
against `citations_n` = 4, which is the repetition-as-breadth pattern the coding scheme was built to
catch.

**Gemini is the case that settles the mechanism.** It stated in its own answer that the paper *"was
later published in The Quarterly Journal of Economics in May 2025."* It cited `oup.com`. It knew the
version of record existed, named it, and linked to its publisher — and then reported 14%, 35% and
5,179 agents, which are the preprint's numbers. The published paper says 15% and 5,172. This is not a
model that failed to find the paper. It is a model that found both and built its answer on the one
the web points at.

## Why this changes the piece

The controls were included to make the design falsifiable. **The first control failed on all four
systems, in a way the myth questions cannot.** A clean, peer-reviewed, top-five-journal version of
record exists, is indexed, and has a DOI — and the retrieval layer returned the three-year-old
preprint with numbers the authors themselves revised.

The failure is also *uniform*, which matters more than it being total. Four systems, four different
retrieval stacks, one shared answer. That rules out an idiosyncratic index and points at the thing
they have in common: the public link graph.

The mechanism is not model failure and should not be written as one. The working paper is **free,
open, and has had three years to accumulate links**. The version of record sits behind Oxford
University Press. The web's link graph rewards the preprint, so the preprint is what comes back —
and with it, the superseded numbers.

That generalises to every paper whose preprint is open and whose version of record is paywalled,
which in economics is most of them. **The number you get is the number that was free.**

That is a better, narrower and more mechanistic finding than "commercial statistics are unsourced",
and it arrived from the question included specifically to disprove the thesis.

## What the pilot does not establish

Stated plainly, because the next runs are designed to test it and not to confirm it.

- **One question, one date, one run each.** Four systems agreeing on one paper is four data points,
  not a rate. The counted dataset runs three times per system per question for exactly this reason.
- **This paper may be unusually preprint-heavy.** *Generative AI at Work* circulated for two years as
  a working paper in a field where that is normal, and was widely read before publication. Q5 (BCG ·
  *Organization Science*) is the immediate test: same shape, different discipline, shorter preprint
  life.
- **The mechanism is inferred, not measured.** "The link graph rewards the preprint" is the best
  available explanation for four-of-four, but the pilot did not count inbound links to either
  version. If the essay asserts the mechanism it has to measure it — which is cheap, and should be a
  named step rather than a rhetorical flourish.

---

## Q5 · ground truth, established 2026-07-27 **before** the run

Written down first, deliberately. The Q4 cell was coded against a ground truth that already existed
in the verified bank; this one had to be built, and building it after seeing four answers would be
coding on the data.

**Version of record.** Dell'Acqua, McFowland III, Mollick, Lifshitz, Kellogg, Rajendran, Krayer,
Candelon & Lakhani (2026). *Organization Science* **37**(2), 403–423.
DOI `10.1287/orsc.2025.21838`. Pagination confirmed against Crossref, not inferred.

**Preprint.** Harvard Business School Working Paper **24-013**, 22 September 2023. Also on SSRN as
`4573321`.

| Claim | Preprint (2023) | Version of record (2026) |
| --- | --- | --- |
| Subjects | 758 **consultants**, "about 7% of the individual contributor-level consultants" | 758 **knowledge workers** |
| Tasks completed | 12.2% more | 12.2% more — **unchanged** |
| Speed | 25.1% more quickly | 25.1% more quickly — **unchanged** |
| **Quality** | **"more than 40% higher quality compared to a control group"** | **The 40% figure does not appear anywhere in the paper.** Table 4 reports **33.9%** (GPT + overview) and **29.9%** (GPT only) |
| **Skill split** | **"those below the average performance threshold increasing by 43% and those above increasing by 17%"** | **Neither figure appears anywhere in the paper** |
| Outside the frontier | 19 **percentage points** less likely to be correct | Abstract says "19% less likely"; the body still says "an average decrease of 19 percentage points" |

**Why this is a stronger version test than Q4.** The productivity headlines are byte-identical across
versions, so no system can be caught out on those. What changed is the *quality* result — and "40%
higher quality" is the single most-quoted number this study produced. Peer review cut it to roughly
30–34% and removed it from the abstract, and deleted the 43%/17% skill-equalisation claim from the
paper entirely.

So Q5 discriminates cleanly. **Any answer containing "40% higher quality", "43%" or "17%" is built on
the preprint, whatever it cites.** An answer citing the *Organization Science* article while quoting
40% is citing a document that does not contain its own number — which is a `provenance_error`, not a
version difference, and is coded as both.

**New guard for the verified bank, `AI-X5`:** *never cite the BCG/Dell'Acqua study as "40% higher
quality" or repeat the 43%/17% skill split — all three are preprint-only and were removed in peer
review. The published quality effects are 33.9% and 29.9% (Table 4).*

## Q5 · the run, 2026-07-27, all four systems

One run each, same prompt, web search requested on all four.

| System | Quality figure returned | Outside-frontier figure | Named as the original source | `version` | `search` |
| --- | --- | --- | --- | --- | --- |
| Perplexity · sonar | **40%** | **23%** | *"The original BCG source is **How People Create and Destroy Value with Generative AI** from Boston Consulting Group"* | preprint | yes |
| ChatGPT · gpt-4.1-mini | **40.2%** | **23%** | Never names the paper. Credits "a study conducted by the Boston Consulting Group" | preprint+ | yes |
| Claude · sonnet-4-5 | **"40 percent of the trial group produced higher quality results"** | 19% | "a working paper titled *Navigating the Jagged Technological Frontier*, September 22, 2023" | preprint | yes |
| Gemini · 2.5-flash | **40%** | not given | "a working paper … published by Harvard Business School. The lead author … is Fabrizio Dell'Acqua" | preprint | yes |

**All four returned the deleted number. Not one of the four cited the paper, in any version.**

That second sentence is the finding. On Q4 the systems at least linked to NBER. Here the citation
sets contain a student newspaper, LinkedIn posts, a Facebook post, a YouTube video, a legal-industry
trade site, an Indian national newspaper, a career-assessment quiz site, a training vendor, general
tech press — and the sponsoring firm's own marketing. Zero links to *Organization Science*, zero to
the SSRN record, and one to an HBS landing page.

### Citation sets, recorded in full so the coding can be disagreed with

- **Perplexity** — 18 slots, **13 unique domains**: bcg.com (×4, incl. media-publications.bcg.com),
  jobcannon.io, thecrimson.com, facebook.com, themelan.com, hbs.edu, venturebeat.com,
  innovationtraining.org, youtube.com, linkedin.com (×3), mi-3.com.au, medium.com, forbes.com.
- **ChatGPT** — **2 slots, 2 domains**: jobcannon.io, bcg.com. Neither is the paper.
- **Claude** — 8 slots, **3 unique domains**: thecrimson.com (×5), linkedin.com, aibusiness.com.
- **Gemini** — 7 slots, **6 unique domains**: innovationtraining.org (×2), mi-3.com.au, jobcannon.io,
  legaldive.com, hindustantimes.com, venturebeat.com.

`interested_share` is **left uncoded** for this cell. The protocol requires each domain to be
classified from its own homepage rather than from memory, and that pass has not been run. What is
already established: **bcg.com is the study's commissioning sponsor publishing about its own study**,
and it sells the advisory service the finding supports. `jobcannon.io` was checked and is a career-
assessment quiz site — under the strict rule it does **not** count as interested, because it does not
sell in the category the claim supports. Recording that is the point: the rule was written to bite
against the coder's convenience, and here it did.

### Three separate corruption mechanisms, all checkable

**1 · Version lag.** Peer review cut the quality result to 33.9% and 29.9% and removed the 40% claim
from the paper. Four of four returned 40%. Same mechanism as Q4, now replicated on a second paper in
a different discipline — which was the specific thing Q5 was run to test.

**2 · Sponsor substitution.** BCG's own publication says *"their performance was 23% worse than those
who didn't use the tool at all"* and *"a level of performance that was 40% higher."* The peer-reviewed
paper says **19 percentage points** and 33.9%/29.9%. Two of four systems reported BCG's **23%** as the
study's finding, and Perplexity named BCG's marketing page as *"the original source"* in those words.
The commissioning firm's promotional page is outranking, and outranking as *provenance*, the
peer-reviewed report of its own experiment.

**3 · False precision, acquired downstream.** `40.2%` appears in **neither** version of the paper —
verified against both full texts. The preprint says "more than 40%" (Table 1: 42.5% and 38%); the
published paper says 33.9% and 29.9%. The figure `40.2%` appears on **jobcannon.io**, a career-quiz
site running a research-statistics page keyed to the working-paper number. ChatGPT returned "40.2%
higher in quality" and cited that page.

A number that was a rounded floor over two conditions became a decimal, on a page whose purpose is to
be found, and came back with a citation attached. **The decimal point is doing the work the evidence
used to do.**

### The one error that is not the assistant's

Claude returned *"40 percent of the trial group produced higher quality results"* — a magnitude
silently converted into a headcount, which is a different claim about a different quantity. It cited
the Harvard Crimson. The Crimson says, verbatim: *"Additionally, 40 percent of the trial group
produced higher quality results."*

Claude reproduced its source faithfully and cited it correctly. **The error is the newspaper's.** That
distinction has to survive into the essay, because it is the whole argument: this is not a story about
models hallucinating. It is a story about a retrieval layer that is accurate with respect to a corpus
that is wrong, and a paper two clicks away that nothing had reason to open.

(Claude also gave "750 consultants" against the true 758. That sentence is cited to a LinkedIn Pulse
post, not to the Crimson, which states 758 correctly. Whether the error originates in the LinkedIn
post or in the answer is **not established** and is recorded as open.)

---

# Appendix B — the surface arm, 2026-07-27

Added because the protocol's own limitation section demanded it: every run above went to an **API
endpoint**, and the essay's premise is about the tool people actually use. So both control questions
were put to the **consumer ChatGPT interface**, same prompt, same day.

The result changed the finding.

## Q5 · consumer surface

It cited **`pubsonline.informs.org/doi/10.1287/orsc.2025.21838`** — the version of record — four times,
alongside the HBS working-paper PDF. It named both versions correctly: *"Harvard Business School
Working Paper No. 24-013 (2023), later published in Organization Science (2026)."* It got the
outside-frontier result right in the correct units — **"19 percentage points"** — which is more precise
than the published abstract manages.

**And it still reported "more than 40% higher in quality" — citing the *Organization Science* page for
it.** The published paper does not contain that claim.

## Q4 · consumer surface

Better still. It reported the preprint's 14% and 5,179, quoted the working paper's abstract verbatim,
named NBER WP 31161 as the original source — which for a question asking for the *original* source is
defensible — and then added, unprompted:

> *"A later peer-reviewed version published in The Quarterly Journal of Economics reports a **15%**
> average productivity increase using a slightly revised sample and analysis."*

That is close to a pass. It is the only response in the study to volunteer the revision.

## The confound, and how it was broken

The obvious objection: the API runs used `gpt-4.1-mini` while the consumer product serves something far
stronger. So the difference might be **model capability**, not **surface**.

Tested directly. Same question, same API, **`gpt-5.5`** — the strongest model available, with reasoning
enabled, at roughly **three times the cost per call**:

| | Cites the version of record? | Quality figure | Outside-frontier figure | Citations |
| --- | --- | --- | --- | --- |
| API · gpt-4.1-mini | no | **40.2%** (exists nowhere) | 23% (BCG's) | jobcannon.io, bcg.com |
| API · **gpt-5.5** | **no — never mentions a published version at all** | over 40% (preprint) | 23% (BCG's) | hbs.edu preprint PDF, bcg.com |
| Consumer ChatGPT | **yes — DOI, four times** | over 40% (preprint) | **19 percentage points** ✓ | informs.org, hbs.edu |

**The stronger, more expensive, reasoning-enabled model did not find the version of record. The
consumer product did.** Capability is not the explanation. The retrieval stack around the model is.

### The artefact worth keeping

`gpt-5.5` returns a visible reasoning summary. On the way to its answer it recorded:

> *"There's a PDF that likely contains relevant figures, like a 12.2% increase in tasks completed,
> 25.1% faster performance, and over 40% improved quality. **I can cite search14 even if it's not fully
> opened yet.**"*

The model states that it will attach a citation to a document it has not read, on the strength of what
it expects that document to contain. It then does exactly that — and the figure it expected to find is
the one peer review removed.

(Reported as what it is: a vendor-surfaced reasoning *summary*, not raw internal state. It is an
observed output and quotable as such.)

## What the surface arm changes

The finding is no longer "retrieval returns the preprint." It is two independent failures that have to
be separated:

1. **Citation** — largely fixed on the consumer surface. It found the version of record, named both
   versions, and linked the DOI.
2. **Content** — not fixed anywhere. Every surface, every model, every run reported a quality figure
   that peer review deleted.

And the combination is the point. On the API, a wrong number arrives cited to a career-quiz site, and a
careful reader can smell it. On the consumer surface, **the same wrong number arrives cited to
Organization Science with a DOI.** It looks impeccable. It is not.

**Better citation practice made the error harder to catch, not less likely to occur.**

## Limits of this arm, stated plainly

- **One system.** Only ChatGPT has a consumer-interface scrape available here. Nothing in this arm
  supports a claim about consumer Claude, Gemini or Perplexity.
- **One run each**, on one day, from one location (United States).
- The API/consumer comparison holds **model constant only in the gpt-5.5 row**. That row is what
  breaks the confound; the rest is descriptive.

---

---

# Appendix C — repeat runs

Collected after the pilot cells, to test the protocol's own falsification condition:
*"Repeat runs disagree wildly. If the same model gives a different `source_class` across three
runs, the headline is variance, not provenance, and it must be reported as such."*

It does not disagree wildly. It disagrees in exactly one place, and that place is informative.

**Run count, stated honestly: two usable runs per system per question, not three.** The protocol asks
for three. Neither cell is complete and neither is described as complete.

---

## Q4 · "What did Brynjolfsson, Li and Raymond find…"

Version of record: *QJE* 140(2), 889–942 (2025) — **15%**, 5,172 agents.
Preprint: NBER WP 31161 (April 2023) — **14%**, 5,179 agents, 34% novices.

| System | Run A | Run B | Cites the version of record |
| --- | --- | --- | --- |
| Perplexity · sonar | 14%, 34% → NBER WP | 14% → NBER WP 31161 | **0 / 2** |
| Claude · sonnet-4-5 | **15% flagged, QJE 140(2), 889–942 cited** | **15% flagged, QJE 140(2), 889–942 cited** | **2 / 2** |
| Gemini · 2.5-flash | 14%, "35% or 38%", 5,179 → NBER WP | 14%, "34% to 35%", 5,179 → NBER WP | **0 / 2** |
| ChatGPT · gpt-4.1-mini | 14%, 34% → NBER, 3 slots all nber.org | 14%, 34% → NBER, 2 slots all nber.org | **0 / 2** |

**Claude passes cleanly, twice.** Verbatim, run B:

> *"It's worth noting that the published version of this paper in The Quarterly Journal of Economics
> (2025) reports a slightly higher figure of 15% on average, likely due to revisions and additional
> data analysis between the working paper and final publication."*

It reports **both** figures, names both versions, cites `academic.oup.com/qje/article/140/2/889/7990658`,
and explains the difference. That is the correct answer, and it is the only system that produced it.

This matters more than a uniform failure would. It establishes that **the version of record is
reachable** — no system can be excused on the grounds that the paywall made it impossible. One
retrieval stack found it, twice, unprompted. The others did not look.

Gemini's error is separate and worth keeping: "35% or 38%" (run A) and "34% to 35%" (run B) for the
novice effect. The preprint says **34%**. Neither 35% nor 38% is in either version. The figure is
drifting run to run.

---

## Q5 · "What did the BCG study find about consultants using GPT-4?"

Version of record: *Organization Science* 37(2), 403–423 (2026) — quality **33.9% / 29.9%**, no 40% claim.
Preprint: HBS WP 24-013 (2023) — **"more than 40% higher quality."**

| System | Run A | Run B | Cites the version of record |
| --- | --- | --- | --- |
| Perplexity · sonar | 40%, 23% — *"The original BCG source is How People Create and Destroy Value with Generative AI"* | 40%, 23% — *"The original source is BCG's own publication"* | **0 / 2** |
| ChatGPT · gpt-4.1-mini | **40.2%**, 23% → jobcannon.io + bcg.com | **40.2%**, 23% → BCG's PDF only, 3 slots 1 domain | **0 / 2** |
| Claude · sonnet-4-5 | "40 percent of the trial group" → the Harvard Crimson | **search failed**; answered from memory, 40% | **0 / 2** |
| Gemini · 2.5-flash | 40% → names WP, cites none of it | 40% → names WP 24-013, cites none of it | **0 / 2** |
| ChatGPT · **gpt-5.5** | 40%, 23% → HBS preprint PDF + bcg.com | — | **0 / 1** |

**Nine API runs. Zero cited the version of record. Every one carried the 40% figure peer review deleted.**

### Two things the repeats established that one run could not

**1 · The 40.2% is stable, and it has come loose from its source.** Run A cited `jobcannon.io`, which
does state 40.2%. Run B cited **BCG's own PDF**, which states **40%**. The same model produced the same
invented decimal twice and attributed it to two different documents, only one of which contains it.
A figure that began as "more than 40%" over two experimental conditions is now circulating with a
decimal place and no fixed origin.

**2 · Perplexity's sponsor substitution is not a slip.** Both runs name BCG's marketing publication as
*the original source*, in those words. It is the stable behaviour of that stack on this question.

### A data-quality problem I am not hiding — now closed

Claude returned `web_search: false` on **2 of its 6 runs** across both questions, despite search being
requested — once saying so in the answer: *"I don't have access to web search at the moment."* Those
runs measure parametric memory, not retrieval, and are excluded from the counted cells. This looks
like intermittent tool provisioning at the endpoint rather than a property of the product.

**Re-run 2026-07-27 17:44, search confirmed active.** The cell now has two valid retrieval runs and
they agree closely:

- *"40 percent of the trial group produced higher quality results"* — again, cited to the Harvard
  Crimson. The propagation of the newspaper's category error is **stable across runs**, not a
  one-off.
- **New in this run, and worse:** *"the lowest performers had the biggest boost — 43% increase versus
  17% for top performers"*, cited to `aibusiness.com`. Those are the **preprint-only skill-split
  figures that peer review removed from the paper entirely** (`AI-X5`). Claude now returns *both*
  deleted artefacts — the 40% and the 43/17 split.
- *"750 Boston Consulting Group consultants"* — again cited to the same LinkedIn Pulse post, in both
  runs. Consistent attribution across two independent runs makes that post the likely origin of the
  error, though it has not been checked at source.
- Citations: `thecrimson.com`, `aibusiness.com`, `linkedin.com`, `axios.com`. **Four domains, none
  academic.** The trade headline it leans on is literally *"Harvard Study: GPT-4 Boosts Work Quality
  by Over 40%"* — the deleted figure encoded in a URL slug.

**Q5 now stands at 9 valid retrieval runs across five model configurations. Zero cited the version of
record. Every one carried a figure the published paper does not contain.**

---

## Why Q4 is passable and Q5 is not

Both papers have an open preprint and a paywalled version of record. Only one gets cited correctly.
The difference is not the paywall — it is **age and path**.

- *Generative AI at Work* has been the version of record since **May 2025**. The NBER landing page
  links forward to it, `academic.oup.com` is indexed, and three years of citations have accumulated
  pointing both ways.
- The *Organization Science* article appeared in **March 2026** — four months before these runs. Its
  preprint has had **three years** to gather links under a working-paper number that trade press,
  vendors and SEO pages all cite.

So the mechanism sharpens: it is not "paywalled papers lose." It is **"the version of record has to
out-age its own preprint before retrieval finds it"** — and during that window, which is years, the
superseded numbers are the answer everyone gets.

That window is where every recently-published finding lives.

---

## What still has to happen before any of this is published as a rate

- A third run on every cell. Two is not three.
- Claude's Q5 cell re-run with search actually enabled.
- The `interested_share` domain pass, from each domain's own homepage.
- The inbound-link count, preprint versus version of record, for both papers — the mechanism above is
  still an inference and the essay must not state it as measured until it is.


---

# Appendix D — the link graph, measured

The protocol committed to this before the essay could state a mechanism: *"'The link graph rewards
the free version' is currently the best explanation for eight of eight, and it is still an
explanation rather than a measurement."*

Now measured. Source: DataForSEO Backlinks, bulk page summary, live index, 2026-07-27.

---

## Q4 · Brynjolfsson, Li & Raymond

| Version | Referring domains | Backlinks | First seen |
| --- | --- | --- | --- |
| **Preprint** — `nber.org/papers/w31161` | **1,884** | 4,585 | 2023-05-01 |
| **Version of record** — `academic.oup.com/qje/article/140/2/889/7990658` | **631** | 1,218 | 2025-02-24 |

**Preprint advantage: 3.0× referring domains, 3.8× backlinks, a 21-month head start.**

## Q5 · Dell'Acqua et al.

| Version | Referring domains | Backlinks | First seen |
| --- | --- | --- | --- |
| **Preprint** — SSRN `4573321` | **1,058** | 2,141 | 2023-09-17 |
| **Preprint** — HBS landing page | **606** | 1,154 | 2023-09-28 |
| **Preprint** — HBS working-paper PDF | **511** | 1,337 | 2023-10-15 |
| **Version of record** — `pubsonline.informs.org/doi/10.1287/orsc.2025.21838` | **53** | 100 | 2026-03-14 |

**Preprint advantage: 20× referring domains against SSRN alone**, and the preprint exists on *three*
separately-linked surfaces, each of which individually outweighs the version of record by an order of
magnitude. The version of record is four months old.

---

## Why this settles the mechanism

The two questions differ in outcome, and the link data predicts the difference precisely.

| | Preprint : VoR referring domains | VoR age at run | Systems citing the VoR |
| --- | --- | --- | --- |
| **Q4** | 3 : 1 | 17 months | **1 of 4** (Claude, on both runs) |
| **Q5** | 20 : 1 | 4 months | **0 of 4**, and 0 of 9 runs |

Where the version of record has had time to accumulate a third as many referring domains as its
preprint, one retrieval stack finds it. Where it has a twentieth, none does.

**This is no longer "paywalled papers lose."** Both versions of record are paywalled; both preprints
are free. What separates them is **link mass and age**. The claim the essay can now make is narrow,
mechanistic and measured:

> A published paper has to out-age its own preprint in the link graph before retrieval will find it.
> Until it does — and that takes years — the numbers you get are the ones the authors already revised.

For Q5 that window is still open, four months after publication and three years after the preprint.
Every finding published this year is inside it.

## A detail that needs interpreting, not asserting

The *Organization Science* page reports **85 of its 100 backlinks as broken**, and `broken_pages: 1`.
On the other five URLs the field is absent. That is consistent with the publisher returning an error
or a block to crawlers, which would compound the link deficit — a version of record that is both
sparsely linked *and* hard to crawl. **It is also consistent with an artefact of one crawler's
access.** It is recorded here as observed and is not used to support any claim until it is checked
against a second source.

## Limits of this measurement

- **One commercial crawler's index, not the web.** DataForSEO's view is a proxy. It is not the index
  any of these four assistants retrieves from, and no vendor publishes theirs.
- **Referring domains are not retrieval rank.** The correlation across two questions is strong and
  directionally right; two points is not a model.
- **Single snapshot**, 2026-07-27. The whole point of the finding is that these numbers move.
- The comparison is **URL-level**. Both papers are reachable at other addresses — repositories,
  mirrors, author pages — that are not counted here. Counting more preprint surfaces would widen the
  gap, not narrow it, so the figures above are conservative in the direction of the argument, which
  is the safer direction to be wrong in.


## Where the study stands after two control cells

Both controls have now run once on all four systems. Both failed, on eight of eight system-question
cells, in the same direction. The four questions included because their provenance was expected to be
bad **have not been run at all** — and on the evidence so far they are the least informative
questions in the set.

The design was built so the controls could kill the thesis. They replaced it with a better one.

**Still to run:** Q4 and Q5 need their two remaining repeat runs each before either enters the counted
dataset — the cells above are single runs and stay labelled as pilots. Then the four non-control
questions, three runs each.

**Added to the queue by these results, and not optional if the essay asserts the mechanism:** count
the inbound links to the preprint versus the version of record for both papers. "The link graph
rewards the free version" is currently the best explanation for eight of eight, and it is still an
explanation rather than a measurement.