← Research

Replication packagePre-registeredRe-sealed 19 Aug 2026CC BY 4.0

Retrieval audit: the data

Everything behind the essay: the protocol as it was registered before the runs, every response with its timestamp and model identifier, the link counts, and the script. If you think the conclusion is wrong, this is what you need to show it.

Get the files Cite the data Read the essay

Appended to every question

“Give the figure and cite the original source.”

Claude
sonnet-4-5
Gemini
2.5-flash
ChatGPT
gpt-4.1-mini
Perplexity
sonar
Four assistants, web search enabled, identical prompts. Every record says how it was collected, in collected_via.
Assistants
4asked the same questions
Questions run
3of six registered; two are controls
Counted responses
36three runs per system, all hand-coded
Licence
CC BY 4.0Reuse it, including to disagree with it.

What the study did

Built so that it could fail.

Four assistants were asked the same six questions, each ending “Give the figure and cite the original source.” Two of the six were controls: questions about papers with a clean, peer-reviewed version of record, included so that the design could fail in the direction that would have narrowed the thesis.

It failed in the other direction. Across thirteen runs on one control, every answer carried a quality figure that the published paper does not contain, and none cited the published paper. The counts behind that, and the referring-domain measurements that explain it, are in the files below.

The protocol was written down first, including the section headed what would make this finding wrong. Every change made after the first run is dated in its changelog, with the reason.

The control cells

Three of twenty-four counted runs cite the version of record.

Both control papers have an open preprint and a paywalled version of record. One stack reached the published paper on every run, which proves it is reachable. No stack reached the other.

Figure 1Did the answer cite the version of record?

Cites itThe published figures, no link to themDoes not

Version-of-record citations per counted run, by system and control question
System Q4 · Brynjolfsson, Li & RaymondQJE 140(2), 2025: 15%, 5,172 agentsQ5 · Dell'Acqua et al.Organization Science 37(2), 2026: 33.9% and 29.9% Q5, the stable behaviour
Run 3Run 4Run 5Run 1Run 3Run 4
Claudesonnet-4-5 The Crimson’s category error, 2 of 3
Gemini2.5-flash 40%, 2 of 3
ChatGPTgpt-4.1-mini The invented decimal, 40.2%, 3 of 3
Perplexitysonar Sponsor substitution, 3 of 3

Counted runs only: verbatim answer text, citations and a timestamp in responses.jsonl, collected 27 Jul 2026. Twelve Q5 runs carried a quality figure the published paper does not contain; with the gpt-5.5 run it is thirteen, and the count does not move.

repeat-runs.md

Q1The question with no answer to find: twelve runs, 121 citation slots, 23 unique domain labels, zero primary sources. No primary document reports the ratio it asks for.

Gemini, Q4 run 515% and 5,172, both version-of-record figures, and the QJE named, with no link to it. Correct content, absent provenance.

The mechanism, measured

A published paper has to out-age its own preprint.

The difference is not the paywall: both versions of record are paywalled and both preprints are free. It is link mass and age.

Figure 2Referring domains, preprint against version of record

Version of recordPreprint

Source: DataForSEO Backlinks, bulk page summary, live index, 27 Jul 2026. One commercial crawler’s index, not the web, and not the index any of these assistants retrieves from.

link-graph.md

Files

Seventeen files, sixteen of them fingerprinted.

Plain text and JSON throughout. Nothing here needs an account or a tool to open. Every file carries the start of its SHA-256 from the manifest.

  • The study, as registered
  • protocol.mdsha256 96ff370b8210…The pre-registered protocol, its changelog, and the four appendices: the control pilot, the surface arm, the repeat runs and the link-graph measurement.
    71 kB Download
  • questions.jsonsha256 18e6de772c80…The six questions, the instruction appended to each, and the ground truth established before each question was run.
    8 kB Download
  • The responses and their coding
  • responses.jsonlsha256 7285e0b7023e…The collected responses, one JSON object per line, with timestamps, model identifiers and citation lists. Records collected before the harness existed carry answer_text: null and say so.
    129 kB Download
  • coding.jsonsha256 7fe1a930e1d7…The hand-coded values. Counts are computed from responses.jsonl, not typed.
    35 kB Download
  • coding-sheet.mdsha256 d03f3f53c220…The human-readable sheet generated from coding.json: 36 counted responses, all hand-coded.
    33 kB Download
  • repeat-runs.mdsha256 c25c6b5aab44…The repeat runs, and the run-to-run variance: including the one system that cited the version of record on all three of its runs.
    9 kB Download
  • q4-runs.jsonsha256 9795682afee0…The Q4 cell in full: figures returned, sources named, citation sets and verdicts per run.Superseded: kept as a record, never counted
    7 kB Download
  • The mechanism
  • link-graph.mdsha256 f6f3b0ff23c7…Referring domains and backlinks for each paper’s preprint and version of record, with the URLs counted.
    4 kB Download
  • The second coder
  • second-coder-packet.jsonsha256 03355dda0676…The packet given to the second coder.
    23 kB Download
  • second-coder-codes.jsonsha256 75496e2f0654…The second coder’s codes.
    2 kB Download
  • second-coder-pass.mdsha256 d21733032543…The pass itself, run by a language model rather than a human reader. It covers 9 of 36 rows, 43 of 54 field comparisons agree, the disagreements stay unreconciled, and no inter-coder reliability figure is claimed.
    8 kB Download
  • The scripts
  • run_audit.pysha256 487e30b591f4…The runner. Calls four assistants, writes every raw response to JSONL, codes nothing. --dry-run prints the exact payloads without calling anything.
    12 kB Download
  • make_coding_sheet.pysha256 584899d0abc0…Regenerates coding-sheet.md from coding.json.
    8 kB Download
  • append.pysha256 08709f41cd21…Appends collected runs to responses.jsonl, flagging the pre-registered discriminators.
    1.9 kB Download
  • domains.pysha256 482548a15a45…Reduces a citation to its registrable domain: from the URL where there is one, from the label where there is not.
    0.6 kB Download
  • Read me, then verify
  • README.mdsha256 7426178dd1c6…How to re-run it, what the results were, and what the study does not establish.
    25 kB Download
  • SHA256SUMS.txtthe manifest itselfSHA-256 for every file above, so you can tell whether what you downloaded is what was published. Verify with sha256sum -c SHA256SUMS.txt after downloading the set.
    1.3 kB Download

Re-running it

Roughly the price of lunch, and an afternoon.

With your own API keys, the whole thing costs roughly the price of lunch and takes an afternoon.

From the package folder
export OPENAI_API_KEY=...  ANTHROPIC_API_KEY=...
export GEMINI_API_KEY=...  PERPLEXITY_API_KEY=...

python3 run_audit.py --dry-run              # see the payloads, call nothing
python3 run_audit.py --only Q4,Q5 --runs 1  # the two controls, once each
python3 run_audit.py --runs 3               # the full set

These four vendors change their web-search parameters more often than they change their models. A 4xx means a builder function needs updating against current documentation; nothing else in the harness depends on it.

What this does not establish

Stated here rather than left for a critic to find.

  • Run counts. Both control questions have three valid runs on all four systems. One Gemini run was discarded and re-run after a length constraint was appended to the frozen prompt; the discard is logged in the changelog rather than quietly dropped.

  • The collected runs went through a third-party endpoint, not through this script’s direct vendor calls. Every record says so in collected_via. A reader re-running it with their own keys is testing the same question against a slightly different path.

  • The link counts are one commercial crawler’s index, not the web, and not the index any of these assistants retrieves from. No vendor publishes theirs.

  • Six questions, chosen against stated criteria: not sampled from a population. Two consistent observations of a mechanism, not a measurement of a field.

  • interested_share is uncoded. The scheme requires each domain to be classified from its own homepage, and that pass has not been run.

Version history

Superseded, never rewritten.

The pre-registered protocol is registered, and registered documents are superseded rather than rewritten. Every change to the package is dated here.

  1. Documentation correction

    The current counted result is Appendix F, coding.json and coding-sheet.md: 36 counted responses across Q1, Q4 and Q5, with three runs per system. Earlier appendix text is retained as history.

  2. Integrity re-seal

    The manifest was regenerated against all 16 files after a route and typography sweep changed package files. No response records or coding values changed.

  3. Re-sealed in full

    Everything published: every collected response, the hand-coded values and the sheet generated from them, the second-coder packet, its codes and the pass itself. The manifest went from eight files to sixteen.

  4. First published

    A partial snapshot: 11 of the 47 collected responses, and no coding.

Licence and citation

Reuse it, including to disagree with it.

Cite the data

Isoglu, S. (2026). Retrieval-provenance audit: what four assistants return when asked where a number comes from. isoglu.com.

CC BY 4.0

Released under CC BY 4.0.

Package
retrieval-audit
Published
27 Jul 2026
Current seal
19 Aug 2026, 16 files
Verify
sha256sum -c SHA256SUMS.txt
Also in German
Abruf-Audit

Elsewhere in the data

The argument, and the two packages beside it.

Contact

Bring the question you're working on.

Research, a guest lecture, or a commercial question.