Retrieval audit — the data

What the study did

Files

Plain text and JSON throughout. Nothing here needs an account or a tool to open.

• protocol.md 43 kB The pre-registered protocol, its changelog, and the four appendices: the control pilot, the surface arm, the repeat runs and the link-graph measurement.
• questions.json 6 kB The six questions, the instruction appended to each, and the ground truth established before each question was run.
• run_audit.py 12 kB The runner. Calls four assistants, writes every raw response to JSONL, codes nothing. `--dry-run` prints the exact payloads without calling anything.
• responses.jsonl 17 kB The collected responses, one JSON object per line, with timestamps, model identifiers and citation lists. Records collected before the harness existed carry `answer_text: null` and say so.
• repeat-runs.md 7 kB The repeat runs, and the run-to-run variance — including the one system that cited the version of record on all three of its runs.
• link-graph.md 4 kB Referring domains and backlinks for each paper’s preprint and version of record, with the URLs counted.
• q4-runs.json 6 kB The Q4 cell in full: figures returned, sources named, citation sets and verdicts per run.
• README.md 12 kB How to re-run it, what the results were, and what the study does not establish.
• SHA256SUMS.txt 717 B SHA-256 for every file above, so you can tell whether what you downloaded is what was published.

Verify with sha256sum -c SHA256SUMS.txt after downloading the set.

Re-running it

With your own API keys, the whole thing costs roughly the price of lunch and takes an afternoon.

export OPENAI_API_KEY=...  ANTHROPIC_API_KEY=...
export GEMINI_API_KEY=...  PERPLEXITY_API_KEY=...

python3 run_audit.py --dry-run              # see the payloads, call nothing
python3 run_audit.py --only Q4,Q5 --runs 1  # the two controls, once each
python3 run_audit.py --runs 3               # the full set

These four vendors change their web-search parameters more often than they change their models. A 4xx means a builder function needs updating against current documentation; nothing else in the harness depends on it.

What this does not establish

• Run counts. Both control questions have three valid runs on all four systems. One Gemini run was discarded and re-run after a length constraint was appended to the frozen prompt; the discard is logged in the changelog rather than quietly dropped.
• The collected runs went through a third-party endpoint, not through this script’s direct vendor calls. Every record says so in collected_via. A reader re-running it with their own keys is testing the same question against a slightly different path.
• The link counts are one commercial crawler’s index, not the web, and not the index any of these assistants retrieves from. No vendor publishes theirs.
• Six questions, chosen against stated criteria — not sampled from a population. Two consistent observations of a mechanism, not a measurement of a field.
• `interested_share` is uncoded. The scheme requires each domain to be classified from its own homepage, and that pass has not been run.