{
 "_comment": [
  "Hand-coded values against section 6 of the protocol. One record per counted response.",
  "citations_n and domains_unique are computed from responses.jsonl by make_coding_sheet.py and are not stored here.",
  "interested_share is null wherever the domain-by-domain pass (each domain classified from its own homepage) has not been run.",
  "Coded by: S. Isoglu, 2026-07-27. Second-coder pass: run 2026-07-28 by a language model, not a human reader: see second-coder-pass.md. The protocol's human requirement was waived by the author 2026-07-28.",
  "Counted API cells: Q1 runs 1-3, Q4 runs 3-5, Q5 runs 1/3/4: twelve per question, all with verbatim text in responses.jsonl. The four Q4 run-1 API records have answer_text: null and are NOT coded here; the gpt-5.5 run and both consumer-surface records are reported in protocol Appendix B (the surface arm and its confound table) and are not counted cells.",
  "version_returned uses one value beyond the protocol's registered vor/preprint/n-a: `preprint+`, for an answer built on the preprint that ALSO carries a figure appearing in neither version (the 40.2%). Registered in the changelog 2026-07-27 20:44, after the control pilot and before Q1.",
  "provenance_error_route (added 2026-07-27 21:0x, one value per provenance_error entry): `routed` = a document in that run's own citation set is established, checked at source, to contain the error; `not_established` = the source is absent from the citation set, was not checked at source, or the erroneous claim is the model's own with no document behind it. Rule fixed before classification. Result: 22 of 36 entries routed, 14 not established.",
  "2026-07-28: the full text of Reichheld & Sasser (1990) was supplied by the author and checked. Two claims the essays rest on are now verified against it rather than against secondary reports: (a) the article contains no acquisition-to-retention cost ratio; (b) its only acquisition-cost figure is the credit-card $51, with that scope. Pagination is still not established: the supplied text is the reprint rendering and carries no volume, issue or page numbers, so the reference ships without them."
 ],
 "coded_by": "S. Isoglu",
 "coded_utc": "2026-07-27",
 "second_coder": {
  "status": "run 2026-07-28, by a language model, NOT the independent human reader the protocol requires",
  "sample": "9 of 36 rows (25%), rule fixed before sampling",
  "agreement": "43 of 54 field comparisons (80%)",
  "report": "second-coder-pass.md: disagreements published unreconciled",
  "material_findings": [
   "source_class is underspecified: named-origin vs retrieved-documents. No source_class rate until the field is split.",
   "Two provenance_error counts are understated in this coding (Q4/chat_gpt/r5, Q5/chat_gpt/r4). Counts left as coded, with the note beside them.",
   "Six of the fourteen not_established route codes are Gemini entries, spread over four rows, unroutable because Gemini returns no citation URLs. The split needs that caveat wherever it is quoted. Corrected 2026-07-28: this note said five, and five is neither the entry count nor the row count.",
   "interested_share coded for the first time, on 9 rows only, by the second coder. No rate may be computed from it."
  ]
 },
 "rows": [
  {
   "q": "Q1",
   "system": "chat_gpt",
   "run": 1,
   "figure_given": "range",
   "figure_verbatim": "up to five times; 5–7 times; 5–25 times",
   "source_named": "yes",
   "source_named_verbatim": "According to Optimove … Joy Subscription notes … Artisan Strategies further elaborates",
   "source_class": "tertiary",
   "interested_share": null,
   "contra_as_support": "no",
   "provenance_error": [],
   "search_used": "yes",
   "version_returned": "n/a",
   "coder_note": "Three vendor pages offered in answer to a request for the original source. No document, author or study is named as an origin; the named entities are the pages it read."
  },
  {
   "q": "Q1",
   "system": "chat_gpt",
   "run": 2,
   "figure_given": "range",
   "figure_verbatim": "5 to 25 times higher; SaaS CAC $1,200 vs CRC $35",
   "source_named": "no",
   "source_named_verbatim": "Industry benchmarks indicate",
   "source_class": "tertiary",
   "interested_share": null,
   "contra_as_support": "no",
   "provenance_error": [],
   "search_used": "yes",
   "version_returned": "n/a",
   "coder_note": "No origin named at all. Both citation slots resolve to one growth-consulting blog; domains_unique 1 against citations_n 2. `source_class` codes the highest class of document offered in support, whether or not it is named as an origin; `none` is reserved for answers returning no citations at all. Under the alternative reading: `none` wherever no origin is named: this would be the one cell in the study where `source_class` varies across runs, which is the protocol's falsification trigger. The rule applied is stated so a reader can apply the other one."
  },
  {
   "q": "Q1",
   "system": "chat_gpt",
   "run": 3,
   "figure_given": "range",
   "figure_verbatim": "5 to 25 times higher; SaaS CAC $1,200 vs CRC $35",
   "source_named": "no",
   "source_named_verbatim": "depending on the industry and business model",
   "source_class": "tertiary",
   "interested_share": null,
   "contra_as_support": "no",
   "provenance_error": [],
   "search_used": "yes",
   "version_returned": "n/a",
   "coder_note": "Same shape as run 2, same single domain. `source_class` codes the highest class of document offered in support, whether or not it is named as an origin; `none` is reserved for answers returning no citations at all. Under the alternative reading: `none` wherever no origin is named: this would be the one cell in the study where `source_class` varies across runs, which is the protocol's falsification trigger. The rule applied is stated so a reader can apply the other one."
  },
  {
   "q": "Q1",
   "system": "claude",
   "run": 1,
   "figure_given": "range",
   "figure_verbatim": "5-25 times more",
   "source_named": "yes",
   "source_named_verbatim": "The original source for this widely referenced statistic is **Bain & Company research conducted by Frederick Reichheld**",
   "source_class": "tertiary",
   "interested_share": null,
   "contra_as_support": "no",
   "provenance_error": [
    "Names Bain & Company / Reichheld research as the origin of the 5–25x cost ratio. No Bain or Reichheld publication reports this ratio; ground truth established 2026-07-27 from primary documents."
   ],
   "search_used": "yes",
   "version_returned": "n/a",
   "coder_note": "The sentence tracks stealthagents.com, which states: 'The foundational figure here comes from Bain and Company research by Frederick Reichheld: acquiring a new customer costs 5 to 25 times more than retaining an existing one.' Checked at source 2026-07-27. The attribution is the page's, reproduced.",
   "provenance_error_route": [
    "routed"
   ]
  },
  {
   "q": "Q1",
   "system": "claude",
   "run": 2,
   "figure_given": "range",
   "figure_verbatim": "5-25 times more; 5x floor / 25x ceiling; retention +5% -> profits +25% to 95%",
   "source_named": "yes",
   "source_named_verbatim": "The statistic originates from research by **Frederick Reichheld at Bain & Company**",
   "source_class": "tertiary",
   "interested_share": null,
   "contra_as_support": "no",
   "provenance_error": [
    "Names Bain & Company / Reichheld as the origin of the 5–25x cost ratio; no such publication exists.",
    "'25% to 95%': checked against the full text of Reichheld & Sasser, 'Zero Defections' (HBR Sept-Oct 1990), supplied by the author 2026-07-28. The article gives the range three ways and none is 25-95%: the exhibit is titled 'Reducing Defections 5% Boosts Profits 25% to 85%'; the body reads 'Reducing defections by just 5% generated 85% more profits in one bank's branch system, 50% more in an insurance brokerage, and 30% more in an auto-service chain'; and the introduction says 'Companies can boost profits by almost 100% by retaining just 5% more of their customers.' The 95% ceiling is downstream drift."
   ],
   "search_used": "yes",
   "version_returned": "n/a",
   "coder_note": "The 5x-floor / 25x-ceiling sentence is near-verbatim stealthagents.com: 'The 5x floor is typical for transactional consumer businesses. The 25x ceiling shows up in complex B2B environments with long sales cycles and substantial onboarding costs.' Checked at source. The 5x-floor/25x-ceiling sentence is near-verbatim stealthagents.com. Separately, the full text of 'Zero Defections' was checked on 2026-07-28: it contains no acquisition-to-retention ratio of any kind, and its only acquisition-cost statement is 'In credit cards, for example, companies spend an average of $51 to recruit a customer and set up the new account.'",
   "provenance_error_route": [
    "routed",
    "not_established"
   ]
  },
  {
   "q": "Q1",
   "system": "claude",
   "run": 3,
   "figure_given": "range",
   "figure_verbatim": "5-25 times more",
   "source_named": "yes",
   "source_named_verbatim": "The original source for this widely-referenced statistic is **Bain & Company research conducted by Frederick Reichheld**",
   "source_class": "tertiary",
   "interested_share": null,
   "contra_as_support": "no",
   "provenance_error": [
    "Names Bain & Company / Reichheld research as the origin of the 5–25x cost ratio; no such publication exists."
   ],
   "search_used": "yes",
   "version_returned": "n/a",
   "coder_note": "Identical in substance to run 1. Same four domains, same named origin.",
   "provenance_error_route": [
    "routed"
   ]
  },
  {
   "q": "Q1",
   "system": "gemini",
   "run": 1,
   "figure_given": "range",
   "figure_verbatim": "5 to 25 times more expensive; 5 times (1990 HBR)",
   "source_named": "yes",
   "source_named_verbatim": "a 1990 Harvard Business Review article by Frederick Reichheld, cited the cost as 5 times more",
   "source_class": "tertiary",
   "interested_share": null,
   "contra_as_support": "no",
   "provenance_error": [
    "Attributes a 5x cost ratio to a 1990 HBR article by Reichheld. Reichheld & Sasser, 'Zero Defections: Quality Comes to Services' (HBR Sept–Oct 1990) contains no acquisition-to-retention cost ratio; acquisition cost appears only in absolute terms ('an average of $51 to recruit a customer').",
    "Names Reichheld alone; the 1990 article is co-authored with W. Earl Sasser Jr."
   ],
   "search_used": "yes",
   "version_returned": "n/a",
   "coder_note": "Names a real, checkable document, and the document does not contain the claim. Every citation resolves through a vertexaisearch redirect; the reader cannot see the destination without following it.",
   "provenance_error_route": [
    "not_established",
    "not_established"
   ]
  },
  {
   "q": "Q1",
   "system": "gemini",
   "run": 2,
   "figure_given": "range",
   "figure_verbatim": "5 to 25 times more expensive",
   "source_named": "yes",
   "source_named_verbatim": "widely cited and supported by research from Bain & Company",
   "source_class": "tertiary",
   "interested_share": null,
   "contra_as_support": "no",
   "provenance_error": [
    "Attributes the ratio to Bain & Company research. No Bain publication reports it."
   ],
   "search_used": "yes",
   "version_returned": "n/a",
   "coder_note": "Organisation named, no document. 12 citation slots resolve to 8 domains, all vendor or agency pages.",
   "provenance_error_route": [
    "not_established"
   ]
  },
  {
   "q": "Q1",
   "system": "gemini",
   "run": 3,
   "figure_given": "range",
   "figure_verbatim": "5 to 25 times more; CAC $200–$1,500 vs retention $15–$85; 80-95% less; retention +5% -> profits +25% to 95%; 60-70% vs 5-20%",
   "source_named": "yes",
   "source_named_verbatim": "stemming from research such as a 1990 Harvard Business Review article by Frederick Reichheld",
   "source_class": "tertiary",
   "interested_share": null,
   "contra_as_support": "no",
   "provenance_error": [
    "Attributes the ratio to the 1990 Reichheld HBR article, which does not contain it.",
    "'25% to 95%': downstream drift from the 25–85% range in Reichheld & Sasser 1990."
   ],
   "search_used": "yes",
   "version_returned": "n/a",
   "coder_note": "Longest answer in the cell and the one with the most unsourced numbers: five further figures appear with no attribution of any kind.",
   "provenance_error_route": [
    "not_established",
    "not_established"
   ]
  },
  {
   "q": "Q1",
   "system": "perplexity",
   "run": 1,
   "figure_given": "range",
   "figure_verbatim": "5 times more; 5 to 25 times",
   "source_named": "yes",
   "source_named_verbatim": "The original source most often traced for this rule of thumb is **Frederick Reichheld's Bain & Company research**",
   "source_class": "tertiary",
   "interested_share": null,
   "contra_as_support": "yes",
   "provenance_error": [],
   "search_used": "yes",
   "version_returned": "n/a",
   "coder_note": "Hedged correctly: it reports the attribution as an attribution ('most often traced'), not as a fact, so no provenance error is coded. contra_as_support: three cited documents dispute the claim they are cited for. hashtagpaid.com ('the stat is…well, kind of old… one source traces its statistics to another source, which quotes another source'); churnmiser.com ('rarely to a controlled study you could reproduce'); releva.ai ('the specific number is dangerously outdated'). All three checked at source 2026-07-27."
  },
  {
   "q": "Q1",
   "system": "perplexity",
   "run": 2,
   "figure_given": "range",
   "figure_verbatim": "5 times more; 5 to 25 times",
   "source_named": "yes",
   "source_named_verbatim": "**Bain & Company / Frederick Reichheld** research, which later summaries say Harvard Business Review repeated as \"anywhere from 5 to 25 times more expensive\"",
   "source_class": "tertiary",
   "interested_share": null,
   "contra_as_support": "yes",
   "provenance_error": [],
   "search_used": "yes",
   "version_returned": "n/a",
   "coder_note": "The HBR paraphrase is accurate: Gallo's 2014 opening sentence reads 'anywhere from five to 25 times more expensive.' Same three contra documents cited."
  },
  {
   "q": "Q1",
   "system": "perplexity",
   "run": 3,
   "figure_given": "range",
   "figure_verbatim": "5 times more expensive; 5–25 times",
   "source_named": "yes",
   "source_named_verbatim": "later articles trace it to **Bain & Company research associated with Frederick Reichheld**, with subsequent writeups noting the claim was popularized through Harvard Business Review",
   "source_class": "tertiary",
   "interested_share": null,
   "contra_as_support": "yes",
   "provenance_error": [],
   "search_used": "yes",
   "version_returned": "n/a",
   "coder_note": "Third consecutive hedge. This cell is the only one in the question that never asserts an origin."
  },
  {
   "q": "Q4",
   "system": "chat_gpt",
   "run": 3,
   "figure_given": "yes",
   "figure_verbatim": "approximately 14%; 34% for less experienced agents",
   "source_named": "yes",
   "source_named_verbatim": "the working paper \"Generative AI at Work\" … published by the National Bureau of Economic Research in April 2023",
   "source_class": "primary",
   "interested_share": null,
   "contra_as_support": "no",
   "provenance_error": [],
   "search_used": "yes",
   "version_returned": "preprint",
   "coder_note": "Four citation slots, one domain. Third consecutive preprint answer; the QJE version is never mentioned."
  },
  {
   "q": "Q4",
   "system": "claude",
   "run": 3,
   "figure_given": "yes",
   "figure_verbatim": "14% (NBER WP); 15% (QJE 140(2), 889–942)",
   "source_named": "yes",
   "source_named_verbatim": "**Published Paper:** … *The Quarterly Journal of Economics*, Volume 140, Issue 2 (May 2025): 889–942",
   "source_class": "primary",
   "interested_share": null,
   "contra_as_support": "no",
   "provenance_error": [],
   "search_used": "yes",
   "version_returned": "vor",
   "coder_note": "Reports both versions, gives correct pagination, cites academic.oup.com, and explains the 14%/15% difference. Third clean pass out of three."
  },
  {
   "q": "Q4",
   "system": "perplexity",
   "run": 3,
   "figure_given": "yes",
   "figure_verbatim": "about 14% on average",
   "source_named": "yes",
   "source_named_verbatim": "their NBER working paper, **\"Generative AI at Work\"**",
   "source_class": "primary",
   "interested_share": null,
   "contra_as_support": "no",
   "provenance_error": [],
   "search_used": "yes",
   "version_returned": "preprint",
   "coder_note": "Six domains including the author's own PDF and BLS. No QJE link in any of the three runs."
  },
  {
   "q": "Q4",
   "system": "gemini",
   "run": 3,
   "figure_given": "yes",
   "figure_verbatim": "14%; 5,179 agents",
   "source_named": "yes",
   "source_named_verbatim": "NBER Working Paper No. 31161, in April 2023",
   "source_class": "primary",
   "interested_share": null,
   "contra_as_support": "no",
   "provenance_error": [],
   "search_used": "yes",
   "version_returned": "preprint",
   "coder_note": "Gives no novice percentage at all, where the narrative runs in Appendix C recorded drifting values ('35% or 38%', '34% to 35%'). Those runs are not counted; the drift reappears in counted run 4 as '13.8%'. 22 slots, 8 domains."
  },
  {
   "q": "Q5",
   "system": "chat_gpt",
   "run": 3,
   "figure_given": "yes",
   "figure_verbatim": "12.2% more tasks; 25.1% faster; 40.2% higher quality; 23% worse",
   "source_named": "no",
   "source_named_verbatim": "A Boston Consulting Group (BCG) study involving 758 consultants",
   "source_class": "tertiary",
   "interested_share": null,
   "contra_as_support": "no",
   "provenance_error": [
    "40.2% appears in neither the preprint nor the version of record. Traced to jobcannon.io, a career-assessment site, which is also the cited source here.",
    "23% is BCG's own marketing figure; the paper reports 19 percentage points."
   ],
   "search_used": "yes",
   "version_returned": "preprint+",
   "coder_note": "Second counted run at 40.2%. The paper is never named in any version. Two slots, two domains: a career-quiz site and the commissioning sponsor.",
   "provenance_error_route": [
    "routed",
    "routed"
   ]
  },
  {
   "q": "Q5",
   "system": "claude",
   "run": 3,
   "figure_given": "yes",
   "figure_verbatim": "12.2% more tasks; 25.1% quicker; 40% higher quality; 19% less likely",
   "source_named": "yes",
   "source_named_verbatim": "the working paper titled \"Navigating the Jagged Technological Frontier\" published on September 22, 2023",
   "source_class": "primary",
   "interested_share": null,
   "contra_as_support": "no",
   "provenance_error": [
    "'750 Boston Consulting Group consultants' against the true 758, cited to the same LinkedIn Pulse post as in counted runs 1 and 4.",
    "40% quality figure was removed in peer review; the published paper reports 33.9% and 29.9%."
   ],
   "search_used": "yes",
   "version_returned": "preprint",
   "coder_note": "Names the preprint correctly, including its date. Drops the Crimson headcount conversion ('40 percent of the trial group') seen in counted runs 1 and 4 and gives the magnitude form instead. Six slots, three domains, none academic.",
   "provenance_error_route": [
    "not_established",
    "routed"
   ]
  },
  {
   "q": "Q5",
   "system": "perplexity",
   "run": 3,
   "figure_given": "yes",
   "figure_verbatim": "12.2% more tasks; 25.1% faster; about 40% higher quality; 23% worse",
   "source_named": "yes",
   "source_named_verbatim": "The original source is BCG's report *How People Create and Destroy Value with Generative AI*",
   "source_class": "tertiary",
   "interested_share": null,
   "contra_as_support": "no",
   "provenance_error": [
    "Names the commissioning sponsor's own marketing publication as the original source of its own experiment.",
    "23% is BCG's figure; the paper reports 19 percentage points.",
    "40% was removed in peer review."
   ],
   "search_used": "yes",
   "version_returned": "preprint",
   "coder_note": "Second counted run naming BCG's own publication as the original source. Sponsor substitution is the stable behaviour of this stack on this question.",
   "provenance_error_route": [
    "not_established",
    "routed",
    "routed"
   ]
  },
  {
   "q": "Q5",
   "system": "gemini",
   "run": 3,
   "figure_given": "yes",
   "figure_verbatim": "12.2% more tasks; 25.1% faster; 40.2% higher quality; 758 consultants",
   "source_named": "yes",
   "source_named_verbatim": "a Harvard Business School Working Paper titled \"Navigating the Jagged Technological Frontier: …\" by Dell'Acqua et al., published in 2023",
   "source_class": "primary",
   "interested_share": null,
   "contra_as_support": "no",
   "provenance_error": [
    "40.2% appears in neither version of the paper. jobcannon.io, checked at source 2026-07-27, states '…produced results rated 40.2% higher in quality…' and is this run's top citation slot."
   ],
   "search_used": "yes",
   "version_returned": "preprint+",
   "coder_note": "Run 3 returns 40.2%: the fabricated decimal: with jobcannon.io in its top citation slot, where runs 1 and 4 both return 40%. Coded as a single occurrence, NOT as the decimal crossing stacks: run 4 went back to 40% with jobcannon.io still in the citation set. One in three is a fluctuation. Names the preprint correctly and cites nothing academic.",
   "provenance_error_route": [
    "routed"
   ]
  },
  {
   "q": "Q4",
   "system": "chat_gpt",
   "run": 4,
   "figure_given": "yes",
   "figure_verbatim": "approximately 14%; 34% for less experienced agents",
   "source_named": "yes",
   "source_named_verbatim": "the working paper \"Generative AI at Work\" … published by the National Bureau of Economic Research in April 2023",
   "source_class": "primary",
   "interested_share": null,
   "contra_as_support": "no",
   "provenance_error": [],
   "search_used": "yes",
   "version_returned": "preprint",
   "coder_note": "Four slots, one domain. Second consecutive preprint answer; the QJE version is never mentioned."
  },
  {
   "q": "Q4",
   "system": "chat_gpt",
   "run": 5,
   "figure_given": "yes",
   "figure_verbatim": "approximately 14%; 34%",
   "source_named": "yes",
   "source_named_verbatim": "the working paper titled \"Generative AI at Work\" … published by the National Bureau of Economic Research in April 2023",
   "source_class": "primary",
   "interested_share": null,
   "contra_as_support": "no",
   "provenance_error": [],
   "search_used": "yes",
   "version_returned": "preprint",
   "coder_note": "Third consecutive preprint answer, four slots, one domain, near-identical wording to runs 3 and 4."
  },
  {
   "q": "Q4",
   "system": "claude",
   "run": 4,
   "figure_given": "yes",
   "figure_verbatim": "14% (NBER WP); 15% (QJE); 5,179 agents",
   "source_named": "yes",
   "source_named_verbatim": "Erik Brynjolfsson, Danielle Li, Lindsey Raymond, \"Generative AI at Work,\" The Quarterly Journal of Economics, Volume 140, Issue 2, May 2025, Pages 889–942",
   "source_class": "primary",
   "interested_share": null,
   "contra_as_support": "no",
   "provenance_error": [],
   "search_used": "yes",
   "version_returned": "vor",
   "coder_note": "Names the QJE article as the original source outright, with correct pagination, and cites academic.oup.com. Reports the preprint's 5,179 agents rather than the published 5,172: the figure count still comes from the working paper."
  },
  {
   "q": "Q4",
   "system": "claude",
   "run": 5,
   "figure_given": "yes",
   "figure_verbatim": "14% (NBER WP); 15% (QJE)",
   "source_named": "yes",
   "source_named_verbatim": "NBER Working Paper No. 31161 (April 2023)**, later published in The Quarterly Journal of Economics, Volume 140, Issue 2 (May 2025)",
   "source_class": "primary",
   "interested_share": null,
   "contra_as_support": "no",
   "provenance_error": [],
   "search_used": "yes",
   "version_returned": "vor",
   "coder_note": "Cites academic.oup.com and gives both figures, but LEADS with 14% and names 'NBER Working Paper No. 31161 (April 2023)' as the original source, flagging the QJE second. Coded `vor` because the answer surfaces and links the version of record; under a stricter reading of 'which version the answer is built on' this row is `preprint`. The call is recorded here because it is the one that makes this cell 3/3 rather than 2/3."
  },
  {
   "q": "Q4",
   "system": "gemini",
   "run": 4,
   "figure_given": "yes",
   "figure_verbatim": "14% on average; 34% for novices; \"Some sources specify the average increase as 13.8%\"",
   "source_named": "yes",
   "source_named_verbatim": "NBER Working Paper 31161, in April 2023 and revised in November 2023",
   "source_class": "primary",
   "interested_share": null,
   "contra_as_support": "no",
   "provenance_error": [
    "'13.8%' appears in neither version of the paper. Attributed only to unnamed 'some sources'."
   ],
   "search_used": "yes",
   "version_returned": "preprint",
   "coder_note": "The drifting-figure behaviour returns in a new form: a third decimal variant, hedged to unnamed sources. `google.com` appears as a citation domain.",
   "provenance_error_route": [
    "not_established"
   ]
  },
  {
   "q": "Q4",
   "system": "gemini",
   "run": 5,
   "figure_given": "yes",
   "figure_verbatim": "15% on average; 5,172 agents",
   "source_named": "yes",
   "source_named_verbatim": "circulated as an NBER working paper (No. 31161) and later published in the Quarterly Journal of Economics",
   "source_class": "primary",
   "interested_share": null,
   "contra_as_support": "no",
   "provenance_error": [],
   "search_used": "yes",
   "version_returned": "vor",
   "coder_note": "The one result that breaks the pattern. Reports BOTH version-of-record figures: 15% and 5,172, and names the QJE publication. But it cites no oup.com slot: its domains are stanford.edu, arxiv.org, revartis.com, itif.org, policycommons.net, richmondfed.org. Correct numbers, no link to the paper that contains them."
  },
  {
   "q": "Q4",
   "system": "perplexity",
   "run": 4,
   "figure_given": "yes",
   "figure_verbatim": "about 14% on average; about 34% for novices; 5,179 agents",
   "source_named": "yes",
   "source_named_verbatim": "their NBER working paper, *Generative AI at Work*",
   "source_class": "primary",
   "interested_share": null,
   "contra_as_support": "no",
   "provenance_error": [],
   "search_used": "yes",
   "version_returned": "preprint",
   "coder_note": "Seven domains, none of them the QJE. Reports the preprint's agent count."
  },
  {
   "q": "Q4",
   "system": "perplexity",
   "run": 5,
   "figure_given": "yes",
   "figure_verbatim": "about 14% on average; about 34% for novices",
   "source_named": "yes",
   "source_named_verbatim": "their NBER working paper, **\"Generative AI at Work\"**",
   "source_class": "primary",
   "interested_share": null,
   "contra_as_support": "no",
   "provenance_error": [],
   "search_used": "yes",
   "version_returned": "preprint",
   "coder_note": "Third consecutive preprint answer with a near-identical citation set."
  },
  {
   "q": "Q5",
   "system": "chat_gpt",
   "run": 1,
   "figure_given": "yes",
   "figure_verbatim": "40.2% higher quality; 23% worse",
   "source_named": "no",
   "source_named_verbatim": "a study conducted by the Boston Consulting Group",
   "source_class": "tertiary",
   "interested_share": null,
   "contra_as_support": "no",
   "provenance_error": [
    "40.2% appears in neither version of the paper; cited to jobcannon.io, which states '…rated 40.2% higher in quality…': checked at source 2026-07-27.",
    "23% is BCG's marketing figure; the paper reports 19 percentage points."
   ],
   "search_used": "yes",
   "version_returned": "preprint+",
   "coder_note": "Two slots, two domains: a career-assessment quiz site and the commissioning sponsor. The paper is never named.",
   "provenance_error_route": [
    "routed",
    "routed"
   ]
  },
  {
   "q": "Q5",
   "system": "chat_gpt",
   "run": 4,
   "figure_given": "yes",
   "figure_verbatim": "12.2% more tasks; 25.1% faster; 40.2% higher quality; 23% decline",
   "source_named": "no",
   "source_named_verbatim": "A Boston Consulting Group (BCG) study, in collaboration with Harvard Business School, MIT Sloan, and Wharton",
   "source_class": "tertiary",
   "interested_share": null,
   "contra_as_support": "no",
   "provenance_error": [
    "40.2% appears in neither version, and this run cites BCG's own PDF for it, a document that states 40%.",
    "23% is BCG's marketing figure; the paper reports 19 percentage points."
   ],
   "search_used": "yes",
   "version_returned": "preprint+",
   "coder_note": "Third counted run at 40.2%, and the second time the decimal is attributed to a BCG document that does not contain it. Both slots resolve to bcg.com. The 'Harvard Business School, MIT Sloan, and Wharton' attribution is broadly right: Mollick is at Wharton, Kellogg at MIT, Lifshitz-Assaf at Warwick, and is NOT coded as an error.",
   "provenance_error_route": [
    "not_established",
    "routed"
   ]
  },
  {
   "q": "Q5",
   "system": "claude",
   "run": 1,
   "figure_given": "yes",
   "figure_verbatim": "\"40 percent of the trial group produced higher quality results\"; 19%",
   "source_named": "yes",
   "source_named_verbatim": "a working paper titled *Navigating the Jagged Technological Frontier*, September 22, 2023",
   "source_class": "primary",
   "interested_share": null,
   "contra_as_support": "no",
   "provenance_error": [
    "'40 percent of the trial group' converts a magnitude into a headcount: a claim about a different quantity. The Harvard Crimson states exactly this, verbatim; the error is the newspaper's and is reproduced faithfully.",
    "40% was removed in peer review.",
    "'750 Boston Consulting Group consultants' against the true 758, cited to a LinkedIn Pulse post: the same post cited in counted runs 3 and 4."
   ],
   "search_used": "yes",
   "version_returned": "preprint",
   "coder_note": "Three slots, three domains, none academic. The Crimson headcount conversion appears in counted runs 1 and 4 (2 of 3); the separate 750-vs-758 count error appears in all three, cited to the same LinkedIn Pulse post each time.",
   "provenance_error_route": [
    "routed",
    "routed",
    "not_established"
   ]
  },
  {
   "q": "Q5",
   "system": "claude",
   "run": 4,
   "figure_given": "yes",
   "figure_verbatim": "12.2% more tasks; 25.1% quicker; \"40 percent of the trial group produced higher quality results\"; 19% less likely",
   "source_named": "yes",
   "source_named_verbatim": "a working paper titled **\"Navigating the Jagged Technological Frontier\"** published by Harvard Business School researchers on September 22, 2023",
   "source_class": "primary",
   "interested_share": null,
   "contra_as_support": "no",
   "provenance_error": [
    "The Crimson headcount conversion, reproduced again.",
    "'750 Boston Consulting Group consultants' against the true 758, cited to the same LinkedIn Pulse post as runs 1 and 3.",
    "40% was removed in peer review."
   ],
   "search_used": "yes",
   "version_returned": "preprint",
   "coder_note": "The headcount conversion returns after being absent in run 3, so it is the majority behaviour of this cell (2 of 3), not a one-off. Five of six slots are the Crimson.",
   "provenance_error_route": [
    "routed",
    "not_established",
    "routed"
   ]
  },
  {
   "q": "Q5",
   "system": "gemini",
   "run": 1,
   "figure_given": "yes",
   "figure_verbatim": "40% higher quality",
   "source_named": "yes",
   "source_named_verbatim": "a working paper … published by Harvard Business School. The lead author … is Fabrizio Dell'Acqua",
   "source_class": "primary",
   "interested_share": null,
   "contra_as_support": "no",
   "provenance_error": [
    "40% was removed in peer review; the published paper reports 33.9% and 29.9%."
   ],
   "search_used": "yes",
   "version_returned": "preprint",
   "coder_note": "Names the working paper and cites none of it: six domains, all trade press, vendor or national newspaper.",
   "provenance_error_route": [
    "routed"
   ]
  },
  {
   "q": "Q5",
   "system": "gemini",
   "run": 4,
   "figure_given": "yes",
   "figure_verbatim": "12.2% more tasks; 25.1% faster; 40% higher quality; 758 consultants",
   "source_named": "yes",
   "source_named_verbatim": "\"Navigating the Jagged Technological Frontier: Field Experimental Evidence…\" was published on September 22, 2023",
   "source_class": "primary",
   "interested_share": null,
   "contra_as_support": "no",
   "provenance_error": [
    "40% was removed in peer review."
   ],
   "search_used": "yes",
   "version_returned": "preprint",
   "coder_note": "Back to 40% after run 3's 40.2%, so the fabricated decimal did NOT stick on this system. jobcannon.io is still in the citation set. 33 slots resolving to 10 domains: the largest repetition-as-breadth gap in the study.",
   "provenance_error_route": [
    "routed"
   ]
  },
  {
   "q": "Q5",
   "system": "perplexity",
   "run": 1,
   "figure_given": "yes",
   "figure_verbatim": "40% higher quality; 23% worse",
   "source_named": "yes",
   "source_named_verbatim": "The original BCG source is **How People Create and Destroy Value with Generative AI** from Boston Consulting Group",
   "source_class": "tertiary",
   "interested_share": null,
   "contra_as_support": "no",
   "provenance_error": [
    "Names the commissioning sponsor's own marketing publication as the original source of its own experiment.",
    "23% is BCG's figure; the paper reports 19 percentage points.",
    "40% was removed in peer review."
   ],
   "search_used": "yes",
   "version_returned": "preprint",
   "coder_note": "Eighteen slots, thirteen domains, zero academic.",
   "provenance_error_route": [
    "not_established",
    "routed",
    "routed"
   ]
  },
  {
   "q": "Q5",
   "system": "perplexity",
   "run": 4,
   "figure_given": "yes",
   "figure_verbatim": "12.2% more tasks; 25.1% faster; about 40% higher quality; 23% drop",
   "source_named": "yes",
   "source_named_verbatim": "The original source is BCG's publication **\"How People Create and Destroy Value with Generative AI\"**",
   "source_class": "tertiary",
   "interested_share": null,
   "contra_as_support": "no",
   "provenance_error": [
    "Third counted run naming the sponsor's marketing publication as the original source.",
    "23% is BCG's figure; the paper reports 19 percentage points.",
    "40% was removed in peer review."
   ],
   "search_used": "yes",
   "version_returned": "preprint",
   "coder_note": "Reddit and Facebook appear in the citation set; hbs.edu appears via an HBS AI-institute blog post, not the working paper. Still zero links to Organization Science.",
   "provenance_error_route": [
    "not_established",
    "routed",
    "routed"
   ]
  }
 ]
}