# Second-coder pass: 2026-07-28 The protocol requires "a pass by an independent reader on a 20% sample, with disagreements reported rather than reconciled away. If agreement is poor, that is published too." **This pass was run by a language model, not a human reader.** The protocol's requirement for an independent human reader was waived on 2026-07-28 (changelog, same date), and no inter-coder reliability figure is claimed anywhere in the study. Disagreements below are reported as found and are not reconciled. **What the second coder received:** `second-coder-packet.json` (the sampled responses, their prompts and their citation lists) and `questions.json` (the pre-registered ground truth). Nothing else. It was instructed not to open `coding.json`, `coding-sheet.md`, `protocol.md` or the drafts, and to check cited documents at source where a code depended on it. It did check them, and several of its findings come from that checking. **Sample.** Rule fixed before sampling: counted rows sorted by (question, system, run); every 4th row from index 2, zero-based. **9 of 36 rows, 25%.** --- ## Agreement Six fields compared per row: `figure_given`, `source_named`, `source_class`, `contra_as_support`, `version_returned`, and the count of `provenance_error` entries. `interested_share` is excluded from the count because the first coder left it null everywhere, so there is nothing to disagree with. **54 comparisons. 43 agree, 11 disagree: 80%.** | Row | Field | First coder | Second coder | | --- | --- | --- | --- | | Q1 · chat_gpt · r3 | `source_named` | no | yes | | Q1 · gemini · r1 | `source_class` | tertiary | **primary** | | Q1 · gemini · r1 | `provenance_error` count | 2 | 1 | | Q1 · perplexity · r2 | `figure_given` | range | yes | | Q1 · perplexity · r2 | `source_class` | tertiary | **primary** | | Q1 · perplexity · r2 | `provenance_error` count | 0 | 1 | | Q4 · chat_gpt · r5 | `provenance_error` count | 0 | **1** | | Q5 · chat_gpt · r4 | `source_named` | no | yes | | Q5 · chat_gpt · r4 | `provenance_error` count | 2 | **4** | | Q5 · gemini · r1 | `provenance_error` count | 1 | 0 | | Q5 · perplexity · r3 | `version_returned` | preprint | preprint+ | Disagreements by field: `provenance_error` count 5, `source_named` 2, `source_class` 2, `figure_given` 1, `version_returned` 1. **Nothing below is reconciled.** Both codings stand as recorded. --- ## The one that is a scheme defect, not a coder error `source_class` split on two of nine rows, and the second coder named the cause: **the scheme never says whether `source_class` describes the source named as the origin or the documents actually retrieved.** In its words: > "I coded the named origin throughout, because otherwise `source_named: yes` + `source_class` carries > no information beyond `citation_domains`. Under the other reading, four rows change." The first coder read it the other way: the class of what was actually produced. On Q1 that is the difference between recording "it named a primary source" and "it produced vendor blogs", and both sentences are true of the same answer. Its sharpest example is a row where the two coders happen to agree on the label and should not: > "Q5/gemini/run1 is the sharpest case: correct primary provenance, zero primary retrieval." **Verdict: the field is underspecified and the study cannot compute a `source_class` rate until it is split in two**: `origin_class` for what the answer claims, `retrieved_class` for what it cites. That is a change to the scheme, so it does not happen retroactively on this dataset. It is recorded here and goes to the next registration. ## Three places the second coder found something the first missed **1 · Q4 · chat_gpt · r5: a misidentified NBER document.** The answer cites "Figure 1 in the study" to `nber.org/sites/default/files/2024-04/2024number1.pdf`, which is the NBER Reporter, not the paper. The Reporter does carry a figure by that name, so the content is real and the document is wrong. The first coder logged no provenance error on this row. **Accepted as a miss.** **2 · Q5 · chat_gpt · r4: two more figures with false provenance.** Checked at source: BCG's PDF contains neither 12.2% nor 25.1%, both of which the answer attributes to it. Those figures are correct about the study and falsely sourced, which is the piece's own distinction applied more strictly than the first coder applied it. **Accepted as a miss.** **3 · Two citations on Q5 · perplexity · r3 are now dead.** `media-publications.bcg.com/BCG-BHI-GenAI-Experimental-Findings.pdf` and `bcg.com/news/25october2023-theres-time-and-place-for-generative-ai` both return 404 as of 2026-07-28. They were live when the response was collected. **Recorded, and it is the study's own thesis arriving on schedule: the corpus moved under the dataset within a day.** ## Two places the coders disagree on what the field means **`provenance_error` scope.** The scheme lists "wrong organisation, wrong year, wrong paper, wrong author": all source-identity attributes. The first coder also logged the deleted 40% figure as a provenance error; the second did not, on the grounds that the cited documents do contain 40%. That accounts for the Q5/gemini/r1 and Q1/gemini/r1 splits in the opposite direction from the two misses above. Neither reading is unreasonable and the scheme does not decide it. **`version_returned: preprint+`.** The second coder applied the definition literally and coded Q5/perplexity/r3 `preprint+`, because 23% appears in neither version of the paper. The value was registered with the 40.2% fabrication in mind. Its note: *"If the intent was narrower, this row should be `preprint`, and the rule needs the words 'and is not the sponsor's figure.'"* Correct, and the rule gets those words at the next registration, not now. ## An artefact that biases the route classification The second coder caught something the first should have: > "All three Gemini rows are structurally unroutable. Gemini returns bare domain labels with > `url: null`. There is no identifiable document to check at source, so by the stated rule every > Gemini provenance error must be `not_established` regardless of how obviously it was inherited … > Coding the same error two different ways because one system returns URLs and the other does not is > an artefact of the citation format, not of the systems' behaviour." **This bites the 22-routed / 14-not-established split reported in the essays.** Five of the fourteen `not_established` entries are Gemini rows, and they are unroutable because of how Gemini reports citations, not because the error was invented. The split is still the honest number under the stated rule, and it now ships with this caveat attached. ## `interested_share`: coded for the first time, by the second coder only The first coder left the field null on all 36 rows. The second coded it on its 9, from homepages, and recorded every borderline call and which way it went: `uplandsoftware.com` and `bcg.com` in, both of which it flags as reversible; `invespcro.com`, `prescientai.com` and `nngroup.com` out. Values, for the sampled rows only: Q1/chat_gpt/r3 **1.0** · Q1/gemini/r1 0.67 · Q1/perplexity/r2 0.56 · Q4 rows **0.0** · Q5/chat_gpt/r4 **1.0** · Q5/gemini/r1 0.17 · Q5/perplexity/r3 0.11. **No rate may be computed from these.** Nine rows, one coder, several self-declared reversible calls. What they are good for is direction, and the direction is that the two commercial questions and the two research questions behave completely differently on this field. --- ## What this pass changes - The `provenance_error` counts for Q4/chat_gpt/r5 and Q5/chat_gpt/r4 are **understated** in the first coding. The counts stay as coded and this note stands beside them. - `source_class` cannot carry a rate until the field is split. - The routed / not-established split needs the Gemini caveat wherever it is quoted. - The protocol's requirement for an independent **human** reader remains open.