{
 "sampling_rule": "Counted rows sorted by (question, system, run); every 4th row from index 2 (0-based). Rule fixed before sampling. 9 of 36 rows = 25%, above the 20% the protocol requires.",
 "instruction": "Code each response independently against the scheme. You have not seen anyone else's codes and must not ask for them.",
 "rows": [
  {
   "id": "Q1/chat_gpt/run3",
   "question_id": "Q1",
   "prompt": "How much more expensive is acquiring a customer than retaining one? Give the figure and cite the original source.",
   "answer_text": "Acquiring a new customer is generally more expensive than retaining an existing one. The cost of acquiring a new customer can be 5 to 25 times higher than retaining an existing one, depending on the industry and business model. ([artisangrowthstrategies.com](https://www.artisangrowthstrategies.com/blog/customer-acquisition-vs-retention-costs-statistics-and-trends-you-should-know?utm_source=openai))\n\nFor example, in the SaaS industry, the average Customer Acquisition Cost (CAC) is $1,200 per customer, while the average Customer Retention Cost (CRC) is $35 per customer. ([artisangrowthstrategies.com](https://www.artisangrowthstrategies.com/blog/customer-retention-cost-vs-acquisition-cost-industry-benchmarks-2026?utm_source=openai))\n\nThis significant difference underscores the importance of balancing acquisition and retention strategies to maximize profitability. ",
   "citations": [
    {
     "title": "Customer Acquisition vs Retention Cost: 2026 Stats (5-25x CAC Gap) | Artisan Strategies",
     "url": "https://www.artisangrowthstrategies.com/blog/customer-acquisition-vs-retention-costs-statistics-and-trends-you-should-know"
    },
    {
     "title": "Customer Retention Cost vs Acquisition Cost: Industry Benchmarks for 2026 | Artisan Strategies",
     "url": "https://www.artisangrowthstrategies.com/blog/customer-retention-cost-vs-acquisition-cost-industry-benchmarks-2026"
    }
   ],
   "citation_domains_for_reference": [
    "artisangrowthstrategies.com"
   ]
  },
  {
   "id": "Q1/gemini/run1",
   "question_id": "Q1",
   "prompt": "How much more expensive is acquiring a customer than retaining one? Give the figure and cite the original source.",
   "answer_text": "Acquiring a new customer can be anywhere from 5 to 25 times more expensive than retaining an existing one. While some older research, like a 1990 Harvard Business Review article by Frederick Reichheld, cited the cost as 5 times more, this figure can vary significantly based on industry, business model, and price point.\n\nThe higher cost of customer acquisition is due to the extensive marketing and advertising efforts needed to attract new customers. This includes investments in digital advertising, content marketing, social media campaigns, and often promotional discounts to convert prospects. In contrast, retention efforts often involve customer success teams, support infrastructure, and targeted retention marketing, which tend to be less expensive as the customer already trusts the brand.",
   "citations": [
    {
     "title": "churnkey.co",
     "url": null
    },
    {
     "title": "postaffiliatepro.com",
     "url": null
    },
    {
     "title": "yotpo.com",
     "url": null
    },
    {
     "title": "churnkey.co",
     "url": null
    },
    {
     "title": "optimove.com",
     "url": null
    },
    {
     "title": "optimove.com",
     "url": null
    },
    {
     "title": "postaffiliatepro.com",
     "url": null
    },
    {
     "title": "churnkey.co",
     "url": null
    },
    {
     "title": "prescientai.com",
     "url": null
    },
    {
     "title": "uplandsoftware.com",
     "url": null
    }
   ],
   "citation_domains_for_reference": [
    "churnkey.co",
    "optimove.com",
    "postaffiliatepro.com",
    "prescientai.com",
    "uplandsoftware.com",
    "yotpo.com"
   ]
  },
  {
   "id": "Q1/perplexity/run2",
   "question_id": "Q1",
   "prompt": "How much more expensive is acquiring a customer than retaining one? Give the figure and cite the original source.",
   "answer_text": "The commonly cited figure is that acquiring a customer costs **5 times more** than retaining one, though some sources put the range at **5 to 25 times** depending on the industry and business model.[1][16]\n\nThe original source most often traced for the **5x** claim is **Bain & Company / Frederick Reichheld** research, which later summaries say Harvard Business Review repeated as “anywhere from 5 to 25 times more expensive.”[2][19]",
   "citations": [
    {
     "title": "Customer Acquisition vs Retention Costs",
     "url": "https://www.optimove.com/resources/learning-center/customer-acquisition-vs-retention-costs"
    },
    {
     "title": "Customer Retention: 5-25x Cheaper Than Acquisition",
     "url": "https://stealthagents.com/research/customer-retention-cost-statistics-2026"
    },
    {
     "title": "Customer Acquisition vs. Retention: Cost Comparison Guide",
     "url": "https://churnkey.co/blog/customer-acquisition-vs-retention-cost-comparison-guide/"
    },
    {
     "title": "Customer Retention Versus Customer Acquisition",
     "url": "https://www.forbes.com/councils/forbesbusinesscouncil/2022/12/12/customer-retention-versus-customer-acquisition/"
    },
    {
     "title": "Customer Acquisition Vs. Retention: Which Drives Profit?",
     "url": "https://www.yotpo.com/blog/cost-of-customer-acquisition-vs-retention/"
    },
    {
     "title": "Customer Acquisition vs Retention Cost: 2026 Stats (5-25 ...",
     "url": "https://www.artisangrowthstrategies.com/blog/customer-acquisition-vs-retention-costs-statistics-and-trends-you-should-know"
    },
    {
     "title": "Customer Acquisition Costs vs Retention Costs: What Matters",
     "url": "https://prescientai.com/blog/customer-acquisition-costs-vs-retention-costs"
    },
    {
     "title": "Cost of customer acquisition vs retention",
     "url": "https://www.upside.com/business/retailer-blog/cost-of-customer-acquisition-vs-retention"
    },
    {
     "title": "Customer Retention vs Acquisition Cost: The Real Math (2026)",
     "url": "https://churnmiser.com/blog/customer-retention-vs-acquisition-cost"
    },
    {
     "title": "Customer Retention vs Acquisition Cost: Beyond 5x",
     "url": "https://releva.ai/blog/customer-retention-vs-acquisition-cost/"
    },
    {
     "title": "CAC vs Retention Cost 2026: The 5-25x Budget Mistake",
     "url": "https://www.artisangrowthstrategies.com/blog/customer-acquisition-cost-vs-retention-cost-analysis-benchmarks"
    },
    {
     "title": "Retention vs Acquisition: The 5x Cost Difference | Regulr Blog",
     "url": "https://regulr.ai/blog/customer-retention-vs-acquisition"
    },
    {
     "title": "The Cost of Customer Acquisition vs Customer Retention",
     "url": "https://uplandsoftware.com/kapost/resources/blog/customer-acquisition-versus-customer-retention/"
    },
    {
     "title": "The Real Cost of Customers: Why Acquisition, Retention ...",
     "url": "https://www.linkedin.com/pulse/real-cost-customers-why-acquisition-retention-loss-shape-diego-rondon-jtccc"
    },
    {
     "title": "What is the Cost of Customer Acquisition vs Customer Retention?",
     "url": "https://www.linkedin.com/pulse/what-cost-customer-acquisition-vs-retention-ian-kingwill"
    },
    {
     "title": "Customer Acquisition Vs Retention Costs",
     "url": "https://www.invespcro.com/blog/customer-acquisition-retention/"
    },
    {
     "title": "Why Customer Retention Costs 5x Less Than Acquisition",
     "url": "https://www.postaffiliatepro.com/blog/why-customer-retention-costs-5x-less-than-acquisition/"
    },
    {
     "title": "Customer Acquisition: What You Need To Know",
     "url": "https://extension.psu.edu/customer-acquisition-what-you-need-to-know"
    },
    {
     "title": "Does it still cost 5X more to acquire customers than ...",
     "url": "https://hashtagpaid.com/banknotes/does-it-still-cost-5x-more-to-acquire-customers-than-to-retain-them-in-2023"
    },
    {
     "title": "Customer Retention Marketing vs. Customer Acquisition Marketing",
     "url": "https://www.outboundengine.com/blog/customer-retention-marketing-vs-customer-acquisition-marketing/"
    }
   ],
   "citation_domains_for_reference": [
    "artisangrowthstrategies.com",
    "churnkey.co",
    "churnmiser.com",
    "forbes.com",
    "hashtagpaid.com",
    "invespcro.com",
    "linkedin.com",
    "optimove.com",
    "outboundengine.com",
    "postaffiliatepro.com",
    "prescientai.com",
    "psu.edu",
    "regulr.ai",
    "releva.ai",
    "stealthagents.com",
    "uplandsoftware.com",
    "upside.com",
    "yotpo.com"
   ]
  },
  {
   "id": "Q4/chat_gpt/run5",
   "question_id": "Q4",
   "prompt": "What did Brynjolfsson, Li and Raymond find about the effect of AI assistance on customer support agent productivity? Give the figure and cite the original source.",
   "answer_text": "In their 2023 study, \"Generative AI at Work,\" Brynjolfsson, Li, and Raymond examined the impact of AI assistance on customer support agent productivity. They found that access to a generative AI-based conversational assistant increased productivity by approximately 14% on average. This improvement was particularly significant among less experienced and lower-skilled agents, who experienced a 34% increase in issues resolved per hour. In contrast, more experienced and highly skilled agents saw minimal impact, with some even experiencing slight declines in quality. ([nber.org](https://www.nber.org/papers/w31161))\n\nThe study also observed that the AI tool facilitated the dissemination of best practices from more capable workers to newer agents, aiding in their development. Additionally, AI assistance improved customer sentiment, reduced requests for managerial intervention, and enhanced employee retention. ([nber.org](https://www.nber.org/papers/w31161))\n\nFor a visual representation of these findings, you can refer to Figure 1 in the study, which illustrates the productivity of customer support agents with and without AI support. ([nber.org](https://www.nber.org/sites/default/files/2024-04/2024number1.pdf))\n\nThe original source of this information is the working paper titled \"Generative AI at Work\" by Brynjolfsson, Li, and Raymond, published by the National Bureau of Economic Research in April 2023. ([nber.org](https://www.nber.org/papers/w31161)) ",
   "citations": [
    {
     "title": "Generative AI at Work | NBER",
     "url": "https://www.nber.org/papers/w31161"
    },
    {
     "title": "Generative AI at Work | NBER",
     "url": "https://www.nber.org/papers/w31161"
    },
    {
     "title": "In This Issue",
     "url": "https://www.nber.org/sites/default/files/2024-04/2024number1.pdf"
    },
    {
     "title": "Generative AI at Work | NBER",
     "url": "https://www.nber.org/papers/w31161"
    }
   ],
   "citation_domains_for_reference": [
    "nber.org"
   ]
  },
  {
   "id": "Q4/gemini/run3",
   "question_id": "Q4",
   "prompt": "What did Brynjolfsson, Li and Raymond find about the effect of AI assistance on customer support agent productivity? Give the figure and cite the original source.",
   "answer_text": "Brynjolfsson, Li, and Raymond found that access to AI assistance increased customer support agent productivity by an average of 14%. This figure is based on a study of 5,179 customer support agents at a Fortune 500 software company. The productivity was measured by the number of customer issues resolved per hour.\n\nTheir research also indicated that the productivity gains were not evenly distributed; less experienced and lower-skilled workers saw the most significant improvements, while highly skilled workers experienced minimal impact on their speed and even slight declines in quality. The study suggests that the AI model helps disseminate the knowledge of more experienced workers, aiding newer employees in improving their performance.\n\nThe original source for these findings is the working paper \"Generative AI at Work\" by Erik Brynjolfsson, Danielle Li, and Lindsey R. Raymond, published by the National Bureau of Economic Research (NBER Working Paper No. 31161) in April 2023.",
   "citations": [
    {
     "title": "nber.org",
     "url": null
    },
    {
     "title": "policycommons.net",
     "url": null
    },
    {
     "title": "itif.org",
     "url": null
    },
    {
     "title": "edwardconard.com",
     "url": null
    },
    {
     "title": "nber.org",
     "url": null
    },
    {
     "title": "mit.edu",
     "url": null
    },
    {
     "title": "itif.org",
     "url": null
    },
    {
     "title": "stanford.edu",
     "url": null
    },
    {
     "title": "nber.org",
     "url": null
    },
    {
     "title": "itif.org",
     "url": null
    },
    {
     "title": "stanford.edu",
     "url": null
    },
    {
     "title": "nber.org",
     "url": null
    },
    {
     "title": "stanford.edu",
     "url": null
    },
    {
     "title": "hrdive.com",
     "url": null
    },
    {
     "title": "nber.org",
     "url": null
    },
    {
     "title": "policycommons.net",
     "url": null
    },
    {
     "title": "stanford.edu",
     "url": null
    },
    {
     "title": "edwardconard.com",
     "url": null
    },
    {
     "title": "nber.org",
     "url": null
    },
    {
     "title": "nngroup.com",
     "url": null
    },
    {
     "title": "policycommons.net",
     "url": null
    },
    {
     "title": "itif.org",
     "url": null
    }
   ],
   "citation_domains_for_reference": [
    "edwardconard.com",
    "hrdive.com",
    "itif.org",
    "mit.edu",
    "nber.org",
    "nngroup.com",
    "policycommons.net",
    "stanford.edu"
   ]
  },
  {
   "id": "Q4/perplexity/run4",
   "question_id": "Q4",
   "prompt": "What did Brynjolfsson, Li and Raymond find about the effect of AI assistance on customer support agent productivity? Give the figure and cite the original source.",
   "answer_text": "Brynjolfsson, Li, and Raymond found that **AI assistance increased customer support agent productivity by about 14% on average**.[2] Their paper on 5,179 agents also reports that the gains were much larger for novice and lower-skilled workers: about **34%**: while experienced, highly skilled workers saw little to no improvement.[2]\n\nThe original source is their NBER working paper, *Generative AI at Work*.[2] The MIT Sloan summary of the same research also reports the **14%** average productivity increase.[1]",
   "citations": [
    {
     "title": "Generative AI and Worker Productivity",
     "url": "https://mitsloan.mit.edu/centers-initiatives/institute-work-and-employment-research/generative-ai-and-worker-productivity"
    },
    {
     "title": "Generative AI at Work",
     "url": "https://www.nber.org/papers/w31161"
    },
    {
     "title": "GENERATIVE AI AT WORK˚ - Danielle Li",
     "url": "https://danielle.li/assets/docs/GenerativeAIatWork.pdf"
    },
    {
     "title": "Generative AI at Work - Íñigo Medina",
     "url": "https://inigomedina.co/library/work/brynjolfsson-li-raymond-generative-ai-at-work"
    },
    {
     "title": "AI improved the productivity of a Fortune 500 software ...",
     "url": "https://www.bls.gov/opub/mlr/2024/beyond-bls/ai-improved-the-productivity-of-a-fortune-500-software-company.htm"
    },
    {
     "title": "Research on a large language assistant by Lindsey ...",
     "url": "https://www.linkedin.com/posts/erikbrynjolfsson_generative-ai-can-boost-productivity-without-activity-7221961645793689600-1l5s"
    },
    {
     "title": "AI Tools Raise the Productivity of Customer-Support Agents",
     "url": "https://www.nngroup.com/articles/ai-productivity-customer-support/"
    }
   ],
   "citation_domains_for_reference": [
    "bls.gov",
    "danielle.li",
    "inigomedina.co",
    "linkedin.com",
    "mit.edu",
    "nber.org",
    "nngroup.com"
   ]
  },
  {
   "id": "Q5/chat_gpt/run4",
   "question_id": "Q5",
   "prompt": "What did the BCG study find about consultants using GPT-4? Give the figure and cite the original source.",
   "answer_text": "A Boston Consulting Group (BCG) study, in collaboration with Harvard Business School, MIT Sloan, and Wharton, investigated the impact of GPT-4 on consultants' performance. The study involved 758 BCG consultants who were assigned to complete 18 realistic consulting tasks. The findings revealed that consultants using GPT-4 completed 12.2% more tasks, worked 25.1% faster, and produced results rated 40.2% higher in quality compared to those who did not use the AI tool. ([web-assets.bcg.com](https://web-assets.bcg.com/pdf-src/prod-live/how-people-create-and-destroy-value-with-gen-ai.pdf))\n\nHowever, the study also highlighted a \"jagged technological frontier.\" When consultants used GPT-4 for tasks within the tool's current capabilities, their performance improved. Conversely, for tasks outside GPT-4's competence, performance declined by 23% compared to the control group. ([web-assets.bcg.com](https://web-assets.bcg.com/pdf-src/prod-live/how-people-create-and-destroy-value-with-gen-ai.pdf))\n\nThese findings underscore the importance of understanding AI's strengths and limitations to maximize its benefits in consulting tasks. ",
   "citations": [
    {
     "title": "How People Create and Destroy Value with Generative AI | BCG",
     "url": "https://web-assets.bcg.com/pdf-src/prod-live/how-people-create-and-destroy-value-with-gen-ai.pdf"
    },
    {
     "title": "How People Create and Destroy Value with Generative AI | BCG",
     "url": "https://web-assets.bcg.com/pdf-src/prod-live/how-people-create-and-destroy-value-with-gen-ai.pdf"
    }
   ],
   "citation_domains_for_reference": [
    "bcg.com"
   ]
  },
  {
   "id": "Q5/gemini/run1",
   "question_id": "Q5",
   "prompt": "What did the BCG study find about consultants using GPT-4? Give the figure and cite the original source.",
   "answer_text": "A study conducted by Harvard Business School in collaboration with Boston Consulting Group (BCG) found that consultants using GPT-4 significantly improved their performance. The study, which involved 758 BCG consultants, showed that those with access to GPT-4 completed 12.2% more tasks, finished tasks 25.1% faster, and produced results that were 40% higher in quality compared to a control group without AI access.\n\nThe original source for this study is a working paper titled \"Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality,\" published by Harvard Business School. The lead author of the paper is Fabrizio Dell'Acqua.",
   "citations": [
    {
     "title": "innovationtraining.org",
     "url": null
    },
    {
     "title": "mi-3.com.au",
     "url": null
    },
    {
     "title": "jobcannon.io",
     "url": null
    },
    {
     "title": "legaldive.com",
     "url": null
    },
    {
     "title": "innovationtraining.org",
     "url": null
    },
    {
     "title": "hindustantimes.com",
     "url": null
    },
    {
     "title": "venturebeat.com",
     "url": null
    }
   ],
   "citation_domains_for_reference": [
    "hindustantimes.com",
    "innovationtraining.org",
    "jobcannon.io",
    "legaldive.com",
    "mi-3.com.au",
    "venturebeat.com"
   ]
  },
  {
   "id": "Q5/perplexity/run3",
   "question_id": "Q5",
   "prompt": "What did the BCG study find about consultants using GPT-4? Give the figure and cite the original source.",
   "answer_text": "The BCG study found that consultants using **GPT-4** completed **12.2% more tasks**, finished **25.1% faster**, and produced work that was **about 40% higher quality** on creative product-innovation tasks.[2][4] The original source is BCG’s report *How People Create and Destroy Value with Generative AI*.[2]\n\nThe same study also found a limit: when consultants used GPT-4 for **business problem solving** outside the model’s capability frontier, performance was **23% worse** than without GPT-4.[2][4]",
   "citations": [
    {
     "title": "BCG-BHI-GenAI-Experimental-Findings.pdf",
     "url": "https://media-publications.bcg.com/BCG-BHI-GenAI-Experimental-Findings.pdf"
    },
    {
     "title": "How People Create and Destroy Value with Generative AI",
     "url": "https://www.bcg.com/publications/2023/how-people-create-and-destroy-value-with-gen-ai"
    },
    {
     "title": "A team of researchers at Harvard, MIT, and Boston Consulting ...",
     "url": "https://www.facebook.com/aliansari0/posts/a-team-of-researchers-at-harvard-mit-and-boston-consulting-group-ran-758-working/27234889272817212/"
    },
    {
     "title": "There's a Time and Place for Generative AI - Boston Consulting Group",
     "url": "https://www.bcg.com/news/25october2023-theres-time-and-place-for-generative-ai"
    },
    {
     "title": "Harvard Study Reveals OpenAI GPT-4 Boosts Creative ...",
     "url": "https://themelan.com/harvard-study-reveals-openai-gpt-4-boosts-creative-consulting-capabilities-but-fails-on-strategic-analysis/"
    },
    {
     "title": "Harvard Business School Partners with BCG on AI Productivity ...",
     "url": "https://www.thecrimson.com/article/2023/10/13/jagged-edge-ai-bcg/"
    },
    {
     "title": "Generative AI Instantly Elevates the Aptitude of Knowledge ...",
     "url": "https://www.bcg.com/press/5september2024-generative-ai-knowledge-workers-consultants"
    },
    {
     "title": "Enterprise workers gain 40 percent performance boost from GPT-4, Harvard study finds",
     "url": "https://venturebeat.com/ai/enterprise-workers-gain-40-percent-performance-boost-from-gpt-4-harvard-study-finds"
    },
    {
     "title": "BCG tests GPT-4, the results will surprise you! (Harvard Study)",
     "url": "https://www.youtube.com/watch?v=HOqk7OyMbpI"
    },
    {
     "title": "This Harvard Business School study found Boston Consulting Group (BCG) consultants who used GPT-4 in their work finished on average 12% more tasks, 25% faster, with 40% higher quality. | Brad Smith",
     "url": "https://www.linkedin.com/posts/bradsmi_harvard-business-school-partners-with-bcg-activity-7134952239290290176-_v6e"
    },
    {
     "title": "Work 25% faster, 40% better: Harvard study finds consulting giant BCG's staff using ChatGPT outperform on every measure – and underachievers get biggest boost | Mi3",
     "url": "https://www.mi-3.com.au/20-09-2023/harvard-business-school-study-bcg-finds-knowledge-workers-using-chat-gpt-outperform"
    },
    {
     "title": "Enterprise workers gain 40 percent performance boost from GPT-4 ...",
     "url": "https://www.linkedin.com/posts/marshallmatt_enterprise-workers-gain-40-percent-performance-activity-7112186312157253632-gGSB"
    },
    {
     "title": "How AI is Redefining Strategy Consulting: McKinsey, BCG, ...",
     "url": "https://medium.com/@takafumi.endo/how-ai-is-redefining-strategy-consulting-insights-from-mckinsey-bcg-and-bain-69d6d82f1bab"
    }
   ],
   "citation_domains_for_reference": [
    "bcg.com",
    "facebook.com",
    "linkedin.com",
    "medium.com",
    "mi-3.com.au",
    "thecrimson.com",
    "themelan.com",
    "venturebeat.com",
    "youtube.com"
   ]
  }
 ]
}