How often do AI models state official economic statistics correctly from memory, with no tools and no retrieval? A pre-registered measurement with frozen question banks, ground truth audited against the primary statistical APIs, and every protocol deviation logged in public. Run quarterly.
Language models answer statistical questions fluently whether or not they know the number. That failure mode is now institutionally documented: the IMF's own paper introducing StatGPT, a retrieval platform backed by the IMF, World Bank, OECD, Eurostat, BIS and the UN statistical system. The paper justifies retrieval precisely because generative models "perform poorly at delivering official statistics" (IMF Departmental Paper 2026/004). Independent studies point the same way: the EBU/BBC study of 3,000 AI news answers found significant issues in 45% of them, and the Tow Center found over 60% of AI search answers carried citation problems.
What none of that research isolates is the specific, checkable case this benchmark measures: a model asserting a precise official figure, such as an inflation rate, a debt ratio or a GDP level, for a specific country and year, from parametric memory alone. Those are the numbers that end up in reports, briefs, and articles. Each one is verifiable against the official series, so the error rate can be measured instead of argued about.
The methodology, scoring bands, and analysis plan are frozen and committed publicly before any model is called. Changes after the fact are logged as numbered deviations in a public file, never silently absorbed.
Question banks are drawn with a seed derived from a pre-announced NIST Randomness Beacon pulse. The announcement is committed before the pulse exists, so the draw cannot be cherry-picked.
Every answer key value is frozen, then independently re-fetched from the primary statistical APIs (World Bank, IMF) by audit code that shares nothing with the serving path. Divergences fail the audit.
By publication covenant, accuracy (Within-Tolerance Rate) is never quoted apart from the Confabulation Rate and the Answer Rate. A model that answers everything confidently and one that declines when unsure are different animals. One number would hide that.
Official statistics get revised. A model whose answer matches an earlier official vintage is never scored as confabulating. StatCite's own verify tool extends the same courtesy to human claims.
Every model's raw responses ship with each run, so every scored verdict can be re-derived from the artifacts. Disputes are checkable, not rhetorical.
Full methodology, covenant, deviations log, question banks, and scoring code: github.com/asokore/statcite/bench.
Six models across three vendors, 100 questions, answered from memory only. The three columns are a single unit, never quote one without the others.
| Model | Within-Tolerance Rate | Confabulation Rate | Answer Rate |
|---|---|---|---|
| claude-opus-5 | 82.0% | 15.5% | 97.0% |
| claude-fable-5 | 81.0% | 15.6% | 96.0% |
| gemini-3-flash-preview | 79.0% | 20.2% | 99.0% |
| gpt-5.5 | 75.0% | 22.7% | 97.0% |
| claude-sonnet-5 | 64.8% | 31.4% | 94.5% |
| claude-haiku-4-5 | 33.3% | 32.5% | 49.4%* |
*Below the pre-registered 70% Answer Rate threshold for interpretation. It is reported for completeness, per the methodology's own rule. At this sample size (Wilson 95% CIs, minimum detectable model-to-model difference roughly 12–15 percentage points) this is not a league table. Read the full R1 report and the post-publication sensitivity analyses. They include a parse-policy sensitivity that affects one model's classification, published there rather than buried.
The plain reading: even the strongest models confabulate precise official statistics at meaningful rates when answering from memory. Roughly one in six to one in three answered-wrongly-with-confidence, depending on the model. That is the baseline verify_stat exists to fix.
Run 2 extends Run 1 with a mechanically applied model roster, a NIST-beacon-seeded fresh draw with a carried contamination-control panel, 25 null probes, an as-deployed settings arm, and a retrieval-delta arm. All of it was pre-registered in the public deviations log before any model was called.
Per the pre-registered publication sequencing, Run 2 results are shared with the covered vendors for a courtesy-preview window before they are published here. No results appear anywhere until that window closes. When they do, the per-question data will be downloadable so every verdict can be checked.