LLM Accuracy on Regulatory Text: Hallucination Risks and How to Control Them

LLMs are as accurate on regulatory and legal text as the architecture around them. The evidence forms a ladder. General-purpose chatbots hallucinated on 58% to 88% of verifiable questions about real federal court cases (Dahl, Magesh, Suzgun and Ho, Journal of Legal Analysis, 2024, testing 2023-era models). Commercial legal research tools built on retrieval-augmented generation still hallucinated on 17% to 33% of queries (Magesh et al., Journal of Empirical Legal Studies, 2025). Grounded summarization of a single supplied document does far better, at roughly 2% to 15% for most models on Vectara's September 2026 leaderboard. What changes from rung to rung is the system around the model.

So hallucination on regulatory text is an architecture problem you engineer down rather than a model property you wait out, and most vendors would rather leave that unargued. The engineering is specifiable, testable and already expected by supervisors under rules written before ChatGPT existed. The deliverable is the Hallucination-Control Checklist for Regulatory AI below: six control domains, every control with spec language for a requirements document, a pass/fail test you can run in an afternoon, and the framework it maps to.

I run RegWatch, a platform that points LLM agents at regulators, so I have a commercial interest in you believing AI can do compliance work. That is why the numbers below include the ugly ones. The oldest studies here tested 2023-era models, and no benchmark is a guarantee for any particular system, mine included. You can still test the system in front of you, and the checklist shows how.

Hallucination rates form a ladder, and architecture decides which rung you're on

Here is the evidence chain, dated, with the caveats each study's critics would raise:

Three things follow that the raw papers never spell out for compliance buyers.

First, the spread within a rung is task difficulty, not luck. A 2025 study of LLM legal explanations for influencer-marketing compliance, run with small models on 1,143 Instagram posts under the Dutch Advertising Code, reached an F1 score of up to 0.93 across the dataset, lost more than 10 points on 95 ambiguous posts, and left out legal citations in 28.57% of the explanations the researchers annotated (Gui et al., arXiv 2510.08111). AIReg-Bench generated 120 fictional but plausible excerpts of AI Act technical documentation with an LLM, had legal experts annotate which Articles each one violates, and tested whether frontier models reproduce the experts' labels; its authors offer the result as "a starting point to understand the opportunities and limitations" of such tools, not as proof they are ready. Operationally: automate the unambiguous with sampling QA, and route ambiguity to humans by design.

Second, Dahl et al. found two failures scarier than wrong answers. The models often failed to correct questions built on false legal premises, and they struggled to predict their own hallucinations, so an ungrounded model's confidence is no guide to whether it is wrong.

Third, nobody on the ladder gets to zero. The best available answer to whether the next model generation fixes this, "not by scale alone," comes from the people with the strongest incentive to say yes.

OpenAI's researchers say accuracy will never reach 100%, and the remedy is behavior, not size

In September 2025, Kalai, Nachum, Vempala and Zhang published "Why Language Models Hallucinate". They argue that hallucination arises from statistical pressures inherent in pretraining, because some facts are not learnable from the training distribution, and that it is compounded by training and evaluation procedures that "reward guessing over acknowledging uncertainty." A model that answers "I don't know" scores zero on many benchmarks, while a model that guesses sometimes scores. Their prescription is to change what gets rewarded, penalizing confident errors and giving credit for appropriate expressions of uncertainty. In OpenAI's own summary, accuracy will never reach 100% because some real-world questions are inherently unanswerable, but hallucination is not inevitable, because a model can abstain when it is uncertain.

For a Chief Risk Officer, this converts an open research question into a procurement one. If hallucination were a maturing-technology problem, the rational move would be to wait. Since the fix is behavioral, the rational move is to demand compensating controls now: grounding, so the model has something true to be faithful to; refusal behavior, so uncertainty surfaces as "not found" instead of a guess; and verification, so the residual slips get caught.

Banking supervisors have long had a frame for this. Models err, and institutions cope through validation, ongoing monitoring and effective challenge. That was the logic of SR 11-7. On 17 April 2026 the Federal Reserve, OCC and FDIC replaced it with revised interagency model risk management guidance, SR 26-2. The new text keeps the principles for traditional statistical and non-generative models, and it states that "Generative AI and agentic AI models are novel and rapidly evolving. As such, they are not within the scope of this guidance," adding that a bank's own risk management and governance practices "should guide the determination of appropriate governance and controls" for tools it does not cover. That gap leaves the control specification for language models yours to write.

The failures that hurt compliance teams are misgrounded citations and wrong dates, not invented statutes

The popular image of AI failure, a chatbot inventing a "Directive (EU) 2019/2087" out of nothing, is real but also the easiest to catch, because a fabricated instrument fails the first lookup. The Magesh study documents the more dangerous type: misgrounded citations, where the cited source exists, resolves, and does not support the claim attached to it. Misgrounding survives the lazy check ("does the link work?") and fails only the expensive one ("does the source actually say this?").

Their diagnosis of why retrieval tools still fail names four causes: naive retrieval that pulls the wrong passages, retrieval of inapplicable authority (right topic, wrong jurisdiction or point in time), sycophancy toward the user's framing, and plain reasoning errors. The mix differed by tool. Naive retrieval and inapplicable authority dominated for Lexis+ AI, while reasoning errors were the largest cause for Westlaw AI-Assisted Research and Ask Practical Law AI, and sycophancy was rare. Better retrieval helps without fixing accuracy on its own, because generation still reasons badly from good sources. That is why verification and human gates belong in the design as well as in the policy.

The second underrated failure is dating. Regulators' pages often display a "last updated" or republication date rather than the original publication date, and an agent that faithfully reads the date on the page will mis-date the item. A mis-dated item does a compliance team as much damage as a hallucinated one: it fires a false "new development" alert or silently compresses an implementation window. No public benchmark we know of measures republication handling, in-force versus proposed labeling, or date-window discipline, which is why the checklist gives dates their own domain instead of treating "hallucination" as one undifferentiated risk.

The Hallucination-Control Checklist gives every control a pass/fail test and a framework hook

Spec language goes in your requirements document or vendor RFP. The test column is deliberately cheap; most take under an hour. The mapping column connects each control to a named source, because the fastest way to fund a control is to show an examiner or an auditor already assumes you have it. Frameworks referenced: NIST AI 600-1 (July 2024); FINRA Notice 24-09 (27 June 2024) and the FINRA 2026 Annual Regulatory Oversight Report (9 December 2025); the FCA's approach to AI; and the research cited above. None of them is a checklist for language models, so "maps to" means "the closest anchor," not "required by."

Domain A: Grounding and retrieval

Domain B: Citation and verification

Domain C: Date and version discipline

Domain D: Human-in-the-loop gates

Domain E: Vendor and procurement questions

Domain F: Monitoring and audit

Twenty-five controls looks heavy, but Domains A to C are properties a system either has or lacks (test once per vendor or build), Domain D is workflow design, and E and F are procurement and operations discipline. A team can run the full test column against one system in about two days.

Regulators expect governance, not a specific control set, and none offers a safe harbor

It is a common misreading to treat FINRA's notice or the FCA's position as a "new AI regulation." Both say existing, technology-neutral rules already apply, which is more demanding in one respect and less prescriptive in another: nobody tells you which controls to run, and everybody expects you to have chosen some.

• FINRA. Notice 24-09 (27 June 2024) says that if a firm uses generative AI tools as part of its supervisory system, its policies and procedures "should address technology governance, including model risk management, data privacy and integrity, reliability and accuracy of the AI model." It adds that it "does not create new legal or regulatory requirements or new interpretations of existing requirements." The 2026 Annual Regulatory Oversight Report (9 December 2025) defines hallucinations as "instances where the model generates information that is inaccurate or misleading, yet is presented as factual information," and describes ongoing monitoring of prompts, responses and outputs and human-in-the-loop review as practices firms may want to consider. In Regulatory Notice 26-14 (9 July 2026, a proposal to modernize Rule 2210 whose comment period closed on 11 September 2026), FINRA says generative AI communication tools can be part of a reasonably designed supervisory system "provided they are vetted, tested and monitored."
• FCA. The AI Update (22 April 2024) says the FCA's rules and core principles do not usually mandate or prohibit specific technologies, and the FCA's AI approach page states: "We do not plan to introduce extra regulations for AI." Firms remain accountable for outcomes whatever produced them. The joint Bank of England and FCA survey (21 November 2024) found that 75% of responding firms were already using AI, with a further 10% planning to within three years.
• US bank supervisors. As described above, the April 2026 interagency guidance (SR 26-2) leaves generative and agentic AI outside its scope and points banks to their own governance.
• NIST AI 600-1 (July 2024) gives the risk its most useful name, confabulation, defined as "the production of confidently stated but erroneous or false content," and maps GOVERN, MAP, MEASURE and MANAGE actions to it. For vendor-neutral internal-policy language, start there.
• EU AI Act (Regulation (EU) 2024/1689, in force 1 August 2024). Article 15 requires accuracy, robustness and cybersecurity for high-risk systems, and Article 15(3) requires the levels of accuracy and the relevant accuracy metrics to be declared in the instructions for use. After the Digital Omnibus, Regulation (EU) 2026/1744, those obligations apply from 2 December 2027 for Annex III systems and 2 August 2028 for Annex I products; the general-purpose AI obligations have applied since 2 August 2025. Be precise about scope: a regulatory-monitoring tool is typically not an Annex III high-risk system (Annex III point 8(a) covers AI used by or for judicial authorities to research and interpret facts and law), so Article 15 does not bind it directly. Use Article 15(3) as a procurement hook instead: ask any vendor to declare accuracy metrics for the intended purpose. If you deploy systems that are in scope, the deadlines are mapped in our EU AI Act compliance checklist for deployers.

And the record for uncontrolled AI output is no longer hypothetical. Damien Charlotin's AI Hallucination Cases database lists 2,097 court decisions worldwide involving hallucinated material as of 30 September 2026. In the database's export, 1,206 entries involve self-represented litigants and 825 involve lawyers, professionals trained in source verification who filed fabricated authority because the output looked checked. The database's scope also covers items other than citations, such as fabricated exhibits. Every filing with a fabricated authority would have died at Domain B, row two: click the citation, read the passage. The compliance analogue is an obligation register entry citing a provision that doesn't say what the entry claims. Nobody benchmarks that in public, so your controls have to catch it.

You can trust AI for grounded, cited, reviewed tasks, and you should not trust it for open-ended recall

Every page ranking for "can you trust AI for compliance" answers "it depends."

Trust it, at rung 2 and with Domain B and D controls, for: summarizing a supplied document; first-pass triage of retrieved items against a documented profile, with human review; comparing two versions of a text you provide; extracting structured fields (dates, thresholds, scope) from a named instrument, with citation pinning; drafting for human editing. These are grounded tasks. Raw error rates on Vectara's summarization benchmark run from about 2% to 15% for most models, far below the 58% to 88% of closed-book legal recall, and verification catches most of the residue. Note the direction of that number: it is not zero, and it moved when the benchmark got harder, so keep the sampling QA in Domain D running.

Do not trust it for: open-ended legal or regulatory research from a chat interface (rung 4, the 58% to 88% rung); any citation you have not resolved and read (the 2,097-decision rung); deadline or in-force-date extraction without Domain C discipline; ambiguous applicability judgments, where measured accuracy drops by double digits; anything uncited. The specific ways an unscoped chat interface breaks on monitoring work are documented, with a reproducible stress test, in Can you use ChatGPT for regulatory horizon scanning?.

The profession has, in aggregate, priced this about right. The Thomson Reuters Institute's 2026 AI in Professional Services Report, a survey of 1,514 professionals conducted in October and November 2025, found that four in ten respondents say their organizations are using generative AI, up from 22% the year before, while among those opposed to applying it, "worries over reliability and accuracy" are the primary reason. Practitioners believe the task fit and distrust the failure mode. Both instincts are right, and human-in-the-loop architecture reconciles them, with review effort concentrated where the evidence says errors cluster, in ambiguity, citations and dates.

Two cheap tests show which rung of the ladder your AI is actually on

Run the two cheapest tests from the checklist against whatever AI touches your compliance workflow today: the non-existent-instrument probe (Domain A) and the 20-output citation resolution audit (Domain B). Together they take half a day and tell you which rung of the ladder you are actually on, as opposed to the rung the sales deck claims. A vendor that declines to run them on a live screen has answered the question anyway.

Publishing a scorecard you do not apply to yourself is the oldest trick in vendor content, so RegWatch should be held to its own checklist. Parts of Domains B, C, D and F describe behavior you can check in the product rather than policy. The Compliance Assistant answers from an organization's own alerts, obligations and policy text, plus the attachments and web search it is given, and its answers carry structured citations (B). Monitoring runs receive an explicit "today" and a lookback floor, a second date check runs before a finding is saved, and each finding keeps its source URL, dates and a verbatim excerpt (C). Triage records every decision to accept, reject or defer, with a written "Why this matters" that a compliance officer can read and overrule, and dismissing an alert requires a reason (D). Decisions go to an append-only audit log that is hash-chained and tamper-evident (F). I would still run the two cheap tests on our own assistant, and I would like prospects to bring the Domain A and B tests to our demos, because no vendor can self-certify its own accuracy. The broader case for the agent architecture is in AI agents for regulatory compliance.

We expect models to keep improving and regulated firms to still be running every control in this checklist five years from now, because accuracy never reaches 100% and supervisory logic, from FINRA's 2026 report to NIST's confabulation profile to the FCA's insistence that existing rules apply, treats accuracy on regulatory text as a property you operate. Other pieces on AI tooling for compliance are collected in the AI compliance hub.

---

This article is general information, not legal advice.