bearout

Why models add things the sources do not contain

A language model does not look anything up while it writes. Conceptually, it produces text one word at a time, using a statistical guess based on the pattern of what came before. The machinery that produces a true sentence produces an invented one in the same way, so a fabricated fact can sound like a real one, and nothing inside the model marks the difference between the two.

Relying on the model’s recall is fragile wherever an answer has to match what particular documents actually say, as it must when the source is a statute and its regulations, the terms of an insurance policy, a compliance manual, or a contract. Some of the material was never in the model’s training data at all, such as a policy written after the model was trained, or a set of private files. Even when it was, generation leans toward whatever was common in that data, so a specific provision or an unusual exception tends to come back smoothed toward the more typical case. Both problems have the same remedy, which is to fetch the relevant documents at the moment a question is asked, pass them to the model, and tell it to answer from those. This is the classic Retrieval Augmented Generation (RAG) approach, and handing the model the exact text at question time removes the need to rely on its recall.

The difficulty is that “answer from these documents” is an instruction rather than a constraint. The sources arrive as more text in the prompt, and the model carries on generating exactly as it did before, leaning on what it was given most of the time but not all of the time. In an ordinary pipeline there is nothing further down the line that catches the inconsistencies.

So whatever the model confabulated survives all the way to the reader. It might be a figure that was never stated, a deadline that sounds plausible, or a qualifier such as “immediately”, “in every case”, or “within 30 days” that quietly changes what the source actually said. Errors of this kind are the dangerous ones because they arrive wrapped in correct material and correct citations, so the answer as a whole looks properly sourced even though one sentence in it is not.

The consequences are no longer hypothetical. More than 1,600 court cases involving AI-fabricated legal citations had been logged as of August 2026, with sanctions, fines, and professional suspensions attached, in a running tally kept by Damien Charlotin’s AI Hallucination Cases database. That count covers one profession, and within it only one kind of mistake. It exists at all because court rulings are public, so a fabricated citation ends up on the record where somebody can find it. Most work is never published that way, and the same mistakes there go uncounted.

bearout is an anti-confabulation architecture distilled from a prototype that answered consumer questions about Ontario’s new-home-warranty scheme, where the cost of a wrong sentence is not a poor demonstration but a homeowner who now holds a false belief about their own rights.

How the gate decides what ships

Many tools in this area are evaluation libraries. They score a batch of answers the system has already produced, and the score goes to a dashboard or a report, where an engineer reads it and decides whether to take action, after the answers have already reached their readers. bearout instead sits in the live path between the model’s draft and the reader and makes the decision itself, so the answer the model drafted is not always the answer that reaches the reader.

The answer is split into sentences, and each sentence is checked against the retrieved sources by a strict judge. A sentence passes only if everything it asserts, including every figure, date, condition, and qualifier, is either stated in the sources or follows directly from them. Rewording the source material is acceptable, but adding anything to it is not.

The behaviour around that check matters more than the check itself.

It fails closed. The gate can also break in its own right, and every such path leads to removal rather than passage. Anything that is not an affirmative pass counts as a failure, so if the judge errors, the network drops, or the reply comes back unreadable, the sentence is treated as unsupported and removed. A gate that jams shut when it breaks still protects the reader, because the worst it can do is withhold an answer. A gate that swings open when it breaks stops protecting anyone at precisely the moment its own machinery has failed.

It will refuse to answer. When too much of the draft has been removed, the whole answer is withheld rather than passed along as a remnant, because a half-answer that has lost its conditions and exceptions still reads as though it were complete, which makes it more dangerous than no answer at all. The caller gets back the verdicts rather than text to display, and has to tell the person that the question could not be answered from the sources at hand.

Every decision is explainable. The gate returns the surviving text together with the exact sentences it removed and its verdict on each one, so a system that keeps those outcomes can say, months later, exactly why a particular sentence disappeared.

Checking the library as well as the answer

Every tool of this kind checks the answer against the retrieved passages, which assumes that those passages are faithful copies of the real source, and quite often they are not.

Documents get amended, scrapers break quietly, and a parser can drop the last three subsections of a section without anything raising an alarm, because a system that indexed 43 of 45 sections looks identical from the outside to one that indexed all 45, at least until somebody happens to ask about the missing two. A grounding gate that is working perfectly will still certify an answer against a passage that no longer matches the law.

For that reason bearout also verifies the library itself against the authoritative source. Ontario publishes its statutes and regulations through a machine-readable interface, and the corpus is checked against that interface in three layers, the later ones catching classes of failure the earlier ones cannot see.

• L1

  Has the source changed? Re-fetch the official text, rebuild the exact passage the system would store today, and compare it against what is currently stored. Any difference means that either the law moved or the parser did.

• L2

  Did we read it correctly? Re-extract the document with a second and deliberately different mechanism, so that every stored passage must also appear in that independent reading. If the same parser were used to check its own work, a consistent bug would pass every time, which is why the second reader shares none of the first one’s machinery.

• L3

  Is anything missing or invented? Build an independent inventory of every section in the official text and compare it against the library in both directions, looking for sections the source has that the library lacks as well as sections the library holds that the source no longer shows.

The most recent run against the warranty prototype’s corpus verified 418 of 418 sections across the nine Ontario statutes and regulations its answers relied on, comparing each stored section against the official text published through the province’s e-Laws service. A perfect score in one run would not mean much on its own. The check is built to run on a schedule, so it would flag the day the stored corpus stopped matching the official text.

How often the gate itself is wrong

The gate’s judgements are themselves made by a language model, and language models are sometimes wrong, so the obvious question is how often the gate is wrong and in which direction it errs. Comparable tools seldom answer that question.

There are exactly two ways for the gate to be wrong, and they are not equally serious. Airport screening offers a familiar comparison, in that one kind of mistake confiscates a harmless water bottle and wastes everyone’s time, while the other kind lets something dangerous through.

Measured on a sealed set of 474 sentences, each labelled with the verdict a correct gate would give it, supported or unsupported, so the gate’s decisions can be marked against a known answer. For the 153 real sentences the system produced, that label was decided by majority vote of three independent AI judges, separate from the judge being measured; the rest were constructed so their labels are known by design. The rates work out to roughly one supported sentence in five removed unnecessarily and roughly one unsupported claim in forty getting through.

The gate is intentionally configured to be wrong in the safe direction far more often than in the harmful one. The judge is instructed to answer “unsupported” whenever it is in doubt, which is where the over-caution comes from, because a missing sentence is a smaller failure than a wrong one in this particular domain. Rewording that instruction moves both rates together rather than improving either one alone. A different domain would want a different point on that trade, and anyone thinking about using this package can see where this one sits before adopting it.

Comparable figures are also hard to find. Several major vendors ship this class of checker, and some of them publish scores on public benchmarks, but a benchmark answers a different question, namely how a detector ranks against other detectors on someone else’s examples rather than how often this gate, running on this corpus, removes a sentence it should have kept.

A grounding checker with no published error rate is asking to be trusted on nothing more than its own say-so, and an unflattering number is not a reason to withhold anything, since the unflattering numbers are the ones that make a disclosure worth reading.

How the measurement is kept reliable

A measurement is only as trustworthy as the method behind it, and when you measure your own work it is easy to make mistakes that are biased in favour of the result. Each of the five practices below exists to block one specific mistake of that kind. They were applied here to the warranty prototype’s numbers, but nothing about them is particular to that project. The published rates describe one judge on one corpus, and anyone adopting the package on a different corpus will need to run the same measurement themselves, with the same discipline, for their own number to mean anything.

1. The test set is sealed

   The 474 sentences are frozen, and a fingerprint of their content identifies the exact set behind every published number, so nobody can quietly swap or reword a test item without the change being visible. The sentences are never published and never used to tune the system, for the same reason an exam stops measuring anything once the students have seen the answer key. What is published is the methodology, the per-item verdicts, and the results, and that is enough for somebody outside the project to recompute the rates without ever seeing a sentence.

2. The labels came from blind judges, not from the system

   The real sentences in the set were labelled by majority vote of three independent AI judges that could not see what the system had decided, and disagreements between them were recorded rather than smoothed over, with five of the 153 coming back split two to one. A stricter second-pass audit then re-checked every label, the ones it contested are set aside for expert review, and a label that changes produces a new fixture version rather than a quiet edit. The trap this avoids is grading an answer sheet you wrote yourself. For this corpus, three independent judges in agreement were considered sufficient, and nothing in the method requires the labellers to be models, so an adopter who wants human labels can run the same review with people.

3. Hard cases were constructed on purpose

   More than half of the set consists of sentences that have been deliberately corrupted in exactly one way, whether by a changed number, a wrong section reference, an inverted condition, an added qualifier, a swapped actor, or a different outcome. These are the near-misses a lazy checker waves through.

4. Changes are declared before they are measured

   Before any modification is tested, the design and the standard it will have to meet are written down and committed. Deciding after the fact what counts as success is how a disappointing result gets rewritten into a promising one, and writing the bar down in advance makes that move impossible to perform quietly.

5. Only two attempts are allowed

   Each candidate design gets at most two tuning passes against the test set, and that limit is fixed in advance. Given unlimited attempts almost anything will eventually produce a good-looking number, since a design that is retuned often enough stops being tested by the set and starts being fitted to it.

A redesign that failed

Checking a whole sentence at once has an obvious weakness, in that a sentence can carry five separate facts and a single shaky clause will condemn the other four along with it. The natural fix is to break each sentence into individual claims, check those separately, and rewrite the answer without the failed ones instead of deleting whole sentences. That gives the judge smaller questions and the reader tidier output, and it was worth testing properly.

It was declared in advance, built, and measured against the same sealed set.

All three runs were scored against the original label set, so the designs are compared on the same ground. That is why the shipped design reads 20.5 and 2.1 here rather than the 20.2 and 2.4 given earlier, which are the same verdicts rescored after two labels were corrected. Every figure is recomputable from the per-item results published alongside the code.

The more elegant design removed roughly twice as many good sentences as the one it was meant to replace, and it was not shipped.

The reason it failed is that a sentence survives only if all of its claims survive, so splitting one sentence into four claims means needing four correct judgements where one would previously have done, and errors accumulate across them rather than cancelling out. That arrangement only pays off if judging a small claim is substantially more reliable than judging a whole sentence, and the measurement showed that it is not. A claim pulled out of its surrounding sentence also loses the context that made it interpretable, which introduces mistakes of its own.

The redesign did fix the specific example that had motivated the whole effort, a case that had been irritating for weeks in which the old approach deleted a plainly correct sentence. Across all 474 items, however, it remained roughly twice as destructive. The compelling example pointed in one direction and the measurement pointed in the other, and the measurement was allowed to decide.

The rejected code ships in the repository, not as a feature but in a folder marked as evidence, alongside its declared standard, its results, and the post-mortem. An account of a failed design is easier to believe when the failure itself can be run.

Boundaries and caveats

Neither the checking nor the refusing is novel. Several commercial products check grounding sentence by sentence, and at least one open-source library refuses by default rather than handing back a score, which is what this gate does. The narrower claim that survives a look at what those products actually document is about the numbers: several publish a benchmark score for their checker, and one publishes precision and recall, but a false-strip and false-pass pair measured on the configuration the product actually ships is not something the field currently offers. The part that appears to be uncontested is verifying the library against its authoritative source.

The numbers describe one domain and one judge. They were measured on consumer legal information, with a specific model, using specific prompt text, and changing the wording of the judge’s instructions moves the operating point. The figures show that this kind of measurement is possible and worth doing, but nobody inherits these particular rates by installing the package.

The judge is replaceable, and probably should be replaced. The judge behind the rates above is a large general-purpose model prompted to verify, and the checking step sits behind a deliberately small interface so that something better fitted can take its place. The most promising candidate is a small open-weight model trained specifically for this verification task, which is the most likely route to a lower false-strip rate. An off-the-shelf detector measured on the same fixture missed nearly a third of the fabrications, so the training would need to be on material like this. Any replacement judge is a different operating point and would need its own measurement.

bearout is an open-source demonstration of these mechanisms, extracted from the prototype they were built in and released under the Apache 2.0 licence. The Python package holds the gate, the corpus verifier, their test suites, the published results, and the experiment that failed. It is hand-rolled on the standard library rather than built on LangChain, LlamaIndex, or any other LLM framework, the core installs with no dependencies at all, and the judge sits behind a single small protocol that any backend can implement. None of it claims to make a system hallucination-free. What it claims is narrower and checkable, which is that every sentence either survives verification against its cited source or is suppressed, and that the verifier’s own error rates are measured and published.