How do we stop a model inventing an answer when the file does not contain one?
A model's default is to answer, because every question in its training data was followed by one. Refusal is a behaviour you build, structure and then test against questions your own documents cannot answer.
Why the default is to answer
A language model does one thing: it continues a pattern. A question is a pattern that, in the material it was trained on, was almost always followed by an answer. Silence is a rare continuation, and a refusal is rarer still. Then instruction tuning adds a second pressure in the same direction — models are rated on helpfulness, and a helpful assistant answers. So the model is not lying to you when it constructs an obligation that the lease does not contain. It is doing the only thing it knows how to do, with the material it has, and it has no separate channel for saying that the material is insufficient. The difference between an answer that is supported and an answer that is plausible is not visible in the prose. Both are fluent, both are grammatical, and in legal language plausibility is high, because contracts are formulaic. A fabricated clause about notice periods reads exactly like a real one. This is why abstention is not a nicety. It is the boundary between a tool that assists a fee-earner and a tool that quietly contaminates a file.
Three different failures that look identical
When a model fails to answer, it may have failed in one of three ways, and the tool will show you the same thing in all three cases. First, the answer is genuinely absent. The file does not contain the notice period, and the honest output is that it does not. Second, the answer is present but was not retrieved: the clause you needed sits in a schedule, and the retrieval step pulled six passages from the body of the document instead. Third, the answer was retrieved and the model still missed it, which is an attention problem, not a retrieval one. Only the first case should produce a refusal. The second and third are tooling defects, and they are worse than a plausible invention in one respect: a confident refusal looks like good governance, so nobody investigates. If your harness only counts refusals and never checks whether the answer was available, you can score a broken retrieval layer as exemplary caution. Count both. A refusal should be a finding about the documents, and you should be able to say which documents were in front of the model when it refused.
How to build refusal into the workflow
Abstention is created by instruction and enforced by structure. The instruction has to be explicit and negative: if the provided excerpts do not contain the answer, say so, and say what you would need instead. Vague encouragement to be careful does not survive contact with a long prompt. Structure does the rest. Ask for an evidence field for every substantive answer — the quoted words and where they came from — and make that field required by the output format. Now abstention has somewhere to live, and the output becomes checkable: if the quote is not in the document, you have caught a problem that a paragraph of prose would have hidden. Ask the model to list the questions it could not answer from the material, so that the gap becomes an agenda item rather than a silence. And keep the retrieval scope visible in the output, so a reader can see what was searched. The last piece is the hardest to maintain: a rule against bridging. Models will find adjacent-but-not-answering material and reason across the gap — the schedule mentions a notice period for a different right, so the model borrows it. Say plainly that adjacent material is not an answer, and test for it.
Testing refusal with unanswerable questions
You cannot test abstention with questions your documents answer. Build a small unanswerable set and treat it as a permanent fixture of your evaluation. Twenty questions is enough to start. Write them against a closed or permissioned matter, and include three kinds. Fully absent: an obligation the documents never touch. Near-miss: an adjacent clause exists but does not answer the question, which is where bridging shows up. False-premise: a question that assumes a fact the documents contradict or never establish, which tests whether the model corrects the premise or accepts it. Run the set with your intended prompt and settings, and mark each output by hand: refused, answered from evidence, or invented. Run the same set again when you change model version, quantisation or retrieval settings, because abstention is the behaviour that degrades first and is least visible when it goes. Then run it in a real workflow, not just in a test harness — behaviour under volume, with a fee-earner who wants an answer, is the only measurement that counts.
Making refusal a record rather than a silent failure
Refusals are findings, and findings belong in the file. A note recording that the documents supplied do not contain the notice provisions is a supervised outcome, and a useful one: it tells the next reader what was checked and what the file lacks. It also protects the firm, because an assertion of absence — the bundle contains no restriction on assignment — is only as good as the completeness of what the model read. So record the refusal, the question and the documents in scope. Distinguish it in the record from a question nobody asked, because a supervisor reading the file later cannot tell the difference otherwise, and the two mean very different things about how the matter was run. One caution. Abstention instructions are a control, not a guarantee. They reduce the rate of invention; they do not eliminate it, and their effect varies between model versions. Nothing in the reviews or the explainers on this site is a substitute for checking, on your own matters, whether the behaviour you asked for is the behaviour you get. Where we do not know how a given model behaves under your conditions, we will say so — and so should your file notes.
What to watch
- A confident answer to a question the retrieved excerpts do not answer, arriving with no hedge at all
- A hedge followed by an invention: the file does not state the notice period, and then a notice period appears in the next sentence
- Counting refusals as success without checking whether the answer was sitting in a document the tool never retrieved
- Bridging from adjacent material — an obligation borrowed from a clause that governs something else
- Abstention tested once in a demonstration with five tidy documents and never re-tested on a real bundle
- An instruction to abstain in the prompt while the output schema has no field in which to record it
Write your refusal rule into the system prompt as an explicit instruction with a named response — if the excerpts do not answer the question, say so and state what is missing. Add a required evidence field for the quoted words and their location, so an invented answer has to carry a quotation that does not exist. Then build a twenty-question unanswerable set from one closed matter and run it whenever the model version, quantisation or retrieval configuration changes. Log refusals in the matter record with the documents in scope, and distinguish them from questions never asked. Do not treat the instruction as a guarantee. Test it.