A legal-tech startup shipped a contract-summarization feature in late 2024. It cited “Section 8.3(b)” of an NDA that had no Section 8.3(b) — the document only ran to Section 6. Nobody caught it before a paying customer did, in front of their own lawyer. The model wasn’t broken. It was doing exactly what autoregressive generation does: producing the most statistically plausible next token, with no built-in mechanism that checks whether “8.3(b)” actually exists in the source text. That gap between fluent and true is the entire reason guardrail and verification layers exist.
What a Verification Layer Actually Does
A hallucination isn’t a bug in the traditional sense — there’s no stack trace, no exception, no failed assertion. The model produces well-formed, confident-sounding text that happens to be false, fabricated, or unsupported by the input it was given. Guardrails don’t fix the model. They wrap it, checking outputs (and sometimes inputs) against something more reliable than the model’s own confidence.
There are three broad layers, and most production systems use some combination of all three.
Input-side guardrails constrain what the model is allowed to reason about in the first place. Retrieval scoping, strict system prompts, and schema-constrained generation (forcing JSON output against a defined schema) all reduce the surface area for fabrication before generation even starts.
Output-side verification checks what came out. The simplest version is a grounding check: does every factual claim in the response trace back to a span in the retrieved context? This is usually implemented as a second LLM call — a smaller, cheaper model reads the source documents and the generated answer, then flags claims it can’t locate. Some teams use natural language inference (NLI) models instead, scoring entailment between source and claim rather than doing a full generative re-read, which is faster and cheaper per call but less able to catch subtle misattributions.
Structural constraints enforce format and citation requirements at generation time rather than after the fact. Forcing every factual sentence to end with a bracketed source reference, then programmatically verifying that reference resolves to a real chunk ID, catches the NDA problem directly — an unresolvable citation is a hard signal, not a probabilistic guess.
The trade-off nobody likes to say out loud: every layer you add increases latency and cost, and none of them get you to zero. A verification layer running on a second model call typically adds 300–900ms and roughly doubles your per-request token cost. You’re buying a lower hallucination rate, not immunity from it.
