Retrieval is easy. Answers that stay inside your documents are not.

Most RAG demos retrieve chunks, call a model, and return fluent text. Fluent is the problem. The model will answer from memory when retrieval misses — and sound certain doing it.

For a founding engineer shipping the first LLM feature, that failure mode kills trust on day one.

The Failure Mode

User asks: “What is our refund policy for annual plans?”

Retrieval returns nothing relevant. The model still answers — pulling from training data or inventing a policy that sounds right.

The user acts on it. Support gets a ticket. Legal gets an email.

RAG without grounding rules is a hallucination machine with a search bar.

Minimum Pipeline

Four steps. No framework required.

1. Chunk documents (fixed size, overlap)
2. Embed chunks → store with source IDs
3. Retrieve top-k for the question
4. Generate answer from chunks only

Pseudocode for the ask endpoint:

def ask(question: str, collection_id: str) -> dict:
    chunks = retrieve(collection_id, question, limit=5)

    if not chunks:
        return {
            "answer": "I could not find this in the documents.",
            "sources": [],
        }

    prompt = build_prompt(
        question=question,
        chunks=chunks,
        rule="Answer only from the sources below. Cite source IDs.",
    )

    answer = llm.complete(prompt)

    return {
        "answer": answer,
        "sources": [{"id": c.id, "excerpt": c.text[:200]} for c in chunks],
    }

The rule in the prompt is not optional. It is the contract.

Grounding Rules

Hard rules for production:

Situation Response
No chunks retrieved “Not found” — no LLM call needed
Chunks retrieved Answer must cite source IDs
Answer cites ID not in retrieval set Reject and retry or fail
Low similarity scores Treat as empty — do not guess

Log every request with: question, chunk_ids, model, latency.

When something goes wrong in week three, you need the audit trail. Regulated domains need it on day one.

Evaluate Before You Tune Prompts

Founding teams skip this. They tweak prompts for a week instead.

Start with ten golden questions — questions you know the answer to, with known source documents.

GOLDEN = [
    {"q": "Refund window for annual plans?", "expect_source": "policy-v2#chunk-14"},
    {"q": "SLA for enterprise tier?", "expect_source": "sla-2026#chunk-3"},
    # ... eight more
]

def eval_retrieval(collection_id: str) -> float:
    hits = 0
    for item in GOLDEN:
        chunks = retrieve(collection_id, item["q"], limit=5)
        if any(item["expect_source"] in c.id for c in chunks):
            hits += 1
    return hits / len(GOLDEN)

Score retrieval hit rate first. If retrieval is below 80%, no prompt engineering will save you.

Fix chunking. Fix embeddings. Fix metadata. Then tune the prompt.

What Changes in Production

Notebook RAG optimizes for impressive answers. Production RAG optimizes for defensible answers.

Defensible means:

  • Every claim traces to a source ID
  • Empty retrieval returns empty answer
  • Logs support debugging and audit
  • Evaluation runs in CI before deploy

The model is the last step, not the first.

Key Takeaways

RAG fails when the model answers without evidence. Grounding rules fix that.

Pipeline: chunk → embed → retrieve → generate with cite-only prompt.

No chunks → say “not found.” Do not call the model.

Evaluate retrieval with golden questions before prompt tuning.

Log chunk IDs on every request. You will need them.

Founding engineers ship trust, not fluency. Grounding is how you ship it.