Retrieval is the easy part. The hard part is evaluation, freshness and knowing when the model should say "I don't know."
Chunk the documents, embed them, drop them in a vector store, wire up a similarity search — most teams can get a RAG demo working in an afternoon. It looks finished. It is not finished. The gap between a demo and a system people trust is almost entirely in the parts that don't show up in a screenshot.
Retrieval quality matters, but it's a solved-enough problem: hybrid search (keyword plus vector), reranking, and sensible chunk sizes get you most of the way there. The harder questions start once the right document is sitting in the context window and the model still has to decide what to do with it.
Most teams ship RAG with no evaluation harness beyond "it looked right when I tried it." That works until the knowledge base grows, the questions get harder, or a document changes underneath the answer. By then nobody notices a regression until a customer does.
A useful evaluation set doesn't need to be huge — fifty to a hundred real questions with known-good answers, scored for both retrieval (did the right chunk come back) and generation (did the model use it correctly) separately. Conflating the two hides which half of the pipeline is actually failing.
Stale documents are the single most common cause of confidently wrong RAG answers. No amount of prompt engineering fixes an embedding for a pricing page that changed six weeks ago and was never re-indexed.
Treat the ingestion pipeline as a first-class piece of the system, not a setup script you run once. Source-of-truth ownership, re-indexing triggers, and a way to invalidate stale chunks matter more than which embedding model you picked.
A model that answers confidently from a weak or irrelevant retrieval is worse than one that declines. Setting a similarity threshold below which the system falls back to "I don't have enough information" is unglamorous, but it's the single change that does the most for user trust.
Ship the abstention path before you ship the clever prompt. It's the difference between a tool people rely on and one they quietly stop using.