The pipeline
Decisions that decide quality
| Decision | Guidance |
|---|---|
| Chunking | Respect document structure (headings, code blocks); add the title and breadcrumbs to each chunk |
| Retrieval | Hybrid (BM25 + vectors) beats either alone; filter by metadata (team, product, date, permissions) |
| Reranking | Retrieve ~50, rerank to ~5. A large relevance boost for little cost |
| Prompting | "Answer only from context; say 'I don't know'; cite sources" |
| Freshness | Incremental re-index on doc change events |
| Security | Enforce the user's permissions at retrieval time, never in the prompt |
Evaluate it like software
Build a golden set of 50–200 real questions with expected sources. Measure retrieval recall@k, answer faithfulness (grounded in context?) and answer relevance. Re-run it on every change to chunking, the embedding model or the prompt. See LLM Evals.
I built an internal RAG knowledge app over our engineering docs: chunk → embed → Pinecone → an LLM answers from retrieved context. It cut down repeated onboarding and "where is this documented?" questions.
Sources & further learning
Videos, courses, docs and books I recommend for this topic.
Related topics
AI Agents & Tool Use
LLMs that plan and act through tools in a loop — workflows vs agents, the agent loop, guardrails, and when not to build an agent.
Model Context Protocol (MCP)
An open protocol that standardises how AI apps connect to tools and data — build a server once, use it from any MCP-capable client.
LLM Evaluation (Evals)
Test AI features like software — golden datasets, code-based and LLM-as-judge graders, regression gates in CI, and production monitoring.