Why
Prompts, models and retrieval settings change constantly. Without evals, every change is "looks fine to me" and regressions reach users silently.
The loop
diagram
Practical rules
- Start with 20–50 real examples, including failures users actually hit.
- Prefer code-based checks where possible. They're cheap and deterministic.
- For LLM judges, use specific rubrics and validate the judge against human labels.
- Track cost and latency alongside quality.
- Feed production failures back into the dataset. The eval set should grow every week.
Sources & further learning
Videos, courses, docs and books I recommend for this topic.
Related topics
AI EngineeringApplied in production
RAG (Retrieval-Augmented Generation) Architecture
Ground LLM answers in your own data — ingestion, chunking, embeddings, vector search, reranking, prompting with citations, and evaluation.
Intermediate5 sources
AI EngineeringLearning
AI Agents & Tool Use
LLMs that plan and act through tools in a loop — workflows vs agents, the agent loop, guardrails, and when not to build an agent.
Advanced4 sources