An Agent You Can't Measure Is One You Can't Trust
The eval harness — not the model — is what let a production RAG agent survive a GPT-4 to GPT-4o migration and cut factual errors ~60%.
Most AI demos are judged by vibes. Someone types a question, the answer looks plausible, everyone nods. That works right up until the system is in front of 100+ support agents making decisions on its output — at which point "looks plausible" is not a quality bar, it's a liability.
When I built a production RAG system at Innovcentric, the pipeline wasn't the hard part. The eval harness was. It's also what made the difference between a clever demo and something I'd put my name on in production.
Why eval is the product, not the QA step
A RAG answer can fail in ways a human reviewer won't catch at a glance: it can be fluent and wrong, cite the right document for the wrong claim, or quietly drift when you swap the underlying model. You cannot eyeball your way out of this at scale. You need a harness that scores every answer the same way, every time.
I built that on RAGAS for the metrics that matter — faithfulness (is the answer grounded in retrieved context?), relevance, and hallucination scoring — wired through LangSmith for trace-level observability so I could see which retrieval or generation step produced a bad answer, not just that one happened.
The migration test
The real proof came during the GPT-4 → GPT-4o migration. A new model is a silent behavior change: cheaper, faster, and subtly different in ways that don't show up until a customer hits the edge case. Without an eval harness, you migrate on faith and find out from a bug report.
With one, the migration became a measurement. I ran the same eval set against both models, watched faithfulness and hallucination scores side by side, and only cut over when the numbers held. Drift monitoring kept watching after the switch. The harness turned a scary upgrade into a routine, reversible decision.
Prompts as version-controlled artifacts
Fifty-plus prompt templates — chain-of-thought, few-shot, instruction-tuned — each with safety guardrails and prompt-injection defense, all version-controlled and reviewed like code. No tuning prompts live in production. A prompt change is a diff that goes through eval before it ships, same as any other code path.
The result
Roughly a 60% cut in factual errors — and, more importantly, the ability to say it was 60% with a number behind it. The eval harness is what made every other claim in the system defensible.
An agent you can't measure is an agent you can't trust. In production, trust is the only metric that pays the bills.