Why AI Products Need Evals
A few months ago, I made a change to a prompt. Small change, a couple of extra lines of instruction, nothing dramatic. I ran it against the five or six examples I always test with, it looked great, and I shipped it.
Two weeks later, someone flagged that a whole category of inputs had started failing. Not loudly, just quietly worse. The kind of regression nobody notices until a user does.
I hadn’t broken anything on purpose. I just had no way of knowing I’d broken something at all. I was testing on vibes: “this looks better to me” was my entire QA process. And if you’ve shipped an AI feature, I’d bet you’ve done the exact same thing — because for a long time, that’s all most of us had.
That’s the gap evals are built to close.
What an eval actually is
Here’s the metaphor I keep coming back to: unit tests tell you if your code runs. Evals tell you if your AI is actually right.
Traditional software is deterministic — same input, same output, every time. So a unit test is simple: does the function return what it’s supposed to? AI systems don’t work that way. The same prompt can produce a slightly different answer each time, and “correct” is often a matter of degree, not a pass/fail switch.
Think of it like a driving test. Passing your license exam once, on a clear day, in an empty lot, doesn’t tell you much about how you’ll drive in heavy rain, at night, in unfamiliar traffic. A good driving test — and a good eval — deliberately covers the range of conditions you’ll actually face, not just the easy case you’re most comfortable with.
An eval, at its core, is a curated set of test cases — representative inputs paired with a way to judge whether the output was good — that you can run every time something changes.
How it actually works
A few pieces come together to make this practical:
A golden dataset. A set of inputs that represent the real range of what your product handles — not just the happy path, but the edge cases, the messy inputs, the ones that broke last time.
A way to score the output. This can be exact-match (does the output match a known-correct answer), a rubric scored by a human, or increasingly, “LLM-as-judge” — using a model to grade another model’s output against criteria you define. Each has trade-offs: exact-match is cheap but rigid, human review is accurate but slow and expensive, LLM-as-judge scales well but needs its own calibration.
A gate, not a one-time check. The real shift is treating evals as something you run before every change ships — a new prompt, a model swap, a tweak to your retrieval logic — rather than something you do once at launch and never revisit.
Tracking over time. A single eval score is a snapshot. What matters more is the trend: did this change move the number up or down, and did it move it down somewhere you weren’t watching?
Ownership. One thing that trips teams up early: who actually maintains the eval set? It can’t just live in one engineer’s head or one notebook they run locally before a demo. The best setups I’ve seen treat the eval set like a shared asset — versioned, reviewed when it changes, and visible to the PM, not just the person running the model. If only one person on the team knows what “good” looks like, you don’t have an eval system, you have a bottleneck.
It’s also worth distinguishing two different rhythms of evaluation, because teams often only build one:
Offline evals — run against your golden dataset before anything ships. These catch regressions before users ever see them.
Online evals — monitoring real production traffic after launch, often via sampling and LLM-as-judge scoring, or through implicit signals like thumbs-up/down, edit rates, or task abandonment. These catch the drift that offline tests can’t, because real usage always finds inputs your golden dataset didn’t anticipate.
Most mature AI products need both. Offline evals are your seatbelt before you ship. Online evals are the mirror that tells you what’s actually happening on the road.
What this looks like in practice
I’ve worked on a document-processing system that extracts structured and unstructured data from a high volume of incoming documents — invoices, forms, and similar — across a lot of different formats. Accuracy sat in the low-to-mid 90s, which sounds solid until you remember what “5-10% wrong” means at real volume: real transactions, real downstream decisions, real people fixing avoidable mistakes.
At one point, a team improved handling for one document type significantly — a genuine win, worth celebrating. But because the eval set spanned document types instead of just the one being tuned, it also surfaced that the change had quietly dragged down accuracy on a different, less common document type. Nobody would have caught that by eyeballing outputs. It only showed up because the held-out eval set was broad enough to catch a regression the team wasn’t even looking for.
What made this catchable at all was fairly unglamorous: a dataset built ahead of time from real historical documents across every category the system handled, not just the ones currently being worked on, with expected outputs already verified. When the change went in, the eval run flagged a drop in one category within minutes — not two weeks later, from a user complaint. That’s the entire difference in outcome: a five-minute fix during development versus a support ticket, a root-cause investigation, and an apology to a customer.
It’s also worth saying plainly: the eval set didn’t design itself. Someone had to decide which document categories mattered enough to include, how many examples per category was “enough” to trust the number, and how to handle documents that were genuinely ambiguous even to a human reviewer. Those are exactly the kinds of calls a PM should be in the room for — because the eval set implicitly defines what your team considers “correct,” and that definition shapes every decision downstream of it.
That’s the real value of evals: not proving you’re right, but catching the ways you’re wrong that you’d never think to check for manually.
What this means for you as a PM
Evals aren’t just an engineering concern — they’re a product decision, and one you should be involved in from the start.
Someone has to decide what “good” means before anyone can optimize toward it. Is a wrong answer worse than a slow one? Is “close enough” acceptable for this use case, or does it need to be exact? Those are product calls, not just technical ones, and if you leave them entirely to engineering, you’ll end up optimizing for whatever’s easiest to measure rather than what actually matters to users.
Evals also give you a real answer the next time a stakeholder asks, “did the roadmap bet pay off?” Without them, your honest answer is “it feels better.” With them, you can say “accuracy went from X to Y on our held-out set, and here’s what regressed along the way.” One of those holds up in a roadmap review. The other doesn’t.
Getting started, without overbuilding it
None of this requires a mature ML platform on day one. If you’re a PM trying to get your team’s first eval off the ground, the smallest useful version looks something like this:
Pull 20–30 real examples from actual usage or support tickets — not ones you invent, ones that already happened. Bias toward the ones that were hard, weird, or previously wrong.
Write down what “correct” means for each one, even roughly. This is the step people skip, and it’s the one that actually forces the product thinking.
Run it before every meaningful change, and write the score down somewhere everyone can see it — not just in a terminal output that disappears.
Grow it as you find new failure modes. Every bug a user reports is a candidate for a new eval example, so the same mistake can’t sneak past you twice.
That’s genuinely enough to start. The elaborate tooling — LLM-as-judge pipelines, automated CI gates, production monitoring dashboards — is worth building once you’ve outgrown a spreadsheet, not before. Don’t let the absence of a fancy eval system be the reason you don’t start one.
Common mistakes
A few patterns I’ve seen repeatedly — including from myself:
Testing only the happy path. The inputs that were easy to think of are rarely the ones that break things.
Treating one eval score as permanent truth. Real usage drifts. A dataset that represented your users six months ago might not represent them today.
No held-out set. If you’re testing on the same examples you tuned your prompt against, you’re not measuring generalization — you’re measuring memorization.
Skipping evals for “small” changes. A two-line prompt tweak is still a change. It deserves the same scrutiny as a model swap, because it can break just as much.
What’s next
Evals give you a way to know whether a single AI system is doing its job. But most interesting AI products today aren’t single systems anymore — they’re multiple agents talking to each other, handing off tasks, coordinating. Once you’re happy that one agent is reliable, the next question is: how do you know the conversation between agents is reliable?
That’s where we’re headed next — into A2A, agent-to-agent systems, and what changes when your product isn’t one AI, but several working together.
See you next Tuesday.
Enjoyed this deep dive? Subscribe to my Substack newsletter for more practical AI insights. If you found the ideas here useful, go ahead and share this article with colleagues or on LinkedIn so more PMs can level-up their AI game. Have your AI product lifecycle story or question? Leave a comment - I would love to hear how you are using Evals in your workflow or help with any challenges you are facing. Together, let’s continue to learn how to get better AI outputs and build amazing products!
Thank you for reading. Happy innovating!