Why Ai Agents need an eval framework

AgentDBA (the database agent I have been building) has more than 1000 automated tests, and until this week none of them measured a single decision the model makes. Every test that runs the agent loop scripts the model’s replies. The suite proves the code is correct when the model behaves. It cannot tell you whether the model behaves.

That gap matters more for an agent than for a chatbot. AgentDBA decides which checks to run on a production Server, what severity to assign, and whether to page someone at 3 a.m. Those are the decisions a customer is trusting. So I built an eval framework to measure them.

Read more: Why Ai Agents need an eval framework

Why tests stop working when a model is in the loop

A unit test rests on one assumption: the same input gives the same output. Run it once, and if it passes you can trust it until the code changes. An agent breaks that assumption three ways:

  • The model is not deterministic. The same question, same data and same prompt can produce a different tool order, a different severity or a different sentence on the next run. Thinking models such as Claude Sonnet 5.5 and Opus 5.5 take no temperature setting, so you cannot pin it down.
  • The model changes under you. A provider update, a new deployment or a switch of model changes behaviour without a line of your code changing.
  • The answer is a judgement, not a value. “Is this the right severity?” has a defensible range, not one correct number.

So the question changes. A test asks “does this work?” An eval asks “how often does this work, and is that rate getting worse?” That is a measurement, and measurement needs three things: a fixed instrument, repeated observations, and a baseline to compare against.

One more principle comes from error analysis, the most useful habit in eval practice: grade the agent on the failures you have actually seen, not a generic list of “AI quality” scores. AgentDBA’s release history records about 20 behaviour defects I have shipped and fixed, such as the agent answering from a stale stored episode without running a single check. Each one becomes a test case.

Code grades, not an LLM

Each eval runs the real agent loop and real tools with a real model, against frozen SQL Server data. The data is stored per stored procedure, so the model can take any path through the tools and still get realistic results. Then plain Python checks what the model decided:

  • Did it call the tool that matches the question before concluding?
  • Is the severity and conclusion category in the accepted range?
  • Would this result page on-call?
  • Did it repeat a stale stored issue instead of checking live data?

These are fixed rules. Given the same trace they give the same verdict every time. That is the point. If the model is the variable you want to measure, the instrument must not vary too. Put an LLM in the judge’s seat and you have two random processes in the measurement, and a change in the score could come from either.

Code graders have three other properties I wanted. They are free, so they can run on every trial. They are auditable: a failure says first_tool: expected error_log, got backup_failures, not “the judge felt this was weaker”. And they cannot be talked round, which matters when the model’s output is fluent and confident.

Why --k 3 matters

Because the grader is fixed, any difference between trials of the same case comes from the model’s decision. --k 3 runs every case three times, so a single run can no longer pass by luck. One trial that gets it right tells you the model can. Three tell you something about whether it does.

Three passes is still a small sample. It shows a case usually works, not that it never fails; a claim like “fails less than 2% of the time” takes around 150 clean trials. So --k 3 is a smoke test while cases are being written. Release runs use k = 5, and a new model gets k = 10 before it becomes the baseline.

First run: Sonnet 5.5 against Opus 5.5

Both models passed all 18 trials: six cases, three runs each. Five run on a healthy server; one adds a stale critical issue in memory.

CaseSonnet 5.5Opus 5.5
Full health checkLOW ×3; tool order varied (2 orders in 3 runs)LOW ×3; same tool order all 3 runs
“Check backups”INFO ×3INFO ×2, LOW ×1
“Check database status”INFO ×3LOW ×2, INFO ×1
Jobs, log health, stale-issue case3/3 each3/3 each
Health check, per trial40,490 tokens in, 597 out, 10.5 s40,719 tokens in, 664 out, 9.3 s
Whole run, 18 trials301,015 in, 7,908 out301,930 in, 8,883 out

On this data the two models read nearly the same input, so cost differences come from price per token and the roughly 12% more output Opus writes.

Order randomness is normal

In the health check, Sonnet ran the three Phase B checks in two different orders across three runs. Opus used the same order every time. Neither is wrong. A health check runs every module once and concludes only after all of them have reported, so the order cannot change the answer. That is why the graders check which tools ran, not the sequence, except where order is the point: on a targeted question, the matching tool must come first.

This is the practical reason to grade outcomes rather than replay a script. A test that expected one exact tool sequence would fail Sonnet a third of the time for a difference that does not matter. An eval that is too strict trains you to ignore it.

The severity split is more interesting. On identical clean data, Sonnet called backups and database status INFO every time. Opus called them LOW in three of six runs. Both are inside the accepted range, so both pass. But it tells me something about my product, not the models: the prompt never says whether “checked, nothing wrong” is INFO or LOW. Two strong models read the gap differently. That is a specification gap, and the fix is a rule in the prompt, not a wider pass mark.

“Opus is more consistent” is tempting, but three runs per case cannot support it. The comparison shows me where to look and where to add runs, not which model wins.

Next: a model as judge, with a narrow job

The code graders tell me whether each model made the right decisions. They cannot tell me whose summary a DBA would rather read at 3 a.m. That is the next phase: a model judge that scores summaries across models on three questions every finding owes the on-call DBA. What is broken, why it broke (or an honest “cannot be determined”), and what to do next.

The judge gets strict limits. It scores quality only and never gates safety. It is calibrated first: I label around 50 outputs by hand, and the judge must agree with me on at least 85%, with its true-positive and true-negative rates reported separately, because a judge that passes everything scores well on a set of mostly good answers. It comes from a different model family from the one being judged, so a model is not grading its own style – but this is for a different day.

Leave a Reply