About Me

Featured

I am passionate about using Cloud Technology.

So any questions/ feedback – please Get In Touch

Copyright ©  in accordance with the Copyright, Designs and Patents Act 1988. All rights reserved. You are free to use any of the content here for personal use but need permission to use it anywhere or by any means (electronic, mechanical, photocopying, recording or otherwise).

Why Ai Agents need an eval framework

AgentDBA (the database agent I have been building) has more than 1000 automated tests, and until this week none of them measured a single decision the model makes. Every test that runs the agent loop scripts the model’s replies. The suite proves the code is correct when the model behaves. It cannot tell you whether the model behaves.

That gap matters more for an agent than for a chatbot. AgentDBA decides which checks to run on a production Server, what severity to assign, and whether to page someone at 3 a.m. Those are the decisions a customer is trusting. So I built an eval framework to measure them.

Read more: Why Ai Agents need an eval framework

Why tests stop working when a model is in the loop

A unit test rests on one assumption: the same input gives the same output. Run it once, and if it passes you can trust it until the code changes. An agent breaks that assumption three ways:

  • The model is not deterministic. The same question, same data and same prompt can produce a different tool order, a different severity or a different sentence on the next run. Thinking models such as Claude Sonnet 5.5 and Opus 5.5 take no temperature setting, so you cannot pin it down.
  • The model changes under you. A provider update, a new deployment or a switch of model changes behaviour without a line of your code changing.
  • The answer is a judgement, not a value. “Is this the right severity?” has a defensible range, not one correct number.

So the question changes. A test asks “does this work?” An eval asks “how often does this work, and is that rate getting worse?” That is a measurement, and measurement needs three things: a fixed instrument, repeated observations, and a baseline to compare against.

One more principle comes from error analysis, the most useful habit in eval practice: grade the agent on the failures you have actually seen, not a generic list of “AI quality” scores. AgentDBA’s release history records about 20 behaviour defects I have shipped and fixed, such as the agent answering from a stale stored episode without running a single check. Each one becomes a test case.

Code grades, not an LLM

Each eval runs the real agent loop and real tools with a real model, against frozen SQL Server data. The data is stored per stored procedure, so the model can take any path through the tools and still get realistic results. Then plain Python checks what the model decided:

  • Did it call the tool that matches the question before concluding?
  • Is the severity and conclusion category in the accepted range?
  • Would this result page on-call?
  • Did it repeat a stale stored issue instead of checking live data?

These are fixed rules. Given the same trace they give the same verdict every time. That is the point. If the model is the variable you want to measure, the instrument must not vary too. Put an LLM in the judge’s seat and you have two random processes in the measurement, and a change in the score could come from either.

Code graders have three other properties I wanted. They are free, so they can run on every trial. They are auditable: a failure says first_tool: expected error_log, got backup_failures, not “the judge felt this was weaker”. And they cannot be talked round, which matters when the model’s output is fluent and confident.

Why --k 3 matters

Because the grader is fixed, any difference between trials of the same case comes from the model’s decision. --k 3 runs every case three times, so a single run can no longer pass by luck. One trial that gets it right tells you the model can. Three tell you something about whether it does.

Three passes is still a small sample. It shows a case usually works, not that it never fails; a claim like “fails less than 2% of the time” takes around 150 clean trials. So --k 3 is a smoke test while cases are being written. Release runs use k = 5, and a new model gets k = 10 before it becomes the baseline.

First run: Sonnet 5.5 against Opus 5.5

Both models passed all 18 trials: six cases, three runs each. Five run on a healthy server; one adds a stale critical issue in memory.

CaseSonnet 5.5Opus 5.5
Full health checkLOW ×3; tool order varied (2 orders in 3 runs)LOW ×3; same tool order all 3 runs
“Check backups”INFO ×3INFO ×2, LOW ×1
“Check database status”INFO ×3LOW ×2, INFO ×1
Jobs, log health, stale-issue case3/3 each3/3 each
Health check, per trial40,490 tokens in, 597 out, 10.5 s40,719 tokens in, 664 out, 9.3 s
Whole run, 18 trials301,015 in, 7,908 out301,930 in, 8,883 out

On this data the two models read nearly the same input, so cost differences come from price per token and the roughly 12% more output Opus writes.

Order randomness is normal

In the health check, Sonnet ran the three Phase B checks in two different orders across three runs. Opus used the same order every time. Neither is wrong. A health check runs every module once and concludes only after all of them have reported, so the order cannot change the answer. That is why the graders check which tools ran, not the sequence, except where order is the point: on a targeted question, the matching tool must come first.

This is the practical reason to grade outcomes rather than replay a script. A test that expected one exact tool sequence would fail Sonnet a third of the time for a difference that does not matter. An eval that is too strict trains you to ignore it.

The severity split is more interesting. On identical clean data, Sonnet called backups and database status INFO every time. Opus called them LOW in three of six runs. Both are inside the accepted range, so both pass. But it tells me something about my product, not the models: the prompt never says whether “checked, nothing wrong” is INFO or LOW. Two strong models read the gap differently. That is a specification gap, and the fix is a rule in the prompt, not a wider pass mark.

“Opus is more consistent” is tempting, but three runs per case cannot support it. The comparison shows me where to look and where to add runs, not which model wins.

Next: a model as judge, with a narrow job

The code graders tell me whether each model made the right decisions. They cannot tell me whose summary a DBA would rather read at 3 a.m. That is the next phase: a model judge that scores summaries across models on three questions every finding owes the on-call DBA. What is broken, why it broke (or an honest “cannot be determined”), and what to do next.

The judge gets strict limits. It scores quality only and never gates safety. It is calibrated first: I label around 50 outputs by hand, and the judge must agree with me on at least 85%, with its true-positive and true-negative rates reported separately, because a judge that passes everything scores well on a set of mostly good answers. It comes from a different model family from the one being judged, so a model is not grading its own style – but this is for a different day.

A Front Door for the DBA Team – AgentDBA UI

AgentDBA began as a command line. That suits one DBA on one server, but it’s a poor way to give a team a read-only advisor. The obvious fix is a web UI, which is also how you create your first real attack surface. So v1.17.0 was built around one question: How do you add a browser without handing out more power?

Continue reading →

The flight recorder: Replaying what an AI agent did on my SQL Server

“What exactly did it do on my server?” is the first question a DBA should ask about any AI agent. AgentDBA answers it from its own records: every decision in a run is written down with who made it, the reason given, what came back and how long it took.

I call it the flight recorder. This post replays one complete health check from those records, using two queries against the agent’s own database.

Read more: The flight recorder: Replaying what an AI agent did on my SQL Server

One health check, 11.2 seconds

The first result set is one row per run. This run was a full health check on local server (lab).

What the row recordsThis run
Triggerhealth_check
Modelgpt-6.1-sol
Decisions6, of which the model made 4
Routedb_status (HIGH) > error_log (LOW) > backup_failures (CRITICAL) > failed_jobs (HIGH) > log_file_health (MEDIUM)
OutcomeConcluded at CRITICAL, closed by the model

Two columns matter before any detail. One says how many of the six decisions the model made. The other says who closed the run. An agent’s record should always separate what the AI chose from what code chose for it.

Six steps, replayed

The second result set is the recorder itself: one row per decision, in order.

StepAt (ms)Decided byActionReason given
122codedb_statusHealth check Phase A gate.
267codeerror_logHealth check Phase A gate.
32,698LLMbackup_failuresTest whether the previously reported backup RPO breaches remain present
45,253LLMfailed_jobsTest whether previously disabled jobs remain disabled…
58,220LLMlog_file_healthCheck current log usage, truncation blockers, and VL…
611,227LLMCONCLUDEConcluding: AREA51-AKS remains in breach of back…

Steps 1 and 2 were not the model’s call. Every health check opens with database status and the error log, and code makes both calls. Had either come back CRITICAL, the agent would have escalated straight away with no model involved. They came back HIGH and LOW, so the run carried on.

From step 3 the model chose, inside a fixed boundary. It had three checks left, each allowed once, and then one verdict to write. It controls the order and the wording, and it has to state a reason with every action.

Where the 11.2 seconds went

The five SQL checks took 413 milliseconds between them. Almost all of the other 10.8 seconds was the model deciding what to do next: four decisions at roughly two and a half to three seconds each. That split matters to a DBA.

From the verdict back to the evidence

Each tool step carries an audit_log_id. It points to one stored row holding two things: the raw resultset SQL Server returned, and the compact summary of it that the model was shown.

Here is the raw resultset stored behind step 3, audit ID 22016.

The model never sees the raw resultset. Each tool reduces its rows to a short, structured summary in ordinary code, and only that summary reaches the model. The raw rows are kept in the audit log, so a reviewer can check the agent’s conclusion against the data the tool actually read.

So a run can be followed end to end: the conclusion, every step that led to it, who decided each step, the reason the model gave, and the data behind it.

Trust or audit

I did not want to ask a DBA to take an AI agent’s word for anything on a production server. The recorder is my answer: one row per decision, kept in your own database, readable with a SELECT.

AgentDBA has a free evaluation at agentdba.ai if interested.

Evidence-bound: what AgentDBA will and won’t link

When the evidence names the database, it says so. When it doesn’t, it stops.

Three databases have no recorded full backup. One backup job failed. Many tools would link the two and call it solved. This one linked the job to one database and said the cause for the other two is not determined.

Continue reading →

Resolving SQL Server Transaction Log Issues Efficiently

Transaction log issues are one of the quietest ways a healthy database turns into an incident. A log file fills up, backups fall behind, and by the time anyone notices it is a 2am page rather than a five-minute fix. AgentDBA runs against SQL Server and explains, in plain language, what is wrong and what to do about it. Below is a real example of it catching one.

Read more: Resolving SQL Server Transaction Log Issues Efficiently

The check

A routine full server check comes back CRITICAL. Two databases have blown through their backup RPO entirely (recovery point objective — how much data would be lost if a restore were needed right now), and three have separate high-severity transaction log problems. Instead of a wall of undifferentiated alerts, AgentDBA groups the findings, ranks them by severity, and attaches a confidence score (51/95, “has supporting evidence, but should be reviewed”) so the DBA knows exactly how much to trust the read before acting on it.

Following the evidence

A quick follow-up question, “check tlog health,” drills straight into the detail: tlog_txn_test has an open transaction holding the log at 71.6% used, model is stuck waiting on a log backup, and tlog_vlf_test has 1,050 virtual log files — a telltale sign of small autogrow increments rather than a properly sized log. Each finding comes with a plain-English resolution, not just a symptom.

None of this is a guess. It is grounded in the same data a DBA would pull by hand — here, the log_reuse_wait_desc column straight from sys.databases, confirming the active transaction and the pending log backup wait.

Underlying evidence — verified directly against SQL Server, not inferred.

Connecting the dots

It also does this unprompted. Asking a separate question, “check database status,” pulls back the same underlying finding and says so explicitly: “you asked about db_status; this finding also draws on log_file_health, corroborated with a direct data link during the same investigation.”
Rather than treating each question as a fresh, isolated lookup, it recognises the two are related and merges the evidence — so the DBA gets one coherent picture instead of having to cross reference two separate answers themselves.

Why this matters for DBAs

  • Time back: what is normally a 15–20 minute manual triage — checking sys.databases, log_reuse_wait_desc, VLF counts, and backup history separately — becomes a single question with a direct answer.
  • Trust, not blind faith: every finding ships with a confidence score and the underlying evidence, so the DBA can see why the agent believes what it believes rather than taking it on faith.
  • Connected context: findings link together across questions instead of resetting with every check, as shown above.

www.agentdba.ai