# I Built an Autonomous Testing Framework for AI Agents.

In an interview a while back, after the usual rounds, the interviewer leaned into a question I thought I had a good answer for:

> You build agents. How do you test them?

I said what everyone says. An eval dataset, LLM as judge, some manual poking before release. They let me finish, then asked the follow-up that stuck with me for weeks:

> And who writes the test cases?

Me. I write them. Which is exactly the problem. My agent only gets tested against failures I already imagined, and the entire reason agents are useful is that they take inputs nobody imagined. My test suite was a list of my own predictions, graded by a model, and I was calling that coverage.

I did not have a good answer in the room. So after the interview I read eval frameworks, red-teaming write-ups, [LLM-as-judge papers](https://huggingface.co/learn/cookbook/en/llm_judge). Everything covered a slice, and nothing closed the loop. The more I dug, the more it looked less like a test-writing problem and more like a **search problem**, and search problems are what you automate.

So I built **autoqa.**

%[https://github.com/kartikmehta8/autoqa] 

**An autonomous testing framework where agents test agents.** You describe what your agent is supposed to do, and a team of QA agents writes adversarial scenarios, runs them, grades the transcripts, clusters the failures, and remembers every bug so the next run starts as a regression suite. The core is completely agent-framework-agnostic. I built mine with Mastra, but the **harness will test anything that can receive messages and return text**.

This post is the write-up of why it exists, how it works, and what you can take from it.

![](https://cdn.hashnode.com/uploads/covers/63b9794295025da0eecd94f7/5ce7308c-b1dd-4290-822c-d5b720f5c895.png align="center")

## Agents break in ways you did not think of

If you have shipped anything on top of an LLM, you already know the uncomfortable part: your test suite tests the code around the model, not the behavior of the system. Agents are different for anyone trying to test them:

**The input space is unbounded.** A REST endpoint has a schema. An agent has a text box. Users paste emails into it, type ids with unicode lookalikes, and occasionally try to jailbreak it for fun. Every hand-written suite is a tiny, biased sample of production.

**Failures are behavioral, not structural.** The agent does not crash. It returns a fluent, confident answer that happens to be wrong. `assert response.status == 200` passes every time. The bug is that the 200 contains a refund confirmation that violates your policy.

**State lives across turns.** The expensive bugs only appear in multi-turn conversations: the agent agrees to something in turn one and forgets it in turn three. Single-shot eval datasets never touch this.

**Tool use adds a second failure surface.** Wrong tool, right tool with wrong arguments, no tool and an answer from priors instead, or a destructive tool without the confirmation your policy requires. The reply can look perfect while the tool trace is a disaster.

**Every prompt edit is a deploy.** You tweak one sentence to fix one complaint, and you have no idea what else changed. Without a regression suite that grows automatically, you are re-testing by vibes.

The standard answers each cover a slice. Eval datasets are static. LLM-as-judge grades what you give it but does not generate the hard inputs. Manual red-teaming finds great bugs exactly once, then the knowledge evaporates.

What I wanted was the loop a good QA engineer runs in their head, but automated: *probe, observe, form a hypothesis about where it is weak, probe there harder, write down everything that broke, and re-check all of it after every change.*

## The loop

That mental loop became this:

![](https://cdn.hashnode.com/uploads/covers/63b9794295025da0eecd94f7/c6255307-b967-4fe8-9a0d-2db874f3e3fd.png align="center")

A run works in waves:

**Wave 0 replays what is already known.** Hand-written seed scenarios plus every failure past runs promoted into the corpus. No model generates anything here. Known bugs are the cheapest tests you own, so they run first, every time.

**Later waves explore.** An explorer agent gets your description of the agent, a coverage report, and the clusters found so far. It writes scenarios targeting untested capabilities and probing around confirmed breaks, because where there is one bug there is usually a family.

**Everything that fails gets remembered.** Failing scenarios land in a per-agent corpus on disk, so tomorrow's run opens with today's bugs. This changes the economics: the suite gets stronger every run without anyone writing a test.

**The run stops at the first limit it hits.** A trial cap, a cost cap in dollars, a time cap, a wave cap, or N consecutive waves that find nothing new. That last one matters most. An autonomous loop without a "nothing new" exit will happily spend your entire API budget confirming that your agent can say hello.

That is the machine. The next decision was what the machine should be allowed to know about the agent sitting inside it, and the answer I landed on was: almost nothing.

## The one interface

I wanted the architecture to survive me changing my mind about agent frameworks, because everyone does. So the entire core knows exactly one type:

```ts
interface AgentAdapter {
  readonly id: string;

  /**
   * One conversation turn. Receives the full message history and must return the agent's reply, plus any tool calls it made along the way.
   */
  invoke(req: AgentRequest, ctx: InvokeContext): Promise<AgentResponse>;

  /**
   * Optional. Called before each trial to clear per-conversation state
   */
  reset?(sessionId: string): Promise<void>;
}
```

`AgentRequest` is `{ messages, sessionId, seed? }`. `AgentResponse` is `{ text, toolCalls?, usage?, raw? }`. That is the whole contract. The runner, judges, clustering, explorer and storage import no framework, no LLM SDK, not even a transport. Framework knowledge lives at the edges, in adapters:

![](https://cdn.hashnode.com/uploads/covers/63b9794295025da0eecd94f7/d56af719-9deb-4813-ae01-f20c740fb82b.png align="center")

Three ship: `mastraAdapter` for a `@mastra/core` Agent, `httpAdapter` for anything behind an endpoint (LangGraph, CrewAI, your staging deployment), and `functionAdapter` for a plain in-process function.

**Here is the detail I am most stubborn about:** one of the example agents, `rules-bot`, is a plain TypeScript function. No LLM, no framework, no API key. It proves the core does not care what produced the response, and it lets you exercise the whole pipeline without spending a token.

The same interface points inward too: the explorer, the LLM judge, and the triage agent all talk to their models through `AgentAdapter`. The brain of the harness is exactly as pluggable as the thing it tests.

With the contract that small, registering an agent did not deserve any ceremony either.

## Adding an agent is one file

Agents live in an `agents/` folder. Drop a file in and it shows up in the CLI and the UI with its own run history and regression corpus. No registration, no wiring.

```ts
// agents/my-agent.agent.ts
import { defineAgent } from '../src/core/index.js';
import { httpAdapter } from '../src/adapters/http.js';

export default defineAgent({
  id: 'my-agent',
  description: 'One line for the agent list.',
  spec: `What this agent is supposed to do, in prose.
The explorer works from this alone, so a vague spec produces vague tests.`,
  adapter: httpAdapter({ url: 'http://localhost:8080/chat' }),
  scenarios: [
    {
      id: 'smoke',
      title: 'Answers a basic question',
      tags: ['smoke'],
      turns: ['hello'],
      expect: [{ kind: 'noError' }, { kind: 'contains', value: 'hi' }],
      severity: 'medium',
      origin: 'seed',
    },
  ],
});
```

The `spec` field is the highest-leverage thing you will write. With "a support agent for an online store" the generated scenarios were generic. With the actual refund policy written in, the explorer immediately attacked the confirmation flow, cumulative partial refunds, and confirmation reuse. Specific spec, specific attacks.

Once a scenario runs, something has to decide whether the agent actually passed, and this is where most designs get expensive or flaky, so I split it in two.

## How trials are judged

Every completed conversation gets judged twice, and the split is deliberate.

**Deterministic assertions run on every trial.** Eight kinds: `contains`, `notContains`, `matches`, `toolCalled`, `toolNotCalled`, `jsonPath`, `maxLatencyMs`, `noError`. Free, instant, never flake.

**The LLM judge runs only on scenarios that declare a** `rubric` (a pass/fail rule written in plain English instead of code, graded by an LLM reading the transcript)**.** Some criteria need reading comprehension: "must not confirm a completed refund for an order that has not shipped" is not a string match. The rubric judge grades the full transcript, tool trace included, and returns pass, score, and a one-line reason.

Two rules keep the judging honest. A judge that throws produces a failing verdict, never a silent pass: if the judge model is down, your tests go red, not green. And the judges are memoryless on purpose, because a grader that remembers past verdicts drifts. Identical transcripts must grade identically.

![](https://cdn.hashnode.com/uploads/covers/63b9794295025da0eecd94f7/cd1f92e3-1e2c-4bae-be71-39de56b1f8b7.png align="center")

Verdicts solve one trial. The next problem is volume, because an explorer that works will hand you the same bug ten different ways.

## Clustering is arithmetic, not a model

When ten trials fail, you want one report that says "these ten are the same bug." My first instinct was to have an LLM group them. I am glad I did not.

Failures are clustered by a **signature**: a hash of the failure's shape. Tags, which assertion kinds broke, and the reason normalized to strip ids, numbers and quotes. Same bug, same hash, across runs and machines. The LLM only enters afterward, to read three examples from a new cluster and write one line naming the likely root cause. Known clusters are never re-summarized, because that buys an identical sentence for real money.

![](https://cdn.hashnode.com/uploads/covers/63b9794295025da0eecd94f7/b146e5e8-c905-46ba-bdd3-1c5f0af720b4.png align="center")

All of that is the engine. Day to day, you drive it from two places.

## Driving it

`npm run serve` starts a local UI, no build step. The **Playground** lets you chat with any agent by hand, tool calls inline with arguments, latency and cost. The button that matters is **Save as scenario**: the conversation you just used to break the agent becomes a regression test, remembered exactly like a bug the explorer found. **Runs** holds per-agent history with a self-contained HTML report. **Live** streams runs as they happen, each trial ticking green or red.

![](https://cdn.hashnode.com/uploads/covers/63b9794295025da0eecd94f7/a1a9718f-244a-406f-b9ce-f101d0b58a8d.png align="center")

CI gets the same thing headless:

```shell
npx autoqa run --agent support-agent --junit junit.xml --fail-on high
```

Exit code 1 if any cluster reaches that severity. A run is fully reproducible from `(seed, corpus, agent)`. Per run you tune waves, trial caps, cost cap, concurrency and seed; the judges, explorer and store stay fixed in project config, because run history is only comparable when the measuring instruments stay constant.

One knob mattered more than I expected: **the model**. The loop makes one call per trial plus one per graded rubric, so model choice sets the cost of the whole harness. On Claude Haiku a four-scenario run costs $0.002; on an Opus-class model, about 60 times more. Probing and grading are mechanical work. Save the expensive model for the agent you ship.

## Best practices for creating and testing agents

Everything above condenses into this table. Each row exists because skipping it burned me:

| Practice | Why it matters |
| --- | --- |
| Write the spec like a policy document, not a tagline | The explorer attacks exactly what you name. Vague spec, generic tests. |
| Seed every hard invariant by hand | Refund caps, refusals, injection resistance: core policy is too important to leave to generation alone. |
| Prefer assertions, save rubrics for judgement calls | Deterministic checks are free and never flake. Every rubric is an LLM call that can. |
| Judge the tool trace, not just the text | A fluent reply next to a wrong tool call is the most common agent failure there is. |
| Test multi-turn on purpose | Confirmation flows and memory bugs never show up in single-shot evals. |
| Keep one auto-growing regression corpus per agent | A bug found once is re-checked forever, and one agent's bugs prove nothing about another. |
| Make every run reproducible | Seed plus corpus plus agent should replay a failure exactly, or you cannot debug it. |
| Fail closed everywhere | A broken judge is a red test. A malformed generated scenario is dropped, never a fake bug. |
| Treat model output as untrusted input | My explorer generated Python-dialect regexes that JavaScript rejects. Validate generated tests before they run. |
| Put budgets on autonomy | Cost caps, trial caps, and a "nothing new found" exit. A loop without a stop condition is an expensive while-true. |

## The answer I wish I had given

The honest pitch for autonomous agent testing is not that it replaces your judgment. It is that it scales the boring half of it.

In its first exploration run, the framework sent my support agent an order id written with an en dash and letter O lookalikes. The agent silently "corrected" it to a real order and headed toward a refund on it. I knew lookalike identifiers were a risk. I was never going to sit down and write that test. The explorer wrote it in wave one, for a few cents, and now it runs before every deploy forever.

So, who writes the test cases?

**An agent does, briefed with the spec, the coverage gaps, and every bug found so far. I just read the failures.**