What makes a testing agent different from a coding agent?

I've been in a lot of discussions lately about independent testing agents, why you'd want one separate from your coding agent, and what makes it different. It isn't really about the model. Part of why coding agents feel so autonomous is that the verification problem was mostly solved for them. They inherit compilers, linters, test runners, CI, source control. Structured feedback that tells them, precisely, whether they're right or wrong. The model is impressive, but it's standing on decades of verification infrastructure it never had to build.

An independent testing agent inherits almost none of that. What it acts on isn't source code, it's a running app: non-deterministic, stateful, changing under it, and nothing volunteers whether it succeeded. There's also a reason you want it independent. A coding agent checking its own work is grading its own homework. You want a separate agent whose only job is deciding whether the product behaves. So its harness has to be built from scratch, and I've been working out what's actually in it.

Start with acting. A testing agent has to do what a user does. It clicks through a web app, taps through a mobile app, calls an API, so the harness has to give it real hands on the product across every surface customers use. But that part is getting commoditized. Any capable model can drive a UI or hit an endpoint now. Executing a test isn't the hard problem anymore.

The hard problem is verification. How does the agent know a login actually worked, a checkout actually completed, a page rendered the way it should? Generating the action is easy-ish; deciding whether the result was right is the whole job. Without a verification layer you haven't built an autonomous tester. You've built an autonomous clicker.

Verification starts with observing, deeply, the way a user would and a tester would. The agent needs screenshots, DOM state, network activity, logs, traces, the runtime behavior. But collecting evidence isn't enough. The harness has to compare what it saw against what was expected and decide whether the behavior was correct, and that judgment is the hard, valuable part.

It also has to do that efficiently. Easy to wave away until you run it at scale. The naive version, where you hand the whole DOM, every screenshot, and all the network traffic to the model on every step and ask "did that look right?", costs enormous tokens and time for one verdict. A good harness runs the cheap deterministic checks deterministically, saves the model for the judgments that need it, and uses what it already knows about the app so it isn't reasoning from raw pixels each run. At the scale a real suite runs, that's the difference between viable and not.

And verification compounds. Every run produces knowledge: which selectors are stable, which flows matter, which failures are expected, which recoveries work. A real harness keeps that and hands it to the next run. Without it, the agent shows up as a brand-new tester every time it opens the browser. It's most of why pointing a general-purpose agent at a browser only gets you so far.

And none of it matters unless people trust the verdict. The agent touches credentials, environments, and real data, so it has to run inside the same controls a person would, and explain itself: what it did, why it decided what it decided, what it saw and concluded. The harness isn't only constraining the agent; it's making its work auditable.

So here's where I've landed. Coding agents got a head start because software already had a verification harness. Compilers, tests, CI, and version control all tell an agent when it's right and wrong. Testing doesn't come with one, so it has to be built. And the hard part was never getting the agent to act. It's getting it to know what happened, judge whether it was right without spending a fortune to do it, and leave behind evidence the rest of us can trust.
share X LinkedIn

More posts

Deleting 400 stale i18n keys and the Bootstrap we'd carried for years

I've been deleting a lot of code lately and I cannot tell you how good it feels. I cleared out about 400 stale translation keys — the kind that accumulate over years and that nobody dares touch, because who still knows what references them? The agent knows; it checks every reference in seconds, so the sweep I'd always filed under "someday" turned into an afternoon. And Eugene and Jeff finally pulled out Bootstrap (BOOTSTRAP! the framework we've been trying to get rid of for years!) — thank you both, truly. I re-ran our UI library market share report afterward and that's when the impact really landed: Bootstrap is finally gone.

What surprised me is how the agents change the math. Migrating class components to function components with hooks, ripping out an old framework, pruning dead i18n — these were always "we should do this eventually" jobs that lost every prioritization fight. The cost of removing code and cleaning up our codebases dropped enough that now we just... do it. I keep catching myself opening diffs hoping they're mostly red.

My ideas died in markdown files, so now one skill interviews me and another ships them

My ideas used to end up as markdown notes in a folder, which is a polite way of saying they ended up nowhere. Filing a real ticket is more effort than I want to spend in the ten seconds I actually have the idea.

Now I say /new-idea and describe it in a sentence. It interviews me — what's the problem, what does done look like, what's explicitly out of scope — and it stops as soon as the ticket is unambiguous, then files it under my epic and assigns it to me.

At the same time, another session — [on a stable worktree, remember?](https://fab.mabl.com/posts/2026-05-06-dario-i-had-20-worktrees-and-no) — is running /idea-loop. It picks the oldest idea up, moves it to in progress, and we build it: plan, tests, review, PR. From there the ticket walks its own states, and a morning check only marks it shipped once a production tag actually contains the commit.

Six AI reviewers improved a PR that shouldn't have existed

A PR landed on our CLI with ~550 lines of new probe code. Its entire product effect was one debug log line, and the number it existed to measure was one the PR's own limitations section admitted it couldn't measure. My six-lens /deep-review panel read it and did good work: safer, quieter, better named. Not one of the six asked whether it should ship at all.

The reviewers weren't the problem. Every mechanism in that panel is anchored — a finding needs a file:line, a fix needs a before/after. "This file shouldn't exist" has no anchor, so there's nowhere to file it, so nobody says it. Each lens does the one thing the harness makes easy: improve the code in front of it.

So every lens now returns a scope verdict — ship-as-is, split, or cut — and that gets settled before any fix is applied. Order matters: hardening code that's about to be deleted makes the deletion look expensive. And a cut doesn't authorize anything. The panel rules; I still press delete.

Chug that data lake

For years the pundits said "data is the new oil," so we hoarded it: API logs, cluster utilization, billing exports, error traces — all in hope that one day we'd have time to actually make use of it. That day is here, and it belongs to the agents. Want to optimize application opex? You could clear your calendar, check into a remote alpine retreat, and spend a week reading the code, the 27-page cloud vendor pricing page, and the 7 pages of "limits" and gotchas — then build a massive spreadsheet crossing all of that with your usage data. But that smells like 2024 thinking. It's 2026: wire up the proper MCP servers and ACLs, offer your agent the code, docs, and data with precise marching orders, and go to bed. Do your vendor's billing docs include set theory notation and lots of linear algebra? Perfect — agents love math. In the morning, review the LLM's citations, pat yourself on the back for saving the operation a few hundred grand — all while you slept.

You're no longer limited by what you know how to do with your data lake, but simply by your own initiative and imagination. LFG.

We started shipping more PRs than we could review, so I wrote /review-with-me.

823 PRs merged across our repos this month. Same 28 days last year: 218. Same number of people.

Everyone got faster once we leaned into the AI tooling, and the review queue got the bill. I stopped being able to keep up with mine, and I don't think I was alone. Writing the code wasn't the slow part anymore. Reviewing it was.

What made review slow wasn't the hard parts, it was getting to them. Most of a big diff is a rename, a function moved between files, a regenerated snapshot. The two hunks that decide whether this ships are buried in the middle of it.

So I wrote /review-with-me. It tells me what the change is for and why, picks the hunks carrying real risk, walks me through them one at a time, and turns whatever I say into inline comments on the PR.