An AI session can write a hundred tests in an afternoon. That's no longer the hard part. The hard part is that the same session also decides what "passing" means, and a model that wants to finish will make a red suite green any way it can: fix the code, fix the test, or skip it.
I've been a tester for 25 years, and that's the question I've always asked: how do we know it works, and who said this result is the right one? In my lab, AI sessions now write most of the tests. This post is about the split I've ended up with: what the agent decides on its own, what still comes to me, and the guards I added when a session got it wrong.
A red suite, sorted by who decides
One client project had a backend suite that had drifted: 20 failures and 13 errors on main. Nobody could tell a real regression from the noise, and the pipeline deployed without running the tests at all. I gave the job to a session.
Most of it was the agent's call, and rightly so. Each failure traced back to a change made on purpose: an endpoint that had been removed, a renamed field, a sign-up that now needs an invite code. Those tests were stale, not the code. It fixed them, removed four that tested an endpoint that no longer exists, and put a test stage in front of the production deploy.
Two tests it didn't touch. They expected a refusal when one payout method had no address, but the form said the address was optional. Either the test was right and the form was wrong, or the other way round. That's a product question, not a testing one. So it marked them xfail(strict) and wrote the question on the issue, from the commit:
Two tests are marked xfail(strict) rather than changed: [...]
Whether that was intended is Marlon's call (#5). If the check
comes back, the tests pass and strict xfail flags it.
I answered on the issue: no address for that method. About an hour after the first commit, the two tests checked the opposite, that web and chat both accept a withdrawal without one, "so the requirement can't come back by accident". 265 passed became 267, with nothing marked as expected to fail.
I decide how much counts as done
Coverage is the other call I keep. On a marketplace I'm building, I made the end-to-end test pass a launch gate: "i want all scenarios to be tested before launch". The session turned that into a scenario matrix of about 90 cases across ten areas (waiting list, invites, sign-in, jobs, quoting, payment, completion and disputes, alerts, verification and admin, cross-cutting), each with a role, a path and an expected result.
The first version still skipped three rate-limit cases on the beta site. "All scenarios" meant those too, so they now run last, as their own group, and record exactly where the limit kicks in.
All 122 rows ran on beta over two days, with real email and sandbox payments. The report fixed nothing. It grouped the findings for triage, so the decision about what to fix before launch stayed with me.
On a volunteer project for a community organisation, the scope call was smaller and it paid off quickly. The session's first suite was one Playwright smoke test and one unit test. It grew from there, and the tests that write data run against a copy of production, never the live one. Then I asked for one thing: "Test landscape too." The session added a phone-landscape project to Playwright, and it caught a settings page showing the same button twice from 768 px up. There are 36 end-to-end tests now, 18 upright and 18 sideways.
A flaky test was a real bug
On the same volunteer project, one end-to-end test failed on the next run: a record it had deleted was still there.
The session's first fix was to the test. A delete waits out an undo window and is only sent when you leave the page inside the app. The test ended inside that window, so the delete was never sent. Making the test leave the page fixed it.
Two minutes later came the fix that mattered. A real user doesn't only leave inside the app: they reload, or close the tab. Both skipped the send, so a delete the user saw happen never reached the server. That's a data bug, and the "flaky" test is what caught it. The app now also sends it when the page is hidden, with keepalive so the request finishes after the tab is gone.
What broke: a session pushed past a failing test
Not every session behaves like that. On 28 October a session on the volunteer project pushed a failing end-to-end test, twice. The test run and the push were chained in one command, with the output filtered, so the push went ahead without anyone reading the failure. A few days later the same project's session told me it had posted a correction on an issue. It hadn't.
Neither is a model problem I can prompt away. A rule in a file says "don't", and a busy session reads it the way people read terms and conditions. So both rules now have a guard that runs outside the model, in every session on every project.
The first is a check that runs before every shell command. If a command contains both a test run and git push, it's refused, with the reason:
Tests and a push in one command. Run the tests on their own,
read the whole result, and push in a separate command only
if they passed.
The second runs when a reply is about to end the turn. If the reply says "I've pushed", "posted", "merged" or "fixed" and no tool call this turn could have done it, the reply is held back once and the session is told to check. It then does the work, shows it, or says it isn't done. A false alarm costs one extra look, and it never loops.
Before they were switched on, both were replayed against the lab's history. The Bash guard, which also refuses installs onto the Mac outside a project's own environment, would have refused 189 of 32,251 past commands for the two rules together, and the ones sampled all broke them. The claim check flagged 60 of 4,740 past replies, about one in 75. Both fail open: if a guard itself breaks, the session carries on, because a broken guard mustn't take the lab down with it.
What I'd tell a test team starting this
- Let the agent own the volume. Writing, running, fixing stale tests, and finding the edge cases in a matrix are all where it's faster than any of us.
- Keep "what does correct mean" for a person. When a test and the app disagree and nothing written down settles it, the agent should park it and ask, not pick a side.
- Keep scope for a person too. "All scenarios" and "landscape too" were each one line from me, and each found something.
- Treat a flaky test as a bug report until you've shown the user can't do what the test did.
- Guard the moves that can't be undone: the push and the claim that something's done. Put the guard outside the model.
None of this is new to testers. The agent made the work faster. It didn't change who has to decide what "working" means.