Does your test test anything? The ten-second revert check
14 November 2026 | 4 min
Green tells you the change did not break the suite. It does not tell you the suite would notice if the change were wrong.
On 4 June 1996 the first Ariane 5 broke up about forty seconds into its flight sequence. Software reused from Ariane 4 converted a 64-bit floating point value to a 16-bit integer, the value did not fit, and both inertial reference units shut down. The Inquiry Board found the units had never been tested with Ariane 5's trajectory data. A rocket and four satellites were lost.
Most of us will never ship code with that much riding on it. The pattern is still the everyday one: a test suite that is green, and a case it was never able to catch.
A test that passes either way is not evidence
A test earns its place by failing when the behaviour it guards is wrong. Plenty of tests never can. The assertion checks something the old code already did. The fixture happens to be sorted, so the sorting fix is never exercised. A mock returns the expected value whatever the code under test does. All of these pass on the day they are written, and all of them read as coverage.
This used to be an occasional annoyance. When an agent writes the change and the test together, it is the normal risk: the test was written by the same process, from the same understanding, to pass against the same code. If that understanding was wrong, both are wrong in the same way, and the suite is green.
The check
Put the code back the way it was, keep the new tests, and run them. If they go red, they depend on the change. If they stay green, they do not test it. With git, when your source lives in src/:
# 1. set the change aside, keeping the tests
git stash push src/
# 2. run the new tests: they should FAIL
npm test
# 3. bring the change back: they should pass again
git stash pop
npm test
That is it. Ten seconds for a small suite, and it answers a question that reading the test cannot.
Details that bite
- Stash the code, not the tests. Pass the source paths to
git stash push. Stashing everything takes the new tests away too, and nothing will fail. - New files are untracked. If the change added a source file, add
-uor the stash leaves it behind and your check runs against half a revert. - Run only the tests you care about. A failure somewhere else in the suite tells you nothing about the new tests. Point the runner at the files you added.
- A compile error is not a red test. If the old code does not build against the new tests, you learned the API changed, not that the assertions work. Revert the behaviour, not the signature.
- Always pop. Put the stash back even when the run surprised you, or you will lose the change you were checking.
When to reach for mutation testing
The revert check is a coarse version of mutation testing: instead of one big mutation (no change at all), tools like Stryker, PIT and mutmut make many small ones, flip a comparison, delete a line, and report which ones your suite failed to notice. They are slower and worth it on code where a quiet mistake is expensive. The revert check is the habit that costs nothing and catches the worst case: a test that could not fail at all.
Make it routine
- Do it for every test you add to cover a fix. That is where a test that cannot fail does the most damage, because it closes the bug report.
- Do it for every test an agent wrote, before you read the test.
- If a test stays green, delete it or fix it, and write down why. A test you know is empty is worse than a gap, because the next reader trusts it.
Where AstraCode fits
AstraCode runs this check itself. When a run has changed both code and tests and your project's test command passes, it puts the run's non-test files back the way they were, runs the tests again with the new tests still in place, and then restores everything. Red means the tests depend on the change. Green means they do not, and the agent is told so instead of reporting success. If anything is unclear, a timeout or a file changed since the run, it touches nothing and says the result is inconclusive. More on how it checks its work in the docs.
An agent that checks its own tests
It runs your tests, then undoes its change and makes sure they fail without it. Free to start, no card.
Also on the blog
- Make a coding agent fail loudly in CI
- npm test | tail exits 0 when your tests fail
- Coding agents are part of your supply chain now: a checklist
- Developers trust AI they can check. Build agents for that.
- When an AI agent says it is done and it is not
- How to review code an agent wrote
- Making an AI agent follow your project's conventions
- We benchmarked eleven models in our own editor. The cheapest one won.
- Fable 5.1 cut cache reads to $0.25. Here is what that saves on a real agent run.
- What vibe coding is, and when it stops working
- What Google Antigravity is, and what it costs
- What GPT-6 Astra actually costs to run a coding agent
- GPT-6 Astra's pricing cliff at 272K tokens, and why agent runs fall off it
- How we're benchmarking GPT-6 Astra for coding (and why scores won't tell you)
- How to use GPT-6 Astra in your editor, and when not to