Blog

Does your test test anything? The ten-second revert check

14 November 2026 | 4 min

Green tells you the change did not break the suite. It does not tell you the suite would notice if the change were wrong.

On 4 June 1996 the first Ariane 5 broke up about forty seconds into its flight sequence. Software reused from Ariane 4 converted a 64-bit floating point value to a 16-bit integer, the value did not fit, and both inertial reference units shut down. The Inquiry Board found the units had never been tested with Ariane 5's trajectory data. A rocket and four satellites were lost.

Most of us will never ship code with that much riding on it. The pattern is still the everyday one: a test suite that is green, and a case it was never able to catch.

A test that passes either way is not evidence

A test earns its place by failing when the behaviour it guards is wrong. Plenty of tests never can. The assertion checks something the old code already did. The fixture happens to be sorted, so the sorting fix is never exercised. A mock returns the expected value whatever the code under test does. All of these pass on the day they are written, and all of them read as coverage.

This used to be an occasional annoyance. When an agent writes the change and the test together, it is the normal risk: the test was written by the same process, from the same understanding, to pass against the same code. If that understanding was wrong, both are wrong in the same way, and the suite is green.

The check

Put the code back the way it was, keep the new tests, and run them. If they go red, they depend on the change. If they stay green, they do not test it. With git, when your source lives in src/:

# 1. set the change aside, keeping the tests
git stash push src/

# 2. run the new tests: they should FAIL
npm test

# 3. bring the change back: they should pass again
git stash pop
npm test

That is it. Ten seconds for a small suite, and it answers a question that reading the test cannot.

Details that bite

When to reach for mutation testing

The revert check is a coarse version of mutation testing: instead of one big mutation (no change at all), tools like Stryker, PIT and mutmut make many small ones, flip a comparison, delete a line, and report which ones your suite failed to notice. They are slower and worth it on code where a quiet mistake is expensive. The revert check is the habit that costs nothing and catches the worst case: a test that could not fail at all.

Make it routine

Where AstraCode fits

AstraCode runs this check itself. When a run has changed both code and tests and your project's test command passes, it puts the run's non-test files back the way they were, runs the tests again with the new tests still in place, and then restores everything. Red means the tests depend on the change. Green means they do not, and the agent is told so instead of reporting success. If anything is unclear, a timeout or a file changed since the run, it touches nothing and says the result is inconclusive. More on how it checks its work in the docs.

An agent that checks its own tests

It runs your tests, then undoes its change and makes sure they fail without it. Free to start, no card.

Start free | Download

Also on the blog