Make a coding agent fail loudly in CI
14 November 2026 | 4 min
In a terminal you read the agent's summary. In CI nobody does, so the exit code is the whole report.
Putting a coding agent in a pipeline looks easy. You install it, pass it a task, and let it work while you do something else. The part that goes wrong is the end of the run, because in CI nobody reads what the agent wrote. A job is green or it is red, and an agent that writes "I could not finish this" and then exits 0 has told the pipeline that everything is fine.
GitLab learned the expensive version of this lesson in 2017: its backups had been failing for a long time, nothing failed loudly, and nobody found out until the day it needed them. A quiet failure is the expensive kind, whether it is a backup script or an agent.
The exit code is the contract
Before you wire any agent into CI, find out what it exits with in each of these cases. If the documentation does not say, run it and find out, because the defaults vary a lot between tools:
- It finished the task. This should be the only case that exits 0.
- It ran out of steps. A run that hit its ceiling did not finish, however reasonable its last message sounds.
- It went round in circles. Repeating the same call until something stops it is a failure, not a long success.
- A request to the model failed. A network error halfway through a run leaves the tree in an unknown state.
- You cancelled it. Conventionally 130, so a script can tell a person stopping it from the run failing.
Test it the way you would test anything else: give it a task that cannot be done in the step limit, run it, and look at $?. If that prints 0, the agent will turn every unfinished run in your pipeline green.
Do not let the agent be the judge
Even an honest exit code only says the agent believes it finished. That is a claim about the run, not about your code. The check that decides whether the change is good should be a separate step the agent cannot skip, edit or talk its way past:
- name: Agent makes the change
run: astracode-agent -p "Fix the failing test in tests/test_billing.py" --permissions auto --max-steps 60
env:
ASTRACODE_API_KEY: ${{ secrets.ASTRACODE_API_KEY }}
- name: Our tests decide
run: npm test
If the agent exits 0 and the tests fail, the job is red, which is what you want. The agent's opinion of its work and your suite's opinion of it are two different signals, and only one of them is allowed to merge code.
Give it a ceiling and a clock
- A step limit. Small tasks need small budgets. A run that needs four times the steps you expected is usually lost, and a ceiling turns that into an honest failure instead of a long bill.
- A job timeout.
timeout-minuteson the job, so a run that hangs on a prompt nobody will answer cannot hold a runner all day. - Read-only by default. Give the agent write access only in the jobs that need it. A review job has no reason to edit files.
Keep push and publish in the workflow
Let the agent edit files and run your commands. Let the workflow's own steps do the things that leave the machine: commit, push, open the pull request, comment. That way the step that publishes is a few lines of YAML you reviewed once, not a decision a model makes in the middle of a run. It also means a run that did not finish can open a draft rather than a pull request that looks ready.
If a step can publish, deploy or push, it should not be the step that reads text a stranger wrote. Issue titles and pull request descriptions are input from the internet, and an agent reads them as instructions.
Parse the result, not the prose
Most agent CLIs have a JSON output mode. Use it in CI. A sentence like "all tests pass" is written for a person; a field that says how the run ended, how many steps it took and which files changed is written for the next step in your pipeline. Fail the job on the field, and post the prose as a comment for the humans.
A checklist for any agent in CI
- Does it exit non-zero when it ran out of steps, looped or errored?
- Does a separate step run your real tests after it?
- Is there a step limit and a job timeout?
- Does the agent lack the token that can push or publish?
- Is the job that reads untrusted text separate from the one that holds secrets?
Where AstraCode fits
The AstraCode CLI is built to this contract. astracode-agent -p exits 0 only when the run finished, 1 when it hit its step ceiling, looped or errored, and 130 when you stopped it. --max-steps sets the ceiling, --output-format json gives the next step how the run ended and what it changed, and under --permissions auto it refuses risky commands and git push, so publishing stays in your workflow. astracode-agent review runs read-only against a branch and fails the job if it ends without reporting. The same agent runs in the editor, your terminal, and inside Zed and JetBrains IDEs.
Put an agent in CI that fails loudly
One install line, an API key from your dashboard, and an exit code your pipeline can trust. Free to start.
Also on the blog
- Does your test test anything? The ten-second revert check
- npm test | tail exits 0 when your tests fail
- Coding agents are part of your supply chain now: a checklist
- Developers trust AI they can check. Build agents for that.
- When an AI agent says it is done and it is not
- How to review code an agent wrote
- Making an AI agent follow your project's conventions
- We benchmarked eleven models in our own editor. The cheapest one won.
- Fable 5.1 cut cache reads to $0.25. Here is what that saves on a real agent run.
- What vibe coding is, and when it stops working
- What Google Antigravity is, and what it costs
- What GPT-6 Astra actually costs to run a coding agent
- GPT-6 Astra's pricing cliff at 272K tokens, and why agent runs fall off it
- How we're benchmarking GPT-6 Astra for coding (and why scores won't tell you)
- How to use GPT-6 Astra in your editor, and when not to