Blog

Make a coding agent fail loudly in CI

14 November 2026 | 4 min

In a terminal you read the agent's summary. In CI nobody does, so the exit code is the whole report.

Putting a coding agent in a pipeline looks easy. You install it, pass it a task, and let it work while you do something else. The part that goes wrong is the end of the run, because in CI nobody reads what the agent wrote. A job is green or it is red, and an agent that writes "I could not finish this" and then exits 0 has told the pipeline that everything is fine.

GitLab learned the expensive version of this lesson in 2017: its backups had been failing for a long time, nothing failed loudly, and nobody found out until the day it needed them. A quiet failure is the expensive kind, whether it is a backup script or an agent.

The exit code is the contract

Before you wire any agent into CI, find out what it exits with in each of these cases. If the documentation does not say, run it and find out, because the defaults vary a lot between tools:

Test it the way you would test anything else: give it a task that cannot be done in the step limit, run it, and look at $?. If that prints 0, the agent will turn every unfinished run in your pipeline green.

Do not let the agent be the judge

Even an honest exit code only says the agent believes it finished. That is a claim about the run, not about your code. The check that decides whether the change is good should be a separate step the agent cannot skip, edit or talk its way past:

- name: Agent makes the change
  run: astracode-agent -p "Fix the failing test in tests/test_billing.py" --permissions auto --max-steps 60
  env:
    ASTRACODE_API_KEY: ${{ secrets.ASTRACODE_API_KEY }}

- name: Our tests decide
  run: npm test

If the agent exits 0 and the tests fail, the job is red, which is what you want. The agent's opinion of its work and your suite's opinion of it are two different signals, and only one of them is allowed to merge code.

Give it a ceiling and a clock

Keep push and publish in the workflow

Let the agent edit files and run your commands. Let the workflow's own steps do the things that leave the machine: commit, push, open the pull request, comment. That way the step that publishes is a few lines of YAML you reviewed once, not a decision a model makes in the middle of a run. It also means a run that did not finish can open a draft rather than a pull request that looks ready.

If a step can publish, deploy or push, it should not be the step that reads text a stranger wrote. Issue titles and pull request descriptions are input from the internet, and an agent reads them as instructions.

Parse the result, not the prose

Most agent CLIs have a JSON output mode. Use it in CI. A sentence like "all tests pass" is written for a person; a field that says how the run ended, how many steps it took and which files changed is written for the next step in your pipeline. Fail the job on the field, and post the prose as a comment for the humans.

A checklist for any agent in CI

Where AstraCode fits

The AstraCode CLI is built to this contract. astracode-agent -p exits 0 only when the run finished, 1 when it hit its step ceiling, looped or errored, and 130 when you stopped it. --max-steps sets the ceiling, --output-format json gives the next step how the run ended and what it changed, and under --permissions auto it refuses risky commands and git push, so publishing stays in your workflow. astracode-agent review runs read-only against a branch and fails the job if it ends without reporting. The same agent runs in the editor, your terminal, and inside Zed and JetBrains IDEs.

Put an agent in CI that fails loudly

One install line, an API key from your dashboard, and an exit code your pipeline can trust. Free to start.

Start free | Download

Also on the blog