How to cut your change-failure rate without losing speed

This is the direct follow-up to What 20 months of AI adoption measured. That post showed our throughput doubled while our change-failure rate doubled with it, and I promised to come back with what actually moves the failure rate down. The honest answer is not a process or a retro. It is a pile of small, boring, automatic checks, and the discipline to keep adding them.

TL;DR:

• As agents took over the keyboard, our change-failure rate doubled period-over-period (~8% → ~17%), and in the worst single month it spiked to 37.5%, over 4× the old baseline.
• The response: about 35 deterministic guardrails (banned commands, agent hooks, pre-commit checks) plus ~60 written rules, most added in one quarter.
• The principle: a written rule is advice the agent can ignore; a tool just runs. A tool executes; an agent gambles.
• The result: as the guardrails went in, the rate came back down even as deploys more than doubled. The direction is right; I can’t call it proof (correlation, and I left before a clean quarter).

The problem

When a person writes a retry loop, they tend to add a ceiling by reflex, because they have been burned before. An agent optimises for “make it pass,” and will happily write a loop with no upper bound, a migration with no rollback, a test cleanup that deletes by prefix. The code reads fine in review. It fails in production.

So the failures are not random. They cluster around the safety a human adds without thinking and an agent leaves out. Once you see that, the fix stops being “review harder” and becomes “make the missing safety impossible to skip.”

What the data shows

The rate spiked to 37.5% in March, then fell back to ~17% by May as the guardrails went up and deploys more than doubled.

Month	Deploys	Failed	Rate	Guardrail files added
Dec 25 – Feb 26	10	0	0%	13 (Feb)
Mar 2026	8	3	37.5%	7
Apr 2026	5	1	20%	25
May 2026	23	4	17.4%	2

Two things stand out. The response was real and fast: the guardrail wall went up in the Feb–April scramble, the month after the rate peaked. And the rate then fell from its March spike: 37.5% in March, down to ~17% by May, while deploy volume more than doubled to 23. The rate went down as the volume went up, which is the direction you want.

I can’t prove the guardrails caused it: the team was maturing, the backlog stabilising, the rate is a lower bound (only the rollbacks App Runner caught), and I left before a clean post-rollout quarter. Correlation, not a verdict. What I can show you exactly is the playbook.

---

A tool executes; an agent gambles.

---

The checklist

Every guardrail starts the same way: a reviewer, human or AI, catches a mistake. From there it can become a written rule the agent is told to follow, or a tool that just runs on every change. The rule is advice; the tool is a guarantee. The whole game is moving each lesson from advice to automation.

Push every catch toward the bottom: a tool that runs itself beats a rule the agent might skip.

1. Give the AI reviewer a structured rulebook

The expensive reviewer becomes useful when you tell it exactly what to flag, in priority order. Our Copilot review file sorts every finding into Critical (blocks the PR), Important, and Suggestion, and names the failure modes that actually take this system down, not generic style.

Generic Copilot: “Looks like this might have an issue.” (or it flags a typo)

Tuned, Critical, blocks the PR: “Critical : event.getFixtureId() can be null for a malformed message; this NPE will crash the Kafka stream.”

A code-reviewer subagent runs the same list before the PR even opens. (This is the previous post in practice: an AI reviewer is only as good as the failure modes you hand it.)

2. Turn the painful lessons into checks, not notes

Our worst self-inflicted incident: a test’s cleanup step deleted roughly 30,000 folders from our S3 bucket with one wrong prefix match. You can respond by writing a rule, “never delete by prefix in a test,” and hope the agent obeys it next time. We did better: we made the mistake impossible. The tests now point at a throwaway mock S3 (LocalStack) that is created and destroyed per run, so a test physically cannot touch real data. The note is advice; the mock is a guarantee.

3. Make the pre-commit hook the checklist nobody can skip

Sixteen checks run on every commit, each one a hard block. One example: it refuses a database migration that ships without a rollback. There is no “I’ll add it later,” because the commit simply does not happen until it’s there.

4. Gate the merge on green checks

Every PR has to pass the test suite, the linters and static analysis (formatting, plus security scans like checkov/tflint on the infra), and a code-coverage floor. And because a flaky test that passes nine times in ten hides a real failure, the suite reruns ten times to flag any non-determinism. “I tested it” stops being a sentence a human says and becomes a number a machine checks.

5. Ban the destructive defaults

Some moves are simply removed from the agent’s reach. It cannot commit to main (a feature branch is required), cannot stage a secret (.env files are blocked), and cannot edit the CI workflow files. No judgement call in the moment, the dangerous option is not on the menu.

6. Tests ship with the code

A hook flags any commit that adds a new class with no test beside it. The agent can still proceed on purpose, but never by omission, which is how the gap usually opens.

The six above are mostly not AI-specific. Small batches, a phased rollout, a feature flag, a cheap revert: none of that is new. What AI changes is that it makes them non-negotiable, because the volume of changes that can go wrong went up. So we wrote the generic ones into AGENTS.md too, as reminders the agent re-applies every time instead of rediscovering them.

The danger zone

ThoughtWorks makes this argument directly: once an agent writes the code, throughput stops being a proxy for healthy work, because an agent can produce unlimited throughput. First-pass acceptance is the first thing I’d wire up on the next project, before the failure rate has a chance to move.

Conclusion

We didn’t cut the failure rate by reviewing harder. We cut the cost of catching a mistake, by moving each one from a person, to a rule, to a tool.

The reviewer is for judgement; everything that repeats belongs to a machine. This is the DORA result in miniature: small batches plus fast, reliable feedback keep change-failure rate down as you ship more, and a guardrail is just that feedback made cheap enough to run on every commit. Charity Majors’ “instrument first” is the same instinct one level up.

What would change my mind: the rate fell from its March peak as the guardrails went in, but I left the project before a clean quarter of post-rollout data, so this is a directional signal, not proof. If another team adopts this playbook and their change-failure rate doesn’t move, the lever was something else. If ours had kept climbing straight through the rollout, these checks were theatre.

Coming next: the signal we never had, first-pass acceptance rate, and how to measure it from the data you already collect.

---

Methodology & limitations (click to expand)

Change-failure rate follows DORA’s four keys (Forsgren, Humble & Kim, Accelerate, 2018).

Data sources

• Deploys and failures: AWS App Runner list-operations, counting START_DEPLOYMENT operations and rollbacks per month.
• Guardrail counts: files and checks in the repo’s .claude/, .github/, and committed pre-commit hook, with first-add dates from git log --diff-filter=A.

Definitions and bounds

• Change-failure rate — rollbacks / deploys per window. A lower bound on real operational failures: it counts only what App Runner auto-rolled-back.
• W1 Apr–Jun 2025 (1/12 = 8.3%), W2 Aug–Nov 2025 (1/14 = 7.1%), W3 Dec 2025–May 2026 (8/46 = 17.4% aggregate; monthly: 0% through Feb, 37.5% Mar, 20% Apr, 17.4% May).
• “Deterministic guardrails” (~35) = 12 permission denies + 7 Claude Code hooks + 16 pre-commit checks (the committed source; the copy installed on one machine lagged at 11). “Rules” (~60) = 17 rule files + 43 Copilot review rules (across the three priority tiers).

What this does not show

• No proven causation. The monthly rate fell in-window, but team maturation and a stabilising backlog ran in parallel, and I left before a clean post-rollout quarter. Correlation, not cause.
• The count measures the process we built, not the quality of the code that shipped.

---