When AI reviews AI
This is the direct follow-up to What 20 months of AI adoption measured. That post showed: review time quadrupled when agents started writing faster than humans could review.
TL;DR:
- Noise fell from 48.9% to 12.5% of the reviewer’s comments: the payoff of a tuned rules file, not a smarter model.
- A real issue comment (not a nitpick) climbed from 39% to 80%.
- The lever: treat
copilot-review-instructions.mdas the primary artifact and iterate on it like tests. - The model never changed.
The problem
When agents write code faster than humans can review it, review becomes the bottleneck. The naive fix: “deploy another AI to review the first one’s work.” The trap: if both use the same model, they agree with each other. LLM evaluators measurably favor their own generations.
The setup: Two independent models, shared instructions philosophy.
- Claude Code (Opus) writes the code, self-reviews with
/review-prskill before pushing (against its own AGENTS.md context) - GitHub Copilot (ChatGPT) reviews the PR independently, reading from
copilot-review-instructions.md(its own rules file, aligned intent) /copilot-reviewskill triages Copilot’s findings: Claude fixes valid ones, replies to everything, signs every reply
Independence is the whole point. One system decides; a completely different system audits.
Examples: what Copilot caught
- Admin check bypassed (security): an
id.startsWith("admin")check let any principal with a crafted prefix (e.g.adminX) gain elevated access; needs a real role/authority check - S3 ARN mismatch (deploy): code used the full bucket ARN where downstream expected just the bucket name; would fail silently in production
- In-memory filtering (performance): a lookup loaded all keys for a record and filtered in memory; push the access check down to the database
- Silent event loss (data integrity): a fully-async publish only logged failures, so callers couldn’t tell an event was dropped; make the outcome observable (await-with-timeout or retry/DLQ)
- Wrong CloudWatch metric (observability): a “failed logins” alarm wired to
DatabaseConnectionsinstead of the login-failure metric, so it would never fire
Our story of tuning the reviewer agent
We tuned the reviewer’s rules file over ~11 weeks and 11 commits, each rule earned from a real PR. Here’s what actually moved the needle.
We started with Copilot’s defaults, and nearly half its output was noise: typos, weak suggestions, false alarms. We were drowning in false positives. 😱
Then we added .github/copilot-review-instructions.md, the file Copilot reads as its review instructions, and iterated on it. Six kinds of rules did the heavy lifting:
1. A confidence floor. The first rule in the file: “Only comment when you have HIGH CONFIDENCE (>80%) that an issue exists. Be concise.” Stops the model from thinking out loud.
2. Suppress known non-issues. “Do NOT flag typos, spelling, or CHANGELOG wording.” The obvious-but-useless comments that train people to tune the bot out.
3. A false-positive allowlist. Patterns that look wrong but aren’t, so the rules got specific (conditioned on @Test or mock in the name) instead of broad.
Examples
throws Exceptionon test stubs: valid, not a bugLocalDateTime.now()in test fakes: intentional- conditioned on
@Test/mockin the name, so production code still gets flagged
4. No repeat comments. “Do NOT repeat a comment already posted on an earlier commit of the same PR.” Kills the spam where the same nit fires on every push.
5. Domain rules from real incidents. The precise rules came from real incidents, written post-mortem. Rules from real pain are specific, not pre-emptive.
Examples
@Scheduledwithout@SchedulerLock: duplicate execution across instancesLEFT JOIN FETCHwith a WHERE on the joined entity: silently wrong resultsInterruptedExceptionswallowed without restoring the interrupt
6. Priority tiers. The file sorts every finding into three buckets: Critical (block the PR), Important (strongly recommend), and Suggestion (nice to have). Labelling severity keeps a merge-blocking bug from sitting in the same flat list as a “rename this variable” nitpick.
The payoff, comparing Copilot vanilla against the tuned reviewer:
| When | Noise | Proper bugs |
|---|---|---|
| Copilot vanilla | 48.9% (217) | 39.2% (174) |
| Tuned Copilot | 12.5% (676) | 80.1% (4,319) |
| Change | ↓ 36.3pp | ↑ 41.0pp |
Noise cut by 73%. Proper bugs more than doubled in share and volume. And the tuned reviewer produced 12× more comments (444 → 5,389), and cleaner ones: re-fired duplicate comments fell from 11% of the vanilla output to 2%, thanks to one rule: “never repeat a comment from an earlier commit.” One caveat I won’t hide: the two samples are about a year apart, so a year of team learning and model upgrades rode along with the rules file. It’s the lever we controlled, not the only thing that moved (more in the methodology).
The gap between vanilla Copilot reviews and a fine-tuned rules file is the difference between “we deployed Copilot” and “we have AI code review.”
The danger zone
The loop closed. The human was left clicking Approve.
Engineers started reading only the summary in the PR description. Then not even that.
Engineers stopped reviewing code.
Humans slowly but surely got out of touch with the actual change.
In the same period our change-failure rate doubled (~8% to ~17%). Bigger release batches were the main driver, but quietly automating ourselves out of review was one of the forces behind it, and the one nobody noticed.
Checklist: how to deploy this at your scale
What we learned: The instructions file becomes your primary artifact. The review tool itself is replaceable; the accumulated team judgment in that rules file, earned from real PRs, is not. Version-control it. Iterate on it. Treat it like you treat tests.
You can probably move faster than we did if you focus on domain rules early and skip some of our false-start phases.
With that foundation:
1. Separate writer from reviewer
One model writes, a different model reviews. Claude Code self-reviews before pushing. Copilot reviews independently. Independence is the whole point.
2. Enable reviews on day one
Deploy Copilot with a basic copilot-review-instructions.md. The noise is real; the signal-to-noise is what improves through iteration.
3. After every PR, suppress one false positive
Wrong throws Exception flag? Add it. CHANGELOG wording complaint? Suppress it. The instructions file grows as a log of “things Copilot shouldn’t flag.”
4. After every incident, add one domain rule
@Scheduled without @SchedulerLock caused a real bug? Write the rule post-incident, not pre-emptively. Rules from real pain are precise.
5. Monthly retro on the instructions file
Not “is the review process working?” but “which rules fired incorrectly this month?” The commit log is your review culture’s history.
6. Make AI-authored comments sign themselves
— responded by claude, — claude, anything. Without signatures you can’t audit who actually decided what. The data-integrity problem in this post exists because we didn’t enforce this.
7. Give the reviewer the PR's intent
A description that says what changed, why, and how it was tested gives the bot (and the human) something to check the diff against. With no stated intent, a reviewer can only catch mechanical bugs, not “this doesn’t do what it claims.”
8. Keep a human approving every PR
The bot reviews; a human still approves. The moment that approval becomes a reflex click (see the danger zone), you’ve automated away the one judgment you can’t outsource. Require an approver who read the diff, not just the summary.
Conclusion
Code from agents is unreviewable at scale by humans. So you need a fine-tuned reviewer agent.
And a tuned reviewer isn’t about less noise, it’s about more signal: it surfaced roughly 640% more architecture-, security- and data-integrity-level issues across the two eras. That’s the difference between “we deployed Copilot” and “we have AI code review.”
The instructions file is where your review culture lives. Iterate on it and treat it like you treat your tests.
What would change my mind: if we froze the rules file for a quarter and noise stayed near 12.5%, then the file wasn’t the lever; a better base model was.
Coming next: How to cut failure rate without losing speed: what release-size guardrails and automated revert criteria the deploy data suggested.
Methodology (click to expand)
Data sources
- Copilot review comment samples: 5,833 categorized comments total (444 from Copilot vanilla, 5,389 after tuning)
- Categorization: manually reviewed comments to establish patterns, then systematic classification by rule (noise: style/nitpick patterns; proper bug: functional issues; architectural: production-risk categories)
- Failure-rate framing follows DORA’s change-fail-rate metric (Forsgren, Humble & Kim, Accelerate, 2018)
- Git history: 11 commits to
.github/copilot-review-instructions.md(188 → 228 lines) over ~11 weeks - Skills:
/review-prand/copilot-reviewfor the feedback loop
Signal-to-noise change (vanilla → tuned)
The same Copilot against refined instructions:
| Category | Copilot vanilla | Tuned Copilot | Change |
|---|---|---|---|
| Noise | 48.9% (217) | 12.5% (676) | ↓ 36.3pp |
| Proper Bug | 39.2% (174) | 80.1% (4,319) | ↑ 41.0pp |
| Architectural | 11.9% (53) | 7.3% (394) | ↓ 4.6pp |
The architectural share fell even though the raw count rose (53 → 394, roughly 640% more), because proper-bug volume exploded faster. Both the “640% more” claim and the falling share are the same data read two ways.
What this post can’t measure
- Bugs Copilot missed (false negatives) are unknowable from this data
- The instructions file teaches the model what to look for, not what it never knew to look for
- Real cost probably includes undiscovered issues
- The “vanilla” and “tuned” samples come from different periods: Copilot vanilla predates the rules file. So the delta also bundles team learning, model upgrades, and a changing PR mix, not the rules file alone. The rules file is the lever we controlled; it isn’t the only thing that moved.