AI models are now good enough to read a diff and say something useful about it. Used well, that gives every pull request a second reviewer that never gets tired and never skims. Used badly, it produces a stream of confident comments that people learn to ignore. The difference is almost entirely in how the review is set up.
What it is good at
- The mechanical misses. An unhandled error path, an off-by-one, a variable shadowed by mistake, a resource that is opened and never closed.
- Consistency. The same rule broken in a second place, a renamed function with one caller left behind, a test that no longer asserts what its name claims.
- Things people stop seeing. Secrets in a config file, a debug flag left on, a to-do comment with no ticket behind it, a dependency added for one line of code.
Where it falls short
An AI reviewer sees the diff and whatever context it is given. It does not know why the feature exists, what the customer asked for, or which trade-offs the team already argued through. It can be confidently wrong, especially about how a platform behaves at runtime. Treat its claims about the outside world as leads to verify, not facts.
Two reviewers, two jobs
One pattern that works well is to run two agents with different briefs:
- a reviewer that checks correctness, security and test coverage, and either approves or requests changes;
- an auditor that reads adversarially: scope creep, deferred work with no tracking issue, claims in the description that the diff does not support.
Each reports a severity. Only the serious findings block the merge; the rest become tracked follow-ups instead of growing the pull request round after round.
Make it a gate, not a vote
A review that can be ignored will be. Wire the verdict in as a required status check, so a blocking finding has to be fixed at the cause before the change can merge. Keep the rules short and written down, so the same finding gets the same verdict every time.
Keep a human on the merge
The AI reviewer raises the floor; it does not replace judgement. The person who merges still owns the change, still reads the diff and still decides. In practice that pairing works: the machine catches the mechanical problems early, and human attention goes to design and intent, where it is worth the most.
Measure it
Track what the reviewer finds that tests and humans did not, and how often its blocking findings turn out to be wrong. If false alarms climb, tighten the brief. If it never blocks anything, it is probably not looking hard enough.