We've been running an internal benchmark of AI code reviewers. The shared-corpus run — every bot on the identical pull request — kept producing the same shape of result no matter what diff we threw at it. Posting the pattern here because it changes how to think about “which AI reviewer is best.”
The setup
A normal pull request: a real code change paired with some unrelated edits to a few markdown files. A good reviewer analyzes the code, flags the unrelated edits, or both. Three bots reviewed it. They produced five findings between them and no two flagged the same thing.
What each bot did
Stripped to the layer each one operated on:
- CodeComments — one finding, on the code. Identified a real side effect the change introduced. Said nothing about the unrelated edits.
- CodeRabbit— one finding, on PR coherence. Noticed the title and contents didn't match. Didn't analyze the code change.
- Greptile— three findings, all on the unrelated edits. Noticed they violated the repo's own documented conventions. Didn't analyze the code either.
What just happened
Three bots. Five findings. Zero overlap.
The union of the five covers the PR completely; no individual bot does. The three reviewers aren't grading the same exam — they're applying fundamentally different lenses:
- Is the code correct?Analyzes the diff's actual behavior and looks for defects the change introduced. Treats PR scope as out of scope.
- Does the PR match its description? Skews toward coherence and metadata. On small diffs with no style hits, meta is most of what gets surfaced.
- Does the diff obey this repo's documented rules?Reads the repo's own conventions as policy and enforces them. Loud when the policy has something to say, quiet when it doesn't.
These lenses are complementary, not competitive. The reason “best AI reviewer” comparisons feel mushy is that the bots aren't trying to do the same job.
Two things we're thinking about
One: repo conventions live in a few well-known files in most codebases, and there's real signal there a reviewer can use. We're looking at how to consume that signal without becoming opinionated about house style.
Two: PR-scope drift — when the title says one thing and the diff does another — is something we deliberately don't check. It's a real cost to make the bot opinionated about what PRs shouldbe about. Whether it's worth that cost is an open question.
The bigger run
Before the shared corpus, we ran each bot across five PRs from its own most-active public repo. Ten PRs total, hand-scored against a strict TP / FP / Nit rubric. Walkthrough-only reviews excluded from comment counts.
| CodeRabbit | Greptile | CodeComments | |
|---|---|---|---|
| Comments / PR | 7.8 | 0.8 | 3.2 |
| TP rate | 51% | 75% | 75% |
| FP rate | 0% | 0% | 16% |
| Noise (nit) rate | 49% | 25% | 9% |
| Duplicate rate | 18% | 0% | 0% |
| P1 recall | 100% | 50% | 40% |
| P2 recall | 90% | 25% | 36% |
Read honestly: CodeRabbit is a thorough, duplicative linter with a strong bug-finding core under a lot of style noise. Greptile is quiet most of the time and accurate when it speaks, but misses real P2s it should have caught. CodeComments sits in the middle on volume, matches Greptile's true-positive rate, dramatically reduces nit noise versus CodeRabbit, and has zero duplicates — at the cost of a non-zero false-positive rate.
Discount our numbers for the obvious bias: we built CodeComments and scored its output. The shared-corpus run is what we'll keep expanding from here.
How the benchmark ran
- Public PRs only, drawn from each bot's most-active public repository.
- Hand-scored against a strict TP / FP / Nit rubric. P1 = real bug, security, or data loss. P2 = correctness or maintainability that should block merge. P3 = style or nit. When in doubt, downgrade.
- Walkthrough-only reviews excluded from comment counts.
- Same-author bias disclosure: we built CodeComments and scored its output.
Where this goes next
The shared corpus is the version that matters. Each new entry sharpens the picture of where the three lenses agree, where they don't, and what each bot still can't see.
If you maintain a repo with all three bots installed and a recent PR where they reviewed the same diff, we'd like to add it. A case where all three said the same thing would be as interesting as another zero-overlap result.