How IonWarp benchmarks AI code review
IonWarp runs each model as an IonWarp reviewer on a fixed set of open-source pull requests with known issues, grades every finding against those issues, and publishes one table per review type: 9 review types, each with its own pull requests.
Updated , the newest snapshot below.
What each review type is measured on
| Review type | PRs | Known issues | Labels | Grader | Snapshot |
|---|---|---|---|---|---|
| Code Review | 50 | 158 | Independent: Martian code-review-bench | Martian Core, Claude Opus 4.5 judge | |
| Security Review | 13 | 18 | Provisional: scoped by IonWarp | IonWarp adapted category-scoped judge, Claude Opus 4.5 | |
| AI Architecture Review | 11 | 6 | Provisional: scoped by IonWarp | IonWarp adapted category-scoped judge, Claude Opus 4.5 | |
| Log & Errors Review | 17 | 31 | Provisional: scoped by IonWarp | IonWarp adapted category-scoped judge, Claude Opus 4.5 | |
| Docs Review | 19 | 17 | Provisional: scoped by IonWarp | IonWarp adapted category-scoped judge, Claude Opus 4.5 | |
| API Performance Review | 18 | 13 | Provisional: scoped by IonWarp | IonWarp adapted category-scoped judge, Claude Opus 4.5 | |
| SOC 2 Review | 20 | 15 | Provisional: scoped by IonWarp | IonWarp adapted category-scoped judge, Claude Opus 4.5 | |
| CI/CD Performance | 18 | 13 | Provisional: scoped by IonWarp | IonWarp adapted category-scoped judge, Claude Opus 4.5 | |
| Cost Review | 20 | 14 | Provisional: scoped by IonWarp | IonWarp adapted category-scoped judge, Claude Opus 4.5 |
Code Review's pull requests and known issues come from the independent Martian code-review-bench. Every other review type uses provisional labels: IonWarp chose which known issues count for that type, and no independent party has checked that choice.
Source: Martian code-review-bench at commit e616e849, the 50 pull requests and known issues Code Review is graded on.
The data is public: Winding-Labs/ai-code-review-benchmark on GitHub.
How a finding is scored
The grader matches each finding a reviewer posts against the pull request's known issues. Precision is the share of findings that match a known issue, recall is the share of known issues found, and F1 is their harmonic mean: 2 × matches ÷ (2 × matches + unmatched findings + missed issues). On the specialist review types, a finding that matches a known issue outside the type's scope counts as neither a match nor a false alarm.
How models are ranked
Sorted by Reviewer Score (F1 counts 3×, time and cost 1× each). Reviewer Score = (3 × F1 + latency + $/run) ÷ 5. Each part is this row's place between the worst (0) and best (100) full-cohort row in this family — higher F1, lower latency and lower $/run score higher. Scored once F1, latency and $/run are all measured and at least 3 rows are.
A review type's best model is its highest-F1 IonWarp model graded on every PR. Two models that show the same F1 are a tie, and a tie goes to the cheaper one, then the faster one.
Cost and time
A model's cost is what its provider charged per review on these pull requests, retries included. An app's cost is an estimate from its list price for 5 seats and 100 reviews a month, marked ~ in the tables. Time is the median (p50) review: for a model, from the reviewer starting to its output; for an app, from the hosted trigger to the posted review.
What these numbers do not show
- IonWarp ran the evaluation; Martian has not certified it.
- A finding that matches a known issue is not independent proof it is right, and an unmatched finding is not proof it is wrong.
- Each model reviewed each pull request once.
- Model and app times start at different points, so compare times within a kind.