Skip to content
IonWarpTry for free

How IonWarp benchmarks AI code review

IonWarp runs each model as an IonWarp reviewer on a fixed set of open-source pull requests with known issues, grades every finding against those issues, and publishes one table per review type: 9 review types, each with its own pull requests.

Updated , the newest snapshot below.

What each review type is measured on

Review typePRsKnown issuesLabelsGraderSnapshot
Code Review50158Independent: Martian code-review-benchMartian Core, Claude Opus 4.5 judge
Security Review1318Provisional: scoped by IonWarpIonWarp adapted category-scoped judge, Claude Opus 4.5
AI Architecture Review116Provisional: scoped by IonWarpIonWarp adapted category-scoped judge, Claude Opus 4.5
Log & Errors Review1731Provisional: scoped by IonWarpIonWarp adapted category-scoped judge, Claude Opus 4.5
Docs Review1917Provisional: scoped by IonWarpIonWarp adapted category-scoped judge, Claude Opus 4.5
API Performance Review1813Provisional: scoped by IonWarpIonWarp adapted category-scoped judge, Claude Opus 4.5
SOC 2 Review2015Provisional: scoped by IonWarpIonWarp adapted category-scoped judge, Claude Opus 4.5
CI/CD Performance1813Provisional: scoped by IonWarpIonWarp adapted category-scoped judge, Claude Opus 4.5
Cost Review2014Provisional: scoped by IonWarpIonWarp adapted category-scoped judge, Claude Opus 4.5

Code Review's pull requests and known issues come from the independent Martian code-review-bench. Every other review type uses provisional labels: IonWarp chose which known issues count for that type, and no independent party has checked that choice.

Source: Martian code-review-bench at commit e616e849, the 50 pull requests and known issues Code Review is graded on.

The data is public: Winding-Labs/ai-code-review-benchmark on GitHub.

How a finding is scored

The grader matches each finding a reviewer posts against the pull request's known issues. Precision is the share of findings that match a known issue, recall is the share of known issues found, and F1 is their harmonic mean: 2 × matches ÷ (2 × matches + unmatched findings + missed issues). On the specialist review types, a finding that matches a known issue outside the type's scope counts as neither a match nor a false alarm.

How models are ranked

Sorted by Reviewer Score (F1 counts 3×, time and cost 1× each). Reviewer Score = (3 × F1 + latency + $/run) ÷ 5. Each part is this row's place between the worst (0) and best (100) full-cohort row in this family — higher F1, lower latency and lower $/run score higher. Scored once F1, latency and $/run are all measured and at least 3 rows are.

A review type's best model is its highest-F1 IonWarp model graded on every PR. Two models that show the same F1 are a tie, and a tie goes to the cheaper one, then the faster one.

Cost and time

A model's cost is what its provider charged per review on these pull requests, retries included. An app's cost is an estimate from its list price for 5 seats and 100 reviews a month, marked ~ in the tables. Time is the median (p50) review: for a model, from the reviewer starting to its output; for an app, from the hosted trigger to the posted review.

What these numbers do not show