Skip to content
IonWarpTry for free

AI code review benchmarks

In these AI code review benchmarks, an app has the highest F1 in 5 of the 6 review types where apps were graded; a model leads Docs Review (Gemini 3.8 Flash, 68.6%). Each review type's best model costs $0.01–$0.42 a review in measured inference; the apps' list prices come to about $1.00–$2.00 a review, an estimate on a different basis.

This page is for teams choosing an AI reviewer for their pull requests, or the model behind one, who want accuracy, price and speed in one place. We run each model as an IonWarp reviewer and put it beside the hosted apps on the same open-source PRs with known issues, so where an app has the highest F1, IonWarp lost on accuracy.

Review typeBest modelF1$ per reviewp50 timeBest appApp F1PRsLabelsSnapshot
Code ReviewMuse Spark 1.356.6%$0.423m 18sQodo v258.5%50Independent
Security ReviewGLM 5.3 Flash66.7%$0.011m 38sQodo v273.3%13Provisional
AI Architecture ReviewDeepSeek V4.1 Flash47.1%$0.056m 14sCodeRabbit72.7%11Provisional
Log & Errors ReviewGemini 3.8 Flash67.7%$0.101m 28sQodo v272.7%17Provisional
Docs ReviewGemini 3.8 Flash68.6%$0.0859 secCodeRabbit55.6% (11/19 PRs)19Provisional
API Performance ReviewMuse Spark 1.356.0%$0.0642 secGreptile v4.176.9% (12/18 PRs)18Provisional
SOC 2 ReviewGLM 5.3 Flash66.7%$0.0134 secNot graded—20Provisional
CI/CD PerformanceDeepSeek V4 Pro72.7%$0.031m 22sNot graded—18Provisional
Cost ReviewDeepSeek V4.1 Flash57.1%$0.0710m 39sNot graded—20Provisional

A review type's best model is its highest-F1 IonWarp model graded on every PR. Two models that show the same F1 are a tie, and a tie goes to the cheaper one, then the faster one. Review times come from runs on Sep 8–24, 2026 (models). Code Review's PRs and labels come from the independent Martian code-review-bench; every other review type uses provisional labels IonWarp scoped itself. Open a review type for every model and app, ranked by Reviewer Score.

How we measure · The best LLM for code review

Which AI code review tool is most accurate?

No single reviewer leads every review type. On Code Review, whose PRs and labels come from the independent Martian code-review-bench, Qodo v2 has the highest F1 of the 4 apps graded, 58.5%, and Muse Spark 1.3 the highest of the 5 models graded on every PR, 56.6%, on the same 50 PRs.

How do you evaluate an AI reviewer?

Give it pull requests whose bugs are already known, then count what it catches and what it gets wrong. Precision tells you how many of its findings matched a known issue, recall how many of the known issues it found, and F1 balances the two.

Review types
9
PRs per review type
11 to 50 open-source PRs
Reviewers graded
9 models and 4 apps
Measured
F1, precision, recall, cost per review and median review time
Labels
Independent on Code Review; provisional on the other 8, scoped by IonWarp
Run by
IonWarp; Martian has not certified it
Trials
One per model and PR
App reviews
Earlier captures of each app's hosted reviews; the models ran fresh on the same PR heads
Data
Winding-Labs/ai-code-review-benchmark on GitHub, CC BY 4.0
Data updated
Published by
IonWarp, built by Ilya Volodarsky at Winding Labs

Does this benchmark test AI coding agents?

No, it tests how well a model or app reviews a pull request, not how well it writes or completes code, so a high score here says nothing about code generation. If you are picking a model to write code, read a benchmark built for that, such as SWE-bench.