Skip to content
IonWarpTry for free

Docs review benchmark: models vs apps (Sep 2026)

Gemini 3.8 Flash has the highest F1 of the 3 models graded on every PR: 68.6%, at $0.08 a review. A model leads: the best of the 4 apps graded, CodeRabbit, has 55.6% F1 on 11 of the 19 PRs.

Reviewer / modelF1PrecisionRecall$ per reviewp50 timePRs
IonWarp Docs · Gemini 3.8 Flash68.6%66.7%70.6%$0.0859 sec19/19
IonWarp Docs · DeepSeek V4 Pro65.2%51.7%88.2%$0.063m 0s19/19
IonWarp Docs · Muse Spark 1.359.5%55.0%64.7%$0.0948 sec19/19
CodeRabbit55.6%55.6%55.6%~$1.507m 43s11/19
Cursor Bugbot36.4%100.0%22.2%~$1.00–1.509m 53s11/19
Greptile v4.128.6%40.0%22.2%~$1.50–2.005m 23s11/19
Qodo v226.7%33.3%22.2%~$1.676m 29s11/19

Sorted by Reviewer Score (F1 counts 3×, time and cost 1× each). Rows not graded on every PR come last. Review times come from runs on Sep 21–23, 2026 (models) and Jan 23–Apr 7, 2026 (apps). App prices (~) are list-price estimates; model prices are measured inference cost per review. Provisional labels: IonWarp chose which issues count as Docs Review findings, and no independent party has checked that choice.

19 PRs · snapshot · How we measure