Docs review benchmark: models vs apps (Sep 2026)
Gemini 3.8 Flash has the highest F1 of the 3 models graded on every PR: 68.6%, at $0.08 a review. A model leads: the best of the 4 apps graded, CodeRabbit, has 55.6% F1 on 11 of the 19 PRs.
| Reviewer / model | F1 | Precision | Recall | $ per review | p50 time | PRs |
|---|---|---|---|---|---|---|
| IonWarp Docs · Gemini 3.8 Flash | 68.6% | 66.7% | 70.6% | $0.08 | 59 sec | 19/19 |
| IonWarp Docs · DeepSeek V4 Pro | 65.2% | 51.7% | 88.2% | $0.06 | 3m 0s | 19/19 |
| IonWarp Docs · Muse Spark 1.3 | 59.5% | 55.0% | 64.7% | $0.09 | 48 sec | 19/19 |
| CodeRabbit | 55.6% | 55.6% | 55.6% | ~$1.50 | 7m 43s | 11/19 |
| Cursor Bugbot | 36.4% | 100.0% | 22.2% | ~$1.00–1.50 | 9m 53s | 11/19 |
| Greptile v4.1 | 28.6% | 40.0% | 22.2% | ~$1.50–2.00 | 5m 23s | 11/19 |
| Qodo v2 | 26.7% | 33.3% | 22.2% | ~$1.67 | 6m 29s | 11/19 |
Sorted by Reviewer Score (F1 counts 3×, time and cost 1× each). Rows not graded on every PR come last. Review times come from runs on Sep 21–23, 2026 (models) and Jan 23–Apr 7, 2026 (apps). App prices (~) are list-price estimates; model prices are measured inference cost per review. Provisional labels: IonWarp chose which issues count as Docs Review findings, and no independent party has checked that choice.
19 PRs · snapshot · How we measure