Skip to content
IonWarpTry for free

CI/CD performance benchmark: models compared (Sep 2026)

DeepSeek V4 Pro has the highest F1 of the 5 models graded on every PR: 72.7%, at $0.03 a review. No app was graded on these 18 PRs.

Reviewer / modelF1PrecisionRecall$ per reviewp50 timePRs
IonWarp CI/CD Performance · DeepSeek V4 Pro72.7%60.0%92.3%$0.031m 22s18/18
IonWarp CI/CD Performance · GLM 5.3 Flash55.0%40.7%84.6%<$0.0150 sec18/18
IonWarp CI/CD Performance · DeepSeek V4.1 Flash59.5%45.8%84.6%$0.0154 sec18/18
IonWarp CI/CD Performance · GPT 5.6 Luna Pro54.5%45.0%69.2%$0.0133 sec18/18
IonWarp CI/CD Performance · Muse Spark 1.344.4%42.9%46.2%$0.0349 sec18/18
CodeRabbitN/AN/AN/A~$1.50N/AN/A
Cursor BugbotN/AN/AN/A~$1.00–1.50N/AN/A
Greptile v4.1N/AN/AN/A~$1.50–2.00N/AN/A
Qodo v2N/AN/AN/A~$1.67N/AN/A

Sorted by Reviewer Score (F1 counts 3×, time and cost 1× each). Rows not graded on every PR come last. Review times come from runs on Sep 21, 2026 (models). App prices (~) are list-price estimates; model prices are measured inference cost per review. Provisional labels: IonWarp chose which issues count as CI/CD Performance findings, and no independent party has checked that choice.

18 PRs · snapshot · How we measure