Skip to content
IonWarpTry for free

SOC 2 review benchmark: models compared (Sep 2026)

GLM 5.3 Flash and DeepSeek V4.1 Flash tie for the highest F1 of the 5 models graded on every PR (66.7%); GLM 5.3 Flash is the cheaper, at $0.01 a review in 34 sec. No app was graded on these 20 PRs.

Reviewer / modelF1PrecisionRecall$ per reviewp50 timePRs
IonWarp SOC 2 · GLM 5.3 Flash66.7%61.1%73.3%$0.0134 sec20/20
IonWarp SOC 2 · GPT 5.6 Luna Pro64.5%62.5%66.7%$0.0737 sec20/20
IonWarp SOC 2 · DeepSeek V4.1 Flash66.7%54.2%86.7%$0.045m 56s20/20
IonWarp SOC 2 · DeepSeek V4 Pro56.3%52.9%60.0%$0.072m 21s20/20
IonWarp SOC 2 · Muse Spark 1.355.2%57.1%53.3%$0.1148 sec20/20
CodeRabbitN/AN/AN/A~$1.50N/AN/A
Cursor BugbotN/AN/AN/A~$1.00–1.50N/AN/A
Greptile v4.1N/AN/AN/A~$1.50–2.00N/AN/A
Qodo v2N/AN/AN/A~$1.67N/AN/A

Sorted by Reviewer Score (F1 counts 3×, time and cost 1× each). Rows not graded on every PR come last. Review times come from runs on Sep 21–24, 2026 (models). App prices (~) are list-price estimates; model prices are measured inference cost per review. Provisional labels: IonWarp chose which issues count as SOC 2 Review findings, and no independent party has checked that choice.

20 PRs · snapshot · How we measure