SOC 2 review benchmark: models compared (Sep 2026)
GLM 5.3 Flash and DeepSeek V4.1 Flash tie for the highest F1 of the 5 models graded on every PR (66.7%); GLM 5.3 Flash is the cheaper, at $0.01 a review in 34 sec. No app was graded on these 20 PRs.
| Reviewer / model | F1 | Precision | Recall | $ per review | p50 time | PRs |
|---|---|---|---|---|---|---|
| IonWarp SOC 2 · GLM 5.3 Flash | 66.7% | 61.1% | 73.3% | $0.01 | 34 sec | 20/20 |
| IonWarp SOC 2 · GPT 5.6 Luna Pro | 64.5% | 62.5% | 66.7% | $0.07 | 37 sec | 20/20 |
| IonWarp SOC 2 · DeepSeek V4.1 Flash | 66.7% | 54.2% | 86.7% | $0.04 | 5m 56s | 20/20 |
| IonWarp SOC 2 · DeepSeek V4 Pro | 56.3% | 52.9% | 60.0% | $0.07 | 2m 21s | 20/20 |
| IonWarp SOC 2 · Muse Spark 1.3 | 55.2% | 57.1% | 53.3% | $0.11 | 48 sec | 20/20 |
| CodeRabbit | N/A | N/A | N/A | ~$1.50 | N/A | N/A |
| Cursor Bugbot | N/A | N/A | N/A | ~$1.00–1.50 | N/A | N/A |
| Greptile v4.1 | N/A | N/A | N/A | ~$1.50–2.00 | N/A | N/A |
| Qodo v2 | N/A | N/A | N/A | ~$1.67 | N/A | N/A |
Sorted by Reviewer Score (F1 counts 3×, time and cost 1× each). Rows not graded on every PR come last. Review times come from runs on Sep 21–24, 2026 (models). App prices (~) are list-price estimates; model prices are measured inference cost per review. Provisional labels: IonWarp chose which issues count as SOC 2 Review findings, and no independent party has checked that choice.
20 PRs · snapshot · How we measure