The RealVuln Benchmark Comparative Report

This report asks one question, and it is narrower than a product comparison: if an autonomous adversary can chain several small, individually unremarkable flaws into a working attack path, how many of those flaws would your current scanning have surfaced at all? Every flaw a scanner misses is a link an agent can use.

RealVuln answers that question with measurements on real application code rather than synthetic test cases. Version 3.1 covers 140 repositories, 66 in Python and 74 in TypeScript/JavaScript, holding 4,138 scored vulnerabilities and 280 false-positive traps. 35 configurations across three categories scanned them. All artifacts, ground truth and scoring code are public.

1. 1

   The tiers are far apart. On the like-for-like two-language stratum, the security-specialized scanner leads at F3 84.4, the best general-purpose LLM reaches 79.5 and the best rule-based tool reaches 10.0. The Python stratum alone gives the same order: 86.4 / 80.0 / 23.1.

2. 2

   Rule-based tools go blind where chaining happens. Semgrep detected 0 of 232 broken-access-control/IDOR findings and 0 of 173 missing-authentication findings. SonarQube also scored 0%. Rowan reached 6.0% and 3.7%. These are the families an agentic attacker composes into a path.

3. 3

   The median scanner misses seven of every ten access-control flaws. Across all 35 runs, median recall on broken access control is 29.7% and on missing authentication 38.2%. The distribution is wide, from 0% to 87.9%, so a vendor’s category label tells you little.

4. 4

   Scale breaks detection. Provisional large-application results show aggregate recall falling from 80.55% on small Python applications to 1.77% on five systems between 263k and 816k lines, and 0.00% on three of them. On five large Java systems the same configuration matched 3 of 295 entries (1.02%), and 0 of 47 on the largest, at over 1M lines.

5. 5

   Cost is not the constraint. The strongest general-purpose LLMs complete the full 741,034-line corpus for $74–$140 per 100,000 lines, and a complete sweep costs $548–$1,041 in API spend. Spending more does not buy detection: the most expensive run in the benchmark scores 52.1.

6. 6

   Validation is the only real fix. Every coverage claim in an RFP or audit pack should be reproduced on repositories of your own size before you rely on it. If a tool cannot demonstrate recall on a 600,000-line system, it has not demonstrated coverage of your estate.

RealVuln runs each scanner against the same pinned commits of real applications and matches the findings to hand-labeled ground truth. A match requires agreement on three fields: file path (normalized, exact), CWE (present in the entry’s accepted-CWE list) and line number (within ten lines of the labeled range). Each ground-truth entry can be matched once. Findings left over are false positives; vulnerabilities left over are false negatives. A scanner that reports an attack chain rather than a single location may declare alternative locations, and it gets credit for each distinct vulnerability the chain genuinely identifies.

False-positive traps are counted at corpus level.

The corpus holds two kinds of target. Fifty repositories are teaching and CTF applications, written by humans and vulnerable on purpose. The other ninety are company-style applications generated by named coding models (Claude Opus 4.7, GPT-5.5, Kimi K2.6 and others) and then seeded with reviewed vulnerabilities. That second kind matters most to institutions now shipping AI-assisted software. The corpus covers eighteen CWE families.

Scoring

The primary metric is F3 (strict), the harmonic mean of precision and recall with β = 3, scaled to 100. That weighting puts recall nine times above precision. The asymmetry is deliberate and fits a bank: a false positive costs an analyst a few minutes, while a missed vulnerability can cost a breach.

Strict aggregation counts any repository a scanner failed to finish as if it had reported nothing, so an unreliable tool cannot hide behind a partial run. F1 and F2 are published alongside F3, so anyone can re-test the conclusions under a different weighting.

Thirty-five scanner runs are recorded in v3.1. Ten completed both language strata, and only those ten support a direct comparison. The rest ran on Python alone and appear on the Python leaderboard. Table 2 and Figure 1 show the comparable set.

Rule-based tools are licensed rather than metered per token, so their figures are not comparable on this axis. Cost is list-price API spend for the scored run, not total cost of ownership.

Reading the table

The ranking is less informative than the detail behind it. At the top of the general-purpose tier, Daybreak Blue’s 81.8% recall and 64.0% precision would have counted as a good result for a specialist tool two years ago. The claim that large models cannot reason about real code no longer holds. Within that same tier, recall runs from 81.8% down to 36.4%, so saying “we use an AI scanner” describes capabilities that differ by a factor of two. Rule-based tools sit apart from that scale. Semgrep’s 9.9% recall means it reported one vulnerability for every ten labeled ones in the corpus.

Reliability is a separate axis. Incomplete runs are common among frontier models. GLM-5.3 finished 136 of 140 repositories, and outside the comparable set several configurations finished only the Python stratum. Strict scoring counts an unfinished repository against the scanner, which is the right treatment for a regulated environment: a scan that intermittently fails to complete is not a control anyone can attest to.

Aggregate scores hide the shape of the failure. Figure 2 breaks recall out by vulnerability family for three archetypes: the best rule-based tool, the best general-purpose LLM and the security-specialized leader. The shortfall is not spread evenly. It concentrates on the families an attacker chains together.

Bold marks the leading value per family. Highlighted rows are the families an adversary composes into a path.

Rule-based detection is confined to syntactic pattern matches. It keeps partial recall on command injection (55.1%), where the pattern is a recognizable call, and falls to zero on authorization, where the flaw is a missing check spread across files. A missing check has no text to match. It is a property of the call path.

The families where rule-based coverage is zero are also the ones that make chaining cheap. An access-control gap needs no memory-corruption primitive to be useful. Put it next to an SSRF (5.4% rule-based recall) or a deserialization flaw and you have a path from the perimeter to a data layer without a single exotic exploit. That is the composition an autonomous agent performs routinely, and it is what the Rings of Risk argument is about.

No scanner dominates every family, ours included. Daybreak Blue beats DevSec Max on the flagship access-control family by 21.5 points (87.9% against 66.4%). DevSec Max leads on SSRF, sensitive data exposure, path traversal and SQL injection, and it leads on overall recall through breadth rather than through superiority in any one class. A reader looking for a single winner is looking for the wrong conclusion. The useful question is which failure modes your estate can absorb.

TypeScript/JavaScript is the harder stratum, though not for the usual reason. The TS/JS corpus is 4.5 times the size of the Python corpus (607,252 lines against 133,782) but carries only 18% more vulnerabilities, 2,236 against 1,902. Density is therefore about a quarter as high. Precision falls for that reason alone, since a scanner’s false positives land in far more clean code.

Δ F3 is TS/JS minus Python, derived from the published values.

The leaders hold their position. DevSec Max moves from 86.4 to 82.7 (−3.7 points), Daybreak Blue from 80.0 to 79.2 (−0.8) and Sol from 77.8 to 72.1 (−5.7). The middle of the field falls by 16 to 23 points: GLM-5.3 by 15.9, Claude Sonnet 5 by 17.5, DeepSeek V4 Flash by 18.2 and DeepSeek V4 Pro by 22.7. For a technology-selection decision, that asymmetry is the most useful signal in the report. A scanner that is adequate on small Python applications is not adequate on a large TypeScript estate, and the degradation is much steeper in the middle of the market than at the top.

Semgrep is the exception. Its recall rises slightly on TS/JS, from 7.0% to 12.3%, though it still tops out at twelve percent.

Every number so far comes from repositories that are small by enterprise standards. The Python applications cluster below 5,550 lines, and the corpus was designed to stay tractable for language models and for hand-labeling. A bank’s core systems are not tractable. The provisional evidence says this is where general-purpose detection fails almost completely.

Applications are pseudonymised, following the Rings of Risk naming. Names, versions and vulnerability identifiers are withheld to respect coordinated vulnerability disclosure; each system is described by what it does, and size is given as a coarse band of non-test lines of code.

Five operational conclusions follow, ordered by the point at which they create exposure.

1. 1

   A clean scan history is not evidence of a clean estate. Semgrep-class tooling missed 3,728 of 4,138 labeled vulnerabilities (90.1%), including every access-control and missing-authentication finding in the corpus. If your assurance file holds years of clean rule-based scan results, do not conclude that the estate is clean. Conclude that the instrument was blind to the flaw classes that matter most.

2. 2

   The backlog understates the exposure it purports to describe. Remediation runs on a queue, and a queue can only hold what was detected. If roughly nine in ten issues never enter it, the queue is a register of the subset your tooling could see, not a register of your vulnerabilities. The distinction becomes material when a regulator, an auditor or an insurer asks you to demonstrate that the register is complete. Documented exposure you have not scheduled is harder to defend than exposure you have not yet looked for.

3. 3

   Prioritize by chainability, not by scanner severity. The families that compose into an agent-executable path (broken access control, missing authorization, SSRF, deserialization) are exactly where syntactic detection returns zero. A backlog ordered by scanner severity, produced by a tool that is blind in those families, will systematically under-prioritize the flaws an autonomous attacker would use first.

4. 4

   Validate vendors at your scale, on your code. The benchmark’s strongest general-purpose configuration returns 0% to 5.56% recall on systems above 260,000 lines. Test any vendor claim of enterprise coverage on a repository of that size before accepting it. A demonstration on a deliberately vulnerable 800-line sample says nothing about a core banking platform. Ask for recall on a system comparable to yours, with the ground truth open to inspection.

5. 5

   Measure remediation closure, not remediation intent. Detection is half the loop. The capability that finds a chain can run it again after the fix, and re-running the attack path is the only evidence that closure is real. Without post-remediation verification, a control is a process control, not a security control.

Any evaluation has limits. These are ours, and they matter for how far the results should be pushed.

Every figure here comes from the released v3.1.0 dataset. The manifest locks the ground-truth content hash, the commit SHA of all 140 repositories and the default prompt hash, so a rerun can be checked against this document instead of taken on trust.

Sources

1. [1]RealVuln benchmark and live dashboard: realvuln.com
2. [2]Repository, ground truth, scanner outputs, scoring pipeline: github.com/kolega-ai/Real-Vuln-Benchmark
3. [3]Methodology paper: RealVuln: Benchmarking Rule-Based, General-Purpose LLM, and Security-Specialized Scanners on Real-World Code, arXiv:2604.13764.
4. [4]Large-application results: The repository-size gap: our first provisional large-app results, RealVuln Journal, Issue 20, 30 September 2026.
5. [5]Language-stratum analysis: The Python-to-TypeScript drop, RealVuln Journal.

Disclaimer. This report is provided for information. It is not legal, regulatory or investment advice. Benchmark results describe how named scanner configurations performed on a specific published corpus at a specific time. They do not predict performance on any institution’s estate. Kolega Labs develops one of the scanners evaluated and wrote the benchmark, so readers should weight the corresponding results accordingly and are encouraged to reproduce or challenge them. All product names belong to their respective owners.