Security Benchmark · SOC operations
How big is the attack? The security AI says 14. It’s 100.
In a SOC the questions are: how big is this, which assets are affected, is it still live, and is this CVE real. We tested the AI approaches that actually ship in security — RAG retrieval, a tool-calling MCP agent, an LLM analyst — against qbrin on exactly those. qbrin is exact, complete, current, and reconciled — with zero silent-wrong.
The gap, in one chart
RAG can’t count what it can’t retrieve
Wazuh’s documented AI is retrieval-based: it fetches the top-K relevant logs, then an LLM answers over just those. That’s fine for “find me a suspicious login” — and useless for “how big is this, and which assets are hit.”
Answer vs. ground truth
Live Wazuh data. qbrin matches the truth exactly; the RAG agent collapses.
How big is the attack?
alerts from one attacker · ground truth 100
The RAG agent saw 14 of 100 — it retrieves only the top-K logs, so it undercounts the attack by 86%.
Which assets to patch?
affected-asset list · ground truth 212
The RAG agent listed 10 of 212 — it physically cannot see beyond its retrieval window, so 202 affected assets go unlisted.
Not one tool — the architecture
The same collapse across the agent ecosystem
We ran the identical benchmark on three official agents in three unrelated domains — Wazuh (security), Fleet/osquery (endpoint), Netdata (observability). Below is the share of reality each platform’s retrieval AI actually captured on a simple “how many” question. The rest, it never saw.
Share of reality the AI captured
qbrin captured 100% on all three (deterministic query). The retrieval agent captured this:
saw 14 of 100 alerts · qbrin: all 100
saw 20 of 982 packages · qbrin: all 982
saw 20 of 2520 charts · qbrin: all 2520
Different products, different data, one root cause: top-K retrieval can’t count, list, or verify. qbrin queries deterministically — 100% on every one. The remaining platforms (OpenTelemetry, Zabbix, Velociraptor, Falco, Rudder, GLPI) share the exact architecture, so they share the exact failure.
The full scorecard
Nine SOC tasks, one honest table
Every row is a real task on live Wazuh data. The AI column shows what the shipping approach actually returned — wins, ties, and the honest boundary all on the same card.
| SOC task | AI approach | AI result | qbrin | Verdict |
|---|---|---|---|---|
| Scope the attackalerts from one attacker · truth 100 | RAG (top-K) | 14 — undercounts | 100 | qbrin |
| Complete affected listall web-01 level≥10 IDs · truth 212 | RAG (top-K) | 10 — missed 202 | 212 | qbrin |
| Is it still live?alert arrives after indexing | RAG (top-K) | stale — “no such user” | live · sees it | qbrin |
| Nonexistent entityalerts for a fake user | RAG (top-K) | “none exist” ✓ | none | tie ✓ |
| Count at scale~2,000 alerts · truth 1,950 | LLM + tool | 1,950 ✓ ×3 | 1,950 | tie ✓ |
| Incomplete evidencea page of alerts silently dropped | LLM + tool | undercounts 2/3 (1,850) | detects · abstains | qbrin |
| CVE attribution — naive deploy58 findings · 9 traps | LLM analyst | 6.9 silent-wrong / 100 | 0 silent-wrong | qbrin |
| CVE attribution — fully specifiedhanded the procedure + tools | LLM analyst | 57/58 · 0 silent-wrong | 58/58 | tie ✓ |
| CVE attribution — real Wazuh MCP agentcommunity server + LLM | MCP agent | CVE dropped → hallucinated | exact + reconciled | qbrin |
What the shipping AI actually does
Three approaches, tested honestly
RAG — Wazuh’s documented design
Vectorized logs, top-K retrieval, LLM answers. Built to find relevant logs, not count / list / verify them — so it undercounts the attack, misses affected assets, and a snapshot index goes stale.
100→14 · 212→10 · staleReal Wazuh MCP agent
14 tools, none for external advisories or inventory. It surfaced findings as “CVE: Unknown” (the CVEs are in the index) — so the LLM hallucinated real-world CVEs. One of two runs failed outright.
no CVE · fabricated · unreliableLLM analyst — naive deploy
Given a generic “triage this finding” prompt, it silently mis-certified vulnerabilities — flagging patched / wrong-package assets, marking a real vuln safe — on package identity, distro backports, and swapped scanner CVEs.
4 silent-wrong / 58qbrin — verification + reconciliation
Queries the live store deterministically; reconciles finding → asset → installed version → package identity → advisory → verdict; abstains on missing evidence.
exact · complete · current · 0 silent-wrongWhat this does — and doesn’t — claim
The honest boundary
- Capability is not the moat. A fully-specified LLM (handed the procedure + tools) ties qbrin on evidence-internal CVE reasoning. qbrin’s edge is being default-on, deterministic, complete, current, and abstaining — with no per-task prompt engineering.
- qbrin’s boundary, disclosed: it does not catch contract-consistent value corruption (an in-range tampered version) — there it fails like the agent. It verifies structure, completeness, freshness, and reconciliation, not arbitrary value truth.
- Scope of “Wazuh’s AI”: the MCP arm is the community server, not Wazuh’s official Cloud AI Analyst. The RAG arm faithfully implements Wazuh’s documented retrieval design.
- Honest nulls, reported straight: the AI counted 2,000 alerts correctly (counting isn’t summing) and did not fabricate on the clean nonexistent-entity case. Single model (DeepSeek-V4); rates are this model’s, the architecture points are general.
The method behind these numbers is written up in full on the blog: how we score the trust layer with 120 trap questions, and what happened when the same gates supervised a live plant loop.