Benchmark · SOC operations
AI in the SOC. The RAG said 14 alerts. It was 100.
In a SOC the questions are: how big is this, which assets are affected, is it still live, and is this CVE real. We tested the AI approaches that actually ship in security — RAG retrieval, a tool-calling MCP agent, an LLM analyst — against qbrin on exactly those. qbrin is exact, complete, current, and reconciled — with zero silent-wrong.
- Dated 2026-07-20
- Stack: self-hosted Wazuh 4.9 + real community MCP server + RAG
- Model held constant (DeepSeek-V4)
The gap, in one chart
RAG can’t count what it can’t retrieve.
Wazuh’s documented AI is retrieval-based: it fetches the top-K relevant logs, then an LLM answers over just those. That’s fine for “find me a suspicious login” — and useless for “how big is this, and which assets are hit.”
Answer vs. ground truth
Live Wazuh data. qbrin matches the truth exactly; the RAG agent collapses.
How big is the attack?
alerts from one attacker · ground truth 100
The RAG agent saw 14 of 100 — it retrieves only the top-K logs, so it undercounts the attack by 86%.
Which assets to patch?
affected-asset list · ground truth 212
The RAG agent listed 10 of 212 — it physically cannot see beyond its retrieval window, so 202 affected assets go unlisted.
Not one tool — the architecture
The same collapse across the agent ecosystem.
We ran the identical benchmark on three official agents in three unrelated domains — Wazuh (security), Fleet/osquery (endpoint), Netdata (observability). Below is the share of reality each platform’s retrieval AI actually captured on a simple “how many” question. The rest, it never saw.
Share of reality the AI captured
qbrin captured 100% on all three (deterministic query). The retrieval agent captured this:
Share of reality captured
Retrieval agent on each platform: share of the true count it saw on a simple “how many” question.
| Platform | Domain | Retrieval agent | qbrin |
|---|---|---|---|
| Wazuh | Security / SIEM | saw 14 of 100 alerts · 14% | all 100 |
| Fleet + osquery | Endpoint / IT | saw 20 of 982 packages · 2.0% | all 982 |
| Netdata | Observability | saw 20 of 2520 charts · 0.8% | all 2520 |
Different products, different data, one root cause: top-K retrieval can’t count, list, or verify. qbrin queries deterministically — 100% on every one. The remaining platforms (OpenTelemetry, Zabbix, Velociraptor, Falco, Rudder, GLPI) share the exact architecture, so they share the exact failure.
The full scorecard
Nine SOC tasks, one honest table.
Every row is a real task on live Wazuh data. The AI column shows what the shipping approach actually returned — wins, ties, and the honest boundary all on the same card.
| SOC task | AI approach | AI result | qbrin | Verdict |
|---|---|---|---|---|
| Scope the attackalerts from one attacker · truth 100 | RAG (top-K) | 14 — undercounts | 100 | qbrin |
| Complete affected listall web-01 level≥10 IDs · truth 212 | RAG (top-K) | 10 — missed 202 | 212 | qbrin |
| Is it still live?alert arrives after indexing | RAG (top-K) | stale — “no such user” | live · sees it | qbrin |
| Nonexistent entityalerts for a fake user | RAG (top-K) | “none exist” ✓ | none | Tie |
| Count at scale~2,000 alerts · truth 1,950 | LLM + tool | 1,950 ✓ ×3 | 1,950 | Tie |
| Incomplete evidencea page of alerts silently dropped | LLM + tool | undercounts 2/3 (1,850) | detects · abstains | qbrin |
| CVE attribution — naive deploy58 findings · 9 traps | LLM analyst | 6.9 silent-wrong / 100 | 0 silent-wrong | qbrin |
| CVE attribution — fully specifiedhanded the procedure + tools | LLM analyst | 57/58 · 0 silent-wrong | 58/58 | Tie |
| CVE attribution — real Wazuh MCP agentcommunity server + LLM | MCP agent | CVE dropped → hallucinated | exact + reconciled | qbrin |
What the shipping AI actually does
Three approaches, tested honestly.
RAG — Wazuh’s documented design
Vectorized logs, top-K retrieval, LLM answers. Built to find relevant logs, not count / list / verify them — so it undercounts the attack, misses affected assets, and a snapshot index goes stale.
Real Wazuh MCP agent
14 tools, none for external advisories or inventory. It surfaced findings as “CVE: Unknown” (the CVEs are in the index) — so the LLM hallucinated real-world CVEs. One of two runs failed outright.
LLM analyst — naive deploy
Given a generic “triage this finding” prompt, it silently mis-certified vulnerabilities — flagging patched / wrong-package assets, marking a real vuln safe — on package identity, distro backports, and swapped scanner CVEs.
qbrin — verification + reconciliation
Queries the live store deterministically; reconciles finding → asset → installed version → package identity → advisory → verdict; abstains on missing evidence.
What this does — and doesn’t — claim
The honest boundary.
The method behind these numbers is written up in full on the blog: how we score the trust layer with 120 trap questions, and what happened when the same gates supervised a live plant loop. For how qbrin handles your own data, read the security page.
See it on a real example.
Book a 20-minute walkthrough. We connect one source, read-only, and show qbrin on a real example from your setup.