A 134-agent adversarial security audit of my own code

The problem

That's 162 thousand lines and no second pair of eyes on them. I direct the build, write the evals and review the result, all with the same head — so what didn't occur to me while building doesn't occur to me while reviewing either. I didn't have a security team to review it for me, so I built one: dozens of agents sweeping the code at once, each from a different angle, and as many again trying to tear down whatever the first ones found.

Constraints

• 01Adversarial by default. A verifier whose job is to confirm findings will confirm noise. Each one's job was to refute, and when unsure the verdict is "not a bug".
• 02Breadth without a single point of view. One prompt reviewing everything sees everything the same way. Many hunters, each with its own lens.
• 03No false criticals. A wrong "CRITICAL" burns trust and a day of someone's work.
• 04Reproducible enough to act on. A finding I can't tie to a file and a failure scenario is a rumour.

Architecture

The shape matters more than the agent count: fan out to find, verify every claim adversarially, keep only what survives a majority of the skeptics. The 134 agents are what that shape costs on a codebase this size. The discipline is that nothing reaches the report without getting past a panel that scores points by throwing it out.

The hard part

The interesting findings were the ones that died. A hunter raised what looked like a real vulnerability: a service-role query apparently bypassing row-level security. Plausible, well argued, wrong. A skeptic followed the call by hand and found the visibility check that runs before it.

If I had believed the hunters, I'd have fixed code that wasn't broken — and touching authorization when you don't have to is how you open a real hole. 41 findings went in and 20 came out. The 21 that died are the reason I take the others seriously: an audit that reports everything it suspects hands me a to-do list, not a diagnosis.

Trade-offs

• Many agents cost real tokens. Shipping a false CRITICAL, or missing a real one, costs more than the compute.
• Verifying adversarially is slower than believing the hunter. That slowness is what turns 41 suspicions into 20 facts.
• This complements static analysis and human review, it doesn't replace either. The audit is strongest on logic and authorization, which is exactly where a linter is blind.

What I'd do differently

I'd close the loop. Every confirmed finding should become a regression test or an eval case, so a hole I patched can't quietly reopen six months later when someone refactors nearby. The audit found the bugs once; it should leave behind the alarm that catches them the second time.