Earlier this year an external penetration test of our attack platform came back with five findings. We build offensive security tooling for a living, so that one stung a little.
The problems
- A repository URL field let a logged-in user hand options to
git. Git will happily run a command you give it through an option like--upload-pack. - Our API rate limiter was configured, but never switched on.
- A login redirect parameter was reported as an open redirect. On retest it stayed on our own domain, so we could not reproduce it.
- The app could edit user records that our identity provider owns.
- A user disabled in the identity provider was not blocked in the app.
Three code fixes merged the next day. Though to be honest: removing the screen that edited users did not remove the API behind it. That took until July.
Before:
var args = $"clone --branch {branch} --single-branch {url} {targetPath}";
After:
// "--" will stop option parsing after it
string[] args = ["clone", "--branch", branch, "--single-branch", "--", url, targetPath];
None of this came from not caring. Every control was there in some shape, and each one was missing a single dimension. The git call had validation, just not for input shaped like an option. The rate limiter existed, it just was not attached to anything. In the fix's own words: "Backend rate limiter was registered but never enforced."
Why this happens
You know what you are building. That does not mean you know what the security world calls it.
"Clone a repository from a URL the user types in" is a feature to a developer. To OWASP it is argument injection and, depending on where the URL points, server-side request forgery (SSRF). If you never think of those two words, you never look up what they require.
Humans and AI agents fail in exactly the same way here. Agents just do it faster, and with more confidence.
That pentest is what started this research. Could we make threat modelling and security-aware implementation nearly automatic every time we work with an AI agent? Features, refactors and bug fixes alike.
First attempt: give the agent the rulebook
I built a knowledge base that serves the OWASP Application Security Verification Standard (ASVS 5.0) to the agent locally. The agent had it installed, listed and reachable.
It did not use it. Asked afterwards why it skipped the lookup while planning an SSRF fix, the agent said:
It crossed my mind given SSRF is an OWASP category, but the vulnerability was already well-characterized in the issue body. The fix is straightforward. OWASP skill would add process overhead without new insight here.
The plan it produced was right in shape and wrong in a way you could check. It proposed an allowlist, which is what ASVS V1.3.6 asks for. It covered domains, paths and ports, and never mentioned protocols, the first thing that requirement lists. It also validated the address before connecting, which the OWASP SSRF Prevention Cheat Sheet documents as bypassable through DNS tricks.
The agent does not know what it does not know, and it cannot tell from the inside.
Second hurdle: the vocabulary
Making the agent look things up only works if it knows what to look up.
A later session got a ticket that read like plumbing: move a stored credential and an outbound URL from one part of the data model to another. The agent's reasoning was "it is a refactor, the attack surface is unchanged." Nothing in that ticket says SSRF, so nothing triggered.
The fix was adding the OWASP Cheat Sheet Series to the knowledge base.
Cheat sheets are organised around what you are building, so a plain description of the feature finds them. So now the agent works like this:
- Describe what the code does with data in three to five plain sentences.
- Search the cheat sheets with those sentences.
- Read which security terms come back: SSRF, XSS, open redirect and so on.
- Search ASVS with those terms for the exact requirement.
- Search our own code for an existing control before writing a new one.
- Cite the requirement ID in the code or the pull request.
In that session, step 5 is what prevented two defects. There was already a shared URL guard and already a redirect callback that re-checks every hop. The agent was about to write a second validator and to "fix" redirects by turning them off.
The check took 5 minutes 56 seconds. It landed 3 hours 20 minutes after the plan it should have shaped, which was only in time because the code was still two tasks away.
Making it permanent
Nobody remembers to look things up every time. The reminders now live in the tooling:
- A hook fires when a prompt names a vulnerability class like SSRF, XSS or path traversal.
- A second hook runs at the start of every session, because most security work arrives as "tokens", "sessions" or "a refactor", words too common to match on.
- The agent has to say which way it went: "Using
owasp-secure-codingto ..." or "Not usingowasp-secure-coding: ..." The second line is the important one. It gives a human something to overrule while the work is still open.
The argument we give the agent is about cost. Reading a requirement is cheap. Rework is paid twice: the wrong fix, then the right one, plus the review that missed it.
The reviewer reads the same rulebook
Our pull request reviewer is a Claude Code agent too. It reads the same knowledge base, and it cites ASVS inline when it finds something.
Our favourite example is close to home. The bug was in the instructions for our own weekly OWASP scan agent. Those instructions read a commit ID out of a GitHub issue and paste it into a git command without checking it. The reviewer tried it in a throwaway repository:
Recorded head | Result |
|---|---|
52e88494 | Works |
$(touch /tmp/PWNED) | The file was created because substitution runs before git parses anything |
--output=/tmp/x | Accepted; there is no -- separator, so an option-shaped value is argument injection |
The reviewer cited ASVS 5.0.0 V1.2.5 (L1) and the OS Command Injection Defense Cheat Sheet for the option-shaped case.
That is the same bug class as the pentest's git finding, caught this time before it merged.
A second example came from a deployment workflow. A cloud tag value was pasted into a shell command that ran as root on our production machines. A hostile tag could send an edited value such as:
vpn-join --hostname=fleet-evil;id>/tmp/pwned
The author pushed a fix four minutes later, before the pull request merged:
if ! [[ "$ts_hostname" =~ ^[A-Za-z0-9-]+$ ]]; then
And a third involved a race condition. Two requests at the same moment could both slip under a per-user limit. The reviewer reproduced it against a real database and cited "CWE-362 (race condition) / ASVS 5.0.0 V2.1.3." The author's reply: "This was a real bug and the reasoning is exactly right."
Across the reviewed work:
- Approximately 12,800 reviewer comments and reviews
- 143 citations of OWASP ASVS
- Approximately 50 threads ending in "Fixed in"
No author called one a false positive. Not every citation is a vulnerability. Some are nits, and the reviewer says so itself: "This is not log injection, and I am not claiming it is."
Where we are today
Every developer at Crimson7 uses the knowledge base in Claude Code daily. The skill we use most is owasp-secure-coding, the six-step loop above. A weekly scan reviews everything that merged.
When we want the full picture, owasp-asvs-verify assesses the whole codebase against ASVS. It goes requirement by requirement and reports which controls are in place and which are missing, with the file and line as evidence.
We also expect it to get more out of a pentest. With the low-hanging fruit already covered, the testers have to look for the less obvious ways into the system.
It does not solve everything. An agent can still look something up after it has already picked its approach. At that point the lookup tends to confirm the choice instead of changing it, and today nothing tells us which order it happened in.
Try it yourself
Setup instructions are in the repository's Install section, for Claude Code, Codex, OpenCode and any other agent that supports MCP (the Model Context Protocol).
Thank you
None of this would exist without the OWASP Foundation. ASVS, the Cheat Sheet Series and the rest of OWASP's guidance are written by its community and published for free, under an open licence (CC BY-SA 4.0) that lets projects like ours build on them. The knowledge base is only as good as the material inside it, and that material is theirs.
owasp-kb is not affiliated with or endorsed by the OWASP Foundation.
