Unlimited password guessing in agent-written backends usually comes down to a login route that does exactly what the prompt asked. It looks up the email, compares the hash, returns a token, and passes every test you thought to write. Nothing in it stops someone from guessing passwords against one account as fast as the server answers.
The handler that looks finished
Here's a typical login handler (Express, trimmed). As far as it goes, it's correct:
app.post("/login", async (req, res) => {
const { email, password } = req.body;
const user = await db.users.findByEmail(email);
if (!user || !(await bcrypt.compare(password, user.passwordHash))) {
return res.status(401).json({ error: "Invalid credentials" });
}
res.json({ token: signToken(user.id) });
});
The happy path works, the wrong-password path returns a 401, and a reviewer skimming the diff sees clean code. The bug is code that was never written, and a diff only shows code that exists.
Why agents skip it
The prompt said "add login", so the agent added login. A rate limit sits outside the feature as described. No test fails without it and no status code looks wrong, so nothing in the agent's own loop tells it something is missing.
In the backendskills author's bench (15 backend tasks, Sonnet 5.5 and Haiku 5.5, 5 runs per task, scored by a hidden check the agent never sees), unlimited password guessing counts as a critical mistake. Two tasks, login-rate-limit and rename-column, accounted for 20 of the 25 critical mistakes in runs without the skills. 10 of the 15 tasks had no critical mistake either way. The failures cluster in a few places, and login is one of them.
How to catch it in review
Start with a test that fails today. Send a run of wrong passwords for one email and assert that you eventually get a 429, then do the same from one IP across many emails. If the endpoint keeps answering 401 forever, there's no limit, whatever the code looks like.
Then add the limit. It takes two limiters on the route, one keyed on the account and one on the IP. With express-rate-limit (v8):
import { rateLimit, ipKeyGenerator } from "express-rate-limit";
// Fail at startup instead of running with NaN if a variable is unset.
function requiredInt(name) {
const value = Number(process.env[name]);
if (!Number.isInteger(value) || value <= 0) {
throw new Error(`${name} must be set to a positive integer`);
}
return value;
}
const windowMs = requiredInt("LOGIN_WINDOW_MS");
const perAccount = rateLimit({
windowMs,
limit: requiredInt("LOGIN_MAX_PER_ACCOUNT"),
keyGenerator: (req) => String(req.body?.email ?? "").trim().toLowerCase(),
standardHeaders: true,
});
const perIp = rateLimit({
windowMs,
limit: requiredInt("LOGIN_MAX_PER_IP"),
keyGenerator: (req) => ipKeyGenerator(req.ip),
standardHeaders: true,
});
app.post("/login", perIp, perAccount, loginHandler);
A few things to check while you're there:
-
What the key is. You need both limits. A single combined key like
${req.ip}:${email}gives every IP its own budget for every account, so an attacker spreading guesses for one account across many addresses gets through, and so does one address trying a common password against many accounts. The per-account limiter stops the first, and the per-IP limiter stops the second.ipKeyGeneratorgroups IPv6 addresses by subnet, so one user can't rotate through their own block of addresses to reset the count. The per-account limit also lets anyone block a user's logins for one window, so keep that window short. - Where the counter lives. The default store is in memory, so with more than one app instance each one counts separately. Use a shared store such as Redis in production, with a separate prefix for each limiter.
- The other costly routes. Password reset, signup, and anything that sends an email or SMS deserve the same treatment.
Pick window and attempt values that fit your users. The test is what keeps the limit from disappearing in the next refactor.
Getting the agent to write it the first time
Review catches it, but it's cheaper if the agent writes the limiter up front. backendskills is an MIT-licensed set of three agent skills: api-endpoint, data-integrity and db-migration. api-endpoint tells the agent to scope every query to the caller, turn bad input into a 4xx, and rate-limit login and costly routes.
In the same bench, critical mistakes went from 25/150 runs without the skills to 2/150 with them (Sonnet 5.5 from 13/75 to 0/75, Haiku 5.5 from 12/75 to 2/75). Runs with the skills cost more and take longer: $0.11 vs $0.06 per run and 23 vs 14 turns on average. 25 runs were excluded because the bench account hit its usage limit.
It's one author's bench on one sample app, so treat it as a signal and check it against your own tasks. The raw runs, tasks and rerun instructions are in bench/. Each skill also has a "Make it yours" section for your team's conventions, which says "not set" as shipped. If an agent still gets something wrong with the skills loaded, the project asks for an issue with the prompt and the diff.
To try it on your next backend task, run npx skills add globalmpc/backendskills. For Codex and other AGENTS.md agents, append dist/AGENTS.md; for Cursor, copy dist/cursor/ into .cursor/rules/.
https://autopromo-mu.vercel.app/go/s6cg2ulk
Written with help from AI.
Top comments (4)
"The bug is code that was never written, and a diff only shows code that exists" - I hit the same failure shape today in a completely different domain.
I link-checked 300 listings in my AI tools directory. The table I built had a column for HTTP status and no column for "was this check ever capable of telling me the truth". 84 listings came back as failures, and two of them were Notion and Writesonic - alive companies whose servers simply do not answer from my network path. The absence was in my harness, not in the data.
So my rule now matches your argument about tests: run a few domains you know are alive through any checker before you believe its failure list. An assertion that expects an absence (a wall of wrong passwords must eventually 429; a liveness pass must eventually fail on a known-dead domain) is worth more than ten happy-path checks, because the happy path is exactly what the prompt described.
Thanks, that's a great example of the same thing. Your link checker had no column for "this failure is wrong", the same way a login route has no line for "stop after N tries". Neither shows up when you only read what's there.
Your rule is a good one: run a few domains you know are alive (and one you know is dead) through the checker first. We do the same for agent-written backend code. Every check in backendskills has a test that must fail on purpose, like a wall of wrong passwords that has to end in a 429. If you want to try those checks on your own agent's code: github.com/globalmpc/backendskills
The per-account limiter only works if the body is parsed before it runs. I tried Express 5 with
express.json()on the route after the limiters (pretty common), andreq.bodywas undefined, so every request got the key"". Five wrong guesses for one email then returned 429 for every other email too, so one person can lock the whole login. I'd parse first and 400 on an empty email.One gotcha for the detection script: don't trust the status code when counting attempts. I've seen rate limiters that keep returning 200 with an error message in the body, and others that return 429 for a while then silently fail open. My counter treated a 200 as "attempt allowed" and undercounted by a lot, so the limiter looked present and wasn't. Assert on the body and on whether guess 1001 actually succeeds, not on what the responses look like. That fail-open test is a ten-minute add to your recipe. I run a paid endpoint-audit service covering this kind of check, disclosure that it's mine, but the body-assertion rule is the part worth stealing.