DEV Community: Hamza Syed

Reviewing AI-generated PRs: the 10-minute checklist

Hamza Syed — Wed, 21 Oct 2026 18:55:35 +0000

AI-generated PRs have a tell: they're too clean. Perfect formatting, thorough comments, confident summaries. And that's exactly why they need a different review: the polish hides the same failure modes every time.

Reviewing AI-generated PRs: a checklist · NotVibesCoded

AI-generated pull requests look confident and complete. That confidence is the hazard. A 10-minute review checklist that catches the specific ways…

notvibescoded.com

The 10-minute checklist

Minutes 0–2: What did it touch that it shouldn't have?

Open the file list first, not the diff. Agents are notorious for scope creep: reformatting untouched files, "fixing" adjacent code, adding dependencies for one helper function. If the PR touches files outside the task, that's your first question, not your first approval.

Minutes 2–4: The tests it wrote for itself

Agents write tests that pass, not tests that protect. Check:

Do the tests assert behavior or implementation? A test that mirrors the code line-by-line will pass forever and catch nothing.
Are the edge cases real? Empty inputs, nulls, permission failures, timeouts, or just the happy path three times?
Would the test fail if the code were wrong? Mutate the logic mentally. If the test still passes, it's decoration.

Minutes 4–6: Error handling and the unhappy path

This is where AI code most consistently underperforms. Look for:

Bare except: / empty catch blocks
Errors swallowed and logged but never surfaced
Retries without backoff or limits
User-facing errors that leak stack traces or internals

Ask: "what happens when this fails?" If you can't answer from reading the code, neither can the on-call engineer.

Minutes 6–8: Security basics

Agents don't think adversarially. Scan for:

SQL/query construction with string interpolation
Missing authorization checks (authentication ≠ authorization)
Secrets or tokens in code, logs, or error messages
Mass assignment / over-permissive input binding

You don't need a full audit, just the OWASP top-10 instincts applied for two minutes.

Minutes 8–10: Would you have written it this way?

Not style, judgment. Did it pick the simple approach or the clever one? Did it add an abstraction for a single use? Did it duplicate something that exists three files over? Agents optimize for completing the task, not for the codebase's future. That's your job, and it's the part that takes the longest to learn and the least time to check once you know the smell.

What to automate

Everything above that's mechanical belongs in CI, not in your 10 minutes:

Scope check: fail if files outside the ticket's area change without explanation
Test mutation score on AI-generated PRs (higher bar than human PRs, not lower)
Secret scanning, obviously
Lint rules for the error-handling patterns your team keeps seeing

Check the description against the diff

The checklist above reviews the code. Spend one of your ten minutes reviewing the story the PR tells about the code, because AI-written descriptions are fluent, confident, and occasionally fictional.

"Added tests for X": open the test file. Count the assertions. Agents regularly describe the tests they intended to write.
"No behavior change" on a refactor: spot-check one call site of anything that moved. Refactors that rename are safe; refactors that rename and adjust a condition are behavior changes wearing a costume.
"Handles the error case": find the error case in the diff. If the handling is a log line, the description and the code disagree, and the code is the one that ships.

This isn't paranoia; it's calibration. The description is generated by the same process that generated the code, with the same confidence and the same blind spots. Trust it exactly as much as you'd trust a summary written by someone who skimmed.

When the approach itself is wrong

The hardest AI PRs to review are the ones where every line is locally fine and the whole thing is wrong. The agent added a utility that already exists two folders over. It solved the problem in the controller when your architecture solves it in the service layer. It introduced a second state-management pattern next to the one the codebase standardized on.

Line-by-line review waves these through, because each line is defensible. The defense is the file list from minutes 0–2 and one question: "If a new hire had written this, where would I tell them this code belongs?" If the answer is "not here," the review is a redirect, not a line edit. Ask for the approach change early and specifically, naming the existing pattern to follow. Agents take structural feedback well; it's the feedback they almost never get, because reviewers default to commenting on lines.

A related tell lives in the commit history. One giant commit named "implement feature" is normal for agents and useless for review; a sequence of small commits, each doing one thing, is reviewable. You can't rewrite the agent's habits, but you can set the bar in your context file: small commits, messages that say why, no unrelated changes in the same commit. Review quality is partly a function of what the reviewer is handed, and shaping the input is cheaper than straining over a bad one.

The mindset shift

Reviewing human code, you ask "is this correct?" Reviewing AI code, ask "what did it not consider?" The agent did the typing. Your review is the engineering.

This checklist is published on NotVibesCoded, alongside field guides on context files for coding agents, database migrations with AI, and shipping AI-built software without the slop.

AI Found Two Critical Flaws Hiding in PHP for a Decade

Hamza Syed — Wed, 21 Oct 2026 17:18:54 +0000

PHP released emergency security updates today for two critical vulnerabilities that allow an attacker to run code on a web server without logging in. The flaws, nicknamed PHP2Shell, are tracked as CVE-2026-109618 and CVE-2026-109619, and both carry a CVSS v4 score of 9.2. They affect the PHP engine itself, which means they sit underneath WordPress, Drupal, Magento, and every other site PHP serves, no matter how careful the code on top is.

PHP2Shell: AI found critical PHP flaws · NotVibesCoded

Two critical remote code execution flaws sat in PHP, the language behind roughly 70% of websites, for more than a decade. AI models helped researchers find…

notvibescoded.com

The detail that should get your attention is not the severity score. It is how the bugs were found, and how quickly they were turned into working attacks once someone looked properly.

What PHP2Shell Actually Is

Both flaws are memory-safety bugs in how PHP handles requests, and both end in unauthenticated remote code execution.

CVE-2026-109618 corrupts memory during out-of-memory handling. When a request runs out of memory mid-setup, PHP's error recovery can leave request data half-initialised, and the cleanup path then treats that broken data as valid.
CVE-2026-109619 is a use-after-free problem with global state. Data from one visitor's request can outlive its memory and be picked up by a later request. One confirmed attack route runs through PHP's session upload-progress feature, which has been enabled by default for over a decade.

Every PHP installation that serves web requests is in scope: PHP-FPM, Apache with mod_php, php-cgi, FrankenPHP, and LiteSpeed. The PHP team rates mod_php as the highest risk because a demonstrated exploit fires from a single unauthenticated request before any application code runs. Login screens and security plugins never get a chance to help.

Wiz, the Google-owned security company that independently found both bugs, says every PHP release back to 5.4 is hit by at least one of them. The flaws have effectively been sitting in the codebase for more than a decade.

How AI Found Them

In August, Ryan Austin, a researcher at the security firm Calif, was auditing PHP's source code with help from GPT-5.6 Sol, one of OpenAI's most advanced models. The audit was part of Patch the Planet, an open-source security programme OpenAI launched in June with Trail of Bits. Austin found one of the flaws. Wiz's researchers, working independently with Google's Gemini 4 Argon, GPT-5.6 Sol, and their own research system, found both.

OpenAI has published its account of the response under the title "Patch the Planet", and calls the bugs "one of the most impactful security issues in recent memory". That is not just marketing. The class of bug involved had been flagged as a problem area decades ago and then largely forgotten, because exploiting it was considered too difficult. AI changed the economics of trying. By the researchers' account, the models helped turn a theoretical weakness into reliable exploitation with as few as a few dozen requests to a server: seconds on small systems, hours on complex ones. Work that would have taken a skilled team months took about a week.

Calif's chief executive, Thai Duong, put the other half of the problem plainly to the Wall Street Journal: "Some dude on Twitter could take the patches and run AI against it and post some exploit script. All hell will break loose." Once a patch ships, anyone can diff it against the old code and ask a model to explain exactly what changed and where the bug was. The same capability that found the flaw can reverse-engineer it.

The Scramble

The coordinated disclosure did not survive contact with the internet. Details of both CVEs circulated on Hacker News and social media on October 20, a day before the planned announcement, and an exploit demonstration dubbed php2shell spread online. PHP pulled its release forward by a day.

On the morning of October 21, Cloudflare's CTO reported early attacks already in progress, roughly six hours before the patches landed, and said the company had deployed emergency firewall rules for all customers, including free ones. He was explicit that this was a stopgap and that only patching solves it. CISA and the FBI urged organisations to patch immediately. In the Netherlands, the national login system DigiD, the parliament website, and the country's own cybersecurity agency took their sites offline as a precaution. Security telemetry counted more than 22.6 million internet-exposed servers running potentially vulnerable PHP.

The fixes shipped at about 13:35 UTC in PHP 8.5.12, 8.4.27, 8.3.36, and 8.2.35.

If You Run PHP, Patch Now

The PHP Foundation's position is blunt: "There are no complete mitigations, updating is the only safe choice."

Update to PHP 8.5.12, 8.4.27, 8.3.36, or 8.2.35. Linux distribution and LTS users should check their vendor for packages, which were shared in advance.
Do not rely on workarounds. Setting session.upload_progress.enabled=0 blocks one confirmed attack path for CVE-2026-109619 and nothing else.
PHP 8.1 and older get no upstream fix. The PHP team released minimal source backports for 7.4, 8.0, and 8.1, and some vendors are backporting further, but an unpatched legacy install that cannot be updated should be taken offline until it can be. A large share of the PHP servers visible on the internet still run unsupported versions, and they are now the softest targets on the web.

The Part That Matters After Today

Strip away the incident response and one fact remains: flaws that survived a decade of human review were found and weaponised in weeks once frontier models were pointed at the code. That cuts in both directions at once. Defenders get an auditor that never gets tired and can read an entire engine looking for one bad assumption. Attackers get the same auditor, plus a patch-diffing machine the moment a fix ships.

This time the defenders moved first, and the coordination worked: researchers, maintainers, hosting providers, and governments had fixes and firewall rules ready before a working exploit was public. OpenAI is explicitly proposing that kind of coordinated response as the model for what comes next, because the discovery side is only going to get faster.

For anyone building software with AI, including the AI-assisted kind covered on this site, the lesson is uncomfortable and useful. The review bar just moved. Code that merely looks right has always been a liability; now there is machinery that can systematically find the places where it is not right, whether you run that machinery on your own code first or someone else runs it for you. Shipping fast is fine. Shipping unreviewed is becoming a measurable risk with a timeline attached.

Originally published on NotVibesCoded, where the PHP2Shell piece covers what happened, how AI found the flaws, and what to patch. Read the full article there, and follow for honest, hands-on coverage of building with AI.

Context files for coding agents: the complete guide

Hamza Syed — Wed, 21 Oct 2026 08:47:47 +0000

Every major coding agent now reads a context file at the start of a session. Claude Code reads CLAUDE.md. Cursor reads AGENTS.md and its own rules files. Copilot reads .github/muse-instructions.md. Aider can be pointed at a conventions file. Different names, same idea: a short document that tells the agent how your project works before it touches a line of code.

Context files for coding agents: a guide · NotVibesCoded

Every coding agent reads a context file, CLAUDE.md, AGENTS.md, Copilot instructions, and most of them are decoration. The complete guide to writing one that…

notvibescoded.com

Most of these files are useless. They’re either empty, a wishlist (“write clean, maintainable code”), or a 500-line brain dump nobody maintains. The agents aren’t the problem. The files are. This guide is the merged, field-tested version of everything we’ve published on the topic: what to put in the file, what to leave out, how to structure it, and how to prompt in a way that makes the file pay off.

One file, several names: don’t maintain five copies

If you use more than one agent (most developers now do), the naming sprawl becomes its own problem: CLAUDE.md for Claude Code, AGENTS.md as the cross-tool standard Cursor and others read, .github/muse-instructions.md for Copilot, a conventions file for Aider. Five copies of the same rules will drift apart, and drifted instructions are worse than missing ones, because each agent then follows a different version of your conventions.

The setup that survives: one canonical file, thin pointers everywhere else. Keep the real content in AGENTS.md (the name the most tools read natively), and make each tool-specific file a stub: “Read AGENTS.md in the repo root and follow it; the rules below are the only additions for this tool.” Tool quirks go in the stub, everything shared lives in exactly one place. This space moves fast, so check your tool’s current docs once a quarter, but the principle doesn’t change: one source of truth, pointers elsewhere.

The bar: every line prevents a real mistake

Think of the context file as onboarding a sharp junior developer who has infinite energy, perfect memory within a session, and zero knowledge of your project. You wouldn’t hand them a 200-page wiki. You’d give them the ten things that prevent the ten most common mistakes.

That’s the standard for every line: does this prevent a real mistake you’ve seen the agent make? If a line is philosophy, cut it. Agents follow concrete constraints. They ignore aspirations, because they were already trying to “write clean code” before you asked.

The five sections that matter

1. Stack and commands

The highest-value section in any context file is the boring one: how to build, test, and lint. Two or three commands the agent will run constantly, with the caveats that matter (“the full suite takes about 4 minutes; run a single file while iterating”). Without this, the agent guesses, and guesses wrong, burning loops on commands that don’t exist in your repo.

2. Architecture in five lines

Where the code lives and how it flows. “app/, Next.js routes. lib/, shared business logic. db/, schema and migrations.” That’s enough. The agent can read the rest of the tree itself; what it can’t infer is which parts are load-bearing and which are legacy.

3. Conventions that differ from the default

Don’t document standard practice, the model already knows it. Document where you diverge: “Database columns are snake_case; TypeScript is camelCase.” “All timestamps are stored as UTC and converted at the UI layer.” “Never use a raw hex value, always a design token.” Rules beat wishes, and conventions with one short example beat conventions described in prose.

4. Things that have bitten you

The highest-value lines in the file, because they’re the ones no model can derive from the code: “The orders table joins to users on email, not user_id.” “Don’t run migrations against the staging database; use the seed script.” “The payments module has no foreign key to users; join on customer_email.” Each line here is a debugging session you never have again.

5. What not to do

Explicit prohibitions work better than you’d expect: “Don’t add a new dependency without asking.” “Don’t refactor files unrelated to the task.” “Don’t write tests for generated code.” Anti-patterns from your actual history, stated as rules, are what separate a context file that changes behavior from one that’s decoration.

A worked example

Here’s a filled-in file for a small SaaS product. It’s 46 lines. Every line is there because it prevents a specific, observed failure.

# Ledgerline (billing SaaS)

## Commands
- Dev: `npm run dev`
- Test: `npm test -- <file>` while iterating; full suite `npm test` (~4 min)
- Lint: `npm run lint` (must pass before committing)

## Architecture
- `app/`, Next.js routes and API handlers
- `lib/`, business logic (billing in `lib/billing/`)
- `db/`, Prisma schema and migrations

## Conventions
- Database columns snake_case; TypeScript camelCase.
- All timestamps UTC in the database, converted in the UI.
- Money is integer cents everywhere. Never floats for amounts.
- UI colors come from design tokens in `lib/tokens.ts`, never raw hex.

## Gotchas
- `orders` joins to `users` on `email`, not `user_id`.
- Webhook handlers must be idempotent: Stripe retries deliveries.
- Don’t run `prisma migrate` against staging; use `npm run db:seed`.

## Don’t
- Don’t add dependencies without asking.
- Don’t refactor outside the task’s scope.
- Don’t edit generated files in `lib/gen/`.

Notice what’s not in it: no tutorial on how billing works, no style philosophy, no secrets. The webhook line alone is worth the whole file. Whoever wrote it clearly lost an afternoon to duplicate Stripe deliveries once, and now no agent on this repo ever will.

Length and layout: write for the skim

Context files compete with your actual code for the agent’s attention. The pattern that holds up:

Root file under ~100-150 lines. Essentials only: commands, hard rules, layout, gotchas.
Details colocated. Directory-specific conventions go in a nested file in that directory (components/AGENTS.md), which most agents pick up when working there.
Link, don’t inline. “See docs/auth.md for the auth flow” beats pasting the auth flow into the root file.
Match reality. If the codebase doesn’t actually follow a convention, writing it down teaches the agent to produce code that clashes with everything around it. Aspirational rules are worse than none.

The file is half the system: prompting that uses it

A good context file handles what’s always true. Your prompts still have to handle what’s true this time. The patterns that survive contact with real, messy codebases:

Point at the code, don’t describe it. “In app/auth/signup.tsx, add email validation matching the pattern in app/auth/login.tsx” beats “add validation to the signup form.” File paths are facts; descriptions are interpretations the agent has to guess at.

State the constraints before the task. Agents optimize for the request as stated. “Don’t add dependencies. Keep the change under 100 lines. This table has 40M rows; the migration must not lock writes.” If you don’t state a constraint, it doesn’t exist.

Ask for a plan on anything non-trivial. For changes touching more than two files: “Propose a plan first. Don’t write code yet.” Reviewing a plan costs a paragraph; catching the same misunderstanding in a 400-line diff costs an afternoon.

Hand over the failing state. “It’s broken” produces guesses. The exact error, the stack trace, the reproduction steps, and what you already tried produce fixes. The agent can’t see your screen, so put your screen in the prompt.

Demand the boring parts explicitly. End significant prompts with: “Include tests for the new behavior, updated docs if the API changed, and a note on what you deliberately didn’t handle.” That last clause is the highest-value sentence in prompting. It forces the agent to surface its own blind spots instead of silently shipping them.

And when the agent gets it wrong twice, suspect the prompt before the model: an unstated assumption, a missing file reference, or two tasks wearing a trench coat. Split, specify, retry.

Maintenance: the file rots faster than the code

Commands change, rules get superseded, tools arrive. Treat staleness as a bug:

Put a last-reviewed date at the top and actually review it: when onboarding a human breaks, or the agent “forgets” the same thing twice.
Delete rules that stopped mattering. A wrong instruction is worse than a missing one.
Grow it by scar tissue: after any session where the agent made a preventable mistake, add the one line that would have prevented it. A month of this produces a file worth more than most documentation.
Test the file, not just the code. Watch a fresh agent session and note every moment it does something the file should have prevented. Each one is a missing or unclear rule. Two sessions of this usually halves your correction rate.

Four ways context files die

The wishlist. Aspirations instead of rules (“write clean code”). The agent nods and changes nothing. Covered above: rules or it didn’t happen.
The novel. 500 lines of everything anyone ever learned about the project. The agent skims, the important rules drown, and nobody dares edit it. The fix is subtraction, not better formatting.
The zombie. Written once at adoption, never updated. Six months later it documents commands that no longer exist and bans a library the team adopted in March. A wrong instruction is worse than a missing one, and zombies are mostly wrong instructions.
The mirror. A copy of the README, or of another tool’s context file, drifting one edit behind the original. Two sources of truth is zero sources of truth. Point, don’t copy.

The checklist

[ ] Build/test/lint commands, verified current, with runtimes noted
[ ] Architecture in five lines or fewer
[ ] 3-5 hard rules and explicit prohibitions
[ ] Conventions that differ from defaults, one example each
[ ] Gotchas from your actual incident history
[ ] Under ~150 lines at root; details nested or linked
[ ] No secrets, no tutorials, no aspirations
[ ] Last-reviewed date at the top

Context engineering isn’t prompt magic. It’s documentation discipline, the same skill that makes a good README, applied to a reader that executes what it reads. The teams getting the most from coding agents aren’t writing cleverer prompts. They’re writing better repos, and then prompting like the file is watching.

This guide is published on NotVibesCoded, where it sits with the rest of the field guides on context engineering, PR review, and shipping AI-built software without the slop.