The research is uncomfortable reading
When NYU researchers tested GitHub Copilot across 89 security-relevant scenarios in 2021, roughly 40% of the generated programs contained a vulnerability. That wasn't a one-off tied to an early model. Veracode's 2025 GenAI Code Security Report evaluated output from over 100 large language models and found about 45% of AI-generated samples contained a security flaw—including OWASP Top 10 issues—with newer, larger models showing no meaningful improvement.
The structural reason matters for how you respond to it. These models are trained to predict plausible code from enormous public corpora, and public code is full of insecure patterns. Without explicit security guidance in the prompt, the default output reflects the average of its training data. That average is not secure.
The failure modes are not exotic. They map almost directly onto the categories that drive traditional code review: SQL injection from string-concatenated queries, command injection, XSS from unescaped output, hardcoded secrets, weak or deprecated cryptography, missing access-control checks, path traversal. The OWASP Top 10 that experienced reviewers already know covers most of what AI assistants get wrong.
The trust gap is as dangerous as the code itself
Stanford's 2022–2023 study found that developers with AI assistant access wrote significantly less secure code than developers without it—and were more confident their code was secure. That pairing is worth sitting with: more trust, worse outcomes.
AI output is fluent, well-formatted, and confidently presented. It reads like a senior engineer wrote it. Reviewers unconsciously apply less scrutiny to code with that register, and automation bias leads engineers to defer to the machine, especially in technologies they're less familiar with—precisely where they're least equipped to catch a subtle flaw. The assistant has no accountability, no threat awareness, and no understanding of your specific deployment environment.
A distinctly AI risk: hallucinated packages
Package hallucination is worth calling out separately because it's specifically an AI problem, not a traditional code quality problem. A 2024 study generated over 205,000 unique hallucinated package names across 16 models. Around 20% of open-source model suggestions—and roughly 5% of commercial model suggestions—pointed to packages that didn't exist.
This creates a supply-chain attack called slopsquatting. An attacker registers a commonly hallucinated package name—names that models invent consistently, not randomly—with malicious code inside. Developers, or AI agents running autonomously, install it on the model's recommendation. The researchers registered one such name and observed thousands of downloads within months.
Every suggested dependency needs verification: does it exist, does it come from the expected maintainer, and is it pinned to a known-good version before it enters a lockfile?
Controls that match the actual risk
The right framing is to treat AI-generated code as untrusted input that happens to be useful—applying the same validation, review, and testing you'd demand of any external contribution.
Automate the predictable catches. Static analysis (Semgrep, CodeQL, Bandit depending on stack), secret scanners, and software composition analysis should run in CI on every AI-assisted change. These tools catch the injection patterns, exposed credentials, and known-vulnerable dependencies that appear most often. They're not new practice—they're existing practice applied with more urgency because assistant-driven volume increases the absolute number of defects entering a codebase.
Keep meaningful human review in the path. A reviewer rubber-stamping fluent code is not a control. The reviewer needs to understand the code, not just read it. Treat assistant output as a draft from an enthusiastic but unaccountable junior contributor: your name goes on the merge, not the model's.
Steer the assistant toward safer patterns. Security context in the prompt—asking for parameterized queries, explicit input validation, least-privilege handling—shifts output toward safer defaults. System instructions for AI coding tools can encode these requirements consistently rather than leaving them to per-session discipline.
A checklist for pre-merge review
Input handling
- Parameterize all queries; no string-concatenated SQL
- Avoid passing user input to the shell; use argument arrays when unavoidable
- Escape output before rendering to HTML
- Reject path traversal patterns; constrain file access to allowed directories
Auth and access control
- Every new endpoint enforces authentication, not just the happy path
- Object-level authorization checked per resource, not only at the route
- No stubbed-out auth with TODOs or permissive defaults
Secrets and configuration
- No hardcoded credentials—assistants frequently invent them as placeholders
- Secrets sourced from environment variables or a secret store
- Debug mode off and TLS verification on in production configuration
Dependencies
- Verify suggested packages exist and come from the expected maintainer
- Pin new dependencies to a specific version and run SCA scan before merge
Cryptography and sensitive data
- No MD5 or SHA-1 for passwords, no ECB mode, no hand-rolled crypto
- PII and secrets absent from logs and outbound API calls
Error handling
- Errors don't expose stack traces, queries, or internal state to end users
- Code fails closed on unexpected states, not open
The gap between code that looks right and code that holds up under pressure is where most AI-assisted security debt accumulates. These controls don't eliminate that gap, but they address the failure modes that research shows appear most often—and that's where the leverage is.
This guide originally appeared on agentpalisade.com. Agent Palisade helps small and mid-sized businesses put AI to work inside the tools they already use — practical automation, internal assistants, and AI security reviews. Book a free 30-minute call.
Top comments (0)