Deepfakes are coming for your KYC flow
I've spent a lot of time lately running suspicious images through AI or Notprofile photos, government ID scans, the kind of stuff that shows up in fraud reports. What I've noticed over the past few months is that the gap between "obviously fake" and "passes a casual check" has basically closed.
The Atlantic piece that dropped last week made that concrete in a way I hadn't seen written up this clearly before: ChatGPT's image generator is being used to produce fake identity documents. Not clumsy Photoshop jobs. Documents that are clearing automated KYC pipelines.
If you've shipped user onboarding with identity verification and a huge slice of devs reading this have this is directly your problem.
TL;DR: Generative AI just made document fraud cheap and scalable. Automated KYC flows that rely on image pattern-matching alone are increasingly inadequate. Here's what the attack surface looks like and what engineers should be thinking about.
What actually changed
For years, fake ID fraud was rate limited by skill. You needed someone who knew Photoshop, knew the security features on a specific state's license, and could produce something that wouldn't get flagged by pixel-level analysis.
GPT-4o's image model changed the cost curve. The model produces realistic textures, plausible holograms, and correctly formatted fields. You don't need skills anymore you need a prompt and maybe a few iterations.
The fraud pattern that's emerging looks roughly like this:
1. Generate a base ID image via text-to-image model
2. Composite a deepfake face (separate tool, increasingly free)
3. Submit via mobile camera capture which adds noise that masks artifacts
4. Liveness check: spoof with a face-swap running on a phone screen
5. Document check: pass, because the image quality clears the automated threshold
Step 4 is what gets me. Liveness detection was supposed to be the backstop. "Is there a real human face in front of the camera right now?" The answer used to be reliably yes or no. Now you can run a real-time face swap locally on a consumer GPU, hold it up to the camera, and a surprising number of liveness checks will pass it.
The KYC stack most of us are actually using
If you've wired up identity verification in the last few years, your stack probably looks something like this:
User uploads ID
→ Document classification (front/back, ID type)
→ OCR extraction (name, DOB, ID number)
→ Template match / security feature check
→ Face match (ID photo vs. selfie)
→ Liveness check (passive or active)
→ Risk score returned → approve / flag / deny
Every one of those steps except maybe OCR extraction is now under active attack. Template matching is fooled by generated images that have the right visual structure. Face matching is fooled by deepfakes. Liveness is fooled by face-swap tools.
The vendors selling these components know this. But "we're working on it" and "our model is continuously updated" doesn't help you when someone is opening accounts in your app today.
What I'd actually change in my stack
This is the numbered section I'd want to find if I were googling this problem at midnight.
Stop treating document image quality as a proxy for authenticity. A generated image can be sharper and more consistent than a real scan. Quality score going up is now mildly suspicious, not reassuring.
Add metadata checks on the upload itself. Real phone camera images have EXIF data, sensor noise profiles, compression artifacts from the camera pipeline. A generated image saved as JPEG and uploaded will have different statistical fingerprints. This isn't foolproof, but it raises the cost.
Cross-reference extracted data, don't just validate format. An ID number that passes a checksum algorithm but doesn't correspond to a real issued document (where that data is available) is a signal. So is a name/DOB combo that doesn't appear anywhere in any public record.
Add friction that's hard to automate at scale. Passive liveness is dead for high-stakes flows. Active liveness (follow this dot, turn your head) is still meaningfully harder to spoof at scale, even if it's not impossible.
Build a manual review queue and actually use it. Automated systems are being beaten. A human looking at a document for 30 seconds catches things the model misses. For any account type that accesses real money, that review step is cheap compared to fraud losses.
Log everything for post-hoc analysis. When a fraud pattern emerges, you want to be able to go back through your approved applications and find accounts that match. You can't do that if you didn't retain the raw evidence.
The platform problem nobody wants to talk about
OpenAI added safeguards to prevent ChatGPT from generating IDs on request. The Atlantic piece goes into how those safeguards are being bypassed specific prompt patterns, regional model variants, fine tuned open-source alternatives.
This is the part that genuinely annoys me: the safety mitigations are playing whack-a-mole with prompts, while the underlying capability — a model that can produce photorealistic documents is out in the world and getting more accessible, not less.
I ran a few of the example images from the fraud reports I've seen through AI or Not and the detection confidence on generated IDs is meaningfully higher than on a real scan, which is useful — but that only helps if you're running detection in your pipeline at all. Most KYC vendors aren't explicitly checking "is this image AI-generated" as a step. They're checking "does this look like a valid document." Those are different questions now.
For anyone building tooling in this space, AI or Not supports image and video detection via API, which means you could add a synthetic-media check as one layer in a multi-signal stack. Not a silver bullet, but it's a signal that wasn't in most fraud stacks two years ago.
The uncomfortable conclusion
The fraud economics have flipped. For most of the history of KYC, the cost to create a convincing fake was higher than the expected value of the fraud, except for sophisticated criminal operations. That's no longer true for a wide range of account types.
If you've built identity verification and you haven't audited it against generated documents and real time face swap tools in the last six months, you're probably running on assumptions that are out of date.
Top comments (0)