DEV Community: Thomas

Realtime deepfake software is a SaaS product now

Thomas — Thu, 07 May 2026 22:21:00 +0000

I've been half-following the deepfake-in-the-wild beat for a while. Most of it has been static image stuff fake profile photos, AI generated headshots on LinkedIn, that kind of thing. I run suspicious images through AI or Not when something looks off, flag it, move on.

But the 404 Media investigation into "HELLO BOSS" software shifted my sense of where the floor actually is. This isn't someone uploading a faked image. This is a live video call where the person on screen is not the person on screen.

What the pipeline actually looks like

The software they describe isn't magic it's a real-time face swap layer that sits between the camera input and whatever video call software the scammer is using. The rough architecture:

[Scammer's webcam]
        ↓
[Face detection + landmark extraction]
        ↓
[Target face model (pre-trained on victim's photos)]
        ↓
[Rendered output frame]
        ↓
[Virtual camera driver — OBS, v4l2loopback, etc.]
        ↓
[Zoom / WhatsApp / Teams / any WebRTC app]

The virtual camera driver is the key piece most people miss. Tools like OBS Virtual Camera on Windows/Mac or v4l2loopback on Linux let you present any video source as a system webcam. The calling app has no idea it's not getting real hardware input.

The face-swap model itself runs inference on every frame typically 24–30 fps which used to require a beefy GPU. Consumer grade hardware can handle it now, and cloud GPU instances are cheap enough that you can rent the compute if you don't own it.

"Hello boss" isn't a technical term it's a script

The name comes from one of the primary use cases: impersonating a company executive on a video call to authorize a wire transfer. Subordinate gets a call from what looks like their CEO on screen, hears a voice that's been cloned or at least pitch-shifted, and gets told to move money somewhere.

The "hello boss" phrasing is the greeting on the other end the scammer picking up a call from someone who thinks they're reaching their boss, not the scammer impersonating the boss outbound. Either way, the social engineering depends entirely on the live video being convincing enough to short-circuit skepticism.

This same stack powers romance scams and "pig butchering" investment fraud, where the point is sustained trust over weeks, not a one time wire transfer. A static fake photo stops working the moment someone asks for a video call. A real time face swap keeps the fiction going.

Why this breaks video KYC assumptions

A lot of identity verification flows have converged on "liveness check + face match" as the standard:

1. User holds up ID document → OCR extracts name, DOB, document number
2. User records short selfie video → liveness detection (blink, turn head)
3. Face on selfie matches face on document → pass

This pipeline assumes the face in the selfie video is the person's real face. Real time face swap defeats step 3 entirely if the attacker pre trains their model on photos of the person whose identity they're stealing. It also defeats liveness checks the swap handles arbitrary head movements and expressions in real time, so asking someone to blink or smile doesn't help.

Some vendors are adding texture analysis, illumination consistency checks, and temporal coherence scoring to catch the artifacts that face swap models still produce at frame boundaries and occlusion edges. But that arms race is already underway and the defenders are not obviously winning.

What I'd actually do differently if I were building this today

Don't trust video alone. Pair any video verification step with a second channel SMS OTP, authenticator app, document scan from a different session so compromising the video doesn't compromise the whole flow.
Log the raw video stream for async review. Real-time detectors aren't reliable enough to be gatekeepers. Use them as signals, not hard blocks, and let a human or a more thorough model review borderline cases after the fact.
Add device fingerprinting. Face swap pipelines route through a virtual camera driver. The camera device name exposed by browser WebRTC APIs (MediaDeviceInfo.label) will often be "OBS Virtual Camera" or similar. That's not a perfect signal, but it's a cheap one worth logging.
Test your liveness checks against an actual face swap. There are open source models you can run locally. If your liveness check passes, you need to know now, not when a fraud team calls you.
Assume the video is synthetic and design accordingly. Treat video verification as a corroborating signal, not a root-of-trust.

Checking clips yourself

When I see video circulating that seems suspicious a celebrity endorsing something, an executive making a statement AI or Not is what I pull up first. It handles video files, not just images, so you can actually run the clip rather than screenshotting a frame and hoping the compression didn't wash out the artifacts. It's not a forensic lab, but it's fast and the confidence scores are useful for triage.

The problem the 404 Media story describes is harder to catch after the fact because there usually isn't a recording it happened on a live call. But for anything that did get recorded, that kind of tooling matters.

The SaaS part is the part that scales

What makes this story different from "deepfakes exist, film at 11" is the distribution model. The software described in the 404 Media piece is sold through Telegram channels at subscription prices. That means:

Low barrier to entry. You don't need to train a model or write code.
Sellers have support channels, update cadences, and refund policies.
Supply scales with demand, not with technical skill.

This is the same trajectory malware took. Ransomware-as-a-service normalized the idea that you could be a criminal without being a programmer. Deepfake-as-a-service is doing the same thing for identity fraud.

The underlying models will keep improving. The virtual camera trick is already table stakes. At some point, real-time voice cloning (already fairly mature) and real-time video swap running together on consumer hardware will be seamless enough that "get on a video call" is no longer a meaningful trust signal.

That's not a prediction it's a present tense engineering problem.

Deepfakes are coming for your KYC flow

Thomas — Tue, 05 May 2026 17:47:06 +0000

Deepfakes are coming for your KYC flow

I've spent a lot of time lately running suspicious images through AI or Notprofile photos, government ID scans, the kind of stuff that shows up in fraud reports. What I've noticed over the past few months is that the gap between "obviously fake" and "passes a casual check" has basically closed.

The Atlantic piece that dropped last week made that concrete in a way I hadn't seen written up this clearly before: ChatGPT's image generator is being used to produce fake identity documents. Not clumsy Photoshop jobs. Documents that are clearing automated KYC pipelines.

If you've shipped user onboarding with identity verification and a huge slice of devs reading this have this is directly your problem.

TL;DR: Generative AI just made document fraud cheap and scalable. Automated KYC flows that rely on image pattern-matching alone are increasingly inadequate. Here's what the attack surface looks like and what engineers should be thinking about.

What actually changed

For years, fake ID fraud was rate limited by skill. You needed someone who knew Photoshop, knew the security features on a specific state's license, and could produce something that wouldn't get flagged by pixel-level analysis.

GPT-4o's image model changed the cost curve. The model produces realistic textures, plausible holograms, and correctly formatted fields. You don't need skills anymore you need a prompt and maybe a few iterations.

The fraud pattern that's emerging looks roughly like this:

1. Generate a base ID image via text-to-image model
2. Composite a deepfake face (separate tool, increasingly free)
3. Submit via mobile camera capture which adds noise that masks artifacts
4. Liveness check: spoof with a face-swap running on a phone screen
5. Document check: pass, because the image quality clears the automated threshold

Step 4 is what gets me. Liveness detection was supposed to be the backstop. "Is there a real human face in front of the camera right now?" The answer used to be reliably yes or no. Now you can run a real-time face swap locally on a consumer GPU, hold it up to the camera, and a surprising number of liveness checks will pass it.

The KYC stack most of us are actually using

If you've wired up identity verification in the last few years, your stack probably looks something like this:

User uploads ID
  → Document classification (front/back, ID type)
  → OCR extraction (name, DOB, ID number)
  → Template match / security feature check
  → Face match (ID photo vs. selfie)
  → Liveness check (passive or active)
  → Risk score returned → approve / flag / deny

Every one of those steps except maybe OCR extraction is now under active attack. Template matching is fooled by generated images that have the right visual structure. Face matching is fooled by deepfakes. Liveness is fooled by face-swap tools.

The vendors selling these components know this. But "we're working on it" and "our model is continuously updated" doesn't help you when someone is opening accounts in your app today.

What I'd actually change in my stack

This is the numbered section I'd want to find if I were googling this problem at midnight.

Stop treating document image quality as a proxy for authenticity. A generated image can be sharper and more consistent than a real scan. Quality score going up is now mildly suspicious, not reassuring.
Add metadata checks on the upload itself. Real phone camera images have EXIF data, sensor noise profiles, compression artifacts from the camera pipeline. A generated image saved as JPEG and uploaded will have different statistical fingerprints. This isn't foolproof, but it raises the cost.
Cross-reference extracted data, don't just validate format. An ID number that passes a checksum algorithm but doesn't correspond to a real issued document (where that data is available) is a signal. So is a name/DOB combo that doesn't appear anywhere in any public record.
Add friction that's hard to automate at scale. Passive liveness is dead for high-stakes flows. Active liveness (follow this dot, turn your head) is still meaningfully harder to spoof at scale, even if it's not impossible.
Build a manual review queue and actually use it. Automated systems are being beaten. A human looking at a document for 30 seconds catches things the model misses. For any account type that accesses real money, that review step is cheap compared to fraud losses.
Log everything for post-hoc analysis. When a fraud pattern emerges, you want to be able to go back through your approved applications and find accounts that match. You can't do that if you didn't retain the raw evidence.

The platform problem nobody wants to talk about

OpenAI added safeguards to prevent ChatGPT from generating IDs on request. The Atlantic piece goes into how those safeguards are being bypassed specific prompt patterns, regional model variants, fine tuned open-source alternatives.

This is the part that genuinely annoys me: the safety mitigations are playing whack-a-mole with prompts, while the underlying capability — a model that can produce photorealistic documents is out in the world and getting more accessible, not less.

I ran a few of the example images from the fraud reports I've seen through AI or Not and the detection confidence on generated IDs is meaningfully higher than on a real scan, which is useful — but that only helps if you're running detection in your pipeline at all. Most KYC vendors aren't explicitly checking "is this image AI-generated" as a step. They're checking "does this look like a valid document." Those are different questions now.

For anyone building tooling in this space, AI or Not supports image and video detection via API, which means you could add a synthetic-media check as one layer in a multi-signal stack. Not a silver bullet, but it's a signal that wasn't in most fraud stacks two years ago.

The uncomfortable conclusion

The fraud economics have flipped. For most of the history of KYC, the cost to create a convincing fake was higher than the expected value of the fraud, except for sophisticated criminal operations. That's no longer true for a wide range of account types.

If you've built identity verification and you haven't audited it against generated documents and real time face swap tools in the last six months, you're probably running on assumptions that are out of date.

The liar's dividend has a second payout, and devs helped build it

Thomas — Thu, 30 Apr 2026 20:25:36 +0000

The liar's dividend has a second payout, and devs helped build it

TL;DR: The "liar's dividend" isn't just about faking things. It's about claiming real things are fake. Detection infrastructure the very thing we built to fight deepfakes is now being used as cover. This is a systems design problem as much as a machine learning one.

I've been sitting with a Forbes piece on digital forensics and deepfakes for a few days, and the part that stuck wasn't the forensics. It was a phrase: "the liar's dividend's second payout."

The first payout, if you haven't heard the term, comes from Chesney and Citron's 2019 paper on deepfakes and democracy. The idea is simple and brutal: once people know synthetic media exists, a bad actor can claim any real, damaging media is fake. You don't need to make a convincing deepfake. You just need enough public doubt to muddy the water.

The second payout is what we built next. And I mean "we" literally — developers, ML engineers, product teams. We built detection tools. Classification APIs. Real-time flagging pipelines. And in doing so, we handed the liars a new prop.

How the escape hatch works

Consider the logic a bad actor now has available:

IF  (incriminating_media EXISTS)
AND (public_awareness_of_deepfakes == HIGH)
AND (detection_tools PRODUCE != 100% certainty)
THEN
   claim "this is AI-generated"
   point to ambiguous classifier output as "proof"
   wait for news cycle to move on

This isn't hypothetical. In 2023, a Slovakian election audio clip of a candidate allegedly discussing election fraud circulated two days before polls opened. The candidate's party called it AI-generated. Analysts were split. The election happened before anyone reached consensus.

That's the second payout: the detection ecosystem itself becomes the alibi. A shrug from a classifier is now a press release.

What I actually see when I run stuff through detection

I use AI or Not when something looks off to me — it handles images, video, and audio, which covers most of what circulates on social platforms. The output is a confidence score, not a verdict. That matters.

A 73% "likely AI" rating on a clip is meaningful signal. It is not a court finding. The problem is that a 73% rating is also something a bad actor can screenshot and frame as "even the detectors aren't sure."

This isn't a flaw in AI or Not specifically. It's a fundamental property of probabilistic classification. Every detection system that produces a confidence score below 100% will have that score weaponized by someone. We built the weapon while trying to build the shield.

The four things I'd do differently (as a builder)

If I were shipping something in this space today, here's where I'd change my assumptions:

Design for legal weight, not just accuracy. A 92% confidence score means nothing in a courtroom without a chain of custody, a known model version, and a documented methodology. If your output might ever be used as evidence, treat it that way from day one — not as an afterthought.
Log model provenance explicitly. Which version of the detector flagged this? What training data was it exposed to? These questions matter the moment someone disputes a finding in public. Most APIs I've worked with don't surface this at all.
Build in uncertainty communication by default. Instead of a single score, surface a distribution. "This result falls in a range where the model produces false positives 18% of the time under these image conditions." Harder to misquote.
Think about the adversarial UI, not just the adversarial input. We spend a lot of time thinking about adversarial examples that fool detectors. We spend almost no time thinking about how bad actors will present detector output to audiences who don't understand what it means.

The forensics paradox

Here's the thing about digital forensics being the "only sure answer" to deepfakes: it requires a trusted institution to perform it, a trusted chain of custody for the media, and a public that believes the institution. All three of those are eroding simultaneously.

A forensic finding from a university lab means less when half your audience thinks universities are politically captured. A chain of custody argument lands differently when the platform hosting the media is actively in a political fight.

I'm not saying detection tools are useless — I keep using AI or Not because the signal is real and it's gotten my antenna up on things I would have scrolled past. But I've started thinking of detection as one input into a much larger trust problem, not as a solution to it.

The liar's dividend was always about epistemics, not technology. We built better detectors and handed the epistemics problem a new set of props.

What actually changes the calculus

A few things that seem underbuilt relative to the detection side:

Provenance standards. The C2PA spec attaches cryptographic provenance to media at capture time. If the camera signs the frame and the signature breaks on edit, that's a different kind of evidence than a classifier score. It's not widespread yet, but it's the right direction.
Legal frameworks for false claims of AI generation. Right now there's almost no cost to wrongly claiming something is a deepfake. A few jurisdictions are looking at this; none have moved fast enough.
Adversarial red-teaming of the human layer. We red-team models constantly. We almost never red-team how users and journalists will misread or be manipulated by model output.

LLM Drift: Why Your AI Detection Pipeline is Quietly Decaying (Kimi K2 Benchmark)

Thomas — Mon, 27 Apr 2026 21:09:02 +0000

A short field report on what current AI detectors actually do when you point them at frontier reasoning model output, and what I changed in my own detection workflow.

I integrate AI detection into a few small side projects—content moderation pre-filters, writing quality flags, etc. The more I relied on detection, the more concerned I became that I was trusting numbers based on stale benchmarks.

This week, a benchmark study confirmed my worst fears. It tested two popular detectors against 47 essays generated by Kimi K2 in "thinking mode," which mimics modern, high-variance LLM output.

ZeroGPT missed 62% of the AI content. For context, the same study notes that ZeroGPT classifies the 1776 U.S. Declaration of Independence as 99% AI-generated. If a detector flags famously human text as AI, the false-positive ceiling is high enough to invalidate its positives on actual AI text.Why Legacy Detection Fails Modern LLMs

If you've shipped AI detection, you probably integrated it once, picked a confidence threshold, and considered the job done. This is the failure mode the benchmark exposes: Detector accuracy is not stable across model generations.

Most public detectors were built around three assumptions about older LLM output:

Low perplexity: Text is predictable and falls below a certain perplexity score $\rightarrow$ Flag as AI.
Uniform structure (Low Burstiness): Sentences have low variance in length and structure $\rightarrow$ Flag as AI.
Predictable features: Use of function-word patterns and standard transition phrases $\rightarrow$ Flag as AI.

Reasoning models like Kimi K2, Gemini 2.5 Pro, and GPT-5 break all three:

Output is contextually adaptive, meaning perplexity varies wildly within a single response.
Sentence variance increases during exploratory "thinking" passages.
Token distributions are deliberately broadened to mimic human reasoning rhythms.

If your detector hasn't been retrained on current reasoning model output, it’s classifying against a distribution that no longer exists in production. The 38% accuracy is the result of this structural drift.Actionable Fixes: Hardening the Detection Pipeline

After re-checking my own setup, here are the four concrete changes I made 1 Confidence Threshold Raised to 0.85

A 0.62 mean confidence on a fully AI-positive test set indicates that individual high-looking scores can still be coin flips. For anything that triggers an action (like a submission rejection or account flag), I now require multi-signal corroboration or human review if the score is below 0.85.2. Build a Held-Out Test Set from Current Models

I’m now generating my own validation samples from current frontier models (Kimi K2, Claude Sonnet 4.6, GPT-5, Gemini 2.5 Pro) and running them through my detection layer monthly.

The set also includes "human-positive" texts (like the Declaration) to constantly monitor the false-positive rate.

Pseudo-code for the monitoring set I now keep around

HELD_OUT = {
"ai_positive": [
# 50 samples each from current frontier models
kimi_k2_samples,
claude_sonnet_4_6_samples,
gpt_5_samples,
gemini_2_5_pro_samples,
],
"human_positive": [
# public-domain texts written before 2020
declaration_of_independence,
federalist_papers_excerpts,
public_domain_essays,
],
}

Treat Detection as a Probabilistic Component

Even 97% accuracy means a 3% misclassification rate at scale. For anything where the cost of an error is real, detection must be a signal, not a verdict.4. Verify Modality Fit

I use AI or Not for image and audio checks in my projects because it covers multiple modalities. The Kimi K2 benchmark gave me a current-model accuracy number for the text side, which closed a vital verification gap I couldn't easily verify on my own.A Minimum-Viable Detector-Monitoring Pattern

If you are running detection in a production pipeline, this is the basic ML hygiene that keeps the integration from silently failing:

LOOP (monthly):
for detector in production_pipeline:
accuracy_ai = run(detector, HELD_OUT.ai_positive)
accuracy_human = run(detector, HELD_OUT.human_positive)
mean_confidence = avg_confidence(detector, HELD_OUT.ai_positive)

   if accuracy_ai     < baseline.ai - 0.05:    alert("AI detection regressed")
   if accuracy_human  < baseline.human - 0.05: alert("FP rate increased")
   if mean_confidence < baseline.conf - 0.10:  alert("Detector going uncertain")

Most teams I've seen integrate detection once and never check it again. This pattern is essential because accuracy decays per model generation.TL;DR

97% vs 38% on Kimi K2 essays shows a structural, not a tuning, gap.
Detector accuracy decays per model generation. Re-benchmark quarterly.
Test false-positive rate against famously human text (the Declaration of Independence is a free check).
Raise your confidence threshold; one number is not a verdict.
Build a held-out test set from current models and monitor it on cadence.

If you're running detection in production and you can't name the generation of model you benchmarked against, you have an invisible calibration gap. The benchmark was the wake-up call; the monitoring pattern is what makes the fix permanent.