LLM abuse detection failed at the message level and worked at the account level — and the scam messages were never going to give the operation away. That is the part worth sitting with.
In early August 2026, OpenAI published a threat-intelligence report, Disrupting a Criminal Scam Operation, describing how it banned a coordinated network of ChatGPT accounts run by a Cambodia-based scam operation. The accounts had been used for investment fraud, romance scams, gambling schemes, and law-enforcement impersonation, and the operation is believed to have reached hundreds of targets.
Read that list and the obvious question is: why didn't a content filter stop this on day one? The answer is the most useful lesson in the whole report, and it generalises far beyond OpenAI. LLM abuse detection almost never works at the level of the individual message, because the individual message is not the crime.
TL;DR
- A romance-scam opener and a legitimate sales email are the same text. Per-request content classifiers cannot separate them, and tuning them until they can would break enormous amounts of ordinary use.
- The Cambodia operation used ChatGPT for personas, translation, and outreach — all of which look benign in isolation.
- What made the network legible was its back office: HR announcements, staff translations, and worker records covering debts, salary deductions, fines, visa status and discipline. That paperwork described debt bondage and attempted escapes.
- The initial lead came from WhatsApp, not from OpenAI's own systems. No single platform sees a whole operation; cross-platform correlation is a detection capability, not a courtesy.
- If you build on top of LLM APIs, your abuse signal lives in grouped queries over your request logs, not in a smarter prompt filter.
What is LLM abuse detection?
LLM abuse detection is the practice of identifying and removing accounts that use a model platform for prohibited purposes, using two distinct layers: a per-request content classifier that scores individual inputs and outputs, and an account-level behavioural system that evaluates the entire corpus of an account's activity over time. The first layer is what most people picture. The second layer is what actually catches organised misuse.
The distinction is not academic. It determines what you can detect at all.
A content classifier answers a question about a string: is this text disallowed? An account-level system answers a question about a pattern: does this collection of requests, over these weeks, from these accounts, look like one operation? Fraud is a property of the second question. It is invisible to the first.
Why the scam text is unclassifiable
Here is the uncomfortable symmetry at the centre of this problem.
Consider a message that opens warmly, references something plausible about the recipient, is written in fluent Vietnamese or Portuguese, and ends with an invitation to keep talking. Is that a romance scam or a recruiter's first-touch message? There is no textual feature that separates them. The register is the same. The personalisation is the same. The call to action is the same.
Now consider the promotional copy for an investment platform. Fraudulent or legitimate? Again: identical surface. The lie isn't in the sentence, it's in whether the platform exists.
💡 Key insight: A classifier can only score what is inside the request. Fraud is defined by facts outside the request — whether the persona is real, whether the product exists, what happens to the victim three weeks later. That information is structurally unavailable at inference time.
This is why "just make the safety filter better" is the wrong instinct. The failure isn't a tuning problem; it's an information problem. Push the threshold down until it catches scam openers, and you have also blocked every sales team, every recruiter, and every non-native speaker asking for help writing a friendly email. The false-positive cost is enormous and falls hardest on exactly the users an LLM helps most.
Content filters remain necessary — they catch the genuinely unambiguous stuff, and they raise the floor. But they are a hygiene layer, not a detection system. Treating them as a detection system is the same category error as treating an AI-generated vulnerability report as evidence: plausible-looking output is being mistaken for a verified signal.
The back office is the tell
So what did make this network visible?
Not the scams. The operations work. Per OpenAI's report and Help Net Security's coverage of it, the same accounts were using ChatGPT to:
- draft internal announcements and translate staff communications
- write recruitment ads offering flights, accommodation, visas and work permits
- maintain worker records covering immigration status, working conditions and discipline
- track debts, salary deductions and fines against individual workers
Taken one at a time, every one of those is a normal business task. HR paperwork is not a crime. Multilingual staff comms are not a crime. Recruitment advertising is not a crime.
Taken together, and read against the content of the records, they describe something else entirely. OpenAI reported that the conversations referenced fake overseas recruitment, debt bondage, restricted movement and attempted escapes — indicators consistent with human trafficking and forced criminality inside scam compounds. Real employers do not ask a chatbot to help document an employee's escape attempt.
That is the shape of the signal. Not a single incriminating prompt, but a corpus that only coheres under one explanation.
There's an irony here that anyone running trust and safety should internalise: the criminals were caught by the least criminal thing they did. The scam output was camouflage-perfect. The admin work — the boring, bureaucratic, obviously-necessary-if-you-run-a-compound work — was unique.
Nobody sees the whole operation
The second structural lesson is in a single clause of the report: the investigation began from a lead shared by OpenAI's peers at WhatsApp.
That ordering deserves more attention than it usually gets. The operation touched at least three surfaces — a messaging platform where victims were contacted, an AI platform where content was produced, and financial rails where money moved. Each platform saw a fragment that was ambiguous on its own. WhatsApp could see coordinated messaging patterns without knowing where the text came from. OpenAI could see the text production without seeing who received it.
Correlation across those fragments is what turned two ambiguous signals into one unambiguous case. OpenAI then closed the loop by passing threat signals back to industry partners and authorities.
The implication for platform builders is uncomfortable but clear: your detection ceiling is set by what you can see, and you cannot see enough. Building intake for external reports — and actually staffing the queue that reads them — is a higher-leverage investment than another 2% of classifier recall. This is the same lesson visible in Anthropic's work on detecting distillation attacks, where the abuse pattern was only legible in aggregate query behaviour rather than in any single call.
INTERPOL's reporting on increasingly sophisticated global financial fraud points the same direction: these are organised, industrialised operations that route around any single chokepoint. A per-vendor ban is a speed bump. Shared indicators are the actual mechanism.
Content-level vs account-level detection
| Content-level | Account-level | |
|---|---|---|
| Unit of analysis | One request or response | An account's full corpus over time |
| Catches | Explicitly disallowed output | Coordinated operations, scaled fraud |
| Misses | Anything benign-looking in isolation | One-off misuse by a clean account |
| Latency | Real time, pre-response | Hours to weeks |
| False-positive cost | High — blocks legitimate users mid-task | Low — review happens offline |
| Implementation | Model or heuristic per request | Aggregation over request logs |
| Right role | Hygiene floor | Actual detection |
The column that matters is the last row. If you only build the left column, you have built a system that stops the careless and waves through the organised.
What to do if you build on LLM APIs
If your product resells model access, accepts untrusted user input, or lets users generate outreach at volume, the upstream provider's safety layer is not your abuse system. It is theirs. It sees your API key, not your users.
Concretely, in rough order of value per hour spent.
The queries to write first
- Aggregate by account before you classify anything. Requests per account per day, distinct languages, distinct personas, ratio of generation to editing. Most coordinated activity is a visible outlier on volume and language spread alone.
- Fingerprint template reuse across accounts. Scam operations run playbooks. Near-duplicate prompt skeletons appearing under many accounts is one of the strongest signals available, and it is a similarity join, not a model.
- Join on shared infrastructure. Payment instrument, signup IP range, email domain, device fingerprint. Operations that are careful about content are frequently careless about billing.
The process to build around them
- Log the boring requests too. The Cambodia case turned on administrative usage. If you sample logs and drop everything your classifier scored as benign, you have deleted the evidence.
- Build an abuse-report intake and answer it. Your best signal will arrive from another platform, a bank, or a victim — and only if there is somewhere to send it.
- Write down what you'd do on a confirmed hit. Ban scope, evidence retention, who you notify. Deciding this during an incident guarantees a bad decision.
Note how little of this is AI work. It is mostly GROUP BY over data you already have. Teams reach for a model here because the problem sounds like an AI problem, and then miss the operations that a three-line aggregation would have surfaced. The same instinct shows up in the gap between a demo and a real deployment described in why production-grade AI agents are harder than they look: the hard part is rarely the model.
Common mistakes
Treating the provider's filter as your abuse system. It has no idea which of your users made the request. You do.
Optimising classifier recall as the primary metric. Recall against what? For fraud, the ground-truth label doesn't exist inside the request. You are optimising a proxy that cannot converge.
Sampling logs down to "interesting" traffic. Interesting is defined by the classifier, which is exactly the thing that misses organised abuse. Keep the boring traffic.
Assuming an account ban ends it. It doesn't. The operation re-registers. Enforcement is a cost-imposition and intelligence-gathering exercise, and it should be measured that way — indicators produced and shared, not accounts closed.
Ignoring the human-harm layer. OpenAI's investigation surfaced trafficking indicators. If your platform ever surfaces something like that, "ban and move on" is not a complete response — it's the point where you involve people whose job this actually is. Threat modelling that stops at your product boundary, rather than the wider system your software sits inside, will keep producing this blind spot.
The takeaway
The instinct after reading a report like this is to ask what the filter should have caught. That's the wrong question, and it will keep being the wrong question as models get better at writing exactly the text everyone else writes.
The right question is whether you can see an account rather than a request. The Cambodia network wasn't undone by a clever prompt filter. It was undone by a tip from another platform and a body of mundane paperwork that only made sense one way.
Content filters are the floor. Behavioural correlation is the detection. Cross-platform intelligence is the ceiling. Most teams have built the first, skipped the second, and never considered the third.
Go run one query against your own logs this week: requests grouped by account, sorted descending, with a distinct-language count beside it. What sits at the top of that list is the conversation you should be having.
FAQ
What is LLM abuse detection?
LLM abuse detection is the practice of identifying accounts using a language model platform for prohibited purposes — fraud, harassment, malware, influence operations — and removing their access. It is usually implemented as two separate layers: a per-request content classifier that blocks obviously disallowed outputs, and an account-level behavioural system that looks at the whole corpus of an account's requests over time. The second layer is the one that catches organised operations, because organised operations rarely send individually incriminating requests.
Why can't a content classifier catch scam messages?
Because a romance-scam opener and a legitimate sales outreach message are the same text. Both are warm, personalised, written in the recipient's language, and designed to start a conversation. The thing that makes one of them fraud is the intent behind the account sending it and what happens over the following weeks — neither of which exists inside the single request the classifier is scoring.
How did OpenAI find the Cambodia-based scam operation?
OpenAI has said the investigation began from a lead shared by its peers at WhatsApp, and that it subsequently passed additional threat signals back to industry partners and relevant authorities. That ordering matters: the initial pointer came from a different platform that saw a different slice of the same operation.
What signals actually revealed the operation?
The most distinctive material was not the scam content at all — it was the back-office work. The accounts used ChatGPT to draft internal announcements, translate staff communications, and maintain worker records covering recruitment, immigration status, debts, salary deductions, fines and discipline. Legitimate businesses do generate HR paperwork, but this paperwork described debt bondage, restricted movement and attempted escapes.
Should I build abuse detection into my own LLM product?
If you resell model access, accept untrusted user input, or let users generate outreach content, then yes — the upstream provider's classifier is not your abuse system, it is theirs, and it does not see your account structure. Start with per-account aggregation: request volume, language distribution, template reuse across accounts, and shared payment or infrastructure fingerprints. Grouped SQL over your request logs catches the majority of coordinated activity.
Does banning accounts actually stop these operations?
Not permanently — a banned network re-registers, and the underlying criminal enterprise continues regardless of which AI vendor it uses. What enforcement buys is friction and intelligence: it raises the operating cost, and the investigation produces indicators that can be shared with other platforms and law enforcement. The realistic goal is degradation and disruption over time, not elimination.
Sources
- OpenAI, "Disrupting a Criminal Scam Operation" — the primary threat-intelligence report: the Cambodia-based network, the four scam types, the WhatsApp lead, and the trafficking indicators.
- Help Net Security, "OpenAI reveals how criminals used ChatGPT to run scams" — the itemised breakdown of persona, translation, promotional and administrative usage.
- INTERPOL, "INTERPOL report warns of increasingly sophisticated global financial fraud threat" — context on the scale and organisation of transnational scam operations.
Written for umesh-malik.com — no-fluff technical writing on AI, Web Dev, and Engineering.
Originally published at umesh-malik.com
Keep reading on umesh-malik.com:
Top comments (0)