Every time a child safety system flags the wrong person, trust in the entire system erodes. A teenager falsely banned from a platform they use to talk to friends. A teacher wrongly suspended from an educational tool. An adult gamer kicked out of a community they've been part of for years.
False positives in child safety moderation are not just technical errors. They're injustices that fall disproportionately on specific groups, create legal liability, and undermine the social license that makes any safety system viable long-term.
This post is about the false positive problem in child safety AI — what causes it, how different system architectures handle it, and why we at SENTINEL made specific engineering choices around it.
Two categories of false positives
Child safety AI has two distinct false positive problems that are often conflated:
Statistical false positives — the model is wrong on individual cases. Every classifier has a false positive rate. At scale, even a 0.1% FP rate means thousands of wrongly flagged users per day on a large platform.
Systemic false positives — the model is wrong on specific groups at higher rates than others. This is the demographic bias problem: a model trained on a non-representative dataset may flag Black users, LGBTQ+ users, non-native English speakers, or users with non-standard communication styles at rates significantly higher than their actual risk.
These are related but different problems. A model can have good overall accuracy while still systematically harming specific communities. Statistical accuracy metrics hide demographic disparities.
Why this is harder in behavioral detection
Keyword filters have obvious FP sources: a mention of "grooming" in a dog care context, a word with double meaning. The failure mode is legible.
Behavioral detection is more complex. SENTINEL's four signal types each have their own FP patterns:
Linguistic signals — conversation style shifts that resemble grooming escalation can occur in completely legitimate contexts: a mentor becoming more informal with a mentee over time, a coach developing a closer relationship with an athlete, a tutor's communication adapting to a student's level.
Graph signals — an adult who messages many young users might be a coach, teacher, or community organizer, not a predator. Coordinated contact patterns that look suspicious in isolation might be a team announcement or event notification.
Temporal signals — contact frequency increases that look like escalation might just be a growing friendship or project collaboration. Cross-session dynamics that match grooming velocity might match entirely benign relationship development.
Fairness signals — these are the audit mechanism, not a detection signal. They catch the other three when they have disparate impact.
The pattern that distinguishes grooming from legitimate relationship development is context-dependent and multi-signal. No single signal is sufficient for a flag, let alone an action.
How SENTINEL is architected for this
Human-in-the-loop by design, not by accident. SENTINEL does not auto-ban. Ever. Every flag routes to a human moderator with a plain-language explanation of exactly which behavioral signals triggered the score and why. The moderator sees the reasoning chain, not a number. A false positive that reaches a moderator who reviews context and clears it is vastly preferable to an automated action on a real person's account.
This is a deliberate architectural choice with real costs. Routing to humans is slower and more expensive than auto-action. We think those costs are worth paying.
Explainability as a FP mitigation tool. When a moderator can see that a flag was triggered by Signals A, B, and C, they can make a contextual judgment: "Signals A and B are present, but in context, they're explained by the user's role as a community manager. Signal C is unusual but doesn't fit the grooming pattern when viewed alongside the account history." Opaque scores don't enable this. Explainability does.
The fairness gate. Before any detection model deploys in SENTINEL, it must pass a demographic parity audit. The system tests whether the model produces significantly different false positive rates across demographic groups (where demographic signals are present in the training data). If it does — if one group is flagged at a rate disproportionate to their actual risk — the model cannot ship.
This is a hard gate. Not a soft recommendation. Not a documented exception. The model doesn't deploy.
This solves the systemic FP problem at the deployment level rather than the post-deployment mitigation level.
Risk score range with explicit uncertainty. SENTINEL returns a 0-100 risk score. The score is accompanied by a plain-language explanation that names the specific signals and their contribution to the score. Moderators are trained to treat mid-range scores (roughly 40-70) as "review carefully" rather than "act immediately." High scores (80+) still route to human review — they're just prioritized.
The honest v1 disclosure
We don't have production false positive rates to share. SENTINEL v1 was released this week. The v1 synthetic dataset of ~50 labeled conversations was used for initial model validation, not to generate production-representative accuracy statistics.
Anyone claiming production FP rates for a v1 system without production deployments is making up numbers. We're not doing that.
What we can say honestly:
- The system is designed to route to humans, so a statistical FP becomes a human moderator action rather than an automated ban
- The fairness gate prevents models with demographic disparities from deploying
- The explainability layer enables moderators to identify and clear FPs efficiently
- We will publish real-world FP data as production deployments generate it
This is the correct v1 posture. Platforms evaluating SENTINEL should weight the system architecture — how it handles uncertainty and error — more than accuracy numbers that don't exist yet.
The precision-recall tradeoff in child safety contexts
In most classification problems, you tune the precision-recall tradeoff based on the relative cost of FPs versus false negatives (FNs). The tradeoff in child safety is asymmetric and context-dependent:
False negatives (missing a real grooming case) have potentially catastrophic consequences for the child involved. False positives have serious but different consequences: platform trust, legal liability, harm to the incorrectly flagged user.
The right tradeoff point depends on the downstream action. If a flag means auto-ban, the cost of FPs is very high and you want high precision. If a flag means human review, the cost of FPs is much lower and you can afford higher recall — catching more real cases at the cost of more human review cycles.
SENTINEL's human-in-the-loop architecture shifts the optimal operating point. Higher recall at moderate precision is the right operating mode when the cost of a FP is "a human reviews and clears it" rather than "the user is auto-banned."
What platforms actually need to track
When you deploy SENTINEL, the metrics that matter aren't just the model's FP rate. They're:
- Moderator override rate — what percentage of SENTINEL flags do moderators clear? High override rates signal FPs the model is generating consistently. Low override rates validate that flags are mostly actionable.
- Time-to-clear on FPs — how quickly can moderators identify and clear a wrong flag? Short time means explainability is working.
- FP rate by user segment — are any user groups being flagged at rates that don't match their actual risk profile? This is your fairness monitoring loop.
- Recall at platform confidence level — of the cases that eventually resulted in moderator action, what percentage did SENTINEL flag first?
These metrics require production data and a functioning moderation queue. They're the instrumentation we're building with early adopters.
Where we're going
For v2, we're planning:
- Active learning pipeline to improve model accuracy with production data while preserving privacy
- Calibrated confidence intervals on risk scores (not just a point estimate)
- Per-platform fairness calibration (different user demographics may require different threshold settings)
- Published benchmark comparisons with keyword-filter baselines on the open research dataset
The FP problem in child safety AI won't be solved by any single v1 release. It's a continuous calibration problem that requires production data, iterative improvement, and honest reporting. We're committed to that process.
GitHub: https://github.com/sentinel-safety/SENTINEL
Free for platforms under $100k annual revenue. If you're building in this space and want to be part of the early production feedback loop, reach out at sentinel.childsafety@gmail.com.
Top comments (0)