Four Layers to Stop a Report System From Becoming a Weapon
Any platform that lets users flag other users for review creates a dual-use tool: it protects genuine victims, and it can be weaponized by bad actors against people doing nothing wrong. I want to walk through the specific layered defenses Finding Astro uses.
Layer 1: account age gating
Accounts under 30 days old cannot file abuse or conflict reports — reports from new accounts are quarantined for manual review instead of processed automatically. This targets a specific attack pattern: creating a fresh account purely to file a retaliatory report, then discarding it.
Layer 2: velocity limiting
More than 3 reports in a single hour from the same source triggers an automatic, temporary block. This is a straightforward rate limit, but it's calibrated around a real behavioral assumption: a genuine user filing multiple legitimate reports in an hour is rare; a bad actor spamming reports to overwhelm a target is the more common cause of that pattern.
Layer 3: geographic clustering detection
More than 3 reports within 500 meters of each other within a week gets flagged as possible coordinated harassment, rather than treated as three independent, unrelated concerns. This catches a pattern the first two layers miss entirely: multiple different accounts, each individually below the velocity threshold, coordinating against the same target or location — a pattern that's invisible if you only look at each report in isolation.
Layer 4: a credibility score with asymmetric weighting
Every user carries a credibility score starting at 100. A report an NGO later confirms as false costs the reporter 15 points. A report confirmed accurate adds 5. The penalty for a bad-faith report is 3x the reward for a good one.
That asymmetry is deliberate, not an oversight. If the penalty and reward were equal, a bad actor filing 10 false reports and having 1 confirmed true would net out roughly even. Weighting the penalty heavier makes bad-faith reporting a losing strategy over any meaningful volume, without requiring every single report to be manually reviewed before action is possible.
Why four layers instead of one strong one
Each layer catches a different attack pattern, and none of them alone would be sufficient. Account-age gating does nothing against an established account. Velocity limiting does nothing against a slow, patient bad actor filing one report a day. Geographic clustering does nothing against a single-account, single-report false accusation. The credibility score does nothing to prevent the first report from causing damage — it only shapes incentives after the fact.
Layering catches more of the pattern space than any single mechanism would, but it also means four separate systems to reason about, maintain, and tune — a real complexity cost against a real abuse-prevention benefit.
Top comments (0)