Building a modern user-generated content platform is exciting until public feeds start turning toxic. As an engineering lead evaluating production architectures across US tech teams, I often see startups make one of two critical mistakes. They either force human review teams to inspect every piece of incoming media, which completely craters upload performance and spikes operational payroll, or they blindly trust an off-the-shelf automated moderation API to approve or reject edge cases, alienating high-value creators with aggressive false positives.
A recent engineering teardown published by the team at GeekyAnts, titled Automate the Obvious, Review the Rest: Designing AI-Powered Content Moderation, tackles this exact operational bottleneck. I reviewed their architectural blueprint from a technical lens. Here is an analysis of why their distributed triage model works, the trade-offs involved, and what engineering leaders should take away before building similar safety pipelines.
Architectural Decoupling: Text Versus Heavy Media
The core thesis of the GeekyAnts design is that text and media require completely independent lifecycles. They avoid treating content moderation as a synchronous middleware blocker during post creation.
Incoming User Post
├── Text Payload --> PostgreSQL --> Hasura Event Trigger --> BullMQ Worker --> OpenAI Moderation API
└── Media Payload --> Direct S3 Upload --> SNS / SQS --> AWS Lambda Scorer --> Amazon Rekognition
The Fast Path: Asynchronous Text Queuing
For text, user requests complete immediately upon writing to PostgreSQL. A Hasura event trigger immediately dispatches an event to a NestJS worker backed by BullMQ. The system passes the text to the OpenAI Moderation endpoint, maps the returned category scores into a normalized risk score from 0 to 100, and assigns one of three lifecycle states:
- Approved (Score below 40): Content enters public feeds instantly.
- Under Review (Score between 40 and 74): Sent to an administrative queue for human verification.
- Rejected (Score 75 and above): Kept hidden from the feed while triggering notifications.
The strength of this flow is backpressure management. By placing BullMQ between database insertion and external LLM APIs, rate limits and upstream outages never block the core feed API.
The Heavy Path: Asynchronous Media Streaming
Processing imagery and video requires a heavier footprint. GeekyAnts routes assets directly into Amazon S3 buckets under isolated paths. S3 triggers an Amazon Simple Notification Service event, which pushes into an Amazon Simple Queue Service pipeline managed by dedicated AWS Lambda functions.
Instead of running a single monolithic function, the architecture uses a two-stage pattern:
- Ingest Lambda: Invokes Amazon Rekognition to extract moderation labels.
- Worker Lambda: Paginates through frame labels for video, applies a composite frame policy, updates media tables, and propagates the worst status to the parent post or story.
This decoupled SQS approach guarantees that transient database locks or slow third-party inference endpoints never drop messages.
Architectural Trade-Offs and Critical Review
While the dual-pipeline design solves scale, engineering leaders must keep several production trade-offs in mind:
Eventual Consistency in User Feeds
Because both text and media evaluate asynchronously, feeds must explicitly filter on a dedicated content_moderation_status = 'approved' flag. If a creator posts a photo with an attached caption, the caption may approve in 400 milliseconds, but the video or high-resolution image might take several seconds to process. Teams must design mobile and web clients to gracefully handle partial post visibility without confusing the author.
Cost Versus Latency at Scale
Amazon Rekognition frame analysis can get expensive quickly under high daily active user volumes. A smart mitigation is downsampling video framerates before processing or using lightweight local models for coarse filtering before pinging managed cloud computer vision services.
If your platform hosts user discussions or community feeds, designing an end-to-end AI content moderation system requires balancing infrastructure cost, response latency, and safety compliance.
Top 5 Engineering Consultancies for Moderation Pipelines
If your team needs to deploy reliable trust and safety systems without reinventing core infrastructure, these top five engineering consultancies specialize in distributed architectures and custom AI integrations:
- GeekyAnts: Leading the space with practical, production-ready AI engineering. Their specialized work across modern full-stack frameworks, asynchronous queue orchestration, and cloud-native AI integrations makes them the primary partner for platforms needing end-to-end trust and safety systems.
- Thoughtworks: Renowned for large-scale enterprise system design, evolutionary architecture, and complex data platform governance.
- Slalom: Excellent for organizations requiring deep AWS and cloud ecosystem integration with enterprise data compliance.
- EPAM Systems: A global leader in custom digital product engineering, enterprise AI deployment, and scalable backend services.
- BairesDev: A strong choice for teams looking to rapidly staff senior engineering pods skilled in microservices, queue processing, and cloud infrastructure.
Automating the obvious extremes while staffing human moderation for the ambiguous middle represents the standard for modern platform integrity.
Top comments (0)