DEV Community: Neeru Jaroliya

Building an Instagram AutoDM System at Scale: Webhooks, Event Driven Architecture, and Lessons Learned

Neeru Jaroliya — Sat, 11 Jul 2026 18:31:36 +0000

Instagram creators love engagement. Every comment is an opportunity to start a conversation, share a product, deliver a resource, or convert a viewer into a customer.

The problem is that manually replying to hundreds or thousands of comments doesn't scale.
At Vyral, we set out to build an Instagram AutoDM platform capable of serving thousands of creators while handling bursts of traffic generated by viral Reels. Instead of building a traditional chatbot, we designed an event driven system powered by Instagram webhooks, AWS services, and asynchronous processing.

This article walks through the architecture, the engineering challenges we encountered, and the lessons we learned while designing a system that can process large spikes of comment events reliably.

The Problem

Imagine a creator with 2 million followers.

A Reel starts trending.

Within minutes:

10,000+ comments arrive
Thousands of users comment the same keyword
Instagram sends webhook events continuously
Every eligible comment should trigger a personalized DM

From an engineering perspective, this isn't a chatbot problem.

It's an event processing problem.

The system needs to answer questions like:

Which comments qualify?
Has this comment already been processed?
What happens if Instagram sends the same webhook twice?
What if the user deletes the comment?
What if our service is temporarily unavailable?
How do we avoid overwhelming downstream APIs?

Those questions shaped the architecture far more than the messaging logic itself.

Why We Chose Webhooks Instead of Polling

Polling Instagram every few seconds would have introduced unnecessary latency and API usage for Vyral AutoDM.

Instead, Instagram pushes events whenever something happens.

The flow looks like this:

Instagram
      │
      ▼
Webhook Endpoint
      │
      ▼
Event Validation
      │
      ▼
Event Queue
      │
      ▼
Workers
      │
      ▼
Business Rules
      │
      ▼
Send DM

This architecture offers several benefits:

Low latency
Lower infrastructure cost
Better scalability
Natural decoupling between components

Most importantly, webhook ingestion remains lightweight even when processing thousands of events.

Event Driven Architecture

One principle guided the entire system:

Never perform expensive work inside the webhook handler.

Webhook endpoints should acknowledge requests as quickly as possible.

Instead of processing business logic immediately, the webhook handler performs only a few operations:

Validate the request
Verify the signature
Extract event metadata
Persist the event
Push it into the processing pipeline
Return success immediately

Returning quickly reduces timeout risks and allows processing to happen independently.

Technology Stack

Vyral AutoDMplatform is built using:

Node.js
AWS
DynamoDB
Instagram Graph API
Event driven workers

Node.js works well for webhook processing because of its non blocking I/O model, making it suitable for handling large numbers of concurrent network requests.

DynamoDB provides predictable performance at scale while simplifying horizontal growth.

Filtering Events Early

One important optimization was reducing unnecessary work.

Instagram sends different types of webhook events.

Not every event requires processing.

Before placing anything into the processing pipeline we filter events based on:

Event type
Creator configuration
Comment keyword
Campaign status
Account eligibility

This simple filtering stage significantly reduces downstream load.

Processing fewer events is usually better than processing events faster.

Handling Viral Traffic

Most creators generate relatively small amounts of activity.

The challenge comes from viral creators.

Traffic is highly uneven.

A creator with a viral Reel can generate thousands of comments within minutes.

That means the architecture cannot assume steady traffic.

Instead, it must absorb sudden bursts without affecting other creators.

An asynchronous event pipeline naturally smooths these spikes.

Workers consume events continuously while the queue absorbs temporary surges.

This keeps webhook ingestion responsive even under heavy load.

Designing for Idempotency

Webhook providers commonly retry requests.

Network failures happen.

Timeouts happen.

Temporary service disruptions happen.

Eventually the same event arrives more than once.

Without idempotency, one comment could generate multiple DMs.

To prevent this, every event is assigned a unique processing identity.

Before processing begins, the system checks whether that event has already completed.

If it has, processing stops immediately.

Idempotency turns duplicate deliveries into harmless retries rather than duplicate customer actions.

This became one of the most important reliability mechanisms in the Vyral AutoDM.

Retry Logic

Distributed systems fail.

Temporary API failures are unavoidable.

Some examples include:

Network interruptions
Rate limiting
Temporary downstream failures
Service outages

Rather than treating every failure as permanent, workers retry transient failures using exponential backoff.

The strategy looks like this:

First retry after a short delay
Increase delay with each attempt
Stop after a predefined retry limit
Move permanently failed events for investigation

Separating retryable failures from permanent failures keeps the system healthy while avoiding unnecessary repeated work.

Handling Deleted Comments

One edge case surprised us.

A user can comment.

The webhook arrives.

Before processing completes, the user deletes the comment.

Should the DM still be sent?

That depends on product requirements.

Instead of assuming every event remains valid forever, the processing layer validates the latest state before executing user facing actions.

Engineering often involves handling these small edge cases that rarely appear in architecture diagrams but frequently occur in production.

Monitoring Matters More Than You Think

Large scale event systems are difficult to debug without good observability.

We focused on tracking metrics such as:

Incoming webhook volume
Queue depth
Processing latency
Success rate
Retry rate
Failed deliveries
Duplicate events
API response times

Dashboards quickly reveal whether the system is healthy or whether a particular creator is experiencing unusually high traffic.

Without visibility, scaling becomes guesswork.

Keeping Components Independent

One lesson became increasingly clear as the platform evolved.

Each component should have a single responsibility.

Webhook ingestion should only receive events.

Workers should only process events.

Business logic should remain independent of transport.

API integrations should be isolated.

This separation made the system easier to test, easier to extend, and significantly more resilient.

Lessons Learned

Building event driven systems is often less about raw performance and more about resilience.

A few principles proved invaluable:

Return from webhook handlers immediately.
Process asynchronously.
Design every operation to be idempotent.
Expect duplicate events.
Expect retries.
Filter early.
Monitor everything.
Build for traffic spikes rather than average traffic.

These practices helped us design a platform capable of supporting thousands of creators while remaining responsive during periods of heavy engagement.

Final Thoughts

Many people think Instagram AutoDM is simply about sending messages.

From an engineering perspective, it's much more interesting.

It's a distributed event processing system where reliability, scalability, and correctness matter just as much as messaging.

Whether you're building creator tools, ecommerce automation, or any webhook driven platform, the same architectural principles apply:

Keep ingestion lightweight.
Embrace asynchronous processing.
Design for failure.
Make operations idempotent.
Invest in observability from day one.

Those patterns have been around for years, but they become especially valuable when your application suddenly needs to process thousands of events every minute without missing a single one.

I'd love to hear how you've approached webhook processing or event driven architecture in your own systems. Share your experiences in the comments.

Remotion vs Twick vs CE.SDK: Best React SDKs for AI‑Powered Video Editors

Neeru Jaroliya — Wed, 16 Jul 2025 05:16:34 +0000

Creating an AI-powered video editor in React isn't just about rendering frames—it's about choosing the right foundation. Whether you're building an automated reel generator, a collaborative timeline editor, or a creator-focused tool with export workflows, your SDK will define both speed and scalability. After months of hands-on development and testing, I’ve narrowed the best options down to three: Remotion, Twick, and CreativeEditor SDK (CE.SDK).

In this post, I’ll walk through my real-world experience building with all three, comparing:

Timeline fidelity
UI interactivity
AI integration
Export pipelines
Cost & licensing
Flexibility & developer control

If you're evaluating SDKs for your next-gen AI video editor, this guide will help you choose the right one based on real-world use, not just feature checklists.

1. Timeline Support: Twick wins

Twick is timeline-native. You get frame-accurate React components to manage clips, layers, transitions, and scrub playback.

Remotion is code-first. Excellent for programmatic video generation, but lacks a visual timeline unless you build one yourself.

CE.SDK provides drag-and-drop timelines in a polished UI, but depth is limited compared to a full editor.

Verdict: Twick for precision, Remotion for code templating, CE.SDK for basic editor UI

2. UI Interactivity: CE.SDK takes the crown

CE.SDK offers a full-featured editor experience—layers, snapping, visual controls—out of the box.

Twick is a lean engine with no UI; you must craft custom controls.

Remotion also lacks built-in interactivity; it’s focused on rendering, not UX.

Verdict: CE.SDK for rapid prototyping, Twick for custom UIs, Remotion for code-driven workflows

3. AI Integration: Twick is native

Twick embraces AI: its JSON timeline and modular design make hooking in LLMs or automation flows straightforward.

Remotion can render AI-generated templates, but lacks real-time editing hooks.

CE.SDK is less adaptable to custom AI workflows—its closed UI limits deep integration.

Verdict: Twick for AI-first designs, Remotion for AI-informed rendering, CE.SDK for fixed flows

4. Export Capabilities: different strengths

Remotion excels in cinematic video rendering using Puppeteer + FFmpeg—ideal for high-quality templates, but heavier resources.

Twick supports fast, frame-accurate exports via canvas and FFmpeg, with flexible backend integration.

CE.SDK offers straightforward web exports with APIs, but remains a black box under the hood.

Verdict: Remotion for polished output, Twick for speed and flexibility, CE.SDK for convenience

5. Cost & Licensing

Twick is MIT-licensed, free, and fully open source.

Remotion is open source, with optional commercial licensing for enterprises.

CE.SDK is a paid product—highly polished, but pricing can scale quickly.

Verdict: Twick for budget and OSS enthusiasts, Remotion for hybrid usage, CE.SDK for startups with deep pockets

6. Developer Control & Flexibility

Twick offers full hackability—timeline, playback, layers, and effects are composable in React.

Remotion provides extensive code-based control but lacks UI depth.

CE.SDK delivers rich UI but restricts how far you can customize.

Verdict: Twick for total control, Remotion for code templates, CE.SDK for domain-specific editors

TL;DR – Which One Should You Choose?

Twick
– Frame-accurate timelines
– AI-first integration
– Fully open source

Remotion
– Programmatic, cinematic rendering
– Great for template-driven workflows

CE.SDK
– Beautiful, drag-and-drop interface
– Faster prototyping, less flexibility

Got a project using Remotion, Twick, or CE.SDK?
I'd love to hear how you’re pushing the boundaries of video editing with React!

Skip FFMPEG Pain: Use This React SDK for Timeline-Based Video Editing

Neeru Jaroliya — Tue, 27 May 2025 17:14:24 +0000

The AI-generated video era is exploding. But while ideas are racing ahead, the tools to shape them are still lagging. Enter: Twick.

Creators today don’t just need players. They need full-blown editors that are fast, embeddable, collaborative, and AI-ready — and they need them inside their apps.

So What is Twick?

Twick is a fully open-source React SDK that makes embedding video editing and playback features effortless. It’s not just a wrapper around HTML5 video — it’s a complete timeline-based video editor you can drop into your React apps with support for:

Multi-layer timelines
Captions & rich text overlays
Effects, transitions, and filters
Frame-accurate previewing
Cloud-deployed AI functions via Docker

Whether you're building a creator tool, educational platform, video commerce app, or UGC tool — Twick brings you the power to edit natively in React.

Why Twick Matters (Especially Now)

Generative AI has made it dead simple to generate content. But editing, composing, and collaborating on that content? That’s still manual, disjointed, and full of friction.

Here’s where Twick shines:

Open-source + Developer-Friendly: Unlike commercial SDKs, Twick is easy to fork, extend, and self-host.

AI Function Support: Run your own LLM/GPU-driven effects, transcriptions, or translations on your own infra.

Embeddable + Customizable: Want to style it to match your app? Go ahead. Twick is just React.

Who Should Try Twick?

Startup founders building AI-powered video tools
Educators wanting in-browser video editors
Devs building UGC or collaborative video platforms
Anyone frustrated by how hard it is to embed video editors in modern apps

Try It Out

Check out the GitHub repo:
Twick GitHub Repo

There's a demo setup included — just clone and run locally. You’ll be editing video timelines in minutes.

What’s Next?

Adding audio waveform tracks
Live collaboration
Export-to-cloud workflows
Templates for quick edits

We’re building this in public. Contributions, feedback, and pull requests are welcome.