DEV Community: Neeru Jaroliya

Preventing Duplicate Side Effects in Event-Driven Systems

Neeru Jaroliya — Sun, 06 Sep 2026 18:38:59 +0000

Event-driven systems are good at moving work asynchronously, but they introduce an uncomfortable property: you rarely control how many times an event gets delivered.

A webhook can be retried. A queue can redeliver a message when a worker crashes. Two workers can process the same message concurrently. The difficult part is not detecting duplicate events. The difficult part is preventing those duplicates from producing duplicate side effects.
For example, consider an automation system where a user action eventually triggers an external API call:

User Action --> Webhook --> Event Queue --> Worker --> External API

If the worker crashes after the external API accepts the request but before our database records the result, we have an ambiguous state.

Retrying is necessary for availability, but retrying the external call may create the same side effect again. This is where most of the interesting engineering work begins.

Idempotency Has to Be Defined at the Business Level

A common implementation is to deduplicate using the provider's event ID: event_id = 123

That is useful, but it is not always enough. The event ID answers: Have I seen this particular event?
What we actually need to answer is: Have I already performed this particular business action?

Those are different questions. For an automation system, an action might be uniquely identified by:
account_id + automation_id + source_event_id + action_type

For example: account_123:automation_42:comment_981:send_dm

Every retry of the same action must produce the same idempotency key. Generating a new UUID for every attempt defeats the entire purpose:

Attempt 1 → key A
Attempt 2 → key B

The system now sees two operations instead of two attempts at one operation.

Don't Use "Check Then Insert"

Use application logic to decide what should happen, and database constraints to guarantee what cannot happen.

The Hard Failure Case**

The real problem appears after the action has been claimed. Consider:

Create action
Call external API
External API succeeds
Worker crashes
Completion state is never written

On recovery: action = processing
The system doesn't know whether the external operation happened. This creates the classic distributed-systems gap:

Our Database External System

commit
   │
   ├──────────────→ API request
   │                    │
   │                 success
   │
   X

worker dies

There is no normal database transaction that can atomically commit our database and an unrelated external API.

So we need to design around the uncertainty rather than pretending it doesn't exist.

Idempotency Keys at the API Boundary

If the external API supports idempotency keys, use them.

The same logical operation should always carry the same key:

action_id = act_12345

Then:

First attempt → act_12345
Retry → act_12345
Retry again → act_12345

The external service can safely treat these as attempts for the same operation.

When the API doesn't support idempotency, we have to maintain the guarantee ourselves.

That usually means storing a durable action record and making the business key unique.

The important distinction is that the event is not the unit of idempotency; the side effect is.

Queues Don't Solve Duplicate Processing

Queues make event-driven systems much easier to scale, but they don't remove this problem.

A worker might do:

receive message
↓
process
↓
external API succeeds
↓
worker crashes before acknowledgement

The queue has no reliable way to know whether the external operation succeeded. It may deliver the message again.

Therefore, every queue consumer that performs an external side effect should be safe to retry.

Webhook-level deduplication is not enough.

Queue-level deduplication is not enough.

The operation itself needs to be idempotent.

State Machines Are More Useful Than a Boolean

A simple:

processed = true

usually isn't enough.

A production action benefits from explicit state:

PENDING
↓
PROCESSING
↓
COMPLETED

PROCESSING
↓
FAILED
↓
RETRY

This lets us distinguish between:

work that hasn't started
work currently being attempted
successfully completed work
retryable failures
permanently failed work

It also gives recovery processes something durable to reason about.

For example, a job stuck in PROCESSING for longer than the expected execution window can be investigated or recovered.

But recovery must still use the same idempotency mechanism. A stale job should never mean "send the request again without checking."

Retries Need Classification

Not every failure should trigger another attempt.

A timeout or 500 usually represents a transient failure.

A 429 generally means we should slow down and retry later.

An invalid request or invalid permission is different. Repeating the same request will not fix it.

A useful mental model is:

Transient failure → retry
Rate limit → backoff + retry
Permanent failure → stop
Unknown failure → investigate safely

Exponential backoff is useful here, especially when many workers encounter the same downstream problem.

Otherwise a temporary outage can turn into a retry storm.

Observability Is Part of Idempotency

When duplicate side effects occur, the most important question is often: Which path caused the second execution?

That is almost impossible to answer if logs contain only: DM sent

We instead want a traceable chain:

event_id
action_id
automation_id
account_id
attempt
worker_id
external_request_id
status

Then we can reconstruct:
Event → Action → Attempt → API Request

and distinguish: duplicate event from: duplicate action from: retry after unknown API outcome

Those are very different failures.

Exactly Once Is Usually the Wrong Goal

We often hear: "We need exactly-once processing."

In practice, guaranteeing exactly-once execution across a webhook provider, queue, database, worker, and external API is extremely difficult.

A more useful design is:

At-least-once delivery
+
Idempotent business actions
+
Durable state
+
Safe retries

The infrastructure may process an event multiple times. The user-visible result should still happen only once. That is the property we actually care about.

What We Use in Practice

When building event-driven automation, the pattern that works well is:

External Event
↓
Validate + persist
↓
Stable event identity
↓
Queue
↓
Create/claim business action
↓
Unique database constraint
↓
Idempotent external request
↓
Update durable state
↓
Metrics + logs

The key lesson is simple:

Don't try to make the entire system exactly-once. Make every important side effect safe to execute more than once.

That shift in thinking changes how you design the database, queues, workers, retries, and API integrations.

And once your system starts processing events at scale, that small design decision can be the difference between a retry being a recovery mechanism and a retry becoming a customer-facing bug.

We encountered these problems while building event-driven automation at Vyral, where user events can trigger external messaging actions. The same patterns apply to payment processing, notifications, order workflows, and almost any system where an event can create an external side effect.

Building an Instagram AutoDM System at Scale: Webhooks, Event Driven Architecture, and Lessons Learned

Neeru Jaroliya — Sat, 11 Jul 2026 18:31:36 +0000

Instagram creators love engagement. Every comment is an opportunity to start a conversation, share a product, deliver a resource, or convert a viewer into a customer.

The problem is that manually replying to hundreds or thousands of comments doesn't scale.
At Vyral, we set out to build an Instagram AutoDM platform capable of serving thousands of creators while handling bursts of traffic generated by viral Reels. Instead of building a traditional chatbot, we designed an event driven system powered by Instagram webhooks, AWS services, and asynchronous processing.

This article walks through the architecture, the engineering challenges we encountered, and the lessons we learned while designing a system that can process large spikes of comment events reliably.

The Problem

Imagine a creator with 2 million followers.

A Reel starts trending.

Within minutes:

10,000+ comments arrive
Thousands of users comment the same keyword
Instagram sends webhook events continuously
Every eligible comment should trigger a personalized DM

From an engineering perspective, this isn't a chatbot problem.

It's an event processing problem.

The system needs to answer questions like:

Which comments qualify?
Has this comment already been processed?
What happens if Instagram sends the same webhook twice?
What if the user deletes the comment?
What if our service is temporarily unavailable?
How do we avoid overwhelming downstream APIs?

Those questions shaped the architecture far more than the messaging logic itself.

Why We Chose Webhooks Instead of Polling

Polling Instagram every few seconds would have introduced unnecessary latency and API usage for Vyral AutoDM.

Instead, Instagram pushes events whenever something happens.

The flow looks like this:

Instagram
      │
      ▼
Webhook Endpoint
      │
      ▼
Event Validation
      │
      ▼
Event Queue
      │
      ▼
Workers
      │
      ▼
Business Rules
      │
      ▼
Send DM

This architecture offers several benefits:

Low latency
Lower infrastructure cost
Better scalability
Natural decoupling between components

Most importantly, webhook ingestion remains lightweight even when processing thousands of events.

Event Driven Architecture

One principle guided the entire system:

Never perform expensive work inside the webhook handler.

Webhook endpoints should acknowledge requests as quickly as possible.

Instead of processing business logic immediately, the webhook handler performs only a few operations:

Validate the request
Verify the signature
Extract event metadata
Persist the event
Push it into the processing pipeline
Return success immediately

Returning quickly reduces timeout risks and allows processing to happen independently.

Technology Stack

Vyral AutoDMplatform is built using:

Node.js
AWS
DynamoDB
Instagram Graph API
Event driven workers

Node.js works well for webhook processing because of its non blocking I/O model, making it suitable for handling large numbers of concurrent network requests.

DynamoDB provides predictable performance at scale while simplifying horizontal growth.

Filtering Events Early

One important optimization was reducing unnecessary work.

Instagram sends different types of webhook events.

Not every event requires processing.

Before placing anything into the processing pipeline we filter events based on:

Event type
Creator configuration
Comment keyword
Campaign status
Account eligibility

This simple filtering stage significantly reduces downstream load.

Processing fewer events is usually better than processing events faster.

Handling Viral Traffic

Most creators generate relatively small amounts of activity.

The challenge comes from viral creators.

Traffic is highly uneven.

A creator with a viral Reel can generate thousands of comments within minutes.

That means the architecture cannot assume steady traffic.

Instead, it must absorb sudden bursts without affecting other creators.

An asynchronous event pipeline naturally smooths these spikes.

Workers consume events continuously while the queue absorbs temporary surges.

This keeps webhook ingestion responsive even under heavy load.

Designing for Idempotency

Webhook providers commonly retry requests.

Network failures happen.

Timeouts happen.

Temporary service disruptions happen.

Eventually the same event arrives more than once.

Without idempotency, one comment could generate multiple DMs.

To prevent this, every event is assigned a unique processing identity.

Before processing begins, the system checks whether that event has already completed.

If it has, processing stops immediately.

Idempotency turns duplicate deliveries into harmless retries rather than duplicate customer actions.

This became one of the most important reliability mechanisms in the Vyral AutoDM.

Retry Logic

Distributed systems fail.

Temporary API failures are unavoidable.

Some examples include:

Network interruptions
Rate limiting
Temporary downstream failures
Service outages

Rather than treating every failure as permanent, workers retry transient failures using exponential backoff.

The strategy looks like this:

First retry after a short delay
Increase delay with each attempt
Stop after a predefined retry limit
Move permanently failed events for investigation

Separating retryable failures from permanent failures keeps the system healthy while avoiding unnecessary repeated work.

Handling Deleted Comments

One edge case surprised us.

A user can comment.

The webhook arrives.

Before processing completes, the user deletes the comment.

Should the DM still be sent?

That depends on product requirements.

Instead of assuming every event remains valid forever, the processing layer validates the latest state before executing user facing actions.

Engineering often involves handling these small edge cases that rarely appear in architecture diagrams but frequently occur in production.

Monitoring Matters More Than You Think

Large scale event systems are difficult to debug without good observability.

We focused on tracking metrics such as:

Incoming webhook volume
Queue depth
Processing latency
Success rate
Retry rate
Failed deliveries
Duplicate events
API response times

Dashboards quickly reveal whether the system is healthy or whether a particular creator is experiencing unusually high traffic.

Without visibility, scaling becomes guesswork.

Keeping Components Independent

One lesson became increasingly clear as the platform evolved.

Each component should have a single responsibility.

Webhook ingestion should only receive events.

Workers should only process events.

Business logic should remain independent of transport.

API integrations should be isolated.

This separation made the system easier to test, easier to extend, and significantly more resilient.

Lessons Learned

Building event driven systems is often less about raw performance and more about resilience.

A few principles proved invaluable:

Return from webhook handlers immediately.
Process asynchronously.
Design every operation to be idempotent.
Expect duplicate events.
Expect retries.
Filter early.
Monitor everything.
Build for traffic spikes rather than average traffic.

These practices helped us design a platform capable of supporting thousands of creators while remaining responsive during periods of heavy engagement.

Final Thoughts

Many people think Instagram AutoDM is simply about sending messages.

From an engineering perspective, it's much more interesting.

It's a distributed event processing system where reliability, scalability, and correctness matter just as much as messaging.

Whether you're building creator tools, ecommerce automation, or any webhook driven platform, the same architectural principles apply:

Keep ingestion lightweight.
Embrace asynchronous processing.
Design for failure.
Make operations idempotent.
Invest in observability from day one.

Those patterns have been around for years, but they become especially valuable when your application suddenly needs to process thousands of events every minute without missing a single one.

I'd love to hear how you've approached webhook processing or event driven architecture in your own systems. Share your experiences in the comments.

Remotion vs Twick vs CE.SDK: Best React SDKs for AI‑Powered Video Editors

Neeru Jaroliya — Wed, 16 Jul 2025 05:16:34 +0000

Creating an AI-powered video editor in React isn't just about rendering frames—it's about choosing the right foundation. Whether you're building an automated reel generator, a collaborative timeline editor, or a creator-focused tool with export workflows, your SDK will define both speed and scalability. After months of hands-on development and testing, I’ve narrowed the best options down to three: Remotion, Twick, and CreativeEditor SDK (CE.SDK).

In this post, I’ll walk through my real-world experience building with all three, comparing:

Timeline fidelity
UI interactivity
AI integration
Export pipelines
Cost & licensing
Flexibility & developer control

If you're evaluating SDKs for your next-gen AI video editor, this guide will help you choose the right one based on real-world use, not just feature checklists.

1. Timeline Support: Twick wins

Twick is timeline-native. You get frame-accurate React components to manage clips, layers, transitions, and scrub playback.

Remotion is code-first. Excellent for programmatic video generation, but lacks a visual timeline unless you build one yourself.

CE.SDK provides drag-and-drop timelines in a polished UI, but depth is limited compared to a full editor.

Verdict: Twick for precision, Remotion for code templating, CE.SDK for basic editor UI

2. UI Interactivity: CE.SDK takes the crown

CE.SDK offers a full-featured editor experience—layers, snapping, visual controls—out of the box.

Twick is a lean engine with no UI; you must craft custom controls.

Remotion also lacks built-in interactivity; it’s focused on rendering, not UX.

Verdict: CE.SDK for rapid prototyping, Twick for custom UIs, Remotion for code-driven workflows

3. AI Integration: Twick is native

Twick embraces AI: its JSON timeline and modular design make hooking in LLMs or automation flows straightforward.

Remotion can render AI-generated templates, but lacks real-time editing hooks.

CE.SDK is less adaptable to custom AI workflows—its closed UI limits deep integration.

Verdict: Twick for AI-first designs, Remotion for AI-informed rendering, CE.SDK for fixed flows

4. Export Capabilities: different strengths

Remotion excels in cinematic video rendering using Puppeteer + FFmpeg—ideal for high-quality templates, but heavier resources.

Twick supports fast, frame-accurate exports via canvas and FFmpeg, with flexible backend integration.

CE.SDK offers straightforward web exports with APIs, but remains a black box under the hood.

Verdict: Remotion for polished output, Twick for speed and flexibility, CE.SDK for convenience

5. Cost & Licensing

Twick is MIT-licensed, free, and fully open source.

Remotion is open source, with optional commercial licensing for enterprises.

CE.SDK is a paid product—highly polished, but pricing can scale quickly.

Verdict: Twick for budget and OSS enthusiasts, Remotion for hybrid usage, CE.SDK for startups with deep pockets

6. Developer Control & Flexibility

Twick offers full hackability—timeline, playback, layers, and effects are composable in React.

Remotion provides extensive code-based control but lacks UI depth.

CE.SDK delivers rich UI but restricts how far you can customize.

Verdict: Twick for total control, Remotion for code templates, CE.SDK for domain-specific editors

TL;DR – Which One Should You Choose?

Twick
– Frame-accurate timelines
– AI-first integration
– Fully open source

Remotion
– Programmatic, cinematic rendering
– Great for template-driven workflows

CE.SDK
– Beautiful, drag-and-drop interface
– Faster prototyping, less flexibility

Got a project using Remotion, Twick, or CE.SDK?
I'd love to hear how you’re pushing the boundaries of video editing with React!

Skip FFMPEG Pain: Use This React SDK for Timeline-Based Video Editing

Neeru Jaroliya — Tue, 27 May 2025 17:14:24 +0000

The AI-generated video era is exploding. But while ideas are racing ahead, the tools to shape them are still lagging. Enter: Twick.

Creators today don’t just need players. They need full-blown editors that are fast, embeddable, collaborative, and AI-ready — and they need them inside their apps.

So What is Twick?

Twick is a fully open-source React SDK that makes embedding video editing and playback features effortless. It’s not just a wrapper around HTML5 video — it’s a complete timeline-based video editor you can drop into your React apps with support for:

Multi-layer timelines
Captions & rich text overlays
Effects, transitions, and filters
Frame-accurate previewing
Cloud-deployed AI functions via Docker

Whether you're building a creator tool, educational platform, video commerce app, or UGC tool — Twick brings you the power to edit natively in React.

Why Twick Matters (Especially Now)

Generative AI has made it dead simple to generate content. But editing, composing, and collaborating on that content? That’s still manual, disjointed, and full of friction.

Here’s where Twick shines:

Open-source + Developer-Friendly: Unlike commercial SDKs, Twick is easy to fork, extend, and self-host.

AI Function Support: Run your own LLM/GPU-driven effects, transcriptions, or translations on your own infra.

Embeddable + Customizable: Want to style it to match your app? Go ahead. Twick is just React.

Who Should Try Twick?

Startup founders building AI-powered video tools
Educators wanting in-browser video editors
Devs building UGC or collaborative video platforms
Anyone frustrated by how hard it is to embed video editors in modern apps

Try It Out

Check out the GitHub repo:
Twick GitHub Repo

There's a demo setup included — just clone and run locally. You’ll be editing video timelines in minutes.

What’s Next?

Adding audio waveform tracks
Live collaboration
Export-to-cloud workflows
Templates for quick edits

We’re building this in public. Contributions, feedback, and pull requests are welcome.