DEV Community: Dev Agrawal

Async Transformations in Reactivity

Dev Agrawal — Thu, 15 May 2025 05:28:49 +0000

Bringing structure to async events — lifecycles, retries, and transactions without losing composability

This is part of the Transformations in Reactivity series.

In Part 1, we focused on composability. In Part 2, we introduced structured scheduling.

In this chapter, we tackle async coordination: giving events the same structured lifecycle semantics Solid’s signals gained with createAsync — without losing the essence of push-based flows.

1. The Async Gap: Signals vs Events

Signals model current state. They’re synchronously derived and perfectly suited for reflecting application state in the UI. But when you introduce async data (like fetches), signals had to evolve mechanisms to handle suspension, pending states, and errors structurally.

Solid 2.0’s createAsync solved this with a structural approach:

Signals throw during unresolved state (Suspense).
Errors propagate up naturally.
Async dependencies remain composable.

Events, however, are inherently push-based and asynchronous. They represent mutations, interactions, and side effects. Yet, they lack the structured semantics signals now enjoy:

No built-in way to track pending state.
No structured error handling.
No notion of scoped retries or isolation.

If signals got structured async reactivity, why not events?

2. What Structured Async for Events Means

In stream-centric systems like RxJS, streams offer next(), error(), and complete(). Useful, but flat. They don’t express:

Pending/loading state.
Scoped retries.
Predictable execution order across async flows.

We need a structural lifecycle for events, much like signals have.

The key:

wait(): Emitted when async work starts.
next(): Emitted for resolved values (once or many times).
error(): Propagates failures structurally.
No complete(): Events are tied to ownership, not upstream closing.

This turns fire-and-forget events into structured flows.

3. `createAsyncEvent`: Declarative Async in the Event Graph

createAsyncEvent models async operations structurally.

Single result:

const onValidated = createAsyncEvent(onSubmit, async (form) => {
  await validateForm(form)
  return form.id
})

Emits wait() at start.
Emits next(result) when resolved.
Emits error(err) on failure.

Multiple results with AsyncIterable:

const onProgress = createAsyncEvent(onStart, async (input) => {
  async function* progress() {
    for (let i = 0; i <= 100; i += 10) {
      await delay(100)
      yield i
    }
  }
  return progress()
})

Emits multiple next() values.
No imperative observer API. Just return values.

4. What About switchMap, concatMap, mergeMap, exhaustMap?

In RxJS:

switchMap cancels previous.
concatMap queues.
mergeMap runs all concurrently.
exhaustMap ignores overlaps.

For events:

We care about mutations, not derived state.
Every mutation should flow through.

Thus:

merge behavior is default.
All handlers run concurrently.

concat, exhaust, and switch can be built in userland:

Queueing.
Busy flags.
Disposing previous subscriptions.

Signals handle derived state with switchMap-like patterns.

Events handle side effects — and merge fits naturally.

5. Observing Async Events Structurally

`createSyncEvent`: Promises from Events

Transforms Handler<T> into Handler<Promise<T>>:

const onSubmitPromise = createSyncEvent(onValidated)

createListener(onSubmitPromise, async (promise) => {
  const result = await promise
  console.log("Validated ID:", result)
})

Every emission creates a new promise reflecting wait, next, error.

`createSyncStream`: AsyncIterables from Events

For multiple emissions:

const onProgressStream = createSyncStream(onProgress)

createListener(onProgressStream, async (stream) => {
  for await (const progress of stream) {
    console.log("Progress:", progress, "%")
  }
})

Each trigger yields an AsyncIterable.
Errors throw during iteration.
Declarative consumption of event-driven streams.

6. Scoped Retry with Event Context

Signals retry by re-accessing. Events propagate downstream.

To enable retries:

Async handlers register retry() into the event context.
Downstream listeners can access:

  createListener(onValidated, {
    error: (err) => getEventContext()?.retry()
  })

This allows isolated, structural retries without global event re-emission.

Event contexts are cloned and propagated across async boundaries, maintaining isolation.

7. Transactions: Deferred Work with `runTransaction`

runTransaction wraps an event flow so that:

wait() and next() are deferred until the transaction completes.
Errors are surfaced via the returned promise.
Side effects batch after completion.

const result = await runTransaction(() => {
  emitSubmit()
  emitOtherAction()
})

Unlike global transitions in React/Solid, each transaction here is fully scoped and supports concurrency.

For full UI transactional updates:

Use runTransaction for event flows.
Use startTransition for signal state forks.
Compose both for predictable isolation.

8. Events as First-Class Async Structures

By giving events:

Structured lifecycles (wait, next, error)
Scoped retries
Transactional isolation

We reach parity with signals in composability and predictability — while keeping the push-based essence.

The API remains minimal:

Return a Promise or AsyncIterable.
Observe flows declaratively.
No boilerplate.

This lets events fully participate in fine-grained, async-reactive UI architectures.

Granular propagation.

Not every listener needs every event.

Next chapter explores Topics — bringing fine-grained updates to event graphs, like stores and projections did for signals.

Scheduling Transformations in Reactivity

Dev Agrawal — Sat, 03 May 2025 19:32:48 +0000

From pure event graphs to safely mutating state

1. Introduction: The Real-World Punch

There’s a saying:

“Every plan is perfect until you get punched in the face.”

In software, that punch often comes the moment your pure logic has to interact with the real world.

In our last post, we built a clean, composable model for events — one based on transformations, ownership, and simplicity. But so far, everything has been pure. We transform values, we log them, we react — but we haven’t changed anything.

That changes now.

Let’s take this innocent-looking example:

const [onEvent, emitEvent] = createEvent()
onEvent(() => console.log(state()))
onEvent(() => setState("world"))
onEvent(() => console.log(state()))

What should this log?

"hello" and then "world"?
"world" twice?
Or "hello" twice?

The answer depends entirely on execution order. And without clear rules, even small graphs like this become hard to reason about.

Push-based systems often run into this:

If you mutate state mid-propagation, every downstream listener sees a different world.

So now we ask: how do we bring consistency and predictability to a reactive event system — even when it mutates?

The answer lies in something we borrowed from signals: phased execution.

2. Mutation Hazards in Push-Based Systems

Push-based reactivity doesn’t have phases. It has flow.

As soon as you emit a value, every subscriber runs immediately — in the order they were defined. That’s fine when you’re just transforming data. But once you introduce mutations, the cracks start to show.

Take this setup:

onEvent(() => console.log(state()))
onEvent(() => setState("world"))
onEvent(() => console.log(state()))

Depending on how these handlers were registered, and in what order they run, the logs might print:

"hello", then "world" — which seems logical
or worse: "world" twice — because the state was changed mid-propagation

There’s no real “before and after” in this model. Everything is firing during the same tick, and nothing is scheduled — so side effects collide with each other, and reads become unpredictable.

Push-based graphs become fragile when reads and writes happen in the same pass.

If we want predictable behavior — if we want to reason about event graphs like pure functions — we need to separate mutation from computation.

Just like signals did.

3. Lessons from Signals — Phased Execution

Signals hit the same wall events do: mixing computation and side effects creates unpredictable behavior.

Solid’s answer was phased execution.

In Solid 2.0, reactivity happens in layers, not all at once:

Pure derivations (memos, derived signals)
- These run first and synchronously during a synthetic "clock cycle".
- No side effects. Just computation.
Side effects (createEffect)
- These are deferred until after the graph stabilizes.
- Used to read from the DOM, notify external systems, or manage local resources.

“All changes to the DOM or outside world happen after all pure computations are complete.”

Even though some reactive work runs immediately, Solid 2.0 internally queues it and flushes the graph once it’s fully consistent. This guarantees that the entire reactive graph stabilizes in a single clock cycle, and only then do effects run.

It’s a small shift, but a powerful one:

Separate pure logic from side effects, and you get predictability.

That’s what we want for events, too.

4. Bringing Phases to Events

If signals needed phased execution to stay predictable, events need the same — maybe even more.

Here’s the structure we adopt for event execution:

Phase 1: Pure event graph

All event handlers (transformers) run immediately.
No state changes, no side effects — just pure mapping.

Phase 2: Mutations

Instead of calling setState() directly inside handlers, you schedule it.
You use createMutation(handler, effectFn) — the handler runs in phase 1, and effectFn is deferred to phase 2.

Phase 3+: Signals and UI

After mutations, the rest of Solid’s reactive system proceeds as usual.
Signals notify, DOM updates happen, effects fire — all downstream of a clean update.

This lets you build composable flows like this:

const onNext = onClick(() => count() + 1)
createMutation(onNext, setCount)

You now know:

onClick runs
onNext transforms the value
setCount(...) is queued, not run immediately
The UI updates after all pure logic completes

5. Blocking Unsafe Patterns

For phased execution to work, we need clear boundaries — but we don’t want to over-restrict expressive patterns either.

State mutations (setState) must never happen in phase 1.

If setState() happens during the pure event phase, downstream handlers may see a changed world — and that breaks everything.

To enforce this, any setState() inside a pure event handler should either:

Throw a dev-time error, or
Be automatically scheduled as a mutation

onClick(() => setState(count() + 1))  // ❌ not allowed directly

Instead, use:

createMutation(onClick, () => setCount(count() + 1))

What about emit()? Emitting from inside a handler is pure — it just expands the graph.

Still, it should be used thoughtfully. Overusing nested emits can make graphs harder to trace.

Mutation breaks the phase model.

Emission just grows the graph.

6. `createMutation` in Practice

This is your gateway to phase 2 — safe, scheduled mutation.

const onNext = onClick(() => count() + 1)
createMutation(onNext, setCount)

Want to perform multiple updates?

createMutation(onSubmit, () => {
  setLoading(true)
  setStatus("submitted")
})

You can also inject logging, analytics, or cleanups — all scoped, all deferred.

7. Sync Parity with Signals

Solid 2.0 uses auto-batching — effects run only after the graph stabilizes.

To match this behavior, we flush all queued mutations automatically after every emit().

emitIncrement()  // queues the mutation
// mutation is flushed right after emit completes

Need to group multiple emits?

batchEvents(() => {
  emitAdd()
  emitSubtract()
})

The queue flushes after the batch — keeping mutation and UI updates consistent with signal behavior.

Mechanism	Signals (2.0)	Events
Default	Auto-batched	Auto-flushed
Manual control	`flushSync()`	`batchEvents()`

8. Final Example — Predictable Flow

const [onIncrement, emitIncrement] = createEvent()
const [count, setCount] = createSignal(0)

const onNext = onIncrement(() => count() + 1)
createMutation(onNext, setCount)

emitIncrement() fires.
onIncrement computes count() + 1.
setCount(...) is queued.
The mutation is flushed after the event cycle.

Want logging?

const onLog = onNext((next) => {
  console.log("Next count:", next)
  return next
})

createMutation(onLog, setCount)

Want to batch updates?

batchEvents(() => {
  emitSave()
  emitClear()
})

Pure. Predictable. Phased.

Next time: async, errors, retries, and full integration with suspense.

Transformations in Reactivity

Dev Agrawal — Sat, 03 May 2025 18:28:00 +0000

What happens when we stop treating events like second-class signals?

1. Introduction

Signals have gone through a quiet renaissance.

Once a niche concept, they’ve become the backbone of fine-grained reactivity across modern frameworks — from SolidJS to Svelte 5, Vue’s Composition API, and beyond. With laziness, ownership, batching, and async support, signals are no longer just state — they’re a reactive architecture.

Meanwhile, events have stayed… familiar.

Push-based systems like RxJS still dominate, with powerful abstractions — and plenty of baggage. They’re expressive, but complex. Powerful, but leaky.

This series explores a different direction:

What if we evolved events the way signals evolved?

What if push and pull reactivity didn’t require different paradigms, just different shapes?

Let’s start small — by rebuilding events from first principles.

2. From Observables to Event Graphs

RxJS gave us a model: streams of values transformed by declarative operators.

It works. But it comes with weight.

Even trivial flows — say, responding to a button click and transforming a value — require a pipeline:

observable.pipe(
  map(x => x + 1),
  filter(x => x < 10),
  tap(console.log)
)

Then comes teardown logic, lifecycle handling, and the occasional “why didn’t this emit?” debugging session.

The idea behind this model isn’t to replace RxJS — it’s to explore what happens when you stop thinking in streams and start thinking in reactive functions.

Instead of operators and pipelines, what if you just… mapped?

3. Solid as Inspiration

SolidJS didn’t invent signals — S.js did.

But Solid made them practical, scalable, and fast. It took a pull-based reactive core and layered on ownership, lifecycles, and structured scheduling — turning signals into a system.

In Solid, signals live inside a graph that disposes itself. When a scope ends — like a component unmounting — all reactive logic inside it is automatically cleaned up. No unsubscribe(). No memory leaks. No manual lifecycle wiring.

That’s the leap: reactivity that lives and dies with its context.

This model works for signals. What happens if we apply the same ideas to events?

4. A New Default: Event Handlers as Transformers

Traditionally, push-based systems treat streams like pipelines. You create a source, then pass it through a chain of operators to get what you want.

But what if you didn’t need a chain?

What if the default behavior of an event was simply: transform a value and hand it off?

const onChange = onClick(() => "clicked")

Just like that, you've turned one event into another. No .pipe(), no .map(), no imported operators — just function composition.

This model simplifies everything. Here’s how you might build a derived counter from a basic event:

const [onIncrement, emitIncrement] = createEvent()
const [count, setCount] = createSignal(0)

const onNextCount = onIncrement(() => count() + 1)

onNextCount((next) => {
  console.log("Next count will be:", next)
})

If you’ve used signals before, this should feel instantly familiar. It’s not a stream — it’s a value transformer, scoped to a reactive lifecycle.

You can compose multiple steps, too:

const onEven = onNextCount((n) => {
  if (n % 2 !== 0) haltEvent()
  return n
})

onEven((value) => {
  console.log("Even count:", value)
})

Each handler returns a new transformed event. That’s it.

No operator chain. No nested observables. No cleanup logic.

And yes, you can still pass values through:

onClick((e) => {
  console.log("Clicked at", e.clientX, e.clientY)
})

5. Filtering with `haltEvent()`

So if we’ve dropped filter, how do we stop an event from propagating?

With one helper: haltEvent().

onInput((value) => {
  if (!isValid(value)) haltEvent()
  return transform(value)
})

Filtering becomes a conditional — not an operator. The behavior is local, readable, and imperative.

And that’s intentional. The more we reduce the operator surface, the more intuitive the graph becomes.

This approach isn’t about losing power — it’s about making the simple path the default.

6. Memory & Lifecycle

Push-based systems historically struggle with memory management.

You subscribe, and then… you unsubscribe. Or you forget. Or you build a helper. Or you use takeUntil. Or you write a useObservable() hook that does the teardown for you.

In RxJS:

const subscription = observable.subscribe(fn)
onDestroy(() => subscription.unsubscribe())

Or maybe:

const stream = interval(1000)
  .pipe(takeUntil(destroy$))

Even internally, you return a cleanup function from your subscription logic:

const interval = setInterval(() => observer.next(), 1000)
return () => clearInterval(interval)

In an ownership model, that all disappears:

const interval = setInterval(() => observer.next(), 1000)
onCleanup(() => clearInterval(interval))

No teardown function needed.

No unsubscribe() anywhere.

The event lives as long as the scope that created it.

This is reactivity with lifecycle built in.

7. Rethinking Completion

Observables give you .complete() as a way to say, “I’m done.” It’s a way for the stream to manage its own lifecycle.

But in UI systems, streams rarely complete.

A click handler doesn’t complete. A keypress doesn’t complete. An internal state change doesn’t complete.

Instead of letting streams declare themselves done, we flip the model:

streams live as long as they’re used. And they stop when their owner stops.

const [onClick, emitClick] = createEvent()
onClick(() => console.log("clicked"))

No manual teardown. No lifecycle ceremony.

When the scope disposes, so does the handler.

This also means events don’t need to be recreated to be reused.

No reinitialization. No cascading unsubscribes. Just permanent event nodes that can be reconnected at will.

8. Higher-Order Events (Sync Only)

Yes — you can build switchMap. But you don’t need to.

An event can just emit another event.

const higherEvent: Handler<Handler<string>> = ...

higherEvent(innerEvent => {
  innerEvent(value => {
    console.log("Inner event value:", value)
  })
})

Each time higherEvent emits, it provides a new innerEvent. You subscribe to it. You decide what to do with the old one.

Want a helper?

const createDynamicEvent = (source: Handler<Handler<T>>): Handler<T> => ...

Now you’ve got a flattened stream of inner values. And you didn’t write a single operator.

This works great for synchronous graphs.

Async adds more complexity — and we’ll get there. Just not today.

9. Final Transformation Example

Here’s the full pattern — from event to transformation to subscriber:

const [onIncrement, emitIncrement] = createEvent()
const onChange = onIncrement(() => count() + 1)

onChange((value) => {
  console.log("Next count will be:", value)
})

You’ve declared an event, transformed it, and used it — all in a few lines.

No operators.
No unsubscribe.
No cleanup code.
Just pure transformation.

This is the baseline:

Events as composable, scoped, reactive functions.

And it’s only the beginning.

Next Up: Scheduling

What happens when events start mutating state?

What happens when the same event graph reads and writes at the same time?

That’s where scheduling comes in.

And in the next post, we’ll show how phase-based execution solves the hardest problem in event reactivity: ordering.

Dynamic Typing is Obsolete

Dev Agrawal — Wed, 13 Dec 2023 06:59:33 +0000

Dynamically typed languages are better than statically typed languages

This statement used to be true for various reasons. It’s not anymore. Let’s dig into why.

Dynamically typed languages and frameworks like Ruby on Rails and LAMP stack blew up in popularity for various reasons. Let’s rewind the clock back a couple decades or so and look at the context.

Software development was dominated by statically typed languages like C++, Java, and C#.

These languages and frameworks had a big learning curve
Code was very verbose because of the ceremony of type definitions
Required expensive specialized tooling that shipped in physical CDs
IDEs were very slow and buggy, especially on older hardware
Feedback loop was slow because of long compilation times
Ecosystems were primarily stewarded by corporations and proprietary software
Frameworks were primarily intended for desktop apps, web apps were an afterthought

Ruby on Rails and LAMP stack were positioned perfectly to sweep and dominate the ecosystem at this point.

Languages like Ruby and PHP had a much smaller learning curve
Dynamic nature and syntactic sugar made it quick and fun to program
No specialized tooling was required, fire up notepad and start building your app
No slow and buggy IDEs, you could literally REPL your code into the terminal
No compile step means faster feedback loops, make a change, hit save, refresh the browser
Ecosystems were extremely open source and community driven
The sole purpose was enabling anyone to make web apps, instead of adding web capabilities to an existing stack

I obviously haven’t been around that long, but those who have remember static typing not as it is today, but as it was back then. It was an indicator of ceremony, slow feedback loops, and proprietary dinosaurs.

Static typing has made a major comeback over the last decade, and almost everything that turned people away from them and towards the comfort of PHP and Ruby has been UNO reversed.

Modern static languages like Typescript have ridiculously low learning curves
Type inference takes out 95% of the ceremony and terse syntax
The most powerful tools also happen to be open source and very well documented
Advancements in hardware and optimization techniques have made IDEs extremely snappy
Compilations are ridiculously quick, especially if the compiler knows the data types beforehand
Even the big bad Microsoft has completely turned around and committed to open source
Typescript is not the only static language with an ecosystem primarily geared towards web development

The factors that blew up Rails and LAMP in popularity are now the same factors that favor ecosystems that offer complete type safety. Dynamic typing has become, to put it strongly, obsolete. They are artifacts of the past.

Yes you can use dynamically typed languages to build real systems for scale. But you will pay the cost of maintenance, performance, and scalability, and eventually will have to rebuild using infrastructure that offers much better guarantees.

Dynamically typed languages are used not because they are dynamically typed, but because they have specific technical benefits (e.g. Elixir for building distributed systems) or ecosystem benefits (e.g. Python for data science).

Starting off with type-safe languages will give you faster feedback loops, more helpful IDE tools, more freedom to make changes without breaking stuff, and better collaboration through internal contracts and documentation at a very low cost.

This article is essentially a summary of this talk "Why Static Typing Came Back” by Richard Feldman. Go watch the full talk for an even bigger history lesson and so many more arguments that I couldn’t even mention here.

Thoughts on Server Components + Websockets

Dev Agrawal — Wed, 13 Dec 2023 06:15:58 +0000

Thoughts on Server Components + Websockets

I've been thinking about this for way too long now, and right now I am convinced that this is potentially an amazing way to build realtime applications. Would love to be proven wrong.
Currently I believe there are 2 ways to achieve it.

Just reload the page

This is what I have working so far. The component listens on a live topic, and when it receives a message it just reloads the page, i.e., load all the server data again.

This is the most bare and simple integration, but also very powerful, because you get security by default. It's like the Stripe integration - server creates a secure session, then tells the client how to connect to it. If the server component is not rendered, the client has no way to connect to a topic and receive unauthorized messages.

This is also not the most efficient - websockets are being used purely as a notification, not to transfer data. Every realtime update is causing a server fetch.

But a huge class of applications can greatly benefit from a very simple drop in "make this page realtime" solution.

RSC over WebSockets

Getting tricker now. This is essentially an entirely different server that is also running Server Components, but with state! This is VERY different from how we use Server Components today which are designed to run in stateless environments.

The DX will be very similar - a server component will return JSX that will be sent to the client as a websocket message where its handled by React. Except there's no navigation - the client just establishes a connection and shows a Suspense fallback, and the server decides when it needs to render the UI.

So how do these components get triggered? Well you can just listen to websocket messages, or have another http api on the side, or subscribe to rabbitmq, or connect to a cloud service.

These server components will require an entirely new concept of Routing defined on top of WebSockets instead of HTTP, and the layer of infrastructure to provide that (like Next.js).

They're also have to be detached from Next.js - the websocket server component has to be mounted on the Client instead of on the Server, because Next.js's routing cannot interfere with it.
(Obviously SSR is a non-issue with websockets)

I don't think there's any solutions to do this today, but if someone is interested in building one let me know. This is kindof a big problem because you need to build a metaframework but for websockets, but you also don't have to deal with SSR so it's slightly easier I guess.

What about Serverless?

Great question.
My implementation of #1 runs on AWS and uses a combination of API Gateway and DynamoDB. "state" on the server doesn't have to be in memory, it can be a database, especially if it's a low latency KV, or even better, a durable object. And the websocket part can use any underlying infrastructure.
It's the same for #2 - stateful RSCs can easily exist on serverless runtimes.

But Why RSCs with WebSockets

The DX of Stateful RSCs is basically the exact thing you imagined when you first heard "server components". It's components that can useState and setState on the server.

This is a very powerful way to build interactive apps, because similar to how RSCs encapsulate the entire http layer, state RSCs can encapsulate the entire websocket layer and give you complete control over how much work is done on the server vs client, and directly connect your UI to server events. Basically if the underlying implementation completely changed from websockets to something else we wouldn't even notice.

Let me know if I'm crazy or is this real.

Clerk Webhooks: Data Sync with Convex

Dev Agrawal — Wed, 13 Dec 2023 06:08:35 +0000

Composing an application out of multiple data sources can be challenging, and while Clerk’s APIs work great for most use cases, webhooks offer the next level of integration that enables you to take full advantage of your existing stack. This post will cover how to synchronize user data from Clerk into your own backend using Webhooks.

Data Sync with Convex

Convex is a real time Backend-as-a-Service Provider that makes building highly dynamic and interactive experiences with React easier than ever.

Clerk offers a first-class integration with Convex. The queries and mutations living in your Convex instance can be authenticated with a token that was created by Clerk. This integration is covered in more detail in the documentation.

Convex is also a great target for synchronizing user data from Clerk using webhooks, since you can take full advantage of Convex’s realtime capabilities for user data.

If you are not sure what webhooks are and how to use webhooks with Clerk, it’s recommended to read this getting started post first.

To demonstrate data synchronization with Convex, we can use Convex’s Clerk starter repo. You can clone and run this repo locally by following the instructions in the repo.

This post will:

Explain how the demo uses data sync with reactive queries together
Show code examples of the webhooks handler implementation in the demo

The Demo

The demo application is a chat room where users can send and receive messages. To access the room, a user needs to sign up for an account using the Clerk integration. Once signed in, the user is redirected to a chat room, which establishes a reactive query to the Convex database. This reactive query looks for the user in the database, and returns a “No Clerk User” if the user is not found, and a “Logged In” if the user is found.

Initially after signing up, the user will not be found in the database. Since the signup process happened with Clerk, Convex has no record of this user.

This is where webhooks come into play. After the user signs up with Clerk, Clerk will send an http request to a pre-defined URL with data about the sign up. This URL will point to a Convex HTTP Action, which will parse the data and create a record in the Convex database for the user. Now the client can query for this user again and receive the expected response.

But since Convex queries are reactive, the client doesn’t need to retry the query, instead it will automatically receive an update from the server with the user data. This will trigger React to re-render the UI, providing the user access to the chat room.

The Code

To look at the webhook implementation, we can open the convex/http.ts file in this repo.

// convex/http.ts

// define the webhook handler
const handleClerkWebhook = httpAction(async (ctx, request) => {
  const event = await validateRequest(request);
  if (!event) {
    return new Response("Error occured", { status: 400 });
  }
  switch (event.type) { ... }
  return new Response(null, { status: 200 });
});

// define the http router
const http = httpRouter();

// define the webhook route
http.route({
  path: "/clerk-users-webhook",
  method: "POST",
  handler: handleClerkWebhook,
});

The handleClerkWebhook function is responsible for… you guessed it… handling the Clerk webhook. It’s a function decorated with httpAction and receives the HTTP request and a context object to access other Convex resources.

This function runs security validation on the request to ensure that it came from a trusted source, and runs some database mutations based on the type of event received. The mutations simply mirror the event - creating, updating, or deleting a user in Clerk simply results in creating, updating, or deleting the same user in Convex. It forms a copy of the Clerk user table within Convex.

// convex/http.ts

  switch (event.type) {
    case "user.created":
    case "user.updated": {
      await ctx.runMutation(internal.users.updateOrCreateUser, {
        clerkUser: event.data,
      });
      break;
    }
    case "user.deleted": {
      const id = event.data.id!;
      await ctx.runMutation(internal.users.deleteUser, { id });
      break;
    }
    default: {
      console.log("ignored Clerk webhook event", event.type);
    }
  }

This allows the most powerful feature of Convex - Queries, to access the user data directly, which means the application doesn’t need to make additional requests to Clerk to fetch the user data (also called the n+1 problem), and the user interface can get notified if the data changes (e.g. if the user changes their name or profile picture).

// convex/users.ts

export const getUser = internalQuery({
  args: { subject: v.string() },
  async handler(ctx, args) {
    return ctx.d
      .query("users")
      .withIndex("by_clerk_id", (q) => q.eq("clerkUser.id", args.subject))7
      .unique();
  },
});

Full Potential

Webhooks are a powerful feature that enable integrations with existing backend tools, allowing you to use their technical potential to the fullest, while also using Clerk's user management capabilities to the fullest. They help glue disjointed parts of the backend together so that you can pick whatever technology suits your use case for storing and processing data.

Ready to Integrate Authentication in Your App?

You can explore the Clerk Webhooks docs to learn more about the webhook events and data that is exposed by Clerk and build your own integrations today!

For more in-depth technical inquiries or to engage with our community, feel free to join our Discord. Stay in the loop with the latest Clerk features, enhancements, and sneak peeks by following our Twitter account, @ClerkDev. Your journey to seamless User Management starts here!

Clerk Webhooks: Getting Started

Dev Agrawal — Wed, 29 Nov 2023 01:30:33 +0000

In an age of software that is composed of decoupled services built by specialized (often outsourced) teams, webhooks play an important architectural role. Webhooks enable us to outsource important user interactions and services to external platforms that are specifically designed for those purposes, and still have our applications be able to respond to those interactions.

For example, Clerk is specifically designed to handle authentication and user management related interactions on behalf of your applications. Webhooks allow Clerk to notify your application of these interactions so that you can implement custom follow-up logic. These interactions include (but are not limited to) signing in, changing profile information, creating an organization, or getting invited to an organization. Clerk ensures that all these important interactions are handled securely and responsively, while giving you a way to implement follow-up logic.

In this blog post we will build a simple webhook integration in a Next.js project. We will also explore the use cases of webhooks with Clerk and discuss how to best handle webhooks for production-ready applications.

This post assumes basic familiarity with Next.js and Clerk.

Setting up Webhooks in Next.js

For this walkthrough we will start with a Next.js project that is already setup with Clerk. However, you can create a webhook subscription in any server-side web framework like Express or Fastify.

If you don’t already have a Next.js project setup with Clerk, you can start by cloning the Clerk Next.js App Router template.

git clone https://github.com/clerkinc/clerk-next-app.git`

This will create a minimal Next.js project fully setup with Clerk’s embedded components and a secured dashboard page.

Create a .env.local file in the root of the project and add your keys here from the Clerk dashboard.

NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY=your_clerk_pub_key
CLERK_SECRET_KEY=your_clerk_secret_key

Let’s create an API route that will handle the Clerk webhook. This handler will live in app/api/clerk/route.ts and will have the URL of your-app.com/api/clerk.

Webhooks are POST requests by default, so let’s create a handler for receiving POST requests.

export async function POST(request: Request) {}

The webhook event data is available as the request payload. Let’s parse it out and log it to the console.

import { WebhookEvent } from "@clerk/nextjs/server";

export async function POST(request: Request) {
  const payload: WebhookEvent = await request.json();
  console.log(payload);
}

Clerk provides a WebhookEvent type for strong typescript inference. For now we will simply assert the webhook payload to be of this type.

Let’s also create a simple GET handler to test if we can access this URL.

import { WebhookEvent } from "@clerk/nextjs/server";

export async function POST(request: Request) {
  const payload: WebhookEvent = await request.json();
  console.log(payload);
}

export async function GET() {
  return Response.json({ message: "Hello World!" });
}

Add this endpoint as a public route to the Clerk middleware

import { authMiddleware } from "@clerk/nextjs";

export default authMiddleware({
  publicRoutes: ["/", "/api/clerk"],
});

export const config = {
  matcher: ["/((?!.*\\..*|_next).*)", "/"],
}

Start the development server.

npm run dev

Now we can navigate to this endpoint in the browser.

That’s all we need to setup a simple webhook endpoint! Now we can provide Clerk with a URL to this endpoint, and we can start receiving events.

However, we are not ready to test this locally yet. While the application is accessible on localhost in our machine, it’s not accessible to anything outside our local network. Since webhook requests are originated from Clerk’s servers, the localhost URL will just point back to the Clerk server instead of our machine.

While we could provide Clerk with the public IP address of our machine, any incoming requests to our local home/work networks will be blocked by firewalls.

To work around this problem, we can use a tunneling service like localtunnel, which creates a secure tunnel to our local machine from the cloud.

Let’s setup localtunnel in a new terminal window.

npm install -g localtunnel

Once the installation is complete, we can create a tunnel to our local server.

lt --port 3000

You should see an output in your terminal like this.

Localtunnel will generate a random URL on the loca.lt domain that will point to your locally running server on port 3000. We can navigate to this new URL in our browser.

Localtunnel has a security mechanism built in to prevent abuse from malicious actors and phishing links. It asks for the public IP address of your local machine to ensure that it’s really you trying to access your application. Follow the instructions on this page to get your public IP address, and submit it on this page.

Once that’s done, we can access our application on this URL. We should also be able to access our webhook endpoint.

If the application doesn’t show up, make sure both the nextjs server and localtunnel are running on separate terminal windows.

Now we are ready to plug our endpoint into Clerk’s dashboard. Go to the Webhooks page, and click Add Endpoint. Enter the localtunnel URL here. You can also add a description for this endpoint.

Here you can select which specific events you want to receive at this endpoint. Let’s leave these unselected for now so that we can receive webhooks for all events. Hit Create.

Once the webhook endpoint is created, we can navigate to the Testing tab. Here we can trigger an example event manually to test our webhook endpoint. Select any event from the dropdown, and hit Send Example.

If your application is running locally and tunneling is set up correctly, you should see a message in your console with the example event.

You will also see a log entry in the Testing tab that shows the status of the webhook.

Even though the webhook was correctly received and logged to the console, the dashboard marks the webhook delivery as failed. To ensure that the delivery is marked as succeeded, we need to return a success response from our endpoint.

export async function POST(request: Request) {
  const payload: WebhookEvent = await request.json();
  console.log(payload);
  return Response.json({ message: "Received" });
}

Let’s send another example event. This time it the webhook should be marked as succeeded.

Now we can test the webhook with real events. Navigate to your application and sign up with a new account. This should log two webhook events in the console - a user.created and a session.created event.

Congratulations, we now have a functioning webhook receiver in our application! We can now further explore what events are triggered by Clerk. For example,

Updating the user profile sends a user.updated event

Signing out of the application sends a session.ended event

Remotely signing out a different device will result in a session.revoked event

Creating a new organization will trigger an organization.created and an organizationMembership.created event

We can also see logs for real webhook events in the dashboard.

When you deploy this app to a cloud environment like Vercel, you will need to create a new endpoint in the Clerk dashboard that points to the deployed application instead of a localtunnel URL.

Note that localtunnel will generate a random URL everytime we start the tunnel, which means everytime we are testing with webhooks locally, we will need to create a new endpoint in the Clerk dashboard. To solve this, we can ask localtunnel to provision a specific URL by adding a --subdomain argument.

lt --port 3000 --subdomain unique-url-name

If the requested URL is available, it will be provisioned for us instead of a randomly generated URL. This will allow us to reuse the same endpoint in the Clerk dashboard everytime we are testing locally.

Webhook Use Cases

So now that we are receiving webhook events from Clerk, what can we do with it?

Data Synchronization

While Clerk’s own database acts as the primary source of truth for authentication and user management, you can create a copy of this data in your own database which gets updated asynchronously through webhooks. This allows you to query your own database for user data instead of relying on Clerk’s APIs, along with flexibility with the data model, complex queries, transactional guarantees, and real time behavior. This mechanism is called data synchronization.

Event Driven Systems

Some advanced use cases might rely on asynchronous workflows and background jobs, such as sending notifications to users or other external systems. These workflows can be triggered to execute when a webhook event is received, for example, sending new users an onboarding email to your application, or subscribing them to mailing lists for software updates and promotional campaigns. Such systems are called event driven systems.

Bring Your Own Email/SMS

Clerk allows you to bring your own Email and/or SMS servers for delivering auth-related messages which would usually be delivered by Clerk. Whenever an auth-related message is created, Clerk sends a webhook event with that message to your application. You can configure Clerk to not deliver these messages in the Clerk dashboard, and instead handle the delivery yourself by listening to the webhook events. (add links to HWR and docs)

We will walk through implementing some of these use-cases in future articles.

Webhook Best Practices

While setting up a simple webhook receiver for testing purposes is straightforward, operating a webhook-driven system in production requires some additional work. Clerk implements webhook deliveries through Svix, a service that specializes in handling webhook deliveries. Svix manages concerns like scalability, logging, retries, security, and delivery guarantees. However, we still need to ensure that our application is setup to receive and process webhooks correctly. Let’s explore some best practices for implementing webhook endpoints for production-ready applications.

Error Handling

Webhook deliveries are marked as successful or failed depending on the HTTP response returned by our webhook endpoint. A failed webhook delivery is retried with exponential backoff until it succeeds, ensuring that our application will never miss an event. The exponential backoff ensures that our application servers are not overloaded with retries and explode server resources.

Our webhook endpoints should be setup to return a successful response (20x) if everything went well. But in case of an error during the processing of a webhook, we can simply throw an error response (40x or 50x) and expect a retry, which often succeeds. We can also monitor the webhook status in the Clerk dashboard, which provides observability into failed webhook deliveries and help discover issues.

// app/api/clerk/route.ts
import { WebhookEvent } from "@clerk/nextjs/server";

export async function POST(request: Request) {
  try {
    const payload: WebhookEvent = await request.json();
    console.log(payload);

    // process the event

    // everything went well
    return Response.json({ message: "Received" });
  } catch (e) {
    // something went wrong
    // no changes were made to the database
    return Response.error();
  }
}

Security

Since webhooks requests can originate on any remote server, webhook endpoints are vulnerable to attacks from malicious actors. Securing webhook endpoints is important to ensure malicious actors cannot make unwanted changes to your application.

Svix adds security measures to webhook requests by adding a secure hash in the request headers created using a secret key. We can use the svix library in our webhook endpoint to validate that the request was sent by a legitimate Svix server and not a malicious actor.

// app/api/clerk/route.ts
import { WebhookEvent } from "@clerk/nextjs/server";
import { headers } from 'next/headers'
import { Webhook } from "svix";

const webhookSecret = process.env.CLERK_WEBHOOK_SECRET || ``;

async function validateRequest(request: Request) {
  const payloadString = await request.text();
  const headerPayload = headers();

  const svixHeaders = {
    "svix-id": headerPayload.get("svix-id")!,
    "svix-timestamp": headerPayload.get("svix-timestamp")!,
    "svix-signature": headerPayload.get("svix-signature")!,
  };
  const wh = new Webhook(webhookSecret);
  return wh.verify(payloadString, svixHeaders) as WebhookEvent;
}

export async function POST(request: Request) {
  const payload = await validateRequest(request);
  console.log(payload);
  return Response.json({ message: "Received" });
}

We can get the webhook signing secret from the Clerk dashboard and store in a CLERK_WEBHOOK_SECRET environment variable.

Now our webhook endpoint is fully secure and ready to be deployed to production servers.

Selective Events

In our walkthrough, we configured Clerk to trigger the webhook endpoint for all events. While this was useful for testing, we want to minimize the use of server resources by only processing events that we actually care about.

For production endpoints, we can apply event filters in the Clerk dashboard by selecting the events that we are processing in the application, and leaving everything else unselected.

Summary

Webhooks play an important role in creating unified user experiences through integrations like data sync and event driven systems. In this article we setup a simple webhook endpoint, explored some use cases, and addressed concerns like security and error handling.

We will dive deep into specific use cases around data sync and event driven systems in future blog posts and walk through implementing them into our projects.

To learn more about webhooks, check out the ultimate webhooks guide, or to learn more about the webhook events exposed by Clerk, read the documentation.

DEV Community: Dev Agrawal

Async Transformations in Reactivity

1. The Async Gap: Signals vs Events

2. What Structured Async for Events Means

3. createAsyncEvent: Declarative Async in the Event Graph

Single result:

Multiple results with AsyncIterable:

4. What About switchMap, concatMap, mergeMap, exhaustMap?

5. Observing Async Events Structurally

createSyncEvent: Promises from Events

createSyncStream: AsyncIterables from Events

6. Scoped Retry with Event Context

7. Transactions: Deferred Work with runTransaction

8. Events as First-Class Async Structures

Next:

Scheduling Transformations in Reactivity

1. Introduction: The Real-World Punch

2. Mutation Hazards in Push-Based Systems

3. Lessons from Signals — Phased Execution

4. Bringing Phases to Events

Phase 1: Pure event graph

Phase 2: Mutations

Phase 3+: Signals and UI

5. Blocking Unsafe Patterns

6. createMutation in Practice

7. Sync Parity with Signals

8. Final Example — Predictable Flow

Transformations in Reactivity

1. Introduction

2. From Observables to Event Graphs

3. Solid as Inspiration

4. A New Default: Event Handlers as Transformers

5. Filtering with haltEvent()

6. Memory & Lifecycle

7. Rethinking Completion

8. Higher-Order Events (Sync Only)

9. Final Transformation Example

Next Up: Scheduling

Dynamic Typing is Obsolete

Thoughts on Server Components + Websockets

Thoughts on Server Components + Websockets

Just reload the page

RSC over WebSockets

What about Serverless?

But Why RSCs with WebSockets

Clerk Webhooks: Data Sync with Convex

Data Sync with Convex

The Demo

The Code

Full Potential

Ready to Integrate Authentication in Your App?

Clerk Webhooks: Getting Started

Setting up Webhooks in Next.js

Webhook Use Cases

Data Synchronization

Event Driven Systems

Bring Your Own Email/SMS

Webhook Best Practices

Error Handling

Security

Selective Events

Summary

3. `createAsyncEvent`: Declarative Async in the Event Graph

`createSyncEvent`: Promises from Events

`createSyncStream`: AsyncIterables from Events

7. Transactions: Deferred Work with `runTransaction`

6. `createMutation` in Practice

5. Filtering with `haltEvent()`