DEV Community: Srijeet Patil

Why We Moved a Fan-Out Operation to Temporal

Srijeet Patil — Sun, 25 Oct 2026 09:34:58 +0000

A quick note before this starts: we use an internal library that wraps Temporal, fully managed by the backend team at the company I work at. So this isn't a Temporal API tutorial, there are no workflow/activity code snippets here. This is the story of the problem that made us need it in the first place, and why a plain queue stopped being enough.

The old shape of the app

The app was built for a simple assumption: one organization, one tenant. Anywhere from 100 to 250 people, all seeing data scoped to their own org. Most operations, fetches, mutations, the usual, ran in the hundreds of milliseconds. Nothing about the system expected to move more data than that.

Long-running work existed, but it meant something narrower than it does now. It ran through SQS: a job gets queued, a worker picks it up, a visibility timeout governs retries if the worker dies mid-task. There was no live progress back to the UI. If something took a while, the frontend just waited, or polled a status column that said pending, running, or done. Good enough, for the scale we were at.

The request that didn't fit

Then a large organization(soon to be onboarded and would bring in good money to the business ;) asked for something the app wasn't built for: not one org, but a parent entity sitting over 100s of smaller sub-organizations underneath it, with thousands of employees across all of them needing access, scoped correctly to their own sub-org.

Structurally, this wasn't a redesign. The data model mostly held. What changed was scale, and scale changed everything downstream of it.

Why minutes, not milliseconds

An operation that used to touch one org's data now had to fan out across every sub-org underneath a parent. A permissions update that previously touched a couple hundred rows could now touch tens of thousands. Queries got heavier. Mutations, especially, started taking real time: not seconds, minutes.

And a mutation that takes minutes is a different kind of problem than a mutation that takes milliseconds. Users expect fast operations to just finish. They don't expect to sit and watch a slow one, they expect to know it's working, see it progress, and trust it'll finish even if something hiccups along the way.

Why a queue alone couldn't carry this anymore

A queue and a worker can run a slow job. What they can't do on their own is tell you much about it while it's running.

With plain SQS, a long mutation was mostly a black box: queued, then eventually done or failed. For a background job nobody's watching, that's fine. For an operation someone is actively waiting on, now taking minutes instead of milliseconds, it wasn't. We needed the UI to show live progress, not just a spinner. We needed a failure in the middle of fanning out across a hundred sub-orgs to not mean starting over, or worse, leaving the update half-applied across some sub-orgs and not others. And we needed the operation to survive a crash or a deploy mid-flight without silently going nowhere.

None of that is impossible to build on raw SQS. People do. But it means hand-rolling state tracking, hand-rolling retry logic per step, hand-rolling the bookkeeping for "which sub-orgs are done and which aren't," and hoping you got all the edge cases. That's a lot of infrastructure to maintain for every team that needs it.

Why Temporal fit

This is where Temporal made sense, not as a queue replacement, but as the layer above it that tracks what a queue can't: the actual state of a long-running operation, durably, with retries and progress built in rather than hand-built.

A migration across a hundred sub-orgs becomes a workflow with per-sub-org steps. Each step can fail and retry independently. The workflow's progress, how many sub-orgs are done, which failed, which are mid-flight, is something Temporal already tracks, not something we invented a table to approximate. And because that state is durable, a crash or a deploy doesn't lose the operation's place. It picks back up.

That's also why it sits across three layers at once, not just the backend. The backend runs the workflow. The data layer gets mutated sub-org by sub-org. And the UI gets to show something true: a live count, not a guess.

What actually changed under the hood

The data model mostly stayed the same. We didn't redesign tables or rethink how org data was structured. What changed was smaller and more specific: a few new columns to track workflow status per sub-org, a few new enum values to represent states that didn't exist before, queued, in_progress, failed, completed, per operation instead of just per job.

That's a deliberately unglamorous detail, but it matters. The architecture shift wasn't a rewrite. It was adding a layer that could track and orchestrate what was already there, not replacing what was there. For a product engineer owning a module on top of this, that's also the part that mattered most day to day: the shape of the data I worked with barely moved. What moved was how operations on that data got run and reported back.

How it fits together

The backend starts the workflow, the workflow fans out into independent, retryable steps per sub-org, and progress flows back to the UI as each one completes. One sub-org step is shown mid-retry, amber, it failed once and is trying again, entirely independent of the other sub-orgs running alongside it. That independence is the part that would've been hardest to hand-roll on a plain queue: one slow or flaky sub-org doesn't block or restart the rest.

The outcome

The honest nuance here: Temporal didn't make the fan-out operation faster. A mutation across a hundred sub-orgs still takes however long it takes, there's no shortcut around the actual volume of data being touched. What changed was the wait itself. Instead of a black box that either finished or silently failed, the UI could show real progress, a failure in one sub-org didn't take down the rest or leave the operation in an ambiguous half-done state, and a crash or deploy mid-run no longer meant starting over.

For users, that's the difference between "this feels broken" and "this is working, and I can see it." The operation was always going to take minutes. What Temporal gave us was the ability to make that time feel safe instead of uncertain.

Why Our App Hung Forever on Some iPads (and Nowhere Else)

Srijeet Patil — Sun, 27 Sep 2026 09:42:03 +0000

The report

We build a canvas-based drawing tool for schools, embedded inside our main mobile app, the app educators, students, and families use day to day. By December 2024, that meant one specific place things could go wrong: a WebView inside the mobile app, loading a React web app inside it.

It was the middle of exam season when the first bug report came in: a few students couldn’t load the web canvas based drawing/sketching app. It showed an infinite loader.

The reports were coming from one school in the Middle East, a lot of students, right as they were trying to submit mid-semester work. And whatever was causing it, we couldn’t reproduce it ourselves.

Why preloading exists

A quick bit of context: loading a web app inside a WebView isn’t free. Every time it happens, the device has to fetch the JS, CSS, and HTML from scratch and it is slow, especially on a mobile connection. So like a lot of apps doing this, we used a common trick: preloading.

The idea is simple. The moment the mobile app starts, we spin up a WebView in the background. Invisible, display: none, and start loading the canvas app before the student ever asks for it. By the time they actually open it, it’s already there, warm and ready.

It’s a good trick, and most of the time, it worked exactly as intended.

Ruling out the obvious

Our first move was the standard one. We assumed it was a firewall issue. Schools often block URLs they don’t recognize, so we asked them to whitelist ours. It’s usually a quick fix. This time, it wasn’t. The bug didn’t budge.

We tried reproducing it ourselves normal network, slow network, nothing. The canvas app loaded fine every time. We turned to Browserstack, picked iPads from the same region, throttled the connection to match, and finally got a partial repro. The app did load slowly. But it loaded. Two to three minutes, not forever.

That wasn’t the bug we were chasing. Students weren’t waiting three minutes, they were stuck indefinitely.

We sent someone from the team on-site. Same result: stuck. Not a firewall, not our test environment, not a fluke. Whatever this was, it was real, and it was only real for them.

Chasing the region

With the obvious explanation gone, we started reaching for regional ones.

We also suspected network interception, heavy-handed filtering at a national level, common enough in some parts of the world. We ruled out that possibility since the pattern didn’t match, and switching the students to mobile data instead of school wifi made no difference.

Someone else wondered if it was just too many students on the same network at once. Again no, the bug hit students independently, not in a pattern that tracked with shared network connection.

Middle Eastern mobile data can be expensive, and expensive sometimes means throttled. But the school sent us a speedtest which showed a steady 20-25 Mbps. Not the problem.

The call with the affected school

We added Sentry logs to track where the load was getting stuck. It wasn’t consistent, sometimes stuck in one part of the React app, sometimes another. We added more logging inside componentDidMount (the codebase predates functional components).

We got on a call with the school. The teacher used a student’s actual device while we watched logs come in live. Clear data, but no answer yet. The load kept stalling at different points each time, somewhere in the list of internal tasks we run to fetch student data.

Then a technical manager on the call, not someone actively working on the project, asked us to disable preloading via a feature flag. Load time dropped from 2-3 minutes to 15 seconds.

That stung a little. The fix was one flag, and it took someone outside the project to think of it.

A fix that explained nothing

We disabled preloading in mobile WebViews globally for our mobile apps, via migration. Load times increased slightly under normal conditions, but the app stopped hanging indefinitely.

It wasn’t the fix that explained anything. We spent months chasing why preloading caused this in the first place:

WKWebView deprioritizing background processes. But the logs were coming in fast enough that the WebView clearly wasn’t starved of CPU.
Bundle size: The canvas app is 3-5MB after extraction. Possible, but we didn’t have hard data connecting this to the failure.
Regional network monitoring: Since this was before the war, so no infrastructure disruption to point to.

Disabling preloading hadn’t even fully fixed the underlying slowness, some users still saw load times up to a minute on the normal, on-demand flow. Preloading wasn’t the root cause. It just turned a slow load into an infinite one.

The real fix, discovered by accident

The real answer showed up months later, indirectly.

Around the middle of the year, my team raised the min_ios_version_supported for our apps. This meant users on older devices either had to update or lose access. We didn’t do this to fix the loading bug, it wasn’t even the primary reason. But in the months after, the infinite-loader reports stopped. They just didn’t come back.

We hadn’t proven this was the fix. We’d removed the population of devices most likely to be affected, and the symptom disappeared with them.

What I think was actually happening

Here’s what I think was actually happening. Older iPads (8th–10th gen) didn’t have the CPU or memory headroom to load the canvas app fast enough, especially while a background WebView was also being preloaded, competing for the same limited resources. On a fast device, that race condition resolves in milliseconds and nobody notices. On a slow device, it can stall indefinitely.

That also explains why webViewWebContentProcessDidTerminate never fired, the WebView wasn’t crashing. It was just starved, still technically alive, never finishing.

Preloading wasn’t the root cause. It just turned an existing problem, old hardware struggling with a 3-5MB app into a worse one. The real fix wasn’t a code change. It was deciding which devices we’d support.

And the regional pattern we chased for weeks? Probably a coincidence of reporting, not geography. We only ever found what people told us about.