DEV Community: Devico Solutions

How to Build a Scalable File-Processing Pipeline: Validation, Metadata Extraction, Queues, and Workers

Vic Tekys — Tue, 15 Sep 2026 09:35:00 +0000

Here's a failure I've seen more than once on upload-heavy products: the endpoint returns 200, the user closes the tab, and twenty minutes later there's no order in the system because a worker choked on a vendor's proprietary container format. Nobody told the user. Nobody told the worker's author either, because the retry policy quietly redelivered the file until it hit a dead-letter queue nobody monitored.

The standard advice — "put a queue in front of it" — doesn't prevent any of this. The queue was never the hard part.

After building ingestion for dental CAD files, construction models, bulk carbon data, and business documents, I've settled on a different frame: a file pipeline is four stages with different latency budgets and different failure semantics. Throughput is decided by how early you reject bad input and how much derived work leaves the request path — not by which broker you pick.

Four stages, not one queue

The four stages are gate, ingest, extract, derive.

Gate is synchronous and cheap: auth, quota, size, extension, magic-byte sniffing. It runs while the user is still watching and its budget is milliseconds.
Ingest moves bytes into object storage and writes the metadata record. It should touch your API process as little as possible.
Extract is asynchronous, per-format parsing: opening the container, pulling metadata, validating structure. This is where the real engineering lives.
Derive produces the artifacts your product actually reads — thumbnails, previews, normalized records, optimized model formats. Also async, and unlike extract, usually cacheable and re-runnable.

The reason to name the stages is that the boundaries between them are where you retry, cache, and shed load. A parse failure shouldn't re-run the upload. A thumbnail failure shouldn't re-run the parse. When everything is one "process file" job, every retry repeats all the work and every failure is opaque.

On the Carbon Management Solution project — bulk emissions data, where a user uploads a single file containing any amount of data that then gets validated and processed — we ran this on serverless AWS with Lambda, SNS, SQS, and DynamoDB, structured so that a certain service is responsible for a certain stage of the process. Stage isolation was the design, not an accident of the stack.

Honest caveat: service-per-stage multiplies your operational surface. Four services means four sets of alarms, four deploy pipelines, four things to reason about at 2 a.m. At low volume, one worker pool consuming from a couple of queues is the right call. Draw the stage boundaries in code first; split the infrastructure when a stage actually needs independent scaling.

Validation is a two-tier problem

The all-async reflex — "accept everything, validate in the worker" — is how you get the silent-failure story from the intro. But the opposite reflex, deep validation in the request handler, is how upload endpoints time out. Validation is two tiers, and the split is not negotiable.

What has to fail while the user is still watching

Anything a user can fix by picking a different file must fail synchronously: wrong extension, oversized file, empty file, exhausted quota, a MIME type that doesn't match the bytes. If the user learns about these hours later from an email, your gate failed at its one job.

The gate checks are cheap because they read a few bytes, not the structure:

import { fileTypeFromBuffer } from "file-type";

const ALLOWED = new Set(["model/stl", "application/zip", "application/pdf"]);

const MAX_BYTES = 500 * 1024 * 1024;

export async function gate(head: Buffer, declaredSize: number, user: User) {

if (declaredSize === 0 || declaredSize > MAX_BYTES) {

throw new UploadError("size_rejected");

}

if (await quotaExceeded(user, declaredSize)) {

throw new UploadError("quota_exceeded");

}

// Sniff real content type from magic bytes — never trust the filename

const sniffed = await fileTypeFromBuffer(head);

if (!sniffed || !ALLOWED.has(sniffed.mime)) {

throw new UploadError("type_rejected");

}

The line to look at is the sniffing: content type comes from the bytes, not the filename. Renamed .exe files and mislabeled archives are a when, not an if.

On BEGO, a German dental CAD/CAM company, we went further and put scenario detection in the order wizard itself: the upload step auto-detects whether the user dropped 3Shape output, exocad output, standalone STLs, or a mixed set, and routes the flow accordingly. That's still gate-tier work — it reads signatures and file lists, not full structure — and it means the user is corrected before the order exists, not after.

What can only fail inside a worker

Structural correctness of a real-world format is unknowable until you parse it. Whether a proprietary archive decrypts, whether the XML inside references parts that exist, whether an STL mesh is watertight — none of that fits a request budget, and none of it should.

So the contract with the user changes shape: the synchronous response means "your file is accepted for processing," never "your file is valid." The pipeline then needs a way to deliver bad news — a per-file status the UI polls or subscribes to, flipping from processing to failed with a reason a human can act on. On BEGO, the order platform tracks per-file status through queue processing for exactly this reason.

The failure mode to design against: sync checks can be spoofed and deep checks can't be rushed. Any team that tries to collapse the two tiers into one ends up with either a slow gate or a lying one.

Metadata extraction is format work disguised as infrastructure

Generic pipeline tutorials treat "extract metadata" as one box in the diagram. In practice it's the majority of the engineering, because it's per-format parsing, and formats are hostile.

On BEGO the extract stage deep-parses laboratory formats — .3ox, .dentalProject, .constructionInfo, .modelInfo— plus encrypted .bego/.begostl archives with decryption and retry logic, pulling out patient and order metadata, materials, colors, and tooth mapping. None of that generalizes. Every one of those parsers is bespoke work against a vendor format that ships no public spec and changes without notice.

On MemoMeister, a document-management SaaS we built from scratch for the German market, extraction went a step further into classification: workers analyzed the keywords, data formats, and other characteristics of an upload to determine the document type and route it to the right user or group. The file's type came from its content, not its name.

Two lessons from shipping that kind of extraction. First, content-derived classification is probabilistic — you will guess wrong, so the design must include a manual-correction path, not just a confidence score. Second, budget for vendor formats changing silently underneath you; a parser that worked for a year is not a parser that works.

This is why "which queue should I use" is usually the wrong first question. The broker is a commodity. The .dentalProject parser is not.

Get bytes out of your API process

One flat infrastructure rule: file bytes should not flow through your API. The client uploads directly to object storage; the API's job is to issue the upload authorization, write the metadata record, and enqueue the job.

BEGO uploads via pre-signed URLs to cloud storage. MemoMeister stored documents on Amazon S3, with workers picking files up after a successful upload. The pattern is the same either way: the API hands out a scoped, expiring write permission and steps out of the data path.

The shape of it, with a commit step:

// 1. API issues the upload — after gate checks, including quota

const key = uploads/${user.id}/${randomUUID()};

const url = await getSignedUrl(

s3,

new PutObjectCommand({

Bucket: BUCKET,

Key: key,

ContentLength: declaredSize, // cap what the URL can write

}),

{ expiresIn: 300 },

);

await db.file.create({ key, userId: user.id, status: "pending_upload" });

// 2. Client PUTs bytes straight to storage

// 3. Client confirms; API verifies the object exists, then enqueues

export async function commit(key: string) {

const head = await s3.send(new HeadObjectCommand({ Bucket: BUCKET, Key: key }));

await db.file.update({ key }, { status: "uploaded", etag: head.ETag });

await queue.send({ key, etag: head.ETag });

}

The part to look at is step 3. Without an explicit commit, you have no moment at which the upload is known-complete, and your extract stage starts firing on half-written objects.

Be honest about what this pattern costs: validation now happens after the write. A client can obtain a URL and never call commit, so you need a sweeper for orphaned objects. Quota has to be enforced before the URL is issued, on the declared size, and re-checked at commit. Direct-to-storage upload is still the right default — it just isn't free.

The queue decisions that actually change throughput

Now the section everyone expects, argued honestly: broker choice matters far less than the policies around it.

BEGO runs its workflow processing on Redis-backed queues under NestJS and Kubernetes. MemoMeister ran RabbitMQ, with some workers calling third-party services like Google Vision and others monitoring database changes to update document state. Carbon Management Solution used SNS and SQS. Three stacks, one set of decisions that mattered:

Queue per stage, not one queue for everything. Extraction backing up shouldn't delay thumbnail generation for files already parsed. Separate queues give you separate backpressure and separate scaling knobs — and keep CPU-bound parsing workers apart from IO-bound workers waiting on third-party APIs, which want completely different concurrency settings.

Visibility timeout longer than your worst file. On SQS-style brokers, a message picked up but not deleted becomes visible again after the timeout. Set it shorter than your slowest legitimate parse and you get two workers processing the same giant file, doubling load exactly when the system is already struggling.

A dead-letter queue you actually watch. Bounded retries, then park the message. An unmonitored DLQ is the silent-failure story again, one hop removed.

One warning on MemoMeister's DB-watching workers: they worked, but they couple the pipeline to your schema — every migration becomes a pipeline change. I'd reach for explicit events first and treat DB-watching as a legacy-integration tool.

Idempotency, retries, and poison files

Standard queues are at-least-once delivery: the broker guarantees a message arrives, not that it arrives once. Operationally that means every worker you write will eventually run twice for the same file. Plan for it or debug it.

The fix is idempotency keyed on the stored object — bucket, key, and ETag — checked before the expensive work:

async function handle(msg: { key: string; etag: string }) {

const claimed = await db.extraction.tryClaim(msg.key, msg.etag); // unique index

if (!claimed) return; // duplicate delivery or a lost race — done

const meta = await parse(msg.key); // the expensive part

await db.extraction.complete(msg.key, msg.etag, meta);

}

Keying on the ETag rather than just the key means a re-uploaded file with new bytes is new work, while a redelivered message for the same bytes is a no-op.

Then there are poison files — the input that kills the worker itself. The classic is the model that OOMs the process: the worker dies before it can nack or delete, the visibility timeout expires, the message comes back, and the next worker dies too. Your fleet is now a crash loop with one file as the ignition. Defenses are boring and essential: a max-receive count routing to the DLQ, memory limits that fail the job instead of the pod, and a size ceiling in the gate so the 9 GB file never enters the pipeline at all.

Derived artifacts are the product

Here's the second assumption worth breaking: the pipeline doesn't exist to store files. It exists to produce cheap-to-read outputs. If the artifact your product serves is still expensive to consume, you optimized the wrong half.

BEGO generates STL thumbnails at extract time so the order UI never touches a mesh. And the clearest evidence I have is Vitus, a Danish constructability platform handling .ifc, .nwd, .rvt, .3dm, and other CAD formats for construction companies like Munck, Femern A/S, VINCI, and COWI. Their viewer struggled not because ingestion was slow, but because the format being loaded was heavy.

The fix was a bundle of read-side changes — migrating models to SVF2, parallel loading, web workers off the main thread, list virtualization, IndexedDB caching. Together they cut model size by 5x and loading time by roughly 2.5x, with multi-model loading 5x faster. To be precise about what those numbers are: client-side model-loading gains from the whole bundle of changes, not a server-side pipeline benchmark, and no single change gets the credit. But that's exactly the point — the wins came from changing the data format and the loading strategy. Worker count had nothing to do with it.

The tax on derived artifacts: they need versioning, and a format migration is a backfill, not a deploy. When Vitus moved to SVF2, every existing model needed re-derivation. Stamp every artifact with the version of the deriver that produced it from day one, so "re-derive everything older than v3" is a query and a queue-fill instead of a forensic project.

Constraints that beat the ideal design

Everything above assumes you get to choose where processing runs. Sometimes you don't, and residency, encryption, and hardware outrank performance.

MemoMeister was restricted to third-party services that stored and processed data within Germany. That single compliance line forced a self-hosted OnlyOffice deployment instead of a managed document service — an architectural decision made by a regulation, not a benchmark.

Visbion is the sharper example: their Image Cube compresses and encrypts DICOM 3.0 medical images on dedicated routing hardware inside mobile scanning trailers, serving NHS Breast Screening Services as the UK's largest installed base of dedicated imaging routing hardware. The heaviest processing happens at the edge, on a device in a trailer, before any cloud pipeline sees a byte — because bandwidth from a mobile trailer and encryption requirements for medical imaging say so. Our work there was consulting on fleet-deployment automation, compression and encryption refinement, and a modular redesign — tuning that constraint-driven architecture, not replacing it with a textbook one.

The trade is real: edge and self-hosted processing give up elasticity and easy observability for compliance and bandwidth. Check these constraints before drawing the architecture, because they don't negotiate afterwards.

Where to spend the effort, in order

If I'm starting an ingestion system today, the order is: a cheap synchronous gate that rejects everything a user can fix on the spot; bytes going straight to object storage with an explicit commit; idempotent workers with a DLQ someone watches; and only then worker scaling and broker tuning. The per-format parsers will take longer than all of the infrastructure combined — budget accordingly.

Reach for the full four-stage split when files are big, formats are hostile, and derived artifacts are what users actually consume. Skip the ceremony when you're thumbnailing avatars — a gate check and one background job is a complete architecture at that size.

And check the read side last, every time. If the artifact coming out of the pipeline is still expensive to consume, no amount of queue work will save you — you've optimized the half of the system your users never touch.

How to Modernize a Legacy SaaS Without Freezing Feature Delivery: Refactor, Replatform, or Rewrite?

Vic Tekys — Wed, 09 Sep 2026 14:30:07 +0000

The moment arrives on every aging SaaS. Sales has commitments on the roadmap, and the codebase is telling you it can't carry them — every feature ships with a regression tax, the framework stopped getting patches years ago, and nobody fully understands the billing module anymore.

The reflex answer is always the same: freeze the roadmap for two quarters, build "v2" on the side, migrate everyone over, resume shipping. I've sat in that meeting more than once.

That reflex is wrong twice over. First, no revenue-generating SaaS actually gets a two-quarter freeze — customers keep filing tickets, competitors keep shipping, and the freeze quietly becomes four quarters. Second, it treats "modernize" as one decision you make once, for the whole system.

It isn't. Refactor, replatform, or rewrite is a per-component verdict. And the way you avoid the freeze is to stop running modernization as a separate project and make it one backlog with feature delivery: build new capabilities as new services, refactor the code you're already touching, and rewrite only what genuinely can't be saved.

The stakes of getting this wrong are documented. In a 2022 Wakefield Research survey of 250 senior IT leaders at large enterprises, 79% of app modernization projects failed, at an average cost of $1.5 million and a 16-month timeline. Those numbers describe big-bang programs at big companies — but the failure mode they capture is exactly the "pause everything, modernize, resume" shape most teams reach for by default.

The false trilemma: you don't choose once for the whole system

Most refactor-vs-rewrite articles frame this as a single fork in the road. Pick a verb, apply it to the platform, done.

Every real modernization I've been close to mixed the verbs — sometimes all three, over years.

On GoodShape, a wellness platform that had been in production for about 15 years, we ran a microservices migration and heavy in-place refactoring at the same time: new capabilities went out as new services while hundreds of bugs got fixed and thousands of code smells got eliminated in the monolith that was still serving traffic.

On Sunlayar, the engagement started around the legacy QMPV solar-design platform — and only after working with it did the honest assessment land: continued maintenance would cost more than a rewrite, so a rewrite is what happened.

Same company doing the work, opposite verdicts. Because the verdict belongs to the component and its situation, not to a philosophy.

One warning before you adopt per-component thinking, though: it needs a whole-system owner. Somebody has to hold the target architecture in their head and decide which extraction happens next. Without that, "decide per component" degrades into a half-migrated estate that stays half-migrated forever — the worst of both worlds, permanently.

Diagnose what's actually failing before picking a verb

"Legacy" is not a diagnosis. It's a bucket for at least four different diseases that get treated very differently:

An obsolete stack — frameworks and runtimes past end-of-life.
Missing documentation — the system's behavior lives in people's heads, some of whom left.
Inconsistent code — written by different developers across different eras, no common standards.
An architecture at its scalability ceiling — the structure itself can't take more load or more change.

The first three are painful, but they're curable in place. Only the fourth justifies structural change — replatforming or rewriting.

This distinction matters because tech debt pain is nearly universal and therefore nearly useless as a signal. In the 2024 Stack Overflow survey, 62.4% of professional developers named technical debt as their top frustration — the number one answer among both ICs and managers. And McKinsey estimated back in 2020 that tech debt amounts to 20–40% of the value of a company's entire technology estate. If frustration were the trigger for a rewrite, everything would get rewritten. The trigger has to be sharper than that.

Architecture at its ceiling vs. code that's merely ugly

The question I actually ask: can this architecture carry the next two years of roadmap? Not "is this code nice." Not "would we build it this way today." Nobody would build any five-year-old system the way they'd build it today; that's not a finding.

GoodShape had all four diseases at once — no documentation, no common coding standards, an obsolete stack, and a monolith that had nearly reached peak performance, with further scaling becoming impossible. That last item is what earned structural change. The other three would have been refactor-shaped on their own.

medneo, a radiology-as-a-service provider operating across Germany and the UK, looked superficially similar — a Radiology Information System degrading under a growing patient load, with regressions creeping in as features accreted. But the architecture could still carry the roadmap. The diseases were slow code, inefficient database access, and outdated frameworks. Refactor-shaped, all of it.

One more bias to name: inside teams overwhelmingly want the rewrite. It's more fun, and it promises escape from code they resent. But fifteen years of production code encodes fifteen years of edge cases, regulatory quirks, and 3 a.m. fixes that nobody remembers making. Joel Spolsky called the full rewrite "the single worst strategic mistake" back in 2000, and the reason he gave — old code has been used, and using code shakes bugs out of it — hasn't aged a day.

Refactor in place — when the foundation still holds

Refactoring is the least glamorous verdict and the one I end up defending most often.

On medneo, the work was exactly as unglamorous as it sounds: optimize hot paths, eliminate unnecessary database calls, upgrade legacy frameworks and libraries to supported versions, and stand up a CI/CD pipeline. Eight engineers, from 2019 to 2022.

The outcomes — significantly reduced average load times, fewer outages, and new rollouts that stopped degrading performance — are qualitative, because the client doesn't publish before/after numbers. I'd rather tell you that plainly than invent a percentage. But the strategic point is quantitative enough: three years of platform improvement happened while the RIS kept serving hospitals, with zero freeze, because none of that work required stopping feature delivery. Refactoring almost never does.

Soapbox, an e-commerce fulfillment platform, ran the same play at a much earlier stage: a React Native warehouse app that had shipped as a prototype — poorly structured code, plenty of bugs, no separation between dev, staging, and production. The team did a global refactoring of the core logic, put a linter in place, set up GitLab and CodePush, and split the environments — and shipped more than 15 releases along the way, including entirely new features like stocktaking, warehouse audit, and barcode scanning. Cleanup and delivery, one backlog, four people.

The limitation is the one from the diagnosis section, and it's hard: refactoring cannot raise an architectural ceiling. If the structure itself is the disease, no amount of in-place cleanup cures it. You'll just have beautiful code that still can't scale.

Replatform incrementally — migration under live traffic

The middle verdict is the one that actually answers this article's title, so it gets the most detail.

GoodShape is the fullest version of it I've worked near. The platform had been in production about 15 years, serving 200+ UK employers — including NHS organizations, Philips, and Heathrow Express — with 24/7 nurse support riding on it. You do not take that offline for a rebuild. The verdict on the monolith was replatform: migrate toward microservices, incrementally, under live traffic. A team of 7, working since 2018, on Java 17 and Spring Boot with React/Redux up front.

The mechanism is what matters, because it's the anti-freeze tactic in concrete form:

New capabilities ship as new services. Push notifications and fitness-tracker data collection — integrating 16 different tracker brands — were built as standalone microservices from day one. The roadmap item was the migration step. Nobody had to choose between them.
Code you're already touching gets refactored. A few hundred bugs fixed and thousands of code smells eliminated — not as a dedicated cleanup phase, but woven through feature work in the monolith.
Documentation gets written as you go, capability by capability, instead of as a doomed up-front archaeology project.
The frontend got split into subprojects and refactored with updated React components — same incremental logic, applied to the UI.

Even purely manual pain got folded in: monthly client reporting had been a hand-built 50-slide PowerPoint; it became an automated reporting feature. Roadmap value and modernization, again the same line item.

If you want the underlying pattern with a name, it's Martin Fowler's strangler fig: new growth wraps the old system until the old system is no longer load-bearing. Stripped to its skeleton, the runtime side is just routing — carve one capability's traffic away from the monolith at the edge:

Everything still goes to the monolith...

location / {

proxy_pass http://legacy-monolith;

}

...except the capabilities that have been extracted.

location /api/notifications/ {

proxy_pass http://notification-service;

}

location /api/tracker-data/ {

proxy_pass http://tracker-service;

}

The routing is the easy part. The discipline is the hard part, and it's the limitation to be honest about: this is a multi-year posture, not a project with an end date — GoodShape's has been running since 2018. You need that whole-system owner deciding which extractions the roadmap actually justifies, or you'll extract whatever's fun instead of whatever's next.

One more honesty note, since it's tempting to oversell: GoodShape won gold at the 2020 ECCCSA awards for application of technology while all this was mid-flight. I'm not claiming the migration caused that — the sources don't, so I won't. The claim I am making is narrower and more useful: the business never had to stop operating, competing, or shipping while its 15-year-old core was being restructured underneath it. And the scalability gains came from the migration plus*the refactoring *plus the process changes — not from microservices as a magic word.

Rewrite — the expensive verdict, and what earns it

Sometimes the honest answer is that the patient doesn't recover in place.

The threshold I use: rewrite when maintaining the old platform costs more than replacing it, and little of it is worth preserving. Both conditions. The first alone describes half the software in production; it's the second that earns the verdict.

Sunlayar met both. The QMPV platform for planning rooftop solar projects ran on obsolete technologies, and as complexity grew it became impossible to add new business functions quickly — the assessment concluded a rewrite would beat continued maintenance. So it was rebuilt as a new platform on Node.js, TypeScript, React, and MongoDB, by a team of 2 developers and 2 QA engineers, in an engagement running since 2014. The rewritten platform outlived its original owner: QMPV was later acquired by Esdec, and the platform is still in production today as EcoFasten's Design Assistant Project Calculator. (Again — the acquisition is a fact of the timeline, not a result I'm attributing to the rewrite.)

BEGO, a German dental company, is the other flavor: not one obsolete platform but several independent portals, with an architecture where every new feature had to work around structural limitations. The verdict was rebuild-as-one-platform — React, TypeScript, NestJS, PostgreSQL, Kubernetes — delivered by a team of about six, with 14 major features, cooperation running since 2025, and a high daily order volume from launch.

But here's the detail that matters most for this article: even BEGO wasn't a pure greenfield rewrite. The proven backend file-processing components were carefully reused. "Rewrite" as a verdict doesn't mean "salvage nothing" — it means salvage selectively, keeping the parts that have earned their production scars.

The limitation: rewrites only look fast in retrospect. Sunlayar's engagement spans over a decade; nothing about these timelines is quick. If someone pitches you a rewrite with a confident six-month schedule, that schedule is the first bug.

The process layer that keeps features shipping

None of the above works without an unglamorous substrate, and it shows up in every one of these cases.

Environment separation, so modernization work and feature work can land in the same codebase without stepping on each other — Soapbox literally could not do both safely until dev, staging, and prod were split. CI/CD, so the cost of shipping small stays near zero — medneo's pipeline is what let optimization work roll out continuously instead of batching into risky releases. Automated linting, so a refactored module and a legacy module stop diverging in style the day after cleanup. Documentation as a habit, which is what GoodShape used to stop the no-docs disease from re-infecting every new service.

The trap: process work reads as slowdown first. It's the line item that gets cut when the roadmap pressure is highest — which is exactly the moment you need it, because it's the only thing that lets one team serve two masters in one codebase.

The question to write down first

The three verdicts aren't symmetric, and choosing between them gets easier once you say the asymmetry out loud.

Refactoring is cheap and reversible — if it turns out to be insufficient, you've lost little and the code is better for whatever comes next. Incremental replatforming trades speed for continuity — slower than the big-bang fantasy, but the platform never stops earning. Rewriting is a one-way door — occasionally still the correct door, as Sunlayar and BEGO show, but you don't walk through it because the old code is annoying. You walk through it when maintenance costs more than replacement and almost nothing is worth saving — and even then, you carry the proven parts with you.

So before picking a verb, write down the answer to one question, per component: can this architecture carry the next two years of roadmap?

Where yes — refactor what you touch, and keep shipping. Where no, but the capability boundaries are extractable — replatform incrementally, and let the roadmap decide the extraction order. Where no, and nothing's worth saving — rewrite, salvage selectively, and be honest with everyone about the years, not months, you just signed up for.

The one answer that's never right is the one that sounds safest in the meeting: freeze everything and build v2 on the side. That's not a modernization strategy. That's the 79%.

How to Keep a Data-Heavy React App Fast

Vic Tekys — Mon, 31 Aug 2026 13:18:24 +0000

A React dashboard that feels perfectly responsive with 200 rows may become almost unusable at 20,000. Memoization is often the first proposed fix, but React.memo cannot help much when the browser is laying out thousands of DOM nodes or a large data transformation is blocking the main thread.

Those problems may produce the same visible symptom—a slow interface—but they have different causes. Rendering too much calls for virtualization. Expensive JavaScript may belong in a Web Worker. Oversized or repeatedly downloaded datasets point toward pagination, caching, or IndexedDB. Media introduces another set of costs.

Getting the diagnosis right matters because users experience far more than the initial page load. Google estimates that roughly 90% of a user's time on a page is spent after it has loaded, when responsiveness depends on how quickly the application handles input, updates data, and paints the next frame.

Find the bottleneck before choosing the optimization

Most performance problems in data-heavy React applications fall into four categories.

Rendering cost grows with the number and complexity of DOM nodes React creates, reconciles, and updates. A table containing 10,000 visible rows is expensive even if every component is neatly organized.

Computation cost comes from JavaScript occupying the main thread. Sorting large arrays, parsing CSV files, filtering records, aggregating chart data, and transforming API responses can all delay keyboard input, scrolling, and rendering.

Network cost usually appears as an oversized response or a chain of dependent requests. A 4 MB JSON payload is one kind of problem; twelve sequential API calls are another.

Media cost becomes significant when an application loads large images, videos, or 3D assets before they are needed—or serves them at a higher resolution than the interface can display.

Memoization has a narrower role than it is often given. React.memo and useCallback can prevent avoidable component updates, while useMemo can prevent the same calculation from being repeated when its dependencies have not changed. None of them makes an unavoidable 400 ms transformation cheap, reduces a large API response, or removes thousands of DOM nodes.

Start with the React Profiler

Record the interaction that feels slow: changing a filter, sorting a table, opening a tab, or selecting a chart range.

The React DevTools Profiler shows which components rendered, how long the commit took, and what triggered the update. A wide flame graph containing hundreds of component renders usually points to a rendering problem. Depending on what the trace shows, the answer may be virtualization, more stable props, narrower state subscriptions, or selective memoization.

The same profiling model applies in React 18 and 19, so upgrading React does not change the basic diagnostic workflow.

Check what is blocking the main thread

Record the same interaction in the Chrome DevTools Performance panel. Look for long tasks—work that occupies the main thread for 50 ms or more.

A long JavaScript block during a calculation, without a corresponding burst of React rendering, suggests a computation bottleneck. Virtualizing a list will not fix it. The calculation may need to be reduced, scheduled differently, or moved to a Web Worker.

Chrome's Long Animation Frames API, available since Chrome 123, adds more detail by attributing slow frames to the scripts responsible for them. That makes it useful when investigating poor Interaction to Next Paint results.

Read the request waterfall

The Network panel answers two different questions:

How much data is being transferred?
Are requests happening in parallel or waiting unnecessarily for one another?

A single 3 MB response calls for pagination, compression, or a smaller field selection. A diagonal sequence of independent requests suggests a waterfall that could be flattened. Client-side caching may improve repeat navigation, but it does not make the first oversized response smaller.

The central distinction is simple: are there too many elements to render, or is one piece of JavaScript preventing the browser from rendering at all? The React Profiler is best suited to the first question; the Performance panel helps answer the second.

Match the symptom to the fix

The techniques in this article are complementary, but they are not interchangeable.

If the page janks while scrolling through a long table or list, the likely cause is rendering overload created by thousands of DOM nodes. Start with list or table virtualization using react-window, TanStack Virtual, or React Virtuoso. Keep in mind that off-screen content will not be present in the DOM, so accessibility, find-in-page, and SEO may require additional work.
If the interface freezes during sorting, filtering, parsing, or aggregation, heavy computation is probably blocking the main thread. Try reducing the amount of work or moving the calculation to a Web Worker. Workers add serialization overhead, cannot access the DOM, and require additional coordination code.
If a large dataset is repeatedly downloaded or kept in memory, consider persisting suitable data in IndexedDB with Dexie.js, idb, or RxDB. This reduces repeated loading and memory pressure, but the application must handle browser eviction, synchronization, schema changes, and stale data.
If the same requests run repeatedly or form a network waterfall, use TanStack Query or SWR together with data prefetching and parallel requests. The trade-off is that caching introduces questions around invalidation, staleness, and memory consumption.
If images, videos, or 3D assets delay interaction, start with compression, right-sizing, lazy loading, and intent-based preloading. Placeholders require additional UX work, while excessive prefetching can waste bandwidth.

A complex application may need several of these changes. A large 3D product viewer, for instance, might use a Worker for calculations, IndexedDB for model data, virtualization for property lists, and preloading for media. The important part is that each technique addresses a measured cost.

Virtualization, pagination, or both?

Virtualization and pagination both limit how much work the browser performs, but they change the product in different ways.

Virtualization maintains the experience of one continuous dataset. It renders only the rows inside or near the viewport and reuses DOM nodes as the user scrolls. A table may contain 50,000 records while keeping only a few dozen row elements mounted.

That makes virtualization a natural fit for authenticated dashboards, chat histories, log viewers, data grids, and infinite feeds. The compromise is that off-screen items do not exist in the DOM. Browser find-in-page cannot discover them, crawlers cannot index them, and some assistive-technology workflows require deliberate focus and navigation handling.

Pagination places a fixed number of items on each page and provides controls for moving between pages. It is usually easier to make crawlable, linkable, and accessible. Public search results, product categories, article indexes, and other SEO-relevant listings generally benefit from this model.

A hybrid approach is often the practical answer. The server returns a bounded page of records, and the client virtualizes that page if it is still large. Payload sizes remain controlled, while the browser avoids mounting more rows than the user can see.

Remember that grids can require two forms of virtualization. Row virtualization helps when there are thousands of records. Column virtualization matters when a financial or analytical grid contains dozens of columns. A library that handles simple lists well may not be equally capable with two-dimensional grids, sticky columns, variable row heights, and keyboard navigation.

Virtualization lets the user scroll through one continuous list while keeping only the visible rows in the DOM. It works well for dashboards, feeds, logs, chat histories, and authenticated applications. The downside is that off-screen content may not be indexed by search engines, while accessibility and browser find-in-page require additional work.
Pagination divides results into separate pages and keeps the number of DOM elements under control. Pages can be indexed, linked, and shared individually. It is usually the better choice for public listings, product directories, article indexes, and search results.
Server pagination combined with client-side virtualization provides a middle ground. The server returns a limited page of data, while the client renders only the visible part of that page. This works well for large authenticated datasets and complex data grids, although it requires managing both pagination and virtualization.

For libraries, react-window remains a focused option for straightforward lists and is actively maintained. Its older sibling, react-virtualized, is rarely the first choice for a new implementation.

TanStack Virtual is headless, which suits teams that want complete control over markup and styling or already use the TanStack ecosystem. React Virtuoso provides more behavior out of the box and is particularly useful for variable-height content.

The library decision should follow the interface requirements. A basic fixed-height list does not need the same machinery as an editable grid with pinned columns, dynamic heights, grouped rows, and accessible keyboard navigation.

Move expensive calculations to Web Workers

When a trace shows a long calculation without a corresponding rendering spike, moving that work off the main thread can restore responsiveness. A Web Worker can sort, parse, aggregate, or transform data while the browser continues to process input and paint frames.

Workers have firm boundaries. They cannot access the DOM, window, or localStorage. They can use fetch, WebSockets, and IndexedDB, which means a worker can retrieve, transform, and persist a large dataset without routing every step through the UI thread.

Messages between the main thread and a worker normally use the structured-clone algorithm. Large values must be copied, and that serialization cost can erase part of the expected performance gain.
Transferable objects avoid that copy for resources such as ArrayBuffer. Ownership moves from one thread to the other, leaving the original buffer unusable on the sending side. For binary files, large typed arrays, and some data-processing pipelines, this can be considerably cheaper than cloning.

SharedArrayBuffer allows genuine shared memory, but it introduces stricter deployment requirements. The page must be cross-origin isolated using COOP and COEP headers, which may conflict with embedded third-party content. Most applications can use transferable objects without taking on that constraint.

The native postMessage interface is manageable for one or two worker operations but becomes tedious as the API grows. Comlink, a small library from Google Chrome Labs, wraps worker communication in a proxy so asynchronous worker functions can be called more like ordinary methods.

A worker pool can help when the application runs many independent jobs. Instead of creating a worker for every task, the application maintains a small number of workers and feeds them work from a queue. The pool size still needs restraint: saturating every available core may speed up the calculation while competing with the browser for resources.

Workers do not reduce layout or painting cost. A page that mounts 10,000 rows will remain expensive after its data transformation moves off-thread. Workers address computation; virtualization addresses rendering.

Use IndexedDB when the browser needs a real local data store

React state is not persistent storage, and localStorage is poorly suited to large datasets. It stores strings, operates synchronously, and commonly offers only a few megabytes per origin. Reading or parsing a large value can block the main thread before the application renders anything useful.

IndexedDB is asynchronous and transactional, with a practical capacity far beyond localStorage, depending on the browser, device, and available disk space. It can store objects, typed arrays, Blob values, and files without manually converting everything to JSON.

That makes it useful for offline-first and local-first products, large working sets that must survive a reload, and expensive data that can be reconstructed but should not be fetched on every visit.

The native IndexedDB API is verbose and event-driven, so most React applications use a wrapper:

Dexie.js provides a promise-based database and query layer.
idb stays close to the native API while replacing the event-based interface with promises.
RxDB adds reactive queries and replication for applications that need live synchronization.

Choosing IndexedDB changes more than storage. Once a working copy lives in the browser, the application needs rules for refreshing it, resolving conflicts, migrating schemas, and distinguishing fresh data from stale data. A local-first architecture is a data-model decision, not a larger version of a cache.

Eviction also has to be expected. Browsers may remove storage under pressure. Safari applies a particularly strict tracking-prevention policy under which script-writable storage, including IndexedDB, can be deleted after seven days without user interaction with the site.

Unless the product explicitly guarantees durable browser storage, treat IndexedDB as a local copy that can be rebuilt. The server—or another controlled persistence layer—should remain the source of truth.

Cache repeat work and remove network waterfalls

After rendering and computation have been addressed, repeated network work often becomes the next visible delay.

TanStack Query and SWR solve several problems that otherwise end up scattered through components: request deduplication, cached results, background refreshes, retry behavior, and stale-data handling. They can return an existing result immediately during repeat navigation instead of showing another loading state for data the user has already seen.

TanStack Query's defaults deserve attention. It considers fetched data stale immediately by default, while unused cache entries are garbage-collected after five minutes. Those settings are safe but can produce unnecessary refetches when the underlying data changes infrequently. staleTime should reflect how quickly each resource can become outdated, not one global guess for the entire application.

Both libraries support the stale-while-revalidate pattern: return cached data first, verify it in the background, and update the interface if the response has changed. It improves perceived performance because repeat visits do not begin with an empty screen.
Prefetching can remove the remaining wait from predictable navigation. Route data can be requested when the user hovers over a link, focuses a control, or reaches a point where the next action is likely. Prefetch too aggressively, however, and the application downloads data the user never requests.

Request structure matters just as much as caching. Independent calls should begin together through Promise.all, parallel queries, or a backend endpoint designed to aggregate them. A five-request waterfall cannot be corrected by memoizing the component that waits for it.
Media benefits from the same discipline. A gallery, 3D viewer, or image-heavy customer interface can preload the next likely asset, retain previously fetched media, and lazy-load everything outside the active view. Service-worker caching through Workbox can extend the approach to unreliable connections and offline use.

Measure whether the change helped

A performance change is only useful if it improves the interaction users actually perform.

Interaction to Next Paint (INP) replaced First Input Delay as a Core Web Vital in March 2024. Unlike FID, which evaluated only the first interaction, INP considers interactions across the visit. A result of 200 ms or less at the 75th percentile is considered good.

That makes INP relevant to data-heavy React products, where roughly 90% of a user's time may occur after the initial load. A dashboard can achieve an excellent Largest Contentful Paint score and still feel slow every time someone filters a table.

Core Web Vitals will not capture every product-specific operation. “Time to render 10,000 rows,” “time to display the first interactive chart,” and “duration of a CSV import” require custom instrumentation.

The Performance API provides the necessary primitives:
performance.mark("filter-start");

// Run the operation.

performance.mark("filter-end");
performance.measure("filter-duration", "filter-start", "filter-end");

Benchmark before and after under the same conditions: identical dataset, device profile, browser version, and network throttling. Comparing a developer laptop with a production user's older machine produces an impressive number, but not a useful one.

Lab and field measurements answer different questions. Lighthouse and local traces provide controlled, repeatable feedback while engineers iterate. Real-user monitoring, CrUX, and the web-vitals library show what happens across the hardware and network conditions users actually have. Lab data helps isolate the change; field data confirms that the improvement survived production.

How these techniques combine in production

The strongest implementations rarely depend on one optimization.

Vitus, a construction-intelligence platform that works with several large 3D models, combined Web Workers, list virtualization, and IndexedDB. Model calculations moved off the main thread. React Virtuoso handled long property and data lists because pagination did not fit the product requirements. Dexie.js stored large model data locally instead of keeping the entire working set in React state.

A separate change—from the previous model format to SVF2 with parallel loading—cut model size roughly 5× and load time about 2.5×. That result came from addressing several layers: computation, rendering, storage, and network transfer.

Datasport, a sports-event platform with chart- and table-heavy dashboards, needed a different combination. Large tables were virtualized because pagination was not suitable, GET requests were cached, and component memoization reduced unnecessary rendering. React Context replaced Redux in areas where a smaller state layer was sufficient. Web Workers were not part of the solution because main-thread computation was not the defining bottleneck.

The Sales Platform, a media-heavy fashion-tech interface, paired virtualization with a dedicated media layer. Virtualized rendering reduced the number of mounted elements by about 70%, while preloading and caching reduced wait times by 50%. SWR managed data caching, and Workbox supported service-worker caching and offline behavior.

The same constraints appear in analytics products. GazeHealth, a mental-health analytics platform, renders a custom D3 Sankey diagram alongside roughly fifteen data visualizations over large datasets. GoodShape, an employee-wellness platform, combines complex charts with AG Grid tables and optimized large-data processing.

The projects use related tools, but not identical stacks. That is the useful lesson: performance work begins with the bottleneck, not with a preferred library.

When specialized performance engineering makes sense

One slow table or blocking filter is usually a bounded problem. An experienced engineer can profile the interaction, identify the expensive work, and implement a focused fix without redesigning the application.

The decision changes when performance problems cross architectural boundaries. Worker APIs, client-side databases, synchronization rules, data-fetching layers, and two-dimensional virtualization all create decisions that other parts of the product will depend on. Those changes need ongoing ownership.

Capacity matters as well. A team may understand the problem but lack time to investigate it without delaying product work. Engineers who have already implemented virtualized grids, worker pipelines, and IndexedDB-backed data layers can often recognize the performance signature faster than a team encountering it for the first time.

External help is most useful when the scope is explicit. A short engagement can address a measured bottleneck. A dedicated team is more appropriate when performance work affects the application's data architecture and will continue across multiple releases.