Processing Millions of Rows: Web Workers in Next.js ⚡

The Thread-Locking Text Filter

In analytical tracking rooms built at Smart Tech Devs, users expect fluid controls when auditing massive textual datasets—such as multi-megabyte app logs, JSON payloads, or data tables containing over 50,000 deep string elements. The default client framework reflex is running filter loops inside a standard React useMemo hook.

However, when a user rapidly types queries to match complex regex patterns against millions of characters, JavaScript's single **Main UI Thread** completely locks up. The browser cannot handle the heavy CPU calculation and update the screen at the same time. The typing indicator freezes, the mouse cursor stutters, and the application triggers a frustrating lag spell. To build a premium experience, you must offload string calculations to a background **Web Worker** and pair it with a **Virtualized List**.

Parallel Processing via Web Workers

Web Workers allow you to spin up entirely separate background execution environments that run parallel to the main browser thread.

By delegating heavy array filter loops to a worker, you completely isolate the computing strain. The user types at a locked 60fps in the input field while the background worker aggressively handles text searching. By combining this with structural viewport virtualization (rendering only the 20 visible items instead of all 50,000 elements into the DOM tree), your app can process massive logs instantly without hitting hardware limits.

Step 1: Architecting the Isolated Worker Script

We build a clean worker script that waits for data filtering events, handles the string mutations natively, and returns the result array.

// public/workers/filterWorker.js

self.addEventListener('message', (event) => {
    const { list, query } = event.data;

    if (!query) {
        self.postMessage(list);
        return;
    }

    // Run heavy regex filtering operations entirely in this background thread
    const lowercaseQuery = query.toLowerCase();
    const filtered = list.filter(item => 
        item.message.toLowerCase().includes(lowercaseQuery) ||
        item.level.toLowerCase().includes(lowercaseQuery)
    );

    // Return the calculated result array directly to the main thread
    self.postMessage(filtered);
});

Step 2: Connecting the Worker Inside a Next.js Component

Inside our client component, we manage the worker lifecycle, passing query changes downward and listening for the results safely.

// components/dashboard/LogAuditTerminal.tsx
"use client";

import React, { useState, useEffect, useRef } from 'react';

interface LogItem { id: string; message: string; level: string; }

export default function LogAuditTerminal({ rawLogs }: { rawLogs: LogItem[] }) {
    const [query, setQuery] = useState('');
    const [visibleLogs, setVisibleLogs] = useState<LogItem[]>(rawLogs);
    const workerRef = useRef<Worker | null>(null);

    useEffect(() => {
        // Initialize the parallel thread worker safely
        workerRef.current = new Worker('/workers/filterWorker.js');

        // Handle the arrival of computed search results
        workerRef.current.onmessage = (event: MessageEvent) => {
            setVisibleLogs(event.data);
        };

        return () => {
            workerRef.current?.terminate();
        };
    }, []);

    useEffect(() => {
        // Dispatch data to the background thread whenever the search query inputs change
        if (workerRef.current) {
            workerRef.current.postMessage({ list: rawLogs, query });
        }
    }, [query, rawLogs]);

    return (
        <div className="p-6 bg-white shadow rounded-xl border max-w-3xl">
            <h3 className="font-bold text-gray-800 mb-4">System Audit Trail Engine</h3>
            
            <input 
                type="text"
                value={query}
                onChange={(e) => setQuery(e.target.value)}
                placeholder="Filter millions of log lines instantly..."
                className="w-full p-3 border rounded-lg mb-4 focus:ring-2 ring-purple-500"
            />

            <div className="bg-gray-900 text-green-400 font-mono p-4 rounded-lg h-96 overflow-y-auto">
                {/* For production, wrap this mapping inside a Virtualized List container */}
                {visibleLogs.slice(0, 100).map(log => (
                    <div key={log.id} className="text-xs py-0.5 border-b border-gray-800">
                        <span className="text-red-400 font-bold">[{log.level}]</span> {log.message}
                    </div>
                ))}
                {visibleLogs.length === 0 && <p className="text-gray-500">No entries match criteria.</p>}
            </div>
        </div>
    );
}

The Client Thread ROI

By moving computing loops out of the main rendering loop, you completely eliminate input freezing. User keystrokes stay incredibly fluid because their input line never shares bandwidth with intense data filtration. Your web platform handles massive analytical tasks cleanly, enabling desktop-grade data interaction speeds right inside standard web viewports.