DEV Community: Utku Akyüz

Virtualizing Huge JSON Files in React: The Technical Breakdown

Utku Akyüz — Wed, 17 Dec 2025 08:07:00 +0000

In my previous article, I detailed the journey that led me to combine the strong logic of json-diff-kit with a completely custom rendering solution. The goal was to build a component that could handle millions of lines of JSON diff data and provide a responsive, intuitive UX. All other existing tools failed to deliver.

This article breaks down the three core engineering challenges I overcame to build virtual-react-json-diff.

Challenge 1: High-Performance Rendering (Goodbye Table, Hello Grid)

The first step was abandoning the rigid <table> structure, which makes both styling and virtualization extremely complex.

I replaced the table structure with a CSS Grid-based diff architecture. This allowed for fluid layouts and vastly improved performance metrics compared to traditional table rendering overhead.

Here is the essential structure for each row component:

<div
  style={{
    display: "grid",
    gridTemplateColumns: "30px 1fr 30px 1fr",
  }}
>
  <div className="cell line-number">{leftPart.lineNumber}</div>
  <div className="cell">{leftPart.text}</div>
  <div className="cell line-number">{rightPart.lineNumber}</div>
  <div className="cell">{rightPart.text}</div>
</div>

The 30px 1fr 30px 1fr template creates four columns (left line number, left content, right line number, right content), eliminating table layout overhead and simplifying the virtualization process via react-window.

Challenge 2: Virtualization vs. Tree Structures (The Conflict)

This was the most complex hurdle. Virtualization libraries like react-window require stability—they perform best when row heights are known and fixed. However, a JSON viewer must allow users to expand and collapse nested objects constantly.

Every time a user collapses a 5,000-line object into a single line, the virtualizer's internal calculations for total scroll height and item positioning break.

The Dynamic Segment-Based Solution

Since I couldn't find a library that offered both deep tree navigation and high-performance virtualization, I engineered a solution myself.

I built a dynamic registry using a segment-based rendering system that separates the raw diff data from the view representation.

Key Data Structures:

Raw diff data: Complete diff results from json-diff-kit.
Segments: Logical groupings that determine what content is currently visible/collapsed.
View arrays: Flattened arrays fed to the virtualizer.

Equal lines are grouped into collapsible segments:


// Part of the core logic to group non-changed lines
export function generateSegments(diff: DiffResult[]): SegmentItem[] {
  const segments: SegmentItem[] = [];
  let i = 0;
  while (i < diff.length) {
    if (isNonEmptyEqualLine(diff[i])) {
      let j = i;
      while (j < diff.length && isNonEmptyEqualLine(diff[j])) j++;
      segments.push({ start: i, end: j, isEqual: true });
      i = j;
    } else {
      segments.push({ start: i, end: i + 1, isEqual: false });
      i++;
    }
  }
  return segments;
}

When a user expands a collapsed segment, the system updates the segments and forces the virtualizer to recalculate:


useEffect(() => {
  const leftBuilt = buildViewFromSegments(segments, rawLeftDiff);
  const rightBuilt = buildViewFromSegments(segments, rawRightDiff);
  setLeftView(leftBuilt);
  setRightView(rightBuilt);
  
  // Critical: Recalculate virtualizer without visual jitter
  listRef.current?.resetAfterIndex(0, true); 
}, [segments, rawLeftDiff, rawRightDiff]);

This dynamic registry allows the viewer to feel like a lightweight text editor, even while managing millions of hidden nodes, maintaining the desired 60 FPS performance.

Challenge 3: Navigation and Search (Don't Get Lost)

Navigating a 50,000-line diff requires tools beyond a standard scrollbar.

The Dual-Minimap

I implemented a dual-minimap system (one for each object) that highlights the changes and synchronizes precisely with the main scrollbar.

The Canvas Optimization: The minimap uses an offscreen canvas for efficient rendering. The diff visualization (coloring the lines) is pre-rendered once (the expensive part). During scrolling, only the lightweight scroll viewport box is composited on top, ensuring smooth performance.


// Pre-render diff on offscreen canvas (only when diff changes)
const diffCanvas = useMemo(() => {
  // ... logic to draw all colored diff lines on canvas
  // The color for each line is based on its type (add, remove, change)
  return offscreen;
}, [leftDiff, rightDiff, height, totalLines]);

// Draw scroll viewport (on every scroll)
const drawScrollBox = useCallback((ctx: CanvasRenderingContext2D) => {
  ctx.clearRect(0, 0, miniMapWidth, height);
  ctx.drawImage(diffCanvas, 0, 0); // Copy pre-rendered diff

  const totalContentHeight = totalLines * ROW_HEIGHT;
  const viewportTop = (currentScrollTop / totalContentHeight) * height;
  ctx.fillStyle = MINIMAP_SCROLL_COLOR;
  ctx.fillRect(0, viewportTop, miniMapWidth, viewportHeight);
}, [diffCanvas, currentScrollTop, totalLines]);

Search & Indexing

Finding a specific value in a virtualized list is hard because the DOM elements might not exist yet.

I solved this by pre-indexing all search results and building a custom scroll-to-row mechanism that interacts directly with react-window's methods.


// When navigating matches, we scroll the virtualizer directly to the indexed position
const navigateMatch = useCallback((direction: "next" | "prev") => {
  const newIndex = direction === "next"
    ? (searchState.currentIndex + 1) % searchState.results.length
    : (searchState.currentIndex - 1 + searchState.results.length) % searchState.results.length;

  const matchIndex = searchState.results[newIndex];
  listRef.current?.scrollToItem(matchIndex, "center");
}, [searchState]);

Search results are also visualized on the minimap by overlaying highlights on the pre-rendered canvas.

Conclusion: A Solution Finally Exists

I kept the package management simple, but open to improvement. The process of code review and cross reviewing with AI tools like Gemini to optimize some logic was incredibly enjoying.

If you are facing a similar problem, where scale is the bottleneck, you don't need to hit the same walls I did. A React Component for JSON comparison is here now.

The virtual-react-json-diff package offers:

Performance: Virtualized rendering for millions of lines.

UX: Minimaps, Search, and clean navigation.

Accuracy: Key-based comparison for arrays.

See the demo and the code
GitHub
Virtual-React-Json-Diff Demo
npm

Hidden Journey Behind virtual-react-json-diff

Utku Akyüz — Wed, 17 Dec 2025 08:03:00 +0000

The Tool I Needed Didn't Exist (And Why I Had to Build It)

Like most of you, I was working on an enterprise-level React/TypeScript project when my client requested a feature that sounded very simple:

"We need to compare two database JSON objects on screen and visualize the differences."

At first glance I thought, "Easy." I would just do a quick search on GitHub or NPM, choose a component library that handles comparison and visualization, and done.

However, after hours of searching, testing demos, and scrolling through issue trackers, I slowly realized an uncomfortable truth.

The tool I needed simply did not exist.

This story is about a huge data comparison challenge, a tight deadline, and how I ended up building the solution myself, resulting in the virtual-react-json-diff NPM package.

Chapter One: The Confidential Project That Started It All

I can't share details about the client project, but the requirement was clear: Compare two versions of database objects from different system states.

The catch? The scale. These are not small config files. These were massive objects. Sometimes a hundred rows, sometimes hundreds of thousands of nested items. When expanded as JSON, we were looking at millions of lines.

The client wanted a clean JSON diff viewer. No specific UI constraints, but it had to be fast.

Chapter Two: Hitting the Wall with Existing Solutions

I started testing every package that came close to "JSON diffing." The results were consistently disappointing:

1. UI and UX Limitations

Most existing components designed for small config files (100-200 lines). They lacked visual hierarchy, navigation aids, or scalable structures for massive datasets.

2. Performance Collapse

Almost none of them could handle large structures. Anything above a few thousand lines caused the interface to freeze or the browser to crash entirely.

3. The Contenders (and why they failed)

react-diff-viewer: Works nicely for small, line based diffs (like Git changes). But for structured JSON? It collapses instantly under the weight of nested objects.
jsondiffpatch: An amazing algorithm under the hood, but the default UI feels extremely dense with very little visual guidance. It was unusable for non-technical users viewing large datasets.
json-diff-kit: This was the turning point. The underlying algorithm was very strong, especially for nested objects. However, the UI was based on HTML Tables.

The Problem with Tables: HTML tables are notoriously difficult to virtualize effectively. When you throw 100,000 rows into a table, the DOM size explodes, and the browser dies.

Chapter Three: The Pivot and an Open Source Contribution

I realized json-diff-kit had the most accurate comparison logic. The creator had developed a solid algorithm that offered several comparison methods.

I made a decision: I would use json-diff-kit for the core logic but completely rewrite the rendering layer.

The Array Key Problem (Fixing the Core Logic)

Then came a critical edge case: comparing arrays of objects. Without a unique key (id), a standard diff algorithm compares by index. If an item is deleted from the middle, the entire diff becomes a mess of "changes" rather than a single deletion.

For an example you can see the issue

I needed to compare arrays based on a specific key (e.g., oid or id). Since no package offered this, I implemented it inside json-diff-kit and sent a Pull Request.

The maintainer reviewed and merged it quickly. This became my first serious open-source contribution.

The key-based comparison is now available via the compare-key method:

const differ = new Differ({
  arrayDiffMethod: "compare-key",
  compareKey: "userId", // or "id", "oid", etc.
  detectCircular: true,
  maxDepth: 20,
});

This allows accurate diffing of arrays where items can be added, removed, or reordered, rather than naive index-based comparison.

What's Next? Going Beyond "Good Enough"

My client was happy. The requirements were met:
✅ No freezing.
✅ Millions of rows rendered (internally).
✅ Key-based array comparison.

But I realized that if I struggled this much to find a tool, others probably were too. I wanted to create a component that wasn't just "good enough" for my client, but useful for the community.

The biggest challenge remained: How do you replace a rigid HTML Table with a fluid, high-performance, and virtualized component that can handle expanding and collapsing a tree of potentially millions of nodes without breaking?

That technical deep dive is in the next article.

=> Continue to the Deep Dive: Virtualizing Millions of JSON Lines at 60 FPS: The Technical Breakdown

See the demo and the code
GitHub
Virtual-React-Json-Diff Demo
npm