Under the Hood: Building a 100% Client-Side CSV Deduplicator with Angular 18+ Signals

#webdev #programming #javascript #angular

Tabular data cleaning is one of the most common prep steps in data engineering and business ops. However, pasting a CSV containing sensitive emails, customer IDs, or corporate financials into a third-party server poses massive security risks.

To solve this, we built a 100% client-side CSV Row Deduplicator on tools.kandz.me. By processing everything in the browser sandbox, your data never touches a server.

But parsing and deduplicating CSV format reliably in raw JavaScript is trickier than a simple string.split(','). Here is a deep dive into how it works under the hood.

1. The Core Architecture & Challenges

At first glance, splitting CSV lines seems trivial. But standard production CSVs present major parsing edge cases:

Quoted Values with Commas: A field like "Doe, John" shouldn't be split at the comma.
Header Preservation: The column names (first row) must remain at the top and never get removed as "duplicates," even if an identical row appears deep inside the dataset (e.g., if multiple CSV files were merged together).
Whitespace Variations: John Doe, john@example.com and John Doe,john@example.com (notice the space) should be evaluated as duplicate rows if white spaces are trimmed.

2. Breaking Down the Parser

To handle quoted cells containing commas, we implement a stateful parser loop instead of a basic .split(',').

function parseCSVLine(line: string): string[] {
  const result: string[] = [];
  let current = '';
  let inQuotes = false;

  for (let i = 0; i < line.length; i++) {
    const char = line[i];
    if (char === '"') {
      inQuotes = !inQuotes; // Toggle quotes state
    } else if (char === ',' && !inQuotes) {
      result.push(current);
      current = '';
    } else {
      current += char;
    }
  }
  result.push(current);
  return result;
}

3. Protecting the Header & Cleaning Merged Files

A major logical bug in naive deduplicators is how they handle the First Row is Header flag. If you shift the header row out of the array before processing, its key signature is never tracked. If an accidental duplicate of that header appears later in the body (very common when concatenating tables), it gets processed as a unique data row.

To fix this, we generate the key signature of the shifted header and add it to our seen tracking set before processing the body:

const seen = new Set<string>();

if (hasHeader && headersRow) {
  let headerKey = '';
  if (colIdx >= 0) {
    headerKey = isTrim ? headersRow[colIdx].trim() : headersRow[colIdx];
  } else {
    const cleanHeader = isTrim ? headersRow.map(c => c.trim()) : headersRow;
    headerKey = cleanHeader.join('|||');
  }
  seen.add(headerKey.toLowerCase());
}

This ensures any repeated headers inside the data rows are instantly stripped out, keeping only the true, anchored header at the top of the output.

4. Reactive State management with Angular Signals

We use Angular 18+ reactive Signals and computed properties to keep calculations fast and clean. When the user changes any parameter (such as toggling case-sensitivity or choosing to deduplicate by a specific column), the engine automatically recalculates the output without manual event listeners or dirty DOM rendering.

By using native input bindings ([value]="csvInput()" and custom target update events) we bypass standard latency, ensuring smooth performance even with large files.

5. Summary

By keeping everything client-side, we combine ultimate data privacy with rapid execution. If you need to clean up messy datasets, give the tool a run.

Try the live tool: tools.kandz.me/csv-deduplicator