Stop Letting Dirty Data Break Your Knowledge Graph: Mastering Entity Resolution & Deduplication in TypeScript

Enterprise knowledge graphs are rarely born pristine. They are typically assembled from fragmented, noisy, and heterogeneous data streams. If you are building modern enterprise AI applications, you have likely run into the ultimate data engineering nightmare: the transition from stochastic language models to zero-hallucination architectures demands absolute structural and semantic integrity. Yet, your underlying data sources are actively fighting against you.

Imagine an enterprise entity representing a multinational corporation. In your European billing ledger, it appears as "Acme Corp Ltd." In your North American procurement system, it is logged as "ACME CORPORATION." Meanwhile, in an unvalidated customer support ticket, a service agent manually typed it in as "Acme C."

To a naive relational database—or worse, a stochastic Large Language Model prone to hallucinations—these variations look like three entirely distinct entities. But to an enterprise Neuro-Symbolic architecture, this is a critical systemic failure known as ontological fragmentation.

If you want to build systems that reliably compute credit risk, trace regulatory exposure, or map supply-chain dependencies without making things up, you need a bulletproof strategy. You need a triad of algorithms: Entity Resolution, Link Prediction, and Graph Deduplication.

Let’s dive deep into how you can implement a deterministic, zero-hallucination data ingestion and deduplication pipeline entirely in TypeScript, without relying on probabilistic guesswork.

---

The Anatomy of Enterprise Identity Fragmentation

To understand why entity resolution is mathematically challenging, we have to look at how identity is modeled across distributed information systems.

In standard web development and modern microservice architectures, dependency resolution frameworks—like npm or yarn—guarantee that package identities are tracked deterministically. They use semantic versioning and cryptographic package locks (package-lock.json or yarn.lock). If two services require lodash@4.17.21, the package manager resolves these strings to an exact cryptographic hash and a specific directory layout in node_modules, entirely eliminating ambiguity.

Enterprise data, however, lacks a global cryptographic lockfile. There is no universal primary key that unifies a customer ID in Salesforce with a UUID in a custom PostgreSQL database and a billing string in Stripe. When these datasets are synchronized into an enterprise knowledge graph, unique real-world entities—people, organizations, devices, or transactions—manifest as disconnected nodes, or worse, conflicting subgraphs.

Consider this fragmented ingestion stream:

• [System A: CRM] Customer #49281 | Name: "Johnathan Doe" | Email: "[e-mail]"
• [System B: Billing] Invoice ID: INV-9938 | Name: "John D." | Email: "[e-mail]"
• [System C: Support] Ticket Author: "[e-mail]" | Name: "J. Doe" | IP: 192.168.1.55

If we construct an initial graph directly from these ingestion streams, our graph database contains three distinct nodes for what is logically a single human being. This structural illusion cascades into downstream reasoning engines. Edges representing financial liabilities, communications, and contracts split across three disconnected components.

Entity Resolution is the systematic process of discovering that these disconnected nodes refer to the same real-world phenomenon and merging them into a single Canonical Entity governed by a rigid ontological URI.

---

String Metrics and Record Linkage: Beyond Exact Matching

Our first line of defense in entity resolution is record linkage using string similarity metrics. In a purely deterministic paradigm, we cannot rely on LLMs to "guess" if two names match. LLMs introduce semantic drift and hallucination risks. Instead, we must employ mathematically rigorous string distance algorithms that calculate edit distances, token intersections, and phonetic alignments.

The Power of Jaro-Winkler

Consider the Jaro-Winkler distance metric, which is heavily utilized in record linkage pipelines. Unlike standard Levenshtein distance (which counts the minimum number of single-character insertions, deletions, or substitutions required to transform one string into another), Jaro-Winkler is specifically designed for short strings like personal names and corporate titles. It places a higher weight on prefixes, acknowledging that human errors and system truncations are far more likely to occur at the end of a string than at its beginning.

When implemented within a TypeScript-based deterministic data ingestion pipeline, these metrics allow us to evaluate millions of node pairs. However, performing an all-pairs comparison ( $O(N^2)$ complexity) across an enterprise database containing tens of millions of records is computationally intractable.

To solve this, enterprise architectures employ Blocking and Canopies. Blocking partitions the dataset into smaller, overlapping buckets based on hard deterministic keys (such as the first three characters of a postal code, the phonetic soundex of a last name, or the primary domain of an email address). Two records are only subjected to expensive Jaro-Winkler calculations if they share at least one blocking key, reducing search time from quadratic to near-linear.

---

Transductive Link Prediction and Graph Embeddings

Once explicit entity resolution has merged obvious duplicates, your knowledge graph will still contain missing edges. In complex enterprise networks, relationships are often latent. Two corporate entities might share identical board members, subsidiaries, patent citations, and procurement patterns, yet lack an explicit corporate-parent edge in the database.

Discovering these missing ontological edges without relying on stochastic text generation requires Transductive Link Prediction via graph structural learning.

Think of this like a high-performance hash map in TypeScript. In backend applications, searching through an unindexed array of objects for a specific ID is an $O(N)$ linear scan. We solve this by projecting our objects into a Hash Map (or a Map<string, T> data structure), where keys map directly to memory buckets via a hashing function, reducing lookup time to $O(1)$ .

Graph embeddings perform a conceptually similar transformation, but for topology and semantics. They project high-dimensional, discrete, non-Euclidean graph structures into a low-dimensional, continuous vector space ( $\mathbb{R}^d$ ).

Just as a hash map preserves equality checks, a high-quality graph embedding model (such as TransE, DistMult, or Node2Vec) preserves structural and relational proximity. If two entity nodes occupy similar topological neighborhoods, their resulting dense vectors will be geometrically close to one another in the continuous vector space.

In transductive link prediction, we train an embedding model on the existing graph structure. The model learns scoring functions for triples $(h, r, t)$ —head entity, relation, and tail entity. If an edge is missing in the discrete graph, but vector arithmetic yields a high confidence score, our deterministic solver can assert that the edge exists with quantifiable mathematical certainty.

---

Graph Deduplication and Ontological Constraint Satisfaction

Deduplication is not merely a string-matching exercise; it is an ontological constraint satisfaction problem. When two entity nodes are identified as duplicates, merging them requires navigating complex graph schemas, multi-valued attributes, and inverse relations without violating the graph's formal ontology (defined in OWL or RDF-S schemas).

Think about managing state transitions in a complex agentic workflow engine, such as LangGraph. A StateGraph maps deterministic computational nodes and state transitions around a mutable, shared Graph State. When an agentic workflow executes, multiple parallel worker nodes might attempt to mutate the shared state simultaneously. Without a strict reducer function or deterministic orchestration handled by a Supervisor Node, race conditions and conflicting state updates corrupt the graph state.

Graph deduplication operates on an identical principle, but applied to structural topology rather than runtime memory. When our deduplication pipeline determines that Node A and Node B are the same entity:

1. Incoming Edges: All incoming edges pointing to Node B must be redirected to Node A.
2. Outgoing Edges: All outgoing edges originating from Node B must be re-anchored to Node A.
3. Literal Properties: Scalar attributes must be reconciled using deterministic conflict-resolution policies (e.g., "most recent timestamp wins" or "highest confidence source wins").

If this merging process is executed without regard for ontological constraints, it can result in logical contradictions. For example, suppose our ontology specifies that a Person can have exactly one primary social security number (ssn), and that the ssn property is functionally unique. If Node A has one SSN and Node B has another, merging them creates an immediate ontological integrity violation.

To prevent this, enterprise graph deduplication pipelines implement Constraint-Driven Merging. Before any merge transaction is committed to the GraphDB, a rule-based validation engine evaluates the proposed canonical state against the ontology. If a functional property violation or disjoint-class violation is detected, the automated merge halts, and the conflicting records route to a deterministic exception queue for human-in-the-loop review.

---

Production Implementation: Building a Deduplication Engine in TypeScript

Let’s put theory into practice. Below is a fully self-contained TypeScript implementation of a deterministic entity resolution pipeline tailored for a SaaS customer data platform. It uses the Jaro-Winkler string similarity algorithm alongside exact-match criteria to identify, merge, and deduplicate customer records without relying on hallucination-prone Large Language Models.

/**
 * Enterprise Knowledge Graph Deduplication & Entity Resolution Engine
 * 
 * This module provides deterministic record linkage and entity resolution 
 * for a SaaS customer profile graph, utilizing Jaro-Winkler string metrics
 * and deterministic property matching.
 */

interface RawCustomerRecord {
  sourceId: string;
  sourceSystem: 'crm' | 'billing' | 'support';
  fullName: string;
  email: string;
  phone: string;
  company: string;
}

interface CanonicalEntity {
  canonicalId: string;
  mergedSourceIds: string[];
  fullName: string;
  email: string;
  phone: string;
  company: string;
  confidenceScore: number;
}

/**
 * Computes the Jaro-Winkler similarity score between two strings.
 * Optimized for short strings like names and identifiers.
 */
function jaroWinklerSimilarity(s1: string, s2: string): number {
  if (s1 === s2) return 1.0;
  if (!s1 || !s2) return 0.0;

  const str1 = s1.toLowerCase();
  const str2 = s2.toLowerCase();
  const len1 = str1.length;
  const len2 = str2.length;

  const maxDist = Math.floor(Math.max(len1, len2) / 2) - 1;
  if (maxDist < 0) return 0.0;

  const match1 = new Array(len1).fill(false);
  const match2 = new Array(len2).fill(false);
  let matches = 0;

  for (let i = 0; i < len1; i++) {
    const start = Math.max(0, i - maxDist);
    const end = Math.min(i + maxDist + 1, len2);

    for (let j = start; j < end; j++) {
      if (!match2[j] && str1[i] === str2[j]) {
        match1[i] = true;
        match2[j] = true;
        matches++;
        break;
      }
    }
  }

  if (matches === 0) return 0.0;

  let transpositions = 0;
  let k = 0;

  for (let i = 0; i < len1; i++) {
    if (match1[i]) {
      while (!match2[k]) k++;
      if (str1[i] !== str2[k]) transpositions++;
      k++;
    }
  }

  const m = matches;
  const t = transpositions / 2;
  const jaro = (m / len1 + m / len2 + (m - t) / m) / 3.0;

  let prefix = 0;
  for (let i = 0; i < Math.min(4, Math.min(len1, len2)); i++) {
    if (str1[i] === str2[i]) {
      prefix++;
    } else {
      break;
    }
  }

  const p = 0.1;
  return jaro + prefix * p * (1.0 - jaro);
}

function normalizeEmail(email: string): string {
  return email.trim().toLowerCase();
}

function normalizePhone(phone: string): string {
  return phone.replace(/\D/g, '');
}

/**
 * Evaluates two customer records to determine if they represent the same real-world entity.
 */
function areRecordsMatching(recordA: RawCustomerRecord, recordB: RawCustomerRecord): boolean {
  const emailA = normalizeEmail(recordA.email);
  const emailB = normalizeEmail(recordB.email);
  if (emailA && emailB && emailA === emailB) {
    return true;
  }

  const phoneA = normalizePhone(recordA.phone);
  const phoneB = normalizePhone(recordB.phone);
  if (phoneA.length >= 10 && phoneB.length >= 10 && phoneA === phoneB) {
    return true;
  }

  const nameSimilarity = jaroWinklerSimilarity(recordA.fullName, recordB.fullName);
  const companySimilarity = jaroWinklerSimilarity(recordA.company, recordB.company);

  if (nameSimilarity >= 0.92 && companySimilarity >= 0.85) {
    return true;
  }

  return false;
}

/**
 * Executes a deterministic entity resolution and deduplication pass over an array of raw records.
 */
function resolveEntities(rawRecords: RawCustomerRecord[]): CanonicalEntity[] {
  const canonicalMap: Map<string, CanonicalEntity> = new Map();
  const assignedSourceIds: Set<string> = new Set();

  for (let i = 0; i < rawRecords.length; i++) {
    const currentRecord = rawRecords[i];

    if (assignedSourceIds.has(currentRecord.sourceId)) {
      continue;
    }

    let baseEntity: CanonicalEntity = {
      canonicalId: `canon_${currentRecord.sourceId}`,
      mergedSourceIds: [currentRecord.sourceId],
      fullName: currentRecord.fullName,
      email: currentRecord.email,
      phone: currentRecord.phone,
      company: currentRecord.company,
      confidenceScore: 1.0,
    };

    assignedSourceIds.add(currentRecord.sourceId);

    for (let j = i + 1; j < rawRecords.length; j++) {
      const targetRecord = rawRecords[j];

      if (assignedSourceIds.has(targetRecord.sourceId)) {
        continue;
      }

      if (areRecordsMatching(baseEntity, {
        sourceId: targetRecord.sourceId,
        sourceSystem: targetRecord.sourceSystem,
        fullName: baseEntity.fullName,
        email: baseEntity.email,
        phone: baseEntity.phone,
        company: baseEntity.company
      }) || areRecordsMatching(currentRecord, targetRecord)) {

        baseEntity.mergedSourceIds.push(targetRecord.sourceId);
        assignedSourceIds.add(targetRecord.sourceId);

        if (targetRecord.fullName.length > baseEntity.fullName.length) {
          baseEntity.fullName = targetRecord.fullName;
        }
      }
    }

    canonicalMap.set(baseEntity.canonicalId, baseEntity);
  }

  return Array.from(canonicalMap.values());
}

// ==========================================
// Execution Example (SaaS Customer Graph)
// ==========================================

const incomingStream: RawCustomerRecord[] = [
  {
    sourceId: "crm_101",
    sourceSystem: "crm",
    fullName: "Jonathan Smith-Wesson",
    email: "[e-mail]",
    phone: "[phone]",
    company: "Enterprise SaaS Corp"
  },
  {
    sourceId: "billing_202",
    sourceSystem: "billing",
    fullName: "Jon Smith-Wesson",
    email: "[e-mail]",
    phone: "5550192834",
    company: "Enterprise SaaS Inc."
  },
  {
    sourceId: "support_303",
    sourceSystem: "support",
    fullName: "Jonathan Smith",
    email: "[e-mail]",
    phone: "[phone]",
    company: "Acme Widgets"
  }
];

const resolvedGraphEntities = resolveEntities(incomingStream);
console.log(JSON.stringify(resolvedGraphEntities, null, 2));

---

Step-by-Step Logic Breakdown

1. Type Definitions & Data Ingestion Setup:
   The code establishes strict TypeScript interfaces (RawCustomerRecord and CanonicalEntity). This guarantees type safety as data flows from unverified external ingestion pipelines into memory, preventing runtime type-coercion errors common in dynamic graph processing.

2. Jaro-Winkler String Metric Calculation (jaroWinklerSimilarity):

   • Evaluates string closeness by determining lengths and calculating a maximum allowable matching distance.
   • Iterates through strings within a dynamic sliding window to tally matching characters and track character-order transpositions.
   • Applies the Winkler adjustment by inspecting up to the first four characters for an exact prefix match, scaling the final score to heavily favor records sharing identical name prefixes.

3. Normalization Helpers (normalizeEmail & normalizePhone):

   • normalizeEmail trims whitespace and forces lowercase conversion, neutralizing case-sensitivity issues.
   • normalizePhone uses regular expressions (/\D/g) to strip out formatting characters like dashes and parentheses, ensuring deterministic comparison across systems with different formatting standards.

4. Deterministic and Heuristic Matching Logic (areRecordsMatching):

   • Rule 1: Checks normalized emails for absolute identity. If two records share an email, resolution is immediate and definitive.
   • Rule 2: Evaluates phone numbers if both are at least 10 digits long, bypassing formatting discrepancies.
   • Rule 3: Falls back to probabilistic heuristics if primary keys differ, combining name similarity (>= 0.92) and company similarity (>= 0.85) using Jaro-Winkler to link records across corporate name variations.

5. Entity Resolution & Clustering Loop (resolveEntities):

   • Maintains an assignedSourceIds tracking set to prevent records from being processed multiple times.
   • Iterates through the raw stream, taking an unassigned record as a seed (baseEntity).
   • Compares the seed against all subsequent unassigned records. If a match is flagged, the source ID is appended to baseEntity.mergedSourceIds, and fields update if higher-fidelity strings are detected.

---

Conclusion: Building Zero-Hallucination Pipelines

By combining string metrics, blocking algorithms, transductive link prediction, and ontological constraint checking, engineers can construct end-to-end data processing pipelines that operate with absolute mathematical predictability.

When you eliminate stochastic guesswork from your data ingestion layers, you ensure that your enterprise knowledge graphs remain clean, connected, and fully auditable. This provides a rock-solid foundation for zero-hallucination neuro-symbolic AI systems that your enterprise stakeholders can actually trust.

The concepts and code demonstrated here are drawn directly from the comprehensive roadmap laid out in the book Neuro-Symbolic AI & Knowledge Graphs, you can find it here. Check also the many other ebooks.