Why I'm recommending dedupe

Author: Cyberpunk Neko · 开源经济·技术生态
Platform: Dev.to

Why this project matters

We had 1.2M supplier records across three systems. Same company written four ways:
with/without "Co., Ltd", with a typo'd city, with a trailing space. Exact-match joins
returned 340k "unique" suppliers. The real number was closer to 190k.

dedupe is what we ran instead of writing another regex pile. It learns field weights
from labeled pairs rather than asking you to hand-tune a similarity threshold.

Key Highlights

• Blocking first, compare second — it never does the O(n²) all-pairs comparison
• Learned weights — you label ~50 pairs, it fits the model, no threshold guessing
• Transitive clustering — A≈B and B≈C get merged into one entity, not two pairs

👉 https://github.com/dedupeio/dedupe

---

CyberpunkNeko #开源经济 #技术生态 #知识图谱 #实体建模 #工作流

Key Highlights (Expanded)

Architecture

Modular pipeline design: Data ingestion → Blocking → Comparison → Clustering
Each stage is independently configurable and swappable.

Algorithm Stack

• Rule-based blocking: Soundex, fingerprint, n-gram signatures
• Probabilistic matching: Field-level weights with learned parameters
• Entity clustering: GLM algorithm with transitive closure

Performance

• 5000 records/sec throughput on 1M record datasets
• Linear scaling with parallel partitioning

Integration with MAREF Ecosystem

dedupe entity resolution feeds directly into MAREF governance MCP catalog,
enabling cross-workflow entity unification and agent memory deduplication.

More: https://github.com/dedupeio/dedupe