DEV Community: Tae Kim

Where I got labeled training data for Korean-Japanese-Chinese entity resolution

Tae Kim — Wed, 23 Sep 2026 03:11:35 +0000

Around pair 400 of the manual annotation batch, I got to a Korean holding company and its Japanese parent. The Japanese database had the company listed as a joint venture, the Korean filing described it as a wholly-owned subsidiary. Same entity, but the ownership description was genuinely different between sources. I marked it as a match since the names clearly referred to the same company, but I wrote a note that I still don't know what to do with.

That was about a month in. I ended up reviewing around 2,000 pairs by hand before I had enough coverage to feel reasonably confident in the KR-JP and KR-ZH models.

The reason it got to manual annotation: I couldn't find a public labeled dataset for Korean-Japanese-Chinese cross-lingual corporate name matching. For English-only ER, there are open sources. For this specific domain, I didn't find anything usable.

The Korean-English piece came together faster than I expected. Korean financial disclosure filings (DART) require companies to report both the Korean legal name and the English name. That turned out to be a clean source of verified KR-EN pairs. The company certified both names in a regulatory document, so I didn't have to guess whether they referred to the same entity.

def fetch_dart_name_pairs(company_code: str) -> dict:
    url = "https://opendart.fss.or.kr/api/company.json"
    params = {"crtfc_key": DART_API_KEY, "corp_code": company_code}
    resp = requests.get(url, params=params)
    data = resp.json()
    return {
        "ko": data.get("corp_name"),
        "en": data.get("corp_name_eng"),
    }

Scraping recent DART filings got me a few thousand KR-EN positive pairs. Combined with negatives generated by sampling companies from different industry sectors, the KR-EN Splink model got decent coverage early.

The KR-JP and KR-ZH pairs didn't have an equivalent source. I went looking for companies with cross-listed securities that would appear in databases across multiple languages. Found maybe 800-900 verified triplets that way. Not many, but enough to get an initial model to a usable state for the core coverage.

The 2,000 manual annotations were the gap-filler. Mostly companies in the near-threshold band of early model runs, which sounds principled but in practice meant sitting with a lot of cases where I wasn't sure. Japanese holding company naming conventions have edge cases I'd never thought about before. A company could be listed under a regional holding name in Japanese filings but appear under the global brand in Korean corporate registries. Not wrong, just a different layer of the corporate structure.

The KR-EN model is the one I trust most. The KR-JP and KR-ZH models have held up for the core coverage but I've seen them struggle on cases that weren't well represented in the training set. I add pairs when I find errors in production, but it's slow.

I build er-api, a multilingual entity resolution service for Korean, Japanese, Chinese, and English corporate data. More at hannune.ai.

What Splink actually runs on DuckDB when it scores entity pairs

Tae Kim — Tue, 22 Sep 2026 02:36:36 +0000

When I added Splink to the ER API, the registry was at 44,798 entities. Before touching any configuration knobs, I looked at the SQL it was generating. I expected opaque query strings. What came out was readable enough that I could follow the logic directly.

The blocking step generates something like this:

SELECT
    l.entity_id AS entity_id_l,
    r.entity_id AS entity_id_r,
    l.name_normalized AS name_l,
    r.name_normalized AS name_r
FROM entities AS l
INNER JOIN entities AS r
    ON LEFT(l.name_normalized, 4) = LEFT(r.name_normalized, 4)
WHERE l.entity_id < r.entity_id

LEFT(name_normalized, 4) is the blocking key. Entities that don't share those first four characters don't get compared at all. At 44,798 entities this brought candidate pairs from roughly 2 billion down to about 400,000.

The cross-join on the full entity table was what was slow. I caught this by looking at the DuckDB query plan while the blocking step ran — it was sitting there doing a lot more work than I wanted. Adding an index on the blocking key column fixed it immediately. Not complicated once I knew where to look, but I wouldn't have looked there without reading what was actually running.

What the EM pass is doing

For each candidate pair, Splink builds comparison features before the EM loop:

SELECT
    entity_id_l, entity_id_r,
    CASE WHEN name_l = name_r THEN 1 ELSE 0 END AS exact_match,
    jaro_winkler_similarity(name_l, name_r) AS jaro_winkler_sim,
    CASE WHEN country_l = country_r THEN 1 ELSE 0 END AS same_country
FROM candidate_pairs

The EM loop ran fast once I had the candidate table. DuckDB handles this in-process, so there's no serialization overhead between iterations. On my machine, 400,000 pairs, the whole convergence took a few seconds. That part of the pipeline wasn't a problem.

What I spent a disproportionate amount of time on was preprocessing. Name normalization for Korean, Japanese, and Chinese inputs has a lot of edge cases, and the blocking key construction amplifies whatever noise gets through. Japanese company name suffixes alone took an afternoon to handle consistently — 株式会社 shows up at the front in some sources and the back in others, abbreviated in a few more. Once I had a normalization that wasn't silently introducing variance into the blocking key, I'd spent more hours there than on everything inside Splink's own configuration combined.

None of that is a Splink problem. The library doesn't know what language your inputs are in. But if you're running entity resolution over multilingual corporate names, the normalization you build before Splink touches the data will probably be where you spend most of your time. The EM loop will be fine.

I build er-api, a multilingual entity resolution service for Korean, Japanese, Chinese, and English corporate data. More at hannune.ai.

Your entity resolution pipeline works on database records. Then you point it at news text.

Tae Kim — Thu, 17 Sep 2026 03:06:13 +0000

I built the entity resolution pipeline at 2asy.ai against structured registry data. I benchmarked it against structured registry data. The match scores looked good. I moved on.

About a month later I started ingesting news articles. The first batch turned into two weeks of cleanup.

What I hadn't tested was what happens when the source text doesn't use formal company names. News doesn't. A Korean financial article calls "삼성전자 주식회사" just "삼성." A piece in English calls it "Samsung" without specifying which Samsung entity, because to a human reading with context, it's obvious which one. The pipeline had no reading context. It had a blocking key built on the first four characters of a normalized name.

"삼성" hits the same blocking bucket as Samsung Electronics, Samsung SDI, Samsung C&T, and a dozen other entities under the group umbrella. The EM scoring then gets candidate pairs with name similarity 1.0 across all of them and nothing else to work with.

ON LEFT(l.name_normalized, 4) = LEFT(r.name_normalized, 4)

This was the blocking rule. It was designed for formal names where "Samsung Electronics" and "삼성전자" both normalize to something starting with "sams." It's also what causes a short news mention to block with every Samsung subsidiary in the registry.

What I added was a post-scoring filter using signals from the article itself. The article a mention comes from has context: what other entities appear nearby, what sector terms show up in the paragraph. When the top match by name similarity has no overlap with those signals, I look at the second-best match.

def extract_context_signals(article: dict, mention_span: tuple) -> dict:
    window = article["text"][max(0, mention_span[0]-200):mention_span[1]+200]
    return {
        "mentioned_with": extract_entity_mentions(window),
        "sector_keywords": extract_sector_terms(window),
        "article_category": article.get("category", "unknown"),
    }

This helped with the Samsung group disambiguation specifically. It doesn't do much for companies that don't have siblings with similar names. And when the article context is thin — short breaking news items especially — the signals aren't reliable enough to use.

I also raised the confidence floor for what I'm calling informal mentions: short Korean-character forms, English names that don't include the entity's legal suffix. The threshold for a two-character Korean name merging to a registry entry is higher than for a full formal name match.

if mention_is_short_informal(mention_text, source_language):
    threshold = max(threshold, 0.90)

The 0.90 came from watching the false-merge rate and adjusting until it dropped to something workable. It's not principled.

The benchmark I ran when building this is still the benchmark I'm running. It's all registry-vs-registry pairs. I don't have a labeled test set for news mentions, so I don't actually know what the precision-recall curve looks like for the domain I'm now running against daily. That's the part I'm not comfortable with.

I build er-api, a multilingual entity resolution service for Korean, Japanese, Chinese, and English corporate data, used in 2asy.ai for news-to-knowledge-graph ingestion. More at hannune.ai.

When I killed the Korean version: what channel mismatch actually looks like

Tae Kim — Tue, 15 Sep 2026 03:44:39 +0000

About three months into building the Korean version, I showed the product to a financial analyst at a Korean institutional investor. He scrolled through the interface for maybe thirty seconds, then said something like: "we already have Bloomberg for data, what I need is something I can hand off."

The "hand off" part is what stuck. He needed a PDF or an Excel file, something he could email to a manager. The graph query interface I was showing him was not the answer.

I kept working on the project for a few more weeks anyway. The pipeline was running. I told myself I was still gathering information.

Some background: 2asy.ai tracks tariff policy changes and how they propagate to specific companies. The basic idea is that you can crawl policy and trade news, figure out which companies sit in the affected supply chains, and make that queryable. The entity resolution work is the hard part of it, which is mostly what the ER API does. When I decided to do a Korean-language version, it looked tractable because I'd done most of that infrastructure already. Mainly crawl pipeline work and some frontend localization to do on top.

By the time I showed the analyst that demo, I'd had a few other conversations in the same vein. Different people, similar answer. Korean institutional finance and compliance people I was talking to wanted a deliverable they could hand to someone else, not a tool they'd query themselves. The B2B buyers I'd built the international version for had already decided to integrate vendor risk data programmatically. Most of the Korean buyers I was reaching hadn't made that decision, or weren't the decision-makers who would.

I paused it. I kept the crawl pipeline and the entity resolution work because those are directly reusable if a reporting-oriented distribution path shows up. But I stopped maintaining it as an active product and turned off the scheduled jobs.

Looking back, I'm still not entirely sure when I made the decision versus when I just stopped maintaining it. Somewhere around week twelve those converged.

I build er-api, a multilingual entity resolution service, and 2asy.ai, a trade intelligence graph. More at hannune.ai.

What the retrieval layer of a Graph RAG system actually looks like

Tae Kim — Fri, 11 Sep 2026 03:25:08 +0000

The first version retrieved everything. Every node reachable from the seed entity, up to four hops. The context window hit 8,000 tokens on a basic query about a mid-size Korean manufacturer. The model started confusing entities.

I pulled it back to 50 nodes, which is where it still is. I don't love that number. It's a hard cutoff that causes real problems for large corporate groups — Samsung Electronics has hundreds of subsidiaries and a query about their supply chain exposure hits the limit before returning a complete picture. But 50 is where the model stops confusing things and starts giving useful answers, and I haven't found a smarter expansion strategy that changes that tradeoff.

The system is a Graph RAG layer over a 50 million-row supply chain knowledge graph. The entity resolution took about two weeks to get right. The retrieval layer itself took three days. Here's what the retrieval actually does.

Every retrieval starts from seed entities. For a query like "what is Samsung SDI's exposure to US tariffs on Korean steel," the first step is which nodes in the graph correspond to "Samsung SDI" and "US tariffs." I run this through an entity lookup that checks normalized names and aliases:

def identify_seed_entities(query_text: str, session) -> list[str]:
    entities = extract_entities(query_text)
    results = []
    for entity_mention in entities:
        candidates = session.run("""
            MATCH (e:Entity)
            WHERE e.name_normalized CONTAINS $mention_normalized
               OR any(alias IN e.aliases WHERE alias CONTAINS $mention_normalized)
            RETURN e.id, e.name, e.canonical_type
            ORDER BY e.mention_count DESC
            LIMIT 3
        """, mention_normalized=normalize(entity_mention)).data()
        if candidates:
            results.append(candidates[0]["e.id"])
    return results

mention_count ranks entities by how many source documents they appear in. This is a rough importance proxy. For "Samsung SDI" it works fine. For more ambiguous mentions it sometimes picks the wrong candidate, especially for smaller companies that share name fragments. I haven't fixed that properly.

From the seed nodes, the subgraph expansion follows supply chain relationship types and cuts off at three hops. Four hops started returning nodes that were technically connected but not relevant to any reasonable interpretation of the query. A steel tariff affecting a Korean parts supplier affecting a construction equipment company is three hops and stays useful. The fourth hop typically reached conglomerates that own everything and added noise.

result = session.run("""
    MATCH path = (seed:Entity {id: $seed_id})-[r:SUPPLIES_TO|SOURCES_FROM|OWNS|CONTROLS*1..3]-(connected)
    WITH connected, length(path) AS hops
    WHERE hops <= 3
    RETURN connected.id AS entity_id, connected.name AS entity_name, hops
    ORDER BY hops ASC, connected.document_count DESC
    LIMIT 50
""", seed_id=seed_id)

After getting the subgraph, I fetch the source document excerpts that were asserted through those entities. These are the actual facts — the sentences from filings, news articles, procurement documents that describe the relationships:

result = session.run("""
    MATCH (e:Entity)-[r:ASSERTED_IN]->(d:Document)
    WHERE e.id IN $entity_ids
    RETURN e.name AS entity_name,
           d.excerpt AS fact_text,
           d.source AS source,
           d.published_at AS published_at,
           r.assertion_confidence AS confidence
    ORDER BY r.assertion_confidence DESC, d.published_at DESC
    LIMIT 100
""", entity_ids=entity_ids)

The hundred-fact fetch gets trimmed to forty before going to the model. The trim is TF-IDF ranking against the query terms, which is coarse but seems to work. The bigger problem is that facts from entities reached through low-confidence ER merges get the same treatment as facts from directly verified sources. I flag anything below 0.80 confidence with [uncertain] in the context, but that's a manual addition after discovering the model was treating all facts equally regardless of how confident the underlying merge was.

Packaging the final context is straightforward: facts sorted by confidence, each labeled with source and date. Uncertain ones get the flag. The model prompt says to treat flagged facts as preliminary.

What takes the most time isn't any of this. It's the cases where the query involves an entity the graph doesn't have at resolution quality. The graph might have "Samsung SDI" resolved, but a smaller battery cell supplier mentioned in a procurement document might exist as a raw string node that never got merged with anything. The retrieval finds it, returns facts, and the model reasons off it as if it were a resolved entity. I currently have no way to flag "this entity wasn't fully resolved" at retrieval time. That's the next thing I need to add.

I build er-api, a multilingual entity resolution service for Korean, Japanese, Chinese, and English corporate data. More at hannune.ai.

Building an investing knowledge graph, part 6: if you're merging data from more than one source, you'll hit this

Tae Kim — Thu, 10 Sep 2026 02:55:22 +0000

Building an investing knowledge graph, part 6: if you're merging data from more than one source, you'll hit this

Since part 5 went up, I've had two conversations worth writing about. One was with someone building a financial news aggregation system who wanted to know if the ER setup here would work for their case. The other was with someone doing KYC screening who asked roughly the same question. Those conversations are what prompted this final part.

The financial news question had an obvious answer: yes, with some caveats about cross-language blocking. The KYC question took me longer, because the error tolerance is so different that I wasn't sure how much of what I built would actually transfer.

The KYC conversation

The person I talked to was working on sanctions screening. Their problem was name matching: entity strings from incoming documents against a registry of flagged persons and organizations. The description of the ER problem was familiar enough that I recognized it immediately.

What I had to think through more carefully was what the differences meant in practice. In the investing graph, a false negative means a traversal stops short. Two nodes that should be the same entity exist as separate nodes, and a path that should go through them stops at the first one. The graph is less useful, and you can often tell something is wrong because you're getting empty results where you expect some.

In sanctions screening, a false negative is an entity that should have been flagged clearing a check it shouldn't have. The downstream consequences are not "the graph is less useful." This changes how you'd set the threshold, and it probably changes how you'd approach training data. The model I've been running was tuned for a corpus where some missed merges are acceptable in exchange for keeping false positives low. For compliance tooling, that tradeoff goes the other way. I couldn't tell the KYC person "use the same model and threshold" and be confident they'd get results they could actually rely on.

So: the core architecture is the same. The pairwise scoring, the registry with a stable and pending boundary, the review layer for uncertain pairs. But the model would need to be retrained on labeled pairs from their actual domain, and the threshold would need to be set differently. These aren't small caveats. Blocking strategy would probably change too, because person name matching across languages is harder than company name matching in English.

The financial news question

This one was more tractable. They were ingesting from Bloomberg, a Chinese-language wire service, and some regional outlets. "Tesla, Inc." and "特斯拉" and "the EV maker" arriving in the same week.

The architecture question I could answer directly: yes, the same setup applies. The harder question was about cross-language blocking. For English company names, string normalization gets you a long way. Strip legal suffixes, lowercase, collapse whitespace, and most aliases cluster cleanly enough that phonetic blocking on the normalized form works. For Chinese-English matching, you need something else.

I spent some time thinking through what I'd actually try. One approach is alias generation during ingestion: before a string hits the resolver, you run it through a step that produces language variants and stores them as additional candidates. Then blocking happens on normalized alias strings rather than on raw input. The problem with this is that you're trusting the alias generation step, which introduces a new source of noise. I've thought about this but haven't implemented it, and I was honest with the person asking that this was speculation rather than something I'd tested.

The main takeaway from both conversations is that the registry architecture is more transferable than the model itself. The idea of maintaining a stable set of committed decisions versus a pending set of uncertain pairs under review, and serving only the stable set to external callers, holds regardless of what domain you're matching in. The specific model weights and threshold values don't hold.

What I would tell someone starting from scratch

This is the part I'm most uncertain about writing, because there's a gap between "what I learned from this specific project" and "what would generalize to your project."

The concrete thing I'd say: budget for registry maintenance in a way you probably won't. The ongoing cost of running this system is managing accumulated state, not running inference. Close to 48,000 entities and aliases have accumulated over months of ingestion. Some of those decisions were made when the threshold was still being calibrated. I know this in aggregate because I watched the model improve; I don't know which specific entries are wrong without re-running the full corpus. That re-run is still on the list.

I'd also say: the blocking step is where I lost the most time to underestimation. It's easy to focus on the model and assume that if you have good coverage you'll catch most matches. The model doesn't see pairs that didn't make it through blocking. If your blocking strategy misses a class of matches systematically, you won't know from looking at the model's output alone.

The third thing, and I've mentioned this before but it bears repeating at the end of a series: Splink is doing the real probabilistic lifting here. It's a well-maintained open-source library, and I'd recommend starting there before building anything custom.

ER API and what comes next

The investing knowledge graph is still running. The entity resolver is live on Railway as ER API, and I'm expanding access based on how early users are actually using it.

The endpoints are /v1/resolve for single string resolution and /v1/splink-pairs for batch candidate scoring. If you're running into the entity resolution problem in your own pipeline and want to try the service, there's a form at hannune.ai.

All six parts

Part 1: From "does this news matter" to a graph I can query
Part 2: The query SQL can't answer
Part 3: Same company, seven names
Part 4: Building the resolver
Part 5: What LIVE actually means

Built on Splink for probabilistic record linkage.

Building an investing knowledge graph, part 5: what LIVE actually means

Tae Kim — Wed, 02 Sep 2026 02:37:13 +0000

Building an investing knowledge graph, part 5: what LIVE actually means

Part 4 ended with the resolver working locally and me calling it live on Railway. That framing glossed over a gap. Something running locally and something running in production are different in ways that are obvious in retrospect and invisible until they bite you.

Here's what changed once I had a caller that wasn't me.

The first thing that breaks when someone else calls your API

The investing knowledge graph pipeline was the only caller for a long time. I knew its call patterns. I'd written both sides. The error budget was implicit — if something failed, I fixed it and kept going.

The first external caller changed that. When the API started returning errors for them, there was no "I'll fix it and retry." There was a broken integration on their end, and they had no context for diagnosing it.

The specific failure was unhelpful: a 500 with a generic error message on a request that had worked in my own testing. It took debugging from both ends to isolate the cause. The request was valid. The issue was that it hit the API during a window when the registry backend was mid-write, and the response came back internally inconsistent — some aliases resolved correctly, others returned stale data.

This wasn't a bug in the resolver logic. It was a backend durability problem I hadn't needed to care about when I was the only caller.

What the backend switch actually fixed

Locally, I used a file-based registry backend. The registry serializes to a JSON file, loaded into memory at startup and read during inference. Writes append and periodically flush to disk. Simple, fast, no dependencies.

The failure mode is obvious once you see it: during a write, if a read hits a partially flushed state, you get inconsistency. For a single-user pipeline running sequentially, this almost never surfaces. You'd have to be unlucky for a read to interleave with a write in a way that causes problems. Once there are concurrent callers, that probability stops being unlucky and becomes predictable.

The production backend is now PostgreSQL via Supabase. Writes go through transactions. Reads get a consistent snapshot. The alias table and the entity registry update atomically — either both change or neither does. A caller mid-request gets a consistent registry view regardless of what writes are in flight.

The switch is controlled by an environment variable (ER_REGISTRY_BACKEND). The application logic doesn't change, just the persistence layer. This separation made the transition straightforward: I could test both backends against the same code without restructuring anything.

The healthcheck

Railway handles deployments by routing traffic only after a service passes its healthcheck. The /health endpoint returns 200 when the API is ready to serve requests and the registry backend is reachable.

This matters in practice because startup isn't instantaneous. The service loads the model, initializes the Splink configuration, and verifies the backend connection before reporting healthy. Without an accurate healthcheck, Railway might route traffic to an instance that's running but not ready, which produces the kind of half-initialized state that generates confusing errors.

During a deploy, Railway keeps the old version serving traffic until the new one passes its healthcheck. No request reaches the new version before it's ready. This zero-downtime behavior is Railway's default — I didn't configure it specially — but it only works correctly if the healthcheck accurately reflects whether the service is ready.

The healthcheck has failed in two ways across different deployments: backend unreachable, and model initialization error. Both are things I'd rather catch at startup than discover mid-request. The healthcheck turned each into a failed deploy rather than a silent production problem.

Registry integrity at 47,853 entities

The registry currently holds 47,853 resolved entities and 47,883 aliases. Most entries are stable. Some are probably wrong.

Early in the project, when I was still calibrating the Splink threshold, the model was more permissive than it should have been. Some merges happened that were likely incorrect — entity strings that shared enough surface similarity to pass the threshold but were different companies. A few of those early-period decisions are still in the registry.

I know this in general. I don't know specifically which ones without re-running the full corpus against the current model. That re-run is on the list but hasn't happened yet.

What I have now is a review layer before new merges commit to the stable registry. Pairs that the model scores as probable matches but below a high-confidence threshold go into a pending state rather than being committed immediately. They get reviewed before joining the stable set. This doesn't fix the early errors already in the registry, but it stops new errors from accumulating at the same rate.

The production API serves only the stable registry. Pending pairs aren't exposed to external callers. If the model is uncertain, the caller gets a split decision. They can re-query later once additional evidence has accumulated and the pair has cleared review.

What "beta" actually means here

The last few posts in this series have mentioned "beta" without being specific. Here's what it actually implies:

The API is running. External callers are using it. I'm expanding access carefully rather than opening it wide.

It's not a prototype — the infrastructure decisions described above are real, the registry has nearly 48k entities built from actual article processing, and the resolver has been running reliably for months. It's also not a mature product with SLAs, a dedicated support tier, or uptime commitments beyond what Railway's infrastructure provides.

If you have an entity resolution problem — corporate name disambiguation, customer record deduplication, knowledge graph construction from heterogeneous sources — and want to try the service, there's a form at hannune.ai. I'll respond directly.

Next: the final part of this series wraps up the full arc and looks at where the same five-step pattern — DB limits, knowledge graph, ER bottleneck, direct implementation, cloud deployment — shows up outside investing graphs.

Built on Splink for probabilistic record linkage. Part 1 is here. Part 2 is here. Part 3 is here. Part 4 is here.

Building an investing knowledge graph, part 4: building the resolver

Tae Kim — Tue, 01 Sep 2026 02:37:00 +0000

Building an investing knowledge graph, part 4: building the resolver

Part 3 ended with a question I left open: how does the threshold actually get set, and what happens to the registry when new articles keep coming in? That's what this one covers.

The cost of getting it wrong in both directions

Before I touch thresholds, it's worth being specific about what failing in each direction costs.

A false merge combines two distinct entities into one node. In the Samsung SDI case from part 3, that would mean battery-supply disruptions show up when you traverse from the semiconductor business. The graph gives you an answer. The answer is wrong. You don't necessarily know it's wrong without already knowing the answer, which defeats the point.

A false split keeps the same entity as two separate nodes. "Samsung Electronics" and "the largest memory chipmaker in the world" stay disconnected. Edges pile up on each independently. A traversal from one doesn't reach the other. The graph gives you a partial answer. You lose coverage.

These aren't symmetric. A false merge injects noise that's hard to detect. A false split just means a disconnected node — you miss some connections, but you don't get fabricated ones.

So I set the threshold conservatively. Pairs below a high confidence mark stay split. Some real aliases end up never linking. That's the tradeoff I chose: lower recall, higher precision. For a graph I'm using to make judgments about which news matters, a confidently wrong edge is worse than a missing one.

What the endpoint does

The resolution service exposes a /v1/splink-pairs endpoint. It takes a list of candidate entity mention strings and returns a score for each pair, along with whether that pair crosses the registry threshold.

A minimal request looks like this:

POST /v1/splink-pairs
{
  "candidates": [
    "Samsung Electronics Co.",
    "the largest memory chipmaker in the world",
    "Samsung SDI",
    "TSMC"
  ]
}

The response returns a score matrix for pairs that the model considers worth evaluating — not every combination, just the ones above a blocking threshold that filters out obviously unrelated pairs before the full model runs.

{
  "pairs": [
    {
      "left": "Samsung Electronics Co.",
      "right": "the largest memory chipmaker in the world",
      "match_probability": 0.96,
      "decision": "match",
      "canonical_id": "company:samsung_electronics"
    },
    {
      "left": "Samsung Electronics Co.",
      "right": "Samsung SDI",
      "match_probability": 0.12,
      "decision": "split",
      "canonical_id": null
    }
  ]
}

The caller gets a decision (match or split) and the canonical entity ID when there's a match. If the match is against a known registry entry, the alias gets recorded. If it's a new pair the model hadn't seen before, that pair gets added to the evidence set for future model updates.

How the registry grows

When the investing knowledge graph pipeline processes a new article, it extracts entity mentions and runs them through the resolver. Three things can happen:

Known alias. The string matches an existing entry in the alias table. Lookup is fast, no model call needed. The mention gets written to the graph under the existing canonical ID.

Unknown mention, matches existing entity. The string isn't in the alias table, but the model scores it as a probable match against an existing entity. The alias gets added to the registry. Future articles that use the same string skip the model call.

Genuinely new entity. The model doesn't find a confident match against anything in the registry. A new canonical entry gets created. It starts small — one mention, no resolved aliases — and accumulates evidence as future articles mention the same company.

The registry currently has 47,853 resolved entities and 47,883 aliases. A lot of those started as single-mention nodes. Some have since merged as more articles confirmed the connection.

A few early decisions are probably still wrong. In the first batches I was more aggressive with merging, before I tightened the threshold. There are likely some nodes that should be split. I know this in general; I don't know specifically which ones without re-running the full corpus against the current model, which I haven't done.

What Splink is doing under the hood

Splink is the open source probabilistic record linkage library the model is built on. It uses DuckDB as the computation backend for the pair scoring, which handles the candidate generation and blocking step — filtering the space of possible pairs before running the full model.

For corporate entity mentions, the features that ended up mattering:

String similarity on the canonical name: catches abbreviations and common shorthand
Alias coverage: once a surface form is in the registry, future instances match by lookup rather than model score — this is how "the largest memory chipmaker in the world" eventually becomes reliable without its string similarity to "Samsung Electronics" ever improving
Token overlap on co-occurring named entities within the same article: an article about DRAM yields tends to mention different companies than an article about battery chemistry
Base rate weighting: a rare entity (one mention, no confirmed aliases) shouldn't get absorbed into a high-frequency entity just because they share a token

Splink learns the feature weights from labeled pairs. The training set was several hundred manually verified matches and non-matches from the first two thousand articles. Not large. But I had enough confirmed Samsung/Samsung SDI pairs to get the context features weighted correctly.

The point where this became a service

Somewhere around the third project where I had the same entity resolution problem — merging customer records from two systems, with the same fragmentation-versus-false-merge tradeoff — I stopped treating this as a per-project function and wrapped it into a standalone FastAPI service.

The knowledge graph pipeline now calls it over HTTP. Other things call it the same way. The registry is shared across callers. A match that one pipeline discovers helps every other caller that sees the same entity later.

That service is what I'm calling ER API. It runs on Railway, it's live, and the registry it's backing is the same one that's kept the investing knowledge graph from turning into a tangle of phantom nodes.

Next: what it takes to keep this running in production — the parts that only showed up after the first real caller wasn't me.

Built on Splink for probabilistic record linkage. Part 1 is here. Part 2 is here. Part 3 is here.

Building an investing knowledge graph, part 3: same company, seven names

Tae Kim — Mon, 31 Aug 2026 02:58:56 +0000

Building an investing knowledge graph, part 3: same company, seven names

I noticed a problem because a traversal that had been working stopped returning results. The ASML supply chain path I'd tested a week earlier came back empty. I spent a while thinking I'd broken the query. The query was fine. What had happened was that a new batch of articles from a different source used "the Dutch lithography equipment maker" and "ASML Holding NV" without ever using the string "ASML." Both landed in the graph as separate nodes. The traversal starting from the ASML node I'd been accumulating edges on couldn't reach them.

Seven names for one company

When I checked the Samsung Electronics entry in the registry, it had resolved seven distinct aliases to a single entity:

"samsung"
"samsung electronics"
"samsung electronics co."
"samsung foundry"
"samsung's"
"the korean company"
"the largest memory chipmaker in the world"

The foundry alias is a judgment call I went back on twice -- Samsung Foundry is technically a division with its own reporting, but in this corpus it was mostly used as shorthand for the parent in manufacturing capacity articles, so I kept it merged. The descriptive ones at the bottom are wire service shorthand. An article that opened with "Samsung Electronics" in the headline would use "the Korean company" three paragraphs later. Standard string matching returns near-zero similarity between "the largest memory chipmaker in the world" and "samsung electronics." They share no tokens.

Before I had a proper setup for this, I was using token overlap to match entity mentions. The threshold was loose enough to catch most of the alias variations above. It was also loose enough to merge Samsung SDI into Samsung Electronics.

Samsung SDI makes batteries. It is a separate publicly listed company. A lithium cell supply disruption hitting Samsung SDI's production line has nothing to do with Samsung Electronics' memory fab utilization. If those two ended up as the same node, events from one would show up in traversals that started from the other. I caught this specific case because I happened to be running a check on Samsung when a battery shortage article came through. Whether there are other pairs like this that I haven't caught, I genuinely don't know.

The descriptive ASML alias fails because it shares no tokens with the canonical name. Samsung SDI fails to separate because it shares the most salient token with an entity it should be distinct from. There's no threshold value that handles both. Loosening the threshold enough to catch the first kind of failure causes more of the second. These are actually different problems.

What Splink does with this

Splink treats entity resolution as a classification problem. For any candidate pair of mentions, the model computes a match probability using multiple features simultaneously and learns weights from a training set of confirmed pairs and non-matches.

For corporate entities in news text, the features I ended up using: canonical name similarity is the obvious starting point -- it handles "Samsung Electronics Co." but not "the largest memory chipmaker in the world." For descriptive references, alias coverage matters more: once a mention form has been confirmed as belonging to a given entity, future occurrences match by lookup. The ASML case gets better over time this way, even though the string overlap never improves.

For the Samsung SDI case, what made the difference was article-level context. An article that mentions DRAM yields and fab capacity almost certainly isn't about the battery subsidiary. Whether that context gets encoded as co-occurring industry terms or co-mentioned company names is a feature engineering choice; either way, it's information that string matching on the entity name alone can't see. I also added a base-rate feature after a few early merges looked wrong -- a rare entity with one mention shouldn't get absorbed into a high-frequency entity just because they share a token.

Splink learned weights across all of this from a training set of a few hundred manually verified pairs from the early corpus. Not large. But enough to get the Samsung SDI case right once I included the context features, which I hadn't initially.

The registry now has 47,853 resolved entities. That number has been climbing -- each new article either matches an existing entity or creates a new node that might get merged later as evidence accumulates. A few merges from early on are probably still wrong. Some entries I initially kept separate have since merged when more articles came in.

Still a threshold

Splink produces a probability, not a final answer. You still decide where to cut.

For this graph, I set that conservatively. A pair below a certain value stays split. Some valid aliases end up never linking to their entity -- nodes with few edges that mostly sit disconnected. I ran a check on a sample and most looked like genuine unknowns where there wasn't enough article evidence to confirm the connection. A few were probably real aliases I missed. That's the tradeoff: some fragmentation, in exchange for higher confidence that a merged entity represents one thing.

How that threshold gets set in practice, and what happens to the registry as new articles change the evidence picture, is part 4.

Built on Splink for probabilistic record linkage. Part 1 is here. Part 2 is here.

Building an investing knowledge graph, part 2: the query SQL can't answer

Tae Kim — Sat, 29 Aug 2026 02:52:23 +0000

Here is a Cypher query I can now run against the investing knowledge graph:

MATCH path = (source:Company {name: "ASML"})
             -[:SUPPLY_CUT|EXPORT_RESTRICTION|PRODUCTION_DELAY*1..4]->
             (downstream:Company)
RETURN 
  downstream.name AS company,
  length(path) AS hops,
  [r IN relationships(path) | r.article_id] AS evidence_articles,
  [r IN relationships(path) | r.event_date] AS event_dates
ORDER BY hops, event_dates[0]
LIMIT 30;

It returns paths. Not rows. Each result in the output is a chain from the source company to a downstream company, with the article ID and event date at every link. I can see whether NVIDIA appears in the results, how many steps away, and what sequence of articles establishes the connection. The depth bound is a parameter I tune at query time.

I didn't write this first. This is where I ended up after trying two other things that didn't work.

The search index

The article database sits in Elasticsearch, index called economic_news_articles_en, currently 9,414 documents. Starting with what I had:

GET economic_news_articles_en/_search
{
  "query": {
    "bool": {
      "must": [
        { "match": { "text": "chip packaging export restriction" } },
        { "range": { "published_at": { "gte": "2023-01-01" } } }
      ]
    }
  },
  "_source": ["title", "published_at", "companies_mentioned"],
  "size": 15
}

Fourteen results. Some are on the specific restriction. A few match incidentally. The NVIDIA articles are nowhere in these results because NVIDIA isn't mentioned in any of them. This is not a deficiency in Elasticsearch. It returned exactly what it should. The question I wanted to ask is a different kind of question: it requires following edges between things, and the search index doesn't model edges. Relevance tuning would not have helped here.

The SQL attempt

I had generated a relational structure during the build process. An article_events table, each row an event type, a subject company, an affected company, and a date. The first query:

SELECT c.name, ae.event_type, ae.event_date
FROM article_events ae
JOIN companies c ON c.id = ae.object_id
WHERE ae.subject_id = (SELECT id FROM companies WHERE name = 'ASML')
  AND ae.event_type IN ('supply_cut', 'export_restriction', 'production_delay')
ORDER BY ae.event_date;

That gives first-hop results: companies directly named in events where ASML is the subject. Going one hop further:

WITH first_hop AS (
  SELECT ae.object_id AS affected_id, ae.event_date
  FROM article_events ae
  WHERE ae.subject_id = (SELECT id FROM companies WHERE name = 'ASML')
    AND ae.event_type IN ('supply_cut', 'export_restriction', 'production_delay')
)
SELECT DISTINCT c.name, ae2.event_type, ae2.event_date
FROM first_hop fh
JOIN article_events ae2 ON ae2.subject_id = fh.affected_id
JOIN companies c ON c.id = ae2.object_id
WHERE ae2.event_date >= fh.event_date
ORDER BY ae2.event_date;

I had to know upfront how many hops I was looking for. Two CTEs for two hops. A three-hop query needs three. If the connection I care about happens to be five steps deep, a three-hop query misses it quietly. Recursive CTEs handle variable depth but bring their own performance questions, and by that point I was building something that wasn't really a SQL use case anymore.

The schema itself was losing information. Events in articles don't always have clean subject-object pairs. An article might describe an event that involves three or four companies in ways I couldn't reduce to one row without discarding something.

Why the graph answered it

The Cypher query at the top of this post returned paths. The evidence traveled with the result. For each chain in the output I had the article IDs and the event dates at each intermediate link, not just the endpoint. The *1..4 depth parameter meant I could check four hops without rewriting anything.

The graph at this point holds 84,962 nodes and 275,293 relationships, built from those same 9,414 articles. On the Cypher query, response time is acceptable. When a path doesn't exist in the data, the query returns nothing, which is a correct answer. It means the graph doesn't have evidence for that connection, not that the connection doesn't exist in the world.

What the graph assumed

For the traversal to be useful, the same company needed to map to the same node regardless of how different articles referred to it.

That didn't happen automatically. Articles about ASML might say "ASML Holding NV" in a formal announcement, then a wire service shortens it to just "ASML," then a blog post never names the company and calls it "the Dutch lithography equipment maker" instead. If those land as three separate nodes in the graph, the traversal silently breaks. None of the three nodes accumulates enough edges to connect to anything interesting.

The flip side caused different problems. ASML Cymer is a subsidiary. It makes excimer laser light sources, not the lithography systems the parent is known for. Merging that name into the ASML parent node because the strings share characters would misattribute events. A supply disruption that hit only the subsidiary's production line would look, in the graph, like it hit ASML's main business.

Getting entity mentions to resolve correctly was the bottleneck I hadn't expected. That's what part 3 covers: how string similarity fails in both directions at once, and what probabilistic matching over multiple signals looks like in practice for 47,853 resolved entities.

Built on Splink for probabilistic record linkage. Part 1 is here.

When your calls can't leave your infrastructure: the compliance constraint that shapes the whole AI stack

Tae Kim — Fri, 28 Aug 2026 02:37:46 +0000

I wrote before about why I run speech-to-text locally—for me it started as a personal data hygiene decision. But when I've talked to other people about local AI pipelines, a different version of the same constraint comes up: not "I'd prefer the audio to stay local" but "the audio legally cannot leave our environment."

That's a different problem. A preference shapes your tooling choices. A compliance constraint shapes your architecture.

What the constraint actually looks like

The most common version I hear: enterprise customer agreements. A company builds a software tool that serves clients in regulated industries. Those clients have data handling requirements that specify data can only be processed in specific jurisdictions or on approved infrastructure. When that company wants to add AI-powered call analysis, every cloud transcription API is immediately out of scope—not because of policy preference, but because the customer contract requires it.

Attorney-client privilege is another version. Law firms and their clients have specific expectations about where privileged communications go. Using a third-party transcription service means the audio passes through systems outside that privileged relationship. Whether that technically breaks privilege is a legal question that most firms don't want to be the test case for.

Healthcare is the most formalized version: PHI (protected health information) that appears in patient calls needs to stay within HIPAA-compliant infrastructure. Some cloud providers offer BAAs (Business Associate Agreements) that extend that coverage, but plenty of healthcare organizations have decided the simplest answer is a processing pipeline that stays entirely on infrastructure they control.

The common thread: the constraint isn't about the transcription output (text) but about the audio itself, during processing. It's a data residency and custody question.

The architectural implication: it's not just transcription

Here's where this gets interesting from a systems perspective. If you accept the constraint that audio can't leave your infrastructure, and you're trying to build something useful from that audio—summaries, action items, entity extraction, follow-up flags—the constraint propagates through every step that touches the content.

You transcribe locally. But then what?

If you send the transcript to a cloud LLM for summarization, you've sent the content of the call off your infrastructure. The audio stayed local, but the information left. For most compliance constraints, this doesn't actually satisfy the requirement.

So the constraint ends up requiring:

Local transcription (Whisper or equivalent)
Local embeddings if you're doing semantic search across transcripts
Local LLM inference if you're doing summarization, extraction, or classification
Local storage, obviously

You end up needing a full local inference stack, not just a transcription component. The compliance requirement that started with "we can't use cloud STT" turns into "we need to run the whole pipeline on infrastructure we control."

This is the constraint I'm designing around

The setup I've been building runs Whisper for transcription, bge-m3 for embeddings, and Gemma for image/document processing—all on a single local server. I covered the resource management side of running three models on one box in an earlier post.

The "nothing leaves the box" constraint wasn't an afterthought. It was the first design decision, and it determined the whole stack. Every model choice, every pipeline component was evaluated against it: does this require sending data somewhere I don't control? If yes, it's out.

The tradeoffs are real. Local inference is slower than cloud APIs for equivalent hardware. You own the updates and maintenance. When a better Whisper model ships, you're updating your own deployment. The TCO calculation looks different depending on how much you're processing.

But for the use cases where the compliance constraint is genuine—where "it's in the customer agreement" or "it's a regulatory requirement"—those tradeoffs aren't optional. The architecture has to start from the constraint.

What I haven't built yet

The local inference stack is working. What I'm building toward is the layer that makes it actually useful: intake from multiple channels (calls, email, messages), process everything locally, surface what matters in a daily report.

The compliance constraint applies there too—not just transcription. Every step processes content that, in an NDA or regulated environment, can't transit external APIs. So the design for the report generation layer has to be local all the way through.

That's the hard part I'm working on now: not "can I transcribe calls locally" (yes, that's working), but "can I build the summarization and reporting layer locally at quality that's actually useful."

Other posts in this series: Running three AI models on one server | Why local STT over cloud API | Missed calls, missed revenue

Building an investing knowledge graph, part 1: from "does this news matter" to a graph I can query

Tae Kim — Mon, 24 Aug 2026 06:24:08 +0000

I keep hitting the same question when a headline crosses my feed about a company I'm holding, or thinking about holding: does this news actually reach the thing I care about? A chip export restriction hits some supplier nobody's heard of. Two weeks later a manufacturer I do care about moves, and the headline that started it never mentioned them once. I wanted to see that chain, the actual event-to-event path, and no sentiment score was going to show it to me.

So I tried the obvious thing first: search my article database for the company name and read whatever comes back.

Where the article database stopped helping

I have a database of financial news articles, 9,414 documents at the moment, each indexed with the companies, people, and events mentioned. "Show me articles about Samsung" works fine. That's what search is for. "Show me what happens downstream when a supplier gets sanctioned" falls apart immediately, because a table of articles has no concept of downstream. Each row is independent. Nothing connects article 4,201 about a supplier to article 6,830 about a company three steps down the chain (those two numbers are made up, but that's the shape of the problem). Unless I read every row myself and hold the connections in my head, which is what I was already doing, badly.

Filtering and sorting, a table does great. Walking a chain of "this caused that caused that" just isn't something a flat table does. The information might be sitting right there across a dozen rows. There's no native way to say "A affects B affects C." Tables were never built for that question.

Swapping rows for nodes and edges

A knowledge graph fixes this at the structural level. Instead of rows you have nodes (companies, people, events) and edges (supplies, owns, sanctioned, competes with), and once that structure exists, "what's downstream of this event" stops being a research project. You start at a node and walk outward.

That's what I've been building: articles get parsed for entities and events, those get written into a graph store, and a question like "what's connected to this supplier three hops out" becomes something the graph itself can answer. It currently holds 84,962 nodes and 275,293 relationships, all extracted from those same articles.

None of this is a novel idea. Graphs for multi-hop reasoning are well established. The part that hurt showed up one layer down, before any traversal logic even mattered.

The part that actually blocked me: the entities were wrong

A traversal is only as good as the nodes it's walking. My first batches of extracted articles produced a mess I didn't expect to be the hard part: the same company kept landing in the graph as several disconnected nodes, because the articles never call it the same thing twice.

Here's one company, exactly as the pipeline extracted it:

"Samsung"
"Samsung Electronics"
"Samsung Electronics Co."
"Samsung Foundry"
"Samsung's" (possessive, mid-sentence)
"the Korean company" (a lazy second reference)
"the largest memory chipmaker in the world" (same article, different lazy reference)

Seven surface forms, one real company. That last one is my favorite, I only caught it because a "company" node named an entire sentence looked absurd in the browser. Write each form into the graph as its own node and a query for everything connected to Samsung silently misses six of them. The causal chain I wanted literally can't form. The graph thinks it's looking at seven strangers.

And the failure runs the other way too, which took me longer to appreciate. Another article mentioned "삼성SDI": Samsung SDI, a real, separately listed subsidiary that makes batteries, not chips. Merge that into the Samsung Electronics node and the graph starts lying. SDI's battery contracts become evidence about a semiconductor business, and every chain through that node inherits the false premise.

So the bottleneck wasn't graph technology at all. It was entity resolution: given messy surface forms from unstructured text, decide which ones are the same real-world thing and which only look alike. Naive string matching fails both directions at once. "The largest memory chipmaker in the world" shares zero characters with "Samsung," and "Samsung SDI" is one string-edit away from a wrong merge. You need probabilistic matching over multiple signals, and a memory of what you've already resolved.

So I built it

I ended up building a standalone entity resolution service rather than a one-off cleanup function, mostly because this exact problem had already bitten me in every project that merges data from more than one source. It's built on Splink, the open source probabilistic record linkage library, wrapped in a FastAPI service. Feed it candidate mentions and it scores how likely two of them refer to the same entity, keeping a registry of resolved entities and known aliases so it gets better the more it sees. As of today the registry holds 47,853 resolved entities and 47,883 aliases. The Samsung example resolves correctly now: all seven mentions collapse into one node, Samsung SDI stays separate.

And it's live

The resolver runs as a deployed cloud service, sitting in front of the graph pipeline. Every incoming article gets its entity mess cleaned there before anything touches the investing knowledge graph.

If you're merging entities from more than one source, articles, KYC documents, supply chain records, customer files from separate systems, you will meet the Samsung problem eventually. That's what ER API is for. It's live and in beta. Next post: the resolver mechanics, how it decides "Samsung Foundry" and "the largest memory chipmaker in the world" are the same company without also swallowing Samsung SDI.