I built a job search API that aggregates live postings by keyword and location, and the first version of the results was embarrassing: search "ML engineer Berlin" and you'd get the same role from the same company four or five times, because it was scraped onto four different boards. Agents consuming the API would waste context window on duplicates and look careless doing it.
The fix was a fuzzy matching pass on (normalized title, normalized company, location) before returning results. Normalizing titles meant lowercasing, stripping punctuation, and collapsing things like "Sr." / "Senior" / "III" — because job boards can't agree on how to spell seniority either. I tuned the similarity threshold by hand: too strict and you miss real duplicates that differ by a comma, too loose and you merge two genuinely different roles at the same company.
What surprised me: the deduplication logic ended up being the actual product, not a cleanup step. The raw feed is the commodity; the deduplicated, structured result (title, company, salary, apply URL) is what makes it usable in an agent pipeline. Nobody wants 40 raw listings. They want the 9 real jobs.
If you're building anything that aggregates external data for agents, budget real time for identity resolution. Your upstream sources will not do it for you, and your downstream agents will absolutely notice when they don't. I ended up packaging the whole thing as https://x402.freeq.one/tools/jobs.html
Top comments (0)