I run a job postings search API that pulls live listings by keyword and location. Early testers kept reporting the same suspicious thing: searching "python berlin" and "rust berlin" both surfaced an identical-looking list, just reordered. At first I assumed my upstream provider was lazy. It wasn't.
The real bug was in how I normalized results before returning them. I was deduplicating on the job title string alone, because "Senior Backend Engineer" felt unique enough within a city. It isn't. Three different companies in the same metro all hire "Senior Backend Engineer" in the same week, and my dedup collapsed them into one record — silently dropping two of the three. The API wasn't returning fewer jobs; it was returning wrong jobs, confidently.
The fix was unglamorous: dedup on a composite key of (title, company, location, apply_url). The apply URL turned out to be the only stable identifier across aggregators — titles get paraphrased, companies get rebranded, locations get geocoded differently, but the application link is the one field nobody rewrites.
What stuck with me: my test suite had been green the whole time because I was testing with synthetic data where every title was distinct. Real-world data is full of collisions, and your dedup key is a product decision, not a hygiene detail. If you're building anything that aggregates listings — jobs, products, events, news — assume the human-readable fields lie to you and find the one field that can't.
I ended up packaging it as https://x402.freeq.one/tools/jobs.html — the composite key fix shipped in the same week the complaints stopped.
Top comments (2)
The repeated-title fixture is a useful correction to the synthetic dataset. I'd add the opposite pair too: one requisition seen through two aggregators, with a paraphrased title or an extra tracking parameter on the apply URL. The four-field key could keep both even though they're the same opening. Then test two genuinely different requisitions that share a generic careers-page link, so URL cleanup doesn't merge them. Keeping source IDs and the merge reason beside the normalized record would make either decision inspectable instead of treating an application URL as guaranteed identity.
The composite-key lesson matches what broke my own merge tests — title collisions inside one city are the norm, not the edge case.
One extra footgun once you key on apply/listing URLs: the “apply” URL and the “job” URL are often different paths on the same ATS. Live check just now — Lever Palantir uses
…/{id}vs…/{id}/apply, Ashby OpenAI uses…/{id}vs…/{id}/application. If one upstream gives you the hosted page and another the apply link, a naive string equality misses the duplicate.Greenhouse is the opposite trap: Stripe’s board returns
absolute_urllikehttps://stripe.com/jobs/search?gh_jid=8172508for every row (725/725). A “strip all query params” normalizer quietly deletes the only stable identifier. Safer: droputm_*/gh_src/ similar, keepgh_jid(or canonicalize to the numeric job id from the JSON).