DEV Community

0xGollum
0xGollum

Posted on Edited on

I built a job search actor that aggregates 5 job boards - a $15/hour vs $50,000/year bug I found along the way

#python #api #webscraping #opensource

This actor pulls job listings from five job boards in one run. Give it a search term, a location, and a salary range, and it returns matching listings, deduplicated across boards, with pay normalized to one comparable figure regardless of whether the source listed it as hourly, weekly, or annual.

Getting that normalization right took two real fixes.

Why it's not a formatting problem, it's a units problem

The instinct is to treat this as a text-parsing issue: extract the number, done. But 15.20 and 30000 aren't the same kind of number. One is dollars-per-hour, the other is dollars-per-year. Comparing them directly is like comparing a speed in km/h to a distance in km — the parser was working fine, the comparison itself was meaningless.

The fix was to annualize every parsed figure at extraction time, based on whatever period word appeared next to it (hour, day, week, month, year):

_ANNUALIZE = {
    "hour": 2080, "hr": 2080,  # 40h/week * 52 weeks
    "day": 260,                 # ~5 working days/week * 52 weeks
    "week": 52,
    "month": 12,
    "year": 1, "yr": 1, "annum": 1,
}

lo, hi = min(a, b) * multiplier, max(a, b) * multiplier
Enter fullscreen mode Exit fullscreen mode

The original wording is still kept around separately for display (so a human still sees "£13.78 - £15.20 per hour", not a slightly surreal "£31,241/year"), but the filter itself only ever compares annualized numbers.

A second one, in the same feature

Testing the same filter against a different source turned up a second, smaller version of the same lesson. One job board tags listings with "full time" (a space). My keyword vocabulary was checking for "full-time" (a hyphen). Same concept, two spellings, one silent miss — a real "Full-Time" job simply never matched.

_JOB_TYPE_RE = {
    jt: re.compile(re.escape(jt).replace("\\-", "[\\s-]"), re.IGNORECASE)
    for jt in _JOB_TYPES
}
Enter fullscreen mode Exit fullscreen mode

Neither bug threw an exception. Neither one showed up in a code review. Both only surfaced by running the filter against real listings from real sources and noticing the result count didn't match expectations — which is the actual reason this project has fixtures captured from live responses rather than hand-written mocks.

Actor page: https://apify.com/0xgollum/global-job-aggregator

Top comments (0)