DEV Community

Rowharbor Data
Rowharbor Data

Posted on Edited on

What I learned pulling 1,650 Gulf jobs from five different ATS feeds

I wanted a simple thing: every open job in the UAE, Saudi Arabia and the rest of the Gulf, straight from the companies hiring, without job-board reposts.

It turns out most companies don't post jobs by hand. Their careers page runs on an applicant tracking system (ATS), and the big ones publish public JSON feeds of every open role. No login, no scraping HTML, no proxies. The feeds exist so careers pages can render, and anyone can read them.

I ended up reading feeds from 48 employers across five systems and got about 1,650 Gulf jobs per run. Here's what I learned.

1. Every ATS has a feed, and every feed is different

ATS Endpoint (public) Quirk
Greenhouse GET boards-api.greenhouse.io/v1/boards/{company}/jobs?content=true Descriptions are HTML, sometimes escaped twice
Lever GET api.lever.co/v0/postings/{company}?mode=json Dates are Unix milliseconds; extra locations hide in categories.allLocations
Ashby GET api.ashbyhq.com/posting-api/job-board/{company}?includeCompensation=true Salary is nested in compensation.summaryComponents
Workday POST {host}/wday/cxs/{tenant}/{site}/jobs Search API, paginated by 20; dates are text like "Posted 3 Days Ago"
SmartRecruiters GET api.smartrecruiters.com/v1/companies/{id}/postings?country=ae Has a country filter (nice!), but descriptions need a second call per job

The cheapest one to work with is Greenhouse: one request, every job, full descriptions.

import httpx

def greenhouse_jobs(company: str) -> list[dict]:
    url = f"https://boards-api.greenhouse.io/v1/boards/{company}/jobs?content=true"
    r = httpx.get(url, timeout=30)
    r.raise_for_status()
    return r.json()["jobs"]

for job in greenhouse_jobs("careem"):
    print(job["title"], "|", job["location"]["name"])
Enter fullscreen mode Exit fullscreen mode

2. "Is this job in the Gulf?" is harder than it looks

Location fields are free text: Dubai, Dubai, UAE, Remote - EMEA, Riyadh, Saudi Arabia; London. My first matcher was a list of substrings: dubai, riyadh, qatar, oman…

Then a job in Romania showed up as Oman.

"R-oman-ia." Substring matching on short country names is a trap. The fix was word boundaries:

import re

GULF = [
    (r"\bdubai\b", "Dubai", "United Arab Emirates"),
    (r"\babu ?dhabi\b", "Abu Dhabi", "United Arab Emirates"),
    (r"\buae\b|united arab emirates", None, "United Arab Emirates"),
    (r"\briyadh\b", "Riyadh", "Saudi Arabia"),
    (r"\bdoha\b|\bqatar\b", None, "Qatar"),
    (r"\bmuscat\b|\boman\b", None, "Oman"),
]
GULF_RE = [(re.compile(p, re.I), city, country) for p, city, country in GULF]

def gulf_match(text):
    for rx, city, country in GULF_RE:
        if rx.search(text or ""):
            return city, country
    return None

gulf_match("Bucharest, Romania")  # None, finally
Enter fullscreen mode Exit fullscreen mode

Also check all locations, not just the first. Plenty of roles list London first and Dubai third.

3. HTML that was escaped twice

Some Greenhouse descriptions come back as <p>About us</p>, and a few were escaped again (<p>). One html.unescape() isn't enough. I unescape in a loop until the text stops changing, then strip tags:

import html, re

def clean(raw):
    text = raw or ""
    for _ in range(3):
        un = html.unescape(text)
        if un == text:
            break
        text = un
    text = re.sub(r"<[^>]+>", " ", text)
    return re.sub(r"\s+", " ", text).strip()
Enter fullscreen mode Exit fullscreen mode

4. Workday doesn't give you dates

Workday returns "postedOn": "Posted 30+ Days Ago". You can turn it into an approximate date, but you can't get better than "about a month ago". Sort by it, but don't promise exact dates.

5. Extracting signals from job text: the false positives teach you the most

Gulf job seekers care about things job boards rarely expose: visa sponsorship, whether Arabic is required, and whether a role is reserved for nationals (Emiratisation and Saudization roles). I extracted these with plain regex rules rather than an AI model, then checked about 165 real postings by hand. The mistakes were the interesting part:

  • "UAE Nationals Preferred" is not "UAE Nationals only." My first rule flagged both. Preferred roles are still open to expats, so I added a negative lookahead for preferred / desirable / a plus.
  • Not every nationals tag is in English. Some AECOM titles say "إماراتيين" ("Emiratis"), and Saudi graduate roles say "Tamheer" (a nationals training programme). Both needed their own patterns.
  • "Arabic cuisine" is not a language requirement. A Sous Chef ad was flagged "Arabic required". The fix: \barabic\b(?!\s+(?:cuisine|food|coffee)).
  • "Over 25 years of experience" is usually the company bragging, not the job requirement. I ignore numbers above 20, and when an ad mentions several, I keep the smallest. If you're filtering for junior roles, you'd rather see a borderline job than miss one.
  • "Spark" isn't always Apache Spark. Hotel sales ads say "spark conversations". I now require context like "Apache Spark", "PySpark" or Spark appearing in a list of tools.

The general lesson: rules are cheap and explainable, but you only find their holes by reading real data. When a rule isn't sure, return null rather than guess.

6. Be polite, and only fetch what you keep

Two things kept my runs fast and friendly to the servers:

  • List first, fetch details later. For Workday and SmartRecruiters, descriptions cost one extra request per job. I fetch the list, filter by location and keywords, sort newest first, and only then fetch details for jobs I actually keep, a few at a time with a small delay.
  • A real User-Agent, a short delay, and backoff on 429. With those in place I haven't been rate-limited by any of these feeds.

7. The numbers

For one full run across 48 employers: about 1,650 Gulf jobs, mostly hospitality (Accor alone had 821), engineering (AECOM 320) and delivery (Delivery Hero 129), plus a long tail of tech and fintech. Salaries were published for only a handful of jobs. Most Gulf employers still don't post pay.


If you want to skip building this yourself, I packaged it as an Apify Actor ($1.50 per 1,000 jobs). It's my own project, and the code examples above work on their own. Questions or ATS quirks you've hit? Drop them in the comments.

Top comments (3)

Collapse
 
ayushdhanani369 profile image
Ayush Dhanani •

The Romania and Oman trap is the detail I liked most. Substring matching looks right until a short country name breaks it. I build Aladdin, a job agent. I use the same rule on the application side. A fresh worker reads the actual page and form itself. If it cannot read the form, or an answer is unknown, it asks the user instead of guessing. Your Arabic cuisine example is the same lesson. Rules are cheap, but only reading the real text finds the holes.

Collapse
 
rowharbor profile image
Rowharbor Data •

Thanks! Agreed. When the text is not clear, leaving the field empty beats guessing.

Collapse
 
ayushdhanani369 profile image
Ayush Dhanani •

Thanks for confirming it. An empty field is honest, and a recruiter can ask about it later. A guessed location is much harder to undo.