Scraping bilingual data: the Arabic/English traps that quietly corrupt your dataset

Most scraping advice assumes one language. Build for a bilingual market and you meet a category of bug that never throws an exception: the request succeeds, the HTML parses, and every field you wanted comes back empty.

Here is what I learned building job datasets that have to work in both Arabic and English.

The bug that returns null instead of failing

I was fetching job detail pages and extracting the stated seniority and employment type. The parser looked for the labels the page uses, matched them, and pulled the adjacent value. It worked in testing.

In production, seniorityStated and employmentType came back null for every record. No error. Status 200. Valid HTML.

The links I was following pointed at a regional subdomain. That subdomain served the Arabic version of the page. My selectors were matching on the English labels — "Seniority level", "Employment type" — and the page said المستوى الوظيفي and نوع التوظيف. Nothing matched. Nothing raised. The fields were simply absent.

The fix was one line: normalize the host before fetching.

import re

# Regional hosts serve localized pages whose labels don't match
# English selectors. Normalize to the canonical host.
url = re.sub(r"https://[a-z]{2}\.example\.com/", "https://www.example.com/", url)

The lesson generalizes: a scraper that finds nothing and a scraper that finds the wrong language look identical from the outside. If a field is null across every single record, suspect locale before you suspect your selector.

Assert on what you expect to find

The defence is cheap. After a parse, check that the fields you rely on actually populated:

critical = ("seniority", "employmentType")
missing = [f for f in critical if not any(r.get(f) for r in records)]
if missing:
    raise RuntimeError(f"No record has {missing} — wrong locale or stale selector?")

A scraper that crashes loudly on an empty column is worth ten that return tidy nulls.

Numerals are not the digits your regex expects

Arabic pages frequently use Eastern Arabic numerals. ٢٠٠ is two hundred.

Python is more helpful here than people assume, which is exactly why this bites. int("٢٠٠") returns 200. "٢٠٠".isdigit() is True. re.findall(r"\d+", ...) finds them, because \d is Unicode-aware by default.

The thing that silently fails is the character class almost everyone actually writes:

import re

text = "أكثر من ٢٠٠ متقدم"          # "more than 200 applicants"

re.findall(r"[0-9]+", text)          # []          <- silently empty
re.findall(r"\d+", text, re.ASCII)   # []          <- same trap
re.findall(r"\d+", text)             # ['٢٠٠']     <- works
int("٢٠٠")                           # 200         <- works

So a scraper that extracts applicant counts with [0-9]+ doesn't crash on an Arabic page. It reports that the page contains no numbers, forever, and you conclude the field is missing rather than that your regex is ASCII-only.

If you want ASCII digits in your output regardless of the source, translate rather than strip:

ARABIC_DIGITS = str.maketrans("٠١٢٣٤٥٦٧٨٩", "0123456789")

def to_int(text):
    return int(re.sub(r"\D", "", text.translate(ARABIC_DIGITS)))

I nearly published the claim that int() raises on Arabic numerals. It doesn't. I only found out because I ran the snippet before shipping the article, which is the entire moral of this post.

Don't let a substring match decide a language

Keyword matching across languages goes wrong in ways that are hard to see. Two real examples from classifying job titles:

• Matching the seniority abbreviation coo as a substring also matches "Coordinator". A programme coordinator became a chief operating officer.
• Matching partner as a substring promoted "Partner Onboarding Specialist" to firm partner.

Word boundaries fix both. But the general point is sharper in a bilingual dataset, where titles mix scripts, transliterations and English loanwords in the same string. re.search(r"\bcoo\b", title), never "coo" in title.

Print the rows behind the number

I was about to publish a statistic saying 18% of roles in a market were executive-level. It felt too high, so I printed the titles in that bucket. Half of them were coordinators and onboarding specialists. The real figure was 16%, and the composition was completely different.

Every heuristic you write will be wrong in a way that flatters your numbers. Print the rows.

Machine translation is for labels, not for meaning

If you add Arabic labels to an English dataset, keep the original. Store title and titleArabic side by side; never overwrite. Translation is lossy, brand names transliterate inconsistently, and someone downstream will want to match on the original string.

For job titles specifically, a translated label is a display convenience. Every join, dedupe and filter should still run on the source-language field.

Encoding, briefly

Set your encoding explicitly at every boundary: reading files, writing files, and printing. On Windows the default console encoding will happily raise UnicodeEncodeError on a perfectly good Arabic string, which sends you hunting for a scraping bug that doesn't exist.

sys.stdout.reconfigure(encoding="utf-8")
pathlib.Path(p).read_text(encoding="utf-8")

Also: a UTF-8 byte order mark at the start of a JSON file will make some parsers reject it outright. If a tool insists your valid JSON is invalid, check for a BOM before you check your syntax.

Where this came from

I maintain scrapers for regional job boards that output company and role labels in both Arabic and English. All of the above is scar tissue from building them.

They're on my Apify page