DEV Community: Get Anything

Scrape Google Play: apps, details & reviews for market research (no API key)

Get Anything — Thu, 17 Sep 2026 06:32:25 +0000

If you're doing app market research — tracking competitors, mining reviews for feature requests, or watching ratings over time — you need structured Google Play data: app details, install counts, ratings, and user reviews. Here's how to get it without the Play Console or any API key.

What's useful to pull

Search apps by keyword or category
App details: installs, rating, price, developer, version, last update, screenshots
Reviews: text, star rating, date, thumbs-up count — the goldmine for product insight
Any country store and language

The easy way

The Google Play Scraper does search + details + reviews across any country store. No API key.

{ "mode": "reviews", "appId": "com.whatsapp", "country": "ae", "maxReviews": 500 }

One clean row per app or review — export to JSON, CSV or Excel, or pull via the API.

What to do with it

Competitor tracking — monitor a rival app's rating and review volume over time
Review mining — run reviews through an LLM to cluster complaints and feature requests
ASO research — see which apps rank for a keyword and how they're positioned
Due diligence — installs, ratings and update cadence at a glance

Connect it to Claude or ChatGPT

Every Apify actor is callable over MCP. Point Claude Desktop at https://mcp.apify.com and ask it to "pull the latest reviews for this app and summarize the top complaints." Guide: Connect Apify actors to Claude & ChatGPT.

Try it: Google Play Scraper. Public store data only.

Get live gold prices in any currency, per gram/tola/karat (no API key)

Get Anything — Thu, 17 Sep 2026 06:31:40 +0000

Most "gold price API" services are paywalled and quote only USD per troy ounce. But if you're building anything for the Gulf or India — a jewelry site, a savings app, a dashboard — you need per gram, per tola, per karat, in AED/INR/SAR. Here's how to get exactly that.

The units that actually matter

Jewelry is sold per gram and by karat (24k/22k/21k/18k), not per ounce.
South Asian markets quote the tola (~11.66 g).
Buyers want their local currency, not USD.

So the useful output is: spot price converted to your currency, then divided into grams/tola and scaled by karat purity.

The easy way

The Gold Price Scraper returns gold, silver, platinum and palladium in any currency, per gram/tola/ounce and by karat — built on live futures + FX. No API key.

{ "metals": ["gold", "silver"], "currency": "AED", "karats": [24, 22, 21, 18] }

Output is one clean row per metal/unit, ready for a site, app or sheet.

Use cases

Jewelry & retail — show today's 22k/gram price in local currency
Savings & fintech apps — value gold holdings live
Dashboards & alerts — track spot moves on a schedule

Connect it to Claude or ChatGPT

Every Apify actor is callable over MCP. Point Claude Desktop at https://mcp.apify.com and ask "what's 22k gold per gram in AED today?" Guide: Connect Apify actors to Claude & ChatGPT.

Try it: Gold Price Scraper.

Get any YouTube video's transcript as clean text for your LLM (no API key)

Get Anything — Thu, 17 Sep 2026 06:30:33 +0000

Feeding a YouTube video to an LLM starts with one thing: the transcript. Here's how to get it as clean, token-efficient text for summarization, Q&A, or a RAG knowledge base — without the YouTube Data API or any key.

Why not the official API?

The YouTube Data API doesn't return caption text without OAuth and extra hoops, and it has quotas. For transcripts, the captions track is public for most videos — you just fetch and clean it.

What you want out

The full transcript as plain text and as Markdown
Cleaned for LLMs: HTML entities decoded, newlines collapsed
Metadata (title, channel, duration) as a bonus
Works with auto-generated captions, falls back across languages

The easy way

The YouTube Transcript Scraper does exactly this. Give it video URLs and it returns pre-cleaned transcript and markdown fields, only charging for videos that actually have captions.

{ "videoUrls": ["https://www.youtube.com/watch?v=..."], "language": "en" }

Pipe the markdown straight into your model:

import requests
items = requests.get("https://api.apify.com/v2/datasets/<DATASET_ID>/items?format=json").json()
context = items[0]["markdown"]

Use cases

Summarize long talks and lectures in one prompt
Q&A over videos — chunk the transcript into a vector DB
Repurpose a video into a blog post or thread
Build a searchable archive of a channel

Connect it to Claude or ChatGPT

Every Apify actor is callable over MCP. Point Claude Desktop at https://mcp.apify.com and ask it to get a video's transcript and summarize it. Guide: Connect Apify actors to Claude & ChatGPT.

Try it: YouTube Transcript Scraper. Public captions only — respect creators' rights.

Connect any Apify scraper to Claude or ChatGPT in 2 minutes (MCP)

Get Anything — Thu, 17 Sep 2026 05:49:02 +0000

You can give Claude, ChatGPT, Cursor, or any MCP client the ability to run real web scrapers and pull live data — search results, web pages behind Cloudflare, business leads, job listings — without writing any integration code. Here's how, using the Apify MCP server.

The one thing you need to know

Apify hosts an MCP server at https://mcp.apify.com. Point any MCP client at it and your AI can discover and run any of thousands of Apify Actors (scrapers), read their results, and adapt to new tasks on the fly.

Claude Desktop (easiest)

Settings → Connectors → Add custom connector
Server URL: https://mcp.apify.com
Your browser opens once to sign in to Apify and approve access.

That's it — ask Claude "search the web for X" or "scrape this URL" and it runs the tool.

Any MCP client (Cursor, VS Code, Claude Code…)

Add this to your MCP config:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com"
    }
  }
}

Prefer a token over OAuth? Add a header:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com",
      "headers": { "Authorization": "Bearer <APIFY_TOKEN>" }
    }
  }
}

Get your token from the Apify Console → API & Integrations.

Load only the tools you want

By default the server loads discovery tools + a web browser. To expose specific scrapers as ready-to-call tools, list them in the URL:

https://mcp.apify.com?tools=get_anything/serp-scraper,get_anything/web-unblocker,get_anything/contact-extractor

A few of mine that are genuinely useful to LLM workflows:

serp-scraper — Google/Bing/DuckDuckGo results as JSON for RAG grounding
web-unblocker — fetch any URL past Cloudflare/DataDome and return clean Markdown
llm-web-scraper — any URL/site → LLM-ready Markdown
contact-extractor — emails/phones/socials from a list of domains
google-maps-leads — local businesses with phone, website & email

No-token discovery mode

For discovery/testing you can connect with no token by loading only the read-only tools:

https://mcp.apify.com?tools=search-actors,fetch-actor-details,search-apify-docs,fetch-apify-docs

Running an Actor (and getting data back) needs a token or OAuth sign-in.

Why this matters

Instead of copy-pasting data into your prompt, your AI fetches it live — grounded, current, and structured. That's the difference between an LLM that guesses and one that knows.

Try it: point Claude Desktop at https://mcp.apify.com and ask it to scrape something. Full docs: Apify MCP server.

Google Maps scrapers give you the phone. Here's how to get the email too

Get Anything — Thu, 17 Sep 2026 05:08:47 +0000

Google Maps is the best source of local business leads — but most Maps scrapers stop at name, address and phone. The field that actually closes deals, email, isn't on Maps at all. Here's how to get it.

The extra step that matters

Maps gives you the business website. So after scraping each listing, take one more hop: fetch the business's own site (and its contact page) and pull the email. That single step turns a directory dump into a real outreach list.

Verified on a live run for coffee shop, Dubai Marina:

Business	Phone	Email	Rating
Roast Speciality Coffee	+971 50 502 5854	info@roastdubai.com	4.6
Roasters Specialty Coffee	+971 58 572 5444	beans@roasterscoffee.ae	4.9
Drinkit coffee	—	ask@drinkit.io	4.8

How it works

Scroll the Maps results feed and collect every place.
Open each place for name, category, address, phone, website, rating, reviews.
For each website, fetch the site + a contact page and extract the email.

Input

{
  "searchQueries": ["marketing agency", "dentist"],
  "location": "Dubai",
  "maxResults": 30,
  "extractEmails": true
}

Use it for

Local lead generation — niche + city prospect lists with email.
Agencies — find local businesses to pitch web/marketing services.
Market research — competitor density, ratings, categories by area.

Try it: Google Maps Leads Scraper on Apify. Public data only — respect GDPR/CAN-SPAM when contacting leads.

Turn a list of domains into a lead list: emails, phones, socials

Get Anything — Thu, 17 Sep 2026 05:07:49 +0000

The most universal sales task there is: "I have a list of company websites — get me their emails and phone numbers." Here's how to do it reliably, and cheaply, without a browser.

Where the data actually hides

Businesses rarely put their email on the homepage. It's on the contact / about / imprint / support page. So the trick isn't just fetching one URL — it's following the few same-domain links that look like contact pages, then scanning those too.

The highest-confidence sources are mailto: and tel: links; after that, scan visible text with a well-filtered regex (drop asset filenames like logo@2x.png, tracking IDs, and example addresses).

Pure HTTP is enough (and cheap)

No browser needed. curl_cffi with Chrome TLS impersonation fetches the pages; BeautifulSoup + regex pull the contacts. That keeps cost per site tiny — which matters when you're processing thousands of domains.

Input / output

{
  "startUrls": [{ "url": "https://example.com" }, { "url": "acme.io" }],
  "crawlContactPages": true,
  "maxPagesPerSite": 5
}

{
  "domain": "apify.com",
  "emails": ["hello@apify.com", "support@apify.com"],
  "phones": ["+1 ..."],
  "socials": { "linkedin": "...", "twitter": "...", "youtube": "..." }
}

Use it for

Lead generation — enrich a list of company domains with reachable contacts.
Sales prospecting — pair it with a Google Maps or directory scraper: one finds the businesses, this gets their contacts.
CRM enrichment — fill missing email/phone fields.

Ready-made: Contact & Email Extractor on Apify. Public pages only — and respect anti-spam / data-protection laws (GDPR, CAN-SPAM) when you reach out.

A search-results scraper for LLMs that doesn't return zero

Get Anything — Thu, 17 Sep 2026 05:06:29 +0000

Everyone building RAG or an LLM agent eventually wants the same thing: live search results, fed into the model. And everyone hits the same wall — the Google-only SERP scrapers keep dying on the /sorry CAPTCHA page the moment they run from a cloud IP.

I tested it directly: from datacenter and even residential cloud IPs, Google's CAPTCHA wall is a coin-flip. So the honest answer isn't "beat Google" — it's use an engine that actually returns data every run.

Bing is the pragmatic default

Bing returns organic results comparable to Google, is far more tolerant of automated access, and exposes the same fields you need: title, URL, snippet, position, related searches. So the SERP Scraper defaults to Bing, with DuckDuckGo as a second reliable engine and Google as best-effort (with an honest disclaimer, not a false promise).

One gotcha: residential IP personalization

Early on, a query for "best web scraping tools" returned… adult Reddit threads. The cause: residential proxy IPs are real users' connections, and their Bing session/personalization leaks into your results. The fix:

clear cookies before each query,
force mkt (market) + SafeSearch,
don't carry any session cookie.

After that, results are clean and query-driven every time.

Input

{
  "queries": ["apify web scraping platform"],
  "engine": "bing",
  "maxResults": 20,
  "country": "us",
  "language": "en"
}

You get one row per organic result:

{ "query": "apify web scraping platform", "engine": "bing", "position": 1,
  "title": "Apify: The largest marketplace of trusted tools for AI", "url": "https://apify.com", "domain": "apify.com" }

Use it for

RAG / LLM grounding — pair it with a URL-to-Markdown reader to build a full "search + read" web browser for your model.
SEO / rank tracking — per country and language.
Keyword research — harvest Related Searches.

Try it: SERP Scraper on Apify. No API key; public search-results data only.

How to tell which anti-bot a website uses (and fetch the page anyway)

Get Anything — Thu, 17 Sep 2026 05:05:22 +0000

If you scrape the web, you know the wall: a 403, a 429, or a "Just a moment…" page that never resolves. Before you sink a day into building a scraper, two questions matter:

Which anti-bot is protecting this site? (Cloudflare, DataDome, PerimeterX/HUMAN, Akamai, Imperva…)
Can I get the page anyway?

Here's a way to answer both in one call.

Detecting the anti-bot vendor

Each vendor leaves fingerprints in the response headers and body:

Vendor	Tell-tale signs
Cloudflare	`cf-ray` header, `Server: cloudflare`, "Just a moment", `challenge-platform`
DataDome	`x-datadome` header, `datadome` cookie, `captcha-delivery.com`
PerimeterX / HUMAN	`_px*` cookies, `px-captcha`
Akamai Bot Manager	`_abck` / `ak_bmsc` cookies
Imperva / Incapsula	`incap_ses` cookie, `x-iinfo` header

If you just want the answer without wiring this up yourself, the Web Unblocker actor returns a protectionDetected field for any URL.

Fetching the page anyway

The strategy that works from cloud IPs:

Fast path — an HTTP request with real Chrome TLS/JA3 impersonation (curl_cffi). Cheap, instant for unprotected pages.
Browser render — if that's blocked, render in a hardened browser (Camoufox — a fingerprint-hardened Firefox) which clears most JavaScript challenges.
Retry on a fresh IP — if it's still blocked, rotate to a new residential IP and try again.

One call, both answers

{
  "startUrls": [{ "url": "https://example.com/protected" }],
  "renderJs": "auto",
  "outputFormat": "html",
  "maxRetries": 2,
  "proxyConfiguration": { "useApifyProxy": true, "apifyProxyGroups": ["RESIDENTIAL"] }
}

You get back the final HTML (or clean Markdown, or a screenshot), the HTTP status, the anti-bot vendor detected, and a bypassed flag telling you whether the real page came through:

{
  "url": "https://example.com/protected",
  "success": true,
  "statusCode": 200,
  "protectionDetected": "Cloudflare",
  "bypassed": true,
  "method": "http"
}

When to use it

You keep getting 403/429 on a target and want the HTML to parse yourself.
You want to know what a site runs before committing to a scraper build.
You need protected pages as clean Markdown for a RAG/LLM pipeline.

Try it here: Web Unblocker on Apify. Only public pages, no logins — respect each site's terms.

GCC Tech Salaries in 2026: How to Track Vision 2030 Hiring & Pay with Live Data

Get Anything — Tue, 28 Jul 2026 04:28:37 +0000

The Gulf is in the middle of the biggest hiring wave in its history. The six GCC states are projected to hire around 2.5 million skilled professionals over three years, Saudi Arabia is targeting 50,000 new tech jobs under Vision 2030, and senior-developer pay in the UAE and Saudi Arabia is reportedly inflating 15–25% a year. Salaries overall are forecast up ~4.6% in Saudi and ~4.1% in the UAE for 2026.

If you're recruiting, job-hunting, or building a jobs product in this market, generic salary surveys are already stale by the time they publish. Here's how to track live GCC salary and hiring data yourself.

The problem: most Gulf job boards hide salary

Bayt, GulfTalent and NaukriGulf rarely publish pay. The two sources that do — Indeed and Glassdoor — sit behind Cloudflare and heavy anti-bot protection, so a normal scraper just hits a "Security Check" page.

The fix: pull salary-tagged jobs across the Gulf

Two ready-made tools get past the anti-bot walls (using a hardened, fingerprint-spoofing browser) and return structured jobs with salary:

Indeed Scraper — GCC Jobs, Salaries & Employers — 70–80% of GCC listings carry a salary range, plus employment type, remote flag and company rating.
Glassdoor Scraper — GCC Jobs & Salary Data — employer-provided salary ranges and company data.

Give each a keyword (e.g. "software engineer", "data scientist", "AI engineer") and one or more countries (UAE, Saudi Arabia, Qatar, Kuwait, Bahrain, Oman).

Example (Indeed) output

{
  "title": "Senior Software Engineer",
  "company": "…",
  "city": "Riyadh",
  "country": "Saudi Arabia",
  "employmentType": "full-time",
  "salaryText": "SAR 25,000 - SAR 35,000 a month",
  "salaryMin": 25000,
  "salaryCurrency": "SAR"
}

Track the trends that matter in 2026

Vision 2030 tech demand — software, full-stack, data science, AI/ML roles in Riyadh, NEOM, Dubai
Tech salary inflation — benchmark senior-dev pay month over month
Saudization — which roles are shifting to Saudi-national-only and how pay moves with it
Sector shifts — tourism, construction, healthcare, education (5,000+ teaching roles a quarter)

Want coverage first, salary second?

Start with the free flagship: GCC Jobs Search — searches Bayt, GulfTalent, NaukriGulf, Foundit and Tanqeeb together with filters for field, experience, city, salary and company, then layer Indeed/Glassdoor for salary depth.

Ask it in plain English

All three plug into ChatGPT/Claude via the Apify MCP server:

"What do senior data engineers earn in Riyadh right now?"

Tools: Indeed GCC · Glassdoor GCC · GCC Jobs Search (free). By Get Anything.

Build a Dubai Real Estate AI Agent with MCP (No Code) — Asking Price vs Sold Price

Get Anything — Tue, 28 Jul 2026 04:27:43 +0000

AI agents are only as good as the tools you give them. Ask ChatGPT or Claude "what's a fair price for a 1-bed in JVC?" and you'll get a vague, out-of-date guess — because the model has no live data. The fix in 2026 is MCP (Model Context Protocol): you connect real data tools to your assistant, and it answers with actual numbers.

Here's how to give an AI assistant two Dubai property superpowers — live asking prices and real sold prices — with no code.

The two tools your agent needs

The gap every property decision hinges on is asking price vs sold price:

Bayut Property Scraper — live listings (what sellers ask): price, beds, baths, area, agent, for sale or rent, across Dubai and the UAE.
Dubai DLD Transactions Scraper — Dubai Land Department registered transactions (what buyers paid): price per sqm, first-sale vs resale, by community — free, no API key.

Both are Apify Actors, and both are MCP-ready.

Connect them in ~2 minutes

Open the Apify MCP server.
Add the two Actors above as tools.
Point your MCP client (Claude Desktop, ChatGPT, Cursor, or any MCP-compatible app) at the server.

That's it — your assistant can now call them.

What you can now ask

"What's the average sold price per sqm for flats in Dubai Marina this month, first-sale vs resale?"

"Find 2-bed apartments for sale in JVC under AED 1.2M, then tell me how that compares to what 2-beds actually sold for there."

"Which communities had the most transactions last month, and what's the off-plan vs resale split?"

The agent calls Bayut for the listings, DLD for the registered prices, and reconciles them — the kind of answer that used to need an analyst and a spreadsheet.

Why this matters in 2026

Dubai just posted its biggest half-year on record (~AED 291.7bn, ~71% off-plan), and prices are up ~9%. In a market moving this fast, static reports are stale on arrival. An MCP agent wired to live listings + registered transactions gives you an always-current answer to "is this priced right?"

Beyond real estate

The same pattern works for hiring: connect the Indeed and Glassdoor GCC scrapers and ask your assistant "what do senior developers earn in Riyadh right now?"

Tools: Bayut · Dubai DLD Transactions. More AI-ready Gulf data tools by Get Anything.

Dubai Off-Plan vs Resale in 2026: What the DLD Transaction Data Actually Shows

Get Anything — Tue, 28 Jul 2026 04:27:42 +0000

Dubai's property market just had its biggest six months on record — 87,800 transactions worth around AED 291.7 billion in H1 2026, with average prices up ~9% and off-plan making up roughly 71% of all volume. Every headline is about off-plan. But if you're actually deploying capital, the question isn't "is off-plan hot?" — it's "first-sale or resale, in which community, at what price per sqm, right now?"

The only data that answers that is registered transaction data from the Dubai Land Department (DLD) — not the asking prices on the portals. Here's how to pull it and read the off-plan vs resale story yourself.

Why "off-plan vs resale" is the metric that matters

The DLD tags every sale with a sale sequence: first sale (off-plan / primary, straight from the developer) or resale (secondary market). That single flag tells you:

Where the primary pipeline is concentrated — first-sale volume clusters in emerging communities with new launches (Dubai South, JVC, Dubai Islands, Dubai Creek Harbour).
The premium/discount between primary and secondary — first-sale price per sqm vs resale price per sqm in the same community, the same month.
Momentum — rising resale volume in an off-plan-heavy area signals an exit market forming.

How to get it (free, no API key)

The DLD publishes this on its public transactions page. Instead of wrangling it by hand, the Dubai DLD Transactions Scraper turns it into clean JSON for any date range — one record per community × sub-type × sale sequence, with pricePerSqmAed, avgTransactionValueAed, transactionCount and totalValueAed, in English and Arabic.

Example: reading the split in JVC

For Jumeirah Village Circle — one of the highest-volume communities in 2026 — the data separates:

{ "community": "JUMEIRAH VILLAGE CIRCLE", "propertySubType": "Flat",
  "saleSequence": "First",  "transactionCount": 118, "pricePerSqmAed": 16218 }
{ "community": "JUMEIRAH VILLAGE CIRCLE", "propertySubType": "Flat",
  "saleSequence": "Second", "transactionCount": 110, "pricePerSqmAed": 12106 }

First-sale (off-plan) clears higher per sqm than resale in the same community — the "new-build premium" you can now quantify per area, per month.

What to build with it

Off-plan heat maps — first-sale volume by community over time
Primary vs secondary price gap trackers
Absorption / exit signals — when resale volume rises in an off-plan area
Valuation comparables grounded in registered prices, not listings

Pair it with live listings

To see asking vs sold in the same community, add the Bayut Property Scraper (live listings: price, beds, area, agent). Asking price from Bayut, sold price from DLD — the full gap in one view.

Ask it in plain English

Both tools plug into ChatGPT/Claude via the Apify MCP server:

"Compare off-plan vs resale price per sqm in Business Bay this quarter."

Tool: Dubai DLD Transactions Scraper · more Gulf data tools by Get Anything.

Scraping bilingual data: the Arabic/English traps that quietly corrupt your dataset

Get Anything — Mon, 27 Jul 2026 11:29:17 +0000

Most scraping advice assumes one language. Build for a bilingual market and you meet a category of bug that never throws an exception: the request succeeds, the HTML parses, and every field you wanted comes back empty.

Here is what I learned building job datasets that have to work in both Arabic and English.

The bug that returns null instead of failing

I was fetching job detail pages and extracting the stated seniority and employment type. The parser looked for the labels the page uses, matched them, and pulled the adjacent value. It worked in testing.

In production, seniorityStated and employmentType came back null for every record. No error. Status 200. Valid HTML.

The links I was following pointed at a regional subdomain. That subdomain served the Arabic version of the page. My selectors were matching on the English labels — "Seniority level", "Employment type" — and the page said المستوى الوظيفي and نوع التوظيف. Nothing matched. Nothing raised. The fields were simply absent.

The fix was one line: normalize the host before fetching.

import re

# Regional hosts serve localized pages whose labels don't match
# English selectors. Normalize to the canonical host.
url = re.sub(r"https://[a-z]{2}\.example\.com/", "https://www.example.com/", url)

The lesson generalizes: a scraper that finds nothing and a scraper that finds the wrong language look identical from the outside. If a field is null across every single record, suspect locale before you suspect your selector.

Assert on what you expect to find

The defence is cheap. After a parse, check that the fields you rely on actually populated:

critical = ("seniority", "employmentType")
missing = [f for f in critical if not any(r.get(f) for r in records)]
if missing:
    raise RuntimeError(f"No record has {missing} — wrong locale or stale selector?")

A scraper that crashes loudly on an empty column is worth ten that return tidy nulls.

Numerals are not the digits your regex expects

Arabic pages frequently use Eastern Arabic numerals. ٢٠٠ is two hundred.

Python is more helpful here than people assume, which is exactly why this bites. int("٢٠٠") returns 200. "٢٠٠".isdigit() is True. re.findall(r"\d+", ...) finds them, because \d is Unicode-aware by default.

The thing that silently fails is the character class almost everyone actually writes:

import re

text = "أكثر من ٢٠٠ متقدم"          # "more than 200 applicants"

re.findall(r"[0-9]+", text)          # []          <- silently empty
re.findall(r"\d+", text, re.ASCII)   # []          <- same trap
re.findall(r"\d+", text)             # ['٢٠٠']     <- works
int("٢٠٠")                           # 200         <- works

So a scraper that extracts applicant counts with [0-9]+ doesn't crash on an Arabic page. It reports that the page contains no numbers, forever, and you conclude the field is missing rather than that your regex is ASCII-only.

If you want ASCII digits in your output regardless of the source, translate rather than strip:

ARABIC_DIGITS = str.maketrans("٠١٢٣٤٥٦٧٨٩", "0123456789")

def to_int(text):
    return int(re.sub(r"\D", "", text.translate(ARABIC_DIGITS)))

I nearly published the claim that int() raises on Arabic numerals. It doesn't. I only found out because I ran the snippet before shipping the article, which is the entire moral of this post.

Don't let a substring match decide a language

Keyword matching across languages goes wrong in ways that are hard to see. Two real examples from classifying job titles:

Matching the seniority abbreviation coo as a substring also matches "Coordinator". A programme coordinator became a chief operating officer.
Matching partner as a substring promoted "Partner Onboarding Specialist" to firm partner.

Word boundaries fix both. But the general point is sharper in a bilingual dataset, where titles mix scripts, transliterations and English loanwords in the same string. re.search(r"\bcoo\b", title), never "coo" in title.

Print the rows behind the number

I was about to publish a statistic saying 18% of roles in a market were executive-level. It felt too high, so I printed the titles in that bucket. Half of them were coordinators and onboarding specialists. The real figure was 16%, and the composition was completely different.

Every heuristic you write will be wrong in a way that flatters your numbers. Print the rows.

Machine translation is for labels, not for meaning

If you add Arabic labels to an English dataset, keep the original. Store title and titleArabic side by side; never overwrite. Translation is lossy, brand names transliterate inconsistently, and someone downstream will want to match on the original string.

For job titles specifically, a translated label is a display convenience. Every join, dedupe and filter should still run on the source-language field.

Encoding, briefly

Set your encoding explicitly at every boundary: reading files, writing files, and printing. On Windows the default console encoding will happily raise UnicodeEncodeError on a perfectly good Arabic string, which sends you hunting for a scraping bug that doesn't exist.

sys.stdout.reconfigure(encoding="utf-8")
pathlib.Path(p).read_text(encoding="utf-8")

Also: a UTF-8 byte order mark at the start of a JSON file will make some parsers reject it outright. If a tool insists your valid JSON is invalid, check for a BOM before you check your syntax.

Where this came from

I maintain scrapers for regional job boards that output company and role labels in both Arabic and English. All of the above is scar tissue from building them.

They're on my Apify page