The Invisible Character Hiding In We Work Remotely's Job Descriptions

#webscraping #python #api #apify

Quick answer

We Work Remotely's job descriptions contain invisible characters that survive HTML-stripping. Specifically: a zero-width joiner (U+200D), used by the site's post editor as filler for what's meant to be an empty paragraph between sections. Python's \s regex class does not match U+200D — it's not whitespace as far as Unicode's own whitespace property is concerned — so a "strip tags, collapse whitespace" cleaning pass leaves it sitting there, invisibly, inside your "clean plain text" field. We found it live in 4 of 91 postings in today's feed, including a real one currently up: UTTR's "Project Manager" listing.

Watching it survive the actual cleaning pipeline 👻

Here's the Actor's real clean_description function — selectolax to strip tags, a regex to collapse whitespace — run against the live RSS feed:

import re, html
from selectolax.parser import HTMLParser

WHITESPACE_RE = re.compile(r"\s+")
ZWJ = "‍"

raw_description = "...‍As a Project Manager at UTTR..."  # from the live <description> tag, unescaped
text = HTMLParser(raw_description).text(separator=" ")
cleaned = WHITESPACE_RE.sub(" ", text).strip()

print(ZWJ in cleaned)        # True
print(cleaned.count(ZWJ))    # 4, in this one description alone

That ‍ between "career." and "As a Project Manager" above isn't a typo in this article — it's the actual invisible character, pasted straight from the live feed, still there after the exact HTML-stripping + whitespace-collapsing logic this Actor ships. If your downstream code does len(description), a substring search, an exact-match dedup, or a hash-based change-detection check, it is now working against a string with 1-4 phantom characters your eyes will never see and most text editors won't show you either.

Why the packed title split is not the trap you'd expect

We Work Remotely packs every job title as "Company: Position" in one RSS <title> field, and this Actor splits it on the first ": " only. The obvious worry is a company name that itself contains a colon — split on the wrong one and you get garbage in both fields. We went looking for a live example and found real ones:

"IxDF - Interaction Design Foundation: Course Director: UX, UI, and AI"
 -> company:  "IxDF - Interaction Design Foundation"
 -> position: "Course Director: UX, UI, and AI"

Splitting on the first ": " gets this right, because the colon that matters — the one separating company from role — always comes first, and a colon legitimately inside the position (as here) comes after. Good design decision, confirmed against a real edge case rather than assumed. We're calling this out specifically because it's the trap that looks scarier than the one that actually bites (the invisible character above), and it's worth knowing which of your assumptions have already been checked against live data and which haven't.

What we handle for you 🛡️

Every fetch goes out with a real Chrome/Firefox TLS fingerprint via curl-cffi impersonation and retries with backoff on 408/429/5xx, even though this feed answers a bare curl with a plain 200 today — that's a standing rule for every target we build against, not a reaction to something WWR does. A malformed RSS item is logged and skipped rather than taking down the run, multi-category requests are merged and de-duplicated by guid before your maxItems cap is applied, and every sparse field — country, state, job_type, region — comes back as null when the source feed leaves it empty, instead of an empty string you have to remember to check for separately.

Output

{
  "guid": "https://weworkremotely.com/remote-jobs/uttr-project-manager",
  "url": "https://weworkremotely.com/remote-jobs/uttr-project-manager",
  "company": "UTTR",
  "position": "Project Manager",
  "category": "Sales and Marketing",
  "job_type": "Full-Time",
  "region": "Anywhere in the World",
  "country": null,
  "state": "New York",
  "skills": [],
  "description": "Headquarters: New York URL: https://www.uttr.com/ UTTR stands for...",
  "logo_url": "https://wwr-pro.s3.amazonaws.com/logos/0171/6351/logo.gif",
  "pub_date": "2026-09-09T15:41:42+00:00",
  "expires_at": "2026-10-09T15:41:42+00:00"
}

from apify_client import ApifyClient

client = ApifyClient("<YOUR_API_TOKEN>")
run = client.actor("DevilScrapes/weworkremotely-jobs-scraper").call(
    run_input={"categories": ["design", "product"], "maxItems": 100}
)
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["company"], "—", item["position"])

Pricing: $0.20 per run plus $0.0015 per row — $1.70 per 1,000 results. A run that finds nothing after your filters still succeeds, and costs only the start fee.

→ We Work Remotely Jobs Scraper on Apify

Built by Devil Scrapes. We rotate real browser TLS fingerprints and retry with backoff on every fetch, and we go looking for the invisible character in the feed before a customer's dedup pipeline finds it for us.

DEV Community