Quick answer: Our release pipeline refused to ship this Actor because it "requested residential proxy". It didn't. The word RESIDENTIAL appeared in a module docstring, in a sentence explaining that the Actor deliberately does not use residential proxy. The gate was grepping source text for a word instead of reading the effective configuration, and it cost the Actor a publication slot.
What was the actual line?
A comment, near enough to this:
"""...uses the shared datacenter pool rather than RESIDENTIAL egress."""
A gate that greps for RESIDENTIAL matches that. It cannot tell a configuration value from prose, and it certainly cannot tell an assertion from its negation — the sentence says the exact opposite of what the gate concluded.
This is the third time the same defect has fired here, one syntax layer deeper each time: first a JSON description field, then a wrapped description= keyword argument, now a module docstring. Each fix taught the regex about one more shape. None of them fixed the design.
Grepping text for a word is not reading configuration. A check whose input is prose will eventually be fooled by prose, and the fix is to change the input, not to sharpen the pattern.
The durable version reads what the Actor will actually send:
def effective_proxy_tier(actor_input: ActorInput) -> str:
"""Resolve the tier the run will really use — from config, not from source text."""
cfg = actor_input.proxy_configuration
if cfg and cfg.apify_proxy_groups:
return ",".join(sorted(cfg.apify_proxy_groups))
return "DATACENTER" # a bare useApifyProxy is datacenter, not residential
That last line is its own trap, and worth stating plainly: a bare useApifyProxy: true with no groups is DATACENTER. Assuming it means residential is how you get a green local run and a 403 in the cloud.
Why does a false positive matter if it fails safe?
Because "fails safe" is doing a lot of work in that sentence. The gate is a money guard — anti-bot-class Actors need extra proof before they ship — so a false positive is genuinely safer than a false negative, and nobody should loosen it casually.
But it isn't free. Apify allows 5 publications per rolling 24 hours. A gate that wrongly holds an Actor doesn't just delay it — it burns a slot that no longer exists by the time the misread is found. The cost of a false positive is measured in capacity, not in minutes.
The honest tension on the day: a second, unrelated Actor was tripping the same gate from prose in a completely different file. It would have been easy to widen the exception list twice and ship both. We fixed one shape, pinned all four known-good shapes and the shapes that must still trip it in a test, and left the other alone rather than loosen a money guard twice in one morning under deadline pressure.
So what is Freelancer.com's real defence?
Considerably less interesting than the gate, which is the joke. https://www.freelancer.com/jobs/<skill> is server-rendered HTML that comes back without a login, without a token and without a browser. The work is in the parsing, not the access:
- Budgets are ranges in mixed currencies (
$250 - $750 USD,₹12500 - ₹37500 INR), so a row needs min, max and currency as separate fields rather than one string. -
Avg Bidis sometimes absent entirely, and an absent average is not0. - The same project appears under multiple skill tags, so a two-skill run must dedup by project ID.
Pagination is a plain ?page=N offset. Two skills × two pages returned 199 rows with zero overlap on a live cloud run — the check that proves the loop walked rather than refetched page 1 four times.
{
"project_id": 39284617,
"title": "Build a Python scraper for real estate data",
"url": "https://www.freelancer.com/projects/python/Build-Python-scraper-for-real/",
"skills": ["Python", "Web Scraping", "Data Mining"],
"budget_min": 250,
"budget_max": 750,
"budget_currency": "USD",
"avg_bid": 412.0,
"bid_count": 27,
"days_left": 6
}
😈 Freelancer.com Projects Scraper pulls live project listings by skill or category — title, full description, skill tags, budget range with currency, average bid, bid count and time remaining — paginated and deduped across skills. We handle the blocks, the retries, the mixed-currency budgets and the cross-skill duplicates, so you get comparable rows instead of strings. $4.00 per 1,000 projects.
FAQ
Do I need a Freelancer.com account?
No. These are public project-listing pages. No login, no API key, no token.
Are budgets comparable across currencies?
They are returned faithfully as budget_min / budget_max / budget_currency rather than converted at a rate that would be stale by the time you read it. Convert at your own reference rate.
Will I get the same project twice if I scrape two skills?
No. Projects are deduped by project ID across every skill in the run.
What if a project has no average bid?
The field is null, never zero — an absent average and an average of zero are different facts.
Does it use residential proxy?
No. It runs on the datacenter pool, which is what the docstring at the top of this article was trying to say before a regex disagreed.
Top comments (0)