DEV Community

AIRabbit
AIRabbit

Posted on

I Scraped 301 Job Ads for $1.75 — The Top Employer Was a Bot

I had forty one tabs open and no idea whether the job market was good or bad.

Was I missing Kubernetes, or German? Were these real companies, or the same recruiter over and over?

No idea. And that's not a discipline problem — it's a format problem.

Every question I cared about was an aggregate question, and you can't answer one of those by scrolling.

So I turned the board into a table. Here's everything I typed:

LinkedIn jobs: AI engineer, Berlin
Enter fullscreen mode Exit fullscreen mode

301 postings. 38 seconds. $1.75.

I didn't write a scraper or touch a single selector.

What's actually running

Three pieces, and only one of them is you.

Piece Job
Apify Hosts ~100 pre-built scrapers ("actors") and keeps them working
The apify-ultimate-scraper skill Tells Claude which actor to pick and how to drive it
You Say what you want, in English

It's a skill, not an MCP server — a folder of instructions Claude reads before running the Apify CLI on your behalf.

You never touch the CLI.

Setup: two commands, once 

npm install -g apify-cli
apify login
Enter fullscreen mode Exit fullscreen mode

The free tier gives you $5/month in credits, which is a few thousand rows. apify login opens a browser.

Install the skill after that and you can forget all of this exists.

The prompt

This is the whole interface.

LinkedIn jobs: AI engineer, Berlin
Enter fullscreen mode Exit fullscreen mode

Behind that line, Claude picked an actor and looked up its input schema. It ran four paginated jobs, noticed two came back empty, and retried them.

Then it deduped 350 rows down to 301 and wrote the CSV.

I watched it happen without typing a command.

The follow-ups matter more than the scrape

The scrape is one line. What I asked next is where the answers came from, because at that point I was talking to a table instead of a feed.

The prompts I actually used, in order:

how many of these are actually in Berlin vs remote-EU?
Enter fullscreen mode Exit fullscreen mode 
what % of the descriptions mention German? use word boundaries,
"rag" is matching "storage"
Enter fullscreen mode Exit fullscreen mode 
group by company — who's posting the most?
Enter fullscreen mode Exit fullscreen mode 
pull out every posting with a salary band, sort by top of range
Enter fullscreen mode Exit fullscreen mode

Each took a few seconds. None of them are answerable on the site itself.

The shift is going from "how do I scrape this" to "what do I want to know."

Steering it when it goes sideways

A few things that got me unstuck:

Situation What to say
Run returns nothing the run succeeded but returned 0 rows — check the log
Wrong actor that actor needs account permissions, find one that doesn't
Suspicious numbers use word boundaries, not substring matching
Want more pull pages 2 through 4 and dedupe on job_id

The second one came up immediately. My first actor needed a one-time permission grant in the Apify Console — Claude hit the error, said so, and switched.

The third one is the mistake I'd have shipped. Substring rag matches "sto*rag*e", which quietly doubled my number.

And worth knowing: SUCCEEDED doesn't mean it worked. An empty dataset that finishes in three seconds is a failed run with a green checkmark on it. Ask for the log, not the status.

Finding 1: "Berlin" isn't Berlin 

Germany (nationwide)     96
Berlin, Berlin, Germany  69
European Union           59
Berlin, Germany          38
EMEA                     27
Enter fullscreen mode Exit fullscreen mode

109 of 301 are actually in Berlin.

Work type says the same thing: 216 Remote, 52 Hybrid, 33 On-site.

"Berlin" is a filter that returns Europe.

Finding 2: Agents ate RAG

Skill % of 301
Python 65%
LLMs 52%
agentic / AI agents 36%
AWS 30%
RAG 25%
Kubernetes 21%
PyTorch 14%
TensorFlow 11%
PhD 5%

PyTorch and TensorFlow combined only tie RAG on its own.

This market is asking for API wiring and production infra, not model training.

Kubernetes looks like a better use of a weekend than backprop.

Finding 3: German blocks less than you'd think

78% of postings never mention German.

If you're hunting in Berlin without it, that's more useful than any advice I'd read on the subject.

And it's invisible from the search page.

Finding 4: the top employer is a bot 

15  Hire Feed
 9  THRYVE
 5  Jobgether
 4  dexter health / SNI / Quik Hire Staffing
Enter fullscreen mode Exit fullscreen mode

Hire Feed posted 15 roles. Senior AI Software Engineer (Remote) EMEA. ML Engineer (Remote) EMEA. DevOps Engineer (Infrastructure) EMEA.

A recruiter aggregator, carpet-bombing the feed.

You can't spot this by scrolling — not because you're not paying attention, but because the platform interleaves those 15 among hundreds.

One group by and it's sitting right there.

Bonus: salary, where it's public

38 of 301 listed a band.

duvo.ai      AI Platform Engineer (EU remote)  €110K–220K
vCluster     AI Infrastructure Engineer        $150K–200K
PulseRise    Voice AI Engineer                  €85K–160K
HiveMQ       Senior AI Fullstack Engineer       €58K–110K
Enter fullscreen mode Exit fullscreen mode

Platform and infra top the range, not the model roles.

And the biggest numbers are in USD — US companies hiring into Europe.

When not to do this

The honest one first: I did this more for fun than out of necessity.

Nobody needed these numbers. I could have opened the site and applied to things like a normal person.

If you're actively hunting, an hour of applications beats an hour of questions.

What I got was the pleasure of turning a vague feeling into a table, plus four facts I wouldn't have noticed otherwise.

That's a real payoff. It just isn't a need, and a guide that pretends otherwise is selling you something.

The rest is more practical:

  • You need private data. No gated profiles, no emails, no connection graph. Public only.
  • You need complete fields. applicant_count was on 113 of 301 rows. skills was empty on nearly all of them. Public pages are lossy.
  • You're scraping people, not postings. Job ads are corporate content. 500 individuals' profiles is a GDPR problem wearing the same clothes.
  • You need it to be allowed. This breaks LinkedIn's ToS. Bans are the normal outcome, not the edge case. hiQ v. LinkedIn means it isn't a CFAA crime in the US — it doesn't make it contract-legal.

Top comments (0)