DEV Community

Devil Scrapes
Devil Scrapes

Posted on

LinkedIn has an /api/ endpoint that needs no login and returns HTML

#webscraping #python #api #linkedin

Quick answer: LinkedIn has a jobs endpoint that needs no login, no cookie and no token: GET /jobs-guest/jobs/api/seeMoreJobPostings/search?keywords=...&location=...&start=0. It returns HTTP 200 and ~30 KB — and it does not return JSON. Despite /api/ in the path, the body is a bare list of HTML <li> cards, which is why people probing it for a JSON object conclude it's broken.

An /api/ path that serves HTML?

Yes, and the naming is the trap. The endpoint exists to feed LinkedIn's own infinite scroll: the front-end asks for the next chunk of the results list and splices the returned markup straight into the DOM. There was never a reason for it to serve JSON.

So the parse target is job-search-card blocks, not keys:

CARD = "job-search-card"

def parse_cards(fragment: str) -> list[ResultRow]:
    """The body is an HTML fragment, not a document and not JSON."""
    soup = BeautifulSoup(fragment, "html.parser")
    return [row for card in soup.select(f"div.{CARD}") if (row := to_row(card))]
Enter fullscreen mode Exit fullscreen mode

It is also a fragment — no <html>, no <head>. Parsers that assume a document mostly cope, but it's worth knowing before you debug an empty selector.

Measured on the live endpoint: 29,996 B on a Chrome TLS profile, 30,034 B on Firefox — near-identical, which is the signal that no fingerprint-based defence is deciding what you get.

How does pagination work, and does it actually paginate?

A plain start offset in steps of 25. The check that matters is not that start=25 returns rows — it's that they are different rows:

start=0 and start=25 returned 10 job-posting IDs each, with zero overlap.

That distinction is the whole game. A surprising number of "paginating" scrapers refetch page 1 repeatedly and return a dataset that looks full and contains one page of jobs duplicated N times. Comparing ID sets across consecutive offsets costs one assertion and catches it immediately.

Our own cloud proof run: 120 rows from 2 keywords × 2 locations × 3 pages, no overlap, no always-empty fields.

Is scraping LinkedIn legal?

The honest answer is that this deserves a real sentence rather than a shrug, because LinkedIn is genuinely litigious about scraping in general — the hiQ v. LinkedIn line of cases exists for a reason.

What's specific here: this endpoint requires no login, so no terms are accepted by a session and no access control is circumvented. It returns public job postings that employers are paying to distribute as widely as possible. And it is the same public surface that rival Actors with a combined six-figure user count have run on for years. That is a materially different posture from scraping member profiles behind an authenticated session, which this Actor does not do and will not do.

Not legal advice. Know your jurisdiction and your own use case.

What does a row look like? 

{
  "job_id": "4021847592",
  "title": "Senior Python Engineer",
  "company": "Example Corp",
  "company_url": "https://www.linkedin.com/company/example-corp",
  "location": "Warsaw, Mazowieckie, Poland",
  "posted_at_text": "2 days ago",
  "job_url": "https://www.linkedin.com/jobs/view/4021847592",
  "keyword": "python developer",
  "search_location": "Poland"
}
Enter fullscreen mode Exit fullscreen mode

Every row carries the keyword and search_location that produced it — so when you run a matrix of searches, you can prove from the dataset alone which branches actually executed instead of trusting the row count.

😈 LinkedIn Jobs Scraper exports job postings by keyword and location from LinkedIn's public guest search — title, company, company URL, location, posting date and job URL — batched across a matrix of keywords and locations, paginated and deduped. We handle the blocks, the retries, the HTML-that-claims-to-be-an-API and the offset walking, so you get rows instead of a fragment. $3.00 per 1,000 jobs.

FAQ

Do I need a LinkedIn account or cookie?
No. The guest endpoint is unauthenticated by design — that is what makes this Actor possible and what keeps it on the public side of the line.

Can I search several keywords and locations at once?
Yes. Pass lists of both and the Actor walks the full matrix, fault-isolated, so one bad combination never zeroes the run.

Does it scrape member profiles?
No. Public job postings only.

How deep can I paginate?
Walk start in steps of 25 up to your per-search page cap. Set the cap by how fresh you need the tail to be.

Why did my own request return HTML when I expected JSON?
Because it always does. See the top of this article — /api/ in the path is not a promise about the content type.

Top comments (0)