DEV Community

Daniel Meshulam
Daniel Meshulam

Posted on

10,789 American employers publish their whole job list in one HTML page

A Paylocity careers page looks like a React app:

https://recruiting.paylocity.com/recruiting/jobs/All/8f0bcd76-3ee7-42f2-b040-83afc6e6d564
Enter fullscreen mode Exit fullscreen mode

Open the network tab and there is no job API call to find. The jobs are
already there, in the HTML, in one global:

import json, re, requests

UA = {"User-Agent": "Mozilla/5.0"}
guid = "8f0bcd76-3ee7-42f2-b040-83afc6e6d564"
html = requests.get(f"https://recruiting.paylocity.com/recruiting/jobs/All/{guid}",
                    headers=UA).text
data = json.loads(re.search(r"window\.pageData\s*=\s*(\{.*?\});\s*\n", html, re.S).group(1))

data["ModuleTitle"]        # 'City of Columbia'
len(data["Jobs"])          # 9
Enter fullscreen mode Exit fullscreen mode

And a job is not a card scraped off a page, it is a record:

{'JobId': 4514780,
 'JobTitle': 'Building and Grounds Maintenance Part Time',
 'PublishedDate': '2026-09-17T13:43:33-05:00',
 'IsInternal': False, 'IsRemote': False,
 'JobLocation': {'Address': '1648 Trottwood Ave.', 'City': 'Columbia',
                 'State': 'TN', 'Zip': '38401', 'Country': 'USA'}}
Enter fullscreen mode Exit fullscreen mode

A street address. Most career systems give you "Columbia, TN" if you are
lucky and "Ridley Park - Parks and Recreation" if you are not.

recruiting.paylocity.com answers 404 to /robots.txt, so there is nothing
there to disallow anything.

Four things that bite

The assignment ends at ;\n. Job descriptions contain semicolons. A
greedy match, or a match that stops at the first ;, both give you a broken
object. The newline is what makes it unambiguous.

A retired board answers 200. When an employer leaves Paylocity the guid
keeps serving a portal page, just without pageData. So "no pageData" is the
proof a board is gone; the status code never says so.

JobId is a global sequence, not a per-board one. Measured over ten
unrelated boards: the ranges interleave (a 40-person academy's ids sit inside
a hospital group's range) and no id appeared at two employers. That matters
because an employer can run several modules, and the same posting shows up on
more than one. Key on the id alone and they collapse correctly.

LocationName is not a location. It is whatever the employer labelled the
site: "Public Works", "Columbia Aquatics & Recreation Center", sometimes a
person's name. The address object is the only place a city comes from.

How many there are

A board is a guid, and nothing derives a guid from a company domain, so the
list has to come from somewhere. The Wayback CDX index holds every URL it has
archived on that host:

https://web.archive.org/cdx/search/cdx?url=recruiting.paylocity.com
  &matchType=domain&collapse=urlkey&fl=original&output=json
  &filter=original:.*[Rr]ecruiting/[Jj]obs/[Aa]ll/.*
Enter fullscreen mode Exit fullscreen mode

Two things about that query cost me boards before they earned them. limit
stops at the first page and looks complete: the first sweep returned exactly
20,000 URLs and 14,836 guids, which is a suspiciously round stopping point.
And the filter is a case-sensitive regex, while the archive holds the path
capitalised as well; fixing the case alone found 1,821 more.

Read live on 2026-09-24: 16,936 candidate boards, 13,581 that still answer,
10,789 with open roles, 146,046 jobs.
One GET each, six at a time, a
quarter of a second apart: 41 minutes for the lot, and not one request
refused.

The employers are the interesting part. This is not the Fortune 500, which
every jobs dataset already has three times over. It is DOCS Health, a Taco
Bell franchise group with 2,219 openings, Farmers Home Furniture, credit
unions, dialysis clinics, car dealerships, county governments, country clubs.
The half of the American labour market that never appears in an enterprise
ATS index.


If you want the rows rather than the code, that is what
ats-jobs-scraper does: a
company name in, every open posting out, from Paylocity and twenty-one other
career systems, as JSON.

Top comments (0)