DEV Community

datalume
datalume

Posted on

How to Scrape Google Jobs in Python (2026): Full Descriptions and Real Apply Links

Google Jobs is the job-listings box that shows up when you search for something like "nurse jobs houston". It aggregates postings from company career sites, LinkedIn, Indeed, ZipRecruiter and hundreds of smaller boards, which makes it one of the best free sources of job data on the web.

Getting that data out with Python in 2026 is harder than most tutorials suggest. Here is what I ran into while building a scraper for it, with code for each step and real numbers from runs on 8 November 2026.

The three traps

  1. Plain HTTP requests get a JavaScript wall. Since early 2025, requests.get("https://www.google.com/search?q=...&udm=8") returns a page that asks you to enable JavaScript instead of results.
  2. There is no page 2. The jobs view shows about 10 jobs. start=10 returns the same 10, and the rest only load by scrolling inside Google's own app.
  3. The apply links look like Google redirects (/goto?url=CAES...). Resolving them one by one is slow, and from datacenter IPs you mostly get a "Verifying your request" page instead of a redirect.

Each one has a workaround that does not involve solving captchas or faking a browser fingerprint.

Step 1: get the HTML

udm=8 is the parameter that opens the Jobs view directly:

from urllib.parse import quote_plus

def jobs_url(query, hl="en", gl="us"):
    return f"https://www.google.com/search?q={quote_plus(query)}&udm=8&hl={hl}&gl={gl}"
Enter fullscreen mode Exit fullscreen mode

Because of the JavaScript wall, you need a SERP API or a SERP proxy that returns the rendered results page as raw HTML. Any provider that gives you the full HTML works. I used Apify's Google SERP proxy, which returns the complete page (about 1 MB, 10 jobs) for each request.

Step 2: parse the job cards

Each job card is a div (class EimVGf at the time of writing) with a data-share-url attribute. That URL contains the job's stable ID (htidocid), which you will use later for deduplication.

from urllib.parse import urlparse, parse_qs
from bs4 import BeautifulSoup

def parse_cards(html):
    soup = BeautifulSoup(html, "html.parser")
    jobs = []
    for card in soup.select("div.EimVGf[data-share-url]"):
        share = card["data-share-url"]
        job_id = parse_qs(urlparse(share).query).get("htidocid", [None])[0]
        title = card.select_one(".tNxQIb")
        company = card.select_one(".a3jPc")
        jobs.append({
            "jobId": job_id,
            "title": title.get_text(strip=True) if title else None,
            "company": company.get_text(strip=True) if company else None,
            "googleJobsUrl": share,
        })
    return soup, jobs
Enter fullscreen mode Exit fullscreen mode

Class names like .tNxQIb are Google's minified names. They change from time to time, so keep the selectors in one place and check them when your fill rate drops.

Step 3: the details live in <template> tags

The full description, the highlights (Qualifications / Benefits / Responsibilities) and the apply buttons are not in the card. They are in a <template id="j<jobId>"> element further down the page.

Here is the trap that cost me an hour: BeautifulSoup's get_text() skips text inside <template>. Your selectors match, but every string comes back empty. Re-parse the template's inner HTML as its own document:

def details(soup, job_id):
    tpl = soup.find("template", id="j" + job_id.rstrip("="))
    if tpl is None:
        return None
    # get_text() ignores strings inside <template>, so parse its contents separately
    return BeautifulSoup(tpl.decode_contents(), "html.parser")
Enter fullscreen mode Exit fullscreen mode

Step 4: real apply URLs without a single extra request

The apply buttons point at /goto?url=.... But the same template has an element with a data-maindata attribute that already contains the real URLs as jobs_posting_urls_1, jobs_posting_urls_2 and so on, in the same order as the buttons:

import re

URL_RX = re.compile(r'\["jobs_posting_urls_(\d+)","(.*?)"\]')

def unescape_js(s):
    return re.sub(r"\\u([0-9a-fA-F]{4})", lambda m: chr(int(m.group(1), 16)), s)

def apply_links(tpl):
    data = next((el["data-maindata"] for el in tpl.select("[data-maindata]")
                 if "jobs_posting_urls" in el["data-maindata"]), "")
    urls = {int(n): unescape_js(u) for n, u in URL_RX.findall(data)}
    links = []
    for i, a in enumerate(tpl.select('a[href^="/goto"]'), 1):
        site = (a.get("title") or "").replace("Apply directly on ", "").replace("Apply on ", "")
        links.append({"site": site, "url": urls.get(i)})
    return links
Enter fullscreen mode Exit fullscreen mode

No redirects, no extra requests, no "Verifying your request" pages.

One caveat. Google merges copies of a job from many boards into one listing, and sometimes it merges a different job by mistake. In one run, an Apple software engineer listing had 8 apply links, and 4 of them pointed to other companies' jobs (at Meta, Anthropic and Paramount). If you need strict links, compare the job title with the URL slug and drop links that clearly name a different job.

Step 5: more than 10 jobs per search

Since there is no page 2, ask Google for different slices of the same search. The Jobs filter chips simply add phrases to the query, so you can repeat the search with those phrases and merge the results by jobId:

FILTERS = ["", "full time", "remote", "part time", "contract",
           "since yesterday", "in the last 3 days", "in the last week", "no degree"]

def collect(query, fetch_html):
    seen = {}
    for f in FILTERS:
        soup, jobs = parse_cards(fetch_html(jobs_url(f"{query} {f}".strip())))
        for job in jobs:
            seen.setdefault(job["jobId"], job)
    return list(seen.values())
Enter fullscreen mode Exit fullscreen mode

On 8 November 2026 I ran this for 20 US searches (software engineer in Austin, nurse in Houston, barista in Portland …):

Metric Result
Requests 180 (20 searches × 9 filters)
Pages that loaded 172 / 180 (95.6%)
Unique jobs per search 65 on average (39 to 83)
Jobs with full description and real apply URL 1,546 / 1,546
Time 7 minutes at 4 requests in parallel

Niche roles return fewer jobs. To get more, split a broad query ("nurse") into specific ones ("ICU nurse", "travel nurse").

Outside the US

The same code works in other countries with hl and gl (e.g. hl=de&gl=de). On the same day, Germany, Japan, the UK, Taiwan, Hong Kong, Singapore, India and France all returned title, company, location, apply URL and description for 10 of 10 jobs. Two differences:

  • Job highlights appear only on US listings.
  • The filter phrases are English, so for other languages use the date filter (tbs=qdr:d, qdr:w, qdr:m) to get different slices.

Google Jobs is not available in South Korea or mainland China, so searches there return nothing.

If you don't want to maintain this

Selectors drift and the details above take time to keep working, so I packaged all of this as an Apify Actor: Google Jobs Scraper. You enter job titles and locations, and it handles the fetching, filter fan-out, deduplication, the template parsing and the titleMatch check on apply links. It costs from $1.70 per 1,000 jobs, and empty searches and duplicates are free.

There are ready-made examples for remote software engineer jobs from the last 24 hours, Houston nurse jobs with salaries and tracking which roles a company is hiring for.


I run datalume, a small set of public-data scrapers on Apify. This article was written with AI assistance; all numbers come from real runs on 8 November 2026. Not affiliated with Google.

Top comments (0)