Build a small job-market tracker with Google Jobs, Python and SQLite

Google Jobs (the job panel you get when you search "data engineer jobs in Austin") pulls together postings from LinkedIn, ZipRecruiter, company career sites, Glassdoor and many smaller boards. That makes it a good single source if you want to follow one slice of the job market over time: a role in a few cities, a skill or a set of companies.

We'll build a tracker that:

1. snapshots Google Jobs for a few searches once a day,
2. stores every posting in SQLite and keeps the date it was first and last seen,
3. prints the new postings, the top hiring companies and the median advertised salary.

It's about 90 lines of Python and uses only the standard library plus apify-client.

Disclosure: the scraper used here is an Apify Actor published by Atlas Data, the author of this post: atlasdata/google-jobs-scraper-reliable. It is pay-per-result and the script sets a cost cap on every run. More examples (Node.js, curl, MCP) are in the GitHub repo.

Setup

pip install apify-client   # tested with apify-client 3.4
export APIFY_TOKEN=...      # https://console.apify.com/settings/integrations

What one result looks like

The input mirrors the Google Jobs filters: queries, location, country, language, datePosted (any, today, 3days, week, month), jobType, remoteOnly, maxResults, plus includeDescription and includeApplyLinks to control how much is returned per job.

Each posting comes back as a flat record. These are the fields we'll use:

Field	Example
jobId	stable ID from Google, used as our primary key
title, companyName, location	Senior Data Engineer, General Motors, Austin, TX
via	where Google found it, e.g. LinkedIn, ZipRecruiter, Talent.com
jobType	Full-time
postedAt	ISO date derived from "3 days ago"
salaryMin, salaryMax, salaryCurrency, salaryPeriod	95000, 112000, USD, year
applyUrl	first apply link
searchQuery	the query that produced the row

salary also keeps the raw text ($95K–$112K a year) in case you want to parse it yourself.

1. The searches

SEARCHES = [
    {"queries": ["data engineer"], "location": "Austin, TX", "country": "us"},
    {"queries": ["data engineer"], "location": "Seattle, WA", "country": "us"},
]
COMMON = {"datePosted": "week", "maxResults": 30, "includeDescription": False}

datePosted: "week" is a good choice for a daily job: each run covers a week of postings, and a missed day doesn't leave a gap. includeDescription: False keeps rows small. Turn it on if you want to count skills in job descriptions later.

A tip on localization: match country (and, if you set it, language) to the market you're searching, e.g. "country": "de" with a German query such as "Softwareentwickler". In our tests, an English query with a forced language: "en" in a non-English market sometimes returned an empty Google Jobs panel. The run's OUTPUT record then reports NO_DATA for that search, so it doesn't fail silently.

2. A table that remembers

db = sqlite3.connect("jobs.db")
db.execute("""
CREATE TABLE IF NOT EXISTS jobs (
    job_id TEXT PRIMARY KEY,
    title TEXT, company TEXT, location TEXT, via TEXT, job_type TEXT,
    salary_min REAL, salary_max REAL, salary_currency TEXT, salary_period TEXT,
    posted_at TEXT, apply_url TEXT, search_query TEXT,
    first_seen TEXT, last_seen TEXT
)""")

first_seen never changes after the first insert. last_seen is refreshed on every run that still finds the job. With these two columns you get "new today" (first_seen = today) and "probably filled or closed" (last_seen older than a few days) without storing full snapshots.

3. Fetch and upsert

def fetch(search: dict) -> list[dict]:
    run = client.actor("atlasdata/google-jobs-scraper-reliable").call(
        run_input={**COMMON, **search},
        max_total_charge_usd=Decimal("0.30"),
        logger=None,
    )
    if run is None or run.status != "SUCCEEDED":
        raise RuntimeError(f"run failed: {run and run.status}")
    return list(client.dataset(run.default_dataset_id).iterate_items())


def upsert(job: dict, today: str) -> bool:
    """Insert a job or refresh last_seen. Returns True when the job is new."""
    is_new = db.execute("SELECT 1 FROM jobs WHERE job_id = ?", (job["jobId"],)).fetchone() is None
    db.execute(
        """INSERT INTO jobs VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)
           ON CONFLICT(job_id) DO UPDATE SET last_seen = excluded.last_seen""",
        (
            job["jobId"], job["title"], job["companyName"], job.get("location"), job.get("via"),
            job.get("jobType"), job.get("salaryMin"), job.get("salaryMax"), job.get("salaryCurrency"),
            job.get("salaryPeriod"), job.get("postedAt"), job.get("applyUrl"), job.get("searchQuery"),
            today, today,
        ),
    )
    return is_new

ON CONFLICT ... DO UPDATE (SQLite 3.24+) turns the insert into an upsert, so the script can be re-run safely any number of times.

4. Simple market stats

    print("\nTop hiring companies (last 30 days):")
    for company, n in db.execute(
        """SELECT company, COUNT(*) FROM jobs WHERE last_seen >= date('now', '-30 day')
           GROUP BY company ORDER BY COUNT(*) DESC LIMIT 10"""
    ):
        print(f"  {n:3d}  {company}")

    yearly = [
        (lo + hi) / 2
        for lo, hi in db.execute(
            """SELECT salary_min, salary_max FROM jobs
               WHERE salary_period = 'year' AND salary_currency = 'USD'
                 AND salary_min >= 10000 AND salary_max IS NOT NULL"""  # skip non-annual figures
        )
    ]

Only some postings include a salary. Print how many did, so a median based on 13 postings isn't read as covering all 60. The salary_min >= 10000 condition is a cheap sanity check that keeps hourly or monthly figures out of an annual median.

Running it

First run (real output):

$ python jobs_tracker.py
Austin, TX: 30 jobs
Seattle, WA: 30 jobs
60 new postings today

Top hiring companies (last 30 days):
    2  ICF
    2  Amazon
    2  Alignerr
    1  Vaspire Technologies
    ...

Median advertised salary (USD/year): 166,500 from 13 of 60 postings that list one

From the second day on, "new postings today" only counts jobs that weren't seen before. Put it in cron:

0 6 * * *  cd /path/to/tracker && APIFY_TOKEN=... python jobs_tracker.py >> tracker.log 2>&1

or use an Apify schedule if you'd rather not keep a machine running. The companion Google Trends post shows how.

Full script

"""Daily Google Jobs snapshot -> SQLite, with simple market stats."""
import os
import sqlite3
from datetime import date
from decimal import Decimal
from statistics import median

from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])

SEARCHES = [
    {"queries": ["data engineer"], "location": "Austin, TX", "country": "us"},
    {"queries": ["data engineer"], "location": "Seattle, WA", "country": "us"},
]
COMMON = {"datePosted": "week", "maxResults": 30, "includeDescription": False}

db = sqlite3.connect("jobs.db")
db.execute("""
CREATE TABLE IF NOT EXISTS jobs (
    job_id TEXT PRIMARY KEY,
    title TEXT, company TEXT, location TEXT, via TEXT, job_type TEXT,
    salary_min REAL, salary_max REAL, salary_currency TEXT, salary_period TEXT,
    posted_at TEXT, apply_url TEXT, search_query TEXT,
    first_seen TEXT, last_seen TEXT
)""")


def fetch(search: dict) -> list[dict]:
    run = client.actor("atlasdata/google-jobs-scraper-reliable").call(
        run_input={**COMMON, **search},
        max_total_charge_usd=Decimal("0.30"),
        logger=None,
    )
    if run is None or run.status != "SUCCEEDED":
        raise RuntimeError(f"run failed: {run and run.status}")
    return list(client.dataset(run.default_dataset_id).iterate_items())


def upsert(job: dict, today: str) -> bool:
    """Insert a job or refresh last_seen. Returns True when the job is new."""
    is_new = db.execute("SELECT 1 FROM jobs WHERE job_id = ?", (job["jobId"],)).fetchone() is None
    db.execute(
        """INSERT INTO jobs VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)
           ON CONFLICT(job_id) DO UPDATE SET last_seen = excluded.last_seen""",
        (
            job["jobId"], job["title"], job["companyName"], job.get("location"), job.get("via"),
            job.get("jobType"), job.get("salaryMin"), job.get("salaryMax"), job.get("salaryCurrency"),
            job.get("salaryPeriod"), job.get("postedAt"), job.get("applyUrl"), job.get("searchQuery"),
            today, today,
        ),
    )
    return is_new


def main() -> None:
    today = date.today().isoformat()
    new_jobs = 0
    for search in SEARCHES:
        jobs = fetch(search)
        new_jobs += sum(upsert(job, today) for job in jobs)
        print(f'{search["location"]}: {len(jobs)} jobs')
    db.commit()
    print(f"{new_jobs} new postings today")

    print("\nTop hiring companies (last 30 days):")
    for company, n in db.execute(
        """SELECT company, COUNT(*) FROM jobs WHERE last_seen >= date('now', '-30 day')
           GROUP BY company ORDER BY COUNT(*) DESC LIMIT 10"""
    ):
        print(f"  {n:3d}  {company}")

    yearly = [
        (lo + hi) / 2
        for lo, hi in db.execute(
            """SELECT salary_min, salary_max FROM jobs
               WHERE salary_period = 'year' AND salary_currency = 'USD'
                 AND salary_min >= 10000 AND salary_max IS NOT NULL"""  # skip non-annual figures
        )
    ]
    total = db.execute("SELECT COUNT(*) FROM jobs").fetchone()[0]
    if yearly:
        print(f"\nMedian advertised salary (USD/year): {median(yearly):,.0f} "
              f"from {len(yearly)} of {total} postings that list one")


if __name__ == "__main__":
    main()

Ideas for extending it

• Time-to-close: julianday(last_seen) - julianday(first_seen) per company shows how fast roles disappear.
• Skills: set includeDescription: True and count mentions of dbt, Spark, Airflow, etc. per week.
• Remote share: run the same query with remoteOnly: True and compare counts over time.
• Where jobs are posted: GROUP BY via shows whether a market leans on LinkedIn, company career sites or niche boards.

The Actor's Store page documents every input and output field, and the GitHub repo has runnable examples. To query jobs from an AI assistant instead, the MCP setup is in atlas-data-mcp.

Comments

[Dexoryn]

I like that this goes beyond collecting job listings and starts tracking how the market changes over time.

Knowing which roles keep appearing, which companies are hiring, and what salaries are being advertised can reveal patterns that a normal job search misses.

Adding skill-demand trends would make this even more useful for deciding what to learn next.