Job listings are the most time-sensitive content on the web. A job posted today might be closed in 3 days. If Google takes 4–8 weeks to crawl it naturally, the listing is dead before it ever ranks.
I run DailyJobFeed — a remote job aggregator that scrapes 10 sources every morning. From day one I knew I couldn't rely on passive crawling. So I built a dual indexing pipeline: the Google Indexing API for Google Jobs, and IndexNow for Bing, Yandex, and DuckDuckGo.
Both fire automatically every morning right after new jobs are inserted. Here's how the approach works.
Why passive crawling isn't enough for job boards
Google's normal crawl cycle for a new domain is weeks. Even for established domains, new pages can sit unindexed for days.
For evergreen content that's fine. For job listings it's fatal — by the time the page ranks, the role is filled.
Two APIs solve this:
| API | Engines | Quota | Best for |
|---|---|---|---|
| Google Indexing API | Google only | 200 URLs/day (free) | Google Jobs JobPosting schema |
| IndexNow | Bing, Yandex, DuckDuckGo, others | 10,000 URLs/batch (free) | Everything else |
They're complementary. Google doesn't participate in IndexNow (yet), so you need both.
Part 1 — Google Indexing API
Google maintains a dedicated Indexing API for pages with JobPosting or BroadcastEvent schema. It's the only Google API that pushes content into the index instead of waiting for crawl.
One-time setup
- Create a Google Cloud project (free, no billing required for this API)
- Enable the Indexing API in the project
- Create a service account and download the JSON key
- Add the service account email as an Owner in Google Search Console for your property
How it works in Python
The general pattern looks like this:
import os
import json
import requests
def get_access_token(key_json: str) -> str:
"""Exchange service account credentials for a bearer token."""
from google.oauth2 import service_account
import google.auth.transport.requests
info = json.loads(key_json)
scopes = ["https://www.googleapis.com/auth/indexing"]
creds = service_account.Credentials.from_service_account_info(info, scopes=scopes)
creds.refresh(google.auth.transport.requests.Request())
return creds.token
def notify_google(urls: list[str], action: str = "URL_UPDATED"):
"""
Ping Google Indexing API for a list of URLs.
action: "URL_UPDATED" for new pages, "URL_DELETED" for expired ones.
"""
key_json = os.environ.get("GOOGLE_INDEXING_KEY_JSON")
if not key_json:
return # no-op without credentials
token = get_access_token(key_json)
endpoint = "https://indexing.googleapis.com/v3/urlNotifications:publish"
headers = {"Authorization": f"Bearer {token}", "Content-Type": "application/json"}
for url in urls:
response = requests.post(
endpoint,
headers=headers,
json={"url": url, "type": action},
timeout=10
)
if response.status_code == 429:
print("Quota hit — stopping batch")
break
Key design decisions
No-op without credentials — if the env var isn't set, the function returns silently. A missing credential never crashes the cron run.
Quota awareness — the free tier is 200 URLs/day. On hitting a 429, stop the batch rather than retrying. Quota increases are available via Google's support form.
URL_DELETED for expired content — when a job listing expires (I serve HTTP 410 Gone after 3 months), pinging URL_DELETED tells Google to drop it from the index immediately rather than waiting for a recrawl to discover the 410.
Part 2 — IndexNow
IndexNow is an open protocol supported by Bing, Yandex, DuckDuckGo, Naver, and others. One POST fans out to all participating engines simultaneously.
Google is "evaluating" IndexNow but doesn't use it yet — which is exactly why you need both APIs.
Setup — one static key file
IndexNow requires a key verification file at your domain root:
# Generate any random string as your key
KEY = "your-random-key-here"
# Host it at:
https://yourdomain.com/{KEY}.txt
# File content must equal the key string exactly
In Nuxt/Vercel, drop the file in public/ and it gets served at root automatically.
Important: deploy the key file before making any API calls — IndexNow returns 403 if the file isn't there yet.
Verify it's live:
curl https://yourdomain.com/your-random-key-here.txt
# Should return: your-random-key-here
Note: the IndexNow key is public by design — it's not a secret. The file just proves you own the domain.
How it works in Python
import os
import requests
INDEXNOW_ENDPOINT = "https://api.indexnow.org/indexnow"
def notify_indexnow(urls: list[str]):
"""Submit URLs to IndexNow — fans out to Bing, Yandex, DuckDuckGo."""
key = os.environ.get("INDEXNOW_KEY")
if not key or not urls:
return # no-op without key
site_url = os.environ.get("SITE_URL", "https://yourdomain.com")
host = site_url.replace("https://", "").rstrip("/")
payload = {
"host": host,
"key": key,
"keyLocation": f"{site_url}/{key}.txt",
"urlList": urls, # up to 10,000 per request
}
response = requests.post(
INDEXNOW_ENDPOINT,
json=payload,
headers={"Content-Type": "application/json"},
timeout=30
)
if response.status_code in (200, 202):
print(f"IndexNow: submitted {len(urls)} URLs")
elif response.status_code == 403:
print("IndexNow 403 — key file not found at keyLocation")
elif response.status_code == 422:
print("IndexNow 422 — host mismatch or malformed URL")
Why the quota difference matters
Google Indexing API caps you at 200 URLs/day on the free tier. IndexNow accepts 10,000 URLs per POST with no meaningful daily cap.
For a new site with a backlog of unindexed pages, IndexNow is the answer. I used it to submit all existing job slugs to Bing in one batch after launch — something the Google API quota wouldn't allow.
Wiring both into your cron
The general pattern in your main scraper script:
def run():
# 1. Clean up expired content
expired_urls = cleanup_expired_jobs()
# 2. Scrape and insert new jobs — returns URLs of newly inserted rows
new_urls = scrape_and_insert()
# 3. Ping Google for new content
if new_urls:
notify_google(new_urls, "URL_UPDATED")
# 4. Ping IndexNow for new content
if new_urls:
notify_indexnow(new_urls)
# 5. Ping IndexNow for expired content
# (skipping Google URL_DELETED here to save quota for new content)
if expired_urls:
notify_indexnow(expired_urls)
Both functions are no-ops if their env vars aren't set, so local dev and staging are unaffected.
Why send expired URLs to IndexNow but not Google
Google's URL_DELETED counts against the 200/day quota. On a site with rolling expiry, expired pages could eat the whole quota before new pages get notified.
IndexNow has no meaningful quota cap, so expired URLs go there freely. Bing revalidates the now-410 URLs and drops them from its index. Google discovers the 410 on its next natural crawl — fast enough given the expiry window is months, not hours.
Results after enabling both
- Google Jobs listings started appearing in Google's job search within hours of the first scraper run (vs. days or weeks of passive crawling)
- GSC indexed page count grew noticeably faster
- Bing started surfacing job pages — small traffic contribution but worth the minimal setup effort
Environment variables
# .env — never commit real values
GOOGLE_INDEXING_KEY_JSON= # full JSON blob from GCP service account
INDEXNOW_KEY= # your random key string
SITE_URL=https://yourdomain.com
Both are optional — the helpers no-op when the var is absent. Set them in your hosting platform's environment variables (Railway, Heroku, Render, etc.).
Should you build this?
If your site has:
- High-volume, time-sensitive content (jobs, events, news, product listings)
- Programmatic page generation at scale
- A crawl budget you can't afford to waste on passive discovery
Yes, absolutely. Both APIs are free, the implementation is straightforward, and the speed difference on indexing is significant — especially for Google Jobs where JobPosting schema + the Indexing API is Google's own recommended combination.
The only real friction is the one-time Google Cloud service account setup. Everything else is an env var and a static file.
Live example: dailyjobfeed.com — remote jobs updated daily, indexed within hours.
Top comments (0)