Ingest SERP Data on a Schedule with Cloudflare Workers (Cron + KV)

#serverless #api #javascript

You don't need a VPS to track rankings overnight. A Cloudflare Worker with a cron trigger can call a SERP API once a day, store each snapshot in KV, and fire a webhook when something moves — for roughly zero dollars at a dozen-keywords scale. This post ships the whole worker in one file.

The design in three lines

A cron trigger fires scheduled() once per day.
Each run fetches results for every keyword in config, writing one KV key per keyword per day.
A rank beyond your threshold (or an API error) posts a compact alert to a webhook.

The worker

// src/index.js — wrangler deploy and forget
const API_URL = "https://api.serpbase.dev/google/search";

const KEYWORDS = [
  { q: "best mechanical keyboard", threshold: 10 },
  { q: "mechanical keyboard vs membrane", threshold: 15 },
  { q: "quiet office keyboard", threshold: 10 },
];
const OWN_DOMAIN = "example.com";
const ALERT_WEBHOOK = "https://hooks.example.com/abc"; // Slack/Discord style

export default {
  async scheduled(controller, env, ctx) {
    const today = new Date().toISOString().slice(0, 10);
    const alerts = [];

    for (const { q, threshold } of KEYWORDS) {
      const resp = await fetch(API_URL, {
        method: "POST",
        headers: {
          "X-API-Key": env.SERPBASE_API_KEY,
          "Content-Type": "application/json",
        },
        body: JSON.stringify({ q, hl: "en", gl: "us" }),
      });
      const data = await resp.json();

      // Errors carry credits_charged: 0 — log and move on, don't spam the webhook
      if (data.status !== 0) {
        console.error(`${q}: ${data.status} ${data.error}`);
        continue;
      }

      const hit = (data.organic || []).find((r) => (r.link || "").includes(OWN_DOMAIN));
      const rank = hit ? hit.rank : null;

      await env.SNAPS.put(`${q}:${today}`, JSON.stringify({
        rank,
        top5: (data.organic || []).slice(0, 5).map((r) => r.link),
      }));

      if (rank === null || rank > threshold) {
        alerts.push(`⚠️ ${q}: ${rank === null ? "not in top 10" : `#${rank}`} (limit #${threshold})`);
      }
    }

    if (alerts.length) {
      await fetch(ALERT_WEBHOOK, {
        method: "POST",
        headers: { "Content-Type": "application/json" },
        body: JSON.stringify({ text: `SERP alerts ${today}\n` + alerts.join("\n") }),
      });
    }
  },
};

Request params, the status/error envelope, and the organic field table are documented in the SerpBase search endpoint docs — the worker adds scheduling and storage, nothing else. The wrangler.toml you need is six lines: name, main, compatibility_date, and a [triggers] crons = ["0 6 * * *"] plus the KV namespace binding SNAPS.

The KV layout (and why one key per day)

Key	Value	Used by
`{q}:{date}`	`{rank, top5}`	The daily write; the alert check
`{q}:history` (optional)	appended array	Trend charts, week-over-week diffs

One key per keyword per day keeps writes atomic and reads trivial: list({ prefix: q + ":" }) walks the whole history. Keep top5 alongside your own rank — when your position drops, the question "who took my slot?" is answerable from the same snapshot without a second API call.

What this costs

Workers' free tier covers this workload comfortably (a dozen fetches a day is nothing). The SERP side is 1 credit per successful request; the prepaid packs run $0.50/1k at entry down to $0.30/1k at the top tier, the $3/month Starter Boost covers 10,000 searches, and new accounts start with 100 free searches — at one keyword per day, that trial alone runs you three months.

Two failure modes to plan for

The webhook becomes the SPOF. If the alert POST fails, the data is still in KV — log the failure and let the next run re-alert only if the condition still holds. Don't build retry state in the worker; KV already remembers yesterday.
Cron runs in UTC. 0 6 * * * is 6:00 UTC. If your users search at 9am in Berlin, your "daily snapshot" should probably land before that, not after.

FAQ

Why KV instead of D1/R2? At one JSON blob per keyword per day, KV is the least machinery that works. Move to D1 when you want SQL across keywords (that's the analytics problem, not the ingestion problem).

Why not check 100 keywords per run? Each is 1 credit; 100/day is ~3,000/month — real money at pack prices, and Worker CPU time starts to matter. Daily snapshots for your top 10–20 head terms, and let the long tail live in your batch pipeline.

Can I add more endpoints (news, maps)? Yes — same envelope, different payload key, and note images/maps cost 2 credits. Factor that into the cron budget.

Deploy it, wire the webhook to your team chat, and the next time a ranking moves you'll hear about it from your own bot — not from a client.

DEV Community