Keywords as code: syncing a keyword list from Git to a keyword database over REST

#api #seo #automation #githubactions

Keyword lists tend to live in spreadsheets, and spreadsheets drift. Someone adds a tab, someone else sorts a column, and three months later nobody knows which version the content plan was built on. If your pages are planned around keywords, the keyword list is part of the site's source, and it can be treated like source: one file in Git, changed through pull requests, synced automatically.

This post builds that sync for the keyword database in Crawl Foundry, using the write endpoints of its REST API. The API basics (keys, envelope, error codes) were covered in an earlier post in this series, so this one goes straight to the parts that make a sync safe to run again and again.

The file

# seo/keywords.yaml
lists:
  - name: Pricing
    tags: [commercial]
    keywords:
      - seo tool pricing
      - pay per use seo tool
  - name: Site audit guides
    tags: [informational, technical]
    keywords:
      - how to do an seo audit
      - find broken links
      - crawled currently not indexed

Lists group keywords by page or topic, tags carry the cross-cutting labels such as intent. Deciding which keywords belong together is a separate question; the post on keyword clustering covers that.

The endpoints

All of these are workspace-scoped and require an Idempotency-Key header:

POST /keywords adds up to 100 keywords per request and returns an ID for each
POST /keyword-lists creates a list
POST /keyword-lists/{listId}/members adds up to 100 keyword IDs to a list
POST /keyword-tags creates a tag with one of seven colors
PATCH /keyword-tags/{tagId}/assignments assigns or unassigns up to 100 keyword IDs

The keyword endpoint is the friendly one for a sync. Its response reports insertedCount, existingCount and restoredCount separately, so sending a keyword that already exists is not an error. You get its ID back either way.

Lists and tags are different. They follow uniqueness rules, and creating one that already exists can be rejected with 409 RESOURCE_CONFLICT. The simplest fix is to create each list and tag once and remember its ID in a small state file committed next to the YAML.

The sync script

// scripts/sync-keywords.mjs
import { readFile, writeFile } from "node:fs/promises";
import { createHash } from "node:crypto";
import { parse } from "yaml";

const BASE = `https://app.crawlfoundry.com/api/v1/workspaces/${process.env.CF_WORKSPACE}`;
const STATE_FILE = "seo/.keyword-sync.json";
const today = new Date().toISOString().slice(0, 10);

async function call(method, path, body) {
  const json = JSON.stringify(body);
  const key = createHash("sha256").update(`${today} ${method} ${path} ${json}`).digest("hex");
  const res = await fetch(BASE + path, {
    method,
    headers: {
      Authorization: `Bearer ${process.env.CRAWLFOUNDRY_API_KEY}`,
      "Content-Type": "application/json",
      "Idempotency-Key": `kw-sync-${key.slice(0, 48)}`,
    },
    body: json,
  });
  const out = await res.json();
  if (!out.success) throw new Error(`${method} ${path}: ${res.status} ${out.code}`);
  return out.data;
}

const chunks = (arr, size = 100) =&gt;
  Array.from({ length: Math.ceil(arr.length / size) }, (_, i) =&gt; arr.slice(i * size, (i + 1) * size));

const config = parse(await readFile("seo/keywords.yaml", "utf8"));
const state = JSON.parse(await readFile(STATE_FILE, "utf8").catch(() =&gt; '{"lists":{},"tags":{}}'));

for (const list of config.lists) {
  const ids = [];
  for (const batch of chunks(list.keywords)) {
    const data = await call("POST", "/keywords", { keywords: batch.map((keyword) =&gt; ({ keyword })) });
    console.log(`${list.name}: ${data.insertedCount} new, ${data.existingCount} existing, ${data.restoredCount} restored`);
    ids.push(...data.items.map((item) =&gt; item.id));
  }

  state.lists[list.name] ??= (await call("POST", "/keyword-lists", { name: list.name })).item.id;
  for (const batch of chunks(ids)) {
    await call("POST", `/keyword-lists/${state.lists[list.name]}/members`, { keywordIds: batch });
  }

  for (const label of list.tags ?? []) {
    state.tags[label] ??= (await call("POST", "/keyword-tags", { label, color: "teal" })).item.id;
    for (const batch of chunks(ids)) {
      await call("PATCH", `/keyword-tags/${state.tags[label]}/assignments`, { operation: "assign", keywordIds: batch });
    }
  }

  await writeFile(STATE_FILE, JSON.stringify(state, null, 2) + "\n");
}

It needs Node 18 or newer for the built-in fetch, plus npm install yaml.

Why the idempotency key looks like that

The key is a hash of the date, the method, the path and the body. Three properties fall out of that:

Retries within a run are free of side effects. If a request times out and you rerun the script the same day, every request that already succeeded gets its original result replayed instead of executing twice.
It survives a crash between two steps. If the script dies after creating a list but before writing the state file, the rerun sends the identical create request with the identical key and gets the original list back, ID included. No duplicate, no conflict.
A changed file is a new request. Add a keyword and the body hash changes, so the key changes.

The date is in there so that a run next week is a genuinely new request rather than a replay of an old one. And the key is hex on purpose: the header only accepts printable ASCII without spaces or commas.

Running it on merge

# .github/workflows/keywords.yml
name: sync-keywords
on:
  push:
    branches: [main]
    paths: ["seo/keywords.yaml"]
permissions:
  contents: write
jobs:
  sync:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
      - run: npm ci
      - run: node scripts/sync-keywords.mjs
        env:
          CRAWLFOUNDRY_API_KEY: ${{ secrets.CRAWLFOUNDRY_API_KEY }}
          CF_WORKSPACE: ${{ vars.CF_WORKSPACE }}
      - name: Commit new list and tag IDs
        run: |
          git config user.name "github-actions[bot]"
          git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
          git add seo/.keyword-sync.json
          git diff --cached --quiet || (git commit -m "chore: update keyword sync state" &amp;&amp; git push)

The path filter only watches the YAML, so the state commit does not trigger another run.

What the sync deliberately does not do

It never removes anything. Deleting a line from the YAML leaves the keyword in the database and in its list. That is a choice, not an omission. A keyword in the database collects history over time, such as tracked positions and metrics, and an accidental deletion in a pull request should not throw that away.

Removals have their own endpoints, POST /keyword-lists/{listId}/member-removals for list membership and an archive call per keyword, and I would keep them as a deliberate, manual step. Additive syncs are safe to automate. Removals deserve a human looking at them.

On cost: the reference documents these organizing calls with the billing classification included. Fetching fresh metrics for the keywords, such as search volume, is a separate paid action, and Crawl Foundry bills that pay-per-use without a subscription obligation. The keyword management page shows what lists, tags and workflow statuses look like in the product, and the keyword research glossary entry covers where the keywords come from in the first place. Every endpoint used here is in the API reference.

DEV Community