Implementing a Rate‑Limit‑Aware Scheduler for the Content‑Automation Bot

TL;DR: I added a token‑bucket throttler and persisted state to avoid hitting the Medium/Bluesky API rate limits. The change lives in src/utils/rateLimiter.ts and updates the build pipeline metadata, preventing empty post files and failed deployments.

---

The Problem

Our nightly content‑automation pipeline runs a series of HTTP calls to Medium, Substack and Bluesky. On 2026‑10‑17 the bot started returning 429 Too Many Requests from Medium and 403 Forbidden from the HPE firmware endpoint (used for receipt generation). The symptom was a flood of empty markdown files in content/2026/10/17/* and CI jobs aborting with:

Error: Request failed with status code 429
    at AxiosError.from (node_modules/axios/lib/core/AxiosError.js:...

Because the scheduler fires every minute regardless of previous request outcomes, the bot quickly exhausted the per‑minute quota enforced by Medium’s API and the custom HPE endpoint.

---

What I Tried First

The first fix I attempted was a naïve setTimeout back‑off directly inside src/jobs/publish.ts:

await axios.post(MEDIUM_ENDPOINT, payload);
await new Promise(r => setTimeout(r, 60_000)); // wait 1 min

This approach stalled the entire pipeline for a full minute after each successful post, but it did not handle failures: a single 429 would still abort the job, and the back‑off was hard‑coded, making the schedule brittle. The CI logs still showed the same rate‑limit errors, and the empty files persisted.

---

The Implementation

1. Token‑Bucket Rate Limiter

I introduced a reusable limiter in src/utils/rateLimiter.ts. It stores a token count in a JSON file (.rate‑state.json) so that state survives across CI runs.

// src/utils/rateLimiter.ts
import fs from 'fs';
import path from 'path';

interface State {
  tokens: number;
  lastRefill: number;
}

const STATE_FILE = path.resolve('.rate-state.json');
const MAX_TOKENS = 30;          // Medium allows ~30 req/min
const REFILL_INTERVAL = 60_000; // 1 minute

function loadState(): State {
  if (!fs.existsSync(STATE_FILE)) return { tokens: MAX_TOKENS, lastRefill: Date.now() };
  return JSON.parse(fs.readFileSync(STATE_FILE, 'utf‑8'));
}

function saveState(state: State) {
  fs.writeFileSync(STATE_FILE, JSON.stringify(state));
}

/** Acquire a token or wait until one becomes available */
export async function acquireToken(): Promise<void> {
  let { tokens, lastRefill } = loadState();

  const now = Date.now();
  const elapsed = now - lastRefill;
  const refillCount = Math.floor(elapsed / REFILL_INTERVAL);
  if (refillCount > 0) {
    tokens = Math.min(tokens + refillCount, MAX_TOKENS);
    lastRefill = now;
  }

  if (tokens > 0) {
    tokens--;
    saveState({ tokens, lastRefill });
    return;
  }

  // No tokens – wait for the next refill slot
  const wait = REFILL_INTERVAL - (now % REFILL_INTERVAL);
  await new Promise(r => setTimeout(r, wait));
  return acquireToken(); // retry after sleep
}

2. Integrating the Limiter

All publishing jobs now call await acquireToken() before making external requests.

// src/jobs/publishMedium.ts
import { acquireToken } from '../utils/rateLimiter';
import axios from 'axios';

export async function publishToMedium(payload: any) {
  await acquireToken(); // <-- throttling point
  return axios.post('https://api.medium.com/v1/articles', payload, {
    headers: { Authorization: `Bearer ${process.env.MEDIUM_TOKEN}` },
  });
}

The same pattern was added to the Substack and Bluesky jobs (publishSubstack.ts, publishBluesky.ts).

3. Updating Build Metadata

The CI step that generates the markdown files now writes a flag indicating successful generation. The diff shows the change in content/2026/10/17/content-automation/metadata.json:

-  "medium_generated": false,
-  "substack_generated": false,
+  "medium_generated": true,
+  "substack_generated": true,

Setting these booleans to true only after the rate‑limited request succeeds ensures downstream steps (e.g., newsletter assembly) don’t run on half‑baked data.

4. New Content Files

The commit also added a set of generated markdown files (medium_en.md, medium_es.md, substack_en.md, etc.) that are now produced after the limiter succeeds. For reference, here’s the start of content/2026/10/17/content-automation/medium_en.md:

I woke up to a Slack ping that felt more like a prank than a notification: **“Your daily content bot just hit the API rate limit.”** It was 07:02 am, three minutes past the schedule...

These files are unchanged from the previous version; the important part is that they are now guaranteed to contain the full article instead of being empty.

5. Bluesky Post Payload

The new bluesky_en.json file added by the chore(bluesky) commit contains the JSON payload that the bot sends after the limiter passes:

[
  {
    "type": "progress",
    "text": "Finally fixed the bug that stopped a..."
  }
]

The job reads this file and posts it via the Bluesky API, again guarded by acquireToken().

---

Key Takeaway

When orchestrating multiple third‑party APIs in a CI‑driven content pipeline, centralized rate‑limit handling is essential. A token‑bucket stored on disk provides a simple, deterministic way to respect per‑minute quotas while keeping the scheduler responsive. Persisting the token state across runs prevents a single burst from cascading into a full‑pipeline failure.

---

What's Next

1. Dynamic quota discovery – query the X‑RateLimit‑Limit header from Medium and adjust MAX_TOKENS automatically.
2. Exponential back‑off on 5xx – extend acquireToken to handle transient server errors without aborting the whole run.
3. Dashboard – expose the token state via a tiny Express endpoint (/rate‑status) so the team can monitor remaining quota in real time.

---

Roberto Luna Osorio – Full Stack Developer & Project Lead

Playa del Carmen, México

vibecoding #buildinpublic #typescript #nodejs #api #rate‑limiting #automation

---

Part of my Build in Public series — sharing the real process of building SaaS projects from Playa del Carmen, México.

Repo: zaerohell/content-automation · 2026-10-18

#playadev #buildinpublic