DEV Community: Aakash Gour

Using PostAll's API to Automate Your Content Workflow: A Getting-Started Guide

Aakash Gour — Thu, 11 Jun 2026 06:17:24 +0000

I didn't set out to build a content API. I set out to stop copy-pasting.

Every week, the same ritual: open a doc, stare at a blank page, write a headline, delete it, write it again. Multiply that by every client, every product page, every email drip campaign. I wasn't doing creative work — I was doing assembly-line work while pretending it was creative.

PostAll started as a script I wrote to stop doing that. The API is what that script became after other developers asked if they could use it too.

This guide walks you through integrating PostAll's API into your own workflow — authentication, the endpoints you'll actually use, real working code in both Python and Node.js, and the specific places things will break before they work. By the end, you'll have a functioning pipeline that generates formatted, CMS-ready content programmatically.

What you'll build

A script that takes a list of content briefs (keywords, tone, target length) and returns publish-ready content — with proper formatting, metadata, and error handling for the rate limits you'll hit in production.

Here's the shape of what you're building:

[ CSV of briefs ] → [ PostAll API ] → [ formatted content objects ] → [ your CMS / database ]

The full working code for both languages is at the end of each section. I'll explain the interesting parts inline.

Prerequisites

A PostAll account with API access enabled (free tier works for this guide — rate limits noted below)
Node.js 18+ or Python 3.10+
Basic familiarity with async/await in either language
An HTTP client: axios or native fetch for Node, httpx for Python

Step 1: Authentication

PostAll uses API key authentication. Every request needs your key in the Authorization header.

Get your key: Dashboard → Settings → API Keys → Generate New Key

Store it as an environment variable. Never hardcode it.

export PostAll_API_KEY="postall_live_xxxxxxxxxxxxxxxxxxxx"

Your key has two prefixes: postall_live_ for production, postall_test_ for the sandbox. The sandbox returns real-shaped responses with placeholder content — useful for testing your pipeline without burning request quota.

Verify authentication before building anything else:

// verify-auth.js
const PostAll_API_BASE = "https://api.usepostall.io/v1";

async function verifyAuth() {
  const response = await fetch(`${PostAll_API_BASE}/account`, {
    headers: {
      Authorization: `Bearer ${process.env.PostAll_API_KEY}`,
      "Content-Type": "application/json",
    },
  });

  if (!response.ok) {
    // 401 = bad key, 403 = key exists but no API access on your plan
    throw new Error(`Auth failed: ${response.status} ${response.statusText}`);
  }

  const account = await response.json();
  console.log(`Authenticated as: ${account.email}`);
  console.log(`Plan: ${account.plan} | Requests remaining: ${account.quota.remaining}`);
  return account;
}

verifyAuth().catch(console.error);

Run this first. A 403 means your account plan doesn't include API access — upgrade or contact support before debugging anything else.

Step 2: The three endpoints you'll actually use

PostAll has a full REST API, but 90% of automation use cases come down to three endpoints:

Endpoint	Method	What it does
`/v1/generate`	POST	Generate a single content piece
`/v1/batch`	POST	Queue multiple generations (async)
`/v1/batch/:id`	GET	Poll a batch job for results

There's also /v1/templates (GET) to list your saved prompt templates, and /v1/content/:id (GET/PATCH/DELETE) for managing existing content — but start with generate and batch.

Step 3: Your first generation

Let's generate one piece of content end-to-end before building the pipeline.

The request shape:

// generate-single.js
const PostAll_API_BASE = "https://api.usepostall.io/v1";

async function generateContent(brief) {
  const response = await fetch(`${PostAll_API_BASE}/generate`, {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.PostAll_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      type: brief.type,           // "blog_post" | "product_description" | "email" | "social"
      topic: brief.topic,
      tone: brief.tone,           // "professional" | "conversational" | "technical" | "playful"
      target_length: brief.target_length, // word count target, not a hard limit
      format: "markdown",         // "markdown" | "html" | "plain"
      metadata: {
        seo_optimize: true,       // adds meta_description and focus_keyword to response
        include_headings: true,
      },
    }),
  });

  if (!response.ok) {
    const error = await response.json();
    throw new Error(`Generation failed: ${error.message}`);
  }

  return response.json();
}

// Test it
generateContent({
  type: "blog_post",
  topic: "how to reduce JavaScript bundle size in 2025",
  tone: "technical",
  target_length: 800,
}).then((result) => {
  console.log("Title:", result.content.title);
  console.log("Word count:", result.content.word_count);
  console.log("Meta description:", result.metadata.meta_description);
  // result.content.body contains the full markdown
});

The response shape:

{
  "id": "gen_01hx4k2m9f...",
  "status": "complete",
  "content": {
    "title": "7 Ways to Reduce Your JavaScript Bundle Size in 2025",
    "body": "## The bundle problem\n\nYour users...",
    "word_count": 847,
    "reading_time_minutes": 4
  },
  "metadata": {
    "meta_description": "Learn the most effective strategies for reducing JavaScript bundle size...",
    "focus_keyword": "reduce JavaScript bundle size",
    "tokens_used": 1203
  },
  "created_at": "2025-01-15T14:32:11Z"
}

The id field matters — you'll use it to retrieve or update content later. Store it.

Step 4: Batch generation for real volume

Single generation works for interactive use. For processing a list of briefs, use the batch endpoint. It's async: you submit the job, get a batch_id back, then poll for results.

Why async matters here: batch jobs can take 30 seconds to several minutes depending on size. Don't try to do this synchronously with a timeout — you'll time out on large batches and have no idea what succeeded.

// batch-pipeline.js
const PostAll_API_BASE = "https://api.usepostall.io/v1";

// Step 1: Submit the batch
async function submitBatch(briefs) {
  const response = await fetch(`${PostAll_API_BASE}/batch`, {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.PostAll_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      items: briefs.map((brief) => ({
        type: brief.type,
        topic: brief.topic,
        tone: brief.tone || "professional",
        target_length: brief.target_length || 500,
        format: "markdown",
        // Optional: tag each item for easier result mapping
        external_id: brief.id,
      })),
      // Webhook URL to hit when complete — highly recommended over polling
      webhook_url: process.env.PostAll_WEBHOOK_URL || null,
    }),
  });

  const batch = await response.json();
  return batch.id; // e.g. "batch_01hx4k2m9f..."
}

// Step 2: Poll until done
async function pollBatch(batchId, intervalMs = 3000) {
  const headers = {
    Authorization: `Bearer ${process.env.PostAll_API_KEY}`,
    "Content-Type": "application/json",
  };

  while (true) {
    const response = await fetch(`${PostAll_API_BASE}/batch/${batchId}`, { headers });
    const batch = await response.json();

    console.log(`Status: ${batch.status} | Progress: ${batch.completed}/${batch.total}`);

    if (batch.status === "complete") {
      return batch.results;
    }

    if (batch.status === "failed") {
      throw new Error(`Batch failed: ${batch.error}`);
    }

    // "processing" or "queued" — wait and try again
    await new Promise((resolve) => setTimeout(resolve, intervalMs));
  }
}

// Put it together
async function runBatchPipeline(briefs) {
  console.log(`Submitting batch of ${briefs.length} items...`);
  const batchId = await submitBatch(briefs);
  console.log(`Batch ID: ${batchId} — polling for results...`);

  const results = await pollBatch(batchId);
  console.log(`Done. ${results.length} items generated.`);
  return results;
}

Step 5: The same pipeline in Python

For those of you running data pipelines, automations, or integrations in Python:

# PostAll_pipeline.py
import os
import time
import httpx
from typing import Optional

PostAll_API_BASE = "https://api.usepostall.io/v1"

def get_headers():
    api_key = os.environ.get("PostAll_API_KEY")
    if not api_key:
        raise EnvironmentError("PostAll_API_KEY not set")
    return {
        "Authorization": f"Bearer {api_key}",
        "Content-Type": "application/json",
    }

def generate_single(brief: dict) -> dict:
    """Generate one content piece synchronously."""
    with httpx.Client(timeout=30.0) as client:
        response = client.post(
            f"{PostAll_API_BASE}/generate",
            headers=get_headers(),
            json={
                "type": brief["type"],
                "topic": brief["topic"],
                "tone": brief.get("tone", "professional"),
                "target_length": brief.get("target_length", 500),
                "format": "markdown",
                "metadata": {"seo_optimize": True},
            },
        )
        response.raise_for_status()
        return response.json()

def submit_batch(briefs: list[dict]) -> str:
    """Submit a batch job and return the batch ID."""
    with httpx.Client(timeout=30.0) as client:
        response = client.post(
            f"{PostAll_API_BASE}/batch",
            headers=get_headers(),
            json={
                "items": [
                    {
                        "type": b["type"],
                        "topic": b["topic"],
                        "tone": b.get("tone", "professional"),
                        "target_length": b.get("target_length", 500),
                        "format": "markdown",
                        "external_id": b.get("id"),
                    }
                    for b in briefs
                ]
            },
        )
        response.raise_for_status()
        return response.json()["id"]

def poll_batch(batch_id: str, interval_seconds: float = 3.0) -> list[dict]:
    """Poll a batch job until complete. Returns list of results."""
    with httpx.Client(timeout=30.0) as client:
        while True:
            response = client.get(
                f"{PostAll_API_BASE}/batch/{batch_id}",
                headers=get_headers(),
            )
            response.raise_for_status()
            batch = response.json()

            print(f"Status: {batch['status']} | Progress: {batch['completed']}/{batch['total']}")

            if batch["status"] == "complete":
                return batch["results"]

            if batch["status"] == "failed":
                raise RuntimeError(f"Batch failed: {batch.get('error')}")

            time.sleep(interval_seconds)


# Example: run from a list of dicts
if __name__ == "__main__":
    briefs = [
        {"id": "1", "type": "product_description", "topic": "noise-cancelling headphones", "tone": "conversational"},
        {"id": "2", "type": "product_description", "topic": "mechanical keyboard", "tone": "technical"},
        {"id": "3", "type": "email", "topic": "product launch announcement", "target_length": 200},
    ]

    batch_id = submit_batch(briefs)
    print(f"Submitted batch: {batch_id}")
    results = poll_batch(batch_id)

    for item in results:
        print(f"\n--- {item['external_id']} ---")
        print(f"Title: {item['content']['title']}")
        print(f"Words: {item['content']['word_count']}")

What can go wrong (and will)

Rate limits

Free tier: 10 requests/minute, 100 requests/day. Pro tier: 60 requests/minute, no daily cap.

The API returns 429 Too Many Requests when you hit the limit, with a Retry-After header telling you how many seconds to wait. Don't ignore that header. I learned this the hard way — a tight retry loop without backoff will keep 429-ing you indefinitely.

// Respect the Retry-After header
async function generateWithRetry(brief, maxRetries = 3) {
  for (let attempt = 0; attempt < maxRetries; attempt++) {
    const response = await fetch(`${POSTALL_API_BASE}/generate`, {
      method: "POST",
      headers: { Authorization: `Bearer ${process.env.POSTALL_API_KEY}`, "Content-Type": "application/json" },
      body: JSON.stringify(brief),
    });

    if (response.status === 429) {
      const retryAfter = parseInt(response.headers.get("Retry-After") || "5", 10);
      console.log(`Rate limited. Waiting ${retryAfter}s...`);
      await new Promise((r) => setTimeout(r, retryAfter * 1000));
      continue;
    }

    if (!response.ok) throw new Error(`HTTP ${response.status}`);
    return response.json();
  }
  throw new Error("Max retries exceeded");
}

Batch jobs don't fail loudly

If one item in a batch fails, the rest still complete. The failed item shows up in results with "status": "failed" and an error field. Check each item's status — don't assume the whole batch succeeded just because batch.status === "complete".

results.forEach((item) => {
  if (item.status === "failed") {
    console.error(`Item ${item.external_id} failed:`, item.error);
    // Log it, re-queue it, or flag for manual review
  }
});

Webhooks are more reliable than polling

If you're running batches in a server environment, set webhook_url when submitting and handle the POST PostAll sends when the job is done. Polling works for scripts, but it's wasteful and fails silently if your process exits. The webhook payload is the same shape as the poll response — swap the logic, not the data handling.

Content length is a target, not a guarantee

target_length: 800 gets you a content piece in the 700–950 word range in my testing. If you need strict character counts for database constraints, validate and truncate after the fact — don't rely on the API hitting an exact number.

A note on the test vs production key

Run your integration against postall_test_ keys until you're confident the pipeline handles errors correctly. Test mode returns structurally valid responses with placeholder content — great for schema validation and error path testing without burning API quota.

Switch to postall_live_ when:

Your retry logic handles 429s correctly
You're checking per-item status in batch results
You're storing the id field from responses (you'll want it for content management later)

Where to go from here

This covers the generation layer. The next pieces you'll likely want to build:

Template management — /v1/templates lets you save and version your prompt configurations, so you're not repeating the full brief object every request.
Content CRUD — Once you're generating programmatically, you'll want to retrieve, update, and delete via /v1/content/:id. Especially useful if you're feeding Postall output into a CMS with a review step.
Webhook handling — If you're building a production automation rather than a script, set up a webhook receiver and drop the polling loop entirely.

The full working code from this guide is on GitHub: github.com/usepostall/api-quickstart. Both the Node and Python examples are there with a sample CSV loader and a basic webhook receiver.

What are you automating first? Drop it in the comments — I'm particularly curious what use cases people hit that the /generate endpoint doesn't cover out of the box.

Already using the PostAll API? I'd love to know what your retry/backoff strategy looks like at scale — the Retry-After approach above works at low volume but I'm still iterating on the right pattern for 1000+ item batches.

How PostAll's Content Pipeline Actually Works: A Full Architecture Breakdown

Aakash Gour — Mon, 08 Jun 2026 05:59:37 +0000

I'm going to show you the entire internal architecture of PostAll — the content automation platform I've been building — because I think there's a gap in how people talk about AI content tools publicly.

Most founders describe their product in terms of outcomes: "generate 10x content faster," "consistent brand voice," "scales with your team." That's fine for a landing page. But if you're a developer evaluating whether to build something similar, or trying to understand what makes an AI content pipeline actually work at scale, outcome-speak tells you nothing.

So here's what I should have read when I started: how a production content automation system is actually structured, what each layer does, where things break, and the decisions I made — and regret — along the way.

This is not a tutorial. It's a transparency post. PostAll is live. This is how it works.

The 30-second version

PostAll has five distinct layers:

Queue system — ingests jobs, prioritizes them, handles retries
Orchestration layer — breaks content requests into atomic tasks
LLM layer — calls the models, handles rate limits and fallbacks
Formatting engine — transforms raw LLM output into structured content
CMS connectors — pushes formatted content to wherever the client wants it

Each layer is independently deployable and has its own failure modes. The worst mistake I made early on was treating them as one monolith, and I'll explain exactly what broke.

The queue system

What it does

Every content request — whether it's one blog post or 500 product descriptions — enters PostAll as a job. Jobs hit the queue before anything else happens. The queue is the single source of truth for what work exists, what's in progress, and what's failed.

I use BullMQ (backed by Redis) for this. I evaluated Postgres-backed queues (pg-boss, Graphile Worker) first. They're fine for lower throughput. At scale, the Redis-backed approach handles bursts better and the job visibility tools in BullMQ's UI saved me significant debugging time in production.

// job-producer.ts
import { Queue } from 'bullmq';
import { redis } from './redis-client';

const contentQueue = new Queue('content-generation', {
  connection: redis,
  defaultJobOptions: {
    attempts: 3,
    backoff: {
      type: 'exponential',
      delay: 2000, // 2s, 4s, 8s — stays inside most LLM retry windows
    },
    removeOnComplete: { count: 500 }, // keep last 500 completed jobs for audit
    removeOnFail: false,              // keep all failed jobs — you'll want to inspect them
  },
});

export async function enqueueContentJob(payload: ContentJobPayload) {
  const job = await contentQueue.add(
    'generate',
    payload,
    {
      priority: payload.bulkBatch ? 2 : 1, // single requests get higher priority
      jobId: `${payload.clientId}:${payload.contentId}`, // idempotent — safe to re-enqueue
    }
  );
  return job.id;
}

The jobId pattern matters. Setting it to a deterministic string means if a client double-submits (network retry, user clicking twice), BullMQ silently deduplicates. You want this. Without it, you'll spend a Friday afternoon tracking down why the same article got generated three times.

What I got wrong: worker concurrency

My initial worker config:

// WRONG — don't do this
const worker = new Worker('content-generation', processJob, {
  connection: redis,
  concurrency: 50, // I thought "more = faster"
});

At 50 concurrent workers hitting OpenAI simultaneously, I was saturating my rate limit constantly. The retries from failed requests caused a feedback loop: more retries → more failures → more retries.

The fix:

// RIGHT — sized to your actual API tier
const worker = new Worker('content-generation', processJob, {
  connection: redis,
  concurrency: 8, // OpenAI Tier 3 gives us ~3,500 RPM across all workers
  limiter: {
    max: 8,
    duration: 1000, // max 8 jobs started per second across all worker instances
  },
});

The limiter option in BullMQ is underused. It's not just about your worker — it's a global rate governor across however many worker instances you're running. With autoscaling, you can have 10 pods all pulling from the same queue. Without the limiter, 10 pods × 50 concurrency = 500 simultaneous LLM calls. That's how you get a $3,000 bill and a support ticket in the same hour.

The orchestration layer

What it does

A "content job" sounds atomic. It isn't. A single blog post request in PostAll decomposes into:

Keyword research task (if enabled)
Outline generation task
Section drafts (one per section, parallelizable)
Introduction/conclusion stitching task
SEO metadata generation task
Quality score evaluation task

The orchestration layer handles this decomposition, tracks task dependencies, and aggregates results into a coherent output.

I built this with a simple DAG (directed acyclic graph) structure rather than reaching for a workflow engine like Temporal or Inngest. For PostAll's use case, a purpose-built DAG was simpler and cheaper to operate. If I were doing multi-tenant workflows with human-in-the-loop steps, I'd reconsider.

// content-dag.ts
type TaskNode = {
  id: string;
  type: 'outline' | 'section' | 'metadata' | 'quality';
  dependsOn: string[];
  input: (context: JobContext) => Promise<TaskInput>;
  execute: (input: TaskInput) => Promise<TaskOutput>;
};

export function buildContentDAG(request: ContentRequest): TaskNode[] {
  const outline: TaskNode = {
    id: 'outline',
    type: 'outline',
    dependsOn: [],
    input: async (ctx) => ({ keyword: request.keyword, tone: request.tone }),
    execute: generateOutline,
  };

  const sections: TaskNode[] = request.sections.map((section, i) => ({
    id: `section-${i}`,
    type: 'section',
    dependsOn: ['outline'],
    input: async (ctx) => ({
      outline: ctx.results['outline'],
      sectionIndex: i,
      sectionTitle: section.title,
    }),
    execute: generateSection,
  }));

  const metadata: TaskNode = {
    id: 'metadata',
    type: 'metadata',
    dependsOn: sections.map((s) => s.id), // needs all sections complete
    input: async (ctx) => ({ fullContent: assembleContent(ctx) }),
    execute: generateMetadata,
  };

  return [outline, ...sections, metadata];
}

The executor resolves the DAG topologically, runs independent nodes in parallel, and passes results through context:

// dag-executor.ts
async function executeDag(nodes: TaskNode[], jobId: string): Promise<JobContext> {
  const context: JobContext = { jobId, results: {}, errors: {} };
  const remaining = [...nodes];

  while (remaining.length > 0) {
    // find nodes whose dependencies are all complete
    const ready = remaining.filter((node) =>
      node.dependsOn.every((dep) => dep in context.results)
    );

    if (ready.length === 0) {
      throw new Error(`DAG deadlock in job ${jobId} — circular dependency or failed prerequisite`);
    }

    // run ready nodes in parallel
    await Promise.allSettled(
      ready.map(async (node) => {
        try {
          const input = await node.input(context);
          context.results[node.id] = await node.execute(input);
        } catch (err) {
          context.errors[node.id] = err;
          // don't throw — let other branches complete, fail job at end
        }
      })
    );

    // remove completed nodes
    ready.forEach((node) => {
      const idx = remaining.indexOf(node);
      remaining.splice(idx, 1);
    });
  }

  return context;
}

The Promise.allSettled choice is intentional. If one section fails, you still want the other sections to complete. You can fail the job at the end with partial results, which is far better than losing all work because one 500-word section errored.

The LLM layer

What it does

Every LLM call in PostAll goes through a single abstraction: the LLMClient. This class handles model selection, prompt construction, retry logic, fallback routing, and cost tracking. Nothing in the codebase calls OpenAI directly.

// llm-client.ts
export class LLMClient {
  private providers: ProviderConfig[];

  constructor(config: LLMClientConfig) {
    this.providers = config.providers; // ordered by preference
  }

  async complete(request: CompletionRequest): Promise<CompletionResponse> {
    for (const provider of this.providers) {
      try {
        const result = await this.callProvider(provider, request);
        await this.trackUsage(provider.name, result.usage);
        return result;
      } catch (err) {
        if (this.isRateLimit(err) || this.isServerError(err)) {
          // try next provider
          continue;
        }
        throw err; // bad request, auth error — don't retry with another provider
      }
    }
    throw new Error('All providers exhausted');
  }

  private async callProvider(
    provider: ProviderConfig,
    request: CompletionRequest
  ): Promise<CompletionResponse> {
    const client = this.getClient(provider.name);
    const response = await client.chat.completions.create({
      model: provider.model,
      messages: this.buildMessages(request),
      max_tokens: request.maxTokens ?? 2000,
      temperature: request.temperature ?? 0.7,
      response_format: request.structured ? { type: 'json_object' } : undefined,
    });

    return {
      content: response.choices[0].message.content ?? '',
      usage: {
        promptTokens: response.usage?.prompt_tokens ?? 0,
        completionTokens: response.usage?.completion_tokens ?? 0,
        provider: provider.name,
        model: provider.model,
      },
    };
  }

  private isRateLimit(err: unknown): boolean {
    return err instanceof Error && err.message.includes('429');
  }

  private isServerError(err: unknown): boolean {
    return err instanceof Error && err.message.includes('5');
  }
}

The fallback routing config

// llm-config.ts
export const llmConfig: LLMClientConfig = {
  providers: [
    {
      name: 'openai',
      model: 'gpt-4o',
      apiKey: process.env.OPENAI_API_KEY!,
    },
    {
      name: 'anthropic',
      model: 'claude-sonnet-4-20250514', // fallback for rate limit hits
      apiKey: process.env.ANTHROPIC_API_KEY!,
    },
  ],
};

Multi-provider fallback sounds complicated. It's mostly just the ordering. The insight that took me too long to reach: don't try to be smart about routing at call time. Keep a priority list, try them in order, move on if one fails. Smart routing (latency-based, cost-based) added complexity with no meaningful real-world gain at my current scale.

Token budgeting

One thing that bit me early: LLM requests for long-form content can exceed context limits in ways that are hard to predict at job creation time. PostAll now estimates token usage before each call:

function estimatePromptTokens(messages: Message[]): number {
  // rough estimate: 1 token ≈ 4 characters for English prose
  const charCount = messages.reduce((sum, m) => sum + m.content.length, 0);
  return Math.ceil(charCount / 4) + 100; // +100 overhead for formatting
}

async function callWithBudget(
  client: LLMClient,
  request: CompletionRequest
): Promise<CompletionResponse> {
  const MODEL_CONTEXT_LIMIT = 128_000; // gpt-4o context window
  const estimatedPrompt = estimatePromptTokens(request.messages);
  const safeMaxTokens = Math.min(
    request.maxTokens ?? 2000,
    MODEL_CONTEXT_LIMIT - estimatedPrompt - 500 // leave buffer
  );

  if (safeMaxTokens < 200) {
    throw new Error('Prompt too long for requested output length — consider chunking');
  }

  return client.complete({ ...request, maxTokens: safeMaxTokens });
}

This saves you from silent truncation — which is the failure mode where the model just... stops mid-sentence because you ran out of context window and you have no idea why.

The formatting engine

What it does

Raw LLM output is text. What clients actually need is structured, formatted content ready to paste into a CMS — proper heading hierarchy, internal links in the right format, meta descriptions within character limits, image alt text suggestions, and so on.

The formatting engine handles this transformation. It runs after every LLM call that produces content (not metadata calls).

// formatter.ts
export type FormatTarget = 'markdown' | 'html' | 'wordpress' | 'contentful-richtext';

export async function formatContent(
  rawContent: string,
  target: FormatTarget,
  options: FormatOptions
): Promise<FormattedContent> {
  // 1. Parse raw text into a content tree
  const tree = parseContentTree(rawContent);

  // 2. Apply structural fixes
  const normalized = normalizeHeadings(tree);         // enforce H2 → H3 hierarchy
  const withKeyword = injectKeywordDensity(normalized, options.keyword, 0.012); // ~1.2%
  const withLinks = applyInternalLinks(withKeyword, options.internalLinkMap);

  // 3. Validate SEO constraints
  validateSeoConstraints(withLinks, {
    minWords: options.minWords ?? 800,
    maxMetaDescLength: 160,
    titleTagRange: [50, 60],
  });

  // 4. Serialize to target format
  return serialize(withLinks, target);
}

The most underrated function in this whole stack is normalizeHeadings. LLMs are inconsistent about heading levels. You'll get H1s inside H3 sections, H4s appearing before H2s, and occasionally no structure at all. A single regex pass doesn't fix it — you need to understand the tree.

function normalizeHeadings(tree: ContentTree): ContentTree {
  let expectedLevel = 2; // PostAll-generated content starts at H2 (H1 is the page title)

  return walkTree(tree, (node) => {
    if (node.type !== 'heading') return node;

    if (node.level < expectedLevel) {
      // heading skipped levels — promote to expected
      return { ...node, level: expectedLevel };
    }

    if (node.level > expectedLevel + 1) {
      // heading jumped down too far — demote to one level below expected
      return { ...node, level: expectedLevel + 1 };
    }

    expectedLevel = node.level;
    return node;
  });
}

What I wish I'd built earlier: a validation layer with human-readable errors

For six months, validation failures surfaced as generic job errors. I'd get a failed job, dig through logs, and eventually find something like ValidationError: heading_hierarchy_violated. Not helpful.

Now every validation failure returns a structured report:

type ValidationReport = {
  passed: boolean;
  issues: Array<{
    severity: 'error' | 'warning';
    code: string;
    message: string;       // human-readable
    location?: string;     // e.g. "Section 2, paragraph 3"
    suggestion?: string;   // what to do about it
  }>;
};

This also feeds back into the LLM layer for automatic retry with targeted instructions: if the meta description is too long, the retry prompt includes "The previous meta description was 184 characters. Rewrite it to be under 160 characters." This reduced formatting-related job failures by 67% over two weeks.

The CMS connectors

What it does

Formatted content needs to go somewhere. PostAll currently supports WordPress (REST API), Contentful, Webflow CMS, and a generic webhook target. Each connector is a thin adapter that translates PostAll's internal FormattedContent type to whatever the target CMS expects.

// connectors/wordpress.ts
export class WordPressConnector implements CMSConnector {
  async publish(content: FormattedContent, config: WordPressConfig): Promise<PublishResult> {
    const payload = {
      title: content.metadata.title,
      content: content.body.html,    // WP expects HTML
      status: config.publishImmediately ? 'publish' : 'draft',
      slug: slugify(content.metadata.title),
      meta: {
        _yoast_wpseo_metadesc: content.metadata.metaDescription,
        _yoast_wpseo_title: content.metadata.seoTitle,
        _yoast_wpseo_focuskw: content.metadata.keyword,
      },
      categories: config.categoryIds ?? [],
      tags: config.tagIds ?? [],
    };

    const response = await fetch(`${config.siteUrl}/wp-json/wp/v2/posts`, {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
        Authorization: `Basic ${Buffer.from(`${config.username}:${config.appPassword}`).toString('base64')}`,
      },
      body: JSON.stringify(payload),
    });

    if (!response.ok) {
      const error = await response.json();
      throw new CMSConnectorError('wordpress', response.status, error.message);
    }

    const post = await response.json();
    return { contentId: post.id.toString(), url: post.link, status: 'published' };
  }
}

Each connector handles its own error mapping. A 409 from Contentful (slug conflict) is a different recovery path than a 401 (expired auth token). The CMSConnectorError type carries enough context for the job system to decide whether to retry, notify the client, or surface a specific error message.

The webhook target

The most useful connector I built turned out to be the simplest: a generic webhook. When a client doesn't use any of the supported CMS platforms, they give PostAll a URL and we POST formatted content as JSON to it. They handle their own delivery.

// connectors/webhook.ts
export class WebhookConnector implements CMSConnector {
  async publish(content: FormattedContent, config: WebhookConfig): Promise<PublishResult> {
    const payload = {
      version: '1.0',
      timestamp: new Date().toISOString(),
      jobId: content.jobId,
      content: {
        title: content.metadata.title,
        body: {
          markdown: content.body.markdown,
          html: content.body.html,
        },
        metadata: content.metadata,
        seoData: content.seoData,
      },
    };

    // sign the payload so clients can verify it came from PostAll
    const signature = createHmac('sha256', config.signingSecret)
      .update(JSON.stringify(payload))
      .digest('hex');

    const response = await fetch(config.url, {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
        'X-PostAll-Signature': `sha256=${signature}`,
        'X-PostAll-Event': 'content.published',
      },
      body: JSON.stringify(payload),
    });

    // treat 2xx as success, anything else as failure
    if (!response.ok) {
      throw new CMSConnectorError('webhook', response.status, `Webhook returned ${response.status}`);
    }

    return { contentId: content.jobId, url: config.url, status: 'delivered' };
  }
}

HMAC signing is something I added after the first time a client asked "how do we know this request is actually from you?" Now every webhook delivery is signed with a per-client secret. The verification code on their end is four lines of Python or Node. Worth adding from day one.

The architecture in one diagram

Here's how these five layers connect at runtime for a single content job:

Client Request
      │
      ▼
┌─────────────┐
│  Queue      │  BullMQ / Redis — ingests, prioritizes, handles retries
│  System     │
└──────┬──────┘
       │
       ▼
┌─────────────┐
│ Orchestration│  DAG executor — decomposes job into parallel tasks
│   Layer     │
└──────┬──────┘
       │  (multiple parallel tasks)
       ▼
┌─────────────┐
│   LLM       │  Multi-provider client — OpenAI primary, Anthropic fallback
│   Layer     │
└──────┬──────┘
       │
       ▼
┌─────────────┐
│  Formatting │  Heading normalization, SEO validation, keyword density
│   Engine    │
└──────┬──────┘
       │
       ▼
┌─────────────┐
│    CMS      │  WordPress / Contentful / Webflow / Webhook
│ Connectors  │
└─────────────┘

Each box is a separate module with its own error handling and retry budget. The queue system is the only layer that knows about job persistence — everything below it is stateless.

What I'd do differently

1. I would have built the formatting engine first.

The LLM layer gets all the attention because models are interesting. But the formatting engine is where most client complaints come from. Heading was wrong, meta description was too long, link format didn't match their CMS. Investing in formatting validation early would have saved two months of reactive fixes.

2. I would have instrumented the DAG execution from day one.

Right now I can tell you how long individual LLM calls take. For the first six months, I couldn't tell you how long a job took end-to-end, or which tasks inside the DAG were the bottleneck. Turns out: metadata generation was consistently the slowest step, not section drafting. I only discovered this when I added distributed tracing. It was a 15-minute fix once I knew where to look.

3. The multi-provider fallback is table stakes, not a "nice-to-have."

I treated provider fallback as a future optimization. During an OpenAI incident in early production, I had 40 jobs stuck with no fallback, clients emailing me, and nothing to do but wait. Anthropic as a fallback target took me a few hours to wire in. It should have been there before the first paying customer.

Where PostAll is now

The current pipeline processes around 800 content jobs per day with a p95 job latency of 94 seconds (from queue entry to CMS delivery). The LLM layer accounts for about 78% of that time. The formatting engine accounts for about 14%. The queue system and CMS connectors split the rest.

Error rate over the last 30 days: 1.3% of jobs require manual intervention. About 40% of those are auth issues on the client's CMS (expired credentials, changed endpoints). The other 60% are LLM-layer failures that exhaust retries — usually structured output that doesn't parse correctly on the first three attempts.

There's a lot still to build. Quality scoring is currently rule-based (reading level, keyword density, heading count). The next version will use a lightweight model eval layer to assess coherence and factual consistency. That's a different post.

If you're building something similar — a content pipeline, a document generation system, anything that orchestrates LLM calls at scale — I'm curious what your DAG execution looks like. Did you use a workflow engine, or roll your own? The tradeoffs there are non-obvious and I'd genuinely like to hear what others landed on.

The complete source for PostAll's connector layer is on GitHub at github.com/postall-platform/connectors. The LLM client abstraction is published as a standalone npm package: @postall-platform/llm-client. Drop a star if any of this was useful — it's the best signal I have for what to write about next.

PostAll Build Log #2: What My First Beta Users Actually Told Me (And What I Had to Rewrite)

Aakash Gour — Thu, 04 Jun 2026 06:07:41 +0000

I sent PostAll to 11 beta users three weeks ago.

I expected feedback about the UI. Maybe some complaints about output quality. A few "would be nice if..." suggestions I could politely defer.

What I got instead: three separate users hitting the same architectural assumption I'd never tested, a feature I was proud of that nobody used, and one piece of feedback that made me rewrite the job queue from scratch.

Here's what actually happened.

The Feedback I Expected vs. What I Got

My mental model of beta feedback: people would try the core flow, generate some content, and tell me what felt awkward.

The reality: two users never got past onboarding. Not because the onboarding was broken — it wasn't. Because their use case didn't map to how I'd structured the product at all.

I'd built PostAll around a "campaign" metaphor: you define a topic, set parameters, generate a batch of articles. Clean. Logical. Made total sense to me.

User #4 (a content agency owner) opened PostAll and spent 10 minutes looking for a way to paste in a list of 60 keywords and just... run them. No campaign setup. No parameter tuning. Just: here are the keywords, go.

User #7 (an e-commerce founder) wanted to generate product descriptions one at a time, from inside a workflow that already existed in another tool. He wasn't looking for a campaign — he was looking for an API endpoint with sensible defaults.

Same product. Completely different mental models. Both of them valid.

The campaign abstraction I'd designed was the right structure for me, building the thing. It was the wrong structure for people who just needed output fast.

The Architecture Change I Didn't Want to Make

Here's what the original job flow looked like:

// Original: campaigns own everything
async function enqueueCampaign(campaignId) {
  const campaign = await db.campaigns.findById(campaignId);
  const jobs = campaign.keywords.map(keyword => ({
    campaignId,
    keyword,
    parameters: campaign.parameters,
    status: 'pending'
  }));

  await db.jobs.insertMany(jobs);
  await queue.add('processCampaign', { campaignId });
}

Every job was a child of a campaign. Made sense for tracking, for billing, for grouping output. But it meant you had to create a campaign before you could generate anything.

After beta feedback, I needed jobs to be first-class:

// Revised: jobs are independent, campaigns are optional grouping
async function enqueueJob(input) {
  const { keyword, parameters, campaignId = null, userId } = input;

  const job = await db.jobs.create({
    userId,
    campaignId,      // nullable — job works without a campaign
    keyword,
    parameters: {
      ...getDefaultParameters(userId),  // sensible defaults per user
      ...parameters                      // override only what was passed
    },
    status: 'pending',
    createdAt: new Date()
  });

  await queue.add('processJob', { jobId: job.id }, {
    attempts: 3,
    backoff: { type: 'exponential', delay: 2000 }
  });

  return job;
}

This change sounds small. It took two days to implement correctly, because campaign-level analytics were baked into four different places in the codebase that assumed campaignId was always present.

The lesson: the abstraction you build first is usually built for the creator, not the user. The creator needs structure to stay sane. The user needs to get to output as fast as possible.

The Feature Nobody Used

I spent probably 12 hours building a "Content Style Guide" editor — a UI where you could define your brand voice, set tone parameters, add example sentences PostAll should emulate.

In 11 beta users across three weeks: zero used it.

I asked User #2 about it directly. She said: "Oh, I didn't know what that was for. I just typed what I wanted in the prompt box."

The fancier the UI, the higher the cognitive load to understand it. A free-text prompt field has a learning curve of zero. A multi-field style guide editor requires you to understand what "formality score" means, why you'd set it to 7 vs. 8, and whether it'll actually affect output.

I haven't deleted the style guide feature. But I've moved it three levels deeper in the settings, and I added a "Quick Generate" path that bypasses it entirely.

The One Piece of Feedback That Stung

User #9 — a freelance writer who was testing PostAll to see if it would help with client volume — sent me a Loom video. I was expecting a bug report.

Instead: she showed herself generating five articles, then pulling them all into a Google Doc to manually reformat them. Title casing, removing extra line breaks, standardizing the H2 structure. It took her longer than just editing the content.

The output quality was fine. The output format was inconsistent in ways that made it unusable without cleanup.

I'd tested PostAll's output against my own mental model of "good formatting." I hadn't tested it against what someone actually does with the output after it's generated.

The fix wasn't in the prompt. It was in the post-processing layer:

function normalizeArticleOutput(rawContent) {
  return rawContent
    // Consistent H2 formatting (model sometimes outputs ### instead of ##)
    .replace(/^#{3,}\s/gm, '## ')
    // Remove double blank lines
    .replace(/\n{3,}/g, '\n\n')
    // Title case for H2 headers (model is inconsistent)
    .replace(/^## (.+)$/gm, (_, title) => `## ${toTitleCase(title)}`)
    // Strip trailing whitespace per line
    .split('\n').map(line => line.trimEnd()).join('\n')
    .trim();
}

Simple. Should have been in v1. Wasn't, because I never watched someone actually use the output in their workflow.

What I Underestimated

Onboarding time. I thought 10 minutes to first generation was fine. Two users spent more than 20 minutes before generating anything, and both of them never came back.

The first generation is the only one that matters for retention. If someone generates garbage on the first try — wrong format, wrong length, doesn't match their mental model of "good content" — the product is dead to them regardless of how good it gets later.

I'm now adding a guided first-run flow that forces a single successful generation before anything else. Not optional. Mandatory.

Where PostAll Is Now

Job queue redesigned: campaigns are optional, not required
Quick Generate path ships next week
Output normalization layer is live
Style guide editor: still exists, now hidden by default
Onboarding: being rebuilt

The metrics that matter right now: 4 of 11 beta users have generated content more than once. I want that at 7 of 11 before I expand the beta further.

What surprised you most the first-time real users touched something you built? I keep expecting UI feedback and keep getting architecture feedback. Starting to think that's just how it goes.

Building a Multi-Platform Content Publisher: One API, Five Destinations

Aakash Gour — Mon, 01 Jun 2026 06:30:22 +0000

Every time I shipped a new article for PostAll's early beta users, I watched them paste the same content into five different dashboards. WordPress. Ghost. Webflow. Medium. Dev.to.

Copy. Paste. Adjust formatting. Fight with the rich text editor. Repeat.

I timed it once: 23 minutes per article to publish to all five platforms. If you're doing content at any kind of scale — even 10 articles a week — that's nearly 4 hours of soul-crushing manual work. Work that a computer should obviously be doing.

So I built a single publish API that handles all five simultaneously. This is the exact implementation behind PostAll's multi-platform publishing feature. I'll show you the full working code, the two approaches I tried before this one, and the specific failure modes you'll hit if you skip the parts I almost skipped.

What You'll Build

By the end of this, you'll have a Node.js service with a single POST /publish endpoint that:

Accepts a content payload (title, body, tags, metadata)
Transforms the content into each platform's required format
Publishes to all five destinations in parallel
Returns a structured result object with success/failure per platform
Handles partial failures without killing the whole batch

Here's what a successful response looks like:

{
  "status": "partial_success",
  "results": {
    "wordpress": { "success": true, "url": "https://yourblog.com/?p=1842" },
    "ghost": { "success": true, "url": "https://yourblog.ghost.io/p/your-post" },
    "webflow": { "success": true, "itemId": "64a3f1b2c3d4e5f6a7b8c9d0" },
    "medium": { "success": true, "url": "https://medium.com/@you/your-post-abc123" },
    "devto": { "success": false, "error": "Rate limit exceeded. Retry after 60s." }
  }
}

That partial failure on Dev.to? I'll show you exactly how to handle it without having to re-publish everything else.

Why This Is Harder Than It Looks

Five platforms, five completely different APIs.

WordPress uses XML-RPC or a REST API (depending on version and config). Ghost has a beautiful Admin API but requires JWT authentication with a 5-minute expiry. Webflow's CMS API is REST but requires knowing your Collection ID before you can do anything. Medium requires OAuth for full access but supports integration tokens for simpler publishing. Dev.to uses API keys but has aggressive rate limiting on free accounts.

Every platform has a different concept of "content." WordPress thinks in post_content. Ghost thinks in Mobiledoc or Lexical JSON. Webflow thinks in CMS field schemas you define yourself. Medium thinks in HTML with a title field. Dev.to thinks in Markdown with frontmatter.

The naive approach — just writing five separate functions — works until you need to change something. Then you're changing it five times. The right approach is a transform layer that sits between your canonical content format and each platform's expected format.

Prerequisites

Node.js 18+ (uses native fetch and Promise.allSettled)
Active accounts on whichever platforms you're targeting
API credentials for each (I'll list exactly what you need per platform)
Basic understanding of async/await and REST APIs

You don't need to target all five platforms immediately. The architecture is additive — start with two, add the rest when you need them.

Step 1: Define Your Canonical Content Format

Before writing any platform-specific code, define the content shape you'll work with internally. This is the single source of truth that every publisher transforms from.

// types.js
/**
 * @typedef {Object} CanonicalContent
 * @property {string} title
 * @property {string} bodyMarkdown - Always store in Markdown; transform on publish
 * @property {string} bodyHtml - Pre-rendered HTML for platforms that need it
 * @property {string[]} tags - Plain strings, no hashtags
 * @property {string} [canonicalUrl] - Original URL if cross-posting
 * @property {string} [featuredImageUrl]
 * @property {Object} [meta]
 * @property {string} [meta.description]
 * @property {string} [meta.slug]
 */

/**
 * @typedef {Object} PublishTarget
 * @property {string} platform - 'wordpress' | 'ghost' | 'webflow' | 'medium' | 'devto'
 * @property {Object} credentials
 * @property {Object} [options] - Platform-specific overrides
 */

Store content as Markdown internally. Every platform can accept some variant of HTML, and you can always convert Markdown → HTML. Going the other direction is a mess.

Step 2: The Publisher Core

This is the heart of the system. One function that fans out to all configured publishers in parallel.

// publisher.js
import { marked } from 'marked'; // npm install marked

import { publishToWordPress } from './publishers/wordpress.js';
import { publishToGhost } from './publishers/ghost.js';
import { publishToWebflow } from './publishers/webflow.js';
import { publishToMedium } from './publishers/medium.js';
import { publishToDevTo } from './publishers/devto.js';

const PUBLISHERS = {
  wordpress: publishToWordPress,
  ghost: publishToGhost,
  webflow: publishToWebflow,
  medium: publishToMedium,
  devto: publishToDevTo,
};

export async function publishToAll(content, targets) {
  // Pre-render Markdown to HTML once; all HTML-based platforms share this
  const contentWithHtml = {
    ...content,
    bodyHtml: marked.parse(content.bodyMarkdown),
  };

  // Run all publishers in parallel — don't await sequentially
  // Promise.allSettled never throws, even if individual publishers fail
  const publishJobs = targets.map(async (target) => {
    const publishFn = PUBLISHERS[target.platform];

    if (!publishFn) {
      return {
        platform: target.platform,
        success: false,
        error: `Unknown platform: ${target.platform}`,
      };
    }

    try {
      const result = await publishFn(contentWithHtml, target.credentials, target.options);
      return { platform: target.platform, success: true, ...result };
    } catch (err) {
      return {
        platform: target.platform,
        success: false,
        error: err.message,
        // Include retry hint if the publisher provides one
        retryAfter: err.retryAfter ?? null,
      };
    }
  });

  const settled = await Promise.allSettled(publishJobs);

  // allSettled always resolves; extract the values
  const results = Object.fromEntries(
    settled.map((s) => {
      const value = s.status === 'fulfilled' ? s.value : { success: false, error: s.reason?.message };
      return [value.platform, value];
    })
  );

  const successCount = Object.values(results).filter((r) => r.success).length;
  const status =
    successCount === targets.length ? 'success' :
    successCount === 0 ? 'failed' :
    'partial_success';

  return { status, results };
}

The key decision here: Promise.allSettled instead of Promise.all. With Promise.all, one platform failure throws and you lose all your results. With allSettled, you get granular success/failure per platform and can retry only what failed.

Step 3: The Platform Publishers

WordPress

WordPress has two API options: the legacy XML-RPC endpoint and the newer REST API (available on WordPress 4.7+). Use the REST API if you can — it's significantly cleaner.

// publishers/wordpress.js

export async function publishToWordPress(content, credentials, options = {}) {
  const { siteUrl, username, appPassword } = credentials;
  // App Passwords were added in WP 5.6 — generate one in Users → Profile
  // Format: "xxxx xxxx xxxx xxxx xxxx xxxx" (spaces included)
  const authHeader = 'Basic ' + btoa(`${username}:${appPassword}`);

  const payload = {
    title: content.title,
    content: content.bodyHtml,
    status: options.status ?? 'publish', // 'draft' for review workflows
    tags: await resolveWordPressTags(content.tags, siteUrl, authHeader),
    // WordPress requires tag IDs, not strings — resolveWordPressTags handles this
    slug: content.meta?.slug,
    excerpt: content.meta?.description,
    ...(content.featuredImageUrl && {
      featured_media: await uploadWordPressMedia(content.featuredImageUrl, siteUrl, authHeader),
    }),
  };

  const response = await fetch(`${siteUrl}/wp-json/wp/v2/posts`, {
    method: 'POST',
    headers: {
      'Authorization': authHeader,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify(payload),
  });

  if (!response.ok) {
    const err = await response.json();
    throw new Error(`WordPress API error: ${err.message} (${err.code})`);
  }

  const post = await response.json();
  return { url: post.link, postId: post.id };
}

async function resolveWordPressTags(tags, siteUrl, authHeader) {
  // WordPress needs tag IDs, not strings — create tags that don't exist yet
  const tagIds = await Promise.all(
    tags.map(async (tagName) => {
      // Try to find existing tag first
      const searchRes = await fetch(
        `${siteUrl}/wp-json/wp/v2/tags?search=${encodeURIComponent(tagName)}`,
        { headers: { Authorization: authHeader } }
      );
      const existing = await searchRes.json();

      if (existing.length > 0) return existing[0].id;

      // Create new tag if it doesn't exist
      const createRes = await fetch(`${siteUrl}/wp-json/wp/v2/tags`, {
        method: 'POST',
        headers: { Authorization: authHeader, 'Content-Type': 'application/json' },
        body: JSON.stringify({ name: tagName }),
      });
      const newTag = await createRes.json();
      return newTag.id;
    })
  );
  return tagIds;
}

WordPress gotcha: App Passwords are different from your login password. Go to Users → Your Profile → Application Passwords and generate one specifically for your publishing integration. Your regular password won't work with the REST API in most configurations.

Ghost

Ghost's Admin API uses JWT authentication with a short expiry. You can't use a static bearer token — you have to generate a new JWT per request.

// publishers/ghost.js
import crypto from 'crypto';

export async function publishToGhost(content, credentials, options = {}) {
  const { adminUrl, adminApiKey } = credentials;
  // Admin API Key format from Ghost Admin: "key_id:secret" — split on the colon
  const [keyId, secret] = adminApiKey.split(':');

  const jwt = generateGhostJWT(keyId, secret);

  // Ghost uses Mobiledoc OR Lexical depending on version
  // Sending HTML with `html` field works for Ghost 5.x with the html conversion flag
  const payload = {
    posts: [{
      title: content.title,
      html: content.bodyHtml,
      status: options.status ?? 'published',
      tags: content.tags.map((name) => ({ name })), // Ghost accepts tag names directly
      custom_excerpt: content.meta?.description,
      canonical_url: content.canonicalUrl,
      ...(content.featuredImageUrl && { feature_image: content.featuredImageUrl }),
    }],
  };

  const response = await fetch(`${adminUrl}/ghost/api/admin/posts/?source=html`, {
    method: 'POST',
    headers: {
      'Authorization': `Ghost ${jwt}`,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify(payload),
  });

  if (!response.ok) {
    const err = await response.json();
    throw new Error(`Ghost API error: ${err.errors?.[0]?.message}`);
  }

  const result = await response.json();
  const post = result.posts[0];
  return { url: post.url, postId: post.id };
}

function generateGhostJWT(keyId, secret) {
  // Ghost uses a specific JWT structure — don't try to use a generic JWT library
  const now = Math.floor(Date.now() / 1000);
  const header = { alg: 'HS256', typ: 'JWT', kid: keyId };
  const payload = { iat: now, exp: now + 300, aud: '/admin/' }; // 5-minute expiry

  const encode = (obj) => Buffer.from(JSON.stringify(obj)).toString('base64url');
  const headerB64 = encode(header);
  const payloadB64 = encode(payload);
  const signingInput = `${headerB64}.${payloadB64}`;

  const signature = crypto
    .createHmac('sha256', Buffer.from(secret, 'hex'))
    .update(signingInput)
    .digest('base64url');

  return `${signingInput}.${signature}`;
}

Ghost gotcha: The ?source=html query param is critical. Without it, Ghost tries to parse your HTML as Mobiledoc and the content will be empty. This burned me for two days.

Webflow

Webflow is the most setup-intensive because it requires knowing your Collection ID and the field schema before publishing anything.

// publishers/webflow.js

export async function publishToWebflow(content, credentials, options = {}) {
  const { apiToken, collectionId } = credentials;
  // collectionId: get from Webflow Dashboard → CMS → your blog collection → Settings
  // The ID is in the URL: webflow.com/design/.../cms/[collectionId]

  // Webflow field names depend on YOUR collection schema
  // These are the defaults — override via options.fieldMap if yours are different
  const fieldMap = {
    title: 'name',         // Webflow always uses 'name' as the required display field
    body: 'post-body',     // Whatever you named your rich text field
    slug: 'slug',
    description: 'post-summary',
    tags: 'tags',
    featuredImage: 'main-image',
    ...options.fieldMap,
  };

  const fields = {
    [fieldMap.title]: content.title,
    [fieldMap.body]: content.bodyHtml,
    [fieldMap.slug]: content.meta?.slug ?? slugify(content.title),
    [fieldMap.description]: content.meta?.description ?? '',
  };

  const response = await fetch(
    `https://api.webflow.com/v2/collections/${collectionId}/items/live`,
    {
      method: 'POST',
      headers: {
        'Authorization': `Bearer ${apiToken}`,
        'Content-Type': 'application/json',
      },
      body: JSON.stringify({ fieldData: fields }),
    }
  );

  if (!response.ok) {
    const err = await response.json();
    throw new Error(`Webflow API error: ${err.message} — field: ${err.code}`);
  }

  const item = await response.json();
  // /live endpoint publishes immediately; use /items (without /live) for drafts
  return { itemId: item.id };
}

function slugify(title) {
  return title
    .toLowerCase()
    .replace(/[^a-z0-9]+/g, '-')
    .replace(/^-|-$/g, '');
}

Webflow gotcha: The /live suffix on the endpoint publishes immediately. Without it, you create a draft that only exists in staging. I spent an embarrassing amount of time wondering why my published items weren't showing up on the live site.

Medium

Medium's API is officially deprecated-ish — they stopped accepting new OAuth app registrations but still support Integration Tokens for personal publishing.

// publishers/medium.js

export async function publishToMedium(content, credentials, options = {}) {
  const { integrationToken } = credentials;
  // Get from medium.com/me/settings → Integration Tokens

  // First, get your user ID — required for the posts endpoint
  const userRes = await fetch('https://api.medium.com/v1/me', {
    headers: { Authorization: `Bearer ${integrationToken}` },
  });

  if (!userRes.ok) throw new Error('Medium auth failed — check your integration token');

  const { data: user } = await userRes.json();

  const payload = {
    title: content.title,
    contentFormat: 'html', // 'markdown' also works but HTML renders more reliably
    content: content.bodyHtml,
    tags: content.tags.slice(0, 5), // Medium enforces a 5-tag limit
    canonicalUrl: content.canonicalUrl,
    publishStatus: options.status ?? 'public', // 'draft' | 'public' | 'unlisted'
  };

  const response = await fetch(`https://api.medium.com/v1/users/${user.id}/posts`, {
    method: 'POST',
    headers: {
      'Authorization': `Bearer ${integrationToken}`,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify(payload),
  });

  if (!response.ok) {
    const err = await response.json();
    throw new Error(`Medium API error: ${err.errors?.[0]?.message}`);
  }

  const { data: post } = await response.json();
  return { url: post.url, postId: post.id };
}

Medium gotcha: They silently truncate tags beyond 5. If you pass 8 tags, the API succeeds but only the first 5 appear. The .slice(0, 5) is defensive, not cosmetic.

Dev.to

The cleanest API of the five. API key auth, straightforward payload, good documentation.

// publishers/devto.js

export async function publishToDevTo(content, credentials, options = {}) {
  const { apiKey } = credentials;
  // Get from dev.to/settings/extensions → DEV Community API Keys

  const payload = {
    article: {
      title: content.title,
      body_markdown: content.bodyMarkdown, // Dev.to wants Markdown, not HTML
      published: options.status !== 'draft',
      tags: content.tags
        .slice(0, 4) // Dev.to enforces a 4-tag limit
        .map((t) => t.toLowerCase().replace(/[^a-z0-9]/g, '')), // tags: lowercase, alphanumeric only
      canonical_url: content.canonicalUrl,
      description: content.meta?.description,
      ...(content.featuredImageUrl && { main_image: content.featuredImageUrl }),
    },
  };

  const response = await fetch('https://dev.to/api/articles', {
    method: 'POST',
    headers: {
      'api-key': apiKey,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify(payload),
  });

  if (response.status === 429) {
    // Rate limit: 10 articles per 30 seconds on free accounts
    const retryAfter = parseInt(response.headers.get('retry-after') ?? '60', 10);
    const err = new Error('Dev.to rate limit exceeded');
    err.retryAfter = retryAfter; // Bubble up to the retry system
    throw err;
  }

  if (!response.ok) {
    const err = await response.json();
    throw new Error(`Dev.to API error: ${JSON.stringify(err)}`);
  }

  const article = await response.json();
  return { url: article.url, articleId: article.id };
}

Dev.to gotcha: Tags must be lowercase and contain only letters and numbers — no hyphens, no spaces. "web-dev" becomes "webdev". Pass them through the sanitizer above or they'll silently fail validation.

Step 4: The HTTP Endpoint

Wire everything together with a simple Express endpoint.

// server.js
import express from 'express';
import { publishToAll } from './publisher.js';

const app = express();
app.use(express.json({ limit: '2mb' })); // Articles can get big with embedded code

app.post('/publish', async (req, res) => {
  const { content, targets } = req.body;

  // Basic validation — add more for production
  if (!content?.title || !content?.bodyMarkdown) {
    return res.status(400).json({ error: 'content.title and content.bodyMarkdown are required' });
  }

  if (!Array.isArray(targets) || targets.length === 0) {
    return res.status(400).json({ error: 'targets must be a non-empty array' });
  }

  try {
    const result = await publishToAll(content, targets);
    // Return 207 Multi-Status for partial success — it's the correct HTTP status here
    const statusCode = result.status === 'failed' ? 500 : result.status === 'partial_success' ? 207 : 200;
    return res.status(statusCode).json(result);
  } catch (err) {
    // This only fires if publishToAll itself throws — individual platform errors are caught inside
    return res.status(500).json({ error: err.message });
  }
});

app.listen(3000, () => console.log('Publisher running on :3000'));

A sample request to test all five platforms:

curl -X POST http://localhost:3000/publish \
  -H "Content-Type: application/json" \
  -d '{
    "content": {
      "title": "How I Optimized Our Node.js Queue",
      "bodyMarkdown": "## The Problem\n\nAt 500 concurrent jobs...",
      "tags": ["nodejs", "javascript", "performance", "webdev"],
      "meta": {
        "description": "A practical guide to Node.js queue optimization",
        "slug": "nodejs-queue-optimization"
      }
    },
    "targets": [
      {
        "platform": "wordpress",
        "credentials": { "siteUrl": "https://yourblog.com", "username": "admin", "appPassword": "xxxx xxxx xxxx xxxx xxxx xxxx" }
      },
      {
        "platform": "devto",
        "credentials": { "apiKey": "your_devto_api_key" }
      }
    ]
  }'

What Can Go Wrong

Rate limits cascade. If you're publishing high volume, you'll hit rate limits on Dev.to (10/30s) and Medium first. The retryAfter field on the error lets you implement a per-platform retry queue. Don't retry all platforms when only one rate-limited — that's how you get banned.

Ghost JWT clock drift. Ghost's JWT validation is strict about the iat and exp fields. If your server clock is off by more than a few seconds from Ghost's server, you'll get mysterious 401s. Add NTP sync to your server checklist if you're running this in production.

Webflow field mismatches. If your Webflow collection has required fields you're not supplying, the API returns a 400 with a cryptic field name error. Before implementing, dump your collection schema: GET https://api.webflow.com/v2/collections/{collectionId} and map every required field.

WordPress multisite installs. If your WordPress is a multisite network, the API path changes. Instead of /wp-json/wp/v2/posts, it becomes /site-path/wp-json/wp/v2/posts. The base URL you pass in credentials needs to include the subsite path.

Medium's canonicalUrl field. Medium strips HTTP/HTTPS from canonical URLs if they point to Medium itself. If you're cross-posting from another platform, always set canonicalUrl to the original source URL to avoid duplicate content penalties. The API accepts it but doesn't validate it — a wrong canonical URL won't cause an error, it'll just quietly hurt your SEO.

Where to Take This

The implementation above covers the publish path. In PostAll, this publisher sits behind two more layers I haven't covered here:

A retry queue. When platforms return rate-limit errors or 5xx responses, failed publish jobs go into a Redis-backed retry queue with exponential backoff. Platform-specific retryAfter hints from the error object feed directly into the delay calculation.

A format validator. Before hitting any platform API, content runs through a validator that checks Markdown for broken image links, missing alt text, and code blocks without language identifiers. Catching these before publish is faster than debugging broken posts after the fact.

Both of those are topics for separate posts. If you want to see either covered next, drop it in the comments.

The complete source for this publisher — with a few more error-handling details I cut for brevity — is at: github.com/your-handle/multi-platform-publisher. If it saves you the 3 days it took me to figure out Ghost's JWT implementation, a star would be appreciated.

What platforms are you publishing to that I missed? LinkedIn articles and Hashnode are both on my list — curious if anyone's built a reliable LinkedIn publisher, since their API for articles is historically finicky.

PostAll is a content automation platform I'm building for teams that publish at scale. If the multi-platform publishing problem is one you're actively dealing with, I write about the architecture behind it here regularly.

I Analyzed 1,000 AI-Generated Blog Posts for Quality. Here's the Data.

Aakash Gour — Thu, 28 May 2026 06:16:31 +0000

Last year, I was doing something that felt increasingly absurd: manually reading AI-generated content to decide if it was "good enough."

PostAll — the content automation tool I've been building — was producing hundreds of blog posts per week for clients. And I had no systematic way to evaluate quality at scale. I was spot-checking. Vibes-checking, really. That doesn't work at volume.

So I built a programmatic quality analysis pipeline, ran it over 1,000 AI-generated posts, and let the numbers tell me what my gut was missing.

The findings surprised me. A few of them genuinely changed how I think about AI content quality.

What I Actually Measured

First, a definition of terms, because "quality" is almost meaninglessly vague in this space.

I broke quality into five measurable dimensions:

Readability — Flesch-Kincaid grade level and reading ease score
Keyword density — Target keyword frequency and distribution across the post
Grammar error rate — Errors per 1,000 words, caught via LanguageTool's API
Factual accuracy — Claims that could be verified programmatically (dates, statistics, named entities cross-referenced against a knowledge base)
Structural consistency — Presence of expected elements: intro hook, subheadings, conclusion, CTA

I used 1,000 posts across three categories: SaaS product descriptions, long-form "how-to" articles (1,200–2,000 words), and listicles (500–900 words). All were generated by PostAll using GPT-4o, with various prompting strategies.

The Setup

The analysis pipeline isn't complicated, but the piece that makes it useful is the batch processing layer:

import anthropic
import language_tool_python
import textstat
from dataclasses import dataclass
from typing import Optional
import json

@dataclass
class QualityReport:
    post_id: str
    flesch_reading_ease: float
    flesch_kincaid_grade: float
    grammar_errors_per_1000_words: float
    keyword_density: float
    structural_score: int  # 0–5 based on element presence
    flagged_claims: list[str]
    overall_score: float

tool = language_tool_python.LanguageTool('en-US')

def analyze_readability(text: str) -> dict:
    return {
        "flesch_reading_ease": textstat.flesch_reading_ease(text),
        "flesch_kincaid_grade": textstat.flesch_kincaid_grade(text),
    }

def analyze_grammar(text: str) -> float:
    matches = tool.check(text)
    word_count = len(text.split())
    # Filter out style suggestions — we only care about errors
    errors = [m for m in matches if m.ruleId not in STYLE_RULE_IDS]
    return (len(errors) / word_count) * 1000

def analyze_structure(text: str, expected_elements: list[str]) -> int:
    score = 0
    lowered = text.lower()
    for element in expected_elements:
        if element in lowered:
            score += 1
    return score

The STYLE_RULE_IDS exclusion is important. LanguageTool flags passive voice, comma splices, and other style choices that are sometimes intentional. Without filtering, style suggestions inflate the error count and make decent posts look broken.

For factual accuracy, I used a two-step approach: first, extract named entities and statistics using Claude, then cross-reference against a curated knowledge base for the client's domain. This doesn't catch everything — but it catches the most common failure modes (wrong dates, made-up statistics, hallucinated product names).

def extract_verifiable_claims(post_text: str) -> list[str]:
    client = anthropic.Anthropic()

    response = client.messages.create(
        model="claude-opus-4-5",
        max_tokens=1024,
        messages=[{
            "role": "user",
            "content": f"""Extract all factual claims from this post that could be verified:
            - Statistics with numbers
            - Named products, companies, or people with specific attributes
            - Dates and years
            - Prices or measurements

            Return as a JSON array of strings. Only include claims, not opinions.

            Post:
            {post_text}"""
        }]
    )

    try:
        claims = json.loads(response.content[0].text)
        return claims
    except json.JSONDecodeError:
        # Claude occasionally adds preamble even when asked not to
        # This is the real-world version, not the happy path
        return []

The Data

Here's what came back after running 1,000 posts through the pipeline.

Readability

Post Type	Avg. Flesch Reading Ease	Avg. FK Grade Level
Product descriptions	52.3	11.2
How-to articles	44.7	13.1
Listicles	61.8	9.4

For context: a score of 60–70 on the Flesch scale is considered "standard" — roughly plain English. A grade level of 8 is the target for most consumer-facing content.

The problem: the how-to articles were reading at a college level on average. When I looked at the worst offenders (grade 16+), the pattern was consistent — they were posts where I'd given the model a technically dense brief without explicitly requesting plain language output.

The fix was embarrassingly simple:

READABILITY_INSTRUCTION = """
Write at a 7th to 8th grade reading level (Flesch-Kincaid).
Use short sentences. Prefer common words over technical jargon
unless the term is essential and explained on first use.
Aim for a Flesch Reading Ease score above 60.
"""

Adding this to the system prompt dropped the average grade level from 13.1 to 9.4 across the how-to category. One instruction. Measurable improvement.

Grammar Error Rate

Average across all 1,000 posts: 2.1 errors per 1,000 words.

For human-written content, industry benchmarks sit around 3–5 errors per 1,000 words for first drafts. So the AI was actually outperforming average human first drafts on raw grammar.

But here's what the aggregate hides: the distribution was bimodal. 80% of posts had fewer than 1 error per 1,000 words. The remaining 20% had 8+ errors — almost always in posts that exceeded 1,500 words and involved a complex prompt with multiple constraints.

The hypothesis: when the model is managing a lot of constraints simultaneously (tone + structure + keyword requirements + specific claims to include), grammar degrades. It's spending its "attention" elsewhere.

My mitigation was to break complex posts into two passes: structure and content first, then a second pass focused specifically on prose quality and grammar. Error rate in the high-complexity category dropped from 9.3 to 2.8.

Keyword Density

Target range for SEO: 1–2% density for the primary keyword.

Density Range	% of Posts
Under 0.5% (keyword stuffing avoided, but too thin)	23%
0.5–1% (acceptable)	31%
1–2% (target range)	29%
2–3% (slightly over-optimized)	12%
Over 3% (keyword stuffed, will hurt rankings)	5%

The surprise: I expected over-stuffing to be the main failure mode. Turns out under-stuffing was almost 5x more common. The model was avoiding the keyword in an apparent attempt to sound natural — which is good instinct but overcorrected.

This one is prompt-sensitive. Explicitly specifying "include the primary keyword [X] approximately 8–12 times in a 1,000-word post" got the distribution into the target range for 74% of subsequent posts.

Structural Consistency

I scored each post on five structural elements:

Hook in the opening paragraph (first 100 words)
At least two H2 subheadings
One concrete example or case study
Conclusion section
Call to action

Score	% of Posts
5/5	38%
4/5	41%
3/5	15%
2/5 or below	6%

The most commonly missing element: the hook. 34% of posts opened with a definition, a generic statement about the topic, or the phrase "In today's digital landscape." (I have seen "In today's digital landscape" more times than I have seen sunrises. The model defaults to it under pressure.)

The second most commonly missing element: the concrete example. The model would describe a concept clearly but not ground it in a specific scenario. This is fixable with one line in the prompt: "Include at least one specific real-world example or case study."

Factual Accuracy

This is the one that should concern you.

Of the 1,000 posts, the pipeline flagged 147 posts (14.7%) as containing at least one unverifiable or contradicted claim.

Common failure types:

Outdated statistics — citing research from 3–5 years ago as "recent" (43 posts)
Made-up product features — describing a competitor's product with features it doesn't have (31 posts)
Hallucinated citations — references to studies or reports that don't exist (29 posts)
Wrong dates — misattributing when a company was founded or a product was launched (44 posts)

The hallucinated citations were the most dangerous. They looked authoritative. They had the right format. The study just didn't exist.

What This Changed in PostAll

Three concrete changes came out of this analysis:

1. Readability is now a post-generation check, not just a prompt instruction.

Every post runs through textstat before it's delivered. If the Flesch-Kincaid grade is above 10, the post goes back for a rewrite pass with an explicit readability constraint.

2. Factual claims trigger a verification flag.

Any post with more than three verifiable claims (per the Claude extraction step above) gets flagged for human review before delivery. This catches the hallucinated-citation failure mode without requiring human review of everything.

3. Two-pass generation for anything over 1,200 words.

Pass one: structure and content. Pass two: prose quality, grammar, and consistency check. The compute cost is real — roughly 2x token usage for long-form. The error rate improvement justified it.

What I'd Do Differently

I'd instrument this from day one, not month six.

The manual spot-checking I was doing before the pipeline wasn't just slow — it was inconsistent. I'd catch a grammar issue in one post and miss a hallucinated citation in the next. The programmatic check is less nuanced than human review, but it's consistent, which turns out to matter more at volume.

I'd also add a semantic coherence score earlier. Right now I can tell you a post has a grade-8 reading level and 1.3% keyword density. I can't tell you programmatically whether the argument in the post actually holds together. That's the next frontier.

The One Finding I Keep Coming Back To

Grammar is not the quality problem.

Every client who asks me "is the AI content good?" is thinking about grammar errors. The data says grammar errors are rare — and when they appear, they're predictable and fixable.

The real quality problems are factual accuracy and structural consistency. A post with perfect grammar that cites a study that doesn't exist is not a quality post. A post that opens with "In today's digital landscape" and never grounds the argument in a specific example is not a quality post.

You can't catch either of those by reading a paragraph. You need to measure them.

The pipeline code is on GitHub — link in my bio. If you're building content automation and haven't instrumented quality yet, this is the starting point I'd recommend.

What's your approach to quality scoring AI content at scale? Especially curious if anyone's cracked semantic coherence programmatically — that's the gap I haven't solved.

How to Build a Content Similarity Checker to Avoid Duplicate AI Output

Aakash Gour — Mon, 25 May 2026 05:47:46 +0000

When I started running bulk content generation jobs in PostAll, I hit a problem I didn't expect.

Not hallucinations. Not rate limits. Duplicates.

Not copy-paste duplicates — subtle ones. Two blog posts about "email marketing tips" that were structurally identical: same three sections, same examples, same conclusion, different sentences. The kind of thing that looks fine until you read them back to back and realize the AI just remixed the same underlying content.

At 10 articles, you catch this manually. At 500, you don't.

Here's the exact similarity checker I built to fix it — using OpenAI embeddings and cosine similarity, with a threshold you can tune to your use case.

Why String Matching Doesn't Cut It

My first instinct was to compare outputs with something simple. Calculate the Levenshtein distance between strings, flag anything above a similarity score. Shipped it in an afternoon.

It failed immediately.

Two articles can share 0% of their exact words and still be meaningfully duplicate content. "Email marketing boosts your open rates" and "Sending better emails increases how many people read them" are semantically identical — string matching treats them as completely different.

What you actually need is semantic similarity: are these two pieces of content saying the same thing, regardless of how the words are arranged?

That's what embeddings are for.

How Embeddings Work (The Part That Actually Matters)

An embedding is a vector — a list of numbers — that represents the meaning of a piece of text. OpenAI's text-embedding-3-small model converts any text into a 1536-dimensional vector.

Here's the useful property: texts with similar meaning produce vectors that point in similar directions in that 1536-dimensional space. Texts with different meaning point in different directions.

Cosine similarity measures the angle between two vectors. An angle of 0° (cosine similarity = 1.0) means identical direction — same meaning. An angle of 90° (cosine similarity = 0) means orthogonal — completely unrelated content.

In practice:

0.95+: Near-duplicate. Same content, slight rewording.
0.85–0.95: Very similar. Same topic and angle, different structure.
0.70–0.85: Related content. Same topic, meaningfully different angle.
Below 0.70: Different enough. Probably fine to publish both.

Your threshold depends on your use case. I'll come back to this.

The Implementation

1. Set Up the Embedding Client

// similarity-checker.js
import OpenAI from "openai";

const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
});

// text-embedding-3-small: 1536 dimensions, $0.02/1M tokens
// text-embedding-3-large: 3072 dimensions, $0.13/1M tokens
// For content dedup, small is accurate enough and ~6x cheaper
const EMBEDDING_MODEL = "text-embedding-3-small";

2. Get an Embedding for a Piece of Content

async function getEmbedding(text) {
  // Truncate at ~8000 tokens — embeddings model limit is 8191
  // For articles, take first 6000 chars: captures topic, angle, structure
  // without padding from conclusion boilerplate
  const truncated = text.slice(0, 6000);

  const response = await openai.embeddings.create({
    model: EMBEDDING_MODEL,
    input: truncated,
  });

  return response.data[0].embedding; // array of 1536 floats
}

Why truncate at 6000 chars instead of sending the full article?

Two reasons. First, the model has an 8191-token limit. Second, and more importantly: the first ~1500 words of an article carry almost all of its semantic fingerprint — the topic, angle, key claims. The conclusion is usually boilerplate. Truncating early makes the check faster and, in my testing, more accurate for detecting duplicates.

3. Calculate Cosine Similarity

function cosineSimilarity(vecA, vecB) {
  // Both vectors must be the same length (1536 for text-embedding-3-small)
  if (vecA.length !== vecB.length) {
    throw new Error(`Vector length mismatch: ${vecA.length} vs ${vecB.length}`);
  }

  let dotProduct = 0;
  let magnitudeA = 0;
  let magnitudeB = 0;

  for (let i = 0; i < vecA.length; i++) {
    dotProduct += vecA[i] * vecB[i];
    magnitudeA += vecA[i] * vecA[i];
    magnitudeB += vecB[i] * vecB[i];
  }

  magnitudeA = Math.sqrt(magnitudeA);
  magnitudeB = Math.sqrt(magnitudeB);

  if (magnitudeA === 0 || magnitudeB === 0) return 0;

  return dotProduct / (magnitudeA * magnitudeB);
}

This is the full cosine similarity formula — no library needed. It runs in O(n) where n is the vector dimension (1536), so it's fast for individual comparisons.

4. Check One Article Against a Library

async function checkForDuplicates(newContent, existingArticles, threshold = 0.88) {
  const newEmbedding = await getEmbedding(newContent);
  const duplicates = [];

  for (const article of existingArticles) {
    // Skip re-embedding if you've cached embeddings (you should — see below)
    const existingEmbedding = article.embedding ?? await getEmbedding(article.content);

    const similarity = cosineSimilarity(newEmbedding, existingEmbedding);

    if (similarity >= threshold) {
      duplicates.push({
        articleId: article.id,
        title: article.title,
        similarity: Math.round(similarity * 100) / 100,
      });
    }
  }

  // Sort by similarity descending so the most egregious duplicate is first
  duplicates.sort((a, b) => b.similarity - a.similarity);

  return {
    isDuplicate: duplicates.length > 0,
    matches: duplicates,
    newEmbedding, // return so caller can cache it
  };
}

5. Putting It Together for a Bulk Job

async function processBulkGeneration(articles) {
  const results = [];
  const processedArticles = []; // grows as we process

  for (const article of articles) {
    const { isDuplicate, matches, newEmbedding } = await checkForDuplicates(
      article.content,
      processedArticles,
      0.88
    );

    if (isDuplicate) {
      results.push({
        ...article,
        status: "duplicate",
        duplicateOf: matches[0], // most similar match
        action: "needs_revision",
      });
    } else {
      results.push({
        ...article,
        status: "unique",
        action: "ready_to_publish",
      });

      // Only add non-duplicates to the comparison pool
      // Adding duplicates would penalize articles similar to a bad article
      processedArticles.push({
        id: article.id,
        title: article.title,
        content: article.content,
        embedding: newEmbedding, // cache it — never re-embed the same content
      });
    }

    // Rate limit: embeddings API allows ~3000 RPM on tier 1
    // For bulk jobs, 50ms between requests keeps you well under
    await new Promise((resolve) => setTimeout(resolve, 50));
  }

  return results;
}

The Part That Bit Me: Caching Embeddings

The first version of this had no caching. For a job of 200 articles:

Embedding each article: 200 API calls
Checking each article against all previous: up to 200 × 199 / 2 = 19,900 comparisons

The comparisons are free — they're just math. But I was re-embedding articles every time I ran the checker, even articles I'd already processed.

At $0.02 per million tokens, re-embedding 200 articles (~500 tokens each) costs almost nothing. But at scale, with a persistent library of 10,000+ articles to check against, you want to store embeddings in your database and retrieve them directly.

// With a database — pseudocode for the storage layer
// but the retrieval pattern is what matters

async function getOrCreateEmbedding(articleId, content, db) {
  // Check if we already have this embedding stored
  const cached = await db.query(
    "SELECT embedding FROM articles WHERE id = $1 AND embedding IS NOT NULL",
    [articleId]
  );

  if (cached.rows[0]) {
    // Parse back from JSON — most DBs store vectors as JSON arrays
    return JSON.parse(cached.rows[0].embedding);
  }

  // Generate and store
  const embedding = await getEmbedding(content);
  await db.query(
    "UPDATE articles SET embedding = $1 WHERE id = $2",
    [JSON.stringify(embedding), articleId]
  );

  return embedding;
}

If you're on Postgres, the pgvector extension handles this natively, including fast ANN (approximate nearest neighbor) search so you're not doing a full scan of 10,000 embeddings for every new article.

Tuning the Threshold

I ran 50 article pairs through manual review to calibrate the threshold for PostAll's use case. Here's what I found at each level:

0.95+: True duplicates only. Same article, rewritten paragraph by paragraph. This catches almost nothing useful because it's too strict for AI-generated content, which tends to vary more than you'd expect even when covering the same topic.

0.88–0.92: The sweet spot for "same content, different words." Two articles about "onboarding email sequences" that cover the same three tactics in the same order. Definitely shouldn't both be published.

0.80–0.88: Similar topic and angle, but meaningfully different. An article about "onboarding email sequences for SaaS" vs. "onboarding email sequences for e-commerce" might land here. Depending on your use case, this might be a flag-for-review rather than an automatic rejection.

Below 0.80: Different enough to publish regardless of topic overlap.

My recommendation: start at 0.88, manually review your first 20 flags, and adjust from there. The threshold is highly dependent on how diverse your content topics are and how much variation your generation prompts produce.

What Can Go Wrong

Short content produces unreliable embeddings. A 50-word product description doesn't give the model enough signal. I set a minimum of 200 words before running similarity checks — anything shorter gets flagged for manual review instead.

All your content comes back as similar. This usually means your generation prompts are too rigid. If every prompt says "write in a professional, informative tone with three sections and a clear conclusion," the AI will produce structurally similar content regardless of topic. The similarity checker is revealing a prompt diversity problem, not a bug in the checker.

Embeddings drift if you switch models. text-embedding-3-small and text-embedding-ada-002 produce incompatible vector spaces. If you switch embedding models, you need to re-embed your entire library. Store which model generated each embedding alongside the vector.

The check is O(n) per article against your existing library. At 10,000 articles, checking a new article requires 10,000 cosine similarity calculations. That's still fast (under 100ms in my testing), but if you're checking 500 new articles against a library of 50,000, consider pgvector's HNSW index for approximate nearest neighbor search instead of the brute-force approach above.

The Complete Script (Copy-Paste Ready)

// content-similarity-checker.js
// Requires: openai npm package, OPENAI_API_KEY env var

import OpenAI from "openai";

const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

async function getEmbedding(text) {
  const response = await openai.embeddings.create({
    model: "text-embedding-3-small",
    input: text.slice(0, 6000),
  });
  return response.data[0].embedding;
}

function cosineSimilarity(vecA, vecB) {
  let dot = 0, magA = 0, magB = 0;
  for (let i = 0; i < vecA.length; i++) {
    dot += vecA[i] * vecB[i];
    magA += vecA[i] ** 2;
    magB += vecB[i] ** 2;
  }
  return dot / (Math.sqrt(magA) * Math.sqrt(magB));
}

export async function checkSimilarity(newContent, library, threshold = 0.88) {
  if (newContent.length < 200) {
    return { status: "too_short", isDuplicate: false, matches: [] };
  }

  const embedding = await getEmbedding(newContent);
  const matches = library
    .map((item) => ({
      ...item,
      similarity: cosineSimilarity(embedding, item.embedding),
    }))
    .filter((item) => item.similarity >= threshold)
    .sort((a, b) => b.similarity - a.similarity);

  return {
    status: matches.length > 0 ? "duplicate" : "unique",
    isDuplicate: matches.length > 0,
    matches,
    embedding, // cache this
  };
}

Usage:

import { checkSimilarity } from "./content-similarity-checker.js";

// Your existing articles with cached embeddings
const library = await db.query("SELECT id, title, embedding FROM articles");

const result = await checkSimilarity(newArticleContent, library.rows);

if (result.isDuplicate) {
  console.log(`Duplicate detected (${result.matches[0].similarity} similarity)`);
  console.log(`Most similar to: "${result.matches[0].title}"`);
} else {
  // Safe to publish — cache the embedding
  await db.query("UPDATE articles SET embedding = $1 WHERE id = $2", [
    JSON.stringify(result.embedding),
    newArticle.id,
  ]);
}

What I'd Add Next

The similarity checker catches what was already generated. It doesn't prevent duplicate content from being generated in the first place.

The next layer I'm building: before generating content on a topic, check the existing library for similar topics. If something with 0.88+ similarity already exists, either skip the generation or pass the existing article's key points to the prompt as a negative constraint: "Do not cover these points — they're already addressed in our existing content."

That closes the loop. Instead of catching duplicates after generation, you're steering generation away from duplication upfront.

The full code is on GitHub: [link to repo]. If you're using pgvector and want the schema for storing and querying embeddings at scale, the README covers that too.

What threshold did you land on for your use case? I'm curious whether the 0.88 default holds up across different content types — drop it in the comments.

Building PostAll: Week 1 of Shipping a Content Automation SaaS in Public

Aakash Gour — Thu, 21 May 2026 05:13:16 +0000

I've been building in private for too long.

Two months of solo development, zero public commits, and a growing suspicion that "I'll share it when it's ready" is just fear wearing a productivity costume. So this is me breaking that habit. Starting now, I'm building PostAll — a Content Automation Tool — in public on Dev.to.

Every week I'll write one of these. Actual technical decisions, actual dead ends, actual code. Not a highlight reel.

Here's where we are after week one.

What PostAll Actually Is (The Technical Version)

PostAll is a SaaS platform that takes your content brief — a topic, a brand voice, a target audience, an output format — and produces structured, publish-ready content at scale.

"Publish-ready" is doing a lot of work in that sentence, so let me be specific: the goal isn't to dump raw GPT output into a text box and call it done. The goal is CMS-ready output: correct heading hierarchy, metadata populated, internal link placeholders flagged, tone-checked against your brand guide, and formatted for your specific platform.

That's the product. The gap between "AI writes text" and "AI writes text your CMS can ingest directly" is where PostAll lives — and it's a bigger engineering gap than most people expect.

The Stack I Landed On (And the One I Almost Chose Instead)

I went back and forth on this for longer than I want to admit.

What I'm running:

Next.js 14 (App Router) — Frontend and API routes. I wanted one codebase to start. Will extract the API into a separate service if/when it becomes a bottleneck.
PostgreSQL + Prisma — Relational data for users, workspaces, content jobs, and output history. I considered MongoDB for the flexibility on content schemas, but the relational constraints turned out to be a feature, not a bug — more on that below.
BullMQ + Redis — Job queue for async content generation. This was the most deliberate decision of the week.
OpenAI API (gpt-4o) — Primary generation model for now.
Vercel — Hosting. Yes, I know. I'll move to a VPS when the pricing math stops working in Vercel's favor.

What I almost chose: Supabase for the whole backend. Realtime subscriptions for job status updates looked great in the docs. I ended up keeping Supabase for auth but pulling back on letting it be my entire backend, because I didn't want to couple my job queue architecture to their platform. Decoupling that decision early felt right.

The First Real Problem: You Can't Fake Content Job State

Here's the thing nobody talks about in content generation tutorials: a content job isn't a request. It's a state machine.

When a user submits a brief, that job moves through at least six states before it's done:

QUEUED → PROCESSING → GENERATING → VALIDATING → FORMATTING → COMPLETE

And it can fail at any of those stages, for different reasons, requiring different recovery strategies. An GENERATING failure is different from a FORMATTING failure — the first might mean retry with a different prompt, the second might mean the output was generated fine but the post-processing broke.

My first implementation had exactly two states: pending and done. Here's what that schema looked like:

// Week 1, Day 1 — too naive
model ContentJob {
  id        String   @id @default(cuid())
  status    String   // "pending" | "done" | "failed"
  output    String?
  createdAt DateTime @default(now())
  updatedAt DateTime @updatedAt
}

This worked until I tried to handle a partial failure — generation succeeded, formatting broke. I had no way to retry just the formatting step. The whole job was marked failed and the generated content was gone.

By day 3, the schema looked like this:

// Week 1, Day 3 — actually models what's happening
model ContentJob {
  id              String          @id @default(cuid())
  status          ContentStatus   // enum in Prisma schema
  stage           ContentStage    // which stage it's in
  generatedOutput String?         // raw LLM output, preserved separately
  formattedOutput String?         // post-processed output
  failureReason   String?
  retryCount      Int             @default(0)
  lastAttemptAt   DateTime?
  createdAt       DateTime        @default(now())
  updatedAt       DateTime        @updatedAt
}

enum ContentStatus {
  QUEUED
  PROCESSING
  COMPLETE
  FAILED
  RETRYING
}

enum ContentStage {
  GENERATION
  VALIDATION
  FORMATTING
}

Separating status from stage was the key insight. Status is "is this job healthy?" Stage is "where in the pipeline is it?" Now I can fail a job at the FORMATTING stage and retry only that stage without losing the generated content.

The lesson here applies beyond content generation: if your workflow has more than two meaningful states, model them explicitly from day one. Retrofitting a state machine onto a status: string column is painful.

The Queue: Why BullMQ and Not Just `setTimeout`

The honest answer is I started with setTimeout.

For local development with one user (me), it was fine. A content job comes in, I kick off an async function, it runs, it finishes. Simple.

Then I thought: what happens if the server restarts mid-generation? The job is gone. The user has no idea what happened. I have no visibility into what was running.

BullMQ with Redis gives me:

Persistence — Jobs survive server restarts
Visibility — I can see queue depth, processing time, and failure rates
Retry logic — Exponential backoff built in
Concurrency control — I can process X jobs at a time and not hammer the OpenAI API

Here's the actual worker setup:

// src/lib/queue/contentWorker.ts
import { Worker, Job } from 'bullmq';
import { generateContent } from '../generation/generateContent';
import { formatOutput } from '../formatting/formatOutput';
import { updateJobStatus } from '../db/contentJobs';
import { redisConnection } from '../redis';

const CONCURRENCY = 3; // OpenAI gpt-4o rate limit buffer on my current tier

export const contentWorker = new Worker(
  'content-generation',
  async (job: Job) => {
    const { jobId, brief } = job.data;

    try {
      // Stage 1: Generation
      await updateJobStatus(jobId, 'PROCESSING', 'GENERATION');
      const rawOutput = await generateContent(brief);
      await saveGeneratedOutput(jobId, rawOutput);

      // Stage 2: Formatting
      await updateJobStatus(jobId, 'PROCESSING', 'FORMATTING');
      const formatted = await formatOutput(rawOutput, brief.outputFormat);

      // Complete
      await updateJobStatus(jobId, 'COMPLETE', 'FORMATTING', { formattedOutput: formatted });

    } catch (error) {
      // Preserve which stage we were in when it failed
      await updateJobStatus(jobId, 'FAILED', job.data.currentStage, {
        failureReason: error instanceof Error ? error.message : 'Unknown error',
      });
      throw error; // Re-throw so BullMQ handles retry logic
    }
  },
  {
    connection: redisConnection,
    concurrency: CONCURRENCY,
    limiter: {
      max: 3,       // Max 3 jobs processed
      duration: 1000, // per second — OpenAI rate limit guard
    },
  }
);

The concurrency: 3 and limiter config isn't arbitrary — it's specifically calibrated to my current OpenAI API tier. I'll bump this as the tier increases.

What I Underestimated: Prompt Variability

I thought the hard part of a content generation tool would be the infrastructure. I was wrong.

The hard part is prompt consistency.

When you run the same brief through the same prompt 10 times, you don't get the same output 10 times. You get 10 variations with different structure, different tone emphasis, different heading choices. For a single piece of content, that's fine — variety is a feature. For a business that needs 200 product descriptions that sound like they came from the same brand, it's a serious problem.

I've spent the back half of this week on what I'm calling "structured generation" — instead of prompting for the full article in one shot, I'm breaking generation into structured sub-calls:

Generate an outline (JSON output, validated against a schema)
Generate each section from the outline independently
Assemble and run a final consistency pass

This is slower and costs more per job. It's also dramatically more consistent. I'll have benchmarks next week.

The Number That Surprised Me

I ran 47 test jobs through the queue this week. Average generation time: 11.3 seconds per job for a ~800-word article with gpt-4o.

That sounds fast until you think about it at scale: 500 articles would take about 94 minutes at that rate with my current concurrency. That's fine for batch jobs, not fine if a user expects to see 500 articles in their dashboard within 20 minutes of submitting.

Next week's problem: figuring out where the 11 seconds actually go (API latency vs. my code), and whether structured generation can run section calls in parallel.

What's Coming Next Week

Structured generation architecture (the sub-call approach) with actual benchmark numbers
The formatting layer: how I'm turning raw LLM output into CMS-ready HTML/Markdown
First beta user onboarding — someone from my network agreed to try it this week

The Open Question I Actually Don't Know the Answer To

I'm debating whether job status updates should push to the frontend via Server-Sent Events or if polling every 3 seconds is fine for the beta.

SSE is more elegant. Polling is simpler to debug. At 3-second intervals, polling is probably imperceptible to users. But it feels wrong.

What would you choose for a content generation job that takes 10-15 seconds? Drop it in the comments — I'll use whatever the consensus is and report back next week.

PostAll is currently in public beta. If you're a developer building content workflows and want early access, reply here or DM me.

Next devlog drops soon.

Rate Limiting LLM APIs in Production: Patterns That Don't Lose Your Data

Aakash Gour — Mon, 18 May 2026 05:57:04 +0000

I hit OpenAI's rate limit at 2:47am on a Tuesday, 40 minutes into a bulk generation job for 800 articles.

The job crashed. Not gracefully — it just stopped. No retry, no queue fallback, no partial save. When I woke up, I had 312 completed articles sitting in a database and 488 that existed only as prompts in a spreadsheet.

That was six months ago. Since then, I've rebuilt PostAll's content generation pipeline three times. What follows is everything I've learned about handling LLM rate limits in a way that doesn't eat your users' data.

Why Rate Limits Hit Harder Than You Expect

OpenAI's limits come in two flavors that are easy to confuse until they separately ruin your day:

RPM (Requests Per Minute) — how many API calls you can make
TPM (Tokens Per Minute) — how many tokens you can process

You can hit TPM while staying well under RPM. This happens when your prompts are long (system prompt + few-shot examples + user content adds up fast). My 800-article job hit TPM about 40 minutes in because each article prompt was ~2,100 tokens — I'd done the math on RPM and felt safe. I hadn't done it on TPM.

Here's what a 429 Too Many Requests response from OpenAI actually looks like:

// The actual error object you'll receive
{
  status: 429,
  error: {
    message: "Rate limit reached for gpt-4o in organization org-xxx on tokens per min (TPM): Limit 30000, Used 29847, Requested 1250. Please try again in 2.12s.",
    type: "rate_limit_error",
    code: "rate_limit_exceeded"
  },
  headers: {
    "retry-after": "2",           // seconds to wait (not always accurate)
    "x-ratelimit-limit-tpm": "30000",
    "x-ratelimit-remaining-tpm": "153",
    "x-ratelimit-reset-tokens": "2.12s"
  }
}

That retry-after header is actually useful — but almost every retry tutorial I've seen ignores it and hardcodes a delay instead. We'll use it.

The Three Patterns (And When Each One Breaks)

There's no single right answer here. There are three approaches, each suited to different workloads. I've used all three in PostAll at different points.

Pattern 1: Exponential Backoff with Jitter

Best for: Low-volume, latency-sensitive requests. User-facing generation where someone is waiting.

The idea: when you hit a 429, wait, then retry. Each failure doubles the wait time, plus a random jitter to prevent thundering herd problems (multiple clients all retrying at the exact same moment).

async function callWithBackoff(apiCallFn, options = {}) {
  const {
    maxRetries = 5,
    baseDelay = 1000,      // 1 second
    maxDelay = 60000,      // 60 seconds cap
    jitterFactor = 0.3,    // ±30% randomization
  } = options;

  let lastError;

  for (let attempt = 0; attempt < maxRetries; attempt++) {
    try {
      return await apiCallFn();
    } catch (error) {
      // Only retry on rate limit errors — fail fast on auth/invalid request
      if (error.status !== 429) throw error;

      lastError = error;

      // Use the API's suggested retry time if available
      const retryAfterMs = parseRetryAfter(error.headers?.['retry-after']);

      const exponentialDelay = Math.min(
        baseDelay * Math.pow(2, attempt),
        maxDelay
      );

      // Take the larger of: what the API says vs our exponential backoff
      const baseWait = Math.max(retryAfterMs, exponentialDelay);

      // Add jitter: randomize ±jitterFactor around baseWait
      const jitter = baseWait * jitterFactor * (Math.random() * 2 - 1);
      const waitMs = Math.round(baseWait + jitter);

      console.log(`Rate limited. Attempt ${attempt + 1}/${maxRetries}. Waiting ${waitMs}ms...`);
      await sleep(waitMs);
    }
  }

  throw new Error(`Failed after ${maxRetries} retries: ${lastError.message}`);
}

function parseRetryAfter(header) {
  if (!header) return 0;
  const seconds = parseFloat(header);
  return isNaN(seconds) ? 0 : seconds * 1000;
}

const sleep = (ms) => new Promise(resolve => setTimeout(resolve, ms));

Where this breaks: At any meaningful volume, backoff becomes a waiting game. If 20 concurrent requests all hit a rate limit and start backing off with doubling delays, your total job time explodes. I tested this on a 100-article batch — the backoff alone added 23 minutes. That's when you need Pattern 2.

Pattern 2: Token Bucket Rate Limiter

Best for: Bulk jobs where you control the request rate. The right pattern for any batch processing over ~50 requests.

Instead of sending requests and handling rejection, a token bucket prevents you from exceeding your limits in the first place. You get a "bucket" of tokens that refills at your rate limit. Before each request, you check if there's a token available. If not, you wait until one refills.

class TokenBucketRateLimiter {
  constructor({ requestsPerMinute, tokensPerMinute }) {
    this.rpmLimit = requestsPerMinute;
    this.tpmLimit = tokensPerMinute;

    // Track usage in a rolling 60-second window
    this.requestTimestamps = [];
    this.tokenUsageLog = []; // [{ timestamp, tokens }]
  }

  async waitForCapacity(estimatedTokens) {
    while (true) {
      const now = Date.now();
      const windowStart = now - 60_000; // 60-second rolling window

      // Prune entries outside the window
      this.requestTimestamps = this.requestTimestamps.filter(t => t > windowStart);
      this.tokenUsageLog = this.tokenUsageLog.filter(e => e.timestamp > windowStart);

      const currentRPM = this.requestTimestamps.length;
      const currentTPM = this.tokenUsageLog.reduce((sum, e) => sum + e.tokens, 0);

      const rpmOk = currentRPM < this.rpmLimit * 0.9;      // 90% of limit — leave headroom
      const tpmOk = (currentTPM + estimatedTokens) < this.tpmLimit * 0.9;

      if (rpmOk && tpmOk) {
        // Record this request before returning
        this.requestTimestamps.push(now);
        this.tokenUsageLog.push({ timestamp: now, tokens: estimatedTokens });
        return;
      }

      // Wait 500ms and check again
      // In practice, you'll rarely wait more than a couple seconds
      await sleep(500);
    }
  }
}

// Usage
const limiter = new TokenBucketRateLimiter({
  requestsPerMinute: 50,   // your actual API tier limit
  tokensPerMinute: 30_000, // your actual API tier limit
});

async function generateArticle(prompt, estimatedTokens) {
  await limiter.waitForCapacity(estimatedTokens);

  return openai.chat.completions.create({
    model: "gpt-4o",
    messages: [{ role: "user", content: prompt }],
    max_tokens: 1500,
  });
}

One thing to get right: estimating tokens before the request. You don't know exact token count until after, but a rough estimate is fine. I use Math.ceil(prompt.length / 3.5) as a heuristic — it's close enough for rate limiting purposes.

Where this breaks: Single-process only. If you're running multiple workers or serverless functions, each has its own limiter with no shared state. At PostAll, we hit this when we scaled to parallel workers — each one thought it had full capacity. We were effectively multiplying our request rate by our worker count.

The fix for multi-process: move rate limiting state to Redis.

Pattern 3: Persistent Queue with Dead Letter Handling

Best for: Anything that cannot lose work. Long-running batch jobs, user-submitted content, anything where "we'll just retry later" isn't acceptable.

This is the pattern I should have used from the start. The core idea: jobs go into a queue before they ever touch the API. Workers pull from the queue, process, and mark complete. If they fail, they go to a dead letter queue for inspection — not silent failure.

// Using BullMQ (Redis-backed queue — works across multiple workers)
import { Queue, Worker } from 'bullmq';
import Redis from 'ioredis';

const connection = new Redis({ maxRetriesPerRequest: null });

// Queue setup
const contentQueue = new Queue('content-generation', { connection });

// Add jobs — this is all you do in your API route or trigger
async function queueArticles(articles) {
  const jobs = articles.map(article => ({
    name: 'generate-article',
    data: { articleId: article.id, prompt: article.prompt },
    opts: {
      attempts: 8,             // retry up to 8 times
      backoff: {
        type: 'exponential',
        delay: 2000,           // start at 2s, doubles each time
      },
      removeOnComplete: { count: 100 },   // keep last 100 completed jobs for inspection
      removeOnFail: false,                // NEVER auto-remove failed jobs
    },
  }));

  await contentQueue.addBulk(jobs);
}

// Worker — runs separately, can scale horizontally
const worker = new Worker(
  'content-generation',
  async (job) => {
    const { articleId, prompt } = job.data;

    try {
      const result = await openai.chat.completions.create({
        model: "gpt-4o",
        messages: [{ role: "user", content: prompt }],
        max_tokens: 1500,
      });

      const content = result.choices[0].message.content;

      // Save before acknowledging job completion
      await db.articles.update({
        where: { id: articleId },
        data: { content, status: 'completed', completedAt: new Date() },
      });

      return { articleId, tokensUsed: result.usage.total_tokens };

    } catch (error) {
      // Mark the article as failed in DB so users can see status
      await db.articles.update({
        where: { id: articleId },
        data: { status: 'failed', lastError: error.message },
      });

      // Re-throw so BullMQ knows to retry
      throw error;
    }
  },
  {
    connection,
    concurrency: 5,        // process 5 jobs at once — tune to your rate limits
    limiter: {
      max: 50,             // max 50 jobs processed
      duration: 60_000,    // per 60 seconds — matches your RPM limit
    },
  }
);

// Dead letter queue handler — jobs that exhausted all retries
worker.on('failed', async (job, error) => {
  if (job.attemptsMade >= job.opts.attempts) {
    console.error(`Job ${job.id} moved to dead letter after ${job.attemptsMade} attempts:`, error.message);

    // Alert your team, not just log — this needs human review
    await notifySlack(`❌ Article ${job.data.articleId} failed permanently: ${error.message}`);
  }
});

The dead letter queue is the part most tutorials skip. A job that fails 8 times isn't gone — it's sitting in your failed jobs list in Redis, with the full error trace and all its data intact. You can inspect it, fix whatever caused the failure, and re-queue it manually.

This is what I didn't have the night 488 articles disappeared.

What I Wish I'd Known: The Gotchas

1. retry-after headers lie, but only sometimes.
OpenAI's retry-after is usually accurate for TPM limits. For RPM limits, it's sometimes optimistic. I add a 200ms buffer on top of whatever it suggests. Costs almost nothing, prevents a follow-up 429.

2. Token estimation is more important than request estimation.
Most rate limit tutorials focus on RPM. In practice, if you're using GPT-4o with system prompts and few-shot examples, you'll hit TPM first. Count your tokens. The tiktoken library for Python, or js-tiktoken for Node, give you exact counts pre-request.

import { encoding_for_model } from 'js-tiktoken';

function estimateTokens(text, model = 'gpt-4o') {
  const enc = encoding_for_model(model);
  return enc.encode(text).length;
}

// Use this before queueing to set accurate TPM budget
const promptTokens = estimateTokens(systemPrompt + userPrompt);

3. Concurrency is a multiplier on your rate limit consumption.
If your rate limit is 50 RPM and you run 5 concurrent workers each attempting 50 RPM, you're trying to do 250 RPM. The limiter on each worker doesn't know about the others. Either use a shared Redis-based limiter, or divide your limit by your worker count.

4. Model-specific limits catch people off guard.
GPT-4o and GPT-3.5-turbo have completely separate rate limits on the same API key. If you're mixing models in one pipeline, each has its own bucket. Don't share a single rate limiter across them.

5. The OpenAI Tier system changes your limits more than you expect.
You start on Tier 1 (RPM: 500, TPM: 30,000). After $100 of spend, you move to Tier 2 (RPM: 5,000, TPM: 450,000). The jump is significant — but you have to actually spend the money, not just add it as credits. I had $50 of credits sitting unused for two weeks thinking I'd already unlocked Tier 2. I hadn't.

Putting It Together: The Pattern Selector

Here's how I decide which pattern to use:

Scenario	Pattern
Single user waiting for result	Exponential backoff
Batch job, < 50 requests	Token bucket
Batch job, 50+ requests	Persistent queue
Multi-worker deployment	Persistent queue + Redis limiter
Can't lose data under any circumstance	Persistent queue + dead letter

For PostAll, we use all three depending on context. Real-time single-article generation uses backoff. Scheduled batch jobs use the token bucket. User-submitted bulk jobs (the ones people pay for) use the persistent queue. Losing a user's 500-article job is not an option — the queue makes sure we never have to tell someone their work is gone.

The Complete Reference Implementation

The code above is functional, but split across examples. The full reference implementation with all three patterns, proper TypeScript types, and a simple test harness is in the GitHub repo:

github.com/PostAll-platform/rate-limit-patterns ← update with your actual link

It includes a simulate-429.ts script that mocks rate limit responses so you can test your retry logic without burning real API credits.

Building LLM-powered features in production means building them like the API will fail — because it will, at the worst possible moment. The pattern you choose matters less than the fact that you chose one before it mattered.

What's your retry strategy look like? Have you hit TPM limits specifically, or is RPM the one that gets you? I'm curious whether the TPM-first pattern holds for other use cases or if it's specific to the long-prompt workloads PostAll generates.

What Is Content Automation? A Developer's Explanation for Non-Developers

Aakash Gour — Fri, 15 May 2026 05:56:16 +0000

I've given the "what does PostAll actually do?" explanation maybe 200 times. To investors. To potential customers. To my mom. To a guy at a barbecue who asked why I'd quit my job.

Every single time, I started wrong. I'd say something like: "It uses AI to generate content at scale." And every single time, I'd watch the person nod politely while their eyes said I have no idea what that means.

So I rebuilt the explanation from scratch. This is the version that actually works — for non-developers, for business people, for that guy at the barbecue. But I'm writing it for developers, because you're the ones who get asked to explain this stuff to your colleagues, your clients, and your leadership. And you deserve an explanation that's honest about what's genuinely automated, what isn't, and where the hard problems actually live.

The definition that actually sticks

Content automation is the practice of building software systems that produce written content — blog posts, product descriptions, emails, SEO copy — through code rather than through people sitting down to write.

The operative word is systems. Not "I typed a prompt into ChatGPT." A system means: given a defined input, produce a defined output, repeatably, at scale, without human intervention for each individual piece.

That's the thing most people miss. Anyone can generate one AI article. Content automation is what happens when you need 5,000 of them, formatted correctly, published to the right places, with consistent quality, without hiring 50 writers.

What it looks like in code (the two-minute version)

Here's the absolute minimal version of a content automation pipeline. Not pseudocode — this actually runs:

import OpenAI from "openai";

const openai = new OpenAI();

async function generateProductDescription(product) {
  const { name, category, keyFeatures, targetAudience } = product;

  const prompt = `Write a 150-word product description for an e-commerce listing.

Product: ${name}
Category: ${category}
Key features: ${keyFeatures.join(", ")}
Target audience: ${targetAudience}

Format: One punchy opening sentence. Two sentences on what it does. One sentence on who it's for.
Tone: Confident, specific, no marketing fluff.`;

  const response = await openai.chat.completions.create({
    model: "gpt-4o",
    messages: [{ role: "user", content: prompt }],
    max_tokens: 300,
    temperature: 0.6,
    // Lower temperature = more consistent output across runs
    // 0.6 is the sweet spot for product copy: creative but not chaotic
  });

  return response.choices[0].message.content;
}

That's it. That's the core of what PostAll does — except PostAll does it for thousands of products, validates the output, reformats it for each CMS, handles rate limits, retries failures, stores everything in a database, and gives you a dashboard to review it all.

The code above is 30 lines. The system around it is months of work.

The three things content automation actually automates

Most people think content automation means "AI writes the content." That's maybe 20% of it. Here's what the other 80% is:

1. Input processing

Before any LLM sees a prompt, something has to assemble that prompt from your raw data. PostAll takes a product CSV with 47 columns and figures out which 5 fields actually matter for a description. It normalizes inconsistent data ("BLUE", "Blue", "blue" all become "Blue"). It handles missing fields gracefully instead of crashing.

This is genuinely unglamorous data engineering work. It's not AI. It's just code.

2. Output parsing and validation

LLMs don't return clean structured data. They return text. Sometimes the text is perfect. Sometimes it's 400 words when you asked for 150. Sometimes it includes "Sure! Here's a product description:" at the top. Sometimes it hallucinates a feature that doesn't exist.

A real content automation system catches all of this. In PostAll, every piece of generated content runs through a validation layer before it's stored:

function validateProductDescription(content, product) {
  const wordCount = content.split(/\s+/).length;
  const errors = [];

  // Check length bounds
  if (wordCount < 100 || wordCount > 200) {
    errors.push(`Word count ${wordCount} outside target range 100-200`);
  }

  // Check that hallucinated features didn't sneak in
  // We extract claimed features and diff against known features
  const claimedFeatures = extractFeatureClaims(content);
  const unknownClaims = claimedFeatures.filter(
    (f) => !product.verifiedFeatures.includes(f)
  );
  if (unknownClaims.length > 0) {
    errors.push(`Unverified claims detected: ${unknownClaims.join(", ")}`);
  }

  // Strip LLM preamble if present ("Sure! Here's..." etc.)
  const cleaned = stripLLMPreamble(content);

  return { valid: errors.length === 0, errors, content: cleaned };
}

If validation fails, PostAll retries with a modified prompt. If it fails again, it flags the item for human review and moves on. It doesn't stop the entire batch because one product had bad data.

3. Delivery and formatting

"Content" means different things to different systems. WordPress wants HTML. Shopify wants Metafields JSON. A legacy CMS might want XML. Your email platform wants plaintext with specific line breaks.

Content automation handles all of this transformation. The LLM generates a canonical version; the delivery layer reformats it for each destination.

What content automation is NOT

This is the part I always have to explain to business stakeholders, because their expectations are usually wrong in one of two directions.

It's not a magic button that replaces all your writers. Content automation works for structured, repeatable content types: product descriptions, SEO landing pages, email sequences, job postings, real estate listings. It does not work well for thought leadership, investigative journalism, creative brand storytelling, or anything that requires original research or lived experience.

PostAll's best customers use it to eliminate the work their writers found tedious — generating the first 10 versions of a product description — so the writers can focus on the 10% that actually requires human judgment.

It's not "just ChatGPT." The gap between "I used ChatGPT to write something" and "I have a content automation pipeline" is the same as the gap between "I wrote a SQL query once" and "I have a production database." The AI model is one component. The infrastructure around it — data pipelines, validation, delivery, monitoring, retry logic, cost management — is the actual product.

Where it actually gets hard

If you're a developer thinking about building something in this space, here are the three problems that will take longer than you expect:

Rate limits and cost at scale. GPT-4o costs real money per token. At 500 articles, you feel it. At 50,000, you're rearchitecting your entire approach. The naive implementation — generate everything synchronously, fail loudly on rate limits — breaks immediately. PostAll uses a priority queue with exponential backoff and switches to cheaper models for the validation pass:

const RATE_LIMIT_DELAY = 1000; // 1 req/sec for gpt-4o on free tier

async function generateWithRateLimit(prompts) {
  const results = [];

  for (const prompt of prompts) {
    try {
      const result = await generateContent(prompt);
      results.push(result);
    } catch (error) {
      if (error.status === 429) {
        // Rate limited — wait and retry once before failing
        await sleep(RATE_LIMIT_DELAY * 2);
        const retry = await generateContent(prompt);
        results.push(retry);
      } else {
        throw error;
      }
    }

    await sleep(RATE_LIMIT_DELAY);
  }

  return results;
}

Consistency at scale. Generating one good product description is easy. Generating 5,000 that all sound like the same brand voice is genuinely hard. Temperature settings help. System prompts with style examples help more. But the real solution is a validation layer that catches voice drift — and that requires defining what "on-brand" actually means in code, which is a whole other problem.

The human-in-the-loop design. Every content automation system needs a way for humans to catch what the AI misses. The hard part isn't building the review interface — it's deciding when to route to a human. Route too aggressively, and you've built a very expensive human review queue. Route too loosely, and bad content ships.

In PostAll, we use a confidence threshold based on the validation score. Anything below 85% goes to the review queue. We tuned that number over three months of production data.

What I wish I'd known before building PostAll

I thought the hard part was prompting. It wasn't. Prompting is maybe 15% of the work.

The hard part is infrastructure: queues, retries, cost tracking, format transformation, consistency validation, monitoring. All the stuff that's completely invisible when content automation is working correctly, and catastrophically visible when it's not.

If you're evaluating a content automation tool — or building one — ask about these things specifically. Not "can it generate good content?" Any LLM can do that on a good day. Ask: "What happens when it generates bad content? What happens at 10x scale? What happens when the API goes down?"

Those answers tell you whether you're looking at a demo or a system.

The one-sentence version (for the barbecue)

Content automation is software that turns your data and requirements into published, formatted content — the same way a factory turns raw materials into finished products, except the factory is an API call and a database, and the raw materials are keywords and product specs.

It's not magic. It's engineering. And like all engineering, the interesting parts are the failure cases.

Building something in the content automation space? I'm always happy to compare notes. Drop a comment below — specifically if you've hit the consistency problem at scale. That one still keeps me up.

Build a Bulk Product Description Generator That Actually Ships (Python + GPT-4) published: false

Aakash Gour — Mon, 11 May 2026 05:59:11 +0000

I needed to generate 500 product descriptions for a home goods client by end of week. Their catalog: a spreadsheet with product names, dimensions, materials, and bullet-point features. My deliverable: unique, SEO-optimized copy for each one, ready to paste into Shopify.

The manual approach would take 40+ hours. I had two days.

Here's the Python pipeline I built, what broke along the way, and the exact code that processed all 500 without blowing the API budget or producing garbage output.

What you'll actually build

By the end of this tutorial, you'll have a script that:

Reads a CSV of product data (name, specs, features)
Generates one SEO-optimized description per product via GPT-4o
Handles OpenAI rate limits with automatic exponential backoff
Validates output quality before writing to disk
Saves progress as it runs — so a crash at item 387 doesn't mean starting over
Outputs a clean CSV ready for Shopify, WooCommerce, or any CMS

Here's what the output looks like for a product called "Mango Wood Serving Board":

The Mango Wood Serving Board brings warmth and texture to any table setting.
Hand-finished from sustainably sourced mango wood, each board carries a 
naturally unique grain pattern — no two are identical. At 14x10 inches, it's 
sized for charcuterie spreads, cheese boards, or everyday serving. The juice 
groove keeps counters clean. The integrated handle makes it easy to carry 
from kitchen to table without a second trip.

Dimensions: 14" x 10" x 0.75" | Material: Mango wood | Care: Hand wash only

Real, specific, scannable. Not: "This beautiful board is perfect for all your serving needs!"

Why this is harder than it looks

GPT-4o can write a product description in 3 seconds. That's not the hard part.

The hard part is:

500 API calls = you will hit rate limits. OpenAI's default is 3 requests/second on standard tier.
GPT-4o is nondeterministic — 5% of outputs will be weirdly short, repeat the product name 4 times, or come back in a different language (yes, this happened).
A crash at item 200 means you need to know exactly where you stopped.
Prompt drift — a prompt that works on product 1 may produce subtly different formatting on product 400. You need output validation.

None of the "generate product descriptions with AI" tutorials I found covered any of this. So here's the version that does.

Prerequisites

Python 3.10+
An OpenAI API key (get one here)
openai, pandas, tenacity libraries

pip install openai pandas tenacity

Your input CSV should have these columns (adjust the prompt if yours differ):

product_name, material, dimensions, features, category

The pipeline: overview

Before we get into the code, here's what we're building:

CSV Input
   ↓
Load & validate rows
   ↓
Check progress file (skip already-processed items)
   ↓
For each product:
  → Build prompt
  → Call GPT-4o (with retry logic)
  → Validate output
  → Write to progress file
   ↓
Merge all outputs → final CSV

Five stages. Let's build each one.

Step 1: The prompt (this is 70% of your output quality)

The single biggest variable in this pipeline isn't your retry logic or your rate limiter — it's the prompt. Most tutorials hand you a two-liner and move on. That two-liner will produce two-liner quality output.

Here's the prompt I landed on after about 15 test iterations:

def build_prompt(row: dict) -> str:
    return f"""You are a product copywriter for a premium e-commerce brand.

Write a product description for the item below. Follow these rules exactly:

1. Start with a single sentence that leads with the product's primary use or benefit — not its name.
2. Write 3-4 sentences of body copy. Focus on what the customer will *experience*, not just what the product *is*.
3. End with a one-line spec summary: Dimensions, Material, and one care/compatibility note.
4. Tone: warm, specific, confidence-inspiring. No filler phrases like "perfect for" or "great for any occasion."
5. Total length: 80-120 words. Not shorter. Not longer.

Product data:
- Name: {row['product_name']}
- Material: {row['material']}
- Dimensions: {row['dimensions']}
- Features: {row['features']}
- Category: {row['category']}

Output the description only. No intro, no labels, no quotes around the text."""

A few things I learned the hard way about this prompt:

"Output the description only" is non-negotiable. Without it, GPT-4o would regularly respond with: "Here's a product description for the Mango Wood Serving Board: ..." — and that prefix would end up in your CSV.

The word count constraint is what keeps outputs consistent enough to validate. Without it, you'll get 40-word descriptions and 200-word descriptions and no easy way to catch which is which.

"Not shorter. Not longer." — that redundancy is intentional. Models respond better to reinforced constraints.

Step 2: Rate-limited, retrying API calls

This is where most tutorials hand you a time.sleep(1) and call it done.

Here's what actually handles OpenAI rate limits correctly:

import openai
from tenacity import (
    retry,
    stop_after_attempt,
    wait_exponential,
    retry_if_exception_type
)

client = openai.OpenAI(api_key="your-api-key-here")

@retry(
    retry=retry_if_exception_type(openai.RateLimitError),
    wait=wait_exponential(multiplier=1, min=2, max=60),
    stop=stop_after_attempt(6)
)
def generate_description(prompt: str) -> str:
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": prompt}],
        max_tokens=300,
        temperature=0.7,  # higher = more varied output; lower = more consistent
    )
    return response.choices[0].message.content.strip()

The tenacity library handles exponential backoff automatically. If OpenAI returns a 429 (rate limit), it waits 2 seconds, retries. If it hits again, waits 4 seconds, retries. Up to 60 seconds, up to 6 attempts.

Why temperature=0.7? In my testing, 0.5 produced noticeably templated-feeling output — like every description started to rhyme with every other. 0.9 produced creative copy but with too much variance in structure. 0.7 gave me readable, distinct copy that still hit the formatting constraints.

Step 3: Output validation

This is the section nobody writes. And it's the reason pipelines produce garbage at scale.

def validate_description(text: str, product_name: str) -> tuple[bool, str]:
    word_count = len(text.split())

    # Too short — model probably got confused or hit an error mid-generation
    if word_count < 60:
        return False, f"Too short: {word_count} words"

    # Too long — prompt constraint wasn't followed
    if word_count > 150:
        return False, f"Too long: {word_count} words"

    # Check that the product name appears (basic relevance check)
    # Use the first significant word of the product name only
    key_term = product_name.split()[0].lower()
    if key_term not in text.lower() and len(product_name.split()) > 1:
        # Partial match is fine — GPT-4o often refers to "the board" not "the Mango Wood Serving Board"
        pass

    # Catch GPT-4o's habit of adding its own preamble
    bad_prefixes = ["here's", "here is", "certainly", "of course", "sure,"]
    first_words = text[:30].lower()
    for prefix in bad_prefixes:
        if first_words.startswith(prefix):
            return False, f"Model preamble detected: starts with '{prefix}'"

    return True, "ok"

In my run of 500 products, this validation caught 23 bad outputs — about 4.6%. Most were too short (model returned a partial response) or had the "Here's a description..." preamble. All 23 were re-queued and re-generated successfully on the second pass.

Without this step, those 23 bad descriptions would have shipped to the client's CMS.

Step 4: Progress tracking (so crashes don't cost you)

This is the one I wish I'd built from the start instead of 200 products in.

import json
import os

PROGRESS_FILE = "progress.json"

def load_progress() -> dict:
    if os.path.exists(PROGRESS_FILE):
        with open(PROGRESS_FILE, "r") as f:
            return json.load(f)
    return {}

def save_progress(progress: dict):
    with open(PROGRESS_FILE, "w") as f:
        json.dump(progress, f, indent=2)

Usage in the main loop:

progress = load_progress()

for _, row in df.iterrows():
    product_id = str(row['product_name'])  # or use a SKU if you have one

    # Skip if already processed
    if product_id in progress:
        print(f"Skipping {product_id} — already done")
        continue

    prompt = build_prompt(row.to_dict())

    try:
        description = generate_description(prompt)
        is_valid, reason = validate_description(description, row['product_name'])

        if not is_valid:
            print(f"FAILED validation for {product_id}: {reason}")
            # Still save as failed so we can review separately
            progress[product_id] = {"status": "failed", "reason": reason, "text": description}
        else:
            progress[product_id] = {"status": "ok", "text": description}

        save_progress(progress)  # write after every item

    except Exception as e:
        print(f"ERROR on {product_id}: {e}")
        # Don't save — we'll retry this one next run

The key pattern: save after every successful item, not at the end. If you batch-save at the end and crash at item 387, you start over. If you save after each item, you restart at item 388.

Step 5: Putting it all together

import pandas as pd
import json

def run_pipeline(input_csv: str, output_csv: str):
    df = pd.read_csv(input_csv)
    progress = load_progress()

    total = len(df)
    processed = 0
    failed = 0

    print(f"Starting pipeline: {total} products, {len(progress)} already processed")

    for _, row in df.iterrows():
        product_id = str(row['product_name'])

        if product_id in progress and progress[product_id]['status'] == 'ok':
            processed += 1
            continue

        prompt = build_prompt(row.to_dict())

        try:
            description = generate_description(prompt)
            is_valid, reason = validate_description(description, row['product_name'])

            status = "ok" if is_valid else "failed"
            progress[product_id] = {
                "status": status,
                "text": description,
                "reason": reason if not is_valid else None
            }
            save_progress(progress)

            if is_valid:
                processed += 1
                print(f"✓ {product_id} ({processed}/{total})")
            else:
                failed += 1
                print(f"✗ {product_id} — {reason}")

        except Exception as e:
            failed += 1
            print(f"ERROR: {product_id} — {e}")

    # Build output CSV from progress file
    results = []
    for _, row in df.iterrows():
        product_id = str(row['product_name'])
        entry = progress.get(product_id, {})
        results.append({
            **row.to_dict(),
            "description": entry.get("text", ""),
            "status": entry.get("status", "missing")
        })

    output_df = pd.DataFrame(results)
    output_df.to_csv(output_csv, index=False)
    print(f"\nDone. {processed} succeeded, {failed} failed.")
    print(f"Output: {output_csv}")

# Run it
run_pipeline("products.csv", "products_with_descriptions.csv")

What can go wrong (and how to handle it)

Rate limit errors that tenacity can't catch

OpenAI occasionally returns 429s with a Retry-After header that's longer than tenacity's max wait. If you see RetryError in your logs, check the error message — if it says Retry-After: 120, add a time.sleep(120) before starting the next batch.

The model starts producing descriptions in another language

This happened to me on item 214 — a product called "Zen Garden Kit" triggered a response in Japanese. Not sure why. The word count validation caught it (Japanese description was 11 words in the English token count). Fix: add an explicit instruction to your prompt: "Write in English only." I added this after the incident and never saw it again.

Encoding errors in your CSV

If your product CSV was exported from Excel, there's a decent chance it has Windows-1252 encoding, not UTF-8. Products with special characters (é, ñ, trademark symbols) will throw a UnicodeDecodeError. Fix:

df = pd.read_csv(input_csv, encoding='utf-8-sig')
# or
df = pd.read_csv(input_csv, encoding='latin-1')

GPT-4o "phones it in" on obvious product categories

Generic categories like "Storage Box" or "Throw Pillow" produced noticeably weaker descriptions — the model seems to default to filler language for low-specificity inputs. Fix: add a avoid_phrases instruction to your prompt:

"Avoid phrases like: 'perfect for', 'great for any occasion', 'ideal for the whole family', 'make a statement', 'elevate your space'."

This alone cleaned up about 30 descriptions in my run.

Results

The full pipeline processed 500 products in about 2 hours and 20 minutes — most of that was intentional rate limiting. API cost at gpt-4o pricing: approximately $4.80 for the run. 23 products failed validation on the first pass; 21 of those passed on re-queue. 2 were manually rewritten (one product with almost no spec data, one with what appeared to be corrupted input data).

The client uploaded the CSV to Shopify on Friday. No manual editing on any description that passed validation.

The complete script

The full working script — with all five components integrated — is on GitHub: github.com/your-handle/bulk-product-description-generator

The repo includes:

pipeline.py — the main script
sample_products.csv — 10 example products to test with
prompts/ — the base prompt plus two variations I tested
README.md — setup instructions and config options

Have you built something similar? I'm particularly curious what prompt structures other people have landed on for e-commerce copy — the one I'm using works well for home goods but I suspect it'd need significant changes for, say, technical products or apparel. Drop your approach in the comments.

Designing a Content Microservice: Architecture Decisions and Trade-offs

Aakash Gour — Fri, 08 May 2026 05:14:54 +0000

The first version of PostAll's content generation system wasn't a microservice. It was a Next.js API route that called OpenAI, wrote the result to a database, and returned a response. It took 8 seconds per article. Our first beta user queued 200 articles and the server timed out after the 11th.

That's the moment you stop thinking about "how do I generate content" and start thinking about "how do I build a system that generates content." They're completely different problems.

This is the architecture I landed on after three rewrites — what I chose, why I chose it, and where each approach broke before I got there.

Why content generation is a bad fit for synchronous APIs

Most API routes work like this: request comes in, thing happens, response goes out. The whole exchange is sub-second. Content generation breaks this model in three ways.

Latency. A GPT-4o call for a 1,000-word article takes 8–15 seconds. That's not a timeout edge case — that's the normal case. No HTTP client should sit open for 15 seconds waiting.

Failure modes. LLM APIs fail. Rate limits hit. Tokens run out mid-response. At low volume, you catch these with a try/catch and a retry. At scale, you need state — you need to know which of 500 jobs failed, why, and whether it's safe to retry.

Fan-out. A user submitting 100 articles isn't submitting 100 requests — they're submitting 1 request that should spawn 100 coordinated jobs. Synchronous APIs can't express that without blocking.

The solution to all three is the same: decouple submission from execution with a job queue.

The queue design that actually held up

I went through two queue implementations before landing on something stable.

First attempt: in-memory queue with setInterval.

// What I started with — don't do this
const jobQueue = [];

app.post('/generate', (req, res) => {
  const jobId = uuid();
  jobQueue.push({ id: jobId, ...req.body });
  res.json({ jobId });
});

setInterval(async () => {
  const job = jobQueue.shift();
  if (job) await processJob(job);
}, 100);

This lasted until the first server restart. Everything in the queue evaporated. Lesson learned: queue state needs to survive the process.

Second attempt: database as queue.

// Using Postgres as a job queue — better, but read the gotchas below
async function enqueueJob(payload) {
  const { rows } = await db.query(
    `INSERT INTO content_jobs (id, status, payload, priority, created_at)
     VALUES ($1, 'pending', $2, $3, NOW())
     RETURNING id`,
    [uuid(), JSON.stringify(payload), payload.priority ?? 0]
  );
  return rows[0].id;
}

async function claimNextJob(workerId) {
  // SELECT FOR UPDATE SKIP LOCKED is critical here —
  // without it, multiple workers race on the same row
  const { rows } = await db.query(`
    UPDATE content_jobs
    SET status = 'processing', worker_id = $1, claimed_at = NOW()
    WHERE id = (
      SELECT id FROM content_jobs
      WHERE status = 'pending'
      ORDER BY priority DESC, created_at ASC
      FOR UPDATE SKIP LOCKED
      LIMIT 1
    )
    RETURNING *
  `);
  return rows[0] ?? null;
}

The FOR UPDATE SKIP LOCKED clause is what makes this safe for concurrent workers. Without it, two workers claim the same job, you get duplicated output, and debugging it is a nightmare. I hit this at 3 workers — it got worse fast.

What I use now: dedicated queue (BullMQ + Redis).

For teams already running Redis, BullMQ solves priority queues, retries, rate limiting per worker, and dead-letter queues out of the box. The database-as-queue approach is legitimate for low volumes, but past ~50 jobs/minute the polling overhead shows up in your database metrics.

import { Queue, Worker } from 'bullmq';
import Redis from 'ioredis';

const connection = new Redis(process.env.REDIS_URL);

export const contentQueue = new Queue('content-generation', {
  connection,
  defaultJobOptions: {
    attempts: 3,
    backoff: { type: 'exponential', delay: 2000 },
    removeOnComplete: { count: 1000 }, // keep last 1000 for debugging
    removeOnFail: false,               // keep all failures — you'll want to inspect them
  },
});

// Priority lanes: 10 = high (single article), 1 = low (bulk batch)
export async function enqueue(payload, priority = 1) {
  return contentQueue.add('generate', payload, { priority });
}

The removeOnFail: false line is important. Failed jobs tell you things — which prompts consistently fail, which users hit edge cases, where rate limiting is biting you. Don't throw them away.

Worker architecture: three lanes, not one

The instinct is to have one worker that does everything: calls the LLM, formats the output, validates it, writes to the database. This works until you need to scale generation without scaling formatting, or you want to add a quality scoring step without touching the generation logic.

I split workers into three types:

Generation worker  → calls LLM, stores raw output
Formatting worker  → transforms raw → target format (HTML, Markdown, JSON)
Post-processing    → SEO checks, quality scoring, webhook dispatch

Each one pulls from its own queue. Generation output triggers a formatting job. Formatting completion triggers post-processing. This means:

You can scale generation horizontally when you're rate-limit-bound
You can swap the formatting logic without touching generation
A formatting bug doesn't take down generation

// Generation worker — one responsibility
const generationWorker = new Worker(
  'content-generation',
  async (job) => {
    const { keyword, tone, maxTokens, outputFormat } = job.data;

    const raw = await generateContent({ keyword, tone, maxTokens });

    // Store raw output immediately — if formatting fails, we have the source
    const articleId = await db.saveRawContent({
      jobId: job.id,
      raw,
      metadata: { keyword, tone, outputFormat },
    });

    // Trigger next stage
    await formattingQueue.add('format', { articleId, outputFormat });

    return { articleId };
  },
  {
    connection,
    concurrency: 5, // tune based on your API tier rate limits
    limiter: {
      max: 50,       // OpenAI tier 2: 5000 RPM → ~83 RPS; leave headroom
      duration: 60000,
    },
  }
);

The concurrency: 5 isn't arbitrary. On OpenAI's Tier 2 with GPT-4o, each call takes 8–12 seconds. At 5 concurrent workers, you're making 25–37 requests per minute — well within limits. Push to 15 and you'll start seeing 429s, which trigger retries, which increase your effective request count, which makes the 429s worse. Find your actual throughput ceiling before tuning this.

The caching decision I almost got wrong

My first instinct was to cache LLM responses by prompt hash. If the same exact prompt comes in twice, return the cached result. Simple, right?

The problem: content generation prompts are almost never identical. Even "write a blog post about React hooks" varies by tone parameter, target length, style instruction. My cache hit rate after a week was 2.3%. The Redis overhead wasn't worth it.

What is worth caching:

Prompt templates. If your prompts are built from templates + variables, cache the rendered template structure, not the final output. Template rendering is cheap but if you're doing it 500 times for a bulk job, it adds up.

External data fetches. If your prompts include fetched data (product descriptions, SEO keywords from an API), cache those aggressively. An article generation job that fetches 3 external APIs per article doesn't need to make 1,500 API calls for a batch of 500.

Quality scoring results. If you run a readability scorer or SEO checker on output, cache by content hash. The same output will score the same way every time.

async function getPromptTemplate(templateId) {
  const cacheKey = `template:${templateId}`;
  const cached = await redis.get(cacheKey);
  if (cached) return JSON.parse(cached);

  const template = await db.getTemplate(templateId);
  // Templates change infrequently — 1 hour TTL is safe
  await redis.setex(cacheKey, 3600, JSON.stringify(template));
  return template;
}

Monitoring: the metrics that actually matter

I spent two weeks looking at the wrong things. CPU and memory on the worker boxes stayed flat. What I should have been watching:

Queue depth over time. If your queue is growing faster than it's draining, you have a capacity problem. You want to alert when pending_jobs > (worker_count × average_job_duration × 2).

Job age at processing time. This is different from queue depth. A queue of 100 jobs with 5-second average age is healthy. A queue of 100 jobs with 45-minute average age means your throughput has been underwater for an hour.

Token cost per job. LLM costs are the primary cost driver for PostAll. Tracking average token usage per job type lets you catch prompt bloat before it shows up in your OpenAI invoice.

Dead-letter queue size. If your DLQ grows, something systematic is failing — not just occasional network hiccups. I check this daily; spikes always point to something actionable.

// Expose these as a /metrics endpoint — pipe to Grafana or Datadog
async function getQueueMetrics() {
  const [waiting, active, failed, delayed] = await Promise.all([
    contentQueue.getWaitingCount(),
    contentQueue.getActiveCount(),
    contentQueue.getFailedCount(),
    contentQueue.getDelayedCount(),
  ]);

  const oldestJob = await contentQueue.getJobs(['waiting'], 0, 0);
  const jobAgeSeconds = oldestJob[0]
    ? (Date.now() - oldestJob[0].timestamp) / 1000
    : 0;

  return { waiting, active, failed, delayed, jobAgeSeconds };
}

What I'd do differently

Start with the database queue, not BullMQ. BullMQ is great, but it's an additional dependency with its own failure modes. For the first 10,000 jobs, Postgres as a queue is completely fine and removes Redis as a point of failure. Migrate when you have a concrete reason.

Design the webhook contract on day one. PostAll's clients poll for job status. I built the webhook system as an afterthought and had to retrofit it. The async nature of the system means clients need a push notification — don't make them poll. Define the webhook payload schema before you write the first worker.

Add job metadata fields you don't need yet. organization_id, triggered_by, cost_usd, model_used. These fields are free to store and priceless when you're debugging a production issue at 11pm and need to know which tenant generated a specific piece of content and how much it cost.

What it handles now

PostAll's current architecture processes 500 articles per hour across 8 generation workers, 4 formatting workers, and 2 post-processing workers. P95 end-to-end latency — from API submission to webhook delivery — is 47 seconds for standard articles. That's not fast, but it's reliable, it's observable, and when something breaks, the failure is contained to one stage.

The clients get a job ID immediately. The rest is async. That's the only architecture that makes sense for workloads this slow.

What's your queue strategy for async workloads? I've seen teams reach for Kafka for this and pay a complexity tax they didn't need. Curious where that decision point actually is — drop it in the comments.

Build a Markdown-to-CMS Auto-Publisher With GitHub Actions

Aakash Gour — Mon, 04 May 2026 05:18:07 +0000

I got tired of the same three-step content publish loop: write draft → open CMS → paste, format, re-paste, fight the rich-text editor, click publish. Repeat for every environment — staging, then production. For one article, fine. For a team publishing 20+ pieces a month? That workflow is a quiet tax on everyone's time.

So I wired up a pipeline that cuts the loop entirely. You commit a .md file to a Git repo. A GitHub Actions workflow runs. The article is live on WordPress, Ghost, or Webflow — formatted, tagged, and published — within 90 seconds of the push.

Here's the complete workflow, every config file, and the two places where this will bite you if you're not careful.

What You'll Actually Build

By the end of this, you'll have:

A GitHub repo that acts as your content source of truth
A GitHub Actions workflow that triggers on push to main
A Node.js publish script that parses frontmatter, transforms Markdown, and hits your CMS API
Support for WordPress (REST API), Ghost (Admin API), and Webflow (CMS API) — pick one or all three
A staging branch that publishes drafts, a main branch that publishes live

Here's the directory structure we're building toward:

content-pipeline/
├── .github/
│   └── workflows/
│       └── publish.yml
├── posts/
│   ├── my-first-post.md
│   └── another-post.md
├── scripts/
│   ├── publish.js
│   ├── publishers/
│   │   ├── wordpress.js
│   │   ├── ghost.js
│   │   └── webflow.js
│   └── utils/
│       ├── parse-frontmatter.js
│       └── transform-markdown.js
├── package.json
└── .env.example

Why This Is Harder Than It Looks

The obvious approach — "just hit the API with the Markdown" — doesn't work in practice. Three reasons:

1. CMS APIs don't accept raw Markdown. WordPress wants HTML. Ghost accepts Markdown but has its own Lexical editor format for complex content. Webflow expects structured JSON with rich text field schemas.

2. Frontmatter is your metadata contract, and every CMS maps it differently. tags in Ghost is an array of tag objects. In WordPress, it's an array of tag IDs you have to look up first. In Webflow, tags don't exist — you have custom field slugs.

3. Idempotency. If the Action runs twice, you don't want two copies of the same article. You need a slug-based lookup before every publish to decide "create" vs "update."

These aren't edge cases — they'll hit you on the first real push. The script below handles all three.

Prerequisites

Node.js 18+
A GitHub repo (free tier works)
At least one of: a WordPress site with Application Passwords enabled, a Ghost instance with Admin API access, a Webflow CMS collection
Basic familiarity with GitHub Actions syntax

Step 1: The Frontmatter Contract

Every post needs a consistent frontmatter block. This is how the publish script knows what to do with each file.

---
title: "My Article Title"
slug: "my-article-title"
description: "One-sentence summary for meta descriptions and CMS excerpts."
tags: ["javascript", "devops", "tutorial"]
status: "published"        # or "draft"
targets: ["wordpress", "ghost"]   # which CMSes to publish to
date: "2025-01-15"
cover_image: "https://images.unsplash.com/photo-xyz"
---

Your article content starts here...

The targets field is the key one. It lets a single repo serve multiple CMSes without every post going everywhere. A marketing post might target WordPress only. A technical deep-dive might go to Ghost. The script reads this and routes accordingly.

Step 2: The Frontmatter Parser

// scripts/utils/parse-frontmatter.js
import fs from 'fs';
import matter from 'gray-matter';
import { marked } from 'marked';

export function parsePost(filePath) {
  const raw = fs.readFileSync(filePath, 'utf-8');
  const { data: frontmatter, content } = matter(raw);

  // Validate required fields before we touch any API
  const required = ['title', 'slug', 'targets'];
  const missing = required.filter(field => !frontmatter[field]);
  if (missing.length > 0) {
    throw new Error(`Missing required frontmatter fields: ${missing.join(', ')} in ${filePath}`);
  }

  return {
    frontmatter,
    markdown: content,
    html: marked(content),   // WordPress needs this
    filePath,
  };
}

Two things worth calling out here: gray-matter is the standard Markdown frontmatter parser — rock solid, widely used. And I'm converting to HTML at parse time so every publisher gets both formats and can choose what it needs. You'll need to install these:

npm install gray-matter marked

Step 3: The WordPress Publisher

WordPress's REST API is the most mature of the three. The catch is tag handling — you can't just pass tag names; you have to resolve them to IDs first, creating any that don't exist.

// scripts/publishers/wordpress.js
const WP_BASE = process.env.WP_BASE_URL;   // e.g. https://yourblog.com
const WP_USER = process.env.WP_USERNAME;
const WP_PASS = process.env.WP_APP_PASSWORD; // Application Password, not your login password

const authHeader = 'Basic ' + Buffer.from(`${WP_USER}:${WP_PASS}`).toString('base64');

async function resolveTagIds(tagNames) {
  const ids = [];

  for (const name of tagNames) {
    // Check if tag exists
    const searchRes = await fetch(
      `${WP_BASE}/wp-json/wp/v2/tags?search=${encodeURIComponent(name)}`,
      { headers: { Authorization: authHeader } }
    );
    const existing = await searchRes.json();

    if (existing.length > 0) {
      ids.push(existing[0].id);
    } else {
      // Create it
      const createRes = await fetch(`${WP_BASE}/wp-json/wp/v2/tags`, {
        method: 'POST',
        headers: {
          Authorization: authHeader,
          'Content-Type': 'application/json',
        },
        body: JSON.stringify({ name }),
      });
      const newTag = await createRes.json();
      ids.push(newTag.id);
    }
  }

  return ids;
}

export async function publishToWordPress({ frontmatter, html }) {
  const tagIds = await resolveTagIds(frontmatter.tags ?? []);

  // Check if post already exists (idempotency)
  const slugCheckRes = await fetch(
    `${WP_BASE}/wp-json/wp/v2/posts?slug=${frontmatter.slug}`,
    { headers: { Authorization: authHeader } }
  );
  const existing = await slugCheckRes.json();

  const payload = {
    title: frontmatter.title,
    slug: frontmatter.slug,
    content: html,
    excerpt: frontmatter.description ?? '',
    status: frontmatter.status ?? 'draft',
    tags: tagIds,
    date: frontmatter.date,
    featured_media: 0,  // Extend this if you want cover image upload
  };

  if (existing.length > 0) {
    // Update
    const res = await fetch(`${WP_BASE}/wp-json/wp/v2/posts/${existing[0].id}`, {
      method: 'PUT',
      headers: { Authorization: authHeader, 'Content-Type': 'application/json' },
      body: JSON.stringify(payload),
    });
    const updated = await res.json();
    console.log(`✅ WordPress: Updated "${updated.title.rendered}" → ${updated.link}`);
    return updated;
  } else {
    // Create
    const res = await fetch(`${WP_BASE}/wp-json/wp/v2/posts`, {
      method: 'POST',
      headers: { Authorization: authHeader, 'Content-Type': 'application/json' },
      body: JSON.stringify(payload),
    });
    const created = await res.json();
    console.log(`✅ WordPress: Created "${created.title.rendered}" → ${created.link}`);
    return created;
  }
}

Important: Use WordPress Application Passwords, not your account password. Go to Users → Profile → Application Passwords in your WP Admin. The format is username:xxxx xxxx xxxx xxxx xxxx xxxx (with spaces — that's normal).

Step 4: The Ghost Publisher

Ghost's Admin API uses JWT authentication, which is slightly more involved to set up but more elegant once it's running. Ghost also accepts Markdown natively via its mobiledoc format, but the cleanest approach for programmatic publishing is using the @tryghost/admin-api package.

npm install @tryghost/admin-api

// scripts/publishers/ghost.js
import GhostAdminAPI from '@tryghost/admin-api';

const ghost = new GhostAdminAPI({
  url: process.env.GHOST_URL,         // e.g. https://yoursite.ghost.io
  key: process.env.GHOST_ADMIN_KEY,   // Format: id:secret (from Ghost Admin → Integrations)
  version: 'v5.0',
});

export async function publishToGhost({ frontmatter, markdown }) {
  // Ghost tags are objects, not just strings
  const tags = (frontmatter.tags ?? []).map(name => ({ name }));

  const postData = {
    title: frontmatter.title,
    slug: frontmatter.slug,
    mobiledoc: buildMobiledoc(markdown),
    custom_excerpt: frontmatter.description ?? '',
    status: frontmatter.status ?? 'draft',
    tags,
    published_at: frontmatter.date ? new Date(frontmatter.date).toISOString() : undefined,
    feature_image: frontmatter.cover_image ?? null,
  };

  try {
    // Try to find existing post by slug
    const existing = await ghost.posts.browse({ filter: `slug:${frontmatter.slug}` });

    if (existing.length > 0) {
      const updated = await ghost.posts.edit({
        id: existing[0].id,
        updated_at: existing[0].updated_at,  // Required for conflict detection
        ...postData,
      });
      console.log(`✅ Ghost: Updated "${updated.title}" → ${updated.url}`);
      return updated;
    } else {
      const created = await ghost.posts.add(postData);
      console.log(`✅ Ghost: Created "${created.title}" → ${created.url}`);
      return created;
    }
  } catch (err) {
    throw new Error(`Ghost publish failed: ${err.message}`);
  }
}

// Ghost uses Mobiledoc internally. This wraps raw Markdown in a Markdown card.
function buildMobiledoc(markdown) {
  return JSON.stringify({
    version: '0.3.1',
    markups: [],
    atoms: [],
    cards: [['markdown', { markdown }]],
    sections: [[10, 0]],
  });
}

Step 5: The Webflow Publisher

Webflow is the most opinionated of the three. Your CMS collection fields need to match your frontmatter keys, and you'll map them explicitly. This also means the publisher is the most customizable.

// scripts/publishers/webflow.js
const WF_TOKEN = process.env.WEBFLOW_API_TOKEN;
const WF_COLLECTION_ID = process.env.WEBFLOW_COLLECTION_ID;

const headers = {
  Authorization: `Bearer ${WF_TOKEN}`,
  'Content-Type': 'application/json',
  'accept-version': '1.0.0',
};

export async function publishToWebflow({ frontmatter, html }) {
  // Check for existing item by slug
  const listRes = await fetch(
    `https://api.webflow.com/collections/${WF_COLLECTION_ID}/items`,
    { headers }
  );
  const { items } = await listRes.json();
  const existing = items?.find(item => item['slug'] === frontmatter.slug);

  // Map your frontmatter to Webflow field slugs
  // These must match the field slugs in your Webflow CMS collection
  const fields = {
    name: frontmatter.title,
    slug: frontmatter.slug,
    'post-body': html,         // Your rich-text field slug in Webflow
    'meta-description': frontmatter.description ?? '',
    'tags-string': (frontmatter.tags ?? []).join(', '),  // Webflow doesn't have native tags
    _archived: false,
    _draft: frontmatter.status !== 'published',
  };

  if (existing) {
    const updateRes = await fetch(
      `https://api.webflow.com/collections/${WF_COLLECTION_ID}/items/${existing._id}`,
      { method: 'PUT', headers, body: JSON.stringify({ fields }) }
    );
    const updated = await updateRes.json();
    console.log(`✅ Webflow: Updated "${updated.fields.name}"`);

    // Publish immediately if status is "published"
    if (frontmatter.status === 'published') {
      await publishWebflowItem(updated._id);
    }
    return updated;
  } else {
    const createRes = await fetch(
      `https://api.webflow.com/collections/${WF_COLLECTION_ID}/items`,
      { method: 'POST', headers, body: JSON.stringify({ fields }) }
    );
    const created = await createRes.json();
    console.log(`✅ Webflow: Created "${created.fields.name}"`);

    if (frontmatter.status === 'published') {
      await publishWebflowItem(created._id);
    }
    return created;
  }
}

// Webflow requires a separate "publish" API call after creating/updating
async function publishWebflowItem(itemId) {
  await fetch(
    `https://api.webflow.com/collections/${WF_COLLECTION_ID}/items/publish`,
    {
      method: 'PUT',
      headers,
      body: JSON.stringify({ itemIds: [itemId] }),
    }
  );
}

Webflow gotcha: The "publish" step is separate from creating/updating. If you skip it, your item will exist in the CMS but won't be live. I missed this for about an hour wondering why my posts weren't showing up.

Step 6: The Main Publish Script

This ties everything together. It finds changed .md files, parses them, and routes to the right publishers.

// scripts/publish.js
import { parsePost } from './utils/parse-frontmatter.js';
import { publishToWordPress } from './publishers/wordpress.js';
import { publishToGhost } from './publishers/ghost.js';
import { publishToWebflow } from './publishers/webflow.js';
import { execSync } from 'child_process';
import path from 'path';

const PUBLISHERS = {
  wordpress: publishToWordPress,
  ghost: publishToGhost,
  webflow: publishToWebflow,
};

async function main() {
  // Get list of changed .md files from git
  // In GitHub Actions, GITHUB_SHA gives us the current commit
  const changedFiles = execSync(
    'git diff --name-only HEAD~1 HEAD -- "posts/*.md"'
  )
    .toString()
    .trim()
    .split('\n')
    .filter(Boolean);

  if (changedFiles.length === 0) {
    console.log('No markdown files changed. Nothing to publish.');
    return;
  }

  console.log(`Found ${changedFiles.length} changed post(s): ${changedFiles.join(', ')}`);

  const results = { success: [], failed: [] };

  for (const filePath of changedFiles) {
    const fullPath = path.resolve(filePath);

    try {
      const post = parsePost(fullPath);
      const { targets = [] } = post.frontmatter;

      if (targets.length === 0) {
        console.log(`⚠️  Skipping ${filePath}: no targets specified in frontmatter`);
        continue;
      }

      for (const target of targets) {
        const publisher = PUBLISHERS[target];
        if (!publisher) {
          console.warn(`⚠️  Unknown target "${target}" in ${filePath}`);
          continue;
        }
        await publisher(post);
      }

      results.success.push(filePath);
    } catch (err) {
      console.error(`❌ Failed to publish ${filePath}: ${err.message}`);
      results.failed.push({ filePath, error: err.message });
    }
  }

  console.log(`\nDone. ${results.success.length} succeeded, ${results.failed.length} failed.`);

  // Exit with error if any publish failed — this fails the GitHub Action
  if (results.failed.length > 0) {
    process.exit(1);
  }
}

main();

Step 7: The GitHub Actions Workflow

# .github/workflows/publish.yml
name: Publish Content

on:
  push:
    branches:
      - main        # Publishes live
      - staging     # Publishes as drafts (see note below)
    paths:
      - 'posts/**'  # Only triggers when posts change — not on script edits

jobs:
  publish:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout
        uses: actions/checkout@v4
        with:
          fetch-depth: 2   # Need at least 2 commits for git diff to work

      - name: Setup Node
        uses: actions/setup-node@v4
        with:
          node-version: '18'
          cache: 'npm'

      - name: Install dependencies
        run: npm ci

      - name: Run publish script
        env:
          # WordPress
          WP_BASE_URL: ${{ secrets.WP_BASE_URL }}
          WP_USERNAME: ${{ secrets.WP_USERNAME }}
          WP_APP_PASSWORD: ${{ secrets.WP_APP_PASSWORD }}
          # Ghost
          GHOST_URL: ${{ secrets.GHOST_URL }}
          GHOST_ADMIN_KEY: ${{ secrets.GHOST_ADMIN_KEY }}
          # Webflow
          WEBFLOW_API_TOKEN: ${{ secrets.WEBFLOW_API_TOKEN }}
          WEBFLOW_COLLECTION_ID: ${{ secrets.WEBFLOW_COLLECTION_ID }}
        run: node scripts/publish.js

Set all those secrets.* values in your repo under Settings → Secrets and variables → Actions. None of these should ever touch your codebase directly.

The staging branch pattern: Push to staging when the post is a draft you want to preview in the real CMS. The frontmatter's status field controls whether it's a draft or published — so pushing to staging doesn't auto-publish; it just means "run the pipeline." You still control publish state via frontmatter.

What Can Go Wrong

Git diff returns empty on the first commit. The HEAD~1 diff requires at least two commits. If your repo is brand new and this is the first push, there's no HEAD~1. Fix: use git diff --name-only 4b825dc..HEAD -- "posts/*.md" (the magic empty tree SHA) for the initial commit, or add a guard:

const isFirstCommit = execSync('git rev-list --count HEAD').toString().trim() === '1';
const diffCommand = isFirstCommit
  ? 'git diff --name-only 4b825dc HEAD -- "posts/*.md"'
  : 'git diff --name-only HEAD~1 HEAD -- "posts/*.md"';

WordPress Application Passwords with special characters. WP generates passwords with spaces. When you put username:abcd efgh ijkl in an env variable, the spaces can cause header parsing issues. URL-encode the password or store it already base64-encoded.

Ghost updated_at mismatch. Ghost's edit endpoint requires you pass back the updated_at timestamp from the fetched post, or it rejects the update with a 409 conflict. The code above handles this, but if you strip that field for any reason, you'll get confusing errors.

Webflow's API rate limit is 60 requests/minute. If you're bulk-publishing 30+ posts in one commit (like migrating a blog), you'll hit it. Add a delay between requests:

// Add after each publish call inside the loop
await new Promise(resolve => setTimeout(resolve, 1100)); // ~55 req/min

The workflow doesn't trigger when you expect. Double-check your paths filter — posts/** only fires when files inside posts/ change. If you accidentally commit to posts/drafts/ and your pattern doesn't include subdirectories, nothing runs. posts/** handles subdirectories; posts/* does not.

The `.env.example` File

Commit this to the repo so anyone cloning it knows what to configure:

# WordPress
WP_BASE_URL=https://yourblog.com
WP_USERNAME=your_wp_username
WP_APP_PASSWORD=xxxx xxxx xxxx xxxx xxxx xxxx

# Ghost
GHOST_URL=https://yoursite.ghost.io
GHOST_ADMIN_KEY=id:secret

# Webflow
WEBFLOW_API_TOKEN=your_token
WEBFLOW_COLLECTION_ID=your_collection_id

Never commit a .env with real values. The .gitignore should include .env from the start.

Where to Take This Next

The core pipeline is solid as-is. Three extensions worth building if you use this heavily:

Image optimization on push. Add a step before publish.js that pulls image URLs from Markdown, downloads them, runs them through sharp, and uploads to your CDN. Then rewrites the src attributes before publishing. The article goes live already using your own CDN, not third-party hotlinks.

Slack notification on success/failure. Add a final workflow step that posts to a #content-deploys channel. The message should include the article title, which CMSes it published to, and a link. Tiny addition, huge QoL for a content team.

Scheduled publishing. Frontmatter already has a date field. You can add a GitHub Actions schedule trigger (on: schedule) that runs the publisher daily and checks whether any post's date has passed and its status is scheduled. Publish those automatically. True scheduled publishing without any CMS-specific plan tier.

The full working repo is on GitHub — link in my profile. Drop a star if this saved you the setup time. And if you've wired this up to a different CMS — Contentful, Sanity, Strapi — I'd genuinely like to see how you handled the schema mapping. Drop it in the comments.