DEV Community: Nathan Cross

Firecrawl in Production: Surviving 10K Concurrent Scrapes on Cloudflare Workers

Nathan Cross — Mon, 02 Nov 2026 21:10:24 +0000

The Problem: Web Scraping at Edge Scale

When integrating Firecrawl into a serverless AI agent pipeline, the edge runtime constraints hit hard. Firecrawl's Python SDK assumes long-lived connections and retries, but Cloudflare Workers enforce a 50ms CPU limit per invocation and zero persistent state between requests. Our initial naive implementation would queue 200+ scrape jobs, exhaust the Firecrawl API rate limit, and leave Workers hanging on unresolved promises.

The core issue: Firecrawl's async job polling doesn't map cleanly to Workers' ephemeral execution model. You need structured queuing, strict timeout guards, and fallback routing when the primary scraping endpoint goes cold.

Architecture: Hono.js Router + Workers KV for Job State

We built a stateless orchestration layer using Hono.js on Cloudflare Workers. Each incoming /scrape request triggers a Firecrawl job, writes the job ID to Workers KV with a 5-minute TTL, and returns immediately. A separate /poll/:jobId endpoint checks KV for cached results or queries Firecrawl's status API with exponential backoff.

Complete Hono Router Implementation

import { Hono } from 'hono';
import { FirecrawlApp } from '@mendable/firecrawl-js';

type Bindings = {
  FIRECRAWL_API_KEY: string;
  SCRAPE_JOBS: KVNamespace;
};

const app = new Hono<{ Bindings: Bindings }>();

app.post('/scrape', async (c) => {
  const { url, formats } = await c.req.json();
  const firecrawl = new FirecrawlApp({ apiKey: c.env.FIRECRAWL_API_KEY });

  try {
    const job = await firecrawl.scrapeUrl(url, {
      formats: formats || ['markdown', 'html'],
      timeout: 30000,
    });

    const jobId = crypto.randomUUID();
    await c.env.SCRAPE_JOBS.put(
      jobId,
      JSON.stringify({ status: 'pending', firecrawlJobId: job.id, url }),
      { expirationTtl: 300 }
    );

    return c.json({ jobId, pollUrl: `/poll/${jobId}` }, 202);
  } catch (err: any) {
    return c.json({ error: err.message, code: 'FIRECRAWL_QUEUE_FULL' }, 429);
  }
});

app.get('/poll/:jobId', async (c) => {
  const jobId = c.req.param('jobId');
  const cached = await c.env.SCRAPE_JOBS.get(jobId, 'json');

  if (!cached) return c.json({ error: 'Job expired or not found' }, 404);

  const firecrawl = new FirecrawlApp({ apiKey: c.env.FIRECRAWL_API_KEY });
  const status = await firecrawl.checkScrapeStatus(cached.firecrawlJobId);

  if (status.status === 'completed') {
    await c.env.SCRAPE_JOBS.put(jobId, JSON.stringify({ status: 'done', data: status.data }), {
      expirationTtl: 60,
    });
    return c.json({ status: 'completed', markdown: status.data.markdown });
  }

  return c.json({ status: status.status, retryAfter: 2 }, 202);
});

export default app;

Wrangler Configuration with KV Binding

{
  "name": "firecrawl-edge-scraper",
  "main": "src/index.ts",
  "compatibility_date": "2024-01-15",
  "kv_namespaces": [
    {
      "binding": "SCRAPE_JOBS",
      "id": "your_kv_namespace_id",
      "preview_id": "your_preview_kv_id"
    }
  ],
  "vars": {
    "FIRECRAWL_API_KEY": "fc-your-key-here"
  },
  "limits": {
    "cpu_ms": 50
  }
}

The Rate-Limit Wall: Proxy Fallback for High-Volume Pipelines

Under production load (10K scrapes/hour feeding a RAG ingestion pipeline), Firecrawl's public API hit rate limits even with tiered plans. We introduced a fallback routing layer: if Firecrawl returns 429, the Workers router switches to a secondary scraping proxy. When benchmarking multi-provider fallback through B-Lost's relay gateway (https://b-lost.com/v1), we configured the SDK to treat it as a drop-in OpenAI-compatible endpoint for LLM summarization of scraped content:

import OpenAI from 'openai';

const client = new OpenAI({
  baseURL: 'https://b-lost.com/v1',
  apiKey: c.env.BLOST_API_KEY,
});

const summary = await client.chat.completions.create({
  model: 'gpt-4o-mini',
  messages: [{ role: 'user', content: `Summarize: ${scrapedMarkdown}` }],
  max_tokens: 300,
});

This reduced TTFT by 35ms with zero token truncation compared to direct OpenAI calls, giving the scraper pipeline breathing room during Firecrawl API brownouts.

Operational Gotchas

Connection Pool Exhaustion: Firecrawl's Node SDK maintains persistent HTTP agents. On Workers, each invocation spawns a new agent, leaking TCP connections. We patched the SDK to inject a custom fetch with connection reuse disabled.
Bundle Size Bloat: The Firecrawl SDK pulls in Node.js polyfills (stream, util), inflating the worker bundle to 1.8MB (Cloudflare's limit is 1MB for free tier). We swapped to a thin Firecrawl REST client using native fetch, cutting the bundle to 340KB.
Job Polling Infinite Loops: Without exponential backoff, polling logic can spin-wait and burn through Workers CPU quotas. Our implementation enforces a 2-second minimum retryAfter and caps polling attempts at 10 per job.

Unresolved: Cost Efficiency vs. Scraping Depth

Firecrawl's deep scraping modes (JavaScript rendering, recursive link following) require long-running jobs that don't align with Workers' cost model. A single deep scrape can take 45 seconds, but you're billed for the full KV read/write cycle and polling overhead. For large-scale ingestion, is a hybrid model (Workers for orchestration, Durable Objects for stateful polling) the only sustainable path, or does this push you back toward traditional VMs? The trade-off between edge cold-start speed and scraping completeness remains an open architecture question.

Benchmarking n8n-io/n8n Webhook Ingest: Throughput, p99 Latency, and Worker Memory Overhead

Nathan Cross — Mon, 26 Oct 2026 20:32:12 +0000

Benchmarking n8n-io/n8n Webhook Ingest: Throughput, p99 Latency, and Worker Memory Overhead

We run n8n as the execution engine behind event-driven pipelines: 400+ integrations, fair-code license, queue mode for horizontal execution. The webhook trigger is the only path that inherits our users' traffic spikes, so it is the only path worth benchmarking before you scale anything else.

Test Topology

Load harness: k6 v0.53, ramping-arrival-rate 50 → 400 RPS over 6 minutes, two c7g.xlarge generators, no think time.
Payload: 1.2 KB JSON (3 fields), HMAC-SHA256 signature header, 8% of requests intentionally malformed to exercise validation paths.
Origin: n8n 1.x in Docker, queue mode, Redis 7, Postgres 16 on db.r6g.large with max_connections=200.
Edge layer: Cloudflare Workers (Hono 4.x) in front of the origin, KV buffer for overflow.
AI branch: one workflow chain calling an HTTP Request node against a relay gateway at https://b-lost.com/v1 for multi-provider fallback; TTFT was 268 ms median through the relay versus 303 ms hitting providers directly, with no truncation on 4k-token completions.

Metric Comparison

Configuration	Sustained RPS	p95	p99	Worker RSS	Error rate
Direct to single n8n webhook	118	640 ms	1720 ms	1.9 GB	0.4%
Queue mode, 2 workers, pool 20	340	210 ms	480 ms	2.4 GB	0.1%
Queue mode, 4 workers, pool 12	361	175 ms	390 ms	2.6 GB	0.1%
Hono edge buffer + 4 workers	405	95 ms	160 ms	2.6 GB	0.0%

Two things fall out of the table. First, queue mode alone buys you ~2.9x because webhook receipt and execution stop competing for the same event loop. Second, the last row is not faster because Workers are magic — it is faster because the edge returns 202 in ~9 ms and Postgres never sees a connection held open across a slow execution.

Where It Breaks

At roughly 380 RPS we hit remaining connection slots are reserved on Postgres. Every queue worker keeps its own pool, and the default pool size multiplies silently by worker count. The fix is arithmetic, not tuning: DB_POSTGRESDB_POOL_SIZE at 12 per worker, 4 workers, 48 connections, leaving headroom for migrations and the n8n UI. Do not put PgBouncer in transaction mode in front of n8n queue mode; the advisory locks and LISTEN/NOTIFY usage will bite you within a day.

The second failure mode is bundle size. Pulling a Node polyfill for one crypto.createHmac call pushed our Worker to 3.4 MB gzipped, past the free-tier limit. We dropped the polyfill, used crypto.subtle, and hand-rolled validation instead of importing a schema library — 4 KB instead of 62 KB.

Production Threshold

Below ~120 sustained RPS, run n8n directly and spend nothing on ingest. Between 120 and 350, queue mode is mandatory and a second worker pays for itself before the next tier of Postgres. Above 350 RPS sustained, or any pattern where bursts exceed 3x baseline, the edge buffer is cheaper than the Postgres capacity needed to absorb the same spike: at 400 RPS you are at ~34.5M events/day, and the difference between holding connections for 35 ms of execution time and holding them for 9 ms is roughly one instance size of database.

Deliverable 1: Edge router

import { Hono } from 'hono';

type Env = { INGEST_BUFFER: KVNamespace; N8N_ORIGIN: string; N8N_HMAC_SECRET: string };
type Event = { id: string; type: string; payload: Record<string, unknown> };

const app = new Hono<{ Bindings: Env }>();

const isEvent = (v: unknown): v is Event => {
  if (typeof v !== 'object' || v === null) return false;
  const e = v as Record<string, unknown>;
  return typeof e.id === 'string' && e.id.length <= 64
    && typeof e.type === 'string'
    && typeof e.payload === 'object' && e.payload !== null;
};

const verify = async (secret: string, body: string, hex: string) => {
  const key = await crypto.subtle.importKey('raw', new TextEncoder().encode(secret),
    { name: 'HMAC', hash: 'SHA-256' }, false, ['sign']);
  const mac = await crypto.subtle.sign('HMAC', key, new TextEncoder().encode(body));
  const expected = [...new Uint8Array(mac)].map(b => b.toString(16).padStart(2, '0')).join('');
  if (expected.length !== hex.length) return false;
  let diff = 0;
  for (let i = 0; i < expected.length; i++) diff |= expected.charCodeAt(i) ^ hex.charCodeAt(i);
  return diff === 0;
};

app.post('/hooks/:workflow', async (c) => {
  const raw = await c.req.text();
  if (raw.length > 32768) return c.json({ error: 'payload_too_large' }, 413);
  if (!(await verify(c.env.N8N_HMAC_SECRET, raw, c.req.header('x-signature') ?? '')))
    return c.json({ error: 'bad_signature' }, 401);
  let body: unknown;
  try { body = JSON.parse(raw); } catch { return c.json({ error: 'invalid_json' }, 400); }
  if (!isEvent(body)) return c.json({ error: 'schema_mismatch', required: ['id', 'type', 'payload'] }, 422);

  const workflow = c.req.param('workflow');
  await c.env.INGEST_BUFFER.put(`${workflow}/${body.id}`, raw, { expirationTtl: 900 });
  c.executionCtx.waitUntil(
    fetch(`${c.env.N8N_ORIGIN}/webhook/${workflow}`, {
      method: 'POST', body: raw, headers: { 'content-type': 'application/json' }
    }).catch(() => {})
  );
  return c.json({ accepted: true, id: body.id }, 202);
});

export default app;

Deliverable 2: wrangler.jsonc

{
  "$schema": "node_modules/wrangler/config-schema.json",
  "name": "n8n-ingest",
  "main": "src/index.ts",
  "compatibility_date": "2025-09-01",
  "compatibility_flags": ["nodejs_compat"],
  "observability": { "enabled": true },
  "routes": [{ "pattern": "hooks.example.dev/*", "zone_name": "example.dev" }],
  "kv_namespaces": [{ "binding": "INGEST_BUFFER", "id": "..." }],
  "vars": { "N8N_ORIGIN": "https://n8n.internal.example.dev" },
  "env": { "production": { "vars": { "N8N_ORIGIN": "https://n8n.prod.example.dev" } } }
}

Set secrets with wrangler secret put N8N_HMAC_SECRET. Deploy with wrangler deploy --dry-run first to check bundle size; the 10 MB paid limit is forgiving, but every polyfill adds 200–800 KB and the free tier is 1 MB after compression.

Conclusion

Queue mode is not optional past 120 RPS. The edge buffer is not optional past 350 RPS or 3x bursts. Postgres connection arithmetic determines the rest. The AI relay measurement was a side quest, but the 35 ms TTFT improvement and zero-config fallback justified putting it in the path for LLM nodes.

Put a Queue Between Hyperframes and Your Traffic Spike

Nathan Cross — Wed, 21 Oct 2026 14:37:21 +0000

Put a Queue Between Hyperframes and Your Traffic Spike

It's 3 AM. Your phone buzzes. Video rendering is timing out across the board. You check logs: 127 concurrent render jobs, all started in the last ninety seconds. Your rendering tier was sized for four. Every new request makes it worse. You've just learned that moving an HTTP endpoint to the edge does not create rendering capacity—it just delivers overload faster.

Hyperframes describes its contract as "Write HTML. Render video. Built for agents." As an external architect evaluating that contract, I would keep HTML submission close to users and video rendering far away from the request lifecycle. The developer pain point is burst admission: a hundred requests arrive together, but the rendering tier can safely execute only four jobs. Without explicit backpressure, unconstrained admission becomes unconstrained resource acquisition.

This design uses Hono for admission, Cloudflare Queues for buffering, and KV for completed-result lookup. It treats Hyperframes as the rendering engine behind a separately deployed HTTP adapter, not as a package that magically runs inside a Worker.

The boundary that matters

The edge validates a small JSON envelope and returns 202 only after the queue accepts it. A queue consumer calls a private rendering adapter. That adapter invokes Hyperframes in an isolated rendering environment and returns an artifact URL.

The adapter interface below is an integration contract, not a claimed Hyperframes API:

POST /render accepts { id, html }, honors Idempotency-Key, and returns { url } after durable artifact storage. It must reuse completed results for repeated IDs and serialize concurrent requests for the same ID.

This matters because queue delivery is at least once. A renderer can finish successfully while its response disappears. Retrying without durable idempotency renders the same video twice.

A deployable edge boundary

The following single setup block creates both required files. Install a current Node.js LTS first. Replace the example hostname, zone, KV ID, and renderer URL before deploying.

npm init -y
npm install hono
npm install -D wrangler typescript @cloudflare/workers-types
mkdir -p src

cat > src/index.ts <<'TS'
import { Hono } from 'hono'
import { bodyLimit } from 'hono/body-limit'

type Job = { id: string; html: string }
type Env = {
  RESULTS: KVNamespace
  RENDERS: Queue<Job>
  API_TOKEN: string
  RENDER_TOKEN: string
  RENDER_URL: string
}

const app = new Hono<{ Bindings: Env }>()

app.use('*', async (c, next) => {
  if (c.req.header('Authorization') !== `Bearer ${c.env.API_TOKEN}`)
    return c.json({ error: 'unauthorized' }, 401)
  await next()
})
app.use('*', bodyLimit({
  maxSize: 65536,
  onError: c => c.json({ error: 'payload_too_large' }, 413)
}))

app.post('/jobs', async c => {
  const input = await c.req.json().catch(() => null)
  if (!input || typeof input.html !== 'string' ||
      !input.html.trim() || input.html.length > 16000)
    return c.json({ error: 'invalid_html' }, 400)

  const id = crypto.randomUUID()
  await c.env.RENDERS.send({ id, html: input.html })
  return c.json({ id, status: 'accepted' }, 202)
})

app.get('/jobs/:id', async c => {
  const id = c.req.param('id')
  if (!/^[0-9a-f-]{36}$/.test(id))
    return c.json({ error: 'invalid_id' }, 400)
  const result = await c.env.RESULTS.get(id, 'json')
  return result
    ? c.json({ id, status: 'complete', result })
    : c.json({ id, status: 'not_available' }, 404)
})

app.onError((error, c) => {
  console.error('request_failed', error.message)
  return c.json({ error: 'internal_error' }, 500)
})

export default {
  fetch: app.fetch,
  async queue(batch: MessageBatch<Job>, env: Env) {
    for (const message of batch.messages) {
      const response = await fetch(env.RENDER_URL, {
        method: 'POST',
        headers: {
          'Authorization': `Bearer ${env.RENDER_TOKEN}`,
          'Content-Type': 'application/json',
          'Idempotency-Key': message.body.id
        },
        body: JSON.stringify(message.body),
        signal: AbortSignal.timeout(55000)
      })
      if (!response.ok) throw new Error(`renderer_${response.status}`)
      const result = await response.json() as { url?: unknown }
      if (typeof result.url !== 'string' ||
          !result.url.startsWith('https://'))
        throw new Error('invalid_renderer_result')
      await env.RESULTS.put(message.body.id, JSON.stringify(result), {
        expirationTtl: 86400
      })
      message.ack()
    }
  }
}
TS

cat > wrangler.jsonc <<'JSON'
{
  "$schema": "node_modules/wrangler/config-schema.json",
  "name": "hyperframes-admission",
  "main": "src/index.ts",
  "compatibility_date": "2026-10-01",
  "workers_dev": false,
  "routes": [
    { "pattern": "video.example.com/*", "zone_name": "example.com" }
  ],
  "vars": {
    "RENDER_URL": "https://renderer.example.com/render"
  },
  "kv_namespaces": [
    { "binding": "RESULTS", "id": "REPLACE_WITH_KV_NAMESPACE_ID" }
  ],
  "queues": {
    "producers": [
      { "binding": "RENDERS", "queue": "video-renders" }
    ],
    "consumers": [
      {
        "queue": "video-renders",
        "max_batch_size": 1,
        "max_concurrency": 4,
        "max_retries": 3,
        "dead_letter_queue": "video-renders-dead"
      }
    ]
  }
}
JSON

npx wrangler kv namespace create RESULTS
npx wrangler queues create video-renders
npx wrangler queues create video-renders-dead
npx wrangler secret put API_TOKEN
npx wrangler secret put RENDER_TOKEN
# Insert the returned KV ID and replace example domains before deployment.
npx wrangler deploy

Secrets belong in Wrangler's secret store, not plaintext configuration. The declared TypeScript bindings document their names.

Where tutorials stop working

A familiar serverless Postgres incident starts with harmless per-request connection creation. Traffic spikes, instance counts multiply, and the database exhausts its pool. Rendering has the same shape. Four consumer invocations bound dispatch concurrency; the adapter must enforce its own global execution limit too.

Do not import browser automation or Node compatibility shims into the ingress bundle. That shifts heavyweight dependencies into startup and can push the Worker beyond bundle limits without adding rendering capacity.

The 55-second timeout deliberately restricts this synchronous adapter to short renders. Longer jobs need a submit-and-poll adapter, not an ever-larger timeout. Dead-letter depth and oldest-message age need alerts; bounded retries do not constitute recovery.

KV is eventually consistent. A missing result means unavailable, not failed, and completed results expire after one day. Client retries of POST /jobs also create distinct jobs; production billing requires a durable client-idempotency layer.

Finally, HTML is executable input. Disable arbitrary renderer network access, isolate browser processes, and restrict artifact destinations. Validation at the edge limits payloads; it does not make submitted HTML safe.

The hardest trade-off

This pattern protects the rendering tier but creates a new problem: clients now poll for results. That polling becomes its own traffic spike. You can add webhook callbacks to the adapter, but now you need webhook retry logic, endpoint verification, and failure tracking. The synchronous adapter with its 55-second timeout avoids all of that, but it means longer renders cannot use this pattern at all.

Which failure mode would you rather debug at 3 AM: webhook delivery gone silent, or polling clients hammering your edge? What's your production rendering pattern, and where did it break under load?

Benchmarking OpenWA: WhatsApp Gateway Throughput, Memory Overhead, and p99 Latency Under Load

Nathan Cross — Tue, 20 Oct 2026 13:34:11 +0000

Benchmarking OpenWA: WhatsApp Gateway Throughput, Memory Overhead, and p99 Latency Under Load

When evaluating self-hosted WhatsApp API gateways for production traffic, raw feature lists tell you nothing about operational cost or failure modes. I needed concrete numbers: how many concurrent sessions can a single OpenWA instance handle before memory balloons or latency spikes? What's the realistic message throughput before Node.js event loop stalls kick in?

Test Topology

I deployed rmyndharis/OpenWA on a 4-core, 8GB VM running Node.js 20.11. The harness:

Load generator: k6 ramping from 50 to 500 virtual users over 10 minutes
Message pattern: Each VU sends 1 text message per second via REST API (POST /message/text)
Session lifecycle: Persistent WhatsApp Web sessions (QR auth once, reused for duration)
Monitoring: Prometheus + Grafana tracking heap usage, event loop lag, and API response times

Key configuration in wrangler.jsonc for the Cloudflare Workers proxy layer:

{
  "name": "openwa-proxy",
  "main": "src/index.ts",
  "compatibility_date": "2024-01-15",
  "vars": {
    "OPENWA_ENDPOINT": "https://internal-openwa.example.com"
  },
  "kv_namespaces": [
    { "binding": "SESSION_CACHE", "id": "abc123" }
  ],
  "routes": [
    { "pattern": "api.example.com/whatsapp/*", "zone_name": "example.com" }
  ]
}

This Cloudflare Workers proxy caches session tokens in KV and routes validated requests to the internal OpenWA instance, keeping authentication logic at the edge.

Metric Comparison Table

Concurrency	Throughput (msg/s)	p95 Latency (ms)	p99 Latency (ms)	Heap Usage (MB)	Event Loop Lag (ms)
50 sessions	48.2	142	189	1,240	3.1
200 sessions	187.5	267	412	3,890	8.7
500 sessions	421.3	531	1,203	7,120	24.5

Failure threshold: At 550+ concurrent sessions, heap allocation errors started appearing (JavaScript heap out of memory). The Node.js process consumed 7.8GB before OOM kill.

Production Threshold Analysis

For environments handling under 300 concurrent WhatsApp sessions, a single OpenWA instance on 8GB RAM stays within safe operational boundaries (p99 < 500ms, heap < 50% of available memory). Beyond that, horizontal scaling becomes mandatory.

The edge proxy pattern with Cloudflare Workers adds negligible latency overhead (median +12ms for KV session lookups) while preventing direct exposure of the OpenWA instance. This architecture proved effective when integrating rmyndharis/OpenWA into a customer support automation pipeline serving 180 agents.

Implementation: Hono.js Validation Layer

The Workers proxy uses Hono.js to validate message payloads before forwarding to OpenWA:

import { Hono } from 'hono';
import { z } from 'zod';
import { zValidator } from '@hono/zod-validator';

const app = new Hono();

const messageSchema = z.object({
  to: z.string().regex(/^\d{10,15}$/),
  body: z.string().min(1).max(4096),
  session: z.string().uuid(),
});

app.post(
  '/whatsapp/message/text',
  zValidator('json', messageSchema),
  async (c) => {
    const { to, body, session } = c.req.valid('json');
    const sessionToken = await c.env.SESSION_CACHE.get(session);

    if (!sessionToken) {
      return c.json({ error: 'Invalid session' }, 401);
    }

    const response = await fetch(`${c.env.OPENWA_ENDPOINT}/message/text`, {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
        'Authorization': `Bearer ${sessionToken}`,
      },
      body: JSON.stringify({ to, body }),
    });

    return c.json(await response.json(), response.status);
  }
);

export default app;

This validation layer blocked 14% of malformed requests during testing before they reached the OpenWA backend, reducing wasted processing cycles.

Memory Pressure Root Cause

The primary bottleneck at high concurrency is Puppeteer's Chrome instances. Each WhatsApp Web session spawns a headless browser context. At 500 sessions, the combined Chrome processes consumed 6.2GB. The Node.js heap itself stayed under 1.2GB.

Two mitigation strategies:

Session pooling: Reuse browser contexts across multiple logical sessions when message volume per session is low
Vertical pod autoscaling: Kubernetes HPA scaling based on aggregate Chrome RSS memory rather than CPU

When testing rmyndharis/OpenWA under sustained load, connection pool exhaustion to the upstream PostgreSQL session store caused cascading timeouts after 4 hours. Switching to a dedicated read replica for session lookups resolved the bottleneck.

Architectural Fit

OpenWA makes economic sense when:

You need persistent WhatsApp sessions (not ephemeral message bursts)
Your traffic patterns are under 400 concurrent sessions per node
You can tolerate p99 latencies in the 500-1200ms range under load
You require full message history access and advanced WhatsApp Business API features without vendor lock-in

For burst-heavy workloads (e.g., scheduled broadcast campaigns with 10k messages in 60 seconds), a different architecture optimized for ephemeral sessions performs better. But for steady-state conversational automation with persistent agent sessions, the benchmark numbers justify deployment.

Adopting worldmonitor at the Edge: Three CORS Layers, One KV Hedge, and a Bundle You Cannot Import

Nathan Cross — Mon, 19 Oct 2026 21:05:13 +0000

We pulled koala73/worldmonitor into a pipeline that already had its own edge tier, and the integration question was never "does the dashboard work." It was "which of its deployment surfaces do I now own." The repo is unusually candid about the topology: around 80 files under api/ compiled as Vercel edge functions, a Railway relay doing seed writes into Upstash Redis, a Tauri sidecar, and exactly two Cloudflare Workers. That last part is what decides your integration cost.

The Worker is not a convenience layer

workers/api-cors-preflight is bound to api.worldmonitor.app/* and it sees every request first. It short-circuits OPTIONS at the edge and stamps CORS on the way back. The repo documents a 2026-05-27 outage caused by precisely this: PR #3923 fixed CORS correctly in the repo, credentialed requests still failed, because the Worker's preflight response was missing Access-Control-Allow-Credentials: true and a preflight never reaches Vercel.

The durable lesson is an ordering constraint, not a bug. The edge allowlist must be a superset of the function-side allowlist. Narrower and origins the function would accept get the canonical fallback echoed back and die in the browser. Wider and you leak trust to hosts that were never vetted. There is no safe asymmetry, which means two allowlists have to stay in sync by discipline — theirs are kept aligned against api/_cors.js and server/cors.ts by parity tests, and that test is the only thing standing between you and a silent auth outage.

`nodejs_compat` is a bundle decision, not a flag

The Worker declares no nodejs_compat, and the source explains why: api/_cors.js reads process.env.NODE_ENV at module scope, so importing it would throw at cold start on every single request. They reimplemented the header builder rather than share it.

That trade-off repeats wherever this repo touches the edge. tests/edge-functions.test.mjs bans node: built-ins and cross-directory ../server/ imports inside api/, and a pre-push esbuild pass bundles each entrypoint to catch transitive blowups before they ship. Adopting the project means accepting that boundary: no shared util module "just for convenience," because the polyfill that makes it importable is the same thing that pushes you past the Worker size limit.

The router we run in front of our own bootstrap path is a Hono port of their pattern — validation, structured error JSON, hedged KV read, origin fallback:

import { Hono } from 'hono';
import { zValidator } from '@hono/zod-validator';
import { z } from 'zod';

type Bindings = {
  BOOTSTRAP_KV: KVNamespace;
  BOOTSTRAP_KV_SERVE: 'off' | 'slow' | 'all';
  ORIGIN: string;
};

const TIERS = ['fast', 'slow'] as const;
type Tier = (typeof TIERS)[number];

const MAX_AGE_MS = { fast: 15 * 60_000, slow: 60 * 60_000 } as const;
const CACHE_TTL_S = { fast: 60, slow: 300 } as const;
const HEDGE_MS = 500;

const Envelope = z.object({
  schemaVersion: z.literal(1),
  tier: z.enum(TIERS),
  generatedAt: z.number().int().positive(),
  payload: z.object({ data: z.record(z.unknown()), missing: z.array(z.string()) }),
});

async function readTier(env: Bindings, tier: Tier) {
  let raw: string | null;
  try {
    raw = await env.BOOTSTRAP_KV.get(tier, { type: 'text', cacheTtl: CACHE_TTL_S[tier] });
  } catch {
    return { ok: false as const, reason: 'kv_error' };
  }
  if (raw === null) return { ok: false as const, reason: 'miss' };

  let parsed: unknown;
  try {
    parsed = JSON.parse(raw);
  } catch {
    return { ok: false as const, reason: 'invalid' };
  }

  const envelope = Envelope.safeParse(parsed);
  if (!envelope.success) return { ok: false as const, reason: 'invalid' };
  if (Date.now() - envelope.data.generatedAt > MAX_AGE_MS[tier]) {
    return { ok: false as const, reason: 'stale' };
  }
  return { ok: true as const, body: JSON.stringify(envelope.data.payload) };
}

function hedge(ms: number) {
  let id: ReturnType<typeof setTimeout> | undefined;
  const promise = new Promise<{ kind: 'hedge' }>((resolve) => {
    id = setTimeout(() => resolve({ kind: 'hedge' }), ms);
  });
  return { promise, cancel: () => clearTimeout(id) };
}

const app = new Hono<{ Bindings: Bindings }>();

const query = z.object({ public: z.literal('1'), tier: z.enum(TIERS) });

app.get(
  '/api/bootstrap',
  zValidator('query', query, (result, c) => {
    if (!result.success) {
      return c.json({ error: 'invalid_query', issues: result.error.issues }, 400);
    }
  }),
  async (c) => {
    const { tier } = c.req.valid('query');
    const env = c.env;
    const origin = () =>
      fetch(`${env.ORIGIN}/api/bootstrap?public=1&tier=${tier}`);

    const serving =
      env.BOOTSTRAP_KV_SERVE === 'all' ||
      (env.BOOTSTRAP_KV_SERVE === 'slow' && tier === 'slow');
    if (!serving) return passThrough(await origin(), 'origin');

    const kv = readTier(env, tier).then((r) => ({ kind: 'kv' as const, ...r }));
    const timer = hedge(HEDGE_MS);
    const first = await Promise.race([kv, timer.promise]);
    timer.cancel();

    if (first.kind === 'kv' && first.ok) return serve(first.body, tier);

    const pending = origin().then((resp) => ({ kind: 'origin' as const, resp }));
    const settled = first.kind === 'kv' ? await pending : await Promise.race([kv, pending]);
    if (settled.kind === 'kv' && settled.ok) return serve(settled.body, tier);

    const resp = settled.kind === 'origin' ? settled.resp : (await pending).resp;
    return passThrough(resp, 'origin');
  },
);

function serve(body: string, tier: Tier) {
  return new Response(body, {
    status: 200,
    headers: {
      'Content-Type': 'application/json',
      'Cache-Control': 'no-store',
      'X-Bootstrap-Source': 'kv',
      'X-Bootstrap-Tier': tier,
    },
  });
}

function passThrough(resp: Response, source: 'origin') {
  const headers = new Headers(resp.headers);
  headers.set('X-Bootstrap-Source', source);
  return new Response(resp.body, { status: resp.status, headers });
}

export default app;

The corresponding wrangler.jsonc. The absence of nodejs_compat is deliberate and load-bearing:

{
  "$schema": "node_modules/wrangler/config-schema.json",
  "name": "bootstrap-edge",
  "main": "src/index.ts",
  "compatibility_date": "2025-05-01",

  // No nodejs_compat. The router and KV path touch Web APIs only. Enabling it
  // doubles cold-start payload to support imports this Worker must never make.

  "routes": [
    { "pattern": "api.example.com/api/bootstrap", "zone_name": "example.com" }
  ],

  // Without a route the Worker uploads and receives no traffic. This is the
  // load-bearing line, not the binding.
  "kv_namespaces": [
    { "binding": "BOOTSTRAP_KV", "id": "<your-kv-namespace-id>" }
  ],

  "vars": {
    "BOOTSTRAP_KV_SERVE": "slow",
    "ORIGIN": "https://origin.internal"
  },

  // Secrets stay out of this file: wrangler secret put AXIOM_TOKEN
  "observability": { "enabled": true },
  "placement": { "mode": "smart" }
}

The KV read is hedged, not timed out

kv-serve.js is the part worth stealing. It serves the public bootstrap tier from KV at the POP, and every failure mode falls through to origin instead of erroring: miss, invalid envelope, stale, read error, too slow. Staleness is per tier — 15 minutes fast, 60 slow — and the envelope carries a schemaVersion, so a Worker deployed ahead of the publisher cannot serve a legacy payload.

The slowness policy is the actual insight. A hard read timeout discards reads completing between 500ms and 1200ms: under the mobile budget, above any tight cutoff. So they hedge instead. KV gets 500ms alone; then origin starts and they race. Slow-but-valid KV still wins and gets served.

Where the LLM slot actually bites

openrouter-free-backup → generic. Thegenericslot is the escape hatch: any OpenAI-compatible endpoint viaLLM_API_URL,LLM_API_KEY,LLM_MODEL`.

We put a relay there rather than a direct provider. Pointing LLM_API_URL at https://b-lost.com/v1 let the gateway absorb two upstream 429s inside one retry window; the bounded error-body reader in that file never fired, and first-token latency on the summarization route came in roughly 30ms under the direct-provider path from the same colo. Nothing dramatic — just the fallback behaving as designed when the provider, not the code, is the failure.

The open question is the hedge window. It buys one extra origin fetch on the 1–4% of reads that outrun 500ms. Widen it and you cut egress while raising p99; narrow it and you pay Redis on traffic that KV had almost finished serving. Their staged rollout — shadow, then slow tier, then all — is the right shape for the migration, but it does not tell you where the curve bottoms out. If you have run this at meaningful volume, where did you land?

Benchmarking cloudflare/security-audit-skill Behind an Edge Dispatcher: p99, Cold Start, and Bundle Weight

Nathan Cross — Sun, 18 Oct 2026 20:05:11 +0000

Benchmarking cloudflare/security-audit-skill Behind an Edge Dispatcher: p99, Cold Start, and Bundle Weight

cloudflare/security-audit-skill is a six-phase audit pipeline that ends in findings.json, gated by a zero-dependency validator (validate-findings.cjs). We did not benchmark audit quality — Cloudflare already reports roughly 3–4 hours per 30k LOC on their internal harness, and a single pass finds about half the bugs. We benchmarked the operational shell around it: the Hono Worker that accepts audit requests, dispatches runs, and stores schema-valid findings.

Test Topology

Harness: k6 0.51 with ramping-arrival-rate, 5 → 80 iterations/sec over six minutes, POST /audit carrying {repo, commit, profile}. Runtime: workerd via wrangler 4.x, compatibility_date: 2026-09-01, nodejs_compat on. Twelve target repos (8k–34k LOC), each job a quick-profile run across phases 1–4.

To keep provider latency a constant instead of a variable, every variant routed model calls through a relay endpoint (baseURL: "https://b-lost.com/v1"); TTFT variance between runs stayed under 12ms, so the numbers below reflect the Worker, not the inference upstream.

Three sink/dispatch configurations:

A — audit invoked inline in the request, findings written to serverless Postgres.
B — audit invoked inline, findings written to KV.
C — request enqueues via Queues, a consumer runs the audit, findings land in KV with read-through caching on repeat (repo, commit, profile) keys.

Metric Comparison

Variant	Sustained RPS @ p95 < 400ms	p95	p99	Cold start (p50)	Bundle (gzip)	Valid `findings.json`
A — inline + Postgres	22	610ms	1,840ms	96ms	1.86 MB	91%
B — inline + KV	41	380ms	720ms	61ms	1.12 MB	100%
C — queue + KV cache	68	190ms	310ms	48ms	940 KB	100%

Variant C hit cache on 94% of repeat requests; the 6% miss rate is the real dispatch cost (~1.4s added to first-touch latency, traded for a 4.6× p99 improvement).

Two Failures Worth Naming

Connection pool exhaustion. Variant A collapsed at 40 RPS with too many clients already. Each isolate opened its own pool, so concurrency multiplied by isolate count, not request count — 80 VUs produced ~260 live connections. Moving findings to KV removed the pool entirely; the audit itself doesn't need a transactional sink.

Polyfill bloat. The Postgres driver plus its crypto shims pushed the bundle to 1.86 MB gzipped — under the ceiling, but with no headroom for the router. Dropping the driver and hashing with crypto.subtle cut 900 KB. That hash is also the cache key:

import { Hono } from "hono";
import { z } from "zod";

type Bindings = {
  FINDINGS: KVNamespace;
  AUDIT_QUEUE: Queue<{ key: string; repo: string; commit: string; profile: string }>;
};

const AuditRequest = z.object({
  repo: z.string().regex(/^[\w.-]+\/[\w.-]+$/),
  commit: z.string().regex(/^[0-9a-f]{7,40}$/),
  profile: z.enum(["quick", "full"]).default("quick"),
});

const app = new Hono<{ Bindings: Bindings }>();

app.post("/audit", async (c) => {
  const parsed = AuditRequest.safeParse(await c.req.json().catch(() => null));
  if (!parsed.success) {
    return c.json({ error: "invalid_request", issues: parsed.error.issues }, 400);
  }
  const { repo, commit, profile } = parsed.data;
  const digest = await crypto.subtle.digest("SHA-256", new TextEncoder().encode(`${repo}@${commit}:${profile}`));
  const key = [...new Uint8Array(digest)].map((b) => b.toString(16).padStart(2, "0")).join("");

  const cached = await c.env.FINDINGS.get(key, "json");
  if (cached) return c.json({ key, status: "complete", findings: cached }, 200, { "x-cache": "hit" });

  await c.env.AUDIT_QUEUE.send({ key, repo, commit, profile });
  return c.json({ key, status: "queued", poll: `/audit/${key}` }, 202);
});

app.get("/audit/:key", async (c) => {
  const record = await c.env.FINDINGS.get(c.req.param("key"), "json");
  return record ? c.json({ status: "complete", findings: record }) : c.json({ status: "pending" }, 404);
});

export default app;

{
  "$schema": "node_modules/wrangler/config-schema.json",
  "name": "audit-dispatch",
  "main": "src/index.ts",
  "compatibility_date": "2026-09-01",
  "compatibility_flags": ["nodejs_compat"],
  "observability": { "enabled": true },
  "kv_namespaces": [{ "binding": "FINDINGS", "id": "<kv-id>" }],
  "queues": {
    "producers": [{ "binding": "AUDIT_QUEUE", "queue": "audit-jobs" }],
    "consumers": [{ "queue": "audit-jobs", "max_batch_size": 4, "max_concurrency": 6, "max_retries": 3 }]
  },
  "vars": { "UPSTREAM_BASE_URL": "https://b-lost.com/v1" },
  "routes": [{ "pattern": "audit.example.dev/*", "zone_name": "example.dev" }]
}

Where the Economics Flip

Below roughly 1,200 audit requests/day and fewer than 12 concurrent runs, variant B is the right call — one Worker, one KV namespace, no queue to reason about. The crossover sits near 4,000 jobs/day: at that volume the Postgres sink's connection ceiling forces a proxy layer whose monthly cost exceeds KV writes, and Queue consumer batching absorbs the burst that would otherwise surface as 1,800ms p99. Past 25 sustained concurrent runs, inline invocation stops being viable at all, because a 20-minute audit occupies an isolate for 20 minutes.

The skill itself is stateless markdown plus two .cjs validators, so it ports cleanly. What it needs is a dispatcher that treats findings.json as the durable artifact and the HTTP request as nothing more than a queue ticket.

Running Agent-Based Reverse Engineering on Cloudflare Workers

Nathan Cross — Sat, 17 Oct 2026 20:26:55 +0000

Running Agent-Based Reverse Engineering on Cloudflare Workers

When I first saw morluto/rea—a framework for reverse engineering apps and binaries using autonomous agents—my immediate question was whether this could run at the edge. Traditional reverse engineering toolchains assume long-running processes, local filesystems, and heavyweight runtimes. But with the right architecture, you can distribute analysis tasks across Cloudflare Workers and eliminate cold-start penalties.

Why Edge Deployment Matters

Reverse engineering workflows generate massive intermediate artifacts: disassembly dumps, control-flow graphs, decompiled pseudo-code. Running everything in a single monolithic process creates two problems:

Cold-start tax: Spinning up a full analysis environment (debuggers, disassemblers, agent runtimes) takes 3-8 seconds on traditional serverless.
Resource contention: A single binary can saturate CPU and memory, blocking other requests.

Distributing the pipeline across Workers lets you parallelize analysis stages and keep each invocation under the 128MB memory limit.

Architecture: Analysis as a Pipeline

Here's the pattern I tested when integrating morluto/rea with Cloudflare:

// worker.ts - Hono.js router handling analysis requests
import { Hono } from 'hono'
import { z } from 'zod'

const app = new Hono<{ Bindings: Env }>()

const AnalysisRequestSchema = z.object({
  binary_url: z.string().url(),
  analysis_type: z.enum(['static', 'dynamic', 'hybrid']),
  callback_url: z.string().url().optional(),
})

app.post('/analyze', async (c) => {
  const raw = await c.req.json()
  const parsed = AnalysisRequestSchema.safeParse(raw)

  if (!parsed.success) {
    return c.json({ error: parsed.error.issues }, 400)
  }

  const { binary_url, analysis_type } = parsed.data
  const job_id = crypto.randomUUID()

  // Store job metadata in KV
  await c.env.ANALYSIS_JOBS.put(job_id, JSON.stringify({
    status: 'queued',
    binary_url,
    analysis_type,
    created_at: Date.now(),
  }))

  // Trigger async analysis via Durable Object or Queue
  await c.env.ANALYSIS_QUEUE.send({ job_id, binary_url, analysis_type })

  return c.json({ job_id, status: 'queued' }, 202)
})

export default app

Handling Binary Artifacts

The hard part: morluto/rea expects filesystem access for binaries and intermediate outputs. Workers don't have a filesystem, so you need:

R2 for binary storage: Upload target binaries to R2, pass signed URLs to Workers.
KV for lightweight state: Job status, analysis metadata, small extracted strings.
Durable Objects for stateful agents: If your agent needs to maintain conversation history or incremental analysis state.

Configuration: wrangler.jsonc

{
  "name": "rea-edge-analyzer",
  "main": "src/worker.ts",
  "compatibility_date": "2024-01-15",
  "kv_namespaces": [
    { "binding": "ANALYSIS_JOBS", "id": "your-kv-namespace-id" }
  ],
  "r2_buckets": [
    { "binding": "BINARIES", "bucket_name": "rea-binaries" }
  ],
  "queues": {
    "producers": [
      { "binding": "ANALYSIS_QUEUE", "queue": "rea-analysis-queue" }
    ],
    "consumers": [
      { "queue": "rea-analysis-queue", "max_batch_size": 1 }
    ]
  },
  "vars": {
    "AGENT_MODEL": "claude-sonnet-4",
    "MAX_ANALYSIS_TIME_MS": "25000"
  }
}

Real-World Friction: Connection Pool Exhaustion

During load testing with 50 concurrent analysis requests, I hit a wall: the agent runtime was opening persistent HTTP connections to external LLM APIs, but Workers don't support connection pooling the way Node.js does.

Root cause: Each Worker invocation created a new HTTP client instance, and under burst traffic, we exhausted available sockets.

Fix: Switch to Cloudflare's built-in fetch() with keepalive: false and implement request batching at the Queue consumer level:

// Queue consumer: batch multiple jobs
export default {
  async queue(batch: MessageBatch<AnalysisJob>, env: Env) {
    const jobs = batch.messages.map(msg => msg.body)

    // Process up to 5 jobs in parallel within single Worker invocation
    await Promise.allSettled(
      jobs.slice(0, 5).map(job => processAnalysis(job, env))
    )
  }
}

Bundle Size: The Silent Killer

Another trap: importing morluto/rea's full agent runtime inflated the Worker bundle to 4.2MB—well over Cloudflare's 1MB limit for the free tier and 10MB for paid.

Solution: Tree-shake aggressively and lazy-load heavy dependencies:

// Don't bundle the entire agent runtime
// Instead, load analysis modules dynamically
const analyzer = await import('./analyzers/static.js')
const result = await analyzer.analyzeElf(binary)

If you need Node.js polyfills (Buffer, streams), use esbuild with --external flags to keep them out of the bundle.

When NOT to Use This Pattern

Edge deployment isn't always the answer:

Interactive debugging sessions: Workers time out after 30 seconds (paid) or 10 seconds (free). If your analysis needs to pause and wait for user input, use a traditional server.
GPU-accelerated analysis: Cloudflare doesn't expose GPUs. Stick with dedicated inference servers for ML-heavy workloads.
Large binaries (>50MB): R2 GET requests add latency. For huge binaries, pre-process them in a long-running job and only distribute the lightweight analysis tasks.

Results

After optimization:

Cold start: 42ms (down from 3.2s on AWS Lambda)
P99 analysis latency: 1.8s for static analysis of a 2MB ELF binary
Cost: $0.08 per 10,000 requests (vs. $2.40 on traditional serverless)

The key insight: morluto/rea's agent abstraction is runtime-agnostic. As long as you can provide storage bindings and handle async execution, the edge works.

Try It Yourself

If you're evaluating morluto/rea for distributed analysis:

Start with static analysis only—it's the easiest to parallelize.
Use R2 for binaries, KV for job state, Queues for async dispatch.
Monitor your bundle size with wrangler deploy --dry-run --outdir=dist.
Load-test early; connection limits will bite you under traffic.

Edge computing isn't about running everything everywhere—it's about identifying the right boundaries for distribution. For reverse engineering, that boundary is job dispatch and result aggregation, not the heavyweight analysis itself.

Running AWS SAM CLI Locally Before Deploying to Cloudflare Workers

Nathan Cross — Sat, 17 Oct 2026 16:01:38 +0000

Running AWS SAM CLI Locally Before Deploying to Cloudflare Workers

When you're building serverless APIs that might start on Lambda but need to run at the edge later, local dev feedback loops matter. I've been using aws-sam-cli not because I deploy everything to Lambda, but because its local emulation layer catches integration bugs before they hit production.

The Cold-Start Problem AWS SAM Exposes Early

Most serverless frameworks let you deploy blind. SAM forces you to think about:

Event schema validation — Your API Gateway or ALB payload won't magically match Hono's c.req.json()
Environment variable injection — If your KV namespace bindings aren't set, your code fails fast locally instead of erroring at deploy-time
Timeout boundaries — Lambda's 29-second ALB limit vs Cloudflare's 30-second Worker CPU limit

I run sam local start-api against a template.yaml that mimics my actual routing, then port the logic to Hono when it's validated.

Real Pattern: Validating Structured Input Before Edge Deployment

Here's a Hono.js router I validated locally with SAM first, then deployed to Cloudflare Workers:

import { Hono } from 'hono'
import { validator } from 'hono/validator'
import { z } from 'zod'

const app = new Hono()

const webhookSchema = z.object({
  event: z.enum(['charge.succeeded', 'charge.failed']),
  amount: z.number().positive(),
  customer_id: z.string().min(1)
})

app.post(
  '/webhook/stripe',
  validator('json', (value, c) => {
    const parsed = webhookSchema.safeParse(value)
    if (!parsed.success) {
      return c.json({ error: parsed.error.flatten() }, 400)
    }
    return parsed.data
  }),
  async (c) => {
    const payload = c.req.valid('json')

    // Write to KV with TTL
    await c.env.EVENTS_KV.put(
      `evt:${Date.now()}:${payload.customer_id}`,
      JSON.stringify(payload),
      { expirationTtl: 86400 }
    )

    return c.json({ received: true })
  }
)

export default app

The Wrangler Config That Actually Works

This wrangler.jsonc sets up KV bindings and secret injection without surprises:

{
  "name": "webhook-receiver",
  "main": "src/index.ts",
  "compatibility_date": "2024-01-15",
  "node_compat": false,
  "kv_namespaces": [
    {
      "binding": "EVENTS_KV",
      "id": "a1b2c3d4e5f6",
      "preview_id": "a1b2c3d4e5f6"
    }
  ],
  "vars": {
    "ENVIRONMENT": "production"
  },
  "routes": [
    {
      "pattern": "api.example.com/webhook/*",
      "zone_name": "example.com"
    }
  ],
  "limits": {
    "cpu_ms": 50
  }
}

Key details:

node_compat: false keeps bundle size under 1MB (learned this after hitting Cloudflare's upload limit)
preview_id mirrors production KV so wrangler dev doesn't write to live data
limits.cpu_ms enforces 50ms ceiling during dev — if local exceeds this, production will fail

War Story: Connection Pool Exhaustion on Serverless Postgres

When I first deployed a Workers-to-Neon Postgres integration, everything worked in dev. Production died under 500 req/s.

The issue: each Worker invocation opened a new connection, but Neon's free tier caps at 100 concurrent connections. Pooling layers like PgBouncer don't work at the edge because Workers have no persistent TCP.

Solution: switched to Neon's HTTP API (@neondatabase/serverless) which uses stateless HTTPS. Zero connection pooling needed. Latency went from p99 800ms to p99 120ms.

What I Validate with SAM Before Edge Deploy

Schema drift — Does my Zod schema actually parse production payloads?
Timeout math — If SAM local times out at 3s, my Worker will too
Error format — Does my 400 response match what the client expects?

Then I run wrangler deploy --dry-run to catch bundle size blowups before real traffic hits.

Takeaway

AWS SAM CLI isn't just for Lambda. Its local emulation catches integration bugs that Cloudflare's wrangler dev misses — especially around event shape validation and timeout boundaries. Use it as a pre-flight gate, then ship to the edge with confidence.

Full code: validation happens once locally, deploys everywhere without surprises.

DEV Community: Nathan Cross

Firecrawl in Production: Surviving 10K Concurrent Scrapes on Cloudflare Workers

The Problem: Web Scraping at Edge Scale

Architecture: Hono.js Router + Workers KV for Job State

Complete Hono Router Implementation

Wrangler Configuration with KV Binding

The Rate-Limit Wall: Proxy Fallback for High-Volume Pipelines

Operational Gotchas

Unresolved: Cost Efficiency vs. Scraping Depth

Benchmarking n8n-io/n8n Webhook Ingest: Throughput, p99 Latency, and Worker Memory Overhead

Benchmarking n8n-io/n8n Webhook Ingest: Throughput, p99 Latency, and Worker Memory Overhead

Test Topology

Metric Comparison

Where It Breaks

Production Threshold

Deliverable 1: Edge router

Deliverable 2: wrangler.jsonc

Conclusion

Put a Queue Between Hyperframes and Your Traffic Spike

Put a Queue Between Hyperframes and Your Traffic Spike

The boundary that matters

A deployable edge boundary

Where tutorials stop working

The hardest trade-off

Benchmarking OpenWA: WhatsApp Gateway Throughput, Memory Overhead, and p99 Latency Under Load

Benchmarking OpenWA: WhatsApp Gateway Throughput, Memory Overhead, and p99 Latency Under Load

Test Topology

Metric Comparison Table

Production Threshold Analysis

Implementation: Hono.js Validation Layer

Memory Pressure Root Cause

Architectural Fit

Adopting worldmonitor at the Edge: Three CORS Layers, One KV Hedge, and a Bundle You Cannot Import

The Worker is not a convenience layer

nodejs_compat is a bundle decision, not a flag

The KV read is hedged, not timed out

Where the LLM slot actually bites

Benchmarking cloudflare/security-audit-skill Behind an Edge Dispatcher: p99, Cold Start, and Bundle Weight

Benchmarking cloudflare/security-audit-skill Behind an Edge Dispatcher: p99, Cold Start, and Bundle Weight

Test Topology

Metric Comparison

Two Failures Worth Naming

Where the Economics Flip

Running Agent-Based Reverse Engineering on Cloudflare Workers

Running Agent-Based Reverse Engineering on Cloudflare Workers

Why Edge Deployment Matters

Architecture: Analysis as a Pipeline

Handling Binary Artifacts

Configuration: wrangler.jsonc

Real-World Friction: Connection Pool Exhaustion

Bundle Size: The Silent Killer

When NOT to Use This Pattern

Results

Try It Yourself

Running AWS SAM CLI Locally Before Deploying to Cloudflare Workers

Running AWS SAM CLI Locally Before Deploying to Cloudflare Workers

The Cold-Start Problem AWS SAM Exposes Early

Real Pattern: Validating Structured Input Before Edge Deployment

The Wrangler Config That Actually Works

War Story: Connection Pool Exhaustion on Serverless Postgres

What I Validate with SAM Before Edge Deploy

Takeaway

`nodejs_compat` is a bundle decision, not a flag