<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Mary Olowu</title>
    <description>The latest articles on DEV Community by Mary Olowu (@itsmarydan).</description>
    <link>https://dev.to/itsmarydan</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F716762%2F54914207-db14-4bf3-b944-8704ebab0094.jpeg</url>
      <title>DEV Community: Mary Olowu</title>
      <link>https://dev.to/itsmarydan</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/itsmarydan"/>
    <language>en</language>
    <item>
      <title>How to Test Stripe Webhooks Locally (Stripe CLI + Replay + Logs)</title>
      <dc:creator>Mary Olowu</dc:creator>
      <pubDate>Wed, 22 Apr 2026 07:43:19 +0000</pubDate>
      <link>https://dev.to/itsmarydan/how-to-test-stripe-webhooks-locally-stripe-cli-replay-logs-7of</link>
      <guid>https://dev.to/itsmarydan/how-to-test-stripe-webhooks-locally-stripe-cli-replay-logs-7of</guid>
      <description>&lt;p&gt;Most Stripe webhook bugs are not business logic bugs. They are reproducibility bugs. If you can't replay the exact event path in under 30 seconds, debugging takes hours instead of minutes — because every attempt involves refreshing the Stripe Dashboard, re-triggering a test, and squinting at logs to figure out whether your handler ran at all.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F78bmt5bsgqz44xdwmkqg.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F78bmt5bsgqz44xdwmkqg.png" alt=" " width="800" height="533"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;This post walks through the local loop that makes this painless: &lt;strong&gt;forward → trigger → replay → inspect.&lt;/strong&gt; Four steps, one terminal window, no guessing.&lt;/p&gt;

&lt;h2&gt;
  
  
  The loop, in full
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Terminal 1 — forward Stripe events to your local handler&lt;/span&gt;
stripe listen &lt;span class="nt"&gt;--forward-to&lt;/span&gt; &lt;span class="s2"&gt;"http://localhost:3000/api/stripe-webhook"&lt;/span&gt;

&lt;span class="c"&gt;# Terminal 2 — trigger real Stripe test events&lt;/span&gt;
stripe trigger payment_intent.succeeded
stripe trigger charge.failed
stripe trigger invoice.payment_failed

&lt;span class="c"&gt;# When something fails, replay the exact event&lt;/span&gt;
stripe events resend evt_1NfP6Q2eZvKYlo2CsKT4a5oS
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That is the whole loop. Everything below is how to make it reliable.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 1 — Give your dev webhook its own path
&lt;/h2&gt;

&lt;p&gt;Don't forward Stripe events to your production webhook path. Use a dedicated dev path:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;stripe listen &lt;span class="nt"&gt;--forward-to&lt;/span&gt; &lt;span class="s2"&gt;"http://localhost:3000/api/stripe-webhook-dev"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Why separate? Because you want to test with lax validation (no signature verification, verbose logging) without loosening your production handler. Keep two paths:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;/api/stripe-webhook&lt;/code&gt; — production. Strict signature verification, minimal logs.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;/api/stripe-webhook-dev&lt;/code&gt; — dev only. Signature check optional, logs every field.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;When &lt;code&gt;stripe listen&lt;/code&gt; starts, it prints a signing secret:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;&amp;gt; Ready! Your webhook signing secret is whsec_abc123...
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Copy that into your dev environment variable (&lt;code&gt;STRIPE_WEBHOOK_SECRET_DEV&lt;/code&gt;). This is NOT the same as the secret from your Stripe Dashboard — the CLI generates its own. This trips up everyone the first time.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 2 — Trigger real events, not made-up payloads
&lt;/h2&gt;

&lt;p&gt;Do not hand-craft JSON payloads. Stripe's &lt;code&gt;trigger&lt;/code&gt; command sends a real event through your account's test data, which means you're testing against payloads that match production exactly.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;# Core payment flows&lt;/span&gt;
stripe trigger payment_intent.succeeded
stripe trigger charge.succeeded
stripe trigger charge.failed

&lt;span class="c"&gt;# Subscription lifecycle&lt;/span&gt;
stripe trigger customer.subscription.created
stripe trigger customer.subscription.updated
stripe trigger customer.subscription.deleted

&lt;span class="c"&gt;# Invoice / billing&lt;/span&gt;
stripe trigger invoice.payment_succeeded
stripe trigger invoice.payment_failed
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Each trigger writes a real event to your Stripe test account. That means every event has a real &lt;code&gt;id&lt;/code&gt; (starts with &lt;code&gt;evt_&lt;/code&gt;) that you can replay later.&lt;/p&gt;

&lt;p&gt;Run all the events your handler cares about at least once. Anything you haven't triggered locally is a surprise waiting in production.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 3 — Replay the exact event, on demand
&lt;/h2&gt;

&lt;p&gt;This is the step most people skip, and it's where debugging speed compounds. When a test fails, you can replay the exact same event by ID:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;stripe events resend evt_1NfP6Q2eZvKYlo2CsKT4a5oS
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Same payload. Same signature. Same timestamp. Your handler sees a byte-for-byte identical request.&lt;/p&gt;

&lt;p&gt;This is gold for two reasons:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Idempotency testing.&lt;/strong&gt; Your handler should be safe to call with the same event ID twice. Replay it five times in a row and confirm you create one record, not five. If you create five, you have a bug.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Deterministic debugging.&lt;/strong&gt; When something fails at 2am, you can add a &lt;code&gt;console.log&lt;/code&gt;, restart your server, and replay the same event. No hunting for a new trigger, no hoping you can reproduce the path.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Step 4 — Validate via logs, not hope
&lt;/h2&gt;

&lt;p&gt;For every event, verify:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;✓ Your handler returned 200 (Stripe will retry otherwise)&lt;/li&gt;
&lt;li&gt;✓ The event ID was logged&lt;/li&gt;
&lt;li&gt;✓ A record was created (first time)&lt;/li&gt;
&lt;li&gt;✓ A replay was skipped (second time — idempotency)&lt;/li&gt;
&lt;li&gt;✓ Unknown event types no-op safely (don't throw)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A minimal handler that makes this visible:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight javascript"&gt;&lt;code&gt;&lt;span class="k"&gt;export&lt;/span&gt; &lt;span class="k"&gt;default&lt;/span&gt; &lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;handler&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;req&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;res&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;event&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;verifySignature&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;req&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt; &lt;span class="c1"&gt;// or skip on dev path&lt;/span&gt;

  &lt;span class="nx"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;`[stripe] &lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;type&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; &lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;`&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;existing&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;findEvent&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;existing&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="nx"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;`[stripe] skipped duplicate &lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;`&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nx"&gt;res&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="na"&gt;received&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="na"&gt;skipped&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt; &lt;span class="p"&gt;});&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;

  &lt;span class="k"&gt;switch &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;type&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;payment_intent.succeeded&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;handlePaymentSucceeded&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;data&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;object&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
      &lt;span class="k"&gt;break&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="k"&gt;case&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;charge.failed&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;handleChargeFailed&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;data&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;object&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
      &lt;span class="k"&gt;break&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
    &lt;span class="nl"&gt;default&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
      &lt;span class="nx"&gt;console&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;`[stripe] no-op for &lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;event&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;type&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;`&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="p"&gt;}&lt;/span&gt;

  &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;recordEvent&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;event&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="nx"&gt;res&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;json&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt; &lt;span class="na"&gt;received&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt; &lt;span class="p"&gt;});&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Three rules this enforces:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Log the event ID on every call.&lt;/strong&gt; When something goes wrong, the event ID is the only key that connects Stripe's dashboard to your logs.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Return 200 even for no-ops.&lt;/strong&gt; Stripe retries non-200 responses. A &lt;code&gt;throw&lt;/code&gt; on an unknown event type will get retried for 3 days and fill up your logs.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Make duplicate detection visible.&lt;/strong&gt; When you replay for testing, you want to SEE that the skip branch fired.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Pre-production checklist
&lt;/h2&gt;

&lt;p&gt;Before you switch Stripe from your CLI forwarder to your real endpoint:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;[ ] Every event type you care about has been triggered locally at least once&lt;/li&gt;
&lt;li&gt;[ ] Each one has been &lt;strong&gt;replayed&lt;/strong&gt; and the idempotency branch ran&lt;/li&gt;
&lt;li&gt;[ ] At least one unknown event type was sent — handler returned 200, did not throw&lt;/li&gt;
&lt;li&gt;[ ] Signature verification works in prod mode with the real Dashboard secret (not the CLI secret)&lt;/li&gt;
&lt;li&gt;[ ] Handler returns 200 in under 5 seconds for all event types (Stripe times out at 30s but backs off aggressively if you're slow)&lt;/li&gt;
&lt;li&gt;[ ] Logs include event ID on every line relevant to the event&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What this costs
&lt;/h2&gt;

&lt;p&gt;Nothing. The Stripe CLI is free. &lt;code&gt;stripe trigger&lt;/code&gt; uses your test mode data. &lt;code&gt;stripe events resend&lt;/code&gt; uses events you already generated. You don't need a tunnel service (ngrok, localtunnel) — &lt;code&gt;stripe listen&lt;/code&gt; does the forwarding itself.&lt;/p&gt;

&lt;h2&gt;
  
  
  What to pair this with
&lt;/h2&gt;

&lt;p&gt;If you want stored event history you can query later (replay a 2-week-old event, audit a customer's event sequence, etc.), you need something between Stripe and your handler that logs every event permanently. Stripe's own Events API only goes back 30 days and can't filter by fields you care about.&lt;/p&gt;

&lt;p&gt;Centrali's webhook triggers do this out of the box — every inbound event is stored in a collection you can query later. But the testing loop above works with any handler, framework, or platform. The important part is the loop.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;If this was useful, I write more about webhook reliability and Stripe integration patterns. Follow me for more.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>stripe</category>
      <category>webhooks</category>
      <category>nextjs</category>
      <category>javascript</category>
    </item>
    <item>
      <title>Stop Writing Custom Scrapers: Index Static Content into Meilisearch with One Config</title>
      <dc:creator>Mary Olowu</dc:creator>
      <pubDate>Wed, 22 Apr 2026 07:42:56 +0000</pubDate>
      <link>https://dev.to/itsmarydan/stop-writing-custom-scrapers-index-static-content-into-meilisearch-with-one-config-742</link>
      <guid>https://dev.to/itsmarydan/stop-writing-custom-scrapers-index-static-content-into-meilisearch-with-one-config-742</guid>
      <description>&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fdh6vnn32lqtyp4q5yt3p.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fdh6vnn32lqtyp4q5yt3p.png" alt=" " width="800" height="533"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;TL;DR&lt;/strong&gt; — &lt;a href="https://github.com/blueinit/content-mill" rel="noopener noreferrer"&gt;content-mill&lt;/a&gt; is an open-source CLI and library that reads static content — MkDocs sites, markdown directories, JSON files, HTML pages — and indexes it into Meilisearch, driven by a YAML config. You define the document shape; it handles extraction, templating, chunking, and atomic zero-downtime re-indexing. You still tune templates and debug extraction for your own content — that part's on you — but you stop maintaining bespoke scraper code.&lt;/p&gt;


&lt;pre class="highlight shell"&gt;&lt;code&gt;npm &lt;span class="nb"&gt;install&lt;/span&gt; @centrali-io/content-mill
&lt;/code&gt;&lt;/pre&gt;

&lt;/blockquote&gt;

&lt;p&gt;If you've ever tried to make your docs, blog posts, or changelogs searchable with Meilisearch, you know the drill: write a custom scraper, parse the content, transform it into the right shape, push it to an index, and hope you don't break search during re-indexing.&lt;/p&gt;

&lt;p&gt;I got tired of writing that glue code for every project, so I built &lt;strong&gt;content-mill&lt;/strong&gt; — a CLI and library that indexes static content into Meilisearch, driven by a YAML config.&lt;/p&gt;

&lt;h2&gt;
  
  
  The problem
&lt;/h2&gt;

&lt;p&gt;Meilisearch is fantastic for search, but getting your content &lt;em&gt;into&lt;/em&gt; it is surprisingly manual. Every docs site, every changelog, every collection of markdown files needs its own extraction pipeline. And if you want zero-downtime re-indexing? That's more code on top.&lt;/p&gt;

&lt;p&gt;Most existing solutions are either tightly coupled to a specific framework (like DocSearch for Algolia) or expect you to run a full crawler. Lighter-weight options exist — usually ad-hoc scripts people write once per project — but nothing I could find that's reusable across source types and explicit about document shape.&lt;/p&gt;

&lt;h2&gt;
  
  
  What content-mill does
&lt;/h2&gt;

&lt;p&gt;You describe your content sources and the document shape you want in a YAML config:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;meili&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;host&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;http://localhost:7700&lt;/span&gt;
  &lt;span class="na"&gt;apiKey&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;${MEILI_MASTER_KEY}&lt;/span&gt;

&lt;span class="na"&gt;sources&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;docs&lt;/span&gt;
    &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;mkdocs&lt;/span&gt;
    &lt;span class="na"&gt;config&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;./mkdocs.yml&lt;/span&gt;
    &lt;span class="na"&gt;index&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;docs&lt;/span&gt;
    &lt;span class="na"&gt;document&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;primaryKey&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;id&lt;/span&gt;
      &lt;span class="na"&gt;fields&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;id&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;slug&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}"&lt;/span&gt;
        &lt;span class="na"&gt;title&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;heading&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}"&lt;/span&gt;
        &lt;span class="na"&gt;content&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;body&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}"&lt;/span&gt;
        &lt;span class="na"&gt;section&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;nav_section&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}"&lt;/span&gt;
        &lt;span class="na"&gt;url&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;path&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}"&lt;/span&gt;
        &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;docs"&lt;/span&gt;
      &lt;span class="na"&gt;searchableAttributes&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="nv"&gt;title&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="nv"&gt;content&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
      &lt;span class="na"&gt;filterableAttributes&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="nv"&gt;section&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="nv"&gt;type&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then run:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;npx @centrali-io/content-mill index &lt;span class="nt"&gt;--config&lt;/span&gt; content-mill.yml
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Once the config matches your content, re-running is a single command. You'll still spend time tuning templates and sanity-checking extraction (use &lt;code&gt;--dry-run&lt;/code&gt; for that) — but you're not maintaining scraper code anymore. content-mill handles extraction, templating, and atomic index swapping, so search never goes down during re-indexing.&lt;/p&gt;

&lt;h2&gt;
  
  
  Four source types, one interface
&lt;/h2&gt;

&lt;p&gt;content-mill ships with adapters for the content formats you're most likely already using:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;mkdocs&lt;/code&gt;&lt;/strong&gt; — Reads your &lt;code&gt;mkdocs.yml&lt;/code&gt;, follows the nav tree, and parses each markdown page. You get &lt;code&gt;nav_section&lt;/code&gt; context so you know which part of the docs each page belongs to.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;markdown-dir&lt;/code&gt;&lt;/strong&gt; — Recursively reads &lt;code&gt;.md&lt;/code&gt; files from a directory. Supports YAML frontmatter, so you can pull version numbers, dates, or any metadata into your search index. Great for changelogs and blog posts.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;json&lt;/code&gt;&lt;/strong&gt; — Reads a JSON array (or directory of JSON files). Every key in each object becomes a template variable. Perfect for structured data you already have lying around.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;html&lt;/code&gt;&lt;/strong&gt; — Reads &lt;code&gt;.html&lt;/code&gt; files, strips scripts/styles/nav/footer, and gives you clean text. Useful for indexing a built static site.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Templating: you control the document shape
&lt;/h2&gt;

&lt;p&gt;The key design decision is that &lt;strong&gt;you&lt;/strong&gt; define what your Meilisearch documents look like. Source adapters extract raw variables (&lt;code&gt;slug&lt;/code&gt;, &lt;code&gt;heading&lt;/code&gt;, &lt;code&gt;body&lt;/code&gt;, &lt;code&gt;path&lt;/code&gt;, &lt;code&gt;frontmatter.*&lt;/code&gt;, etc.), and you map them to fields using &lt;code&gt;{{ template }}&lt;/code&gt; syntax:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;fields&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;id&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;slug&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}-{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;chunk_index&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}"&lt;/span&gt;
  &lt;span class="na"&gt;title&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;chunk_heading&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}"&lt;/span&gt;
  &lt;span class="na"&gt;content&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;chunk_body&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}"&lt;/span&gt;
  &lt;span class="na"&gt;excerpt&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;body&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;|&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;truncate(200)&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}"&lt;/span&gt;
  &lt;span class="na"&gt;url&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;path&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}#{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;chunk_heading&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;|&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;slugify&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Filters like &lt;code&gt;truncate&lt;/code&gt;, &lt;code&gt;slugify&lt;/code&gt;, &lt;code&gt;lower&lt;/code&gt;, &lt;code&gt;upper&lt;/code&gt;, and &lt;code&gt;strip_md&lt;/code&gt; can be chained with pipes. This means you're not locked into someone else's schema — your search index looks exactly the way your frontend expects.&lt;/p&gt;

&lt;h2&gt;
  
  
  Chunking for granular results
&lt;/h2&gt;

&lt;p&gt;Whole-page results are often too broad for docs search. content-mill can split pages by heading level:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;chunking&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;strategy&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;heading&lt;/span&gt;
  &lt;span class="na"&gt;level&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;2&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This turns one long page into multiple documents — one per &lt;code&gt;##&lt;/code&gt; section — each with its own &lt;code&gt;chunk_heading&lt;/code&gt;, &lt;code&gt;chunk_body&lt;/code&gt;, and &lt;code&gt;chunk_index&lt;/code&gt;. Your search results can now link directly to the relevant section instead of dumping users at the top of a page.&lt;/p&gt;

&lt;h2&gt;
  
  
  Zero-downtime re-indexing
&lt;/h2&gt;

&lt;p&gt;Every indexing run uses Meilisearch's index swap:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Documents go into a temp index (&lt;code&gt;docs_tmp&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;Atomic swap with the live index (&lt;code&gt;docs&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;Old index gets cleaned up&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;If something fails mid-way, your live index is untouched. No maintenance windows needed.&lt;/p&gt;

&lt;h2&gt;
  
  
  CI/CD in two lines
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="c1"&gt;# GitHub Actions&lt;/span&gt;
&lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Index docs&lt;/span&gt;
  &lt;span class="na"&gt;env&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;MEILI_MASTER_KEY&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;${{ secrets.MEILI_MASTER_KEY }}&lt;/span&gt;
  &lt;span class="na"&gt;run&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;npx @centrali-io/content-mill index --config content-mill.yml&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Hook this into your release pipeline and your search index stays in sync with every deploy.&lt;/p&gt;

&lt;h2&gt;
  
  
  Use as a library
&lt;/h2&gt;

&lt;p&gt;Don't need the CLI? Import it directly:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;loadConfig&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;indexAll&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;@centrali-io/content-mill&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;config&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;loadConfig&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;./content-mill.yml&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;indexAll&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;config&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;dryRun&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt; &lt;span class="p"&gt;});&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Or build the config object in code if you prefer programmatic control.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why not docs-scraper, DocSearch, or a custom crawler?
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;docs-scraper&lt;/strong&gt; (the Meilisearch-native option) is a Scrapy-based web crawler. Works well for live sites, heavy for "I already have markdown in a repo."&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Algolia DocSearch&lt;/strong&gt; is excellent, but framework-specific and indexes into Algolia — not useful if you've chosen Meilisearch.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Custom scrapers&lt;/strong&gt; work fine for one project. Painful when you have three of them to maintain across different repos.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;content-mill is intentionally narrow: static content in, Meilisearch out, config-driven shape in between. If you're not already on Meilisearch, use something else.&lt;/p&gt;

&lt;h2&gt;
  
  
  Getting started
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;npm &lt;span class="nb"&gt;install&lt;/span&gt; @centrali-io/content-mill
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ol&gt;
&lt;li&gt;Create a &lt;code&gt;content-mill.yml&lt;/code&gt; with your Meilisearch connection and source definitions&lt;/li&gt;
&lt;li&gt;Run with &lt;code&gt;--dry-run&lt;/code&gt; first to preview the extracted documents&lt;/li&gt;
&lt;li&gt;Run for real and check your Meilisearch dashboard&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The full config reference and source type examples are in the &lt;a href="https://github.com/blueinit/content-mill" rel="noopener noreferrer"&gt;README on GitHub&lt;/a&gt;.&lt;/p&gt;




&lt;p&gt;content-mill is MIT-licensed and open source. If you use Meilisearch and have static content to index, try it — and if your source type isn't covered (AsciiDoc, RST, Notion export, whatever), &lt;a href="https://github.com/blueinit/content-mill/issues" rel="noopener noreferrer"&gt;open an issue&lt;/a&gt; and I'll look at adding an adapter.&lt;/p&gt;

</description>
      <category>meilisearch</category>
      <category>search</category>
      <category>typescript</category>
      <category>opensource</category>
    </item>
    <item>
      <title>Stop Writing Custom Scrapers: Index Any Static Content into Meilisearch with One Config File</title>
      <dc:creator>Mary Olowu</dc:creator>
      <pubDate>Wed, 25 Mar 2026 05:16:16 +0000</pubDate>
      <link>https://dev.to/itsmarydan/stop-writing-custom-scrapers-index-any-static-content-into-meilisearch-with-one-config-file-2g65</link>
      <guid>https://dev.to/itsmarydan/stop-writing-custom-scrapers-index-any-static-content-into-meilisearch-with-one-config-file-2g65</guid>
      <description>&lt;p&gt;If you've ever tried to make your docs, blog posts, or changelogs searchable with Meilisearch, you know the drill: write a custom scraper, parse the content, transform it into the right shape, push it to an index, and hope you don't break search during re-indexing.&lt;/p&gt;

&lt;p&gt;I got tired of writing that glue code for every project, so I built &lt;strong&gt;content-mill&lt;/strong&gt; — a CLI and library that indexes static content into Meilisearch from a single YAML config.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Problem
&lt;/h2&gt;

&lt;p&gt;Meilisearch is fantastic for search, but getting your content &lt;em&gt;into&lt;/em&gt; it is surprisingly manual. Every docs site, every changelog, every collection of markdown files needs its own extraction pipeline. And if you want zero-downtime re-indexing? That's more code on top.&lt;/p&gt;

&lt;p&gt;Most existing solutions are either tightly coupled to a specific framework (like DocSearch for Algolia) or require you to write a full crawler. If you just have some markdown files and a Meilisearch instance, there's nothing lightweight that bridges the gap.&lt;/p&gt;

&lt;h2&gt;
  
  
  What content-mill Does
&lt;/h2&gt;

&lt;p&gt;You describe your content sources and the document shape you want in a YAML config:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;meili&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;host&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;http://localhost:7700&lt;/span&gt;
  &lt;span class="na"&gt;apiKey&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;${MEILI_MASTER_KEY}&lt;/span&gt;

&lt;span class="na"&gt;sources&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;docs&lt;/span&gt;
    &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;mkdocs&lt;/span&gt;
    &lt;span class="na"&gt;config&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;./mkdocs.yml&lt;/span&gt;
    &lt;span class="na"&gt;index&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;docs&lt;/span&gt;
    &lt;span class="na"&gt;document&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;primaryKey&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;id&lt;/span&gt;
      &lt;span class="na"&gt;fields&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;id&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;slug&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}"&lt;/span&gt;
        &lt;span class="na"&gt;title&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;heading&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}"&lt;/span&gt;
        &lt;span class="na"&gt;content&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;body&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}"&lt;/span&gt;
        &lt;span class="na"&gt;section&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;nav_section&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}"&lt;/span&gt;
        &lt;span class="na"&gt;url&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;path&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}"&lt;/span&gt;
        &lt;span class="na"&gt;type&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;docs"&lt;/span&gt;
      &lt;span class="na"&gt;searchableAttributes&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="nv"&gt;title&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="nv"&gt;content&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
      &lt;span class="na"&gt;filterableAttributes&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="nv"&gt;section&lt;/span&gt;&lt;span class="pi"&gt;,&lt;/span&gt; &lt;span class="nv"&gt;type&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then run:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;npx @centrali-io/content-mill index &lt;span class="nt"&gt;--config&lt;/span&gt; content-mill.yml
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That's it. content-mill reads your sources, extracts content, applies your field templates, and pushes everything to Meilisearch with atomic index swapping (so search never goes down during re-indexing).&lt;/p&gt;

&lt;h2&gt;
  
  
  Four Source Types, One Interface
&lt;/h2&gt;

&lt;p&gt;content-mill ships with adapters for the content formats you're most likely already using:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;mkdocs&lt;/code&gt;&lt;/strong&gt; — Reads your &lt;code&gt;mkdocs.yml&lt;/code&gt;, follows the nav tree, and parses each markdown page. You get &lt;code&gt;nav_section&lt;/code&gt; context so you know which part of the docs each page belongs to.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;markdown-dir&lt;/code&gt;&lt;/strong&gt; — Recursively reads &lt;code&gt;.md&lt;/code&gt; files from a directory. Supports YAML frontmatter, so you can pull version numbers, dates, or any metadata into your search index. Great for changelogs and blog posts.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;json&lt;/code&gt;&lt;/strong&gt; — Reads a JSON array (or directory of JSON files). Every key in each object becomes a template variable. Perfect for structured data you already have lying around.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;html&lt;/code&gt;&lt;/strong&gt; — Reads &lt;code&gt;.html&lt;/code&gt; files, strips scripts/styles/nav/footer, and gives you clean text. Useful for indexing a built static site.&lt;/p&gt;

&lt;h2&gt;
  
  
  Templating: You Control the Document Shape
&lt;/h2&gt;

&lt;p&gt;The key design decision is that &lt;strong&gt;you&lt;/strong&gt; define what your Meilisearch documents look like. Source adapters extract raw variables (&lt;code&gt;slug&lt;/code&gt;, &lt;code&gt;heading&lt;/code&gt;, &lt;code&gt;body&lt;/code&gt;, &lt;code&gt;path&lt;/code&gt;, &lt;code&gt;frontmatter.*&lt;/code&gt;, etc.), and you map them to fields using &lt;code&gt;{{ template }}&lt;/code&gt; syntax:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;fields&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;id&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;slug&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}-{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;chunk_index&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}"&lt;/span&gt;
  &lt;span class="na"&gt;title&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;chunk_heading&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}"&lt;/span&gt;
  &lt;span class="na"&gt;content&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;chunk_body&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}"&lt;/span&gt;
  &lt;span class="na"&gt;excerpt&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;body&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;|&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;truncate(200)&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}"&lt;/span&gt;
  &lt;span class="na"&gt;url&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;path&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}#{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;chunk_heading&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;|&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;slugify&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Filters like &lt;code&gt;truncate&lt;/code&gt;, &lt;code&gt;slugify&lt;/code&gt;, &lt;code&gt;lower&lt;/code&gt;, &lt;code&gt;upper&lt;/code&gt;, and &lt;code&gt;strip_md&lt;/code&gt; can be chained with pipes. This means you're not locked into someone else's schema — your search index looks exactly the way your frontend expects.&lt;/p&gt;

&lt;h2&gt;
  
  
  Chunking for Granular Results
&lt;/h2&gt;

&lt;p&gt;Whole-page results are often too broad for docs search. content-mill can split pages by heading level:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;chunking&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;strategy&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;heading&lt;/span&gt;
  &lt;span class="na"&gt;level&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="m"&gt;2&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This turns one long page into multiple documents — one per &lt;code&gt;##&lt;/code&gt; section — each with its own &lt;code&gt;chunk_heading&lt;/code&gt;, &lt;code&gt;chunk_body&lt;/code&gt;, and &lt;code&gt;chunk_index&lt;/code&gt;. Your search results can now link directly to the relevant section instead of dumping users at the top of a page.&lt;/p&gt;

&lt;h2&gt;
  
  
  Zero-Downtime Re-indexing
&lt;/h2&gt;

&lt;p&gt;Every indexing run uses Meilisearch's index swap:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Documents go into a temp index (&lt;code&gt;docs_tmp&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;Atomic swap with the live index (&lt;code&gt;docs&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;Old index gets cleaned up&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;If something fails mid-way, your live index is untouched. No maintenance windows needed.&lt;/p&gt;

&lt;h2&gt;
  
  
  CI/CD in Two Lines
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="c1"&gt;# GitHub Actions&lt;/span&gt;
&lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Index docs&lt;/span&gt;
  &lt;span class="na"&gt;env&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;MEILI_MASTER_KEY&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;${{ secrets.MEILI_MASTER_KEY }}&lt;/span&gt;
  &lt;span class="na"&gt;run&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;npx @centrali-io/content-mill index --config content-mill.yml&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Hook this into your release pipeline and your search index stays in sync with every deploy.&lt;/p&gt;

&lt;h2&gt;
  
  
  Use as a Library
&lt;/h2&gt;

&lt;p&gt;Don't need the CLI? Import it directly:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="nx"&gt;loadConfig&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nx"&gt;indexAll&lt;/span&gt; &lt;span class="p"&gt;}&lt;/span&gt; &lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;@centrali-io/content-mill&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;config&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;loadConfig&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="s1"&gt;./content-mill.yml&lt;/span&gt;&lt;span class="dl"&gt;'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nf"&gt;indexAll&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;config&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="na"&gt;dryRun&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;false&lt;/span&gt; &lt;span class="p"&gt;});&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Or build the config object in code if you prefer programmatic control.&lt;/p&gt;

&lt;h2&gt;
  
  
  Getting Started
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;npm &lt;span class="nb"&gt;install&lt;/span&gt; @centrali-io/content-mill
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;ol&gt;
&lt;li&gt;Create a &lt;code&gt;content-mill.yml&lt;/code&gt; with your Meilisearch connection and source definitions&lt;/li&gt;
&lt;li&gt;Run with &lt;code&gt;--dry-run&lt;/code&gt; first to preview the extracted documents&lt;/li&gt;
&lt;li&gt;Run for real and check your Meilisearch dashboard&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The full config reference and source type examples are in the &lt;a href="https://github.com/blueinit/content-mill" rel="noopener noreferrer"&gt;README on GitHub&lt;/a&gt;.&lt;/p&gt;




&lt;p&gt;content-mill is MIT licensed and open source. If you're using Meilisearch and have static content to index, I'd love to hear how it works for your use case. Issues and PRs welcome on &lt;a href="https://github.com/blueinit/content-mill" rel="noopener noreferrer"&gt;GitHub&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;br&gt;
&lt;em&gt;Tags: #meilisearch #search #typescript #opensource&lt;/em&gt;&lt;br&gt;
&lt;/p&gt;

</description>
      <category>meilisearch</category>
      <category>typescript</category>
      <category>opensource</category>
    </item>
    <item>
      <title>Exponential vs Linear: How to Tell If Your Event-Driven Trigger Is Looping</title>
      <dc:creator>Mary Olowu</dc:creator>
      <pubDate>Sun, 15 Mar 2026 23:46:07 +0000</pubDate>
      <link>https://dev.to/itsmarydan/exponential-vs-linear-how-to-tell-if-your-event-driven-trigger-is-looping-1gc</link>
      <guid>https://dev.to/itsmarydan/exponential-vs-linear-how-to-tell-if-your-event-driven-trigger-is-looping-1gc</guid>
      <description>&lt;h1&gt;
  
  
  Exponential vs Linear: How to Tell If Your Event-Driven Trigger Is Looping
&lt;/h1&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fne8kgwa53mqcx103xz5x.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fne8kgwa53mqcx103xz5x.png" alt=" " width="800" height="533"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The Core Idea
&lt;/h2&gt;

&lt;p&gt;When you're building rate limits for event-driven triggers, you face a fundamental problem: how do you set a threshold that catches loops without blocking legitimate high-volume workloads?&lt;/p&gt;

&lt;p&gt;The answer is that loops and legitimate traffic have fundamentally different growth characteristics:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Legitimate triggers scale linearly with user actions.&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;1 user creates 1 order → 1 trigger execution&lt;/li&gt;
&lt;li&gt;50 users create 50 orders per minute → 50 trigger executions per minute&lt;/li&gt;
&lt;li&gt;The ratio is always 1:1. Trigger executions track user actions.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Recursive loops scale exponentially from a single user action.&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;1 user creates 1 record → trigger fires → function creates another record → trigger fires again&lt;/li&gt;
&lt;li&gt;After 10 seconds: 100+ executions&lt;/li&gt;
&lt;li&gt;After 60 seconds: 700+ executions&lt;/li&gt;
&lt;li&gt;All from 1 user action. The trigger is its own input.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This isn't a subtle distinction. It's the difference between a line and an exponential curve. And it means your rate limit doesn't need to be clever — it just needs to sit in the massive gap between the two curves.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why This Matters for Rate Limit Design
&lt;/h2&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Ff2ywwvjgjzhq6vlavatu.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Ff2ywwvjgjzhq6vlavatu.png" alt=" " width="800" height="533"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;A rate limit of 100 executions per 60 seconds:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Never blocks legitimate traffic.&lt;/strong&gt; Even a high-volume e-commerce system processing 80 orders per minute sits under the limit.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Always catches loops.&lt;/strong&gt; A recursive loop hits 100 executions in under 8 seconds.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The gap between "highest legitimate volume" and "slowest possible loop" is enormous. You don't need machine learning or anomaly detection. You just need basic arithmetic.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Math
&lt;/h2&gt;

&lt;p&gt;A recursive trigger loop doubles (at minimum) with each iteration. If one trigger execution creates one record, and that record fires one trigger:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Time&lt;/th&gt;
&lt;th&gt;Executions (cumulative)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Iteration 1&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Iteration 2&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Iteration 3&lt;/td&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Iteration 10&lt;/td&gt;
&lt;td&gt;1,024&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Iteration 16&lt;/td&gt;
&lt;td&gt;65,536&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Even with network latency and compute overhead slowing each iteration to 100ms, you hit 100 executions in ~7 seconds. With faster execution (10ms per iteration), you hit 100 in under a second.&lt;/p&gt;

&lt;p&gt;Meanwhile, the highest legitimate trigger volume we've seen across our platform is ~80 executions per minute per trigger — and that's a busy e-commerce workspace during a flash sale.&lt;/p&gt;

&lt;p&gt;The gap is 10x-100x. Your rate limit has a lot of room.&lt;/p&gt;

&lt;h2&gt;
  
  
  What About Burst Traffic?
&lt;/h2&gt;

&lt;p&gt;The natural objection: "What about a bulk import? A user imports 500 records at once, and each fires a trigger."&lt;/p&gt;

&lt;p&gt;This is a valid concern but a different problem:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Bulk imports via API&lt;/strong&gt; publish a single aggregate event (&lt;code&gt;records_bulk_created&lt;/code&gt;), not 500 individual events. Event-driven triggers don't match on the aggregate event, so they don't fire at all.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Batch operations from compute functions&lt;/strong&gt; do publish individual events. But even 500 trigger executions from a batch operation is a one-time burst, not a sustained loop. If your rate limit window is 60 seconds, the burst registers once. A loop registers continuously.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;If batch-triggered functions need to fire triggers&lt;/strong&gt;, the rate limit should be configurable per-trigger. Default 100/60s works for 99% of cases. The 1% that needs more can raise it.&lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Implementing the Test
&lt;/h2&gt;

&lt;p&gt;The simplest implementation is a Redis counter with a TTL:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight typescript"&gt;&lt;code&gt;&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="kd"&gt;function&lt;/span&gt; &lt;span class="nf"&gt;isWithinRateLimit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;triggerId&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kr"&gt;string&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="nb"&gt;Promise&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nx"&gt;boolean&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;key&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;`trigger_rate:&lt;/span&gt;&lt;span class="p"&gt;${&lt;/span&gt;&lt;span class="nx"&gt;triggerId&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;`&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
  &lt;span class="kd"&gt;const&lt;/span&gt; &lt;span class="nx"&gt;count&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;redis&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;incr&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;key&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="k"&gt;if &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;count&lt;/span&gt; &lt;span class="o"&gt;===&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;await&lt;/span&gt; &lt;span class="nx"&gt;redis&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;expire&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;key&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;60&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
  &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nx"&gt;count&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That's it. Six lines. The &lt;code&gt;INCR&lt;/code&gt; is atomic (no race conditions across instances), the &lt;code&gt;EXPIRE&lt;/code&gt; handles cleanup, and the threshold separates linear from exponential with a 10x margin.&lt;/p&gt;

&lt;h2&gt;
  
  
  Beyond Rate Limiting
&lt;/h2&gt;

&lt;p&gt;Rate limiting is the safety net, not the whole solution. For a complete defense:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Block obvious loops at configuration time.&lt;/strong&gt; When a user creates a trigger on &lt;code&gt;record_created&lt;/code&gt; for collection X, and the function calls &lt;code&gt;api.createRecord('X', ...)&lt;/code&gt;, reject it with a clear error. This is prevention, not detection.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Track causality at runtime.&lt;/strong&gt; Propagate a &lt;code&gt;sourceTriggerId&lt;/code&gt; through event chains so you can identify self-loops without waiting for the rate limit to trip. The user gets a "recursive loop detected" message instead of a vague "rate limit exceeded."&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Rate limit as the catch-all.&lt;/strong&gt; For cross-trigger chains (A→B→A) and exotic patterns that bypass the first two layers.&lt;/p&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;We wrote a detailed post about implementing all three layers: &lt;a href="https://medium.com/@olowu.marydan/how-we-stopped-recursive-trigger-loops-from-melting-our-compute-fleet-498a4cb3e5d0" rel="noopener noreferrer"&gt;How We Stopped Recursive Trigger Loops From Melting Our Compute Fleet&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Takeaway
&lt;/h2&gt;

&lt;p&gt;If your platform has event-driven triggers, ask yourself: can a trigger's output become its own input? If yes, you need loop protection. And the simplest, most reliable loop protection is a rate limit set in the gap between linear user-driven traffic and exponential recursive behavior.&lt;/p&gt;

&lt;p&gt;That gap is enormous. Use it.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;Building event-driven infrastructure? We'd love to hear about your trigger architecture challenges. Reach out on [Twitter/X] @centraliio or drop a comment.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>eventdrivenarchitecture</category>
      <category>recursionprevention</category>
      <category>platformengineering</category>
      <category>ratelimiting</category>
    </item>
  </channel>
</rss>
