DEV Community: MarcoBlch

My Stripe Webhooks Were Failing 96% of the Time

MarcoBlch — Tue, 22 Sep 2026 19:05:54 +0000

Originally published on meridianbuild.dev, my engineering blog where I write up the real bugs and decisions behind the products I build.

FastPass only works if money moves correctly. A sender pays, the money sits in escrow, and a Stripe webhook is the thing that tells my backend the payment actually happened. So when I finally opened the webhook logs one evening, my stomach dropped. Around 96 percent of them were failing. Money was landing in Stripe and my app had no idea.

The setup

FastPass is a pay to reach platform. Someone pays to guarantee a reply from a busy person, the money is held in escrow, and it gets released or refunded based on what happens next. Stripe and Stripe Connect handle the money and the 75/25 split between the recipient and the platform. Every state change, a payment captured, a transfer made, a refund issued, arrives as a Stripe webhook. If the webhooks do not land, the whole machine freezes.

The webhook lives in a Supabase Edge Function, which runs on Deno, not Node. Hold that thought.

The bug

Every webhook starts by verifying Stripe's signature, so I know the call is really from Stripe and not a forged request. The textbook line looks like this:

const event = stripe.webhooks.constructEvent(body, signature, webhookSecret)

That line was fine on my laptop. In production it threw on every real event. Verification failed, so I rejected the event, so the payment update never ran. The dashboard showed green deploys, Stripe showed successful charges, and my database quietly fell behind reality.

The trap

constructEvent verifies the signature with an HMAC. In Node, that HMAC is computed synchronously. But Deno, and every edge runtime (Vercel Edge, Cloudflare Workers, Supabase Functions), only gives you the Web Crypto API, and Web Crypto is asynchronous. The synchronous helper has no synchronous crypto to call, so it simply cannot do the work. It does not warn you. It throws, and you are left blaming your webhook secret.

So "96 percent of my webhooks are invalid" was never a Stripe problem and never a security problem. It was me calling a synchronous function in a place where crypto only exists in async form.

The fix is one word

const event = await stripe.webhooks.constructEventAsync(
  body,
  signature,
  webhookSecret
)

constructEventAsync reaches for the Web Crypto API, awaits it, and returns the verified event. Same inputs, same security, one await. The moment it shipped, signature verification went green and the backlog of stuck payments started clearing.

Two things I added while I was in there:

Idempotency. Stripe retries webhooks, and now that they were actually being accepted, I did not want the same event processed twice. A double payout is a very direct way to lose money. Before doing anything, I check a webhook_events table for the event id and skip if I have seen it:

const { data: seen } = await supabase
  .from('webhook_events').select('id').eq('event_id', event.id).single()
if (seen) return ok({ skipped: true, reason: 'already_processed' })

Return 200 even on a rejected signature. Stripe retries anything that is not a 2xx, aggressively. During the incident that meant retry storms piling onto a broken endpoint. Now a bad signature is logged and answered with 200, so Stripe stops hammering while I investigate. It is one of the rare cases where a 200 on a failure is correct, because the alternative is a self inflicted denial of service.

The takeaway

If you move a webhook, or any signature check, into an edge or Deno runtime and it starts failing for no obvious reason, suspect the crypto before you suspect your secret. Edge runtimes expose Web Crypto, which is async only. SDKs that grew up on Node ship synchronous helpers that quietly do not work there. The fix is almost always the same shape: find the ...Async variant and await it. Stripe has constructEventAsync, and most libraries have an equivalent once you go looking.

The scariest bugs are the ones where nothing crashes. My app was up, Stripe was happy, money was moving, and the only symptom was a number in a log I had not thought to read. I hit the mirror image of this on the same project, a sitemap that quietly advertised the wrong pages, where everything returned 200 and was still wrong.

If you want to see what it powers: FastPass lets you get paid to answer messages instead of drowning in a free inbox.

Why Every Funnel Event in My Analytics Read Zero

MarcoBlch — Tue, 22 Sep 2026 12:35:00 +0000

📖 Originally published on meridianbuild.dev my engineering blog documenting the real bugs and decisions behind OutfitMaker.

The worst kind of broken instrumentation is the kind that reports a number. A blank dashboard makes you check the wiring. A dashboard full of zeros makes you think your product is just quiet.

For as long as I'd had funnel tracking in OutfitMaker, every custom event I cared about read zero. Signup: 0. Activation: 0. First Item Uploaded: 0. The onboarding-choice events: 0. Meanwhile pageviews ticked up normally and real people were clearly signing up, uploading clothes, and getting outfit suggestions. I'd half-convinced myself the funnel was just thin. It wasn't thin. It was lying.

Here's the teardown. It took two PRs and turned out to be four separate bugs stacked on top of each other.

How the events were supposed to fire

OutfitMaker runs Kaunta, a self-hosted, privacy-friendly analytics script, loaded once per page. Automatic pageviews work out of the box. For funnel milestones I used a server-driven pattern: a controller sets flash[:analytics_event] = "Signup", and the layout turns that flash into a kaunta.track() call on the next page.

The layout snippet looked like this and it had been copy-pasted into three layouts:

<script src="<%= kaunta_script_url %>"
  data-website-id="<%= kaunta_website_id %>"
  async defer></script>

<% if flash[:analytics_event].present? %>
<script>
  document.addEventListener('DOMContentLoaded', function() {
    if (window.kaunta) {
      window.kaunta.track('<%= flash[:analytics_event] %>', { locale: '<%= I18n.locale %>' });
    }
  });
</script>
<% end %>

It reads fine. It is also wrong in three independent ways, each one enough on its own to zero out the event.

Bug 1: Turbo never re-fires DOMContentLoaded

Every one of these milestones arrives on a page reached by a redirect_to after a form POST. You submit the signup form, the server creates the user, sets the flash, and redirects you to the next page. In a Hotwire app that redirect is a Turbo visit, not a full document load.

DOMContentLoaded fires exactly once, on a cold load. Turbo swaps the <body> and does not re-evaluate <head> scripts on navigation. So on the page that actually carried the flash, the listener was being registered after the event it was waiting for had already passed — and then never ran again. The track() call was real, reachable code that simply never executed.

This is the same family of silent Turbo failure I wrote about with duplicate frame ids: nothing throws, the HTTP response is a clean 200, and the gap is a layer of intent the framework doesn't share with you.

Bug 2: the async race

Suppose the page did cold-load a direct visit, say. The Kaunta loader is async defer, so window.kaunta is usually still undefined when DOMContentLoaded runs. The guard:

if (window.kaunta) { /* track */ }

is a coin flip you lose most of the time. When the script wasn't ready, the event was dropped on the floor with no retry. No error, no queue, nothing. So even the rare cold-load path was unreliable.

Bug 3: the layout the funnel actually lands on

The snippet lived in the application, landing, and blog layouts. But a successful signup redirects to onboarding_path, which renders under the onboarding layout and that layout had no Kaunta at all. Not the loader, not the trigger.

So for the single most important event in the whole funnel, Signup, all three bugs were academic. The page it landed on couldn't have tracked anything even if the script and the timing had been perfect. Signup wasn't unreliable. It was structurally impossible.

The fix: fire from Stimulus, share one partial

The repair (9dec828, PR #85) moves the trigger out of the <head> and into a Stimulus controller that runs from the <body>. Stimulus connect() runs on every Turbo navigation, which kills Bug 1. And instead of a one-shot guard, it polls for window.kaunta, which kills Bug 2:

import { Controller } from "@hotwired/stimulus"

export default class extends Controller {
  static values = {
    name: String,
    props: { type: Object, default: {} },
  }

  connect() {
    if (!this.nameValue) return
    // Skip Turbo's cached *preview* render so a snapshot that still
    // contains this element doesn't double-fire on the preview pass.
    if (document.documentElement.hasAttribute("data-turbo-preview")) return
    if (this.fired) return
    this.waitForKaunta(25) // ~5s max (25 × 200ms)
  }

  waitForKaunta(attemptsLeft) {
    if (this.fired) return
    if (window.kaunta && typeof window.kaunta.track === "function") {
      this.fired = true
      window.kaunta.track(this.nameValue, this.propsValue)
    } else if (attemptsLeft > 0) {
      this.pollTimer = setTimeout(() => this.waitForKaunta(attemptsLeft - 1), 200)
    }
  }
}

The data-turbo-preview check is the kind of detail you only learn by getting it wrong: Turbo renders a cached preview of a page before the fresh version arrives, and if your tracking element is in that snapshot it'll fire on the preview too. Bail on the preview pass; fire on the real render.

The flash is surfaced by a tiny partial that just mounts the controller:

<% if flash[:analytics_event].present? %>
  <div hidden
       data-controller="kaunta-event"
       data-kaunta-event-name-value="<%= flash[:analytics_event] %>"
       data-kaunta-event-props-value="<%= { locale: I18n.locale }.to_json %>"></div>
<% end %>

Bug 3 gets fixed by construction: the loader and the trigger become two shared partials (shared/_kaunta_head and shared/_kaunta_event), rendered in all four user-facing layouts application, landing, blog, and the previously-naked onboarding. That deletes the three-way copy-paste and closes the gap in the same move. Three layouts each lost ~30 lines.

I can't run the suite on the box I write these from (no Ruby, no Node it validates on deploy), so I locked the server side of the contract with an integration test instead of trusting the browser behaviour to a comment:

test "first item upload emits the kaunta-event trigger with the event name" do
  post wardrobe_items_path, params: { wardrobe_item: { image: image } }
  follow_redirect!

  assert_select "[data-controller='kaunta-event']", count: 1
  assert_select "[data-kaunta-event-name-value='First Item Uploaded']", count: 1

  # Regression guard: the old, broken inline tracker must be gone.
  assert_no_match(/addEventListener\(['"]DOMContentLoaded['"]/, response.body,
    "flash analytics event must not be tracked via a DOMContentLoaded script")
end

The assert_no_match is the part I care about most. It's not testing what the page does; it's testing that the broken pattern can never come back through a careless copy-paste.

Bug 4: Signup was still zero

I deployed, watched First Item Uploaded and the onboarding-choice events start landing events that had never been anything but zero and felt good about it. Then I looked again the next day. Signup was still zero.

The first three bugs were dead. This was a fourth one wearing their clothes.

The cause: 100% of OutfitMaker's signups come through Google OAuth. Nobody uses the email/password form. And the Signup event was only ever set in RegistrationsController#create the email/password path. The OAuth path, OmniauthCallbacksController, set no event and, it turned out, sent no admin notification either. Which also explained a separate mystery: I never got an email when someone signed up. I'd been blind to my own new users because I'd only instrumented the door nobody walks through.

The fix (dca116a, PR #88) pulls the side effects into a shared concern both controllers include:

module SignupSideEffects
  extend ActiveSupport::Concern

  private

  def handle_new_signup(user)
    user.update_column(:locale, I18n.locale.to_s)
    flash[:analytics_event] = "Signup"
    return if Rails.env.test?

    UserMailer.welcome_email(user).deliver_later
    AdminMailer.new_user_signup(user).deliver_later unless user.admin?
  end
end

Two subtleties in the OAuth side made this more than a one-liner.

Telling a new signup from a returning login. from_omniauth both creates new users and logs in existing ones, and it'll also link OAuth to an existing email account. Only a genuinely new record should fire Signup. Rails 7's previously_new_record? is exactly that signal true only on the request that created the row:

if @user.previously_new_record?
  handle_new_signup(@user)
  sign_in(@user, event: :authentication)
  redirect_to onboarding_path(step: 1)
else
  flash[:notice] = t("flash.omniauth.success", provider: provider)
  sign_in_and_redirect @user, event: :authentication
end

Flash survives exactly one redirect. Devise's sign_in_and_redirect would send a new user to root, which then redirects to onboarding two hops. flash[:analytics_event] only survives one. The second redirect would silently eat it, and I'd be right back to a zeroed Signup for a subtly different reason. So the new-signup branch redirects straight to onboarding in a single hop, and the flash lives long enough for the Stimulus trigger to fire.

I covered the new-vs-returning logic at the model level too, because it's the load-bearing assumption a brand-new user, a returning user, and an OAuth-linked existing account each get their own test asserting whether previously_new_record? is set.

The result

After the second deploy, Kaunta started showing real signups where there had only ever been a flat line and the rest of the chain (Activation, the post-wow upgrade prompts) lit up with it. OutfitMaker is small enough that these are modest counts, but the point isn't the magnitude. It's that the dashboard finally describes the product instead of contradicting it.

The takeaway

Instrumentation that silently reports zero is worse than none, because you act on it. I spent real time wondering why my "funnel was thin" when the funnel was fine and the funnel meter was unplugged. Two rules I'd hand my past self:

In a Turbo app, never fire analytics from a <head> DOMContentLoaded script. Your most important events arrive on Turbo visits, where that listener is already too late. Fire from a Stimulus connect() and wait for your script to be ready instead of guarding once and giving up.
A funnel's top must cover every path into it. I instrumented the signup flow I wrote first and assumed it was the signup flow. The one my users actually used was untouched. Count the doors before you trust the turnstile.

The quiet bugs are the expensive ones. A crash gets fixed the same day. A zero that should be a number can sit there for as long as you're willing to believe your product is just quiet.

My AI Was Blind in Production for Weeks and Nothing Crashed

MarcoBlch — Fri, 11 Sep 2026 08:25:35 +0000

📖 Originally published on meridianbuild.dev my engineering blog documenting the real bugs and decisions behind OutfitMaker.

The whole pitch of OutfitMaker's outfit suggestions is that the AI sees your wardrobe. Not a list of tags like "blue oxford shirt, brown chinos" the actual photos of your actual garments, so Gemini can reason about color, texture, cut and how two specific pieces look together. That's the difference between a suggestion engine and a search filter, and it's the multimodal path I built the feature around.

For a few weeks in production, it wasn't seeing anything at all.

The feature still worked. Users opened the app, asked for an outfit, and got outfits back. Nothing errored. Nothing 500'd. Sentry was quiet. The only thing wrong was that every suggestion was being generated by a model that had received exactly zero images it was captioning off the text labels and nothing else. My "multimodal AI" was, in the most literal sense, blind, and the app was too polite to tell me.

The one log line that gave it away

I wasn't hunting for this. I was reading production logs for something unrelated when the same line kept scrolling past:

Multimodal suggestion: 0/25 items with images

Twenty-five items in the user's wardrobe, zero of them reaching Gemini with a photo attached. Every request. Right above it, the reason:

Failed to encode image for item 4867:
`identify -format %m %w %h %b ...` failed with status: 127

status: 127 is the shell's way of saying command not found. Whatever was trying to run identify couldn't find identify.

identify is an ImageMagick binary. And that's the moment the whole thing clicked into place.

Two libraries that both do "images"

OutfitMaker leans on two completely separate image toolchains, and I had stopped thinking of them as separate.

libvips (via ruby-vips) powers Active Storage variants the thumbnails and resized wardrobe images the app serves all day.
ImageMagick (via MiniMagick) is what two specific services shell out to: OutfitSuggestionService#encode_item_image, which measures and encodes a garment photo before handing it to Gemini, and WatermarkService, which stamps shared Look Previews.

MiniMagick doesn't bundle ImageMagick. It's a thin Ruby wrapper that shells out to the identify and convert binaries and expects them to be on the PATH. If they're not there, it doesn't raise a nice LoadError at boot it fails at call time, per image, with a shell exit code.

Here's the part that stung: this is invisible to your Gemfile. bundle install is perfectly happy. MiniMagick the gem is installed. The dependency that's actually missing lives one layer down, in the operating system, and nothing in Ruby's dependency graph knows or cares.

The Dockerfile was telling the truth; I just hadn't read it

The runtime stage of the multi-stage Dockerfile installed exactly what Active Storage needed and nothing more:

RUN apt-get update -qq && \
    apt-get install --no-install-recommends -y curl libvips postgresql-client python3 libgomp1 && \
    rm -rf /var/lib/apt/lists /var/cache/apt/archives

libvips is there. imagemagick is not. So in the production container, identify genuinely did not exist.

Why didn't I catch it in development? Because my laptop has ImageMagick installed the way most dev machines accumulate it — from some Homebrew formula years ago, pulled in as a transitive dependency of something else. Locally, identify was on the PATH, MiniMagick was happy, images encoded, everything looked fine. The bug only existed in the one environment I couldn't see into by running identify in a terminal: the slim production image.

This is the recurring lesson of shipping Rails in Docker. Your production runtime is a separate contract from your Gemfile, and the slimmer you make the image, the more of that contract you're signing implicitly. Every --no-install-recommends is a small bet that you listed everything you actually need.

The fix is one word

 RUN apt-get update -qq && \
-    apt-get install --no-install-recommends -y curl libvips postgresql-client python3 libgomp1 && \
+    apt-get install --no-install-recommends -y curl imagemagick libvips postgresql-client python3 libgomp1 && \
     rm -rf /var/lib/apt/lists /var/cache/apt/archives

That's the entire change in PR #81 (b57fb72). I checked on ruby:3.3.5-slim first that apt-get install imagemagick actually provides identify (ImageMagick 6.9.11, about 30MB with dependencies), deployed, and watched the log line flip from 0/25 to items actually carrying image data into the model. Watermarking, which was silently broken the same way, came back at the same time.

One word in a Dockerfile. Weeks of a flagship feature quietly running at a fraction of its quality.

What I actually took away

The one-line fix is not the interesting part. Three things are.

Silent degradation is worse than a crash. A crash gets a Sentry alert, a red graph, a bug report. This got none of that, because from the outside the feature "worked" it returned outfits. The failure was in quality, which no exception tracker measures. If I'd had a single assertion that treated 0/25 items with images as an error condition instead of an info log, I'd have known in minutes, not weeks. Degraded-but-successful is the hardest failure mode to see, and it's exactly the one AI features love to hit.

Shell-out dependencies need a home in your health checks. A gem that wraps a system binary is a dependency your language tooling can't verify. The honest fix isn't just "add the package" it's making the runtime prove it has what it needs. A boot-time check that identify exists would have turned a silent production degradation into a failed deploy.

The real cleanup is to not need ImageMagick at all. libvips was already in the image for Active Storage, and it can do everything those two MiniMagick call sites need. The follow-up isn't to keep patching the runtime it's to migrate encode_item_image and WatermarkService off MiniMagick and onto ruby-vips, so the dependency I forgot to install becomes a dependency I no longer have. The best way to stop forgetting a thing is to delete the thing.

If you run a multimodal feature in production, go check your logs for the equivalent of 0/25. The model will never complain that you forgot to send it the picture. It'll just quietly do its best with half the input and hand you back something plausible which is the most dangerous thing an AI can do.

More OutfitMaker war stories in the same vein: the Gemini swap where Vertex refused to return images, and the TOCTOU race in my Gemini rate limiter.

My Sidekiq Worker OOM'd Four Times. I Fought It Four Ways Before Buying RAM

MarcoBlch — Wed, 09 Sep 2026 07:40:19 +0000

📖 Originally published on meridianbuild.dev my engineering blog documenting the real bugs and decisions behind OutfitMaker.

The feature that kept killing my background worker was the feature working exactly as designed.

OutfitMaker's strongest activation loop is what I call the closet dump: a new user photographs their whole wardrobe in one sitting and uploads it in a burst. When it fires, it's great — the app is now clearing something like 500 items a day. But each uploaded garment kicks off an ImageAnalysisJob, and each of those does something quietly expensive: it shells out to rembg to strip the background, which spawns a Python subprocess that loads a ~170MB U2Net model into memory.

One model load is fine. Five at once, on a small box, is not.

OOM #1 the burst that killed the worker

On August 20 the Sidekiq worker OOM-crashed mid-burst. A 70-item upload fanned out into ImageAnalysisJobs, Sidekiq ran them at concurrency 5, and five simultaneous U2Net loads was roughly a gigabyte of model weights materializing at the same instant. The Linux OOM killer did what it does, the worker died, and every job in flight died with it — 610 items stranded in pending, which I had to re-enqueue by hand in prod.

The tempting fix is "lower the concurrency." I didn't reach for that first, because it would tax every job to contain a problem that only one job type causes. The other jobs in the queue are network-bound Gemini calls — they sit around waiting on I/O and use almost no memory. Throttling them to protect rembg is collateral damage.

Defense 1: a Redis semaphore that caps rembg at one concurrent run. A SET NX EX lock (rembg:exclusive) means only one background removal executes at a time; the other worker threads keep churning through the Gemini-bound jobs, so throughput barely moves while peak memory drops about 5×. The details that made it safe in production:

If a job waits past a 90-second budget for the lock, it skips background removal. That's fine rembg has always been best-effort; callers already handle a nil result, and retry_analysis can add the cutout later.
If Redis itself is unreachable, the job proceeds unguarded rather than blocking analysis on a dead dependency.
The lock carries a TTL, so a worker that dies holding it doesn't wedge the whole pipeline the lock just expires.

Twenty-one green test runs, shipped. Peak memory way down. Done, I thought.

OOM #2 the single run that was too big by itself

Two days later, August 22, a different OOM signature. The semaphore was working the logs showed lock-busy skips exactly as designed but now single, serialized rembg runs were being OOM-killed by the container's cgroup. The tell was a Background Removal Failed: line with an empty stderr, repeating every 30 seconds or so. Empty stderr means the process didn't error; it got killed.

rembg's memory scales with input pixels. A modern phone shoots 12-megapixel photos, and pushing 12MP through U2Net inference spikes to multiple gigabytes even one at a time. Serializing hadn't helped because the problem wasn't concurrency anymore; it was the size of a single input.

Defense 2: downscale the image to ≤1600px on the long side before handing it to rembg. U2Net operates on roughly 320px internally feeding it a 12MP original buys you literally nothing in output quality. Downscaling first caps the memory spike, and as a bonus makes each run faster (which means fewer of those 90-second lock-busy skips). Small images pass through untouched; if the resize fails, it falls back to the original. Best-effort all the way down.

This is the one I'd call a real fix, not a workaround and at the time it felt like the end of it. My own commit note said it out loud: "the memory-plan upgrade question: NOT needed — 8GB is ample once inputs are sane."

I was about to be proven half wrong.

Defense 3 and OOM #4 the software ladder runs out

There was a quieter third defense in between: setting MALLOC_ARENA_MAX=2. Ruby under glibc will happily spin up a memory arena per thread, and the fragmentation adds up on a multi-threaded Sidekiq process. Pinning the arena count is a well-worn Rails-in-production knob that trims baseline bloat for free.

And then, on August 23 at 19:48, OOM #4 with all three prior defenses live.

The culprit this time was the one structural thing I'd been avoiding. Sidekiq queue weights don't cap per-queue concurrency. During an upload burst, nothing stops all the worker threads from running ImageAnalysisJob simultaneously, each holding a full-resolution image buffer in Ruby memory (phones are shooting 24MP now) alongside the serialized rembg. The downscale protected the Python subprocess; it didn't protect the Ruby heap holding the originals.

Defense 4: drop Sidekiq concurrency from 5 to 3. Three threads still clear ~500 items a day comfortably, because the jobs are Gemini-bound they spend most of their time waiting on the network, not computing. Bursts drain slower, but the worker survives them. I wrote the honest line into the config comment: if OOM #5 happens past this, the software ladder is exhausted the Railway Pro upgrade becomes the correct spend.

That's the sentence that mattered. Four defenses in, I'd stopped pretending the box was big enough.

The part nobody likes to write: I bought RAM

Here's the thing the concurrency cut exposed. Dropping from 5 to 3 threads slowed burst draining exactly during activation the closet-dump moment, which is the single most important thing a new user does. My memory defenses were now actively degrading the business outcome they existed to protect. A user dumps their whole closet, and my worker politely trickles through it three at a time because I was scared of an OOM.

So on August 25 I moved the service to Railway Pro and verified a 32GB limit on both containers. Then I went back and undid the survival taxes but only those:

Sidekiq concurrency went back to 5.
The rembg lock became a 2-slot semaphore instead of one (a per-thread token with a compare-and-delete release via EVAL, because redis-client has no eval sugar).
The downscale-to-1600px and MALLOC_ARENA_MAX=2 stayed.

That last line is the whole lesson. Speed restored, risk not re-introduced. 423 tests, zero failures.

What I'd actually take from five days of this

Separate the correctness fixes from the survival taxes. Downscaling a 12MP image before a model that works at 320px is just correct it should have been there from day one, and it stays forever. Capping concurrency to survive a too-small box is a tax: it works, but it costs you exactly where you can least afford it, and you should undo it the moment you have headroom. When I got more RAM, I kept the first kind and reversed the second. If I'd treated them all as "the fix," I'd have paid the activation tax indefinitely for no reason.

A ladder of cheap fixes is the right instinct right up until it isn't. Each defense was individually correct and individually shippable, and I'd make the same first three calls again. But there's a failure mode where you keep climbing the software ladder past the rung where the honest answer is "the machine is too small." I almost did. The tell was defense #4 starting to hurt the product. Buying RAM isn't an admission of failure; it's a line item, and sometimes it's cheaper than the fifth clever workaround plus the activation you're leaking while you build it.

ML subprocesses are memory bombs, and your Ruby heap is a second one. The rembg model got all my attention because it was the loud failure. But OOM #4 was Ruby holding full-resolution image buffers across five threads a cost that had nothing to do with Python. If you fan out image work in Sidekiq, both sides of the shell-out are spending memory, and only measuring one of them is how you get a fourth OOM after three fixes.

If you want more of these, I've written about wrapping those same external calls in circuit breakers and the concurrency race hiding in my Gemini rate limiter the same worker, different ways to make it hurt.

I Added Circuit Breakers to a Rails App and Had to Patch the Gem Twice

MarcoBlch — Sun, 06 Sep 2026 15:53:24 +0000

📖 Originally published on meridianbuild.dev my engineering blog documenting the real bugs and decisions behind OutfitMaker.

OutfitMaker leans on a lot of external services. Outfit suggestions, wardrobe image analysis, missing-item detection and trip planning all call Gemini through Vertex AI. The "Look Preview" feature calls a second Google API for image generation (a whole story on its own I wrote about that swap here). Product images come from Replicate. Affiliate suggestions come from Amazon via RapidAPI.

Every one of those is a thing that can go down, get slow, or start rate-limiting me without warning. And when one does, the failure mode in a Rails app is ugly: a Sidekiq job retries, hammers the dead service, ties up a worker, the retry queue backs up, and the failure spreads to features that have nothing to do with the broken provider.

A circuit breaker is the standard fix. It's a small state machine that sits in front of an external call. After a set number of failures inside a time window, it "opens" and while it's open, calls fail instantly instead of waiting on a dead service. After a cooldown it goes "half-open," lets one call through to test the water, and either closes (recovered) or opens again. The point isn't to make failures disappear. It's to make them cheap and contained instead of expensive and contagious.

I shipped circuit breakers across all six integrations in two phases in early May 2026 — Vertex services first (b7f9fd3, PR #63), then Look Preview, Replicate and Amazon (b637695, PR #67). I reached for the breaker_machines gem, pinned to 0.10.3. The DSL is clean and the wiring took an afternoon.

Then I tried to actually make a circuit trip, and the afternoon turned into a week.

Decision one: separate circuits, not one big Vertex circuit

Four of my services hit Gemini Vertex AI on the same gemini-2.5-flash model. The tempting design is a single :gemini_vertex circuit they all share, because they genuinely share fate one Vertex outage breaks all four.

I gave each service its own circuit instead. The reasoning is in a design note in the shared mixin:

# DESIGN NOTE separate Vertex circuits despite shared model
#   1. Per-feature blast radius. The April 23 incident came from
#      ImageAnalysisJob under load. A shared circuit would have opened
#      and silently degraded outfit suggestions and trip planning by
#      contagion — features paid users depend on.
#   2. Per-feature observability. Sentry breadcrumbs and metrics tag by
#      circuit name. Separate circuits = a glance tells you which feature
#      tripped, no log mining.
#   3. Per-feature cost of false positives. A false-positive open on
#      MissingItemDetector returns [] (graceful). A false-positive open on
#      OutfitSuggestionService blocks a paywalled flow.

A shared circuit is one fewer thing to configure, but it couples the blast radius of every feature to the noisiest one. The whole reason I was adding breakers was to stop one feature's failure from spreading. A shared circuit would have quietly re-introduced exactly that.

To keep four near-identical declarations DRY without coupling them, there's a class-method helper:

def gemini_vertex_circuit(circuit_name,
                          service_exception:,
                          network_errors:,
                          failures: 3,
                          within: 1.minute,
                          reset_after_seconds: 30,
                          &block)
  circuit circuit_name do
    threshold failures: failures, within: within
    reset_after reset_after_seconds.seconds, jitter: 0.25
    handle(*network_errors, service_exception)
    instance_exec(&block) if block
  end
end

Replicate gets a different template failures: 5 within 1.hour instead of 3 within 1.minute because its traffic is roughly one invocation a day right now, so a one-minute window is statistically unreachable, and its published rate limit makes short failure clusters more likely than a real outage. The thresholds describe the service, not a global default.

Decision two: where circuit state lives

Circuit state has to be shared across processes my web dynos and Sidekiq workers all need to agree that a circuit is open. So it goes in Redis. The non-obvious part is which Redis.

I gave it a dedicated database (db 1), separate from Rails.cache (db 0):

# Why a separate Redis DB instead of reusing Rails.cache?
#   1. Rails.cache.clear (and Rack::Attack key churn) wipe everything in db 0.
#      Circuit breaker state must survive cache flushes losing it during an
#      outage would re-arm the breaker mid-incident and let traffic stampede
#      a service that's already down.
#   2. Namespace ("bm") is a defense in depth, not a substitute for db
#      isolation: ActiveSupport's :redis_cache_store only namespaces keys, it
#      does not isolate the Redis DB.

The TTL is 24 hours long enough to outlast a real multi-hour provider outage, because a short TTL would silently drop circuit state at 3am during a low-traffic night, right before morning traffic resumes. There's also a build-time branch: when assets compile with SECRET_KEY_BASE_DUMMY set and no Redis, the store falls back to a NullStore so the breaker becomes a no-op. No traffic at build time means no state to track.

All of that is design. None of it is what cost me the week. The week went to discovering that with this exact setup, the circuit never actually tripped.

Bug one: the breaker counted to zero forever

I wrote a test that fired enough failures to cross the threshold and asserted the circuit opened. It didn't. The failure count stayed at zero no matter how many exceptions I threw.

breaker_machines' cache adapter counts failures by calling increment on the cache store and then reading the value back. With RedisCacheStore, increment issues a raw Redis INCR, which stores a plain string "3". But the read comes back through ActiveSupport's default deserialization path, which tries to un-marshal that string, fails, and returns nil. nil.to_i is 0. The counter is structurally incapable of going up.

The fix is four lines read the counter raw and coerce it:

module BreakerMachinesCacheRedisFix
  def get_window_count(key, window_seconds)
    if @cache.respond_to?(:increment)
      @cache.read(key, raw: true).to_i
    else
      super
    end
  end
end

BreakerMachines::Storage::Cache.prepend(BreakerMachinesCacheRedisFix)

Finding the four lines took the better part of two days. The patch itself is guarded so it can't rot silently:

unless defined?(BreakerMachines::VERSION) && BreakerMachines::VERSION == "0.10.3"
  raise "breaker_machines_cache_patch is pinned to 0.10.3, currently loaded: ..."
end

unless BreakerMachines::Storage::Cache.instance_method(:get_window_count).arity == 2
  raise "BreakerMachines::Storage::Cache#get_window_count signature changed; ..."
end

If I ever bump the gem, the app refuses to boot until I've re-checked whether the bug still exists. A monkey patch you forget about is worse than the bug it fixed. (This is the same instinct as the atomic rewrite in my Gemini rate-limiter post: when a read-then-write straddles a process boundary, the boundary is where the bug hides.)

Bug two: a fresh worker forgot the circuit was open

With the counter fixed, circuits tripped correctly. Then I tested the scenario that actually matters in production: a worker boots into a world where the circuit is already open, set by some other process. It should refuse calls immediately. Instead, its first call sailed straight through to the dead service.

breaker_machines does try to handle this. Its Circuit#initialize reads the stored status from Redis and assigns self.status = "open". The problem is ordering. Right after initialize returns, the underlying state_machines gem runs its own initialize_states lifecycle hook, which resets @status back to the initial value :closed because the manual assignment never tripped the flag that tells state_machines "this attribute is already set." The restore happens, then gets quietly overwritten.

The trace, captured in the patch's own documentation:

[restore] stored=#<Status status=:open, opened_at=...>
[restore] after assign: status=open
[trace status= called with "closed"] caller=[
  "state_machines/machine/state_methods.rb:88:in `write'",
  "state_machines/machine/state_methods.rb:35:in `initialize_state'",
  "state_machines/machine_collection.rb:36:in `block in initialize_states'"
]
FINAL: closed

The fix is to restore again, after the lifecycle is done:

module BreakerMachinesStateRestoreFix
  def initialize(name, options = {})
    super
    restore_status_from_storage if @storage
  end
end

BreakerMachines::Circuit.prepend(BreakerMachinesStateRestoreFix)

super runs the full chain — including the state_machines reset. Then I re-apply the stored status, this time with nothing left to clobber it. The restore is just a read-and-assign, so calling it twice is harmless.

Why does this matter enough to patch a gem? Because of how the failure scales. Every fresh worker that boots during an outage and Sidekiq autoscaling spins up more workers exactly when things are failing pays one wasted call to the dead service before its in-memory state catches up. Redeploying to ship a fix during an outage produces a whole fleet of forgetful workers. The cost is small per worker and real in aggregate, and it's worst at the exact moment you most need the breaker to hold. At one worker it's a rounding error; at five workers plus frequent deploys it's the pattern the breakers existed to remove.

Because it's a Module#prepend on the base Circuit class, the fix applied to all six circuits at once with zero per-service changes — one Railway restart.

What "open" looks like to a user

A tripped circuit shouldn't show a stack trace. The fallback on the outfit-suggestion circuit distinguishes the two cases the gem lumps together:

fallback do |error|
  # The breaker_machines fallback fires on EVERY whitelisted error, not
  # just open-state calls. Two distinct cases:
  #   1. Circuit OPEN — error is BreakerMachines::CircuitOpenError. Replace
  #      with a localized "temporarily unavailable" message, tag Sentry with
  #      the circuit name.
  #   2. Circuit CLOSED but raised an in-whitelist exception re-raise
  #      unchanged so the controller's existing rescue handles it like before.
  if error.is_a?(BreakerMachines::CircuitOpenError)
    # ... localized message + Sentry tag ...

Open circuit means "we already know this is down, here's a calm message." A closed circuit that raised means the real failure should flow through untouched. Collapsing those two into one generic error would have either hidden real bugs or shown scary copy for a known, handled state.

The numbers I can and can't give you

What I can stand behind, because it's in the repo: the gem is pinned to 0.10.3, both patches carry version guards that fail the boot on an unverified bump, and the two patch test files are 87 and 163 lines the state-restore one simulates two processes to prove a fresh circuit sees the stored open state. The two bugs are reproducible on 0.10.3, and I confirmed the counter bug is still present in the 0.10.8 source.

What I can't give you is a clean "incidents prevented" graph. OutfitMaker isn't at the scale where provider outages hit daily, and I'm not going to dress up the breakers' value with numbers I don't have. The honest framing: this was insurance bought before the fire, and most of the work was discovering the policy didn't pay out until I patched it twice.

Takeaway

A circuit breaker is a small state machine that fails fast and contains the blast radius when an external service goes down and for an AI-heavy app riding on three different Google and third-party APIs, it's not premature optimization, it's table stakes. But "add the gem" is the 20% of the work. The 80% is the boring, specific reality underneath: your cache store serializes counters in a way the adapter didn't expect, and your circuit state evaporates the instant a worker restarts unless the restore runs after the state-machine lifecycle, not during it.

If you're wiring breakers into a multi-process Rails deploy, write the two tests that actually matter before you trust the library: one that proves a circuit trips under your real cache store, and one that proves a freshly booted process sees a circuit another process already opened. Mine both failed against a popular, well-written gem. The breakers in OutfitMaker only do their job because those two tests forced the patches that made them true.