My AI Was Blind in Production for Weeks and Nothing Crashed

📖 Originally published on meridianbuild.dev my engineering blog documenting the real bugs and decisions behind OutfitMaker.

The whole pitch of OutfitMaker's outfit suggestions is that the AI sees your wardrobe. Not a list of tags like "blue oxford shirt, brown chinos" the actual photos of your actual garments, so Gemini can reason about color, texture, cut and how two specific pieces look together. That's the difference between a suggestion engine and a search filter, and it's the multimodal path I built the feature around.

For a few weeks in production, it wasn't seeing anything at all.

The feature still worked. Users opened the app, asked for an outfit, and got outfits back. Nothing errored. Nothing 500'd. Sentry was quiet. The only thing wrong was that every suggestion was being generated by a model that had received exactly zero images it was captioning off the text labels and nothing else. My "multimodal AI" was, in the most literal sense, blind, and the app was too polite to tell me.

The one log line that gave it away

I wasn't hunting for this. I was reading production logs for something unrelated when the same line kept scrolling past:

Multimodal suggestion: 0/25 items with images

Twenty-five items in the user's wardrobe, zero of them reaching Gemini with a photo attached. Every request. Right above it, the reason:

Failed to encode image for item 4867:
`identify -format %m %w %h %b ...` failed with status: 127

status: 127 is the shell's way of saying command not found. Whatever was trying to run identify couldn't find identify.

identify is an ImageMagick binary. And that's the moment the whole thing clicked into place.

Two libraries that both do "images"

OutfitMaker leans on two completely separate image toolchains, and I had stopped thinking of them as separate.

• libvips (via ruby-vips) powers Active Storage variants the thumbnails and resized wardrobe images the app serves all day.
• ImageMagick (via MiniMagick) is what two specific services shell out to: OutfitSuggestionService#encode_item_image, which measures and encodes a garment photo before handing it to Gemini, and WatermarkService, which stamps shared Look Previews.

MiniMagick doesn't bundle ImageMagick. It's a thin Ruby wrapper that shells out to the identify and convert binaries and expects them to be on the PATH. If they're not there, it doesn't raise a nice LoadError at boot it fails at call time, per image, with a shell exit code.

Here's the part that stung: this is invisible to your Gemfile. bundle install is perfectly happy. MiniMagick the gem is installed. The dependency that's actually missing lives one layer down, in the operating system, and nothing in Ruby's dependency graph knows or cares.

The Dockerfile was telling the truth; I just hadn't read it

The runtime stage of the multi-stage Dockerfile installed exactly what Active Storage needed and nothing more:

RUN apt-get update -qq && \
    apt-get install --no-install-recommends -y curl libvips postgresql-client python3 libgomp1 && \
    rm -rf /var/lib/apt/lists /var/cache/apt/archives

libvips is there. imagemagick is not. So in the production container, identify genuinely did not exist.

Why didn't I catch it in development? Because my laptop has ImageMagick installed the way most dev machines accumulate it — from some Homebrew formula years ago, pulled in as a transitive dependency of something else. Locally, identify was on the PATH, MiniMagick was happy, images encoded, everything looked fine. The bug only existed in the one environment I couldn't see into by running identify in a terminal: the slim production image.

This is the recurring lesson of shipping Rails in Docker. Your production runtime is a separate contract from your Gemfile, and the slimmer you make the image, the more of that contract you're signing implicitly. Every --no-install-recommends is a small bet that you listed everything you actually need.

The fix is one word

 RUN apt-get update -qq && \
-    apt-get install --no-install-recommends -y curl libvips postgresql-client python3 libgomp1 && \
+    apt-get install --no-install-recommends -y curl imagemagick libvips postgresql-client python3 libgomp1 && \
     rm -rf /var/lib/apt/lists /var/cache/apt/archives

That's the entire change in PR #81 (b57fb72). I checked on ruby:3.3.5-slim first that apt-get install imagemagick actually provides identify (ImageMagick 6.9.11, about 30MB with dependencies), deployed, and watched the log line flip from 0/25 to items actually carrying image data into the model. Watermarking, which was silently broken the same way, came back at the same time.

One word in a Dockerfile. Weeks of a flagship feature quietly running at a fraction of its quality.

What I actually took away

The one-line fix is not the interesting part. Three things are.

Silent degradation is worse than a crash. A crash gets a Sentry alert, a red graph, a bug report. This got none of that, because from the outside the feature "worked" it returned outfits. The failure was in quality, which no exception tracker measures. If I'd had a single assertion that treated 0/25 items with images as an error condition instead of an info log, I'd have known in minutes, not weeks. Degraded-but-successful is the hardest failure mode to see, and it's exactly the one AI features love to hit.

Shell-out dependencies need a home in your health checks. A gem that wraps a system binary is a dependency your language tooling can't verify. The honest fix isn't just "add the package" it's making the runtime prove it has what it needs. A boot-time check that identify exists would have turned a silent production degradation into a failed deploy.

The real cleanup is to not need ImageMagick at all. libvips was already in the image for Active Storage, and it can do everything those two MiniMagick call sites need. The follow-up isn't to keep patching the runtime it's to migrate encode_item_image and WatermarkService off MiniMagick and onto ruby-vips, so the dependency I forgot to install becomes a dependency I no longer have. The best way to stop forgetting a thing is to delete the thing.

If you run a multimodal feature in production, go check your logs for the equivalent of 0/25. The model will never complain that you forgot to send it the picture. It'll just quietly do its best with half the input and hand you back something plausible which is the most dangerous thing an AI can do.

---

More OutfitMaker war stories in the same vein: the Gemini swap where Vertex refused to return images, and the TOCTOU race in my Gemini rate limiter.