DEV Community: Menshikov Vasil

How I Stopped Being Scared of Rails Migrations (and Learned to Love Zero-Downtime Deploys)

Menshikov Vasil — Mon, 10 Aug 2026 08:12:30 +0000

I used to have a little ritual before every risky deploy: write the up, write a matching down, and tell myself that if anything went wrong I could just roll it back. It was a comfort blanket. And like most comfort blankets, it was hiding the fact that I was cold. This is the story of how I learned that rails zero downtime migrations aren't about being able to undo things - they're about designing so you never need to.

Why this bugged me for years

The migration that finally broke the spell looked completely, boringly safe in review. It added a compound index to speed up product search:

class OptimizeProductSearch < ActiveRecord::Migration[7.0]
  def up
    add_index :products, [:category_id, :price, :created_at],
              name: 'idx_products_category_price_date'
  end

  def down
    remove_index :products, name: 'idx_products_category_price_date'
  end
end

The up built the index over 15-45 minutes under shared locks - slow queries, a sad p95, but survivable. The trap was the down. When a deploy went sideways and the pipeline cheerfully auto-rolled-back, the DROP INDEX grabbed an exclusive lock for a few seconds. On our busiest table, that was a brief but total outage. Sit with that for a second: the rollback - the thing I'd added specifically to keep us safe - was the incident. I found a writeup that argues exactly this point, that in high-load systems a rollback-first mindset manufactures the worst failure modes, and reading it was a little embarrassing, because it named a thing I'd been doing to myself for years.

Once I saw it, I saw it everywhere. Dropping an index or column freezes the fast path with exclusive locks at exactly the moment traffic is highest. Data migrations lose information, so recreating the original is wishful thinking. Rolling code back while the DB is half-migrated leaves a ghost state nobody can reason about mid-incident. And teams that fear schema changes quietly stop shipping the refactors they actually need - which is the slow, invisible cost that hurts most.

The irreversibility one deserves a concrete example, because people underestimate it. We had a migration that normalized free-form addresses into city, country, and postal_code, then dropped the original address column. The down gamely tried to reconstruct the address by string-joining the pieces back together - but the formatting, the ordering, the apartment numbers were simply gone. That migration was irreversible without data loss, full stop. Writing a fantasy down for it was worse than useless; it handed me false confidence I'd cash in at the worst possible time.

The thing that finally clicked

The fix turned out to be a discipline, not a gem. It's what Martin Fowler calls Parallel Change, and what database folks call expand and contract. Every change becomes a little sequence of small, individually-deployable steps, each one compatible with the app version before it. You expand - add new structures without blocking, and leave the old ones alone. You migrate - backfill, dual-write, move reads over gradually. Then you contract - remove the legacy stuff, but only after a full compatibility window has passed.

That compatibility window is the part everyone skips and everyone regrets. The new schema has to keep supporting the previous app version for at least one deploy cycle. Do that, and rolling code back is a non-event - the old code still runs perfectly against the expanded schema, no ghost state, no cold sweat. The whole trick is just refusing to collapse those three steps back into one clever migration, no matter how much your inner tidiness gremlin wants you to.

Dual-writes are the bridge

During the window I write to both the old and new structures, so either app version reads consistent data, and I move reads over slowly:

class User < ApplicationRecord
  has_many :user_statuses, -> { order(:effective_from) }

  def status=(new_status)
    if has_attribute?(:status)
      write_attribute(:status, new_status)  # legacy field
    end
    user_statuses.create!(                  # always maintain the new system
      status_type: UserStatus.status_types[new_status] || 0,
      effective_from: Time.current
    )
  end
end

When touching every write path felt too risky, I leaned on a database trigger to auto-sync legacy writes into the new table for the length of the rollout, then dropped the trigger in the contract phase. Either way the rule is the same, and it's the whole point: nobody, ever, reads a half-populated new table.

Never take a lock you can't afford

The other half of zero-downtime is refusing to grab a lock that'll hurt. The rule the Rails migrations guide and strong_migrations both nudge you toward: add nullable columns with no default (fast), backfill asynchronously in batches, then add the NOT NULL constraint in a separate later migration.

class AddColumnWithoutDowntime < ActiveRecord::Migration[7.0]
  disable_ddl_transaction!

  def up
    add_column :large_table, :new_field, :string  # fast, no default
    queue_background_migration('FillNewFieldJob')  # backfill in batches
    # change_column_null later, once backfill completes
  end
end

The background job walks the table in ordered batches and re-enqueues itself until it runs out of rows, so a table with tens of millions of rows never holds one long transaction or a table-wide lock. On Postgres the same instinct means building indexes concurrently - CREATE INDEX CONCURRENTLY skips the write-blocking lock, at the cost of a second table scan. Cheap trade. Take it.

The rehearsal that ended most of my incidents

Here's the embarrassingly cheap thing that fixed the most pain: I started running migrations against production-sized data in staging before they ever touched production. A change that runs in 40 milliseconds on a seed database can run for 40 minutes against 50 million rows, and there is no way to feel that difference on your laptop. So I'd generate a production-scale dataset, run each pending migration inside a transaction I rolled back for repeatability, and record duration, memory delta, and any blocking queries it kicked off. The output was a short report the team actually read before sign-off. Nine times out of ten the rehearsal caught the problem - a missing concurrent flag, an unbatched backfill - back when it was still boring to fix, which is the only time fixing anything is fun.

Make the safety rails automatic

Discipline that depends on everyone remembering is discipline that evaporates at 2 AM. So I pushed the rules into tooling. If you're on Rails, the strong_migrations gem already codifies most of this and is the fastest possible start - it fails the build on unsafe operations and prints the safe rewrite right there in your face. I layered a little duration estimator on top. A CI check refuses the deploy if a recent migration adds a NOT NULL column with no default, creates a blocking index, or is estimated to run long:

if total_estimated_time > 300  # 5 minutes
  unless ENV['FORCE_LONG_MIGRATION'] == 'true'
    puts 'Set FORCE_LONG_MIGRATION=true to proceed'
    exit 1
  end
end

I also instrument the migrations themselves - subscribing to sql.active_record notifications, logging any migration query over a second and pushing its duration to StatsD, so a slow migration shows up on a dashboard instead of in an incident channel. And before anything risky runs in production, a safety-net step snapshots the affected tables so there's a credible recovery point. Estimators, circuit breakers around big data migrations, a staging rehearsal task - you build the set once and reuse it across every service, and it pays you back forever.

How it feels now

Forward-only evolution didn't just change my migrations, it changed how the deploy button feels. Instead of fear I have a checklist: design for backward compatibility, expand first and contract later, split the scary change into small ones so data migrates asynchronously while reads drift over, and recover forward with fixes and checkpoints instead of fantasy rollbacks. None of it is clever. It's mostly the willingness to turn one terrifying migration into three boring ones - and I've made my peace with boring.

The part I didn't expect was cultural. When schema changes stopped being scary, the team stopped avoiding them, and all those long-postponed refactors we'd been quietly routing around finally shipped. That's the real payoff. If your database changes still page your on-call, start with just two things: the compatibility window, and the pre-deploy check. They're the two highest-leverage moves, and they're the ones that let you sleep.

Sources & further reading

Martin Fowler - Parallel Change (expand and contract): https://martinfowler.com/bliki/ParallelChange.html
strong_migrations - catch unsafe migrations in development (GitHub): https://github.com/ankane/strong_migrations
PostgreSQL - CREATE INDEX, including the CONCURRENTLY option: https://www.postgresql.org/docs/current/sql-createindex.html
Rails - Active Record Migrations guide: https://guides.rubyonrails.org/active_record_migrations.html
A thorough end-to-end zero-downtime schema evolution playbook someone published, estimators and staging rehearsal task included: https://dorokhovich.com/blog/rails-database-schema-evolution?utm_source=devto&utm_medium=article&utm_campaign=success-story&utm_content=rails-database-schema-evolution

The Bug That Made Me Fall Back in Love With rack.response_finished

Menshikov Vasil — Sun, 09 Aug 2026 08:50:43 +0000

There is a particular kind of production bug that doesn't feel like a bug. It feels like weather. Every few days something leaks, a worker gets cranky, p99 creeps up during a traffic spike, and you shrug and restart it because it clears up on its own. For the better part of a year that was my relationship with rack.response_finished - except I didn't know that was the name of my problem yet. I just knew our Rails app was quietly bleeding resources and I kept blaming the wrong things.

This is the story of how it finally clicked, why it bugged me for so long, and how I migrated a high-traffic app to rack.response_finished without spending a single night watching a dashboard with my stomach in knots.

Why this bugged me for years

Our symptoms were the vague kind that make you feel a little crazy. Occasional resource leaks under load. Tail latency that crept up when traffic spiked and settled back down when it didn't. Metrics that never quite matched what I believed was happening. We streamed large files and Server-Sent Events, so the Rack triplet [status, headers, body] was returning long before the last byte ever reached the client.

Here's the piece I genuinely did not appreciate for months: our middleware was freeing resources - closing DB connections, clearing caches, wiping thread-locals - well before the client had the full response. I found a lovely deep-dive on exactly what happens between returning a Rack triplet and delivering that last byte, and reading it was the moment the fog lifted. All those vague symptoms suddenly had one concrete cause. We'd been treating a lifecycle bug as an infrastructure flake - restarting workers, bumping pool sizes, side-eyeing the load balancer. It was none of that. It was an assumption I'd baked into how we wrapped response bodies, and I'd never once questioned it.

The thing I'd been doing without thinking

For years, if you wanted a "run this when the request is done" hook, you wrapped the body in Rack::BodyProxy. It's a tidy little object that calls your block when the wrapped body closes. Elegant on paper. The trouble is every middleware that wanted a callback added its own wrapper, so a single request ended up dragging a Russian-doll stack of proxies around:

original_body = ["Hello World"]
body = Rack::BodyProxy.new(original_body) { logger.info "Request finished" }
body = Rack::BodyProxy.new(body) { metrics.record_latency }
body = Rack::BodyProxy.new(body) { cleanup_thread_locals }
body = Rack::BodyProxy.new(body) { close_db_connections }

The Shopify Rails infrastructure folks wrote up this exact pain in "Friendship Ended with Rack::BodyProxy", and honestly it was validating to read, because it named three things that had been nagging at me.

First, allocation pressure. Every proxy is one more object. At tens of thousands of requests a second, even tiny per-request allocations add up - and because these callbacks capture closures, the objects hang around long enough to get promoted into older GC generations, which is exactly where you don't want churn.

Second, timing you can't trust. #close gets called by the server, sure, but the spec never promised it happens after the client has everything. Depending on the server and buffering, my callbacks could fire before, during, or after transmission. I was cleaning up at a moment I couldn't actually pin down.

Third - and this is the one that was actually paging me at 3am - exceptions skipped cleanup entirely. If the body raised while iterating, the proxy's callback might just never run:

class ProblematicBody
  def each
    yield "Part 1"
    raise "Something went wrong"  # BodyProxy#close might not be called
    yield "Part 2"
  end
end

That right there was the leak. A stream raises halfway through a big download, the cleanup silently gets skipped, and the resource just... stays open. Multiply by traffic and you get weather.

The thing that finally clicked

The replacement is so much calmer, and I mean that emotionally as much as technically. Instead of N proxy objects, there's one standard key in env holding an array of callbacks:

def call(env)
  callbacks = env["rack.response_finished"] ||= []
  callbacks << lambda do |env, status, headers, error|
    # Runs AFTER complete response delivery
    cleanup_resources
    log_metrics(status, headers)
  end

  @app.call(env)
end

What sold me wasn't the elegance, it was the promise. Unlike BodyProxy#close, these callbacks are guaranteed to run in three cases: a clean finish after all data is sent, an application exception even if the body never started iterating, and a server exception during network trouble. Each callback gets (env, status, headers, error), and per Rack's Lint spec they fire in reverse registration order, so you can branch on what actually happened. It shipped as part of Rack 3.x, and Puma added real server-side support (puma#3681) so it isn't a quiet no-op in production.

The one guarantee I kept coming back to: the callbacks run even when the body raises mid-stream. That single sentence killed my entire class of leaks. I remember reading it and feeling almost annoyed at how simple the fix was.

How I migrated without holding my breath

I couldn't flip everything at once - our services spanned Rack versions, and I'm allergic to big-bang changes on infrastructure I can't fully see. So the critical middleware learned to speak both dialects and pick whichever the server offered:

class SafeMigrationMiddleware
  def initialize(app)
    @app = app
  end

  def call(env)
    status, headers, body = @app.call(env)

    if env["rack.response_finished"]
      register_new_callback(env)
    else
      body = wrap_with_proxy(body)  # fallback
    end

    [status, headers, body]
  end

  private

  def register_new_callback(env)
    callbacks = env["rack.response_finished"] ||= []
    callbacks << method(:cleanup_resources)
  end

  def wrap_with_proxy(body)
    Rack::BodyProxy.new(body) { cleanup_resources }
  end

  def cleanup_resources(*)  # accepts any number of args
    logger.info "Request completed"
  end
end

The cleanup_resources(*) splat is on purpose - the new API hands you four arguments, the proxy hands you none, and a splat lets one method serve both paths without a branch. Small thing, but it made the diff read cleanly, which matters to me more than I'll admit.

Then I refused to guess. I put a StatsD counter on each path so I could watch, in real numbers, how much traffic had moved to the new mechanism:

if env["rack.response_finished"]
  StatsD.increment('middleware.response_finished.new_api')
  register_new_callback(env)
else
  StatsD.increment('middleware.response_finished.fallback')
  body = wrap_with_proxy(body)
end

The mistake I almost shipped

Callbacks run in the same thread as the request. That makes thread-local cleanup delightful, and it makes slow work a trap. My first draft casually fired off a notification email inside the callback, which is a wonderful way to block a worker for seconds at a time. Fast cleanup belongs in the callback; anything with I/O belongs in a background job:

# Good: fast cleanup
callbacks << lambda do |env, status, headers, error|
  Thread.current[:request_id] = nil
  ActiveRecord::Base.clear_active_connections!
end

Rails itself gets happier here too: ActionDispatch::Executor can now reliably clear thread-locals right after the response completes, and gems like rack-timeout, newrelic_rpm, skylight, and sentry-ruby line up their timing and errors far more accurately.

How it feels now

I want to be honest about the size of the win, because it's easy to oversell a refactor you're proud of. Average latency barely moved - most requests were never the problem. What changed was the shape of things. Objects allocated per request dropped on our streaming endpoints, major GC ran a little less often, and the p99 tail got quieter during spikes because fewer long-lived closures were being promoted into old-gen. And the leak alerts that used to page me during big downloads simply stopped, because the cleanup now runs even when the body raises.

The order of operations mattered as much as the code. I audited every middleware that touched BodyProxy, bumped Rack to 3.x in development first, shipped the dual-API middleware, and let the StatsD counters tell me when the new path dominated. I only pulled the fallback after a service had run for weeks at effectively 100% new-API with zero callback errors. I left it in longer than felt necessary, on purpose - rushing that last step is exactly how you turn a calm migration into a scary one. Profiling before and after with memory_profiler and ruby-prof gave me actual numbers instead of a vibe, which is the difference between "trust me" and a graph.

If I could hand one note back to the version of me who kept restarting workers: the leak was never infrastructure. It was an abstraction I'd stopped questioning because it looked so tidy. BodyProxy was elegant right up until it wasn't, and the replacement is simpler and more honest about what it promises. That combination - less clever, more trustworthy - is the trade I'll take every single time now. If you run a busy Rack app, start the dual-API pattern today; you get to learn the new mechanism at zero risk, and future-you gets to sleep.

Sources & further reading

Rack - Add rack.response_finished to Rack::Lint (PR #1952, the spec and reverse-order contract): https://github.com/rack/rack/pull/1952
Rack CHANGELOG (where rack.response_finished and Rack 3.x changes land): https://github.com/rack/rack/blob/main/CHANGELOG.md
Puma - Add support for rack.response_finished (server-side implementation, PR #3681): https://github.com/puma/puma/pull/3681
Rails at Scale - "Friendship Ended with Rack::BodyProxy" (Shopify's writeup of the same migration): https://railsatscale.com/2025-08-26-friendship-ended-with-rack-bodyproxy/
Rack::BodyProxy API documentation (RubyDoc): https://www.rubydoc.info/gems/rack/Rack/BodyProxy
A production migration writeup someone put together, with the full dual-API middleware and monitoring metrics laid out: https://dorokhovich.com/blog/rack-response-finished?utm_source=devto&utm_medium=article&utm_campaign=success-story&utm_content=rack-response-finished

How I Rescued My Broken Local Kubernetes Development Environment in One Afternoon (Docker + k3d + Tilt)

Menshikov Vasil — Sat, 08 Aug 2026 17:16:58 +0000

There's a specific flavor of shame in a local Kubernetes development environment that only you can't get working. Everyone else on the team runs kubectl get pods and gets an answer. You run it and get connection refused, again, and you quietly start to wonder if the problem is you. It wasn't me, it turned out - it was that I'd never actually built the thing, I'd just accreted it. This is the story of how I tore my broken setup down and rebuilt it, deliberately, in a single afternoon.

Why this bugged me for years

I'd inherited a FastAPI service - Python 3.12, listening on 8080, backed by PostgreSQL - that the team ran on "local Kubernetes." That phrase meant something different on every laptop, and mine was easily the worst offender. kubectl installed three different ways. A Docker Desktop that half-started on a good day. A helm binary that wasn't on my PATH. And, crucially, no cluster at all. Every single kubectl command greeted me with the same connection refused, and I'd learned to just... route around it, which is the worst possible response and exactly the one I kept choosing.

I burned two full mornings guessing. Then I got tired of guessing and followed a single opinionated write-up that treats the workstation itself as the deliverable - the idea being that you're not done until kubectl get pods -A responds without errors against a real local cluster. That reframing did something to my brain. If you want the full chapter with every OS variant, someone wrote up the exact workstation setup here, and it's the map I wish I'd had two mornings earlier.

The thing that finally clicked: install bottom-up

The mindset shift that fixed everything was embarrassingly simple - install in dependency order. Docker first, because nothing runs without it, then kubectl, k3d, helm, and Tilt last. On my Mac that collapsed to almost one command:

# Docker Desktop from docker.com (daemon + GUI), then:
brew install kubectl helm k3d tilt

k3d is the quiet hero of this whole story. It's just k3s packaged into Docker containers - k3s being a lightweight, CNCF-certified Kubernetes distribution - so a local cluster is featherweight next to "real" Kubernetes. If you're agonizing over k3d vs kind vs minikube, here's what decided it for me: k3d spins up a cluster in under five seconds, ships a built-in image registry that Tilt adores, and barely touches RAM, whereas minikube boots a whole VM and kind - Kubernetes-IN-Docker, originally built to test Kubernetes itself - sits somewhere in the middle. For a tight inner dev loop I rebuild dozens of times a day, and those saved seconds compound into genuinely real time. The one hard requirement is Docker 20.10.5 or newer; k3d's own repository lists a working Docker install as the sole prerequisite. Everything else is just host headroom - 8 GB of RAM is a comfortable floor, 16 GB means you never think about it, and you'll want 10 to 20 GB of free disk for images and layers.

On Linux the shape is identical, except you install Docker Engine with no GUI and you must add yourself to the docker group or every command demands sudo:

sudo usermod -aG docker $USER
# then log out and back in so the group takes effect

One caution I learned the hard way: those curl ... | bash installers for k3d, helm, and Tilt are convenient, but you're piping someone else's code from the internet straight into a shell. On a work machine I now download the script, skim it, then run it - or reach for the apt/dnf packages the official docs offer for kubectl and helm, which are cleaner to maintain long-term anyway.

Naming what each tool actually does

Here's the confession at the heart of this: before that afternoon, I genuinely could not have told you which tool did what, and that fuzziness is exactly why my setup was so fragile. The mental model that finally stuck goes like this. Docker builds images and runs the containers everything else sits inside. k3d spins up the cluster itself - k3s in Docker - and tears it down just as fast. kubectl and helm manage resources and packaged charts inside that cluster. Tilt runs the fast inner dev loop, watching your files and rebuilding and redeploying on every save, and its Live Update can even sync changed files straight into a running container, collapsing edit-build-push-deploy into seconds. And k9s is the optional terminal UI that replaces a wall of kubectl commands.

Once I could name the job of each binary, the errors stopped being mysterious and started being addressable. A connection refused is Docker. A stuck rebuild is Tilt. A missing chart is helm. The abstraction layers finally lined up in my head, and honestly that was more valuable than any single command I learned.

The smoke test that ends the guessing

This is the part I wish I'd run on day one instead of day three. Rather than debugging four tools in isolation, you prove the whole chain works together at once: create a throwaway cluster, switch to it, look at the system pods, delete it.

k3d cluster create dev
k3d kubeconfig merge dev --kubeconfig-switch-context
kubectl get pods -A
k3d cluster delete dev

When that kubectl get pods -A printed coredns, traefik, and metrics-server all sitting in Running, I actually said it out loud to my empty room: the workstation was ready. That single sequence is the entire difference between "I think it's installed" and "I proved it works," and I've never set up a machine without it since. Naming the cluster dev and reusing it later is deliberate too - k3d makes clusters so cheap to create and destroy that a named, reusable dev cluster becomes the stable target for the rest of your toolchain.

The gotchas that used to eat my mornings

Almost every failure at this stage is one of a small handful of classics, and just running the list saved me hours. The most common by far is simply that the Docker daemon isn't running - k3d and Tilt both die with connection refused, so start Docker Desktop or sudo systemctl start docker and wait for it to fully come up. After a reboot you may hit port already in use, in which case just recreate the cluster.

The trickiest one, and the one that bit me hardest, is too many open files. Pods and Tilt both watch files, and they collide with system inotify limits; the symptom shows up as too many open files in pod logs or the kubelet. On a Linux or Docker host the fix is:

sudo sysctl fs.inotify.max_user_watches=1048576
sudo sysctl fs.inotify.max_user_instances=8192

Make it survive reboots by putting those in a file under /etc/sysctl.d/, and as a fallback for stubborn machines you can create a k3d cluster with only a server node and no agents, which cuts the total open files.

Windows has its own trap: do all your dev work inside WSL2, including the repo itself. A teammate kept the code on C:\ and ran the tools from WSL2, and the result was a crawling filesystem and Tilt file sync that silently refused to trigger uvicorn --reload. Move the repo into the WSL2 home directory and the whole problem evaporates. And two quieter ones round it out - if Docker Desktop refuses to start with a virtualization complaint, enable hardware virtualization (VT-x / AMD-V, plus SLAT for WSL2) in BIOS/UEFI, and if you try to squeeze into 4 GB, expect OOM once PostgreSQL and a few pods pile in. Give Docker 8 GB or more under Settings then Resources and the random pod deaths stop.

The little upgrade: k9s made me delete half my muscle memory

Instead of hammering kubectl get pods, kubectl describe, and kubectl logs all day, I now open k9s - a terminal UI that continually watches the cluster - and navigate with arrow keys. Logs, restarts, deletes, all from one screen.

brew install derailed/k9s/k9s

For anyone still finding their feet, it's a genuinely lovely visual replacement for dozens of repetitive commands, and I felt a small pang deleting keystrokes I'd spent years memorizing.

How it feels now

The change is night and day, and it's less about any single number than about the shift from flailing to knowing. What used to be two mornings of guessing is now a checklist-driven afternoon. kubectl get pods -A returns system pods in Running instead of connection refused. Tilt hot reload, which used to be silently broken behind that WSL2 path issue, now fires on every save. Debugging a pod is arrow keys in k9s instead of three kubectl invocations each time. And the biggest one, the one that's hard to put in a table: I'm actually confident it works now, because I watched the smoke test pass with my own eyes.

With the toolchain proven, the natural next step is spinning up a real dev cluster for the service - built-in registry, port forwarding, the works - rather than a throwaway. But if you're setting this up for the first time, please don't improvise the way I did for two mornings. Install bottom-up, run the smoke test before you touch a line of application code, and keep the gotcha list within reach.

The thing that genuinely stuck with me is this: a local Kubernetes environment is not "done" when the binaries are installed. It's done when you've watched the smoke test pass. I'd spent years treating my dev environment as something that accreted rather than something I built, and the fix was simply to build it on purpose, prove it, and only then start stacking real work on top. Prove it first. Everything after that is easier.

Sources & further reading

k3d - k3s in Docker, official documentation: https://k3d.io/
k3d install requirements and cluster options (GitHub): https://github.com/k3d-io/k3d
kind - Kubernetes IN Docker, official documentation: https://kind.sigs.k8s.io/
Tilt Live Update tutorial (official docs): https://docs.tilt.dev/tutorial/5-live-update.html
k9s - Kubernetes terminal UI (GitHub): https://github.com/derailed/k9s
A full field-tested workstation write-up someone put together, with every OS variant and the complete gotcha list

A Local Kubernetes Tools Comparison That Finally Unfroze Me: Stop Comparing Them, Sort Them Onto Shelves

Menshikov Vasil — Fri, 07 Aug 2026 08:53:58 +0000

Here is the confession I don't love making: I once lost an entire week to a decision that should have taken an afternoon. My team wanted local Kubernetes, I volunteered to pick our stack, I opened a "getting started" guide, and got hit with an incantation - k3d, kind, minikube, Helm, Kustomize, Tilt, Skaffold, DevSpace, Telepresence, mirrord, k9s. I froze. Every local Kubernetes tools comparison I read stacked a dozen names side by side as if they were rival answers to one question, and I couldn't pick anything. What eventually unfroze me wasn't a better comparison - it was realizing they don't belong on the same shelf.

Why this bugged me for a whole week

I'm a fairly decisive person, which made the paralysis extra humiliating. I figured picking a stack would be a lazy afternoon of reading. Instead I kept hitting the same wall: do I need k3d or Helm? Is Tilt an alternative to minikube? Where on earth does Telepresence fit? Every post read like one giant menu where everything competed with everything, and I genuinely could not tell whether I was choosing one tool or ten.

The reframing that saved me came from an overview of the local Kubernetes tooling landscape someone wrote up: each tool handles its own little piece, and once you sort them onto shelves, the whole picture goes quiet. Let me hand you the shelves, because they cost me a week and they might save you one.

The first fork nobody told me about: local cluster vs remote connection

Before you compare any specific tool, there's a bigger split I didn't even know existed. There are two fundamentally different ways to get a near-production loop. The first is to run all of Kubernetes on your own machine - spin up a small but real cluster locally with k3d, kind, minikube, Docker Desktop, or Rancher Desktop, and deploy with the same manifests you'll use in prod. Fully local, no network dependency. That's the path we took. The second is to keep your code local but plug it into a remote cluster, so your local process behaves as if it were a Pod living on staging. That's the world of Telepresence (which builds a VPN-style tunnel, needs root, and modifies the cluster), mirrord (which injects into your process via LD_PRELOAD or DYLD_INSERT_LIBRARIES, needs no root, and mirrors by default so it's safe for shared staging), and Gefyra (VPN-based, no root, Docker-only).

The remote approach is genuinely great, but it needs a live remote cluster to maintain and network access - an extra layer of magic I didn't want to impose on a team still finding its feet. So: everything local. That one decision alone eliminated three tools from my "must choose now" list, and I felt the paralysis loosen its grip a little.

The foundation shelf: Docker and kubectl

These sit under everything else. Docker is the container engine, and here's the fact that surprised half my team: local clusters run their Kubernetes nodes as ordinary Docker containers. A node is a container with Kubernetes inside it, and inside that run your Pods. No Docker, no cluster. kubectl, meanwhile, is the official CLI and your main channel to the cluster - apply manifests, read logs, forward ports, shell in. Every accelerator like Tilt or Skaffold and every visual tool like k9s runs on top of kubectl, calling the same Kubernetes API underneath, so it's worth actually knowing even if you end up living inside Tilt all day. A small bonus that trips people up: Kustomize is built right into kubectl via kubectl apply -k.

The cluster shelf: k3d, kind, minikube, Docker/Rancher Desktop

This shelf holds the "box" your app goes into, and all of these give you a real Kubernetes API - they just differ in startup speed, resource appetite, and features. k3d is a wrapper that runs k3s, a minimal certified Kubernetes, inside Docker; it has the fastest startup, the lowest memory footprint, and a handy built-in registry, which makes it my pick for everyday dev (with the honest caveat that it's a community project, not an official Rancher/SUSE product). kind, Kubernetes IN Docker, runs upstream Kubernetes via kubeadm - CNCF-certified, multi-node and HA capable, and built to test Kubernetes itself and CI conformance. minikube runs Kubernetes in a VM or via a docker driver with a big addon ecosystem and remains the classic learning tool. Docker Desktop offers single-node Kubernetes behind a checkbox, convenient if you already run it but limited in configuration. And Rancher Desktop is a GUI app on k3s for native local dev and testing. The 2025-2026 consensus lands on kind or k3d for the best balance of speed and capability, and k3d in particular starts fast and stays light, which is exactly why it's so pleasant to recreate constantly while you're still learning and breaking things.

The manifest shelf: Helm and Kustomize

Production manifests usually live as Helm charts or Kustomize overlays, and this is the whole reason a local cluster beats docker-compose - it accepts these same formats. Helm is the Kubernetes package manager: your app becomes a chart with a Chart.yaml, a values.yaml, and a templates/ directory of Go templates, and in return you get versioning, releases, rollbacks, dependencies, and repos. It's been a CNCF graduated project since 2020. Kustomize takes the opposite bet - no templates at all; you write base manifests and apply overlays, which are patches, per environment. It's built into kubectl since 1.14, though on its own it has no packaging, versioning, or rollback, so people bolt on Argo CD or Flux for that. In practice the two complement each other: Helm to package and distribute, Kustomize to tidily patch per environment.

The accelerator shelf: Tilt, Skaffold, DevSpace

The bare loop - edit, build, push, deploy, look - is excruciatingly slow in Kubernetes, and this shelf exists to automate the building, updating, and watching. Tilt (Apache-2.0) automates watch-build-update, and its signature move is Live Update, syncing changed files straight into the running container without a full rebuild; you configure it in a Tiltfile written in Starlark, a Python-like language, and it ships a genuinely clear web UI that's a big plus for beginners. Skaffold, from Google, is a build/push/deploy pipeline with file sync, a dedicated skaffold debug command, and profiles, deploying via kubectl, Helm, or Kustomize. DevSpace is a CLI that leans into bidirectional sync, hot reload, and in-cluster dev containers. Two adjacent names worth not confusing with these: Okteto is about cloud dev environments, which is closer to the remote approach, and Garden is graph-based multi-service automation, which is overkill for a single service.

The convenience shelf: k9s and friends

Once you have more than one Pod, typing kubectl get, logs, and describe all day gets old fast. k9s is a terminal UI - a "visual kubectl" - with vim-style navigation, live Pod, node, and deployment views, and hotkeys (l for logs, s for shell, d for describe). My favorite touch: it color-codes contexts, so you can paint prod an alarming red and never wreck it by muscle memory again. Also worth installing when the day comes are kubectx and kubens for fast context and namespace switching, stern for multi-pod log streaming, and krew, the kubectl plugin manager.

Why we landed on k3d + Tilt

Once the tools were on shelves, the choice basically made itself, because local Kubernetes dev really only has two core pains: you need a realistic cluster that accepts real production manifests, and you need a fast edit-to-result loop. k3d covers the first - a real Kubernetes API via k3s, fast startup, modest memory, so we get genuine parity with our Helm and Kustomize manifests instead of a docker-compose surrogate, and without turning laptops into space heaters. It's cheap to recreate, which matters enormously while a team is learning. Tilt covers the second - Live Update compresses the loop from minutes to seconds, codifies the whole setup in a Tiltfile, deploys through real manifests, and shows a web UI where builds, logs, and state all live in one place. Together they're fully local, open source, and free.

The alternatives are all worthy, and I want to be fair to them: Skaffold and DevSpace solve the same loop and are worth trying if their style fits your team, Telepresence and mirrord are the remote approach if you'd rather lean on a shared cluster, and Okteto and Garden solve adjacent problems entirely. For a single service headed toward its first deploy, k3d + Tilt was simply the shortest, clearest path.

How it feels now

The before-and-after is almost embarrassing in how much of it was in my head. Before the map, my mental model was "ten rival tools, one agonizing choice," the decision took a paralyzed week, and the stack stayed stubbornly undecided the whole time. After the map, the tools sat on distinct shelves, the decision collapsed into an afternoon, and I had a stack - k3d + Tilt - that I could actually explain the reasons for.

What stays with me is why I was stuck, because it wasn't complexity. I was overwhelmed because I kept comparing a cluster to a package manager to a file-syncer as though they answered the same question, and they simply don't. The moment I stopped ranking them and started sorting them - foundation, cluster, manifest, accelerator, remote, convenience - the choice made itself. If you're standing at the same frozen starting line, don't reach for one more comparison table. Reach for shelves. That reframing is the thing I'd hand back to the paralyzed version of me who lost a week he didn't need to.

Sources & further reading

How Tilt's Live Update Fixed My Kubernetes Dev Loop and Gave Me an Hour a Day Back

Menshikov Vasil — Thu, 06 Aug 2026 09:29:19 +0000

Change one line of Python. docker build. docker push. kubectl rollout restart. Wait for the pod. Check the logs. Repeat, forty times a day. I actually did the arithmetic on what that ritual was costing me one afternoon, and the number was ugly enough that I nearly gave up developing on Kubernetes entirely. Instead I found Tilt's Live Update, rebuilt my Kubernetes dev loop around it, and got roughly an hour a day back. This is how that went.

Why this bugged me for weeks

My team had just moved local development onto a k3d cluster for parity with prod. Good call in principle - and in practice my day looked like this for every single code change to our FastAPI service myapp:

docker build -t k3d-registry.localhost:5000/myapp .
docker push k3d-registry.localhost:5000/myapp
kubectl rollout restart deploy/myapp -n myapp
kubectl logs -f deploy/myapp -n myapp   # wait, watch, hope

A minute or two, minimum, per edit. Ten edits and half an hour of my life had dissolved into progress bars. But the raw time wasn't even the worst of it - the context switching was. By the time the pod finally came up, I'd forgotten what I was checking in the first place. Kubernetes had taken my tight, genuinely joyful edit-run loop and turned it into molasses, and I could feel my patience for the whole platform draining by the day.

I was one bad afternoon away from ripping the app back out of the cluster - and throwing away all the parity we'd just bought - when I came across a write-up on using Tilt for a fast local Kubernetes dev loop. It described the exact pain I was drowning in, and it gave me a way out.

The three-line Tiltfile that started it

Tilt's config is a file called Tiltfile (no extension) written in Starlark, which is basically a trimmed-down Python. Run tilt up and it executes top to bottom, builds a graph of what to build and deploy, then watches your files and rebuilds only what changed. My first working version was almost embarrassingly short:

# Tiltfile

# How to build the myapp image from the current directory
docker_build('k3d-registry.localhost:5000/myapp', '.')

# What to deploy — our existing manifests
k8s_yaml(['k8s/deployment.yaml', 'k8s/service.yaml'])

# Fine-tuning: forward the container's port 8080 to localhost:8080
k8s_resource('myapp', port_forwards='8080:8080')

Three functions carry almost any Tiltfile, all documented in Tilt's API reference: docker_build for how to build, k8s_yaml for what to deploy, and k8s_resource for the fine-tuning like port-forwards and grouping. The clever bit is how Tilt ties them together - it scans the YAML, finds the workload, matches the built image to the manifest by tag, swaps in a uniquely-tagged fresh build, and deploys. The one rule I had to internalize immediately: the tag in docker_build must exactly match the image: in deployment.yaml. Mismatch it and Tilt cheerfully builds one image while the Deployment asks for another - instant ImagePullBackOff, and a confusing ten minutes before you realize what you did.

After tilt up, my service was live at localhost:8080 with no manual kubectl port-forward. Already better. But the real prize came next.

Live Update: the feature that gave me my day back

This is the part that actually changed things. The normal rebuild-image-then-redeploy-pod cycle takes tens of seconds even for a tiny service - Docker layers, registry push, cluster pull, pod restart, all of it. Live Update skips the whole dance: instead of rebuilding, Tilt copies changed files directly into the running container and, if needed, runs commands there. Seconds, not minutes.

You configure it inside docker_build via live_update, and the steps run in a strict order:

docker_build(
    'k3d-registry.localhost:5000/myapp',
    '.',
    live_update=[
        # if deps change, just rebuild the whole image
        fall_back_on(['requirements.txt']),
        # sync code instantly into /code/app (the WORKDIR from our Dockerfile)
        sync('./app', '/code/app'),
        # reinstall deps only if that file changed
        run('pip install -r requirements.txt', trigger=['requirements.txt']),
    ],
)

Tilt's decision tree is refreshingly simple, and the Live Update reference spells out the exact ordering: a fall_back_on file changed means a full rebuild; a file matched by a sync gets a fast live update; a file in the context but covered by no sync triggers a full docker build; an untracked file does nothing. Two rules tripped me up until I read them twice. sync paths must live inside the build context - if Tilt is watching it, you can sync it - and run() cannot come before sync(), because you have to put the files in place before you act on them. Also worth knowing: the very first deploy is always a full one, since Live Update needs an already-running container to copy into.

Hot reload closes the loop

Live Update drops the new file into the container, but how does the running process actually pick it up? For us, delightfully, FastAPI via uvicorn hot-reloads on its own. The launch command:

uvicorn app.main:app --host 0.0.0.0 --port 8080 --reload --reload-dir app

Per uvicorn's settings docs, --reload runs an internal watcher that restarts the process when .py files change, and --reload-dir app narrows the watch so it doesn't twitch on every temp file. So when your framework self-reloads, sync alone is enough: files land in the container, uvicorn's watcher notices, the app reloads, and there's nothing else to wire up.

If your stack can't hot-reload - a Go binary, or uvicorn without --reload - the synced files just sit there while the old process runs stale code, which is the classic "my changes didn't apply" trap that'll cost you an afternoon of confusion. For that case, Tilt's restart_process extension gives you docker_build_with_restart, which re-runs your entrypoint after each sync. For our FastAPI service plain --reload was simpler, and since it's dev-only, we never ship --reload in the production image.

The dashboard I didn't know I needed

tilt up also brings up a web UI at localhost:10350, and it quietly ended my tab-juggling. On one screen I get every resource with two statuses - did the build and deploy succeed, and what is the pod doing right now - along with the Pod ID behind a copy button, endpoints as clickable links, and the part I use constantly: filterable logs, by build versus runtime, by level, by keyword. When a pod won't start, I stopped grep-ing through kubectl logs across three terminals and just filtered the dashboard. There's a Trigger Update button for manual rebuilds, and a manual mode if you'd rather save often and deploy on demand:

trigger_mode(TRIGGER_MODE_MANUAL)
k8s_resource('myapp', trigger_mode=TRIGGER_MODE_AUTO)  # but keep this one automatic

Is Tilt the only option? My honest take

Tilt isn't alone here - it usually gets compared with Skaffold, from Google, and DevSpace. All three build, deploy, and sync for a fast loop, and the real differences come down to UI-first versus CLI-first and Starlark versus YAML. Tilt bets on the visual dashboard and Live Update; the Starlark config is flexible but a steeper climb than YAML, and it shines for beginners and mixed-experience teams precisely because you can see the whole cluster state. Skaffold is CLI-only with YAML config, file sync, profiles, and a dedicated skaffold debug - familiar and declarative, but no visual panel. DevSpace is a YAML CLI with two-way sync, reverse port-forward, and dev containers.

If your team lives in YAML and loves the terminal, Skaffold or DevSpace is a great fit, and I won't pretend that choice is anything but subjective. For getting a newcomer developing comfortably on Kubernetes, though, the dashboard won it for me - less blind fumbling in the terminal, more actually understanding what's happening.

How it feels now

The per-edit time went from a minute or two down to a second or two, and the list of things I run by hand collapsed from build-push-restart-logs to simply saving the file. The image only rebuilds when dependencies change now, not on every keystroke. Logs and status live on one dashboard instead of scattered across N terminals. The port-forward is declared once in the Tiltfile instead of manually re-established every time a pod restarts. And my focus, which used to get shredded on every deploy, stays intact.

The arithmetic that sold me was blunt: forty edits times roughly ninety seconds saved is about an hour a day I got back, and that's before you count the context-switching tax, which I suspect was the bigger number anyway. The whole setup took a single afternoon.

What lingers, though, isn't the hour. It's that the pain was never Kubernetes - it was my loop. A tiny Tiltfile, Live Update, and uvicorn's --reload turned the whole experience from molasses back into the tight edit-run rhythm I'd been quietly grieving, and I got to keep cluster parity while doing it. If you're still rebuilding images by hand and watching progress bars, that's the trade I'd make again in a heartbeat: an afternoon of setup for a loop that finally respects your attention.

Sources & further reading

The Kubernetes Inner Dev Loop That Was Quietly Eating My Sprint - and How I Got Back to Seconds

Menshikov Vasil — Wed, 05 Aug 2026 10:05:16 +0000

For about six months, my Kubernetes inner dev loop was a productivity black hole, and I mean that almost literally: time went in and nothing came out. I was the backend person on a small payments squad, and I got to the point where I quietly dreaded touching the cluster at all. This is the story of what was actually wrong, why it bugged me for so long, and the setup that finally gave me back a dev loop measured in seconds instead of minutes.

Why this bugged me for so long

Our service - I'll just call it myapp, a FastAPI HTTP API on port 8080 backed by PostgreSQL - ran beautifully on a laptop. uvicorn --reload picked up every change in a fraction of a second, the way good local dev should feel. Then we "just" shipped it to Kubernetes, and the instant loop turned into this:

# 1. build the image
docker build -t registry.internal/myapp:dev .
# 2. push to the registry
docker push registry.internal/myapp:dev
# 3. apply the manifests
kubectl apply -f k8s/
# 4. wait for the new Pod and check it's alive
kubectl rollout status deployment/myapp
kubectl logs -f deployment/myapp

Four steps that simply did not exist when we ran things locally. Every single pass was two to five minutes of build, push, wait, check. Multiply that by dozens of iterations a day across the team and you get whole afternoons of people staring at kubectl rollout status. And here's the part that really got under my skin: the wait was juuust long enough that I'd flip to Slack every time. So the true cost was never the four minutes. It was the train of thought that derailed on every deploy, and never quite got back on the rails.

A slow loop is only half the misery, though. The other half is the bugs you genuinely cannot see on a laptop. We shipped a change that passed every local test and then watched the Pod get OOMKilled in staging, because we'd never set a memory limit locally - memory is effectively infinite on a dev box. Another release sat there "running but not responding" because a readiness probe was failing and Kubernetes had quietly yanked the Pod out of the Service endpoints. Nothing in my local world had prepared me for either.

The idea that reframed everything: shift-left, but with fidelity

I got tired of band-aids and went looking for a systematic answer. Someone had written up exactly why the Kubernetes inner loop gets slow and how to close the local-versus-cluster gap, and it put a name to the thing I'd been feeling in my gut: the more your test environment differs from prod, the more bugs leak downstream, where each one costs an order of magnitude more to fix.

The line that really stuck with me was this - testing earlier in a CI container that doesn't match your cluster isn't shift-left. It's just failing faster in the wrong environment. Real shift-left for Kubernetes means validating against real cluster conditions: real resource limits, real probes, real services in the namespace, early, on your own machine. That reframed the whole problem for me, and it pointed at two concrete tools instead of a vague "do better."

The first is k3d, a lightweight wrapper that runs CNCF's k3s inside Docker. Not an emulation of a cluster - an actual one, small enough to live on a laptop. The second is Tilt, whose live-update performs an in-place update of the containers in your cluster, syncing your code straight into a running Pod so iteration drops back to seconds.

What I actually rolled out

I named the cluster dev, put everything in a myapp namespace, and used k3d's built-in registry at k3d-registry.localhost:5000. My first lesson landed the hard way: the local cluster cannot see images from your local Docker daemon. I built myapp:dev, expected the cluster to pick it up, and got a Pod wedged in ImagePullBackOff instead. k3d nodes run their own containerd, isolated from your Docker. You either push to a registry the cluster can reach or import the image explicitly - the k3d registries guide walks through wiring up a local registry the cluster can actually pull from.

The second lesson was the :latest tag trap. If your manifest references myapp:latest, Kubernetes defaults imagePullPolicy to Always and the kubelet re-pulls on every launch even when the image is sitting right there. The Kubernetes image docs spell it out: omit imagePullPolicy with a :latest tag and it becomes Always, while a fixed tag defaults to IfNotPresent. I switched to specific tags - myapp:dev, myapp:<gitsha> - and imagePullPolicy: IfNotPresent, and a whole category of mysterious slowness just evaporated.

Then Tilt took over the loop, and this is the part I still find a little magical. Instead of build then push then apply then wait, saving a file synced the change straight into the running Pod. The numbers matched exactly what the tool promised - Tilt's own writeup frames live-update as deploying code to running containers in seconds, not minutes, and syncing files into a running Pod with hot reload brought my iteration back down to roughly one to five seconds, better than a 95% cut. One detail I'm glad I got right: in the cluster image I run fastapi run, not uvicorn --reload. Reload is dev-only overhead and a leak risk; the speed comes from Tilt's live-update, not from reload inside the container.

How it feels now

The headline is easy to quote - two-to-five-minute iterations became one-to-five-second ones, and "build, push, apply, wait, check" collapsed into "save the file." But honestly the second-order effects were the ones that changed how I work. Because a change now took seconds to see, I started testing smaller increments, one hypothesis at a time, instead of batching five changes into a single painful deploy and then bisecting which one broke. Pull requests got smaller. Review comments got sharper, because a reviewer could pull the branch and watch it run in a real cluster in under a minute.

And the OOMKilled and readiness-probe bugs that used to ambush us in staging now surfaced on the author's laptop, where the person with all the context was sitting right there to fix them. That's the whole economic argument for shift-left in one sentence: a bug caught in the inner loop costs a coffee's worth of attention, while the same bug caught after git push costs a CI run, a reviewer's time, and sometimes a rollback. I didn't measure it to the decimal, but the number of "why is staging broken?" Slack threads dropped to nearly zero within a sprint, and I stopped bracing every time I opened the cluster.

A few things bit hard enough to cost a day each, and I'll pass them on so they don't cost you one. ImagePullBackOff on day one had nothing to do with a bad image - the cluster just couldn't see it, and understanding image delivery into k3d up front would have saved me an afternoon. Running --reload inside the container "because it was fast locally" fought Tilt and masked the real loop, so kill it. Don't confuse the inner and outer loops: this whole exercise is about the inner loop, the single-developer edit-build-run cycle before git push, and the outer loop of CI, GitOps, and integration tests is a genuinely separate beast. And don't skip resource limits locally - the entire point of a production-like local setup is that limits and probes exist on your laptop, so leaving them out just reopens the gap you were trying to close.

I had to actively resist the itch to also "fix" that outer loop in the same push. It's real work, but it's a different problem with different tools, and bundling it would have stalled everything. I fixed the inner loop first, shipped it, and let the win speak before touching anything downstream. Once the loop was fast, I layered the rest of a production-like setup on piece by piece - real Deployment and Service manifests, PostgreSQL inside the cluster, ConfigMaps and Secrets instead of hardcoded values, health probes - one production-like layer at a time (there's a full end-to-end assembly linked in Sources below).

What I keep coming back to isn't the speed number, satisfying as it is. It's that I started using the cluster again instead of avoiding it. A dev loop you dread is a dev loop you route around, and every workaround quietly costs you the fidelity you were supposed to be buying. Getting seconds back didn't just make me faster - it made me willing, and that turned out to be the whole point.

Sources & further reading

k3d-io/k3d — Little helper to run CNCF's k3s in Docker
k3d docs — Using image registries with k3d
Tilt docs — Smart Rebuilds with Live Update
Tilt blog — Fast Kubernetes Development with Live Update
Kubernetes docs — Container Images & imagePullPolicy
A full local-Kubernetes inner-loop write-up someone put together, covering the end-to-end k3d-to-production-like assembly

How I Cut Python Docker Image Size From 1.2 GB to a Lean, Secure Container

Menshikov Vasil — Tue, 04 Aug 2026 08:24:46 +0000

When we moved our tracking API to Kubernetes, the container turned out to be the weakest link in the whole thing. It was a 1.2 GB image that took two full minutes to rebuild on every code change and then, adding insult to injury, flat-out refused to run in our local k3d cluster. If you've ever wanted to reduce a Python Docker image size and wondered why your "working" image won't load into a cluster, this is the afternoon where I fixed all three problems at once - and stopped being quietly embarrassed by my own Dockerfile.

Why this bugged me for weeks

Our service, myapp, is a Python 3.12 and FastAPI HTTP API on port 8080 that talks to PostgreSQL. The first Dockerfile I ever wrote for it is the first Dockerfile anyone writes, and it technically works:

FROM python:3.12
WORKDIR /code
COPY . .
RUN pip install -r requirements.txt
CMD python -m uvicorn app.main:app --host 0.0.0.0 --port 8080

Here's the uncomfortable truth: every single line is a small mistake. The full-fat python:3.12 base is enormous. COPY . . before pip install means a one-character code edit invalidates the dependency layer and reinstalls the world. The shell-form CMD quietly breaks signal handling. And it runs as root. I lived with all of this for longer than I'd like to admit, treating the slow rebuilds as just the cost of doing business, until I got fed up and went hunting for a canonical, production-minded reference. Someone had written up a walkthrough on writing a small, fast, secure Dockerfile for a FastAPI service and loading it into k3d, and honestly it restructured how I think about every instruction in the file.

The cheapest 90% win: fix your layer order

Each Dockerfile instruction is a layer, Docker caches them, and - this is the part that matters - if a layer changes, every layer after it is invalidated too. I was editing application code dozens of times a day and almost never touching dependencies, yet my ordering forced a full pip install on every build. The fix is embarrassingly simple: install what rarely changes before copying what changes constantly.

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt   # heavy, rarely changes
COPY ./app ./app                                      # light, changes often

That one reorder dropped my rebuild from around two minutes to a few seconds on a code-only change, because the dependency layer now comes straight from cache. It's the single most common Dockerfile anti-pattern in existence, and I'd been cheerfully living inside it for months.

The multi-stage build

The naive image ships everything into the final result - compilers, dev headers, pip caches - all of it dead weight at runtime. A multi-stage build installs dependencies in a build stage and copies only the finished virtualenv into a clean runtime image:

# --- Stage 1: build ---
FROM python:3.12-slim AS build
ENV PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1 PIP_NO_CACHE_DIR=1
WORKDIR /code
RUN python -m venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH"
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# --- Stage 2: runtime ---
FROM python:3.12-slim
ENV PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1 PATH="/opt/venv/bin:$PATH"
WORKDIR /code
COPY --from=build /opt/venv /opt/venv
COPY ./app ./app
EXPOSE 8080
CMD ["fastapi", "run", "app/main.py", "--port", "8080"]

How much you save depends entirely on your dependencies - with pure wheels the gain is modest, but we had a couple of packages that compiled from source, and dropping the build toolchain shrank the image substantially. Switching from python:3.12 to python:3.12-slim did the rest.

The small details that make it production-grade

The write-up drilled a handful of these in, and every one of them has bitten someone I know. Use fastapi run, not a bare uvicorn --reload - the reload flag is dev-only overhead in a cluster image, and FastAPI's own container guide uses fastapi run, which starts Uvicorn with sane production settings. Write CMD in exec form, as an array, so the app becomes PID 1 and receives SIGTERM from Kubernetes directly and shuts down gracefully; shell form wraps it in /bin/sh, the signal never arrives, and the Pod gets hard-killed on timeout. Set PYTHONUNBUFFERED=1, or your logs get stuck in a buffer and never surface in kubectl logs - I lost an hour to "why is my container silent" before I understood that one. Add an unprivileged user with adduser --disabled-password --uid 10001 appuser and then USER appuser, because root-by-default violates least privilege and it pairs later with securityContext.runAsNonRoot: true in the manifest. And add a .dockerignore, because without one I was shipping .git, a local .venv, and nearly a .env straight into the build context.

The bug that stole a full day: ImagePullBackOff

With a beautiful new image built, I ran docker build, applied the manifest, and the Pod sat in ImagePullBackOff forever. It seems completely obvious that a freshly built local image would be visible to the cluster. It is not, and this cost me a full day of my life. k3d nodes run their own containerd, isolated from your Docker daemon - an image sitting in Docker is invisible to the cluster, and the kubelet just keeps failing to pull it from a remote registry that doesn't have it.

The fastest fix for a one-off test is a direct import:

docker build -t myapp:dev .
k3d image import myapp:dev -c dev

Pair that with imagePullPolicy: IfNotPresent in the manifest so Kubernetes doesn't reach for the network anyway. For ongoing work we moved to k3d's built-in registry, where the same name k3d-registry.localhost:5000/myapp:dev works for both push from the host and pull from inside the cluster. The full delivery model - both paths, plus why *.localhost resolves - is covered in the companion write-up linked in Sources below.

Two more touches that paid off

First, a HEALTHCHECK that doesn't need curl. I wanted Docker to know whether the service was alive during local runs, but python:3.12-slim has no curl, and neither does distroless. So I did the check with Python, which is guaranteed to be in the image:

HEALTHCHECK --interval=30s --timeout=10s --start-period=60s --retries=3 \
  CMD ["python", "-c", "import urllib.request,sys; sys.exit(0 if urllib.request.urlopen('http://localhost:8080/healthz').getcode()==200 else 1)"]

Second, actually understanding what EXPOSE does and doesn't do. For an embarrassingly long time I thought EXPOSE 8080 published the port. It doesn't - the Dockerfile reference is explicit that EXPOSE doesn't actually publish anything; it's pure metadata, documentation for whoever reads the file. Real publishing happens with -p in docker run, or with Service and Ingress objects in Kubernetes. Internalizing that killed a whole afternoon of "why can't I reach the port" confusion, purely because I stopped expecting EXPOSE to do a job it was never designed for.

I also standardized on running a single worker per Pod and scaling with replicas, rather than cramming Gunicorn plus multiple Uvicorn workers into one container. In Kubernetes the cluster is the process manager, and letting it own concurrency kept my image simpler and my resource limits meaningful. For local development it's not even a question - one process is plenty.

How it feels now

The difference is night and day, and it's satisfying in a way that's hard to overstate. The base went from the full python:3.12 to a multi-stage python:3.12-slim. A rebuild on a code change went from roughly two minutes of full pip install to a few seconds served from cache. The container runs as uid 10001 appuser instead of root. Signal handling went from broken - the shell-form CMD swallowing SIGTERM - to correct, with the app as PID 1 receiving signals directly. And the image, which used to greet me with ImagePullBackOff every time, now imports and pushes into k3d cleanly.

For local dev I deliberately stayed on slim, because it's genuinely easy to debug. When I hardened the image for production later, I moved the runtime to distroless (gcr.io/distroless/python3-debian12) - no shell, no package manager, far fewer CVEs - building the deps on slim and copying them across. The trade-off is real: docker exec ... sh no longer works, so you debug with kubectl debug and ephemeral containers instead. That's a fair price for the reduced attack surface, but it's a choice worth making consciously rather than by accident.

What stuck with me most is how tangled up all of this had felt when it was really one thing. I'd been treating "make it smaller," "make it faster," "make it secure," and "make it actually run in the cluster" as four separate chores I'd get to someday. They were never four projects. They were one Dockerfile, done with a little more care than the copy-paste version I'd been shrugging past for months - and the version of me who kept restarting slow builds would not believe how good a well-ordered Dockerfile feels to live with.

Sources & further reading

Docker docs — Multi-stage builds
Docker docs — Optimizing builds with cache (layer invalidation)
Docker docs — Dockerfile reference: EXPOSE and CMD exec form
GoogleContainerTools/distroless — Minimal images with no shell or package manager
FastAPI docs — FastAPI in Containers - Docker
A local-Kubernetes containerization write-up someone put together, with the complete annotated Dockerfile and the slim-vs-alpine-vs-distroless decision

How a Test Coverage Ratchet Finally Fixed the Codebase Everyone Was Afraid to Touch

Menshikov Vasil — Mon, 03 Aug 2026 09:25:03 +0000

I have never seen a metric sit as stubbornly still as our test coverage did. Twenty-five percent. For two years. It wasn't drifting up, it wasn't drifting down, it just sat there like furniture nobody wanted to move. And the thing that eventually fixed it wasn't a rewrite or a hero week - it was a small, almost boring idea called a test coverage ratchet, which I'll get to. First I want to tell you why this bugged me for so long, because the psychology turned out to matter more than the config.

Why this bugged me for years

I joined a team maintaining a ~400,000-line JavaScript and TypeScript monolith. Coverage was 25% and everyone knew it. It's not that people didn't care - I watched them care, out loud, in retros. It's that every attempt to fix it failed the exact same way. Someone would pitch a "refactoring sprint," leadership would grudgingly hand over two weeks, the team would rewrite one module, a production incident would eat half the time, and the whole thing would quietly evaporate. Coverage: still 25%.

What really got under my skin was watching genuinely excellent engineers make a one-line change to a function they clearly understood, and then refuse to clean up the obvious mess sitting right next to it. When I asked why, the answers were always the same flavor: "it's always worked this way," "better not touch it," "honestly it'd be easier to rewrite than to understand." That's not laziness. Once I stopped reading it as laziness, everything reframed.

The thing I finally understood: it's fear, not incompetence

Teams don't rot from a lack of knowledge. They rot from fear. When a codebase is big and fragile, every change feels dangerous, so people start programming defensively - the smallest possible edit, a workaround instead of a fix. And it feeds on itself: the worse the code gets, the less anyone wants to touch it, which makes it worse. Psychologists have a name for it, learned helplessness, that state where you stop trying to change a situation even when you actually could. Michael Feathers puts the technical half of it bluntly in Working Effectively with Legacy Code - "legacy code is simply code without tests" - which is exactly why the fear is rational. With no tests, every edit really is a gamble.

And here's the thing that clicked for me: you cannot fix learned helplessness with a two-week sprint. A sprint says "all or nothing," and since "all" is impossible, the brain quietly hears "nothing." The way out is the opposite of a sprint - tiny, achievable, basically-guaranteed-to-succeed steps. That's the whole spirit of incremental constraints, and it lines up perfectly with Robert C. Martin's Boy Scout Rule: leave every file you touch a little better than you found it. Not perfect. A little better. Rename one variable, split one bloated function, delete one bit of duplication.

The rule we actually wrote down

We put one sentence on the team wiki: every change should leave the code in a better state than before the change. Then we made it concrete. New files had to meet a real bar - tests for public methods, no exceptions. Modified files couldn't lose coverage, and ideally gained a few points. Critical bug fixes shipped with a regression test that reproduced the bug. Refactors shipped with a test proving behavior hadn't changed.

The magic is entirely in the asymmetry. We never asked anyone to go improve the 300,000 lines of legacy code. We asked only that whatever you newly wrote or happened to touch that day met the bar. Legacy code you never open never blocks you. That one boundary is what made the whole thing feel possible instead of doomed.

Making it real: the ratchet lives in CI

A rule nobody enforces is just a nice feeling on a wiki. So we encoded the ratchet in three layers.

The first is a two-tier jest coverage threshold. The trick is a per-path override - jest's coverageThreshold takes both a global block and path/glob-specific blocks, and a glob's files are held to their own bar independently of the global one. We pinned the global numbers at today's levels so they could never slide backward, while new feature directories answered to 80%:

// jest.config.js
module.exports = {
  collectCoverageFrom: ['src/**/*.{js,ts,tsx}'],
  coverageThreshold: {
    global: {
      branches: 25, // current level - never decrease
      functions: 30,
      lines: 35,
      statements: 35
    },
    // Everything created after we flipped the switch
    'src/features/**/*.ts': {
      branches: 70,
      functions: 80,
      lines: 80,
      statements: 80
    }
  }
}

Every few weeks, once the global numbers had drifted upward on their own, we'd bump the global floor up to meet them. That's the whole idea of a ratchet - it only ever clicks in one direction, and it can't click back.

The second layer is a quality gate that judges only the diff. This pattern has a name too: diff coverage, sometimes called patch coverage. You hold a high bar only to the lines a pull request adds or changes, and you cheerfully ignore the legacy sea around them. Codecov ships this as a first-class check - its codecov/patch status "only measures lines adjusted in the pull request" - so you can demand 80% on new lines without touching the rest of the repo. Ready-made ratchet tools exist as well; jest-coverage-ratchet reads your coverage summary and nudges each threshold up to the current level so it can only ever go higher. All of these are lovely. But a tool with no culture behind it just becomes another gate people learn to game, so we wrote a thin wrapper of our own to keep the numbers legible to the team:

# .github/workflows/quality-gate.yml
name: Quality Gate
on: [pull_request]
jobs:
  quality-check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0  # needed for diff analysis
      - name: Check coverage for changed files
        run: |
          CHANGED_FILES=$(git diff --name-only origin/main...HEAD | grep -E '\.(js|ts|tsx)$')
          if [ ! -z "$CHANGED_FILES" ]; then
            npm run test:coverage
          fi

We rolled it out with a phase I'd call "measurement without judgment" - run coverage, ESLint, and duplication analysis purely to see where we stood, with zero blame attached to any number. That framing mattered enormously. Nobody feels attacked by a dashboard they helped set up.

The third layer is ESLint with the same asymmetry - strict complexity limits on new directories, warnings-only on the legacy ones, so src/features/** answered to a complexity ceiling of 8 and 30-line functions, while src/legacy/** got gentle warnings at 15 and 100. Same shape, different tool.

The trick that unlocked the genuinely scary files

Some functions were terrifying - a 200-line calculateDiscount nobody fully understood. You can't refactor what you can't describe, and this is where Feathers' characterization test earns its keep: it "characterizes the actual behavior of a piece of code". You don't test what the code should do. You test what it currently does, whatever that is, warts and all, and pin it in place before you dare touch anything:

describe('calculateDiscount - current behavior', () => {
  it('VIP user with promo XYZ123', () => {
    const result = calculateDiscount({ type: 'VIP' }, [{ price: 100 }], 'XYZ123');
    // We don't know WHY it's 0.25, we're just pinning current behavior
    expect(result.discount).toBe(0.25);
  });
});

The emotional shift here is real. Once a tangled function is wrapped in characterization tests, it just stops being scary. You've got a net that screams the instant behavior changes, so you can finally carve it into small, testable pieces with your shoulders down.

How it feels now

Eleven months later, global line coverage had climbed from 25% to 61%, and the code we shipped that quarter was sitting around 84% - up from maybe 10% before. We reverted far fewer PRs for regressions, and "safe-ing" a scary legacy function went from a thing we simply avoided to something you could knock out in an afternoon. Most tellingly, the number of engineers willing to touch calculateDiscount went from exactly one to most of the team.

But the number I care about most is the one we never scheduled: zero dedicated refactoring sprints. Coverage climbed because ordinary feature work now dragged quality up with it, one touched file at a time, and nobody had to be a hero.

We hit a few walls worth naming. Don't set the new-code bar at 100% - we tried 90% and people gamed it with trivial assertions; 80% forced real tests without inviting malicious compliance. Bump the global floor by hand, in its own PR, on purpose - we automated it once and a hot-fix that happened to touch a well-covered file caused flaky failures. And treat "measurement without judgment" as load-bearing: the first time a manager used the coverage dashboard to single someone out in a review, trust cratered for a month. Kill that instinct early and loudly.

If you want the long version - the full psychology, every config, the pre-commit hooks I didn't have room for - someone wrote up the whole incremental-constraints playbook here, and it's worth a read before you pitch your next doomed sprint.

What stays with me isn't the graph. It's that "better not touch it" has basically vanished from our standups. We didn't make anyone braver by asking them to be brave. We just made the next small step safe enough that bravery stopped being the requirement - and it turns out an entire team quietly leaving files a little better than they found them will outrun any refactoring sprint you could ever schedule.

Sources & further reading

The brew services Setup That Finally Got Docker Desktop Off My Mac (and Made Local Postgres Feel Instant)

Menshikov Vasil — Sun, 02 Aug 2026 08:39:15 +0000

For a long time the first fifteen minutes of my workday belonged to a whale. I'd open the laptop, hear the fans spin up, watch "Docker Desktop is updating," and wait - all so I could run a single local Postgres container that my Mac was perfectly capable of running on its own. The morning three people on my team showed up late to standup for the exact same reason ("my fans won't stop and Postgres won't bind," "I ran docker compose up and my battery's already at 40%"), something in me finally snapped. This is the brew services setup that got Docker Desktop off our machines, and honestly, made local development feel joyful again.

Why this bugged me for years

Our local stack was not exotic. One docker-compose.yml with Postgres, Redis, and MySQL. That's it. And yet Docker Desktop's VM taxed every machine the entire day - idle RAM I could feel, CPU burned on file-sync I never asked for, a licensing question hanging over the whole thing, and that update nag that always seemed to block me at the worst possible moment.

The licensing part had teeth, too. Under the Docker Subscription Service Agreement, Docker Desktop needs a paid subscription once your company is big enough - the free tier caps out under 250 employees and $10M revenue. We were over one of those lines, which meant Docker Desktop wasn't only a performance tax, it was a per-seat bill for the privilege of running one Postgres. That combination gnawed at me for a genuinely embarrassing amount of time before I did anything about it.

What finally reframed it was saying the obvious thing out loud: on a Mac, you already own a first-class service supervisor. It's called launchd, it's what Apple uses to run its own daemons, and Homebrew speaks it fluently. I'd been renting a Linux VM to babysit a database my operating system already knew how to babysit.

Why brew services instead of a container

The pitch is almost too simple. Homebrew installs Postgres, Redis, MySQL, MongoDB, Nginx, RabbitMQ, Elasticsearch, Kafka - the whole cast of local dev dependencies - as native formulae. Then brew services wraps macOS's own launchd so those processes start at login, get supervised, and stop cleanly (the full command surface lives in the Homebrew manpage). No VM. No file-sync layer. No 4GB toll booth in front of a single database.

I want to be honest about the trade-off, because I treated this as a decision and not a religion. Containers give you isolation, reproducibility, multiple versions side by side, and real production parity. Homebrew gives you native performance, near-zero overhead, and a Mac-first experience for the ninety-percent case where you just need a Postgres to develop against. Both of those are true at once. If you want the longer version of that decision, with all the commands and the honest tradeoffs laid out, someone wrote up the whole Homebrew-vs-containers case here and it's a good companion read.

And yes, I did the responsible thing and evaluated the usual Docker Desktop alternatives first. Colima, OrbStack, and Podman are all genuinely lighter than Docker Desktop and worth knowing - Colima especially gives you a Docker-compatible runtime with minimal fuss. But they're still containers. They shave the licensing and some of the RAM; they don't change the fundamental fact that you're running a Linux VM to host a database your Mac can run natively. For a single local Postgres or Redis, brew services isn't a lighter container runtime. It's no container at all - which is the part most "Docker Desktop alternatives 2025" roundups quietly skip.

The setup: four commands and a launchd file

Here's the entire day-to-day surface. Four commands:

# Start a service
brew services start postgresql

# Stop a service
brew services stop postgresql

# Restart a service
brew services restart postgresql

# List all services and their status
brew services list

That last one is the command I actually live in. It hands you an at-a-glance view of what's running and where its config lives:

$ brew services list
Name       Status  User    Plist
mysql      started username /Users/username/Library/LaunchAgents/homebrew.mxcl.mysql.plist
postgresql stopped
redis      started username /Users/username/Library/LaunchAgents/homebrew.mxcl.redis.plist

Our old onboarding doc had a two-page section titled "Installing and troubleshooting Docker Desktop." The new version is a shell snippet:

brew install postgresql@16 redis mysql
brew services start postgresql@16
brew services start redis
brew services start mysql

A new hire now goes from a fresh laptop to a working database stack in about the time it takes to fetch coffee. No account, no login, no license-acceptance dialog. The first time I watched someone new run that snippet and just... have a database, I felt a little pang about all the mornings I'd lost to the whale.

What's actually happening underneath

I didn't want to hand anyone a magic command they couldn't reason about, so we documented the mechanism. When you run brew services start postgresql, Homebrew generates and registers a .plist with launchd - the same supervision layer Apple uses for system daemons. Apple's Technical Note TN2083 on Daemons and Agents is the canonical reference for how login agents like this get loaded, and thoughtbot's walkthrough of starting and stopping services with Homebrew is a friendly plain-English companion. The generated plist is refreshingly readable:

<?xml version="1.0" encoding="UTF-8"?>
<plist version="1.0">
<dict>
  <key>Label</key>
  <string>homebrew.mxcl.postgresql</string>
  <key>ProgramArguments</key>
  <array>
    <string>/usr/local/opt/postgresql/bin/postgres</string>
    <string>-D</string>
    <string>/usr/local/var/postgres</string>
  </array>
  <key>RunAtLoad</key>
  <true/>
  <key>KeepAlive</key>
  <true/>
  <key>StandardErrorPath</key>
  <string>/usr/local/var/log/postgres.log</string>
</dict>
</plist>

RunAtLoad starts it at login. KeepAlive restarts it if it dies. Logs land at a predictable path you can just tail. Once people saw that this was only launchd doing what it already does for the whole OS, the "but is it reliable?" worries quietly dissolved.

How it feels now

I'm not going to pretend "trust me, it feels faster" is data, so we did measure it across a handful of volunteer machines for a couple of weeks. The headline is the memory. The local database stack went from something like 4GB of idle RAM under Docker Desktop to under 200MB with brew services - and getting roughly 4GB back on a 16GB MacBook is the difference between a laggy editor and a responsive one. Two people told me they'd stopped closing their browser to "make room" for the dev stack, which tells you everything about the world we'd been living in.

The rest followed. Cold database start dropped from nearly half a minute to under two seconds. The fans, which used to be a constant background hum, became a rare event. Onboarding went from around fourteen steps to four. And the annual license-management chore simply vanished. None of these numbers are exotic; they're just what happens when you stop running a VM to do a job your OS does natively.

The gotchas nobody puts in the README

It wasn't frictionless, and I'd be lying if I pretended otherwise. Version pinning is now on you - a container pins its version in the compose file, but Homebrew installs a version and a stray brew upgrade can move it out from under you mid-sprint. We standardized on the versioned formula, postgresql@16, and left a note in the Brewfile so nobody accidentally jumps a major version.

A service that "won't start" is almost always a stale socket or a log you haven't read yet. My debugging loop is boring and reliable: glance at brew services list, then actually read the log, then poke the service itself rather than trusting the status column.

brew services list
tail -f /usr/local/var/log/postgres.log
ps aux | grep postgres
launchctl list | grep postgres

Nine times out of ten the log names the culprit outright - a leftover postmaster.pid, a port already bound, a data dir from an older major version.

The sudo brew services trap is worth calling out too. Running a service as your user (a login agent) versus as root (a system daemon) puts the plist in different directories and changes when it starts; TN2083 above explains exactly why. For local dev you almost always want the user-level agent. We deleted one sudo somebody had copy-pasted from a random blog and half of our "it doesn't start at boot" complaints evaporated. And finally: your data now lives on the host with no volume abstraction, which is simpler but means a careless brew uninstall can take your local data with it. Our rule became a one-liner - your local DB is disposable, seed it from a script, never keep anything you can't regenerate.

Where Docker still wins, and why I kept it

I did not declare war on Docker. I just retired Docker Desktop as the default local-database runtime. The instant you need several services wired together for an integration test, or byte-for-byte production parity, containers are still exactly right. Our whole policy collapsed into one sentence: Homebrew for quick local databases, Docker for multi-service integration tests. Both live in the repo; you reach for the lighter one by default. That single sentence ended months of "should we standardize on containers or not" bikeshedding, because it was never either/or - most jobs are just lighter than we'd assumed.

We're tightening a couple of loose ends now, mostly a checked-in Brewfile so brew bundle reproduces the exact service set - and because Homebrew Bundle can start services declaratively with restart_service: true, that closes most of the reproducibility gap containers otherwise own.

What stays with me, though, isn't the RAM graph. It's that my laptop is quiet in the morning now. There's a particular kind of craft in noticing the tool you've been tolerating for years and asking whether it's actually earning its keep - and if your machine sounds like a jet engine every morning to run one Postgres and a Redis, I'd gently suggest it isn't.

Sources & further reading

How to Generate requirements.txt From What Your Code Actually Imports (pipreqs vs pip freeze)

Menshikov Vasil — Sat, 01 Aug 2026 16:15:03 +0000

There's a file in most Python projects that everyone commits and nobody reads, and for two years ours was quietly lying to us. If you've ever wondered how to generate a requirements.txt that reflects what your code actually needs - rather than every stray package that ever wandered into your virtualenv - this is the story of how I finally stopped trusting pip freeze for that job and switched to pipreqs. Our file went from 214 packages to 23, and honestly the bigger relief was emotional: I could finally open it and believe what it told me.

Why this bugged me for years

Every Python engineer knows the ritual. Finish a feature, run pip freeze > requirements.txt, commit, move on. It works. Right up until it doesn't.

Ours had metastasized. A new hire opened a PR and asked the most innocent question in code review: "Why does this API service depend on jupyter, matplotlib, and black?" It didn't. Those were tools somebody had pip install-ed into the shared venv months earlier and never removed. And pip freeze does not care in the slightest what your code imports - it snapshots everything installed in the environment. Our so-called source of truth was 90% noise, and I'd been squinting past it for so long I'd stopped seeing it.

That noise was not harmless, which is what finally pushed me to act. Docker builds crawled, because all 214 packages got resolved and installed on every cold build. Our vulnerability scanner shrieked about CVEs in libraries we never once imported. And worst of all, nobody could tell which dependencies were real, so nobody dared delete any of them. That last one is the quiet killer - a file so untrustworthy that fear freezes it in place. (I've watched the exact same fear freeze legacy test suites, so I recognized the smell immediately.)

So I went looking for a better way to generate the thing, and the tool I'd criminally underused was pipreqs, whose one-line pitch in its GitHub README is to "generate pip requirements.txt file based on imports of any project" - the exact opposite of what pip freeze does.

In fairness to pip freeze

I don't want to trash pip freeze, because it isn't wrong - it's just answering a different question than I was asking. The official pip docs are refreshingly blunt about this: it "reports what is installed; it does not compute a lockfile or a solver result." It answers "what is installed here," not "what does this code need." In a pristine, single-purpose virtualenv those two sets are identical. In a real, long-lived dev environment they drift apart embarrassingly fast.

The canonical flow is exactly what you'd expect:

# macOS/Linux
source venv/bin/activate
pip freeze > requirements.txt

If your venv is clean and dedicated to one project, that's genuinely the right tool - it captures exact versions of everything, which is precisely what you want for a fully reproducible environment. Our problem was simply that our venv had become a junk drawer, and we'd been using a drawer-inventory tool to describe a shopping list.

The thing that finally clicked

pipreqs takes the opposite approach, and once I understood it I felt a little silly for not switching sooner. It statically parses your .py files, finds the actual import statements, maps them to PyPI packages, and writes only those. Here's the whole sequence:

# 1. install it (once)
pip install pipreqs

# 2. from the project root, scan the code
cd /path/to/your/project
pipreqs .

# 3. regenerating over an existing file? force it
pipreqs . --force

The . is just the current directory - pipreqs walks it, reads the imports, and emits a requirements.txt containing only the libraries your code truly touches. The README documents the flags worth knowing: --savepath to write elsewhere, --print to dump to stdout, and --diff to compare an existing file against the project's real imports. That's the entire change. The generated file went from 214 lines to 23 - the same 23 packages the app had been importing all along, now finally the only 23 in the file.

The payoff rippled outward faster than I expected. The final Docker image roughly halved, dropping from around 1.3 GB to a little over 600 MB. A cold pip install in CI went from nearly three minutes to under a minute. Our scanner's findings on dependencies fell from thirty-one to six, and every one of those six was now a package we actually used, which meant we could actually triage them. And for the first time, "can I delete this dependency?" became a question with an answer instead of a shrug.

Where pipreqs bites (because it's not a silver bullet)

I'd be a bad engineer if I sold you pipreqs as pure upside, so here's where it drew blood - and the first one is a well-known limit of static analysis, not a bug. Dynamic and conditional imports simply get missed. If you reach for importlib.import_module(name) or import inside a try/except, pipreqs never sees it. We had exactly one - a plugin loaded by string name - and it broke at runtime until we pinned it by hand.

The other traps are gentler. Import name isn't always package name: import cv2 is really opencv-python, import yaml is PyYAML. pipreqs handles most of these through its mapping, but verify anything exotic. It infers versions from your environment or PyPI, so keep running it inside your activated venv to get versions that match what you actually tested against. And test/dev-only tools legitimately vanish from the output - which is the point, not a defect. If you want pytest and black tracked, give them their own requirements-dev.txt rather than mourning that pipreqs dropped them.

Our pragmatic landing spot: pipreqs generates the runtime requirements.txt, a hand-maintained requirements-dev.txt holds the tooling, and CI diffs the generated file to catch drift.

Rolling it out across twelve services without a big-bang

I'm allergic to regenerating every manifest in one heroic PR and praying, because bloated dependency files are load-bearing in ways you only discover when something 404s in prod at 2am. So we staged it. For each service we first saved the old pip freeze output as requirements.legacy.txt, so if pipreqs missed something we had the full list to diff against. Then we generated into a separate file with pipreqs . --force --savepath requirements.pipreqs.txt and reviewed what fell out by hand - ninety percent of the drops were obviously dev tooling, and the remaining ten percent got a human look. We rebuilt each image in a scratch container and ran the full suite plus a smoke test against the real boot path, not just unit tests, specifically to flush out the dynamic-import failures early. Then we promoted one service, let it bake in production for a week, and only then fanned out to the rest.

Exactly one service broke, and it broke exactly where I'd feared: our notification worker loads channel plugins by string name via importlib, so pipreqs never saw slack_sdk or twilio. The scratch-container smoke test caught it before any customer did. We dropped those two into an explicit requirements.extra.txt and concatenated it during the build. Cheap lesson, caught in the right place.

The reproducibility question everyone asks

The most common objection I heard internally was fair: "pip freeze guarantees exact versions and full transitive pinning - pipreqs only gives me top-level packages, isn't that less reproducible?"

The honest answer is that the two tools solve different halves of the problem, and the Python community has since standardized that split. pip freeze output plays the role of a lockfile - the exact, fully-resolved state you deploy. As of PEP 751 and the pylock.toml spec, a lockfile's entire job is "specifying dependencies to enable reproducible installation," pinning exact versions, URLs, and hashes. pipreqs shines as the other half - a manifest, the human-readable statement of intent about what your code needs. Our whole mistake had been asking one file to do both jobs, and letting the lockfile role bloat the manifest into something unreadable. If you want the long version of that manifest-versus-lockfile distinction and the staged rollout, someone wrote up the whole twelve-service migration here and it's a good deep-dive.

Our fix keeps both files: pipreqs writes the intent-level requirements.txt, and we resolve and pin into a separate lockfile at build time. The manifest is readable and reviewable again, and reproducibility lives where it belongs. We wired a guard into CI too - regenerate the file, and fail the build if it drifts from what's committed:

pipreqs . --force --savepath /tmp/req.check
diff <(sort requirements.txt) <(sort /tmp/req.check)

If a new import lands without updating the manifest, the diff is non-empty and the pipeline goes red. No more mystery dependencies sneaking in - or, more to the point, lingering unnoticed for two years.

How it feels now

We're piloting uv, Astral's Rust-based package manager, and weighing whether pyproject.toml should become the single source of dependency truth with requirements.txt generated as a lockfile artifact - which is where most modern Python workflows are clearly heading. But I keep coming back to how small the actual fix was. One tool swap turned a 214-line liability into a 23-line file that documents exactly what the service needs, and that's it.

What I didn't expect was the feeling. That file stopped being a place packages go to die and became something I actually read in review. If yours has that same graveyard smell, run pipreqs . against your project right now and diff it against what's committed. I'd bet you'll be a little startled - and maybe a little relieved - at how little your code actually imports.

Sources & further reading

pipreqs — the tool itself, README and flags (--savepath, --print, --diff, --force)
pip freeze — official pip documentation ("reports what is installed; it does not compute a lockfile")
PEP 751 / pylock.toml specification — the standardized lockfile format, Python Packaging User Guide
Writing your pyproject.toml — Python Packaging User Guide
uv — Astral's fast package/project manager with universal lockfiles
A full write-up of this migration someone put together — the staged rollout across twelve services, the CI drift check, and the manifest-vs-lockfile split

Hacker News + dev.to (thought-leadership framing of the inner-loop problem drives the whole funnel)

Menshikov Vasil — Wed, 22 Jul 2026 08:48:27 +0000

The Kubernetes inner dev loop is slow AND lies to you — here's why

If you've ever shipped a service to Kubernetes for the first time, you know the feeling: the edit→result loop that took a fraction of a second with uvicorn --reload suddenly takes 2–5 minutes, and code that "worked locally" dies in the cluster for reasons you never had to think about.

This is the opener of a new series on local Kubernetes development. It's a mental-model piece, not a tutorial — the goal is to name the two distinct problems most people blur together.

Key takeaways

The inner loop is the loop you crank dozens of times an hour: edit → build → run → check → fix, before you ever git push. Keeping it in the seconds is what protects your flow.
Kubernetes wedges four extra steps into that loop: docker build, update manifest, docker push, apply and wait for the Pod. That's the 2–5 min/iteration tax — vs ~1–5 s with file-sync/hot-reload (a 95%+ reduction).
The local cluster can't see images in your local Docker daemon. Skip the push/import step and you get a Pod stuck in ImagePullBackOff.
The :latest trap: myapp:latest defaults to imagePullPolicy: Always, so the kubelet re-pulls every start. Use specific tags + IfNotPresent.
Low fidelity is the other half of the pain: OOMKilled, CPU throttling, NetworkPolicy blocks, readiness failures, RBAC denials — a whole class of bugs invisible under bare uvicorn or docker compose.
Reframe shift-left as environment fidelity. As Testkube puts it: "testing earlier in a CI container that doesn't match your cluster isn't shift-left — it's just failing faster in the wrong environment."

The running example throughout the series is myapp: a Python 3.12 + FastAPI HTTP API on port 8080 that depends on PostgreSQL, run on a local k3d cluster.

Full article: https://dorokhovich.com/blog/local-k8s-inner-dev-loop?utm_source=devto&utm_medium=syndication&utm_campaign=local-k8s-inner-dev-loop