DEV Community: Menshikov Vasil

ImagePullBackOff in k3d even though `docker build` succeeded? Here's why (a k8s error field guide)

Menshikov Vasil — Wed, 16 Sep 2026 11:01:56 +0000

The most disorienting beginner moment in k3d: you build an image with docker build, the cluster says ImagePullBackOff, and you're sure the image exists. It does — in Docker. k3d nodes run on containerd, which is isolated from your Docker daemon. Docker has the image; the cluster can't see it.

Takeaways

Almost every status is diagnosed with the same three commands: kubectl describe pod (State/Reason/Events), kubectl logs --previous, kubectl get events --sort-by=.lastTimestamp.

ImagePullBackOff: typo in image/tag, private registry without imagePullSecrets, Docker Hub rate limit (toomanyrequests), or the k3d containerd isolation. Fix the last with k3d image import myapp:dev -c dev, or (better) a local registry — and reference the full name k3d-registry.localhost:5000/myapp:dev in the manifest. An incomplete image name is a top cause of a repeat BackOff.
CrashLoopBackOff: kubectl logs --previous for the dead instance's real error; check Last State: Terminated Exit Code. Common: app bug at startup, missing env/config, unavailable dependency, OOM, too-strict liveness, or exit code 0 (a long-running service that "exited successfully" — usually a wrong entrypoint).
Pending: FailedScheduling names the cause — usually insufficient CPU/memory for requests on a small local cluster. Also nodeSelector/affinity, taints, unbound PVC, hostPort clash.
OOMKilled (exit 137 = 128+9): container-level (exceeded limits.memory) or the sneaky node-level OOM — k3d nodes live in a Docker VM, so combined limits over the VM's memory kill Pods even when each app is within its own limit. Don't overcommit.
Service silent, Pods Running: walk Service -> endpoints -> Pod. Empty endpoints = selector mismatch (labels are case-sensitive) or port mismatch (targetPort vs containerPort). Running 0/1 = readiness failing (removed from endpoints, not restarted).
Tilt not updating: synced path outside build context, file not covered by sync(), fall_back_on file changed (forces rebuild), or run() before sync(). FastAPI trap: without a process restart the synced code lands but uvicorn keeps the old code in memory — use --reload or docker_build_with_restart.
Disk eaten: kubelet image GC is lazy (only above 85% disk). docker system df, then prune incrementally — but -a can wipe images the cluster needs and --volumes can delete your local PostgreSQL data.

Full article: https://dorokhovich.com/blog/local-k8s-troubleshooting?utm_source=devto&utm_medium=syndication&utm_campaign=local-k8s-troubleshooting

Local Kubernetes Dev — Part 15: Common problems and how to fix them

Menshikov Vasil — Mon, 14 Sep 2026 09:58:16 +0000

"I built the image with docker build, but in k3d it's ImagePullBackOff. How? The image exists!"

Because k3d nodes run on containerd, and it's isolated from your Docker daemon. Docker has the image, the cluster doesn't. The most disorienting moment for a k3d newcomer, and it's fixed two ways: k3d image import, or (better) a local registry with the full name k3d-registry.localhost:5000/myapp:dev in the manifest.

In the new article (part 15) — a field guide to the statuses you hit every week, all diagnosed with the same three commands:
• ImagePullBackOff / ErrImagePull (+ the k3d containerd trap);
• CrashLoopBackOff (and why logs --previous is the key);
• Pending / FailedScheduling;
• OOMKilled (137) — and the sneaky node-level OOM inside the Docker VM;
• Service is silent: a break in the Service → endpoints → Pod chain (selector / ports / readiness);
• Tilt not picking up changes;
• k3d/Docker ate the disk.

Bookmark it: https://dorokhovich.com/blog/local-k8s-troubleshooting?utm_source=devto&utm_medium=syndication&utm_campaign=local-k8s-troubleshooting

Why `kubectl apply` doesn't roll out your `:latest` image (and the bridge from local to CI)

Menshikov Vasil — Sat, 12 Sep 2026 20:26:02 +0000

The counterintuitive pipeline bug: you push a new image under the same :latest tag, run kubectl apply, and nothing deploys. Kubernetes only triggers a rollout when the pod template changes — the line image: myapp:latest didn't change, so as far as the cluster is concerned there's nothing to do, even though the registry image is now different.

Takeaways

latest is three problems in one: it's mutable (no reproducibility); re-applying doesn't roll out (template unchanged); and a Pod restart may silently pull a different, possibly broken image. Prod changes version "by itself."
Use immutable tags that point to one specific build: short SHA (sha-abc1234), semver (v1.2.3), or a digest (@sha256:...) for maximum reproducibility. In CI, docker/metadata-action generates these from git context and writes OCI labels (org.opencontainers.image.revision, etc.) tying the image to a commit.
One base, per-environment overlays to avoid config drift: Helm (Go templates + values-prod.yaml; multiple -f merge last-wins — order matters) or Kustomize (plain YAML base/ + overlays/; the images: field swaps tags without editing the Deployment). Hybrid is common: third-party charts via Helm, your own services via Kustomize.
Minimal push-based CI (GitHub Actions): checkout -> buildx -> login -> metadata-action -> build-push-action -> deploy with an immutable tag (kubectl set image / helm upgrade / kustomize build | apply).
Be deliberate about the local/prod split. Reused as-is: same Dockerfile/image, same charts/base, probes. Prod-only: real registry + immutable tags, HPA/PDB/anti-affinity, real Secrets/TLS, monitoring. Must NOT carry over: Tilt live-update/file-sync, uvicorn --reload, exposed debug ports — they're a security hole in prod.
Push-CI's three weaknesses point to GitOps: CI holds the cluster keys (attack surface), Git-vs-cluster drift isn't tracked, and there's no automatic rollback.

Full article: https://dorokhovich.com/blog/local-k8s-preparing-for-deployment-and-ci?utm_source=devto&utm_medium=syndication&utm_campaign=local-k8s-preparing-for-deployment-and-ci

Local Kubernetes Dev — Part 14: Preparing for deployment and CI

Menshikov Vasil — Thu, 10 Sep 2026 12:40:35 +0000

"I pushed a new image under the same :latest, ran kubectl apply — and nothing changed in the cluster." Why?

Kubernetes triggers a rollout only if the pod template changed. The line image: myapp:latest didn't change — so as far as the cluster is concerned there's nothing to deploy, even though the registry already holds a different image. Worse, on a pod restart latest can pull a new (possibly broken) version — on its own, without a deploy from you. Prod changes versions "all by itself."

In the new article (part 14) — the bridge from a fast local loop to a repeatable deploy:
• one manifest base + per-environment overlays (Helm values vs Kustomize patches), no copy-paste and no config drift;
• why latest in prod is pain, and what to use instead: immutable tags (sha, semver, digest) + docker/metadata-action;
• a minimal CI on GitHub Actions: build → push → apply;
• an explicit breakdown: what moves from local, what's prod-only, and what you must NOT drag into prod (Tilt live-update, uvicorn --reload, debug ports);
• the three weaknesses of push-based CI (keys in CI, drift, manual rollback) — and why they lead to GitOps.

Read it: https://dorokhovich.com/blog/local-k8s-preparing-for-deployment-and-ci?utm_source=devto&utm_medium=syndication&utm_campaign=local-k8s-preparing-for-deployment-and-ci

Kubernetes Secrets are base64, not encryption — and 4 more config gotchas

Menshikov Vasil — Thu, 27 Aug 2026 20:38:52 +0000

If you take one thing from this: a Secret committed to git with a real password is a leaked password. base64 is encoding, not encryption — it reverses with one command.

echo 'czNjcjN0' | base64 -d   # prints: s3cr3t

The Kubernetes docs say it plainly: Secrets are stored unencrypted in etcd by default, just base64-encoded.

Takeaways

ConfigMap vs Secret: ConfigMap is for non-sensitive key/value data (1 MiB limit, no secrecy, must share the Pod's namespace). Secret is for passwords/keys/tokens. Use stringData: so the API base64-encodes values for you.
Three ways to inject values: env + valueFrom (one key), envFrom (all keys), or a volume mount (each key becomes a file). The trap: env is fixed at Pod start and never updates on a ConfigMap/Secret change — you must kubectl rollout restart. A volume mount does update automatically (minus subPath), but your app has to re-read the file.
Keep secrets out of git: minimum is a committed secret.example.yaml template + real secret.yaml in .gitignore. For GitOps, encrypt before committing with Sealed Secrets / SOPS / External Secrets. Sealed Secrets are bound to namespace+name; back up the controller's private key or your committed SealedSecrets become unreadable.
Per-environment values without copy-paste: Kustomize configMapGenerator/secretGenerator add a content hash suffix to the name — change the contents, the name changes, the Deployment reference updates, and you get an automatic rolling update. Or Helm: one chart + values-dev.yaml/values-prod.yaml with only the diffs.
What actually protects secrets in prod: Encryption at Rest in etcd, plus least-privilege RBAC. Note that list on secrets implicitly exposes their contents.

Full article: https://dorokhovich.com/blog/local-k8s-configuration-and-secrets?utm_source=devto&utm_medium=syndication&utm_campaign=local-k8s-configuration-and-secrets

Local Kubernetes Dev — Part 10: Configuration and secrets

Menshikov Vasil — Tue, 25 Aug 2026 12:41:10 +0000

"I'm using a Secret — so my password is protected." Nope.

A Kubernetes Secret stores values in base64. Many people mistake this for encryption — and happily commit a manifest with a real password to git. But base64 unwraps with a single command:

echo 'czNjcjN0' | base64 -d # → s3cr3t

So a Secret with a real password in your repo = a leaked password. And git remembers everything: delete the file later and the value still lives on in history.

In the new article I go through it step by step:
• how ConfigMap differs from Secret and why base64 ≠ encryption;
• three ways to pass values into a Pod (env, envFrom, volume) and the trap of "I changed the ConfigMap but the Pod still runs on the old env";
• how to avoid committing a secret: templates + .gitignore for local, Sealed Secrets for prod;
• how to keep different values for dev and prod without copy-pasting manifests (Kustomize secretGenerator / Helm values);
• what actually protects secrets in prod: Encryption at Rest and RBAC (where even list permission on secrets exposes their contents).

Part 10 of the local Kubernetes series. Read it and stop confusing encoding with encryption: https://dorokhovich.com/blog/local-k8s-configuration-and-secrets?utm_source=devto&utm_medium=syndication&utm_campaign=local-k8s-configuration-and-secrets

How I Stopped Being Scared of Rails Migrations (and Learned to Love Zero-Downtime Deploys)

Menshikov Vasil — Mon, 10 Aug 2026 08:12:30 +0000

I used to have a little ritual before every risky deploy: write the up, write a matching down, and tell myself that if anything went wrong I could just roll it back. It was a comfort blanket. And like most comfort blankets, it was hiding the fact that I was cold. This is the story of how I learned that rails zero downtime migrations aren't about being able to undo things - they're about designing so you never need to.

Why this bugged me for years

The migration that finally broke the spell looked completely, boringly safe in review. It added a compound index to speed up product search:

class OptimizeProductSearch < ActiveRecord::Migration[7.0]
  def up
    add_index :products, [:category_id, :price, :created_at],
              name: 'idx_products_category_price_date'
  end

  def down
    remove_index :products, name: 'idx_products_category_price_date'
  end
end

The up built the index over 15-45 minutes under shared locks - slow queries, a sad p95, but survivable. The trap was the down. When a deploy went sideways and the pipeline cheerfully auto-rolled-back, the DROP INDEX grabbed an exclusive lock for a few seconds. On our busiest table, that was a brief but total outage. Sit with that for a second: the rollback - the thing I'd added specifically to keep us safe - was the incident. I found a writeup that argues exactly this point, that in high-load systems a rollback-first mindset manufactures the worst failure modes, and reading it was a little embarrassing, because it named a thing I'd been doing to myself for years.

Once I saw it, I saw it everywhere. Dropping an index or column freezes the fast path with exclusive locks at exactly the moment traffic is highest. Data migrations lose information, so recreating the original is wishful thinking. Rolling code back while the DB is half-migrated leaves a ghost state nobody can reason about mid-incident. And teams that fear schema changes quietly stop shipping the refactors they actually need - which is the slow, invisible cost that hurts most.

The irreversibility one deserves a concrete example, because people underestimate it. We had a migration that normalized free-form addresses into city, country, and postal_code, then dropped the original address column. The down gamely tried to reconstruct the address by string-joining the pieces back together - but the formatting, the ordering, the apartment numbers were simply gone. That migration was irreversible without data loss, full stop. Writing a fantasy down for it was worse than useless; it handed me false confidence I'd cash in at the worst possible time.

The thing that finally clicked

The fix turned out to be a discipline, not a gem. It's what Martin Fowler calls Parallel Change, and what database folks call expand and contract. Every change becomes a little sequence of small, individually-deployable steps, each one compatible with the app version before it. You expand - add new structures without blocking, and leave the old ones alone. You migrate - backfill, dual-write, move reads over gradually. Then you contract - remove the legacy stuff, but only after a full compatibility window has passed.

That compatibility window is the part everyone skips and everyone regrets. The new schema has to keep supporting the previous app version for at least one deploy cycle. Do that, and rolling code back is a non-event - the old code still runs perfectly against the expanded schema, no ghost state, no cold sweat. The whole trick is just refusing to collapse those three steps back into one clever migration, no matter how much your inner tidiness gremlin wants you to.

Dual-writes are the bridge

During the window I write to both the old and new structures, so either app version reads consistent data, and I move reads over slowly:

class User < ApplicationRecord
  has_many :user_statuses, -> { order(:effective_from) }

  def status=(new_status)
    if has_attribute?(:status)
      write_attribute(:status, new_status)  # legacy field
    end
    user_statuses.create!(                  # always maintain the new system
      status_type: UserStatus.status_types[new_status] || 0,
      effective_from: Time.current
    )
  end
end

When touching every write path felt too risky, I leaned on a database trigger to auto-sync legacy writes into the new table for the length of the rollout, then dropped the trigger in the contract phase. Either way the rule is the same, and it's the whole point: nobody, ever, reads a half-populated new table.

Never take a lock you can't afford

The other half of zero-downtime is refusing to grab a lock that'll hurt. The rule the Rails migrations guide and strong_migrations both nudge you toward: add nullable columns with no default (fast), backfill asynchronously in batches, then add the NOT NULL constraint in a separate later migration.

class AddColumnWithoutDowntime < ActiveRecord::Migration[7.0]
  disable_ddl_transaction!

  def up
    add_column :large_table, :new_field, :string  # fast, no default
    queue_background_migration('FillNewFieldJob')  # backfill in batches
    # change_column_null later, once backfill completes
  end
end

The background job walks the table in ordered batches and re-enqueues itself until it runs out of rows, so a table with tens of millions of rows never holds one long transaction or a table-wide lock. On Postgres the same instinct means building indexes concurrently - CREATE INDEX CONCURRENTLY skips the write-blocking lock, at the cost of a second table scan. Cheap trade. Take it.

The rehearsal that ended most of my incidents

Here's the embarrassingly cheap thing that fixed the most pain: I started running migrations against production-sized data in staging before they ever touched production. A change that runs in 40 milliseconds on a seed database can run for 40 minutes against 50 million rows, and there is no way to feel that difference on your laptop. So I'd generate a production-scale dataset, run each pending migration inside a transaction I rolled back for repeatability, and record duration, memory delta, and any blocking queries it kicked off. The output was a short report the team actually read before sign-off. Nine times out of ten the rehearsal caught the problem - a missing concurrent flag, an unbatched backfill - back when it was still boring to fix, which is the only time fixing anything is fun.

Make the safety rails automatic

Discipline that depends on everyone remembering is discipline that evaporates at 2 AM. So I pushed the rules into tooling. If you're on Rails, the strong_migrations gem already codifies most of this and is the fastest possible start - it fails the build on unsafe operations and prints the safe rewrite right there in your face. I layered a little duration estimator on top. A CI check refuses the deploy if a recent migration adds a NOT NULL column with no default, creates a blocking index, or is estimated to run long:

if total_estimated_time > 300  # 5 minutes
  unless ENV['FORCE_LONG_MIGRATION'] == 'true'
    puts 'Set FORCE_LONG_MIGRATION=true to proceed'
    exit 1
  end
end

I also instrument the migrations themselves - subscribing to sql.active_record notifications, logging any migration query over a second and pushing its duration to StatsD, so a slow migration shows up on a dashboard instead of in an incident channel. And before anything risky runs in production, a safety-net step snapshots the affected tables so there's a credible recovery point. Estimators, circuit breakers around big data migrations, a staging rehearsal task - you build the set once and reuse it across every service, and it pays you back forever.

How it feels now

Forward-only evolution didn't just change my migrations, it changed how the deploy button feels. Instead of fear I have a checklist: design for backward compatibility, expand first and contract later, split the scary change into small ones so data migrates asynchronously while reads drift over, and recover forward with fixes and checkpoints instead of fantasy rollbacks. None of it is clever. It's mostly the willingness to turn one terrifying migration into three boring ones - and I've made my peace with boring.

The part I didn't expect was cultural. When schema changes stopped being scary, the team stopped avoiding them, and all those long-postponed refactors we'd been quietly routing around finally shipped. That's the real payoff. If your database changes still page your on-call, start with just two things: the compatibility window, and the pre-deploy check. They're the two highest-leverage moves, and they're the ones that let you sleep.

Sources & further reading

Martin Fowler - Parallel Change (expand and contract): https://martinfowler.com/bliki/ParallelChange.html
strong_migrations - catch unsafe migrations in development (GitHub): https://github.com/ankane/strong_migrations
PostgreSQL - CREATE INDEX, including the CONCURRENTLY option: https://www.postgresql.org/docs/current/sql-createindex.html
Rails - Active Record Migrations guide: https://guides.rubyonrails.org/active_record_migrations.html
A thorough end-to-end zero-downtime schema evolution playbook someone published, estimators and staging rehearsal task included: https://dorokhovich.com/blog/rails-database-schema-evolution?utm_source=devto&utm_medium=article&utm_campaign=success-story&utm_content=rails-database-schema-evolution

The Bug That Made Me Fall Back in Love With rack.response_finished

Menshikov Vasil — Sun, 09 Aug 2026 08:50:43 +0000

There is a particular kind of production bug that doesn't feel like a bug. It feels like weather. Every few days something leaks, a worker gets cranky, p99 creeps up during a traffic spike, and you shrug and restart it because it clears up on its own. For the better part of a year that was my relationship with rack.response_finished - except I didn't know that was the name of my problem yet. I just knew our Rails app was quietly bleeding resources and I kept blaming the wrong things.

This is the story of how it finally clicked, why it bugged me for so long, and how I migrated a high-traffic app to rack.response_finished without spending a single night watching a dashboard with my stomach in knots.

Why this bugged me for years

Our symptoms were the vague kind that make you feel a little crazy. Occasional resource leaks under load. Tail latency that crept up when traffic spiked and settled back down when it didn't. Metrics that never quite matched what I believed was happening. We streamed large files and Server-Sent Events, so the Rack triplet [status, headers, body] was returning long before the last byte ever reached the client.

Here's the piece I genuinely did not appreciate for months: our middleware was freeing resources - closing DB connections, clearing caches, wiping thread-locals - well before the client had the full response. I found a lovely deep-dive on exactly what happens between returning a Rack triplet and delivering that last byte, and reading it was the moment the fog lifted. All those vague symptoms suddenly had one concrete cause. We'd been treating a lifecycle bug as an infrastructure flake - restarting workers, bumping pool sizes, side-eyeing the load balancer. It was none of that. It was an assumption I'd baked into how we wrapped response bodies, and I'd never once questioned it.

The thing I'd been doing without thinking

For years, if you wanted a "run this when the request is done" hook, you wrapped the body in Rack::BodyProxy. It's a tidy little object that calls your block when the wrapped body closes. Elegant on paper. The trouble is every middleware that wanted a callback added its own wrapper, so a single request ended up dragging a Russian-doll stack of proxies around:

original_body = ["Hello World"]
body = Rack::BodyProxy.new(original_body) { logger.info "Request finished" }
body = Rack::BodyProxy.new(body) { metrics.record_latency }
body = Rack::BodyProxy.new(body) { cleanup_thread_locals }
body = Rack::BodyProxy.new(body) { close_db_connections }

The Shopify Rails infrastructure folks wrote up this exact pain in "Friendship Ended with Rack::BodyProxy", and honestly it was validating to read, because it named three things that had been nagging at me.

First, allocation pressure. Every proxy is one more object. At tens of thousands of requests a second, even tiny per-request allocations add up - and because these callbacks capture closures, the objects hang around long enough to get promoted into older GC generations, which is exactly where you don't want churn.

Second, timing you can't trust. #close gets called by the server, sure, but the spec never promised it happens after the client has everything. Depending on the server and buffering, my callbacks could fire before, during, or after transmission. I was cleaning up at a moment I couldn't actually pin down.

Third - and this is the one that was actually paging me at 3am - exceptions skipped cleanup entirely. If the body raised while iterating, the proxy's callback might just never run:

class ProblematicBody
  def each
    yield "Part 1"
    raise "Something went wrong"  # BodyProxy#close might not be called
    yield "Part 2"
  end
end

That right there was the leak. A stream raises halfway through a big download, the cleanup silently gets skipped, and the resource just... stays open. Multiply by traffic and you get weather.

The thing that finally clicked

The replacement is so much calmer, and I mean that emotionally as much as technically. Instead of N proxy objects, there's one standard key in env holding an array of callbacks:

def call(env)
  callbacks = env["rack.response_finished"] ||= []
  callbacks << lambda do |env, status, headers, error|
    # Runs AFTER complete response delivery
    cleanup_resources
    log_metrics(status, headers)
  end

  @app.call(env)
end

What sold me wasn't the elegance, it was the promise. Unlike BodyProxy#close, these callbacks are guaranteed to run in three cases: a clean finish after all data is sent, an application exception even if the body never started iterating, and a server exception during network trouble. Each callback gets (env, status, headers, error), and per Rack's Lint spec they fire in reverse registration order, so you can branch on what actually happened. It shipped as part of Rack 3.x, and Puma added real server-side support (puma#3681) so it isn't a quiet no-op in production.

The one guarantee I kept coming back to: the callbacks run even when the body raises mid-stream. That single sentence killed my entire class of leaks. I remember reading it and feeling almost annoyed at how simple the fix was.

How I migrated without holding my breath

I couldn't flip everything at once - our services spanned Rack versions, and I'm allergic to big-bang changes on infrastructure I can't fully see. So the critical middleware learned to speak both dialects and pick whichever the server offered:

class SafeMigrationMiddleware
  def initialize(app)
    @app = app
  end

  def call(env)
    status, headers, body = @app.call(env)

    if env["rack.response_finished"]
      register_new_callback(env)
    else
      body = wrap_with_proxy(body)  # fallback
    end

    [status, headers, body]
  end

  private

  def register_new_callback(env)
    callbacks = env["rack.response_finished"] ||= []
    callbacks << method(:cleanup_resources)
  end

  def wrap_with_proxy(body)
    Rack::BodyProxy.new(body) { cleanup_resources }
  end

  def cleanup_resources(*)  # accepts any number of args
    logger.info "Request completed"
  end
end

The cleanup_resources(*) splat is on purpose - the new API hands you four arguments, the proxy hands you none, and a splat lets one method serve both paths without a branch. Small thing, but it made the diff read cleanly, which matters to me more than I'll admit.

Then I refused to guess. I put a StatsD counter on each path so I could watch, in real numbers, how much traffic had moved to the new mechanism:

if env["rack.response_finished"]
  StatsD.increment('middleware.response_finished.new_api')
  register_new_callback(env)
else
  StatsD.increment('middleware.response_finished.fallback')
  body = wrap_with_proxy(body)
end

The mistake I almost shipped

Callbacks run in the same thread as the request. That makes thread-local cleanup delightful, and it makes slow work a trap. My first draft casually fired off a notification email inside the callback, which is a wonderful way to block a worker for seconds at a time. Fast cleanup belongs in the callback; anything with I/O belongs in a background job:

# Good: fast cleanup
callbacks << lambda do |env, status, headers, error|
  Thread.current[:request_id] = nil
  ActiveRecord::Base.clear_active_connections!
end

Rails itself gets happier here too: ActionDispatch::Executor can now reliably clear thread-locals right after the response completes, and gems like rack-timeout, newrelic_rpm, skylight, and sentry-ruby line up their timing and errors far more accurately.

How it feels now

I want to be honest about the size of the win, because it's easy to oversell a refactor you're proud of. Average latency barely moved - most requests were never the problem. What changed was the shape of things. Objects allocated per request dropped on our streaming endpoints, major GC ran a little less often, and the p99 tail got quieter during spikes because fewer long-lived closures were being promoted into old-gen. And the leak alerts that used to page me during big downloads simply stopped, because the cleanup now runs even when the body raises.

The order of operations mattered as much as the code. I audited every middleware that touched BodyProxy, bumped Rack to 3.x in development first, shipped the dual-API middleware, and let the StatsD counters tell me when the new path dominated. I only pulled the fallback after a service had run for weeks at effectively 100% new-API with zero callback errors. I left it in longer than felt necessary, on purpose - rushing that last step is exactly how you turn a calm migration into a scary one. Profiling before and after with memory_profiler and ruby-prof gave me actual numbers instead of a vibe, which is the difference between "trust me" and a graph.

If I could hand one note back to the version of me who kept restarting workers: the leak was never infrastructure. It was an abstraction I'd stopped questioning because it looked so tidy. BodyProxy was elegant right up until it wasn't, and the replacement is simpler and more honest about what it promises. That combination - less clever, more trustworthy - is the trade I'll take every single time now. If you run a busy Rack app, start the dual-API pattern today; you get to learn the new mechanism at zero risk, and future-you gets to sleep.

Sources & further reading

Rack - Add rack.response_finished to Rack::Lint (PR #1952, the spec and reverse-order contract): https://github.com/rack/rack/pull/1952
Rack CHANGELOG (where rack.response_finished and Rack 3.x changes land): https://github.com/rack/rack/blob/main/CHANGELOG.md
Puma - Add support for rack.response_finished (server-side implementation, PR #3681): https://github.com/puma/puma/pull/3681
Rails at Scale - "Friendship Ended with Rack::BodyProxy" (Shopify's writeup of the same migration): https://railsatscale.com/2025-08-26-friendship-ended-with-rack-bodyproxy/
Rack::BodyProxy API documentation (RubyDoc): https://www.rubydoc.info/gems/rack/Rack/BodyProxy
A production migration writeup someone put together, with the full dual-API middleware and monitoring metrics laid out: https://dorokhovich.com/blog/rack-response-finished?utm_source=devto&utm_medium=article&utm_campaign=success-story&utm_content=rack-response-finished

How I Rescued My Broken Local Kubernetes Development Environment in One Afternoon (Docker + k3d + Tilt)

Menshikov Vasil — Sat, 08 Aug 2026 17:16:58 +0000

There's a specific flavor of shame in a local Kubernetes development environment that only you can't get working. Everyone else on the team runs kubectl get pods and gets an answer. You run it and get connection refused, again, and you quietly start to wonder if the problem is you. It wasn't me, it turned out - it was that I'd never actually built the thing, I'd just accreted it. This is the story of how I tore my broken setup down and rebuilt it, deliberately, in a single afternoon.

Why this bugged me for years

I'd inherited a FastAPI service - Python 3.12, listening on 8080, backed by PostgreSQL - that the team ran on "local Kubernetes." That phrase meant something different on every laptop, and mine was easily the worst offender. kubectl installed three different ways. A Docker Desktop that half-started on a good day. A helm binary that wasn't on my PATH. And, crucially, no cluster at all. Every single kubectl command greeted me with the same connection refused, and I'd learned to just... route around it, which is the worst possible response and exactly the one I kept choosing.

I burned two full mornings guessing. Then I got tired of guessing and followed a single opinionated write-up that treats the workstation itself as the deliverable - the idea being that you're not done until kubectl get pods -A responds without errors against a real local cluster. That reframing did something to my brain. If you want the full chapter with every OS variant, someone wrote up the exact workstation setup here, and it's the map I wish I'd had two mornings earlier.

The thing that finally clicked: install bottom-up

The mindset shift that fixed everything was embarrassingly simple - install in dependency order. Docker first, because nothing runs without it, then kubectl, k3d, helm, and Tilt last. On my Mac that collapsed to almost one command:

# Docker Desktop from docker.com (daemon + GUI), then:
brew install kubectl helm k3d tilt

k3d is the quiet hero of this whole story. It's just k3s packaged into Docker containers - k3s being a lightweight, CNCF-certified Kubernetes distribution - so a local cluster is featherweight next to "real" Kubernetes. If you're agonizing over k3d vs kind vs minikube, here's what decided it for me: k3d spins up a cluster in under five seconds, ships a built-in image registry that Tilt adores, and barely touches RAM, whereas minikube boots a whole VM and kind - Kubernetes-IN-Docker, originally built to test Kubernetes itself - sits somewhere in the middle. For a tight inner dev loop I rebuild dozens of times a day, and those saved seconds compound into genuinely real time. The one hard requirement is Docker 20.10.5 or newer; k3d's own repository lists a working Docker install as the sole prerequisite. Everything else is just host headroom - 8 GB of RAM is a comfortable floor, 16 GB means you never think about it, and you'll want 10 to 20 GB of free disk for images and layers.

On Linux the shape is identical, except you install Docker Engine with no GUI and you must add yourself to the docker group or every command demands sudo:

sudo usermod -aG docker $USER
# then log out and back in so the group takes effect

One caution I learned the hard way: those curl ... | bash installers for k3d, helm, and Tilt are convenient, but you're piping someone else's code from the internet straight into a shell. On a work machine I now download the script, skim it, then run it - or reach for the apt/dnf packages the official docs offer for kubectl and helm, which are cleaner to maintain long-term anyway.

Naming what each tool actually does

Here's the confession at the heart of this: before that afternoon, I genuinely could not have told you which tool did what, and that fuzziness is exactly why my setup was so fragile. The mental model that finally stuck goes like this. Docker builds images and runs the containers everything else sits inside. k3d spins up the cluster itself - k3s in Docker - and tears it down just as fast. kubectl and helm manage resources and packaged charts inside that cluster. Tilt runs the fast inner dev loop, watching your files and rebuilding and redeploying on every save, and its Live Update can even sync changed files straight into a running container, collapsing edit-build-push-deploy into seconds. And k9s is the optional terminal UI that replaces a wall of kubectl commands.

Once I could name the job of each binary, the errors stopped being mysterious and started being addressable. A connection refused is Docker. A stuck rebuild is Tilt. A missing chart is helm. The abstraction layers finally lined up in my head, and honestly that was more valuable than any single command I learned.

The smoke test that ends the guessing

This is the part I wish I'd run on day one instead of day three. Rather than debugging four tools in isolation, you prove the whole chain works together at once: create a throwaway cluster, switch to it, look at the system pods, delete it.

k3d cluster create dev
k3d kubeconfig merge dev --kubeconfig-switch-context
kubectl get pods -A
k3d cluster delete dev

When that kubectl get pods -A printed coredns, traefik, and metrics-server all sitting in Running, I actually said it out loud to my empty room: the workstation was ready. That single sequence is the entire difference between "I think it's installed" and "I proved it works," and I've never set up a machine without it since. Naming the cluster dev and reusing it later is deliberate too - k3d makes clusters so cheap to create and destroy that a named, reusable dev cluster becomes the stable target for the rest of your toolchain.

The gotchas that used to eat my mornings

Almost every failure at this stage is one of a small handful of classics, and just running the list saved me hours. The most common by far is simply that the Docker daemon isn't running - k3d and Tilt both die with connection refused, so start Docker Desktop or sudo systemctl start docker and wait for it to fully come up. After a reboot you may hit port already in use, in which case just recreate the cluster.

The trickiest one, and the one that bit me hardest, is too many open files. Pods and Tilt both watch files, and they collide with system inotify limits; the symptom shows up as too many open files in pod logs or the kubelet. On a Linux or Docker host the fix is:

sudo sysctl fs.inotify.max_user_watches=1048576
sudo sysctl fs.inotify.max_user_instances=8192

Make it survive reboots by putting those in a file under /etc/sysctl.d/, and as a fallback for stubborn machines you can create a k3d cluster with only a server node and no agents, which cuts the total open files.

Windows has its own trap: do all your dev work inside WSL2, including the repo itself. A teammate kept the code on C:\ and ran the tools from WSL2, and the result was a crawling filesystem and Tilt file sync that silently refused to trigger uvicorn --reload. Move the repo into the WSL2 home directory and the whole problem evaporates. And two quieter ones round it out - if Docker Desktop refuses to start with a virtualization complaint, enable hardware virtualization (VT-x / AMD-V, plus SLAT for WSL2) in BIOS/UEFI, and if you try to squeeze into 4 GB, expect OOM once PostgreSQL and a few pods pile in. Give Docker 8 GB or more under Settings then Resources and the random pod deaths stop.

The little upgrade: k9s made me delete half my muscle memory

Instead of hammering kubectl get pods, kubectl describe, and kubectl logs all day, I now open k9s - a terminal UI that continually watches the cluster - and navigate with arrow keys. Logs, restarts, deletes, all from one screen.

brew install derailed/k9s/k9s

For anyone still finding their feet, it's a genuinely lovely visual replacement for dozens of repetitive commands, and I felt a small pang deleting keystrokes I'd spent years memorizing.

How it feels now

The change is night and day, and it's less about any single number than about the shift from flailing to knowing. What used to be two mornings of guessing is now a checklist-driven afternoon. kubectl get pods -A returns system pods in Running instead of connection refused. Tilt hot reload, which used to be silently broken behind that WSL2 path issue, now fires on every save. Debugging a pod is arrow keys in k9s instead of three kubectl invocations each time. And the biggest one, the one that's hard to put in a table: I'm actually confident it works now, because I watched the smoke test pass with my own eyes.

With the toolchain proven, the natural next step is spinning up a real dev cluster for the service - built-in registry, port forwarding, the works - rather than a throwaway. But if you're setting this up for the first time, please don't improvise the way I did for two mornings. Install bottom-up, run the smoke test before you touch a line of application code, and keep the gotcha list within reach.

The thing that genuinely stuck with me is this: a local Kubernetes environment is not "done" when the binaries are installed. It's done when you've watched the smoke test pass. I'd spent years treating my dev environment as something that accreted rather than something I built, and the fix was simply to build it on purpose, prove it, and only then start stacking real work on top. Prove it first. Everything after that is easier.

Sources & further reading

k3d - k3s in Docker, official documentation: https://k3d.io/
k3d install requirements and cluster options (GitHub): https://github.com/k3d-io/k3d
kind - Kubernetes IN Docker, official documentation: https://kind.sigs.k8s.io/
Tilt Live Update tutorial (official docs): https://docs.tilt.dev/tutorial/5-live-update.html
k9s - Kubernetes terminal UI (GitHub): https://github.com/derailed/k9s
A full field-tested workstation write-up someone put together, with every OS variant and the complete gotcha list

A Local Kubernetes Tools Comparison That Finally Unfroze Me: Stop Comparing Them, Sort Them Onto Shelves

Menshikov Vasil — Fri, 07 Aug 2026 08:53:58 +0000

Here is the confession I don't love making: I once lost an entire week to a decision that should have taken an afternoon. My team wanted local Kubernetes, I volunteered to pick our stack, I opened a "getting started" guide, and got hit with an incantation - k3d, kind, minikube, Helm, Kustomize, Tilt, Skaffold, DevSpace, Telepresence, mirrord, k9s. I froze. Every local Kubernetes tools comparison I read stacked a dozen names side by side as if they were rival answers to one question, and I couldn't pick anything. What eventually unfroze me wasn't a better comparison - it was realizing they don't belong on the same shelf.

Why this bugged me for a whole week

I'm a fairly decisive person, which made the paralysis extra humiliating. I figured picking a stack would be a lazy afternoon of reading. Instead I kept hitting the same wall: do I need k3d or Helm? Is Tilt an alternative to minikube? Where on earth does Telepresence fit? Every post read like one giant menu where everything competed with everything, and I genuinely could not tell whether I was choosing one tool or ten.

The reframing that saved me came from an overview of the local Kubernetes tooling landscape someone wrote up: each tool handles its own little piece, and once you sort them onto shelves, the whole picture goes quiet. Let me hand you the shelves, because they cost me a week and they might save you one.

The first fork nobody told me about: local cluster vs remote connection

Before you compare any specific tool, there's a bigger split I didn't even know existed. There are two fundamentally different ways to get a near-production loop. The first is to run all of Kubernetes on your own machine - spin up a small but real cluster locally with k3d, kind, minikube, Docker Desktop, or Rancher Desktop, and deploy with the same manifests you'll use in prod. Fully local, no network dependency. That's the path we took. The second is to keep your code local but plug it into a remote cluster, so your local process behaves as if it were a Pod living on staging. That's the world of Telepresence (which builds a VPN-style tunnel, needs root, and modifies the cluster), mirrord (which injects into your process via LD_PRELOAD or DYLD_INSERT_LIBRARIES, needs no root, and mirrors by default so it's safe for shared staging), and Gefyra (VPN-based, no root, Docker-only).

The remote approach is genuinely great, but it needs a live remote cluster to maintain and network access - an extra layer of magic I didn't want to impose on a team still finding its feet. So: everything local. That one decision alone eliminated three tools from my "must choose now" list, and I felt the paralysis loosen its grip a little.

The foundation shelf: Docker and kubectl

These sit under everything else. Docker is the container engine, and here's the fact that surprised half my team: local clusters run their Kubernetes nodes as ordinary Docker containers. A node is a container with Kubernetes inside it, and inside that run your Pods. No Docker, no cluster. kubectl, meanwhile, is the official CLI and your main channel to the cluster - apply manifests, read logs, forward ports, shell in. Every accelerator like Tilt or Skaffold and every visual tool like k9s runs on top of kubectl, calling the same Kubernetes API underneath, so it's worth actually knowing even if you end up living inside Tilt all day. A small bonus that trips people up: Kustomize is built right into kubectl via kubectl apply -k.

The cluster shelf: k3d, kind, minikube, Docker/Rancher Desktop

This shelf holds the "box" your app goes into, and all of these give you a real Kubernetes API - they just differ in startup speed, resource appetite, and features. k3d is a wrapper that runs k3s, a minimal certified Kubernetes, inside Docker; it has the fastest startup, the lowest memory footprint, and a handy built-in registry, which makes it my pick for everyday dev (with the honest caveat that it's a community project, not an official Rancher/SUSE product). kind, Kubernetes IN Docker, runs upstream Kubernetes via kubeadm - CNCF-certified, multi-node and HA capable, and built to test Kubernetes itself and CI conformance. minikube runs Kubernetes in a VM or via a docker driver with a big addon ecosystem and remains the classic learning tool. Docker Desktop offers single-node Kubernetes behind a checkbox, convenient if you already run it but limited in configuration. And Rancher Desktop is a GUI app on k3s for native local dev and testing. The 2025-2026 consensus lands on kind or k3d for the best balance of speed and capability, and k3d in particular starts fast and stays light, which is exactly why it's so pleasant to recreate constantly while you're still learning and breaking things.

The manifest shelf: Helm and Kustomize

Production manifests usually live as Helm charts or Kustomize overlays, and this is the whole reason a local cluster beats docker-compose - it accepts these same formats. Helm is the Kubernetes package manager: your app becomes a chart with a Chart.yaml, a values.yaml, and a templates/ directory of Go templates, and in return you get versioning, releases, rollbacks, dependencies, and repos. It's been a CNCF graduated project since 2020. Kustomize takes the opposite bet - no templates at all; you write base manifests and apply overlays, which are patches, per environment. It's built into kubectl since 1.14, though on its own it has no packaging, versioning, or rollback, so people bolt on Argo CD or Flux for that. In practice the two complement each other: Helm to package and distribute, Kustomize to tidily patch per environment.

The accelerator shelf: Tilt, Skaffold, DevSpace

The bare loop - edit, build, push, deploy, look - is excruciatingly slow in Kubernetes, and this shelf exists to automate the building, updating, and watching. Tilt (Apache-2.0) automates watch-build-update, and its signature move is Live Update, syncing changed files straight into the running container without a full rebuild; you configure it in a Tiltfile written in Starlark, a Python-like language, and it ships a genuinely clear web UI that's a big plus for beginners. Skaffold, from Google, is a build/push/deploy pipeline with file sync, a dedicated skaffold debug command, and profiles, deploying via kubectl, Helm, or Kustomize. DevSpace is a CLI that leans into bidirectional sync, hot reload, and in-cluster dev containers. Two adjacent names worth not confusing with these: Okteto is about cloud dev environments, which is closer to the remote approach, and Garden is graph-based multi-service automation, which is overkill for a single service.

The convenience shelf: k9s and friends

Once you have more than one Pod, typing kubectl get, logs, and describe all day gets old fast. k9s is a terminal UI - a "visual kubectl" - with vim-style navigation, live Pod, node, and deployment views, and hotkeys (l for logs, s for shell, d for describe). My favorite touch: it color-codes contexts, so you can paint prod an alarming red and never wreck it by muscle memory again. Also worth installing when the day comes are kubectx and kubens for fast context and namespace switching, stern for multi-pod log streaming, and krew, the kubectl plugin manager.

Why we landed on k3d + Tilt

Once the tools were on shelves, the choice basically made itself, because local Kubernetes dev really only has two core pains: you need a realistic cluster that accepts real production manifests, and you need a fast edit-to-result loop. k3d covers the first - a real Kubernetes API via k3s, fast startup, modest memory, so we get genuine parity with our Helm and Kustomize manifests instead of a docker-compose surrogate, and without turning laptops into space heaters. It's cheap to recreate, which matters enormously while a team is learning. Tilt covers the second - Live Update compresses the loop from minutes to seconds, codifies the whole setup in a Tiltfile, deploys through real manifests, and shows a web UI where builds, logs, and state all live in one place. Together they're fully local, open source, and free.

The alternatives are all worthy, and I want to be fair to them: Skaffold and DevSpace solve the same loop and are worth trying if their style fits your team, Telepresence and mirrord are the remote approach if you'd rather lean on a shared cluster, and Okteto and Garden solve adjacent problems entirely. For a single service headed toward its first deploy, k3d + Tilt was simply the shortest, clearest path.

How it feels now

The before-and-after is almost embarrassing in how much of it was in my head. Before the map, my mental model was "ten rival tools, one agonizing choice," the decision took a paralyzed week, and the stack stayed stubbornly undecided the whole time. After the map, the tools sat on distinct shelves, the decision collapsed into an afternoon, and I had a stack - k3d + Tilt - that I could actually explain the reasons for.

What stays with me is why I was stuck, because it wasn't complexity. I was overwhelmed because I kept comparing a cluster to a package manager to a file-syncer as though they answered the same question, and they simply don't. The moment I stopped ranking them and started sorting them - foundation, cluster, manifest, accelerator, remote, convenience - the choice made itself. If you're standing at the same frozen starting line, don't reach for one more comparison table. Reach for shelves. That reframing is the thing I'd hand back to the paralyzed version of me who lost a week he didn't need to.

Sources & further reading

How Tilt's Live Update Fixed My Kubernetes Dev Loop and Gave Me an Hour a Day Back

Menshikov Vasil — Thu, 06 Aug 2026 09:29:19 +0000

Change one line of Python. docker build. docker push. kubectl rollout restart. Wait for the pod. Check the logs. Repeat, forty times a day. I actually did the arithmetic on what that ritual was costing me one afternoon, and the number was ugly enough that I nearly gave up developing on Kubernetes entirely. Instead I found Tilt's Live Update, rebuilt my Kubernetes dev loop around it, and got roughly an hour a day back. This is how that went.

Why this bugged me for weeks

My team had just moved local development onto a k3d cluster for parity with prod. Good call in principle - and in practice my day looked like this for every single code change to our FastAPI service myapp:

docker build -t k3d-registry.localhost:5000/myapp .
docker push k3d-registry.localhost:5000/myapp
kubectl rollout restart deploy/myapp -n myapp
kubectl logs -f deploy/myapp -n myapp   # wait, watch, hope

A minute or two, minimum, per edit. Ten edits and half an hour of my life had dissolved into progress bars. But the raw time wasn't even the worst of it - the context switching was. By the time the pod finally came up, I'd forgotten what I was checking in the first place. Kubernetes had taken my tight, genuinely joyful edit-run loop and turned it into molasses, and I could feel my patience for the whole platform draining by the day.

I was one bad afternoon away from ripping the app back out of the cluster - and throwing away all the parity we'd just bought - when I came across a write-up on using Tilt for a fast local Kubernetes dev loop. It described the exact pain I was drowning in, and it gave me a way out.

The three-line Tiltfile that started it

Tilt's config is a file called Tiltfile (no extension) written in Starlark, which is basically a trimmed-down Python. Run tilt up and it executes top to bottom, builds a graph of what to build and deploy, then watches your files and rebuilds only what changed. My first working version was almost embarrassingly short:

# Tiltfile

# How to build the myapp image from the current directory
docker_build('k3d-registry.localhost:5000/myapp', '.')

# What to deploy — our existing manifests
k8s_yaml(['k8s/deployment.yaml', 'k8s/service.yaml'])

# Fine-tuning: forward the container's port 8080 to localhost:8080
k8s_resource('myapp', port_forwards='8080:8080')

Three functions carry almost any Tiltfile, all documented in Tilt's API reference: docker_build for how to build, k8s_yaml for what to deploy, and k8s_resource for the fine-tuning like port-forwards and grouping. The clever bit is how Tilt ties them together - it scans the YAML, finds the workload, matches the built image to the manifest by tag, swaps in a uniquely-tagged fresh build, and deploys. The one rule I had to internalize immediately: the tag in docker_build must exactly match the image: in deployment.yaml. Mismatch it and Tilt cheerfully builds one image while the Deployment asks for another - instant ImagePullBackOff, and a confusing ten minutes before you realize what you did.

After tilt up, my service was live at localhost:8080 with no manual kubectl port-forward. Already better. But the real prize came next.

Live Update: the feature that gave me my day back

This is the part that actually changed things. The normal rebuild-image-then-redeploy-pod cycle takes tens of seconds even for a tiny service - Docker layers, registry push, cluster pull, pod restart, all of it. Live Update skips the whole dance: instead of rebuilding, Tilt copies changed files directly into the running container and, if needed, runs commands there. Seconds, not minutes.

You configure it inside docker_build via live_update, and the steps run in a strict order:

docker_build(
    'k3d-registry.localhost:5000/myapp',
    '.',
    live_update=[
        # if deps change, just rebuild the whole image
        fall_back_on(['requirements.txt']),
        # sync code instantly into /code/app (the WORKDIR from our Dockerfile)
        sync('./app', '/code/app'),
        # reinstall deps only if that file changed
        run('pip install -r requirements.txt', trigger=['requirements.txt']),
    ],
)

Tilt's decision tree is refreshingly simple, and the Live Update reference spells out the exact ordering: a fall_back_on file changed means a full rebuild; a file matched by a sync gets a fast live update; a file in the context but covered by no sync triggers a full docker build; an untracked file does nothing. Two rules tripped me up until I read them twice. sync paths must live inside the build context - if Tilt is watching it, you can sync it - and run() cannot come before sync(), because you have to put the files in place before you act on them. Also worth knowing: the very first deploy is always a full one, since Live Update needs an already-running container to copy into.

Hot reload closes the loop

Live Update drops the new file into the container, but how does the running process actually pick it up? For us, delightfully, FastAPI via uvicorn hot-reloads on its own. The launch command:

uvicorn app.main:app --host 0.0.0.0 --port 8080 --reload --reload-dir app

Per uvicorn's settings docs, --reload runs an internal watcher that restarts the process when .py files change, and --reload-dir app narrows the watch so it doesn't twitch on every temp file. So when your framework self-reloads, sync alone is enough: files land in the container, uvicorn's watcher notices, the app reloads, and there's nothing else to wire up.

If your stack can't hot-reload - a Go binary, or uvicorn without --reload - the synced files just sit there while the old process runs stale code, which is the classic "my changes didn't apply" trap that'll cost you an afternoon of confusion. For that case, Tilt's restart_process extension gives you docker_build_with_restart, which re-runs your entrypoint after each sync. For our FastAPI service plain --reload was simpler, and since it's dev-only, we never ship --reload in the production image.

The dashboard I didn't know I needed

tilt up also brings up a web UI at localhost:10350, and it quietly ended my tab-juggling. On one screen I get every resource with two statuses - did the build and deploy succeed, and what is the pod doing right now - along with the Pod ID behind a copy button, endpoints as clickable links, and the part I use constantly: filterable logs, by build versus runtime, by level, by keyword. When a pod won't start, I stopped grep-ing through kubectl logs across three terminals and just filtered the dashboard. There's a Trigger Update button for manual rebuilds, and a manual mode if you'd rather save often and deploy on demand:

trigger_mode(TRIGGER_MODE_MANUAL)
k8s_resource('myapp', trigger_mode=TRIGGER_MODE_AUTO)  # but keep this one automatic

Is Tilt the only option? My honest take

Tilt isn't alone here - it usually gets compared with Skaffold, from Google, and DevSpace. All three build, deploy, and sync for a fast loop, and the real differences come down to UI-first versus CLI-first and Starlark versus YAML. Tilt bets on the visual dashboard and Live Update; the Starlark config is flexible but a steeper climb than YAML, and it shines for beginners and mixed-experience teams precisely because you can see the whole cluster state. Skaffold is CLI-only with YAML config, file sync, profiles, and a dedicated skaffold debug - familiar and declarative, but no visual panel. DevSpace is a YAML CLI with two-way sync, reverse port-forward, and dev containers.

If your team lives in YAML and loves the terminal, Skaffold or DevSpace is a great fit, and I won't pretend that choice is anything but subjective. For getting a newcomer developing comfortably on Kubernetes, though, the dashboard won it for me - less blind fumbling in the terminal, more actually understanding what's happening.

How it feels now

The per-edit time went from a minute or two down to a second or two, and the list of things I run by hand collapsed from build-push-restart-logs to simply saving the file. The image only rebuilds when dependencies change now, not on every keystroke. Logs and status live on one dashboard instead of scattered across N terminals. The port-forward is declared once in the Tiltfile instead of manually re-established every time a pod restarts. And my focus, which used to get shredded on every deploy, stays intact.

The arithmetic that sold me was blunt: forty edits times roughly ninety seconds saved is about an hour a day I got back, and that's before you count the context-switching tax, which I suspect was the bigger number anyway. The whole setup took a single afternoon.

What lingers, though, isn't the hour. It's that the pain was never Kubernetes - it was my loop. A tiny Tiltfile, Live Update, and uvicorn's --reload turned the whole experience from molasses back into the tight edit-run rhythm I'd been quietly grieving, and I got to keep cluster parity while doing it. If you're still rebuilding images by hand and watching progress bars, that's the trade I'd make again in a heartbeat: an afternoon of setup for a loop that finally respects your attention.

Sources & further reading

The Kubernetes Inner Dev Loop That Was Quietly Eating My Sprint - and How I Got Back to Seconds

Menshikov Vasil — Wed, 05 Aug 2026 10:05:16 +0000

For about six months, my Kubernetes inner dev loop was a productivity black hole, and I mean that almost literally: time went in and nothing came out. I was the backend person on a small payments squad, and I got to the point where I quietly dreaded touching the cluster at all. This is the story of what was actually wrong, why it bugged me for so long, and the setup that finally gave me back a dev loop measured in seconds instead of minutes.

Why this bugged me for so long

Our service - I'll just call it myapp, a FastAPI HTTP API on port 8080 backed by PostgreSQL - ran beautifully on a laptop. uvicorn --reload picked up every change in a fraction of a second, the way good local dev should feel. Then we "just" shipped it to Kubernetes, and the instant loop turned into this:

# 1. build the image
docker build -t registry.internal/myapp:dev .
# 2. push to the registry
docker push registry.internal/myapp:dev
# 3. apply the manifests
kubectl apply -f k8s/
# 4. wait for the new Pod and check it's alive
kubectl rollout status deployment/myapp
kubectl logs -f deployment/myapp

Four steps that simply did not exist when we ran things locally. Every single pass was two to five minutes of build, push, wait, check. Multiply that by dozens of iterations a day across the team and you get whole afternoons of people staring at kubectl rollout status. And here's the part that really got under my skin: the wait was juuust long enough that I'd flip to Slack every time. So the true cost was never the four minutes. It was the train of thought that derailed on every deploy, and never quite got back on the rails.

A slow loop is only half the misery, though. The other half is the bugs you genuinely cannot see on a laptop. We shipped a change that passed every local test and then watched the Pod get OOMKilled in staging, because we'd never set a memory limit locally - memory is effectively infinite on a dev box. Another release sat there "running but not responding" because a readiness probe was failing and Kubernetes had quietly yanked the Pod out of the Service endpoints. Nothing in my local world had prepared me for either.

The idea that reframed everything: shift-left, but with fidelity

I got tired of band-aids and went looking for a systematic answer. Someone had written up exactly why the Kubernetes inner loop gets slow and how to close the local-versus-cluster gap, and it put a name to the thing I'd been feeling in my gut: the more your test environment differs from prod, the more bugs leak downstream, where each one costs an order of magnitude more to fix.

The line that really stuck with me was this - testing earlier in a CI container that doesn't match your cluster isn't shift-left. It's just failing faster in the wrong environment. Real shift-left for Kubernetes means validating against real cluster conditions: real resource limits, real probes, real services in the namespace, early, on your own machine. That reframed the whole problem for me, and it pointed at two concrete tools instead of a vague "do better."

The first is k3d, a lightweight wrapper that runs CNCF's k3s inside Docker. Not an emulation of a cluster - an actual one, small enough to live on a laptop. The second is Tilt, whose live-update performs an in-place update of the containers in your cluster, syncing your code straight into a running Pod so iteration drops back to seconds.

What I actually rolled out

I named the cluster dev, put everything in a myapp namespace, and used k3d's built-in registry at k3d-registry.localhost:5000. My first lesson landed the hard way: the local cluster cannot see images from your local Docker daemon. I built myapp:dev, expected the cluster to pick it up, and got a Pod wedged in ImagePullBackOff instead. k3d nodes run their own containerd, isolated from your Docker. You either push to a registry the cluster can reach or import the image explicitly - the k3d registries guide walks through wiring up a local registry the cluster can actually pull from.

The second lesson was the :latest tag trap. If your manifest references myapp:latest, Kubernetes defaults imagePullPolicy to Always and the kubelet re-pulls on every launch even when the image is sitting right there. The Kubernetes image docs spell it out: omit imagePullPolicy with a :latest tag and it becomes Always, while a fixed tag defaults to IfNotPresent. I switched to specific tags - myapp:dev, myapp:<gitsha> - and imagePullPolicy: IfNotPresent, and a whole category of mysterious slowness just evaporated.

Then Tilt took over the loop, and this is the part I still find a little magical. Instead of build then push then apply then wait, saving a file synced the change straight into the running Pod. The numbers matched exactly what the tool promised - Tilt's own writeup frames live-update as deploying code to running containers in seconds, not minutes, and syncing files into a running Pod with hot reload brought my iteration back down to roughly one to five seconds, better than a 95% cut. One detail I'm glad I got right: in the cluster image I run fastapi run, not uvicorn --reload. Reload is dev-only overhead and a leak risk; the speed comes from Tilt's live-update, not from reload inside the container.

How it feels now

The headline is easy to quote - two-to-five-minute iterations became one-to-five-second ones, and "build, push, apply, wait, check" collapsed into "save the file." But honestly the second-order effects were the ones that changed how I work. Because a change now took seconds to see, I started testing smaller increments, one hypothesis at a time, instead of batching five changes into a single painful deploy and then bisecting which one broke. Pull requests got smaller. Review comments got sharper, because a reviewer could pull the branch and watch it run in a real cluster in under a minute.

And the OOMKilled and readiness-probe bugs that used to ambush us in staging now surfaced on the author's laptop, where the person with all the context was sitting right there to fix them. That's the whole economic argument for shift-left in one sentence: a bug caught in the inner loop costs a coffee's worth of attention, while the same bug caught after git push costs a CI run, a reviewer's time, and sometimes a rollback. I didn't measure it to the decimal, but the number of "why is staging broken?" Slack threads dropped to nearly zero within a sprint, and I stopped bracing every time I opened the cluster.

A few things bit hard enough to cost a day each, and I'll pass them on so they don't cost you one. ImagePullBackOff on day one had nothing to do with a bad image - the cluster just couldn't see it, and understanding image delivery into k3d up front would have saved me an afternoon. Running --reload inside the container "because it was fast locally" fought Tilt and masked the real loop, so kill it. Don't confuse the inner and outer loops: this whole exercise is about the inner loop, the single-developer edit-build-run cycle before git push, and the outer loop of CI, GitOps, and integration tests is a genuinely separate beast. And don't skip resource limits locally - the entire point of a production-like local setup is that limits and probes exist on your laptop, so leaving them out just reopens the gap you were trying to close.

I had to actively resist the itch to also "fix" that outer loop in the same push. It's real work, but it's a different problem with different tools, and bundling it would have stalled everything. I fixed the inner loop first, shipped it, and let the win speak before touching anything downstream. Once the loop was fast, I layered the rest of a production-like setup on piece by piece - real Deployment and Service manifests, PostgreSQL inside the cluster, ConfigMaps and Secrets instead of hardcoded values, health probes - one production-like layer at a time (there's a full end-to-end assembly linked in Sources below).

What I keep coming back to isn't the speed number, satisfying as it is. It's that I started using the cluster again instead of avoiding it. A dev loop you dread is a dev loop you route around, and every workaround quietly costs you the fidelity you were supposed to be buying. Getting seconds back didn't just make me faster - it made me willing, and that turned out to be the whole point.

Sources & further reading

k3d-io/k3d — Little helper to run CNCF's k3s in Docker
k3d docs — Using image registries with k3d
Tilt docs — Smart Rebuilds with Live Update
Tilt blog — Fast Kubernetes Development with Live Update
Kubernetes docs — Container Images & imagePullPolicy
A full local-Kubernetes inner-loop write-up someone put together, covering the end-to-end k3d-to-production-like assembly