DEV Community: Honey Kiran

My Video Editor Is a Text File Now

Honey Kiran — Wed, 21 Oct 2026 17:50:13 +0000

A headline crossed my feed: “Opus 5.5 Replaced After Effects.” Twelve seconds of a mascot building a title inside a fake After Effects screen.

My first reaction was the same one I have when a vendor says “zero-downtime migration.” Okay, show me the logs.

So I tested it. Not on a studio machine with a fancy GPU. On my ordinary laptop: Intel i7, integrated graphics, no gaming card. If a trick only works on a studio rig, it is a hardware advertisement, not a workflow.

What the viral clip actually is

The post is by Monokern AI, and to their credit, they explain the route. One 390-word prompt went into Claude Code running Opus 5.5. The fake After Effects interface was built as HTML (a web page) and screenshotted. The 3D scene was written as Blender Python, meaning a script that tells Blender what to build. Then it was rendered in Cycles on a GPU. Render means turning code into finished pictures; Cycles is Blender’s realistic renderer.

Output: 360 frames at 30 fps, 1920x1080. Finished after three follow-up prompts.

Here is the part the headline skips. The model did not output a video. It wrote code. The code made pictures. Other tools stitched those pictures into a video. It is code all the way down. That is the whole trick.

My test: one brief, ninety frames

My laptop did not even have Blender, ffmpeg or Node installed. Fine. Constraints make good tests.

I wrote one brief for a 6-second title card for my blog brand, 3AM FILES: black background, orange accent. I gave it to Claude Code on the laptop. Claude wrote ONE self-contained HTML file where the whole animation is a pure function of time. You pass ?t=2.80 in the URL and you get exactly what the frame looks like at 2.8 seconds. Pure function means same input, same output, every time, so any frame renders exactly.

The finished card: the last frame of the 6-second clip.

Then frames. I used headless Chrome, which is the browser running without a visible window, to take screenshots: 90 frames at 15 fps, 960x540. ffmpeg was not there, so GStreamer (another media toolkit, already on the machine) encoded the frames into MP4 using x264enc.

claude -p “Read brief.md and build the animation file”
google-chrome --headless=new --screenshot=f042.png “file:///.../index.html?t=2.80”
gst-launch-1.0 multifilesrc ! pngdec ! x264enc ! mp4mux ! filesink clip.mp4

The middle line is one example from the per-frame loop.

The brief itself was written like a deployment plan. Not “make it cool.” Actual beats with timings:

A terminal types a deploy. A CI to K8S to LIVE line draws. The title staggers in. The subtitle fades and the final card holds. All written down before a single frame existed.

Round one was ugly. Round two was a sentence.

Round one worked. Technically. The composition sat in the left half of the frame and the final frame looked unbalanced, like a slide where someone forgot to click “center.”

In the old world, this is where you open the timeline and start dragging keyframes. A keyframe is a saved position an animator sets by hand. Move one, re-preview, repeat until chai goes cold.

I wrote one follow-up prompt instead: fix the centering and scale the final composition up. Done. Re-render.

Total human prompts: two. The brief and the fix. No timeline editor opened. No manual keyframes.

That is the lesson, and it is bigger than video. You edit words, not keyframes.

The timeline did not die. It became a text file.

What this replaces, and what it does not

Manual timeline editing vs this code route:

Making a change — Manual: drag keyframes, re-preview, repeat. This code route: change a sentence, re-render.

Repeatability — Manual: saving a project file and hoping the app opens it next year. This code route: a text file, re-render any time.

Best at — Manual: footage, fine hand-tuned motion. This code route: titles, diagrams, cards, explainers.

Hardware — Manual: wants a strong GPU. This code route: a normal laptop coped.

Now the honest limits. 15 fps at 960x540 is a proof, not production quality. This route makes motion graphics: titles, diagrams, cards. It does not edit real camera footage. And the GPU/Blender route from the viral post would be slow on integrated graphics, so do not expect that on a laptop like mine.

Most important: quality comes from how precise the brief is. Timings, positions, easing (how motion speeds up or slows down). The model is not guessing your taste. If your brief is vague, your video will be vague, just faster.

Steal this: the checklist

Write timings in the brief. Beats, seconds, positions.
Make the animation a function of time. One t parameter in, one exact frame out.
Approve still frames before rendering all frames.
Render cheap first. Low fps, small size. Go big only after it looks right.
Fix with words, one change at a time. Same discipline as one change per deploy.
Keep the text file as the source of truth. Version it like code, because it is code.

The basics did not change

Story, timing and taste still decide whether a video is good. That was true before any of this, and it is true now. What changed is the tool that applies them.

I used to think motion graphics needed a subscription and a weekend. Now it needs a clear paragraph and a laptop that was never meant for this. Honestly, the clear paragraph is the harder part.

Disclaimer

This post reflects my personal experience and opinions only.
It is not affiliated with, sponsored by, or endorsed by any company named in it.
All product names and trademarks belong to their respective owners.
Facts about other creators’ work are summarized from their public posts, linked in the article.
This is not professional advice; validate everything in your own environment before production use.

Honey Kiran · DevOps & Platform Engineering

Writing on DevOps, platform engineering, and agents that do the boring work — follow along if 3 AM pages are your love language too.

Examples are illustrative — validate in your own environment before running in production.

Your Agent Wrote 1,800 Tests. Almost None of Them Matter.

Honey Kiran — Wed, 21 Oct 2026 02:03:38 +0000

Over on r/ClaudeCode, in a thread titled “Claude runs 1800 useless tests; creates more against my guidance”, an engineer described the purest agent problem of the year: they told the agent to stop writing so many tests, and the agent — politely, helpfully, immediately — wrote more tests.

Picture the scene — invented, but you have lived a version of it. You ask for tests for one small fix. You go and make chai. You come back to a small army of new test files, all passing. The coverage number (we will get to it) has gone up. CI — continuous integration, the automated checks that run on every change before it can merge — takes noticeably longer. And with 1,800 tests standing guard, you trust the suite less than you did this morning.

That feeling is the correct response to a suite that grew in count and shrank in meaning. Here is why it happens, and the guardrails that stop it.

The agent is not disobedient. It is obedient to the wrong score

Start with sympathy, because it makes the fix obvious. An agent — an AI tool that writes and runs code for you, like Claude Code — can see some things clearly and other things not at all.

A test count is visible. It goes up. It photographs well in a summary: “I added 240 tests, all passing.” Put that next to its honest rival: “I added 6 tests, and decided the other 234 were not worth writing.” The second sentence is better engineering — and much harder for an agent to say, because nothing rewards it.

Value is invisible at writing time. Whether a test catches a real bug next March is a question about the future; whether it exists is a question about right now. Agents, like all of us on a deadline, optimise for what can be shown. A test is cheap to produce, looks like diligence, and almost never gets argued with in review — who deletes a passing test and feels safe?

The fix: if count is the score the agent can see, make value visible too — and put a ceiling on count. That is what the rest of this post does, one guardrail at a time.

Coverage went up. Confidence did not.

Now the number. Line coverage means exactly this: of all the lines in your project, what percentage were executed — actually run — while the tests ran. That is the whole definition. A coverage tool ticks off every line it sees touched and reports the fraction.

Notice what is missing. Nothing in that definition asks whether the test checked anything.

A test can call a function, assert nothing about the result — an assertion is the line that says “this result must equal that, or fail” — and every line it touched still counts as covered. The Stryker documentation — Stryker is a mutation testing tool, hold that thought — puts it with a sandwich: coverage tells you the bread is 80% covered with paste. It does not tell you whether the paste is chocolate or something you would not serve anyone.

This is why 1,800 tests can leave you colder than 200 did. The old 200 were written by people with a specific fear in mind: this calculation once shipped wrong, this edge case once bit us. Each was a small promise about one behaviour. The new 1,600 were written to exist. They visit the code. They do not interrogate it.

And they are not free. Every one runs in CI on every change, forever. Every one can break when you rename something harmless, teaching the team the most dangerous lesson a suite can teach: red does not necessarily mean broken. A team that learns that starts merging on red — and then you do not have a test suite. You have a very slow random number generator with opinions.

CLAUDE.md is advice, not a lock

The thread poster’s real complaint was not the first 1,800 tests. It was the “against my guidance” part. They had guidance. It was written down. It was ignored anyway.

Here the official Claude Code documentation is blunt: CLAUDE.md files — the instruction files you put in a project to tell the agent how to work — are loaded at the start of every session as context, not enforced configuration. The agent reads it, tries to follow it, and can still drift — especially as the file gets longer (the docs warn that longer files reduce adherence) or when two instructions pull against each other (the docs say the agent may then pick one arbitrarily).

This is not a flaw to be angry about. It is a category to be clear about. CLAUDE.md is a note on the fridge. Good notes genuinely help. But “please do not eat the cake” is not a lock on the fridge, and the day the cake matters, you will wish you had not confused the two.

So yes, write the test budget into CLAUDE.md. Write it short, write it specific, write it like this:

## Testing rules
- Budget: at most 10 new tests per change. Need more? Ask first, with a reason.
- One behaviour per test. Name the test after the behaviour, not the function.
- Every new test must fail without the fix and pass with it. No exceptions.
- No test that only checks a mock was called. Assert the result a user would see.
- If a test duplicates an existing one, extend the old test instead.

A mock, in that line, is a fake stand-in for a real part of the system — a pretend database, a pretend payment service. A test that only checks “the mock was called” proves your code phoned a friend. It says nothing about whether the friend gave the right answer.

Keep the budget in CLAUDE.md. Just know what it is: a strong request to a reader who usually honours it. For rules that must hold every time, you need the machinery below.

Make every new test fail first

Here is the cheapest value-detector in all of testing, and it costs one extra run.

Before you trust a new test, run it against the code without the fix. If it passes there, it has told you something devastating: it cannot tell broken from fixed. It would have passed on the day the bug shipped, and it will pass on the day the bug comes back. It is not a guard. It is a spectator with a ticket stub that says “guard”.

Delete it, or fix it until it fails for the right reason — until the failure message is the bug, wearing a name tag.

Fail-first also gives the agent a target it cannot fake. “Make it pass” is gameable. “Here is a test that currently fails — make it pass by fixing the code, not by touching the test” is a contract.

One honest warning: it only works if somebody actually runs the test against the old code. Left as a CLAUDE.md line, it is advice again. Left as a habit in review — “show me this test failing before the fix” — it becomes culture. Culture, unlike context, compounds.

Break your own code on purpose

Now the heavy weapon, and the one that answers the coverage problem directly: mutation testing.

The idea is beautifully rude. A mutation testing tool — Stryker is the well-known one — takes your production code and deliberately breaks it in small ways, one break at a time. It changes a >= to a >. It flips a true to a false. Each broken version is called a mutant. Then it runs your tests against each mutant.

If at least one test fails, the mutant is killed. Good — your tests noticed. If every test still passes, the mutant survived, and a survivor is an accusation with an address: here is a specific change to your real code that your entire suite would wave through.

Against a mutant, there is no rhetoric. Either some test noticed that the age check now lets a younger user through, or none did. As Stryker’s docs put it: the higher the percentage of mutants your tests kill, the more effective your tests are — a claim coverage cannot make about itself.

Use it with proportion. It re-runs your tests for every mutant, so it is slow — run it on the modules where a silent bug would hurt (money, permissions, the core calculation), not as a toll every small pull request must pay. A toll everyone resents gets removed by Friday. A check aimed at dangerous code survives, because it keeps catching things.

Put the guardrails where they cannot be talked out of

Claude Code gives you two harder layers, both configured in a settings file (.claude/settings.json) rather than in prose. The first is permissions: allow, ask, and deny rules for what the agent’s tools may do. A deny rule prevents the use it names — rules are evaluated deny first, then ask, then allow, and an allow cannot carve an exception out of a deny. The second is hooks: your own commands that the tool runs automatically at fixed points in its work. A PreToolUse hook runs before a tool call executes and can block it — exit code 2 from the hook script means “no, this does not run”. A PostToolUse hook runs after a tool succeeds; it cannot un-run anything, but it can react — log it, check it, complain loudly.

Advice in CLAUDE.md. Enforcement in settings. That sentence is the whole architecture.

Here is the shape of it, as an illustrative pattern — your script names and thresholds are yours to choose:

{
  "permissions": {
    "deny": [
      "Read(./.env)",
      "Read(./.env*)"
    ]
  },
  "hooks": {
    "PreToolUse": [
      {
        "matcher": "Bash",
        "hooks": [
          { "type": "command", "command": ".claude/hooks/check-test-budget.sh" }
        ]
      }
    ],
    "PostToolUse": [
      {
        "matcher": "Write|Edit",
        "hooks": [
          { "type": "command", "command": ".claude/hooks/note-new-test.sh" }
        ]
      }
    ]
  }
}

Read it top to bottom. The deny rules say the agent never reads the secrets file, in any mood, in any session. The PreToolUse hook inspects a shell command before it runs and can veto it — that is where a script counting new test files against the budget belongs, because a veto cannot be talked out of anything. The PostToolUse hook cannot veto a file just written, but it can count it and put the running total in front of everyone while there is still time to act.

Then the last gate, the one nobody can charm: CI itself. A CI gate (a check the build must pass before merging) is team policy in pipeline code. Two gates fit this problem: fail the build when a change adds an absurd number of tests with no explanation, and run the mutation check on critical modules so a suite that only looks strong cannot merge quietly. The agent does not experience these as suggestions. That is rather the point.

The test-bloat checklist

Pin this one. Work it the next time an agent offers you a stadium of tests.

Set a test budget: A number, per change — at most 10 new tests is a sane start. Over the budget needs a reason, stated first.
One behaviour per test: If the name needs the word “and”, it is two tests wearing one coat.
Fail first: Run every new test against the code without the fix. A test that passes on broken code proves nothing — delete or repair it.
Assert the outcome, not the mock: Check the result a user would see, not that a fake dependency received a phone call.
Delete the duplicate: If a new test guards a behaviour an old test already guards, extend the old test instead.
Keep CLAUDE.md short and specific: It is context, not law — longer files get followed less reliably. Every line must earn its place.
Move must-never rules to hooks and deny: Anything that must hold regardless of what the agent decides belongs in settings — a PreToolUse hook that can block, a deny rule that beats every allow.
Gate it in CI: Make unexplained suite growth and a falling mutation result on critical code fail the build. A gate does not negotiate.
Mutate the critical code: Run mutation testing where a silent bug hurts most. Every surviving mutant is a named weakness — fix those before adding any new test.
Review the count, not just the diff: Ask out loud, every time: “How many tests did this add, and what does each one promise?”

Coverage counts the lines your tests touched. It never asks whether they were holding anything.

Fewer tests. Stronger promises.

The goal was never 1,800 tests, and it was never 18 either. The goal is a suite you believe on the worst morning of the year — when it goes red at the moment you least want it to, and you stop and look, because its red has always meant something.

That suite comes from budgeting tests, failing them first, breaking the code underneath them on purpose, and putting the rules that matter where a polite paragraph cannot be talked out of them. Do that, and the next gift the agent brings you will be small, sharp, and worth keeping.

Staging warning: try new hooks, deny rules and CI gates on a staging or throwaway branch first — a PreToolUse hook that blocks too much will block your real work just as faithfully, and you want to learn its temper where it costs nothing.

Disclaimer

This post describes my personal experience with my own systems and accounts, in my own words. It is a first-person account of what I observed and what I do in my own work — not a statement of any company's intentions, policies, or internal decisions. Where I do not know something, the post says so.
I am not affiliated with, endorsed by, or speaking for any company, product, or vendor named in this post. All product and company names are trademarks of their respective owners and are mentioned only to describe the experience accurately.
Any error messages, documentation excerpts, or figures quoted here are reproduced from my own records or from the public sources linked in the post, believed accurate at the time of writing. Product behavior changes over time; verify against current official documentation before relying on any detail.
Nothing in this post is professional, financial, legal, or security advice for your specific situation. Examples are illustrative — validate in your own environment before running anything in production.
Corrections are welcome. If any company named here publishes an official statement, or if any detail is wrong, I will update this post and note the correction. The goal is an accurate public record, not a verdict.

Honey Kiran · DevOps & Platform Engineering

Claude Code, AI agents, CI/CD, testing, and DevOps — explained the practical way.

How many tests did your agent write this week — and if you deleted half of them tonight, which half would you actually miss? Tell me in the comments.

Examples are illustrative — validate in your own environment before running in production.

OpsBites: Your ‘Private' EKS Cluster Still Bills Like a Public One

Honey Kiran — Wed, 21 Oct 2026 02:03:24 +0000

The requirement arrives in one sentence, usually from security: “The cluster must not be reachable from the internet.”

Fine. Sensible, even. You turn off the public API endpoint, put the nodes in private subnets, and let the engineers in over the VPN. That part of private EKS works exactly as advertised, and AWS documents it well.

Then somebody — usually finance, usually a month later — asks what “private” costs. And the room goes quiet, because the honest answer is: privacy on AWS is not a setting. It’s a subscription.

Privacy on AWS is not a setting. It’s a subscription.

A thread on r/devops this week asked the classic version of the question: we run EKS inside a corporate environment, our network is on-premise, how do we connect to a private cluster? The connectivity answer is the easy half. Here’s the half that shows up on the invoice.

What ‘private’ changes — and what it doesn’t

With the public endpoint disabled, kubectl from the office reaches the API server privately, over VPN or Direct Connect. Nodes live in subnets with no route to the internet. Good. That’s the security posture everyone wanted.

But your cluster still talks to AWS all day long. Nodes register with EC2. Pods pull images from ECR. The autoscaler asks STS for tokens. Logs flow to CloudWatch. Load balancers get created. On a public cluster, those calls stroll out through the internet gateway and nobody thinks about them. In a private subnet there is no stroll. Every single one of those calls needs a private path you provision — a VPC endpoint — or it doesn’t happen at all.

AWS’s own doc for private EKS clusters lists the endpoints a limited-internet cluster needs. Count them with me: EC2, ECR api, ECR dkr, S3, Elastic Load Balancing, CloudWatch Logs, STS, EKS, EKS Auth, the OIDC endpoint, and Route 53. Eleven friends. Most of them are interface endpoints, and interface endpoints have a price list.

The math nobody does in the design meeting

PrivateLink pricing, US regions: an interface endpoint costs $0.01 per hour, for every AZ you put it in , plus $0.01 per GB of data processed. Sounds like nothing. Now do the multiplication everyone skips.

Line itemThe mathStanding cost / month8 interface endpoints × 3 AZs8 × 3 × $0.01/hr≈ $175EKS control plane$0.10/hr≈ $733 NAT gateways (yes, still)3 × $0.045/hr, before data≈ $991 Application Load Balancer$0.0225/hr, before LCU≈ $16S3 gateway endpointgateway type — free$0 Total, before any data moves≈ $363

Three details in that table deserve a stare:

The S3 endpoint is free — because it’s a gateway endpoint, not an interface one. No hourly charge, no per-GB charge. It’s the only free seat in the house, and it carries the heaviest traffic in the building, because ECR stores image layers in S3. Take it on day one, every time.

You still pay for NAT. Private doesn’t mean nothing ever leaves: package mirrors, external APIs, vendor webhooks. Whatever egress remains rides NAT gateways at $0.045/hr each plus $0.045 per GB — and a partial hour bills as a full one. One per AZ, because a single shared NAT is a cross-AZ fee machine and a quiet single point of failure.

ALB is cheap; NLB is cheaper; neither is free. $0.0225 an hour for either, with LCU charges on top — $0.008 per LCU-hour on the ALB, $0.006 on the NLB. Small numbers, permanent numbers.

Then the data starts moving

The hourly meter is the rent. The per-GB meter is the toll — and it punishes the lazy route. One terabyte of image pulls through a NAT gateway costs about $45 in processing fees alone. The same terabyte through endpoints costs about $10 — and because image layers actually stream out of S3, most of that weight rides the free gateway endpoint anyway. Endpoints in every AZ also keep the traffic local; let an endpoint live in one AZ while your nodes sit in another and you’ll meet cross-AZ transfer charges, the toll booth nobody remembers building.

The honest middle

None of this argues against going private. Compliance is real, auditors are realer, and “the Kubernetes API was on the internet” is a sentence nobody wants in an incident report. The bite is narrower:

Price the privacy in the design doc , not in the first invoice review. $363 a month standing is nothing to a platform budget — unless nobody budgeted it, in which case it’s a surprise, and surprises are what actually get clusters decommissioned.
Codify the endpoints in Terraform from day one. Endpoints added later, by hand, during an outage, are how you end up with three of eleven and a mystery.
Dev clusters don’t need the full cathedral. Fewer AZs, a shared-endpoint strategy, a smaller footprint — the meter multiplies by AZs, so spend the multiplication where it matters.

Three traps that make it worse

Endpoints in one AZ only. Cheaper on the hourly line — until every node in the other two AZs routes its calls across, and the cross-AZ toll quietly eats the saving. It also builds a dependency nobody drew: one AZ hiccup away from pods that can’t pull images.
Forgetting the S3 gateway endpoint. Miss this one free line and ECR layer pulls — the fattest traffic in the cluster — detour through NAT at $0.045 a gigabyte. It’s the most expensive free thing you’ll ever forget to click.
Cathedral spec on dev clusters. Three AZs, eleven endpoints, full ceremony, for a cluster three people use to test Helm charts. The meter multiplies by AZs whether the cluster is production or a hobby with a pipeline.

Private was never going to be free. It just sends the bill in a shape the design meeting didn’t draw: every endpoint, times every AZ, forever. Draw it anyway — the architecture diagram and the invoice are the same picture, and you’re better off seeing it first.

Disclaimer

This post describes my personal experience with my own systems and accounts, in my own words. It is a first-person account of what I observed and what I do in my own work — not a statement of any company’s intentions, policies, or internal decisions. Where I do not know something, the post says so.
I am not affiliated with, endorsed by, or speaking for any company, product, or vendor named in this post. All product and company names are trademarks of their respective owners and are mentioned only to describe the experience accurately.
Any error messages, documentation excerpts, or figures quoted here are reproduced from my own records or from the public sources linked in the post, believed accurate at the time of writing. Product behavior changes over time; verify against current official documentation before relying on any detail.
Nothing in this post is professional, financial, legal, or security advice for your specific situation. Examples are illustrative — validate in your own environment before running anything in production.
Corrections are welcome. If any company named here publishes an official statement, or if any detail is wrong, I will update this post and note the correction. The goal is an accurate public record, not a verdict.

Examples in this post are illustrative unless marked as documented patterns. Pricing figures are the providers’ published US list prices and vary by region — verify against the current pricing pages before budgeting.

Tags: Kubernetes · Amazon EKS · AWS · VPC · DevOps

Your Liveness Probe Is a Kill Switch on a Timer

Honey Kiran — Wed, 21 Oct 2026 02:02:10 +0000

Pull up a chair. Today on r/devops, someone posted a thread titled: “Kubernetes probes keep restarting services that just need more time.” I read that title and sighed into my chai, because every engineer who has run Kubernetes for more than a week has watched this exact tragedy.

You deploy your app. The pod goes Running. You smile. Ten seconds later the restart counter ticks to 1. Then 2. Then the status flips to CrashLoopBackOff — Kubernetes throwing its hands up: I tried restarting this thing, it keeps failing, so now everyone waits.

This Substack is reader-supported. To receive new posts and support my work, consider becoming a free or paid subscriber.

You check the logs, expecting a stack trace. There is none. The last line says something innocent like Loading configuration... or Warming up local cache...

Your app was not broken. It was not deadlocked. It was busy getting ready for work. But the cluster saw a container that had not answered yet, assumed it was dead, and shot it.

Slow is not dead. Your cluster just cannot tell the difference until you teach it.

Three probes, three questions, three punishments

In Kubernetes, health checks are called probes. They are not run by some wizard in the control plane. They are run by the kubelet — the agent that lives on every node and babysits the containers assigned to it.

The kubelet runs three probes. They are not interchangeable. Each asks a different question, and each carries a different punishment.

ProbeQuestion it asksWhat happens when it fails Startup Has it finished starting?Nothing — until its whole budget runs out. Then the container is killed and restarted. Liveness Is it deadlocked?The kubelet kills the container and restarts it, subject to the pod’s restart policy. Readiness Can it take traffic right now?Nothing is killed. The pod’s IP leaves the Service’s endpoints, so traffic stops until it recovers.

Readiness is the gentle one: no traffic for you until you feel better. It runs for the container’s whole life, and its punishment costs you a few seconds of capacity, never the container.

Liveness is capital punishment: death and restart. It exists for exactly one situation — an app so deadlocked that no amount of waiting will save it, where the only cure is off and on again.

Attach that punishment to a container that is still booting, and you have not built a health check. You have built a kill switch on a timer.

The 30-second fuse

Why does this happen out of the box? The defaults. Every probe carries the same knobs:

initialDelaySeconds — wait before the first check. Default: 0.
periodSeconds — how often the check repeats. Default: 10.
timeoutSeconds — how long a check may take before it counts as failed. Default: 1.
successThreshold — successes needed to call it healthy. Default: 1.
failureThreshold — failed checks in a row before the punishment lands. Default: 3.

Now add a standard livenessProbe to a slow-booting app and touch nothing else, because defaults feel harmless. Here is the math the kubelet runs. The container starts; the first check fires almost immediately and fails — the port is not open yet, the app is still reading its config. Strike one. Ten seconds later, strike two — it is loading dependencies now. Ten seconds after that, strike three. Roughly thirty seconds into the boot, the kubelet kills the container and starts a fresh one — which boots the same slow way, and gets shot at the same thirty-second mark.

Repeat that a few times and the kubelet starts backing off exponentially between restarts. That is your CrashLoopBackOff.

The classic band-aid is initialDelaySeconds: 60. I dislike it. It is a blind guess wearing a number. If the app boots in 8 seconds, the pod sits unchecked for most of a minute. If a busy node or a slow disk stretches one boot to 65 seconds, the guess fails and the kill switch fires anyway.

A liveness probe should be your last line of defense against a dead process — not a test of whether your app can boot in thirty seconds flat.

The fix: give the boot its own probe

Kubernetes has a probe built for exactly this: the startup probe. Its rule is simple and beautiful — while a startup probe is configured and has not yet succeeded, Kubernetes does not run the liveness or readiness probes at all. They sit on the bench and wait.

The startup probe runs only during boot, and its patience is a budget you compute:

Startup budget = failureThreshold × periodSeconds

The official Kubernetes documentation’s own example sets failureThreshold: 30 and periodSeconds: 10 — 30 × 10 = 300 seconds, a five-minute runway. If your app answers at second 40, the startup probe succeeds once, retires forever, and liveness and readiness take over the watch. If the app truly hangs and burns all thirty checks, only then is it killed.

apiVersion: v1
kind: Pod
metadata:
  name: slow-starter-demo
spec:
  containers:
  - name: app
    image: example-app:1.0
    ports:
    - containerPort: 8080
    startupProbe:
      httpGet:
        path: /healthz
        port: 8080
      periodSeconds: 10
      failureThreshold: 30 # 30 x 10s = 300s to finish booting
    livenessProbe:
      httpGet:
        path: /healthz
        port: 8080
      periodSeconds: 10
      failureThreshold: 3 # strict again — this one is for deadlocks
    readinessProbe:
      httpGet:
        path: /ready
        port: 8080
      periodSeconds: 5

Notice the trick: the startup probe and the liveness probe ask the same endpoint. The startup probe is just the patient version, standing guard over the boot so liveness never has to be lenient. Your liveness stays tight — a genuinely deadlocked app still gets caught and restarted in about thirty seconds. Fast recovery, no friendly fire.

Liveness discipline: never check the database

One more trap, and it is the most dangerous one in this whole topic. Kubernetes gives you four probe mechanisms — httpGet (success means a status from 200 to 399), tcpSocket, exec, and grpc (stable since Kubernetes 1.27). The danger is not the mechanism. It is what your endpoint checks.

The classic mistake: the /healthz handler checks the process, then pings the database, then the cache, then a downstream API — and returns 500 if any of them sulk. Wire that to a liveness probe and walk through a database hiccup with me. The database stops accepting connections for a minute. Every pod’s liveness check now fails. Every kubelet in the cluster does its duty and kills its pods — all of them, at roughly the same time. The fresh pods boot straight back into the same dead database, fail the same check, and get killed again, while their boot storms hammer the very database that was trying to recover.

The Kubernetes documentation warns about this in plain words: incorrect liveness probes lead to cascading failures — restarts under high load, failed client requests, and extra workload dumped on the pods still standing.

So the rule, stated flat: liveness checks only the app itself. Is the process breathing? Is it deadlocked? Nothing outside the container’s own skin. Dependency checks belong in the readiness probe — when the database goes down, readiness fails, traffic drains away, and the pod sits quietly with its connections and memory intact, waiting. Nobody gets shot over a dependency.

The new slow starters

Slow boots are not a legacy problem. They are getting worse.

JVM services warming up and apps hydrating big in-memory caches were already blowing past a thirty-second fuse years ago. Now add the newest tenants of our clusters: model servers. A GPU inference server can spend minutes loading model weights into memory before it can answer a single request — and that is a healthy, normal boot. Drop one behind default liveness settings with no startup probe and it will never serve a request in its life. It will load weights, get shot, load weights, get shot.

The defaults suit a small, fast service. Plenty of what we run in 2026 is neither. Bridging that gap is not the app’s job. It is ours.

The probe audit

Run your manifests against this list:

Every slow starter has a startupProbe . If a boot takes more than a few seconds, liveness never guards the boot window again.
The startup budget covers the worst boot you have measured — on a cold, busy node, not on your laptop. Budget = failureThreshold × periodSeconds.
Startup and liveness share one endpoint. Same /healthz, two patience levels. No extra endpoint to maintain.
Liveness checks nothing external. No database, no cache, no downstream API. If it reaches outside the container, move that check to readiness.
Readiness checks the real dependencies. If the app cannot serve honestly without its database, readiness should know before your users do.
timeoutSeconds is honest. A busy single-threaded event loop can easily blow a 1-second timeout under load — and a timeout counts as a failure.
Heavy exec probes stay off dense nodes. Every exec check forks processes inside the container; on a crowded node that overhead is real. Prefer httpGet, tcpSocket, or grpc.
Alert on restart counts, not just availability. A pod killed and restarted all day can still show green between executions. Restarts are the smoke; go look for the fire.

The takeaway

A kubelet has no emotional intelligence. It holds a stopwatch and a restart button, and it does exactly what your numbers tell it to do. So set the numbers like someone who has read them: a startup probe with a real budget for the boot, a liveness probe that only ever asks “are you deadlocked?”, and readiness carrying every dependency check. Slow is not dead — stop letting a thirty-second fuse treat it that way.

Honey Kiran · DevOps & Platform Engineering

I write about SRE, Kubernetes, Terraform, and platform engineering — the defaults that bite, the incidents that teach, and the guardrails that hold.

What is the worst boot time hiding in your cluster right now — and which probe is guarding it? Tell me in the replies; the slowest honest answer wins.

Disclaimer

This post describes my personal experience with my own systems and accounts, in my own words. It is a first-person account of what I observed and what I do in my own work — not a statement of any company’s intentions, policies, or internal decisions. Where I do not know something, the post says so.
I am not affiliated with, endorsed by, or speaking for any company, product, or vendor named in this post. All product and company names are trademarks of their respective owners and are mentioned only to describe the experience accurately.
Any error messages, documentation excerpts, or figures quoted here are reproduced from my own records or from the public sources linked in the post, believed accurate at the time of writing. Product behavior changes over time; verify against current official documentation before relying on any detail.
Nothing in this post is professional, financial, legal, or security advice for your specific situation. Examples are illustrative — validate in your own environment before running anything in production.
Corrections are welcome. If any company named here publishes an official statement, or if any detail is wrong, I will update this post and note the correction. The goal is an accurate public record, not a verdict.

Examples are illustrative — validate in your own environment before running in production.

This Substack is reader-supported. To receive new posts and support my work, consider becoming a free or paid subscriber.

My AI Agent Drove My iPhone for a Day. Then I Parked It.

Honey Kiran — Mon, 19 Oct 2026 03:30:34 +0000

Saturday, coffee in hand, the dangerous kind of simple idea in my head.

On my phone sits an app I reach for constantly — no API, no web version worth the name, everything locked behind its own screens. Every engineer I know is busy wiring AI agents into terminals, browsers and incident channels. But agents that can only talk are half an agent, and the phone is where the world actually transacts. If an agent is ever going to be useful beyond a chat window, it has to work where the work is: inside apps like that one.

So: give my AI agent (Muse — the same one that runs my blog pipeline) hands on my real iPhone. Open the app, read its screens, tap through it with me in the loop. Eyes and fingers, under orders — nothing gets tapped behind my back.

Down this road, before you start:

The exact build and link fixes for Xcode 27.1 / iOS 27 — including two gates nobody warns you about.
Two small Objective-C patches for private Apple methods iOS 27 silently deleted.
A 14-step copy-paste recipe ending in an automation server on your phone.
The proof: no cable, control over Tailscale, a live screen read, one remote tap.
And the honest bill — the friction that made me park it the same day.

What I wanted my agent to do

The standard tool for handing an agent an iPhone is WebDriverAgent (WDA). Plain words: a test-runner app that moonlights as a remote control. Born at Facebook, now maintained by the Appium project. You build it in Xcode (Apple’s development environment), sign it with your own Apple ID, and install it on the phone like any app you wrote yourself.

Once it runs, WDA starts an HTTP server on port 8100. That is the whole trick — the phone becomes a tiny web API. Send a GET for the source and it walks the current app’s view hierarchy — every button, label and field — and hands it back as XML. Send a POST with coordinates and it taps the glass. No computer vision, no screenshot-guessing. Just HTTP.

Or at least, that is how it is supposed to go.

The morning: a “successful” test that never ran

The rig: a current iPhone on iOS 27.0.1, an Apple-silicon Mac. Even step zero had a cover charge — the Mac had to hop from macOS 26.5.1 to 26.7.1 first, because current Xcode refuses to install below macOS 26.6.

Then the first self-inflicted wound. The WDA source cloned that morning was branch 3.17 of the old Facebook repo — a vintage tree, discovered to be vintage only in the evening. Maintenance had long moved to Appium. Half the day’s fires trace back to that: old code, brand-new operating system.

Signing was easy: personal Apple ID, free “Personal Team”, automatic signing, unique bundle IDs — the stock com.facebook.* IDs can’t be provisioned by a personal team, so they became com.yourname.WebDriverAgentRunner and com.yourname.WebDriverAgentLib.

Hit build. Three brick walls in under five minutes.

Wall 1 — the test runs died in 0.6 seconds: IPHONEOS_DEPLOYMENT_TARGET is set to 10.0, supported range is 15.0 to 27.1.x. Xcode 27 looked at a deployment target from the iOS 10 era and laughed. Both targets went to 15.0.

Wall 2 — the project’s base config (IOSSettings.xcconfig) shipped GCC_TREAT_WARNINGS_AS_ERRORS = YES paired with -Weverything. Against the new iOS 27 SDK headers, warnings breed like rabbits, and every one was a build error. That went to NO.

Wall 3 — the linker, on Apple’s private automation framework: Cannot link directly with 'XCTAutomationSupport' because product being built is not an allowed client of it. XCTAutomationSupport is the engine room under Apple’s UI-testing stack, protected by an allowlist — only binaries presenting an approved client name may link it. The surgery: build WebDriverAgentLib as a static library (MACH_O_TYPE = staticlib), link XCTAutomationSupport into the Runner with the PrivateFrameworks search path, and pass -Wl,-client_name,XCTest so the bundle presents itself as client “XCTest” — an approved name.

By midday the build went green: “Testing…”, app installed, Stop button live. I walked away pleased.

All afternoon Xcode showed the test running, and every fresh start was rejected — “There is already a test running.” When the session record finally closed it showed 20,301.4 seconds: 5.6 hours of “testing.” Nothing was listening on port 8100. No runner process existed on the phone. The evening’s evidence proved the test bundle could not load at all — the morning “success” was a session record that never died, not a server that ever ran.

The build was green, the app was installed, and the test had been “running” for 20,301 seconds. It had never loaded.

The evening: two gates and a missing framework

That evening I stopped trusting the GUI and asked the phone directly, via devicectl (Apple’s command-line device tool). Root Cause A, in its own words: Installing cryptex com.apple.MobileAsset.DDI is disallowed because developer mode is not enabled. (CoreDeviceError 12040/12052)

Developer Mode was OFF. Without it, the Developer Disk Image — the “DDI”, iOS’s bundle of developer services — cannot mount, the phone lists as connected (no DDI), and every launch dies at step 0. Fix: Settings → Privacy & Security → Developer Mode → ON, survive the reboot, confirm again after it.

Next, Root Cause B. The build now installed the runner; iOS refused to launch it. Xcode’s own alert: The application could not be launched because the Developer App Certificate is not trusted. Fix: Settings → General → VPN & Device Management → trust the Apple Development certificate under my own Apple ID. Two gates, both on the phone, neither mentioned by any happy-path tutorial I have ever read.

Then Root Cause C, the real one. A detached launch died instantly (exit code 83), and xcodebuild showed why: Library not loaded: @rpath/XCTAutomationSupport.framework/XCTAutomationSupport.

We had LINKED the framework but never EMBEDDED it — and it lives in none of the searched places: not the app bundle, not /System/Developer/Library/Frameworks, not the cryptex copies. It lives in PrivateFrameworks, and nothing had told the binary’s runpath (the @rpath list the loader searches at launch) to look there. Self-inflicted, too: the static merge made the test binary reference the framework directly, discarding the search paths the stock dynamic library carries.

The fix: two Runner build settings — add /System/Developer/Library/PrivateFrameworks and /private/preboot/Cryptexes/OS/System/Developer/Library/PrivateFrameworks to LD_RUNPATH_SEARCH_PATHS. On the rebuild: bundle loaded, runner started, automation screen up, and the console printed the line you wait all day for: ServerURLHere->http://<phone-wifi-ip>:8100<-ServerURLHere.

iOS 27 removed two methods. We patched both.

The server lived about three and a half minutes before it crashed with an NSInvalidArgumentException: Enqueue Failure: -[__NSXPCInterfaceProxy_XCTMessagingChannel_RunnerToDaemon _XCT_exchangeProtocolVersion:reply:]: unrecognized selector.

Plain words: WDA asks the phone’s test daemon “which protocol version do you speak?” In iOS 27, the daemon no longer answers that call. The old code — FBTestmanagerdVersion in FBXCodeCompatibility.m — asked with no guard and no error handling, so the exception executed the runner. The rude part: the proxy still claims to support the call. The interface declares it; the daemon just dropped it.

That number is informational only — it lands in session info, nothing steers by it — so a fallback of 0 is harmless. My agent read the source, found the send site, and patched it: respondsToSelector check with an early return, the exchange wrapped in @try/@catch, 0 kept on failure.

Crash 2 surfaced on the first real screen read (GET /source): -[XCUIApplicationProcess waitForQuiescenceIncludingAnimationsIdle:]: unrecognized selector. WDA waits for the app to go idle — “quiescence” — before snapshotting, through another private method iOS 27 removed. There is a settings knob, waitForIdleTimeout, that sounds exactly like the fix. We set it to 0. Nothing changed: the screen-read path funnels through a different timeout into the same raw send, in XCUIElement+FBUtilities.m (fb_waitUntilStableWithTimeout:). The knob exists. The knob is decorative. Same patch: guard it, catch it, and if the method is gone, skip the wait — best-effort.

// The pattern that fixes both iOS 27 crashes: never call a private
// method you have not checked for, and never let the check kill you.
if ([object respondsToSelector:@selector(thePrivateMethod:)]) {
  @try {
    [object thePrivateMethod:];
  } @catch (NSException *exception) {
    // Selector vanished mid-flight — fall back. Do not crash the runner.
  }
}

The recipe

Everything above, compressed into the checklist I wish had existed on Saturday morning. Every name and value is load-bearing.

# Project / target settings that made it build on Xcode 27.1 / iOS 27
IPHONEOS_DEPLOYMENT_TARGET = 15.0 # both targets (was 10.0)
GCC_TREAT_WARNINGS_AS_ERRORS = NO # IOSSettings.xcconfig
MACH_O_TYPE = staticlib # WebDriverAgentLib only

# Runner target
OTHER_LDFLAGS = "-Wl,-client_name,XCTest"
FRAMEWORK_SEARCH_PATHS = "$(PLATFORM_DIR)/Developer/Library/PrivateFrameworks"
LD_RUNPATH_SEARCH_PATHS = "/System/Developer/Library/PrivateFrameworks" "/private/preboot/Cryptexes/OS/System/Developer/Library/PrivateFrameworks"

Install current Xcode plus the iOS platform component; pair the phone and trust the Mac.
Clone the maintained Appium WebDriverAgent repo — not an old branch of the original.
Personal Team signing on both targets; unique bundle IDs (com.yourname.*).
Deployment target 15.0 on both targets.
IOSSettings.xcconfig: GCC_TREAT_WARNINGS_AS_ERRORS = NO.
WebDriverAgentLib: MACH_O_TYPE = staticlib.
Runner: link XCTAutomationSupport, add PrivateFrameworks to FRAMEWORK_SEARCH_PATHS, and OTHER_LDFLAGS += "-Wl,-client_name,XCTest".
Runner: LD_RUNPATH_SEARCH_PATHS += both PrivateFrameworks paths from the settings block.
On the phone: Developer Mode ON (Settings → Privacy & Security; expect a reboot and a second confirmation).
On the phone: trust the developer certificate (Settings → General → VPN & Device Management).
Patch FBXCodeCompatibility.m (FBTestmanagerdVersion): respondsToSelector guard + @try/@catch, fallback 0.
Patch XCUIElement+FBUtilities.m (fb_waitUntilStableWithTimeout:): guard + @try/@catch around the quiescence send; skip the wait if the method is gone.
Start with the xcodebuild test command below (~90 seconds to server-up). Optional, once, while cabled: tick “Connect via network” in Xcode’s Devices window so later starts run over Wi-Fi.
Drive it: GET /status → POST /session with empty capabilities (target app in the foreground) → GET /source → POST /wda/tap/0 with coordinates read from the XML.

The proof: no cable, one tap, real data

Control never needed the Mac relay we had built — a USB port-forwarder that never answered cleanly and turned out to be furniture. Plain curl from the agent’s machine, straight to the phone over Tailscale, the private mesh VPN where every device keeps a stable private address:

# Start (Mac, ~90 seconds to server-up). Full path — bare xcrun may
# point at the Command Line Tools and find nothing.
/Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild test \
  -project ~/Documents/WebDriverAgent/WebDriverAgent.xcodeproj \
  -scheme WebDriverAgentRunner \
  -destination id=<your-device-udid>

# Drive it from anywhere (here: over Tailscale, no cable)
curl http://<phone-tailscale-ip>:8100/status
curl -X POST http://<phone-tailscale-ip>:8100/session \
  -d '{"capabilities":{"alwaysMatch":{},"firstMatch":[{}]}}'
curl http://<phone-tailscale-ip>:8100/session/<session-id>/source
curl -X POST http://<phone-tailscale-ip>:8100/session/<session-id>/wda/tap/0 \
  -d '{"x":67,"y":904}'

GET /status answered: "WebDriverAgent is ready to accept commands", ready: true, iOS 27.0.1 — and the patch’s fingerprint, "testmanagerdVersion": 0. It answered again and again. A session was created, and /status stayed healthy after session creation — the runner had survived the exact call that killed it before the patch.

One driving landmine, learned live: creating a session bound to SpringBoard (the home-screen process) timed out and wedged the single-threaded server solid. Attach to a normal app in the foreground. Never target system apps.

Then the full loop. “Connect via network” was on. The cable run stopped, the cable came OUT, the start command ran again — session hosted over Wi-Fi. The agent attached to a target app (com.example.targetapp) and read its current screen back live — every label and figure exactly as displayed.

Element search by accessibility ID is broken on this tree (Invalid key path wdName specified in predicate format string), so the coordinates came straight from the source XML: a tab’s rectangle, center (67, 904). One POST to /wda/tap/0 — the tab switched on the phone in my hand — and the next screen read back cleanly: a list, every row and count intact.

Read + tap, end to end, zero cables anywhere in the workflow.

Why I parked it the same day

The control never missed. Every status, session, read and tap landed, all evening, over a VPN, on a phone with no cable in it. As a technology demo, flawless.

And that same evening I told the agent to switch it off and park the whole approach. The bill, unsoftened:

While a session runs, the phone is not your phone. The automation screen owns the display. You are carrying a test rig, not a handset. There is no background mode — iOS does not offer one for this.
Every start needs the Mac awake and a human pasting a Terminal command. Ninety seconds, yes — but a Mac, a human, and a paste, every single time. That same evening, starting it by driving Xcode’s GUI remotely failed twice. The paste is the only start that ever worked.
Free signing expires every 7 days. Miss the weekly re-run and the runner is dead until somebody re-signs it. A paid developer account stretches that to a year — for $99 a year, rent on automating your own pocket hardware.
The stack is an old branch held together by two source patches and build-setting surgery. iOS 27 deleted two private methods it stood on and told nobody. The next update can delete two more, and each one costs another evening exactly like this one.

The technology verdict and the ownership verdict are different verdicts. If this ever comes back, it comes back on a dedicated spare phone on a dock, next to an always-on Mac — never on the daily driver. Until then: parked, with the recipe left running in the one place it costs nothing — this page.

Save the recipe if you are heading down this road anyway. When your agent finally reads your phone screen, enjoy the ten minutes. Then count what it costs to keep.

Honey Kiran · DevOps & Platform Engineering

I write about DevOps, platform engineering, Kubernetes, and putting AI agents to work — with guardrails, receipts, and the occasional parked project.

Running an agent against your own phone, or thinking about it? Tell me where yours broke — the replies are where the real field notes collect.

_Disclaimer

This post describes my personal experience with my own systems and accounts, in my own words. It is a first-person account of what I observed and what I do in my own work — not a statement of any company's intentions, policies, or internal decisions. Where I do not know something, the post says so.
I am not affiliated with, endorsed by, or speaking for any company, product, or vendor named in this post. All product and company names are trademarks of their respective owners and are mentioned only to describe the experience accurately.
Any error messages, documentation excerpts, or figures quoted here are reproduced from my own records or from the public sources linked in the post, believed accurate at the time of writing. Product behavior changes over time; verify against current official documentation before relying on any detail.
Nothing in this post is professional, financial, legal, or security advice for your specific situation. Examples are illustrative — validate in your own environment before running anything in production.
Corrections are welcome. If any company named here publishes an official statement, or if any detail is wrong, I will update this post and note the correction. The goal is an accurate public record, not a verdict.

Examples are illustrative — validate in your own environment before running in production._

Cloud Run Is Reliable. Your Defaults Are the Risk.

Honey Kiran — Mon, 19 Oct 2026 03:29:06 +0000

Over on r/googlecloud, in a thread titled “How reliable is Cloud Run, really?”, an engineer was doing the homework before a serious move: a mission-critical, latency-sensitive service, currently on Heroku, possibly moving to Cloud Run — and one question underneath it all. Can I trust this thing?

It is the right question. It is also, forgive me, slightly the wrong question. Cloud Run (Google’s serverless container platform — you give it a container, it runs it, scales it, and bills you for the work) is reliable in the ways Google controls. The part that will actually decide your first month is the part Google does not control: the settings you leave at their defaults because the deploy worked on the first try and everyone went home happy.

Picture it — illustrative scene, but you have seen this film. The migration goes beautifully. Then the first quiet morning arrives, traffic dips, the platform politely scales everything to zero to save you money, and the first real user of the day pays for it in one slow request. Nobody’s dashboard turns red. The average still looks lovely. Only the slowest users know. They always know.

So let us read the fine print together. I did the reading so you do not have to lose your afternoon to twelve documentation tabs.

First, what the SLA actually promises

SLA means Service Level Agreement — the written promise, with money behind it. For Cloud Run services, Google’s SLA promises a monthly uptime percentage of 99.95% for a normal, non-GPU service in most regions. Stockholm is the odd one out at 99.9%, and a GPU service without zonal redundancy (extra copies spread across zones, so one zone failing does not take you down) sits at 99.5%. Fine numbers. Now read what counts as “down”.

Downtime, in this document, means an error rate above one percent — and only counting requests that fail with a 5xx error (a server-side failure) because of Cloud Run’s own infrastructure, with a minimum of 100 valid requests in the measurement window. If Google misses, the remedy is a financial credit against future bills — 10%, 25%, or 50% depending on how far it fell — and you have to ask for it within 30 days, with logs.

Notice two things, because they matter more than the headline number. One: errors caused by your own software are excluded. If your container crashes, that is your downtime, not Google’s, and the SLA stays silent. Two: the SLA counts errors, not slowness. A request that takes nine painful seconds and eventually returns a success counts as uptime. Your user experienced a failure. The contract experienced a triumph. Both are telling the truth, which is exactly why “is Cloud Run reliable?” needs a better follow-up: reliable at what, measured how, and whose settings?

The SLA promises the platform will answer. It never promised your defaults would let it answer fast.

Cold starts: the platform’s politest trap

A cold start is the delay when a request arrives and no running instance (a live copy of your container) exists to take it, so the platform has to start one first — pull things together, boot your process, run your startup code — while the request waits. By default, a Cloud Run revision (one immutable version of your service; every deploy creates a new one) with no traffic scales to zero instances. Zero is beautiful on the bill and brutal on the first request after a lull.

The fix everybody reaches for is minimum instances: you tell Cloud Run to keep a set number of instances warm and ready even when nothing is arriving. Set it to 1 and the quiet-morning problem mostly disappears for a small service. But be clear about the trade, because there are two catches hiding in it.

Catch one: warm is not free. Instances kept running by minimum instances do incur billing costs — a lower idle rate if you are on request-based billing, the full rate if you are on instance-based billing. You are renting a taxi that waits outside your house all night. Very convenient. Not charity.

Catch two: it is a best-effort target, not a guarantee — Google’s own words. Zone capacity trouble, infrastructure rebalancing, a container that crashes on startup, or a quota wall can all leave you temporarily below your floor. So minimum instances reduce cold starts; they do not delete the concept. Design as if a cold start can still happen occasionally — because it can — and make your startup path lean: small image, lazy loading of heavy things you do not need on every request, and startup CPU boost turned on (the --cpu-boost flag, which gives an instance extra CPU while it starts). Then measure your own cold start in staging. Do not copy anyone else’s number off the internet, including mine. There is no honest universal number; runtimes and images differ too much.

Concurrency: 80 strangers in one small room

Concurrency means how many requests one instance handles at the same time. This is where Cloud Run quietly differs from the old one-request-at-a-time mental model. The console default is 80 concurrent requests per instance, and you can raise it to a maximum of 1,000. If you deploy with the command line or Terraform, a brand-new service defaults to 80 times the number of vCPUs (the virtual CPU cores you gave the container) — so for a 1-vCPU service, that is also 80. Either way, picture it: 80 requests sharing one container, one CPU allowance, one memory allowance, and whatever database connections your code opens.

For a light, I/O-bound service (one that mostly waits on other systems — a quick database read, an API call, then done), 80 can be perfectly sensible and wonderfully cheap: fewer instances, less bill. For a CPU-heavy service (each request actually crunches — image work, heavy JSON, crypto, reports), 80 is a queue disguised as a setting. Requests do not fail. They wait politely behind each other inside your container, and your latency tail — the slowest few percent of requests, the p95 and p99 your heaviest users live in — grows a long, ugly beard.

One more nuance: the number is a ceiling, not a promise. If an instance’s CPU is already highly used, Cloud Run may send it fewer requests and scale out instead. And if your code cannot handle parallel requests safely — shared global state, not thread-safe — the docs’ advice is blunt: set concurrency to 1. Correct beats fast.

Here is a sane starting deploy for a latency-sensitive service — note that every interesting choice is explicit, nothing is left to vibes:

gcloud run deploy my-service \
  --image us-docker.pkg.dev/PROJECT_ID/REPO/my-service:TAG \
  --region us-central1 \
  --min-instances 1 \
  --max-instances 20 \
  --concurrency 40 \
  --cpu 1 \
  --memory 512Mi \
  --timeout 60s \
  --cpu-boost

Read it like a contract: one warm instance always ready, at most twenty instances so a spike cannot stampede the database, forty concurrent requests per instance because we measured instead of guessing, and a 60-second timeout matched to the real work. Your numbers will differ. The habit must not.

CPU only exists while a request is looking at it

This is the default that surprises the most migrants, so sit with it for a second. Cloud Run has two billing settings. The default is request-based billing — the setting previously called “CPU only allocated during request processing”. Exactly what it sounds like: an instance gets CPU while it is processing a request, and is charged when it processes requests, starts, and shuts down. Between requests, your code effectively sits in a dark room.

The alternative is instance-based billing — previously “CPU always allocated”, switched on with --no-cpu-throttling — where CPU is allocated for the entire life of the instance and you are charged for that entire life. More money. Also, sometimes, the only correct answer.

Why? Because of everything your service does after replying. Flush the last log lines, finish the metric, close out the background task — under request-based billing, all of that crawls or stalls the moment the response leaves, because the CPU went home. Like a chef who may only move while a customer is watching: the dish got served; the kitchen never got cleaned. If your service does real background work, choose instance-based billing on purpose — or move that work to a proper queue. Do not discover the dark room in production.

Timeouts: the 504 that leaves the body behind

Every request gets a timeout: how long the service may take before Cloud Run gives up on it. The default is 5 minutes (300 seconds), and you can extend it up to a maximum of 60 minutes (3,600 seconds). For most migrated web services this is plenty of headroom — which is precisely why people stop thinking about it, right up until a client, a load balancer, or a framework setting somewhere else in the chain gives up much earlier and everyone debugs the wrong layer.

And here is the detail worth memorising: on timeout, Cloud Run closes the connection and returns an HTTP 504 — but it does not terminate the container. Your code may keep processing the dead request, burning CPU your live requests needed. So set the timeout just above your honest worst case, bail out early inside long handlers, and if a job regularly wants near an hour, it is asking to be a background job, not a web request.

Revisions: the part Cloud Run genuinely does beautifully

Now the good news, and it is genuinely good. Every change — new image, new setting — creates a new revision, and Cloud Run lets you choose which revisions get traffic and in what percentage. Deploy the new revision with no traffic, look at it directly, then send it 5%. Watch it. Then 50%, then all of it. If it misbehaves at any point, rollback is not a redeploy and a prayer; it is one traffic command sending 100% back to the previous revision:

# deploy the new revision, but give it no traffic yet
gcloud run deploy my-service \
  --image us-docker.pkg.dev/PROJECT_ID/REPO/my-service:TAG \
  --region us-central1 \
  --no-traffic --tag canary

# let the canary revision taste 5 percent of real traffic
gcloud run services update-traffic my-service \
  --region us-central1 \
  --to-tags canary=5

# rollback: send everything back to the previous revision
gcloud run services update-traffic my-service \
  --region us-central1 \
  --to-revisions PREVIOUS_REVISION=100

Two honest caveats. Traffic changes are not instantaneous, and requests already in flight are not dropped during a switch — they finish on whichever revision took them. That is good behaviour, but it means “we rolled back” and “the bad revision stopped mattering” are a few moments apart, not the same instant. And a rollback only rescues you if the previous revision is actually still good and your data changes were backwards-compatible. A new revision writing a new database shape that the old revision cannot read will make your beautiful one-command rollback into a one-command second outage. Rehearse the rollback in staging, with the database change included, before you need it at speed.

Staging warning: test every one of these settings in a staging service first — minimum instances, concurrency, timeout, traffic split, and the rollback command itself — with the same defaults you plan for production. A rollback you have never run is a rumour, not a plan.

The pre-migration reliability checklist

Pin this. Ten questions, answered in staging, before Heroku becomes a memory:

Is minimum instances set — or is cold the plan? If the service is latency-sensitive, a warm floor (often 1 to start) is a decision, not an accident. Know its idle cost and its best-effort limits.
Does concurrency match your code? Load-test at your chosen value. CPU-heavy means lower; light and I/O-bound can go higher; unsafe in parallel means 1.
CPU mode: chosen, or inherited? Request-based by default. If anything must run outside a request — background flushes, threads, agents — switch to instance-based billing deliberately.
Is the timeout longer than your real work — and not longer? Default 5 minutes, maximum 60. Add headroom, add early bail-out, and move hour-long work out of requests entirely.
Can your database survive your maximum instances? Instances times connections per instance is a number. Compute it before a traffic spike computes it for you, then cap max instances to fit.
Does every deploy land with no traffic first? New revision dark, checked directly, then 5%, then watched at each step up. No big-bang cutovers.
Has the rollback command been run, not just written? One command back to the previous revision — rehearsed, with the data change in the rehearsal.
Do you know what the SLA excludes? Your code’s errors are yours. Slow-but-successful is “uptime”. Plan your own error budget on top of the 99.95%.
Is all background work finished inside the request? If not, it needs instance-based billing or a queue. Test what happens after the response is sent, not just before.
Did the load test happen in staging first? Cold starts, the latency tail, and the rollback — all three observed in staging, none of them debuted in production.

Trust the platform. Audit the defaults. So, back to that thread’s question: how reliable is Cloud Run, really? Reliable — measurably, contractually, 99.95%-with-credits reliable — at the layer Google runs. Your service’s reliability is a separate number, written almost entirely in six settings most teams never open: minimum instances, concurrency, CPU allocation, timeout, maximum instances, and how traffic moves between revisions.

None of this is an argument against migrating. Cloud Run honours the platform bargain — with sharper tools and sharper edges. Keep the bargain. Just read the settings page like it is part of the contract, because for your users, it is the only part they will ever feel.

Honey Kiran · DevOps & Platform Engineering

Cloud, Kubernetes, DevOps, SRE, and platform engineering — explained the practical way.

Moving something latency-sensitive to Cloud Run — or already there and wondering which default is quietly taxing you? Tell me which setting bit you first in the comments.

Examples are illustrative — validate in your own environment before running in production.

Disclaimer

This post describes my personal experience with my own systems and accounts, in my own words. It is a first-person account of what I observed and what I do in my own work — not a statement of any company’s intentions, policies, or internal decisions. Where I do not know something, the post says so.
I am not affiliated with, endorsed by, or speaking for any company, product, or vendor named in this post. All product and company names are trademarks of their respective owners and are mentioned only to describe the experience accurately.
Any error messages, documentation excerpts, or figures quoted here are reproduced from my own records or from the public sources linked in the post, believed accurate at the time of writing. Product behavior changes over time; verify against current official documentation before relying on any detail.
Nothing in this post is professional, financial, legal, or security advice for your specific situation. Examples are illustrative — validate in your own environment before running anything in production.
Corrections are welcome. If any company named here publishes an official statement, or if any detail is wrong, I will update this post and note the correction. The goal is an accurate public record, not a verdict.

Your Image Tag Says ‘latest’. Your Rollback Says Good Luck.

Honey Kiran — Mon, 19 Oct 2026 03:28:00 +0000

Over on r/devops, a thread titled “How does your organisation handle CI/CD and image tagging for microservices?” asked a simple question — how do you tag your container images across a fleet of services? It was crossposted to r/kubernetes, and the answers split into two camps: teams with a scheme, and teams with a habit. The habit, almost every time, is one word: latest .

Picture the scene — illustrative, but you have lived a version of it. A deploy goes out at night. Something is wrong, nobody knows what yet, and the call is the sensible one: roll back. Somebody re-applies the manifest from before the deploy. The manifest says image: api:latest. The pods come back green. Everyone exhales.

Except the rollback rolled nothing back. Between the good deploy and the bad one, another build pushed to latest. The tag moved. The cluster pulled whatever latest pointed at that minute — the broken build, or a third build nobody had tested. The team did the right thing, quickly, and production still got a surprise. That is not a people problem. That is a naming problem.

A tag is a nickname, not a name

Start from the floor: this whole mess is two definitions wearing one coat.

A container image (the packaged app — code, runtime, and files bundled into one runnable unit) lives in a registry (the store that holds images and hands them out when a cluster asks). When you push an image, you give it a tag: a short human-readable label after the colon, like api:1.4.2 or api:latest.

Here is the part nobody reads on the box: a tag is just a pointer. By default, tags are mutable (changeable). Push a new image with the same tag and the tag quietly leaves the old image and points at the new one. The old image is still in the registry, but it has lost its label. Kubernetes documentation says it plainly: tags can be moved to point to different images, but digests are fixed.

The digest is the image’s other identity. It is a SHA-256 hash (a fingerprint computed from the content itself — change even one byte and you get a completely different fingerprint) written as sha256: followed by 64 characters. You reference it with an @: api@sha256:.... A digest does not point at the content. In a real sense, it is the content’s name, because it is calculated from it.

And latest? latest is not special software magic. It is simply the tag you get when you do not specify any tag at all. Push or pull an image with no tag and everyone — Docker, Kubernetes — assumes you mean latest. It carries no promise about being newest, best, or tested. It only means “whatever this tag pointed at when you asked.”

One more uncomfortable truth: a tidy tag does not save you. api:1.4.2 looks like a version and reads like a promise, but unless the registry locks it, that tag is exactly as movable as latest. A nickname that looks formal is still a nickname.

A tag tells you what someone called it. A digest tells you what it actually is.

Why ‘latest’ breaks rollbacks first

A rollback (returning production to the previous, known-good version) has one job: run the exact code that ran before. Not similar code. Not code with the same label. The exact code.

With a moving tag, you cannot even name that code. “Roll back to latest“ says nothing, because latest is a question, not an answer — and the registry will answer it fresh every time you ask. If a build lands mid-incident, your rollback target moves while you are rolling back. It is like giving someone directions to “the blue house” on a street where a painter is working his way down the block.

Kubernetes makes the trap spring faster. The imagePullPolicy setting (the rule that decides when the kubelet — the agent on each node that runs pods — downloads an image) is defaulted for you when you leave it out. Leave it out with :latest, or with no tag at all, and the default is Always: every new pod re-asks the registry what the tag means right now. Leave it out with a specific tag or a digest, and the default is IfNotPresent: pull only if the node does not already have it. So a latest deployment does not even fail consistently — old pods keep the old image, new pods get the new one, and for a while your fleet is running a mixture that exists in no manifest anywhere. Kubernetes’ own guidance is blunt: avoid :latest in production, because it is harder to track which version is running and harder to roll back properly.

Then comes the audit question: what code was running at 2 PM? With digests recorded, that is a lookup. With latest, it is archaeology — push logs, pipeline timelines, and hope. Auditors are not famous for accepting hope as evidence.

Tag like you mean it: SHA plus semver

So what should CI/CD (continuous integration and continuous delivery — the automated pipeline that builds, tests, and ships every change) stamp on each image? Two tags, doing two different jobs.

First, every build gets the full Git SHA as a tag. The Git SHA (the fingerprint Git gives each commit — each saved snapshot of the source code) is unique to that exact source. Tag api:a91f... (use the full SHA in real life, not the short form) and anyone can walk from a running image back to the precise code that produced it. No two different builds ever share it, because no two different commits share it. This is the tag for machines and for tracing.

Second, a release also gets a semver tag, for humans. Semver (semantic versioning) is the three-number scheme MAJOR.MINOR.PATCH: raise MAJOR for changes that break compatibility, MINOR for features that break nothing, PATCH for backward-compatible bug fixes. The spec also carries a rule that fits this whole post: once a version is released, its contents must not be modified — changes ship as a new version. v1.4.2 tells a human what kind of change this is at a glance. A real job — just not the job of proving identity.

Here is the shape of it in a pipeline. Read the comments — the last step is the one most pipelines skip, and it is the one that matters most:

IMAGE=registry.example.com/shop/api
SHA=$(git rev-parse HEAD)
VERSION=1.4.2

docker build -t $IMAGE:$SHA -t $IMAGE:v$VERSION .
docker push $IMAGE:$SHA
docker push $IMAGE:v$VERSION

# Capture the digest - this is the identity you deploy
docker inspect --format='{{index .RepoDigests 0}}' $IMAGE:$SHA
# Illustrative output:
# registry.example.com/shop/api@sha256:1a2b3c...(full 64-character hash in real output)

Two tags go up: the SHA tag, unique forever, and the release tag, readable by humans. Then the pipeline captures the digest from the pushed image and records it with the build — in the build output, the release notes, wherever your team actually looks. That recorded digest is now the deployable fact. Everything after this point should use it.

Deploy the digest, not the nickname

Now the deploy side. A Deployment (the Kubernetes object that keeps a set of identical pods running and replaces them when the spec changes) should carry the image in its full form: name, tag, and digest, joined like this: name:tag@sha256:digest.

Why keep the tag in the string at all? For the human reading the manifest at 2 AM — the tag says “release 1.4.2.” But when tag and digest are both present, Kubernetes pulls only by the digest. The tag is a comment living inside the address. Humans read the nickname; the machine obeys the fingerprint.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  namespace: shop
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
        - name: api
          image: registry.example.com/shop/api:1.4.2@sha256:REPLACE_WITH_THE_DIGEST_FROM_YOUR_BUILD
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8080

Notice imagePullPolicy: IfNotPresent sitting happily next to the digest pin. That pairing makes sense precisely because a digest never changes: if the node already holds that exact fingerprint, there is nothing newer to fetch — newer does not exist. Compare that with Always plus latest, where every pod start is a fresh roll of the dice.

And rollback? Rollback becomes the most boring operation in your runbook: take the previous recorded digest, put it back in the manifest, apply. No re-pushing old images. No re-tagging anything to make yesterday’s build wear today’s label. No guessing. Boring is not a side effect here. Boring is the feature.

Lock the registry door — and sign it, if you can

Everything above is discipline. Discipline is good, but discipline that a tired human can accidentally break at release time needs a lock behind it. Registries sell exactly that lock: immutable tags (tags the registry itself refuses to move once pushed).

On Amazon ECR (Amazon’s container registry), the tag setting defaults to MUTABLE — overwrites allowed. Set ImageTagMutability to IMMUTABLE and a second push under an existing tag is refused, with an ImageTagAlreadyExistsException. The mistake stops being silent and becomes a loud, cheap failure in the pipeline, where failures belong. On Google Artifact Registry (Google Cloud’s registry), immutable tags mean you cannot delete a tagged image, remove a tag, move a tag to another digest, or push a tag another version already uses. Docker Hub offers the same idea — all tags immutable, or only specific ones.

Two honest warnings before you flip the switch on everything you own. First, flip it repository by repository, and tell the team before you do — the first person to learn about it should not learn from a failed release push. If your flow genuinely needs one moving label in development, keep that in a development repository and keep production locked. Second —

Staging warning: test the whole flow in staging first. Turn immutability on in a staging repository, push the same tag twice on purpose, and watch the second push fail. Then practise a digest rollback there. A lock you have never tested is just a new way to be surprised, and production charges full price for surprises that staging would have given you free.

If you can go one step further, sign the image. Signing (attaching a cryptographic signature — a tamper-evident seal proving who produced an artifact and that it has not changed since) is what cosign, from the Sigstore project, does for container images. The rule that matters: always sign the digest, never just the tag — sign a tag and you might seal something you did not intend, because the tag can move after you sign it. Verification then checks that the signature’s digest matches the image in front of it. Signing does not fix tagging. It sits on top of digest pinning and answers the next question: not just “is this the exact image?” but “is this the exact image we built?”

# Sign the digest, never just the tag
cosign sign --key cosign.key registry.example.com/shop/api@sha256:REPLACE_WITH_THE_DIGEST_FROM_YOUR_BUILD

# Verify before anything deploys it
cosign verify --key cosign.pub registry.example.com/shop/api@sha256:REPLACE_WITH_THE_DIGEST_FROM_YOUR_BUILD

The tagging checklist

Pin this where your pipeline lives. Work it top to bottom for one service first — one service done properly teaches the whole fleet the pattern.

Never deploy latest : Not in production, not in staging — latest, or no tag at all, is for a quick test on your own laptop and nothing else.
Tag every CI build with the full Git SHA: Unique, automatic, and traceable back to the exact source that produced the image.
Add a semver tag on releases, for humans: v1.4.2 tells people what kind of change shipped — MAJOR breaks, MINOR adds, PATCH fixes.
Record the digest from every build: Capture it in the build output and keep it with the release record. No record, no rollback.
Deploy by digest: The manifest carries name:tag@sha256:digest — humans read the tag, Kubernetes pulls the digest.
Roll back by restoring the previous digest: Never by re-pushing or re-tagging an old image to make it look current.
Turn on immutable tags in the registry: ECR IMMUTABLE, Artifact Registry immutable tags, or Docker Hub’s setting — repository by repository.
Test in staging first: Push the same tag twice and watch it fail; practise the digest rollback before production needs it.
Sign the digest if your pipeline can: cosign sign the digest, verify before deploy — proof of what it is and whose it is.
Keep latest in its place: If you keep it at all, it is a convenience label in development only — never a deploy reference, never a rollback plan.

Make rollback boring again

None of this is exotic: a SHA tag, a semver tag, a recorded digest, a manifest that deploys the digest, a registry that refuses to move tags, a signature if you can. An afternoon of pipeline work — mostly deleting habits, not adding machinery.

What you get back is what latest quietly stole: certainty. What is running? That digest. What ran yesterday? The previous digest. Roll it back — done, because the previous digest was written down before anyone needed it.

Let latest live in development, where surprises are cheap tuition. In production, let the manifest say the fingerprint. Nicknames are for friends. Rollbacks need names.

Honey Kiran · DevOps & Platform Engineering

Kubernetes, Docker, CI/CD, DevOps, SRE, and platform engineering — explained the practical way.

Quick check before you close this tab: what do your production manifests reference right now — latest, a tag, or a digest? Tell me in the comments, and if it is latest, tell me what your rollback plan calls it.

Examples are illustrative — validate in your own environment before running in production.

Disclaimer

This post describes my personal experience with my own systems and accounts, in my own words. It is a first-person account of what I observed and what I do in my own work — not a statement of any company’s intentions, policies, or internal decisions. Where I do not know something, the post says so.
I am not affiliated with, endorsed by, or speaking for any company, product, or vendor named in this post. All product and company names are trademarks of their respective owners and are mentioned only to describe the experience accurately.
Any error messages, documentation excerpts, or figures quoted here are reproduced from my own records or from the public sources linked in the post, believed accurate at the time of writing. Product behavior changes over time; verify against current official documentation before relying on any detail.
Nothing in this post is professional, financial, legal, or security advice for your specific situation. Examples are illustrative — validate in your own environment before running anything in production.
Corrections are welcome. If any company named here publishes an official statement, or if any detail is wrong, I will update this post and note the correction. The goal is an accurate public record, not a verdict.

Your Terragrunt CI Re-Downloads the Internet on Every Run

Honey Kiran — Mon, 19 Oct 2026 03:26:52 +0000

Somebody on r/Terraform is watching their Terragrunt CI re-download the internet on every single run — and every obvious fix has a trap in it.

The setup, from the thread: a multi-component Terragrunt project (Terragrunt is a wrapper that keeps Terraform code DRY — don’t repeat yourself — across many components), CI on GitLab running on Kubernetes runners, providers and modules re-downloaded every run. Downloads are slow and sometimes flaky. They tried caching to S3 as compressed archives — the compress and decompress time ate the saving. Pre-baking every provider into the runner image? By their estimate that image lands around 80 GB. A shared EFS volume is now on the table.

Friend, you do not have a download problem. You have a cache-placement problem. And Terraform has a very specific opinion about where provider binaries may safely be shared — ignore it and your parallel runs will corrupt each other’s downloads in fun, non-reproducible ways.

Every init downloading the same provider is not thoroughness. It is your pipeline paying the same toll twenty times a day.

What is actually being downloaded, again and again

A provider is the plugin binary that lets Terraform talk to a cloud — the AWS provider alone is a large binary, and a multi-component project downloads the same providers once per component, per run, because each component initialises in its own working directory. Modules (reusable Terraform building blocks) get fetched per component too, into each unit’s .terragrunt-cache. Multiply by every pipeline, every day. That is the bill — in minutes, and in flaky failures when a registry hiccup fails a job that changed three lines.

Trap 1: the plain plugin cache directory, shared by parallel runs

Terraform’s own shared plugin cache — plugin_cache_dir in the CLI config, or the TF_PLUGIN_CACHE_DIR environment variable — sounds like the answer: point every run at one folder, download once. Read the fine print in Terraform/OpenTofu’s own docs, though: that cache directory is not guaranteed to be concurrency safe, and the provider installer’s behaviour with multiple concurrent init calls is undefined. Two runners populating the same cold cache at the same moment can meet a half-unpacked provider. Terragrunt’s own performance docs add the other half: at large scale, filesystem lock contention on that shared directory becomes the bottleneck. It works in a demo. CI is not a demo.

Trap 2: zip it, ship it, unzip it

The thread’s S3 proof-of-concept failed for a boring reason: compression is a tax you pay twice — once packing, once unpacking — on every run, warm or cold. A cache that costs minutes to restore is just a slower download wearing a costume. Same family: pre-baking providers into the runner image trades download time for an image so large the runner spends its life pulling it instead. The thread’s ~80 GB estimate is their number, not a law of physics — but the direction is right: the image becomes the download.

The fix Terragrunt already built: the Provider Cache Server

Terragrunt ships a provider cache server made for exactly this shape — many units, parallel runs, one shared cache. Turn it on per run:

terragrunt run --all --provider-cache plan

or with environment variables, which is how you will actually do it in CI:

TG_PROVIDER_CACHE=1 \
TG_PROVIDER_CACHE_DIR=/shared/terragrunt-provider-cache \
terragrunt run --all plan

What happens then: Terragrunt serves providers to its own Terraform/OpenTofu runs from the shared cache directory, downloading from the registry only on a genuine miss, and the cache directory uses the same layout as Terraform’s plugin cache — so an existing warmed cache folder can be reused by pointing --provider-cache-dir (or TG_PROVIDER_CACHE_DIR) at it. By default it caches providers from registry.terraform.io and registry.opentofu.org; private registries can be added with --provider-cache-registry-names / TG_PROVIDER_CACHE_REGISTRY_NAMES. One warning from experience-shaped common sense: the cache dir must be a filesystem your runners actually share (your EFS idea works for this), and it must survive between runs — a cache on an ephemeral runner disk that dies with the pod is a diary, not a cache.

On newer OpenTofu (1.10 and later), Terragrunt also has an automatic provider cache dir mode that sets TF_PLUGIN_CACHE_DIR to a central location for you, since that OpenTofu line handles concurrent cache access safely — Terragrunt’s docs position the cache server as the pick for Terraform and older OpenTofu, or when you need the advanced sharing. Know which of the two you turned on; running both by accident is how cache behaviour becomes folklore.

And the modules?

Providers are the heavy half, but do not ignore the other download: Terragrunt fetches each unit’s module source into its per-unit .terragrunt-cache. Persisting that between CI runs helps, but key the cache on your lockfiles and config hashes, and prune it — Terragrunt and env0-style guidance both warn the cache directory grows large in CI if you never clean it. A cache with no eviction plan is a disk-full incident with a waiting period.

The CI caching checklist

Cache server on: TG_PROVIDER_CACHE=1 set for Terragrunt runs in CI.
Shared, persistent dir: TG_PROVIDER_CACHE_DIR points at storage that outlives the runner pod (shared volume, not pod disk).
Private registries listed: any registry beyond the two defaults added via TG_PROVIDER_CACHE_REGISTRY_NAMES.
No naked shared plugin cache under parallelism: plain TF_PLUGIN_CACHE_DIR sharing is not concurrency-guaranteed — that is the cache server’s job.
Module cache keyed + pruned: .terragrunt-cache persisted with a lockfile-based key and a size cap.
No zip archives: if restoring the cache means decompressing it, you rebuilt the download with extra steps.
No provider-stuffed runner image: the image stays small; the cache lives outside it.
Staging warning: test cache behaviour in staging first — a poisoned shared cache fails every pipeline at once.

The fix, in one line: stop downloading providers per run and stop sharing the naive cache folder across parallel inits — switch on Terragrunt’s provider cache server, park its directory on storage that survives the runner, and let the registry become the fallback instead of the routine.

Disclaimer

This post describes my personal experience with my own systems and accounts, in my own words. It is a first-person account of what I observed and what I do in my own work — not a statement of any company’s intentions, policies, or internal decisions. Where I do not know something, the post says so.
I am not affiliated with, endorsed by, or speaking for any company, product, or vendor named in this post. All product and company names are trademarks of their respective owners and are mentioned only to describe the experience accurately.
Any error messages, documentation excerpts, or figures quoted here are reproduced from my own records or from the public sources linked in the post, believed accurate at the time of writing. Product behavior changes over time; verify against current official documentation before relying on any detail.
Nothing in this post is professional, financial, legal, or security advice for your specific situation. Examples are illustrative — validate in your own environment before running anything in production.
Corrections are welcome. If any company named here publishes an official statement, or if any detail is wrong, I will update this post and note the correction. The goal is an accurate public record, not a verdict.

Honey Kiran · DevOps & Platform Engineering

Writing on monitoring, SRE, Terraform, Kubernetes and platform engineering — one hard-won lesson at a time.

Tell me in the replies: what does your CI re-download every run that it has no business downloading twice?

Examples are illustrative — validate in your own environment before running in production.