DEV Community: Amaresh Pelleti

Redis 8.6: What Changed in Performance and Event Streams

Amaresh Pelleti — Fri, 16 Oct 2026 12:42:55 +0000

Redis 8.6 went GA in February 2026 and the latest patch, 8.6.7, shipped September 17 with security fixes only — the feature set has been stable for months. The headline feature isn't a speed number. It's a real fix for a problem every Streams-as-a-queue user has hit: what happens when your producer times out waiting for an XADD reply and doesn't know if the message landed. Here's Redis 8.6 explained with the commands that actually change how you'd build on it, verified against Redis's own release notes and idempotency docs.

Redis 8.6 Stream Idempotency: What XADD IDMP Actually Fixes

If you've built a queue on Redis Streams, you've hit this: a producer calls XADD, the connection drops before the reply arrives, and the producer has no way to know whether the message was written. Retry, and you risk a duplicate. Don't retry, and you risk losing the message. Per Redis's own docs, this is exactly the gap Redis 8.6 closes with idempotent message processing, new in this release.

Two modes, both added to XADD:

# Manual mode — you supply both producer ID and message ID
XADD mystream IDMP producer-1 iid-1 * field value

# Automatic mode — Redis derives the message ID from the field/value content
XADD mystream IDMPAUTO producer-1 * field value

Resend the exact same (producer-id, iid) pair — whether because the producer retried after a timeout or crashed and restarted — and Redis returns the ID of the original entry instead of writing a duplicate. IDMPAUTO skips supplying your own ID and lets Redis hash the field/value pairs instead, so identical content gets deduplicated automatically; the tradeoff is the hash calculation, which Redis's own docs note makes IDMPAUTO "slightly slower" than manual IDMP.

Retention is tunable per stream via XCFGSET:

XCFGSET mystream IDMP-DURATION 300 IDMP-MAXSIZE 1000

IDMP-DURATION (1–86,400 seconds, default 100) controls how long Redis remembers an ID; IDMP-MAXSIZE (1–10,000, default 100) caps how many IDs it tracks per producer, and is the harder limit of the two — Redis evicts the oldest tracked IDs once IDMP-MAXSIZE is hit, regardless of how much time is left on IDMP-DURATION. Redis's own sizing guidance: set IDMP-DURATION to roughly how long your producer takes to recover from a crash and resume sending, and size IDMP-MAXSIZE as mark-delay (ms) × messages/ms + margin — their own worked example is 1K msgs/sec at an 80ms mark-delay needing IDMP-MAXSIZE around 100.

This isn't free, and Redis is specific about the cost: 2–5% lower throughput on XADD compared to the non-idempotent path, with under 1.5% additional memory overhead. For a queue where a duplicate message means double-charging a customer or double-sending a notification, that's a cheap trade. For a high-throughput logging stream where an occasional duplicate is harmless, it's not worth turning on. XINFO STREAM now reports iids-duplicates so you can actually measure how often it's catching something, rather than guessing.

One real caveat from the release notes, not a hypothetical: avoid IDMP/IDMPAUTO if you're running appendonly yes with the non-default aof-use-rdb-preamble no — Redis flags this combination as a known limitation to be fixed in a later patch.

New Eviction Policies: volatile-lrm and allkeys-lrm

Redis 8.6 adds two eviction policies — volatile-lrm (keys with a TTL) and allkeys-lrm (any key) — that evict based on least recently modified, not least recently used. The distinction matters for read-heavy caches: standard LRU treats a read as "recently used" and protects the key from eviction, while LRM only resets the clock on a write. That means a key you read constantly but never update is exactly as evictable as one nobody's touched, which is the behavior you actually want if "modified recently" is a better proxy for relevance than "read recently" in your workload (a cached computed result, for instance, versus a hot but static config value you're not trying to protect from eviction just because it gets read a lot).

Hot Key Detection: the HOTKEYS Command

Redis 8.6 also ships a new HOTKEYS command for real-time detection and reporting of frequently accessed keys — useful for finding the one key that's skewing load on a cluster shard before it becomes an incident, rather than reconstructing it after the fact from slow-log guesswork.

Like our recent look at PostgreSQL 19, this is a release where the real-world impact comes down to one or two specific changes, not the full feature list.

What Redis Doesn't Quantify

Redis's own materials describe "substantial memory reduction for hashes and sorted sets" (from unifying field/value and score/value into single structs) and "substantial performance improvements" across more than 20 optimizations — but neither the official 8.6 release notes nor the "what's new" page attaches a specific percentage to either claim. Worth knowing if you're deciding whether to benchmark your own workload before upgrading: Redis isn't hiding a number here, there just isn't one published. The one area where Redis did publish a number is the idempotency overhead above (2–5% throughput, <1.5% memory) — because that's a cost, and costs get measured.

Should You Upgrade to Redis 8.6?

Running Streams as a queue or event log where duplicates are costly: yes, specifically for IDMP/IDMPAUTO. This is the first Redis-native answer to a problem people have previously solved with external dedup tables.
Read-heavy caching workload with a mix of hot-but-static and actively-updated keys: worth testing allkeys-lrm against your current allkeys-lru policy — it's a config change, not a migration.
Running Redis via Docker Compose: a version bump is a good moment to double-check your compose file doesn't recreate the container unnecessarily on every up — unrelated to 8.6 itself, but worth catching while you're already touching the service definition.
Running appendonly yes with aof-use-rdb-preamble no: hold off on IDMP/IDMPAUTO specifically until the known limitation is patched; the rest of 8.6 is unaffected.
Just want the performance improvements: they're real per Redis's own notes, just not independently quantified — benchmark your own workload rather than expecting a specific percentage.

Frequently Asked Questions

Q: Does Redis 8.6 guarantee exactly-once delivery for Streams?
A: No — it guarantees at-most-once production (no duplicate entries from a retried XADD). Consumer-side processing guarantees are a separate concern handled by consumer groups and acknowledgments, which idempotent production doesn't change.

Q: What's the performance cost of enabling IDMP?
A: Per Redis's own docs: 2-5% lower XADD throughput, under 1.5% additional memory, with IDMPAUTO slightly slower than manual IDMP due to its hash calculation.

Q: Is allkeys-lrm a replacement for allkeys-lru?
A: Not a direct replacement — a different eviction signal. LRU tracks reads, LRM tracks writes. Pick LRM when "recently modified" better predicts what's still relevant in your workload than "recently accessed" does.

Q: What's the latest Redis 8.6 patch?
A: 8.6.7, released September 17, 2026 — security fixes only (an unauthenticated-cluster-bus-port issue and a few crash fixes). The feature set has been unchanged since the 8.6.0 GA in February 2026.

Quick Summary:

Redis 8.6 (GA February 2026, latest patch 8.6.7) adds idempotent Streams production via XADD IDMP/IDMPAUTO — real duplicate-message prevention for queue-style use cases
Cost is measured and specific: 2-5% lower XADD throughput, <1.5% extra memory; tune retention with XCFGSET's IDMP-DURATION/IDMP-MAXSIZE
New volatile-lrm/allkeys-lrm eviction policies evict by last-modified time instead of last-accessed time
New HOTKEYS command surfaces frequently-accessed keys in real time
"Substantial" memory/performance gains are real per Redis's own notes but not independently quantified — benchmark your own workload

If you're running Streams as a real queue, XADD IDMP is the first Redis-native fix for the retry-duplicate problem — worth adopting even if you skip everything else in this release.

Cilium mTLS ztunnel: Sidecar-Free Encryption Explained

Amaresh Pelleti — Thu, 15 Oct 2026 12:37:20 +0000

Cilium mTLS ztunnel doesn't need a SPIRE server — and that's actually the second-biggest news here. The bigger story: Cilium's older, SPIRE-based "Mutual Authentication" feature is now officially deprecated, with removal targeted for v1.21. If you adopted it, you're migrating whether you planned to or not. I fetched the real docs and the maintainer's own deprecation issue to find out what that migration actually involves — and the real gotcha isn't SPIRE at all. (New to Cilium or Kubernetes networking generally? Start with our Kubernetes complete guide.)

What Changed: SPIRE-Based Mutual Auth Is Being Removed

Cilium shipped a SPIRE-based "Mutual Authentication" feature back in 2023 (as a Beta), requiring a SPIRE server deployed alongside Cilium (authentication.mutual.spire.enabled=true) to issue and verify identity certificates. Per the maintainer's own deprecation issue, opened by a Cilium committer: "Given the lack of progress on Mutual Auth, and the presence and at least equal stability of the new feature, as of Cilium v1.20, Cilium committers are marking the Mutual Auth feature as deprecated, and will remove the feature in a later version, most likely v1.21." The agent-side removal PR is already merged.

The replacement is ztunnel Transparent Encryption — also Beta, but actively developed, and it doesn't ask you to run SPIRE at all.

Cilium mTLS ztunnel: How It Actually Works (No SPIRE Needed)

Per Cilium's ztunnel documentation, ztunnel is "a purpose-built per-node proxy that provides transparent Layer 4 mTLS encryption and authentication for pod-to-pod communication." Instead of a SPIRE server, it needs a set of Kubernetes secrets generated with a short OpenSSL script the docs provide — a bootstrap key pair (secures the connection between ztunnel and Cilium's certificate server) and a CA key pair (the root for ephemeral, in-memory client certificates ztunnel issues on request):

openssl genrsa -out bootstrap-private.key 2048
# ... (bootstrap cert generation, see docs)
openssl genrsa -out ca-private.key 2048
# ... (CA cert generation, see docs)

kubectl --namespace kube-system create secret generic cilium-ztunnel-secrets \
  --from-file=bootstrap-private.key=bootstrap-private.key \
  --from-file=bootstrap-root.crt=bootstrap-root.crt \
  --from-file=ca-private.key=ca-private.key \
  --from-file=ca-root.crt=ca-root.crt

Then enable it at install time:

cilium install 1.20.2 \
  --set encryption.enabled=true \
  --set encryption.type=ztunnel

One hard precondition the docs call out: Cluster Mesh must be disabled first — ztunnel "is currently not compatible with Cluster Mesh," and enabling both together throws a validation error.

Enrolling Namespaces (Not Pods)

Once ztunnel is running, nothing is encrypted until you opt namespaces in. Per the docs, enrollment is a namespace label:

kubectl label namespace <namespace-name> io.cilium/mtls-enabled=true

Every existing pod in that namespace gets enrolled (ztunnel pods excepted), iptables rules get installed in each pod's network namespace to redirect traffic to the local ztunnel proxy, and every future pod created in the namespace is enrolled automatically. Disenrolling is the same label, removed: kubectl label namespace <namespace-name> io.cilium/mtls-enabled-.

That namespace-level granularity sounds convenient for a gradual rollout. It's also where the real migration risk is.

The Real Migration Gotcha: Partial Enrollment Breaks Traffic

The docs are explicit about this, in one line easy to skim past: "Traffic between workloads is only supported when both the source and destination endpoints are enrolled in ztunnel. Communication between an enrolled workload and a non-enrolled workload is not supported."

In practice: if you enroll payments but haven't gotten to orders yet, and payments calls orders, that call breaks — not silently degrades to unencrypted, just doesn't work. A namespace-by-namespace rollout plan has to account for every namespace a given workload talks to, not just the one you're migrating. For anything beyond a small cluster, map your service-to-service dependencies before you label the first namespace, not after something stops responding.

The Bigger Gotcha: It Can Silently Break L4 NetworkPolicy

This is the one that catches people who already did their homework on the namespace-enrollment rule. Per the docs: "ztunnel interferes with Cilium network policy as traffic is encrypted before it leaves the pod, meaning L4 policies won't work except for directly targeting the ztunnel HBONE port (15008)."

If you're running Kubernetes NetworkPolicies or Cilium-native L4 policies to restrict which pods can talk to which ports, enabling ztunnel on a namespace can quietly stop those policies from doing their job — because by the time your policy would evaluate the traffic, it's already encrypted and tunneled through port 15008. This doesn't throw an error. It just means an L4 rule you're relying on may no longer enforce what you think it does. Re-verify your NetworkPolicy behavior after enrolling a namespace, not just your mTLS connectivity.

Cilium mTLS ztunnel: Other Real Limitations (Confirmed in the Docs)

TCP only. UDP and other protocols aren't redirected through ztunnel — if a service depends on UDP, it isn't covered by this encryption.
Namespace-level enrollment only. No per-pod opt-in; everything in an enrolled namespace is in.
Host-networked pods are skipped. No network namespace path means no enrollment — they won't error, they'll just stay unencrypted.
Requires iptables support in the kernel. Minimal container runtimes without iptables can't use it.

mTLS secures what's in transit between pods; it doesn't tell you what a compromised pod does once it's running. Pair it with runtime visibility — see our Falco on Kubernetes guide for the detection side of that equation.

Frequently Asked Questions

Q: Do I need to run a SPIRE server for Cilium's native mTLS?
A: No. ztunnel generates and manages its own certificates from a bootstrap/CA key pair you create once via a provided OpenSSL script — no external PKI or SPIRE deployment required.

Q: Is Cilium's SPIRE-based Mutual Authentication feature going away?
A: Yes. Per the Cilium maintainers' own deprecation issue, it was marked deprecated as of v1.20, with removal "most likely" in v1.21. The agent-side removal code is already merged.

Q: Can I migrate one namespace at a time?
A: Technically yes via the namespace label, but only if every namespace that workload talks to is migrated together — traffic between an enrolled and a non-enrolled workload isn't supported.

Q: Will my existing NetworkPolicies still work after enabling ztunnel?
A: Not necessarily. L4 policies stop enforcing as expected once ztunnel encrypts traffic before it leaves the pod, except for rules that directly target ztunnel's HBONE port (15008). Re-test policy behavior after enrollment.

Q: What Cilium version is this in?
A: ztunnel Transparent Encryption is Beta as of Cilium 1.20.2, the current stable release at time of writing.

Quick Summary:

Cilium's older SPIRE-based Mutual Authentication is deprecated as of v1.20 and slated for removal in v1.21 — this isn't optional for long
The replacement, ztunnel Transparent Encryption, needs no SPIRE server — it generates its own bootstrap/CA certs via a provided script
Enrollment is per-namespace (io.cilium/mtls-enabled=true), not per-pod
Real gotcha #1: traffic between an enrolled and a non-enrolled workload isn't supported — map dependencies before migrating
Real gotcha #2: L4 NetworkPolicy can silently stop enforcing once ztunnel encrypts traffic, except for rules targeting port 15008
Cluster Mesh is incompatible with ztunnel and must be disabled first; only TCP is covered, not UDP

Cilium mTLS ztunnel is where this migration ends up either way: if you're still running the SPIRE-based mutual auth, the removal timeline makes it a when, not an if — plan the namespace rollout around your actual service dependency graph, and re-test NetworkPolicy behavior before you call it done.

Grafana Tempo 3.1: Query-Based Trace Redaction Explained

Amaresh Pelleti — Wed, 14 Oct 2026 12:20:15 +0000

Grafana Tempo 3.1 shipped on 2026-09-30 with a feature that matters the moment something goes wrong, not when everything's fine: redaction by query. Before this release, scrubbing a leaked secret out of your traces meant enumerating trace IDs one at a time. Now you can hand Tempo a TraceQL query and it finds and removes every matching trace. Here's the exact command, verified against Grafana's own docs, plus the rolling-upgrade gotcha that can make a "successful" redaction silently incomplete.

What's Actually New in Grafana Tempo 3.1

Per Grafana's release notes, Tempo 3.1's headline changes are:

Redaction by query — remove traces matching a TraceQL query instead of enumerating trace IDs.
Kafka ingestion improvements — authenticated, TLS-encrypted Kafka connections (SASL SCRAM-SHA-256, SCRAM-SHA-512, OAUTHBEARER, AWS_MSK_IAM), rack-aware fetching via client_rack to cut cross-zone transfer costs, and ingest.kafka.producer_compression for Azure Event Hubs' required gzip compression.
TraceQL metrics extrapolation and arithmetic — extrapolate rates from head-sampled traces so metrics reflect real traffic volume, and combine aggregations with arithmetic to compute ratios.
vParquet5 becomes the default block format. Existing vParquet4 blocks stay readable with no migration needed; you can pin the old format with storage.trace.block.version: vParquet4 if you need to.

This article focuses on redaction, since it's the feature most teams will actually need to reach for under pressure — and the one where getting the details wrong has real consequences. If you're building out observability more broadly, see our guide to real-time log monitoring for the logs side of the same problem.

The Real Incident: A Secret Leaked Into a Span Attribute

The scenario this feature exists for: an API key, a password, or a customer PII field ends up in a span attribute — logged by an overly verbose instrumentation library, a debug flag left on, or a bug in a custom span processor. It's now sitting in your trace storage, queryable by anyone with Explore access, until you do something about it.

Before Tempo 3.1, your options were either live with it until the retention period expired, or find every offending trace ID manually and redact them one by one. Query-based redaction skips that — useful whether you're running Tempo standalone or as part of a full Kubernetes observability stack.

Grafana Tempo 3.1 Redaction: The Exact Command

Per Tempo's tempo-cli documentation, the redact command has two modes. Trace-ID mode (the old way, still supported, up to 1000 IDs per request):

tempo-cli redact --tenant=<TENANT_ID> --trace-id=<TRACE_ID> [--trace-id=<TRACE_ID> ...] <scheduler-address>

Query mode (the new way):

tempo-cli redact --tenant=<TENANT_ID> --query=<TRACEQL_QUERY> --start=<START> --end=<END> <scheduler-address>

For the leaked-API-key scenario, assuming the key landed in a span attribute called http.request.header.authorization:

tempo-cli redact \
  --tenant=prod \
  --query='{ span.http.request.header.authorization = "sk-live-abc123" }' \
  --start=now-7d \
  --end=now \
  --dry-run \
  tempo-scheduler.prod.svc:9095

Run with --dry-run first — it evaluates the query and reports a match count without rewriting anything. Once the count looks right, drop --dry-run to actually redact.

A real constraint worth knowing before you write the query: Tempo's docs specify the TraceQL support here is narrower than a normal TraceQL search — redaction "supports only single spanset filters with = comparisons on resource.* or span.* attributes, joined by && or ||." You can't use the full TraceQL grammar (no duration comparisons, no nested spanset logic) — just flat equality checks on resource or span attributes.

What Redaction Actually Does — and Doesn't Do

This is the part generic release-note summaries skip: redaction doesn't strip one field out of a trace. Per the docs, the command "rewrites all blocks for a tenant that contain specific trace IDs. The traces are dropped from the new blocks." The entire matching trace is removed from object storage, not just the offending attribute. If you need to keep the rest of the trace and only scrub one field, this feature doesn't do that — it's all-or-nothing at the trace level.

Tempo's own safety warnings are worth repeating verbatim because they change how you should actually run this in production:

"Redaction rewrites blocks in object storage and can't be undone." There's no soft-delete, no recovery window.
Stop ingesting the sensitive data at its source first — fix the instrumentation bug before you clean up its output, or you're redacting a moving target.
Wait roughly 10 minutes for current blocks to flush to storage before running the redaction, so you're not racing data still in memory.
For query-based redaction, verify your match set in Grafana Explore first, then confirm with --dry-run before dropping it.
A single pass is never fully complete on a live system. Traces still held by block-builders (not yet flushed) aren't covered by a redaction run — you may need a second pass.

The Rolling-Upgrade Gotcha

This is the finding that doesn't show up if you only read the headline feature list. Per Grafana's own 3.1 release notes:

Translate that: if you're mid-rollout — some Tempo workers upgraded to 3.1, some still on 3.0 — and you submit a redaction with --start/--end to limit it to a narrow time window, any worker still running the old version ignores your time window entirely and redacts every trace matching the query across the whole block, with no error or warning telling you it happened. You could intend to scrub one afternoon's worth of traces and instead silently wipe weeks of data, discovering it only when someone asks where their traces went.

The practical rule: finish the Tempo 3.1 rollout across every scheduler and worker before you run your first windowed redaction. If you're not certain every component is upgraded, run trace-ID-based redaction instead — it doesn't have this failure mode. It's the same discipline covered in our broader list of DevOps tools worth having in your incident-response toolkit before you need them.

Frequently Asked Questions

Q: Does Tempo 3.1's redaction remove just the sensitive field, or the whole trace?
A: The whole trace. Per Tempo's docs, matching traces are dropped entirely from the rewritten blocks — there's no partial-field redaction.

Q: Can I undo a redaction if I made a mistake?
A: No. Tempo's docs state explicitly that redaction "rewrites blocks in object storage and can't be undone." Always run --dry-run first and verify the match count.

Q: What TraceQL syntax does the --query flag actually support?
A: A narrower subset than normal TraceQL search: single spanset filters using = comparisons on resource.* or span.* attributes, combined with && or || only. No duration filters, no nested spansets.

Q: Is it safe to use --start/--end windowed redaction during a rolling upgrade?
A: No — per Grafana's own warning, a worker still on an older Tempo version ignores the time window and redacts every match in the block with no error. Wait until every scheduler and worker is on 3.1 first.

Quick Summary:

Tempo 3.1 (GA'd 2026-09-30) adds redaction by TraceQL query, alongside Kafka auth/TLS improvements, TraceQL metrics arithmetic, and vParquet5 as the new default block format
The redact command: tempo-cli redact --tenant=<id> --query=<traceql> --start=<t> --end=<t> --dry-run <scheduler-address>
Redaction removes the entire matching trace from storage, not just one attribute, and can't be undone
Query syntax is limited to flat = comparisons on resource.*/span.* attributes joined by &&/||
Don't run windowed (--start/--end) redaction mid-rollout — an old worker silently ignores the window and redacts the whole block

Grafana Tempo 3.1's redaction feature is worth having ready before you need it — check your Tempo version across every scheduler and worker now, not during the incident where you actually need to use it.

YAML vs JSON for LLMs: Real Token Counts, Real Gotchas

Amaresh Pelleti — Tue, 13 Oct 2026 11:46:16 +0000

YAML vs JSON for LLMs isn't just a style preference — it changes how many tokens you pay for on every request. I ran both formats through a real tokenizer on identical data instead of repeating the "20-ish percent" figures floating around online, and the gap is bigger than you'd guess, with one real parsing bug that can silently corrupt an LLM's output if you're not careful.

YAML vs JSON for LLMs: The Real Numbers, Measured

Most "YAML saves tokens" claims cite a number without showing their work. So I built two test cases — a nested config object and a 10-row uniform table — and ran both formats through OpenAI's cl100k_base tokenizer (the GPT-4-era encoding, a standard reference point; exact counts shift slightly on Claude's or Gemini's own tokenizers, but the ranking between formats doesn't).

Nested config (not a list — a single object with child objects):

Format	Tokens	Characters
JSON	99	322
YAML	66	223

YAML cut 33.3% of the tokens off the same config, just by dropping braces, commas, and quoted keys.

Uniform table (10 rows, 4 fields each — the shape most AI agent data actually takes: user lists, logs, tool outputs):

Format	Tokens	Characters
JSON	323	963
YAML	221	590
TOON	97	235

Here the picture changes. YAML still beats JSON by 31.6%, but TOON — a format built specifically for this case — beats JSON by 70% and beats YAML by 56.1%.

The reason is structural, not cosmetic. YAML's - id: 1\n name: Alice syntax repeats every field name on every row. TOON declares the field list once (users[10]{id,name,role,active}:) and streams bare values after it. For a one-off config file that repetition barely matters. For a 500-row tool result going into an agent's context window on every turn, it adds up fast.

YAML vs JSON vs TOON: Which Should You Use?

Nested, non-uniform data (app config, a single object with child objects): YAML and TOON are nearly identical here — TOON's spec explicitly borrows YAML's indentation syntax for non-tabular data, so there's no real token advantage to switching formats for this case. Pick YAML; it's the one more tools already read.
Uniform arrays of objects (user lists, log lines, query results, tool outputs): TOON is the clear winner. This is specifically the case it was designed to compress.
Anything hitting an API, a database, or existing software: JSON. It's universal, strict, and every library expects it. Don't reach for YAML or TOON here — convert at the LLM boundary, not before it.

A practical pattern: keep your actual system in JSON, and only convert to YAML or TOON for the text you hand to the model — then have the model hand JSON back for anything that needs to go back into code.

The YAML Gotcha Nobody Warns You About

YAML's compactness comes from a looser grammar than JSON's, and that looseness has a well-documented failure mode: the Norway problem. In YAML 1.1 (still what many libraries implement, including older PyYAML), bare words like no, yes, on, and off parse as booleans, not strings. A country-code list like:

geoblock_regions:
  - dk
  - fi
  - is
  - no
  - se

silently becomes ["dk", "fi", "is", false, "se"] — Norway (no) turns into False. Version-looking strings have the same problem: 10.23 parses as the float 10.23, not the string "10.23", which breaks version-comparison logic that expected text.

This isn't theoretical for this site's audience specifically. GitHub Actions workflow files use on: as a top-level key (on: push). Per a real bug report against the Go YAML library, a generic YAML 1.1 parser reads that key as the boolean true, not the string "on" — so any tooling that parses workflow YAML with a non-GitHub-aware library can silently drop or mishandle the trigger config. GitHub's own runner handles on: correctly because it's YAML-1.2-aware, but if you're writing your own workflow linter, generator, or an LLM-based tool that reads .github/workflows/*.yml, this is exactly the kind of bug that passes code review and fails in production. See our GitHub Actions YAML guide for the rest of the syntax.

The fix, per that same thread, is one of two things: quote the key ("on": push) or use a YAML 1.2-compliant parser. YAML 1.2.2 — the current spec, revised 2021-10-01 — dropped the bare on/off/yes/no boolean shortcuts specifically because of bugs like this one. If you're feeding YAML into an LLM and expect it to reliably reason about or regenerate that YAML, this gotcha matters doubly: the model has almost certainly also been trained on YAML 1.1-style parsing assumptions.

YAML vs JSON for LLMs: Why Token Count Matters More for Agents

A single chat message where YAML saves 60 tokens over JSON is immaterial. It compounds somewhere specific: AI agents that re-send accumulated context — conversation history, tool outputs, retrieved documents — on every turn. If an agent calls a tool 20 times in a session and each tool result is a JSON array of rows, you're paying the JSON tax 20 times over, not once. That's the same economics covered in our prompt compression guide: the lever that matters is the one you pull on every request, not the one you pull once.

Frequently Asked Questions

Q: Is YAML actually faster for an LLM to parse, or just fewer tokens?
A: This article only measured token count, which is the cost and context-window lever. Parsing speed is a separate question and depends on the runtime, not the model.

Q: Does TOON replace YAML?
A: No — they overlap only on uniform tabular data. For nested, irregular structures (most real config files), TOON's own spec uses YAML-style indentation, so there's no efficiency gain from switching.

Q: Why does "no" become false in YAML?
A: YAML 1.1 treats a set of bare words (y, n, yes, no, on, off, true, false) as boolean literals when unquoted. YAML 1.2.2 removed this behavior from the spec, but many parsers (older PyYAML, Go's go-yaml v2) still implement 1.1 rules for compatibility.

Q: Should I quote every YAML value to avoid this?
A: Quote any value that could be misread as a different type — country codes, version strings, and anything resembling yes/no/on/off. For a config key like GitHub Actions' on:, quote the key itself: "on": push.

Quick Summary:

Measured with a real tokenizer: YAML saves ~30-33% of tokens vs JSON on both nested config and tabular data
TOON pulls further ahead specifically on uniform tabular data — 70% fewer tokens than JSON, 56% fewer than YAML, in this test
For nested, non-uniform data, YAML and TOON are about the same size — pick YAML for compatibility
The Norway problem (no → false) is real and has broken actual tooling, including GitHub Actions' on: key in non-GitHub-aware parsers
JSON still wins for anything touching an API or database — convert formats only at the LLM boundary

If you're weighing YAML vs JSON for LLMs in your own pipeline, the honest answer is "it depends on the shape of the data," not "YAML always wins" — measure your own payloads with a tokenizer before committing, and watch for unquoted values that look like booleans.

Terraform State Explained: Locking, Backups, and Recovery

Amaresh Pelleti — Mon, 12 Oct 2026 14:05:19 +0000

Terraform state causes more real incidents than almost anything else in the tool — a stuck lock, a corrupted file, a state rm that touched the wrong resource. Most of the anxiety is unnecessary once you know two things most tutorials skip: Terraform backs up state automatically before every change, and it can't be turned off. Here's terraform state explained with the commands that actually matter, verified against HashiCorp's own docs.

Terraform State Backups: The One You Didn't Know You Had

Per Terraform's state documentation, Terraform "stores each workspace's state in a local file named terraform.tfstate, and a backup of the previous state in terraform.tfstate.backup." That backup isn't optional. Per the state command reference, every command that modifies state "write[s] backup files," and backups for state modification "can not be disabled" — HashiCorp's own words, because of how sensitive state content is.

Practically: before you panic about a bad terraform state rm or state mv, check for terraform.tfstate.backup first. It's sitting right next to your state file, one command edit behind.

Terraform State Commands That Actually Matter

Six subcommands live under terraform state, and you'll use maybe three of them regularly:

Command	What it does
`terraform state list`	Lists every resource currently tracked in state
`terraform state show <resource>`	Shows the full recorded attributes for one resource
`terraform state mv <src> <dst>`	Moves a resource to a new address in state, without touching real infrastructure
`terraform state rm <resource>`	Removes a resource from state tracking, without destroying it
`terraform state pull`	Retrieves and prints the raw remote state
`terraform state replace-provider`	Updates which provider a resource is associated with

mv and rm are the two people reach for under pressure — renaming a resource in code without triggering a destroy/recreate, or un-tracking something Terraform shouldn't manage anymore. Both are state-only operations: nothing in your real infrastructure changes. That's also exactly why the automatic backup exists — a bad mv or rm is a state-file mistake, and the backup is how you undo it.

State Locking: The DynamoDB Pattern Is Deprecated

If you learned Terraform's S3 backend from an older tutorial, it told you to provision a DynamoDB table for state locking. That guidance is now out of date. Per the S3 backend docs, DynamoDB-based locking is explicitly deprecated: "Locking can be enabled via S3 or DynamoDB. However, DynamoDB-based locking is deprecated and will be removed in a future minor version."

The current approach is native S3 locking, enabled with one config key:

terraform {
  backend "s3" {
    bucket       = "my-terraform-state"
    key          = "prod/terraform.tfstate"
    region       = "us-east-1"
    use_lockfile = true   # native S3 locking — no DynamoDB table needed
  }
}

The lock lives alongside your state as a file with a .tflock suffix. If you're still running a DynamoDB table purely for Terraform locking, it's no longer required — use_lockfile = true replaces the dynamodb_table setting, and one less piece of infrastructure to maintain and pay for.

When a Lock Won't Release: force-unlock

A crashed apply, a killed CI job, or a network drop mid-run can leave a lock in place with nothing actually running. Per the force-unlock reference, the command is:

terraform force-unlock [options] LOCK_ID

The important line in HashiCorp's own docs: "This command does not modify your infrastructure." It only removes the lock record — it doesn't touch state content or real resources, which is why it's safe to run once you've confirmed nothing else is actually mid-apply. One real caveat worth knowing before you reach for it: "Local state files cannot be unlocked by another process" — force-unlock is a remote-backend feature. If you're on purely local state, a stuck lock means finding and clearing the lock file yourself, not running this command.

A Practical Recovery Workflow

Put together, here's the actual sequence for the two most common state incidents:

"I ran terraform state rm on the wrong resource."

Don't run terraform apply yet — that's what would actually recreate or destroy something based on the now-wrong state.
Check terraform.tfstate.backup — it has the pre-rm version.
Restore the specific resource block from the backup, or use terraform state push with a corrected copy if you're on a remote backend.

"A lock won't release and nothing is actually running."

Confirm no CI job or teammate has an apply genuinely in progress — check your CI dashboard, not just your own terminal.
Get the lock ID from the error message Terraform printed when the lock conflict occurred.
Run terraform force-unlock LOCK_ID. Remember this doesn't work for purely local state.

Neither of these requires touching real infrastructure. That's the point of how Terraform separates state operations from apply operations — it's worth knowing before the 2am incident, not during it.

For the tool that automates a lot of this multi-environment state and backend setup, our Terragrunt vs Terraform guide covers remote_state blocks that auto-generate backend config like the one above. If you're deciding between engines entirely, our OpenTofu vs Terraform guide and our look at what's new in OpenTofu 1.13 cover the state-encryption feature Terraform doesn't have natively.

Frequently Asked Questions

Q: Does Terraform back up state automatically?
A: Yes. Per HashiCorp's own docs, every state-modifying command writes a backup to terraform.tfstate.backup, and this cannot be disabled.

Q: Do I still need a DynamoDB table for Terraform state locking?
A: No, not for new setups. DynamoDB-based S3 locking is deprecated. Use use_lockfile = true in your S3 backend config for native locking instead.

Q: Does terraform force-unlock destroy anything?
A: No. Per HashiCorp's docs, it explicitly does not modify your infrastructure — it only removes the lock record. It also doesn't work on local (non-remote) state files.

Q: What's the difference between terraform state rm and terraform destroy?
A: state rm only removes a resource from Terraform's tracking — the real infrastructure is untouched. destroy actually deletes the real resource. They're easy to confuse in a panic; state rm is the safe one.

Q: Where is Terraform state stored by default?
A: Locally in terraform.tfstate unless a remote backend (S3, HCP Terraform, etc.) is configured. HashiCorp recommends a remote backend for anything beyond solo experimentation.

Quick Summary:

Terraform state modification commands (state mv, state rm) always write an undeletable backup to terraform.tfstate.backup first
DynamoDB-based S3 state locking is deprecated — use use_lockfile = true for native S3 locking instead
terraform force-unlock LOCK_ID removes a stuck lock without touching infrastructure, but doesn't work on local state
The safe recovery path for a bad state edit is the .tfstate.backup file, not terraform apply
state rm and terraform destroy are not the same thing — only one of them touches real infrastructure

Check terraform state list before any bulk state operation — confirming what's actually tracked takes ten seconds and prevents the kind of terraform state mistake that needs a recovery workflow in the first place.

Terragrunt vs Terraform: What's Actually Different

Amaresh Pelleti — Fri, 09 Oct 2026 12:11:46 +0000

Terragrunt isn't a Terraform competitor. It's a thin wrapper that runs on top of OpenTofu or Terraform, and the confusion about what it actually adds is real enough that "terragrunt vs terraform workspaces" and "terragrunt vs terraform modules" are both real searches. Here's what Terragrunt actually does, verified against its current docs and a real CLI run — including one command-syntax change that will break a tutorial written more than a year ago.

Terragrunt vs Terraform: What Terragrunt Actually Adds

Per Terragrunt's own overview, its job is solving code duplication across multiple environments and regions. A team running the same infrastructure in dev, staging, and prod normally either copies the same .tf files three times or reaches for workspaces. Terragrunt's answer is four features:

DRY configuration. An include block lets every environment inherit shared provider and backend settings from one root config, instead of repeating them.
Remote state generation. A remote_state block auto-generates the backend.tf file before every run, so nobody hand-writes backend config per environment.
Module dependencies. A dependency block lets one unit read another unit's outputs — Terragrunt runs terragrunt output on the dependency first, then feeds those values in.
Dependency-ordered run --all. Terragrunt builds a directed acyclic graph from your dependency blocks and runs commands across every unit in the correct order automatically.

None of this replaces OpenTofu or Terraform. Terragrunt still shells out to one of them for every actual plan/apply — it's an orchestration layer, not a second IaC engine.

The Command That Changed: run-all Is Gone

If you learned Terragrunt from an older tutorial, this will break on you. Terragrunt went through a CLI redesign that folded the old run-all and graph commands into a single run command with flags. We tested this directly on Terragrunt 1.1.6:

$ terragrunt run --all plan --non-interactive
INFO  The following units will be run, starting with dependencies and then their dependents:
.
╰── .
[.] tofu: No changes. Your infrastructure matches the configuration.

❯❯ Run Summary  1 units  564ms
   Succeeded    1

$ terragrunt run-all plan --non-interactive
ERROR unknown command: "run-all". Terragrunt no longer forwards unknown commands by
default. Use 'terragrunt run -- run-all ...' or a supported shortcut.

That's not a deprecation warning — on a current install, the old syntax is a hard error. The fix is mechanical: terragrunt run-all apply becomes terragrunt run --all apply, and every other run-all <command> invocation follows the same pattern.

Terragrunt vs Terraform vs OpenTofu: Which Engine Runs

Terragrunt doesn't force a choice between OpenTofu and Terraform. Per its engine documentation, without the experimental Engine feature enabled, "Terragrunt will determine how IaC updates will be performed by doing things like invoking the tofu/terraform binary directly" — it uses whichever binary it finds. We confirmed this: with only tofu installed and no terraform binary on the test machine, terragrunt init correctly invoked tofu for every step, and Terragrunt's own --help output labels its command shortcuts "OpenTofu shortcuts" first.

If you need to pin a specific engine explicitly rather than relying on autodetection, that's what the experimental engine block is for:

# requires: export TG_EXPERIMENTAL_ENGINE=1
engine {
  source  = "github.com/gruntwork-io/terragrunt-engine-opentofu"
  version = "v0.1.0"
}

For most setups, autodetection is enough — install whichever binary you want (tofu or terraform), and Terragrunt uses it without extra configuration.

A Minimal DRY Setup

This is the actual pattern the DRY features exist for — one root config, environment-specific units that inherit from it:

infra/
├── terragrunt.hcl          # root: shared provider + remote_state config
├── dev/
│   └── terragrunt.hcl      # include { path = find_in_parent_folders() }
├── staging/
│   └── terragrunt.hcl
└── prod/
    └── terragrunt.hcl

Each environment's terragrunt.hcl is a few lines — an include block plus whatever's actually different about that environment (instance size, region, a different dependency output). The provider block, backend config, and remote state setup live once, in the root file.

What Terragrunt Doesn't Solve

Terragrunt adds a real dependency to your toolchain and a second config language layer (HCL generating HCL) on top of the one you already have. Two things worth knowing before adopting it:

It's still Gruntwork's tool. Per its docs, Terragrunt is maintained by Gruntwork, not a vendor-neutral foundation the way OpenTofu is under the Linux Foundation. That's not a red flag — it's a mature, 9.9k-star project with paid enterprise support available — but it's a different governance model than the engine it wraps.
The Engine feature (explicit OpenTofu/Terraform pinning) is still experimental, gated behind TG_EXPERIMENTAL_ENGINE=1. Autodetection covers the common case; don't build a workflow around the pinning syntax expecting it to be stable yet.

If your actual problem is "I have one root module and want per-environment variable values," Terraform/OpenTofu's built-in .tfvars files or workspaces solve that without adding a new tool. Terragrunt earns its place once you have multiple modules with real dependencies between them, or enough environments that copy-pasted backend blocks have become their own maintenance problem.

If you haven't settled the engine question yet, our OpenTofu vs Terraform guide covers licensing and compatibility in more depth — Terragrunt works the same way on top of either. If you're documenting whichever module structure you land on, our terraform-docs GitHub Action guide covers keeping that in sync in CI, and our Terraform Helm provider migration guide covers a real breaking-change migration if Helm is part of your stack.

Frequently Asked Questions

Q: Is Terragrunt a replacement for Terraform or OpenTofu?
A: No. Terragrunt is an orchestration wrapper — it still invokes the tofu or terraform binary for every actual infrastructure change. It doesn't have its own execution engine outside the experimental Engine feature.

Q: Does terragrunt run-all still work?
A: No, not on current versions. We tested it directly on Terragrunt 1.1.6 and got ERROR unknown command: "run-all". Use terragrunt run --all <command> instead — the CLI redesign folded run-all into run as a flag.

Q: Terragrunt vs Terraform workspaces — what's the actual difference?
A: Workspaces let one set of .tf files manage multiple named state instances from the same config. Terragrunt instead uses separate directories (units) per environment, each with its own remote_state block and its own real backend, inheriting shared settings via include. Workspaces are simpler for small setups; Terragrunt's directory-per-environment model scales better once environments need genuinely different configuration, not just different variable values.

Q: Does Terragrunt work with OpenTofu?
A: Yes, and its CLI help labels the primary command shortcuts "OpenTofu shortcuts." Without the experimental Engine feature, Terragrunt auto-detects and invokes whichever of tofu or terraform is on your system PATH.

Q: Do I need Terragrunt if I'm already using OpenTofu?
A: Only if you're managing multiple environments or modules with real dependencies between them. OpenTofu alone is enough for a single module with .tfvars-based environment differences.

Quick Summary:

Terragrunt is an orchestration wrapper, not an alternative to OpenTofu/Terraform — it still shells out to one of them for every actual change
Core features: DRY config via include, auto-generated remote state, dependency blocks for cross-module outputs, and dependency-ordered run --all
terragrunt run-all is gone on current versions — confirmed via direct test, it's now a hard error, not a warning. Use terragrunt run --all <command>
Terragrunt auto-detects tofu or terraform on PATH; explicit engine pinning is still an experimental feature
Worth adopting once you have multiple environments or modules with real dependencies; skip it for a single module that just needs different variable values

Check docs.terragrunt.com before pinning a Terragrunt version in CI — the CLI redesign is still an active migration, and more run-all-style syntax changes could land before it's fully settled.

OpenTofu 1.13: What's New and What Terraform Still Doesn't Have

Amaresh Pelleti — Thu, 08 Oct 2026 12:43:15 +0000

OpenTofu 1.13.0 shipped on September 30, 2026. Officially-supported Windows on ARM64, a set of new functions, and upgrades to the state-encryption system it shipped ahead of Terraform. Here's what actually changed, verified against the official release changelog, plus a close look at state encryption itself — still the single biggest feature gap between OpenTofu and Terraform.

What's Actually New in OpenTofu 1.13

Per the official v1.13 changelog:

Windows ARM64 is now officially supported. Previously unofficial or community-built; OpenTofu now ships official binaries for it.
New convert function for type conversion, plus a set of functions with an assume... prefix that "give OpenTofu additional hints about what's expected as the final result of an unknown value" — useful for reducing unknown-value noise in plan output.
Linting support (experimental), enabled with a -lint flag on supported commands.
Symbol libraries (experimental) — a new capability to define reusable functions and types in hcl-lang based libraries.
Unicode 17 for all string processing.
Plan files now include the provider schemas needed to render the plan, which matters if you consume .tfplan files outside the CLI.

What Breaks in OpenTofu 1.13: Upgrade Notes

Three things worth checking before you upgrade a CI pipeline to 1.13:

The winrm provisioner connection type is removed. If anything in your config still uses WinRM for Windows provisioning, it's gone — OpenSSH is the supported path on modern Windows.
base64gzip output changed. Results are "equivalent to but not equal to" previous output, due to optimized DEFLATE compression — don't diff old and new base64gzip output expecting a byte match.
macOS 13 Ventura is now the minimum. And 1.13 is explicitly called out as the last release series shipping official 32-bit CPU builds.

None of these are exotic — they're the kind of thing that silently breaks a pipeline that hasn't been touched in a year.

The Feature Terraform Still Doesn't Have: State Encryption

OpenTofu's state and plan encryption isn't new to 1.13 — it shipped in an earlier release, and 1.13 extends it — but it remains the clearest structural difference between OpenTofu and Terraform. Per OpenTofu's encryption docs, it's stable, GA functionality, not experimental: "If an attacker were to gain access to your state file, they should not be able to read it and use the sensitive values (e.g. access keys) contained in the state file."

The config lives inside the terraform block:

terraform {
  encryption {
    key_provider "pbkdf2" "mykey" {
      passphrase = var.state_passphrase  # minimum 16 characters
    }
    method "aes_gcm" "mymethod" {
      keys = key_provider.pbkdf2.mykey
    }
    state {
      method = method.aes_gcm.mymethod
    }
    plan {
      method = method.aes_gcm.mymethod
    }
  }
}

Supported key providers go beyond a passphrase: AWS KMS, GCP KMS, Azure Vault, OpenBao's Transit Secret Engine (compatible with HashiCorp Vault 1.14), and a generic external-program option for anything else. 1.13 specifically extended this: GCP KMS gained additional_authenticated_data support, AWS KMS gained an encryption_context field, and OpenBao gained an associated_data argument — all ways to bind the encryption to additional context so a stolen key alone isn't enough to decrypt a state file from a different environment.

Terraform has no equivalent built-in feature. State-at-rest protection in Terraform means encrypting the storage backend itself (S3 bucket encryption, for instance) — the state content isn't encrypted by Terraform's own logic the way OpenTofu's encryption block does it.

Two Quiet But Real Bug Fixes

Two fixes in 1.13 are worth knowing even if they don't affect your specific setup:

connection.script_path escaping now prevents command injection. If a provisioner script path was ever built from a variable, this closes a real injection vector.
SSH proxy connection errors now report properly instead of crashing the run — previously a connection failure could crash rather than surface a readable error.

Should You Upgrade Now?

If you're not using WinRM provisioners and you're on macOS 13+ already, there's no real reason to hold back — the breaking changes are narrow and well-documented. If you're on an older macOS or a 32-bit build target, plan the migration now: 1.13 is the last release series supporting 32-bit builds, and there won't be a grace period on the next major bump.

For a broader look at whether OpenTofu is the right call over Terraform for your team at all — licensing, provider registry compatibility, migration steps — see our OpenTofu vs Terraform guide. And if Terragrunt is part of your stack, it works the same way on top of either engine — see our Terragrunt vs Terraform guide for what it actually adds. If you're keeping module docs in sync as part of that CI pipeline, our terraform-docs GitHub Action guide covers automating that.

Frequently Asked Questions

Q: When did OpenTofu 1.13 release?
A: September 30, 2026, per the official GitHub releases page.

Q: Does OpenTofu 1.13 support Windows on ARM64?
A: Yes, officially for the first time in this release, per the changelog.

Q: Is OpenTofu's state encryption a new feature in 1.13?
A: No — it's an existing, stable feature that 1.13 extended with additional key-provider options (GCP KMS authenticated data, AWS KMS encryption context, OpenBao associated data). The core encryption capability predates this release.

Q: Does Terraform have state encryption like OpenTofu?
A: Not as a built-in feature. Terraform relies on backend-level encryption (like S3 bucket encryption) rather than encrypting state file contents itself the way OpenTofu's encryption block does.

Q: What breaks when upgrading to OpenTofu 1.13?
A: The winrm provisioner connection type is removed, base64gzip output changed (not byte-identical to older output), and the minimum macOS version is now 13 Ventura.

Quick Summary:

OpenTofu 1.13.0 released September 30, 2026: official Windows ARM64 support, new convert/assume... functions, experimental linting and symbol libraries
Breaking on upgrade: WinRM provisioner removed, base64gzip output changed, macOS 13+ required, last release with 32-bit builds
State and plan encryption remains OpenTofu's biggest structural advantage over Terraform — stable, GA, supports AWS/GCP/Azure KMS, OpenBao, and passphrase-based keys
1.13 extended encryption with additional-authenticated-data options for GCP KMS, AWS KMS, and OpenBao
Two security-relevant bug fixes: a connection.script_path injection vector closed, SSH proxy errors now report instead of crashing

Check the official OpenTofu changelog before pinning 1.13 in CI if you're running WinRM provisioners or 32-bit builds — those are the two changes actually capable of breaking a pipeline.

Switching Cloud Accounts: AWS, GCP and Azure Fast

Amaresh Pelleti — Wed, 07 Oct 2026 12:34:06 +0000

Switching cloud accounts across AWS, GCP and Azure is three unrelated problems wearing one trenchcoat. AWS calls them named profiles. GCP calls them configurations. Azure barely gives you a name for it at all — you're switching subscriptions inside one login, or isolating state directories if you actually need two separate identities. Nobody remembers three different mechanisms, so most people end up running aws sts get-caller-identity before every deploy just to check they're not about to hit the wrong account.

This is the actual mechanism behind each one, verified on the CLI, plus a shell function that switches all three at once so you stop needing to remember any of it.

Switching Cloud Accounts on AWS: Named Profiles

AWS profiles, documented here, live in two files: ~/.aws/config (settings) and ~/.aws/credentials (keys), both INI-formatted with [profile name] sections (the default profile skips the profile prefix in config, but not in credentials):

# ~/.aws/config
[default]
region = us-east-1

[profile work]
region = us-east-1
sso_start_url = https://example.awsapps.com/start
sso_region = us-east-1
sso_account_id = 111122223333
sso_role_name = DeveloperAccess

[profile personal]
region = us-west-2

List what's actually configured, and check what a given profile resolves to, before you trust it:

$ aws configure list-profiles
default
work
personal

$ aws configure list --profile personal
NAME       VALUE                    TYPE                    LOCATION
profile    personal                 manual                  --profile
access_key ****************EST2     shared-credentials-file
secret_key ****************EST2     shared-credentials-file
region     us-west-2                config-file             ~/.aws/config

Every AWS CLI command takes --profile name, but typing that on every command gets old fast. Set AWS_PROFILE instead and every subsequent command in that shell uses it:

$ export AWS_PROFILE=personal
$ aws configure list
NAME       VALUE      TYPE  LOCATION
profile    personal   env   AWS_PROFILE

AWS_PROFILE beats whatever's in the config file, which is exactly what you want for a per-terminal override.

⚠️ The SSO gotcha: an SSO-based profile with no cached session fails with Error loading SSO Token: Token for <url> does not exist — not a permissions error, not a typo, just an expired or missing session. Run aws sso login --profile work to fix it, not aws configure.

Switching Cloud Accounts on GCP: Named Configurations

gcloud's configurations are the equivalent — a full bundle of account, project and defaults, switchable by name. :

$ gcloud config configurations create work
Created [work].

$ gcloud config configurations create personal
Created [personal].

$ gcloud config configurations list
NAME      IS_ACTIVE  ACCOUNT  PROJECT  COMPUTE_DEFAULT_ZONE  COMPUTE_DEFAULT_REGION
default   True
personal  False
work      False

Set values inside a specific configuration without activating it first:

$ gcloud config set project work-project-id --configuration=work
Updated property [core/project].

Switch which one is active:

$ gcloud config configurations activate personal
Activated [personal].

And the same environment-variable escape hatch as AWS exists here too — CLOUDSDK_ACTIVE_CONFIG_NAME overrides the active configuration for that shell without touching the persisted state:

$ CLOUDSDK_ACTIVE_CONFIG_NAME=work gcloud config get-value project
Your active configuration is: [work]
work-project-id

This matters for scripts and CI: you can run a one-off command against a different configuration without permanently activating it and without a --configuration flag on every gcloud subcommand that supports one.

Switching Cloud Accounts on Azure: Subscriptions, Not Profiles

Azure doesn't really do named profiles. az login authenticates one identity, and that identity can see every subscription its account has access to. Switching "accounts" day to day usually means switching subscriptions within that one login:

$ az account list --output table
Name          SubscriptionId    State    IsDefault
Work-Prod     11111111-...      Enabled  True
Work-Staging  22222222-...      Enabled  False

$ az account set --subscription "Work-Staging"

That covers most people. But if you genuinely need two separate identities — a work tenant and a personal tenant, logged in simultaneously without one az login clobbering the other — Azure CLI reads its entire state from a directory controlled by AZURE_CONFIG_DIR, documented at learn.microsoft.com (default ~/.azure). Point it somewhere else and you get a fully isolated Azure CLI, tokens and all:

$ AZURE_CONFIG_DIR=~/.azure-work az login
$ AZURE_CONFIG_DIR=~/.azure-personal az login

Each one logs in and caches its session independently. Set AZURE_CONFIG_DIR before any az command and that command runs against that identity's isolated state, the same shape as AWS_PROFILE and CLOUDSDK_ACTIVE_CONFIG_NAME.

The Part Nobody Automates: Switching All Three at Once

Here's the actual fix for "the problem is remembering them": one shell function that sets all three environment variables together, so a single word switches your entire terminal's cloud identity.

# ~/.bashrc or ~/.zshrc
cloudenv() {
  case "$1" in
    work)
      export AWS_PROFILE=work
      export CLOUDSDK_ACTIVE_CONFIG_NAME=work
      export AZURE_CONFIG_DIR="$HOME/.azure-work"
      ;;
    personal)
      export AWS_PROFILE=personal
      export CLOUDSDK_ACTIVE_CONFIG_NAME=personal
      export AZURE_CONFIG_DIR="$HOME/.azure-personal"
      ;;
    *)
      echo "usage: cloudenv work|personal" >&2
      return 1
      ;;
  esac
  echo "cloudenv -> $1"
}

$ cloudenv work
cloudenv -> work
$ cloudenv personal
cloudenv -> personal
$ cloudenv bogus
usage: cloudenv work|personal

Add more cases as you add accounts. This is a plain shell function, not a plugin or a tool to install — it works in bash and zsh as-is, and every AWS, gcloud and az command you run after calling it picks up the right identity automatically, because all three tools read these same environment variables.

Make the Active Account Visible in Your Prompt

The environment-variable approach has one real failure mode: you switch, forget you switched, and run something against the wrong account an hour later. Fix it by putting the active profile in your prompt so it's always visible, not something you have to remember to check:

# ~/.bashrc
PS1='[\${AWS_PROFILE:-none}] \w \$ '

That renders as [work] ~/project $ — updates automatically every time cloudenv changes AWS_PROFILE, no extra step required. Extend the same pattern to show $CLOUDSDK_ACTIVE_CONFIG_NAME if you're switching GCP configs just as often.

What This Doesn't Solve

This pattern switches default identity for a shell. It doesn't stop you from overriding it per-command with --profile or --subscription, and it doesn't replace credential rotation or least-privilege IAM — a cloudenv work that quietly has admin access everywhere is still a cloudenv work you should fix at the IAM layer, not just the shell layer.

If you're also juggling Kubernetes clusters alongside these cloud accounts, that's a separate switching problem with its own tool — see our kubectx guide for the equivalent pattern with kubectl contexts. And if .bashrc itself has become the thing you're afraid to touch, our Linux file permissions guide and finding what's using a port cover two other everyday shell headaches the same way — real commands, real gotchas, not just theory.

Frequently Asked Questions

Q: What's the difference between AWS_PROFILE and --profile?
A: --profile name sets it for one command. export AWS_PROFILE=name sets it for every command in that shell session until you change or unset it. Both read from the same ~/.aws/config and ~/.aws/credentials files.

Q: Does gcloud have something like AWS_PROFILE?
A: Yes — CLOUDSDK_ACTIVE_CONFIG_NAME. Set it as an environment variable and it overrides whichever configuration is currently active via gcloud config configurations activate, without changing the persisted state.

Q: How do I run two Azure identities at once without logging out?
A: Point AZURE_CONFIG_DIR at two different directories and run az login in each. Azure CLI reads its entire state — tokens included — from that directory, so each one stays fully isolated.

Q: Why does my AWS SSO profile fail with a token error?
A: The cached SSO session expired or was never created. Run aws sso login --profile <name> to fix it — aws configure won't help here, since the profile's settings are already correct.

Q: Is a shell function safer than a tool like direnv for this?
A: A manual cloudenv work is explicit — you know exactly when your identity changed. Directory-based auto-switching (direnv, etc.) is convenient but can silently activate the wrong account just by cd-ing into a folder. For anything touching production, explicit beats automatic.

Quick Summary:

AWS: named profiles in ~/.aws/config/credentials, switch with --profile or export AWS_PROFILE=name
GCP: named configurations via gcloud config configurations, override per-shell with CLOUDSDK_ACTIVE_CONFIG_NAME
Azure: subscriptions within one login via az account set, or fully isolated identities via AZURE_CONFIG_DIR
All three read from environment variables, so one shell function (cloudenv) can switch all three together
Put the active profile in PS1 so you see which account you're in instead of having to remember or re-check it

Stop running aws sts get-caller-identity as a paranoia check before every command. Switching cloud accounts shouldn't require remembering three different mechanisms — put the account name in your prompt, and you'll see it before you need to ask.

Kubernetes OOMKilled (Exit Code 137): Causes and Fixes

Amaresh Pelleti — Tue, 06 Oct 2026 11:31:49 +0000

Kubernetes OOMKilled with exit code 137 means the Linux kernel killed your container's process for using more memory than it was allowed. Sometimes that's the container's own memory limit. Sometimes the whole node ran out of memory and your container lost the lottery. The status looks the same, but the fix is completely different, so the first job is working out which one you hit.

This guide shows how to confirm an OOMKilled container, tell a limit kill from a node kill, and fix the common causes, including the JVM default that trips up most Java workloads. If you need a refresher on how requests and limits work, see our Kubernetes resource requests and limits guide, or start from the complete Kubernetes guide.

What Does OOMKilled Exit Code 137 Mean in Kubernetes?

Exit code 137 is 128 + 9. Per the Bash manual, when a process dies from a fatal signal N, the exit status is 128+N. Signal 9 is SIGKILL, which a process can't catch or clean up after.

OOMKilled is the reason Kubernetes records when that SIGKILL came from the kernel's out-of-memory killer. The Kubernetes memory docs show exactly what you'll see in the pod status:

lastState:
  terminated:
    exitCode: 137
    reason: OOMKilled

⚠️ Note: Exit code 137 on its own doesn't prove an OOM kill. Anything that sends SIGKILL produces 137, including a pod that ignored SIGTERM for longer than its grace period (see our Kubernetes graceful shutdown guide). Only trust it as memory-related when the reason says OOMKilled.

How to Confirm a Kubernetes OOMKilled Container

Start with the pod. A container that keeps getting OOMKilled usually shows a climbing restart count and eventually CrashLoopBackOff:

kubectl get pod my-app-7d9f8 -n prod
kubectl describe pod my-app-7d9f8 -n prod

In the describe output, look under the container's Last State. You want Terminated, Reason: OOMKilled, Exit Code: 137. To pull just that field across all containers in the pod:

kubectl get pod my-app-7d9f8 -n prod \
  -o jsonpath='{range .status.containerStatuses[*]}{.name}{"\t"}{.lastState.terminated.reason}{"\t"}{.lastState.terminated.exitCode}{"\n"}{end}'

Then check how much memory the container actually uses against its limit. This needs metrics-server installed in the cluster:

kubectl top pod my-app-7d9f8 -n prod --containers
kubectl get pod my-app-7d9f8 -n prod -o jsonpath='{.spec.containers[*].resources}'

If usage sits close to the limit just before each restart, you've found your answer.

Container Limit Kill vs Node Out-of-Memory Kill

There are two different ways a container ends up OOMKilled, plus one lookalike that isn't an OOM kill at all.

What happened	What triggers it	What you see
Container limit kill	Container uses more than its own `limits.memory`	`OOMKilled`, 137, container restarts
Node OOM kill	The whole node runs out of memory before the kubelet can react	`OOMKilled`, 137, often a container that was under its limit
Eviction (not an OOM kill)	Node memory drops below the kubelet's eviction threshold	Pod status `Evicted`, pod rescheduled elsewhere

Container limit kills are the common case. The kernel enforces limits.memory through the container's cgroup. The resource management docs point out a subtlety: memory limits are enforced reactively. A container can briefly exceed its limit, and the kill happens when the kernel detects memory pressure.

Node OOM kills happen when the node itself runs out of memory before the kubelet can evict anything. The kernel's OOM killer then picks a victim across the whole node, and the kubelet biases that choice by Quality of Service class through oom_score_adj:

QoS class	`oom_score_adj`	Killed
Guaranteed	-997	Last
Burstable	Between 3 and 999, based on the memory request	In between
BestEffort	1000	First

This is why a pod with no memory requests or limits (BestEffort) gets killed first on a busy node, even if it's small. To check the node side, look for kernel OOM events. The Kubernetes docs example shows them as OOMKilling warnings on the node, though whether they appear as events depends on your cluster's node monitoring:

kubectl describe node <node-name> | grep -i -A2 oom
kubectl get events -A --field-selector reason=OOMKilling

Evictions look different. The pod status says Evicted, not OOMKilled, and the pod is replaced rather than restarted in place. If that's what you see, the fix is node capacity and requests, not container limits.

Why Java Containers Get OOMKilled (and Why They Sometimes Don't Use Their Memory)

Java is the most common OOMKilled search for a reason. A modern JVM sizes its heap from the container's memory limit, but by default it only takes a quarter of it. The Java documentation lists the -XX:MaxRAMPercentage default as 25 percent.

That causes two opposite problems:

Heap too small. With a 2 GiB limit, the default max heap is about 512 MiB. The app throws java.lang.OutOfMemoryError while the container still has plenty of memory. That's a JVM error, not an OOMKilled.
Heap too big. Someone sets -Xmx or MaxRAMPercentage=100 to "use all the memory." But the JVM also needs memory outside the heap: metaspace, thread stacks, the JIT code cache, direct buffers. Heap plus everything else exceeds the limit, and the kernel kills the container with 137.

A common middle ground is to leave headroom for non-heap memory:

env:
  - name: JAVA_TOOL_OPTIONS
    value: "-XX:MaxRAMPercentage=75"
resources:
  requests:
    memory: "2Gi"
  limits:
    memory: "2Gi"

Tune the percentage from real measurements. Apps with many threads or large direct buffers need more non-heap headroom than 25 percent.

Other Causes of Kubernetes OOMKilled Errors

Memory-backed emptyDir volumes. If you mount an emptyDir with medium: Memory, files written there count against the container's memory limit. The docs note the kubelet tracks tmpfs emptyDir volumes as container memory use, not local storage. A job that writes a large temp file to a RAM-backed volume gets OOMKilled even though the app itself is small.

A real memory leak. If memory climbs steadily until each kill, raising the limit only delays the next restart. Graph the container's memory over a few restarts. A sawtooth that always climbs to the limit is a leak, and the fix is in the code.

A limit copied from another service. Limits often get set once from a template and never measured. Check actual peak usage under load before assuming the app is misbehaving.

Sidecars. Each container has its own limit. An Envoy or logging sidecar can be the one getting OOMKilled while your app container is fine. The jsonpath command above shows which container it was.

How to Fix a Kubernetes OOMKilled Container

Match the fix to the cause:

Limit too low for real usage: raise limits.memory in the Deployment, based on measured peak usage plus headroom.
Node OOM kills: set memory requests that reflect real usage, so the scheduler doesn't overpack nodes. For critical workloads, set requests equal to limits so the pod is Guaranteed and killed last.
Java: set MaxRAMPercentage deliberately instead of relying on the 25 percent default or -Xmx guesses.
Leak: fix the code. Raising the limit buys time, not a solution.

Raising Memory Without Recreating the Pod

In-place pod resize has been stable since Kubernetes v1.35, according to the resize documentation. You can raise a running container's memory through the pod's resize subresource. It requires kubectl v1.32 or newer.

This changes one running pod only. It is useful for buying time during an incident:

kubectl patch pod my-app-7d9f8 -n prod --subresource resize --patch \
  '{"spec":{"containers":[{"name":"app","resources":{"requests":{"memory":"1Gi"},"limits":{"memory":"1Gi"}}}]}}'

⚠️ Important: For pods managed by a Deployment, this patch is lost the next time the pod is replaced, because new pods come from the Deployment template. Update the Deployment too, or the OOMKilled restarts come back on the next rollout.

Keep a few resize rules in mind. The pod's QoS class can't change through a resize, so a Guaranteed pod must keep requests equal to limits. Lowering a memory limit is best-effort: if current usage is above the new limit, the kubelet skips the resize and leaves it "In Progress." Windows pods don't support in-place resize.

Right-sizing memory also affects your bill. Our Kubernetes cost optimization guide covers finding over-provisioned workloads without starving them.

Frequently Asked Questions

Q: What does exit code 137 mean in Kubernetes?
A: Exit code 137 means the process was killed by signal 9 (SIGKILL), because the exit status for a fatal signal is 128 plus the signal number. When the reason is OOMKilled, the kernel's out-of-memory killer sent it. Without that reason, 137 can also come from a pod that ignored SIGTERM past its grace period.

Q: How do I check why a Kubernetes pod was OOMKilled?
A: Run kubectl describe pod <name> and look at the container's Last State for Reason: OOMKilled and Exit Code: 137. Then compare kubectl top pod <name> --containers with the container's memory limit. If usage was under the limit, check the node for kernel OOM events.

Q: Is OOMKilled the same as a pod being evicted?
A: No. OOMKilled means the kernel killed a container, and the kubelet restarts it in the same pod. Eviction means the kubelet removed the whole pod because the node was short on memory, and the pod status shows Evicted.

Q: Can I increase a pod's memory limit without restarting it?
A: Yes, on Kubernetes v1.35 or later, with kubectl v1.32 or later. Patch the pod's resize subresource. But for Deployment-managed pods, also update the Deployment, or the change is lost when the pod is replaced.

Q: Why does my Java app get OOMKilled when the heap is smaller than the limit?
A: The JVM uses memory outside the heap for metaspace, thread stacks, the code cache and direct buffers. Heap plus non-heap can exceed the container limit. Leave headroom, for example -XX:MaxRAMPercentage=75 instead of 100.

Quick Summary:

Exit code 137 is 128 + 9 (SIGKILL); only the OOMKilled reason ties it to memory
A container limit kill, a node OOM kill and an eviction look similar but need different fixes
On node OOM, BestEffort pods die first (oom_score_adj 1000) and Guaranteed pods last (-997)
The JVM defaults to a max heap of 25 percent of the container limit, and non-heap memory still counts
In-place resize (stable since v1.35) raises memory on a running pod, but update the Deployment too

Next time a pod shows Kubernetes OOMKilled, run kubectl describe pod and kubectl top pod --containers side by side before touching any limits. Those two outputs tell you which of the three cases you're in.

GitHub Actions Workflow Not Triggering? Every Cause and Fix

Amaresh Pelleti — Mon, 05 Oct 2026 12:06:08 +0000

A GitHub Actions workflow not triggering almost never means GitHub is broken. It means one of about a dozen rules quietly filtered your event out: the workflow file is on the wrong branch, a filter didn't match, the commit was pushed with GITHUB_TOKEN, or the schedule got disabled. None of these produce an error. The run simply never appears.

This guide covers every reason for a GitHub Actions workflow not triggering, in the order worth checking. Each one comes with the exact rule from GitHub's docs and the fix. If you're still learning workflow syntax, read how GitHub Actions workflow files are structured first.

GitHub Actions Workflow Not Triggering? Quick Diagnosis

Match your symptom to the likely cause, then jump to that section.

Symptom	Most likely cause
No "Run workflow" button	`workflow_dispatch` file isn't on the default branch
Scheduled workflow never runs	File not on default branch, or auto-disabled after 60 days
Scheduled run is late or missing	Queue delay at the top of the hour
Push from another workflow triggers nothing	Push used `GITHUB_TOKEN`
Runs on branches, never on tags	`branches` filter defined without `tags`
Pushed a change, nothing ran	`paths` filter didn't match, or `[skip ci]` in the message
Fork PR shows "approval required"	First-time contributor approval
Workflow missing from the Actions tab	Invalid YAML, disabled workflow, or Actions turned off

Before anything else, check whether GitHub created a run at all:

# List recent runs for one workflow, including disabled ones
gh run list --workflow ci.yml --all --limit 10

# Filter by the event you expected
gh run list --workflow ci.yml --event push --branch main

If a run exists with status skipped or action_required, the trigger worked. Something later stopped it. But if there's no run at all, a rule filtered the event out first. That distinction cuts the list below in half.

Workflow Not Triggering Because It's Not on the Default Branch

Many events only fire if the workflow file exists on the default branch. That includes workflow_dispatch, schedule, workflow_run, issues, issue_comment, delete, and most other non-push events. The events reference repeats the same note on each: "This event will only trigger a workflow run if the workflow file exists on the default branch."

This catches people constantly. You add a new workflow_dispatch workflow on a feature branch. Then you open the Actions tab, and there's no "Run workflow" button. The workflow is fine. It just isn't on main yet.

Fix: merge the workflow file to the default branch first. After that, you can dispatch it against any branch:

gh workflow run deploy.yml --ref my-feature-branch -f environment=staging

GitHub's docs spell out the order: the button appears once the file is on the default branch, and once the workflow has run at least once, you can dispatch it against any branch or tag via the API or gh.

⚠️ Note: push and pull_request are the exceptions. They run from the workflow file on the branch you pushed, which is why a CI workflow can work on a feature branch while a new manual workflow can't.

Scheduled GitHub Actions Workflow Not Triggering

Scheduled workflows have four separate failure modes, and none of them show an error.

1. The file isn't on the default branch. Scheduled workflows only run on the default branch, using its latest commit. A cron on any other branch never fires.

2. It was disabled after 60 days. In a public repository, GitHub automatically disables scheduled workflows when there's been no repository activity for 60 days. Forks of public repos also start with scheduled workflows disabled. Re-enable it:

gh workflow list --all          # disabled workflows are hidden without --all
gh workflow enable nightly.yml

3. It ran, just late. The docs are blunt about this: schedule events "can be delayed during periods of high loads," the start of every hour is a high-load time, and under enough load "some queued jobs may be dropped." Everyone writes 0 * * * *. Don't.

4. The time is off by hours. Cron runs in UTC unless you say otherwise. GitHub now supports an IANA timezone per schedule, so you can stop converting by hand:

on:
  schedule:
    - cron: '17 5 * * 1-5'   # 5:17 AM, weekdays, off the top of the hour
      timezone: "America/New_York"

With a daylight-saving timezone, a run that falls in the skipped spring-forward hour moves to the next valid time. For example, a 2:30 AM schedule runs at 3:00 AM that day. The shortest interval GitHub allows is every 5 minutes.

Why Does a Push From Another Workflow Not Trigger Anything?

Events created with the repository's GITHUB_TOKEN don't start new workflow runs. GitHub does this on purpose to prevent infinite loops. Otherwise, a workflow that pushes a commit would trigger itself forever. The GITHUB_TOKEN docs list the exceptions.

So say your release workflow commits a version bump and pushes it with GITHUB_TOKEN. Your on: push workflow won't run for that commit. There are two exceptions:

workflow_dispatch and repository_dispatch always create runs, even when sent with GITHUB_TOKEN.
When a workflow opens or updates a pull request with GITHUB_TOKEN, the opened, synchronize and reopened events create runs in an approval-required state. Someone with write access has to click "Approve workflows to run."

Fix: trigger the next workflow explicitly with workflow_dispatch, or use a GitHub App token or a fine-grained personal access token for the push. The App token is the better choice for shared repos because it isn't tied to a person's account.

Branch, Tag and Path Filters That Silently Skip Runs

Filters are the biggest single reason for a workflow not triggering. Three rules from the workflow syntax reference cause most of it.

Defining branches alone disables tags. If you define only branches/branches-ignore, the workflow won't run for tag pushes, and vice versa. Define neither and it runs for both. So this workflow never runs on v1.2.0:

on:
  push:
    branches: [main]
    # add this to also run on release tags:
    tags: ['v*']

branches and paths must both match. When you set both, the workflow runs only when a push matches the branch and changes a matching file. A docs-only commit to main with paths: ['src/**'] does nothing.

Path filters have diff limits. GitHub builds the changed-file list with a two-dot diff for pushes and a three-dot diff for pull requests. So if the diff has more than 3,000 files and your matches aren't in the first 3,000, the workflow won't run. On the other hand, a push with more than 1,000 commits always runs. Large merges and monorepo refactors hit both limits.

⚠️ Important: A workflow skipped by a branch or path filter leaves its required checks stuck in Pending, which blocks the PR from merging. If you require a check that's path-filtered, the usual fix is a small always-running job that reports success when the real job is skipped.

Also note that GitHub won't create push events for tags when more than three tags are pushed at once. Pushing a batch of tags with git push --tags can trigger nothing.

Skip Keywords in the Commit Message

Any of these in the commit message (or the HEAD commit of a PR) stops push and pull_request workflows:

[skip ci]
[ci skip]
[no ci]
[skip actions]
[actions skip]

A skip-checks: true trailer does the same. This bites teams whose release tooling adds [skip ci] to automated commits. Then someone squash-merges a PR, and the combined message still contains it.

Skip keywords only apply to push and pull_request. They don't stop pull_request_target, schedule or workflow_dispatch.

Pull Requests From Forks Waiting for Approval

When a first-time contributor opens a pull request on a public repository, a maintainer with write access may need to approve the workflow run before it starts. The run exists, with status action_required, but nothing executes until someone approves it on the PR page.

Separately, fork PRs get no secrets except GITHUB_TOKEN, and that token is read-only. That doesn't stop the workflow from triggering, but a deploy step that needs a secret will fail, which is easy to mistake for a trigger problem. Our GitHub Actions security guide covers why pull_request_target is risky as a workaround.

workflow_run Chains That Stop Partway

workflow_run lets one workflow start after another finishes. Two limits cause silent gaps:

It only fires if the listening workflow is on the default branch (same rule as above).
You can't chain more than three levels. In GitHub's example, A → B → C → D → E → F runs A through D, and E and F never run.

Also, the requested activity type doesn't fire on re-runs. If you re-run the upstream workflow, a downstream workflow filtered to types: [requested] won't start again. Use completed for most chains.

The Workflow Doesn't Show Up at All

If the workflow is missing from the Actions tab entirely, check these:

Location and extension. The file must be in .github/workflows/ with a .yml or .yaml extension.
Invalid YAML. A syntax error means GitHub can't parse the triggers. The Actions tab shows the error on the workflow page after you push. Catch it earlier by running actionlint locally.
The workflow is disabled. gh workflow list hides disabled workflows. Use gh workflow list --all.
Actions is disabled for the repo or org. Check Settings → Actions → General. If you see "GitHub Actions is currently disabled for this repository," GitHub's docs note that changing these settings may not restore access. The account can be in a GitHub-controlled disabled state, which needs support.

GitHub Actions Workflow Not Triggering: The Full Checklist

When you have a GitHub Actions workflow not triggering, work through this in order:

Does gh run list --workflow <file> --all show a run? If yes, check its status instead.
Is the workflow file on the default branch? Required for schedule, workflow_dispatch, and workflow_run.
Is the workflow enabled? gh workflow list --all.
Do branches, tags and paths all match what you pushed?
Did the commit message contain a skip keyword?
Was the event created with GITHUB_TOKEN?
Is it a fork PR waiting for approval?
For schedules: UTC vs your timezone, top-of-the-hour delays, 60-day auto-disable.

Once it runs, make it a required check so it can't be skipped silently. Our unit testing with GitHub Actions guide shows the setup, and the complete Git and GitHub Actions guide ties the whole workflow together.

Frequently Asked Questions

Q: Why doesn't my workflow_dispatch workflow show a Run workflow button?
A: The workflow file isn't on the default branch yet. GitHub only shows the "Run workflow" button when the file exists on the default branch. Merge it to main, then you can dispatch it against any branch with gh workflow run <file> --ref <branch>.

Q: Why is my GitHub Actions cron schedule not running?
A: Scheduled workflows only run from the default branch, run in UTC unless you set a timezone, can be delayed or dropped at the top of the hour, and are auto-disabled in public repos after 60 days without activity. Check gh workflow list --all to see if it's disabled.

Q: Why doesn't a commit pushed by a workflow trigger other workflows?
A: Events created with the repository's GITHUB_TOKEN don't start new workflow runs, to prevent recursive loops. Use workflow_dispatch or repository_dispatch to chain workflows, or push with a GitHub App token instead.

Q: Why does my workflow run on branches but not on tags?
A: If you define only a branches filter under push, GitHub won't run the workflow for tag pushes. Add a tags filter, such as tags: ['v*'], to cover both.

Q: Why are my required checks stuck in Pending?
A: A workflow skipped by a path filter, branch filter or skip keyword leaves its checks pending, which blocks merging. Add a job that always runs and reports success, or avoid path-filtering required checks.

Quick Summary:

schedule, workflow_dispatch and workflow_run only fire when the workflow file is on the default branch
Pushes made with GITHUB_TOKEN don't start new runs, except workflow_dispatch and repository_dispatch
A branches-only filter silently ignores tag pushes, and branches + paths must both match
Public-repo schedules auto-disable after 60 days of inactivity and can be delayed at the top of the hour
Scheduled workflows accept an IANA timezone, so you no longer have to convert cron times to UTC by hand

Next time you're stuck with a GitHub Actions workflow not triggering, start with gh run list --workflow <file> --all. Whether a run exists tells you which half of this list to check.

Ollama vs vLLM vs llama.cpp: Which Local LLM Engine?

Amaresh Pelleti — Thu, 01 Oct 2026 12:20:51 +0000

Ollama vs vLLM vs llama.cpp comes down to one question: how many people will hit the model at once? For one developer on a laptop, pick Ollama or llama.cpp. For a GPU server with many concurrent users, pick vLLM. Below, I explain why. I also benchmarked Ollama and llama.cpp on the same model file, and the defaults changed the result more than the engine did.

If you haven't installed any of these yet, start with our LM Studio and Ollama setup guide.

Ollama vs vLLM vs llama.cpp at a Glance

	Ollama	llama.cpp	vLLM
What it is	Model manager and local server	C/C++ inference engine, CLI and server	High-throughput serving engine
Best at	Easiest local setup	Running anywhere, fine control	Many concurrent requests on GPUs
Model format	Its own library, or import GGUF / Safetensors	GGUF	Hugging Face weights; GGUF "highly experimental" via a plugin
Hardware	macOS, Windows, Linux; CPU, NVIDIA, AMD, Apple Silicon	CPU, Metal, CUDA, HIP, Vulkan, SYCL and more	NVIDIA, AMD and Intel GPUs, TPUs, CPUs; experimental on Apple Silicon
API	Native API plus OpenAI-compatible	OpenAI-compatible (`llama serve`)	OpenAI-compatible, Anthropic Messages API, gRPC
Concurrency default	1 request per model	Parallel slots, auto-sized	Continuous batching
Setup effort	Lowest	Low to medium	Highest

What Each Engine Actually Is

llama.cpp is the engine underneath much of the local-LLM world. It's a plain C/C++ implementation with no dependencies. Per its README, it targets "a wide range of hardware." It runs GGUF files and supports 1.5-bit to 8-bit quantization. As of v0.5.0 it ships a single llama command, so the old llama-server workflow is now llama serve:

# Chat with a model straight from Hugging Face
llama cli -hf ggml-org/Qwen3-4B-GGUF

# OpenAI-compatible server on port 8080
llama serve -m ./Qwen3-4B-Q4_K_M.gguf --port 8080

Ollama wraps local inference in a model manager: ollama pull, ollama run, a model library, and a background service. It exposes both its own API and an OpenAI-compatible one on port 11434. It trades some control for convenience, which is why most people start here. Our Ollama API guide covers the endpoints.

vLLM is a serving engine built for throughput. Its headline features are PagedAttention for KV-cache memory, continuous batching, chunked prefill and prefix caching. It also splits models across GPUs with tensor, pipeline and expert parallelism. So it's what you run behind an internal API that dozens of users or agents call at once.

uv pip install vllm
vllm serve Qwen/Qwen3-4B

Ollama vs llama.cpp: A Real Benchmark on the Same Model File

To compare Ollama and llama.cpp fairly, I loaded the exact same GGUF file into both: Qwen3-4B at Q4_K_M, 2.5 GB. The machine was an Apple M1 with 16 GB of RAM. I ran Ollama 0.34.4 and llama.cpp 0.5.0 with default settings. Each engine generated 256 tokens for the same prompt. First I sent one request at a time, then four at once.

Setup	Single request	4 concurrent (aggregate)	Wall time for 4 requests
Ollama, default (`OLLAMA_NUM_PARALLEL=1`)	18.7 tok/s	18.9 tok/s	54 s
llama.cpp `llama serve`, default (4 slots)	13.7 tok/s	28.9 tok/s	35 s
Ollama with `OLLAMA_NUM_PARALLEL=4`	17.8 tok/s	35.0 tok/s	29 s

Three things stand out.

For one user, Ollama was faster. It generated about 18.7 tokens per second against 13.7 for llama.cpp on the same file. The startup logs show different defaults: Ollama picked a 4,096-token context from available VRAM, while llama serve auto-configured 4 slots with a 40,960-token context each. Same weights, different setup.

Under load, Ollama's default queues. With four requests at once, default Ollama served them one after another. Aggregate throughput stayed at about 19 tokens per second, and the last user waited nearly a minute. llama.cpp's parallel slots split the work, so each request ran slower (8 tokens per second) but all four finished in 35 seconds.

One setting flips the result. With OLLAMA_NUM_PARALLEL=4, Ollama handled the same four requests in 29 seconds at 35 tokens per second aggregate, the best result in the test.

I also threw away an earlier run. It showed llama.cpp far ahead, but the Mac was busy (load average around 9), and the numbers didn't hold up on a rerun. Benchmark on an idle machine, and run each test more than once.

⚠️ Note: These are small-model numbers on one laptop. They show the shape of the difference, not a universal ranking. Run the same test on your own hardware before choosing.

Ollama vs vLLM vs llama.cpp Under Concurrent Load

For a single user, the gap between engines comes mostly from configuration: context size, slot count and quantization. Model size and memory bandwidth set the ceiling, and Ollama and llama.cpp share much of the same low-level code.

Concurrency is where they split:

Ollama processes one request per model at a time by default. Its FAQ lists OLLAMA_NUM_PARALLEL with a default of 1, and notes that memory use scales with OLLAMA_NUM_PARALLEL × OLLAMA_CONTEXT_LENGTH. Extra requests wait in a queue, and past OLLAMA_MAX_QUEUE the server returns a 503.
llama.cpp runs several parallel "slots" in llama serve. The -np / --parallel flag defaults to auto.
vLLM batches requests continuously, adding new ones to the running batch as others finish. Combined with PagedAttention, this is what lets one GPU serve many users without the queue growing.

If Ollama is your backend and several people or agents share it, raise the parallel setting before blaming the model:

OLLAMA_NUM_PARALLEL=4 ollama serve

Each parallel slot needs its own context memory, so watch RAM or VRAM when you raise it.

The vLLM Memory Default That Surprises People

vLLM claims most of the GPU as soon as it starts. The --gpu-memory-utilization flag defaults to 0.92, meaning vLLM takes 92 percent of GPU memory for the model and its KV cache, per the engine arguments reference. That's intentional: the spare memory becomes KV cache, which is what makes high concurrency possible.

It also means a second vLLM instance, or anything else on that GPU, fails to allocate. Run two instances on one GPU and you have to split it yourself:

vllm serve Qwen/Qwen3-4B --gpu-memory-utilization 0.45 --port 8000
vllm serve Qwen/Qwen3-1.7B --gpu-memory-utilization 0.45 --port 8001

Can vLLM Run GGUF Models or Run on a Mac?

Mostly no, for now. vLLM's GGUF support is labelled "highly experimental and under-optimized" in its own docs, and it now needs a separate vllm-gguf-plugin package. If your models are GGUF files, llama.cpp or Ollama is the natural fit.

On Macs, vLLM's installation docs describe experimental support for Apple Silicon that must be built from source and runs on the CPU. In practice, Apple Silicon users should pick Ollama or llama.cpp, both of which use Metal on the GPU.

Ollama vs vLLM vs llama.cpp: Which One Should You Use?

One developer, local machine, just want it working: Ollama. ollama run is the fastest path from nothing to a working model, and the hardware requirements guide tells you what fits.
Maximum control, unusual hardware, or embedding inference in your own tooling: llama.cpp. You pick the exact quantization, the context size, the slot count and the backend.
A shared endpoint for a team or an application with real traffic, on NVIDIA or AMD GPUs: vLLM. Its batching pays off as soon as requests overlap.
Small team sharing one Ollama box: Ollama can work, but set OLLAMA_NUM_PARALLEL above 1 and test under load first.

A common path is to prototype with Ollama, then move the same model to vLLM when it becomes a service. All three expose an OpenAI-compatible API, so switching usually means changing a base URL and model name, not rewriting code.

Frequently Asked Questions

Q: Is Ollama built on llama.cpp?
A: Ollama runs GGUF models using the ggml/llama.cpp stack, and it has also added its own engine work, including MLX on Apple Silicon. In our benchmark on the same GGUF file, the differences came from defaults (context size and parallel slots), not from the engine itself.

Q: Is vLLM faster than Ollama?
A: For many concurrent requests on a GPU, usually yes, because vLLM batches requests continuously and manages KV-cache memory with PagedAttention. For a single user on a laptop, the difference is small, and vLLM is much harder to run on a Mac.

Q: Can I use vLLM with GGUF files?
A: Only experimentally. vLLM's docs describe GGUF support as highly experimental and under-optimized, and it requires the separate vllm-gguf-plugin package. For GGUF models, use llama.cpp or Ollama.

Q: Why does Ollama slow down with multiple users?
A: By default Ollama processes one request per model at a time (OLLAMA_NUM_PARALLEL defaults to 1), so extra requests queue. Raise OLLAMA_NUM_PARALLEL, keeping in mind that each parallel slot needs its own context memory.

Quick Summary:

Ollama vs vLLM vs llama.cpp is mostly a concurrency decision, not a single-request speed decision
On the same GGUF file, default Ollama won for one user (18.7 vs 13.7 tok/s), llama.cpp won under 4 concurrent requests, and OLLAMA_NUM_PARALLEL=4 beat both
Ollama defaults to OLLAMA_NUM_PARALLEL=1; llama.cpp auto-sizes parallel slots; vLLM batches continuously
vLLM takes 92 percent of GPU memory by default and treats GGUF and Apple Silicon as experimental
All three expose an OpenAI-compatible API, so moving between them is mostly a config change

If you're choosing between Ollama vs vLLM vs llama.cpp today, start with Ollama on your laptop, measure with the concurrency you actually expect, and move to vLLM when the queue becomes the bottleneck.

PostgreSQL 19: What's Actually Shipping, What Got Cut

Amaresh Pelleti — Wed, 30 Sep 2026 13:02:53 +0000

PostgreSQL 19 is late. The project released Beta 4 on September 24, 2026, reverted several headline features along the way, and now plans a release candidate in early October, with general availability possibly also in October. If you planned around SQL/PGQ graph queries, GROUP BY ALL or partition merge/split, those are out. REPACK, pg_plan_advice and parallel autovacuum are still in.

Here's what changed. It's taken from the PostgreSQL project's own Beta 4 announcement and release notes, plus Snowflake's engineering writeup of the revert cycle.

Where the Release Stands Now

The project normally ships a major version every fall, usually in late September. This year, the Beta 4 announcement says the next step is a release candidate "in early October," and that GA "may also occur in October." The announcement explains the delay directly: the community "chose to remove some features from the PostgreSQL 19 release" so the release stays reliable and close to schedule.

Snowflake's post, written a week before Beta 4, counted 53 reverts since beta began, compared with about 44 for PostgreSQL 18. The difference this year is timing. Many reverts landed late and hit headline features, several after in-depth reviews that used AI tools to find bugs and build reproducible test cases.

What PostgreSQL 19 Reverted

The Beta 4 announcement lists these reverts since Beta 3:

SQL/PGQ (property graph queries)
Online enabling and disabling of data checksums
Temporal UPDATE/DELETE ... FOR PORTION OF
ALTER TABLE ... MERGE PARTITIONS and SPLIT PARTITIONS
The new pg_get_role_ddl(), pg_get_tablespace_ddl() and pg_get_database_ddl() functions (removed)

Earlier in the cycle, per Snowflake's writeup, the project also reverted:

GROUP BY ALL: a post-commit review found it missed special handling of ORDER BY entries with nondefault equality semantics, which produced wrong results. It is absent from the current release notes.
Non-text output formats for pg_dumpall
JSON_TABLE ON ERROR cascading to columns
Fast defaults for domains with nonvolatile constraints
Logical replication database-specific snapshots
Nested query tracking in pg_stat_statements

Snowflake notes that property graphs, GROUP BY ALL and partition merge/split are now PostgreSQL 20 material.

⚠️ Note on lz4: Snowflake's post lists the default TOAST compression change to lz4 as reverted, citing missing buildfarm support. But the official release notes, dated September 14, still list "Change the default TOAST compression method from pglz to the more efficient lz4." The two sources disagree, so check the final release notes before relying on the new default. You can always set default_toast_compression yourself.

What Is Still Shipping in PostgreSQL 19

These features are in the release notes and have survived the revert cycle so far:

Feature	What it does
`REPACK`	Reclaims disk space and reorganizes a table, combining `VACUUM FULL` and `CLUSTER`, with a concurrent option
`pg_plan_advice`	New extension for stabilizing and controlling query planner decisions
Parallel autovacuum	Autovacuum can use parallel workers, set with `autovacuum_max_parallel_workers` and a per-table `autovacuum_parallel_workers`
`INSERT ... ON CONFLICT DO SELECT ... RETURNING`	Returns conflicting rows instead of doing nothing
`IGNORE NULLS` / `RESPECT NULLS`	For `lead()`, `lag()`, `first_value()`, `last_value()` and `nth_value()`

Snowflake's post, published before Beta 4, flagged REPACK, the fast-path foreign key checks and postgres_fdw statistics import as at risk. Beta 4 then shipped "several fixes for the new REPACK command" and "several fixes to the performance improvement feature for the foreign key constraint checks," which suggests both are still in. Nothing is final until GA, though.

Why It Slipped: More Bugs Found Earlier

Snowflake's writeup points to AI-assisted review as a real factor: more people are using AI tools to find bugs and produce reproducible test cases, and the fixes are large enough that pushing features to a later release made sense. It gives security patching as a parallel example: PostgreSQL used to average a couple of CVEs per release, while the August 2026 patch release for PostgreSQL 18 had 28. The same post notes the project had no official AI contribution policy at the time of writing.

The takeaway isn't that the release is in trouble. The process is working as designed: features that weren't ready came out, and the reasons are public on the hackers mailing list. If you're troubleshooting a current Postgres deployment in the meantime, our PostgreSQL troubleshooting guide covers the common problems.

What to Do Before GA

Stay on PostgreSQL 18 for production. Snowflake's post calls it the safe upgrade target today, and 19 isn't at release candidate yet.
Grep your code for reverted features. Look for GROUP BY ALL, GRAPH_TABLE / property graph syntax, FOR PORTION OF, and MERGE PARTITIONS / SPLIT PARTITION in migrations, ORM-generated SQL and internal tools.
Test the features you plan to use on the beta. REPACK and the foreign key changes were still getting fixes in Beta 4, so test them against your own workload.
Check the final release notes at GA before relying on anything flagged above, especially the lz4 default.

For partitioning without the reverted merge/split commands, our PostgreSQL partitioning guide covers what works today. Also, if you run Postgres on Kubernetes, see PostgreSQL on Kubernetes with CloudNativePG for how major-version upgrades are handled there.

Frequently Asked Questions

Q: When will PostgreSQL 19 be released?
A: The project's Beta 4 announcement (September 24, 2026) plans a release candidate in early October and says general availability may also occur in October. That's later than the usual late-September release.

Q: Was GROUP BY ALL removed from PostgreSQL 19?
A: Yes. It was reverted after a post-commit review found it mishandled ORDER BY entries with nondefault equality semantics, producing wrong results. It doesn't appear in the current PostgreSQL 19 release notes.

Q: Is SQL/PGQ graph query support in PostgreSQL 19?
A: No. The Beta 4 announcement lists "Revert SQL/PGQ (property graph query) support." Snowflake's writeup describes it as PostgreSQL 20 material.

Q: Is REPACK still in PostgreSQL 19?
A: As of Beta 4, yes. It's in the release notes, and Beta 4 shipped several fixes for it. It was flagged as at risk earlier, so confirm in the final release notes.

Q: Should I upgrade to PostgreSQL 19 now?
A: Not in production. It's still in beta. Use PostgreSQL 18 for production and test your workloads on the PostgreSQL 19 beta.

Quick Summary:

PostgreSQL 19 Beta 4 shipped September 24, 2026; a release candidate is planned for early October and GA may follow in October
Reverted: SQL/PGQ, GROUP BY ALL, partition merge/split, FOR PORTION OF, online checksum toggling, the new pg_get_*_ddl() functions and several smaller items
Still shipping: REPACK, pg_plan_advice, parallel autovacuum, ON CONFLICT DO SELECT, IGNORE NULLS in window functions
The lz4 TOAST default is unclear: Snowflake reports it reverted, the September 14 release notes still list it
Stay on PostgreSQL 18 for production and grep your code for the reverted syntax now

Check the PostgreSQL 19 release notes again at GA before you finalize a migration plan. They are the source that settles every open question above.