<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: juan barraza</title>
    <description>The latest articles on DEV Community by juan barraza (@juan_barraza_dde18726622b).</description>
    <link>https://dev.to/juan_barraza_dde18726622b</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4244790%2F48a0e5b8-7632-42d8-9d75-da9b0ce3134a.png</url>
      <title>DEV Community: juan barraza</title>
      <link>https://dev.to/juan_barraza_dde18726622b</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/juan_barraza_dde18726622b"/>
    <language>en</language>
    <item>
      <title>From Vanilla Kubernetes to OpenShift: What Actually Breaks</title>
      <dc:creator>juan barraza</dc:creator>
      <pubDate>Sun, 15 Nov 2026 12:37:29 +0000</pubDate>
      <link>https://dev.to/juan_barraza_dde18726622b/from-vanilla-kubernetes-to-openshift-what-actually-breaks-3ofe</link>
      <guid>https://dev.to/juan_barraza_dde18726622b/from-vanilla-kubernetes-to-openshift-what-actually-breaks-3ofe</guid>
      <description>&lt;p&gt;OpenShift is Kubernetes — same API, same pods, same kubectl mostly works. And yet, workloads that run perfectly on vanilla Kubernetes or managed services like EKS/GKE routinely fail on OpenShift, and the error messages rarely point at the real cause. I move workloads between both worlds in telecom environments, and the failure points are always the same short list.&lt;/p&gt;

&lt;p&gt;This is the migration guide I'd give any team before their first deployment to OpenShift.&lt;/p&gt;

&lt;h2&gt;
  
  
  The big difference: OpenShift doesn't trust your container
&lt;/h2&gt;

&lt;p&gt;Vanilla Kubernetes runs containers as whatever user the image declares — usually root. OpenShift runs them with a &lt;strong&gt;random, high-numbered UID&lt;/strong&gt; enforced by Security Context Constraints (SCCs). This is the root of almost every "works everywhere but OpenShift" failure.&lt;/p&gt;

&lt;p&gt;Your app tries to write to a directory owned by root, or bind to a privileged port, or execute a file it doesn't own — and dies with a permission error that looks like a bug in your code.&lt;/p&gt;

&lt;h3&gt;
  
  
  The fix, in your Dockerfile
&lt;/h3&gt;

&lt;p&gt;Make every writable path owned by the root group (GID 0) and group-writable:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight docker"&gt;&lt;code&gt;&lt;span class="k"&gt;RUN&lt;/span&gt; chgrp -R 0 /app &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; chmod -R &lt;span class="nv"&gt;g&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;u /app
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;g=u&lt;/code&gt; gives the group the same permissions as the user. OpenShift runs the container with an arbitrary UID that is always a member of the root group (GID 0), so your app can now write wherever the image user could. Never use &lt;code&gt;chown 1000:1000&lt;/code&gt; — the runtime UID is random and you can't predict it.&lt;/p&gt;

&lt;p&gt;Also: set a numeric non-root USER in your Dockerfile (e.g. USER 1001). OpenShift will still override it with its own UID, but it proves the image doesn't need root. And don't rely on writing outside your app directory.&lt;/p&gt;

&lt;h2&gt;
  
  
  Routes instead of Ingress resources
&lt;/h2&gt;

&lt;p&gt;On vanilla Kubernetes you expose services with an Ingress resource plus an ingress controller. OpenShift uses &lt;strong&gt;Routes&lt;/strong&gt;, built on its HAProxy router:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;oc expose service my-service --hostname myapp.example.com
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;or in YAML:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;apiVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;route.openshift.io/v1&lt;/span&gt;
&lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Route&lt;/span&gt;
&lt;span class="na"&gt;metadata&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;my-app&lt;/span&gt;
&lt;span class="na"&gt;spec&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;to&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;kind&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Service&lt;/span&gt;
    &lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;my-service&lt;/span&gt;
  &lt;span class="na"&gt;tls&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;termination&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s"&gt;edge&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Good news: An edge Route without its own certificate uses the router's default wildcard certificate — one less cert-manager headache for internal apps. If you must keep standard Ingress manifests, OpenShift supports them, but Routes are the native path — and the one your cluster admin expects.&lt;/p&gt;

&lt;h2&gt;
  
  
  API objects that don't exist on vanilla Kubernetes
&lt;/h2&gt;

&lt;p&gt;OpenShift-specific resources you'll meet immediately:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Routes&lt;/strong&gt; (shown above)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Projects&lt;/strong&gt; — namespaces with annotations and quotas; &lt;code&gt;oc new-project&lt;/code&gt; is the normal entry point&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;DeploymentConfigs&lt;/strong&gt; — the legacy OpenShift deployment object, deprecated since OpenShift 4.14. Prefer standard Deployments unless you need triggers; DCs exist mostly for backward compatibility&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;ImageStreams&lt;/strong&gt; — a layer over container registries that tracks tags and enables rollbacks. You can ignore them and reference images directly, but builds (&lt;code&gt;oc start-build&lt;/code&gt;) produce ImageStreams&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The kubectl/oc reality
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;kubectl&lt;/code&gt; works against OpenShift for standard resources. You'll want &lt;code&gt;oc&lt;/code&gt; for the rest:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;oc login https://api.mycluster.example.com:6443
oc project my-namespace
oc get routes
oc adm policy who-can delete pods -n my-namespace
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;oc adm policy&lt;/code&gt; subcommands matter more than people expect: the default project template often grants developers edit on their own project but nothing on others, and SCC permissions (&lt;code&gt;oc adm policy add-scc-to-user&lt;/code&gt;) are how you handle the rare workload that genuinely needs elevated privileges.&lt;/p&gt;

&lt;h2&gt;
  
  
  The pre-flight checklist
&lt;/h2&gt;

&lt;p&gt;Before migrating a workload, run this against your manifests:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Does the image run as non-root with random UID?&lt;/strong&gt; Fix with &lt;code&gt;chgrp 0&lt;/code&gt; + &lt;code&gt;g=u&lt;/code&gt; on every writable path.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Any hostPath, privileged, or runAsUser: 0?&lt;/strong&gt; These require a custom SCC. Ask whether they're truly needed — most of the time they're legacy assumptions.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Any privileged ports (&amp;lt;1024)?&lt;/strong&gt; Move to a higher port and let the Route/Service handle exposure.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Ingress manifests?&lt;/strong&gt; Plan the Route equivalents.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Helm charts?&lt;/strong&gt; Look for hardcoded runAsUser or fsGroup values — they often fall outside the project's allowed UID range and get rejected.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Storage classes and CSI?&lt;/strong&gt; Standard, but confirm the cluster's default StorageClass — OpenShift clusters often ship with their own.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Any Docker socket or host access?&lt;/strong&gt; Will be rejected by the restricted SCC without an explicit exception.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Summary
&lt;/h2&gt;

&lt;p&gt;OpenShift breaks vanilla Kubernetes workloads for one main reason: it runs containers with an arbitrary UID under Security Context Constraints. Fix your Dockerfiles (&lt;code&gt;chgrp -R 0&lt;/code&gt;, &lt;code&gt;chmod -R g=u&lt;/code&gt;, no root), replace Ingress with Routes, learn the handful of OpenShift-specific objects, and run the pre-flight checklist before the first deploy. The migration itself is much faster — when you know the list of things that break.&lt;/p&gt;

&lt;p&gt;One last piece of advice: rehearse the migration first with a dry run in a test cluster or namespace. Don't go straight to production. Test first, and please, never on a Friday.&lt;/p&gt;

</description>
      <category>openshift</category>
      <category>kubernetes</category>
      <category>devops</category>
      <category>containers</category>
    </item>
    <item>
      <title>Kafka Consumer Lag Is an Operational Signal, Not Just a Metric</title>
      <dc:creator>juan barraza</dc:creator>
      <pubDate>Sat, 14 Nov 2026 17:39:40 +0000</pubDate>
      <link>https://dev.to/juan_barraza_dde18726622b/kafka-consumer-lag-is-an-operational-signal-not-just-a-metric-5dmc</link>
      <guid>https://dev.to/juan_barraza_dde18726622b/kafka-consumer-lag-is-an-operational-signal-not-just-a-metric-5dmc</guid>
      <description>&lt;p&gt;In every event-driven system I've operated, there's one metric that predicts user-visible trouble before anything else: consumer lag. Not CPU, not memory, not pod count — lag. If your consumers can't keep pace with the topic, everything downstream of that offset is silently falling behind while your dashboards stay green.&lt;br&gt;
Remember why this matters: the front end of your system serves real requests and signaling messages. When lag grows, those answers arrive late — or don't arrive at all — and the impact lands directly on end subscribers.&lt;/p&gt;

&lt;p&gt;This is the guide I wish someone had given me the first time I had to explain at 2 AM why notifications were "sent" but nobody received them.&lt;/p&gt;
&lt;h2&gt;
  
  
  What consumer lag actually is
&lt;/h2&gt;

&lt;p&gt;Lag is the distance between the end of the log and the last offset your consumer group has committed:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;lag = log end offset − committed offset
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Per partition, per consumer group. It is not one number — it's a vector. A group can be perfectly caught up on 15 partitions and hopelessly behind on one, and the average will hide it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Measuring it: three levels of effort
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Level 1 — The CLI, right now, from anywhere:&lt;/strong&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kafka-consumer-groups.sh \
  --bootstrap-server &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;broker:9092&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; \
  --group &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;group-id&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; \
  --describe
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The output gives you CURRENT-OFFSET, LOG-END-OFFSET, LAG, and CONSUMER-ID per partition. This is your first look: is the lag big, and is it concentrated on specific partitions?&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Level 2 — Continuously, with kafka-exporter + Prometheus:&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The &lt;a href="https://github.com/danielqsj/kafka_exporter" rel="noopener noreferrer"&gt;kafka-exporter&lt;/a&gt; exposes &lt;code&gt;kafka_consumergroup_lag&lt;/code&gt; per group and partition. The alert that actually works is not on absolute lag but on its trend:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;deriv(kafka_consumergroup_lag[10m]) &amp;gt; 0
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Lag that grows continuously is an incident. A spike that recovers is a batch job or a traffic burst — different conversation.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Level 3 — Lag as a first-class SLO.&lt;/strong&gt; Define it per group: "notification dispatch group must stay under 10,000 messages of lag." That converts an infrastructure metric into a promise someone can be paged on.&lt;/p&gt;

&lt;h2&gt;
  
  
  Reading lag like an operator
&lt;/h2&gt;

&lt;p&gt;Four patterns, four different actions:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;1. Lag grows steadily on all partitions.&lt;/strong&gt; Consumers are slower than producers. Check consumer processing time before scaling anything — if each message takes 200 ms and you're receiving 100 msg/s with 10 consumers, no amount of restarts will save you. Scale consumers or parallelize processing.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Lag grows on one partition only.&lt;/strong&gt; Classic partition key skew. One hot key (one big customer, one busy region) is hammering a single partition. Scaling consumers doesn't help — the partition is the ceiling. The fix is upstream: review the partition key.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Lag spikes and recovers.&lt;/strong&gt; Usually a batch producer or a restart. If it recovers within your SLO, tune the alert, don't chase the incident.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. Lag jumps with no producer change.&lt;/strong&gt; Suspect a rebalance storm. Consumers leaving and joining the group repeatedly pause consumption during each rebalance. Check &lt;code&gt;kafka-consumer-groups.sh --describe&lt;/code&gt; for changing CONSUMER-IDs, and look at session timeout vs processing time — a consumer that takes longer than its session timeout to poll will get kicked, rejoin, and repeat.&lt;/p&gt;

&lt;h2&gt;
  
  
  The scaling ceiling most people discover too late
&lt;/h2&gt;

&lt;p&gt;Here's the trap: you see lag, you scale consumers, lag stays. Why? &lt;strong&gt;A consumer group can never have more active consumers than partitions.&lt;/strong&gt; If your topic has 6 partitions and you run 12 consumers, 6 of them are idle. Before scaling consumers, check the partition count:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kafka-topics.sh --bootstrap-server &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;broker:9092&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; --topic &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;topic&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; --describe
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If you need more parallelism than partitions, you need to repartition the topic — which is a much bigger decision (offsets, ordering, consumer state).&lt;/p&gt;

&lt;h2&gt;
  
  
  A quick health one-liner
&lt;/h2&gt;

&lt;p&gt;When I want a single command that answers "is anything falling behind right now," across all groups:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kafka-consumer-groups.sh \
  --bootstrap-server &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;broker:9092&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; \
  --all-groups --describe 2&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;/dev/null \
  &lt;span class="o"&gt;|&lt;/span&gt; awk &lt;span class="s1"&gt;'$6 ~ /^[0-9]+$/ &amp;amp;&amp;amp; $6 &amp;gt; 1000 {print $1, $2, $3, $6}'&lt;/span&gt; \
  &lt;span class="o"&gt;|&lt;/span&gt; sort -k4 -nr &lt;span class="o"&gt;|&lt;/span&gt; head -20
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Any partition with more than 1000 messages behind (group, topic, partition, lag), sorted worst first. Zero output means you can go back to sleepp.&lt;/p&gt;

&lt;h2&gt;
  
  
  Summary
&lt;/h2&gt;

&lt;p&gt;Consumer lag is the earliest honest signal in an event-driven system. Measure it per partition, alert on its trend rather than its absolute value, know the four patterns (global slowness, key skew, bursts, rebalances), and remember the partition ceiling before you scale consumers. Treat lag as an SLO and it will tell you about incidents before your users do.&lt;/p&gt;

&lt;p&gt;Note: this article was drafted with AI assistance and reviewed, edited and tested by me based on my own operations experience.&lt;/p&gt;

</description>
      <category>kafka</category>
      <category>devops</category>
      <category>monitoring</category>
      <category>observability</category>
    </item>
    <item>
      <title>A Field Guide to CrashLoopBackOff: Beyond kubectl describe</title>
      <dc:creator>juan barraza</dc:creator>
      <pubDate>Sat, 14 Nov 2026 17:17:39 +0000</pubDate>
      <link>https://dev.to/juan_barraza_dde18726622b/a-field-guide-to-crashloopbackoff-beyond-kubectl-describe-10if</link>
      <guid>https://dev.to/juan_barraza_dde18726622b/a-field-guide-to-crashloopbackoff-beyond-kubectl-describe-10if</guid>
      <description>&lt;p&gt;If you operate Kubernetes, CrashLoopBackOff becomes the error you see most often and diagnose worst. &lt;br&gt;
The name is misleading: CrashLoopBackOff is not the problem. It's the kubelet's restart policy with exponential backoff telling you that your container keeps dying. &lt;br&gt;
The real error is somewhere else, and &lt;code&gt;kubectl describe&lt;/code&gt; alone rarely shows it.&lt;/p&gt;

&lt;p&gt;This guide is the workflow I use in production environments, ordered from the fastest signals to the deepest ones. Every step produces evidence you can keep — which matters when you need to escalate to another team or even show in front of your customer when the problem is "beyond your control" :-)&lt;/p&gt;
&lt;h2&gt;
  
  
  Step 1: Read the exit code before anything else
&lt;/h2&gt;

&lt;p&gt;The container's exit code tells you which family of problem you're in:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl get pod &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;pod-name&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; -o &lt;span class="nv"&gt;jsonpath&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;'{.status.containerStatuses[0].lastState.terminated.exitCode}'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For pods with multiple containers, list them first with&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl get pod &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;pod-name&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; -o &lt;span class="nv"&gt;jsonpath&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;'{.status.containerStatuses[*].name}'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;and adjust the index.&lt;/p&gt;

&lt;p&gt;The codes that appear constantly in practice:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Exit 0 in a loop&lt;/strong&gt; — the process finished successfully but the container expects it to keep running. Common with batch-style entrypoints, wrong PID 1, or an entrypoint that daemonizes and exits.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Exit 1&lt;/strong&gt; — application error. Read the logs (next step).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Exit 137&lt;/strong&gt; — SIGKILL. Almost always OOMKilled. Confirm with:
&lt;/li&gt;
&lt;/ul&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl get pod &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;pod-name&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; -o &lt;span class="nv"&gt;jsonpath&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;'{.status.containerStatuses[0].lastState.terminated.reason}'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;If it says &lt;code&gt;OOMKilled&lt;/code&gt;, no amount of log reading will help — raise the memory limit or fix the leak.&lt;br&gt;
NOTE FROM THE FIELD: When I say "fix the leak", I mean, take care of the internal issue that is using and killing your memory. It has been pretty common for me to find bad sizing issues, but also an I/O process that can't write and has nowhere to flush, so memory usage collapses.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Exit 143&lt;/strong&gt; — SIGTERM, graceful shutdown rejected or too slow.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Exit 126/127&lt;/strong&gt; — permission denied or command not found in the entrypoint. Your image and your runtime disagree.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2&gt;
  
  
  Step 2: Read the previous logs, not the current ones
&lt;/h2&gt;

&lt;p&gt;The current logs are usually empty or useless — the container just restarted. The crash is in the &lt;em&gt;previous&lt;/em&gt; container instance:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl logs &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;pod-name&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; --previous --tail=100
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;This single flag is the most underused command in Kubernetes troubleshooting. The stack trace, the connection refused, the missing file — it's almost always in there.&lt;/p&gt;

&lt;p&gt;If the logs are empty, the process is dying before writing anything: check the exit code again (126/127 points to the entrypoint), and check whether the log driver of your runtime is actually shipping stdout.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 3: The five causes that cover most real cases
&lt;/h2&gt;

&lt;p&gt;In order of how often I find them in production:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;1. Configuration injected at the wrong moment.&lt;/strong&gt; The pod starts before the ConfigMap/Secret exists, or an env var references a key that was renamed. The app crashes on startup, the next attempt finds the resource (if a controller created it late) or crashes forever. Check:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl describe pod &lt;span class="o"&gt;&amp;lt;&lt;/span&gt;pod-name&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; grep -A5 -i &lt;span class="s2"&gt;"environment\|mounts"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;2. A dependency that isn't ready.&lt;/strong&gt; Databases, Kafka brokers, LDAP servers. The app has no retry logic and dies on first connection refused. The fix is an initContainer or a startup probe — not more replicas.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;3. Liveness probe killing a slow starter.&lt;/strong&gt; The app needs 90 seconds to warm up; the liveness probe starts failing at 30. Kubernetes kills it; the loop begins. The tell: the container dies at a suspiciously regular interval, right after startup. Use a startupProbe for slow applications and keep liveness for steady-state.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;4. Running as non-root against root-owned code.&lt;/strong&gt; Hardened images (or an OpenShift-style cluster with restricted security contexts) run your container as a random UID. Your app tries to write to &lt;code&gt;/tmp&lt;/code&gt; or read a root-owned file and gets permission denied. Exit code 1 or 126.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;5. OOMKilled.&lt;/strong&gt; Exit 137. Already covered — but note that a JVM or Python process inside a container with a memory limit needs its heap configured against the &lt;em&gt;limit&lt;/em&gt;, not the node's memory.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 4: Events and timing
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;kubectl get events --field-selector involvedObject.name=&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;pod-name&lt;span class="o"&gt;&amp;gt;&lt;/span&gt; --sort-by=.lastTimestamp
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Look at the &lt;em&gt;pattern in time&lt;/em&gt;, not just the messages: a container killed exactly every N seconds is a probe or a dependency timeout. A container killed at random intervals is a resource or data-dependent failure.&lt;/p&gt;

&lt;h2&gt;
  
  
  A reusable evidence collector
&lt;/h2&gt;

&lt;p&gt;When this happens across many pods, I don't type these commands one by one. One loop, saved output, timestamped — everything an escalation needs:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="c"&gt;#!/usr/bin/env bash&lt;/span&gt;
&lt;span class="nv"&gt;POD&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$1&lt;/span&gt;
&lt;span class="nv"&gt;OUT&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;"crashloop-${POD}-$(date +%Y%m%d-%H%M%S)"&lt;/span&gt;
mkdir -p &lt;span class="s2"&gt;"$OUT"&lt;/span&gt;
kubectl get pod &lt;span class="s2"&gt;"$POD"&lt;/span&gt; -o yaml &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="s2"&gt;"$OUT/pod.yaml"&lt;/span&gt;
kubectl describe pod &lt;span class="s2"&gt;"$POD"&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="s2"&gt;"$OUT/describe.txt"&lt;/span&gt;
kubectl logs &lt;span class="s2"&gt;"$POD"&lt;/span&gt; --previous --tail=200 &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="s2"&gt;"$OUT/previous.log"&lt;/span&gt; 2&lt;span class="o"&gt;&amp;gt;&amp;amp;&lt;/span&gt;1
kubectl logs &lt;span class="s2"&gt;"$POD"&lt;/span&gt; --tail=50 &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="s2"&gt;"$OUT/current.log"&lt;/span&gt; 2&lt;span class="o"&gt;&amp;gt;&amp;amp;&lt;/span&gt;1
kubectl get events --field-selector involvedObject.name=&lt;span class="s2"&gt;"$POD"&lt;/span&gt; \
  --sort-by=.lastTimestamp &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="s2"&gt;"$OUT/events.txt"&lt;/span&gt;
tar czf &lt;span class="s2"&gt;"${OUT}.tgz"&lt;/span&gt; &lt;span class="s2"&gt;"$OUT"&lt;/span&gt;
&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"Evidence package: ${OUT}.tgz"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Run it, attach the tarball to the ticket, and you've done in 10 seconds what normally takes 20 minutes of back-and-forth.&lt;/p&gt;

&lt;h2&gt;
  
  
  Summary
&lt;/h2&gt;

&lt;p&gt;CrashLoopBackOff is a symptom with a short list of real causes. Exit code first, previous logs second, then the five usual suspects: config timing, missing dependencies, aggressive probes, UID permissions, and memory limits. The commands are cheap; the discipline of collecting evidence in order is what separates a 5-minute diagnosis from an afternoon of guessing.&lt;/p&gt;

</description>
      <category>kubernetes</category>
      <category>devops</category>
      <category>debugging</category>
      <category>linux</category>
    </item>
  </channel>
</rss>
