DEV Community: BrainTally

A practical Kubernetes troubleshooting checklist

BrainTally — Sat, 17 Oct 2026 11:59:47 +0000

A pod is not running, or is running but serving errors, and you have a terminal and a few minutes. Trying things at random wastes them. Kubernetes already records most of what you need; the skill is reading it in the right order. This checklist moves from the cheapest, most informative checks to the more expensive ones. Commands assume kubectl is pointed at the right cluster and namespace, so confirm that first with kubectl config current-context.

Step 1: look at the state

kubectl get pods -o wide
kubectl get deploy,rs,svc

The STATUS, READY and RESTARTS columns tell you which branch of this checklist to follow. The -o wide output also shows which node each pod landed on, which matters later.

Step 2: read the status like a diagnosis

Status	What it usually means	Next check
Pending	The scheduler has not placed the pod	`describe pod`, events
ContainerCreating	Image pull, volume mount or network setup in progress or stuck	events
ImagePullBackOff / ErrImagePull	Wrong image name or tag, missing registry credentials, registry unreachable	events, image reference, pull secret
CrashLoopBackOff	The container starts, exits, and is restarted with growing delays	previous logs, exit code
CreateContainerConfigError	A referenced ConfigMap or Secret (or key) is missing	events
Running but not Ready	Readiness probe failing	probe config, logs
Evicted	The node removed the pod, often under resource pressure	node conditions

Note that "CrashLoopBackOff" is not itself a cause. It is the kubelet backing off between restarts of a container that keeps exiting. By default the delay starts at 10 seconds, doubles each time, and is capped at 5 minutes; it resets after the container runs for 10 minutes without problems.

Step 3: read the events

kubectl describe pod NAME
kubectl get events --sort-by=.lastTimestamp

The Events section at the bottom of describe is usually where the answer is. For a Pending pod, look for scheduler messages such as insufficient CPU or memory, unmatched node selectors or affinity, untolerated taints, or unbound PersistentVolumeClaims. Events are kept for one hour by default (--event-ttl), so check them early.

Also in describe, under Containers, read State, Last State, Reason and Exit Code. OOMKilled means the container exceeded its memory limit. Exit codes above 128 are 128 plus the signal number: 137 is SIGKILL (9), 143 is SIGTERM (15).

Step 4: read the logs, including the previous container

kubectl logs POD
kubectl logs POD --previous
kubectl logs POD -c CONTAINER
kubectl logs -l app=myapp --tail=50

For a crash loop, the current container may have just started and have nothing to show, so --previous is what you want. For pods with several containers, name the container with -c, and remember init containers: a failing init container keeps the pod from starting.

Step 5: check configuration and wiring

Environment and config: kubectl exec POD -- env and confirm values from ConfigMaps and Secrets arrived.
Service selectors: kubectl get endpointslices -l kubernetes.io/service-name=SVC. An empty result means no ready pod matches the Service selector. Compare kubectl get pods --show-labels with the selector.
Ports: check targetPort on the Service against the port your container listens on.
Probes: an aggressive liveness probe restarts a healthy but slow-starting app. A startup probe is the documented fix.
Permissions: kubectl auth can-i list pods --as=system:serviceaccount:NAMESPACE:NAME -n TARGET_NAMESPACE.

If an image has no shell, kubectl debug -it POD --image=busybox --target=CONTAINER adds an ephemeral debug container alongside the target.

Step 6: check resources and node pressure

kubectl describe node NODE
kubectl top nodes
kubectl top pods

In describe node, read Conditions: MemoryPressure, DiskPressure or PIDPressure set to True mean the kubelet may evict pods. Also read Allocated resources: requests add up even when usage is low, so a cluster can be "full" for the scheduler while the CPUs are idle. Requests drive scheduling; limits are enforced at runtime (CPU is throttled, memory can be OOM-killed).

Step 7: check the rollout

kubectl rollout status deploy/NAME
kubectl rollout history deploy/NAME
kubectl rollout undo deploy/NAME

If the problem began with a deploy, consider rolling back first, then investigating. Restoring service and diagnosing are separate jobs.

Common mistakes

Running kubectl logs on a crash-looping pod and stopping when it is empty. Use --previous.
Debugging in the wrong namespace or context.
Treating CrashLoopBackOff as the cause rather than the symptom.
Raising a memory limit without checking whether the app leaks.
Changing several things at once.
Reading events too late; they are gone after an hour by default.

How to tell it is fixed

kubectl get pods shows all containers READY and a restart count that has stopped climbing.
kubectl rollout status reports success.
The Service's EndpointSlice lists the pod addresses, and a request from inside the cluster succeeds.
No new warning events for several minutes.

Want to check how well this sticks? Free 10-question Kubernetes skills check - scenario questions with an explanation for every answer, no sign-up: braintally.com/technical-assessments/kubernetes/check

Originally published on BrainTally.

AWS IAM mistakes teams keep making

BrainTally — Sat, 17 Oct 2026 11:05:16 +0000

Most AWS security incidents do not start with an exotic exploit. They start with an over-broad policy, a key that was never rotated or a role that trusts more than it should, each created because it was the fastest way to make something work. Here are the IAM mistakes that recur most often, what to do instead, and how to check that the fix held.

1. Wildcards in actions and resources

{ "Effect": "Allow", "Action": "*", "Resource": "*" }

That is administrator access. Less obvious variants are s3:* on every bucket, or ec2:* for a pipeline that only starts and stops instances. Name the actions and narrow the resource:

{
  "Effect": "Allow",
  "Action": ["s3:GetObject", "s3:PutObject"],
  "Resource": "arn:aws:s3:::example-bucket/uploads/*"
}

Object-level actions such as s3:GetObject apply to the object ARN (bucket/prefix/*), while s3:ListBucket applies to the bucket ARN - mixing these up is a common reason a "tight" policy fails and someone swaps in a wildcard. IAM Access Analyzer can generate a policy from CloudTrail activity; test it, and don't treat the first working policy as the final one.

2. Long-lived access keys

A leaked key keeps working until someone notices. Prefer temporary credentials:

Workloads on AWS: instance profiles, ECS task roles, or a Kubernetes service account mapped to a role.
CI/CD: OpenID Connect federation (sts:AssumeRoleWithWebIdentity) - no stored key.
People: federated sign-in through IAM Identity Center.

Audit what's left with aws iam generate-credential-report and aws iam get-credential-report.

3. Loose trust policies

Trusting arn:aws:iam::123456789012:root trusts the whole account. Name the specific role instead, use sts:ExternalId for third parties, and for GitHub Actions OIDC roles always restrict the sub claim:

"Condition": {
  "StringEquals": {
    "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
    "token.actions.githubusercontent.com:sub": "repo:my-org/my-repo:ref:refs/heads/main"
  }
}

4. Unrestricted iam:PassRole

iam:PassRole on "Resource": "*" lets a user hand compute a more powerful role than their own. Restrict it to specific role ARNs and add the iam:PassedToService condition. Treat iam:CreatePolicyVersion, iam:AttachUserPolicy and iam:PutRolePolicy the same way.

5. No guardrails above individual policies

Permissions boundaries cap what a role's policies can grant; SCPs cap whole accounts; an explicit deny anywhere wins. Effective permissions are the intersection.

6. Shared users and unused identities

Shared logins destroy auditability. Use last-accessed data or Access Analyzer's unused-access findings to remove old users, roles and keys. The root user: MFA, no access keys, rarely used.

7. Skipping MFA and visibility

Require MFA (passkeys or security keys where you can), keep CloudTrail logs somewhere workload roles can't delete, and review Access Analyzer findings.

A quick audit for this week

Find policies with "Action": "*" or "Resource": "*" and justify each.
Generate a credential report; delete or rotate old keys.
Read the trust policy of every role assumable from outside the account.
List who holds iam:PassRole and IAM-write permissions.
Move one CI pipeline from a stored key to OIDC.

Want to see how well this sticks? I put together a free 10-question AWS skills check - scenario questions with an explanation for every answer, no sign-up: braintally.com/technical-assessments/aws/check

Originally published on BrainTally.