Kubernetes Logging in 2026: From kubectl logs to a Pipeline That Scales

TL;DR: In Kubernetes, logs die with the pod — and the pods you most need logs from are the ones that just died. The fix is an architecture, not a tool: applications write structured logs to stdout/stderr, a node-level agent (Fluent Bit, usually) collects and enriches them with Kubernetes metadata, and a backend (Loki, OpenSearch, or your cloud's) stores them with retention you chose on purpose. Add correlation IDs so a log line connects to a request and a trace, control the volume before it controls your budget, and test the pipeline by killing a pod and finding its last words.

Originally published on DevOps Enginer.

The problem: the evidence deletes itself

On a VM, a crashed application's logs sat on disk until someone looked. In Kubernetes, the sequence runs the other way: the pod crashes, gets rescheduled or replaced, the node eventually cleans up, and the logs — the primary evidence of what happened — evaporate on a schedule driven by everything except your investigation.

kubectl logs papers over this for exactly one generation: it shows the current container, and with -p the previous one. Before that? Gone. On another node after a reschedule? Gone. From the DaemonSet that was OOM-killed at 3 a.m.? Never existed as far as you can prove.

So the troubleshooting guide in this series starts with logs — and production logging is the discipline of making sure they are somewhere else before the pod takes them away.

The model: stdout is the application's only logging API

The rule from which everything else follows: a containerised application logs to stdout and stderr, and nowhere else. No log files inside the container, no application-managed rotation, no "the file is on the volume somewhere." The container runtime captures those two streams to files on the node (/var/log/containers/…), and the platform takes it from there.

This buys three things:

• Uniformity. Every workload, every language, same contract. The pipeline never needs per-application file paths.
• Rotation for free (with a caveat below): the kubelet and runtime rotate the node-side files; the application cannot fill its own container filesystem with logs.
• Streams stay separable. stderr is not automatically "errors" — plenty of frameworks log everything to stdout — but having both captured beats parsing one merged file inside a dying container.

The application-side obligations that make this work: one event per line, and structured — JSON lines with a timestamp, a level, a message, and the fields a future searcher will filter on (service, version, request ID). Plain-text prose logs are a tax every incident pays at the worst exchange rate.

The collector: a node-level agent

The standard shape is one log collector per node, running as a DaemonSet, tailing the runtime's log files, enriching each record with Kubernetes metadata (namespace, pod, container, labels — fetched from the API), parsing structure, and shipping to the backend.

The usual choice in 2026 is Fluent Bit: small, fast, CNCF-graduated, written for exactly this job. Fluentd (its older sibling) remains capable, especially where a team already runs its plugin ecosystem; a few platforms ship their own agents. What matters more than the brand:

• Metadata enrichment is non-negotiable. A log line without namespace/pod/labels is nearly unsearchable at cluster scale. Verify your pipeline adds them — silently missing metadata is the most common logging bug in production clusters.
• Buffering and backpressure. When the backend is unreachable, the agent must buffer (to disk, with a cap) and retry — and its buffer behaviour must not be allowed to fill the node disk it is protecting you from. Size it, alert on it.
• The agent is production software. It runs on every node, touches every log line, and fails silently when misconfigured. It belongs in the observability stack's alerting (the observability guide covers the wider stack): DaemonSet pods healthy, shipping lag near zero, drop counters at zero.

A note on audit logs: the API server's audit stream is a different animal — who did what to the cluster, the security record from the security-basics guide — collected at the control plane, retained separately, and never mixed into the application pipeline.

Where logs land: pick the backend by the questions you ask

• Loki — indexes only labels, stores compressed log chunks in object storage. Cheap to run, excellent when your queries are "this service, this namespace, this time window, containing this string" — which is most incident queries. Pairs with Grafana next to your metrics. Its discipline: labels are precious and few (service, namespace, level, environment). A user ID or request ID in a label explodes the index (the cardinality trap); those belong in the log line or in structured fields, not labels.
• Elasticsearch / OpenSearch — full-text indexing of everything: expensive in storage and operations, unmatched when you genuinely need arbitrary field search across huge volumes and long retention.
• Cloud-native sinks — CloudWatch Logs, Google Cloud Logging, Azure Monitor: zero extra infrastructure on the matching managed service, pricing that rewards a retention policy, and query languages your team may like less than LogQL. For many EKS/GKE/AKS clusters this is the pragmatic default, with the node agent either replaced by the provider's or forwarding to it.

Any of the three works. The expensive mistakes are the two defaults nobody chose: keeping everything forever, and discovering your per-GB ingestion bill during the incident that tripled log volume.

The sidecar pattern: when node-level isn't enough

Some applications will not log to stdout — legacy software writing to files, multi-line stack traces an application insists on assembling itself, or a workload needing local pre-processing. The pattern: a sidecar container in the same pod that reads the file from a shared volume and writes it to its own stdout (so the node agent picks it up normally), or ships directly to the backend. It works, at the price of a container per pod — resources, complexity, one more thing to upgrade. Reach for it per-workload, knowingly, not as the cluster default.

Log survival details that bite

• Rotation is configured at the node, and it has a size. Runtime/kubelet log rotation (commonly around 10 MiB per file, a handful of files) means a chatty container's history on the node may cover minutes, not hours. That is fine because the agent ships lines as they are written — but it is why "the agent was down for an hour" translates directly into "that hour's logs exist only where the agent later recovered them, if it could." Agent health is log retention.
• Node disk is a shared fate. Logs, images, and emptyDir volumes all eat the same disk; a runaway logger can trigger disk-pressure evictions of innocent pods (the storage guide covers the disk side). Resource discipline — requests and limits from that guide — applies to your loggiest workloads first.
• kubectl logs still matters. For live debugging it is the fastest tool that exists: kubectl logs -l app=api --since=10m --all-containers=true across a label selector, --previous for the crashed instance, --follow during a rollout. The pipeline is for history and search; kubectl is for now. Engineers should be fluent in both.

Correlation: logs that connect to everything else

A log stream alone answers "what did this service print." Production debugging asks a bigger question: "what happened to this request." The bridge is a correlation ID: generated at the edge (or taken from an incoming header), propagated through every service call, and printed in every log line and trace span. With that one discipline, an incident moves from grepping five services by timestamp to one query per system, all keyed by the same ID — and the metrics–traces–logs triad from the observability guide becomes a single investigation instead of three.

Two supporting habits:

• Levels with meaning. ERROR should mean "a human may need to act," not "a retry happened." When everything is an error, nothing is; alert on error rate changes, not on the existence of error lines.
• Never log secrets or personal data. Tokens, passwords, full request bodies with credentials, unnecessary personal fields — logs are copied, retained, and widely readable by design; treat them as the least-trusted storage you own. Redaction at the application beats scrubbing in the pipeline, and both beat explaining a credential found in a log bucket (the ConfigMaps and Secrets guide covers the secret-handling side).

Volume, retention, and the bill

Log volume scales with traffic and verbosity, and both spike during incidents — exactly when you can least afford dropped data or a surprise invoice. The controls, in the order teams usually meet them:

1. Retention tiers. Hot storage (fast search) for days-to-weeks; cheaper object storage for months; deletion after the period your compliance actually requires. "Forever" is a decision too — an expensive one nobody remembers making.
2. Drop or sample the noise. Health-check access logs from internal probes, debug output from one misbehaving workload, a namespace that logs every cache hit — filter at the agent, where dropping is free, rather than at the backend, where you have already paid to ingest.
3. Per-namespace visibility. Know which workloads produce the volume. The cost guide in this series makes the same point about compute; logs deserve the same per-team accounting, because the fix is always a conversation with one workload's verbosity.

Prove it: the kill-a-pod test

Like backups and disruption budgets before it in this series, a logging pipeline is a rumour until exercised. The test is five minutes:

1. Pick a workload. In its logs, find (or generate) a unique marker line — a distinctive request ID works.
2. Delete the pod. Wait for the node to forget it ever existed.
3. From the backend alone, retrieve that pod's final log lines, filtered by namespace, pod name, and the marker. Check the metadata is attached and the timestamps are ordered and correct.

If step 3 fails, you do not have a logging pipeline — you have a logging intention. Run the test after every agent upgrade and every backend change, and monitor the agent continuously in between; silent gaps are this system's natural failure mode.

Common mistakes

• Applications logging to files inside containers. The data is invisible to the platform, dies with the pod, and fills the container filesystem on the way out.
• Unstructured prose logs. Every incident becomes a regex-writing contest. JSON lines with stable fields, from day one.
• High-cardinality labels. Request IDs, user IDs, and pod IPs as index labels: the backend slows, costs climb, and the fix (re-indexing your mental model) is painful. Labels identify the stream; fields describe the event.
• No one owns agent health. The DaemonSet falls behind or dies on two nodes; the gap is discovered mid-incident, which is the only time it cannot be fixed.
• Debug level in production, forever, because one incident needed it once. Verbosity is a per-workload dial with a cost meter attached.
• Secrets in logs. The widest-read storage in your stack is the worst place for the most sensitive strings you own.
• Retention set by the tool's default instead of by your actual compliance and debugging needs — too short loses investigations, too long buys nothing but storage bills.

Where this fits in the series

Logging is the third pillar the observability guide stands on, and the raw material the troubleshooting guide consumes first. It leans on the storage guide's disk discipline, the requests-and-limits guide's resource accounting, the network policies guide's protection for the shipping path, and the security-basics guide's audit and redaction rules. Get it right and it disappears — every other guide in this series quietly assumes the logs were there when needed.

FAQ

Fluent Bit or Fluentd?
For a new cluster pipeline, Fluent Bit: lighter, faster, and the current centre of gravity for Kubernetes log collection. Fluentd remains reasonable where its plugin ecosystem or an existing deployment carries weight. The architecture matters more than the choice.

Loki or Elasticsearch?
If your queries are service-and-time-window shaped (most incident work), Loki's label model is dramatically cheaper to run. If you need deep, arbitrary full-text analytics over long retention, Elasticsearch/OpenSearch earns its cost. Many clusters start on their cloud provider's sink and graduate to one of these when query needs or bills demand it.

Should the application ever ship logs directly to the backend?
Almost never. It couples application deploys to logging infrastructure, duplicates shipping logic in every service, and loses the node agent's uniform metadata and buffering. The stdout contract exists precisely to avoid this.

How long should logs be retained?
Long enough to cover your slowest investigation and your compliance floor — commonly 2–4 weeks hot and a few months in cheap archive for application logs, longer for audit logs under policy. Decide per class of log, write it down, and let the tiers enforce it automatically.

Do managed services (EKS, GKE, AKS) solve this for me?
They provide a destination (CloudWatch, Cloud Logging, Azure Monitor) and sometimes an agent option, but the architecture decisions remain yours: stdout discipline, metadata, correlation IDs, retention, and cost control. The provider removes the backend operations, not the design.