A Running Container Doesn't Mean a Healthy Service

A container is running. A health check is failing. A worker has exited.

Those observations answer different questions. Treating them as interchangeable makes an investigation harder, especially when several services change state close together.

I'm Gökhan, one of the developers behind CoreTrailKit at Core Trail Labs. This is a technical note from a controlled local Docker test we used while developing the app—not a customer incident or a production outage.

The test: several failures, not one proven chain

The environment contained three demo repositories and 14 real Docker services. We deliberately introduced several failure conditions:

• An order-worker exceeded its memory limit and was OOM-killed.
• Selected health checks were made to fail, with explanatory output supplied by the demo fixtures.
• A shipping-api process stopped with exit code 1.

These were controlled inputs. In particular, a health-check message mentioning an unavailable dependency was part of the fixture; it was not independent proof that the worker's memory failure caused every other symptom.

That distinction matters when reading a convincing-looking dashboard.

Process state and health are separate observations

Docker gives a container with a configured health check a health status in addition to its normal container status. A process can remain running while its health check fails. Docker's HEALTHCHECK reference explains that distinction.

The reverse deserves caution too: passing a particular probe does not establish that every user-facing operation works. It tells us that the configured probe passed.

For an existing container, a useful starting point is to inspect its state and recent application output:

# Replace CONTAINER_ID with the actual container name or ID.
docker inspect --type container --format '{{json .State}}' CONTAINER_ID
docker logs --timestamps --since 10m CONTAINER_ID

These are inspection examples, not captured output from the screenshots. The Docker references for inspect and logs describe the commands and options.

Keep the questions separate: is the process running, what does the probe report, and what does the application say? Do not combine three partial answers into a stronger conclusion than the evidence supports.

An alarm should identify the evidence, not invent the explanation

Here is the actual alarm screen from our test:

Real application screenshot from the controlled demo. The health-check explanations were supplied by the fixtures.

For the worker, the Docker connector checks the reported OOMKilled flag rather than treating any exit as a memory failure. The stopped shipping service is a different observation: exit code 1 does not explain the underlying application bug.

Likewise, a failing health check is evidence of a failed probe. Its output can be a useful lead, but the text itself may still need verification against the application and its dependencies.

The next investigation step should follow the evidence available—not the most plausible story we can tell about it.

A timeline preserves context, not causality

The timeline captured changes observed during the watch session:

Several visible entries share the same displayed minute. Their position on the screen is not proof of a precise causal order.

This view is useful for asking what was observed around the time a problem appeared. It does not establish that the first visible entry caused the next one, or that every underlying transition was captured.

To investigate a suspected dependency failure, I would look for more specific evidence: timestamped application output, matching request identifiers, or records from both sides of the interaction. A nearby timestamp is a reason to investigate, not a completed diagnosis.

What to capture before restarting

The practical takeaway does not require a particular dashboard. Before changing the failing environment, capture:

• The affected container's identity and current state.
• Health-check status and available probe output.
• Exit information and any explicit OOM indication.
• Relevant timestamped application logs.
• What you changed yourself, and when.

Be careful when sharing that material: logs and probe output can contain credentials or personal data.

In our test, we knew how the failures had been introduced. In a real investigation, we often do not. Keeping observations, fixture-supplied explanations, and causal conclusions separate is what makes the record useful later.

When a local service fails, what context do you most often discover you should have captured before restarting it?