When Docker Says 'Started' But Consul Isn't Ready
It's 2:00 AM. Your CI pipeline just failed for the third time with "connection refused" from Consul, but the container logs show Started fifteen seconds before the test ran. The container is up. The port is bound. Consul's process is running. Yet your application can't read the service catalog because no Raft leader has been elected. docker compose up told you everything was ready. It lied.
The gap between "container started" and "Consul can serve reads" is a race condition that wastes CI minutes and breaks local development feedback loops. When evaluating hashicorp/consul for orchestration, I want the first test request to tell me whether the system works, not whether my timing was lucky. The useful boundary is a real catalog read, not just an open socket.
One stack, two readiness checks
This setup demonstrates layered health checks: Compose waits for Consul to answer a Raft peer query before starting the application, then the application owns its own readiness by making a consistent catalog read with a two-second deadline. The peer query tells you whether Consul is ready to have dependents. The catalog read tells you whether those dependents can actually use Consul. Those are different questions, and keeping both makes startup failures easier to locate.
Paste this into an empty directory with Docker Compose v2. It writes the service, Dockerfile, and docker-compose.yml, then waits for both health checks. Nothing publishes a host port. The last command runs the same probe Compose uses; exit status zero means the full path worked.
cat > main.go <<'EOF'
package main
import (
"net/http"
"os"
"time"
)
var client = &http.Client{Timeout: 2 * time.Second}
func ok(url string) bool {
res, err := client.Get(url)
if err != nil {
return false
}
defer res.Body.Close()
return res.StatusCode == http.StatusOK
}
func main() {
if len(os.Args) == 2 && os.Args[1] == "probe" {
if !ok("http://127.0.0.1:8080/ready") {
os.Exit(1)
}
return
}
http.HandleFunc("/ready", func(w http.ResponseWriter, r *http.Request) {
if !ok("http://consul:8500/v1/catalog/services?consistent") {
http.Error(w, "Consul catalog unavailable", http.StatusServiceUnavailable)
return
}
w.WriteHeader(http.StatusOK)
})
s := &http.Server{Addr: ":8080", ReadHeaderTimeout: 2 * time.Second}
panic(s.ListenAndServe())
}
EOF
cat > Dockerfile <<'EOF'
# syntax=docker/dockerfile:1
FROM golang:1.26-alpine AS build
WORKDIR /src
COPY main.go .
RUN --mount=type=cache,target=/root/.cache/go-build \
CGO_ENABLED=0 go build -trimpath -o /service main.go
FROM scratch
COPY --from=build /service /service
USER 65532:65532
ENTRYPOINT ["/service"]
EOF
cat > docker-compose.yml <<'EOF'
services:
consul:
image: hashicorp/consul:1.22.0
command: ["agent", "-server", "-bootstrap-expect=1", "-node=consul", "-client=0.0.0.0", "-data-dir=/consul/data"]
networks: [backend]
volumes: ["consul-data:/consul/data"]
cpus: "1.0"
mem_limit: 512m
restart: unless-stopped
healthcheck:
test: ["CMD", "consul", "operator", "raft", "list-peers"]
interval: 5s
timeout: 3s
retries: 6
start_period: 15s
app:
build: .
depends_on:
consul:
condition: service_healthy
networks: [backend]
cpus: "0.5"
mem_limit: 64m
restart: unless-stopped
read_only: true
cap_drop: [ALL]
security_opt: ["no-new-privileges:true"]
healthcheck:
test: ["CMD", "/service", "probe"]
interval: 5s
timeout: 3s
retries: 3
networks:
backend:
driver: bridge
internal: true
volumes:
consul-data:
EOF
docker compose up --build --wait --wait-timeout 90
docker compose exec -T app /service probe
What this fixes, and what it doesn't
A startup delay is not a readiness check. Five seconds might be plenty on a warm laptop and too short on a CPU-constrained CI runner restoring Consul state from disk. With layered health checks, docker compose ps shows which side failed. If Consul is unhealthy, start with its logs and Raft state. If Consul is healthy but the app isn't, investigate the app-to-Consul path and the catalog response.
After startup, Compose reports health changes but doesn't restart a container just because it becomes unhealthy. That matters here: restarting the app won't repair a missing Raft leader. The application's readiness check surfaces the problem without creating restart churn.
Two Docker failure patterns are worth avoiding while fixing the startup race. One is a root process paired with a writable host bind mount: a compromised process can modify mounted host files without needing a container escape. Root by itself is not an escape. This app runs as a numeric non-root user, has no host mounts, and gets a read-only root filesystem. The other pattern is putting COPY . . before dependency installation: a source-only edit invalidates the dependency layer and turns a small change into a full rebuild. Here, a source edit reruns go build, but BuildKit keeps the compiler cache on the same builder. A fresh CI builder still starts cold.
The named volume preserves Consul state across ordinary restarts, which helps local iteration but can hide first-boot problems in CI. Use a fresh Compose project per run when bootstrap behavior is what you're testing. The CPU and memory settings are starting budgets, not capacity estimates. If readiness starts failing under load, check throttling and memory pressure before extending the timeout. A longer grace period can hide the same contention your application will face on its next restart.
The single-server tradeoff
This is a single-server sandbox, not a highly available Consul deployment. The private bridge reduces exposure but does not replace ACLs or TLS, and -bootstrap-expect=1 deliberately removes quorum failure from the exercise. For shared deployments, use a supported patched image pinned by digest and a properly secured multi-server cluster.
Keep the application-owned readiness check, but here's the hard question: should you make Consul a hard dependency if requests really need a fresh catalog read, or should the app serve from a last-known-good snapshot during brief leader elections? The first choice makes every Consul outage an application outage. The second choice tolerates stale reads but requires snapshot logic and state reconciliation.
Which failure mode have you chosen for your service mesh, and what production incident made you pick it?
Top comments (0)