DEV Community

Ramesh Yara
Ramesh Yara

Posted on Originally published at Medium

Five Things Kubernetes Quietly Added to My Containers (And How Each One Broke a Confluent Service)

#kubernetes #kafka #devops #java

Originally published on Medium.

When I moved a full Confluent Kafka stack into a Kubernetes cluster, five things broke. Each one looked like a configuration mistake I had made. None of them were.

Every failure came from something Kubernetes added to my containers without being asked — a default behavior that is correct in the general case and a silent hazard for these specific images. Here is what each one did and the fix.

1. Kubernetes injected a variable that crashed the Kafka broker

The broker came up for a few seconds, then died. The log said the advertised listener address was pointing to a TCP socket URL — something like tcp://10.96.0.1:9092 — instead of a hostname. I had set the advertised listener explicitly. Something had overridden it.

Kubernetes had.

By default, every pod in a namespace receives environment variables for every Service in that namespace. For a Service named kafka, Kubernetes injects KAFKA_PORT=tcp://10.96.0.1:9092, KAFKA_SERVICE_HOST=10.96.0.1, and several more. This is a legacy feature from the Docker links era, on by default for backwards compatibility.

The Confluent Kafka image maps every environment variable prefixed with KAFKA_ directly to a broker configuration key. The injected KAFKA_PORT variable landed in that mapping, overrode the advertised listener I had configured, and produced an invalid broker config. The crash log blamed the configuration value — not the source of it.

The Confluent Schema Registry image has the same convention with SCHEMA_REGISTRY_* prefixed variables. A Service named schema-registry injects SCHEMA_REGISTRY_PORT, which the image reads as a configuration for its deprecated PORT key.

The fix is one field:

spec:
  enableServiceLinks: false
Enter fullscreen mode Exit fullscreen mode

Both the broker and Schema Registry came up cleanly once service link injection was disabled on their pod specs.

2. CPU limits without requests made every pod Pending

After the broker came up, all eight pods went Pending simultaneously. kubectl describe pod showed Insufficient cpu on every one of them. The node had 2 CPU cores and nothing was running on it.

The cause was how Kubernetes handles resource specs with only limits defined. When a container specifies resources.limits.cpu but omits resources.requests.cpu, Kubernetes silently sets the request equal to the limit. With eight pods each carrying an implicit 500m CPU request, the scheduler saw 4000m of demand against a 2000m node. All eight were Unschedulable before a single one started.

The fix is an explicit minimal request:

resources:
  requests:
    cpu: 1m
    memory: 512Mi
  limits:
    cpu: 500m
    memory: 1Gi
Enter fullscreen mode Exit fullscreen mode

1m is nominal — it tells the scheduler that the container needs one millicore of guaranteed scheduling capacity, not 500m. The scheduler placed all eight pods immediately.

The rule generalizes: omitting requests does not mean "no CPU requirement." It means "requirement equals limit." For a stack of JVM services alongside Kafka and a database, the implicit math almost always exhausts the node before a single pod starts.

3. The default probe timeout was 1 second — and the JVM needed 3

With pods scheduled, the next issue surfaced: MySQL and PostgreSQL were intermittently NotReady, causing cascading Connection refused errors in the Spring Boot services.

The default timeoutSeconds for Kubernetes probes is 1 second. Under Docker Desktop and WSL2, mysqladmin ping and pg_isready routinely take 1–3 seconds to respond even on fully initialized databases. Overlayfs plus a constrained CPU budget introduces latency that doesn't exist on a local host process.

When a 1-second timeout fires against a database that takes 1.5 seconds to respond, Kubernetes marks the pod NotReady and removes it from the Service endpoint list. Any connection pool — Hikari — that opens a connection during that window hits Connection refused from the cluster's DNS resolver, not from the database itself. The error looks like a connectivity problem in the application.

Raising the timeout resolved it:

readinessProbe:
  timeoutSeconds: 5
startupProbe:
  timeoutSeconds: 5
Enter fullscreen mode Exit fullscreen mode

The 1-second default is appropriate for bare-metal or cloud clusters where probe responses are in the sub-100ms range. In a local WSL2 environment it fires spurious failures that cascade through every service that depends on the probed component.

4. Keycloak's health endpoint wasn't where the probe was looking

Keycloak also had a probe problem, but a different one.

In Keycloak's production mode (start), the health endpoints live on the management port — port 9000. In development mode (start-dev), only port 8080 is opened. Port 9000 is silent.

My probe was checking /health/ready on port 9000. It was timing out against a port that didn't exist, leaving Keycloak permanently NotReady.

Fixing the port was necessary but not sufficient. /health/ready on port 8080 returns 404 in both modes. The correct readiness signal for Keycloak in this setup is the realm endpoint:

readinessProbe:
  httpGet:
    path: /realms/ai-microservices
    port: 8080
Enter fullscreen mode Exit fullscreen mode

The realm endpoint returns 200 only after Keycloak's Liquibase migration has run and the realm has been imported from the ConfigMap. It verifies that the actual downstream dependency — token issuance — is ready, not just that the process started.

The startup window also needs to account for Keycloak's real initialization time. On WSL2, start-dev takes 8–20 minutes from pod creation to a valid realm response. The startup probe failureThreshold needs to be set accordingly:

startupProbe:
  httpGet:
    path: /realms/ai-microservices
    port: 8080
  failureThreshold: 120
  periodSeconds: 15
Enter fullscreen mode Exit fullscreen mode

5. A rolling update killed Schema Registry — permanently

With everything stable, I issued a kubectl apply to update the Schema Registry deployment. The new pod came up alongside the old one — standard rolling update behavior — and Schema Registry died in that window and never recovered.

Schema Registry uses Kafka as its backing store and participates in a Kafka consumer group to elect a leader that coordinates reads and writes to the _schemas topic. When two Schema Registry pods run simultaneously — even briefly during a rolling update — both attempt to join the consumer group. Kafka triggers a partition rebalance.

The KafkaGroupLeaderElector thread that manages Schema Registry's leadership does not survive that rebalance. It exits with LEADER_NOT_AVAILABLE and does not restart. All subsequent requests to register or retrieve schemas fail until the pod is manually restarted.

With a single replica, every rolling update creates a two-pod window. The fix is to use Recreate:

spec:
  strategy:
    type: Recreate
Enter fullscreen mode Exit fullscreen mode

Recreate terminates the old pod before starting the new one. No two-pod window, no rebalance. At single-replica scale the tradeoff is brief unavailability during the update — acceptable because the platform's transactional outbox absorbs the gap rather than losing events.

What these five have in common

Each failure was documented Kubernetes behavior. None of them were bugs. Each one was a default designed for a broad set of workloads that happened to be wrong for this specific combination of images and topology.

Service link injection exists to help containers discover other services. Implicit request-equals-limit exists to prevent pods from being unschedulable when only limits are set. The 1-second probe timeout exists because most health endpoints are fast. Rolling update semantics exist because most applications can run multiple instances during a deploy.

Confluent images that interpret KAFKA_* environment variables as configuration, JVM startup latency under WSL2, Keycloak's development-mode port behavior, and Schema Registry's single-leader election model are each correct on their own. The failures come from the intersection.

The practical checklist that came out of this:

  • enableServiceLinks: false on every Confluent image (Kafka, Schema Registry) — and any image that reads its own name as an env var prefix.
  • Explicit requests.cpu on every container, even if it's nominal.
  • timeoutSeconds: 5 on readiness and startup probes in any local cluster.
  • Read the image's health endpoint documentation, not the Kubernetes docs, to find the correct probe path and port for each version.
  • strategy: Recreate for any single-replica service that participates in leader election.

All five are now set in every manifest I write for this stack.

This platform — Spring Boot 4, Java 21, Kafka, Keycloak, and a full in-cluster observability stack — is open-source: github.com/Rummy43/ai-microservices-platform

Top comments (0)