DEV Community

Ramesh Yara
Ramesh Yara

Posted on Originally published at Medium

Three Layers Deep: Getting Confluent Schema Registry to Authenticate with MSK Serverless on EKS

#kubernetes #aws #kafka #java

The Setup

When you deploy Confluent Schema Registry on EKS against an MSK Serverless cluster, you're combining three things that each have their own opinions about authentication:

  1. Confluent's Docker image — assumes it controls its own environment
  2. Kubernetes — helpfully injects environment variables from service discovery
  3. MSK Serverless — requires SASL/OAUTHBEARER (IAM) with a specific JAR on the JVM classpath

In isolation, each is fine. Together, they'll fight you in exactly three layers. Here's what I found and how I fixed each one.

The Environment

  • EKS 1.31 (Fargate managed node group)
  • MSK Serverless (us-east-1)
  • confluentinc/cp-schema-registry:7.5.0 as the base image
  • MSK IAM auth via IRSA (IAM Roles for Service Accounts)
  • aws-msk-iam-auth-2.3.2-all.jar for the SASL callback handler

Layer 1: Kubernetes Poisons the Schema Registry Config

Symptom: Schema registry pod crashes immediately on startup. Logs show something like:

Schema Registry environment variables are not properly configured
SCHEMA_REGISTRY_PORT must be set to a valid port number
Enter fullscreen mode Exit fullscreen mode

Root cause: Kubernetes injects environment variables for every Service in the same namespace. If there's a service named schema-registry, Kubernetes adds:

SCHEMA_REGISTRY_PORT=tcp://10.100.200.50:8081
SCHEMA_REGISTRY_PORT_8081_TCP=tcp://10.100.200.50:8081
SCHEMA_REGISTRY_PORT_8081_TCP_ADDR=10.100.200.50
Enter fullscreen mode Exit fullscreen mode

Confluent's startup scripts map any env var prefixed with SCHEMA_REGISTRY_ into the config file. So SCHEMA_REGISTRY_PORT=tcp://10.100.200.50:8081 gets translated to port=tcp://10.100.200.50:8081 — which is not a valid port number, causing the crash.

Fix: Add enableServiceLinks: false to the pod spec:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: schema-registry
spec:
  template:
    spec:
      enableServiceLinks: false   # prevents K8s service env var injection
      containers:
        - name: schema-registry
          image: your-ecr/schema-registry:7.5.0-msk
          # ...
Enter fullscreen mode Exit fullscreen mode

This tells Kubernetes to skip environment variable injection from service discovery for this pod. Your own explicitly-defined env vars (like SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS) are unaffected.

Same issue hits Kafka broker pods. Any Confluent image that uses the KAFKA_ or SCHEMA_REGISTRY_ env var convention will be crashed by K8s service link injection. Always set enableServiceLinks: false on these deployments.

Layer 2: The MSK IAM Auth JAR Doesn't Reach the JVM

Symptom: Schema registry starts but immediately fails to connect to MSK Serverless:

ERROR org.apache.kafka.clients.NetworkClient - [Producer clientId=...] 
Connection to node -1 (b-1.xxx.kafka.us-east-1.amazonaws.com/...) failed 
authentication due to: SSL handshake failed
Enter fullscreen mode Exit fullscreen mode

Or worse — it silently retries PLAINTEXT/SSL when what MSK Serverless requires is SASL/OAUTHBEARER.

Root cause: MSK Serverless only supports IAM (SASL/OAUTHBEARER). The Confluent 7.5 base image doesn't include the aws-msk-iam-auth JAR. You need to add it to the JVM classpath. But there's a catch: Confluent's entrypoint scripts call java directly and ignore the CLASSPATH environment variable.

Setting CLASSPATH=/path/to/aws-msk-iam-auth.jar in your deployment manifest does nothing.

Fix: Build a custom image that bakes the JAR directly into the directory Confluent's scripts scan:

FROM confluentinc/cp-schema-registry:7.5.0

# Confluent's kafka-run-class hardcodes /usr/share/java/schema-registry/*
# and does NOT honour the CLASSPATH env var — the JAR must be here.
USER root
ADD https://repo1.maven.org/maven2/software/amazon/msk/aws-msk-iam-auth/2.3.2/aws-msk-iam-auth-2.3.2-all.jar \
    /usr/share/java/schema-registry/aws-msk-iam-auth.jar
RUN chmod 644 /usr/share/java/schema-registry/aws-msk-iam-auth.jar
USER appuser
Enter fullscreen mode Exit fullscreen mode

Using ADD with the Maven Central URL means you don't need to manage the JAR locally — Docker pulls it at build time. The USER root / USER appuser sandwich is required because Confluent's base image runs as appuser (uid 1000) by default, but writing to /usr/share/java/schema-registry/ requires root.

Then configure the SASL settings in your env vars:

env:
  - name: SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS
    value: "b-1.xxx.kafka.us-east-1.amazonaws.com:9098"
  - name: SCHEMA_REGISTRY_KAFKASTORE_SECURITY_PROTOCOL
    value: "SASL_SSL"
  - name: SCHEMA_REGISTRY_KAFKASTORE_SASL_MECHANISM
    value: "AWS_MSK_IAM"
  - name: SCHEMA_REGISTRY_KAFKASTORE_SASL_JAAS_CONFIG
    value: "software.amazon.msk.auth.iam.IAMLoginModule required;"
  - name: SCHEMA_REGISTRY_KAFKASTORE_SASL_CLIENT_CALLBACK_HANDLER_CLASS
    value: "software.amazon.msk.auth.iam.IAMClientCallbackHandler"
Enter fullscreen mode Exit fullscreen mode

The key is that /usr/share/java/schema-registry/ is the directory Confluent's launch scripts automatically add to the JVM classpath via wildcard (-cp /usr/share/java/schema-registry/*). Putting your JAR there means it's picked up without any script modification.

Why not CLASSPATH? The Confluent Docker image uses launch scripts that construct the java command internally. The CLASSPATH env var is only used if java is invoked without explicit -cp/-classpath flags. Confluent sets its own -cp, so your CLASSPATH override is ignored.

Layer 3: The Ensure Script Fails Pre-Flight Checks

Symptom: Pod starts, gets past the JAR loading, but then hangs for minutes before failing with a connectivity error — even though MSK is accessible. Or it fails immediately with an obscure error from the Confluent ensure script.

Root cause: The Confluent Docker image runs /etc/confluent/docker/ensure as a pre-flight check before launching Schema Registry. This script attempts to connect to Kafka to verify bootstrap connectivity. In the EKS environment, it may run before your IRSA credentials are fully available, or it uses a different code path that doesn't pick up the IAM auth callback handler.

Fix: Override the ensure script with a no-op via a ConfigMap subPath mount:

# ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
  name: schema-registry-scripts
data:
  ensure: |
    #!/bin/bash
    # No-op: pre-flight connectivity check skipped in EKS.
    # IRSA credentials are injected at runtime; the ensure script
    # runs before the IAM token exchange is complete.
    exit 0
Enter fullscreen mode Exit fullscreen mode 
# In the Deployment spec
volumeMounts:
  - name: schema-registry-scripts
    mountPath: /etc/confluent/docker/ensure
    subPath: ensure
volumes:
  - name: schema-registry-scripts
    configMap:
      name: schema-registry-scripts
      defaultMode: 0755
Enter fullscreen mode Exit fullscreen mode

The subPath: ensure mount replaces only that single file in the container's filesystem — the rest of /etc/confluent/docker/ is untouched. The defaultMode: 0755 ensures the script is executable.

Is this safe? The ensure script's job is to verify connectivity before startup. Since Schema Registry itself will retry connectivity on startup with proper retry/backoff logic, skipping the pre-flight check doesn't lose safety — it just removes a redundant check that doesn't understand your auth setup anyway.

The Complete Custom Image 

FROM confluentinc/cp-schema-registry:7.5.0

# Confluent's kafka-run-class hardcodes /usr/share/java/schema-registry/*
# and does NOT honour the CLASSPATH env var — the JAR must be here.
USER root
ADD https://repo1.maven.org/maven2/software/amazon/msk/aws-msk-iam-auth/2.3.2/aws-msk-iam-auth-2.3.2-all.jar \
    /usr/share/java/schema-registry/aws-msk-iam-auth.jar
RUN chmod 644 /usr/share/java/schema-registry/aws-msk-iam-auth.jar
USER appuser
Enter fullscreen mode Exit fullscreen mode

Build and push to ECR:

docker build -t <account-id>.dkr.ecr.us-east-1.amazonaws.com/ai-platform/schema-registry:7.5.0-msk \
  -f docker/schema-registry/Dockerfile docker/schema-registry/

aws ecr get-login-password --region us-east-1 | \
  docker login --username AWS --password-stdin <account-id>.dkr.ecr.us-east-1.amazonaws.com

docker push <account-id>.dkr.ecr.us-east-1.amazonaws.com/ai-platform/schema-registry:7.5.0-msk
Enter fullscreen mode Exit fullscreen mode

The Full Deployment Manifest 

apiVersion: apps/v1
kind: Deployment
metadata:
  name: schema-registry
  namespace: microservices
spec:
  replicas: 1
  strategy:
    type: Recreate   # rolling update with 2 concurrent pods is toxic on MSK consumer groups
  selector:
    matchLabels:
      app: schema-registry
  template:
    metadata:
      labels:
        app: schema-registry
    spec:
      enableServiceLinks: false   # Layer 1: prevents K8s service env var injection
      serviceAccountName: schema-registry-sa  # IRSA service account
      containers:
        - name: schema-registry
          image: <account-id>.dkr.ecr.us-east-1.amazonaws.com/ai-platform/schema-registry:7.5.0-msk
          ports:
            - containerPort: 8081
          env:
            - name: SCHEMA_REGISTRY_HOST_NAME
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: SCHEMA_REGISTRY_LISTENERS
              value: "http://0.0.0.0:8081"
            - name: SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS
              value: "b-1.xxx.kafka.us-east-1.amazonaws.com:9098"
            - name: SCHEMA_REGISTRY_KAFKASTORE_SECURITY_PROTOCOL
              value: "SASL_SSL"
            - name: SCHEMA_REGISTRY_KAFKASTORE_SASL_MECHANISM
              value: "AWS_MSK_IAM"
            - name: SCHEMA_REGISTRY_KAFKASTORE_SASL_JAAS_CONFIG
              value: "software.amazon.msk.auth.iam.IAMLoginModule required;"
            - name: SCHEMA_REGISTRY_KAFKASTORE_SASL_CLIENT_CALLBACK_HANDLER_CLASS
              value: "software.amazon.msk.auth.iam.IAMClientCallbackHandler"
          startupProbe:
            httpGet:
              path: /subjects
              port: 8081
            failureThreshold: 30
            periodSeconds: 10   # 300s total — _schemas topic sync can take a while
          livenessProbe:
            httpGet:
              path: /subjects
              port: 8081
            initialDelaySeconds: 60
            periodSeconds: 30
          volumeMounts:
            - name: schema-registry-scripts
              mountPath: /etc/confluent/docker/ensure
              subPath: ensure   # Layer 3: override ensure script
      volumes:
        - name: schema-registry-scripts
          configMap:
            name: schema-registry-scripts
            defaultMode: 0755
Enter fullscreen mode Exit fullscreen mode

Why Each Layer Was Non-Obvious

Layer What made it hard to spot
enableServiceLinks: false The error message blames SCHEMA_REGISTRY_PORT being invalid — doesn't mention Kubernetes service injection at all
Custom Dockerfile with JAR The CLASSPATH env var looks like it should work; the fact that Confluent ignores it is undocumented
No-op ensure script The ensure failure is a pre-flight check exit, which looks like a connection error — masks that IRSA credentials weren't ready

The real lesson: when you combine a third-party container image with a managed cloud service and a container orchestrator, each one adds an invisible layer of convention. The debug process is peeling them back one at a time — which is exactly why the error messages never tell you the right thing.

The Smoke Test

After all three layers are resolved, Schema Registry starts clean and the MSK Serverless connection goes through IAM:

# From inside the EKS cluster
kubectl exec -n microservices deploy/user-service -- \
  curl -s http://schema-registry:8081/subjects
# -> []  (empty on first boot, connection confirmed)
Enter fullscreen mode Exit fullscreen mode

And the full path: POST /api/v1/users through the ALB → user-service → MSK Serverless (via Schema Registry for Avro serialization) → notification-service consumer. HTTP 201.

Have you run into similar layering conflicts when combining third-party container images with managed cloud services on Kubernetes? Which of these three layers caught you most off-guard?

Top comments (0)