DEV Community: Lee Yao

Why Spark Couldn't Read from Kafka: A Real Debugging Journey Across PySpark, Hadoop, Docker, and Kafka

Lee Yao — Mon, 10 Aug 2026 00:14:45 +0000

I thought this would be a simple task.

I already had a Python Kafka producer running. Kafka was up in Docker. The topic existed, and I could send a message into it successfully.

The next step sounded straightforward:

Python Producer
      ↓
Kafka
      ↓
Spark Structured Streaming

All I wanted Spark to do was read a JSON message from a Kafka topic.

Instead, I ran into one error after another.

At first, it looked like one problem:

Spark cannot read Kafka.

It was not one problem.

It turned into a chain of failures across several different layers:

Python / PySpark
      ↓
Spark runtime
      ↓
Kafka connector
      ↓
Hadoop / Windows
      ↓
Docker
      ↓
Kafka networking
      ↓
Ivy dependency resolution

The useful part of this experience was not any single fix. It was learning how to separate the layers and stop treating every error as a problem in my Python code.

This is the full debugging path.

What I Was Building

This was part of an financial data engineering project.

The batch side of the project already looked roughly like this:

Financial Data Source
      ↓
Python ingestion
      ↓
AWS S3
      ↓
Snowflake
      ↓
dbt
      ↓
Financial anomaly models

I wanted to add a streaming extension for newly arriving financial events.

For the first version, I kept it intentionally simple:

Python Kafka Producer
      ↓
Kafka topic: financial_events
      ↓
Spark Structured Streaming

The producer sent a simulated financial event:

{
  "company_id": "COMPANY_001",
  "company_name": "Sample Company",
  "report_type": "quarterly_report",
  "reporting_date": "2026-08-08",
  "event_id": "FIN-20260808-001",
  "source": "simulated_financial_event"
}

Kafka accepted the message successfully.

I could even read it with Kafka's console consumer.

So Kafka itself was working.

Then Spark entered the picture.

Failure #1: PySpark Worked, but `spark-submit` Didn't

I installed PySpark:

pip install pyspark

Then I installed Java 17 and verified it:

java -version

After reopening my terminal, Java was available.

I tested Spark directly through Python:

python -c "from pyspark.sql import SparkSession; spark = SparkSession.builder.master('local[*]').appName('sec-test').getOrCreate(); print(spark.version); spark.stop()"

Spark started and returned a version.

That told me something important:

Python
   ↓
PySpark
   ↓
JVM
   ↓
Spark

was basically working.

So I moved on and tried to run the streaming job with spark-submit.

Observed error:

Could not find valid SPARK_HOME
PySpark was not found in your Python environment
Failed to find Spark jars directory

My first reaction was that PySpark might not have installed correctly.

But that did not match the evidence.

Python had already imported PySpark and created a SparkSession successfully.

So this was not simply:

PySpark is broken.

The problem was the Windows spark-submit launcher and how it was discovering the pip-installed Spark environment.

That distinction mattered.

Instead of spending time changing the application code, I temporarily bypassed the launcher and tried running the Python script directly:

python scripts/spark_kafka_stream.py

That got me past the first layer.

And immediately exposed the second one.

Failure #2: Spark Didn't Know What `kafka` Meant

My streaming code contained something like:

kafka_df = (
    spark.readStream
    .format("kafka")
    .option("kafka.bootstrap.servers", "localhost:9092")
    .option("subscribe", "financial_events")
    .load()
)

Observed error:

Failed to find data source: kafka

Why this error mattered:

This is an important error to understand.

It does not mean:

Spark tried to connect to Kafka and Kafka rejected the connection.

It means something earlier than that.

Spark did not have the Kafka data source implementation available at all.

In other words:

Spark
  ↓
.format("kafka")
  ↓
"Where is the Kafka connector?"

The Python pyspark package alone is not enough.

Spark's Kafka integration is implemented through JVM-side connector JARs.

For Structured Streaming, I needed:

spark-sql-kafka-0-10

The dependency I initially used was:

org.apache.spark:spark-sql-kafka-0-10_2.13:4.2.0

The _2.13 part matters too. It refers to the Scala binary version the package was built for.

This was one of those moments where the relationship between Python, Spark, Scala, and the JVM became much more concrete.

I was writing Python, but underneath it the stack looked more like:

Python
   ↓
PySpark API
   ↓
Spark JVM
   ↓
Scala/JVM Kafka connector
   ↓
Kafka client

After adding the connector, Spark successfully resolved and downloaded the Kafka package and its dependencies.

That looked promising.

Then Spark failed again.

Failure #3: The Kafka Connector Worked — Windows Did Not

Once the Kafka JARs were available, I expected the application to start.

Observed error:

Did not find winutils.exe

followed by:

HADOOP_HOME and hadoop.home.dir are unset

and eventually:

ERROR SparkContext: Error initializing SparkContext

How I diagnosed it:

The stack trace was the key.

It showed calls such as:

org.apache.hadoop.util.Shell.getWinUtilsPath
org.apache.hadoop.fs.FileUtil.chmod
org.apache.spark.util.Utils.fetchFile

and eventually surfaced back in Python as a Py4JJavaError.

This was a good reminder not to stop reading at the top-level exception.

If I had only looked at:

Py4JJavaError

I might have assumed I had a Python-to-Java communication problem.

But the real cause was deeper in the stack trace:

Caused by:
HADOOP_HOME and hadoop.home.dir are unset

Spark itself is not Hadoop, but Spark uses Hadoop libraries for a number of filesystem-related operations.

On Windows, some of those Hadoop utilities expect Windows-specific support such as winutils.exe.

Decision point:

At this point I had two choices.

I could keep Spark running directly on Windows and start configuring:

HADOOP_HOME
winutils.exe
Windows Hadoop binaries
PATH

Or I could ask a more architectural question:

Should I really spend time making a distributed data processing engine behave like a native Windows application?

For this project, I chose not to.

Spark, Kafka, Airflow, and most of the surrounding data engineering ecosystem are much more naturally deployed on Linux.

So instead of patching my local Windows environment, I moved Spark into Docker.

The architecture became:

Windows Host
   |
   |-- Python Kafka Producer
   |
Docker
   |
   |-- Kafka
   |
   |-- Spark

I pulled the official Spark image and added it to Docker Compose.

Inside the container, Spark reported:

Spark version 4.0.1
Scala version 2.13.16
OpenJDK 17

Now I had a clean Linux Spark environment.

The Windows/Hadoop compatibility problem was gone.

But Spark still could not simply use my existing Kafka configuration.

That brought me to the most important networking issue in this whole debugging session.

Failure #4: `localhost` Meant the Wrong Machine

My Kafka broker was originally configured like this:

KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092

That worked perfectly for my Python producer running on Windows:

Windows Python
      ↓
localhost:9092
      ↓
Docker port mapping
      ↓
Kafka

But Spark was no longer on Windows.

Spark was now inside another Docker container.

Inside the Spark container:

localhost

means:

the Spark container itself

It does not mean the Windows host, and it does not automatically mean the Kafka container.

This seems obvious once you see it, but it is one of the easiest Docker networking mistakes to make.

Docker Compose provides DNS between services.

So I checked whether Spark could resolve the Kafka service:

docker exec sec-spark getent hosts kafka

It returned an internal Docker IP for kafka.

That confirmed Docker service discovery was working.

I then tested the internal Kafka port:

docker exec sec-spark bash -c "echo > /dev/tcp/kafka/29092 && echo KAFKA_PORT_OK"

and got:

KAFKA_PORT_OK

So Spark could reach Kafka over the Docker network.

However, Kafka networking has another detail that makes this more interesting.

Kafka's `advertised.listeners` Problem

Kafka is different from a simple HTTP service.

A Kafka client does not just open one TCP connection to the address you give it and stay there forever.

The first address is the bootstrap address.

For example:

kafka.bootstrap.servers=kafka:29092

The client connects to that broker and asks for cluster metadata.

Kafka then tells the client which broker addresses it should use afterward.

Those addresses come from:

advertised.listeners

This means a configuration can behave like this:

Spark
  ↓
connects successfully to kafka:29092
  ↓
Kafka responds:
"Use localhost:9092 for this broker"
  ↓
Spark tries localhost:9092
  ↓
localhost = Spark container
  ↓
connection fails

So simply making the first socket connection work is not enough.

The broker must advertise an address that the client can actually reach from its own network.

I had two different kinds of clients:

Client 1:
Python Producer on Windows

Client 2:
Spark inside Docker

They needed two different addresses.

I changed Kafka to expose two listeners:

KAFKA_LISTENERS: PLAINTEXT_HOST://:9092,PLAINTEXT_DOCKER://:29092,CONTROLLER://:9093

KAFKA_ADVERTISED_LISTENERS: PLAINTEXT_HOST://localhost:9092,PLAINTEXT_DOCKER://kafka:29092

KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT,PLAINTEXT_DOCKER:PLAINTEXT

KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT_DOCKER

Now the network paths were clear.

From Windows:

Python Producer
      ↓
localhost:9092
      ↓
Kafka

From Docker:

Spark
      ↓
kafka:29092
      ↓
Kafka

This was probably the most valuable part of the whole troubleshooting process.

Before this, I understood Docker ports and Kafka bootstrap servers separately.

After debugging it, I understood why Kafka's advertised.listeners has to describe the broker from the client's point of view.

That is a much more useful mental model.

Failure #5: Spark Could Reach Kafka, but Ivy Could Not Write Its Cache

At this point I finally had:

Spark in Linux Docker
      ↓
Docker DNS
      ↓
Kafka

The network was working.

I ran Spark with the Kafka connector:

/opt/spark/bin/spark-submit \
  --packages org.apache.spark:spark-sql-kafka-0-10_2.13:4.0.1 \
  /opt/sec_project/scripts/spark_kafka_stream.py

And got another failure.

Observed error:

/nonexistent/.ivy2.5.2/cache/...
No such file or directory

The log showed:

Ivy Default Cache set to:
/nonexistent/.ivy2.5.2/cache

Then Java threw a FileNotFoundException.

This was not a Kafka connectivity problem.

It was not a Spark streaming problem either.

It was a dependency management problem.

What Ivy Was Doing

When Spark sees:

--packages org.apache.spark:spark-sql-kafka-0-10_2.13:4.0.1

it needs to resolve and download that package and its transitive dependencies.

Spark uses Ivy/Maven dependency resolution for this.

A simplified view is:

spark-submit
      ↓
Ivy
      ↓
Maven Central
      ↓
Kafka connector JARs
      ↓
local cache

Normally Ivy uses a cache under the user's home directory.

But inside this Spark container, the effective home path led Ivy to:

/nonexistent/.ivy2.5.2

That directory was not usable.

So the connector existed on Maven Central, the network worked, and the package name was correct — but Spark still could not start because Ivy could not save its dependency metadata.

Fix:

The fix was simple once the real layer was identified.

I explicitly gave Ivy a writable directory:

--conf spark.jars.ivy=/tmp/.ivy2

The command became:

docker exec -it sec-spark /opt/spark/bin/spark-submit \
  --conf spark.jars.ivy=/tmp/.ivy2 \
  --packages org.apache.spark:spark-sql-kafka-0-10_2.13:4.0.1 \
  /opt/sec_project/scripts/spark_kafka_stream.py

This time the log changed from:

Ivy Default Cache set to:
/nonexistent/.ivy2.5.2/cache

to:

Ivy Default Cache set to:
/tmp/.ivy2/cache

Spark successfully downloaded and loaded the Kafka connector, Kafka client, Hadoop client libraries, and related dependencies.

Then I finally saw:

Running Spark version 4.0.1
OS info Linux
Java version 17.0.16
Submitted application: SEC Kafka Stream

That was the first time the entire runtime had actually made it through startup.

Then I Got `Batch: 0`

Spark started the streaming query and printed:

Batch: 0

The table was empty.

After everything I had just fixed, an empty table was almost suspicious.

But this time there was no error.

That difference mattered.

The streaming query was alive.

The Kafka connector was loaded.

The broker was reachable.

The topic existed.

There simply was no message in the newly recreated topic yet.

Structured Streaming was doing exactly what it was supposed to do.

It had created a micro-batch, found no new Kafka records, and waited.

So I left the Spark terminal running.

Then I opened another terminal and ran:

python scripts/kafka_producer.py

The producer sent the financial event.

I went back to the Spark terminal.

And there it was:

Batch: 1

with:

message_key = COMPANY_001
topic       = financial_events
partition   = 0
offset      = 0

and the full financial event JSON.

That was the moment the pipeline actually worked:

Python Producer
      ↓
Kafka
      ↓
Spark Structured Streaming
      ↓
Micro-batch
      ↓
financial event

What Looked Like One Problem Was Actually Several

The original symptom was:

Spark cannot read Kafka.

But that description was almost useless.

The actual failures were distributed across different layers:

Runtime / launcher — spark-submit could not correctly discover the pip-installed local Spark environment.
Dependency — Spark did not have the Kafka datasource connector.
OS / Hadoop compatibility — Windows Spark execution hit HADOOP_HOME / winutils.exe behavior.
Container networking / Kafka metadata — localhost meant different things on the host and inside Docker, while Kafka needed to advertise an address each client could actually reach.
Dependency cache — Ivy tried to write into /nonexistent, preventing the Kafka connector from loading.

And after all five were fixed, Batch: 0 was not another failure.

It was simply an empty streaming batch.

Then Batch: 1 proved the end-to-end integration.

The Biggest Lesson: Debug by Layer

The most useful thing I learned was to stop asking:

Why isn't Spark reading Kafka?

That question is too broad.

A better troubleshooting flow is:

Can Spark start?
      ↓
Is the JVM healthy?
      ↓
Does Spark know the Kafka datasource?
      ↓
Is the connector loaded?
      ↓
Can the Spark container resolve Kafka?
      ↓
Is the Kafka port reachable?
      ↓
Is the broker advertising the right address?
      ↓
Does the topic exist?
      ↓
Are there records after the current offset?
      ↓
Is the streaming query actually running?

Each answer eliminates an entire class of problems.

That is much faster than randomly changing application code.

Read the Root Cause, Not Just the Last Exception

Another lesson was how to read stack traces.

For example:

Py4JJavaError

was not the real problem.

Further down:

Caused by:
HADOOP_HOME and hadoop.home.dir are unset

was the useful information.

Likewise:

FileNotFoundException

did not mean my Python script was missing.

The path:

/nonexistent/.ivy2.5.2/cache

showed that the failure belonged to Spark's dependency cache.

The pattern I want to keep using is:

Read the error
      ↓
Find the first meaningful "Caused by"
      ↓
Identify the failing layer
      ↓
Test that layer separately
      ↓
Only then change the configuration or code

That sounds simple.

In practice, it changes troubleshooting completely.

The Final Local Architecture

The working development setup now looks like this:

Windows Host
│
├── Python Kafka Producer
│       │
│       └── localhost:9092
│
└── Docker Compose Network
        │
        ├── Kafka
        │     ├── Host listener:   localhost:9092
        │     └── Docker listener: kafka:29092
        │
        └── Spark
              │
              └── Spark Structured Streaming
                    ↓
                 financial_events

The next step is to parse the Kafka JSON payload into typed Spark columns and continue toward Snowflake.

But getting one message from the Python producer all the way into a Spark micro-batch was already a useful engineering exercise.

I started the day thinking I was debugging one Kafka connection.

I ended up debugging five different systems that happened to fail on the same path.

And that is probably the part of distributed data engineering that tutorials don't show often enough.

Suggested DEV Tags

#dataengineering #apachekafka #apachespark #docker

Debugging a Multi-Container Airflow Pipeline: Kafka Network Isolation and the YAML Indentation Trap

Lee Yao — Mon, 11 May 2026 06:09:41 +0000

After getting Kafka, Spark, and Snowflake all working individually, I thought wiring them together in Airflow would be the easy part. It was not. What followed was an afternoon of containers failing silently, misleading error messages, and one of those bugs where the fix is two characters but finding it takes two hours.

This post covers the two main issues I hit when building an Airflow DAG to orchestrate a CMS Medicare streaming pipeline. Both problems are worth understanding because they'll come up any time you try to connect services across multiple Docker Compose projects.

The Setup

The pipeline I was building looks like this:

CMS API → Kafka Producer → Kafka → Spark Streaming → Snowflake → dbt → dbt test

All orchestrated by a single Airflow DAG with four tasks:

run_cms_producer >> run_spark_streaming >> run_dbt_models >> run_dbt_tests

The tricky part: Kafka and Spark live in one Docker Compose project (CMS_project), and Airflow lives in a completely separate Docker Compose project (DOT_project/airflow). Two separate projects, two separate Docker networks, and a whole set of problems that come with that.

Problem 1: `NoBrokersAvailable` — Kafka Network Isolation

What the error looked like

The first task, run_cms_producer, kept failing immediately. The error in the Airflow task log:

kafka.errors.NoBrokersAvailable: NoBrokersAvailable

This error means the Kafka client tried to connect to the broker, got nothing back, and gave up. The fix seems obvious: check the broker address. I did. It looked right. The container names were correct. The ports were open. And yet — nothing.

Why it happened: Docker Compose network isolation

When you run docker-compose up in a directory, Docker creates a private network for that project. By default, it's named after the folder: cms_project_default for CMS_project, and airflow_default for DOT_project/airflow.

Containers in cms_project_default can talk to each other freely. Containers in airflow_default can talk to each other freely. But containers in different projects cannot reach each other by default — the networks are completely separate.

The first fix I tried was adding Airflow's containers to the CMS network:

# In Airflow's docker-compose.yml
networks:
  default:
    name: airflow_default
  cms_network:
    external: true
    name: cms_project_default

This seemed to work — docker inspect confirmed the Airflow scheduler was in both networks. But the NoBrokersAvailable error kept showing up.

The real problem: how Kafka advertises itself

Here's where it gets less obvious. Kafka doesn't just sit there and accept connections. When a client first connects, Kafka sends back a list of addresses the client should use for all future communication. This list is called the advertised listeners.

In my CMS_project/docker-compose.yml, the Kafka configuration looked like this:

KAFKA_LISTENERS: INTERNAL://0.0.0.0:29092,EXTERNAL://0.0.0.0:9092
KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:29092,EXTERNAL://localhost:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL

This sets up two listeners:

INTERNAL on port 29092 — intended for containers inside the same Docker Compose project
EXTERNAL on port 9092 — intended for connections from the host machine (your Windows terminal)

When the Airflow container tried to connect to kafka:29092, Kafka initially accepted the connection (because Airflow was now on the same Docker network). But then Kafka sent back its advertised address for that listener: kafka:29092. When the Airflow container tried to use that address for subsequent requests, it hit the same broker again — which was fine.

So why was it still failing?

The answer is subtle. Even though Airflow was on the cms_project_default network, Kafka was treating it as an internal client. The INTERNAL listener was designed for containers inside cms_project — and those containers have a specific network configuration that resolves kafka to the right IP. Airflow's container, being from a different Compose project, had slightly different network resolution behavior. The connection would time out at the API version check stage, which is the very first thing the Kafka client does before it can do anything else.

The fix: a third dedicated listener

The cleanest solution is to add a third listener specifically for cross-project Docker container connections:

KAFKA_LISTENERS: INTERNAL://0.0.0.0:29092,EXTERNAL://0.0.0.0:9092,DOCKER://0.0.0.0:39092
KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:29092,EXTERNAL://localhost:9092,DOCKER://kafka:39092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT,DOCKER:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL

And expose the new port:

ports:
  - "9092:9092"
  - "39092:39092"

Now there are three separate listeners, each for a different kind of client:

Listener	Port	Who uses it
`INTERNAL`	29092	Containers inside `cms_project` (Spark, etc.)
`EXTERNAL`	9092	Host machine (your terminal, local Python scripts)
`DOCKER`	39092	Containers from other Docker Compose projects (Airflow)

In the Airflow DAG, the producer task now uses the DOCKER listener:

run_producer = BashOperator(
    task_id="run_cms_producer",
    bash_command="KAFKA_BROKER=kafka:39092 python /opt/airflow/cms_producer.py",
    execution_timeout=timedelta(minutes=15),
)

And in cms_producer.py, the broker address reads from the environment variable:

import os
KAFKA_BROKER = os.environ.get("KAFKA_BROKER", "localhost:9092")

After this change, the connection worked immediately.

How to debug Kafka connectivity issues

Before changing any configuration, verify the actual connection from inside the container that's having trouble. Don't guess based on port numbers or network diagrams:

# Copy a test script into the Airflow container
docker cp test_kafka.py airflow-airflow-scheduler-1:/tmp/test_kafka.py

# Run it from inside the container
docker exec airflow-airflow-scheduler-1 python /tmp/test_kafka.py

Where test_kafka.py contains:

from kafka import KafkaConsumer
c = KafkaConsumer(bootstrap_servers='kafka:39092', consumer_timeout_ms=3000)
print('Connected OK')

This tells you immediately whether the network path is open, without having to trigger a full DAG run and wait for logs.

Problem 2: The YAML Indentation Trap

What the error looked like

Once the Kafka connection was fixed, run_cms_producer and run_spark_streaming started succeeding. But run_dbt_models failed with:

Runtime Error
  Could not find profile named 'cms_dbt'

dbt couldn't find the profile I had added. I checked the file — cms_dbt was right there in profiles.yml. Or so I thought.

Why it happened: YAML nesting is invisible

Here's what the profiles.yml actually contained:

dbt_dot_flights:
  outputs:
    dev:
      type: snowflake
      account: "{{ env_var('SNOWFLAKE_ACCOUNT') }}"
      # ... other fields ...
  target: dev
  cms_dbt:                    # ← THIS IS THE PROBLEM
    target: dev
    outputs:
      dev:
        type: snowflake
        account: FZPFTPF-LOB40082
        # ...

cms_dbt was indented under dbt_dot_flights. In YAML, indentation defines the structure — so instead of being a separate top-level profile, cms_dbt became a field inside the dbt_dot_flights profile. dbt looked for a top-level key called cms_dbt, didn't find one, and reported that the profile was missing.

What the file should have looked like:

dbt_dot_flights:
  target: dev
  outputs:
    dev:
      type: snowflake
      # ...

cms_dbt:                      # ← Top-level, no indentation
  target: dev
  outputs:
    dev:
      type: snowflake
      account: FZPFTPF-LOB40082
      # ...

The difference is two spaces of indentation, which is completely invisible when you're skimming a file looking for a specific key.

Why YAML indentation bugs are so hard to catch

Most syntax errors give you a loud, obvious error message. YAML indentation errors often don't — the file is syntactically valid, it just means something different from what you intended. A misindented block doesn't cause a parse error; it causes incorrect data structure, which only surfaces as a logical error later when something tries to use that data.

In this case, profiles.yml parsed without any complaint. dbt read it, built the profile registry, and simply didn't find cms_dbt as a top-level key — because it wasn't one.

The quick way to catch YAML structure bugs

Before you commit any YAML file, dump it as parsed Python to see exactly what structure it produces:

python -c "import yaml; import json; print(json.dumps(yaml.safe_load(open('profiles.yml')), indent=2))"

This shows you the actual data structure the parser sees, not what you think you wrote. A misindented block shows up immediately as a nested key instead of a top-level key.

For profiles.yml specifically, the top-level keys should be the profile names. If you run the command above and see cms_dbt nested inside dbt_dot_flights instead of at the top level, you've found the bug.

The second dbt failure: a typo in `dbt_project.yml`

After fixing the indentation, run_dbt_models failed again with a different error:

No materialization 'table2' was found for adapter snowflake!

This one was a straightforward typo. In dbt_project.yml:

models:
  cms_dbt:
    marts:
      +materialized: table2    # ← Should be 'table'

table2 is not a valid dbt materialization type. The valid options are view, table, incremental, and ephemeral. One character difference, and dbt can't find the materialization.

The Debugging Workflow That Actually Works

After going through all of this, here's the approach I'd use from the start next time:

Step 1: Test network connectivity before writing any DAG code.

Don't assume two containers can talk to each other just because they're on the same network. Actually test it from inside the source container before you do anything else.

docker exec <source-container> python -c "
from kafka import KafkaConsumer
c = KafkaConsumer(bootstrap_servers='<target>:<port>', consumer_timeout_ms=3000)
print('OK')
"

Step 2: Validate YAML files immediately after editing.

Any time you touch a YAML file, run the parser dump to verify the structure:

python -c "import yaml; import json; print(json.dumps(yaml.safe_load(open('yourfile.yml')), indent=2))"

Two seconds of checking saves an hour of debugging.

Step 3: Read the Airflow task logs, not the DAG-level logs.

When a task fails, the DAG-level logs just say "failed." The actual error is in the task-specific log file. In the Airflow UI: click the task (the colored square) → click "Logs." That's where the real error message is.

Step 4: For Kafka specifically, understand the three listener types.

If you're running Kafka in Docker and connecting from multiple environments (host machine, same-project containers, cross-project containers), you need a separate listener for each. One listener cannot serve all three use cases cleanly.

Summary

Problem	Root Cause	Fix
`NoBrokersAvailable` from Airflow	Two Docker Compose projects on separate networks; Kafka's `INTERNAL` listener not designed for cross-project connections	Add a third `DOCKER` listener on a dedicated port (39092) for cross-project container access
`Could not find profile 'cms_dbt'`	YAML indentation error — `cms_dbt` was nested inside `dbt_dot_flights` instead of being a top-level key	Fix indentation; validate with `yaml.safe_load`
`No materialization 'table2'`	Typo in `dbt_project.yml`	Change `table2` to `table`

The Kafka network issue is the one worth spending time understanding. Docker Compose network isolation is intentional and useful, but it creates real headaches when you need services from different projects to communicate. Knowing that Kafka has separate listener types — and that you can add custom ones — gives you a clean, explicit solution rather than hacks like network_mode: host.

The YAML bugs are embarrassing in retrospect, but they're also genuinely easy to miss. The fix is always the same: don't trust your eyes on YAML indentation, trust the parser.

Why My Spark Container Keeps Exiting — Docker PID 1 and the Daemon Trap

Lee Yao — Thu, 07 May 2026 04:41:08 +0000

I spent an embarrassing amount of time staring at my terminal, watching Spark containers start and immediately die. Three different attempts, three different failure modes, all in the same afternoon. If you're setting up Spark inside Docker and your container just... vanishes, this post is for you.

The Setup

I'm building a CMS Medicare streaming pipeline — pulling hospital charge data from the CMS public API, pushing it through Kafka, processing it with Spark Structured Streaming, and landing the results in Snowflake. The whole stack runs in Docker Compose. Kafka and ZooKeeper came up without a hitch. Spark did not.

Here's what my docker-compose.yml looked like at the start:

services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.4.0
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181

  kafka:
    image: confluentinc/cp-kafka:7.4.0
    depends_on: [zookeeper]
    ports:
      - "9092:9092"
    environment:
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

  spark:
    image: bitnami/spark:3.5
    depends_on: [kafka]
    environment:
      SPARK_MODE: master

  spark-worker:
    image: bitnami/spark:3.5
    depends_on: [spark]
    environment:
      SPARK_MODE: worker
      SPARK_MASTER_URL: spark://spark:7077

Looked reasonable enough. It wasn't.

Attempt 1 — The Image That No Longer Exists

Error response from daemon: failed to resolve reference
"docker.io/bitnami/spark:3.5": not found

bitnami/spark:3.5 had been pulled from Docker Hub. I tried 3.5.3. Gone. Tried bitnami/spark:3. Also gone. The entire Bitnami Spark image line had been removed with no notice.

This is the first thing worth remembering before we even get to the real problem: third-party images on Docker Hub can disappear at any time. There is no deprecation warning, no migration guide. For anything that needs to be reproducible, you either pin to a verified digest or mirror the image in a private registry.

I switched to the Apache official image: apache/spark:3.5.1-python3. That one pulled fine.

Attempt 2 — Wrong Environment Variables

I updated the image name but kept the same environment variable:

spark:
  image: apache/spark:3.5.1-python3
  environment:
    SPARK_MODE: master

docker-compose up -d reported all containers as "Started." But docker ps only showed two running — Kafka and ZooKeeper. The Spark containers had already exited.

The problem: SPARK_MODE is a Bitnami-specific environment variable. The Apache official image has never heard of it.

Bitnami's image ships with a custom entrypoint script that reads SPARK_MODE and decides whether to launch a master or worker. It's a convenience layer Bitnami built on top of vanilla Spark. The Apache official image has none of this. Its default entrypoint (/opt/entrypoint.sh) simply executes whatever command you pass in. If you don't pass a meaningful command, it finishes and exits.

The lesson: switching between images from different publishers is not just swapping the image: field. Different publishers package the same software with different entrypoints, different environment variables, and different directory layouts. Before you can use an image correctly, you need to understand how that specific image expects to be started.

Attempt 3 — The Real Trap: `start-master.sh`

Spark comes bundled with start-master.sh. That seems like the right tool:

spark:
  image: apache/spark:3.5.1-python3
  command: /opt/spark/sbin/start-master.sh

Same result. "Started." No Spark container.

The container was starting. Spark Master was launching. And then everything was shutting down within a fraction of a second. To understand why, you need to know one foundational Docker rule.

The Core Rule: Docker Containers Live and Die with PID 1

Every container has a main process — specified by CMD, ENTRYPOINT, or command in your Compose file. Inside the container, this process gets PID 1. When PID 1 exits, the container exits. No exceptions.

PID 1 is running  →  container is running
PID 1 exits       →  container exits immediately

Now look at what start-master.sh actually does internally (simplified):

#!/bin/bash
nohup java -cp $SPARK_CLASSPATH org.apache.spark.deploy.master.Master &
echo "Master started."
exit 0

See that &? It puts the Spark Master process into the background. The shell script (PID 1) spawns a child Java process, prints a message, and calls exit 0. The moment it does that, Docker kills the container and everything inside it — including the Spark Master that just started.

Here's the exact timeline:

t=0.0s  Container starts; PID 1 = start-master.sh (bash)
t=0.1s  Bash forks a Java process (Spark Master) into the background
t=0.2s  Bash script reaches exit 0 → PID 1 terminates
t=0.2s  Docker detects PID 1 exit → tears down the container
t=0.2s  The background Java process is killed along with it

Spark Master was alive for about 0.2 seconds.

start-master.sh was written for bare-metal servers and VMs, where you start a background daemon and the OS keeps it alive after the startup script exits. Docker doesn't work that way. Docker is watching PID 1 and only PID 1.

Why Kafka and ZooKeeper Didn't Have This Problem

Confluent's images use exec in their entrypoints:

exec kafka-server-start /etc/kafka/server.properties

In bash, exec replaces the current process with the specified command. The shell doesn't fork a child — it becomes Kafka. Kafka inherits PID 1, runs in the foreground, and blocks indefinitely.

Image	What PID 1 Does	Result
`cp-kafka`	`exec kafka-server-start` (foreground, blocking)	✅ Container stays alive
`cp-zookeeper`	`exec zookeeper-server-start` (foreground, blocking)	✅ Container stays alive
`apache/spark` + `start-master.sh`	Forks Java to background with `&`, script exits	❌ Container exits immediately

The entire difference: & versus exec.

Four Ways to Fix It

Fix A: `tail -f /dev/null`

spark:
  image: apache/spark:3.5.1-python3
  command: ["tail", "-f", "/dev/null"]
  volumes:
    - ./spark-apps:/opt/spark-apps

tail -f /dev/null watches a file that never gets new content. PID 1 blocks forever. Submit jobs via docker exec:

docker exec my-spark-container \
  /opt/spark/bin/spark-submit \
  /opt/spark-apps/my_job.py

Best for: local development, one-off job submission.

Fix B: Run the Spark Master Class Directly

command: >
  bash -c "
  /opt/spark/bin/spark-class org.apache.spark.deploy.master.Master
  --host spark --port 7077 --webui-port 8080
  "

Skips the wrapper script entirely. The Master process runs in the foreground as PID 1.

Best for: when you actually need a running Master/Worker cluster.

Fix C: Custom Entrypoint Script

#!/bin/bash
# custom-entrypoint.sh
/opt/spark/sbin/start-master.sh   # starts daemon in background
tail -f /opt/spark/logs/*         # blocks + streams logs to stdout

volumes:
  - ./custom-entrypoint.sh:/opt/custom-entrypoint.sh
command: bash /opt/custom-entrypoint.sh

Master auto-starts, container stays alive, and you get log output via docker logs.

Best for: when you want Spark to auto-start and want logs accessible.

Fix D: Use a Docker-Friendly Image

jupyter/pyspark-notebook handles all of this correctly out of the box. Their entrypoints are built around exec from the start.

Best for: quick prototyping. Tradeoff: you depend on a third party to keep the image available.

Summary

Docker containers exit when PID 1 exits. Always.
start-master.sh backgrounds Spark with & and exits — which kills the container.
Confluent's images use exec, making the service itself PID 1 and keeping the container alive.
The fix: ensure PID 1 is a foreground process that never returns.

Three patterns to spot in any startup script:

command & — background execution, PID 1 exits shortly after → container dies
exec command — replaces PID 1, container lives as long as the process does → container survives
nohup command & — classic daemon pattern, same problem as & in Docker → container dies

Docker containers are not VMs. On a VM, daemonizing a process and exiting the startup script is completely normal. In Docker, the startup script is the container. Once you internalize that, most "why does my container keep exiting" questions answer themselves.

DEV Community: Lee Yao

Why Spark Couldn't Read from Kafka: A Real Debugging Journey Across PySpark, Hadoop, Docker, and Kafka

What I Was Building

Failure #1: PySpark Worked, but spark-submit Didn't

Failure #2: Spark Didn't Know What kafka Meant

Failure #3: The Kafka Connector Worked — Windows Did Not

Failure #4: localhost Meant the Wrong Machine

Kafka's advertised.listeners Problem

Failure #5: Spark Could Reach Kafka, but Ivy Could Not Write Its Cache

What Ivy Was Doing

Then I Got Batch: 0

What Looked Like One Problem Was Actually Several

The Biggest Lesson: Debug by Layer

Read the Root Cause, Not Just the Last Exception

The Final Local Architecture

Suggested DEV Tags

Debugging a Multi-Container Airflow Pipeline: Kafka Network Isolation and the YAML Indentation Trap

The Setup

Problem 1: NoBrokersAvailable — Kafka Network Isolation

What the error looked like

Why it happened: Docker Compose network isolation

The real problem: how Kafka advertises itself

The fix: a third dedicated listener

How to debug Kafka connectivity issues

Problem 2: The YAML Indentation Trap

What the error looked like

Why it happened: YAML nesting is invisible

Why YAML indentation bugs are so hard to catch

The quick way to catch YAML structure bugs

The second dbt failure: a typo in dbt_project.yml

The Debugging Workflow That Actually Works

Summary

Why My Spark Container Keeps Exiting — Docker PID 1 and the Daemon Trap

The Setup

Attempt 1 — The Image That No Longer Exists

Attempt 2 — Wrong Environment Variables

Attempt 3 — The Real Trap: start-master.sh

The Core Rule: Docker Containers Live and Die with PID 1

Why Kafka and ZooKeeper Didn't Have This Problem

Four Ways to Fix It

Fix A: tail -f /dev/null

Fix B: Run the Spark Master Class Directly

Fix C: Custom Entrypoint Script

Fix D: Use a Docker-Friendly Image

Summary

Failure #1: PySpark Worked, but `spark-submit` Didn't

Failure #2: Spark Didn't Know What `kafka` Meant

Failure #4: `localhost` Meant the Wrong Machine

Kafka's `advertised.listeners` Problem

Then I Got `Batch: 0`

Problem 1: `NoBrokersAvailable` — Kafka Network Isolation

The second dbt failure: a typo in `dbt_project.yml`

Attempt 3 — The Real Trap: `start-master.sh`

Fix A: `tail -f /dev/null`