DEV Community

Lee Yao
Lee Yao

Posted on

Why Spark Couldn't Read from Kafka: A Real Debugging Journey Across PySpark, Hadoop, Docker, and Kafka

#dataengineering #apachekafka #apachespark #docker

I thought this would be a simple task.

I already had a Python Kafka producer running. Kafka was up in Docker. The topic existed, and I could send a message into it successfully.

The next step sounded straightforward:

Python Producer
      ↓
Kafka
      ↓
Spark Structured Streaming
Enter fullscreen mode Exit fullscreen mode

All I wanted Spark to do was read a JSON message from a Kafka topic.

Instead, I ran into one error after another.

At first, it looked like one problem:

Spark cannot read Kafka.

It was not one problem.

It turned into a chain of failures across several different layers:

Python / PySpark
      ↓
Spark runtime
      ↓
Kafka connector
      ↓
Hadoop / Windows
      ↓
Docker
      ↓
Kafka networking
      ↓
Ivy dependency resolution
Enter fullscreen mode Exit fullscreen mode

The useful part of this experience was not any single fix. It was learning how to separate the layers and stop treating every error as a problem in my Python code.

This is the full debugging path.

What I Was Building

This was part of an financial data engineering project.

The batch side of the project already looked roughly like this:

Financial Data Source
      ↓
Python ingestion
      ↓
AWS S3
      ↓
Snowflake
      ↓
dbt
      ↓
Financial anomaly models
Enter fullscreen mode Exit fullscreen mode

I wanted to add a streaming extension for newly arriving financial events.

For the first version, I kept it intentionally simple:

Python Kafka Producer
      ↓
Kafka topic: financial_events
      ↓
Spark Structured Streaming
Enter fullscreen mode Exit fullscreen mode

The producer sent a simulated financial event:

{
  "company_id": "COMPANY_001",
  "company_name": "Sample Company",
  "report_type": "quarterly_report",
  "reporting_date": "2026-08-08",
  "event_id": "FIN-20260808-001",
  "source": "simulated_financial_event"
}
Enter fullscreen mode Exit fullscreen mode

Kafka accepted the message successfully.

I could even read it with Kafka's console consumer.

So Kafka itself was working.

Then Spark entered the picture.

Failure #1: PySpark Worked, but spark-submit Didn't

I installed PySpark:

pip install pyspark
Enter fullscreen mode Exit fullscreen mode

Then I installed Java 17 and verified it:

java -version
Enter fullscreen mode Exit fullscreen mode

After reopening my terminal, Java was available.

I tested Spark directly through Python:

python -c "from pyspark.sql import SparkSession; spark = SparkSession.builder.master('local[*]').appName('sec-test').getOrCreate(); print(spark.version); spark.stop()"
Enter fullscreen mode Exit fullscreen mode

Spark started and returned a version.

That told me something important:

Python
   ↓
PySpark
   ↓
JVM
   ↓
Spark
Enter fullscreen mode Exit fullscreen mode

was basically working.

So I moved on and tried to run the streaming job with spark-submit.

Observed error:

Could not find valid SPARK_HOME
PySpark was not found in your Python environment
Failed to find Spark jars directory
Enter fullscreen mode Exit fullscreen mode

My first reaction was that PySpark might not have installed correctly.

But that did not match the evidence.

Python had already imported PySpark and created a SparkSession successfully.

So this was not simply:

PySpark is broken.

The problem was the Windows spark-submit launcher and how it was discovering the pip-installed Spark environment.

That distinction mattered.

Instead of spending time changing the application code, I temporarily bypassed the launcher and tried running the Python script directly:

python scripts/spark_kafka_stream.py
Enter fullscreen mode Exit fullscreen mode

That got me past the first layer.

And immediately exposed the second one.

Failure #2: Spark Didn't Know What kafka Meant

My streaming code contained something like:

kafka_df = (
    spark.readStream
    .format("kafka")
    .option("kafka.bootstrap.servers", "localhost:9092")
    .option("subscribe", "financial_events")
    .load()
)
Enter fullscreen mode Exit fullscreen mode

Observed error:

Failed to find data source: kafka
Enter fullscreen mode Exit fullscreen mode

Why this error mattered:

This is an important error to understand.

It does not mean:

Spark tried to connect to Kafka and Kafka rejected the connection.

It means something earlier than that.

Spark did not have the Kafka data source implementation available at all.

In other words:

Spark
  ↓
.format("kafka")
  ↓
"Where is the Kafka connector?"
Enter fullscreen mode Exit fullscreen mode

The Python pyspark package alone is not enough.

Spark's Kafka integration is implemented through JVM-side connector JARs.

For Structured Streaming, I needed:

spark-sql-kafka-0-10
Enter fullscreen mode Exit fullscreen mode

The dependency I initially used was:

org.apache.spark:spark-sql-kafka-0-10_2.13:4.2.0
Enter fullscreen mode Exit fullscreen mode

The _2.13 part matters too. It refers to the Scala binary version the package was built for.

This was one of those moments where the relationship between Python, Spark, Scala, and the JVM became much more concrete.

I was writing Python, but underneath it the stack looked more like:

Python
   ↓
PySpark API
   ↓
Spark JVM
   ↓
Scala/JVM Kafka connector
   ↓
Kafka client
Enter fullscreen mode Exit fullscreen mode

After adding the connector, Spark successfully resolved and downloaded the Kafka package and its dependencies.

That looked promising.

Then Spark failed again.

Failure #3: The Kafka Connector Worked — Windows Did Not

Once the Kafka JARs were available, I expected the application to start.

Observed error:

Did not find winutils.exe
Enter fullscreen mode Exit fullscreen mode

followed by:

HADOOP_HOME and hadoop.home.dir are unset
Enter fullscreen mode Exit fullscreen mode

and eventually:

ERROR SparkContext: Error initializing SparkContext
Enter fullscreen mode Exit fullscreen mode

How I diagnosed it:

The stack trace was the key.

It showed calls such as:

org.apache.hadoop.util.Shell.getWinUtilsPath
org.apache.hadoop.fs.FileUtil.chmod
org.apache.spark.util.Utils.fetchFile
Enter fullscreen mode Exit fullscreen mode

and eventually surfaced back in Python as a Py4JJavaError.

This was a good reminder not to stop reading at the top-level exception.

If I had only looked at:

Py4JJavaError
Enter fullscreen mode Exit fullscreen mode

I might have assumed I had a Python-to-Java communication problem.

But the real cause was deeper in the stack trace:

Caused by:
HADOOP_HOME and hadoop.home.dir are unset
Enter fullscreen mode Exit fullscreen mode

Spark itself is not Hadoop, but Spark uses Hadoop libraries for a number of filesystem-related operations.

On Windows, some of those Hadoop utilities expect Windows-specific support such as winutils.exe.

Decision point:

At this point I had two choices.

I could keep Spark running directly on Windows and start configuring:

HADOOP_HOME
winutils.exe
Windows Hadoop binaries
PATH
Enter fullscreen mode Exit fullscreen mode

Or I could ask a more architectural question:

Should I really spend time making a distributed data processing engine behave like a native Windows application?

For this project, I chose not to.

Spark, Kafka, Airflow, and most of the surrounding data engineering ecosystem are much more naturally deployed on Linux.

So instead of patching my local Windows environment, I moved Spark into Docker.

The architecture became:

Windows Host
   |
   |-- Python Kafka Producer
   |
Docker
   |
   |-- Kafka
   |
   |-- Spark
Enter fullscreen mode Exit fullscreen mode

I pulled the official Spark image and added it to Docker Compose.

Inside the container, Spark reported:

Spark version 4.0.1
Scala version 2.13.16
OpenJDK 17
Enter fullscreen mode Exit fullscreen mode

Now I had a clean Linux Spark environment.

The Windows/Hadoop compatibility problem was gone.

But Spark still could not simply use my existing Kafka configuration.

That brought me to the most important networking issue in this whole debugging session.

Failure #4: localhost Meant the Wrong Machine

My Kafka broker was originally configured like this:

KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
Enter fullscreen mode Exit fullscreen mode

That worked perfectly for my Python producer running on Windows:

Windows Python
      ↓
localhost:9092
      ↓
Docker port mapping
      ↓
Kafka
Enter fullscreen mode Exit fullscreen mode

But Spark was no longer on Windows.

Spark was now inside another Docker container.

Inside the Spark container:

localhost
Enter fullscreen mode Exit fullscreen mode

means:

the Spark container itself
Enter fullscreen mode Exit fullscreen mode

It does not mean the Windows host, and it does not automatically mean the Kafka container.

This seems obvious once you see it, but it is one of the easiest Docker networking mistakes to make.

Docker Compose provides DNS between services.

So I checked whether Spark could resolve the Kafka service:

docker exec sec-spark getent hosts kafka
Enter fullscreen mode Exit fullscreen mode

It returned an internal Docker IP for kafka.

That confirmed Docker service discovery was working.

I then tested the internal Kafka port:

docker exec sec-spark bash -c "echo > /dev/tcp/kafka/29092 && echo KAFKA_PORT_OK"
Enter fullscreen mode Exit fullscreen mode

and got:

KAFKA_PORT_OK
Enter fullscreen mode Exit fullscreen mode

So Spark could reach Kafka over the Docker network.

However, Kafka networking has another detail that makes this more interesting.

Kafka's advertised.listeners Problem

Kafka is different from a simple HTTP service.

A Kafka client does not just open one TCP connection to the address you give it and stay there forever.

The first address is the bootstrap address.

For example:

kafka.bootstrap.servers=kafka:29092
Enter fullscreen mode Exit fullscreen mode

The client connects to that broker and asks for cluster metadata.

Kafka then tells the client which broker addresses it should use afterward.

Those addresses come from:

advertised.listeners
Enter fullscreen mode Exit fullscreen mode

This means a configuration can behave like this:

Spark
  ↓
connects successfully to kafka:29092
  ↓
Kafka responds:
"Use localhost:9092 for this broker"
  ↓
Spark tries localhost:9092
  ↓
localhost = Spark container
  ↓
connection fails
Enter fullscreen mode Exit fullscreen mode

So simply making the first socket connection work is not enough.

The broker must advertise an address that the client can actually reach from its own network.

I had two different kinds of clients:

Client 1:
Python Producer on Windows

Client 2:
Spark inside Docker
Enter fullscreen mode Exit fullscreen mode

They needed two different addresses.

I changed Kafka to expose two listeners:

KAFKA_LISTENERS: PLAINTEXT_HOST://:9092,PLAINTEXT_DOCKER://:29092,CONTROLLER://:9093

KAFKA_ADVERTISED_LISTENERS: PLAINTEXT_HOST://localhost:9092,PLAINTEXT_DOCKER://kafka:29092

KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT,PLAINTEXT_DOCKER:PLAINTEXT

KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT_DOCKER
Enter fullscreen mode Exit fullscreen mode

Now the network paths were clear.

From Windows:

Python Producer
      ↓
localhost:9092
      ↓
Kafka
Enter fullscreen mode Exit fullscreen mode

From Docker:

Spark
      ↓
kafka:29092
      ↓
Kafka
Enter fullscreen mode Exit fullscreen mode

This was probably the most valuable part of the whole troubleshooting process.

Before this, I understood Docker ports and Kafka bootstrap servers separately.

After debugging it, I understood why Kafka's advertised.listeners has to describe the broker from the client's point of view.

That is a much more useful mental model.

Failure #5: Spark Could Reach Kafka, but Ivy Could Not Write Its Cache

At this point I finally had:

Spark in Linux Docker
      ↓
Docker DNS
      ↓
Kafka
Enter fullscreen mode Exit fullscreen mode

The network was working.

I ran Spark with the Kafka connector:

/opt/spark/bin/spark-submit \
  --packages org.apache.spark:spark-sql-kafka-0-10_2.13:4.0.1 \
  /opt/sec_project/scripts/spark_kafka_stream.py
Enter fullscreen mode Exit fullscreen mode

And got another failure.

Observed error:

/nonexistent/.ivy2.5.2/cache/...
No such file or directory
Enter fullscreen mode Exit fullscreen mode

The log showed:

Ivy Default Cache set to:
/nonexistent/.ivy2.5.2/cache
Enter fullscreen mode Exit fullscreen mode

Then Java threw a FileNotFoundException.

This was not a Kafka connectivity problem.

It was not a Spark streaming problem either.

It was a dependency management problem.

What Ivy Was Doing

When Spark sees:

--packages org.apache.spark:spark-sql-kafka-0-10_2.13:4.0.1
Enter fullscreen mode Exit fullscreen mode

it needs to resolve and download that package and its transitive dependencies.

Spark uses Ivy/Maven dependency resolution for this.

A simplified view is:

spark-submit
      ↓
Ivy
      ↓
Maven Central
      ↓
Kafka connector JARs
      ↓
local cache
Enter fullscreen mode Exit fullscreen mode

Normally Ivy uses a cache under the user's home directory.

But inside this Spark container, the effective home path led Ivy to:

/nonexistent/.ivy2.5.2
Enter fullscreen mode Exit fullscreen mode

That directory was not usable.

So the connector existed on Maven Central, the network worked, and the package name was correct — but Spark still could not start because Ivy could not save its dependency metadata.

Fix:

The fix was simple once the real layer was identified.

I explicitly gave Ivy a writable directory:

--conf spark.jars.ivy=/tmp/.ivy2
Enter fullscreen mode Exit fullscreen mode

The command became:

docker exec -it sec-spark /opt/spark/bin/spark-submit \
  --conf spark.jars.ivy=/tmp/.ivy2 \
  --packages org.apache.spark:spark-sql-kafka-0-10_2.13:4.0.1 \
  /opt/sec_project/scripts/spark_kafka_stream.py
Enter fullscreen mode Exit fullscreen mode

This time the log changed from:

Ivy Default Cache set to:
/nonexistent/.ivy2.5.2/cache
Enter fullscreen mode Exit fullscreen mode

to:

Ivy Default Cache set to:
/tmp/.ivy2/cache
Enter fullscreen mode Exit fullscreen mode

Spark successfully downloaded and loaded the Kafka connector, Kafka client, Hadoop client libraries, and related dependencies.

Then I finally saw:

Running Spark version 4.0.1
OS info Linux
Java version 17.0.16
Submitted application: SEC Kafka Stream
Enter fullscreen mode Exit fullscreen mode

That was the first time the entire runtime had actually made it through startup.

Then I Got Batch: 0

Spark started the streaming query and printed:

Batch: 0
Enter fullscreen mode Exit fullscreen mode

The table was empty.

After everything I had just fixed, an empty table was almost suspicious.

But this time there was no error.

That difference mattered.

The streaming query was alive.

The Kafka connector was loaded.

The broker was reachable.

The topic existed.

There simply was no message in the newly recreated topic yet.

Structured Streaming was doing exactly what it was supposed to do.

It had created a micro-batch, found no new Kafka records, and waited.

So I left the Spark terminal running.

Then I opened another terminal and ran:

python scripts/kafka_producer.py
Enter fullscreen mode Exit fullscreen mode

The producer sent the financial event.

I went back to the Spark terminal.

And there it was:

Batch: 1
Enter fullscreen mode Exit fullscreen mode

with:

message_key = COMPANY_001
topic       = financial_events
partition   = 0
offset      = 0
Enter fullscreen mode Exit fullscreen mode

and the full financial event JSON.

That was the moment the pipeline actually worked:

Python Producer
      ↓
Kafka
      ↓
Spark Structured Streaming
      ↓
Micro-batch
      ↓
financial event
Enter fullscreen mode Exit fullscreen mode

What Looked Like One Problem Was Actually Several

The original symptom was:

Spark cannot read Kafka.

But that description was almost useless.

The actual failures were distributed across different layers:

  1. Runtime / launcher — spark-submit could not correctly discover the pip-installed local Spark environment.
  2. Dependency — Spark did not have the Kafka datasource connector.
  3. OS / Hadoop compatibility — Windows Spark execution hit HADOOP_HOME / winutils.exe behavior.
  4. Container networking / Kafka metadata — localhost meant different things on the host and inside Docker, while Kafka needed to advertise an address each client could actually reach.
  5. Dependency cache — Ivy tried to write into /nonexistent, preventing the Kafka connector from loading.

And after all five were fixed, Batch: 0 was not another failure.

It was simply an empty streaming batch.

Then Batch: 1 proved the end-to-end integration.

The Biggest Lesson: Debug by Layer

The most useful thing I learned was to stop asking:

Why isn't Spark reading Kafka?

That question is too broad.

A better troubleshooting flow is:

Can Spark start?
      ↓
Is the JVM healthy?
      ↓
Does Spark know the Kafka datasource?
      ↓
Is the connector loaded?
      ↓
Can the Spark container resolve Kafka?
      ↓
Is the Kafka port reachable?
      ↓
Is the broker advertising the right address?
      ↓
Does the topic exist?
      ↓
Are there records after the current offset?
      ↓
Is the streaming query actually running?
Enter fullscreen mode Exit fullscreen mode

Each answer eliminates an entire class of problems.

That is much faster than randomly changing application code.

Read the Root Cause, Not Just the Last Exception

Another lesson was how to read stack traces.

For example:

Py4JJavaError
Enter fullscreen mode Exit fullscreen mode

was not the real problem.

Further down:

Caused by:
HADOOP_HOME and hadoop.home.dir are unset
Enter fullscreen mode Exit fullscreen mode

was the useful information.

Likewise:

FileNotFoundException
Enter fullscreen mode Exit fullscreen mode

did not mean my Python script was missing.

The path:

/nonexistent/.ivy2.5.2/cache
Enter fullscreen mode Exit fullscreen mode

showed that the failure belonged to Spark's dependency cache.

The pattern I want to keep using is:

Read the error
      ↓
Find the first meaningful "Caused by"
      ↓
Identify the failing layer
      ↓
Test that layer separately
      ↓
Only then change the configuration or code
Enter fullscreen mode Exit fullscreen mode

That sounds simple.

In practice, it changes troubleshooting completely.

The Final Local Architecture

The working development setup now looks like this:

Windows Host
│
├── Python Kafka Producer
│       │
│       └── localhost:9092
│
└── Docker Compose Network
        │
        ├── Kafka
        │     ├── Host listener:   localhost:9092
        │     └── Docker listener: kafka:29092
        │
        └── Spark
              │
              └── Spark Structured Streaming
                    ↓
                 financial_events
Enter fullscreen mode Exit fullscreen mode

The next step is to parse the Kafka JSON payload into typed Spark columns and continue toward Snowflake.

But getting one message from the Python producer all the way into a Spark micro-batch was already a useful engineering exercise.

I started the day thinking I was debugging one Kafka connection.

I ended up debugging five different systems that happened to fail on the same path.

And that is probably the part of distributed data engineering that tutorials don't show often enough.

Suggested DEV Tags

#dataengineering #apachekafka #apachespark #docker

Top comments (0)