PDF, ZIP and binary files streaming over Kafka with names intact.

Day 07 of the WKafka Open-Source Engineering Series.

Don't provision a cloud bucket when all you need is sending small PDFs and ZIPs down your pipeline. WKafka supports format='file' out of the box.

The Pain Points We Faced

• Building S3/MinIO upload/download pipelines just to move 500KB report files
• Lost original filenames and mime-types when converting files to raw bytes
• High latency polling for cloud object storage propagation

The Implementation

# Producer
kafka.produce_file(topic="reports", file_path="audit_2026.pdf")

# Consumer
@kafka.consumer(topic="reports", format="file")
def on_report(msg):
    msg.save_file(target_dir="received_pipeline_files/")

Why This Architecture Wins

• format='file': Native file serializer bundles filename and binary data.
• msg.save_file(): One-liner to write the incoming file to disk preserving name.
• Sub-Second Delivery: Direct broker streaming without external storage bottlenecks.

Verification & Status

Tested and verified with Apache Kafka against real broker clusters (see EXAMPLES_STATUS.md in repository). Compatible with Python 3.9 through 3.14 with strict typing.

• GitHub: https://github.com/wisrovi/wkafka
• PyPI: https://pypi.org/project/wkafka

Kafka #Python #DataEngineering #OpenSource #Wisrovi