Benthos Has a Free Stream Processor — Connect, Transform, and Route Data Between Services

Benthos (now Redpanda Connect) is a declarative stream processing tool for ETL, routing, and transformation.

What You Get for Free

• 200+ connectors — Kafka, RabbitMQ, S3, PostgreSQL, HTTP, NATS, and more
• Processing pipeline — filter, map, parse, enrich data in-flight
• Bloblang — powerful data transformation language
• Error handling — dead letter queues, retry strategies
• Batching — configurable batch sizes and timing
• Rate limiting — control throughput per pipeline
• Metrics — Prometheus metrics out of the box
• YAML config — entire pipeline in one config file
• Single binary — no dependencies

Quick Start

# pipeline.yaml
input:
  kafka: { addresses: ["localhost:9092"], topics: ["raw-events"] }
pipeline:
  processors:
    - bloblang: |
        root.user = this.user_id
        root.action = this.event_type.lowercase()
        root.timestamp = now()
output:
  postgresql: { dsn: "postgres://user:pass@host/db", table: "events" }

benthos -c pipeline.yaml

Why Data Engineers Choose It

Custom ETL scripts are fragile. Kafka Streams is Java-only:

• Declarative — YAML config, no code to maintain
• Bloblang — powerful transformations without code
• Connectors — any-to-any data movement
• Single binary — deploy anywhere, no runtime

A data team maintained 8 Python ETL scripts moving data between Kafka, PostgreSQL, and S3. They replaced all 8 with Benthos YAML configs — same transformations, zero custom code to maintain.

---

Need Custom Data Solutions?

I build production-grade scrapers and data pipelines for startups, agencies, and research teams.

Browse 88+ ready-made scrapers on Apify → — Reddit, HN, LinkedIn, Google, Amazon, and more.

Custom project? Email me: [e-mail] — fast turnaround, fair pricing.