Apache Pulsar Has a Free API: The Messaging System That Scales Beyond Kafka

#pulsar #messaging #kafka #streaming

Kafka was revolutionary in 2011. But it stores compute and storage together, making scaling painful. Pulsar separates them — and adds features Kafka still doesn't have.

What Is Apache Pulsar?

Apache Pulsar is a distributed messaging and streaming platform. It separates serving (brokers) from storage (BookKeeper), letting you scale each independently.

docker run -p 6650:6650 -p 8080:8080 apachepulsar/pulsar:latest bin/pulsar standalone

Producer/Consumer

import pulsar

client = pulsar.Client('pulsar://localhost:6650')

# Producer
producer = client.create_producer('persistent://public/default/my-topic')
producer.send(b'Hello Pulsar!')

# Consumer
consumer = client.subscribe('persistent://public/default/my-topic', 'my-subscription')
msg = consumer.receive()
print(f"Received: {msg.data().decode()}")
consumer.acknowledge(msg)

client.close()

Why Pulsar Over Kafka

Feature	Kafka	Pulsar
Architecture	Coupled compute+storage	Separated (brokers + BookKeeper)
Multi-tenancy	No	Yes (built-in)
Geo-replication	MirrorMaker (complex)	Built-in, one line
Message queuing	No (only streaming)	Yes (shared subscription)
Delayed messages	No	Yes (native)
Dead letter queue	No	Yes (built-in)
Schema registry	Confluent (separate)	Built-in
Topic scaling	Partition-bound	Segment-based (infinite)

Key Pulsar Features

1. Multi-tenancy — isolate teams, environments, projects:

persistent://tenant/namespace/topic
persistent://marketing/analytics/click-events
persistent://engineering/staging/user-events

2. Geo-replication — replicate topics across data centers:

bin/pulsar-admin topics set-replication-clusters   --clusters us-east,eu-west,ap-southeast   persistent://public/default/my-topic

3. Tiered storage — offload old data to S3/GCS automatically:

broker.conf:
managedLedgerOffloadDriver=aws-s3
s3ManagedLedgerOffloadBucket=pulsar-offload

4. Functions — serverless stream processing:

# Pulsar Function — processes messages in-flight
class ExclamationFunction(Function):
    def process(self, input, context):
        return f"{input}!!!"