DEV Community

Cover image for Deploying Apache Pulsar as a Self-Hosted Google Pub/Sub Alternative
Vultr profile image Sanskriti Harmukh
Sanskriti Harmukh for Vultr

Posted on with Aashish Chaurasiya • Originally published at docs.vultr.com

Deploying Apache Pulsar as a Self-Hosted Google Pub/Sub Alternative

#docker #messaging #streaming #selfhosted

Apache Pulsar is a distributed messaging/streaming platform combining high-throughput delivery, durable storage, and built-in serverless compute (Pulsar Functions), a self-hosted alternative to Google Pub/Sub with multi-tenancy and tiered storage built in. This guide deploys Pulsar via Docker, sets up multi-tenancy, deploys a Pulsar Function, configures an IO connector, enables JWT auth, and covers migrating off Pub/Sub.

Pulsar vs. Google Pub/Sub

  • Deployment: Pub/Sub is fully managed on GCP. Pulsar runs on any infra you control (bare metal, VMs, Kubernetes).
  • Multi-tenancy: Pub/Sub uses GCP projects for isolation. Pulsar has native tenants/namespaces/topic-level policies.
  • Storage: Pub/Sub abstracts storage entirely. Pulsar uses Apache BookKeeper with optional tiered storage to object storage.
  • Serverless: Pub/Sub integrates with Cloud Functions. Pulsar Functions run inline with the broker, no external system needed.
  • Pricing: Pub/Sub charges per volume/operation. Pulsar is free beyond infra cost.

Concept Mapping

Google Pub/Sub Apache Pulsar Description
Topics Topics Named channels for publishing
Subscriptions Subscriptions Named consumers reading from topics
Push Subscriptions Pulsar Functions Use a Function to forward messages to an HTTP endpoint
Ordering Keys Key-Shared Subscriptions Ordered delivery for same-key messages
Dead-Letter Topics Dead-Letter Topics Stores unprocessable messages
IAM Policies Authorization Policies RBAC for topics/namespaces
Projects Tenants/Namespaces Tenants = project-level isolation; namespaces subdivide a tenant

Architecture:

  • Brokers — stateless, handle routing/connections, no direct storage
  • BookKeeper (Bookies) — durable, replicated storage layer
  • Metadata store — cluster coordination; standalone Docker deployments use RocksDB, not external ZooKeeper
  • Pulsar Manager — web UI for tenants/namespaces/topics/monitoring
  • Pulsar Functions — inline serverless stream processing

Prerequisites: a Linux server, Docker + Docker Compose, a domain A record (e.g. pulsar.example.com).

Deploy with Docker

Standalone mode runs broker, BookKeeper, and metadata store in one container. The broker advertises localhost, so pulsar-admin/pulsar-client commands run via docker exec rather than from outside.

$ mkdir -p ~/pulsar/{data,conf,connectors,functions}
$ cd ~/pulsar
Enter fullscreen mode Exit fullscreen mode

Environment file:

$ nano .env
Enter fullscreen mode Exit fullscreen mode 
PULSAR_DOMAIN=pulsar.example.com
ACME_EMAIL=admin@example.com
PM_USERNAME=admin
PM_PASSWORD=PULSAR_MANAGER_PASSWORD
PM_EMAIL=admin@example.com
Enter fullscreen mode Exit fullscreen mode 
$ chmod 777 data
Enter fullscreen mode Exit fullscreen mode

Admin bootstrap script — creates the Pulsar Manager admin account on first run:

$ nano init-admin.sh
Enter fullscreen mode Exit fullscreen mode 
#!/bin/sh
set -e
CSRF=$(curl -s http://pulsar-manager:7750/pulsar-manager/csrf-token)
curl -s \
  -H "X-XSRF-TOKEN: $CSRF" \
  -H "Cookie: XSRF-TOKEN=$CSRF;" \
  -H "Content-Type: application/json" \
  -X PUT http://pulsar-manager:7750/pulsar-manager/users/superuser \
  -d "{\"name\":\"$PM_USERNAME\",\"password\":\"$PM_PASSWORD\",\"description\":\"admin\",\"email\":\"$PM_EMAIL\"}"
Enter fullscreen mode Exit fullscreen mode 
$ chmod +x init-admin.sh
$ sudo usermod -aG docker $USER
$ newgrp docker
Enter fullscreen mode Exit fullscreen mode

Docker Compose:

$ nano docker-compose.yml
Enter fullscreen mode Exit fullscreen mode 
services:
  traefik:
    image: traefik:v3.7.0
    container_name: traefik
    command:
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"
      - "--certificatesresolvers.le.acme.email=${ACME_EMAIL}"
      - "--certificatesresolvers.le.acme.storage=/letsencrypt/acme.json"
      - "--certificatesresolvers.le.acme.httpchallenge=true"
      - "--certificatesresolvers.le.acme.httpchallenge.entrypoint=web"
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "./letsencrypt:/letsencrypt"
    restart: unless-stopped

  pulsar:
    image: apachepulsar/pulsar:4.2.2
    container_name: pulsar
    command: bin/pulsar standalone --advertised-address localhost
    ports:
      - "6650:6650"
      - "8080:8080"
    volumes:
      - ./data:/pulsar/data
      - ./connectors:/pulsar/connectors
      - ./functions:/pulsar/functions
    environment:
      - PULSAR_MEM=-Xms2g -Xmx2g -XX:MaxDirectMemorySize=1g
    restart: unless-stopped

  pulsar-manager:
    image: apachepulsar/pulsar-manager:v0.4.0
    container_name: pulsar-manager
    environment:
      - SPRING_CONFIGURATION_FILE=/pulsar-manager/pulsar-manager/application.properties
    depends_on:
      - pulsar
    healthcheck:
      test: ["CMD-SHELL", "wget -qO- http://localhost:7750/pulsar-manager/csrf-token >/dev/null 2>&1"]
      interval: 10s
      timeout: 5s
      retries: 12
      start_period: 45s
    labels:
      - "traefik.enable=true"
      - "traefik.http.services.pulsar-manager.loadbalancer.server.port=9527"
      - "traefik.http.routers.pulsar-manager.rule=Host(`${PULSAR_DOMAIN}`)"
      - "traefik.http.routers.pulsar-manager.entrypoints=websecure"
      - "traefik.http.routers.pulsar-manager.tls=true"
      - "traefik.http.routers.pulsar-manager.tls.certresolver=le"
    restart: unless-stopped

  pulsar-manager-init:
    image: curlimages/curl:8.19.0
    container_name: pulsar-manager-init
    depends_on:
      pulsar-manager:
        condition: service_healthy
    restart: "no"
    volumes:
      - ./init-admin.sh:/init-admin.sh:ro
    entrypoint: ["sh", "/init-admin.sh"]
    environment:
      - PM_USERNAME
      - PM_PASSWORD
      - PM_EMAIL
Enter fullscreen mode Exit fullscreen mode 
$ docker compose up -d
$ docker compose ps
$ docker compose logs pulsar
$ docker logs pulsar-manager-init
Enter fullscreen mode Exit fullscreen mode

pulsar-manager-init should print {"message":"Add super user success, please login"} and then exit — it's a one-shot job.

Access Pulsar Manager

Pulsar pre-creates two tenants: public (default namespace) and pulsar (internal system topics).

  1. Visit https://pulsar.example.com, log in.
  2. New Environment: name local, Service URL http://pulsar:8080, Bookie URL http://pulsar:6650, Confirm.
  3. Click into the environment — sidebar shows Tenants, Namespaces, Topics, Tokens.

Configure Multi-Tenancy 

$ docker exec -it pulsar bin/pulsar-admin tenants create my-tenant \
    --admin-roles admin \
    --allowed-clusters standalone
$ docker exec -it pulsar bin/pulsar-admin namespaces create my-tenant/production
$ docker exec -it pulsar bin/pulsar-admin namespaces set-retention my-tenant/production \
    --size 10G \
    --time 7d
$ docker exec -it pulsar bin/pulsar-admin namespaces set-message-ttl my-tenant/production \
    --messageTTL 3600
$ docker exec -it pulsar bin/pulsar-admin topics create persistent://my-tenant/production/orders
$ docker exec -it pulsar bin/pulsar-admin topics list my-tenant/production
Enter fullscreen mode Exit fullscreen mode

Check the NAMESPACES tab under my-tenant in Pulsar Manager to confirm.

Deploy a Pulsar Function

Transform/filter/route messages inline, no external stream processor needed.

$ nano ~/pulsar/functions/uppercase_function.py
Enter fullscreen mode Exit fullscreen mode 
from pulsar import Function

class UppercaseFunction(Function):
    def process(self, input, context):
        return input.upper()
Enter fullscreen mode Exit fullscreen mode 
$ docker exec -it pulsar bin/pulsar-admin functions create \
    --name uppercase \
    --tenant public \
    --namespace default \
    --inputs persistent://public/default/input-topic \
    --output persistent://public/default/output-topic \
    --py /pulsar/functions/uppercase_function.py \
    --classname uppercase_function.UppercaseFunction
$ docker exec -it pulsar bin/pulsar-admin functions status \
    --tenant public \
    --namespace default \
    --name uppercase
Enter fullscreen mode Exit fullscreen mode

Test it:

$ docker exec -it pulsar bin/pulsar-client produce persistent://public/default/input-topic \
    --messages "hello world"
$ docker exec -it pulsar bin/pulsar-client consume persistent://public/default/output-topic \
    --subscription-name test-sub \
    --num-messages 1 \
    --subscription-position Earliest
Enter fullscreen mode Exit fullscreen mode

Output should show content:HELLO WORLD.

Set Up an IO Connector

The standard image ships without connector NARs — download what you need. This demos the file source connector.

$ curl -L -o ~/pulsar/connectors/pulsar-io-file-4.2.2.nar \
    https://downloads.apache.org/pulsar/pulsar-4.2.2/connectors/pulsar-io-file-4.2.2.nar
$ docker compose restart pulsar
Enter fullscreen mode Exit fullscreen mode

Wait for the broker (repeat until it returns ["standalone"]):

$ curl -s http://localhost:8080/admin/v2/clusters
Enter fullscreen mode Exit fullscreen mode 
$ docker exec -it pulsar bin/pulsar-admin sources available-sources
Enter fullscreen mode Exit fullscreen mode

Should list file.

Configure and deploy:

$ mkdir -p ~/pulsar/data/input-files
$ chmod 777 ~/pulsar/data/input-files
$ nano file-source-config.yaml
Enter fullscreen mode Exit fullscreen mode 
configs:
  inputDirectory: /pulsar/data/input-files
  recurse: false
  keepFile: true
  fileFilter: '[^\.].*'
  minimumFileAge: 0
Enter fullscreen mode Exit fullscreen mode 
$ docker cp file-source-config.yaml pulsar:/pulsar/
$ docker exec -it pulsar bin/pulsar-admin sources create \
    --name file-source \
    --tenant public \
    --namespace default \
    --destination-topic-name persistent://public/default/file-data \
    --source-type file \
    --source-config-file /pulsar/file-source-config.yaml
$ docker exec -it pulsar bin/pulsar-admin sources status \
    --tenant public \
    --namespace default \
    --name file-source
Enter fullscreen mode Exit fullscreen mode

Test it:

$ printf "log entry 1\nlog entry 2\n" > ~/pulsar/data/input-files/test.txt
$ docker exec -it pulsar bin/pulsar-client consume persistent://public/default/file-data \
    --subscription-name file-reader \
    --num-messages 2 \
    --subscription-position Earliest
Enter fullscreen mode Exit fullscreen mode

Each message includes file.name/file.path/file.modified.time metadata. Use a fresh --subscription-name if you repeat this — an existing subscription resumes where it left off rather than replaying.

Configure JWT Authentication 

$ docker exec -it pulsar bin/pulsar tokens create-secret-key \
    --output /pulsar/data/my-secret.key
$ docker exec pulsar bin/pulsar tokens create \
    --secret-key file:///pulsar/data/my-secret.key \
    --subject admin > ~/pulsar/data/admin-token.txt
$ docker exec pulsar bin/pulsar tokens create \
    --secret-key file:///pulsar/data/my-secret.key \
    --subject app-client > ~/pulsar/data/app-client-token.txt
$ docker compose stop pulsar
$ docker run --rm apachepulsar/pulsar:4.2.2 cat /pulsar/conf/standalone.conf > conf/standalone.conf
$ nano conf/standalone.conf
Enter fullscreen mode Exit fullscreen mode

Append:

authenticationEnabled=true
authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken
tokenSecretKey=file:///pulsar/data/my-secret.key

authorizationEnabled=true
authorizationProvider=org.apache.pulsar.broker.authorization.PulsarAuthorizationProvider
superUserRoles=admin

brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationToken
brokerClientAuthenticationParameters=file:///pulsar/data/admin-token.txt
Enter fullscreen mode Exit fullscreen mode

The last two let internal components (like the Functions worker) authenticate once JWT is on.

Mount the config:

$ nano docker-compose.yml
Enter fullscreen mode Exit fullscreen mode

Add under the pulsar service's volumes:

      - ./conf/standalone.conf:/pulsar/conf/standalone.conf
Enter fullscreen mode Exit fullscreen mode 
$ docker compose up -d pulsar
Enter fullscreen mode Exit fullscreen mode

Grant and test:

$ docker exec -it pulsar bin/pulsar-admin \
    --auth-plugin org.apache.pulsar.client.impl.auth.AuthenticationToken \
    --auth-params "token:$(cat ~/pulsar/data/admin-token.txt)" \
    namespaces grant-permission public/default \
    --role app-client \
    --actions produce,consume
$ docker exec -it pulsar bin/pulsar-client \
    --auth-plugin org.apache.pulsar.client.impl.auth.AuthenticationToken \
    --auth-params "token:$(cat ~/pulsar/data/app-client-token.txt)" \
    produce persistent://public/default/test-topic \
    --messages "authenticated message"
Enter fullscreen mode Exit fullscreen mode

Authenticate Pulsar Manager Too

Without this, the UI 401s on every request once broker auth is on.

$ docker run --rm --entrypoint cat apachepulsar/pulsar-manager:v0.4.0 \
    /pulsar-manager/pulsar-manager/application.properties > pulsar-manager-application.properties
$ nano pulsar-manager-application.properties
Enter fullscreen mode Exit fullscreen mode 
backend.jwt.token=YOUR_ADMIN_TOKEN
jwt.broker.token.mode=SECRET
jwt.broker.secret.key=file:///pulsar-manager/broker-secret.key
Enter fullscreen mode Exit fullscreen mode

Replace YOUR_ADMIN_TOKEN with the contents of ~/pulsar/data/admin-token.txt.

$ nano docker-compose.yml
Enter fullscreen mode Exit fullscreen mode

Add under pulsar-manager:

volumes:
  - ./pulsar-manager-application.properties:/pulsar-manager/pulsar-manager/application.properties:ro
  - ./data/my-secret.key:/pulsar-manager/broker-secret.key:ro
  - ./pulsar-manager-data:/pulsar-manager/pulsar-manager/dbdata
Enter fullscreen mode Exit fullscreen mode

The bind mount survives container recreation (a plain docker cp doesn't); the dbdata mount preserves the admin login and registered environments.

$ mkdir -p pulsar-manager-data
$ docker cp pulsar-manager:/pulsar-manager/pulsar-manager/dbdata/. pulsar-manager-data/
$ docker compose up -d pulsar-manager
Enter fullscreen mode Exit fullscreen mode

Verify the Deployment

All pulsar-admin/pulsar-client calls below need --auth-plugin org.apache.pulsar.client.impl.auth.AuthenticationToken --auth-params "token:$(cat ~/pulsar/data/admin-token.txt)" once auth is on.

$ nc -zv SERVER-IP 6650
$ curl -s -H "Authorization: Bearer $(cat ~/pulsar/data/admin-token.txt)" \
    http://localhost:8080/admin/v2/clusters
Enter fullscreen mode Exit fullscreen mode

Expect ["standalone"].

$ docker exec -it pulsar bin/pulsar-admin --auth-plugin org.apache.pulsar.client.impl.auth.AuthenticationToken --auth-params "token:$(cat ~/pulsar/data/admin-token.txt)" topics create persistent://public/default/verify-test
$ docker exec -it pulsar bin/pulsar-admin --auth-plugin org.apache.pulsar.client.impl.auth.AuthenticationToken --auth-params "token:$(cat ~/pulsar/data/admin-token.txt)" topics list public/default
$ docker exec -it pulsar bin/pulsar-client --auth-plugin org.apache.pulsar.client.impl.auth.AuthenticationToken --auth-params "token:$(cat ~/pulsar/data/admin-token.txt)" produce persistent://public/default/verify-test --messages "test message"
$ docker exec -it pulsar bin/pulsar-client --auth-plugin org.apache.pulsar.client.impl.auth.AuthenticationToken --auth-params "token:$(cat ~/pulsar/data/admin-token.txt)" consume persistent://public/default/verify-test --subscription-name verify-sub-1 --num-messages 1 --subscription-position Earliest
$ docker exec -it pulsar bin/pulsar-admin --auth-plugin org.apache.pulsar.client.impl.auth.AuthenticationToken --auth-params "token:$(cat ~/pulsar/data/admin-token.txt)" functions status --tenant public --namespace default --name uppercase
Enter fullscreen mode Exit fullscreen mode

A healthy function shows "running" : true. Use a fresh --subscription-name if you re-run the consume check.

Migrating from Google Pub/Sub

Topics/subscriptions: create persistent://tenant/namespace/topic-name per Pub/Sub topic. Pull subscriptions → Shared/Key_Shared; push subscriptions → Pulsar Functions forwarding to the target HTTP endpoint.

App code: swap the Pub/Sub client library for Pulsar's client (Java, Python, Go, Node.js, C++, C# all supported).

Schemas: Pulsar has a built-in schema registry — see the schema docs for Avro/Protobuf migration.

Push subscriptions: replace with a Pulsar Function that forwards to your HTTP endpoint — see the Functions docs.

Dead-letter topics: configure dead-letter policies on subscriptions to mirror Pub/Sub's max-delivery-attempts behavior.

Data migration: dual-write to both systems during cutover; backfill historical data by exporting Pub/Sub messages to Cloud Storage and replaying into Pulsar; disable Pub/Sub producers once consumers are caught up.

Watch for:

  • Ordering — use Key_Shared subscriptions to match Pub/Sub ordering-key behavior
  • Retention — Pub/Sub defaults to 7 days; set matching Pulsar namespace retention
  • Auth — replace GCP IAM with JWT tokens or OAuth 2.0
  • Monitoring — Pulsar exposes Prometheus metrics at http://localhost:8080/metrics/
  • Throughput — Pub/Sub has per-project quotas; Pulsar's ceiling depends on your infra sizing
  • Exactly-once — Pub/Sub is at-least-once by default; Pulsar supports exactly-once via transactional producers

Next Steps

Pulsar is running with multi-tenancy, a working Function, an IO connector, and JWT auth. From here:

  • Add tiered storage to offload cold data to object storage
  • Scale beyond standalone mode to a real broker/bookie/ZooKeeper cluster for production HA
  • Explore sink connectors to export processed data to external databases

For the full guide, visit the original article on Vultr Docs.

Top comments (0)