DEV Community

Cover image for Deploying Redpanda Kafka-Compatible Streaming Platform on Ubuntu 24.04
Vultr profile image Sanskriti Harmukh
Sanskriti Harmukh for Vultr

Posted on with Aashish Chaurasiya

Deploying Redpanda Kafka-Compatible Streaming Platform on Ubuntu 24.04

#kafka #devops #streaming #linux

Redpanda is a Kafka-API-compatible streaming platform written in C++ with no JVM and no ZooKeeper. This guide installs Redpanda on Ubuntu 24.04, secures it with a Let's Encrypt certificate and SASL/SCRAM authentication, tunes the kernel for production, verifies with a producer/consumer test, and exposes Redpanda Console behind Nginx basic auth. By the end, you'll have a secured, production-tuned single-node Redpanda cluster with a web console.

Prerequisite: Ubuntu 24.04 server sized per Redpanda's CPU/memory requirements, non-root sudo user, and a domain A record (e.g. redpanda.example.com).

Install Redpanda 

$ sudo apt update
$ curl -1sLf 'https://dl.redpanda.com/nzc4ZYQK3WRGd9sy/redpanda/cfg/setup/bash.deb.sh' | sudo -E bash
Enter fullscreen mode Exit fullscreen mode

Warning: Only run vendor setup scripts you trust — piped curl | sudo bash runs with root privileges.

$ sudo apt install redpanda -y
$ rpk --version
Enter fullscreen mode Exit fullscreen mode

Open the Firewall

Port Service Purpose
9092 Kafka API Producer/consumer traffic
8082 Pandaproxy (HTTP) REST access for non-Kafka clients
8081 Schema Registry Avro/Protobuf schema versioning
9644 Admin API Monitoring, config, health checks
33145 Internal RPC Inter-node communication 
$ sudo ufw allow 9092,8082,8081,9644,33145/tcp
$ sudo ufw allow 80/tcp
$ sudo ufw allow 443/tcp
$ sudo ufw reload
Enter fullscreen mode Exit fullscreen mode

Issue a Let's Encrypt Certificate

Redpanda ships with plaintext networking by default, fine for a lab, not for anything else.

$ sudo apt install certbot -y
$ DOMAIN=redpanda.example.com
$ EMAIL=admin@example.com
$ sudo certbot certonly --standalone -d $DOMAIN --non-interactive --email $EMAIL
Enter fullscreen mode Exit fullscreen mode

Certbot stores certs under /etc/letsencrypt/live, readable only by root. Redpanda runs as its own redpanda user, so copy the certs into a dedicated directory:

$ sudo mkdir /etc/redpanda/certs
$ sudo cp /etc/letsencrypt/live/$DOMAIN/fullchain.pem /etc/redpanda/certs/node.crt
$ sudo cp /etc/letsencrypt/live/$DOMAIN/privkey.pem /etc/redpanda/certs/node.key
$ sudo cp /etc/letsencrypt/live/$DOMAIN/chain.pem /etc/redpanda/certs/ca.crt
$ sudo chown -R redpanda:redpanda /etc/redpanda/certs/
$ sudo chmod 400 /etc/redpanda/certs/node.key
$ sudo chmod 444 /etc/redpanda/certs/node.crt /etc/redpanda/certs/ca.crt
Enter fullscreen mode Exit fullscreen mode

Automate renewal — Let's Encrypt certs expire every 90 days, so wire a deploy hook that refreshes Redpanda's copies and restarts the service:

$ sudo nano /etc/letsencrypt/renewal-hooks/deploy/redpanda-renewal.sh
Enter fullscreen mode Exit fullscreen mode 
#!/bin/bash
DOMAIN="redpanda.example.com"

cp /etc/letsencrypt/live/$DOMAIN/fullchain.pem /etc/redpanda/certs/node.crt
cp /etc/letsencrypt/live/$DOMAIN/privkey.pem /etc/redpanda/certs/node.key
cp /etc/letsencrypt/live/$DOMAIN/chain.pem /etc/redpanda/certs/ca.crt

chown -R redpanda:redpanda /etc/redpanda/certs/
chmod 400 /etc/redpanda/certs/node.key
chmod 444 /etc/redpanda/certs/node.crt /etc/redpanda/certs/ca.crt

systemctl restart redpanda
Enter fullscreen mode Exit fullscreen mode 
$ sudo chmod +x /etc/letsencrypt/renewal-hooks/deploy/redpanda-renewal.sh
$ sudo certbot renew --dry-run
Enter fullscreen mode Exit fullscreen mode

Tune for Production 

$ sudo rpk redpanda mode production
$ sudo rpk redpanda tune all
$ sudo systemctl start redpanda-tuner
Enter fullscreen mode Exit fullscreen mode

tune all optimizes I/O schedulers and CPU settings, takes ~30 seconds. The tuner service reapplies these on every reboot since kernel tuning doesn't persist otherwise.

Bootstrap Authentication

Set security policy before the first startup so it's enforced from the very first boot.

$ sudo nano /etc/redpanda/.bootstrap.yaml
Enter fullscreen mode Exit fullscreen mode 
enable_sasl: true
admin_api_require_auth: true
superusers:
  - admin
Enter fullscreen mode Exit fullscreen mode

Note: This file is only read on first startup. Later changes go through the Admin API or rpk cluster config.

Set the superuser's password:

$ echo "RP_BOOTSTRAP_USER=admin:STRONG_PASSWORD:SCRAM-SHA-256" | sudo tee -a /etc/default/redpanda
Enter fullscreen mode Exit fullscreen mode

Configure the Node 

$ sudo cp /etc/redpanda/redpanda.yaml /etc/redpanda/redpanda-backup.yaml
$ sudo nano /etc/redpanda/redpanda.yaml
Enter fullscreen mode Exit fullscreen mode

Replace the contents (swap redpanda.example.com for your domain):

redpanda:
  data_directory: /var/lib/redpanda/data
  seed_servers: []

  rpc_server:
    address: 0.0.0.0
    port: 33145
  advertised_rpc_api:
    address: redpanda.example.com
    port: 33145

  kafka_api:
    - name: tls_listener
      address: 0.0.0.0
      port: 9092
      authentication_method: sasl
  advertised_kafka_api:
    - name: tls_listener
      address: redpanda.example.com
      port: 9092
  kafka_api_tls:
    - name: tls_listener
      key_file: /etc/redpanda/certs/node.key
      cert_file: /etc/redpanda/certs/node.crt
      truststore_file: /etc/redpanda/certs/ca.crt
      enabled: true
      require_client_auth: false

  admin:
    - address: 0.0.0.0
      port: 9644
  admin_api_tls:
    - enabled: true
      key_file: /etc/redpanda/certs/node.key
      cert_file: /etc/redpanda/certs/node.crt
      truststore_file: /etc/redpanda/certs/ca.crt

rpk:
  tune_network: true
  tune_disk_scheduler: true
  tune_disk_nomerges: true
  tune_disk_write_cache: true
  tune_disk_irq: true
  tune_cpu: true
  tune_aio_events: true
  tune_clocksource: true
  tune_swappiness: true
  coredump_dir: /var/lib/redpanda/coredump
  tune_ballast_file: true
Enter fullscreen mode Exit fullscreen mode

This enables SASL-over-TLS on the Kafka API (9092), TLS on the Admin API (9644), advertises both under your public domain, and applies the full rpk tuning profile.

$ sudo systemctl start redpanda
$ sudo systemctl status redpanda
Enter fullscreen mode Exit fullscreen mode

Create an rpk Profile

Run rpk as your regular user (not sudo) so it picks up your profile.

$ nano ~/rpk-profile.yaml
Enter fullscreen mode Exit fullscreen mode 
kafka_api:
  sasl:
    user: admin
    password: STRONG_PASSWORD
    mechanism: SCRAM-SHA-256
  tls:
    enabled: true
    ca_file: /etc/redpanda/certs/ca.crt
  brokers:
    - redpanda.example.com:9092

admin_api:
  tls:
    enabled: true
    ca_file: /etc/redpanda/certs/ca.crt
  addresses:
    - redpanda.example.com:9644
Enter fullscreen mode Exit fullscreen mode 
$ rpk profile create production --from-profile ~/rpk-profile.yaml
$ rpk cluster health
Enter fullscreen mode Exit fullscreen mode 
CLUSTER HEALTH OVERVIEW
=======================
Healthy:                          true
Unhealthy reasons:                []
Controller ID:                    0
All nodes:                        [0]
Nodes down:                       []
Cluster UUID:                     65e28fbb-c1b4-42c3-b0b7-fd82608699f3
Enter fullscreen mode Exit fullscreen mode

Test with a Producer and Consumer 

$ rpk topic create test-topic
Enter fullscreen mode Exit fullscreen mode 
TOPIC       STATUS
test-topic  OK
Enter fullscreen mode Exit fullscreen mode 
$ echo "Hello Redpanda" | rpk topic produce test-topic
Enter fullscreen mode Exit fullscreen mode 
Produced to partition 0 at offset 0 with timestamp 1699618845123.
Enter fullscreen mode Exit fullscreen mode 
$ rpk topic consume test-topic --num 1
Enter fullscreen mode Exit fullscreen mode

Deploy Redpanda Console 

$ sudo apt install redpanda-console -y
$ sudo nano /etc/redpanda/redpanda-console-config.yaml
Enter fullscreen mode Exit fullscreen mode 
server:
  listenPort: 8080
  listenAddress: 127.0.0.1

kafka:
  brokers:
    - redpanda.example.com:9092
  tls:
    enabled: true
    caFilepath: /etc/redpanda/certs/ca.crt
  sasl:
    enabled: true
    username: admin
    password: STRONG_PASSWORD
    mechanism: SCRAM-SHA-256

redpanda:
  adminApi:
    enabled: true
    urls:
      - https://redpanda.example.com:9644
    tls:
      enabled: true
      caFilepath: /etc/redpanda/certs/ca.crt
    authentication:
      basic:
        username: admin
        password: STRONG_PASSWORD
Enter fullscreen mode Exit fullscreen mode 
$ sudo systemctl enable redpanda-console
$ sudo systemctl start redpanda-console
Enter fullscreen mode Exit fullscreen mode

Front the Console with Nginx Basic Auth

Redpanda Console CE has no built-in login — put Nginx basic auth in front of it.

$ sudo apt install nginx -y
$ sudo apt install apache2-utils -y
$ sudo htpasswd -c /etc/nginx/.htpasswd admin
Enter fullscreen mode Exit fullscreen mode 
$ sudo nano /etc/nginx/sites-available/redpanda-console
Enter fullscreen mode Exit fullscreen mode 
server {
    listen 80;
    server_name redpanda.example.com;
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl;
    server_name redpanda.example.com;

    ssl_certificate /etc/letsencrypt/live/redpanda.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/redpanda.example.com/privkey.pem;

    location / {
        auth_basic "Restricted Access";
        auth_basic_user_file /etc/nginx/.htpasswd;
        proxy_pass http://127.0.0.1:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}
Enter fullscreen mode Exit fullscreen mode 
$ sudo ln -s /etc/nginx/sites-available/redpanda-console /etc/nginx/sites-enabled/
$ sudo nginx -t
$ sudo systemctl restart nginx
Enter fullscreen mode Exit fullscreen mode

Open https://redpanda.example.com, enter the basic-auth credentials to reach the console dashboard.

Next Steps

Redpanda is running with TLS, SASL auth, production tuning, and a secured console. From here you can:

  • Add more brokers and set seed_servers for a multi-node cluster
  • Wire Kafka Connect or a schema registry client against port 8081
  • Point existing Kafka producers/consumers at Redpanda. The wire protocol is compatible, no client changes needed

For the full guide with additional tips, visit the original article on Vultr Docs.

Top comments (0)