DEV Community: Bouchaala Reda

Creating a simple Message Bus: Episode 3

Bouchaala Reda — Fri, 03 Apr 2026 18:42:00 +0000

I's been a while since the last post, hey! Let's waste no time and dive straight in.

This is the one everything has been building toward.

We have a producer that sends messages to the broker. We have a broker that stores them in topic queues. All that's missing is the consumer — the thing that actually reads those messages.

Let's build it.

The Consumer

// internal/consumer/consumer.go

type Consumer struct {
    host  string
    port  string
    topic string
}

Simple. It knows where the broker is and which topic it cares about.

The interesting part is Subscribe:

func (c *Consumer) Subscribe() ([]string, error) {
    conn, err := net.Dial("tcp", net.JoinHostPort(c.host, c.port))
    if err != nil {
        return nil, err
    }
    defer conn.Close()

    fmt.Fprintf(conn, "%s\n", c.topic)

    var messages []string
    scanner := bufio.NewScanner(conn)
    for scanner.Scan() {
        messages = append(messages, scanner.Text())
    }

    return messages, nil
}

We connect to the broker.
We send the topic name we want to subscribe to, terminated with a newline.
We read back whatever the broker sends us, line by line.
We return the messages.

That's the entire consumer. Pretty much the mirror image of the producer.

Teaching the Broker About Consumers

The broker needs a new port for consumers to connect to. Let's add that to the config first:

// internal/broker/config.go

type BrokerConfig struct {
    ProducerHost string
    ProducerPort string

    ConsumerHost string
    ConsumerPort string
}

And now ConsumerListen:

// internal/broker/broker.go

func (broker *Broker) ConsumerListen() error {
    listener, err := net.Listen("tcp", net.JoinHostPort(broker.config.ConsumerHost, broker.config.ConsumerPort))
    if err != nil {
        return err
    }

    for {
        conn, err := listener.Accept()
        if err != nil {
            return err
        }

        scanner := bufio.NewScanner(conn)
        scanner.Scan()
        topicName := scanner.Text()

        log.Printf("consumer subscribed to topic %s", topicName)

        broker.mu.Lock()
        topic, exists := broker.topics[topicName]
        if !exists || topic == nil {
            broker.mu.Unlock()
            conn.Close()
            continue
        }

        for topic.Queue.Len() > 0 {
            message := topic.Queue.Dequeue()
            fmt.Fprintf(conn, "%s\n", string(message.Data))
        }
        broker.mu.Unlock()

        conn.Close()
    }
}

We listen on the consumer port.
A consumer connects and sends us a topic name.
We look up the topic.
If it exists, we dequeue all its messages and write them back, one line each.
We close the connection and wait for the next consumer.

Remember When I Said No Goroutines?

Yeah, about that.

ProducerListen and ConsumerListen both block forever in a loop. We need both running at the same time, which means we finally have to reach for a goroutine.

But that brings a problem: both methods touch broker.topics, and doing that from multiple goroutines without protection is a data race. So we add a sync.Mutex to the broker:

// internal/broker/broker.go

type Broker struct {
    config  *BrokerConfig
    decoder decoder.Decoder
    topics  map[string]*Topic
    mu      sync.Mutex
}

And we lock around every access to topics — in HandleNewMessage and ConsumerListen.

I said I'd avoid goroutines as much as possible, and we did. But this one we actually need.

Putting it all together

In the broker CLI, we run ProducerListen in a goroutine and let ConsumerListen block the main thread:

// cmd/broker/broker.go

go func() {
    log.Println("Broker: start listening for incoming producer messages...")
    err := b.ProducerListen()
    if err != nil {
        log.Fatalf(err.Error())
    }
}()

log.Println("Broker: start listening for incoming consumer messages...")
err = b.ConsumerListen()
if err != nil {
    log.Fatalf(err.Error())
}

Two new flags: -chost (default 127.0.0.1) and -cport (default 9991).

And the consumer CLI is as straightforward as the producer:

// cmd/consumer/consumer.go

client := consumer.New(brokerHost, brokerPort, topic)

messages, err := client.Subscribe()
if err != nil {
    log.Fatalf(err.Error())
}

for _, message := range messages {
    log.Printf("[%s] %s", topic, message)
}

Flags: -host, -port (default 9991), -topic (required).

Testing it out

Open three terminals. Rebuild first:

make

Start the broker:

./build/broker

Send a few messages from the second terminal:

./build/producer -topic orders -message "first order"
./build/producer -topic orders -message "second order"
./build/producer -topic orders -message "third order"

Now consume them from the third terminal:

./build/consumer -topic orders

You should see:

2024/06/10 11:00:01 [orders] first order
2024/06/10 11:00:01 [orders] second order
2024/06/10 11:00:01 [orders] third order

All the pieces, finally together.

What's next

This is a working message bus, but a very basic one. A few interesting directions to take it:

A consumer that keeps listening instead of pulling once and exiting (push vs. pull)
Multiple consumers on the same topic
Persistence: right now messages are gone when the broker restarts

For now though, this is exactly what I set out to build. Something short and simple, something I understand end to end.

But I doubt I'll end the series here, we'll tackle the points above (especially number 1) in future posts for sure, so stay tuned!

Hope it was useful in any way. See you in the next one!

Creating a simple Message Bus: Episode 2

Bouchaala Reda — Sun, 02 Jun 2024 09:52:58 +0000

Hello again to this series where I try to build a Message Bus, in the simplest way possible to understand the architecture and how it works.

In the first episode of this series, we talked about what a Message Bus is, why it's useful, and started off with the producer part of the code.

If you haven't read that one, please do as we're going to build on top of it.

Here's a simple diagram to help us see the components (Broker, Consumer, Producer) and how they interact with each other.

In this episode, I'm going to implement the broker part responsible of handling a new message, and storing it somewhere.

In other words, we're going to fully implement the green lines in the diagram above.

In the next episode however, we're going to tackle the consumers and that's when (hopefully, remember I don't know exactly how things are going... just going with the flow) all the pieces come together.

Enough talk, let's create out broker:

// internal/broker/broker.go

type Broker struct {
    config *BrokerConfig

    // utils
    decoder decoder.Decoder

    // core
    topics map[string]*Topic
}

The config field is used to group all configuration in one place instead of having the Broker struct bloated with fields.
Our broker needs to be able to "decode" messages sent by the producers, so a Decoder interface is needed, we'll implement it next.
The topics field is a map between topic names, and Topic objects. We'll get back to this shortly.

1. Broker config

Here's how the BrokerConfig looks like:

// internal/broker/config.go
type BrokerConfig struct {
    // fields relevant to producers
    ProducerHost string
    ProducerPort string
}

For the time being, it only contains fields relevant to the producer part, but we'll add to it along the way.

2. Decoder

The interface of the decoder is

// internal/shared/decoder/decoder.go

package decoder

import "mbus/internal/apiv1"

type Decoder interface {
    Decode([]byte) (*apiv1.Message, error)
}

And we implement it using msgpack:

// internal/shared/decoder/msgpack.go
package decoder

import (
    "mbus/internal/apiv1"

    "github.com/vmihailenco/msgpack"
)

type MsgpackDecoder struct {
}

func (*MsgpackDecoder) Decode(data []byte) (*apiv1.Message, error) {
    var msg apiv1.Message

    err := msgpack.Unmarshal(data, &msg)
    if err != nil {
        return nil, err
    }

    return &msg, nil
}

3. Topics

Let's zoom inside the broker's internals to understand what map looks like.

We receive a message from a producer on the "orders" topic (in read life, the message might be a JSON object describing the order).
We route it to the appropriate topic object using our topics map.
We add the message to the topic and diaptch

Note: Dispatch is a term used a lot in both event-based and messaging systems, and it just means delivering the event/message to all its listeners/consumers.

Because we don't have the consumer part of the system build yet, I wanted to just save the messages on the topic object itself until we build the consumer part.

To do that, we need the topic object to have a queue (First In, First Out) that we add messages to.

We can build a simple queue using a Go slice, but it's always best to use language-provided tools and mechanisms in its standard libary. We can use container/list package which implements a doubly-linked list as a queue.

Implementing a queue as a doubly-linked list is better in terms of memory reuse compared to a slice... We're not building Kafka as to worry about memory optimizations but just saying.

Alright. I can hear you say "Enough talk, show me the code."

Let's first implement a queue using container/list:

// internal/shared/queue/queue.go

package queue

import (
    "container/list"
    "mbus/internal/apiv1"
)

type MessageQueue struct {
    list *list.List
}

func New() *MessageQueue {
    return &MessageQueue{
        list: list.New(),
    }
}

func (q *MessageQueue) Enqueue(message *apiv1.Message) {
    q.list.PushBack(message)
}

func (q *MessageQueue) Dequeue() *apiv1.Message {
    if q.list.Len() == 0 {
        return nil
    }

    message := q.list.Front()
    q.list.Remove(message)
    return message.Value.(*apiv1.Message)
}

func (q *MessageQueue) Len() int {
    return q.list.Len()
}

Simple and effective.

Alright, with that done here's our Topic:

// internal/broker/topic.go
package broker

import (
    "mbus/internal/apiv1"
    "mbus/internal/shared/queue"
)

type Topic struct {
    Name  string
    Queue *queue.MessageQueue
}

func NewTopic(name string) *Topic {
    return &Topic{
        Name:  name,
        Queue: queue.New(),
    }
}

func (topic *Topic) Dispatch(message *apiv1.Message) {
    // for now, we only save the message in a queue
    topic.Queue.Enqueue(message)
}

Again, quite simple stuff here.

Now that we have everything ready, let's implement the broker part responsible of handling a new message from producers.

// internal/broker/broker.go

func (broker *Broker) ProducerListen() error {
    listener, err := net.Listen("tcp", net.JoinHostPort(broker.config.ProducerHost, broker.config.ProducerPort))
    if err != nil {
        return err
    }

    for {
        // Accept new connection
        conn, err := listener.Accept()
        if err != nil {
            return err
        }

        // read it
        data, err := io.ReadAll(conn)
        if err != nil {
            return err
        }

        message, err := broker.decoder.Decode(data)
        if err != nil {
            return err
        }

        // do something with the message
        // for now, just print it
        broker.HandleNewMessage(message)

        // close it off
        conn.Close()
    }
}

Alright, let's see what ProducerListen does:

It creates a listener on the host/port combination provided in the config (we'll get them from command-line arguments once we implement the cmd for the broker)
We start by accepting a connection on the TCP listener we created. Mind you, this call will block (i.e. put our program to sleep) until a new connection is established.
After that, we read all data from that connection
We decode the raw data into a apiv1.Message object
We call HandleNewMessage with the message (we'll work on that function shortly)
Close off the connection.
And we keep doing steps 2-6 over and over again.

Our ProducerListen call is blocking in two ways:

The call to listener.Accept will block the main thread (the main application path) so our broker won't be able to do anything else but to wait as well. That's kind of bad because we can't send/receive data to/from consumers.
After we get a connection, you can see that we do a bunch of operations on it: read it, decode it and handle it. During this time, we can't receive new messages from producers because our main thread is busi handling the message we got. So if all those steps take 1 second to finish, we can only around receive 1 message per second. Our broker throughput is not its selling point.

Both of these points can be solved by introducing goroutines which allow us to take some of the work to the "background" and not halt the entire application when doing a blocking operation.

Introducing goroutines however, will complicate things a bit because we will need to protect ourselves from race conditions using synchronization services such as a Mutex or Semaphore.

But I said in the begenning of these series: everything will be as simple as it can be.

And I meant it. I want to avoid goroutines as much as possible, unless we need to.

Our code is trivial and easy to understand that way. Which is the goal of this series anyway.

Once we finalize everything we can gradually start making things better and better.

Alright, the HandleNewMessage is quite simple:

// internal/broker/broker.go
func (broker *Broker) HandleNewMessage(message *apiv1.Message) {
    // get the topic
    topic, exists := broker.topics[message.Topic]

    // if it does not exist, create it
    if exists == false || topic == nil {
        log.Printf("creating topic %s", message.Topic)
        broker.topics[message.Topic] = NewTopic(message.Topic)
    }

    // add the message to the topic
    log.Printf("new message dispatched on topic %s", message.Topic)
    broker.topics[message.Topic].Dispatch(message)
}

We test whether we have a topic registerd with the message topic
If not, we create one using NewTopic
We call Dispatch on the topic.

If you recall from earlier, our Dispatch function from the Topic class only adds the message to a queue.

Putting it all together

We got all the pieces, now we just need to glue them together with a command-line program:

// cmd/broker/broker.go

package main

import (
    "flag"
    "log"
    "mbus/internal/broker"
)

var (
    produceHost string
    producePort string
)

func main() {
    parseFlags()

    config := &broker.BrokerConfig{
        ProducerHost: produceHost,
        ProducerPort: producePort,
    }

    broker, err := broker.New(config)
    if err != nil {
        log.Fatalf(err.Error())
    }

    // Listen for producers sending in messages
    log.Println("Broker: start listening for incoming producer messages...")
    err = broker.ProducerListen()
    if err != nil {
        log.Fatalf(err.Error())
    }
}

func parseFlags() {
    flag.StringVar(&produceHost, "phost", "127.0.0.1", "The host to listen to for producer messages")
    flag.StringVar(&producePort, "pport", "9990", "The port to listen to for producer messages")

    flag.Parse()
}

We parse command-line flags (host and port where producers send messages to)
We create a BrokerConfig object with the config we got.
We create a broker instance with our config object
Then we start listening for new messages!

Testing it out

At the end of each episode, we try to test our code to see if it works.

We do manual testing here. Nice.

Let's add this piece of code to our broker:

// internal/broker/broker.go

func (broker *Broker) TestingThisOut() {
    for _, topic := range broker.topics {
        if topic == nil {
            continue
        }

        for topic.Queue.Len() > 0 {
            message := topic.Queue.Dequeue()

            log.Printf("Received new message from topic '%s': '%s'", topic.Name, string(message.Data))
        }
    }
}

This will allow us to print any new messages we have in the queue of any topic that has any.

And let's add this to our broker command-line, between creating the broker and ProducerListen:

// cmd/broker/broker.go
// broker.New call

    ticker := time.Tick(3 * time.Second)
    go func(ticker <-chan time.Time) {
        for {
            select {
            case <-ticker:
                broker.TestingThisOut()
            }
        }
    }(ticker)

// calling ProducerListen here.

This code, will run forever in the background, periodically every 3 seconds, and will call TestingThisOut function (well thought of name) which will print messages stored in queues of topics.

Now we're ready to test.

Open two terminals. Make sure you run make to rebuild the project.

Run the broker in one of them:

./build/broker

And send a message in the second terminal

./build/producer -topic orders -message "look ma! a new order"

Get back to the terminal where you ran your broker, and you should see something like this:

2024/06/02 10:39:43 Broker: start listening for incoming producer messages...
2024/06/02 10:39:52 creating topic orders
2024/06/02 10:39:52 new message dispatched on topic orders
2024/06/02 10:39:55 Received message from topic orders: look ma! a new order

And that concludes this episode.

See you in the next one!

Creating a simple Message Bus: Episode 1

Bouchaala Reda — Sun, 26 May 2024 20:53:40 +0000

In my perpetual quest to be a better engineer and to understand tools/architectures better, I decided to start building stuff.

Build a message bus, database, reverse proxy... etc.
Whatever. Just build something I'm interested in learning more of.

To not think about this as a huge task, I decided to commit myself to build stuff in the simplest way possible. No fancy shenanigans.

Start small and simple. Add more along the way.

I'll be working with Go, not because I'm a Go expert, but because I like it and I feel like it helps with my productivity. I'll probably be learning more about Go along the way. Two birds with one stone kind of thing.

I also want to point out, that I write a post after I'm done with a portion of the code, sort of journaling my way through it.

which means that, the code could be incomplete or does not work (as I'm writing this, I'm thinking but that's what tests are for, but lets leave the tests for some other time). And it also means I'll probably be jumping between files a lot.

I wanted to start with a message bus.

Let's define it and start this series of posts by creating the project structure and maybe a bit more.

A message bus, is a messaging system that allows different systems to communicate with each other via sending and receiving of these messages.

So this message bus, is a system (also called the broker) that allows senders (also called producers) to send messages (just data, it could contain anything) to receivers (also called consumers).

In other words,

A producer prepares a messages, points it to the broker and says "here, deliver this message please" to this destination
The broker gets the message and delivers it to one or more consumers that are subscribing to said destination.

(image source: here)

Project layout

So we have three actors: a broker, consumer and producer.

Let's start by creating an empty project structure. I'll call the go module mbus. Short and nice.



# Create the dir and cd into it
mkdir mbus
cd mbus

# Create the go module
go mod init mbus

# Create the project layout
mkdir cmd internal build
mkdir cmd/{broker,producer,consumer}
mkdir internal/{broker,producer,consumer}

Our base project layout is created.

To make our lives easier, let's create a very simple Makefile



all: clean build

.PHONY: build
build:
    go build -o build/broker cmd/broker/broker.go
    go build -o build/producer cmd/producer/producer.go
    go build -o build/consumer cmd/consumer/consumer.go

.PHONY: clean
clean:
    rm -f build/broker
    rm -f build/consumer
    rm -f build/producer

So running make in the command line will rebuild our project. You could use something like gowatch, but again I'm keeping it simple.

Message structure

Let's define what "message" is in our application.

It needs to have some data, it could be json, it could be Base64 encoded image... we don't know and we don't care.
It needs to have some sort of destination name, for us to know where to send it to. In the "message bus" world, it's often called a "topic" or a "routing key" if you want to sound like a real nerd. I like "routing key" but let's use topic since it's shorter.

The message will be our contract between all parties, so let's call it apiv1 and put it inside internal, like so



mkdir internal/apiv1
touch internal/apiv1/message.go



// internal/apiv1/message.go

package apiv1

type Message struct {
    Data  []byte
    Len   int
    Topic string
}

func NewMessage(topic string, data []byte) *Message {
    return &Message{
        Data:  data,
        Len:   len(data),
        Topic: topic,
    }
}

Nice and simple.

The Len field is something we might not use, but when dealing with slices it's always a good idea to keep the length of it around. We'll see, if we don't need it we can just remove it later on.

Now, let's create the "producer" part of the app and call it a day.

Producing messages

If you remember from our intro, a producer is very simple: it has a message and a topic, and it just sends them off to a broker.

Knowing that, let's create a command line app that will accept a host and port pair to point it to the broker, a topic and a message.



// cmd/producer/producer.go
package main

import (
    "flag"
    "log"

    "mbus/internal/producer"
)

var (
    brokerHost string
    brokerPort string
    topic      string
    message    string
)

func main() {
    parseFlags()

    client := producer.New(brokerHost, brokerPort)

    err := client.Publish(topic, message)
    if err != nil {
        log.Fatalf(err.Error())
    }
}

func parseFlags() {
    flag.StringVar(&brokerHost, "host", "127.0.0.1", "Broker host")
    flag.StringVar(&brokerPort, "port", "9990", "Broker port")
    flag.StringVar(&topic, "topic", "", "Topic to produce the message for")
    flag.StringVar(&message, "message", "", "The message contents")

    flag.Parse()

    if topic == "" {
        log.Fatalf("please provide a topic")
    }

    if message == "" {
        log.Fatalf("please provide a message to be sent")
    }
}

Parsing flags to get command line arguments,
Creating a client by using mbus/internal/producer package (which we'll create after this)
Publishing the message to the topic, using the client.

The interesting stuff is at internal/producer/producer.go which we'll create in a minute, first I want to show you what a Producer looks like.



// internal/producer/producer.go
type Producer struct {
    host string
    port string

    conn net.Conn

    encoder encoder.Encoder
}

The first two fields are there to know where the broker is in our network.
The second field, represents the TCP connection to the broker.
The next one is the encoder. More on this bellow.

In order for us to send a Message object down the wire, we need to properly encode it to binary. We have a bunch of options in Go, but I'll go with msgpack. (Offical Website)

The encoder.Encoder is an interface so we can swap out the msgpack implementation with another one.

I'm used to a lot of OOP so that encoded is embedded inside the publisher (composition), but I realize that maybe that's not the best way to things all the time.

But it works for now, so let's leave it be.



// Creating a shared folder for all shared things
mkdir -p internal/shared/encoder

The Encoder interface is pretty simple:



// internal/shared/encoder/encoder.go

package encoder

import "mbus/internal/apiv1"

type Encoder interface {
    Encode(*apiv1.Message) ([]byte, error)
}

Let's create a msgpack encoder, but first let's install the msgpack package:



go get -u github.com/vmihailenco/msgpack



// internal/shared/encoder/msgpack.go
package encoder

import (
    "mbus/internal/apiv1"

    "github.com/vmihailenco/msgpack"
)

type MsgpackEncoder struct {
}

func (e *MsgpackEncoder) Encode(msg *apiv1.Message) ([]byte, error) {
    data, err := msgpack.Marshal(msg)
    if err != nil {
        return nil, err
    }

    return data, nil
}

Pretty simple stuff.

Now let's get back to our producer by creating a constructor method:



// internal/producer/producer.go

func New(host, port string) *Producer {
    return &Producer{
        host:    host,
        port:    port,
        conn:    nil,
        encoder: &encoder.MsgpackEncoder{},
    }
}

Here, we create a new Publisher and use MsgpackEncoder for decoding.

Now, let's add a method to the Publisher so we can start publishing messages:



// internal/producer/producer.go
func (c *Producer) Publish(topic, message string) error {
    // we could connect in the New function before returning
    // but it's better to defer it and call it here, whenever
    // the user tries to publish a message.
    err := c.connect()
    if err != nil {
        return err
    }

    msg := apiv1.NewMessage(topic, []byte(message))

    data, err := c.encoder.Encode(msg)
    if err != nil {
        return err
    }

    n, err := c.conn.Write(data)
    if err != nil {
        return err
    }

    if n != len(data) {
        return errors.New("could not write all data")
    }

    return nil
}

func (c *Producer) connect() error {
    conn, err := net.Dial("tcp", net.JoinHostPort(c.host, c.port))
    if err != nil {
        return nil
    }

    c.conn = conn
    return nil
}

Again very simple.

We connect to the broker, create a Message object, encode it, and send it to the broker using the connection established.

That's it. Producer part done. I told you the producer is the easiest one.

Next one will be the broker.

But first, let's at least manually test (since we don't have unit tests, lazy me) that our producer is actually sending stuff somewhere.

For that, we can use netcat for this. Run this command in another terminal:



nc -l -p 9990 -t 127.0.0.1

This will tell netcat (nc) to listen for TCP connections, on 127.0.0.1 port 9990. Kind of like a temporary test double for our broker 😁

Now, let's compile our app and run the producer:



make
./build/producer -topic sales -message hey

You should see something printed on the terminal where you ran nc

Done, test coverage 100%.

Jokes aside, we'll probably add tests in another episode.

But for now, we'll call it a day.

Like I said at the start of this post, I'm just starting out with this so I don't really know where I'm going with this, and that's part of the fun. But it also means you could find the code not working or incomplete.

In any way, if you find a mistake or have some feedback, I'd love to hear it.

until then, see you in another episode!

How To Safely Verify MACs With Go And PHP Examples

Bouchaala Reda — Thu, 23 Feb 2023 23:06:51 +0000

I wanted to write this short article to explain the recent learning I had while reading on Message Authentication Codes (MACs).

First, let's briefly explain what MACs are and how they can be useful to us.

MACs (Message Authentication Code) are cryptographic algorithms that use a hash function and a secret key (they are also called symmetric cryptographic algorithms) to generate what's called an authentication tag or just tag, which is used to verify the integrity (the message wasn't changed) & authenticity (the sender's identity) of the message given knowledge of the secret key.

The most widely used MAC in the wild is what's called an HMAC: Hash-based MAC. Like its name suggests, it's a way to use hash function with a secret key. HMACs can be used with SHA128, SHA256, SHA512... etc. The result of an HMAC function is called hmac or tag. I'll use the lowercase hmac from here on in this article.

In other words, if you and I agree on a secret key and a hash algorithm to use we can effectively share messages, that only we can verify the authenticity & integrity of. No one can tamper with our messages in transit.

Let's start off by creating an example Go program that creates an HMAC of some data, and then verifies it, using SHA256 hashing algorithm and a strong, randomly generated secret key.

package main

import (
    "crypto/hmac"
    "crypto/sha256"
    "encoding/hex"
    "fmt"
    "strconv"
)

var secretKey = "randomly generated, secure secret key"
var data = "This is private information"

func hmac(data, secretKey string) string {
    hash := hmac.New(sha256.New, []byte(secretKey))
    hash.Write([]byte(data))

    return hex.EncodeToString(hash.Sum(nil))
}

func verify(data, hmac, secretKey string) bool {
    // Recompute the signature
    correctHmac := sign(data, secretKey)

    // Check if they are equal
    return correctHmac == hmac
}

func main() {
    hmac := hmac(data, secretKey)
    fmt.Printf("Data: %s\nHMAC: %s", data, hmac)

    fmt.Println()

    correct := verify(data, hmac, secretKey)
    fmt.Printf("\nIs HMAC %s correct ? %s", hmac, strconv.FormatBool(correct))
}

And here's a PHP example of the same functionality:

<?php

const secretKey = "randomly generated, secure secret key";
const data = "This is private information";

function hmac(string $data, string $secretKey): string {
    return hash_hmac(algo: 'sha256', data: $data, key: $secretKey);
}

function verify(string $data, string $hmac, string $secretKey): bool {
    $correctHmac = sign($data, $secretKey);

    return $correctHmac === $hmac;
}

function run() {
    $hmac = hmac(data, secretKey);
    print sprintf("Data: %s\nHMAC: %s", data, $hmac);

    print "\n";

    $correct = verify(data, $hmac, secretKey);
    print sprintf("Is HMAC %s correct ? %s", $hmac, $correct ? "true" : "false");
}

run();

Both programs output are:

Data: This is private information
HMAC: cf824e1bfbc628aca8709483622f9c2aaf7ac9e57d50623d225cc76d8ad578fc
Is HMAC cf824e1bfbc628aca8709483622f9c2aaf7ac9e57d50623d225cc76d8ad578fc correct ? true

The Problem with our code

Using == to compare hashes makes us vulnerable to a type of attack called Timing Attack. Because the == comparison returns as soon as it detects that the two hmacs differ, it's not suitable for this use case.

Because, an attacker can recreate the valid hmac by measuring how long it takes for the comparison to finish.

To put this into better perspective, let's make an example. Say we have 8 bit hmacs and each bit takes exactly 1ms to compare it to the other hmac bit. The valid signature looks like this: 0001 1010

If I send you the correct hmac, then it should take exactly 8ms for you to verify that it is in fact valid.
If I now send you 1000 0000, you'll reply in less than 1ms that it's incorrect, because the first bit is incorrect. Remember it takes 1ms to verify each bit.
If I send you 0001 1001, your reply will take about 6ms because the first 6 bits are correct, but not the last 2.
... etc. and I keep doing this until you reply with 8ms which then I know that it's the valid hmac.

And like that, I was able to re-create the valid hmac by updating my forged hmac and measuring how long it took you (the secret key owner) to verify it. Of course this is an overly simplified example, authentication tags produced by MACs should be at least 128 bit to be strong enough and to avoid collisions.

The takeaway though is that MAC tags comparison/verification should ALWAYS be done in constant time. In our simplified example, whether the hmacs were equal or not it should always take the same 8ms to return with the result.

Note that this also applies to Digital Signatures (the asymmetric brother to MAC). Signatures also need to be verified in constant time to avoid timing attacks.

The Solution

I purposefully used the == operator in both languages to illustrate this problem but we should always use good and well known libraries when it comes to cryptographic operations and algorithms.

In Go, we can verify hmacs in constant-time using hmac.Equal() from crypto/hmac package.
In PHP, we can compare hmacs and hashes using hash_equals() function.

Let's rewrite our Go & PHP examples using the correct way. I'll just rewrite the verify function as it's all we need.

func verify(data, signature, secretKey string) bool {
    // Recompute the signature
    correctSignature := sign(data, secretKey)

    // Check if they are equal
    return hmac.Equal([]byte(correctSignature), []byte(signature))
}

and the PHP example will look like this

function verify(string $data, string $signature, string $secretKey): bool {
    $correctSignature = sign($data, $secretKey);

    return hash_equals($correctSignature, $signature);
}

How to do constant-time comparison

Let's dive a bit deep and see how hmac.Equals() is implemented. The source code reveals that it actually uses a function from the crypto/subtle package called ConstantTimeCompare, which is implemented like this:

func ConstantTimeCompare(x, y []byte) int {
    if len(x) != len(y) {
        return 0
    }

    var v byte
    for i := 0; i < len(x); i++ {
        v |= x[i] ^ y[i]
    }

    return ConstantTimeByteEq(v, 0)
}

We return early if the two slices are not of equal length, because in that case we know that entirely different hashing algorithms were used.
Next, create a byte variable v which will hold our result.
Then we iterate over the slices and we XOR each slice element and the result is ORed with the variable v. That is, if the two bytes we're comparing are different the XOR will be 1, ORed with v, v will be equal to 1, and will stay 1 for the entirety of the loop. v will only be 0 if all bytes from the two slices are the same.
Then, taking another precaution, and we also compare the two bytes v and 0 in a time constant manner using ConstantTimeByteEq.

Whether the two slices were equal or not, the function takes the exact same time to return, which makes it timing-attack safe.

And that's it! Thanks for reading and until the next episode, have a great day!

Secret Key Encryption with Go using AES

Bouchaala Reda — Fri, 20 Jan 2023 14:41:45 +0000

I was working on a personal Go project when I needed the ability to encrypt & decrypt arbitrary pieces of data (strings or JSON payloads or whatever) using a secret key. This article is the result of all the learning I had in the last couple of days while trying to achieve that.

TL;DR

Secret key (or Symmetric) encryption requires the use of a block cipher such as AES. AES by itself can only encrypt/decrypt 16 byte long data (which is its block size), hence the need to use a block cipher mode.

GCM block cipher mode is one of many modes. It is a method that uses AES in a way that enables us to encrypt/decrypt arbitrary sized data. It also adds message authentication (integrity).

The complete working code example is here.

A Cryptography primer

Before diving in, I'd like to define some Cryptography terminology.

A secret key is basically an agreed upon value between communicating parties (machines, you and your friends ... or any combination of that).
Plaintext is the text that you want to encrypt so that no one else can read it unless that someone has the secret key.
Ciphertext is the encryption result of the plaintext.
A cipher is the algorithm that does the actual work of encrypting and decrypting our data.

What we are talking about here is called Symmetric encryption (also called Secret key encryption) whereby we use the same key to encrypt our plaintext to ciphertext and to revert the ciphertext back to plaintext.

Cipher types

Now let's talk a bit about ciphers. Symmetric ciphers belong to two main categories:

Stream ciphers: operate by encrypting each bit (or byte) of the plaintext at a time, producing the ciphertext.
and Block ciphers: operate by encrypting fixed-sized blocks of plaintext.

Stream and Block ciphers are the building blocks (no pun intended) of more complicated and siphisticated cryptographic utilities such as MACs, hash functions, symmetric-key digital signature schemes and much more.

Now let's take one example of a Block cipher and dive more into it: AES, Advanded Encryption Standard.

AES is a block cipher that takes a fixed-size key and fixed-size plaintext, and returns fixed-size ciphertext. AES has three variants that are selected based on the secret key length, all of which use a fixed-sized block of 16 bytes (or 128 bits).

Secret Key Length	AES Variant	Block Size
16 bytes (128 bits)	AES-128	16 bytes (128 bits)
24 bytes (192 bits)	AES-192	16 bytes (128 bits)
32 bytes (25 6bits)	AES-256	16 bytes (128 bits)

Because the block size of AES is set to 16 bytes, the plaintext must be at least 16 bytes long. Which causes a problem for us since we want to be able to encrypt/decrypt arbitrary sized data.

Let's put that aside for now and let's have a look at an example of encrypting some data using AES.

Let's give it a try.

package main

import (
    "crypto/aes"
    "fmt"
)

var (
    // We're using a 32 byte long secret key
    secretKey string = "N1PCdw3M2B1TfJhoaY2mL736p2vCUc47"
)

func encrypt(plaintext string) (string) {
    aes, err := aes.NewCipher([]byte(secretKey))
    if err != nil {
        panic(err)
    }

    // Make a buffer the same length as plaintext
    ciphertext := make([]byte, len(plaintext))
    aes.Encrypt(ciphertext, []byte(plaintext))

    return string(ciphertext)
}

func main() {
    // This will successfully encrypt.
    ciphertext := encrypt("This is some sensitive information")
    fmt.Printf("Ciphertext: %x \n", ciphertext)

    // This will cause an error since the
    // plaintext is less than 16 bytes.
    ciphertext = encrypt("Hello")
}

Block Cipher Modes

Now back to our problem: using AES alone will not be enough to get what we want which is to: Encrypt arbitrary sized messages.

This is where Block Cipher modes (or Block modes of Operation) come in. A Block Cipher mode is a method that uses the Block cipher to solve a particular problem. There's a lot of Block Cipher modes available and the majority of them solve the problem of "How do I encrypt an arbitrary sized message". That's exactly what we want!

I'm by no means a Cryptography expert so I can't and I won't compare block cipher modes or dive into their details. But here's what we need to undertstand:

Block ciphers by themselves have two limitations:

They can only encrypt/decrypt fixed-sized data
They only provide confidentiality which means one looking at ciphertext cannot possibly revert it back to plaintext without knowing the secret key

Block cipher modes were created to solve those two limtations:

All block cipher modes solve the size limitation.
Some modes only solve the size limitation and nothing else, like ECB, CBC & CTR.
Other modes were created to also add authentication or message integrity (these are called combined modes) like: CCM, GCM & TKW.

By using one of the modes that also provide authentication, we are effectively using Authenticated Encryption, as opposed to Block encryption. Here's a recap to get the full picture:

AES is a Block Cipher that gives us confidentiality. This is *Block Encryption *.
When using AES alone, you can only encrypt/decrypt data that is 16 bytes long (which the size of an AES block).
Using AES with CBC mode for example, aleviates the size limitation. This is also Block Encryption.
Using AES with GCM (a combined mode) aleviates the size limitation but also gives us message authentication (integrity). This is Authenticated Encryption.

Alright, we'll be using GCM mode bacause it's one of the most widely adopted symmetric block cipher modes. GCM requires an IV (initialization vector) that should ALWAYS be randomly generated (the term used here is nonce, which is pretty much the same). We're just using a random string in our example.

Using AES with GCM

Let's see how we can do that in Go:

package main

import (
    "crypto/aes"
    "crypto/cipher"
    "crypto/rand"
    "fmt"
)

var (
    // We're using a 32 byte long secret key.
    // This is probably something you generate first
    // then put into and environment variable.
    secretKey string = "N1PCdw3M2B1TfJhoaY2mL736p2vCUc47"
)

func encrypt(plaintext string) string {
    aes, err := aes.NewCipher([]byte(secretKey))
    if err != nil {
        panic(err)
    }

    gcm, err := cipher.NewGCM(aes)
    if err != nil {
        panic(err)
    }

    // We need a 12-byte nonce for GCM (modifiable if you use cipher.NewGCMWithNonceSize())
    // A nonce should always be randomly generated for every encryption.
    nonce := make([]byte, gcm.NonceSize())
    _, err = rand.Read(nonce)
    if err != nil {
        panic(err)
    }

    // ciphertext here is actually nonce+ciphertext
    // So that when we decrypt, just knowing the nonce size
    // is enough to separate it from the ciphertext.
    ciphertext := gcm.Seal(nonce, nonce, []byte(plaintext), nil)

    return string(ciphertext)
}

func decrypt(ciphertext string) string {
    aes, err := aes.NewCipher([]byte(secretKey))
    if err != nil {
        panic(err)
    }

    gcm, err := cipher.NewGCM(aes)
    if err != nil {
        panic(err)
    }

    // Since we know the ciphertext is actually nonce+ciphertext
    // And len(nonce) == NonceSize(). We can separate the two.
    nonceSize := gcm.NonceSize()
    nonce, ciphertext := ciphertext[:nonceSize], ciphertext[nonceSize:]

    plaintext, err := gcm.Open(nil, []byte(nonce), []byte(ciphertext), nil)
    if err != nil {
        panic(err)
    }

    return string(plaintext)
}

func main() {
    // This will successfully encrypt & decrypt
    ciphertext1 := encrypt("This is some sensitive information")
    fmt.Printf("Encrypted ciphertext 1: %x \n", ciphertext1)

    plaintext1 := decrypt(ciphertext1)
    fmt.Printf("Decrypted plaintext 1: %s \n", plaintext1)

    // This will successfully encrypt & decrypt as well.
    ciphertext2 := encrypt("Hello")
    fmt.Printf("Encrypted ciphertext 2: %x \n", ciphertext2)

    plaintext2 := decrypt(ciphertext2)
    fmt.Printf("Decrypted plaintext 2: %s \n", plaintext2)
}

As you can see, we can now encrypt and decrypt any arbitrary sized data! If you want to store the ciphertext somewhere, you'd have to encode it with encoding/base64 or encoding/hex or something similar.

This is just an example and shouldn't be used as is in anything real.

In a real world project, one should probably use a better library to encrypt/decrypt sensitive information (like libsodium), but by figuring out how to do that with Go's standard library and by using symmetric cryptography primitives such as block ciphers and block modes, you understand more about they they relate to each other.

Again, I am by no means a security expert. This article is just me sharing my learnings so please let me know in the comments if there's anything I missed, anything you didn't understand or any other general feedback!

Thanks for reading and have a lovely day!

Using OpenStreetMap For Your Web Projects

Bouchaala Reda — Thu, 18 Aug 2022 19:02:00 +0000

Note: This is an old article I published in my personal blog on August 2019. I now publish articles on dev.to so I stopped my blog and moved this article here.

What's wrong with Google Maps?

Nothing. But... let's consider this scenario:
You're working on a web project, say an e-commerce platform for a real-estate company based in Algeria (where I currently live). One of the project's requirements is to show a map of the product's location, on its details page.

You could go with Google Maps, but since they changed to a pay-as-you-go pricing model, you should probably think twice before just jumping in.

Using Google Maps could be the perfect solution in your case but for me it wasn't. I had to look for an alternative.

OpenStreetMap to the rescue

First of all, what is OpenStreetMap ?
OpenStreetMap is a collaborative project, that aims to provide free, continuously updated geographic data of the world.

So can we use their maps? and how? Let's have a look.

OpenStreetMap usage requirements

OpenStreetMap data is licensed Open Data Commons Open Database License (ODbL),
and the map itself is licensed under CC BY-SA 2.0. That means...

To be able to use OpenStreetMap in your websites, you need adhere by two perfectly reasonable requirements

You must display a proper attribution text in/around your map. Since we're using the map and the data, we'll add © OpenStreetMap contributors text inside the map (more on that later), and link it to the copyright page.
When expecting high traffic (in our case we are), you must avoid flooding OpenStreetMap servers. You can read more about the tile server usage policy here.

Show me the code

Enough talk, we will now setup a map, with respect to the requirements mentioned above.

I'm going to use NodeJS to setup a map tile server proxy, I found this library that can help us with that.

Let's install everything we'll be needing

npm install --save tilestrata tilestrata-proxy tilestrata-disk tilestrata-headers

Let's review what we installed:

Tilestrata is the main server code.
TileStrata Proxy will help us proxy requests to OpenStreetMap servers.
TileStrata Disk, will help us cache the map tiles, to avoid flooding the proxied servers.
TileStrata Headers to help us with client-side caching of map tiles.

Let's first setup a config file

module.exports = {
    // Host:Port to listen on
    host: '127.0.0.1',
    port: 12345,

    cache: {
        lifetime: 3600 * 24 * 7 * 4, // Cache for a month
        refresh: 3600 * 24 * 14, // 14 days refresh

        // The map-server script needs to be lunched from the project root
        dir: './map_tiles_cache',
    },
};

and now the server code:

var tilestrata = require('tilestrata');
var headers    = require('tilestrata-headers');
var proxy      = require('tilestrata-proxy');
var disk       = require('tilestrata-disk');
var config     = require('./config');
var util       = require('util');

var strata = tilestrata();
strata.layer('osm') // highlight-line
    .route('*.png')
    .use(disk.cache({
        dir: config.cache.dir,
        refreshage: config.cache.refresh,
        maxage: config.cache.lifetime,
    }))
    .use(proxy({
        uri: 'http://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png', // highlight-line
        subdomains: ['a', 'b', 'c'],
    }))
    .use(headers({ // highlight-line
        'Cache-Control': 'public, max-age=' + config.cache.lifetime, // highlight-line
        'Expires': config.cache.lifetime, // highlight-line
    }));

process.stdout.write(util.format(
    "Map server listening on http://%s:%s\nPress Ctrl-C to quit.\n",
    config.host, config.port
));
strata.listen(config.port, config.host);

Let's review the highlighted lines

We called our map layer osm, you can call this whatever you want.
We proxy the request to OpenStreetMap tile servers, the variables are
- s means sub-domain, and it's picked at random from the arrow bellow the highlighted line. Note that this is a required behavior.
- z is the zoom level, x and y and numbers indicating the column and row the map. You don't have to worry about all this parameters, they will be passed automatically by our client map library. We'll still be using latitudes and longitudes like we're used to.
Then we setup client side caching, using the Cache-Control and Expires header.

Now that our map server code is done, let's start it and use our map!

mkdir map_tiles_cache # for caching files
node mapserver.js

Let's create a simple page that displays our map. We'll be using Leaflet JS for that.

<html>
<head>
    <title> OpenStreenMap Example </title>
    <meta charset="utf-8">
    <link rel="stylesheet" type="text/css" href="https://cdnjs.cloudflare.com/ajax/libs/leaflet/1.5.1/leaflet.css">
    <script src='https://cdnjs.cloudflare.com/ajax/libs/leaflet/1.5.1/leaflet.js'></script>
</head>

<body>
    <div id="map" style="height: 700px; width: 100%;"></div>

    <script>
        var map = L.map('map', {
            center: [36.7597907,3.0665139], // The capital of Algeria
            zoom: 9, // default zoom level
        });

         L.tileLayer('http://127.0.0.1:12345/osm/{z}/{x}/{y}.png', {
             attribution: '&copy; <a href="https://www.openstreetmap.org/copyright">OpenStreetMap contributors</a>'
        }).addTo(map);
    </script>
</body>

And we're done! We have ourselves a free, flexible map to use in our website.
Checkout the Leaflet website for more documentation on how to ineract with your new map.

One more thing...

Depending on the use case, the cache disk space can grow very large.

As an example, The north side of Algeria, viewed from from zoom level 6 all the way up to zoom level 18 (which is the maximum zoom level supported by OpenStreetMap) will be around 30GB (You can calculate your own area with this calculator tool from Geofabrik).

You have two options to fix that

Delete the cache folder periodically.
Setup map boundaries for the map.

The first option is self-explanatory, so let's look at the second one.

From the docs, LeafletJS has map boundaries,
which restricts the view to the given geographical bounds,
bouncing the user back if they try to pan outside the view.

Let's set that up. I got the bounds for Algeria here (you can get bounds for any country).

// The bounds object
// representing a fixed rectangular area
var bounds = L.bounds(
    L.point(-8.704895, 18.92874), 
    L.point(12.03598, 37.77284)
);

// Set the max bounds
map.setMaxBounds(bounds);

And we're done!

Of course everything has its pros and cons, and you should decide what's better suited for you own needs.

But as you can see, it was pretty easy to setup a map server based on OpenStreetMap and get started with it, so it's good to have it in your arsenal.

And that's about it! I hope this post was of help to you.
Make sure to drop a comment if you have any question or feedback. Thanks!

Migrating a WordPress blog from subdirectory to subdomain without loosing URL structure with Nginx

Bouchaala Reda — Thu, 18 Aug 2022 18:43:32 +0000

If you just want to look at the full Nginx config used jump to the end of the post.

From an SEO perspective using a subdirectory or subdomain for your blog/site is a subject of debate. Have a look at this article for an example.

The goal of this post however, is to help you to physically move your WordPress site from your subdirectory to a subdomain without actually having to change your URL structure (i.e. still looks like it's a subdirectory).

The reasons why you'd want to do this may vary. Here are mine:

We had a WP blog that lived in the same repository as the main website (let's call it site.com).
In order for the SEO team to manage the blog, they had to go through the tech team every time they needed something changed/added.
We have a lot of blog traffic, and we didn't want to lose any traffic nor SEO credit by moving everything from site.com/blog to blog.site.com.

So the requirements for this migration are now clear:

Move the WP blog from the site.com repository to a managed WordPress hosting provider like HostGator, Hostingr... etc.
The blog under site.com/blog should still be accessible by visitors as normal, and it should serve the new managed blog at blog.site.com.
blog.site.com MUST NOT be accessible directly by visitors, only via site.com/blog.
Blog traffic must be served with HTTPS.

Requirement number 1 is quite straightforward. We are left with requirements 2 to 4.

The way I did was by adding an Nginx reverse proxy on the main website (site.com) to serve the blog contents from blog.site.com as if it was hosted under site.com/blog. So let's make a start on that.

Here's a diagram of how things work. We'll dive into the Nginx config next.

Nginx Reverse Proxy Config

Basic reverse proxy config looks like this



# Any URL path that starts with blog will be using this config block.
location /blog {
    # Request paths coming into our main site will be /blog/something
    # But we want to send requests to blog.site.com as /something
    # So we use rewrite to strip /blog/ from the request path
    rewrite /blog/(.*) /$1  break;

    proxy_pass http://blog.site.com;
}

This is the base config we can work with. Nginx basically catches any request made to /blog and fetches the contents from blog.site.com. A reverse proxy at its simplest form.

Problem 1: We're not using HTTPS

But as you can see, we're using HTTP and not HTTPS and that does not fulfill requirement number 4. So let's configure Nginx to use HTTPS.

Let's configure Nginx to use HTTPS & secure it using HTTP Basic Auth.



location /blog {
    # ...
    rewrite /blog/(.*) /$1  break;

    # SSL config for proxy
    proxy_ssl_server_name on;                   # 1
    proxy_ssl_session_reuse on;                 # 2
    proxy_set_header Host blog.site.com;        # 3
    proxy_set_header X-Forwarded-Proto https;   # 3
    proxy_set_header X-Forwarded-Port 443;      # 3
    proxy_set_header X-Real-IP $remote_addr;    # 3
    proxy_set_header X-Forwarded-Host $host;    # 3

    proxy_set_header Authorization "Basic {CREDENTIALS}"; #4

    # Proxy
    proxy_pass https://blog.site.com;
}

Here's an explanation of the directives we added to our config:

proxy_ssl_server_name on; Will force Nginx to use TLS SNI (Server Name Indication) which is required in this case because we are trying to serve two different websites with two different SSL certificates in one server using one IP address. With this directive, Nginx knows which SSL certificate to use. Note that Support for SNI was introduced in Nginx 1.7.0. So make sure you're using 1.7.0+.
proxy_ssl_session_reuse on; Will re-use the previous negotiated connection to do an abbreviated SSL handshake which is better than doing a full handshake each time we try to connect, the latter is CPU intensive. This is a performance improvement.
The proxy_set_header directive is used to set some required and informational headers to be sent along with the request to blog.site.com. Eg: setting the correct Host header.
Here we set the Authorization header to a Basic (HTTP Basic Auth). The username/password need to be configured at blog.site.com level (if you're using managed WP hosting, they'll definitely have an HTTP Basic Auth section somewhere in the site config). CREDENTIALS is just a placeholder, replace it with a base64 encoding of username:password.

Our Nginx config is now ready, and it will start serving the blog traffic from blog.site.com as requested.

Problem 2: Incorrect links in blog pages

We're now faced with another problem. The WP blog in blog.site.com will have all page links point to blog.site.com/page-url exposes our managed blog and breaks our requirements.

That means that when a visitor first opens our blog, everything will look good, but whenever the visitor clicks on any link on the page, they'll be redirected to blog.site.com/page-url. Definitely not what we want.

Fortunately, Nginx can help us with that as well. The solution is basically to use Nginx's ngx_http_sub_module which will help us modify the response from blog.site.com by replacing string occurrences with other ones, before sending it to the visitor. Let's see how we might do that by adding to our previous Nginx config



location /blog {
    # ...
    rewrite /blog/(.*) /$1  break;

    # SSL config for proxy
    proxy_ssl_server_name on;                   # 1
    proxy_ssl_session_reuse on;                 # 2
    proxy_set_header Host blog.site.com;        # 3
    proxy_set_header X-Forwarded-Proto https;   # 3
    proxy_set_header X-Forwarded-Port 443;      # 3
    proxy_set_header X-Real-IP $remote_addr;    # 3
    proxy_set_header X-Forwarded-Host $host;    # 3

    proxy_set_header Authorization "Basic {CREDENTIALS}"; #4

    # 5
    proxy_set_header Accept-Encoding "";

    # 6
    sub_filter_once off;
    sub_filter_last_modified on;
    sub_filter_types text/html text/css text/xml text/javascript application/json;
    sub_filter 'blog.site.com' 'site.com/blog';

    # 7
    sub_filter 'src="/wp-content/' 'src="/blog/wp-content/';

    # 8
    sub_filter 'http:' 'https:';

    # Proxy
    proxy_pass https://blog.site.com;
}

Let's explain what we added there:

Disable response compression which is required to be able to change the response.
We substitute every occurrence of blog.site.com in the response with site.com/blog on all HTML, CSS, JS, XML & JSON response types.
We prefix absolute asset URLs sent by WP (/wp-content/) with /blog/
We just replace all insecure links with secure ones so that we don't get any browser errors, since we are using HTTPS on our main site and also between site.com & blog.site.com.

Problem 3: Incorrect blog redirects

The last problem we have is a tricky one to solve. The only good (I say good here because I probably could've used if directive, but we know that it causes problems when used in a location block) solution I found to work is kind of a hack, so if you have better ideas please let me know in the comments.

The problem is, whenever the blog actually returns a redirect response, the link in the Location header will be incorrect in some cases. Sometimes the blog returns relative URLs without https:// (eg: /faq-page), and sometimes it returns absolute URLs that are complete and start with https://. We only need to replace Location header links if they relative links and not absolute ones.

With the help of Nginx's ngx_http_map_module and the map directive (think of it as a simple switch/case statement), we can create a dynamic variable (Its value depends on other values/variables) that will be hold the prefix that we need to add to the Location header link.

Add this section before the server block of your Nginx config:



map $upstream_http_location $_upstream_http_location_prefix { # 1
  default $upstream_http_location; # 2
  "~^/"                 "/blog";   # 3
  "~*^http"             "";        # 4
  "~*^((?!http|\/).)*"  "/blog/";  # 5
}

Let's explain what each line does:

The first line uses the map directive to create a dynamic variable called $_upstream_http_location_prefix whose value depend on $upstream_http_location variable. The latter holds the Location value sent by the upstream (our blog.site.com).
The default value of the variable shall be the Location header value itself. This is just to be on the safe side, although this is probably never going to happen because our case statements are quite mutually exclusive.
If the Location header value starts with /, then the prefix will be /blog.
If the Location header value starts with http then the prefix will be empty.
If Location header value neither starts with / nor http then the prefix will be /blog.

Now that we have our Location header prefix ready, we can use it in out location /blog block as so.



location /blog {
    # ...

    proxy_hide_header Location; # 1
    add_header Location "$_upstream_http_location_prefix$upstream_http_location"; # 2

    # Proxy
    proxy_pass https://blog.site.com;
}

We first hide the original Location header sent to us by the upstream (our blog).
Add a new Location header whose value is OUR_CALCULATED_PREFIX + ORIGINAL VALUE and by doing that we effectively re-wrote the header value depending on what it starts with.

Problem 4: Visitors can access WP admin login via /blog

I really didn't want the admin login page to be accessible via site.com/blog so I think we'll be better off if we just hide it completely. Anyone who's interested in logging in to WP admin site need to go to blog.site.com, login using HTTP Basic Auth then login using his/her WP account credentials.

We can easily do that by adding a couple of location blocks



# Return the blog's 404 page when accessing WP login/admin
location /blog/wp-admin { return 301 /blog/404; }
location /blog/wp-login.php { return 301 /blog/404; }

# Prevent the blog's robots.txt from being proxied
location /blog/robots.txt { return 404; }

location /blog {
    # ...
}

Final Nginx config

Here's the full Nginx config if you want to copy and paste the whole thing.

Before the server block, add this:



map $upstream_http_location $_upstream_http_location_prefix { # 1
  default $upstream_http_location; # 2
  "~^/"                 "/blog";   # 3
  "~*^http"             "";        # 4
  "~*^((?!http|\/).)*"  "/blog/";  # 5
}

Then inside the server block of you site add this. Make sure to replace site.com with your website URL.



# Return the blog's 404 page when accessing WP login/admin
location /blog/wp-admin { return 301 /blog/404; }
location /blog/wp-login.php { return 301 /blog/404; }

# Prevent the blog's robots.txt from being proxied
location /blog/robots.txt { return 404; }


location /blog {
    # strip /blog/ from the request path
    rewrite /blog/(.*) /$1  break;

    # SSL config for proxy
    proxy_ssl_server_name on;                   # 1
    proxy_ssl_session_reuse on;                 # 2
    proxy_set_header Host blog.site.com;        # 3
    proxy_set_header X-Forwarded-Proto https;   # 3
    proxy_set_header X-Forwarded-Port 443;      # 3
    proxy_set_header X-Real-IP $remote_addr;    # 3
    proxy_set_header X-Forwarded-Host $host;    # 3

    # Set HTTP Basic auth for connecting to the blog.
    proxy_set_header Authorization "Basic {CREDENTIALS}"; #4

    # Correct Location header (if present).
    proxy_hide_header Location;
    add_header Location "$_upstream_http_location_prefix$upstream_http_location";

    # Disable response compression so we can change it.
    proxy_set_header Accept-Encoding "";

    # Change the response's links to correct ones.
    sub_filter_once off;
    sub_filter_last_modified on;
    sub_filter_types text/html text/css text/xml text/javascript application/json;
    sub_filter 'blog.site.com' 'site.com/blog';
    sub_filter 'src="/wp-content/' 'src="/blog/wp-content/';
    sub_filter 'http:' 'https:';

    # Proxy
    proxy_pass https://blog.site.com;
}

This Nginx config is what something I worked on and is operational at time of writing this post. The blog being served by this Nginx reverse proxy averages 1.2+ million unique visitors per month, and is doing just fine. So I can safely say that this solution is well tested in the real world.

That's it, thanks for reading the article and make sure to drop a comment if you have any questions or feedback!

Dynamic PostgreSQL credentials using HashiCorp Vault (with PHP Symfony & Go examples)

Bouchaala Reda — Sun, 14 Aug 2022 13:18:06 +0000

I was playing around with HashiCorp Vault, trying to integrate dynamic secrets (one of many features Vault offers) with a web application.

Basically, have a web application connect to a database (PostgreSQL) using dynamically generated credentials (username & password), that you can rotate whenever you want and it'll all be transparent to your app.

Vault handles the credentials generation (and thus creating a corresponding username & password in PostgreSQL) and expiration (and thus, removing the username from the DB).

I'm by no means a Vault expert, this was actually my first hands-on with it. I thought I'd share this article with an example to:

Be better at writing technical articles
Learn to better articulate & communicate my thoughts
Showcase real example of using Vault with your application as there aren't many.

Anyway, let's get into it.

Vault Initial Setup

First we need to setup some stuff in Vault. Installing Vault is really easy so go ahead and do that first.

Once you get Vault installed on your system, you can run the dev server with

vault server -dev

In the output, you'll get the three important information:

The API endpoint (which is the same as the UI URL). If you ran the dev server without any arguments this is probably http://127.0.0.1:8200.
The unseal token. This is used to unseal and Vault from its sealed state. Whenever Vault is rebooted and/or initialized, it starts in a sealed state so you'll need to unseal it first. We don't have to worry about this because when using dev server, Vault is already initilized and unsealed.
The root token. The token we'll use to authenticate our requests to the API. This is only a good idea when running a dev server and trying out some stuff, but in the real world the root token is only there for emergencies and for initial setup of users/policies... etc.

Now that we have a Vault server running, leave that Terminal open and open a new one (or a new Tmux pane or whatever).

Let's configure our access to Vault.

# COnfigure our access
export VAULT_ADDR="http://127.0.0.1:8200"
export VAULT_TOKEN="THE ROOT TOKEN YOU GOT FROM RUNNING SERVER"

# verify the connection 
vault status

You should see that it says "Initialized: true" & "Sealed: false".

PostgreSQL Initial Setup

Let's do some initial setup on our PostgreSQL database. I'm using a VM (created using Vagrant) of Ubuntu 22.04 and I installed PostgreSQL 14 inside of it. Checkout out this article on DigitalOcean on how to install it.

Once it's installed let's do some setup:

# Inside the VM
sudo -i -u postgres

# Open PostgreSQL cli
psql

# and create a new "vault" user. 
CREATE ROLE "vault" WITH SUPERUSER LOGIN ENCRYPTED PASSWORD 'vault-password';

After that, let's make sure PostgreSQL accepts remote connections from outside the VM.

Open /etc/postgresql/14/main/postgresql.conf file and update listen_addresses to 0.0.0.0
Open /etc/postgresql/14/main/pg_hba.conf and add this line host all all 0.0.0.0/0 md5.

That basically tells the PostgreSQL server to listen to remote connections and not just locally (1), and the HBA config allows all users to connect from anywhere using their passwords (2).

Again, this is not a good idea for a production system as this exposes your PostgreSQL instance too much. We're only playing around here so it's okay.

Configuring Vault to use our PostgreSQL database

Now that we have both Vault & Postgres initialized, let's configure Vault to connect and manage Postgres credentials.

Connect Vault to Postgres

Vault can manage secrets using its Secrets Engines which range from AWS, GCP, Key Value, LDAP, SSH, databases... and so on. See the complete list on their docs.

Secrets Engines are Vault components that store, generate & encrypt secrets. The one that we are interested in is the database engine.

The database engine supports a wide varity of database flavors including but not limited to PostgreSQL, MySQL, Redshift and Elasticsearch.

Let's enable the engine and configure it to use our Postgrs database.

# Enable the database secrets engine
vault secrets enable database # 1

# Configure the engine to connect to our Postgres database using the user we created earlier.
vault write database/config/application_db \ # 2
    plugin_name=postgresql-database-plugin \ # 3
    allowed_roles="dbuser" \                 # 4
    connection_url="postgresql://{{username}}:{{password}}@192.168.56.101:5432/postgres" \ # 5
    username="vault" \ # 6
    password="vault-password"

Let's explain what we did here:

We enabled the secrets engine in Vault under the path database.
We are writing new config data to our database engine, we called this config application_db but you can call it whatever you want.
The plugin name to use. We're using PostgreSQL here.
The allowed roles that are to be associated with this config. Role here refers to Vault role not Postgres role. We'll create the Vault role after this.
Setup the connection URL to Postgres. 192.168.56.101 is my VM's IP address so change that to yours (or 127.0.0.1 if you're running it locally). We use the default postgres database to connect since our user vault does not have a databse nor that it needs one.
Supply the username & password for the vault user we created earlier.

Now, let's create a Vault role that will manage the credential creation in both Vault & Postgres.

When creating a role, we supply:

Creation statements: Vault will use this to know how to create the user in Postgres whenever we ask for new credentials.
Revocation statements: Vault will execute these commands against Postgres whenever the credentials have expired (TTL reached).
TTL (Time-To-Live) of the credentials. Once the credentials expire Vault will execute the Revokation statements and will remove the credentials from its storage.

Here's how to do that with Postgres:

vault write database/roles/dbuser 
    db_name="app" \
    max_ttl="10m" \
    creation_statements="CREATE USER \"{{name}}\" WITH SUPERUSER ENCRYPTED PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
    revocation_statements="REVOKE ALL PRIVILEGES ON ALL TABLES IN SCHEMA public FROM \"{{name}}\"; DROP OWNED BY \"{{name}}\"; DROP ROLE \"{{name}}\";"

For clarity here's what the creation & revocation statements look like:

-- Creation
CREATE USER "{{name}}" WITH SUPERUSER ENCRYPTED PASSWORD '{{password}}' VALID UNTIL '{{expiration}}';
GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO "{{name}}";

-- Revokation
REVOKE ALL PRIVILEGES ON ALL TABLES IN SCHEMA public FROM "{{name}}";
DROP OWNED BY "{{name}}";
DROP ROLE "{{name}}";

Basically, we created a Vault role that:

Expires in 10 minutes post creation.
Creates a Postgres user with superuser privileges, that has access to all tables in the public schema.
Properly drops the Postgres user upon expiration.

Of course, SUPERUSER privileges are not needed and even dangerous to give to a user that only needs SELECT, UPDATE, INSERT & DELETE privileges. In production, you'll need to tune the creation statements to a more secure one: Remove SUPERUSER & only give the least needed permissions to only neccessary tables.

I added the SUPERUSER privilege and set a short TTL just for the sake of example.

Now that we have a Vault role setup, let's actually try creating some credentials, by simply reading the from database/creds/dbuser, like so:

vault read database/creds/dbuser

That will output

The Lease ID, which you can use to re-new the lease on the credentials and not have them expire.
The Lease duration, in our case it's 10 minutes.
The generated username & password!

Let's test it out using psql

psql -h192.168.56.101 -u v-root-truly-Jixx1aASFjSmjjYF2Vin-1660473935 -d postgres -p

Supply the generated password and you should be in!

Example applications

Now, the strategy we're using here is our example applications need to connect to Vault and read/generate new credentials and use them before the current ones expire (i.e. before every 10 minutes).

Let's do two example application to showcase how that's done.
One using Go and one using PHP (w/ Symfony framework).

But first, let's create a table and populate it with some dummy data. With psql:

CREATE DATABASE app;
\c app
CREATE TABLE users (id serial, name VARCHAR);
INSERT INTO users (name) VALUES ('Jack Reacher');

Go example!

The Go application will expose an HTTP endpoint that will just simply
get all users from our Postgres database using Vault generated credentials.

Create a new directory and execute the following.

go mod init go-vault-example
go get -u "github.com/mittwald/vaultgo"
go get -u "github.com/gorilla/mux"
go get -u "github.com/lib/pq"

That will install our app dependencies which are

Mux for our HTTP endpoint
pq to enable Postgres support while using database/sql package to query our Postgres database.
Vault Go which is a library we'll use to read credentials from our Vault server.

Here's our complete Go app:

package main

import (
    "database/sql"
    "net/http"
    "strings"
    "fmt"

    vault "github.com/mittwald/vaultgo"
    "github.com/gorilla/mux"
    _ "github.com/lib/pq"
)

type VaultCreds struct {
    Data struct {
        User        string `json:"username"`
        Password    string  `json:"password"`
    } `json:"data"`
}

type DbConnection struct {
    Dbname string
    Host string
    Port int
    User string 
    Password string
}

// Read (generate) credentials from our Vault server.
// Don't forget to update your Vault address and token.
func getDBConnectionConfig() DbConnection {
    c, err := vault.NewClient("VAULT SERVER ADDRESS", vault.WithCaPath(""), vault.WithAuthToken("VAULT AUTH TOKEN"))
    if err != nil {
        panic(err)
    }

    key := []string{"v1", "database", "creds", "dbuser"}
    options := &vault.RequestOptions{}
    response := &VaultCreds{}

    err = c.Read(key, response, options)
    if err != nil {
        panic(err)
    }

    return DbConnection{
        Dbname: "app",
        Host: "192.168.56.101",
        Port: 5432,
        User: response.Data.User,
        Password: response.Data.Password,
    }
}

// This function opens up a new Postgres connection to our server and returns it.
func openConnection() *sql.DB {
    config := getDBConnectionConfig()
    connStr := fmt.Sprintf("host=%s port=%d user=%s password=%s dbname=%s sslmode=disable", config.Host, config.Port, config.User, config.Password, config.Dbname)

    db, err := sql.Open("postgres", connStr)
    if err != nil {
        panic(err)
    }

    err = db.Ping()
    if err != nil {
        panic(err)
    }

    fmt.Printf("Connected to PostgreSQL db using user <%s> and password <%s>\n", config.User, config.Password);
    return db
}

// Factory to create the function that handles the index request "/".
// It queries the database and return a join of all names in the users table. Pretty simple.
func newIndexHandler(db *sql.DB) func(http.ResponseWriter, *http.Request) {
    return func(w http.ResponseWriter, r *http.Request) {
        rows, err := db.Query(`SELECT name FROM users`)
        if err != nil {
            panic(err)
        }
        defer rows.Close()


        names := []string{}
        for rows.Next() {
            var name string
            err = rows.Scan(&name)
            if err != nil {
                panic(err)
            }

            names = append(names, name)
        }

        w.WriteHeader(http.StatusOK)
        fmt.Fprint(w, strings.Join(names, ", "))
    }
}

func main() {
    // Setup DB connection
    db := openConnection()
    defer db.Close();

    // Setup router 
    r := mux.NewRouter()
    r.HandleFunc("/", newIndexHandler(db))
    http.Handle("/", r)

    // Start listening
    fmt.Println("Listening on 127.0.0.1:8002")
    http.ListenAndServe("127.0.0.1:8002", r)
}

That's a pretty basic Go program that fetches data from our users table in Postgres.
The only interesting part is, it read the credentials from our Vault server and uses them to connect.
Our app is completely oblivious to how those credentials are generated and when. It just reads them and uses them.

go build
./go-vault-example

#
# Output is something like
# Connected to PostgreSQL db using user <v-root-truly-OQZIsxFo71ytoXb3aIKo-1660475644> and password <4u-A74pocYWqzUaSqW5L>                                                                     │
# Listening on 127.0.0.1:8002
#

Visit http://127.0.0.1:8002 in your browser and you'll see some data output.

Now, the only time the credentials are read (and thus generated) is when we start our program. If we wanted to re-generate a new pair of credentials, we'll need to stop it and run it again which isn't really a good idea.
So let's use system signals for that by adding this snippet right before starting the HTTP server.

// Setup reload signal using SIGHUP
// Sending SIGHUP signal to our process will make it close the current DB connection
// and open a new one with newly generated credentials.
signals := make(chan os.Signal, 1)
signal.Notify(signals, syscall.SIGHUP)
go func() {
    for {
        <-signals

        fmt.Println("Reloading: Terminating current connection and creating a new one.")
        db.Close()
        db = openConnection()
    }
}()

// Start Listening
// ... etc.

Don't forget to add os, syscall & os/signals imports.

That snippet runs a goroutine that waits for SIGHUP signal.
Upon receiving it, it closes the current database connection and creates a new one, effectively re-generating new credentials.

Build and run the app again, and let's try it out.

PID=$(pgrep go-vault-example) # Get the PID of our process
kill -HUP $PID # Send a SIGHUP signal

Our program will print something like:

Reloading: Terminating current connection and creating a new one.
Connected to PostgreSQL db using user <v-root-truly-63msxAJoWUb1QJ50hxfL-1660476797> and password <oaY-a1bl4IShVzcKEoNp>

Since we have our credentials expire after 10 minutes, we just need to send a SIGHUP signal to our program before that happens.
We can set up a cron job to do so that runs every 8 minutes or something like that.

To conclude:

We start our program, and it generates a new pair of credentials.
We reload our program every 8 minutes and by that we generate new credentials.
The old credentials will always expire and be removed from our database (that leaves 2 minutes of the old user not being used by anyone).

That's it, that's a basic example of how you can use Vault dynamic credentials in your Go app.

PHP Example (w/ Symfony Framework)

Let's create a new Symfony 6.1 minimal application, with Doctrine ORM, Vault PHP Client & a couple of its dependencies.

composer create-project symfony/skeleton:"6.1.*" php-vault-example
cd php-vault-example
composer require orm
composer require csharpru/vault-php
composer require alextartan/guzzle-psr18-adapter
composer require laminas/laminas-diactoros

Symfony by default uses YAML files for configuration but that doesn't really fit us for this use case. We need to use PHP config files because that gives us the ability to run some custom code to get the database credentials from Vault.
First, let's remove the connection config from config/packages/doctrine.yml

doctrine:
    # Remove the "dbal" line and its children.
    dbal:

Then create a new file config/packages/db_connection.php like so

<?php

use App\Init\VaultConfig;
use Symfony\Config\DoctrineConfig;

$config = new VaultConfig();
$credentials = $config->getDbConfig();

return static function (DoctrineConfig $doctrine) use ($credentials) {
    $dbal = $doctrine->dbal()
        ->connection('default')
        ->driver('pdo_pgsql')
        ->dbname('app')
        ->serverVersion('14')
        ->user($credentials['username'])
        ->password($credentials['password'])
        ->host('192.168.56.101')
        ->port('5432');
};

We still don't have the VaultConfig class yet so let's create that in src/Init/VaultConfig.php

<?php

namespace App\Init;

use GuzzleHttp\Psr7\Uri;
use Laminas\Diactoros\RequestFactory;
use Laminas\Diactoros\StreamFactory;
use Vault\Client;
use Vault\AuthenticationStrategies\TokenAuthenticationStrategy;

class VaultConfig {

    public const VAULT_ADDR = 'http://127.0.0.1:8200';

    public const VAULT_TOKEN = 'hvs.mjlW8faLW7GHQdbxAc3t8aEz';

    public const CREDENTIALS_PATH = '/database/creds/dbuser';

    private Client $client;

    public function __construct() {
        $client = new Client(
            new Uri(self::VAULT_ADDR),
            new \AlexTartan\GuzzlePsr18Adapter\Client(),
            new RequestFactory(),
            new StreamFactory()
        );

        try {
            $authenticated = $client->setAuthenticationStrategy(new TokenAuthenticationStrategy(self::VAULT_TOKEN))
                ->authenticate();
        } catch (\Exception $e) {
            die($e->getMessage());
        }

        if (!$authenticated) {
            die("Could not authenticate to Vault server at: " . self::VAULT_ADDR);
        }

        $this->client = $client;
    }


    public function getDbConfig(): array {
        try {
            $response = $this->client->read(self::CREDENTIALS_PATH);
        } catch (\Exception $e) {
            die($e->getMessage());
        }

        return $response->getData();
    }
}

Make sure to update the VAULT_ADDR & VAULT_TOKEN to your values.
Now let's create a simple controller that will read data from our users table.

Create a new file src/Controller/IndexController.php

<?php

namespace App\Controller;

use Doctrine\DBAL\Connection;
use Symfony\Bundle\FrameworkBundle\Controller\AbstractController;
use Symfony\Component\HttpFoundation\JsonResponse;
use Symfony\Component\Routing\Annotation\Route;

class IndexController extends AbstractController {
    #[Route('/', name: 'app_index')]
    public function index(Connection $connection): JsonResponse  {
        $users = $connection->executeQuery("SELECT * FROM users")
            ->fetchAllAssociative();

        return $this->json([
            'data' => $users,
            'connection' => [
                'user' => $connection->getParams()['user'],
                'password' => $connection->getParams()['password'],
            ]
        ]);
    }
}

Let's start a local PHP server and see our response

php -S localhost:8000 -t public

Visit http://localhost:8000 you'll see a JSON response containing our users, and also the currently used PostgreSQL username/password.

Now, Symfony actually caches the config data after executing it the first time, so our credentials are only generated once and then cached which is good thing because we definetly don't want our database credentials be generated on every request.
So if we wanted to generate a new pair, all we need to do is clear the symfony cache!

./bin/console cache:clear

If you visit your web page again, you'll see a new pair of credentials being used. Same as we did for our Go app, we just need to clear the cache before our credentials expire, and we'll be good.
To conclude:

We start our PHP application, and it reads the config for the first time and generate new credentials.
We clear Symfony cache every 8 minutes or so, so that the config is re-read and new credentials get generated & used.
The old credentials will always expire and be removed from our database (that leaves 2 minutes of the old user not being used by anyone).

Final notes & recap

I haven't tested this approach in a production environment and I'll sure that will bring up some nice challenges, such as what happens when your application is getting a lot of concurrent requests and you rotate your keys? You could get some downtime everytime you rotate keys so that's something to keep an eye for.
The TTLs we used are obviously not realistic and are only meant for testing. In real life you'd want a longer TTL depending on your use cases.
As I said in the beginning of the article, I'm by no means a Vault or Go expert (I mainly work with PHP). I only wanted to write this article to share examples of using Vault inside an application.
If you notice anything wrong or something could be done better, I'd love your feedback.

Let's recap what we did

Let's do a recap of what we did today:

We set up a local dev Vault server to test things out with.
We set up Vault to be able to talk to PostgreSQL and dynamically generate credentials using the database secrets engine.
We tested out the integration by generating new credentials and seeing them also created in PostgreSQL. Whenever the credentials expired we did notice that the user disappeared from our Postgres database.
We created a basic Go application that reads the dynamic credentials from Vault and uses them to query the database.
We then added a mechanism to our Go app to be able to rotate keys by using system signals (SIGHUP)
We then created a simple PHP application using Symfony framework, and we did same thing as in Go.

Thanks for reading!