Hacking Caddy

I've been using nginx as my go to reverse proxy for quite some time and I've been quite happy with it. For the most part, it's easy to use and I have a pretty standard config that I copy and re-use for most of my projects. I usually set up SSL, compression and passing requests to an application. I found Caddy a couple days ago and the code examples looked great! Caddy sets a lot of the stuff that I was explicitly setting in nginx automatically and this made me want to give it a shot.

Switching from nginx to caddy

Here is my 48 line nginx configuration for one of my projects. To be completely honest the gzip stuff is something I found and copied. I never learned what it was really doing so I imagine there is stuff that I can cut out there.

But regardless, it works and is what I currently use. Having defaults I think is really important and ideally nginx would have a configuration that I could easily just use for compression would be great.

A simple configuration with a proxy pass, ssl certificate and compression enabled:

server {
    gzip  on;
    gzip_http_version 1.0;
    gzip_comp_level 2;
    gzip_min_length 1100;
    gzip_buffers     4 8k;
    gzip_proxied any;
    gzip_types
        text/css
        text/javascript
        text/xml
        text/plain
        text/x-component
        application/javascript
        application/json
        application/xml
        application/rss+xml
        font/truetype
        font/opentype
        application/vnd.ms-fontobject
        image/svg+xml;

    gzip_static on;
    gzip_proxied        expired no-cache no-store private auth;
    gzip_disable        "MSIE [1-6]\.";
    gzip_vary           on;

    listen 7088 ssl http2;
    server_name _;

    ssl_certificate /etc/ssl/certs/selfsigned.crt;
    ssl_certificate_key /etc/ssl/selfsigned.key;

    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_prefer_server_ciphers on;
    ssl_ciphers AES256+EECDH:AES256+EDH:!aNULL;

    location / {
        proxy_pass http://localhost:6002;
        proxy_redirect off;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }

    location /static {
        alias /home/nivethan/bp/commentless/public/;
    }
}

Now to caddy.

The below configuration is a 12 line version of the above! To be fair, I could also put the gzip stuff in nginx higher up so that the server block is shorter but I like that with caddy, I don't even need to think about that stuff.

192.168.1.70:7088 {
    encode gzip

    handle /static/* {
        root /static/* /home/nivethan/bp/commentless/public
        uri strip_prefix /static
        file_server
    }

    reverse_proxy localhost:6002
}

The handle part however bothers me as nginx makes it very simple to add aliases and when I searched around for similar functionality for caddy, I found that there wasn't anything besides sort of hacking an alias together. The other thing was why is gzip not a default? I assume if a browser can't handle gzip, caddy will return the uncompressed files so it should be safe to always have it on.

I think with some minor changes, caddy could actually be a 4 line configuration for most of my projects.

The goal configuration would be:

192.168.1.70:7088 {
    alias /static /home/nivethan/bp/commentless/public
    reverse_proxy localhost:6002
}

All I should have to tell my web server is the port to proxy to and the directory to serve static assets out of. I think the above configuration would be my idea, there is some parts I could probably remove, like I always use /static for my static files so that could be set automatically. I also have a static IP so that could also be removed. But I didn't want to add too much code as I might come up with a better way to do all this.

Hacking Caddy

In my eyes, adding an alias command shouldn't really require digging deep into the internals of caddy. Really before the Caddyfile is processed, we can rewrite the alias line into a set of lines that Caddy can already understand. This means that I just need to find where caddy reads in the config file and to replace a line starting with alias with multiple lines. This should be a pretty simple change.

First a caveat, I haven't programmed in Go, I've read a couple things and I've tried my hand at going through the go examples on the main website but I usually give up and go back to trusty node or python to quickly work something. This is to say that I am writing really bad code here, I'm just doing some string munging and the code I'm writing is quite fragile. I would be comfortable using this for my personal projects but I wouldn't roll this out to production. :)

Caddy has their installation and build instructions on their site, as long as you read it, it should be prety straightforward to build and run caddy. I, however, do not read and wasted some time doing go build in the wrong folder.

The steps to build caddy from source is:

> git clone https://github.com/caddyserver/caddy.git
> cd caddy/cmd/caddy
> go build

This will generate a caddy executable in the current directory. In the same directory, I wrote my ideal Caddyfile:

192.168.1.70:7088 {
    alias /static /home/nivethan/bp/commentless/public
    reverse_proxy localhost:6002
}

This should throw the following error:

2022/01/02 00:36:48.544 INFO    using adjacent Caddyfile
run: adapting config using caddyfile: Caddyfile:3: unrecognized directive: alias

Perfect! I also did make sure caddy was working with a proper configuration just to make sure which it did. So now I had a way to build caddy and run it. The next step was to start reading code and poke around. I needed to figure out where caddy was reading in the file so I started backwards and grepped around looking for keywords that I thought would be unique. Stuff like reverse_proxy and strip_prefix were pretty easy to find and after reading a few files, I found that it all flowed from the obivous place. The caddy/cmd/main.go file is where the config file get's read in and then passed around in caddy.

• The smart thing to search for would have been os.ReadFile or some variation on that but for whatever reason I knew I wanted to find everything where a file is read in but I didn't know the syntax for it.

Inside that file, after the config gets read in, we can intercept it and modify it and then pass it on. The change I'm trying to make is just a simple rewrite so I can get away with making my change just here. I think it could be cool to add the alias properly which I imagine involves updating the directives.go file which seems to be the file with all of the parameters and there seems to be a pretty good structure to how these directives are parsed. Maybe one day!

For now, I'm just going to hack it into place.

The below code will go in caddy/cmd/main.go, in the loadConfig function after the config has been read in fully and all error checking has been done. For me, this is around line 156.

At this point in the code, we have a config variable with the bytes of a valid config file. After the code I add, we should still have a config variable in bytes with a valid config file. Just a little longer.

    var configString = strings.Split(string(config), "\n")
    var lines []string

    for _, element := range configString {
        var line = strings.TrimSpace(element)

        if strings.HasPrefix(line, "alias") {
            var tokens = strings.Split(line, " ")
            var source = tokens[1]
            var target = tokens[2]
            line = fmt.Sprintf("encode gzip\n handle %[1]s/* {\n root %[1]s/* %[2]s\n uri strip_prefix %[1]s\n file_server\n }", source, target)
        }

        lines = append(lines, line)
    }

    var c = strings.Join(lines, "\n")
    config = []byte(c)

The first line of this snippet is to convert the bytes to a string and then split it it on the newline character. I then loop through each line and trim it. I then check to see if the line starts with alias. If it does, then I split that line on spaces to get a set of tokens. The first token is the keyword, the second token is the source and the last token is the destination. I want to map a URL that references source to a file in the destination.

The next line is the rewrite, I want to change the line to be the handle logic from before. I also adde the encode gzip part here as well as really I want that to be a default. I usually have just one alias in my configurations anyway so this as good a place as any to stick it in.

The next line simply appends the line to an array of lines.

Once the loop ends, I join the lines together and cast it into a byte before putting it back in the config variable.

Now we can rebuild caddy and run it:

> go build
> sudo ./caddy run

Voila! We shouldn't see any errors and if we navigate to our application, we can confirm that everything is getting gzipped and static files are getting served properly!

I was pretty happy with how this turned out as now my config is actually even smaller. I think this might even be the smallest it can be. It's really a 2 line configuration, one specifying the reverse proxy port and the other specifiying where to find static files.

Closing Thoughts

I used print debugging to find out how things were happening and in what order, I didn't find the actual print command so I used the caddy.Log().Info() mostly. This worked fine but sometimes my lines would get lost caddy's actualy outputs.

The caddy codebase is really readable and it was pretty easy to jump into. I mainly used grep to search around for things and using CoC with vim basically let me not even refer to any go documentation because I got it all from it. I don't know how much of this is from go being a simple language and having a default formatter or if caddy is just structured really well. Probably a mixture of both.

It was pretty cool to rely on the code completion to tell me about the TrimSpace function and the Join function. go is a bit weird in that everything seems to be a function that you pass things into, rather than a class that you call a method on. I'm using to doing something like, array.append or array.push vs array = append.

There is also something to be said about the fact that go has a reputation as a beginner friendly language and being a small language. I still don't feel comfortable building nginx from source and hacking away at C. Go on the other hand felt so much like javascript that there was no fear.

I'm surprised at how easy and quick it was to make this change. I wonder where the real place to make this change would be if I wanted this to be part of the caddy project. This is very much a bad patch job.

One issue with my code right now is that it is finnicky in regards to having a trailing slash in the alias paths. If you add one, I imagine it will cause issues with how the prefix is being stripped. I could add some logic to make sure I remove a trailing slash on paths before doing the rewrite but I'll leave that footgun in. There is also the fact that paths can have spaces and I'm splitting on them to get the tokens. The real way to get tokens would be to parse the line character by character and building tokens up but I don't have spaces in MY paths so I'm not worried about it.

I wish go had named parameters in the sprintf, I don't like that it's using indexes. I wish there was template literals in go like in javascript. Using an append function for an array is strange. for loop syntax is weird. I'm sure if I spent more time with go, it would all make sense but initial reaction is that it's strange.

Using the --watch option with caddy is pretty great. I can add a new project without having to restart my webserver which is a nice quality of life thing.

Overall this was a fun diversion from doing work.