DEV Community

Gabriel Wu
Gabriel Wu

Posted on

Custom Tailscale DERP Server

#devops #linux #networking #tutorial

Deploying a Custom Tailscale DERP Server with DNS-01 SSL (When Ports 80/443 are Blocked)

Setting up a self-hosted Tailscale DERP (Designated Encrypted Relay for Packets) node is the best way to ensure low-latency, private connections. But if your VPS provider or local ISP blocks ports 80 and 443, the built-in Let’s Encrypt automation will fail, leaving you in a “no viable challenge type found” loop.

Here is the battle-tested, three-tier configuration to get your DERP node running on a custom port (e.g., 3443) with a valid SSL certificate.

Phase 1: Cloudflare Side (The DNS Validator)

Since we can’t use HTTP-01 challenges (because port 80 is blocked), we use DNS-01. This proves you own the domain by adding a temporary TXT record via API.

  1. Create an API Token:
    • Go to My Profile > API Tokens > Create Token.
    • Use the Edit zone DNS template.
    • Set Zone Resources to your specific domain (e.g., example.com).
  2. Copy the Token: You will only see this once.
  3. DNS Record: Ensure you have an A record pointing your subdomain (e.g., derp.example.com) to your VPS IP.

Phase 2: VPS Side (The Engine Room)

We will use Docker for the DERP server and a Python virtual environment for Certbot and its Cloudflare DNS plugin, avoiding conflicts with system packages.

1. Prepare the Virtual Environment

Install the necessary Python headers and create a dedicated space for Certbot:

# Install dependencies
sudo apt update
sudo apt install python3-venv python3-pip -y

# Create the venv in /opt/certbot
sudo python3 -m venv /opt/certbot/
sudo /opt/certbot/bin/pip install --upgrade pip
Enter fullscreen mode Exit fullscreen mode

2. Install Certbot & Cloudflare Plugin

Install the packages into the virtual environment and symlink the binary so you can run certbot from anywhere:

sudo /opt/certbot/bin/pip install certbot certbot-dns-cloudflare

# Create a symlink for ease of use (use -f to overwrite if exists)
sudo ln -sf /opt/certbot/bin/certbot /usr/bin/certbot
Enter fullscreen mode Exit fullscreen mode

3. Configure Credentials

Create a secure directory for your Cloudflare API token (in root’s home, since certbot runs as root):

sudo mkdir -p /root/.secrets/
sudo nano /root/.secrets/cloudflare.ini
Enter fullscreen mode Exit fullscreen mode

Inside cloudflare.ini, add:

dns_cloudflare_api_token = YOUR_CLOUDFLARE_TOKEN_HERE
Enter fullscreen mode Exit fullscreen mode

Security check: Run sudo chmod 400 /root/.secrets/cloudflare.ini to restrict access.

4. Request the Certificate (DNS-01 Challenge)

Trigger the challenge. Certbot will create a temporary TXT record via Cloudflare, verify it, and pull your certs—no port 80 required:

sudo certbot certonly \
  --dns-cloudflare \
  --dns-cloudflare-credentials /root/.secrets/cloudflare.ini \
  --dns-cloudflare-propagation-seconds 60 \
  -d derp.example.com \
  --agree-tos \
  --register-unsafely-without-email
Enter fullscreen mode Exit fullscreen mode

Optional: Add --email your@example.com if you want Let’s Encrypt expiry notices.

5. Copy Certs and Deploy the Container

The container reads certs from /app/certs. Certbot stores them under /etc/letsencrypt/live/. Copy (or symlink) them into /root/derper-data with the names the image expects, then start the container. One-time setup:

SOURCE="/etc/letsencrypt/live/derp.example.com"
DEST="/root/derper-data"
mkdir -p ${DEST}
cp ${SOURCE}/fullchain.pem ${DEST}/derp.example.com.crt
cp ${SOURCE}/privkey.pem ${DEST}/derp.example.com.key
chmod 644 ${DEST}/derp.example.com.crt
chmod 600 ${DEST}/derp.example.com.key
Enter fullscreen mode Exit fullscreen mode

6. Start the Container

Since we use manual mode, mount the certificate directory to the path fredliang/derper expects: /app/certs.

docker run -d \
  --name derper \
  --restart always \
  --net=host \
  -v /root/derper-data:/app/certs \
  -v /root/derper-data:/var/lib/derper \
  -e DERP_DOMAIN=derp.example.com \
  -e DERP_ADDR=:3443 \
  -e DERP_CERT_MODE=manual \
  -e DERP_HTTP_PORT=-1 \
  -e DERP_VERIFY_CLIENTS=false \
  fredliang/derper:latest
Enter fullscreen mode Exit fullscreen mode

Note: DERP_VERIFY_CLIENTS=false lets anyone use your DERP server. To restrict access to your Tailnet members, set to true and add the tailscaled socket mount:

  -v /var/run/tailscale:/var/run/tailscale:ro \
  -e DERP_VERIFY_CLIENTS=true \
Enter fullscreen mode Exit fullscreen mode

Important: This requires tailscaled running on the host AND the Docker image’s derper version must match the host’s tailscaled version. Version mismatches cause verification failures. For simplicity, most self-hosted setups use false since TLS still encrypts all traffic and DERP only relays already-encrypted WireGuard packets.

Phase 3: Tailscale Side (The Controller)

Now that the server is humming on port 3443, we need to tell the Tailscale network how to find it.

  1. Log into your Tailscale Admin Console.
  2. Edit your ACL JSON to include the derpMap.

Crucial: You must specify the DERPPort as 3443. With OmitDefaultRegions: true, your custom DERP is the only option—if it fails, clients have no relay fallback. With OmitDefaultRegions: false, Tailscale may prefer default DERP regions over your custom node (due to latency heuristics or reachability issues), potentially causing unexpected routing. Test with tailscale netcheck after setup to confirm your node is selected.

"derpMap": {
    "OmitDefaultRegions": true,
    "Regions": {
        "901": {
            "RegionID": 901,
            "RegionCode": "My-Custom-Node",
            "Nodes": [
                {
                    "Name": "1",
                    "RegionID": 901,
                    "HostName": "derp.example.com",
                    "DERPPort": 3443,
                    "IPv4": "YOUR_VPS_IP",
                    "InsecureForTests": false
                }
            ]
        }
    }
}
Enter fullscreen mode Exit fullscreen mode

Phase 4: Verification – The “200 OK” Moment

Before you celebrate, run this from your local machine (Mac/Linux/Windows). Replace derp.example.com with your hostname if different:

curl -Iv https://derp.example.com:3443
Enter fullscreen mode Exit fullscreen mode
  • Success: You see HTTP/1.1 200 OK and SSL certificate verify ok.
  • Failure: If it hangs on Client Hello, your VPS firewall (iptables) is blocking port 3443, or the Docker container cannot read the certificate files.

Finally, run tailscale netcheck on your local device. When you see your custom Region ID with a low latency number, you are officially off the grid!

Phase 5: Automation – Renewal and Deploy Hook

With the venv-based Certbot, use an explicit deploy hook for renewal. Create /root/update_derp_cert.sh, make it executable (chmod +x /root/update_derp_cert.sh), then use:

#!/bin/bash
# Path to your Let's Encrypt live directory
SOURCE="/etc/letsencrypt/live/derp.example.com"
DEST="/root/derper-data"

# Copy and rename for the container
cp ${SOURCE}/fullchain.pem ${DEST}/derp.example.com.crt
cp ${SOURCE}/privkey.pem ${DEST}/derp.example.com.key

# Ensure permissions are correct
chmod 644 ${DEST}/derp.example.com.crt
chmod 600 ${DEST}/derp.example.com.key

# Restart container to pick up new certs
docker restart derper
Enter fullscreen mode Exit fullscreen mode

Then add a crontab entry via crontab -e to check for renewal daily:

# Check for renewal daily at 3 AM
0 3 * * * /usr/bin/certbot renew --quiet --deploy-hook /root/update_derp_cert.sh
Enter fullscreen mode Exit fullscreen mode

Your DERP node is now a set-it-and-forget-it powerhouse.

Top comments (0)