I went on to setup my own public DNS node built on top of PiHole using Cloudflared and DNSCrypt and private VPN node support with Wireguard in cloud with log monitoring using Grafana/Loki and container monitoring/management using WeaveWork Scope and Portainer all automated with Ansible, Packer and Terraform π. This is similar to the setup I run at home of an Ad blocking VPN with DNS over HTTPS on a Raspberry Pi4 , so why do it again ? π
- This is automated so the stack can be created in under 5mins from scratch in the cloud π
- It's more complex when all your routing is behind a home router, this is a cloud instance can kill and provision again.
- It was OK when I was running the VPN when away from home locally connecting to my home network blocking all those pesky ads. Being abroad meant taking speed hit due to traffic routing out and coming back in.
- Now I am able to use my own public DNS node and block all ads while keeping traffic on local connection utilising all available speed πͺ when I don't need to keep data encrypted over a public network.
- Also it exposes only my IP to my DNS server sending cloud IP to the DNS services adding another layer of privacy, poor man's DNS privacy π¬
So cutting the tirade short by the end you should have an idea of how to have a setup of your own beyond just those tutorials that teach you how to install PiHole with a VPN that you can't even upgrade without destroying your setup. What you should have would be something like below which will be deployed using container images fetched from a container registry.
My personal node is sitting in Hetzner using a CX11 cloud instance type. It is based on 1 vCPU and 2GB RAM and comes with 20GB SSD and 20TB Bandwidth. All for 2.66 Euro(excl VAT), unbelievable ini't ?
I am going to assume some basic understanding of the tools used for this setup. So first of what we need to do is build a base image snapshot upon which we will build our application server image which I call JARVIS. To do so we will use Packer. I am not going to go in to Ansible part as it depends on what you want in your snapshot you can just point the playbook_file
to your playbook.
Packer
Base image
To build the base image make sure you generate the Hetzner API TOKEN and set it as an environment variable. This recipe will build an image on Debian 10 and save it as debian-base-snapshot
. Two things to note are that you should use the location
and server_type
where you will run the final server. The things ansible playbook installs in this step for me are copying ssh keys, installing docker and other dev tools required as per my needs.
To build run following command
packer build packer.json
{
"variables": {
"hcloud_token": "{{env `HCLOUD_TOKEN`}}"
},
"builders": [
{
"token": "{{ user `hcloud_token` }}",
"server_name": "base-packer",
"snapshot_name": "debian-base-snapshot",
"snapshot_labels": { "name": "debian-base-snapshot" },
"type": "hcloud",
"image": "debian-10",
"location": "nbg1",
"server_type": "cx11",
"ssh_username": "root"
}
],
"provisioners": [
{
"type": "shell",
"inline": [
"sleep 30",
"apt-get update",
"apt-get -y upgrade",
"apt-get update && apt-get install -y wget curl gcc make python python-dev python-setuptools python-pip libffi-dev libssl-dev libyaml-dev"
]
},
{
"type": "ansible",
"extra_arguments": ["--vault-password-file=~/.helsing_ansible_vault_pass"],
"playbook_file": "../../../../ansible/base.yml"
}
]
}
Jarvis image
Once you have the base snapshot we will use that in place of Hetzner provided snapshot to build our server snapshot used to run it. You can use the same token used in previous step for this as well. The recipe is very similar to the one used for base snapshot. It uses its own playbook installing what is needed in this image, One of the things I do is git clone the source repo in this stage.
packer build packer.json
{
"variables": {
"hcloud_token": "{{env `JARVIS_HCLOUD_TOKEN`}}"
},
"builders": [
{
"token": "{{ user `hcloud_token` }}",
"server_name": "jarvis-packer",
"snapshot_name": "debian-jarvis-snapshot-base",
"type": "hcloud",
"image_filter": {
"with_selector": [
"name==debian-base-snapshot"
],
"most_recent": true
},
"snapshot_labels": { "name": "debian-jarvis-snapshot-base" },
"location": "nbg1",
"server_type": "cx11",
"ssh_username": "root"
}
],
"provisioners": [
{
"type": "shell",
"inline": [
"sleep 30",
"apt-get update",
"apt-get -y upgrade",
"apt-get update && apt-get install -y wget curl gcc make python python-dev python-setuptools python-pip libffi-dev libssl-dev libyaml-dev"
]
},
{
"type": "ansible",
"extra_arguments": ["--vault-password-file=~/.helsing_ansible_vault_pass"],
"playbook_file": "../../../../ansible/jarvis.yml"
}
]
}
Terraform
Last I worked with Terraform was while setting up my stack and the latest version at that time was 0.12
so working on this I upgraded to the latest 0.14.8
and it requires that you setup version.tf
version.tf
terraform {
required_providers {
hcloud = {
source = "hetznercloud/hcloud"
version = "~> 1.25.2"
}
}
required_version = "~> 0.14"
}
Here is the recipe that creates the infrastructure it uses the snapshot created earlier. Hetzner has recently launched their Firewall
simplifying managing ports. This will create 1 server instance of cx11 and a firewall with the defined rules attached to it for traffic.
provider.tf
tarraform plan -out=jarvis.out
tarraform apply "jarvis.out"
variable "JARVIS_HCLOUD_TOKEN" {}
provider "hcloud" {
token = var.JARVIS_HCLOUD_TOKEN
}
data "hcloud_image" "jarvis_image" {
with_selector = "name=debian-jarvis-snapshot-base"
}
data "hcloud_ssh_keys" "all_keys" {
}
resource "hcloud_firewall" "jarvis_firewall" {
name = "jarvis-firewall"
rule { // SSH
direction = "in"
protocol = "tcp"
port = "22"
source_ips = [
"0.0.0.0/0",
"::/0"
]
}
rule { // HTTP
direction = "in"
protocol = "tcp"
port = "80"
source_ips = [
"0.0.0.0/0",
"::/0"
]
}
rule { // DNS
direction = "in"
protocol = "udp"
port = "53"
source_ips = [
"0.0.0.0/0",
"::/0"
]
}
rule { // DNS
direction = "in"
protocol = "tcp"
port = "53"
source_ips = [
"0.0.0.0/0",
"::/0"
]
}
rule { // HTTPS
direction = "in"
protocol = "tcp"
port = "443"
source_ips = [
"0.0.0.0/0",
"::/0"
]
}
rule { // WIREGUARD
direction = "in"
protocol = "udp"
port = "52828"
source_ips = [
"0.0.0.0/0",
"::/0"
]
}
rule { // GRAFANA
direction = "in"
protocol = "tcp"
port = "13443"
source_ips = [
"0.0.0.0/0",
"::/0"
]
}
rule { // SCOPE
direction = "in"
protocol = "tcp"
port = "15443"
source_ips = [
"0.0.0.0/0",
"::/0"
]
}
rule { // PORTAINER
direction = "in"
protocol = "tcp"
port = "19443"
source_ips = [
"0.0.0.0/0",
"::/0"
]
}
}
resource "hcloud_server" "jarvis_server" {
name = "jarvis-hetzner"
image = data.hcloud_image.jarvis_image.id
server_type = "cx11"
labels = { "name" = "jarvis-hetzner" }
location = "nbg1"
ssh_keys = data.hcloud_ssh_keys.all_keys.ssh_keys.*.name
firewall_ids = [hcloud_firewall.jarvis_firewall.id]
}
That sorts out our infrastructure, now comes the important part what are we going to run and how. It is going to run 12 services using Docker Compose.
- OpenResty
- HAProxy
- PiHole
- Grafana
- Loki
- Promtail
- Prometheus
- Scope
- Portainer
- Cloudflared
- DNSCrypt
- WireGuard
All Dockerfiles setup in folder as such for convenience to be used in docker-compose file
Docker Compose
First of we need to create docker-compose.yml file that we will use to build container images that can then be tagged and pushed to a registry later to be pulled for running.
This file is used to create images with all the configs and settings required by each image that you can run as first step before tagging and pushing to registry. Lets call this file docker-compose-ci.yml and we build images as following
docker-compose -f docker-compose-ci.yml build
version: "3.7"
services:
#OPENRESTY 10.0.3.2
openresty:
build:
context: ./openresty
dockerfile: ./Dockerfile
image: jarvis/openresty
#HAPROXY 10.0.3.6
haproxy:
build:
context: ./haproxy
dockerfile: ./Dockerfile
args:
BASIC_AUTH_USERNAME: ${BASIC_AUTH_USERNAME}
BASIC_AUTH_PASSWORD: ${BASIC_AUTH_PASSWORD}
BASIC_AUTH_REALM: ${BASIC_AUTH_REALM}
HAPROXY_HTTP_SCHEME: ${HAPROXY_HTTP_SCHEME}
HAPROXY_STATS_URI: ${HAPROXY_STATS_URI}
HAPROXY_STATS_REFRESH: ${HAPROXY_STATS_REFRESH}
image: jarvis/haproxy
#PIHOLE 10.0.3.3
pihole:
build:
context: ./pihole
dockerfile: ./Dockerfile
args:
TZ: ${TZ}
WEBPASSWORD: ${WEBPASSWORD}
DNS1: 10.0.3.4#5053 # cloudflared IP Address
DNS2: 10.0.3.5#5053 # DNSCrypt IP Address
image: jarvis/pihole
#CLOUDFLARED 10.0.3.4
cloudflared:
build:
context: ./cloudflared
dockerfile: ./Dockerfile
args:
TZ: ${TZ}
TUNNEL_DNS_UPSTREAM: ${TUNNEL_DNS_UPSTREAM}
image: jarvis/cloudflared
#DNSCRYPT 10.0.3.5
dnscrypt:
build:
context: ./dnscrypt
dockerfile: ./Dockerfile
image: jarvis/dnscrypt
#WIREGUARD 10.0.3.7
wireguard:
build:
context: ./wireguard
dockerfile: ./Dockerfile
args:
PUID: ${PUID}
PGID: ${PGID}
TZ: ${TZ}
PEERS: ${PEERS}
PEERDNS: 10.0.3.3#53 #pi-hole IP Address
image: jarvis/wireguard
#GRAFANA 10.0.3.8
grafana:
build:
context: ./grafana
dockerfile: ./Dockerfile
args:
GF_SECURITY_ADMIN_PASSWORD: ${GF_SECURITY_ADMIN_PASSWORD}
image: jarvis/grafana
#LOKI 10.0.3.9
loki:
build:
context: ./loki
dockerfile: ./Dockerfile
image: jarvis/loki
#PROMTAIL 10.0.3.10
promtail:
build:
context: ./promtail
dockerfile: ./Dockerfile
image: jarvis/promtail
#SCOPE 10.0.3.11
scope:
build:
context: ./scope
dockerfile: ./Dockerfile
args:
ENABLE_BASIC_AUTH: ${ENABLE_BASIC_AUTH}
BASIC_AUTH_USERNAME: ${BASIC_AUTH_USERNAME}
BASIC_AUTH_PASSWORD: ${BASIC_AUTH_PASSWORD}
image: jarvis/scope
#PROMETHEUS 10.0.3.12
prometheus:
build:
context: ./prometheus
dockerfile: ./Dockerfile
image: jarvis/prometheus
#PORTAINER 10.0.3.13
portainer:
build:
context: ./portainer
dockerfile: ./Dockerfile
image: jarvis/portainer
Once we have all the images built, tagged and pushed (we will get to the tagging and pushing part when we see deployment, I am using Gitlab repos so they come with private registry). We use docker-compose with right set of port, volume and command mapping to bring all the services up. Read through the file to make yourself comfortable
docker-compose up
version: "3.7"
services:
#OPENRESTY 10.0.3.2
openresty:
image: registry.gitlab.com/jarvis/openresty:amd64
container_name: jarvis_openresty
networks:
network:
ipv4_address: 10.0.3.2
aliases:
- jarvis_openresty
depends_on:
- haproxy
expose:
- "80"
- "443"
ports:
- "80:80"
- "443:443"
- "13443:13443"
- "15443:15443"
- "19443:19443"
volumes:
- ./openresty/ssl:/usr/local/openresty/nginx/conf/config/ssl
#HAPROXY 10.0.3.6
haproxy:
image: registry.gitlab.com/jarvis/haproxy:amd64
container_name: jarvis_haproxy
networks:
network:
ipv4_address: 10.0.3.6
aliases:
- jarvis_haproxy
depends_on:
- pihole
- grafana
- prometheus
- scope
expose:
- "80"
- "18081"
- "18443"
- "13000"
- "15000"
- "19000"
#PIHOLE 10.0.3.3
pihole:
image: registry.gitlab.com/jarvis/pihole:amd64
container_name: jarvis_pihole
volumes:
- "pihole_data:/etc/pihole"
- "pihole_dnsmasq_data:/etc/dnsmasq.d"
- "/dev/null:/var/log/pihole.log:ro"
depends_on:
- cloudflared
- dnscrypt
expose:
- "80/tcp"
- "67/udp"
ports:
- "53:53/tcp"
- "53:53/udp"
environment:
- DNSMASQ_LISTENING=all
- IPv6=false
- PIHOLELOG=/dev/null
networks:
network:
ipv4_address: 10.0.3.3
aliases:
- jarvis_pihole
dns:
- 127.0.0.1
- 1.1.1.1
cap_add:
- NET_ADMIN
#CLOUDFLARED 10.0.3.4
cloudflared:
image: registry.gitlab.com/jarvis/cloudflared:amd64
container_name: jarvis_cloudflared
expose:
- "49312/tcp"
- "5053/udp"
networks:
network:
ipv4_address: 10.0.3.4
aliases:
- jarvis_cloudflared
#DNSCRYPT 10.0.3.5
dnscrypt:
image: registry.gitlab.com/jarvis/dnscrypt:amd64
container_name: jarvis_dnscrypt
expose:
- "5053/tcp"
- "5053/udp"
volumes:
- "dnscrypt_data:/config"
networks:
network:
ipv4_address: 10.0.3.5
aliases:
- jarvis_dnscrypt
#WIREGUARD 10.0.3.7
wireguard:
image: registry.gitlab.com/jarvis/wireguard:amd64
container_name: jarvis_wireguard
volumes:
- "wireguard_data:/config"
- "/lib/modules:/lib/modules"
depends_on:
- pihole
ports:
- "52828:51820/udp"
cap_add:
- NET_ADMIN
- SYS_MODULE
networks:
network:
ipv4_address: 10.0.3.7
aliases:
- jarvis_wireguard
depends_on:
- pihole
#GRAFANA 10.0.3.8
grafana:
image: registry.gitlab.com/jarvis/grafana:amd64
container_name: jarvis_grafana
volumes:
- "grafana_data:/var/lib/grafana:rw"
- ./grafana/config/datasource.yml:/etc/grafana/provisioning/datasources/datasource.yml
expose:
- "3000"
networks:
network:
ipv4_address: 10.0.3.8
aliases:
- jarvis_grafana
#LOKI 10.0.3.9
loki:
image: registry.gitlab.com/jarvis/loki:amd64
container_name: jarvis_loki
expose:
- "3100"
command: -config.file=/etc/loki/local-config.yaml
networks:
network:
ipv4_address: 10.0.3.9
aliases:
- jarvis_loki
#PROMTAIL 10.0.3.10
promtail:
image: registry.gitlab.com/jarvis/promtail:amd64
container_name: jarvis_promtail
volumes:
- /var/log:/var/log
command: -config.file=/etc/promtail/config.yml
networks:
network:
ipv4_address: 10.0.3.10
aliases:
- jarvis_promtail
# SCOPE 10.0.3.11
scope:
image: registry.gitlab.com/jarvis/scope:amd64
container_name: jarvis_scope
networks:
network:
ipv4_address: 10.0.3.11
aliases:
- jarvis_scope
expose:
- "4040"
pid: "host"
privileged: true
labels:
- "works.weave.role=system"
volumes:
- "/var/run/docker.sock:/var/run/docker.sock:rw"
command:
- "--probe.docker=true"
- "--weave=false"
#PROMETHEUS 10.0.3.12
prometheus:
image: registry.gitlab.com/jarvis/prometheus:amd64
container_name: jarvis_prometheus
volumes:
- "prometheus_data:/var/lib/prometheus:rw"
expose:
- "9090"
networks:
network:
ipv4_address: 10.0.3.12
aliases:
- jarvis_prometheus
#PORTAINER 10.0.3.13
portainer:
image: registry.gitlab.com/jarvis/portainer:amd64
container_name: jarvis_portainer
restart: always
networks:
network:
ipv4_address: 10.0.3.13
aliases:
- jarvis_portainer
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- portainer_data:/data
expose:
- "9000"
networks:
network:
driver: bridge
ipam:
config:
- subnet: 10.0.3.0/24
volumes:
wireguard_data: {}
pihole_data: {}
pihole_dnsmasq_data: {}
cloudflared_data: {}
dnscrypt_data: {}
grafana_data: {}
prometheus_data: {}
portainer_data: {}
For automating the deployment phase so that you don't need to go in to your server to run all the docker commands I found a very nice OSS not very widely know called Stack Up
Stack Up
Stack Up is a simple deployment tool that performs given set of commands on multiple hosts in parallel. It reads Supfile, a YAML configuration file, which defines networks (groups of hosts), commands and targets.
Demo
Note: Demo is based on this example Supfile.
Installation
$ go get -u github.com/pressly/sup/cmd/sup
Usage
$ sup [OPTIONS] NETWORK COMMAND [...]
Options
Option
Description
-f Supfile
Custom path to Supfile
-e
, --env=[]
Set environment variables
--only REGEXP
Filter hosts matching regexp
--except REGEXP
Filter out hosts matching regexp
--debug
, -D
Enable debug/verbose mode
--disable-prefix
Disable hostname prefix
--help
, -h
Show help/usage
--version
, -v
Print version
Network
A group of hosts.
# Supfile
networks:
production:
hosts:
- api1.example.com
- api2.example.com
- api3.example.com
staging:
# fetch dynamic list of hosts
inventory: curl http://example.com/latest/meta-data/hostname
$ sup production COMMAND
will run COMMAND on api1
, api2
andβ¦
It makes things quite convenient, it is similar to ansible in a sense that you just have to define instructions in YAML and it executes it over SSH. Always remember no matter what new kid on the block software there might be BASH is always the King of the hood. In below file you can see different stages defined that can be grouped together in a target to create a pipeline. Its a quick and simple CI/CD pipeline that you can control from your command line.
version: 0.5
env:
ENV: <set var>
PWD: <set var>
CR_USER: <set var>
CR_PAT: <set var>
CONTAINER_REGISTRY: <set var>
PROJ_ID: <set var>
networks:
production:
hosts:
- root@server
commands:
connect:
desc: Check host connectivity
run: uname -a; date; hostname
once: true
build:
desc: Build Docker image
run: cd $PWD/$PROJ_ID && git pull && source ~/.bashrc && docker-compose -f docker-compose-ci.yml build
once: true
docker_login:
desc: Login to Gitlab container registry
run: docker login $CONTAINER_REGISTRY -u $CR_USER -p $CR_PAT
once: true
docker_tag:
desc: Tag images for CI registry
run: >-
docker images | grep "^${PROJ_ID}_" | awk '{print $1}' | xargs -I {} echo {} |
xargs -I {} docker image tag {} $CONTAINER_REGISTRY/{}:latest
once: true
docker_push:
desc: Push images to CI registry
run: >-
docker images | grep "^${CONTAINER_REGISTRY}" | awk '{print $1}' | xargs -I {}
echo {} | xargs -I {} docker image push {}:latest
once: true
restart:
desc: Restart docker containers
run: systemctl restart docker-compose@$PROJ_ID
once: true
docker_ps:
desc: List docker process
run: sleep 30; docker ps
once: true
test:
desc: test
run: uname -a; date; hostname
once: true
targets:
deploy:
- connect
- build
- docker_login
- docker_tag
- docker_push
- restart
- docker_ps
sup production deploy
Now a little bit over logging and monitoring that you can run from anywhere on the globe. Grafana is pretty amazing even though it might seems daunting to use for a personal project it sure makes your life comfortable. Loki along with promtail is super efficient for streaming your logs while using prometheus to capture metrics from HAProxy and OpenResty gives some nice insights beyond logs. Also if you want you can setup Dashboards over prometheus metrics or loki logs.
haproxy
nginx
loki
For container monitoring Scope and Portainer both are an overkill :P but I like Scope's UI and metric presentation while Portainer's management of Docker stack is unbeatable, this allows me to not go to the server for any reason and allows me to even debug my containers from the browser as they both allow you to exec in to your running containers from the browser.
scope
Some times we just underestimate how low resources we need to run so many services and how we can learn so many new things because of the amazing OSS community and their contributions.
I hope you find this useful and pushes you to increase your privacy, block those pesky ads and tracking or learn to play around and setup your own cloud infrastructure.
Top comments (0)