How to quickly setup an infra [Part 1]

This is the first part of a serie where I would like to present a simple way to setup an infra for either

• an ambitious personal project
• a small SME/a startup in it's infancy.

Why this serie?

Good question! 😁 You could go with the heroku-like approach, or the hyperscaler one. I am sure there are tons of other tutorials out there to help you do so.

But the approach that I am going to present is ultra cheap (only 1 VM needed) and has some pedagogical merits 😊.

Plus in an era where companies are more and more moving away from the cloud, you never know, it might proves useful 😋.

 What are we aiming at?

By the end of this serie, you should have a stack looking like so

This stack includes:

• Traefik as a reverse proxy
• PostgreSQL as a sql db and pgAdmin as its admin interface
• Bookstack and its associated db for documenting your product
• keycloak and traefik forward-auth to protect apps that cannot implement authentication without keycloak
• minio back and front to stored/exchange data
• Ofelia as a job scheduler
• Private pypi to store your precious private python libraries between collegues.
• dozzle for log aggregation
• uptime-kuma for monitoring and alerting

I'll talk (python) web app setup in another subsequent serie 😊.

Where to start?

Well, thanks to the work we did at EcoAct, we got you covered:

• This documentation page and the ones related are the content I am going to present in this serie. This post corresponds to this page.
• The associated code can be found in this open source (MIT licence) repo.

What are we going to learn today?

Setting up a VM with one simple script! Yes I know, it surely could be done with ansible, but fur such a simple use case it seems overkill (plus one has to install ansible before being able to use it, by contrast with bash 😂).

TL DR: clone the repo, launch the setup script. The end! 😂 Now the details.

Actual setup

There are countless blog posts to help you decide which cloud provider is the best for you, and I won't add to this literature here 😊.

I therefore just assume you have a Virtual Machine (VM) Cloud setup with ssh access.

Here I provide a simple bash script to setup a freshly bought VM, assuming an Ubuntu Operating System (this OS choice is important for the firewall setup).

Launching the setup script

After having accessed with ssh to your VM, just git clone ecodev-infra. Then in the main folder, launch

make setup-vm <YOURUSER> 

Where <YOURUSER> is the username you used for connecting to the VM. The bash script executed will

• Install docker and related components
• Add <YOURUSER> to the sudo and docker group of the VM (might need to disconnect and reconnect at the end of the setup for this to take effect)
• Setup ufw (uncomplicated firewall) to work with docker (more on that below)
• block all but 22, 80 (tcp only) and 443 (tcp only) ports.
• Setup a decent history (english resource on the topic) memory size.

🚨 Technical topic incoming 🚨

Why going into all this trouble with ufw?

As explained here way better than I could, the standard ufw setup won't forbid access to docker exposed ports on your VM, even ports explicitely blocked with an ufw rule!!.

This has to do with technicalities related to iptables explained in the linked posts.

To make ufw work with docker, one has to use this (awesome!) solution. This is the reason of the sudo cp after.rules /etc/ufw/after.rules line in the setup.sh script.

You might be as curious as I was and wondering why there is no Open Source docker container responsible for dealing with firewall issues.

To the best of my knowledge, this has to do with the fact that firewell related stuff is really low level and needs to live close to the VM kernel. One example out of the numerous conversations on the topic.

The script in its whole splendor

If you're too curious 😋 to wait to go inspect the EcoAct infra repo (though I still highly encourage you to do so), here is the srcipt

#!/bin/bash

# This is a VM setup for ubuntu only OS
# Provide as argument your username (in order to add you to docker and sudo group)

echo "Installing docker"
sudo apt-get update
sudo apt-get install ca-certificates curl gnupg make
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg
echo   "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
       "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" |   sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

echo "Add access rights for current user" $1
sudo usermod -a -G docker $1
sudo usermod -a -G sudo  $1

echo "Setting up ufw to be docker compatible"
sudo ufw enable

echo "Edit ufw rules to be docker compatible"
# ref: https://github.com/chaifeng/ufw-docker?tab=readme-ov-file#solving-ufw-and-docker-issues
sudo cp after.rules /etc/ufw/after.rules

sudo ufw reload
echo "Setup ufw: block all but 22, 80 (tcp only) and 443 (tcp only) ports"
sudo ufw default deny
sudo ufw allow 22
sudo ufw route allow proto tcp from any to any port 80
sudo ufw route allow proto tcp from any to any port 443
echo "Turning ufw log"
# ufw is known to be glutton in terms of memory
sudo ufw logging off

echo "Restarting docker"

sudo systemctl restart docker

echo "Setup decent history size"

echo "export HISTSIZE=100000" >> .bashrc
echo "export HISTFILESIZE=100000" >> .bashrc
echo "export HISTCONTROL=ignoreboth:erasedups" >> .bashrc

echo "All done!"

I like what I read!

That's awesome! If you want to encourage me to continue the serie, do not hesitate to ⭐ ecodev-infra, best way to aknowledge the hard work we put at EcoAct to create this repo.

What's the (hopefully no so the distant) future?

We would like to do a full pythonic equivalent of Tiangolo's work. Of course Tiangolo repo is the place to start if you're new to modern python, but the work we hope to build will rely on dash for the frontend part, to have the full web stack within python! Plus the infra bricks are a little more numerous here, since we want to do more than just we app development (remember: build a full-fledged infra stack for a small structure)

The first brick is there and the associated documentation there. But one step at a time! Let's first setup the infra, then I'll talk python! 🥰