Odysseas Lamtzidis

Posted on Apr 26, 2020

Home webserver setup on a Raspberry pi, using balena and Nginx

#nginx #balena #raspberrypi #webserver

The post was originally posted on my blog, which is actually hosted on this setup, at my home.

Introduction

In this post, we will be using a spare Raspberry pi 4 to host our very own website using the internet connection of our house.

We will start with some introductory terms to get a lay of the land and then we will continue with the tutorial itself.

If you are familiar with the relevant terms (IP, Domain Name, etc.), go ahead and jump to Let's get to it.

Introduction
Table of Contents
Let's get to it
Comments

Static Website

So, this idea came to me when I was considering alternatives for a new blog I wanted to start. Up to this point, I used Github pages which hosts for free any static website that belonged to an organization, a project or a person.

For those who are not familiar with web programming, as we read from Wikipedia:

A static web page (sometimes called a flat page or a stationary page) is a web page that is delivered to the user's web browser exactly as stored, in contrast to dynamic web pages which are generated by a web application.

So, the website is not supported by a back-end web application, but it's only a set of .html, .css, .js files that the server sends to the browser for the user to view the website. Wordpress sites, for example, are not static, since it's supported by a PHP server, a SQL database and various other components.

Webserver

In our case, to keep things as simple and as lightweight as possible, we will be serving a static website using a static webserver, Nginx. It is one of the oldest and most performant webservers, allowing us to serve up to 1000 users without our Raspberry breaking a sweat.

Nginx is a very robust web server, allowing the user to perform a myriad of different uses, from serving a static website to performing inverse proxy. Its configuration is very straightforward (as we will see below) and as we will find out soon enough and in essence we will simply dictate the server to serve a set of static files (our website), each time there is a connection at a specific port. (What is a computer port?)

Blogging software - Jekyll

As we are lazy, we don't want to write a blog website from scratch, as it would entail considerable overhead for each new post we want to make. What we want, is a framework that will have a certain theme and which will generate the static files of the blog for us, allowing us to focus solely on the content of the blog.

Luckily for us, there is a very easy-to-use framework, called Jekyll. It was created by Github's co-founder Tom Preston-Werner. As we read from the project's repository readme.md:

Jekyll is a simple, blog-aware, static site generator perfect for personal, project, or organization sites. Think of it like a file-based CMS, without all the complexity. Jekyll takes your content, renders Markdown and Liquid templates, and spits out a complete, static website ready to be served by Apache, Nginx or another web server. Jekyll is the engine behind GitHub Pages, which you can use to host sites right from your GitHub repositories.

The power of Jekyll is that it is super easy to use, so easy, that you don't even need programming knowledge (Verified from personal experience). In essence, you configure a Jekyll theme using a central configuration file and then you write the blog posts in markdown format (More on markdown here).

Domain

Now that we have the core pieces of the website, it is time to think about the domain name and the potential issue of dynamic IP.

If this doesn't sound familiar, let's spend a minute for a computer science 101 super-mini-course.

What's an IP

Each computer that is connected to a network is identified by a unique address, or IP, very much like your home address. The Internet is a global network of computers, thus each server has an IP.

As your home router is connected to the internet, it has it's own IP address, which you can find using a service like whatsmyipaddress. At the same time, the router creates a local network in which all your computers at home are connected, thus each computer has a local IP and all your computers, since they are connected to the Internet through the router, will have the same global IP, that of the router.

But what it has to do with domains?

Because it is hard for a person to remember an IP, there are services that have Huge registries, in which an IP is tied to a human-understandable word, or Domain. When you pay for a domain name, you pay to register your IP to these registries and tie that IP to the domain name that you have bought.

Now, each time someone enters that domain name, the computer automatically connects to several Domain Name Registries searching for an IP that is tied to that specific domain name. When it is found, the browser connects to the server using the IP and loads the content.

The problem arises when our IP is not static but it changes continuously, in other words, it's dynamic.

Dynamic IPs aka "The Plot Thickens"

Many ISPs (Internet Service Providers) around the world offer a dynamic IP, meaning that the IP doesn't stay the same but changes now and then, according to the policies of each ISP. This creates a challenge, as the domain name will have to point to a new IP each time our IP changes.

Luckily for us, most domain name providers offer a service called Dynamic DNS. This service allows the customer to use their API to update the IP to which the domain name must point to. We will be using a small program called ddclient which supports most of the known domain name providers.

Wait a minute, I don't have a domain name

If you don't have a domain, go ahead and grab one from one of the major Domain Name providers (just google buy domain name). Make sure that the Domain Name provider that you choose supports Dynamic DNS and ddclient. This guide was tested using Namecheap.

balena.io

At this point, it is apparent that we will be needing to install and configure a bunch of software not only to bootstrap the website but to also keep it up to date. To do that, we will be using balena.io to develop and deploy our software to the Raspberry pi with the ease and speed of using a cloud-service provider.

That's right, using balena to provision and manage our embedded IoT device, we will be having the same tools and workflows that one would expect from AWS.

So what's the deal, exactly?

The team behind balena.io were the first people to port docker to the Raspberry family, showcasing how the container visualization paradigm could serve the domain of the Internet of Things.

Balena now offers a full feature-set that enables us to manage-literally thousands- devices, such as a Raspberry pi, as easily as ever

We will develop our application as a multi-container application, meaning that the distinct services from which the project is constructed will run as distinct containers, completely isolated one from another.

You can think the ~~docker engine~~ balena-engine (our optimized for the IoT version of docker) as an oven where you can bake both a fish and a cake at the same time, while each will taste and smell just fine when you take them out.

In the same sense, each service can run independently, without having to worry with incompatible libraries or different versions. They will ~~taste~~ work just fine.

Finally, balena allows us to

easily access the device's logs
ssh into the host OS or one of the containers
push a new release by simply running a command.

This last part is pure black magic.

You simply define your application in a docker-compose.yaml, you define a couple of Dockerfiles and then you just push your project. balena takes care of building the project specifically for your device on its build-servers and then it simply sends the built project to the device. The smart supervisor is responsible for downloading and setting up your application according to the docker-compose file.

Developing and managing IoT devices have never been so easy and beautiful. Here is a sneak peek of the dashboard for our device:

Disclaimer: I work at balena.io in the product team. Thus, you could say that I am a bit biased.

Complimentary Software:

Certbot

Certbot is a service offered by letsencrypt, a nonprofit Certificate Authority providing TLS certificates for anyone who may ask. This way, users will be able to connect securely on our website, using https.

We will be using the certbot-CLI program to request a certificate for our website. In essence, certbot will place a special file for our webserver to serve. When the authority tests the website, it will find the specific file and verify that the website (and thus the domain) is indeed ours. Giving us a certificate for 90 days.

Netdata

Netdata is a monitoring agent that runs on the device, aggregates, visualizes and presents various data about the operation of the machine. From fairly simple, such as RAM usage, to more complicated such as CPU interrupts. Moreover, it has collectors for specific apps that can auto-detect if it's running and start gathering data.

Although Netdata is fairly complex and customizable, we will be using it because:

It's super light (about 5% CPU consumption) and thus ideal for the constraint nature of a Raspberry pi 4.
It can auto-detect nginx and start gathering data using the stub_page of the webserver. You can read more about stub_page here.

Architecture

We will be using the multi-container functionality of the platform, thus the project will consist of several different containers, each one running a specific component. This architecture enables us to isolate one component from another, facilitating the management and configuration of the application.

Components

webserver: Runs the nginx service and the certbot for SSL generation
ddclient: Runs the ddiclient service
netdata monitoring: Runs an instance of Netdata Monitoring software to overview the load of the server.

Let's get to it

Provisioning the Device

The first order of business would be to provision our device, a Raspberry pi 4.

To do that, we need a new account at balena.io and we need to head over to the Get Started Guide of the balena platform and finish it. It will prompt you to install a demo-app in your device, but that's ok. We need you to get familiar with the platform before we continue.

Disclaimer: After you finish with the guide, don't turn off the device, we will need it for later. Just leave it be. Ok? Cool.

Go ahead, I'll wait.

One Note: The first 10 devices on balena.io are for Free, so using your new account will be more than enough for this project.

Before going forward, we assume that:

You have balena-CLI installed
You have balena-etcher installed
You have logged in balena-CLI
You have logged in balena dashboard

Installing the Software

Jekyll

Although this blog post focuses mainly on setting up a balena-powered raspberry pi webserver, we want to give some insight into how to create a website in the first place.

Visit Jekyll's website and follow the Get started guide.
Get yourself familiar with the Jekyll templating engine
Search for a Jekyll theme that is appealing to you:
1. http://jekyllthemes.org/
2. https://jekyllthemes.io/
Download the theme locally and configure it according to the theme's and Jekyll's documentation.
Build the website according to the theme's documentation, the source files will be placed in a directory called _site.
Upload the files inside _site to a Github Repository.

Disclaimer: If you haven't used Github again:

1) Follow ths guide to create a new repository to Github.

2) Follow this guide to upload all your website's source files to the repository you just created. Simply drag and drop all of them as it is shown in the guide.

3) Congratulations! You have your very first project on Github!

Installing nginx-in-balena

To install the software, we did all the heavy lifting for you. We aggregated all the relevant software and made sure that it can support the Raspberry pi 4 with a balenaOS 64bit. You only have to go download a local copy of the project:


git clone https://repo

cd repo

We are now into the project's directory. Let's configure it!

Possibly, the same software will run without problems on a Raspberry pi 3 with either 32 or 64 bit OS. If you test it successfully on a Raspberry pi 3, please do leave a comment and we will update the blog-post accordingly.

Configuring the Software

Environment Variables

Go here and download the project's repository locally. If you are not familiar with Github, please refer to their documentation.
cd into the repository using a terminal program (cmd in windows).
Open the file .balena/balena.yml using your favorite code editor. If you don't have one, go ahead and install Visual Studio Code
In this file, we need to change a couple of environment variables which we make available to the services via their respective Dockerfiles. Go ahead and change the variables:
1. In order to find the REPO_ZIP_URL, go to your Github repository, click on clone or download and then right-click on Download ZIP and click on Copy Link Address.
2. Fill in the domain of your website according to this certbot example. Please note that www.domain.com and domain.com are two different domains, it is best to include both.

REPO_ZIP_URL=https://github.com/OdysLam/odyslam.github.io/archive/master.zip

REPO_NAME=odyslam.github.io

CERTBOT_MAIL=hi@odyslam.me

CERTBOT_DOMAIN_1=www.example.com

CERTBOT_DOMAIN_2=example.com

This environment variables are not expected to change while the server runs, thus we prefer to define them at build time.

On the other hand, there are 2 environment variables that can be set using balena dashboard and when the nginx container reloads, it will pick them up.

SYNC_WEBSITE: If this environment variable is set to "1", the container will always download the latest version of the website every time it restarts.
CERTBOT_FORCED: If this environment variable is set to "1", the container will always request a new certification every time it restarts. If the current certification is still valid, it will simply inform the user that the certification is up-to-date and will exit.

You can read more about environment variables in balena, in the documentation.
You can read more about build-time secrets in balena, in the documentation.

nginx configuration

Run the commands bellow to generate a private key that will be used by nginx for SSL related functionality. As it might take some minutes, go ahead and read about it in this Stack Overflow Question. Welcome to the world of cryptography.


cd nginx

openssl dhparam -out dhparam.pem 2048

cd ..

Using a text editor, open nginx.conf which will find the nginx directory. Head over to the following excerpt and replace the www.example.com and example.com with your domain name.


server {
listen 443 ssl http2;
server_name www.example.com example.com

server {

listen 80;
listen [::]:80;
server_name www.example.com example.com;

If you want to read more about the nginx configuration file and what the various fields mean, you can read more about it here:

Digital ocean article
Nginx Beginner's Guide

ddiclient configuration

Create a configuration file for ddclient using a text editor:
1. You can find examples in the ddclient's documentation
2. If you used Namecheap, you can find an example configuration in the namecheap documentation
Place the configuration file you just created into the ddclient folder

Configuring the environment

It's time to move on configuring our environment. In our case we need to allow ingoing connections from the router and make sure that the server will always have a static ip in the local network.

Static IP

Balena allows us to set our device to static IP in a breeze.

Find out the IP form that the router uses to assign IPs.
1. Go to your balenaCloud dashboard, you can find the IP of your device at the summary page.
Use a text editor to open the file static-ip that you will find in the directory tools of the repository you downloaded.
Replace the field ROUTER_IP with the IP of the router.
Replace the field DEVICE_IP with the IP of your balena device with a small change. Change the digits after the last . to 100.
Close the text editor.

Here is an example, note that here the IP has the format 192.168.1.X


[connection]

id=my-ethernet

type=ethernet

interface-name=eth0

permissions=

secondaries=



[ethernet]

mac-address-blacklist=



[ipv4]

#This is the important line
address1=192.168.1.100/24,192.168.1.1 

dns=8.8.8.8;8.8.4.4;

dns-search=

method=manual



[ipv6]

addr-gen-mode=stable-privacy

dns-search=

method=auto

Port Forwarding

Since we are building a server, we need to allow people to be able to connect to our Raspberry Pi. Normally a home router will block all ingoing connections (connections that are not initiated from a device inside the local network), thus we need to create a rule and tell the router that each connection that is made to a specific port should be forwarded to the same port of the Raspberry.

This is because when someone attempts to connect to our IP, in essence, he tries to connect to our router since it functions as the gateway that stands between the Internet and our local network. Thus we want to tell the router that any time someone tries to continue to the ports that we will specify, in essence, he wants to connect to our server, thus the router must forward the connection to the Raspberry pi.

In other words, we need to forward ports 80 and 443 to the Raspberry pi.

Visit portforward
Find your router
Follow instructions and forward the ports to the device's IP

We are almost there, let's deploy the device.

Deploy

Now that we have everything configured, let's start deploying the various components.

Deploy the project to the device

Open balenaCloud and create a new application for Raspberry pi 4. You can name it whatever you want. We will assume that you name it bananas.
Download an development image for that application. Don't bother filling in your wifi credentials, we will be using ethernet as wifi is not very reliable for a webserver.
Burn the image using the best image format app in the world, balena-etcher.
Pull out the sd card and re-insert it in the computer. Copy the file named static-ip we created earlier into the system-connections directory of the sd card.
Insert the card into the Raspberry pi and connect it to power.
Go to the balena-nginx directory, where your docker-compose.yaml is located, and push the project


balena push bananas

Generating the SSL certificate

To generate the SSL certificate, we don't have to do anything.

The server will detect the absence of the certificates and will run the certbot service to register your website. Afterward, it will simply start the server and will serve your website.

The certificates will be saved into a persistent directory with a functionality called named volumes. This enables the device to persist your certificates (or any data in that directory)

Updating your certificates

You are very organized and want to plan ahead? Sure, cerbot will email you a couple of days before your certificates are invalidated so you can renew.

When you receive that e-mail, you only need to:

Log in to balena dashboard
Go to device summary
Open ssh session to the nginx container
Type certbot renew
Done

Push new content to the website

All we have to do in order to push new content to the website is to update the source files in the directory from which we have configured nginx to serve our website. In our case /usr/share/nginx/html/.ping

To do that, we have a script inside the nginx container, which downloads the latest version of our website from the GitHub repository we have defined.

To run the script:

SSH into nginx container, preferably using balena dashboard
/update-blog.sh

But, what happens if we want to add new content to our website?

We have to upload the new source files into our GitHub Repository and then we have to run the update-blog.sh script in order to download the new version in the server.

So, in order to push new content to the website:

Change the website's source files
Upload the new files to the Repository and add a commit message to describe the changes
From balena dashboard, ssh into the nginx container and run: /update-blog.sh

Push new content to the website - For advanced users

The whole process has be automated, and you can simply run the script deploy-dev-loca.sh from the local directory of the project, like this:

./deploy-dev-local.sh "<commit message>"

The field <commit message> must be replaced with a commit message for the addition to the GitHub repository, just like you did when you uploaded the files using the website of GitHub.

Before you can you can use the script, you have to open the file using your favorit text editor and replace the following fields:

# If you use 
export J_OUTPUT= Absolute path to the directory of the website's source files

export REPO = Absolute path to the directory of the website's source files

export DEV_UUID= UUID of the device, can be found from the device's dashboard

Comments

Top comments (2)

Odysseas Lamtzidis • May 29 '20

Hey, I just saw this message. You just made my day with your comment. Thank you so much!

Blacknight318 • Mar 13 '21

Running into an issue with missing dhparam.pem file, I already have an nginx reverse proxy going, any help modifying to bypass certbot for setup would be appreciated.