DEV Community: bowlerdesign.tech

Archiving ProtonMail Emails on a headless Ubuntu instance

bowlerdesign.tech — Mon, 16 Nov 2020 11:17:00 +0000

Introduction

I wanted to be able to store all of my ProtonMail emails locally for archival purposes, with the intention of running a local content search whenever I needed something retrieving.

I'm going to talk through my steps. There's plenty of tutorials out there for email services such as; Gmail, Hotmail etc. But not much around ProtonMail, especially using a headless instance.

ProtonMail bridge

ProtonMail Bridge is an open-source piece of software built by the Proton Team to create a dummy IMAP server locally, with a sole purpose of decrypting your emails locally to enable you to use a desktop email client such as ThunderBird.

ProtonMail Bridge is a paid feature though, with a minimum requirement of the ProtonMail Plus plan ($5/month). ProtonMail Bridge is not available on the ProtonMail Free plan.

Overview

We're going to set up a Cron job on a headless Ubuntu instance to periodically pull down all our emails. This instance will be running ProtonMail Bridge and a tool named Offlineimap, which will be used to store all your emails in a plaintext, searchable format.

Getting Started

Step 1: Setting up your Linux server

You'll need to either have an existing server instance or create one. I use a Proxmox instance running on a server in my loft. You could also use something like Digital Ocean to run your local archive. Using the following link will give you $100 worth of credits for 60 days to play around with, just sign up using this link.

Once you have the server set up, or have logged in. You'll need to do some updates and run some prerequisite installs.

sudo apt-get update
sudo apt-get upgrade

Step 1: gpg and pass

To use ProtonMail Bridge, you need to first set up the dependencies. These dependencies are pass (A password management system), which has a dependency of gpg (A key management system).

Installing GPG

GPG should be installed on your system as part of the update and upgrade commands that you initially ran after provisioning your server. If it's not, you can install GPG by running the following command.

sudo apt-get install gpg

Once GPG is installed, you need to generate a GPG key.

gpg --gen-key

This will run you through a wizard of adding your full name and email address. Also requiring you to create a passphrase for your gpg key. Add a password to satisfy the wizard, but we'll have to remove this later.

Unfortunately, ProtonMail Bridge will not work with a password protected GPG key in headless mode. This is a serious security risk, so make up your own opinion on this, do some research on what implications this has.

Once GPG has finished, you'll have a public key output. It'll look something like the following

7BDD29402175BC627671356BE8AC4A1C3C5J6357

We now need to remove the passphrase associated with this key.

Run the following

gpg --edit-key 7BDD29402175BC627671356BE8AC4A1C3C5J6357

Once you have executed that command, you'll be in the context of gpg

Run then next command

passwd

You'll now need to enter the passphrase you created earlier, which will unlock the GPG key.

There will now be another console prompt to enter a new passphrase.

Leave this blank.

You'll be asked to confirm that you want a blank passphrase, it'll warn you about the security implications.

Hit 'Y' or Yes to confirm this implication.

Finally, quit the GPG context by typing q and hitting Enter.

Installing Pass

Now it's time to install Pass, which is a ProtonMail Bridge dependency.

sudo apt-get install pass

We'll then need to initialise a new pass instance with your previously generated gpg key id.

pass init 7BDD29402175BC627671356BE8AC4A1C3C5J6357

That's, all the setup required for initialising pass.

Step 2: Installing ProtonMail Bridge

Now it's time to install ProtonMail Bridge.

Run the following command to install the latest version of ProtonMail Bridge at time of writing this post.

wget https://protonmail.com/download/beta/protonmail-bridge_1.5.0-1_amd64.deb

Then install

dpkg -i protonmail-bridge_1.5.0-1_amd64.deb

There may be some more missing dependencies when installing. To fix this, run the following

sudo apt --fix-broken install

Step 3: Initialising ProtonMail Bridge

We now need to log in to ProtonMail Bridge, to do this, run the following command

protonmail-bridge --cli

ProtonMail Bridge will start up and ask you to log in with an email and a password.

This will be your standard ProtonMail username and password.

Step 4: Getting your ProtonMail IMAP Username and Password

Whilst in the context of the ProtonMail Bridge session, run the following command.

info

You'll see an output like the following.

            Welcome to ProtonMail Bridge interactive shell
                              ___....___
    ^^ __..-:'':__ :..: __:'':-..__
                  _.-: __:.-:'': : : :'':-.:__ :-._
                .':.-: : : : : : : : : :._:'.
             _ :.': : : : : : : : : : : :'.: _
            []: : : : : : : : : : : : : :[]
            []: : : : : : : : : : : : : :[]
   :::::::::[]: __:__ : __:__ : __:__ : __:__ : __:__ : __:__ :__:[]:::::::::::
   !!!!!!!!![]!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!![]!!!!!!!!!!!
   ^^^^^^^^^[]^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^[]^^^^^^^^^^^
            [] []
            [] []
      jgs [] []
    ~~^_~^~/ \~^-~^~ _~^-~_^~-^~_^~~ -^~_~^~-~_~-^~_^/ \~^ ~~_ ^
>>> info
Configuration for {your email}
IMAP Settings
Address: 127.0.0.1
IMAP port: 1143
Username: {your email}
Password: {your password}
Security: STARTTLS

SMTP Settings
Address: 127.0.0.1
IMAP port: 1025
Username: {your email}
Password: {your password}
Security: STARTTLS

Take a note of the values {your email} and {your password}. You'll need these later.

Step 5: Setting up Offlineimap

I'm going to be using an open-source tool I found on Github. Written by peterrus.

Run the following command to pull down the repo from Github.

git clone https://github.com/peterrus/protonmail-export-linux.git

The cd into the cloned repo

cd protonmail-export-linux

Next, you need to install offlineimap

sudo apt-get install offlineimap

Once installed, in the protonmail-export-linux directory, there will be a offlineimaprc file. Open this with your favorite text editor. I'll be using vim in this example.

vim offlineimparc

You'll see a file open that looks something like the following

# Feel free to modify this file to your needs
# See https://github.com/OfflineIMAP/offlineimap/blob/master/offlineimap.conf for a full reference

[general]
accounts = Protonmail
metadata = ./offlineimap-metadata

[Account Protonmail]
localrepository = ProtonmailLocal
remoterepository = ProtonmailRemote

# Feel free to change this, or disable it by commenting it out
postsynchook = notify-send 'Protonmail export done'

[Repository ProtonmailLocal]
type = Maildir
localfolders = ./protonmail-export
# If you (accidentally) delete mail locally, it won't get synced to Protonmail
sync_deletes = no

[Repository ProtonmailRemote]
type = IMAP
# Change this to the value provided in the Protonmail bridge
remoteuser = {your email}
remotepass = {your password}
remotehost = localhost
remoteport = 1143
ssl = no
starttls = no
# If you delete mail on Protonmail, this deletion also get's synced to the archive
expunge = yes
# Don't try to sync local changes to Protonmail, we just want a backup
readonly = True

Leave everything how it is, apart from the section where I have specified, {your email} and {your password}. These will need to be replaced with your ProtonMail Bridge email and password that I got you to take note of earlier.

Save your changes and exit your text editor.

Step 6: Setting up screen

I use screen to run services in the background. Both protonmail-bridge and offlineimap need to run simultaneously.

sudo apt-get install screen

Step 7: Creating a startup script

I made it easy for myself and created a startup script to execute both protonmail-bridge and offlineimap

Here's my script, simply create a new file named start-sync.sh in your home directory. Giving it execute permissions by running

sudo chmod +x start-sync.sh

The start-sync.sh file contents should look like this.

#! /bin/bash
echo "killing existing screen sessions"
pkill screen
echo "Starting bridge"
screen -d -m protonmail-bridge --cli
echo "Starting sync"
sleep 10
screen -d -m offlineimap -c /home/{your user}/protonmail-export-linux/offlineimaprc

Change the location of your offlineimaprc file in that shell script.

Step 8: Starting Offlineimap

Now it's time to execute your start-sync.sh

./start-sync.sh

Now, both of your services should have started up. To see them, run the following command

screen -r

You'll see an output like the following

ed@mail:~# screen -r
There are several suitable screens on:
        6415..mail (12/05/20 18:06:26) (Detached)
        6308..mail (12/05/20 18:06:16) (Detached)
Type "screen [-d] -r [pid.]tty.host" to resume one of them.
ed@mail:~#

Then run

screen -r 6415 // 6415 being the id of the top process on the screen -r output

You should see an output of a successful authentication with ProtonMail Bridge, and a list of emails currently being downloaded.

Copy message UID 2081 (23/3445) ProtonmailRemote:All Mail -> ProtonmailLocal
Folder INBOX [acc: Protonmail]:
etc...

Detach from this screen by hitting Ctrl + a and Ctrl + d.

Step 9: Where are my emails?

Whilst your emails are downloading, they will be saved to ~/protonmail-export. You should see a list of your email folders in this directory, under the cur subdirectory.

Step 10: Setting up a cron job to pull new emails

Now it's time to set this job up to run each day, or each hour, whatever you'd like to pick.

Run the following

sudo crontab -e

At the very bottom of your crontab file, paste in the following command.

0 0 * * * /home/{your user}/start-sync.sh

I use a tool called crontab.guru to assist with picking a schedule to run my command.

My command will run at midnight each day.

That's all

And hopefully, everything should be up and running successfully. Let me know if you receive any issues in to comments, I'll be happy to help you out.

You could now sync all your emails to Dropbox, or something like Syncthing, which I have recently written about here.

Schiit Modi 2 - Fixing Windows not recognizing USB device

bowlerdesign.tech — Sun, 27 Sep 2020 16:31:02 +0000

Sorry for the off-topic post, but this has been bugging me for months

Having recently purchased a Schiit Modi 2 from eBay, just to try out before commiting to something a little more expensive (and reliable).

The Modi 2 has some real issues with Windows 10, I belive the Modi 3 solves all of these issues.

There's an error message when plugging the Modi into Windows which is something like "USB device not recognized, the last USB device you connected to this computer malfunctioned, and Windows does not recognize it."

The solution

A hairdryer... yes, I'm being serious. Apply heat directly to the Schiit Modi 2 for 10 - 15 seconds. Then try connecting the Modi to your PC. Ridiculous, I know, but it worked.

Now, I can't take any credit for this solution, the real MVP is vavvo966 who posted the following

There's another method which includes replacing the chip, but this involves soldering 100 tiny legs, which I'm not a fan of, but you can find some information here

Self-hosting a Wireguard VPN, the easy way

bowlerdesign.tech — Sun, 13 Sep 2020 19:49:07 +0000

Introduction

We're going to cover setting up a Wireguard VPN on your home server or cloud service. For secure remote access to your internal network, or a cheap, secure connection to a cloud service for some increased privacy when browsing online.

VPN's don't make you anonymous, there's a lot of stigma around this. Here's some helpful information if you want to read into this some more.

Setting up Wireguard, the easy way

I initially found setting up Wireguard confusing. Keys kept getting mixed up, I had no way of sending public keys between devices so that I could set up a client on my mobile device etc..

After around 30 seconds of lazy Duck Duck Go'ing (Not quite the same ring to it). I found a script on Github provided by angristan. Here's the repo.

It's just a bash script which does all of the config for you, but still providing user prompts for entering the public server IP and choosing a preferred DNS address for the server.

Step 1: Clone and execute the Wireguard Installer

After ssh'ing to your server, whether it's local, or cloud-hosted. If you're thinking of cloud hosting your Wireguard VPN for some privacy, I'd highly recommend using Digital Ocean. Using the following link will give you $100 worth of credits for 60 days to play around with, just sign up using this link.

Anyway, after you have accessed your machine, we need to pull down the Wireguard installer code from Github. We're just going to curl it.

curl -O https://raw.githubusercontent.com/angristan/wireguard-install/master/wireguard-install.sh

Now we just need to change the file permissions to allow execution of the new .sh scipt we've just downloaded.

sudo chmod +x wireguard-install.sh

Finally, execute the Wireguard Installer

sudo ./wireguard-install.sh

Step 2: Configuring Wireguard

This is the easy part.

You'll see below the process of setting up Wireguard using the Wireguard Installer. All of the values below were picked for me, I just had to hit return a few times.

➜ Ed sudo ./wireguard-install.sh
Welcome to the WireGuard installer!
The git repository is available at: https://github.com/angristan/wireguard-install

I need to ask you a few questions before starting the setup.
You can leave the default options and just press enter if you are ok with them.

IPv4 or IPv6 public address: 37.120.198.182
Public interface: eth2
WireGuard interface name: wg0
Server's WireGuard IPv4: 10.66.66.1
Server's WireGuard IPv6: fd42:42:42::1
Server's WireGuard port [1-65535]: 57281
First DNS resolver to use for the clients: 176.103.130.130
Second DNS resolver to use for the clients (optional): 176.103.130.131

Okay, that was all I needed. We are ready to setup your WireGuard server now.
You will be able to generate a client at the end of the installation.
Press any key to continue...

After running through those steps above, the Wireguard Installer will do its thing and set up Wireguard for you. You'll eventually be left with a prompt to set up a new client.

Tell me a name for the client.
The name must consist of alphanumeric character. It may also include an underscore or a dash.
Client name: Phone
Client's WireGuard IPv4: 10.66.66.2
Client's WireGuard IPv6: fd42:42:42::2

Here's my config, I just entered a name and the rest was generated for me.

What's also really cool, is that a QR code gets generated in the console window, which you can scan with your new device.

You'll also have a .conf file generated for you to copy to your device.

Portforwarding

Remember we had to specify a port? We'll need to forward that. There are a million different tutorials on the web for how to port-forward for your router.

Here's a handy guide.

Final steps

All that's now left to do is to set up Wireguard on your device. Simply download the required app/program onto your machine and either scan the provided QR code or import that .conf file into your client.

Then enable your VPN. Let me know how it goes.

How to self-host Bitwarden Password Manager

bowlerdesign.tech — Sun, 30 Aug 2020 18:10:05 +0000

Introduction

This article will cover setting up your own self-hosted Bitwarden instance with Docker and configuring ngnix to allow for public exposure for cross-device access to your vault.

What is Bitwarden?

Bitwardenis a free and open-source password management service that stores sensitive information such as website credentials in an encrypted vault. The Bitwarden platform offers avariety of client applicationsincluding a web interface, desktop applications, browser extensions, mobile apps, and a CLI.

I use Bitwarden as my main password vault. It stores my card details for automating the filling out of payment forms. Saves me from having to find or remember my card details. I also use Bitwarden for storing all of my passwords.

Having Bitwarden as a public endpoint means that I can connect to my password vault using the Bitwarden app on Android, specifying my self hosted instance.

Setting up the Bitwarden Server

This section of the tutorial is to set up the main Bitwarden 'hub'. This will be a publicly exposed Bitwarden API that will live on your server.

Step 1: Setting up your Linux server

You'll need to either have an existing server instance or create one. I use a Proxmox instance running on a server in my loft. You could also use something like Digital Ocean to host your Bitwarden Server. Using the following link will give you $100 worth of credits for 60 days to play around with, just sign up using this link.

Once you have the server set up, or have logged in. You'll need to do some updates and run some prerequisite installs.

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install docker.io
sudo apt-get install docker-compose

Step 2: Provisioning your Bitwarden Server

Next, you'll need to create a new folder, this will house your Bitwarden Server, you can call it anything memorable. I'll just call mine bitwarden

cd ~
mkdir bitwarden
cd bitwarden

Next, you'll need to create a docker-compose.yml file. This is an orchistration file which docker-compose will use to provision your Docker instance.

touch docker-compose.yml

Next, you'll need to edit your docker-compose.yml file and paste in the following content.

# docker-compose.yml
version: '3'

services:
  bitwarden:
    image: bitwardenrs/server
    restart: always
    ports:
      - 80:80
    volumes:
      - ./bw-data:/data
    environment:
      WEBSOCKET_ENABLED: 'true' # Required to use websockets
      SIGNUPS_ALLOWED: 'true' # set to false to disable signups

I'm using bitwarden_rs as it's written in Rust, faster and more reliable. Also entirely opensource with a heavy user-base.

Save your docker-compose.yml file and exit back to your bitwarden directory.

Step 3: Running your Bitwarden Server locally

Now, you have everything provisioned for running your Bitwarden Server.

The next thing to do is run it.

sudo docker-compose up -d

This will start up your Bitwarden Server inside Docker, it may take some time to pull down the images.

You can eventually see your instance running by executing the following

sudo docker ps

This will list your running instance.

If all is well, you can locally view your Bitwarden Server by navigating to http://localhost:PORT. Or from another machine by using your ip address instead of localhost

You should see something that looks like the following.

Finally, you'll just need to register for an account on your new hosted instance.

Click the Create Account button

Then fill out your details. If you have an existing Bitwarden account, you'll still have to create a new account on this instance. You can then Export and Import between accounts.

The last thing to do is hit Submit

Step 4: Exposing your new server publicly

This part may sound scary, but it is required to allow your Bitwarden Clients (Android, iOS, Chrome extension etc) to connect to your server.

We're going to be using nginx.

Setting up nginx

Nginx is a reverse proxy that allows you to point incoming web traffic to your new Bitwardeb server.

Firstly, install nginx if you haven't already

sudo apt-get install nginx

If you have UFW installed, you will have to Allow Nginx through your local firewall.

I have a tutorial for setting up UFW here

sudo ufw app list

Output
---

Available applications:
  Nginx Full
  Nginx HTTP
  Nginx HTTPS
  OpenSSH

As you can see, there are three profiles available for Nginx:

Nginx Full : This profile opens both port 80 (normal, unencrypted web traffic) and port 443 (TLS/SSL encrypted traffic)
Nginx HTTP : This profile opens only port 80 (normal, unencrypted web traffic)
Nginx HTTPS : This profile opens only port 443 (TLS/SSL encrypted traffic)

It is recommended that you enable the most restrictive profile that will still allow the traffic you’ve configured. Since we will be configuring SSL for our server we will need to allow traffic on port 443.

You can enable this by typing:

sudo ufw allow 'Nginx HTTPS'

Next thing to do is just double check your nginx server is up and running

sudo systemctl status nginx

You should see something that looks like the following

● nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
   Active: active (running)
...

The next part allows us to take incoming traffic and point it to your container instance. Allowing you to expose your Bitwarden server to the internet.

Navigate to /etc/nginx/

cd /etc/nginx/

Use your favorite text editor and open the following file with sudo

sudo vim nginx.conf

I use the following code for my syncing server

server {
    listen 443;
    listen [::]:443;
    server_name vault.bowlerdesign.tech;

    location / {
        proxy_pass http://127.0.0.1:3000; # bitwarden server address
    }
}

Port-forwarding

You will need to port forward your instance to allow public access to your instance. This will involve googling how to port forward from your router.

You'll need to point port 443 to your instance where nginx is set up.

Linking Bitwarden Server with your public domain

You will also need to set up a public domain name. This can then be used to call your new public instance with port 443 exposed.

For example, I would set up a subdomain on bowlerdesign.tech to be vault.bowlerdesign.tech. Notice this is also the domain I specified in my nginx config above.

Here's something to search for with regards to setting up a domain name

Setting up Certbot

Certbot allows us to generate SSL certificates for free with Let's Encrypt. It's simple to install and use. Even hooks in with nginx, meaning that there's no more manual configuration required.

To install Certbot, simply run the following command

sudo apt-get install certbot

Then, to set up your SSL certificate, run

sudo certbot

Follow the instructions, select your domain name from the nginx list.

Also, select redirect as this will upgrade any http requests to https.

Step 5: Connecting to your new Bitwarden instance from a client.

I'm going to use the Firefox Bitwarden Plugin for this part of the tutorial. But the process is identical for all Bitwarden clients.

First, if you haven't already, install your chosen Bitwarden client and open it.

In the top left corner, click the cog icon

You'll then get some configuration. Simply add your full url into the Server URL field

Like so, then just hit Save and log in as normal

That's it

Pretty easy right?

Please don't hesitate to get in touch in the comments if you get stuck. I'd be more than happy to help out with any issues you may face.

Also, if this helped, please consider buying me a beer! It helps with server costs and providing these blog posts.

Thanks for reading!

How to completely self host Standard Notes

bowlerdesign.tech — Tue, 25 Aug 2020 18:29:25 +0000

Introduction

This article will cover setting up your own self-hosted Standard Notes instance and routing your instance through nginx to allow for public exposure.

We'll also go over self-hosting Standard Notes Extensions to allow you to use extensions within your instance, such as a Markdown editor or secure spreadsheets

What is Standard Notes

Standard Notes is a free, open-source, and completely encrypted notes app. Being open-source, allows anyone to self-host their own Standard Notes server. Which means that you own all of your data, on your server.

Standard Notes is great for a secure, private, encrypted note-taking solution. Even without self-hosting, all notes are E2E (end-to-end) encrypted. This means that nobody, other than yourself, can view the notes that you have written.

I have written about Standard Notes in my My Self Hosted Note Syncing Journey Once Switching to Iphone blog post.

Self-hosting Standard Notes is probably for the paranoid and the curious. It was the latter!

It's relatively simple and gives you peace of mind knowing that only you own your data.

Setting up Standard Notes Syncing Server

Step 1: Setting up the syncing server

You'll need to set up a new Linux server. I'll be using a Ubuntu Server instance which I self-host in Proxmox.

This new server will need to have Docker and Docker Compose installed.

These can be installed by running the following commands on your server after you have ssh'd to your server

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install docker.io
sudo apt-get install docker-compose

Next, you'll need clone the syncing-server repo down to your home directory.

git clone https://github.com/standardnotes/syncing-server.git

In the syncing-server directory, you'll need to create a new file named docker-compose.yml

cd syncing-server
touch docker-compose.yml

Open this file with your favorite editor, I use Vim.

You'll need to paste the following yaml code into the docker-compose.yml file. This is what Docker Compose reads to provision your Docker container.

version: '3'
services:
  app:
    build: .
    depends_on:
      - db
    command: bash -c "bundle exec rails db:migrate && bundle exec rails server -b 0.0.0.0"
    env_file: .env
    restart: unless-stopped
    ports:
      - ${EXPOSED_PORT}:3000
  db:
    image: mysql:5.7
    environment:
        MYSQL_DATABASE: '${DB_DATABASE}'
        MYSQL_USER: '${DB_USERNAME}'
        MYSQL_PASSWORD: '${DB_PASSWORD}'
        MYSQL_ROOT_PASSWORD: '${DB_ROOT_PASSWORD}'
    expose:
      - '3306'
    restart: always
    volumes:
      - std_notes_db:/var/lib/mysql
volumes:
  std_notes_db:

Save docker-compose.yml and exit back to the folder.

You'll also need the create a .env file. This is where your environment variables will be stored.

touch .env

Again, edit this file with your favorite editor and paste in the following.

# Rails Settings

EXPOSED_PORT=3000
RAILS_ENV=production
RAILS_LOG_TO_STDOUT=false

# Database Settings

DB_PORT=3306
DB_HOST=127.0.0.1
DB_DATABASE=standard_notes_db
DB_USERNAME=std_notes_user

# Please change this!
DB_PASSWORD=changeme123

# Please change this!
DB_ROOT_PASSWORD=changeme123

DB_POOL_SIZE=30
DB_WAIT_TIMEOUT=180

SECRET_KEY_BASE=changeme123

# Disable user registration
#DISABLE_USER_REGISTRATION=true

# Datadog
DATADOG_ENABLED=false

Save and exit that file.

Now, ensure that you're in the syncing-server directory and run the following command.

docker-compose up -d

This will start docker, using the docker-compose.yml file and run the server in detached mode.

You should hopefully have an instance running at http://localhost:3000

Your MySQL Data will be written to your local disk at /var/lib/mysql - Be sure to back this up.

Step 2: Setting up Nginx

Nginx is a reverse proxy that allows you to point incoming web traffic to your new Standard Notes syncing server. If you only intend on using Standard Notes locally, you can skip this section entirely.

Installation

sudo apt install nginx

If you have UFW installed, you will have to Allow Nginx through your local firewall.

I have a tutorial for setting up UFW here

sudo ufw app list


Output
---

Available applications:
  Nginx Full
  Nginx HTTP
  Nginx HTTPS
  OpenSSH

As you can see, there are three profiles available for Nginx:

Nginx Full : This profile opens both port 80 (normal, unencrypted web traffic) and port 443 (TLS/SSL encrypted traffic)
Nginx HTTP : This profile opens only port 80 (normal, unencrypted web traffic)
Nginx HTTPS : This profile opens only port 443 (TLS/SSL encrypted traffic)

It is recommended that you enable the most restrictive profile that will still allow the traffic you’ve configured. Since we haven’t configured SSL for our server yet in this guide, we will only need to allow traffic on port 80.

You can enable this by typing:

sudo ufw allow 'Nginx HTTP'

Checking your webserver is up and running

sudo systemctl status nginx

You should see something that looks like the following

● nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
   Active: active (running)
...

Step 3: Pointing Nginx to your syncing server

The next part allows us to take incoming traffic and point it to your container instance. Allowing you to expose your syncing server to the internet.

This will allow you to hook up Standard Notes on any device and sync your notes privately.

If someone gets hold of your public endpoint, your notes will still be safe as they are encrypted with your Standard Notes username and password.

Navigate to /etc/nginx/

cd /etc/nginx/

Use your favorite text editor and open the following file with sudo

sudo vim nginx.conf

I use the following code for my syncing server

server {
    listen 80;
    listen [::]:80;
    server_name notes.bowlerdesign.tech;

    location / {
        proxy_pass http://127.0.0.1:3000; # syncing-server address
    }
}

Step 4: Port-forwarding

You will need to port forward your instance to allow public access to your instance. This will involve googling how to port forward from your router.

You'll need to point port 80 to your instance where nginx is set up.

Step 5: Linking Standard Notes with your public domain

You will also need to set up a public domain name. This can then be used to call your new public instance with port 80 exposed.

For example, I would set up a subdomain on bowlerdesign.tech to be notes.bowlerdesign.tech. Notice this is also the domain I specified in my nginx config above.

Here's something to search for with regards to setting up a domain name

Setting up Certbot

Certbot allows us to generate SSL certificates for free with Let's Encrypt. It's really really simple to install and use. Even hooks in with nginx, meaning that there's no more manual configuration required.

To install Certbot, simply run the following command

sudo apt-get install certbot

Then, to set up your SSL certificate, run

sudo certbot

Follow the instructions, select your domain name from the nginx list.

Also select redirect as this will upgrade any http requests to https.

Step 6: Connecting to your Standard Notes syncing server from Standard Notes

Congratulations on making it this far, now it's time to hook up Standard Notes with your new private syncing server.

Standard Notes is available on multiple platforms. The easiest way for this tutorial is to use their web interface.

Navigate to https://app.standardnotes.org/

You'll be greeted with something that looks like the following screenshot

You will need to sign up with a new account as this will be syncing to your server.

Click Register

Before filling out your registration credentials, select Advanced Options

Now fill in your new server url or IP address.

Then go back to the registration form and enter an Email and Password

Finally, click Register

Easy right? You now have a working self-hosted Standard Notes server.

Setting up self-hosted Standard Notes extensions

Now that you have Standard Notes working, you can add and delete notes at your heart's desire. But wouldn't it be cool to also use the Standard Notes Extensions to fully embrace the power of Standard Notes?

You can leave it here, or follow the below process to self-host your extensions too, utilising the power of Markdown editors or Secure Spreadsheets!

Firstly, by self-hosting your extensions, you're bypassing the need to purchase a license. The Standard Notes team mainly works from a 'Visionary' point of view. I heavily encourage you to purchase a license to support the great work that the team are doing. Here's a quote from the Standard Notes team themselves.

"Most of our extensions are open-source and available for self-hosting. You can also learn to develop your own extensions by following the guides on this site. However, we encourage you to support the sustainability and future development of this project by purchasing a subscription." ~ Sustainability of Standard Notes

If however, you would still like to proceed with hosting your Extensions, then please follow the steps below.

Step 1: Pulling the Standard Notes Extensions Repository

I will be using the Standard Notes Extention repository by iganeshk who has provided their own README.md for setting up Standard Notes Extensions. I'll be using their instructions in my steps below. All credit for the setup goes to iganeshk

Let's begin.

Firstly, start by ssh'ing into your new server that we set up in the previous section.

We'll need to pull down the required repository

git clone https://github.com/iganeshk/standardnotes-extensions.git
cd standardnotes-extensions

Ensure that you have Python3 and pip installed on your server as this is used to install the required dependencies for the extensions repository.

sudo apt-get install -y python3 python3-pip

Then use pip3 to install the required dependencies. You may have to reboot before this command works.

pip3 install -r requirements.txt

Step 2: Configuring the Standard Notes Extensions Repository

You'll now need to create a .env file to store your environment variables for the repository.

Ensure you're in the standardnotes-extensions directory

cd ~/standardnotes-extensions

Then create the .env file

touch .env

Then open with your favorite editor

vim .env

Paste in the following example from iganeshk

env.sample

Edit your domain value, and Github username and TOKEN

The Github integration is to allow you to clone down the Standard Notes extensions from the Standard Notes Github page.

# Github Credentials
# Generate your token here: https://github.com/settings/tokens
# No additional permission required, this is just to avoid github api rate limits
#

domain: https://domain.com/extensions

github:
  username: USERNAME
  token: TOKEN

Save and close that file. Ensuring that it is named .env

Step 3: Running Standard Notes Extensions from your server

You'll now need to execute the service.

This can be done by running the following command.

python3 build_repo.py

This will generate a /public directory, which you will serve.

Step 4: Routing with nginx

Now, with that static folder generated, we just need to point nginx to that /public folder to serve the extensions.

As before in the previous section, edit your nginx config

Navigate to /etc/nginx/

cd /etc/nginx/

Use your favorite text editor and open the following file with sudo

sudo vim nginx.conf

You'll see your previously added nginx config for serving the syncing server

server {
    listen 80;
    listen [::]:80;
    server_name notes.bowlerdesign.tech;

    location / {
        proxy_pass http://127.0.0.1:3000; # syncing-server address
    }
}

Under the server section, you will need to add the following configuration, again taken from iganeshk.

location ^~ /extensions {
    autoindex off;
    alias /path/to/standardnotes-extensions/public;
    # CORS HEADERS
    if ($request_method = 'OPTIONS') {
       add_header 'Access-Control-Allow-Origin' '*';
       add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
       #
       # Custom headers and headers various browsers *should* be OK with but aren't
       #
       add_header 'Access-Control-Allow-Headers' 'DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range';
       #
       # Tell client that this pre-flight info is valid for 20 days
       #
       add_header 'Access-Control-Max-Age' 1728000;
       add_header 'Content-Type' 'text/plain; charset=utf-8';
       add_header 'Content-Length' 0;
       return 204;
    }
    if ($request_method = 'POST') {
       add_header 'Access-Control-Allow-Origin' '*';
       add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
       add_header 'Access-Control-Allow-Headers' 'DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range';
       add_header 'Access-Control-Expose-Headers' 'Content-Length,Content-Range';
    }
    if ($request_method = 'GET') {
       add_header 'Access-Control-Allow-Origin' '*';
       add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
       add_header 'Access-Control-Allow-Headers' 'DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range';
       add_header 'Access-Control-Expose-Headers' 'Content-Length,Content-Range';
    }
  }

Your nginx.conf should now look like the following

server {
    listen 80;
    listen [::]:80;
    server_name notes.bowlerdesign.tech;

    location / {
        proxy_pass http://127.0.0.1:3000; # syncing-server address
    }

    location ^~ /extensions {
    autoindex off;
    alias /path/to/standardnotes-extensions/public;
    # CORS HEADERS
    if ($request_method = 'OPTIONS') {
       add_header 'Access-Control-Allow-Origin' '*';
       add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
       #
       # Custom headers and headers various browsers *should* be OK with but aren't
       #
       add_header 'Access-Control-Allow-Headers' 'DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range';
       #
       # Tell client that this pre-flight info is valid for 20 days
       #
       add_header 'Access-Control-Max-Age' 1728000;
       add_header 'Content-Type' 'text/plain; charset=utf-8';
       add_header 'Content-Length' 0;
       return 204;
    }
    if ($request_method = 'POST') {
       add_header 'Access-Control-Allow-Origin' '*';
       add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
       add_header 'Access-Control-Allow-Headers' 'DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range';
       add_header 'Access-Control-Expose-Headers' 'Content-Length,Content-Range';
    }
    if ($request_method = 'GET') {
       add_header 'Access-Control-Allow-Origin' '*';
       add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
       add_header 'Access-Control-Allow-Headers' 'DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range';
       add_header 'Access-Control-Expose-Headers' 'Content-Length,Content-Range';
    }
  }
}

Make sure you change the alias to point to your public folder. Remember, this is in the /standardnotes-extensions/public location.

For me it would be

alias /home/ubuntu/standardnotes-extensions/public

Close and save the nginx.conf

That's it for config.

The last thing you'll need to do in terms of setup is to verify the endpoint.

Step 5: Verifying the Standard Notes Extensions endpoint

Navigate to your previous url and add extensions/index.json to the end of the url.

For example

http://notes.bowlerdesign.tech/extensions/index.json

You should see a list of json objects.

If, you don't see the endpoint correctly, ensure that the /public folder has the correct permissions from nginx to read the files.

Step 6: Adding Extensions to Standard Notes

Nearly there. We just need to add our new endpoint to our existing Standard Notes client.

Navigate back to https://app.standardnotes.org and log in as we did before.

In the bottom corner, select Extensions

You'll see a field that reads Enter Extended Code this is where you'll usually paste your Extensions API Key. But all you need to do is paste your new extensions url.

Then hit Submit Code

And, we're done

That's it, providing everything worked, you should now be able to add extensions to your Standard Notes instance.

Now, this process is complicated, so please don't hesitate to get in touch in the comments if you get stuck. I'd be more than happy to help out with any issues you may face.

Thanks for reading, please donate to Standard Notes as, without them, none of this would be possible.

Also, if this helped, please consider buying me a beer! It helps with server costs and providing these blog posts.

How to Set Up a Headless Syncthing Network

bowlerdesign.tech — Mon, 03 Aug 2020 18:31:00 +0000

Overview of Syncthing

Syncthing is a P2P (peer to peer) network that allows you to keep your files synchronised.

Syncthing aims to replace services such as: Dropbox, Google Drive and OneDrive. By putting the user in control of their own data. Syncthing enables the user to set up multiple 'nodes' which can communicate with eachother.

A node can be, your personal computer, a work machine, a machine in another location or even a central server.

Syncthing will encrypt all data when it is being transported to your other nodes. Nodes don't just have to be on your local network, you can sync between nodes over the internet. Encrypted, of-course.

Use case

I use Syncthing as my storage solution. I have a 'central' Syncthing server which lives in my loft. I quoted 'central' as the loft server is just a node, there's nothing special about it other than having a backup cron job running on it.

I then have multiple other nodes, such as: my main laptop, a work machine and even my Android phone. I have spoken about Syncthing on Android in my My Self Hosted Note Syncing Journey Once Switching to Iphone post.

With the multiple nodes set up, I can choose folders that can sync. I'm currently writing this blog post on my personal laptop, which is syncing to my blog server as I type (well, hit save).

You can then specify which nodes, the folder you created, to be shared with. So in my case, I am sharing my blog with my personal laptop and my 'central' server, just to backup my data.

Setting up Syncthing

I'm going to show you how to set up Syncthing in the context of my setup. So having that 'central' server.

My 'central' server is running Ubuntu Server, so the following instructions are tailored for Debian distros.

Firstly, you need to install the correct package locations in order to download Syncthing correctly.

sudo apt-get install curl apt-transport-https

curl -s https://syncthing.net/release-key.txt | sudo apt-key add -

echo "deb https://apt.syncthing.net/ syncthing stable" | sudo tee /etc/apt/sources.list.d/syncthing.list

Next, update your local repository with the new Syncthing repository

sudo apt-get update
sudo apt-get upgrade

sudo apt-get install syncthing

The next thing you want to do is enable Syncthing as a system service. This will allow start-up on boot and continuous execution in the background of your server.

sudo systemctl enable syncthing@username.service

sudo systemctl start syncthing@username.service

So this will start Syncthing on your Ubuntu Server. Problem is, Ubuntu Server is usually headless. So at the moment there is no way to access the Syncthing GUI to configure your folders and nodes.

The way we can change this to allow local access from another machine is to change the serve address.

When you ran the Syncthing command above, a syncthing folder was created under your /home/username/.config folder.

Inside that will be a config.xml file.

Open this with a text editor, something like Vim or Nano. It really doesn't matter.

Within config.xml there is a section that looks like this:

...
<gui enabled="true" tls="false" debugging="false">
    <address>127.0.0.1:8384</address>
    <apikey>...</apikey>
    <theme>default</theme>
</gui>
...

Under the address attribute, you need to update the ip address.

0.0.0.0 will allow all machines on your local network to be able to access the Syncthing GUI.

...
<gui enabled="true" tls="false" debugging="false">
    <address>0.0.0.0:8384</address>
    <apikey>...</apikey>
    <theme>default</theme>
</gui>
...

Save and close config.xml and then restart Syncthing by running the following command.

sudo systemctl restart syncthing@username.service

Second Node

For Syncthing to work, you need to set up another Syncthing node. Simply follow the above steps again on another machine, or jump to the bottom of this page quickly and install Syncthing on Windows or Android

Accessing the Syncthing GUI

Now with all that running, you should be able to access the Syncthing GUI by navigating to:

http://{your_server_ip}:8384

Once entering your ip-address or hostname, you should be presented with something that looks like this

Adding Devices (nodes)

To add a new Device, select the Add Remote Device button, which will open up a new popup window.

The Add Device window will pop up. You may notice that there is an existing ID which is available to click.

Syncthing automatically polls your local network looking for existing nodes, depending on how your other nodes are set up, you may or may not see this.

If you don't see the ID, simply navigate to your other Syncthing node (let's call it Syncthing B), select Actions > Show ID from the top right menu.

Here you will see the Syncthing B Syncthing ID. Copy this ID or use the QR code to scan the ID. And paste the ID into the Device ID field back on the Syncthing A node.

Complicated at first, but once you get the hang of Syncthing, it'll be easier.

Click Save on that window and it'll add Syncthing B as a new device.

Sharing Folders

Now, on Syncthing A click the Add Folder button on the left hand side of the screen. This will pop up with a new window.

Give your new folder a name and change the path to be relative to the folder on your machine that you want to backup. I tend to keep all of my synced folders under /home/ed/Syncthing just so that I know where everything is.

Navigate to the Sharing Tab

Here are the devices you have added. I have a couple more than you probably will just because this is my existing system. Simply select the device that you added and click Save

Accepting the new device

Next, on Syncthing B, there will be a new call-to-action at the top of the page. This is a request for you to allow Syncthing A to add Syncthing B as a device. Simply click Add.

Then, shortly after accepting the new device, you'll get another popup which is Syncthing A requesting to share the folder you have just create.

Click Add

Finally, you now need to choose a location for this folder to sync to.

Click Save and that's it.

Testing

Feel free to test by adding a file to the Syncthing A folder and wait for the global state to change on Syncthing B

Setting up other Syncthing nodes

Syncthing works on everything apart from iPhone. There are whispers of something coming to iOS in the future, but not right now.

Setting up another node on a Linux machine

Setting up a node on another Linux machine is as simple as just copying the steps above. If the machine isn't headless, such as a desktop machine, you can ignore the part about changing the ip address in the config.xml

Setting up another node on a Windows machine

Setting up on Windows is relativley easy. You can either go the manual route of downloading the Syncthing exe. Or by opting for the simple solution which is installing SyncTrayzor which adds a system tray icon and runs Syncthing in the backgroud automatically.

Setting up another node on an Android device

Syncthing has apps in the Play Store and F-Droid which can be downloaded and installed.

Thanks

Let me know if this has helped in the comments below.

Also, feel free to check out my other blog posts here

Configuring Vim for Node Js Development

bowlerdesign.tech — Tue, 28 Jul 2020 18:58:00 +0000

Using Vim

If you have come across this article, I probably don't need to explain why you need to use Vim as your development IDE.

You're probably here because you want to use Vim, but not sure how to integrate it into your current workflow.

I was in the same position a few months ago.

The easiest way is to install a vim extension into your current IDE of choice, this will bring all of the features of vim into your current workflow.

After getting used to Vim, in something that you're familiar with. You'll probably want to jump-ship into full-blown Vim.

That's what this blog post is for.

The Primeagen

A lot of my configuration and interest in jumping over to Vim came from The Primeagen.

This guy is a machine with Vim.

He's currently a Netflix Engineer who only works in Vim for his day-to-day work. Regularly streaming over on Twitch and uploading on YouTube.

He has recently produced a series on Vim As Your Editor. I highly recommend that you go and watch it.

Plugins

Vim is highly configurable with user-created plugins.

Here's a list that I recommend to use with Node.Js:

vim-fugitive
- This integrates Git seemlessly into Vim
coc.nvim
- This allows code completion and intellisense for Javascript in Vim
nerdcommenter
- A simple plugin to comment out lines of code using Vim bindings
ale
- A syntax highlighter for Vim
nerdtree
- A file tree view for selecting files and navigating folder structure in your projects
lightline
- A lightweight status bar to show progress in your buffer and other useful information
fzf
- A file-finding plugin to quickly search your project and swtich buffers
gruvbox
- The only theme you should be using. Heavily influenced by The Primeagen
vim-buftabline
- Introduces 'tabs' into Vim for keeping track of your open buffers

Installing plugins is easy in Vim. I use a plugin manager called vim-plug.

.vimrc.plug

Your plugin installation lives in a file called .vimrc.plug.

Here's a copy of my .vimrc.plug as an example

" .vimrc.plug

call plug#begin('~/.vim/plugged')
" Git
Plug 'tpope/vim-fugitive'

" Code completion
Plug 'neoclide/coc.nvim', {'branch': 'release'}

" Code commenter
Plug 'preservim/nerdcommenter'
"
" Syntax highlighting
Plug 'dense-analysis/ale'

" NERDTree
Plug 'preservim/nerdtree'
"
" Statusbar
Plug 'itchyny/lightline.vim'
"
" Finder
Plug 'junegunn/fzf', { 'do': { -> fzf#install() } }
Plug 'junegunn/fzf.vim'

" File finder
Plug 'vifm/vifm.vim'

" Theme
Plug 'morhetz/gruvbox'

" Tabs
Plug 'ap/vim-buftabline'

call plug#end()

set background=dark
colorscheme gruvbox

Just install those plugins using :PlugInstall

.vimrc

.vimrc is your main Vim config file. You probably know this already.

Here's mine:

syntax on

set guicursor=
set noshowmatch
set relativenumber
set nohlsearch
set hidden
set noerrorbells
set tabstop=4 softtabstop=4
set shiftwidth=4
set expandtab
set smartindent
set nu
set nowrap
set smartcase
set noswapfile
set nobackup
set undodir=~/.vim/undodir
set undofile
set incsearch
set termguicolors
set scrolloff=8
set modifiable

" Give more space for displaying messages.
set cmdheight=2

" Having longer updatetime (default is 4000 ms = 4 s) leads to noticeable
" delays and poor user experience.
set updatetime=50

" Don't pass messages to |ins-completion-menu|.
set shortmess+=c

set colorcolumn=100
highlight ColorColumn ctermbg=0 guibg=lightgrey

map <C-n> :NERDTreeToggle<CR>
map <C-p> :Files<CR>
map <C-f> :Rg<CR>
map <C-t> :e <cfile><cr>
map <S-Tab> :bn<CR>
map <F5> :setlocal spell! spelllang=en_gb<CR>

let loaded_matchparen = 1
let mapleader = " "

" CoC
" GoTo code navigation.
nmap <leader>gd <Plug>(coc-definition)
nmap <leader>gy <Plug>(coc-type-definition)
nmap <leader>gi <Plug>(coc-implementation)
nmap <leader>gr <Plug>(coc-references)
nmap <leader>rr <Plug>(coc-rename)
nmap <leader>g[ <Plug>(coc-diagnostic-prev)
nmap <leader>g] <Plug>(coc-diagnostic-next)
nmap <silent> <leader>gp <Plug>(coc-diagnostic-prev-error)
nmap <silent> <leader>gn <Plug>(coc-diagnostic-next-error)
nnoremap <leader>cr :CocRestart

" Sweet Sweet FuGITive
nmap <leader>gj :diffget //3<CR>
nmap <leader>gf :diffget //2<CR>
nmap <leader>gs :G<CR>

" Search and replace hotkey
nnoremap H :%s//gc<left><left><left>

" Move highlighted text up and down
vnoremap J :m '>+1<CR>gv=gv
vnoremap K :m '<-2<CR>gv=gv

" Import plugins
if filereadable(expand("~/.vimrc.plug"))
    source ~/.vimrc.plug
endif

" Status bar config
set statusline+=%#warningmsg#

" Fix files automatically on save
let g:ale_fixers = {}
let g:ale_javascript_eslint_use_global = 1
let g:ale_linters = {
  \'javascript': ['eslint'],
  \'vue': ['eslint', 'stylelint', 'tsserver'],
\}

let g:ale_fixers = {
  \'javascript': ['prettier', 'eslint'],
  \'vue': ['eslint', 'stylelint'],
\}

let g:ale_linters_explicit = 1
let g:ale_sign_column_always = 1
let g:ale_sign_error = '>>'
let g:ale_sign_warning = '--'
let g:ale_fix_on_save = 1

" Close NERDTree when closing the last buffer
autocmd bufenter * if (winnr("$") == 1 && exists("b:NERDTree") && b:NERDTree.isTabTree()) | q | endif

fun! TrimWhitespace()
    let l:save = winsaveview()
    keeppatterns %s/\s\+$//e
    call winrestview(l:save)
endfun

autocmd BufWritePre * :call TrimWhitespace()

command! -bang -nargs=* Rg
  \ call fzf#vim#grep(
  \ 'rg --column --line-number --no-heading --color=always --smart-case -- '.shellescape(<q-args>), 1,
  \ fzf#vim#with_preview(), <bang>0)

There is some initial setup prior to using this configuration. Such as:

Running :CocInstall coc-tsserver
Installing fzf sudo apt-get install fzf
Installing Rg sudo apt-get install ripgrep

Final points

Vim is a steep learning curve. Get used to Vim by using a plugin in your existing IDE, once comfortable, switch to Vim.

Let me know if this post has helped you in the comments below, I'd appreciate your feedback.

Setting up Cucumber, Nightwatch.js and Selenium Grid for highly scalable remote Chrome execution from Buddy

bowlerdesign.tech — Tue, 21 Jul 2020 11:26:00 +0000

Technical Overview

Nightwatch.js - Node.js powered end-to-end testing framework
Cucumber - BDD testing framework that sits on top of Nightwatch.js
Selenium Grid - Highly scalable browser automation for running tests
Buddy - CI/CD pipeline

End Goal

The outcome of setting up this pipeline will allow us to execute end-to-end tests from anywhere, in any development environment.

Having Selenium Grid as a public endpoint also allows us to execute these end-to-end tests from our existing CI/CD pipeline.

Selenium Grid

Selenium Grid, in short, is a central server that our test framework (Nightwatch.js) will be able to send requests to. Selenium Grid can spin up multiple Nodes which will allow for test execution against remote-controlled headless Chrome Instances.

The beauty of Selenium Grid is that it is highly scalable. Each Node can have a defined amount of Chrome instances (or Firefox, Internet Explorer, and Opera). And a grid can register an infinite amount of Nodes.

Nodes can either live on the same instance as the Selenium Grid Hub (the central processing endpoint), or the Nodes can live on separate server instances to allow for fine-tuned server scaling for each node.

Nightwatch.js

We wanted to integrate Nightwatch.js into our end-to-end testing pipelines because it integrates with our existing microservices flawlessly.

Tests are simple to write, with the comprehensive API developer documentation. And is fully supported by the community if we run into anything tricky.

Cucumber

Cucumber allows us to write tests in a human-readable, user story kind of manner. This works great for our inhouse testers as it can be tricky for them to write tests from a technical mindset. Having the tests as user stories already integrate with our current working methods. This also helps us as a team to think about test scenarios from the perspective of a user.

Here's an example of a Cucumber test:

Feature: Google

    Scenario: Google webpage loads

        Given I navigate to Google
        And I log in with a valid-user
        And I use the search filter to search for "Test Automation"
        Then the title is "Google - Test Automation"

Cucumber will 'translate' this into a full test and pass it over to Nightwatch, which will execute the tests.

Putting it all together

Setting up Cucumber to use Nightwatch.js

This was tricky at first as Nightwatch.js is a standalone test runner, it works without Cucumber. Tests can be written by only using Nightwatch.js if BDD isn't needed. But we required Cucumber as we wanted to involve our testers as much as possible.

Installing the necessary npm packages

To be able to run Cucumber with Nightwatch.js, we need some packages from npm.

We'll need the following:

cucumber
cucumber-pretty
nightwatch
nightwatch-api

These can be installed by running the following command (global install)

sudo npm i -g cucumber cucumber-pretty nightwatch nightwatch-api

Setting up your node environment to execute end-to-end tests

Next, we need to add a build script to our package.json to allow us to run Cucumber from npm

// package.json
"scripts": {
        ...
        "e2e-test": "cucumber-js --require cucumber.conf.js --require tests/**/step-definitions --format node_modules/cucumber-pretty tests/*"
    },

We also need to create a cucumber.conf.js file to specify our Cucumber settings.

// cucumber.conf.js
const { setDefaultTimeout, AfterAll, BeforeAll } = require('cucumber');
const { createSession, closeSession } = require('nightwatch-api');

setDefaultTimeout(60000);

BeforeAll(async () => {
  await createSession();
});

AfterAll(async () => {
  await closeSession();
});

Finally, we need to create a nightwatch.conf.js file to add our execution settings

// nightwatch.conf.js
module.exports = {
    "src_folders" : ["tests/"],

    "selenium" : {
        "start_process" : false, // Important for running test execution on a remote Selenium Grid
        "host" : "seleniumgrid.example.com", // Endpoint for Selenium Grid
        "port" : 443,
        "use_ssl": true
    },

    "test_settings" : {
        "default" : {
            "selenium_port": 443,
            "selenium_host": "seleniumgrid.example.com", // Endpoint for Selenium Grid
            "use_ssl": true,
            "desiredCapabilities": {
                "browserName": "chrome", // Specified Browser
                "javascriptEnabled": true,
                "acceptSslCerts": true,
                "chromeOptions": {
                    "w3c": false, // Important
                    "args": ["--no-sandbox", "--disable-setuid-sandbox", "--disable-gpu", "--disable-dev-shm-usage"], // Important
                },
                // This is to pass config when using Nodes
                "goog:chromeOptions": {
                    "w3c": false, // Important
                    "args": ["--headless", "--no-sandbox", "--disable-setuid-sandbox", "--disable-gpu", "--disable-dev-shm-usage"], // Important
                }
                // Both chromeOptions objects are required for this to work
            },
        },
    }
}

That's all the configuration needed, your folder structure should now look like this:

project_root/
  package.json
  cucumber.conf.js
  nightwatch.conf.js

Folder Structure

Cucumber is very particular about folder structure. The recommended folder structure is the following:

project_root
├ cucumber.conf.js
├ nightwatch.conf.js
├ package.json
└ tests
    └ features
        ├ ebay.feature
        ├ google.feature
        └ step-definitions
            ├ ebay.js
            ├ google.js
            └ shared.js

Provisioning Selenium Grid

This is where we create the Selenium Grid server. We host ours in AWS EC2 on a T2.micro Ubuntu Server instance. We tested with CentOS but could not get everything working properly. Ubuntu Server works flawlessly and is highly recommended if you're looking at setting this up.

Here is a script that I wrote to provision Selenium Grid on Ubuntu Server. It can copy this script and save it as a .sh file. Then execute this file in the scope of your Linux instance. It will require some tweaking of the paths to match up with your server.

#! /bin/bash

# This script requires default-jre and unzip to be installed
# Run the following as Sudo
# `sudo apt-get update && sudo apt-get install -y default-jre unzip xvfb screen`

# Enable execute permissions with
# `sudo chmod +x setup.sh`

# Please execute this script as your user
# e.g `sudo -u your_user ./setup.sh`

cd $HOME

mkdir TestAutomation
cd TestAutomation

# Download Chrome
wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb

# Install Chrome
sudo apt install -y ./google-chrome-stable_current_amd64.deb

# Remove deb file
rm google-chrome-stable_current_amd64.deb

# Download Selenium Grid
wget https://selenium-release.storage.googleapis.com/3.141/selenium-server-standalone-3.141.59.jar

# Download Chrome Driver
wget https://chromedriver.storage.googleapis.com/83.0.4103.39/chromedriver_linux64.zip

unzip chromedriver_linux64.zip

rm chromedriver_linux64.zip

# Create spawn files
export DISPLAY=:0.0
echo "screen -d -m java -jar /home/ubuntu/TestAutomation/selenium-server-standalone-3.141.59.jar -role hub" > ../spawn-hub.sh
echo "screen -d -m xvfb-run --auto-servernum --server-num=0 java -Dwebdriver.chrome.driver=/home/ubuntu/TestAutomation/chromedriver -Dwebdriver.chrome.bin=/usr/bin/google-chrome -jar /home/ubuntu/TestAutomation/selenium-server-standalone-3.141.59.jar -role node -hub http://localhost:4444/grid/register -browser browserName=chrome,maxInstances=10" > ../spawn-node.sh

# Set spawn file permissions
chmod +x ../spawn-*

Executing the generated spawn-hub.sh will result in Selenium Grid spinning up on the port 4444. The grid will then listen for Nodes.

To spin up a Node, execute the generated script spawn-node.sh. This command can be executed multiple times to spin up multiple Nodes.

TIP: If you want your Nodes on separate instances, copy the above provisioning script and execute it on your Node instances. But only execute the spawn-node.sh script.

Tying it together

Now all that is needed is to expose your Selenium Grid server to the web. I won't go into detail on how to do this, as it's different for everyone.

Make sure that you update your nightwatch.conf.js file to point to your new hosted endpoint.

Executing the tests

All that is now needed to run these tests is the following command:

npm run e2e-test

To recap...

This will run the Cucumber command to find all cucumber tests and their respective step-definitions.
Pass them to Nightwatch to act as the test handler.
Nightwatch will pass those tests to our remote Selenium Grid instance.
Selenium Grid will distribute the tests to the test Nodes.
The test Nodes will spin up Google Chrome and execute their assigned tests.
Once finished, passing back the test output to Nightwatch.js and displaying the output in the npm console.

Chrome Plugins

It's also worth mentioning that this execution pipeline allows for a full instance of Google Chrome to run, which means that you can use browser plugins as part of your execution pipeline. This is a rare thing to achieve but is possible using the above steps.

Adding all this to Buddy

Adding to Buddy was relatively simple. We created a small pipeline to pull down the latest change from this test repository.

Buddy allows an action to call another pipeline. So as part of our microservice pipeline, we call the external tests pipeline to execute the tests.

- action: "Run Selenium Grid Tests"
    type: "RUN_NEXT_PIPELINE"
    wait: true
    comment: "Triggered by $BUDDY_PIPELINE_NAME execution #$BUDDY_EXECUTION_ID"
    trigger_condition: "ALWAYS"
    revision: "HEAD"
    variables:
    - key: "TEST_TAGS"
      value: "\"@google\""
    next_project_name: "automationtests"
    next_pipeline_name: "Run Selenium Grid Tests"

We used Tags to be able to execute specific tests as part of our deployment, that's the TEST_TAGS variable you can see in the above snippet.

Here's our pipeline for running the test project:

- pipeline: "Run Selenium Grid Tests"
  trigger_mode: "MANUAL"
  ref_name: "master"
  ref_type: "BRANCH"
  trigger_condition: "ALWAYS"
  actions:
  - action: "Format Test Tags"
    type: "BUILD"
    working_directory: "/buddy/automationtests"
    docker_image_name: "library/ubuntu"
    docker_image_tag: "18.04"
    execute_commands:
    - "TAGS=$TEST_TAGS"
    - "NODE_ENV=$ENV"
    volume_mappings:
    - "/:/buddy/automationtests"
    trigger_condition: "ALWAYS"
    shell: "BASH"
  - action: "Find & replace"
    type: "REPLACE"
    local_path: "package.json"
    replacements:
    - replace_from: "{TAGS}"
      replace_to: "$TAGS"
    trigger_condition: "ALWAYS"
  - action: "Run tests with Selenium Grid"
    type: "BUILD"
    working_directory: "/buddy/automationtests"
    docker_image_name: "library/node"
    docker_image_tag: "12"
    execute_commands:
    - "# yarn install"
    - "npm install"
    - ""
    - "npm run e2e-test-buddy --silent -- tests/* >> test-output.txt"
    volume_mappings:
    - "/:/buddy/automationtests"
    trigger_condition: "ALWAYS"
    shell: "BASH"
    execute_every_command: true
  variables:
  - key: "NODE_ENV"
    settable: true

That's it

All done, you should now have remote test execution.

Please get in touch with us if you have any issues

Thanks for reading.

My Self Hosted Note Syncing Journey Once Switching to Iphone

bowlerdesign.tech — Tue, 25 Feb 2020 11:32:00 +0000

Recently moving to the iPhone meant that I had to re-think and re-design my existing note-taking solution, this is a quick writeup of my journey.

My existing solution with Syncthing and VS Code

I loved my original solution. I had recently discovered Syncthing and got it set up on my self hosted solution in my loft. I had a central sync server and then a node on each of my devices, including my Huawei P20 Pro.

I could keep my extensive markdown nodes on every device and sync them without even thinking about it. It just worked, I wasn't tied into a service as the nodes are written in Markdown.

Everything just worked.

Switching to iPhone

I made the switch to the iPhone due to privacy concerns with Android. I have recently boycotted all my Google services and switched over to a self-hosted solution. Part of this meant getting rid of my Android.

Everything started great with the iPhone, the interface is great, everything just works.

Apart from Syncthing.

There's currently no iOS client for Syncthing.

So what now? I started looking at other app alternatives, Bear, Markdown Notes, etc.. but all of them locked down where the files were saved on the iPhone, I couldn't access them from a file explorer on the iPhone. Something that could easily be done with Android.

Bumped into Resilio, not self-hosted, but achieves the whole decentralized syncing that I loved about Syncthing. Cool, I managed to get my notes syncing to a Resilio folder on my iPhone. Bingo!

But wait, what happens when I try and edit a note? It opens a copy of the note in a markdown app. Not ideal, I just want to view and edit notes, how hard can it be!

I give up.

Standard Notes

I had a love-hate relationship with Standard Notes. I loved the UI, the syncing was pretty good, it has an iOS app! and it can be self-hosted, amazing.

But a lot of the features are paid. So if I wanted to write in Markdown, which I do, I had to pay for the extensions. I had to think to myself, how badly do I want notes on my iPhone?

I went without notes for a week, it nearly killed me. So I bit the bullet and settled for one month at £9.99, I thought I'd give it a try.

Loved it, I heavily use Markdown Basic, Simple Task Editor and Secure Spreadsheets extensions. But can I justify £10 a month?

Standard Notes Extensions

Standard Notes being open-source meant that I could have a poke around in its git repository. I did a global search on Github for "Standard Notes" and I bumped into a repository by Github User jonhadfield who has collected some awesome Standard Notes extensions. Browsing his README.md I spotted the repository Extensions Repository Builder. This would allow me to self-host my extensions rather than paying the monthly fee!

I set up Extensions Repository Builder on my internal host and bingo, it just worked, amazing.

Full E2E Sync

So after a very long journey of pulling my hair out and re-evaluating how much these notes meant to me. I finally had an end-to-end solution for storing my notes and syncthing between all of my devices.

Where I'm going next

I have a did.txt file which I log to every day with what I did that day, it helps with our daily standups so that I can remember what I did the previous day. On jonhadfield's Github, there was also a link for standardnotes-fs which allows you to mount your Standard Notes as files using fuse. This means I can set up my did.txt command to edit my Did note.

I'd also like to hook up iOS reminders to a reminders section in Standard Notes. I own an Apple Watch and I tend to use Siri on it quite often. So I need to work out some way to sync these with my standard notes.

How to Route Nginx Through a Child Nginx Configuration - Jumpbox to Web Server Configuration

bowlerdesign.tech — Thu, 02 Jan 2020 11:34:00 +0000

I had a specific use case where I needed to run a docker instance, which had it's own configured nginx instance.

The issue was that I already use nginx on my JumpBox.

The fix was relatively self explanatory, but I wanted somewhere to write down the issues that I had to tackle.

The Solution

So the solution is just to point nginx (JumpBox) to the other nginx instance (docker).

Just to let you know, in my case, there are separate physical servers. The configuration will not differ, but you may need to internally expose ports between hosts if you have something like UFW installed. I have a tutorial here on how to set up UFW.

So on my Host B I have a web instance running on port 8080. This web instance is an nginx reverse proxy that is pointing to another docker instance within the same docker network on the machine.

Host A is my existing nginx reverse proxy.

The quick solution is to forward your call from Host A to Host B.

Like so

server {
  server_name blog.bowlerdesign.tech;

  location / {
    proxy_pass http://host_running_blog:8080;
  }

I ran into issues here, I couldn't get css loading properly when navigating to blog.bowlerdesign.tech.

I was missing headers in the request. Here's an ideal configuration for setting up a website.

server {
  server_name blog.bowlerdesign.tech;

  location / {
    proxy_pass http://host_running_blog:8080;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto scheme;
    proxy_pass_header Content-Type;
  }

This will forward the necessary headers for your host.

Setting Up UFW on Ubuntu Server

bowlerdesign.tech — Sun, 15 Dec 2019 11:29:00 +0000

UFW (Uncomplicated Firewall) is a program that allows you to internally control ports on your Linux instance. This gives you the ability to forward ports from your machine.

The common use of a firewall is to control the ports that have access from the outside world, for instance, running a website would need ports 80/443 exposed on your network to be able to route your site.

UFW is different, think of port forwarding, but between local instances. You can lock down internal exposure to port 22 (ssh) for example.

Why? What's the point?

Security.

In the coming weeks, I'll be writing blog posts on how I have set up my jumpbox server. UFW plays a key part in my setup. I have used UFW to only open port 22 on my jumpbox server. I can ssh in, but that's it. No other ports can be attacked or sniffed.

From the jumpbox server I can then only ssh into my other internal instances. This means that if I wanted to ssh into server B I would have to go via the jumpbox A.

I think I get it, how can I install it?

We're running Ubuntu Server on a Raspberry Pi. But these instructions are for all Debian instances, the Raspberry Pi is irrelevant for this tutorial.

Let's install

Install UFW

sudo apt-get install ufw

Check the status of UFW

sudo ufw status

You should see that UFW is disabled

Let's allow some ports, it's really important that you allow your ssh port, otherwise you can lose access when we get round to enabling UFW. The default ssh port is 22 unless you have changed the default port.

sudo ufw allow 22

You can use the above command to allow the necessary ports for your instance. We're just going to stick with port 22 for this example.

Let's enable UFW

sudo ufw enable

That's it

You now have a firewall running on your local instance, locked down to be only accessible by port 22.

In the future, if you're running services on this box, you'll need to expose any other ports that you want to have access outside of your machine. Let's say you set up OpenVPN, you have to expose port 1194 on the machine it's running on.

Thanks for reading

Thanks for reading, hope I've helped in some way!