DEV Community: Łukasz Wolnik

Reclaiming free disk space from a private Docker repository

Łukasz Wolnik — Wed, 09 Oct 2024 20:34:47 +0000

My private Docker repository just hit 200 GB in size. Below is an instruction on how to reclaim most of that disk space back by deleting old registry's tags.

First mark tags for deletion based on year. There's a regex that will match all tags from years 2021-2023:

#!/bin/bash

# Docker repository details
REGISTRY="https://docker.foo.com"
REPOSITORY="foo-frontend"
USERNAME="username"
PASSWORD="password"

# Get all tags
tags_json=$(curl -sk -u "$USERNAME:$PASSWORD" "$REGISTRY/v2/$REPOSITORY/tags/list")
tags_to_delete=$(echo "$tags_json" | jq -r '.tags[]' | grep -E '^202[123][0-9]{4}$')

# Delete tags
echo "Deleting tags from 2021 and 2022:"
while IFS= read -r tag; do
    echo "Deleting tag: $tag"
    ./delete-tag.sh "$tag" "$REPOSITORY"
    echo "Tag $tag deletion process completed."
done <<< "$tags_to_delete"

echo "All deletion processes completed."

Now the script delete-tag.sh that gets digest for a given tag and marks it for deletion:

#!/bin/bash

if [ $# -eq 0 ]; then
    echo "Error: Please provide a TAG value as an argument."
    exit 1
fi

# Set your registry URL, repository name, and credentials
REGISTRY_URL="https://docker.foo.com"
REPO_NAME="$2"
TAG="$1"
USERNAME="username"
PASSWORD="password"

echo "Received TAG: $TAG"

echo "Fetching manifest for ${REPO_NAME}:${TAG}"
MANIFEST=$(curl -s -k -u "${USERNAME}:${PASSWORD}" \
  -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \
  "${REGISTRY_URL}/v2/${REPO_NAME}/manifests/${TAG}")

echo "Debug - Manifest output:"
#echo "$MANIFEST"

if [[ $MANIFEST == *"errors"* ]]; then
  echo "Error fetching manifest:"
  echo "$MANIFEST" | jq .
else
  echo "Manifest found. Extracting digest..."
  CURL_OUTPUT=$(curl -s -k -u "${USERNAME}:${PASSWORD}" \
    -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \
    -I "${REGISTRY_URL}/v2/${REPO_NAME}/manifests/${TAG}")

  echo "Debug - Curl output for digest:"
#  echo "$CURL_OUTPUT"

  DIGEST=$(echo "$CURL_OUTPUT" | grep -i Docker-Content-Digest | awk '{print $2}' | tr -d '\r\n')

  echo "Digest: ${DIGEST}"
  echo "Attempting to delete..."

  DELETE_RESPONSE=$(curl -s -i -k -u "${USERNAME}:${PASSWORD}" \
    -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \
    -X DELETE "${REGISTRY_URL}/v2/${REPO_NAME}/manifests/${DIGEST}")

  echo "Delete response:"
#  echo "$DELETE_RESPONSE"

  if [[ $DELETE_RESPONSE == *"202"* ]]; then
    echo "Tag deleted successfully."
  else
    echo "Failed to delete tag. Check the response for more details."
  fi
fi

echo "Script execution completed."

Lastly run docker ps to identify registry:2 container id and run:

docker exec <container_id> bin/registry garbage-collect /etc/docker/registry/config.yml

Run below to confirm there's now more free space on your disk:

df -h

Getting all repos from a Docker Registry

curl -k -u username:password -X GET https://docker.foo.com/v2/_catalog

Getting all tags for a given regsitry

curl -k -u username:password -X GET http://docker.foo.com/v2/bar/tags/list

Speed boost to Flipper debugger on Apple M1

Łukasz Wolnik — Thu, 30 Jun 2022 22:54:50 +0000

TL;DR Download the Apple silicon builds of Flipper here: https://github.com/chiragramani/FlipperReleases/releases

Flipper's official builds are painfully slow on Apple M1 silicon.

The app was so slow that it hindered my daily work. And the worst bit was that not only my CPU was struggling with the Flipper but it also made an iOS app I was working on painfully slow too!

Intel inside

It turned out that Flipper has been running through Rosetta emulator.

Universal build to the rescue

Thankfully there are good people out there that prepared a Universal App build for Flipper that runs natively on M1 silicon to a great success.

Here's the PR https://github.com/facebook/flipper/pull/3553 for a reference.

Installation

UPDATE: the para-official builds are now available at https://github.com/chiragramani/FlipperReleases/releases

Build M1 silicon-ready Flipper app by running below on your Terminal:



git clone https://github.com/aarongrider/flipper.git
cd flipper
git checkout m1-universal
cd desktop
yarn
yarn build --mac

Your Apple binary will be at ../dist/mac-universal/Flipper.app.

Run it by double clicking it in the Finder and you'll immediately see that Flipper and most importantly an app you're developing runs much faster now.

You can be happy about your M1 machine again.

Basic HTTP authentication in Traefik 2

Łukasz Wolnik — Mon, 31 May 2021 12:21:38 +0000

In order to secure your website with Basic HTTP authentication in Traefik do the following.

1. Download `apache2-utils`



sudo apt install apache2-utils

2. Generate password and copy its output



htpasswd -nB adam

The n option will display the hash in stdout and the B option will use more secure encryption.

devto password's hash is shown below:



adam:$2y$05$h9OxLeY20/5uiXjfPgdRxuFlrfqBf2QifYDgrwsR6rAEgX3/dpOGq

3. Replace $ with $$



adam:$$2y$$05$$h9OxLeY20/5uiXjfPgdRxuFlrfqBf2QifYDgrwsR6rAEgX3/dpOGq

4. Create a new middleware in Traefik for HTTP basic auth for your HTTPS entrypoint.



  labels:
    - "traefik.http.middlewares.yourservice-basicauth.basicauth.users=adam:$$2y$$05$$h9OxLeY20/5uiXjfPgdRxuFlrfqBf2QifYDgrwsR6rAEgX3/dpOGq"

Remember to use the escaped double $.

5. Create a new chain for your HTTPS connection.

You may have existing rules for your HTTPS route, e.g. compressing, etc.

So create a new chain where you'll combine your existing middlewars and the newly created yoursite-basicauth.

In your docker-compose.yml file:



  labels:
    - "traefik.http.middlewares.yourservice-https-chain.chain.middlewares=yourservice-basicauth,yourservice-other-middleware-remove-or-replace-with-yours"
    - "traefik.http.routers.yourservice-https.middlewares=yourservice-https-chain"

Restart your service and your website will now prompt browsers for username and password.

Cerebral Debugger in Next.js

Łukasz Wolnik — Mon, 29 Mar 2021 18:37:36 +0000

If you are getting below error:

ReferenceError: WebSocket is not defined
    at Devtools.createSocket

when trying to hook Cerebral Debugger up in Next.js try to refactor your bootstrap code from this:

const devTools =
  process.env.NODE_ENV === "production"
    ? null
    : {
        devtools: DevTools({
          host: "localhost:8585",
        }),
      };

to this:

let devTools = null;

if (process.env.NODE_ENV !== "production" && typeof window !== "undefined") {
  devTools = {
    devtools: DevTools({
      host: "localhost:8585",
    }),
  };
}

Now you're be able to use Cerebral Debugger whilst still using Next.js when calling below:

export default App(({ app }) => {
  app.on("initialized", () => {
    app.getSequence("initialize")();
  });
  return {
    state,
    sequences,
    providers,
  };
}, devTools);

MySQL replication over SSH

Łukasz Wolnik — Sun, 06 Dec 2020 15:02:41 +0000

Below is a quick and dirty solution for keeping a database synced with a master copy over SSH that you can set up in 30 minutes on existing databases.

It's a much easier setup (but unsecure) than a proper real-time MySQL replication one. Additionally you have full control over timing of syncing as you can use a cronjob to initiate the syncing process.

Below mysql and mysqldump commands are optmised to work with MySQL instances running inside a Docker container.

Master setup

Create a file named make-snapshots.sh with below content in your master server and then make it executable with chmod +x ./make-snapshots.sh.

#!/bin/bash
export MYSQL_PWD=PasswordMaster1
# Read the last snapshotted ID
source last_tablename_id
# Remove the old snapshot
rm dump_tablename.sql
# Write snapshot
mysqldump --protocol=TCP --ssl-mode=DISABLED -u username -t --where "id > $LAST_TABLENAME_ID" dbname tablename > dump_tablename.sql
# Remove the last snapshot
rm last_tablename_id
# Extract and write last ID
echo "export LAST_TABLENAME_ID=`grep -oP "\),\((\d+)" dump_tablename.sql | tac | head -1 | cut -b 4-`" > last_tablename_id

Before running your first snapshot create last_tablename_id with below:

export LAST_TABLENAME_ID=0

This will ensure that the first snapshot will contain all existing data, i.e. as the mysqldump SQL will equal to WHERE id > 0.

How does the snapshot bit work

Thanks to the --where parameter passed to the mysqldump which tells to backup all rows in a table meeting a criteria.

After each mysqldump backup our script is checking what was the last exported ID from a table which is then saved in last_tablename_id file. Each subsequent runs will read that number and only include rows that have not been exported yet.

The grep -oP "\),\((\d+) is searching for occurrences of ),(123 string inside the dump. Then tac is reversing the grep's output, head -1 is taking just the first line and cut is removing the starting ),( characters leaving the the number of a first column (which is assumed to be an id - a primary key).

Crontab

Add a daily snapshot creation at 2:00 AM by running crontab -e and adding below row into your crontab.

0 2 * * * cd ~/master && ./make-snapshots.sh

Slave setup

On your slave server, i.e. the one that will pull the newest data from the master database, create a file named pull-snapshots.sh and make it executable with chmod +x.

#!/bin/bash
export MYSQL_PWD=PasswordSlave1
# Remove last snapshot
rm dump_tablename.sql 
# Pull a snapshot from the master server
scp username@99.0.0.1:~/master/dump_tablename.sql .
mysql --protocol=TCP --ssl-mode=DISABLED -u username dbname < dump_tablename.sql

For scp to run without prompting for password you need to setup key-based authentication between the two servers.

Crontab

Add below line into your crontab so it runs 30 minutes later than the master's one, e.g. at 2:30 AM.

30 2 * * * cd ~/your/script/path && ./pull-snapshots.sh

Voilà! Your database is now synced with the master!

Further improvements:

don't rely on grep as if there wasn't any new entries in a database then the last_tablename_id will be missing the id, e.g. LAST_TABLENAME_ID=. It's better to rely on the mysql query just before running the mysqldump.
create a special user in the slave's DB that is only authorised for doing INSERT in your database in case your master server is compromised and someone has injected a harmful SQL into the dump_*.sql files. You don't want to run DROP DATABASE, etc.
check for errors in MySQL dump and do not overwrite last_tablename_id file if there was an error (to keep the last successful exported ID)
if you need anything more then just use a proper replication setup

Install rtl88x2bu WiFi adapter drivers on Ubuntu 20.10

Łukasz Wolnik — Wed, 02 Dec 2020 00:16:46 +0000

Ubuntu 20.10 based distros do not support the rtl88x2bu Wifi USB adapter out of the box.

Internet connection via Bluetooth

There's a catch 22 when installing a rtl88x2bu WiFi adapter driver. First you need to download it from the Internet but your Linux is not ready to connect to a WiFi.

Instead of using another PC to download a driver and copy to a USB flash drive simply use Bluetooth tethering on your mobile:

buy a Bluetooth dongle for your desktop PC
enable Bluetooth tethering and turn on a hotspot on your mobile
pair and connect to your mobile phone from your PC

Installing rtl88x2bu drivers

If you are looking for rtl88x2bu drivers for Linux you have probably already seen the github.com/cilynx/rtl88x2bu repo. Unfortunately its build process did not work for me on my Ubuntu 20.10-based system. The source files simply did not compile (both 5.3.1 and 5.8.1).

Error! Bad return status for module build on kernel
Consult /var/lib/dkms/rtl88x2bu/5.3.1/build/make.log for more information.

./rtl88x2bu-5.3.1/include/rtw_security.h:252:8: error: redefinition of ‘struct sha256_state’
  252 | struct sha256_state {

I have found another solution in the following repo instead: github.com/morrownr/88x2bu.

Below are instructions for Ubuntu taken from that repo's README:

sudo apt-get update
git clone https://github.com/morrownr/88x2bu.git
cd 88x2bu
sudo ./install-driver.sh
sudo reboot

node-sass considered harmful

Łukasz Wolnik — Wed, 04 Nov 2020 17:52:00 +0000

... to your developer and Continuous Integration experience due to its painfully slow installation process.

The node-sass during its installation npm install node-sass will:

Trigger the node-gyp if it won't be able to find binaries for your operating system.
Use Python 2.7 to run some scripts.
Run a painfully slow postinstall process (adds up to 1 minute to your CI build time).

So if your team is using various operating systems (Windows/Linux/MacOS) or using a Continuous Integration system just replace it with a much better alternative sass.

sass is just a JavaScript package and has zero external dependencies.

Create React App

I guess the reason why node-sass is still so popular is due to the Create React App support for SASS. But you don't have to eject a CRA if you don't want to compromise your installation process or a build time. Simply use below command inside your app.

TL;DR

Transparently replace your node-sass with sass and enjoy faster builds and quicker installs:

rm -rf node_modules
npm install node-sass@npm:sass -D --legacy-peer-deps

Simplify your monorepo with npm 7 workspaces

Łukasz Wolnik — Sat, 31 Oct 2020 00:56:39 +0000

This month npm has released a major version of their package manager npm 7. It shipped with support for workspaces.

Why is it big news? Because npm is the only package manager that comes bundled with every NodeJS. To use yarn or pnpm you have to take an additional step and install them first.

Read on and you'll find out how to use npm 7 workspaces in a real-world scenario and learn that using workspaces the npm's way is very different to yarn's.

Monorepo use cases

A monorepo is a term describing a single git repository that contains many projects.

The most common reason to set up a monorepo is to streamline work within a dev team that maintains multiple apps that are using a shared piece of code, for example a common User Interface library.

Imagine a team that develops two React apps that shares some common UI elements like inputs, selectors, accordions, etc. It would be nice to extract that UI in form of React components and prepare building blocks that are ready to use for all members of the team.

Apart from that it's just more convenient to have all your source files opened in a single IDE instance. You can jump from project to project without switching windows on your desktop.

I just want that nice button in my app too

Let's say I want to build two independent React apps called app1 and app2 that will use a common component from a common UI library called ui. And I want both apps to hot reload whenever I edit a file inside the UI library.

By independent I mean that app1 doesn't know anything about app2 and vice-versa.

Below is a setup that is compatible with npm 7 workspaces.

Defining workspaces in npm 7

I am prefixing all packages' names with @xyz so they will be distinctive from the official npm registry. Just change it to yours.

This is the most crucial part of the whole setup. Insert below inside your root folder's package.json to set up a monorepo.

{
    "name": "@xyz/monorepo",
    "private": true,
    "version": "1.0.0",
    "workspaces": [
        "./common/*"
    ]
}

The new "workspaces" property lets npm know that I want to track any packages inside ./common folder and automatically symlink them in the root's node_modules when I run npm install.

From now on whenever our React apps will use import Foo from "@xyz/ui" the NodeJS will find it in ./node_modules/common/@xyz/ui that points to ./common/ui folder that contains our library. Perfect! No need for npm link anymore with the workspaces.

Without workspaces the React app would complain that it cannot find a module named @xyz/ui and would start looking for it in the npm official registry.

Take the first step

To test our setup let's share a text from the ui library and import that string into our React app.

Create the common UI library's package.json:

{
    "name": "@xyz/ui",
    "version": "1.0.0",
    "private": true,
    "main": "index.js"
}

and index.js file that will export a string:

const version = "This comes from UI! 1.0.0"

export default version;

Time to import that string into our apps.

To quickly create React apps inside our monorepo. Just use the newly released Create React App 4:

mkdir apps
cd apps
npx create-react-app app1
npx create-react-app app2

Now you'd think that we need to add our ui library to the app. In yarn it would look like below:

yarn workspace app1 add @xyz/ui

But with npm we don't need to add any dependency at all.

Just go to your App.js file in both app1 and app2 apps and add the below code to display a string from our UI library:

...
import testString from "@xyz/ui"; 
...
    <span>{testString}</span>
...

To test it use the below commands:

# create a symlink to the @xyz/ui in the root folder
npm install
# go to the app's folder
cd apps/app1
# For CRA 4 you may need to add SKIP_PREFLIGHT_CHECK=true to .env file
# And use the --legacy-peer-deps flag as many packages hasn't been updated yet to officially support React 17
npm install --legacy-peer-deps
npm run start

and from another terminal window:

cd apps/app2
npm install
npm run start

You'll see the This comes from UI! 1.0.0 text rendered inside both of your React apps!

Export React JSX components

Now if you'd try to export a JSX component the React apps will complain that they cannot parse the JSX. You need to transpile JSX code from the common UI package first.

You can use a basic Webpack 5 setup:

common/ui/package.json

{
    "name": "@xyz/ui",
    "version": "0.2.0",
    "private": true,
    "module": "build/ui.bundle.min.js", # Changed main to module
    "scripts": {
        "build": "webpack --config webpack.prod.js",
        "build-watch": "webpack --config webpack.prod.js --watch",
    },
    ... # webpack 5 dependencies
}

common/ui/babel.config.js

module.exports = {
  presets: [
    [
      "@babel/preset-react",
      {
        targets: {
          node: "current",
        },
      },
    ],
  ],
};

common/ui/webpack.prod.js

const path = require("path");

module.exports = {
  entry: {
    index: { import: "./src/index.js" }
  },
  module: {
    rules: [
      {
        test: /\.jsx?$/,
        exclude: /node_modules/,
        loader: "babel-loader",
      },
    ],
  },
  output: {
    filename: "ui.bundle.min.js",
    path: path.resolve(__dirname, "build"),
    // Below two important lines!
    library: 'xyzUI',
    libraryTarget: 'umd'
  },
};

Our simple component:

common/ui/src/index.js

import React from "react";

const UIExample = ({ text = "" }) => {
  return (
    <div>
      <h1>Shared UI library {text}</h1>
    </div>
  );
};

export default UIExample;

Import the UIExample component inside your React app using below:

apps/app1/src/App.js

...
import UIExample from "@xyz/ui";
...
    <div>
        <UIExample text="from app1" />
    </div>
...

Make sure the UI library is transpiled on every code change:

cd common/ui
npm run build-watch

Run the app1 in a separate terminal window and take notice that whenever you edit the UI component the webpack dev server will automatically reload it with the newest version thanks to the webpack watch running in the background.

cd apps/app1
npm run start

Demo

Below I'm editing the common UI component UIElement and upon saving both React apps are automatically refreshed with the updated component:

Summary

With the newest npm 7 and its support of workspaces it is now possible to have a monorepo without a need of any external tools like @react-workspaces or nx.

Just remember that npm has a different philosophy than yarn. For example you cannot run a script inside a workspace from the monorepo's root folder.

Recognize also that @xyz/app1 and @xyz/app2 weren't defined in the monorepo's package.json workspaces property. Only the modules that will be exported need to be there (@xyz/ui).

The npm 7 workspaces are mainly providing discovery for the modules. I wish it had been emphasised in release notes and that the npm's help examples were a bit more complex. I hope this article fills this gap for the time being.

Bonus

Check out my gif-css-animation-monorepo repository that shows how I made the animation for this article using an HTML page.

Self-hosted Continuous Delivery that doesn't cost a fortune 💰 - Part 2

Łukasz Wolnik — Sat, 25 Jan 2020 01:05:49 +0000

Overview

In the first part we set up Docker and installed Traefik on our VPS (Virtual Private Server).

What's left in this final part of setting up your own Continuous Delivery pipeline is:

setting up your web app
installing Drone
connecting GitHub to Drone
setting up an auto-update over SSH

Setting up a new web app in Traefik

I think it was worth to install Traefik just for being able to very quickly deploy any Docker image and link a (sub)domain to it with Let's Encrypt's SSL.

It allows you to add a new NodeJS/React/Wordpress web apps or MySQL/PostgreSQL instances in a matter of minutes.

Because once Traefik is set up adding a new piece of software boils down to setting up a new A type in a DNS zone for your domain that will point to your server's IP address and adding another entry in docker-compose.yml file in sites directory. It's quite close to an expierience with using Netlify.

To start, create a new directory inside your home directory called sites:

cd ~
mkdir sites
cd sites

Create a docker-compose.yml file and paste below into it:

version: '3'

services:
  fooapp: # **** EDIT HERE ****
    image: docker.wolnik.co.uk/fooapp # **** EDIT HERE ****
    labels:
      - traefik.backend=fooapp # **** EDIT HERE ****
      - traefik.frontend.rule=Host:fooapp.com # **** EDIT HERE ****
      - traefik.docker.network=web
      - traefik.port=80
    networks:
      - web
  mysql:
    image: mysql:8.0.3
    restart: always
    ports:
      - 3306:3306
    environment:
      MYSQL_ROOT_PASSWORD: apasswordheretochange # **** EDIT HERE ****
      MYSQL_DATABASE: yourdbname # **** EDIT HERE ****
    networks:
      - web
  adminer:
    image: dockette/adminer
    labels:
      - traefik.frontend.rule=Host:mysql.foo.com # **** EDIT HERE ****
      - traefik.docker.network=web
      - traefik.port=80
    restart: always
    ports:
      - 8080:80 # Here I'm mapping adminer's default 8080 port to 80
    networks:
      - web

networks:
  web:
    external: true

There's quite a lot going on above but actually the only new part that deals with Traefik is adding labels so that Traefik can assign a domain for your container and route to the right port.

Run your containers with (you must be in ~/sites directory):

docker-compose up -d

The -d flag means that docker-compose will run in background so you can exit your SSH session and your containers will still run.

Installing Drone

Setting up GitHub OAuth App

Go to GitHub. Open Settings → Developer options → OAuth Apps.

Then fill out the fields:

Homepage URL: https://drone.yourdomain.com
Authorization callback URL: https://drone.yourdomain.com/login

Once created copy the GitHub client id and secret:

Drone Server

Create a subfolder in your home folder.

cd ~ && mkdir drone && cd $_

Create a docker-compose.yml file and paste into it below:

version: "3"

networks:
  web:
    external: true
  internal:
    external: false

services:
  drone:
    image: drone/drone:1
    labels:
      - traefik.backend=drone
      - traefik.frontend.rule=Host:drone.yourdomain.com # **** EDIT HERE ****
      - traefik.docker.network=web
      - traefik.port=80
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /var/lib/drone:/home/yourusername/traefik/drone/data # **** EDIT HERE ****
    environment:
      - DRONE_GITHUB_SERVER=https://github.com
      - DRONE_GITHUB_CLIENT_ID=********** # **** EDIT HERE ****
      - DRONE_GITHUB_CLIENT_SECRET=**************************** # **** EDIT HERE ****
      - DRONE_RUNNER_CAPACITY=2
      - DRONE_RPC_SECRET=THIS_RANDOIM_STRING_NEED_TO_BE_THE_SAME # **** EDIT HERE ****
      - DRONE_SERVER_HOST=drone.yourdomain.com # **** EDIT HERE ****
      - DRONE_SERVER_PROTO=https
      - DRONE_TLS_AUTOCERT=true
      - DRONE_AGENTS_ENABLED=true
    networks:
      - internal
      - web
  runner:
    image: drone/drone-runner-docker:1
    restart: always
    ports:
      - "3000:3000"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      - DRONE_RPC_PROTO=https
      - DRONE_RPC_HOST=drone.yourdomain.com # **** EDIT HERE ****
      - DRONE_RPC_SECRET=THIS_RANDOM_STRING_NEED_TO_BE_THE_SAME # **** EDIT HERE ****
      - DRONE_RUNNER_CAPACITY=2
      - DRONE_RUNNER_NAME=drone.yourdomain.com THIS_STRING_NEED_TO_BE_THE_SAME # **** EDIT HERE ****
    networks:
      - internal
      - web

Go to drone.yourdomain.com URL and you should see Drone UI running. Asking you to login via your GitHub.

Login to Docker on VPS

Make sure your VPS can connect to your Docker Hub (or your own Docker repo) by running:

docker login -u user

or if you have your own Docker repo at docker.yourdomain.com:

docker login -u user docker.yourdomain.com

You can confirm that you are logged in if there's an auth section in ~/.docker/config.json file. And of course if you are not asked for authentication when pulling image from Docker.

Setting up Continuous Delivery in Drone

First create a script on your VPS that pulls Docker images and restarts docker-compose.yml in ~/sites directory. Name it update-all.sh and paste below into it:

docker pull docker.wolnik.co.uk/fooapp # **** EDIT **** Docker image name
docker pull docker.yourdomain.com/fooapp # **** EDIT/DELETE **** Another web app
docker-compose up -d

Insert there all images that you are intending to update over Drone. So that docker-compose will use the latest Docker images of your web apps.

Drone SSH user

Above update-all.sh script must be invoked by Drone runner after completing building and pushing new Docker images once they are build.

The best way is to create a separate user for this task on your VPS:

adduser drone
sudo usermod -aG docker drone
chown drone: ~/sites/update-all.sh
chmod +x ~/sites/update-all.sh

.drone.yml

In order for Drone to know how to build your web app add .drone.yml file inside root of your Git repo, i.e.

Paste below into .drone.yml file inside your Git repo, add and commit to your master branch.

pipeline:
  build:
    image: plugins/docker
    registry: docker.wolnik.co.uk
    repo: docker.wolnik.co.uk/fooapp
    username:
      from_secret: docker_username
    password:
      from_secret: docker_password
    tags:
      - 1.0.0
      - 1.0
      - latest
  ssh:
    image: appleboy/drone-ssh
    host: fooapp.com
    username:
      from_secret: ssh_username
    password:
      from_secret: ssh_password
    port: 22
    command_timeout: 2m
    script:
      - cd /home/yourdominaname/sites/ && ./update-all.sh

In above file we set up two steps that will build and deploy our web app.

First step - build - will pull the latest Git repo from master and build Docker image based on its Dockerfile. Then it will push the image to your Docker registry.

Second step - ssh - will connect to your server over SSH, go to your sites directory and run update-all.sh script which in turn will pull your Docker images and restart docker-compose so that your web app is using the latest - just built - Docker image.

Important if you are using your own Docker repo!

Remember that both registry URL and repo must match, i.e. repo can't be just fooapp if registry has a domain docker.yourdomain.com. It needs to be then docker.yourdomain.com/fooapp in order to work.

Setting up secrets in Drone

A .drone.yml contains four secret strings that forms two credentials. One for Docker repo (DockerHub or your own Docker repo) and SSH access.

Go to your Drone UI and set up 4 secrets that will replace the placeholders in your Git repo's .drone.yml file.

Final words

Please be patient while fine tuning your .drone.yml and Drone secrets. Sometimes it's tricky to get the Drone to pick up the lastest GitHub's commit.

But it's all worth it once you can see a running gear like below for your own repo:

P.S. Of course this is just a start. Now you can extend your Drone's pipeline with additional step like running tests before deploying anything on a live site.

Self-hosted Continuous Delivery that doesn't cost a fortune 💰 - Part 1

Łukasz Wolnik — Fri, 24 Jan 2020 21:54:05 +0000

Overview

In this series I want to demonstrate how to set up an open-sourced, self-managed continuous delivery solution that can be run on a private server for free thanks to Drone and Traefik.

The goal is to update a web app on a live website by just running:

git push origin master

on your dev machine.

That's it! After a couple of minutes a website should be automatically updated with changes that have just been pushed to a Git repo.

Below image depicts this process from pushing a commit to serving an HTTPS website.

Requirements

You need to have a Linux server to host and run the required software. You can buy one for example from OVH.

Tech stack

The presented continuous delivery's pipeline contains:

1. GitHub

A hosted Git repo with hooks that notifies a build server about a new commit.

2. Drone

A build server that will listen to new commits, build images and push them to a Docker repo.

3. Traefik

A load balancer and a reverse proxy, that runs a Docker's image of a web app and routes a web domain foo.bar.com to a container that runs a Docker image build with Drone.

Installing Docker

To start we need a Docker running on a server.

sudo apt-get update
sudo apt install apt-transport-https ca-certificates curl software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable"
sudo apt update
apt-cache policy docker-ce
sudo apt install docker-ce
sudo systemctl status docker

Press Ctrl-C to quit.

Below will allow your non-root user to access Docker.

sudo usermod -aG docker yourusername

Installing docker-compose

sudo curl -L https://github.com/docker/compose/releases/download/1.21.2/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose

Installing Traefik

For more detailed installation refer to the article I based this one on [1]

Go to your home directory and create a folder:

cd ~
mkdir traefik
cd traefik

Set up a password for your Traefik dashboard using apache2-utils.

sudo apt-get install apache2-utils
htpasswd -Bc password.txt yourusername

Open password.txt and copy your password hash from there. You will need it in the next step.

Create a traefik.toml file and paste below content into it:

vim traefik.toml

defaultEntryPoints = ["http", "https"]

[entryPoints]
  [entryPoints.dashboard]
    address = ":8080"
    [entryPoints.dashboard.auth]
      [entryPoints.dashboard.auth.basic]
        users = ["$output_of_password.txt_file_created_above"] # ***** EDIT HERE *****
  [entryPoints.http]
    address = ":80"
      [entryPoints.http.redirect]
        entryPoint = "https"
  [entryPoints.https]
    address = ":443"
      [entryPoints.https.tls]

[api]
entrypoint="dashboard"

[acme]
email = "youremail@email.com" # ***** EDIT HERE *****
storage = "acme.json"
entryPoint = "https"
onHostRule = true
  [acme.httpChallenge]
  entryPoint = "http"

[docker]
domain = "docker.yourdomain.com" # ***** EDIT HERE *****
watch = true
network = "web"

sudo docker network create web
touch acme.json
chmod 600 acme.json

Run Traefik:

vim docker-compose.yml

And paste below:

version: '3'

services:
  traefik:
    image: traefik:1.7.6-alpine
    restart: always
    ports:
      - "80:80"
      - "443:443"
    labels:
      - traefik.frontend.rule=Host:traefik.foo.bar.com # ***** EDIT HERE *****
      - traefik.port=8080
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./traefik.toml:/traefik.toml
      - ./acme.json:/acme.json
    networks:
      - web

networks:
  web:
    external: true

Run Traefik

docker-compose up -d

Traefik Dashboard

To confirm that your Traefik installation was successful you should be able to access traefik.foo.bar.com URL that you specified in above docker-compose.yml file.

You should see a website like below:

Conclusion

Well done! You've got half of your continuous delivery pipeline set up. In the next article I will show how to install Drone build server and how to connect the trio: GitHub, Drone and Traefik.

[1] https://www.digitalocean.com/community/tutorials/how-to-use-traefik-as-a-reverse-proxy-for-docker-containers-on-ubuntu-18-04

Using Raspberry Pi 4 with a power adapter only

Łukasz Wolnik — Thu, 27 Jun 2019 08:58:07 +0000

Story

I ordered my Raspberry Pi 4 one hour after its world premiere on Monday morning from a train to London. I rushed through the https://thepihut.com/ checkout process to ensure that I got my board before they run out of stock and, more importantly, while I still got a 3G signal on my mobile.

I admit I got prompted by the website to buy an HDMI cable which I kindly refused to. After all I also have a Raspberry Pi Zero W which uses a Mini-HDMI port so I was sure I was fully covered and I didn't need yet another HDMI cable.

How wrong I was! The next evening I received the board I noticed my Mini-HDMI is huge compared to the Micro-HDMI connectors that Raspberry Pi 4 use now.

I quickly ordered a Micro-HDMI cable with a next-day delivery but I still wanted to use my newly arrived board now. No, I mean it. NOW!

Headless mode

Luckily I knew there was a way to run the previous version of Raspberry Pi in a headless mode, i.e. without a display, keyboard and mouse. So I thought that maybe it would be possible to install the operating system without connecting it to a display too.

And it was indeed. Here is how I did it. The key part was to skip the NOOBS installer entirely (even with its silentinstall option which didn't work for me) and clone the Raspbian image directly onto the SD card.

All you need to start enjoying your new board with only a power adapter at your disposal is:

a Unix-based computer with an SD card reader
an SD card

Download Raspbian

Go to https://www.raspberrypi.org/downloads/raspbian/ and start downloading Raspbian Buster Lite zip file.

Prepare your SD card

Insert your SD card to your card reader and run sudo fdisk -l to check its /dev/sdX path.

In my case the SD card reader was at /dev/sde.

Now run an app called gparted and select your device from a drop-down in the top-right.

If your SD card is a brand new one and has not been formatted yet you might need to create a partition table. Simply click Device → Create Partition Table… from the main menu.

You can now create a new partition. Just create a single one with the maximum capacity and format it using ext4 filesystem.

Install Raspbian

Extract an 2019-06-20-raspbian-buster-lite.img file from the downloaded Raspbian Buster Lite zip package.

Write the image onto the SD card using below command:

sudo dd if=2019-06-20-raspbian-buster-lite.img of=/dev/sde bs=4M conv=fsync

Enable SSH and setup WiFi

The above dd utility has created two partitions on your SD card: boot and roofs.

Go to the boot “directory in your file browser and create an empty file named ssh. This will enable the SSH protocol for you Raspberry.

Create another file in the same boot path named wpa_supplicant.conf and paste below content to it:

country=GB
ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1

network={
    ssid="Your network name, e.g. BTHub1-123QWE"
    psk="Y0ur_P455w0rd"
    key_mgmt=WPA-PSK
}

Replace country, ssid and psk values to match your WiFi settings.

Grande finale

Your SD card is now ready. Insert it to your Raspberry Pi SD card slot at the bottom of the board.

Connect a power adapter.

Scan your local network (in my case 192.168.1.*) to see what IP address has been assigned to the Raspberry Pi.

sudo nmap -sS -p 22 192.168.1.0/24 | grep "Nmap scan"

You can power off the board to see which of the IP addresses will disappear from the list to identify the Pi.

Connect to the board:

ssh pi@192.168.1.101

The password is raspberry.

You're back in the game! Change your password with passwd and enjoy!

Bonus! Change WiFi after installation

Let's say you have changed your WiFi password but your Raspberry is no longer in a pre-installation mode. Don't worry! It's much easier to change WiFi password for a Raspberry that has an operating system already installed.

Simply eject an SD card from your Raspberry and read it on your PC's card reader. Then locate the /etc/wpa_supplicant/wpa_supplicant.conf file and replace your WiFi settings there. Remember to sudo yourself in order to access and overwrite this file.

DEV Community: Łukasz Wolnik

Reclaiming free disk space from a private Docker repository

Getting all repos from a Docker Registry

Getting all tags for a given regsitry

Speed boost to Flipper debugger on Apple M1

Intel inside

Universal build to the rescue

Installation

Basic HTTP authentication in Traefik 2

1. Download apache2-utils

2. Generate password and copy its output

3. Replace $ with $$

4. Create a new middleware in Traefik for HTTP basic auth for your HTTPS entrypoint.

5. Create a new chain for your HTTPS connection.

Cerebral Debugger in Next.js

MySQL replication over SSH

Master setup

How does the snapshot bit work

Crontab

Slave setup

Crontab

Further improvements:

Install rtl88x2bu WiFi adapter drivers on Ubuntu 20.10

Internet connection via Bluetooth

Installing rtl88x2bu drivers

node-sass considered harmful

Create React App

TL;DR

Simplify your monorepo with npm 7 workspaces

Monorepo use cases

I just want that nice button in my app too

Defining workspaces in npm 7

Take the first step

Export React JSX components

Demo

Summary

Bonus

Self-hosted Continuous Delivery that doesn't cost a fortune 💰 - Part 2

Overview

Setting up a new web app in Traefik

Installing Drone

Setting up GitHub OAuth App

Drone Server

Login to Docker on VPS

Setting up Continuous Delivery in Drone

Drone SSH user

.drone.yml

Setting up secrets in Drone

Final words

Self-hosted Continuous Delivery that doesn't cost a fortune 💰 - Part 1

Overview

Requirements

Tech stack

1. GitHub

2. Drone

3. Traefik

Installing Docker

Installing docker-compose

Installing Traefik

Traefik Dashboard

Conclusion

Using Raspberry Pi 4 with a power adapter only

Story

Headless mode

Download Raspbian

Prepare your SD card

Install Raspbian

Enable SSH and setup WiFi

Grande finale

Bonus! Change WiFi after installation

1. Download `apache2-utils`