DEV Community: Mayeu

What's a good way of sending a big directory structure to a server using Ansible?

Mayeu — Wed, 19 Jun 2019 00:00:00 +0000

Ansible has a copy module that works really well for moving a small set of files around. But it can rapidly slow down your playbook since it does not scale well with hundreds of files. So how can you move a big set of files and folder without losing too much time? Well, you use another module namedsynchronize for this.

synchronize uses a tool named rsync under the hood, so you first have to ensure it is installed on your machine, and on the targeted host (it comes with most Unixes). If you are using macOS or Ubuntu, it should be already there. Otherwise start by installing it on your computer.

To be sure it is installed on your targeted server you should add a task to install it before you use the synchronize module in your playbook. Here it is an example for Ubuntu or Debian hosts:

- name: Install rsync running synchronize later
  apt:
    name: rsync
    state: present

By default synchronize just require a src and a dest, but like a lot of Ansible modules there are many more options for finer control.

Here is a minimal example of using the module:

- name: Synchronize a local folder to my remote server
  synchronize:
    src: path/to/my/local/folder
    dest: /absolute/path/to/my/destination/folder

And with that you can copy a gazillion files (at least) without worrying that your copy will be slow!

Happy syncing 👋

This post was originally published on my website: mayeu.me.

How to Read the Content of an External File With Ansible

Mayeu — Thu, 13 Jun 2019 00:00:00 +0000

Do you want to load up JSON data from a file directly into your Ansible playbook, tasks or roles? Did you start by trying to read it via a shell or command module and wondered if there was a better way? Well, there is, in the form or lookup plugins. Read on to see how to use it.

Ansible comes with a whole bunch of Lookup plugins that allow loading stuff from the outside your play. The one we are interested in today is thefile one. So, how do I load the content of my file with this?

Simply enough:

vars:
  file_contents: "{{lookup('file', 'path/to/file.json')}}"

And bam, you have the content in the file_contents variable!

As you see, we are calling the lookup() function, to which we are passing the plugin we want to use has the first argument (in that case file), and then the path to our file. And just with that we loaded our file content!

Note that the file must be on your Ansible controller (i.e.: the computer where you are running the ansible command).

You can directly use this in any module without the need for an intermediate variable:

- name: Read the content of a file instead of copying it directly
  copy:
    content: "{{lookup('file', 'path/to/file.json')}}"
    dest: /etc/config/file.json

And voilà 🙂

This post was originally published on my website: mayeu.me.

How to Trigger Any Action When a File or Folder Changes on Macos on the Cheap

Mayeu — Tue, 14 May 2019 05:16:08 +0000

Did you ever want to trigger an action when a file changed? For example, automatically move a file that has just been downloaded? The action could be a script, app, anything really. Here I will describe how to do exactly that on macOS with only the tools provided by the default system.

Side note: I know there are GUI app that does that in a much simpler manner; here I just want to highlight how to do that manually with what macOS is providing to us.

But before we start, I would like to stress the fact that this method may not be ideal for anything critical to take place. Quoting the docs:

⚠️ IMPORTANT: Use of this key is highly discouraged, as filesystem event monitoring is highly race-prone, and it is entirely possible for modifications to be missed. When modifications are caught, there is no guarantee that the file will be in a consistent state when the job is launched.

This basically means that:

if there is too much change happening at once, some may be missed and your action won’t trigger
You can’t be sure that your action will find the file in the exact state that triggered the script

But I believe that for day-to-day workflows, which are not computationally demanding, those caveats are not really a problem.

How to watch for changes?

For this we will be using launchd which is the service manager running on macOS. It has two main tasks:

Booting the system
Managing daemon and agents

In launchd lingo, an agent is a service run on a per user basis, and a daemon is a system service.

launchd has multiple levels of configurations, some are system-wide and privileged, but you can also use it to run unprivileged user tasks. There are five folders in which you can find those definitions:

~/Library/LaunchAgents: per user agents provided by the user.
/Library/LaunchAgents: per user agents provided by the administrator.
/Library/LaunchDaemons: systemwide daemons provided by the administrator.
/System/Library/LaunchAgents: per user agents provided by Apple.
/System/Library/LaunchDaemons: systemwide daemons provided by Apple.

In our case, we will be using a small launchd agent that is going to be stored in ~/Library/LaunchAgents. We will see later on how we can load or unload our agent to active or deactivate it.

The test Setup

Let’s get some action going on, hop in your terminal and go to a clean folder for our test. From here I will assume that the folder you are using is ~/file-watching-test, if you are using another one don’t forget to adapt your paths in the rest of the article!

$ mkdir ~/file-watching-test
$ cd ~/file-watching-test

In this folder we are going to create an empty watched file that will be monitored by launchd:

$ touch ~/file-watching-test/watched

We are also going to create the script that will be called when the watched file changes, open the ~/file-watching-test/script.sh file in your favourite editor and add:

#!/bin/sh
echo "$(date): 🐈 I has be summoned" >> ~/file-watching-test/result

Let’s break out what is happening here:

echo "$(date): 🐈 I has be summoned":
- this echo command will get the return of the date command executed via $()
- print <current date>: 🐈 I has be summoned to the standard output.

You can try it directly in your shell to see what is happening. Then:

>>: this is a redirection operator. This one takes whatever has been printed on the standard output (so not the errors, if any) and append it to the file it points toward. With this we can log all the execution of our script.
And finally, ~/file-watching-test/result is our result file that will receive the output of our echo command.

Now, make sure the script can be executed with chmod +x ~/file-watching-test/script.sh.

To sum up, we have the following hierarchy:

$ tree file-watching-test
file-watching-test
├── script.sh      # Our script
├── watched        # The file we are watching

Creating a service file for `launchd`

We want our test service to:

Watch the ~/file-watching-test/watched file
When a change happens, we want to execute ~/file-watching-test/script.sh

Before I show you the service file, be warned that launchd use XML for service declaration, so brace yourself.

Here is our me.mayeu.watchtest.plist file:

<?xml version=“1.0” encoding=“UTF-8”?>
<!DOCTYPE plist PUBLIC “-//Apple//DTD PLIST 1.0//EN” “http://www.apple.com/DTDs/PropertyList-1.0.dtd”>
<plist version=“1.0”>
<dict>
        <key>Label</key>
        <string>me.mayeu.watchtest</string>
        <key>ProgramArguments</key>
        <array>
                <string>/Users/m/file-watching-test/script.sh</string>
        </array>
        <key>WatchPaths</key>
        <array>
                <string>/Users/m/file-watching-test/watched</string>
        </array>
</dict>
</plist>

OK let’s break that down, and be sure to adapt the various paths to the one you are using! If you are unsure, use the pwd command in your shell when you are in the file-watching-test folder to print absolute path of the folder.

The very first part is not really interesting since it is a declaration of the format and the document type definition (DTD):

<?xml version=“1.0” encoding=“UTF-8”?>
<!DOCTYPE plist PUBLIC “-//Apple//DTD PLIST 1.0//EN” “http://www.apple.com/DTDs/PropertyList-1.0.dtd”>

Then, we state that the file is using the plist format version 1.0. A plist is the named used for Property List files that are use to store configuration, services, serialised objects and more in macOS. We also declare that this plist contains a dictionary (dict):

<plist version=“1.0”>
<dict>
...
</dict>
</plist>

Inside those key we are going to declare our job. First, we have to provide a unique label to identify the job. Here I use me.mayeu.watchtest:

<key>Label</key>
<string>me.mayeu.watchtest</string>

By convention (not only but we don’t need to dive to deep in this), one uses the reversed domain following by some name representing the app. This ID is also used to identify the application and other related resources.

For example, Evernote uses com.evernote.Evernote as an application ID.

Then, we declare the program we are going to run using the ProgramArguments key:

<key>ProgramArguments</key>
<array>
    <string>/Users/m/file-watching-test/script.sh</string>
</array>

This key take an array listing all the arguments. In our case it is just the path to the script, but if we wanted to execute git commit -m “My commit message” we would do this like:

<key>ProgramArguments</key>
<array>
    <string>/usr/bin/git</string>
    <string>commit</string>
    <string>-m</string>
    <string>My commit message</string>
</array>

And finally, we declare when to run the program, in that case we use the WatchPaths key that take an array of paths to watch:

 <key>WatchPaths</key>
 <array>
    <string>/Users/m/file-watching-test/watched</string>
 </array>

Done! We have our new service.

If you are curious of what launchd can do, you can find all the valid key detailed in the launchd.plist manpage: man 5 launchd.plist. launchd should also be used for time-based jobs instead of using cron. See the Apple documentation for more about that.

Trigger it!

Almost there! We have all the pieces we need so let’s copy our service file to the right place:

$ cp me.mayeu.watchtest ~/Library/LaunchAgents/

We will now tell launchd to load it:

$ launchctl load ~/Library/LaunchAgents/me.mayeu.watchtest

Now that launchd has loaded our service, we can change our watched file by adding content in it:

$ echo ‘trigger the watcher’ > test/watched`

And this should have triggered the script; thus we will have some content in the result file:

$ cat ~/file-watching-test/result
Thu Apr 11 12:39:41 +07 2019: 🐈 I has be summoned

🎉

If we trigger it again, we should see a new line:

$ echo ‘trigger the watcher’ > test/watched`
$ cat ~/file-watching-test/result
Thu Apr 11 12:39:41 +07 2019: 🐈 I has be summoned
Thu Apr 11 12:42:55 +07 2019: 🐈 I has be summoned

We are done with this test! To clean behind ourselves we are going to:

unload the service
delete the service plist
delete our test folder

$ launchctl unload ~/Library/LaunchAgents/me.mayeu.watchtest
$ rm -rf ~/Library/LaunchAgents/me.mayeu.watchtest
$ rm -rf ~/file-watching-test

This is it! You now have some basis on using launchd, and maybe you learned one or two things about your shell and shell scripts at the same times 🙂

One last point, you can also watch folders using this method and thus act on any changes that happen under those folders, may it be deleted files, new files, new folders, etc.

Have fun! And if you build something cool with this, do not hesitate to poke me about this on Twitter or by email: m [-at-] mayeu [-dot-] me.

This post was originally published on mayeu.me.

Fix Mistakes in Shell Commands Easily With `fc`

Mayeu — Mon, 13 May 2019 00:00:00 +0000

From time to time, it can be quite cumbersome to fix a mistyped command in your shell. What if you could fix the command you just typed in your favourite editor? Well, that’s exactly what fc command is all about, and it is standard to all Unix system (this includes GNU/Linux & macOS). Let see how that work.

From Wikipedia:

fc is a standard program on Unix that lists, edits and reexecutes commands previously entered to an interactive shell.

Using fc looks something like this:

You type a non-working/wrong command
You type fc
The last typed command (before fc) is open in your editor
You fix the command, save, and quit
When you exit the editor, the edited command is executed

Here is an interactive example in which I use the wrong path and fix it via fc (click on the image to see it in action):

Of course this example is a little bit contrived because you should use autocompletion to navigate folders and not type your path entirely.

Checking my history, I have found this big command (which check if Homebrew’s ZSH path is part of the /etc/shells file, and if not, add it):

export zshpath="$(brew --prefix)/bin/zsh"; \
grep -q "^${zshpath}" /ec/shells || \
sudo -E sh -c "echo '$zshpath' \>\> /etc/shells"

It is not obvious, but there is a mistake right in the middle: /ec/shellsinstead of /etc/shells. With fc fixing this is really easy to fix since I will have access to my whole editor and not only my shell movement and shortcut.

But that’s not all fc can also take an argument to fix an arbitrary command from your shell history, not only the last one.

Use fc -l to list your history

$ fc -l
6213 rm -rf test
6214 brew install asciinema
6215 vim ~/bin/mac-bootstrap
6216 exit
6221 ls
6222 mkdir right-path
6228 asciinema rec
6229 cd wrong-path
6230 fc
6231 cd right-path
6232 pwd

Then you can use fc <history number> to fix a specific command:

$ fc 6214 # Will open “brew install asciinema” in your editor

There is not much more to this tool, but you can find the full documentation of fc here if want to check out the other arguments.

This post was originally published on my website: mayeu.me.

How to Import a Managed Digital Ocean Kubernetes cluster in Rancher

Mayeu — Sat, 27 Apr 2019 00:00:00 +0000

This post contains affiliate links to Digital Ocean. It will get you $100 to test out the platform, and if at some point you start using it for real I will earn $25.

Digital ocean (DO) recently announced their managed offering for kubernetes. Rancher is a tool that helps you manage multiple Kubernetes clusters in one interface but it does not yet support the new DO offering natively, and I was wondering if one could import a DO managed cluster in your Rancher instance.

Turn out you can, here is how.

TL;DR: just follow the importer

If you already know how to use the importer for existing clusters then you are good to go. If not, just go there, chose the “Existing Cluster Import” option and follow the steps to import your DO managed cluster 🙂

Illustrated Guide

Here is a step by step guide on doing the import. It assumes you already have a Rancher instance running somewhere. (Here is a nice and straightforward guide on running Rancher on DO).

Your very first step is to go create a managed cluster on DO. For that connect to your account, and in the sidebar select the “Kubernetes” menu. When there, follow the guide to create your cluster.

Now that your cluster is provisioning you can download its configuration. We are going to use it to connect to it via kubectl.

Open your rancher interface, and click on the “Add Cluster” button

Select the “Import existing cluster”

The importer will give you a kubectl command to run against your Digital Ocean cluster. Use the previously downloaded config file to run it:

$ kubectl --kubeconfig=config-from-do.yml apply -f \ https://your.rancher.domain/v3/import/lotofgibberishrlasxtowg.yaml

When the configuration has been applied, your cluster should be imported and available in your rancher interface

And voilà, you have imported a Digital Ocean managed cluster in your Rancher 🎉

This post was originally published on mayeu.me.

Économiser du temps et de la bande passante avec un cache Docker local

Mayeu — Tue, 04 Dec 2018 06:00:00 +0000

Régulièrement, j'utilise Docker 🐳 dans des VMs (vagrant) ou sur d'autres
machine de mon réseau local, et je me retrouve à télécharger plusieurs fois la
même image sur ces différentes machines. En plus du gâchis de bande passante,
ça devient rapidement une grosse perte de temps sur de petites connexions !
Pour régler ce problème j'utilise maintenant un registre (« registry ») Docker
qui tourne localalement et qui cache de manières transparentes toutes les
images récupérées par Docker. Voici comment mettre cela en place.

D'abord, un peu de préparation. Nous allons créer un dossier qui va être
utilisé par le registre pour stocker toutes ces données. Ce dossier peut-être
n'importe où sur votre machine, personnellement je l'ai mis dans /var/lib :

$ sudo mkdir /var/lib/docker-registry

Nous allons ajouter la configuration par défaut du registre dans ce dossier, et
pour cela on va directement l'extraire depuis l'image Docker :

$ sudo docker run -it --rm registry:2       \
       cat /etc/docker/registry/config.yml  \
       > /var/lib/docker-registry/config.yml

La configuration pourra être différente selon le moment ou vous la récupérez, à
la création de cet article elle ressemblait à ça :

version: 0.1
log:
  fields:
    service: registry
storage:
  cache:
    blobdescriptor: inmemory
  filesystem:
    rootdirectory: /var/lib/registry
http:
  addr: :5000
  headers:
    X-Content-Type-Options: [nosniff]
health:
  storagedriver:
    enabled: true
    interval: 10s
    threshold: 3

Pour activer la fonction de proxy du serveur, il faut ajouter la configuration
suivante au fichier yaml:

proxy:
  remoteurl: https://registry-1.docker.io

La clé remoteurl peut pointer vers n'importe quel registre, ici j'ai mis
celui de docker par défaut.

La configuration finale ressemble à ça:

version: 0.1
log:
  fields:
    service: registry
storage:
  cache:
    blobdescriptor: inmemory
  filesystem:
    rootdirectory: /var/lib/registry
http:
  addr: :5000
  headers:
    X-Content-Type-Options: [nosniff]
health:
  storagedriver:
    enabled: true
    interval: 10s
    threshold: 3
proxy:
  remoteurl: https://registry-1.docker.io

Maintenant que la configuration est finie, on peut démarrer notre registre. À
noter que j'ai utilisé l'option --restart=always dans la commande suivante
pour m'assurer que le registre démarre automatiquement avec le démon Docker.

$ sudo docker run --restart=always -p 5000:5000                         \
         --name v2-mirror -v /var/lib/docker-registry:/var/lib/registry \
         --detach registry:2 serve /var/lib/registry/config.yml

On utilise -v pour monter le dossier créé précédemment dans l'image, et on
démarre le registre avec l'option serve suivit du chemin vers le fichier de
configuration.

Assurons-nous d'abord que le container est bien lancé avec docker ps:

$ sudo docker ps
CONTAINER ID   IMAGE        CREATED          STATUS          PORTS                    NAMES
67425da4ea4c   registry:2   32 seconds ago   Up 29 seconds   0.0.0.0:5000->5000/tcp   v2-mirror

On peut ensuite lister le contenu du registre (vide) via curl :

$ curl http://localhost:5000/v2/_catalog
{"repositories":[]}

Maintenant que notre registre est actif, on va configurer docker pour qu'il
l'utilise. Pour cela il faut éditer le fichier /etc/docker/daemon.json pour
ajouter la configuration suivante (qui doit être du JSON valide) :

{
    "registry-mirrors": ["http://localhost:5000"]
}

Il est possible que ce fichier (voir dossier) n'existe pas sur votre système.
Si c'est le cas, vous pouvez le créer manuellement (avec l'utilisateur root).
Une fois la configuration changée il faut redémarrer docker. Ici j'assume que
vous avez un système qui utilise Systemd:

$ sudo systemctl restart docker

Nous sommes maintenant prêts à réaliser notre premier download pour vérifier que
le proxy fonctionne correctement:

$ sudo docker pull redis
Using default tag: latest
latest: Pulling from library/redis
f17d81b4b692: Pull complete
b32474098757: Pull complete
8980cabe8bc2: Pull complete
e614c66c2b9c: Pull complete
6eb43ec9256b: Pull complete
394ecf5f46d4: Pull complete
Digest: sha256:f30f134bd475d451ce3207fb128bcef8ff87d0f520a39cac0c4ea285819c42a9
Status: Downloaded newer image for redis:latest

~ took 40s

Vérifions que l'image est maintenant dans notre registre local:

$ curl http://localhost:5000/v2/_catalog
{"repositories":["library/redis"]}

Et voilà ! Nous venons de cacher notre première image 🎉. Vérifions maintenant
que le cache fonctionne comme il faut. En premier lieu, on va effacer l'image
de notre démon docker:

$ sudo docker rmi redis
Untagged: redis:latest
Untagged: redis@sha256:f30f134bd475d451ce3207fb128bcef8ff87d0f520a39cac0c4ea285819c42a9
Deleted: sha256:415381a6cb813ef0972eff8edac32069637b4546349d9ffdb8e4f641f55edcdd
Deleted: sha256:2a5a57892da005399e6ce7166c5521cdca43a07872f23995e210bde5dae2640e
Deleted: sha256:85e1fabde4fd4d6df993de44ef3e04d15cd69f9d309c0112c6a5054a6dc8351a
Deleted: sha256:2725175b62c7479ee209454110e8293080b9711e4f0a29219e358d1afba88787
Deleted: sha256:7ae66985fd3a3a132fab51b4a43ed32fd14174179ad8c3041262670523a6104c
Deleted: sha256:bf45690ef12cc54743675646a8e0bafe0394706b7f9ed1c9b11423bb5494665b
Deleted: sha256:237472299760d6726d376385edd9e79c310fe91d794bc9870d038417d448c2d5

Et on la récupère de nouveau:

sudo docker pull redis
Using default tag: latest
latest: Pulling from library/redis
f17d81b4b692: Pull complete
b32474098757: Pull complete
8980cabe8bc2: Pull complete
e614c66c2b9c: Pull complete
6eb43ec9256b: Pull complete
394ecf5f46d4: Pull complete
Digest: sha256:f30f134bd475d451ce3207fb128bcef8ff87d0f520a39cac0c4ea285819c42a9
Status: Downloaded newer image for redis:latest

~ took 13s

Ce qui nous a pris 3x moins de temps ! 👍 Le téléchargement était quasiment
instantané, et seule la décompression a pris du temps.

Avec ça nous avons maintenant un cache local transparent pour toutes les images
Docker que l'on télécharge. On peut maintenant pointer les différentes VM ou
machine du réseau vers ce cache, et profiter du temps gagné pour faire des
choses utiles plutôt que de télécharger des octets depuis internet :)

Un effet secondaire intéressant de ce cache, et que si un docker pull échoue
au milieu du téléchargement, les images intermédiaires déjà téléchargées seront
conservées dans le cache, et donc il ne sera pas nécessaire de les télécharger
de nouveau. Vous pouvez vérifier ça en stoppant un pull et en le relançant
avec et sans le proxy.

Cet article a été initialement publié sur mayeu.me.

Saving Time and Bandwidth by Caching Docker Images With a Local Registry

Mayeu — Tue, 04 Dec 2018 06:00:00 +0000

On a regular basis I use Docker 🐳 in VM (vagrant), or on various machines of my local network. This lead me to download images on one machine while those were already downloaded on another one. Beside the waste of bandwidth, on a lowspeed or crowded connection this is also a huge waste of time! To solve this issue, I am now running a local registry on my laptop that automatically cache any images I request via docker.

To achieve this, we are going to run the official Docker registry in proxy mode, and then we will instruct our Docker daemon to use this local registry as its default one.

First we need to create a folder that will be used by the registry to store all the images and data it needs. It can be anywhere on your machine, I personally put it in the somewhat standard /var/lib folder:

$ sudo mkdir /var/lib/docker-registry

To ensure we have an up-to-date configuration file of the current registry version, we can directly extract it from the docker image (and at the sametime, pull the image):

$ sudo docker run -it --rm registry:2 cat \
       /etc/docker/registry/config.yml > /var/lib/docker-registry/config.yml

Depending on your version of the docker registry image, the configuration maybe slightly different from mine. At the time of creation of this article it looked like this:

version: 0.1
log:
  fields:
    service: registry
storage:
  cache:
    blobdescriptor: inmemory
  filesystem:
    rootdirectory: /var/lib/registry
http:
  addr: :5000
  headers:
    X-Content-Type-Options: [nosniff]
health:
  storagedriver:
    enabled: true
    interval: 10s
    threshold: 3

To activate the proxy behaviour of the registry we have to had the following key in the configuration:

proxy:
  remoteurl: https://registry-1.docker.io

Where remoteurl is the URL of the remote registry to use by default. Here it is Docker’s official one.

The final configuration looks like this:


version: 0.1
log:
  fields:
    service: registry
storage:
  cache:
    blobdescriptor: inmemory
  filesystem:
    rootdirectory: /var/lib/registry
http:
  addr: :5000
  headers:
    X-Content-Type-Options: [nosniff]
health:
  storagedriver:
    enabled: true
    interval: 10s
    threshold: 3
proxy:
  remoteurl: https://registry-1.docker.io

With the configuration done, we can start our local registry. Note that I have added a --restart=always option flag in the command, so that each time the Docker daemon will start the registry will also start automatically.

$ sudo docker run --restart=always -p 5000:5000 \
         --name v2-mirror -v /var/lib/docker-registry:/var/lib/registry \
         --detach registry:2 serve /var/lib/registry/config.yml

In the command we use -v to mount our previously created registry folder in the image, and we start the registry via serve with the configuration file as the only parameter.

We can check that it is running:

$ sudo docker ps
CONTAINER ID IMAGE CREATED STATUS PORTS NAMES
67425da4ea4c registry:2 32 seconds ago Up 29 seconds 0.0.0.0:5000->5000/tcp v2-mirror

We can query the content of our empty registry with curl:

$ curl http://localhost:5000/v2/_catalog
{"repositories":[]}

Now that we have our local registry running we have to configure the Docker daemon so it will use it instead of the default one. This requires a change inthe /etc/docker/daemon.json file (as root):

{
    "registry-mirrors": ["http://localhost:5000"]
}

This file and folder may not exist on your system yet. If so you can safely create it. After the change we need to restart the daemon. Assuming your system uses Systemd it should look like:

$ sudo systemctl restart docker

We can now try to download an image to see if it correctly use our caching proxy:

$ sudo docker pull redis
Using default tag: latest
latest: Pulling from library/redis
f17d81b4b692: Pull complete
b32474098757: Pull complete
8980cabe8bc2: Pull complete
e614c66c2b9c: Pull complete
6eb43ec9256b: Pull complete
394ecf5f46d4: Pull complete
Digest: sha256:f30f134bd475d451ce3207fb128bcef8ff87d0f520a39cac0c4ea285819c42a9
Status: Downloaded newer image for redis:latest

~ took 40s

Let’s check the content of the registry:

$ curl http://localhost:5000/v2/_catalog
{"repositories":["library/redis"]}

It seems that we just cached our first image 🎉. Let’s try the caching then!First we delete the redis image from our docker daemon:

$ sudo docker rmi redis
Untagged: redis:latest
Untagged: redis@sha256:f30f134bd475d451ce3207fb128bcef8ff87d0f520a39cac0c4ea285819c42a9
Deleted: sha256:415381a6cb813ef0972eff8edac32069637b4546349d9ffdb8e4f641f55edcdd
Deleted: sha256:2a5a57892da005399e6ce7166c5521cdca43a07872f23995e210bde5dae2640e
Deleted: sha256:85e1fabde4fd4d6df993de44ef3e04d15cd69f9d309c0112c6a5054a6dc8351a
Deleted: sha256:2725175b62c7479ee209454110e8293080b9711e4f0a29219e358d1afba88787
Deleted: sha256:7ae66985fd3a3a132fab51b4a43ed32fd14174179ad8c3041262670523a6104c
Deleted: sha256:bf45690ef12cc54743675646a8e0bafe0394706b7f9ed1c9b11423bb5494665b
Deleted: sha256:237472299760d6726d376385edd9e79c310fe91d794bc9870d038417d448c2d5

And we pull it again:

sudo docker pull redis
Using default tag: latest
latest: Pulling from library/redis
f17d81b4b692: Pull complete
b32474098757: Pull complete
8980cabe8bc2: Pull complete
e614c66c2b9c: Pull complete
6eb43ec9256b: Pull complete
394ecf5f46d4: Pull complete
Digest: sha256:f30f134bd475d451ce3207fb128bcef8ff87d0f520a39cac0c4ea285819c42a9
Status: Downloaded newer image for redis:latest

~ took 13s

Nice, 3x time quicker! 👍 With the download part was almost instant, and the image decompression took most of the time!

And done! We now have a local cache for all our Docker images that will start each time the docker daemon is started. We can point all our VM or machines in the network to it (by making listen to the outside). And we can now enjoy more time to focus on what matter to us and less on downloading bits from the internet :)

A nice effect of this caching is that intermediate images are also cached. Which is really useful with unstable Internet access. Because now when the connection is going to timeout in the middle of a pull, all the bits already downloaded will be cached so you can continue your pull where you left it! You can try that by starting to pull and image, stop it, then start it again. With the default daemon without cache this leads to a downloading all of the intermediate image again, but not with the proxy 👏

This post was originally published on mayeu.me.

Running Systemd commands on remote host from your local host

Mayeu — Mon, 22 Oct 2018 05:00:00 +0000

While I was reading the pretty awesome Archlinux wiki for something completely different, I found out that one can launch any systemctlcommand to a remote host via the --host (or -H) flag.

So let say you want to check the cron process on a server named kitten:

$ systemctl -H root@kitten status cron
● cron.service - Regular background program processing daemon
   Loaded: loaded (/lib/systemd/system/cron.service; enabled; vendor preset: enabled)
   Active: active (running) since Sat 2018-07-28 11:36:37 CST; 2 months 25 days ago
     Docs: man:cron(8)
 Main PID: 1045
    Tasks: 1 (limit: 4643)
   CGroup: /system.slice/cron.service
           └─1045 /usr/sbin/cron -f

Nice right? Under the hood this use SSH so you must have access to the host. And you can, of course, use any systemctl command, not just status ;)

Have fun 👋

This post was originally published on mayeu.me.

Lancer une commande systemd sur un serveur distant depuis la machine locale

Mayeu — Mon, 22 Oct 2018 05:00:00 +0000

Alors que j’étais en train de lire l’excellent wiki d’Archlinux pour quelque chose de complètement différent, j’ai découvert qu’il était possible d’utiliser l’option --host (ou -H) pour lancer une commande desystemctl directement sur un serveur distant.

Par exemple, disons que vous vouliez vérifier la santé du processus cron sur un serveur nommé chaton:

$ systemctl -H root@chaton status cron
● cron.service - Regular background program processing daemon
   Loaded: loaded (/lib/systemd/system/cron.service; enabled; vendor preset: enabled)
   Active: active (running) since Sat 2018-07-28 11:36:37 CST; 2 months 25 days ago
     Docs: man:cron(8)
 Main PID: 1045
    Tasks: 1 (limit: 4643)
   CGroup: /system.slice/cron.service
           └─1045 /usr/sbin/cron -f

Pratique non ? Les commandes sont exécutées par SSH, il est donc nécessaire d’avoir l’accès au serveur préalablement. Et ça n’est pas limité à status n’importe quelle commande de systemctl va pouvoir être lancée comme cela ;)

Have fun 👋

Cet article a été initialement publié sur mayeu.me.

How to set the language of a post?

Mayeu — Sun, 30 Sep 2018 02:44:40 +0000

Hello 👋,

I see in the preference that there is a Languages option, and my blog uses French and English so almost all my post are in this two languages (two separated posts, not both language on the same posts).

I am wondering how does that work with post I make federate on dev.to, should I add a language tag or a specific declaration in the front matter? The documentation does not says any thing about the language (yet?).

Nettoyer les données non utilisées par Docker

Mayeu — Mon, 24 Sep 2018 00:00:00 +0000

Attention: lisez l’article en entier avant de taper une commande que vous pourriez regretter

Si vous utilisez Docker (que ça soit en production ou sur votre machine dedéveloppement) vous avez surement accumulé beaucoup de données maintenant inutiles. Il existe des outils comme docker-gc pour nettoyer tout ça, mais depuis l’API 1.25 docker intègre la commande prune qui pourrait suffire amplement.

L’usage de base est plutôt simple:

$ sudo docker system prune
WARNING! This will remove:
        - all stopped containers
        - all networks not used by at least one container
        - all dangling images
        - all build cache
Are you sure you want to continue? [y/N] y

Deleted Containers:
deleted: sha256:ea43728b2d10e7b0fe24036f9531caac96bd02f779b95a6620110f00ccd3b002
deleted: sha256:022db612b3070971ce7d51778806a1f995a9c3aa1a741a6c0be0bca603787387
...approximativement 2 gajillion de hashes...

Total reclaimed space: 5.64GB

Ça va enlever pas mal de choses, mais malheureusement pas tout ce qui est inutilisé (les images sans containers par exemple ne sont par retirées avec cette commande). Pour cela il faut ajouter l’option --all:

$ sudo docker system prune --all
WARNING! This will remove:
        - all stopped containers
        - all networks not used by at least one container
        - all images without at least one container associated to them
        - all build cache
Are you sure you want to continue? [y/N] y
... moar hashes...

Total reclaimed space: 26.78GB

C’est pas mal tout ça, mais pas encore idéal. Garder un peu de cache c’est quand même utile pour accélérer les choses. Pour cela il y a l’option--filter (disponible uniquement à partir de l’API 1.28). Au moment ou j’écris ces lignes il n’y a que deux filtres disponibles: until etlabel. Dans mon cas j’utilise surtout until qui va tout nettoyer jusqu’à une certaine date (utilisant un timestamp).

Évidemment, je ne m’amuse pas à taper un timestamp, mais j’utilise date pour le générer:

$ sudo docker system prune --all --filter until=$(date -d "1 month ago" +%s)

Ici j’utilise une substitution avec $( commande ) qui permet d’exécuter une commande pour injecter son résultat sous forme de chaînes de caractères dans une autre commande. La commande date se décompose en date -d <description de la date> <format de sortie>. Ici le format de sortie +%s signifie “timestamp”. Jetez un oeil à man date si vous voulez en savoir plus.

Et voilà 😃 J’ai rarement besoin de données plus vieilles qu’un mois (et même 1 mois est très conservateur, mais comme j’ai rarement accès à un bon débit je garde le plus possible).

Finalement, si vous avez l’âme d’un aventurier vous pouvez ajouter l’option--force qui va lancer la commande sans confirmation nécessaire. Je l’utilise sur mon serveur d’intégration continue dans une tâche cron:

$ sudo docker system prune --force --all --filter until=$(date -d "1 week ago" +%s)

Joyeux ménage ♻️

Une note à propos de sudo: sur ma machine de développement (qui est un système GNU/Linux), j’utilise Docker exclusivement via sudo, car être dans le group docker revient à donner les droits root à son utilisateur. Cela est dû au fait que docker est un daemon privilégié sur le système. Plus d’info dans la documentation officielle : Docker security.

Cet article a été initialement publié sur mayeu.me.

Cleaning Unused Docker Data Without External Tools

Mayeu — Mon, 24 Sep 2018 00:00:00 +0000

Warning: read the full article before typing commands you may regret.

If you are a docker user (may it be in production or on your development machine), you may have accumulated quite a bit of now useless data. There are some existing tools to clean your daemon of unused images and containers (like docker-gc), but since the docker API 1.25 docker has an easy to use prune command that could me good enough for you.

Basic usage is pretty simple:

$ sudo docker system prune
WARNING! This will remove:
        - all stopped containers
        - all networks not used by at least one container
        - all dangling images
        - all build cache
Are you sure you want to continue? [y/N] y

Deleted Containers:
deleted: sha256:ea43728b2d10e7b0fe24036f9531caac96bd02f779b95a6620110f00ccd3b002
deleted: sha256:022db612b3070971ce7d51778806a1f995a9c3aa1a741a6c0be0bca603787387
... approximately 2 gajillion hashes later...

Total reclaimed space: 5.64GB

That’s nice, but it does not clean everything unused. For that you should add the --all option to remove every image and not only the dangling ones:

$ sudo docker system prune --all
WARNING! This will remove:
        - all stopped containers
        - all networks not used by at least one container
        - all images without at least one container associated to them
        - all build cache
Are you sure you want to continue? [y/N] y
... moar hashes...

Total reclaimed space: 26.78GB

Of course that may not be ideal. Having caches is pretty useful after all. And there is just the option for that: --filter (only available starting at API 1.28). At the time of my writing there are only two filters: untiland label. In my case I only use the until one which allows you to clean everything older than a specified timestamp.

Of course I don’t put a timestamp in the command, I use date to generate a timestamp based on a human readable date:

$ sudo docker system prune --all --filter until=$(date -d "1 month ago" +%s)

This command uses a substitution via $( command ), this allows you to get the result of a command and inject it in another one as a string. The date command itself is decomposed as: date -d <time description> <time format>, where+%s is the format to get a timestamp. You can check man date for more info.

And voilà 😃 I rarely need the cache to be older than a month on my laptop (and one month is already pretty conservative, but that’s because I rarely have a good internet so I cache as much as possible).

Finally, if you feel adventurous enough you can add the --force option so that the command will not ask for confirmation. I use the following in acronjob on my continuous integration server:

$ sudo docker system prune --force --all --filter until=$(date -d "1 week ago" +%s)

Happy cleaning ♻️

A note about sudo: on my dev machine (which is a GNU/Linux system) I only use docker via sudo because not doing so mean that your user has the same power as root all the time. This is due to the fact that the docker daemon is a privileged process running as root. See the official Docker security page.

This post was originally published on mayeu.me.

DEV Community: Mayeu

What's a good way of sending a big directory structure to a server using Ansible?

How to Read the Content of an External File With Ansible

How to Trigger Any Action When a File or Folder Changes on Macos on the Cheap

How to watch for changes?

The test Setup

Creating a service file for launchd

Trigger it!

Fix Mistakes in Shell Commands Easily With `fc`

How to Import a Managed Digital Ocean Kubernetes cluster in Rancher

TL;DR: just follow the importer

Illustrated Guide

Économiser du temps et de la bande passante avec un cache Docker local

Saving Time and Bandwidth by Caching Docker Images With a Local Registry

Running Systemd commands on remote host from your local host

Lancer une commande systemd sur un serveur distant depuis la machine locale

How to set the language of a post?

Nettoyer les données non utilisées par Docker

Cleaning Unused Docker Data Without External Tools

Creating a service file for `launchd`