DEV Community: Farzad Sadeghi

Make the zsh Prompt Go Faster

Farzad Sadeghi — Sun, 29 Dec 2024 16:36:31 +0000

How to make the zsh prompt faster

In this post, we are going to take a look at 3 ways of making the zsh prompt return faster.

We will talk about:

plugin precompilation
caching
async functions

There are obviously other methods, of course, but you can find them on almost any other blog talking about zsh prompts so for the sake of brevity, I'm not going to repeat them(an example: use fnm instead of nvm or why would you not use mise). There is also a faster syntax highlighter for zsh. so on and so forth.

Also:

Using the color red for all segments in your zsh prompt will make it go faster because, as everyone should know by now, "red ones go faster"

Before we start though, a bit of reasoning on why I would be doing the things the way I am doing them. A prompt written entirely in a faster language will be faster, yes, but I'm not willing to write a couple of thousand lines of code to get my current shell prompt(the major issue here being maintenance throughout the years,i.e. technical debt). My shell prompt is pretty critical for me so I want to be in complete control over it which means I am unwilling to use ready-made prompts with thousands of lines of code which don't have half the segments that I need.

And finally:

If you say you care about your zsh prompt's performance then that means that you already have `zmodload zsh/zprof` in your rc file.

If you talk about performance, then you have to have benchmarks and profilers. zprof is built in.

Plugin Precompilation

zsh can compile zsh scripts using the builtin zcompile into wordcode. This will have the effect of having faster parsing.
The way we use this to get a faster prompt is to explicitly ask zsh to compile certain chunky plugins(think your syntax highlighters and completion plugins) into wordcode.

For example:

zcompile-many ~/.oh-my-zsh/custom/plugins/zsh-autosuggestions/{zsh-autosuggestions.zsh,src/**/*.zsh}
zcompile-many ~/zsh-async.git/v1.8.5/async.zsh

You can read more about zcompile in man 1 zshall, under SHELL BUILTIN COMMANDS, under zcompile.

Caching

Eval Caching

First we will talk about evalcaching: caching the results of evals. This is a thing because some heavier/older version managers use this to inject themselves into your shell and some of them just take too much time.
For those kinds of plugins you can use evalcache.
After sourcing the script, you can use it like so:

_evalcache rbenv init -

We basically just swap out all instances of eval with _evalcache.

Admittedly, this is very limited in scope and it has no smart way of redoing the cache. A function is provided, _evalcache_clear, which will clear the cache, which in turn will result in the cache being regenerated.

Ye Old Result Caching

You can also just cache some results in a file and then just read from a file. Now how you update the cache is really up to you. You can use a user service, you can use cron jobs. You can even run it in the prompt itself or run the function asynchronously which we will get to in the next section.

Below is a simple example of implementing such a cache:

caching_function() {
  CACHE_OUTPUT="$1"
  TIME_CACHE=$2

  # check if the cache result is too old
  if [ $(($(stat --format=%Y "$CACHE_OUTPUT") + TIME_CACHE)) -gt "$(date +%s)" ]; then
    # the cache is still valid, we don't need to do anything
    :
  else
    # the cache value is too old
    if OUTPUT=$("your function that generates the output goes here"); then
      if [ -n "${OUTPUT}" ]; then
        echo "${OUTPUT}" >"${CACHE_OUTPUT}"
      fi
    fi
  fi

  cat "${CACHE_OUTPUT}"
}

Our function has two arguments:

The first arg tells it where to store the cache. I keep mine under /tmp.
The second is the amount of time the cache result will be valid for.

The function will check the last time the file was updated and compare that using the CACHE_TIME var and the current time. If the cache value is new enough, we just return the cache value and we are done. If the cache is old, we run the function that generates the result, update the cache and return the new result.

Async Functions

This is probably the most important one. Again, no surprises here. We will use a zsh async library,zsh-async, to have some async segments in our zsh prompt.

Let's take this zsh function from my prompt:

docker_compose_running_pwd() {
  local cwd="$1"
  local list=$(docker compose ls --format json | jq '.[].ConfigFiles' | tr -d '"')
  local array=("${(@f)$(echo ${list})}")
  for elem in "${array[@]}"; do
    if [[ "${cwd}" == $(dirname "${elem}") ]];then
      echo "\U1F7E6"
      return
    else
        ;
    fi
  done
}

This function prints a blue square if there is a docker compose file running from the current path we are in and outputs nothing if we are not.

Under normal circumstances this does not require to be an async segment but as soon as you switch your docker context to a remote docker host, then you will start to feel the pain.

We first define a start function that registers an async worker with the library and then define a callback function that gets called when the async runner returns:

_async_dcpr_start() {
  async_start_worker dcpr_info
  async_register_callback dcpr_info _async_dcpr_info_done
}

The start function has nothing special in it. We just register an async worker and then register a callback function that will be called when the runner returns. Please do note that async_register_callback will require two arguments, the name of the async runner and the name of our callback function.

Next we will define our callback function:

_async_dcpr_info_done() {
  #first part
  local job=$1
  local return_code=$2
  local stdout=$3
  local more=$6
  #second part
  if [[  $job == '[async]' ]]; then
    if [[ $return_code -eq 2 ]]; then
      _async_dcpr_start
      return
    fi
  fi
  #third part
  dcpr_info_msg=$stdout
  #fourth part
  [[ $more == 1 ]] || set-prompt && zle reset-prompt
}

The callback functions gets 6 arguments(copied from here):

$1 job name, e.g. the function passed to async_job
$2 return code
- Returns -1 if return code is missing, this should never happen, if it does, you have likely run into a bug. Please open a new issue with a detailed description of what you were doing.
$3 resulting (stdout) output from job execution
$4 execution time, floating point e.g. 0.0076138973 seconds
$5 resulting (stderr) error output from job execution
$6 has next result in buffer (0 = buffer empty, 1 = yes)
- This means another async job has completed and is pending in the buffer, it's very likely that your callback function will be called a second time (or more) in this execution. It's generally a good idea to e.g. delay prompt updates (zle reset-prompt) until the buffer is empty to prevent strange states in ZLE.

The function itself is straightforward. I like to rename the shell arguments so i have to deal with a name a couple of months from the time of writing the function and not some random numbers.

Next is the part where we handle the errors. These are the errors returned by the async runner. I like to call the start function of the async job again to get it to start again. This is not, generally speaking, a good idea since if something is broken and a rerun won't fix it, you end up in a loop. This is essentially my version of having the async job "failing loudly".

The third part is where we assign the stdout that our function, docker_compose_running_pwd, made to a "global" variable. This variable,dcpr_info_msg, is the variable that we will use in our prompt.

The fourth part is the most important part. The first condition checks for an empty buffer. If the buffer is not empty, it means we have more async jobs that have finished and are waiting for their turn, in which case it will not update the prompt. If, however the prompt buffer is empty, then we will set and reset the prompt to get an updated prompt displayed when the buffer is empty.

This is ideal, because, first and foremost, the library's documentation tells us to do that to avoid having zle ending up in a weird state. The other reason would be to avoid unnecessary zle resets to cut down on needless flickering and to save us some CPU cycles.

set-prompt is a function that I use to set the actual prompt. You may not need to call that or whatever equivalent that you have. In my case, I have to do it since my prompt lies on the fancier side of prompts and a change in the amount of characters in the prompts(because an async function is providing its output after the initial prompt display) will need to be taken into account so that's why i run the prompt after everything(all the async functions) has returned, so that the final character lengths are known and correct.

In the next section, we simply call the init function of the library and then call our own start function after that:

async_init
_async_dcpr_start

In this section we add our async job to the precmd hook for zsh so that our async job runs on the precmd hook on every prompt. The precmd hook is executed before the prompt is displayed.

Also please do note that this is where we actually tell the async runner what function to actually run. Moreover, this is where we pass any arguments that may or may not be needed by the said function.

Do keep in mind that the async execution environment our function will run in is not the same as the one your shell prompt will be run in. This means that env vars will not carry over. In our example our docker compose function cannot get the current working directory by just accessing the $PWD env var so we will have to pass $PWD to it as a function argument manually.

add-zsh-hook precmd (){
  async_job dcpr_info docker_compose_running_pwd $PWD
}

This final part serves two purposes. First, it clears the prompt var on changing a directory, so that we don't get a wrong result until we get a new result on a new prompt. Second, this also serves as the definition for our global var that we will use in the prompt.

add-zsh-hook chpwd() {
  dcpr_info_msg=
}

Here's everything put together:

docker_compose_running_pwd() {
  local cwd="$1"
  local list=$(docker compose ls --format json | jq '.[].ConfigFiles' | tr -d '"')
  local array=("${(@f)$(echo ${list})}")
  for elem in "${array[@]}"; do
    if [[ "${cwd}" == $(dirname "${elem}") ]];then
      echo "\U1F7E6"
      return
    else
        ;
    fi
  done
}

_async_dcpr_start() {
  async_start_worker dcpr_info
  async_register_callback dcpr_info _async_dcpr_info_done
}

_async_dcpr_info_done() {
  #first part
  local job=$1
  local return_code=$2
  local stdout=$3
  local more=$6
  #second part
  if [[  $job == '[async]' ]]; then
    if [[ $return_code -eq 2 ]]; then
      _async_dcpr_start
      return
    fi
  fi
  #third part
  dcpr_info_msg=$stdout
  #fourth part
  [[ $more == 1 ]] || set-prompt && zle reset-prompt
}

async_init
_async_dcpr_start

add-zsh-hook precmd (){
  async_job dcpr_info docker_compose_running_pwd $PWD
}

add-zsh-hook chpwd (){
  dcpr_info_msg=
}

A Disposable Firefox Instance

Farzad Sadeghi — Wed, 26 Jun 2024 19:30:08 +0000

Making a Disposable Firefox Instance

We want to make a disposable firefox instance.

Why firefox? well the only other choice is chromium really. Mozilla are no choir boys either. Basically we are choosing between the lesser of two evils here. There is also the whole google killing off manifest v2.

Qutebrowser and netsurf are solid but for this one, we will choose something that has more compatibility.

Now let's talk about the requirements and goals for this lil undertaking of ours:

Requirements and Goals

We want:

the instance to be ephemeral. This will prevent any persistent threat to remain on the VM.
the instance to be isolated from the host.
to prevent our IP address from being revealed to the websites we visit.

We will not be:

doing any fingerprint-resisting. In case someone wants to do it, here's a good place to start: arkenfox's user.js
we are trying to keep our IP from being revealed to the websites we visit. We don't care whether a VPN provider can be subpoenaed or not. Otherwise, needless to say, use your own VPN server but that will limit the IP choices you have. Trade-offs people, trade-offs. There is also the better choice, imho, which is use a SOCKS5 proxy.

Implementation

Isolation and Sandboxing

We will be practicing compartmentalization. This makes it harder for threats to spread. There are more than one way to do this in the current Linux landscape. We will be using a virtual machine and not a container. Needless to say, defense in depth is a good practice so in case your threat model calls for it, one could run firefox in a container inside the VM but for our purposes running inside a virtual machine is enough.

To streamline the process, we will be using vagrant to provision the VM. Like already mentioned, we will use Vagrant's plugin for libvirt to build/manage the VM which in turn will use qemu/kvm as the hypervisor.

We value transparency so we will use an open-source stack for the virtualisation: Vagrant+libvirt+qemu/kvm

The benefits of using an open-source backend include:

we don't have to worry about any backdoors in the software. There is a big difference between "they probably don't put backdoors into their software" and "there are no backdoors on this piece of software"(the xz incident non-withstanding)
we don't have to deal with very late and lackluster responses to security vulnerabilities

Yes. We just took shots at two specific hypervisors. If you know, you know.

Now lets move on to the base for the VM.

We need something small for two reasons: a smaller attack surface and a smaller memory footprint(yes. A smaller memory-footprint. We will talk about this a bit later).

So the choice is simple if we are thinking of picking a linux distro. We use an alpine linux base image. We could pick an openbsd base. That has the added benefit of the host and the guest not running the same OS which makes it harder for the threats to break isolation but for the current iteration we will be using alpine linux.

IP Address Leak prevention

The choice here is rather simple:

We either decide to use a VPN or a SOCKS5 proxy. You could make your own VPN and or SOCKS5 proxy. This IS the more secure option but will limit the ip choices we have. If your threat model calls for it, then by all means, take that route. For my purposes using a VPN provider is enough. We will be using mullvad vpn. Specifically, we will be using the openvpn config that mullvad generates for us. We will not be using the mullvad vpn app mostly because a VPN app is creepy.

We will also be implementing a kill-switch for the VPN. In case the VPN fails at any point, we don't want to leak our IP address. A kill-switch makes sure nothing is sent out when the VPN fails.
We will use ufw to implement the kill-switch feature. This is similar to what tails OS does as in, it tries to route everything through tor but it also blocks any non-tor traffic, thus ensuring there are no leaks. We will be doing the same.

Non-Persistance

We are running inside a VM so in order to achieve non-persistence we could just make a new VM instance, run that and after we are done with the instance, we can just destroy it. We will be doing just that but we will be using a tmpfs filesystem and put our VM's disk on that. This has a couple of benefits:

RAM is faster than disk. Even faster than an nvme drive
RAM is volatile

One thing to be wary of is swap. In our case we will be using the newer tmpfs which will use swap if we go over our disk limit so keep this in mind while making the tmpfs mount. Please note that there are ways around this as well. One could use the older ramfs but in my case this is not necessary since I'm using zram for my host's swap solution. This means that the swap space will be on the RAM itself so hitting the swap will still mean we never hit the disk.

To mount a tmpfs, we can run:

sudo mount -t tmpfs -o size=4096M tmpfs /tmp/tmpfs

Remember we talked about a smaller memory footprint? This is why. An alpine VM with firefox on top of it is smaller both in disk-size and memory used(mostly because of alpine using libmusl instead of glibc).

The above command will mount a 4GB tmpfs on /tmp/tmpfs.

Next we want to create a new storage pool for libvirt so that we can specify the VM to use that in Vagrant.

virsh pool-define-as tmpfs_pool /tmp/tmpfs

and then start the pool:

virsh pool-start tmpfs_pool

Implementing the Kill-Switch Using UFW

The concept is simple. We want to stop sending packets to any external IP address once the VPN is down.

In order to achieve this, we will fulfill a much stricter requirement. We will go for a tails-like setup, in that the only allowed external traffic will be to the IP address of the VPN server(s).

Here's what that will look like:

ufw --force reset
ufw default deny incoming
ufw default deny outgoing
ufw allow in on tun0
ufw allow out on tun0
# enable libvirt bridge
ufw allow in on eth0 from 192.168.121.1 proto tcp
ufw allow out on eth0 to 192.168.121.1 proto tcp
# server block
ufw allow out on eth0 to 185.204.1.174 port 443 proto tcp
ufw allow in on eth0 from 185.204.1.174 port 443 proto tcp
ufw allow out on eth0 to 185.204.1.176 port 443 proto tcp
ufw allow in on eth0 from 185.204.1.176 port 443 proto tcp
ufw allow out on eth0 to 185.204.1.172 port 443 proto tcp
ufw allow in on eth0 from 185.204.1.172 port 443 proto tcp
ufw allow out on eth0 to 185.204.1.171 port 443 proto tcp
ufw allow in on eth0 from 185.204.1.171 port 443 proto tcp
ufw allow out on eth0 to 185.212.149.201 port 443 proto tcp
ufw allow in on eth0 from 185.212.149.201 port 443 proto tcp
ufw allow out on eth0 to 185.204.1.173 port 443 proto tcp
ufw allow in on eth0 from 185.204.1.173 port 443 proto tcp
ufw allow out on eth0 to 193.138.7.237 port 443 proto tcp
ufw allow in on eth0 from 193.138.7.237 port 443 proto tcp
ufw allow out on eth0 to 193.138.7.217 port 443 proto tcp
ufw allow in on eth0 from 193.138.7.217 port 443 proto tcp
ufw allow out on eth0 to 185.204.1.175 port 443 proto tcp
ufw allow in on eth0 from 185.204.1.175 port 443 proto tcp
echo y | ufw enable

First, we forcefully reset ufw. This makes sure we are starting from a known state.

Second, we disable all incoming and outgoing traffic. This makes sure our default policy for some unforseen scenario is to deny traffic leaving the VM.

Then we allow traffic through the VPN interface, tun0.

Finally, in my case and because of libvirt, we allow traffic to and from the libvirt bridge, which in my case in 192.168.121.1.

Then we add two rules for each VPN server. One for incoming and one for outgoing traffic:

ufw allow out on eth0 to 185.204.1.174 port 443 proto tcp
ufw allow in on eth0 from 185.204.1.174 port 443 proto tcp

eth0 is the interface that originally had internet access. Now after denying it any access, we are allowing it to only talk to the VPN server on the server's port 443.

Needless to say, the IP addresses, the ports and the protocol(tcp/udp which we are not having ufw enforce) will depend on the VPN server and your provider.

Note: make sure you are not doing DNS request out-of-band in regards to your VPN. This seems to be a common mistake and some VPN providers don't enable sending the DNS requests through the VPN tunnel by default which means your actual traffic goes through the tunnel but you are kindly letting your ISP(if you have not changed your host's DNS servers) know where you are sending your traffic to.

After setting the rules, we enable ufw.

Sudo-less NTFS

In order to make the process more streamlined and not mistakenly keep an instance alive we need to have a sudo-less NTFS mount for the VM.

Without sudo-less NTFS, we would have to type in the sudo password twice, once when the VM is being brought up and once when it is being destroyed. Imagine a scenario when you close the disposable firefox VM, thinking that is gone but in reality it needs you to type in the sudo password to destroy it, thus, keeping the instance alive.

The solution is simple. We add the following to /etc/exports:

"/home/user/share/nfs" 192.168.121.0/24(rw,no_subtree_check,all_squash,anonuid=1000,anongid=1000)

This will enable the VM to access /home/user/share/nfs without needing sudo.

The Vagrantfile

Here is the Vagrantfile that will be used to provision the VM:

ENV['VAGRANT_DEFAULT_PROVIDER'] = 'libvirt'
Vagrant.require_version '>= 2.2.6'
Vagrant.configure('2') do |config|
  config.vm.box = 'generic/alpine319'
  config.vm.box_version = '4.3.12'
  config.vm.box_check_update = false
  config.vm.hostname = 'virt-disposable'

  # ssh
  config.ssh.insert_key = true
  config.ssh.keep_alive = true
  config.ssh.keys_only = true

  # timeouts
  config.vm.boot_timeout = 300
  config.vm.graceful_halt_timeout = 60
  config.ssh.connect_timeout = 15

  config.vm.provider 'libvirt' do |libvirt|
    # name of the storage pool, mine is ramdisk.
    libvirt.storage_pool_name = 'ramdisk'
    libvirt.default_prefix = 'disposable-'
    libvirt.driver = 'kvm'
    # amount of memory to allocate to the VM
    libvirt.memory = '3076'
    # amount of logical CPU cores to allocate to the VM
    libvirt.cpus = 6
    libvirt.sound_type = nil
    libvirt.qemuargs value: '-nographic'
    libvirt.qemuargs value: '-nodefaults'
    libvirt.qemuargs value: '-no-user-config'
    # enabling a serial console just in case
    libvirt.qemuargs value: '-serial'
    libvirt.qemuargs value: 'pty'
    libvirt.qemuargs value: '-sandbox'
    libvirt.qemuargs value: 'on'
    libvirt.random model: 'random'
  end

  config.vm.provision 'update-upgrade', type: 'shell', name: 'update-upgrade', inline: <<-SHELL
    set -ex
    sudo apk update && \
      sudo apk upgrade
    sudo apk add firefox-esr xauth font-dejavu wget openvpn unzip iptables ufw nfs-utils haveged tzdata
    mkdir -p /vagrant && \
      sudo mount -t nfs 192.168.121.1:/home/devi/share/nfs /vagrant
  SHELL

  config.vm.provision 'update-upgrade-privileged', type: 'shell', name: 'update-upgrade-privileged', privileged: true, inline: <<-SHELL
    set -ex
    sed -i 's/^#X11DisplayOffset .*/X11DisplayOffset 0/' /etc/ssh/sshd_config
    sed -i 's/^X11Forwarding .*/X11Forwarding yes/' /etc/ssh/sshd_config
    rc-service sshd restart

    ln -fs /usr/share/zoneinfo/UTC /etc/localtime

    #rc-update add openvpn default
    mkdir -p /tmp/mullvad/ && \
      cp /vagrant/mullvad_openvpn_linux_fi_hel.zip /tmp/mullvad/ && \
      cd /tmp/mullvad && \
      unzip mullvad_openvpn_linux_fi_hel.zip && \
      mv mullvad_config_linux_fi_hel/mullvad_fi_hel.conf /etc/openvpn/openvpn.conf && \
      mv mullvad_config_linux_fi_hel/mullvad_userpass.txt /etc/openvpn/ && \
      mv mullvad_config_linux_fi_hel/mullvad_ca.crt /etc/openvpn/ && \
      mv mullvad_config_linux_fi_hel/update-resolv-conf /etc/openvpn && \
      chmod 755 /etc/openvpn/update-resolv-conf
    modprobe tun
    echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.d/ipv4.conf
    sysctl -p /etc/sysctl.d/ipv4.conf
    rc-service openvpn start || true
    sleep 1
  SHELL

  config.vm.provision 'kill-switch', communicator_required: false, type: 'shell', name: 'kill-switch', privileged: true, inline: <<-SHELL
    # http://o54hon2e2vj6c7m3aqqu6uyece65by3vgoxxhlqlsvkmacw6a7m7kiad.onion/en/help/linux-openvpn-installation
    set -ex
    ufw --force reset
    ufw default deny incoming
    ufw default deny outgoing
    ufw allow in on tun0
    ufw allow out on tun0
    # allow local traffic through the libvirt bridge
    ufw allow in on eth0 from 192.168.121.1 proto tcp
    ufw allow out on eth0 to 192.168.121.1 proto tcp
    # server block
    ufw allow out on eth0 to 185.204.1.174 port 443 proto tcp
    ufw allow in on eth0 from 185.204.1.174 port 443 proto tcp
    ufw allow out on eth0 to 185.204.1.176 port 443 proto tcp
    ufw allow in on eth0 from 185.204.1.176 port 443 proto tcp
    ufw allow out on eth0 to 185.204.1.172 port 443 proto tcp
    ufw allow in on eth0 from 185.204.1.172 port 443 proto tcp
    ufw allow out on eth0 to 185.204.1.171 port 443 proto tcp
    ufw allow in on eth0 from 185.204.1.171 port 443 proto tcp
    ufw allow out on eth0 to 185.212.149.201 port 443 proto tcp
    ufw allow in on eth0 from 185.212.149.201 port 443 proto tcp
    ufw allow out on eth0 to 185.204.1.173 port 443 proto tcp
    ufw allow in on eth0 from 185.204.1.173 port 443 proto tcp
    ufw allow out on eth0 to 193.138.7.237 port 443 proto tcp
    ufw allow in on eth0 from 193.138.7.237 port 443 proto tcp
    ufw allow out on eth0 to 193.138.7.217 port 443 proto tcp
    ufw allow in on eth0 from 193.138.7.217 port 443 proto tcp
    ufw allow out on eth0 to 185.204.1.175 port 443 proto tcp
    ufw allow in on eth0 from 185.204.1.175 port 443 proto tcp

    echo y | ufw enable
  SHELL

  config.vm.provision 'mullvad-test', type: 'shell', name: 'test', privileged: false, inline: <<-SHELL
    set -ex
    curl --connect-timeout 10 https://am.i.mullvad.net/connected | grep -i "you\ are\ connected"
  SHELL
end

Provisioning

We will be using the vagrant shell provisioner to prepare the VM.

The first provisioner names update-upgrade does what the name implies. It installs the required packages.

The next provisioner, update-upgrade-privileged, enables X11 forwarding on openssh, sets up openvpn as a service and starts it and finally sets the timezone to UTC.

The third provisioner, kill-switch, sets up our kill-switch using ufw.

The final provisioner runs the mullvad test for their VPN. Since at this point we have set up the kill-switch we wont leak our IP address to the mullvad website but that's not important since we are using our own IP address to connect to the mullvad VPN servers.

Interface

how do we interface with our firefox instance. ssh or spice?

I have gone with ssh. In our case we use ssh's X11 forwarding feature. This choice is made purely out of convenience. You can go with spice.

Timezone

We set the VM's timezone to UTC because it's generic.

haveged

haveged is a daemon that provides a source of randomness for our VM. Look here.

QEMU Sandbox

From man 1 qemu:

-sandbox arg[,obsolete=string][,elevateprivileges=string][,spawn=string][,resourcecontrol=string]
    Enable Seccomp mode 2 system call filter. 'on' will enable syscall filtering and 'off' will disable it. The default is 'off'.

CPU Pinning

CPU pinning alone is not what we want. We want cpu pinning and then further isolating those cpu cores on the host so that only the VM runs on those cores. This will give us a better performance on the VM side but also provide better security and isolation since this will mitigate side-channel attacks based on the CPU(the spectre/metldown family, the gift that keeps on giving).

In my case, I've done what I can on the host-side to mitigate spectre/meltdown but I don't have enough resources to ping 6 logical cores to this VM. If you can spare the resources, by all means, please do.

No Passthrough

We will not be doing any passthroughs. It is not necessarily a choice made because of security, but merely out of a lack of need for the performance benefit that hardware-acceleration brings.

Launcher Script

#!/usr/bin/dash
set -x

sigint_handler() {
  local ipv4="$1"
  xhost -"${ipv4}"
  vagrant destroy -f
}

trap sigint_handler INT
trap sigint_handler TERM

working_directory="/home/devi/devi/vagrantboxes.git/main/disposable/"
cd ${working_directory} || exit 1

vagrant up
disposable_id=$(vagrant global-status | grep disposable | awk '{print $1}')
disposable_ipv4=$(vagrant ssh "${disposable_id}" -c "ip a show eth0 | grep inet | grep -v inet6 | awk '{print \$2}' | cut -d/ -f1 | tr -d '[:space:]'")

trap 'sigint_handler ${disposable_ipv4}' INT
trap 'sigint_handler ${disposable_ipv4}' TERM

echo  "got IPv4 ${disposable_ipv4}"
xhost +"${disposable_ipv4}"
ssh \
  -o StrictHostKeyChecking=no \
  -o Compression=no \
  -o UserKnownHostsFile=/dev/null \
  -X \
  -i".vagrant/machines/default/libvirt/private_key" \
  vagrant@"${disposable_ipv4}" \
  "XAUTHORITY=/home/vagrant/.Xauthority firefox-esr -no-remote" https://mullvad.net/en/check/
xhost -"${disposable_ipv4}"
vagrant destroy -f

The script is straightforward. It brings up the VM, and destroys it when the disposable firefox instance is closed.

Let's look at a couple of things that we are doing here:

The shebang line: we are using dash, the debian almquist shell. It has a smaller attack surface. It's small but we don't need all the features of bash or zsh here so we use something "more secure".
we add and remove the IP of the VM from the xhost list. This allows the instance to display the firefox window on the host's X server and after it's done, we remove it so we don't end up whitelisting the entire IP range(least privilege principle, remember?).
we use -o UserKnownHostsFile=/dev/null to prevent the VM from adding to the host's known hosts file. There are two reasons why we do this here. One, the IP range is limited, we will eventually end up conflicting with another IP that lives on your hostsfile that was a live and well VM as some point but is now dead so libvirt will reassign its IP address to our disposable instance which will prompt ssh to tell you that it suspects there is something going on which will prevent the ssh command from completing successfully which will in turn result in the VM getting killed. Two, we will stop polluting the hostsfile by all the IPs of the disposable VM instances that we keep creating so that you won't have to deal with the same problem while running other VMs.
we register a signal handler for SIGTERM and SIGINT so that we can destroy the VM after we created it and we one of those signals. This helps ensure a higher rate of confidence in the VM getting destroyed. This does not guarantee that. A SIGKILL will kill the script and that's that.

Notes Regarding the Host

A good deal of security and isolation comes from the host specially in a scenario when you are running a VM on top of the host. This is an entirely different topic so we won't be getting into it but here is a good place to start. Just because it's only a single line at the end of some random blogpost doesn't mean its not important. Take this seriously.

We are using somebody else's vagrant base image. Supply-chain attacks are a thing so it is very much better to use our own base image.

As a starting you can look here. This is how the base image we are using is created.

Docker, Linux, Security. Kinda.

Farzad Sadeghi — Mon, 20 May 2024 02:27:25 +0000

Docker, Linux, Security. Kinda.

We will be exploring some Linux features in the context of a docker application container. Another way of explaining it would be to say we will talk about how to make more secure application containers.
We will not talk about firewall and apparmor because they are tools that enhance security on the host in general and not specific to a docker application container. A secure host means a more secure application container but that is discussion for another post.

We will focus on Linux containers since FreeBSD containers are still experimental(see here and here). Yes, windows containers exist.

We will not discuss performance. Here be performance penalties, but again that is not the focus of this post.

Before we begin, Linux docker containers are Linux. They are using most of the functionality that existed before application containers in the form of docker were a thing. Knowing Linux better means you know Linux Docker containers(application containers is a more correct term) better. We will see this point throughout this post.

Base Image

We start with the first building block of a new docker image, The base image. By far the most used base images are the Alpine docker base image, followed by Debian and Ubuntu docker base images.
These distros have two major differences that we want to focus on:

C standard library implementation
the userspace utility implementation

Debian and Ubuntu(we are not forgetting that Ubuntu itself is a Debian derivative) both use glibc, as in gnu's libc implementation. Alpine uses musl-libc as its C standard library implementation.

The major difference here which will come into play later on again is glibc has been around for much longer, so it has to keep backwards compatibility for a much longer period of time and for far more many things. Also the general attitude with the glibc team is that they have to support everything since if they don't then who will?

Libmusl on the other hand, does not try to support everything under the sun, a relatively newer project, comparatively, and, they keep their codebase lean.

As a result not all applications are supported by libmusl but a good number of them are.

In simpler terms, libmusl has a far smaller attack surface compared to glibc.

On to our second point, which is the cli utilities' implementation. Debian and Ubuntu use gnu's Coreutils while Alpine uses Busybox(remember, we are talking about the most used application container bases. You can install a desktop version of Alpine with GNU coreutils).

Here we have the same situation as before, The GNU coreutils are bigger, do more and have a larger attack surface. Busybox is smaller, does not support as many features as GNU Coreutils but does support enough of them to make them useful. Needless to say, busybox is small and hence, it has a smaller attack surface.

To get a feel for how this plays out in the real world, you can look at some of the popular images that come in both Debian and Alpine flavours on dockerhub. Take a look at the number of reported vulnerabilities for both bases. The theme we observe is simple. The bigger the attack surface the bigger the number of vulnerabilities.

Alpine images are small, lean and functional, just like libmusl and busybox but there are still quite a few things on an alpine image that are extraneous. We can take them out and have a perfectly functioning application container.

That's how we get distroless.

Distroless base images follow the same pattern as alpine base docker images, as in, less functionality while still keeping enough functionality to be able to do the job and minimize the attack surface.
Minimizing a base image like this means that the base images are very specialized so we have base images for golang, python, java and the like.

Dokcer Runtimes

By default docker uses containerd which in turn uses runc for the runtime. There are two additional runtimes that we want to focus on who try to provide a more secure runtime environment for docker.

gvisor
kata

gvisor

gVisor creates a sandbox environment. Containers interact with the host through this sandboxed environment.

gvisor has two components. Gofer and Sentry. Sentry is a kernel that runs the containers and intercepts and responds to system calls made by the application so as not to have an application directly control the syscalls that it makes.

Gofer handles filesystem access(not /proc) for the application.

The application is a regular application. gVisor aims to provide an environment equivalent to Linux 4.4. gvisor presently does not implement every system call, /proc file or /sys file.

Every sandbox environment gets its own instance of Sentry. Every container in the sandbox gets its own instance of Gofer.

gVisor currently does not support all system calls. You can find the list of supported system calls for amd64 here.

  -------------
  |Application|
  -------------
       |system calls
       |
    --------   9p    -------
    |Sentry|<------->|Gofer|
    --------         -------
         | limited    |system
         | syscalls   |calls
        ---------------
        | Host Kernel |
        ---------------
               |
               |hardware

kata

Kata creates a sandbox environment for containers to interact with as proxy, not too dissimilar to gvisor but the main point of difference is that kata uses a VM to achieve this.

gVisor and katacontainers allow us to implement defense in depth when it comes to application containers and host system security.

Capabilites and Syscalls

Let's talk about capabilities for a bit.

From man 7 capabilities:

For the purpose of performing permission checks, traditional UNIX implementations distinguish two
categories of processes: privileged processes (whose effective user ID is 0, referred to as
superuser or root), and unprivileged processes (whose effective UID is nonzero).  Privileged
processes bypass all kernel permission checks, while unprivileged processes are subject to full
permission checking based on the process's credentials (usually: effective UID, effective GID, and
supplementary group list).

Starting with Linux 2.2, Linux divides the privileges traditionally associated with superuser into
distinct units, known as capabilities, which can be independently enabled and disabled.
Capabilities are a per-thread attribute.

Capabilities give you a more granular control over which privileges to give instead of just root and non-root.

Docker let's us choose which capabilities to give to a container. So we can for example allow a non-privileged process to bind to privileged ports using capabilities.

As an example, a simple application making calls to API endpoints and writing results back to a database does not require any capabilities. It can run under a non-privileged user with no capabilities and do all the tasks that it needs to do.
That being said, determining which capabilities are required can be a bit challenging when it comes to certain applications since there is no straightforward way of achieving this. In certain cases we can get away with dropping all capabilities, running our application and then trying to figure out, based on the received error messages, which capability is missing and needs to be given to the application. But in certain cases this may not be feasible or practical.

From man 2 sycalls:

The system call is the fundamental interface between an application and the Linux kernel.

The Linux kernel lets us choose which ones of these interface calls can be allowed to be made by an application. We can essentially filter which syscalls are allowed and which ones are not on a per application basis. Docker enables this functionality with an arguably more friendly approach.

Capabilities and syscall filtering are tools to implement principle of least privilege. Ideally, we would like to allow a container to only have access to what it needs and just that. Not more, and obviously not less.

capabilities in the wild

Capabilities are a Linux feature, docker allows us to use that with application containers. We'll look at a very simple example of how one can set capabilities for a regular executable on Linux.

man 8 setcap lets us set capabilities for a file.

syscall Filtering in the wild

As an example we will look at man 1 bwrap.
Bubblewrap allows us to sandbox an application, not too dissimilar to docker. Flatpaks use bubblewrap as part of their sandbox.

Bubblewrap can optionally take in a list of syscalls to filter.

The filter is expressed as a BPF(Berkley Packet Filter program - remember when I said docker gives you a friendlier interface to seccomp?) program.

Below is a short program that defines a BPF program that can be passed to an application using bwrap that lets us log all the sycalls the application makes to syslog.

#include <fcntl.h>
#include <seccomp.h>
#include <stdbool.h>
#include <unistd.h>

void log_all_syscalls(void) {
  scmp_filter_ctx ctx = seccomp_init(SCMP_ACT_LOG);
  seccomp_arch_add(ctx, SCMP_ARCH_X86_64);
  seccomp_export_bpf(ctx, 1);
  seccomp_export_pfc(ctx, 2);
  seccomp_release(ctx);
}

int main(int argc, char **argv) {
  log_all_syscalls();
}

Building is straightforward. Just remember to link against libseccomp with -lseccomp.

gcc main.c -lseccomp

Running the above code we get this:

 > 5@#
# pseudo filter code start
#
# filter for arch x86_64 (3221225534)
if ($arch == 3221225534)
  # default action
  action LOG;
# invalid architecture action
action KILL;
#
# pseudo filter code end
#

#!/usr/bin/dash

TEMP_LOG=/tmp/seccomp_logging_filter.bpf

./a.out > ${TEMP_LOG}

bwrap --seccomp 9 9<${TEMP_LOG} bash

Then we can go and see where the logs end up. On my host, they are logged under /var/log/audit/audit.log and they look like this:

type=SECCOMP msg=audit(1716144132.339:4036728): auid=1000 uid=1000 gid=1000 ses=1 subj=unconfined pid=19633 comm="bash" exe="/usr/bin/bash" sig=0 arch=c000003e syscall=13 compat=0 ip=0x7fa58591298f code=0x7ffc0000AUID="devi" UID="devi" GID="devi" ARCH=x86_64 SYSCALL=rt_sigaction
type=SECCOMP msg=audit(1716144132.339:4036729): auid=1000 uid=1000 gid=1000 ses=1 subj=unconfined pid=19633 comm="bash" exe="/usr/bin/bash" sig=0 arch=c000003e syscall=13 compat=0 ip=0x7fa58591298f code=0x7ffc0000AUID="devi" UID="devi" GID="devi" ARCH=x86_64 SYSCALL=rt_sigaction
type=SECCOMP msg=audit(1716144132.339:4036730): auid=1000 uid=1000 gid=1000 ses=1 subj=unconfined pid=19633 comm="bash" exe="/usr/bin/bash" sig=0 arch=c000003e syscall=13 compat=0 ip=0x7fa58591298f code=0x7ffc0000AUID="devi" UID="devi" GID="devi" ARCH=x86_64 SYSCALL=rt_sigaction
type=SECCOMP msg=audit(1716144132.339:4036731): auid=1000 uid=1000 gid=1000 ses=1 subj=unconfined pid=19633 comm="bash" exe="/usr/bin/bash" sig=0 arch=c000003e syscall=13 compat=0 ip=0x7fa58591298f code=0x7ffc0000AUID="devi" UID="devi" GID="devi" ARCH=x86_64 SYSCALL=rt_sigaction
type=SECCOMP msg=audit(1716144132.339:4036732): auid=1000 uid=1000 gid=1000 ses=1 subj=unconfined pid=19633 comm="bash" exe="/usr/bin/bash" sig=0 arch=c000003e syscall=13 compat=0 ip=0x7fa58591298f code=0x7ffc0000AUID="devi" UID="devi" GID="devi" ARCH=x86_64 SYSCALL=rt_sigaction
type=SECCOMP msg=audit(1716144132.339:4036733): auid=1000 uid=1000 gid=1000 ses=1 subj=unconfined pid=19633 comm="bash" exe="/usr/bin/bash" sig=0 arch=c000003e syscall=14 compat=0 ip=0x7fa5859664f4 code=0x7ffc0000AUID="devi" UID="devi" GID="devi" ARCH=x86_64 SYSCALL=rt_sigprocmask
type=SECCOMP msg=audit(1716144132.339:4036734): auid=1000 uid=1000 gid=1000 ses=1 subj=unconfined pid=19633 comm="bash" exe="/usr/bin/bash" sig=0 arch=c000003e syscall=13 compat=0 ip=0x7fa58591298f code=0x7ffc0000AUID="devi" UID="devi" GID="devi" ARCH=x86_64 SYSCALL=rt_sigaction
type=SECCOMP msg=audit(1716144132.339:4036735): auid=1000 uid=1000 gid=1000 ses=1 subj=unconfined pid=19633 comm="bash" exe="/usr/bin/bash" sig=0 arch=c000003e syscall=1 compat=0 ip=0x7fa5859ce5d0 code=0x7ffc0000AUID="devi" UID="devi" GID="devi" ARCH=x86_64 SYSCALL=write
type=SECCOMP msg=audit(1716144132.339:4036736): auid=1000 uid=1000 gid=1000 ses=1 subj=unconfined pid=19633 comm="bash" exe="/usr/bin/bash" sig=0 arch=c000003e syscall=1 compat=0 ip=0x7fa5859ce5d0 code=0x7ffc0000AUID="devi" UID="devi" GID="devi" ARCH=x86_64 SYSCALL=write
type=SECCOMP msg=audit(1716144132.339:4036737): auid=1000 uid=1000 gid=1000 ses=1 subj=unconfined pid=19633 comm="bash" exe="/usr/bin/bash" sig=0 arch=c000003e syscall=270 compat=0 ip=0x7fa5859d77bc code=0x7ffc0000AUID="devi" UID="devi" GID="devi" ARCH=x86_64 SYSCALL=pselect6

Docker allows us to do the same. We can give docker a seccomp profile to filter out the syscalls that are not required for a specific container.

You can find the default docker seccomp profile here.

Namespaces

A namespace wraps a global system resource in an abstraction that makes it appear to the processes
within the namespace that they have their own isolated instance of the global resource.  Changes
to the global resource are visible to other processes that are members of the namespace, but are
invisible to other processes.  One use of namespaces is to implement containers.

From man 7 namespaces.
You can think of namespaces as almost the same thing as a namespace does in some programming languages.

Docker uses its own namespaces for the containers so as to further isolate the application containers from the host system.

Namespaces in the Wild

As an example let's look at the script provided below. Here we are creating a new network namespace. The new interface is provided by simply connecting an android phone for USB tethering. Depending on the situation you have going on and the udev naming rules the interface name will differ but the concept is the same. We are creating a new network namespace for a second internet provider, which in this case, is our android phone. We then use this network namespace to execute commands in the context of this specific network namespace. Essentially, we can choose which applications get to use our phone internet and which ones use whatever it is we were previously connected to.

#!/usr/bin/env sh
PHONE_NS=phone_ns
IF=enp0s20f0u6

sudo ip netns add ${PHONE_NS}
sudo ip link set ${IF} netns ${PHONE_NS}
sudo ip netns exec ${PHONE_NS} ip link set ${IF} up
sudo ip netns exec ${PHONE_NS} ip link set dev lo up
sudo ip netns exec ${PHONE_NS} dhclient ${IF}

$ sudo ip netns exec home_ns curl -4 icanhaveip.com
113.158.237.102
$ curl -4 icanhasip.com
114.201.132.98

HINT: The IP addresses are made up. The only thing that matters is that they are different.

Since we have the android phone's interface on another namespace the two cannot interfere with each other. This is pretty much how docker uses namespaces.

Without a network namespace we would have to make a small VM, run a VPN on the VM and then make a socks5 proxy to the VM from the host and then have applications pass their traffic through a socks5 proxy with varying degrees of success.

NOTE: since we are not running the script on a hook, you might blow out your net having two upstreams at the same time. In which case, run the script, then restart NetworkManager or whatever you have.

SBOM and Provenance Attestation

What is SBOM?
NIST defines SBOM as a “formal record containing the details and supply chain relationships of various components used in building software.".

It contains details about the components used to create a certain piece of software.

SBOM is meant to help mitigate the threat of supply chain attacks(remember xz?).

What is provenance?

The provenance attestations include facts about the build process, including details such as:

    Build timestamps
    Build parameters and environment
    Version control metadata
    Source code details
    Materials (files, scripts) consumed during the build

source

Example

Let's review all that we learned about in the form of a light exercise.

For the first build, we use a non-vendored version.
Vendoring means that you store your dependencies locally. This means you are in control of your dependencies. You don't need to pull them from a remote. Even if one or more of your dependencies One of the more famous examples is Lua. The Lua foundation actually recommend vendoring your Lua dependency.

Vendoring helps with build reproducability.

We will use milla as an exmaple. It's a simple go codebase.

FROM alpine:3.19 as builder
RUN apk update && \
      apk upgrade && \
      apk add go git
WORKDIR /milla
COPY go.sum go.mod /milla/
RUN go mod download
COPY *.go /milla/
RUN go build

FROM alpine:3.19
ENV HOME /home/user
RUN set -eux; \
  adduser -u 1001 -D -h "$HOME" user; \
  mkdir "$HOME/.irssi"; \
  chown -R user:user "$HOME"
COPY --from=builder /milla/milla "$HOME/milla"
RUN chown user:user "$HOME/milla"
ENTRYPOINT ["home/user/milla"]

The first docker image build is fairly simple. We copy the source code in, get our dependencies and build a static executable. As for the second stage of the build, we simply put the executable into a new base image and we are done.

The second build which is a vendored build with a golang distroless base. We copy over the source code for the project and all its dependencies and then do the same as before.

FROM golang:1.21 as builder
WORKDIR /milla
COPY go.sum go.mod /milla/
RUN go mod download
COPY *.go /milla/
RUN CGO_ENABLED=0 go build

FROM gcr.io/distroless/static-debian12
COPY --from=builder /milla/milla "/usr/bin/milla"
ENTRYPOINT ["milla"]

Below You can see an example docker compose file. Milla can optionally use a postgres database to store messages. We also include a pgadmin instance. Now let's talk about the docker compose file.

services:
  terra:
    image: milla_distroless_vendored
    build:
      context: .
      dockerfile: ./Dockerfile_distroless_vendored
    deploy:
      resources:
        limits:
          memory: 128M
    logging:
      driver: "json-file"
      options:
        max-size: "100m"
    networks:
      - terranet
    user: 1000:1000
    restart: unless-stopped
    entrypoint: ["/usr/bin/milla"]
    command: ["--config", "/config.toml"]
    volumes:
      - ./config.toml:/config.toml
      - /etc/localtime:/etc/localtime:ro
    cap_drop:
      - ALL
    runtime: runsc
  postgres:
    image: postgres:16-alpine3.19
    deploy:
      resources:
        limits:
          memory: 4096M
    logging:
      driver: "json-file"
      options:
        max-size: "200m"
    restart: unless-stopped
    ports:
      - "127.0.0.1:5455:5432/tcp"
    volumes:
      - terra_postgres_vault:/var/lib/postgresql/data
      - ./scripts/:/docker-entrypoint-initdb.d/:ro
    environment:
      - POSTGRES_PASSWORD_FILE=/run/secrets/pg_pass_secret
      - POSTGRES_USER_FILE=/run/secrets/pg_user_secret
      - POSTGRES_INITDB_ARGS_FILE=/run/secrets/pg_initdb_args_secret
      - POSTGRES_DB_FILE=/run/secrets/pg_db_secret
    networks:
      - terranet
      - dbnet
    secrets:
      - pg_pass_secret
      - pg_user_secret
      - pg_initdb_args_secret
      - pg_db_secret
    runtime: runsc
  pgadmin:
    image: dpage/pgadmin4:8.6
    deploy:
      resources:
        limits:
          memory: 1024M
    logging:
      driver: "json-file"
      options:
        max-size: "100m"
    environment:
      - PGADMIN_LISTEN_PORT=${PGADMIN_LISTEN_PORT:-5050}
      - PGADMIN_DEFAULT_EMAIL=${PGADMIN_DEFAULT_EMAIL:-devi@terminaldweller.com}
      - PGADMIN_DEFAULT_PASSWORD_FILE=/run/secrets/pgadmin_pass
      - PGADMIN_DISABLE_POSTFIX=${PGADMIN_DISABLE_POSTFIX:-YES}
    ports:
      - "127.0.0.1:5050:5050/tcp"
    restart: unless-stopped
    volumes:
      - terra_pgadmin_vault:/var/lib/pgadmin
    networks:
      - dbnet
    secrets:
      - pgadmin_pass
networks:
  terranet:
    driver: bridge
  dbnet:
volumes:
  terra_postgres_vault:
  terra_pgadmin_vault:
secrets:
  pg_pass_secret:
    file: ./pg/pg_pass_secret
  pg_user_secret:
    file: ./pg/pg_user_secret
  pg_initdb_args_secret:
    file: ./pg/pg_initdb_args_secret
  pg_db_secret:
    file: ./pg/pg_db_secret
  pgadmin_pass:
    file: ./pgadmin/pgadmin_pass

We are assigning memory usage limits for the containers. We are also limiting the size of the logs we are keeping on disk.

One thing that we did not talk about before is the networking side of compose.

As can be seen, the postgres and pgadmin container share one network while the postgres container and milla share another network. This makes it so that milla and pgadmin do not have access to each other. This is inline with principle of least privilege. Milla and pgadmin don't need to talk to each other so they can't do that.

Also we refrain from using host networking.

We are also binding the open ports to the host's localhost interface. This does not let us connect to the endpoints directly. In our example we don't need the ports to be exposed to the internet but we will need access to them. What we can do is bind the open ports to the host's localhost and then use ssh to forward the ports onto our own machine, assuming the docker host is a remote.

ssh -L 127.0.0.1:5460:127.0.0.1:5455 user@remotehost

While building milla, for the second stage of the build, we made a non-privileged user and our now mapping a non-privileged user on the host to that user. We are removing all capabilities from milla since milla will be making requests and has no server functionality. Milla will only need to bind to high-numbered ports which does not require a special privileges.

We run both postgres and milla with gvisor's runsc runtime since it's possible to do so.

Finally we use docker secrets to put the secrets into the container's runtime environment.

Now onto the attestations.

In order to view the SBOM for the image we will use docker scout.

docker scout sbom milla
docker scout sbom milla_distroless_vendored

The SBOMs can be viewed here and here respectively.

Now lets look at the provenance attestations.

docker buildx imagetools inspect terminaldweller/milla:main --format "{{ json .Provenance.SLSA }}"

And here you can look at the result.

After NTP Comes NTS. After NTS comes sdwdate.

Farzad Sadeghi — Mon, 29 Apr 2024 20:29:20 +0000

After NTP Comes NTS. After NTS comes sdwdate.

Well for this one I will be talking a bit about NTP and NTS.

Unlike the DNS post there isn't much going on here.

NTP is plain-text, NTS uses TLS so if our requests are tampered with, we can know.

There is the "oooh, you cant see what I'm sending now" but in this case its NTP so the content being secret is not necessarily more important than making sure the content has not been modified(guarantee of integrity).

So far so good.

But before we go any further, lets talk about what we are trying to achieve here, in other works, what requirements are we trying to satisfy here:

REQ-001: The NTP(NTS) requests shall be anonymous
REQ-002: It shall be evient when an NTP(NTS) requests has been tampered with
REQ-003: It should not be known which time servers are being used upstream by the client

If you are wondering why any of this even matters you can have a look here.

Now talk about the problem. The protocol is fine. We are sending TCP with TLS here. That's brilliant. We get all this:

* Identity: Through the use of a X.509 public key infrastructure, implementations can cryptographically establish the identity of the parties they are communicating with.
* Authentication: Implementations can cryptographically verify that any time synchronization packets are authentic, i.e., that they were produced by an identified party and have not been modified in transit.
* Confidentiality: Although basic time synchronization data is considered nonconfidential and sent in the clear, NTS includes support for encrypting NTP extension fields.
* Replay prevention: Client implementations can detect when a received time synchronization packet is a replay of a previous packet.
* Request-response consistency: Client implementations can verify that a time synchronization packet received from a server was sent in response to a particular request from the client.
* Unlinkability: For mobile clients, NTS will not leak any information additional to NTP which would permit a passive adversary to determine that two packets sent over different networks came from the same client.
* Non-amplification: Implementations (especially server implementations) can avoid acting as distributed denial-of-service (DDoS) amplifiers by never responding to a request with a packet larger than the request packet.
* Scalability: Server implementations can serve large numbers of clients without having to retain any client-specific state.
* Performance: NTS must not significantly degrade the quality of the time transfer. The encryption and authentication used when actually transferring time should be lightweight.

Excerpt from RFC 8915

If we find a client that lets us use a SOCKS5 proxy, then we can send our NTS requests over Tor and then call it a day.

REQ-002 and REQ-003 are being satisfied by using TLS. The missing piece is REQ-001, anonymizing the requests.

This is not something for the protocol to handle so then we have to look for a client that support a SOCKS5 proxy.

Unfortunately chrony and ntpd-rs do not support SOCKS5 proxies.

for ntpd-rs look here

Which means our setup is not complete.

Implementation

We will be using ntpd-rs as the client.

We will also setup one NTS server using ntpsec.

[observability]
log-level = "info"
observation-path = "/var/run/ntpd-rs/observe"

[[source]]
mode = "nts"
address = "virginia.time.system76.com"

[[source]]
mode = "nts"
address = "mmo1.nts.netnod.se"

[[source]]
mode = "nts"
address = "ntppool1.time.nl"

[[source]]
mode = "nts"
address = "ntp1.glypnod.com"

[[source]]
mode = "nts"
address = "ntp3.fau.de"

[synchronization]
single-step-panic-threshold = 1800
startup-step-panic-threshold = { forward="inf", backward = 1800 }
minimum-agreeing-sources = 3
accumulated-step-panic-threshold = 1800

[[server]]
listen = "127.0.0.1:123"

[[server]]
listen = "172.17.0.1:123"

[[server]]
listen = "192.168.121.1:123"

[[server]]
listen = "10.167.131.1:123"

[[server]]
listen = "[::1]:123"

nts enable
nts key /etc/letsencrypt/live/nts.dehein.org/privkey.pem
nts cert /etc/letsencrypt/live/nts.dehein.org/fullchain.pem mintls TLS1.3
nts cookie /var/lib/ntp/nts-keys
nts-listen-on 4460
server 0.0.0.0 prefer

server ntpmon.dcs1.biz nts  # Singapore
server ntp1.glypnod.com nts # San Francisco
server ntp2.glypnod.com nts # London

tos maxclock 5

restrict default kod limited nomodify noquery
restrict -6 default kod limited nomodify noquery

driftfile /var/lib/ntp/ntp.drift

statsdir /var/log/ntpstats/

version: "3.9"
services:
  filebrowser:
    image: ntpsec
    build:
      context: .
    deploy:
      resources:
        limits:
          memory: 128M
    logging:
      driver: "json-file"
      options:
        max-size: "50m"
    networks:
      - ntsnet
    ports:
      - "4460:4460/tcp"
    restart: unless-stopped
    entrypoint: ["ntpd"]
    command: ["-n", "-I", "0.0.0.0", "-d", "5"]
    volumes:
      - ./ntp.conf:/etc/ntp.conf:ro
      - /etc/letsencrypt/live/nts.dehein.org/fullchain.pem:/etc/letsencrypt/live/nts.dehein.org/fullchain.pem:ro
      - /etc/letsencrypt/live/nts.dehein.org/privkey.pem:/etc/letsencrypt/live/nts.dehein.org/privkey.pem:ro
      - vault:/var/lib/ntp
    cap_drop:
      - ALL
    cap_add:
      - SYS_NICE
      - SYS_RESOURCE
      - SYS_TIME
networks:
  ntsnet:
volumes:
  vault:

What comes after NTS

Above we looked at NTP and NTS. We failed to find a client that supports SOCKS5 but that's a trivial matter.

What is not trivial, however, is how NTS and NTP work, and by that I mean you will still have to ask a server to tell you the time.

Doing so over Tor or other anonymizing networks should be fine but we can choose to try out another method of doing things.

Enter sdwdate

sdwdate

It still has the same flaw as NTP/NTS as in we still have to trust a server not to lie please look here.

Personally, It is a bit of a disappointment that the protocol that's supposed to be oh-so-much-shinier and newer than NTP has the same flawed mechanism as NTP.

Now granted having hardware that tells you the time so that you can share that with everyone else is not something trivial or readily-available but this only makes sdwdate desirable in the absence of an NTS client that support SOCKS5 proxy.

Once that is done, the larger user pool of NTS/NTP will offer more protection against the smaller userbase of sdwdate.

sdwdate gives a table of comparison between itself and NTP. Let's take at look at that:

Let's take a look at sdwdate. It is a roller-coaster. And I do mean that. So don't make up your mind until the very end.
There is a comparison between NTP and sdwdate made here by kicksecure themselves.

category	sdwdate	ntp
written in memory-safe language	Yes	No
distributed trust	Yes	No
secure connection by default	Yes	No
gradual clock adjustments	Yes	Yes
daemon	Yes	Yes
functional over tor	Yes	No
tor not required	No	Yes
client, time fetcher	Yes	Yes
Server, time provider	No	Yes
AppArmor profile	Yes	Yes
systemd security hardening,seccomp	Yes	?
drop-in config folder	Yes	No
proxy support	Yes	No
possible to secure by default on GNU/Linux distribution level	Yes	No
secure	Yes	No
optional GUI	Yes	No

memory-safety: I mean its good and all that sdwdate uses a memory-safe language(python) but NTP is a protocol. Not sure how NTP is bound to a single programming language. The one client we mentioned before uses rust which guarantees memory safety.
secure connection by default: NTS uses TLS v1.3 . Not sure why sdwdate is being compared against NTP and not NTS.
functional over Tor: again, NTS uses TCP which can pass through a SOCKS5 proxy as is implemented by the current incarnation of Tor. Also, not sure, but are we comparing against the NTP protocol or a specific implementation?
Tor not required: what if I want to use i2p or yggdrasil to sync time over? Why does it have to be Tor?
apparmor profile: not sure why this is even included. You can write one for NTP implementations.
systemd security hardening, seccomp: same as above. You can do it for NTP/NTS implementations as well.
drop-in config folder: what's a folder? Is that supposed to be a directory? Second, what does that even mean? And third, who is writing these? The only kind of people who make this sort of mistake are people who use MS Windows more than Linux. This is official kicksecure documentation. You have Windows users writing these for the ultra secure and hardened "Linux", I'll say it again, "Linux", distro?
proxy support: again, NTS uses TCP so it supports SOCKS5 proxies as well but for whatever reason we are comparing against NTP(though whether we are comparing against the protocol or an implementation is something left to be decided by the next generation of humans)
possible to secure by default on GNU/Linux distribution level: whats the GNU/Linux distribution level? What does this even mean? You can secure it on the OS level? I mean it's software so I would hope that it would be possible to secure it on the software level.
secure: what are the criteria? Secure against what? And again, why are we comparing to NTP and not NTS?
optional GUI: again not sure why we keep zig-zagging between comparing implementations and the protocols. In conclusion, why is that table even there? What purpose does it even serve?

If we were going to base our judgement on the documentation provided on kicksecure's website, I am sorry to say that sdwdate does a very poor job but fortunately that's not all there is to it.

Now let's go take a look at the github README for the project:

At randomized intervals, sdwdate connects to a variety of webservers and extracts the time stamps from http headers (RFC 2616))

This is our first spark of brilliance.

The second spark is when we consider the practical meaning of only being able to use Tor v3 addresses. Like a wise man once said:

amateurs practice something until they can get it right. pros practice something until they can't get it wrong.

The result of using only Tor v3 addresses is that you cannot leak your real IP address no matter what happens. You either have a working Tor proxy in which case the IP address will be that of the exit node or none at all.

Now we know we definitely are dealing with a very promising solution.
'sdwdate' extracts the time stamp in the http header so we are not asking a known NTP server about the time, we are just doing a normal http request.

DISCLAIMER

Although unrelated, it is worth noting that the kicksecure docs are pretty good even if you are not planning on using kicksecure.

What to do with your DNS when ODoH's Trust-Me-Bruh Model doesn't work for you

Farzad Sadeghi — Thu, 29 Feb 2024 17:45:56 +0000

DNS.

Domain Name System.

We all use it. We all need it. But most people are still using it like its the early 2000s.

What do I mean by that?

Ye good ole UDP on port 53.

And your ISP will tell ya you don't need to worry about your privacy because they swear on boy scout honor that they don't log your DNS queries. Right ....

It's 2024. We have come a long way. We have DoH, DoT, ODoH, DNSCrypt and more.

We're going to talk about all of these for a little bit and then finally I'm going to share what I am doing right now.

Problem Statement

Plain jane DNS, i.e., sending your request using UDP without any sort of encryption, has been the norm for almost ever. Even right now that is what most people are doing.
That might have been oh-so-cool in the 80s but It doesn't fly anymore.
So we ended up with DoH and DoT. DNS-over-HTTPS and DNS-over-TLS. They are both self-explanatory. Instead of doing unencrypted requests over UDP, we do a TCP request using HTTPS or TLS.

So far so good. DoH and DoT are definitely improvements over RFC 1035 but let's take a step back and see what we are trying to defend against. Without a structure, we are not doing much more than just magic granted to us by the flying spaghetti monster.

Let's review our threat model.What are we trying to achieve here? What are the threats and who are the threat actors?
Who are we safeguarding our DNS queries against? Men-in-the-middle? Our internet provider? The authoritative DNS server that we use?

Statement: We want to have a private and anonymous DNS solution. That means:

Requirement 001:

The DNS queries shall only be viewed by the authoritative DNS server(We can up this requirement later by running our own authoritative DNS server but for now we are going to stick with our current requirement).

This naturally means that your internet provider and other men-in-the-middle are not allowed to snoop on what we are querying.

Requirement 002:

The DNS queries shall be anonymous. This means the authoritative DNS server that is getting our DNS queries shall not be able to identify the source of the query.

There is more than one way to "identify" the source of the query. We only mean the source as in the IP address that made the DNS query.

This second requirement is what ODoH is trying to solve. ODoH tries to separate the identity of the source of the DNS query from the query itself.

ODoH stands for oblivous DoH. It add an "oblivious" proxy in middle of the source of the DNS query and the server. This way the proxy can send the queries in bulk for example to try to mask who sent what when. I'm summarizing here but what ODoH is trying to do can be summarized by this:

ODoH tries to separate the identity of the source of the query from the query itself by adding a proxy in the middle

Below you can see

        --- [ Request encrypted with Target public key ] -->
   +---------+             +-----------+             +-----------+
   | Client  +-------------> Oblivious +-------------> Oblivious |
   |         <-------------+   Proxy   <-------------+  Target   |
   +---------+             +-----------+             +-----------+
       <-- [   Response encrypted with symmetric key   ] ---

ripped straight from RFC 9230

The main problem with this sort of a solution is that there is always an element of "trust-me-bruh" to the whole situation.

How can we trust that the proxy provider and the server are not colluding?

We could run our own oblivious proxy but then if it's just you and your friends using the proxy, then your proxy is not obfuscating much, is it now?

And then there is the "oblivious" aspect of the solution. How can we enforce that? How can you verify that?

Trust Me Bruh. We don't Log anything ...

We have cryptography, We have zk. I think we can do better than just blind trust.

Objectively speaking, and I'm not accusing anyone of anything so it's just a hypothetical but if someone would give me some money and they asked me to come up with a system which let's them practically monopolize access to DNS queries, I would propose ODoH.

It has enough mumbo jumbo tech jargon(end-to-end-encrypted, ...) to throw off your average layman and lul them into a false sense of security and privacy but it doesnt prevent the proxy and server provider from colluding. After all the technnical jargon, you end up with "it's safe" and "it's private" because "you can trust us".

Now we can see that DoH, DoT and ODoH are all better than baseline DNS queries over UDP without encryption but they can't satisfy both of our requirements.

Solution

Now let's talk about the solution I at the time of writing this blog post.

DoH or DoT is good enough to satisfy Requirement001 but they need something a little extra to be able to satisfy Requirement002.

For that, we use an anonymizing network like tor. DoT and DoH both work over TCP so we can use any SOCKS5 proxy here that ends up being a Tor proxy.

What I mean is you can use a the Tor running on your host or you can use ssh -L to use Tor running on a VPS. That way, your internet proviedr can't know you're using Tor at all.

With your DNS queries going over Tor, we can satisfy Requirement002.

Tor is not the only solution here but I use Tor. There is more than one anonimyzing network out there and there are protocols that do this also.

Right now we have an outline in our head:

We need to only use TCP for DNS and send everything over a Tor SOCKS5 proxy.
we will be using DoT or DoH. This will be useful in two ways. One we ensure we are using TCP for DNS which is what most SOCKS5 implementations support(even though they should support UDP because it's SOCKS5 and not SOCKS4 but that's another can of worms)

There is more than one way to do this but I have decided to use dnscrypt-proxy.

We will not be using dnscrypt for the dnscrypt protocol though you could elect to use that as the underlying DNS protocol.

dnscrypt-proxy lets's us use a SOCKS5 proxy through which the DNS queries will be sent. We will use a Tor SOCKS5 proxy here. You can choose which protocols should be enabled and which ones should be disabled.

There are two points:

one, enable the tcp only option, since we dont want to use plain jane UDP queries.
two, I have asked dnscrypt-proxy to only use DNS servers that support DNSSEC.

I recommend going through all the available options in the dnscrypt-proxy.toml file. It is one of those config files with comments so it's pretty sweet. There are quite a few useful options in there that you might care about depending on your needs.

Implementation

Right now I run dnscrypt-proxy on a small alpine linux VM. I made it fancier by running the VM on a tmpfs storage pool. Basically mine is running entirely on RAM.

I used to have dnscrypt-proxy running on a raspberry pi and had my openwrt router forward DNS queries to that raspberry pi.

There is obviously no best solution here. Just pick one that works for you.

Here you can find the vagrantfile I use for the DNS VM I use:

ENV['VAGRANT_DEFAULT_PROVIDER'] = 'libvirt'
Vagrant.require_version '>= 2.2.6'
Vagrant.configure('2') do |config|
  config.vm.box = 'generic/alpine319'
  config.vm.box_version = '4.3.12'
  config.vm.box_check_update = false
  config.vm.hostname = 'virt-dns'

  # ssh
  config.ssh.insert_key = true
  config.ssh.keep_alive = true
  config.ssh.keys_only = true

  # timeouts
  config.vm.boot_timeout = 300
  config.vm.graceful_halt_timeout = 60
  config.ssh.connect_timeout = 30

  # shares
  config.vm.synced_folder '.', '/vagrant', type: 'nfs', nfs_version: 4, nfs_udp: false

  config.vm.network :private_network, :ip => '192.168.121.93' , :libvirt__domain_name => 'devidns.local'

  config.vm.provider 'libvirt' do |libvirt|
    libvirt.storage_pool_name = 'ramdisk'
    libvirt.default_prefix = 'dns-'
    libvirt.driver = 'kvm'
    libvirt.memory = '256'
    libvirt.cpus = 2
    libvirt.sound_type = nil
    libvirt.qemuargs value: '-nographic'
    libvirt.qemuargs value: '-nodefaults'
    libvirt.qemuargs value: '-no-user-config'
    libvirt.qemuargs value: '-serial'
    libvirt.qemuargs value: 'pty'
    libvirt.random model: 'random'
  end

  config.vm.provision 'reqs', type: 'shell', name: 'reqs-install', inline: <<-SHELL
    sudo apk update &&\
      sudo apk upgrade &&\
      sudo apk add tor dnscrypt-proxy privoxy tmux
  SHELL

  config.vm.provision 'reqs-priv', type: 'shell', name: 'reqs-priv-install', privileged: true, inline: <<-SHELL
    cp /vagrant/torrc /etc/tor/torrc
    cp /vagrant/dnscrypt-proxy.toml /etc/dnscrypt-proxy/dnscrypt-proxy.toml
    #cp /vagrant/config /etc/privoxy/config
    rc-service tor start
    sleep 1
    #rc-service privoxy start
    #sleep 1
    rc-service dnscrypt-proxy start
  SHELL
end

It's pretty straightforward. We use an alpine linux VM as base. Make a new interface on the VM with a static IP and have dnscrypt-proxy receive DNS queries through that interface and IP only. I don't change the port number(53) because of certain applications(you know who you are) refusing to accept port for a DNS server's address.

You could also make it spicier by using privoxy. Maybe I make a post about that later.

How to get your SMS on IRC

Farzad Sadeghi — Wed, 31 Jan 2024 13:11:14 +0000

How to get your SMS on IRC

It's not really a continuation of the "one client for everything" post but it is in the same vein. Basically, in this post we are going to make it so that we receive our SMS messages on IRC. More specifically, it will send it to a IRC channel.

In my case this works and is actually secure, since the channel I have the SMS going to is on my own IRC network which only allows users in after they do a successful SASL authentication.

The general idea is this:

We run an app on our phone that will send the SMS to a web hook server
The web hook server has an IRC client that will send the message to the IRC channel

security considerations

SMS vs RCS

For forwarding the SMS I get on my cellphone from my cellphone to the web hook server, i use android_income_sms_gateway_webhook. This app does not support RCS(see #46).

For this to work, make sure your phone has RCS disabled unless you use another app that supports RCS.

web hook server connection

The app will be connecting to our web hook server. The ideal way I wanted to do this would be to connect to a VPN, only through which we can access the web hook server. But its android not linux. I dont know how I can do that on android so that's a no go.

Next idea is to use local port mapping using openssh to send the SMS through the ssh tunnel. While that is very feasible without rooting the phone, a one-liner in termux can take care of it but automating it is a bit of a hassle.

Currently the only measure I am taking is to just use https instead of http.

Since we are using only tls we can use the normal TLS hardening measures, server-side. We are using nginx as the reverse proxy. We will also terminate the tls connection on nginx.

We will be using pocketbase for the record storage and authentication. We can extend pocketbase which is exactly how we will be making our sms web hook.

Pocketbase will give us the record storage and authentication/registration we need. We will use girc for our IRC library. My personal IRC network wll require successful SASL authentication before letting anyone into the network so supporting SASL auth(PLAIN) is a requirement.

We can use basic http authentication using our chosen app. We can configure the JSON body of the POST request our web hook server will receive.

The default POST request the app will send looks like this:

For the body:

{
  "from": "%from%",
  "text": "%text%",
  "sentStamp": "%sentStamp%",
  "receivedStamp": "%receivedStamp%",
  "sim": "%sim%"
}

And for the header:

{ "User-Agent": "SMS Forwarder App" }

We get static cerdentials so we can only do basic http auth. We dont need to encode the client information into the security token so we'll just rely on a bearer-token in the header for both authentication and authorization.

Authentication and Authorization

In our case, the only resource we have is to be able to post anything on the endpoint so in our case, authentication and authorization will be synonimous.

We can put the basic auth cerdentials in the url:

https://user:pass@sms.mywebhook.com

Also do please remember that on the app side we need to add the authorization header like so:

{"Content-Type": "application/json"; "Authorization": "Basic base64-encoded-username:password"}

As for the url, use your endpoint without using the username and passwor in the URI.

Dev works

You can find the finished code here.

Here's a brief explanation of what the code does:

We launch the irc bot in a goroutine. The web hook server will only respond to POST requests on /sms after a successful basic http authentication.

In our case there is no reason not to use a randomized username as well. So effectively we will have two secrets this way. You can create a new user in the pocketbase admin panel. Pocketbase comes with a default collection for users so just create a new entry in there.

The code will respond with a 401 for all failed authentication attempts.
We dont fill out missing credentials for non-existant users to make timing attacks harder. Thats something we can do later.

Deployment

events {
  worker_connections 1024;
}
http {
  include /etc/nginx/mime.types;
  server_tokens off;
  limit_req_zone $binary_remote_addr zone=one:10m rate=30r/m;
  server {
    listen 443 ssl;
    keepalive_timeout 60;
    charset utf-8;
    ssl_certificate /etc/letsencrypt/live/sms.terminaldweller.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/sms.terminaldweller.com/privkey.pem;
    ssl_ciphers HIGH:!aNULL:!MD5:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_session_cache shared:SSL:50m;
    ssl_session_timeout 1d;
    ssl_session_tickets off;
    ssl_prefer_server_ciphers on;
    tcp_nopush on;
    add_header X-Content-Type-Options "nosniff" always;
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
    add_header X-Frame-Options SAMEORIGIN always;
    add_header X-XSS-Protection "1; mode=block" always;
    add_header Referrer-Policy "no-referrer";
    fastcgi_hide_header X-Powered-By;

    error_page 401 403 404 /404.html;
    location / {
      proxy_pass http://sms-webhook:8090;
    }
  }
}

version: "3.9"
services:
  sms-webhook:
    image: sms-webhook
    build:
      context: .
    deploy:
      resources:
        limits:
          memory: 256M
    logging:
      driver: "json-file"
      options:
        max-size: "100m"
    networks:
      - smsnet
    restart: unless-stopped
    depends_on:
      - redis
    volumes:
      - pb-vault:/sms-webhook/pb_data
      - ./config.toml:/opt/smswebhook/config.toml
    cap_drop:
      - ALL
    dns:
      - 9.9.9.9
    environment:
      - SERVER_DEPLOYMENT_TYPE=deployment
    entrypoint: ["/sms-webhook/sms-webhook"]
    command: ["serve", "--http=0.0.0.0:8090"]
  nginx:
    deploy:
      resources:
        limits:
          memory: 128M
    logging:
      driver: "json-file"
      options:
        max-size: "100m"
    image: nginx:stable
    ports:
      - "8090:443"
    networks:
      - smsnet
    restart: unless-stopped
    cap_drop:
      - ALL
    cap_add:
      - CHOWN
      - DAC_OVERRIDE
      - SETGID
      - SETUID
      - NET_BIND_SERVICE
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
      - /etc/letsencrypt/live/sms.terminaldweller.com/fullchain.pem:/etc/letsencrypt/live/sms.terminaldweller.com/fullchain.pem:ro
      - /etc/letsencrypt/live/sms.terminaldweller.com/privkey.pem:/etc/letsencrypt/live/sms.terminaldweller.com/privkey.pem:ro
networks:
  smsnet:
    driver: bridge
volumes:
  pb-vault:

timestamp:1706042815 version:1.1.0 https://blog.terminaldweller.com/rss/feed https://raw.githubusercontent.com/terminaldweller/blog/main/mds/lazymakefiles.md

One Chat Client for Everything

Farzad Sadeghi — Fri, 10 Nov 2023 16:25:40 +0000

One Client for Everything

Foreword
Two ways of solving this
The web app way
gui or terminal client
Matrix or IRC

Foreword

First let's talk about the problem we're trying to solve here. I want to have a unified interface into all the communication forms that I use.

I can't be bothered to have different clients open all the time. I want to have one client that takes care of all things mostly well.

Two ways of solving this

There is generally two ways one can try to solve this. number one is to just use a browser. Almost all forms of comm nowadays have a web client so basically one way of solving our problem is to a dedicated browser that has all the clients open. Mind you, there are even specialized and more lightweight browser offerings specifically geared towards this use-case but still this option is not ideal in terms of resources and the interface you're getting is not really unified.

The web app way

An example that comes to mind for this sort of solution is rambox though they are no longer offering a FOSS solution. I'm just mentioning them as an example of what's being offered out there as a ready-to-use solution.

Although this way of doing things is very resource-intensive, this is the complete way of doing things. What I mean by that is that by using the official web apps, you will not be compromising on any features that the clients offer since you will be using the official clients.

gui or terminal client

The second way of going about and solving this is to pick a very good client that supports a protocol with a lot of bridges and then bridge everything through to the app of that one protocol.

Currently there are only three protocols that have enough facilities for bridging to make this feasible. IRC, Matrix and XMPP.

I'm adding XMPP for the sake of completion but in terms of practicality XMPP doesn't have nearly as many bridges as IRC and Matrix.

So this basically narrows down our choice to either IRC or Matrix.

Now lets look at the clients that are available for these two protocols.

Matrix or IRC

The last requirement on my side is that i would rather use a unified terminal keyboard-based client than a web application client. That being said, i definitely expect to use a web client since using a terminal client on a smart phone is pretty much just pain. A lot of pain.

Unfortunately at the time of writing this post, Matrix has no terminal client that comes close to either irssi or weechat, both terminal clients originally only supporting IRC but later advertising themselves as multi-chat clients.

Also as an added bonus, starting from the next irssi release which should be irssi v1.5 one can elect not to build the IRC module at all while building irssi.

Matrix and IRC both have a rich ecosystem of bridges. Matrix has a growing fan base which means more and more bridges or tools with similar functionality will be releases for it. Contrast that with IRC where that number seems to be smaller than Matrix but still is very much alive and well.

bitlbee-libpurple

it'll be bitlbee

bitlbee is a bridge software for IRC. The distinguishing feature for bitlbee is that the way it bridges other protocols to IRC is by masquerading as an ircd.

You could also use libpurple as the backend for bitlbee (link).

libpurple has an origin story similar to libreadline. Basically it used to live inside pidgin, but later on it was turned into a library so that other applications could use it as well.

List of protocols supported by libpurple:

aim
bitlbee-discord
bitlbee-mastodon
bonjour
eionrobb-icyque
eionrobb-mattermost
eionrobb-rocketchat
facebook
gg
hangouts
hehoe-signald
hehoe-whatsmeow
icq
irc
jabber
matrix
meanwhile
novell
otr
simple
sipe
skypeweb
slack
steam
telegram-tdlib
zephyr

matterbridge

matterbridge is an everything-to-everything bridge.

Please keep in mind that with matterbridge, you don't get the full functionality of a protocol as in you get no private messages and such. You get the ability to join public chat rooms or whatever they call it in that protocol.

bridge ircds

matterircd

a mattermost bridge that emulates an ircd as the name implies.

matrix2051

another bridge that emulates an ircd, but for matrix.

irslackd

a bridge to slack that emulates an ircd.

docker compose

Here's the original Dockerfile. You can find mine here.

And here's the docker compose file I use that goes with that:

version: "3.8"
services:
  bitlbee:
    image: devi_bitlbee
    deploy:
      resources:
        limits:
          memory: 384M
    logging:
      driver: "json-file"
      options:
        max-size: "100m"
    networks:
      - bitlbeenet
    ports:
      - "127.0.0.1:8667:6667"
      - "172.17.0.1:8667:6667"
    restart: unless-stopped
    user: "bitlbee:bitlbee"
    command: ["/usr/sbin/bitlbee", "-F","-n","-u","bitlbee","-c","/var/lib/bitlbee/bitlbee.conf", "-d","/var/lib/bitlbee"]
    dns:
      - 9.9.9.9
    volumes:
      - ./conf/bitlbee.conf:/var/lib/bitlbee/bitlbee.conf:ro
      - userdata:/var/lib/bitlbee
      - /home/devi/.cache/docker-bitlbee/signald/run:/var/run/signald
      - /etc/ssl/certs:/etc/ssl/certs:ro
  signald:
    image: signald/signald:stable
    deploy:
      resources:
        limits:
          memory: 384M
    logging:
      driver: "json-file"
      options:
        max-size: "100m"
    networks:
      - signalnet
    ports:
      - "127.0.0.1:7775:7775"
      - "172.17.0.1:7775:7775"
    restart: unless-stopped
    dns:
      - 9.9.9.9
    volumes:
      - /home/devi/.cache/docker-bitlbee/signald/run:/signald
      - /etc/ssl/certs:/etc/ssl/certs:ro
    environment:
      - SIGNALD_ENABLE_METRICS=false
      - SIGNALD_HTTP_LOGGING=true
      - SIGNALD_VERBOSE_LOGGING=true
      - SIGNALD_METRICS_PORT=7775
      - SIGNALD_LOG_DB_TRANSACTIONS=true
  matterircd:
    image: 42wim/matterircd:latest
    deploy:
      resources:
        limits:
          memory: 384M
    logging:
      driver: "json-file"
      options:
        max-size: "100m"
    networks:
      - matterircdnet
    ports:
      - "127.0.0.1:7667:7667"
      - "172.17.0.1:7667:7667"
    dns:
      - 9.9.9.9
    restart: unless-stopped
    command: ["--conf", "/matterircd.toml"]
    volumes:
      - ./matterircd.toml:/matterircd.toml:ro
networks:
  bitlbeenet:
  signalnet:
  matterircdnet:
volumes:
  userdata:
  matterircddb:

timestamp:1699398469
version:0.1.0
https://blog.terminaldweller.com/rss/feed

colo - 256 terminal colors in almost all formats

Farzad Sadeghi — Mon, 30 Jan 2023 18:22:17 +0000

I spend most of my time in the terminal. Colorizing things makes detecting important pieces of information faster and more efficient in the terminal.

Colo is a little script that helps with choosing colors for the terminal applications. It can output the colors in different formats, rgb,numbers, hex, hsi so one can quickly choose the colors that one wants to use.

terminaldweller / colo

A simple color script which print the 256 terminal colors in rgb,hex,numbered, ...

colo

A simple script that prints out the 256 terminal colors in different formats.
It can print the numbers, the hex value, rgb,hsi and the ansi escape sequence.

colo --help
usage: colo [-h] [--ansi] [--hsi] [--rgb] [--number] [--name] [--hex]

optional arguments:
  -h, --help  show this help message and exit
  --ansi      bool
  --hsi       bool
  --rgb       bool
  --number    bool
  --name      bool
  --hex       bool

How to get

pipx install colo

Also since this is a very small package without any dependencies you could always use pipx run without "installing the package":

pipx run colo

View on GitHub

I wrote it to make my life in the terminal easier.

I guess the "feature" that colo has is that supports printing the colors in different formats and I can get it with a simple pip install colo on a new machine/vm/container.

Hope someone else finds it useful.

keyoxide

Farzad Sadeghi — Mon, 30 Jan 2023 14:52:42 +0000

openpgp4fpr:9E20464F1CCF3B103249FA93A6A0F5158B3881DF

DEV Community: Farzad Sadeghi

Make the zsh Prompt Go Faster

How to make the zsh prompt faster

Plugin Precompilation

Caching

Eval Caching

Ye Old Result Caching

Async Functions

A Disposable Firefox Instance

Making a Disposable Firefox Instance

Requirements and Goals

Implementation

Isolation and Sandboxing

IP Address Leak prevention

Non-Persistance

Implementing the Kill-Switch Using UFW

Sudo-less NTFS

The Vagrantfile

Provisioning

Interface

Timezone

haveged

QEMU Sandbox

CPU Pinning

No Passthrough

Launcher Script

Notes Regarding the Host

Docker, Linux, Security. Kinda.

Docker, Linux, Security. Kinda.

Base Image

Dokcer Runtimes

gvisor

kata

Capabilites and Syscalls

capabilities in the wild

syscall Filtering in the wild

Namespaces

Namespaces in the Wild

SBOM and Provenance Attestation

Example

Further Reading

After NTP Comes NTS. After NTS comes sdwdate.

After NTP Comes NTS. After NTS comes sdwdate.

Implementation

What comes after NTS

sdwdate

DISCLAIMER

Links

What to do with your DNS when ODoH's Trust-Me-Bruh Model doesn't work for you

Problem Statement

Solution

Implementation

How to get your SMS on IRC

How to get your SMS on IRC

security considerations

SMS vs RCS

web hook server connection

Authentication and Authorization

Dev works

Deployment

One Chat Client for Everything

One Client for Everything

Table of Contents

Foreword

Two ways of solving this

The web app way

gui or terminal client

Matrix or IRC

bitlbee-libpurple

matterbridge

bridge ircds

matterircd

matrix2051

irslackd

docker compose

colo - 256 terminal colors in almost all formats

terminaldweller / colo

A simple color script which print the 256 terminal colors in rgb,hex,numbered, ...

colo

How to get