DEV Community: Daniel Ziltener

BastilleBSD Jail Manager - Containers Done Light

Daniel Ziltener — Fri, 07 May 2021 12:28:46 +0000

What is BastilleBSD?

Bastille is an open-source system for automating deployment and management of containerized applications on FreeBSD.
-- bastillebsd.org

BastilleBSD is, at its core, a set of scripts to manage your FreeBSD jails. It handles things like creation, starting, stopping, port forwarding, and using and composing templates. It allows you to manage both "normal" and VNET jails, and if you use ZFS as your file system, it helps you to take snapshots from running jails and lets you restore them, even on other machines, too. And it's in the pipeline to even support Linux as a jail guest.

What are jails?

Jails are a FreeBSD technology that started back in the 90s. The developers needed something more powerful and safe than a simple chroot to isolate web applications and do things like having multiple different versions of PHP running on the same server. If you want to know a bit more about the history behind it, see Jails - High value but shitty Virtualization. A jail is a virtual environment that takes a directory of the host's file system as its root, and uses the host's kernel, which makes it a very lightweight solution.

You can actually use jails without a tool like BastilleBSD, they're built-in to FreeBSD, like the jails-inspired LXC is in the Linux kernel. But while that is possible and sometimes done, it is just more convenient to have tools to manage it and hide away some of the low level verbosity.

What will this series teach?

Starting next week, we will dive into setting up the host machine, setting up different kinds of jails, how to create and use templates, how to test these templates, and how to snapshot/restore jails.

Updated: Write and auto-deploy daemons on FreeBSD

Daniel Ziltener — Wed, 28 Apr 2021 16:28:54 +0000

Goal

Our goal is to write an rc.d file that allows us to start anything as a daemon, and an accompanying settings file, and to get to know the commands to use for automated deploy.

Settings File

We will start with the settings file. Technically you can do in there whatever you want; if you have any sanity left, you will simply use it to set environment variables. This makes it easy to follow the third rule of the Twelve-Factor App. And it really is that simple, you just add export statements, and make "dummy values" you will replace in your CI/CD script using sed, or replace them with variables and use RENDER in Bastille:

export HTTP_PORT=3000
export POSTGRES_IP=SED_PGSQL_IP
export POSTGRES_USERNAME=SED_PGSQL_USER
export POSTGRES_PASSWORD=SED_PGSQL_PASS

This file has to be stored with the same name as your actual service file, but under /usr/local/etc/rc.conf.d/. The RC system will auto-detect it.

Daemon file

This one is going to be a little bit longer. Let's first take a look at the whole script:

#!/bin/sh

# PROVIDE: location_service
# REQUIRE: LOGIN DAEMON NETWORKING
# BEFORE: some_other_service
# KEYWORD: shutdown

. /etc/rc.subr

name=location_service
rcvar=${name}_enable
pidfile="/var/run/${name}.pid"
pidfile_child="/var/run/${name}_jvm.pid"
logfile="/var/log/${name}.log"
location_service_chdir="/usr/local/share/location-service"

command="/usr/sbin/daemon"
start_cmd="location_service_start"
procname="daemon"

load_rc_config ${name}
: ${location_service_enable:=no}

location_service_start() {
    /usr/sbin/daemon -r -f -P ${pidfile} -p ${pidfile_child} -t ${name} -o ${logfile} /usr/local/bin/java -jar location-service.jar
}

run_rc_command "$1"

Main variables

There is some magic going on with the variable names, but that aside, it is a pretty standard shell script file. In this case, we are working with a location service that is to be run by the JVM, but it really doesn't matter - you can run literally anything as a service this way.

PROVIDE is the name of the daemon; this is placed here again so potentially other daemons can REQUIRE it.
REQUIRE are the daemons/targets that should be loaded before this one. The values should be quite self-explanatory.
BEFORE is the inverse of REQUIRE; use it to tell the rc system to start this daemon before the mentioned one(s).
KEYWORD can be either nojail (will be ignored if inside a jail), nostart (automatic startup procedure will ignore the script. In conjunction with the shutdown keyword, this can be used to write scripts that do something only at system shutdown), and shutdown (service needs to be stopped before system shutdown).
. /etc/rc.subr loads subroutines that enable the "magic" to run the service.
rcvar is the flag that you can set to enable or disable your service in the rc.conf file.
The two pidfiles are holding the process id of the daemon and the actual process, so the system can keep track of it.
and the location_service_chdir (or whatever_you_name_it_chdir) sets the working directory.

The Daemon command

FreeBSD ships with a program aptly called daemon, and we are using it to run our normal program as one. We thus tell the rc system to use /usr/sbin/daemon as the main command, and tell it that the process started by it - procname - will be called "daemon". We also tell it to run a custom starting function, start_cmd, which in this case is called "location_service_start".

Configure the daemon

We configure the daemon by setting the flags when starting it. We have the following flags:

-r: Restarts the process if it dies
-f: Redirects standard input/output and standard error to /dev/null
-p: the pidfile for the process you want to have daemonized
-P: the pidfile of the daemon process itself - that is what the rc system needs
-o: the log file to log to
-t: gives the process a name (optional)
-u: (not used here): the user to run the process as

This service file has to be stored under /usr/local/etc/rc.d and be made executable.

Automatic deployment

We almost have it - the only thing missing are the commands for your CI/CD. It is really simple once you have both files at their respective locations.

You can set the config variables using sed -i.bak "s/SED_PGSQL_IP/${your_ci_var}/g" /usr/local/etc/rc.conf.d/location_service. Remember, if your variable contains the "/" character, you can use something else for sed, whatever character follows the "s" will be the separator.

Enable the service using sysrc -f /etc/rc.conf "location_service_enable=YES".

And finally, don't forget to actually (re-)start it using service location_service restart.

Have fun!

A Must-Read History Lesson For Devs: "The Eternal Mainframe"

Daniel Ziltener — Mon, 26 Apr 2021 10:01:16 +0000

Many younger developers run from hype to hype and are completely imprinted on their marketing overlords. The dev community grew too quickly to be contained by its own culture and history, as it - in my opinion - should have been. Because there are important lessons to be learned from the past that spans decades. "The Cloud" isn't a 21st century invention. Neither are "thin clients" and having all your data in said cloud. One of the best articles about this is the essay "The Eternal Mainframe"by Rudolf Winestock.

I'd love to make dev.to aware of this piece of writing and like to hear your opinions on it!

You can find it here.

How to share your dev box with others in 5 minutes - Caddy edition

Daniel Ziltener — Wed, 21 Apr 2021 14:08:23 +0000

Sometimes you want to share what you are working on right now with your coworkers without either calling them over or pushing and waiting until it is deployed on the staging server. Then you take a look and find a bunch of complicated tools, proprietary helpers like ngrok, and a whole lot of similar stuff.

Yet all you need is a reverse proxy and SSH. And here's how it's done, by example of Caddy.

Main Server Configuration

Add something like that to your Caddy file:

mymachine.dev.example.com {
  reverse_proxy https://localhost:11010 {
    transport http {
      tls_insecure_skip_verify
    }
  }
}

Dev Box Configuration

On the dev box, make sure the webserver either reacts to the domain set up on the main server, or to every request in general. Since this is Caddy, I had to also disable the automatic certificate generation - I generate self-signed certs -, and the HTTPS redirect.

{
  auto_https off
}
localhost, devbox.local, mymachine.dev.example.com {
  tls /etc/ssl/localcerts/bundle.crt /etc/ssl/localcerts/server.key
  # Rest of your config
}

And finally, as the last step, open up the SSH tunnel and reverse-forward the port:

/usr/bin/ssh -N -T -R 11010:localhost:443 user@example.com

N means "don't execute a remote command", T means "don't allocate a terminal", and R means to forward the port in reverse - from the target to the local machine.

That is all!

RC Service File

We're using FreeBSD machines set up using Bastille at work, so I created a simple RC file so the forwarding is done automatically at every devbox start:

#!/bin/sh

# PROVIDE: sompani_tunnel
# REQUIRE: LOGIN DAEMON NETWORKING
# KEYWORD: shutdown

. /etc/rc.subr

name=sompani_tunnel
rcvar=${name}_enable
pidfile="/var/run/${name}.pid"
pidfile_child="/var/run/${name}_jvm.pid"
logfile="/var/log/${name}.log"
sompani_tunnel_chdir="/usr/local/share/location-service"

command="/usr/sbin/daemon"
start_cmd="sompani_tunnel_start"
procname="daemon"

load_rc_config ${name}
: ${sompani_tunnel_enable:=no}

sompani_tunnel_start() {
    /usr/sbin/daemon -r -f -P ${pidfile} -p ${pidfile_child} -t ${name} -o ${logfile} /usr/bin/ssh -N -T -R 11005:localhost:443 sompani-live
}

run_rc_command "$1"

What is Functional Programming?

Daniel Ziltener — Wed, 20 Jan 2021 22:00:13 +0000

So you are interested in this fancy thing called functional programming. Good! Because that is what I want to teach you. Functional programming has been around for many decades, but only quite recently became the "hot new thing". And due to that, unlike so many other "hot new things" that happen to be bloated, overly complicated solutions to problems nobody really has, it is a quite simple, straightforward approach.

But why functional programming?

Functional programming is a paradigm that emphasizes simplicity. Unlike object oriented programming, it separates data and logic, and in its pure form tries to keep logic and mutable state apart as good as possible, facilitating reasoning about the code, debugging, and testing.

But today, we won't actually write code yet. Today, we want to get you into the mindset and thinking patterns to make the first lines of code as easy as possible for you!

No really, what is it?

Okay, enough with the rambling about the good old days. What actually is functional programming? Well, what the community generally understands as functional programming these days is often synonymous with "pure functional programming". It is a programming paradigm that takes a lot of ideas from how mathematics work. But don't worry, you don't need any kind of degree or to be especially talented in maths to learn it!

...Oh, looks like we got a visitor! It's Vodblob McSteele. Vodblob is an absolute expert in multiple areas of programming and got famous through multiple books and courses, and decided to help me out in this one.

The alien adjusts its glasses Nice to meet you! I'm Vodblob, I dedicated my life to studying all programming paradigms in existence. Some joke there is one for every eye I have, but so far there are still less. But today I am here to teach you functional programming! I can give you a few examples that make you think like a functional programmer in no time - without a single line of code to write! It takes a little slate and a chalk pen out of its pocket and draws two separate circles on it.

Vodblob points at the left circle The base idea is that you work with functions that take one (or more) argument, give a value back, and do nothing else. We call those pure functions, and for our explanation, they are all inside that left circle. A real life example for this would be your water heater in the office: you put in a liter of cold water, wait a bit, and get a liter of boiling water to pour into your tea cup. Always the same process. Unless the power is out - then you get nothing. But that applies to programming, too!

But the problem with that approach is that when you only have pure functions, you get a program that takes an input and produces an output and some hot air - and nothing else. Usually you need more than that, because usually, you want that program to interact with you or with some changing data.

It points at the right circle For that, we have impure functions. Those do more than the pure ones discussed above: they might give you a different result each time, and they might even change the program state, display something on your screen, or other things in the program's environment, like updating file contents! A bit like eating chips: each time you reach into the bag, you get out a slightly different chip, you modify the bag's state, and at some point you might even reach in and find - nothing!

As you've seen, I drew two distinct, separate circles to represent these groups, and for good reason. We want to keep those two categories separate unless interaction is absolutely necessary, because it is much easier to reason about, test, and debug pure functions and chained-together pure functions.

Vodblob takes a sip of tea out of its cup and pauses for a few seconds. It then takes the chalk pen again and draws a circle that encompasses both other circles.

Lastly, functional programming is data centric. Your program state tends to be just data structures - things like numbers, text strings, lists, and key-value maps -, and you use your vast collection of functions to extract, modify, and change the state, and make decisions based on these structures. This is the common denominator between the pure and impure functions, and their way of communicating. It is the background noise of the little universe the pure and impure bubbles live in.

Functions themselves can be data, too - you can store functions inside these data structures, and pass them as arguments to other functions (which then become higher-order functions). This last part is what allows you to do elegant data manipulation by simply combining functions you already have, and have them be applied to all kinds of data. This is very powerful!

Recap

Vodblob harrumphs, thinks for a few seconds, and continues.

Alright. I think this is it for today. It might be, purely looking at the number of words, a short lesson, but it is a lot of stuff to digest. To recap:

Pure Function: A function that, given X, always returns Y and does nothing else. Example: 2+2=4.
Impure Function: A function that, given X, might give changing results and changes the program state or the environment. Example: display text, write file, increase counter.
Data Structure: The way data is kept in memory. This could be text, numbers, lists, key-value-maps, ...
Program State / Environment: The former is mutable data inside your program, the latter is everything from stuff in databases you program can access to what's shown on screen.
Data centric: your program state is just data structures, and your program uses that data to make decisions and alter said data.
Higher-order functions: Functions that take other functions as argument to operate on data.

What now?

You might not have 100% understood a few things, but that is fine. You got a first insight, some food for thought, and the rest will come naturally as we progress.

Next time, we will look at a functional programming language and start writing simple code. Until then, your homework is to look at the recap and let your mind run wild. What can you imagine doing with the possibilities listed above? What data structures can you imagine existing? Be creative! It doesn't have to be realistic, after all you don't know yet what the languages actually work like.

Vodblob wipes his slate clean, packs his backpack, and shakes your hand.

Oh to be a blank slate again, to learn it all again with fresh eyes... Well, in any case, we'll hopefully see each other again soon!

The tech to go from zero to shipping in just 3 months

Daniel Ziltener — Thu, 30 Jul 2020 01:23:44 +0000

As we all know, there are two things no startup ever has enough of: time and money. The constraints are even tighter when you are doing a spin-off as a startup, having to maintain the existing project and starting a new one. We are a team of three people, with me as the sole developer, and we started three months ago completely from scratch. On the tech side, I could mainly rely on almost a decade of Clojure experience, and on a framework I haven't used before called Fulcro. Here is what we used:

Hosting: Google Cloud Services and Cloudinary

I am generally not that much of a fan of clouds, but as a startup, we have a lot of free credits available for Google Cloud, so we opted to just spin up a few machines. I provisioned three FreeBSD machines using Ansible, a build server, a dev server and a production server, and created a managed PostgreSQL instance. The deployment from the build to the dev and production servers is being done using the gcloud command line tools, as I didn't want to rely on IP addresses. It was quick and hassle-free to set all this up.

Apart from that, we use Cloudinary as our binary storage for PDF and image files used in our product.

Database: Datomic on PostgreSQL

Datomic is a sophisticated database that handles structuring the data with flexible schemas, uses a Datalog flavour as query language, is tightly coupled with Clojure, and stores the data in time-key-value-action tuples. It marks entries as retracted instead of deleting and keeps track of every change, timestamping it, allowing for "time travel" without any additional software or logic. It is available as a cloud offering on AWS, or as an On-Prem offering for you to self-host. We went with the free single-server license for Datomic Pro for self hosting, using PostgreSQL as storage backend.

Programming Language: Clojure

Clojure, and its sibling ClojureScript, is a modern, functional Lisp. It provides concise code, minimal syntax, and immutable and flexible data structures. The macro system allows for extensibility, and since it compiles to both the JVM and JavaScript, it is possible to write pure-Clojure full-stack programs that even share code between frontend and backend. Dependency handling and compilation is handled using the excellent Shadow-CLJS tool that also supports NPM packages for the frontend.

Framework: Fulcro

If you have spent any amount of time in the Clojure world, you know that Clojurians usually dislike using frameworks, instead insisting on mixing-and-matching libraries to form the application. Frameworks tend to feel constraining and lock you into a cage. But Fulcro and its Rapid Application Development addon is a quite different beast. While there is a certain core you'll want to use in every project, almost everything about it is optional. For example, you can write your frontend in something completely different. But we went with full-stack Fulcro, which is using React underneath to keep data in sync and allows us to use React components whenever we want to.

RAD comes with powerful functionalities to validate and even generate the database schema (currently, Datomic and SQL backends are available), and based on that is able to generate reports and forms even with subforms to quickly have a way to manage all the data.

To top it all off, there is an addon for Chromium called Fulcro Inspect that allows you to see the entire normalized in-browser database that keeps the program state, the transactions made and even allows you to run queries.

Planning: Trello and GitLab Issues

We kept and keep track of what is going on and what we are doing using Trello and GitLab Issues.

For that we made two Trello boards: one for the product planning that is mostly for the higher-level picture and groups features the way it makes sense for the business side of things, and a development board using the Kanban Template where I create smaller tasks in a way that makes sense to me as a developer, and I link the cards on the development board to the ones on the product board so we don't lose track of each other.

And finally, we are using GitLab Issues for exactly what you'd expect - to keep track of and fix bugs in features that are already developed.

Development: Spacemacs ;)

At least for now, all code so far of this project has been written in Emacs using the cutting-edge branch of Spacemacs. It provides an excellent Clojure environment thanks to CIDER, and has ways to integrate GitLab (using the Magit-extension Forge) and Trello (using org-trello). After years of keeping my own Emacs configuration up-to-date I finally made the actually quite small step over to Spacemacs-land, because my config already gravitated towards it anyway, and so far I enjoyed every second with it.

This last thing probably didn't make it significantly faster, and there are great Clojure addons for Vim (Fireplace), IntelliJ (Cursive-Clojure) and VSCodium (Calva), but you know Emacs people - they just have to mention they're using Emacs. :)

Write and auto-deploy daemons on FreeBSD

Daniel Ziltener — Tue, 14 Jul 2020 19:54:31 +0000

Goal

Our goal is to write an rc.d file that allows us to start anything as a daemon, and an accompanying settings file, and to get to know the commands to use for automated deploy.

Settings File

export HTTP_PORT=3000
export POSTGRES_IP=SED_PGSQL_IP
export POSTGRES_USERNAME=SED_PGSQL_USER
export POSTGRES_PASSWORD=SED_PGSQL_PASS

This file has to be stored with the same name as your actual service file, but under /usr/local/etc/rc.conf.d/. The RC system will auto-detect it.

Daemon file

This one is going to be a little bit longer. Let's first take a look at the whole script:

#!/bin/sh

# PROVIDE: location_service
# REQUIRE: LOGIN DAEMON NETWORKING
# KEYWORD: shutdown

. /etc/rc.subr

name=location_service
rcvar=${name}_enable
pidfile="/var/run/${name}.pid"
pidfile_child="/var/run/${name}_jvm.pid"
logfile="/var/log/${name}.log"
location_service_chdir="/usr/local/share/location-service"

command="/usr/sbin/daemon"
start_cmd="location_service_start"
procname="daemon"

load_rc_config ${name}
: ${location_service_enable:=no}

location_service_start() {
    /usr/sbin/daemon -r -f -P ${pidfile} -p ${pidfile_child} -t ${name} -o ${logfile} /usr/local/bin/java -jar location-service.jar
}

run_rc_command "$1"

Main variables

. /etc/rc.subr loads subroutines that enable the "magic" to run the service.
rcvar is the flag that you can set to enable or disable your service in the rc.conf file.
The two pidfiles are holding the process id of the daemon and the actual process, so the system can keep track of it.
and the location_service_chdir (or whatever_you_name_it_chdir) sets the working directory.

The Daemon command

Configure the daemon

We configure the daemon by setting the flags when starting it. We have the following flags:

-r: Restarts the process if it dies
-f: Redirects standard input/output and standard error to /dev/null
-p: the pidfile for the process you want to have daemonized
-P: the pidfile of the daemon process itself - that is what the rc system needs
-o: the log file to log to
-t: gives the process a name (optional)
-u: (not used here): the user to run the process as

This service file has to be stored under /usr/local/etc/rc.d and be made executable.

Automatic deployment

We almost have it - the only thing missing are the commands for your CI/CD. It is really simple once you have both files at their respective locations.

Enable the service using sysrc -f /etc/rc.conf "location_service_enable=YES".

And finally, don't forget to actually (re-)start it using service location_service restart.

Have fun!

Adding custom toolbar buttons to Fulcro 3 Workspaces cards

Daniel Ziltener — Mon, 08 Jun 2020 17:55:32 +0000

Recently I came across the problem that I wanted to use Workspaces' toolbar functionality - but with Fulcro 3 cards. There is no official documentation of doing this, so I ended up taking the standard Fulcro 3 card as a base and creating this custom card.
I used the following imports:

(ns util
  (:require [com.fulcrologic.fulcro.algorithms.merge :as m]
            [nubank.workspaces.core :as ws]
            [nubank.workspaces.model :as wsm]
            [nubank.workspaces.ui :as ui]
            [nubank.workspaces.ui.core :as uc]
            [com.fulcrologic.fulcro.dom :as dom]
            [nubank.workspaces.card-types.fulcro3 :as ct.fulcro]))

And created this card:

(defn custom-toolbar-card [fulcro-class & {::keys [toolbar-items card-width card-height] :as props}]
  {::wsm/card-width card-width
   ::wsm/card-height card-height
   ::wsm/align {:flex 1}
   ::wsm/init
   (fn [card]
     (let [fulcro-card (ct.fulcro/fulcro-card-init card {::ct.fulcro/wrap-root? true
                                                         ::ct.fulcro/root       fulcro-class})
           card-id     (::wsm/card-id card)
           app         (::ct.fulcro/app fulcro-card)]
       (assoc fulcro-card
              ::wsm/render-toolbar
              (fn []
                (dom/div
                 (mapv #(uc/button {:onClick ((:fn %) app)} (:text %)) toolbar-items)
                 (uc/button {:onClick #(ct.fulcro/inspector-set-app card-id)}
                            "Inspector")
                 (uc/button {:onClick #(ui/restart-card card-id)}
                            "Restart"))))))})

Note that I added the standard card-with and card-height parameters as well - you can leave those out - and added the default Fulcro 3 buttons.

Buttons can now be added in the format {:fn (fn [app-of-the-card] ...) :text "Button Text")}.

You can use it like this:

(ws/defcard login-card
            (util/custom-toolbar-card
             account-forms/LoginForm
             ::util/card-width 6
             ::util/card-height 16
             ::util/toolbar-items
             [{:fn (fn [app]
                     #(m/merge-component! app account-forms/LoginForm {:ui/error "The credentials you entered are incorrect."}))
               :text "Trigger error"}]))

In this example, I made a card for a login form (account-forms/LoginForm) which is a Fulcro 3 component. I added a toolbar item with the button text "Trigger error" that, when clicked, calls the fn, which is given the fulcro app of the card, updating the LoginForm to set a login error text.