DEV Community: José Miguel Parrella

Transparency and user agency as principles for distributing and consuming open source software packages

José Miguel Parrella — Tue, 08 Jun 2021 19:19:47 +0000

More people are building more kinds of software that is consumed more often and in more places than ever, and the likelihood of that software being open source or having an open source dependency is very high compared to just a decade ago.

These forces (software publishers, software kinds and software use cases) along with network evolution, how software is monetized, and how people and organizations use technology put pressure on the software distribution technology and techniques such as software packages, package managers, and software repositories.

I've been researching Linux and open source package management for a while and I'm very excited about many of those technologies and their applications, from distri and systemd/mkosi to libostree and spack. Unsurprisingly, many of these are prompting us to revise how we think about distributions.

Dirk RIEHLE

@dirkriehle

What is an open source distribution?

10:25 AM - 28 May 2021

Inspired by The Laws of Software Installation, here's an attempt to elaborate on the "contract" between software authors and users, particularly in open source software where the volume and composition possibilities create a vast and complex ecosystem of its own.

Our expectations of transparency for software packages have evolved. What used to be minimal metadata regarding the publisher, description of the software, and signatures has been enhanced with things like licensing information and is fully evolving into Software Bill of Materials: the Nutrition Facts label that allows every software package to explain how it came to be, where it comes from, who and what was involved in making that happen.

In an open source world, consent builds upon transparency. We have been influenced by certain use cases, such as mobile apps, where "permissions" (both stated by the developer and enforced by the platform) have become the expectation.

All of this means different things to different people: some users might want to know if the package lays itself out in the filesystem according to FHS, others need to know if the application will self-update or install publisher certificates, if it needs network egress to ship telemetry at runtime, if it pulls additional dependencies out-of-band to a local cache, if it changes environment variables, starts automatically on boot, need to run in a privileged container, ships with and relies on LSM integration and more.

In general, we lack a standardized mechanism to carry behavioral ("permissions") information in a package, let alone prove them or make policy decisions such as allowing a pull into a build or in production based on the organization's choices around, say, network egress or vendoring.

Interfaces are another interesting aspect of software packages that can be easily overlooked. Beyond the most basic operations (install, remove and maybe update) package managers don't share semantics let alone have a harmonized approach to automation: allowing themselves to be handled by the system in user-defined ways.

Hooks, triggers and other artifacts are regularly abused to achieve certain automation goals such as preseeding configuration or performing certain provisioning steps right after install, sometimes overreaching in terms of administrative privileges usage with broad security implications.

Package managers inevitably grow in sophistication to meet certain user needs (see dpkg triggers) but in general, whether you are a software publisher, integrator, IT organization, a developer or a system administrator you are expected to learn the inner workings and nuances of each package system instead of relying on standardized interfaces, which results in wildly varying user experiences and thinly spread resources.

A good way to illustrate the complexity of this today is parsing through the several thousand lines of Ansible code devoted to dealing with APT or DNF, or how basic operations such as listing Linux packages or Go modules are handled.

These two attributes, transparency and interfaces, are only worth investing in if they give users agency. Of course, there are other attributes that make a high quality software package, particularly a judicious use of resources and testing for alternative configurations while providing sensible defaults.

Finally, I think the jury's out for at least two topics: vendoring and component duplication, and automatic upgrades. While there will always be very good arguments against both in several use cases I expect more of both as a result of growing software "kinds" in a world where there isn't a lot of contention for storage or networking.

It seems there's only one thing we love more than the open source components that we use and build upon daily, and that's the package manager that we use to acquire those components. There's so much exciting activity happening in this space, that is easy to focus on technical differences such as package formats or installation mechanics when looking at package systems. With this post, I've tried to make the case for considering user agency, quality, transparency and interfaces as key tenets of the "contract" between software publishers, distributors and users.

Summary

Growth of software publishers, software kinds and software use cases along with changes in software monetization and network evolution are changing what users expect from software packages
While fun and necessary innovation is happening, package formats, implementation choices and the intricacies of how a software package is installed aren't necessarily where we'll meet future user expectations or how we give them agency
Packages should not only describe what the software is, but how it was made and where it came from; packages should also describe their behaviors: metadata becomes a contract
Package operations (across the entire lifecycle and well beyond installation) should be automatable via interfaces and said interfaces should allow for user-defined options: alternatives, configurations, installation paths, etc.
Quality remains critical: packages must guarantee their removeability, keep promises across the lifecycle and provide users with control on mutating logic (e.g., triggers) and resource usage, noting that user's attention is also a finite resource that increases security risk when depleted

Are you a software publisher or distributor? What steps are you taking to make your packages more transparent and give users more control? Are you a developer? I always love to hear new things that people have learned about software packaging and distribution. Input is always welcome!

Dapr, the hard way

José Miguel Parrella — Thu, 13 Feb 2020 00:23:31 +0000

This weekend I wanted to catch up with Dapr, the Distributed Application Runtime. As a sysadmin by trade, I have little knowledge of application model theory and limited hands-on experiences with some of the challenges that Dapr seeks to address, so I was a little bit more inclined to figure out how things were put together, which proved useful for me to understand the value of Dapr.

TL;DR: what the Dapr software does is stand up a localhost endpoint next to your application. This endpoint, which speaks HTTP and gRPC, provides a standard, simple API (the Dapr spec) through which your application and any other application that interacts with yours can invoke methods, store and retrieve state, publish and subscribe to events, and more.

Why dapr?

The reasons why you'd want such a thing are best described in the website and the Azure Fridays two-part interview with the team, but I'll give it a shot.

At first glance,none of the things Dapr can do are necessarily new to developers. Everyone consumes events and stores state. What's new is the ability to enable large, complex, distributed applications without a lot of focus on the (traditionally hard) implementation details.

Historically, ways through which developers bring things like session stores into their applications include:

1) Framework facilities
2) Middleware
3) External services

Examples of framework facilities that can achieve what Dapr does include the session management capabilities in Laravel for PHP developers, or things like Catalyst::Plugin::Session for Perl developers.

Those facilities offer the most idiomatic option for developers but they come at a cost: it's harder to part ways with the framework, in some cases it isn't ready for a distributed world (the most simplistic facilities are instance-bound, live in memory, etc.) and even for the ones that are more sophisticated, someone still has to write and maintain the code that does state persistence, interact with the backends, etc.

Middlewares are a generalization of this. They tend to be less language dependent, but they also tend to require more specialized operator knowledge. Some have rich language bindings, like the Java bindings for many Apache projects and Java-adjacent middlewares. With externally managed services, the issue tends to be around the need to import and keep track of SDKs, which can introduce challenges from poor developer experience to severe API lag, missing documentation and more.

Imagine a world where your application can be written in any language (or many) and where you can always count on a local endpoint that offers you the building blocks to make your application highly decoupled and distributed (including in Kubernetes).

This would be without taking any SDK dependencies (and perhaps even dropping some!) and just using HTTP/gRPC and JSON. All that you need to know as a developer is to know how to target the Dapr spec.

And even if ultimately all state must be persisted somewhere, and/or a pub/sub server must be somehow available, you can delegate all of these decisions to someone else in your team, and those things can change from underneath you without you changing your application: want to use a managed CosmosDB instance instead of a bring-your-own Redis one? Dapr can do that.

Dapr and `podman`

I'm running Debian sid and I wanted to run dapr in my local machine. I followed the installation instructions (the install script basically fetches the latest release from GitHub) and ran dapr init. This failed (at least in version 0.3.0) because I don't have docker in my machine. Since I wanted to use podman, I proceeded to take a look at the code and see how I could make that happen.

In standalone mode (i.e., not Kubernetes) dapr init does three key things: check that docker is installed, fetch the daprd binary and prepare the runtime. So I went ahead and changed the test logic for testing and calling docker (see #257) in my cloned repo, and rebuilt dapr with go build.

Once I ran dapr init with the resulting binary, I had local containers running Dapr's placement service and Redis as the default state store in standalone mode:

$ dapr init 
⌛  Making the jump to hyperspace...
✅  Downloading binaries and setting up components...
✅  Success! Dapr is up and running

$ podman ps | grep dapr
231a26a31000  docker.io/library/redis:latest   redis-server          4 minutes ago  Up 4 minutes ago  0.0.0.0:6379->6379/tcp    dapr_redis
f49eb9753492  docker.io/daprio/dapr:latest                           4 minutes ago  Up 4 minutes ago  0.0.0.0:50005->50005/tcp  dapr_placement

Dapr also fetches daprd which is the component that actually starts with each application instance. The Redis container supports it (for storing state) and the placement container is used for actors -- more on this later.

Dapr in action

The easiest way to see Dapr in action in standalone mode is to run one of the samples (don't forget to install sample dependencies with npm install), for example:

$ dapr run --app-id alpha --log-level error --app-port 3000 -- node app.js
ℹ️  Starting Dapr with id alpha. HTTP Port: 34651. gRPC Port: 34655
✅  You're up and running! Both Dapr and your app logs will appear here.
== APP == Node App listening on port 3000!

As you can see, I start my Node.js application as an argument of dapr, and I use --app-port to bind daprd to the application port. What I can do now is start querying a Dapr endpoint via localhost:34651, calling my Node.js application methods from dapr, persisting and querying state, and I can do this over HTTP or gRPC, with the dapr CLI or a tool like curl:

$ dapr invoke --app-id alpha --method neworder --payload '{"data": { "orderId": "42" } }'
$ curl http://localhost:45651/v1.0/invoke/alpha/method/order ; echo
{"orderId":"42"}

The interesting thing is when you read the code of the /neworder method in my application, it also calls Dapr for persistence (and yes, daprPort is passed as an environment variable) which makes the entire programming model decoupled and simple:

app.post('/neworder', (req, res) => {
    const state = [{
      key: req.body.data,
      value: req.body.data.orderId
    }];

    fetch(stateUrl, { // http://localhost:${daprPort}/v1.0/state/${stateStoreName}
        method: "POST",
        body: JSON.stringify(state),
        headers: {
            "Content-Type": "application/json"
        }
    }).then((response) => {
    ...

And this also means that instead of curl or a CLI, I could have written a consumer in any other language. In fact, Dapr's components extend to pub-sub, input/output bindings which are great for triggers and other events, state and secret stores, tracing exporters, OAuth authorization and, perhaps dramatically, Virtual Actors support, helping abstract the implementation details of things such as concurrency control.

Check out all the supported integrations in the components-contrib repo!

Beyond standalone `daprd`

Of course, things get far more interesting when you deploy this to Kubernetes. A simple dapr init --kubernetes will use your current kubeconfig to deploy the Dapr elements to your cluster. You can deploy a redis chart, or use a managed Redis service or other state store. I end up with a Dapr-enabled cluster:

$ kubectl get pods  | grep dapr
dapr-operator-68f7dcb454-zjhdj           1/1     Running   0          4d1h
dapr-placement-6d77d54dc6-ww5rb          1/1     Running   0          4d1h
dapr-sidecar-injector-86d6ccf956-7r85k   1/1     Running   0          4d1h

The distributed calculator sample is a great way to see Dapr in action in a Kubernetes cluster. You'll get a React-based calculator that can persist state to Dapr and that calls services in multiple languages for each operation, all over Dapr. Once you deploy this sample, you'll end up with a bunch of services:

$ kubectl get svc
NAME                        TYPE           CLUSTER-IP     EXTERNAL-IP      PORT(S)            AGE
addapp-dapr                 ClusterIP      10.0.66.128    <none>           80/TCP,50001/TCP   4d1h
calculator-front-end        LoadBalancer   10.0.15.166    158.51.155.210   80:30366/TCP       4d1h
calculator-front-end-dapr   ClusterIP      10.0.79.13     <none>           80/TCP,50001/TCP   4d1h
divideapp-dapr              ClusterIP      10.0.160.36    <none>           80/TCP,50001/TCP   4d1h
...

You'll notice that each service has a -dapr sidecar, and that the React frontend doesn't know the cluster IP or service names of each operation's corresponding service. This is a key aspect of Dapr beyond standalone mode: Kubernetes-aware service discovery, along with mDNS capabilities for non-Kubernetes environments, that developers don't need to implement in their code.

And what do you say? Passing around the output of something like jc? Using dapr as the backend for something like deskconn? Running dapr run --app-id ncapp --app-port 3000 -- socat -v tcp-l:3000,fork exec:'/bin/cat'? Sure! Why not?

Summary

Dapr offers an excellent path to getting rid of redundant logic with brittle implementations and enable large teams to start doing things like service invocation, decoupled state stores and event-driven programming while reducing the 3rd party code and SDK footprint in the codebase.

It could be particularly exciting when coupled with OAM/Rudr (watch Mark Russinovich's Dapr, Rudr, OAM interview at Microsoft Ignite) to help further separate developer and operator concerns in Kubernetes clusters.

I learned a thing (or five) in the process of checking it out. Give it a try, reach out to the team and say hi!

A quick guide to podman and toolbox in Debian (and maybe Ubuntu)

José Miguel Parrella — Sat, 01 Feb 2020 19:11:27 +0000

Over the last couple years I've been spending a lot of time playing with containerized development environments such as WSL2 and Crostini. I also run Fedora in a NUC to try and keep up with systemd, cgroup2, podman and other technologies. But since I brought my Debian laptop to FOSDEM20, I wanted to play with Podman and Toolbox natively.

Why `podman` is important

Remember when docker bundled daemon and tools? Although it was eventually decoupled, many of us learned a formulaic usage of the docker command and it's not unusual to find the legacy packages in many of our systems today.

That was certainly the case for me: I knew that we needed to decouple to achieve rootless containers, registry-side building and interchangeable runtimes. But the options expanded rapidly and since I was wary of keeping too many tools with overlapping experiences in my system, I continued to rely on the docker-ce packages.

I believe podman is a credible replacement for my needs, but if you want to play around with toolbox then podman is a requirement, even if one that you'll be happy with. All of this stack is integrated and tested in Fedora before other RPM-based distros (let alone Debian derivatives) so it does take a little bit of work but the results and basic functionality is pretty much comparable. Let's begin.

Installing `podman` and `toolbox` in Debian

While I was attending FOSDEM, two speakers (one from SUSE, one from Red Hat) wondered if podman and toolbox were available for Debian and derivatives, such as Ubuntu. They assumed so, but weren't quite sure. (A word of warning: there's a snap by Ondrej called "toolbox", but this is not what we're discussing here.)

For podman, openSUSE's Kubic builds deb packages that work in Debian and Ubuntu. This is the current installation method, and it's what I used. Albeit rocky, there's ongoing work to get this package officially into Debian.
For toolbox, which is a shell script, you can fetch a release and place the script in your PATH. Make sure you install flatpak, as that's needed (there could be other dependencies, but in my reasonably vanilla desktop system, I was only missing a sudo apt install flatpak -y)

One last thing, part of the rootless magic relies on user namespaces so make sure you echo 1 | sudo tee /proc/sys/kernel/unprivileged_userns_clone and understand the security implications of that.

What can `toolbox` do?

In Fedora Silverblue, toolbox is used to provide a mutable working environment on top of a (mostly) immutable operating system such as Silverblue or CoreOS (you can watch rishi's presentation for full impact)

In our case, since we're running Debian in an environment I regularly mutate, we care less about that aspect but we still want a working environment that's easy to step in and out of.

Either way, you might be wondering how that is any different from having a pet docker run -it ... /bin/bash or a playground VM whether that's with libvirt or lxd, in a public cloud, VPS provider or somewhere else. Or something custom you have with a mix of pyenv or nix-shell...

The difference with toolbox is that it overlays this environment on top of your profile, carries your shell settings and helps users resolve just like in the host. So you wouldn't need to worry about things like mounting a 9p filesystem or sync'ing files and adjusting ownership, etc.

So when you enter the toolbox environment, you feel like you're in your regular environment, but things you change beyond your profile are kept to the container. Here's an example:

bureado@crucia:~$ echo hi-from-host > hello
bureado@crucia:~$ htop -v

Command 'htop' not found, but can be installed with:

sudo apt install htop

bureado@crucia:~$ toolbox enter --container debian-toolbox-latest
bureado@toolbox:~$ cat hello && echo hi-from-container > hello
hi-from-host
bureado@toolbox:~$ htop -v
htop 2.2.0 - (C) 2004-2019 Hisham Muhammad
Released under the GNU GPL.

bureado@toolbox:~$ logout
bureado@crucia:~$ cat hello
hi-from-container

In this example you see how I switched from my laptop (crucia) to the container (toolbox) sharing files and changes in my profile but keeping any additions (in this case, a previous apt install htop I made in toolbox) to the container.

The role of `podman`

As I mentioned earlier, toolbox is a script. A 2.5K+ SLOC script with close to 60 mentions to podman. So podman is a real hero here. If I run toolbox list, I get:

bureado@crucia:~$ toolbox list
IMAGE ID      IMAGE NAME                       CREATED
b31c77acc328  localhost/debian-toolbox:latest  About an hour ago
1787a6a86277  localhost/ubuntu-toolbox:latest  3 hours ago

CONTAINER ID  CONTAINER NAME         CREATED            STATUS                IMAGE NAME
267d9c17c3f8  debian-toolbox-latest  About an hour ago  Up About an hour ago  localhost/debian-toolbox:latest
baf2ed3ece9b  ubuntu-toolbox-latest  3 hours ago        Up 2 hours ago        localhost/ubuntu-toolbox:latest

This lists two OCI container images and two actual containers running. Doesn't that sound like the output of docker ps? Well, that's what podman can replicate:

bureado@crucia:~$ podman images
REPOSITORY                 TAG        IMAGE ID       CREATED             SIZE
localhost/debian-toolbox   latest     b31c77acc328   About an hour ago   445 MB
localhost/ubuntu-toolbox   latest     1787a6a86277   3 hours ago         340 MB
docker.io/library/ubuntu   19.04      c88ac1f841b7   2 weeks ago         72.4 MB
docker.io/library/debian   unstable   0e26bcfa03fc   5 weeks ago         122 MB
bureado@crucia:~$ podman ps
CONTAINER ID  IMAGE                            COMMAND               CREATED            STATUS                PORTS  NAMES
267d9c17c3f8  localhost/debian-toolbox:latest  toolbox --verbose...  About an hour ago  Up About an hour ago         debian-toolbox-latest
baf2ed3ece9b  localhost/ubuntu-toolbox:latest  toolbox --verbose...  3 hours ago        Up 3 hours ago               ubuntu-toolbox-latest

And of course, podman allows me to do operations like start and stop and run -- all rootless. In fact, I can take a Dockerfile and build it with podman, as you see below.

bureado@crucia:~$ podman build . -t debian-toolbox
STEP 1: FROM docker.io/library/debian:unstable
STEP 2: ENV NAME=debian-toolbox VERSION=unstable
--> Using cache 96545d7a49c3a47a39cb9f2fc8c6b40d5240b02dfa1a0c2ac9efcf976d67d44c
...

I encourage you take a look at podman which also runs on Mac (and even WSL2) by virtue of its remote-client support.

There are also underlying technologies to podman such as conmon and you can learn more about it replaying this FOSDEM session: Podman - The Powerful Container Multi-Tool.

Where did the image come from?

You probably noticed from my output that there are two (debian|ubuntu)-toolbox-latest containers that are using two (debian|ubuntu)-toolbox images. Where did those come from? (And yes, this also means this article is probably helpful if you use Ubuntu instead of Debian.)

This image is supposed to be functionally close to your actual host working environment (to provide consistency) and, in fact, it needs two special LABELs asserting so in order for toolbox to ingest them.

Here's an example of a Debian image for toolbox where you can see the additional packages being installed and the labels being declared.

Once you do all of that (with podman), you instruct toolbox to recognize this image and create a standby container which you can enter. Altogether, it looks like this:

podman build -t debian-toolbox -f Dockerfile
toolbox create -i localhost/debian-toolbox:latest
toolbox enter -c debian-toolbox-latest

And everytime you toolbox enter, you can mutate that system without polluting your main one - except for files in your profile. That's it! I learned a good deal about the underlying technology just going through this process and reading the code.

In the coming months, I'll be evaluating this against my current setup of using nix and Python's venv while looking at more emerging technology that can be applied to this space, from other tools in this stack like buildah and skopeo to things like MicroK8s.

Let me know if you find this useful and/or interesting, and comments always welcome!

Getting started with distri on Azure

José Miguel Parrella — Wed, 04 Sep 2019 17:55:33 +0000

You might have come across Michael Stapelberg's Linux package managers are slow write-up earlier this month. His hypothesis is that indexes, hooks/triggers and archive operations are a big source of issues with conventional Linux package managers.

Concurrently, Michael announced distri, a Linux distro design to experiment with different approaches in these areas. I've been playing around with distri after getting it up and running on Azure. Here's what I learned.

What's in it?

In its current release, jackherer, distri ships Linux 5.1.9, systemd, and 170+ other packages including Python 2.7, Python 3, Perl, Golang and Docker, all the way to Xorg and i3.

distri is also a command-line tool used to build packages and the distro (images, repos) itself. The online repo contains said images and binary packages. The binary packages are SquashFS files, with metadata.

What makes it unique?

In addition to the distri tool and the packaging definitions for 400+ packages, the GitHub repo contains the code (Go) for a FUSE filesystem that assembles images from the local software store as a working, read-only filesystem for the running instance.

So if you inspect the root of the filesystem, you'll notice that /bin, /share and /lib (and their /usr counterparts) point to /ro. And, you guessed it, /ro is handled by this FUSE component.

Each package:version pair ships in a separate directory under /ro. Exchange directories are used for places where each package is expected to contribute files, such as /usr/include.

The part that makes distri unique is that this FUSE component lazily loads the images as needed. The SquashFS images, roughly equivalent to binary packages such as debs and rpms, live in a separate store under /roimg.

What all of this implies is that package manager operations are very fast, because they involve transporting an SquashFS file (and textproto metadata) to the local software store, and then letting the FUSE component work its magic by assembling it in the filesystem "view". Here's Michael showing how that works

Neither the concept of one folder per package or even the lazy loading with FUSE concepts are new (Nix, AppFS) but I don't think I have seen both put together in a modern way. Michael says that this method provides just enough FHS compatibility for 3rd-party applications to work, such as Chrome or Visual Studio Code.

Behind the scenes

About 80% of the packages in the distri repo use the cbuilder build method, but other available and popular methods include gomodbuilder and perlbuilder.

One of Michael's topics of interest is hooks and triggers, which he argues makes deserialization, idempotency, upstream standardization, etc., much more harder. Only one package in distri has a trigger, and that is openssh for host key generation.

The packaging is very streamlined (respective to upstream) with only about 60 patches, 130 manual extra build flags, 115 manually declared runtime dependencies and 295 custom build steps across over 400 packages.

Conceptually similar repos that you might want to compare include nixpkgs, Spack, Homebrew formulae or the Photon specs.

This is me building nano in distri. I use the distri scaffold tool to create a textproto file, declare my two build dependencies and go:

...this will end with something like:

2019/09/04 16:59:04   step 0: 2m31.51560212s (command: [${DISTRI_SOURCEDIR}/configure --host=x86_64-pc-linux-gnu --prefix=${DISTRI_PREFIX} --sysconfdir=/etc --disable-dependency-tracking])
2019/09/04 16:59:04   step 1: 18.112800177s (command: [make -j8 V=1])
2019/09/04 16:59:04   step 2: 2.581871913s (command: [make install DESTDIR=${DISTRI_DESTDIR} PREFIX=${DISTRI_PREFIX}])
[...]
2019/09/04 16:59:05 nano runtime deps: ["file-amd64-5.34-3" "glibc-amd64-2.27-3" "zlib-amd64-1.2.11-3" "ncurses-amd64-6.1-5"]
[...]
2019/09/04 16:59:05 package successfully created in /root/distri/build/distri/pkg/nano-amd64-4.3-1.squashfs

You can then place this package in your local store, where the FUSE component will pick it up, making the nano command avaiable in your $PATH. Neat!

Beyond packaging

distri runs on Azure pretty much out of the box. Images can be converted to the VHD format, and serial console can be enabled optionally, but because it doesn't let root SSH into it by default (which is good) you might want to consider the steps I documented which include uploading a disk and creating a VM with the Azure CLI.

I also wrote a basic /etc/os-release for distri, because one is currently not included. Here's what that looks like:

root@distri0:~# hostnamectl 
   Static hostname: distri0
         Icon name: computer-vm
           Chassis: vm
        Machine ID: 201aa7a551c14f299bb973f1ce206503
           Boot ID: e511d5411c804a7e8f24288025219796
    Virtualization: microsoft
  Operating System: distri (jackherer)
            Kernel: Linux 5.1.9
      Architecture: x86-64

More demos and cool things to try are on the GitHub repo. If you're interested in distri, I suggest you join the mailing list or join the conversation on GitHub Issues for the repo.

If you're in Europe and attending All Systems Go I'd love to meet and talk more about Linux and package management. And if you're in the USA and attending All Things Open join me for a session on The Future of Linux Distros in the Cloud, October 14th, 2019 in Raleigh.

5 cosas que aprendí en KubeCon Barcelona

José Miguel Parrella — Mon, 03 Jun 2019 21:04:44 +0000

Hace un par de semanas visité Barcelona para participar en la KubeCon + CloudNativeCon Europe 2019.

Si no pudiste ir, muchas de las láminas así como los vídeos de las sesiones ya están disponibles.

Estas son 5 cosas (algunas bastante inesperadas) que aprendí en KubeCon Europe, Barcelona 2019.

Kubernetes usa controladores para poder escalar mejor

Kubernetes es un proyecto de software muy complejo, con más de 2 millones de líneas de código, múltiples componentes y mecanismos de extensión. Y sigue creciendo.

En su charla The Kubernetes Control Plane for Busy People Who Like Pictures, Daniel Smith explica por qué en un sistema distribuído como Kubernetes es muy difícil de escalar su monitoreo con una máquina de estado convencional.

Mediante el uso de ciclos de control, Kubernetes puede agregar nuevas características y escalar de forma lineal en vez de exponencial.

Además, en su charla describe varias categorías de controladores (clásicos, inyección, biyección, unión) e incluso describe como ejemplos varios de los controladores más populares. No me esperaba que iba a interesarme por teoría de control en una charla de Kubernetes en Barcelona...

`topologyKey` ayuda a mejorar la disponibilidad de tus aplicaciones en la nube

Improving Availability for Stateful Applications in Kubernetes de Michelle Au fue mi charla favorita de este evento.

La razón es que en vez de asumir que la audiencia está escogiendo el mejor mecanismo de almacenamiento para sus aplicaciones en Kubernetes, Michelle describe cómo cada tipo de almacenamiento (ya sea tradicional o de nube) encaja (o no) con Kubernetes.

Si tienes una NAS, una SAN, un disco administrado en la nube o algo distinto, Kubernetes utiliza conceptos y lógica diferentes para garantizar que el estado de tus aplicaciones está distribuído y es resiliente.

Por ejemplo, el uso del valor failure-domain en topologyKey te permite asignar zonas a los recursos de manera que Kubernetes esté consciente de dónde está tu almacenamiento y qué hacer en caso de fallos.

CNAB agrupa los elementos que describen una aplicación moderna compleja

Un tópico del que hablo en The Future of Linux Packaging es que hoy en día una aplicación ya no está compuesta por un paquete o un proyecto con código fuente, sino que se ha vuelto un concepto complejo que a veces requiere múltiples servidores o sistemas, servicios administrados y más.

Chris Crone dio una excelente introducción a CNAB en Barcelona donde explicó que hoy en día desplegar una aplicación requiere de varios pasos incluyendo herramientas como apt, terraform, helm, kubectl, aws|gcloud|az y más.

CNAB ataca este problema agrupando los recursos que describen una aplicación y creando el concepto de una imagen "lanzadera" que despliega la aplicación. No dejes de darle una mirada a su presentación y darle feedback al equipo de CNAB.

(Por cierto, ¿ya probaste Helm 3?)

No es suficiente solo con dejar de usar `root` en tus contenedores

Si ya trabajas con contenedores de seguro te has preguntado cómo puedes confiar en todos los layers que tu runtime se baja cuando construyes tus imágenes. Es un tema muy importante, porque al fin y al cabo se trata de tu aplicación, de los datos de tus usuarios y de tus sistemas de producción donde podría estar entrando código malicioso o defectuoso.

En Rootless, Reproducible, and Hermetic: Secure Container Build Showdown, Andrew describe varios vectores de ataque que incluyen el uso de un FROM malicioso, ataques al servidor donde se construye la imagen a través de RUN, uso de --privileged e incluso ataques dentro del contenedor.

Además, Andrew revisa nueve herramientas distintas (diferentes a docker build) que si bien funcionan sin necesidad de root, tienen distintos niveles de soporte para Docker y Kubernetes así como distintos niveles de protección. Andrew también habla sobre el proceso de estandarizar la interfaz de construcción de imágenes a través del Container Build Interface.

Slack es el principal mecanismo de colaboración de Kubernetes

Por último, me pasé por State of Kubernetes Contributor Community donde Paris Pittman habló sobre la salud del proyecto.

Aquí descubrí que Slack es el principal medio de colaboración de Kubernetes (las listas de correo, websites y sobre todo GitHub o Twitter vienen muy por detrás...) y cómo ya me había visto la presentación de Rael García sobre el grupo de trabajo de documentación donde también está el equipo kubernetes-docs-es, decidí pasarme por Slack.

¡Y vaya comunidad! Al cabo de un par de días ya había hecho mis primeros pull requests y empecé a escribir un documento de tips and tricks que quizás también te anime a contribuir al proyecto.

¿Qué has aprendido sobre Kubernetes recientemente? Compártelo en Twitter

Open source sustainability is a 🌎 debate

José Miguel Parrella — Sat, 04 May 2019 20:27:52 +0000

For the past decade, I've received a daily ping with open source news coverage. It's a lot. One minute I'm reading a Reddit thread on distros, then LWN coverage on Python 2.7 or a Jepsen report on a new product when SJVN tweets a new ZDNet article on corporate contributions.

This coverage is, almost always, written in English by and for US audiences. At first glance, this seems inconsequential as anyone that started with open source in non-English speaking countries two decades ago made do even with the sorry state of documentation, i18 and l10n back then.

But during the second half of 2018, as the debate dug deeper into open source sustainability (a broad collection of issues including licensing, business models, ethics, diversity & inclusion, etc.) it became apparent that the premises and analysis could use a global perspective.

Earlier this year I set about testing whether there is a homogeneous understanding of the underlying concepts of this debate.

I interviewed a couple dozen open source influencers and community leaders from around the globe, and shared my findings at the Open Source Leadership Summit.

My goal was not to test or even find the answers. For that, the sample needs to be several orders of magnitude larger. My goal was to test the questions.

A key takeaway was that even the words don't translate well. What do "income inequality", "wealth redistribution", "freeloading", "loss-leader", "strip mining" or "sustainability" translate to in Arabic, French or Spanish? It's "free" vs. "gratis" all over again! On top of that, many discussions assume a finance/economics background which in turn is a reflection of the VC-driven reality of open source in the US.

I wondered how much weight non-US influencers put on themes such as open source licensing or business models compared to income inequality or survival of the maintainer. Maybe the quality of VC funding isn't a big concern simply because it isn't available to begin with, but do non-US audiences place any weight on the role of freeloaders or the age differences between contributors? I discovered my Twitter network polled in a very different way than my non-US friends.

I also wanted to understand whether some of the prevailing models such as open source foundations or codes of conduct met the expectations of this group. Was this a classical hammer ergo nail situation? I wondered what people that agree open source has an income inequality problem believe about the role of foundations and competition: if I strongly believe in only one problem do I also believe in only one "savior"?

In the US and other developed markets, there's been no shortage of proposed solutions to the components of the sustainability problem, from new licenses, subscription models, distributed funding, foundations, etc.

But are those solutions enforceable or viable outside of the US? Are there any discussion forums for this problem south of the equator? Can a distributed funding model disburse funds in countries like Egypt? What about Nepal, or Venezuela? (all in the Top 10 for largest public GitHub org growth)

There are areas where I plan to research more, including in the role of government and regulators. My upbringing in the open source world had a lot of this, back in the OpenXML and open source legislation days south of the equator, and I wonder how that has evolved in a cloud world.

Possibly the most sobering takeaway for me was that when talking to non-US audiences, it seems impossible to talk about project sustainability without talking about maintainer survival. And whether that meant monetarily or from a recognition standpoint, lots of verbatims pointed out to expectations for the project health to sustain the individual developer's life.

Nowadays, most project leaders understand that they need to listen to and care for a global audience if they want their project to achieve global scale: there are more GitHub contributors outside of the US than in the US and this gap continues to increase every year.

Similarly, I believe open source thought leaders addressing questions of open source sustainability need to bring this discussion to a global audience.

If you have a sample you'd like to poll on this topic, check out the resources from my talk. I would love to hear your findings or chat with you on this topic, so feel free to reach out on Twitter or elsewhere!

Getting started with Photon OS on Azure

José Miguel Parrella — Sun, 10 Mar 2019 00:54:17 +0000

Photon OS is an open source, minimal Linux container host that is optimized for cloud-native applications. On Azure, it ships with just over a hundred packages including systemd, cloud-init and docker but Photon offers over a thousand packages in their repos like Go, .NET Core, Postgres, Tomcat, Zookeeper or Kubernetes!

This is a simplified version of the Azure quickstart experience in the official Photon OS documentation. It was tested with Photon OS 3.0 GA and azure-cli version 2.0.38.

Prerequisites

Download the VHD for Azure from GitHub and extract with tar xf in your local system
Ensure you have the latest Azure CLI (az) available in your working system and that you've logged in with az login

Getting started

Just like any other custom Linux VHD, you'll create a few workspaces, upload the VHD and create a new VM based on it.

Here's a script that simplifies the VM creation. It assumes that an SSH keypair is available in the profile of whatever user runs this script. It expects the downloaded and extracted VHD as an argument, for example: ./script.sh photon-azure-3.0-26156e2.vhd:

$ ./script.sh ./photon-azure-3.0-26156e2.vhd                                                      
+ set -e                                                                                                                  
+ GROUP=photon-rg                                                                                                         
+ STORAGE=phrg999                
+ LOCATION=southcentralus                                      
+ VM_NAME=photon-vm                                                           
+ STORAGE_CONTAINER=vhds                                                                                                  
+ IMAGE_PATH=./photon-azure-3.0-26156e2.vhd                                                                               
+ basename ./photon-azure-3.0-26156e2.vhd                                                                                
+ IMAGE_NAME=photon-azure-3.0-26156e2.vhd                                                                                 
+ az group create -n photon-rg -l southcentralus                     
+ az storage account create -n phrg999 -g photon-rg  
...                                  
+ az vm create -n photon-vm -g photon-rg --os-type linux --image https://phrg999.blob.core.windows.net/vhds/photon-azure-3.0-26156e2.vhd --use-unmanaged-disk --storage-account phrg999                                                             
{                                                          
...                     
  "powerState": "VM running",
...

The az vm create command will output (in JSON, by default) information about the newly created VM, including a public IP address you can use to SSH into the VM. If you missed it, run az vm list-ip-addresses -o table.

bureado@photon4 [ ~ ]$ systemd-analyze 
Startup finished in 1.278s (kernel) + 2.177s (initrd) + 14.546s (userspace) = 18.002s
multi-user.target reached after 14.532s in userspace
bureado@photon4 [ ~ ]$ cloud-init --version
/usr/bin/cloud-init 18.3
bureado@photon4 [ ~ ]$ uname -a
Linux photon4 4.19.15-3.ph3 #1-photon SMP Mon Feb 25 14:48:35 UTC 2019 x86_64 GNU/Linux
bureado@photon4 [ ~ ]$ curl -s -H Metadata:true "http://169.254.169.254/metadata/instance?api-version=2017-08-01" | jq .compute.location
"westus2"
bureado@photon4 [ ~ ]$

Cleaning up

If you used a unique resource group name in the $GROUP variable above, you can remove all resources by running az group delete -g <group name> --yes. This will remove all resources created by the script, including the uploaded blob, but won't delete it from your local working folder.

Have fun!

A few of Microsoft's snow tracks in open source engagement

José Miguel Parrella — Tue, 12 Feb 2019 05:16:00 +0000

My colleague Jeff recently shared an outstanding model to assess organizational readiness, challenges and aspirations when it comes to participating in open source.

I was familiar with this model since Jeff has driven it at Microsoft for a while and I've had a chance to learn from him and his team, and I was immensely happy to see him share it in long form with the world.

Shortly after there was a provocative follow-up question by LFDL's Ibrahim Haddad: how do you advance in this ladder? Ibrahim offered some ideas that range from securing executive support to being involved in the TODO Group and having a comprehensive component inventory.

Inpired by all of this, I wanted to share some of my experiences after ~10 years driving open source change at Microsoft. Many learnings are our own, some come from our partners and even from customers that often cross-pollinate our mutual open source journeys.

Embrace the branches

The journey to pervasive open source engagement in an organization isn't linear. Imagine this.

Some years from now we'll reflect back on how hard it was to move past the initial frustration with open source. You helped hire a dream team of open sourcerers but not much change was happening. Everyone was on their own mission, and at times you felt hopeless. But today, what a difference! You operate proactively, control your own destiny and people know you and come to you for guidance.

Until they don't. Often times, a new "hype" branch will stem from a more mature stage in your open source journey triggered by a competitor, a new product, a new market or just the ups and downs of this crazy industry of ours.

Resist the need to exercise extreme control over branches. Ask your mentors what would be a reasonable level of structure you can offer to the branch, then aim even lower: anything growing rapidly or moving a lot of resources carries politics that will only drag you down.

At Microsoft, I've had the privilege to collaborate on projects that range from bland components in places like Windows Update to cutting-edge research projects and anything in between whether under the Office brand, the Microsoft brand or even no brand.

I work on Azure, but in my head I've known at least 4 Azures since PDC09. I lived through Microsoft Open Technologies, Port 25, the Open Source Technology Center, the Enterprise Open Source Group and many other, more obscure acronyms.

Each one of those branched in hype more than once but all of those branches contributed to the corporate culture and organizational knowledge and proficiency we have on open source today.

So branches eventually merge. And the best that you can do is to showcase the business value of engagement maturity and capture the learnings: at times, I've had to engage my colleagues at the Microsoft Library or the Microsoft Archives to "deposit" some knowledge that is too fragile for our document management and retention systems.

Secure a bench

There's no escaping it: executive support is core to the success of your open source strategy, regardless of the organization. Maybe the CEO is your sponsor and the ideologist behind your open source strategy. But even if she isn't, you don't always need "inorganic" (authoritative, mandated, dictated) executive edicts.

Sometimes you can create the conditions for executive support without calling in the air cover. You should still actively advocate but by bringing an external perspective, maybe shining a light on what a competitor or an adjacent industry are doing with open source, you can generate enough energy for executive support.

That's why I often suggest that internal open source advocates remain fully informed of their context: industry, trends, market research and intelligence. I fully believe a market research engine focused on open source technologies is part of the charter of those in charge of open source strategy, in no small part because they can attract and retain not only tactical executive support but a support bench that can serve the strategy long term and across the spectrum.

Invest in the program

You can have the vision and the sponsors, maybe even the hero hires, but if you don't have the program and the resources for the program, you'll always depend on heroics and coin tosses - and open source always outpaces chance.

Ibrahim recently shared his take on the responsibilities of an open source program office. I don't see program offices as the place "where open source happens" but rather the place "where we invest so that open source happens".

That doesn't mean it's where all the decisions are made, or where all the talent is hired. It's not where other teams dump or outsource their duties. It shouldn't be a bottleneck and it only makes sense if it scales to "community scale". It's not only a place for people to come, but a launchpad from where people goes to the rest of the organization.

And even with the program and the funding it's in the execution of an open source program (the tooling, the processes, the compliance efforts...) that a competitive differentiator can be made. No wonder so many in the industry collaborate in the TODO Group to do so: to elevate the initial bar.

There is no shortage of debate on what success looks like for an open source program office. But I think the program must enable the organization to know where they are in the engagement spectrum at any given point in time, anticipate where they want to be and what is in the critical path to get there.

Without the program, the organization is instrument flying and everything is opportunistic. You wouldn't short change your organization in security response or customer support - don't do so in the open source program either.

Care about the product

Whether it's disruption, competitiveness, engineering economics or something else, organizations look at open source to derive value from it and make them better at doing what they do. This is why open source programs should deeply care about the product, whether that's a solution, a technology portfolio, a societal value proposition or something else.

The program will often times need to have an opinion on whether open sourcing is appropriate for a particular initiative, or how exactly to adopt what types of open source components to get something done. They need to understand the business imperatives for the product, and need to read the product tea leaves.

To me, this means spending quality time with product leaders, developers/engineers and everyone else that makes the business tick. You might not establish a functional relationship with those individuals, but a personal one is worth the time investment.

What are their top technical dilemmas? What keeps them up at night? What resource challenges do they have? What are some types of fire drills or distractors they face? And then, more specifically, what types of communities their members are part of, where do they get external direction or pressures?

I might be biased since I'm a product person, but in growing open source programs I believe having product management types can be really helpful, maybe even developing "product advocates" (on a rotational basis) that can represent and advocate for product when the program conducts business with their natural partners.

This might derive into actual work for the open source program office. For example, if different teams are building in diverging build systems, or obtaining components from disparate sources, or lack alignment in terms of tooling, the open source program office is often times expected to resource, develop and maintain services for product units. Product is always a customer of the program office.

Finally, caring about the product as an open source subject matter expert also means being able to discourage the business from doing things that only look good in form.

I spent the first half of my career at Microsoft advocating for open source by hyping it up, and the second half advocating for it by keeping teams away from pitfalls. For example, our team has written "why NOT" guidance that sets realistic expectations for teams.

This raises the bar so that the projects where we invest become more clear and our participation in the community is deeper. It also strengthens the case for resource asks, or makes it clear there wasn't a case to begin with.

There is business value, in pushing for maturity and sophistication: teams can no longer afford to be the "hype laggards" of an organization, let alone an industry, so you're helping them hit the market at an advanced stage, too.

Develop the culture

I'm often reminded by Stephen Walli of the famous "culture eats strategy for breakfast" lemma. At some point you realize your organization is well disciplined and resourced to execute most of the things we expect an open source program to do (it might even be in autopilot) but there's no purchase order you can approve for culture.

An open source culture permeates the entire organization. It should be an integral part of recruitment and talent development: that certainly hasn't gone unnoticed for Facebook. Conversely, the absence of it -or lack of authenticity- can alienate your value chain. If you sell software, it certainly aggravates the enterprise sales problem described by Luke Kanies.

At Microsoft, I spend a significant amount of my time talking to others about open source culture and taking concrete steps towards developing said culture. Developing an open source culture should be an observable event, and while you can measure many aspects of it, I've often found success best described in terms of moments of truth that should be as easy to digest and understand as the culture itself.

A powerful mechanism I've found is to structure most training and communications around the culture theme. Often times, this takes the form of sharing learnings: finding culturally-relevant events in our open source journey that can serve to illustrate what we mean by a fledging (or failing) culture instead of trying to enumerate or dictate the aspects of the culture that we find strategically convenient. It is true: culture is on a low-strategy diet.

In times of change it's hard for organizations to find their internal or external voice and this is why functions like PR or Marketing are such an important part of developing an open source culture. In absence of cultural tenets and a strong "why we do this", those functions operate opportunistically with the results that we all love to critique.

Something that has worked wonders for us is developing talent as ambassadors of the culture, equipping them with that voice. PR and Marketing and other functions can facilitate this, understanding that we lead with a transparency and enablement spirit.

When speaking, I spend little time introducing myself before showing a collage of thousands of Microsoft contributors on GitHub. When it comes to culture, my goal is to speak on their behalf and the other way around. Over time, I've found this speaks louder and more institutionally than one-offs, and one-offs can go wrong at any time, but especially when your maturity tree has a lot of hype branchs and people organically stay on message, less because of the policing and more because the message makes sense.

Don't go in alone

Yours is not the only organization trying to derive value from open source. Your competitors certainly are, but so is everyone else in your value chain. Chances are that anyone that Accounts Receivable or Accounts Payable is touching is also trying to do something with open source. Therefore, units like partner and business development, customer success, customer support, etc., must all think ecosystem first.

And by ecosystem of course I mean commercial and community partnerships. We talk about open source foundations and some working groups as if they're extensions of the standards work of the nineties, but ecosystem is far more complicated than that. Venture capital is a reality in open source today, and we can expect academia, government and system integrators to play an increasingly stronger role in this phase of the cycle.

Final thoughts

If you've read this far, you've probably noticed I've mentioned a good number of functions that, in any medium- to large-sized organization, are performed by different (maybe even competing) divisions and disciplines, so you might be wondering about how open source change agents in an organization can or should work with their colleagues.

I've been fortunate enough to have worked with open source at Microsoft across the entire org chart: field sales, corporate marketing, business planning, talent development, partnerships, product strategy... and with and across disciplines ranging from legal to customer support. In those capacities, I've interacted with different shapes of an "open source program" sometimes running in parallel and sometimes not running at all. And I've done so since 2010, under evolving leadership and evolving internal perceptions about open source at Microsoft.

I can talk at length about those experiences in the hallway track at many upcoming open source conferences and I certainly look forward to collaborating with thought leaders to document some learnings. But one of the most inspiring leaders I know at Microsoft summarized what best describes the mechanics of an impactful cross-boundary open source collaboration: being humble, helpful and harmless.

This is why sharing cultural learnings is so powerful: you let someone else tell the story for you, and it's invariably helpful because it's immediately relatable. This also means that I proactively look for opportunities to find out those learnings. I proactively seek to participate in RCAs and post-mortems, and in those efforts I mostly stay silent (=harmless)

Yet I can confidently say that it is in and through those painful moments that I've felt we've made the broadest, most authentic and longer-lasting cultural changes with open source at Microsoft.

In summary:

The journey to pervasive open source engagement isn't linear. Don't try to control a hype event, just observe it.
Sometimes the case for executive support is self-evident. Develop an open source market research engine.
You can have the vision and the sponsors, but you can't shortchange your program. Operational excellence is the next frontier of differentiation.
Product empathy isn't optional for an open source program office. Knowing what's a non-ideal case for open source can be very valuable: people can't afford to lag in the engagement model.
Lead with culture and don't go in alone!

DEV Community: José Miguel Parrella

Transparency and user agency as principles for distributing and consuming open source software packages

Summary

Dapr, the hard way

Why dapr?

Dapr and podman

Dapr in action

Beyond standalone daprd

Summary

A quick guide to podman and toolbox in Debian (and maybe Ubuntu)

Why podman is important

Installing podman and toolbox in Debian

What can toolbox do?

The role of podman

Where did the image come from?

Getting started with distri on Azure

What's in it?

What makes it unique?

Behind the scenes

Beyond packaging

5 cosas que aprendí en KubeCon Barcelona

Kubernetes usa controladores para poder escalar mejor

topologyKey ayuda a mejorar la disponibilidad de tus aplicaciones en la nube

CNAB agrupa los elementos que describen una aplicación moderna compleja

No es suficiente solo con dejar de usar root en tus contenedores

Slack es el principal mecanismo de colaboración de Kubernetes

Open source sustainability is a 🌎 debate

Getting started with Photon OS on Azure

Prerequisites

Getting started

Cleaning up

A few of Microsoft's snow tracks in open source engagement

Embrace the branches

Secure a bench

Invest in the program

Care about the product

Develop the culture

Don't go in alone

Final thoughts

Dapr and `podman`

Beyond standalone `daprd`

Why `podman` is important

Installing `podman` and `toolbox` in Debian

What can `toolbox` do?

The role of `podman`

`topologyKey` ayuda a mejorar la disponibilidad de tus aplicaciones en la nube

No es suficiente solo con dejar de usar `root` en tus contenedores