DEV Community: James Hunt

Implementing Non-Trivial Containerized Systems Part 4: Adding a Web Interface

James Hunt — Tue, 22 Jun 2021 18:07:03 +0000

This is the fourth part of a multi-part series on designing and building non-trivial containerized solutions. We're making a radio station using off-the-shelf components and some home-spun software, all on top of Docker, Docker Compose, and eventually, Kubernetes.

In this part, we've got a working system, our own rebuildable images, and a portable Docker Compose deployment recipe. What more could we want? A web interface for managing the tracks that we're streaming would be nice...
The Icecast web interface is, well, usable, but it does leave a fair amount to be desired. This is the part of systems design where I usually take a step back, see what's missing, and then focus my software development energies on solving those deficiencies.

As a radio station operator, I would like to...:

Exert fine-grained control over the tracks that my radio station plays.
Use my web browser to add new YouTube tracks to my radio station. That's it; I'm a simple man, with simple needs. I can write a small-ish web app to do these things, and give it read-write access to the /radio volume.

And here it is:

I call it MixBooth. It's on GitHub too.

MixBooth is a small Vue.js front-end backed by an even smaller Go HTTP (REST) API. The backend piece consists of only three endpoints:

GET /playlist

This one retrieves the current playlist. The Vue bits render this on the bottom half, under the Current Track Line-Up heading. The checkboxes are there so that you, intrepid Radio Disk Jockey that you are, can remove tracks from the rotation, and add them back in, by way of our next endpoint:

PUT /playlist

Tired of the same song popping up? Yank the cassette!

Finally, to get new stuff into the station, from YouTube, we have:

POST /upload

For a deeper dive into the code, check out the GitHub repository.

To containerize this thing, we're first going to look at our Dockerfile. Keep in mind I specifically wrote this piece of software to work with this radio station deployment. I was "filling in the gaps" so to speak. This is a long Dockerfile – the longest we've seen so far – so we'll take it in parts.

Our first part is the api stage. Here, we build the Go backend application, using tooling that should be familiar to most Go programmers (even terrible ones like me):

FROM golang:1.15 AS api
WORKDIR /app
COPY go.mod .
COPY go.sum .
COPY main.go .
RUN go build

Our next stage (in the same Dockerfile) builds the Vue component, using tooling that is recognizable to Node developers (but maybe not to the aformentioned Go rockstars):

FROM node:15 AS ux
WORKDIR /app
COPY ux .
RUN yarn install
RUN yarn build

Finally, in the last stage, we'll tie it all back together, copying in assets from our build stages, with some Ubuntu packaging:

FROM ubuntu:20.04
RUN apt-get update \
 && DEBIAN_FRONTEND=noninteractive  apt-get install -y python3 python3-pip ffmpeg \
 && pip3 install youtube-dl \
 && apt-get remove -y python3-pip \
 && apt-get autoremove -y \
 && rm -rf /var/lib/apt/lists/*

COPY --from=api /app/mixbooth /usr/bin/mixbooth
COPY --from=ux  /app/dist     /htdocs
COPY            ingest        /usr/bin

EXPOSE 5000

ENV HTDOCS_ROOT=/htdocs
CMD ["mixbooth"]

Remember: we need Python (and PIP!) for the youtube-dl bits; this web UI literally shells out to run youtube-dl when you ask to ingest new tracks.

In fact, let's take a closer look at that ingest script we're copying in.

#!/bin/bash
set -eu

mkdir -p /tmp/ytdl.$$
pushd /tmp/ytdl.$$
for url in "$@"; do
  youtube-dl -x "$url"
  for file in *; do
    ffmpeg -i "$file" -vn -c:a libopus "$file.opus"
    mv "$file.opus" $RADIO_ROOT/
    echo "$RADIO_ROOT/$file.opus" >> $RADIO_ROOT/playlist.m3u
  done
done
popd
rm -rf /tmp/ytdl.$$

I want to point out that instead of hard-coding the radio files mountpoint to something like /radio, I chose to rely on the $RADIO_ROOT environment variable instead. We'll use this in our next section, when we add the web interface container into our larger deployment.

Composing the Web UI

Let's get this web interface into the mix from a Docker perspective, shall we?

Here's the Compose file we ended up with from the last post:

---
version: '3'
services:
  icecast:
    image: filefrog/icecast2:latest
    ports:
      - '8000:8000'
    environment:
      ICECAST2_PASSWORD: whatever-you-want-it-to-be

  source:
    image: filefrog/liquidsoap:latest
    command:
      - |
        output.icecast(%opus,
          host = "icecast",
          port = 8000,
          password = "whatever-you-want-it-to-be",
          mount = "pirate-radio.opus",
          playlist.safe(reload=120,"/radio/playlist.m3u"))

    volumes:
      - $PWD/radio:/radio

Let's add a new web service, using the published Docker image:

---
version: '3'
services:
  # in addition to the other services, here's a new one:
  web:
    image: filefrog/mixbooth:latest
    environment:
      RADIO_ROOT:      /radio
      MIXBOOTH_STREAM: '//{host}:8000/pirate-radio.opus'
    ports:
      - 5000:5000
    volumes:
      - $PWD/radio:/radio

This will spin up the (public) MixBooth image, and bind it on port 5000. We point to the same host path ($PWD/radio) as we used for the LiquidSoap container – we need them to both be looking at the exact same files, so that we can add new tracks for the stream source to pick up, and modify the playlist it uses.

We also added the $RADIO_ROOT environment variable, since MixBooth has no preconceived notions of where the audio files ought to go. We also set the funny-looking $MIXBOOTH_STREAM environment variable like so:

      MIXBOOTH_STREAM: '//{host}:8000/pirate-radio.opus'

This is highly-specific to what MixBooth does. The embedded player for the radio station, is little more than an HTML 5 <audio> element with the appropriate stream source elements. The heavy lifting is done by the browser – thankfully! However, much as it was clueless about where the audio tracks should live, it likewise flummoxed by where, precisely, one would go to listen to those tracks.

This $MIXBOOTH_STREAM environment variable encodes that information, but it does so with some late-binding templating. More precisely, the {host} bit will be replaced, by the Javascript in the visitor's browser, with whatever hostname they used to access the web interface itself. Come in by IP? Hit the Icecast endpoint by IP. Used a domain name and TLS? Listen in secure comfort, oblivious to the numbers that underpin the very Internet.

These were both conscious design decisions made (by me) while implementing this missing piece of the puzzle. By abstracting the site- and station-specific configuration out of the code, and even out of the "configuration" (such as it is), I was able to make the deployment more cohesive and explicit. Since you can't not spell out precisely where the wiring goes, the resulting docker-compose.yml is much easier to understand.

(If you're into software engineering self-reflection, this forthrightness is aimed squarely at reducing Action At A Distance.)

Here's the final Compose file; take it for a spin and see what you think!

--------
version: '3'
services:
  icecast:
    image: filefrog/icecast2:latest
    ports:
      - '8000:8000'
    environment:
      ICECAST2_PASSWORD: whatever-you-want-it-to-be

  source:
    image: filefrog/liquidsoap:latest
    command:
      - |
        output.icecast(%opus,
          host = "icecast",
          port = 8000,
          password = "whatever-you-want-it-to-be",
          mount = "pirate-radio.opus",
          playlist.safe(reload=120,"/radio/playlist.m3u"))
    volumes:
      - $PWD/radio:/radio

  web:
    image: filefrog/mixbooth:latest
    environment:
      RADIO_ROOT:      /radio
      MIXBOOTH_STREAM: '//{host}:8000/pirate-radio.opus'
    ports:
      - 5000:5000
    volumes:
      - $PWD/radio:/radio

Next time, we'll pick this whole deployment up and dump it onto the nearest convenient Kubernetes cluster – stay tuned!

Implementing Non-Trivial Containerized Systems - Part 3: Deploying Containers Together

James Hunt — Fri, 11 Jun 2021 22:17:03 +0000

This is the third part of a multi-part series on designing and building non-trivial containerized solutions. We're making a radio station using off-the-shelf components and some home-spun software, all on top of Docker, Docker Compose, and eventually, Kubernetes.

In this part, we're going to take the notes we took during the last part (you did take notes, didn't you?) and distill them down into some infrastructure-as-code. It all starts with Docker Compose.

Welcome back! If you've been following along, you should now have a directory of .opus audio files and two containers streaming audio out to anyone who connects. If you haven't been following along, go read [the] [other parts] [first]. Then come back.

The Dockerfiles we have made so far are eminently portable. Anyone can take those little recipes and rebuild our container images, more or less. The images themselves are also portable – we could docker push them up to DockerHub and let the world enjoy their container-y goodness.

What's not so portable is how we run and wire those containers up. That part is actually quite proprietary, dense, and easily forgotten. Let's rectify that. Let's use Docker Compose.

Docker Compose is a small-ish orchestration system for a single Docker host. It lets us specify how we want our containers run; what ports to forward, what volumes to mount, etc. It also has a few tricks up its sleeve that will make our lives better. Best of all, Docker Compose recipes can be shared with others who also like building bespoke Internet radio stations. We are legion. And we use Docker.

What's in a recipe? It starts, as most things in the container world do, with a YAML file:

$ cat docker-compose.yml
---
version: '3'

We'll be using version 3 of the Compose specification. That's not terribly important right now, but there are certain concepts that won't work if we're not on a new enough version.

NOTE: If you have been following along, you've probably got containers still running from our previous experiments. Luckily, we specified the --rm flag when we ran these containers, so all we need to do to clean up is stop them:

$ docker stop icecast
$ docker stop liquid

Docker Compose deals in containers, but it does so in scalable sets of identical containers that it calls "services". For our immediate purposes, we won't be scaling these things out, so a service is effectively a container, and vice versa.

We'll start by adding our first service / container: Icecast2 itself.

---
version: '3'
services:
  icecast:
    image: filefrog/icecast2:latest
    ports:
      - '8000:8000'
    environment:
      ICECAST2_PASSWORD: whatever-you-want-it-to-be

Essentially, we're taking the docker run command and committing it to a file, in YAMLese. To get this container running, we'll use the docker-compose command:

$ docker-compose up

This starts the containers up in "foreground mode" - their output will be multiplexed onto your terminal. As we add more containers (err... services), Docker Compose will start color-coding their output to help us keep them separate.

To get your shell prompt back, terminate the foreground process with a Ctrl-C.

That was so easy, let's add our LiquidSoap service / container into the mix:

--------
version: '3'
services:
  icecast:
    image: filefrog/icecast2:latest
    ports:
      - '8000:8000'
    environment:
      ICECAST2_PASSWORD: whatever-you-want-it-to-be

  source:
    image: filefrog/liquidsoap:latest
    command:
      - |
        output.icecast(%opus,
          host = "10.128.0.56",
          port = 8000,
          password = "whatever-you-want-it-to-be",
          mount = "pirate-radio.opus",
          playlist.safe(reload=120,"/radio/playlist.m3u"))

    volumes:
      - $PWD/radio:/radio

This time, we're going to give docker-compose a -d flag, so that it forks into the foreground and keeps on running, while we get our shell prompt back to get EVEN MORE WORK DONE.

$ docker-compose up -d

Since Docker Compose is just creating containers using Docker, we can use the docker CLI to do all the things we're already accustomed to. For example, to see what LiquidSoap is up to, we can check the docker logs:

$ docker logs radio_source_1
docker logs radio_source_1
2021/03/30 13:16:36 >>> LOG START
2021/03/30 13:16:36 [main:3] Liquidsoap 1.4.4
2021/03/30 13:16:36 [main:3] Using: bytes=[distributed with OCaml 4.02 or above] pcre=7.4.6 sedlex=2.3 menhirLib=20201216 dtools=0.4.1 duppy=0.8.0 cry=0.6.4 mm=0.5.0 xmlplaylist=0.1.4 lastfm=0.3.2 ogg=0.5.2 vorbis=0.7.1 opus=0.1.3 speex=0.2.1 mad=0.4.6 flac=0.1.5 flac.ogg=0.1.5 dynlink=[distributed with Ocaml] lame=0.3.4 shine=0.2.1 gstreamer=0.3.0 frei0r=0.1.1 fdkaac=0.3.2 theora=0.3.1 ffmpeg=0.4.3 bjack=0.1.5 alsa=0.2.3 ao=0.2.1 samplerate=0.1.4 taglib=0.3.6 ssl=0.5.9 magic=0.7.3 camomile=1.0.2 inotify=2.3 yojson=1.7.0 faad=0.4.0 soundtouch=0.1.8 portaudio=0.2.1 pulseaudio=0.1.3 ladspa=0.1.5 dssi=0.1.2 camlimages=4.2.6 srt.types=0.1.1 srt.stubs=0.1.1 srt.stubs=0.1.1 srt=0.1.1 lo=0.1.2 gd=1.0a5
2021/03/30 13:16:36 [gstreamer.loader:3] Loaded GStreamer 1.16.2 0
2021/03/30 13:16:36 [frame:3] Using 44100Hz audio, 25Hz video, 44100Hz master.
... etc ...

That was fairly painless. All of the things we can express in terms of a docker run command can be specified, in YAML (of course), for Docker Compose. This means we'll never get to the point of wanting to "wrap up" all the Docker stuff we've been playing with into something more serious.

There are a few things Docker Compose gives us that we don't (easily) get with pure docker run: a bit of DNS. Recall from the last post that when we told LiquidSoap where to stream to, we had to use the actual IP of the Docker host. With Docker Compose, our containers all live on the same virtual bridge, and we get free DNS for connecting them together. From the container, the name icecast will resolved to the internal bridge IP address of the icecast containers. This means we can improve our compose recipe:

--------
version: '3'
services:
  icecast:
    image: filefrog/icecast2:latest
    ports:
      - '8000:8000'
    environment:
      ICECAST2_PASSWORD: whatever-you-want-it-to-be

  source:
    image: filefrog/liquidsoap:latest
    command:
      - |
        output.icecast(%opus,
          host = "icecast",
          port = 8000,
          password = "whatever-you-want-it-to-be",
          mount = "pirate-radio.opus",
          playlist.safe(reload=120,"/radio/playlist.m3u"))

    volumes:
      - $PWD/radio:/radio

This final version of our recipe is completely portable; you don't have to adhere to my personal network numbering scheme to make use of this deployment asset – just download it and compose it up!

Before I let you go, we should verify that everything is still working. The Icecast web interface should still load, and you should be able to listen to those two goofballs on Rent / Buy / Build (the podcast) talk about how to source the components of your Cloud-Native platform.

Enjoy!

Implementing Non-Trivial Containerized Systems - Part 2: Containerizing With Docker

James Hunt — Fri, 04 Jun 2021 17:45:18 +0000

This is the second part of a multi-part series on designing and building non-trivial containerized solutions. We're making a radio station using off-the-shelf components and some home-spun software, all on top of Docker, Docker Compose, and eventually, Kubernetes.

In this part, we're going to take the architecture we cooked up in the last part, and see if we can't get it running in containers, using nothing but Docker itself.
Today, we're going to build a radio station. We drew up the architecture in our last post:

Roll up your sleeves, we're gonna start building some containers.

Sourcing Audio - youtube-dl

Before we can broadcast audio, we have to have audio, and we're going to get our audio from YouTube. This, then, is our first container.

We're going to start with Ubuntu 20.04 (Focal Fossa) base image. I like this image not only because it's familiar and homey; but also for access to apt for package management, and (more importantly) all those sweet, sweet Debian packages. Case in point, we need Python (youtube-dl is written in Python):

FROM ubuntu:20.04
RUN apt-get update \
 && apt-get install -y python3 python3-pip \
 && pip3 install youtube-dl

We're going to set up an ENTRYPOINT in our image, so that when we docker run it, we can pass it arguments, and use the container as if it were youtube-dl itself.

FROM ubuntu:20.04
RUN apt-get update \
 && apt-get install -y python3 python3-pip \
 && pip3 install youtube-dl

WORKDIR /radio
ENTRYPOINT ["youtube-dl"]

Let's run what we've got so far:

$ docker build -t youtube-dl .
Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM ubuntu:20.04
 ---> f63181f19b2f
Step 2/2 : RUN apt-get update  && apt-get install -y python3 python3-pip  && pip3 install youtube-dl
 ---> Running in cbc3f65f6028
<a whole bunch of apt-get output...>
Collecting youtube-dl
  Downloading youtube_dl-2021.3.14-py2.py3-none-any.whl (1.9 MB)
Installing collected packages: youtube-dl
Successfully installed youtube-dl-2021.3.14
Removing intermediate container cbc3f65f6028
 ---> 66648c847e34
Successfully built 66648c847e34
Successfully tagged youtube-dl:latest

$ docker run youtube-dl --version
2021.03.14

Since we used an ENTRYPOINT, the --version argument we gave to the container got passed through youtube-dl. This is important, since this will be our primary means of interacting with youtube-dl.

Let's try to download some back episodes of my podcast, Rent / Buy / Build. You can find those here.

$ docker run youtube-dl https://youtu.be/D7hxDYBoHvQ?list=PLGDWYlG32yKoYAuREw9eSLFIdWJZ_3rW8
[youtube:tab] Downloading playlist PLGDWYlG32yKoYAuREw9eSLFIdWJZ_3rW8 - add --no-playlist to just download video D7hxDYBoHvQ
[youtube:tab] PLGDWYlG32yKoYAuREw9eSLFIdWJZ_3rW8: Downloading webpage
[youtube:tab] PLGDWYlG32yKoYAuREw9eSLFIdWJZ_3rW8: Downloading webpage
[download] Downloading playlist: Rent / Buy / Build Episodes
[youtube:tab] playlist Rent / Buy / Build Episodes: Downloading 1 videos
[download] Downloading video 1 of 1
[youtube] D7hxDYBoHvQ: Downloading webpage
[youtube] D7hxDYBoHvQ: Downloading MPD manifest
[download] Destination: Episode 0 - What's All This Then-D7hxDYBoHvQ.mp4
[download] 100% of 12.76MiB in 00:0296MiB/s ETA 00:003
[download] Finished downloading playlist: Rent / Buy / Build Episodes

Looks like we got the file, but where did it go? Into the ether. When our container process (youtube-dl) exits (i.e. finishes downloading and transcoding the video), the root filesystem for the container is wiped away. This is a feature, honest. If we want to keep any of the files we create in the container, we're going to need to bind-mount in some persistent(-ish) storage. If you recall, we set our WORKDIR to /radio, so if we can mount a directory from outside of the container, we can keep those files around longer.

$ mkdir radio
$ docker run -v $PWD/radio:/radio youtube-dl https://youtu.be/D7hxDYBoHvQ?list=PLGDWYlG32yKoYAuREw9eSLFIdWJZ_3rW8
[youtube:tab] Downloading playlist PLGDWYlG32yKoYAuREw9eSLFIdWJZ_3rW8 - add --no-playlist to just download video D7hxDYBoHvQ
[youtube:tab] PLGDWYlG32yKoYAuREw9eSLFIdWJZ_3rW8: Downloading webpage
[youtube:tab] PLGDWYlG32yKoYAuREw9eSLFIdWJZ_3rW8: Downloading webpage
[download] Downloading playlist: Rent / Buy / Build Episodes
[youtube:tab] playlist Rent / Buy / Build Episodes: Downloading 1 videos
[download] Downloading video 1 of 1
[youtube] D7hxDYBoHvQ: Downloading webpage
[youtube] D7hxDYBoHvQ: Downloading MPD manifest
[download] Destination: Episode 0 - What's All This Then-D7hxDYBoHvQ.mp4
[download] 100% of 12.76MiB in 00:0218MiB/s ETA 00:004
[download] Finished downloading playlist: Rent / Buy / Build Episodes

$ ls -l radio
total 13312
-rw-r--r-- 1 jhunt staff 13383936 Mar 22 14:30 "Episode 0 - What's All This Then-D7hxDYBoHvQ.mp4"

Yay! Files! Too bad they're video files, not audio files. To fix that, let's throw ffmpeg into our container image.

FROM ubuntu:20.04
RUN apt-get update \
 && DEBIAN_FRONTEND=noninteractive apt-get install -y python3 python3-pip ffmpeg \
 && pip3 install youtube-dl

WORKDIR /radio
ENTRYPOINT ["youtube-dl"]

(Note: I had to add that DEBIAN_FRONTEND=noninteractive bit to keep the package manager from bothering me with terminal prompts about timezone data.)

With ffmpeg, we can take the files that come out of youtube-dl, drop the video components, and convert the audio into a common format (which will be useful when we get to LiquidSoap programming).

$ docker run -v $PWD/radio:/radio --entrypoint ffmpeg youtube-dl \
  -i "Episode 0 - What's All This Then-D7hxDYBoHvQ.mp4" \
  -vn -c:a libopus \
  "Episode 0 - What's All This Then-D7hxDYBoHvQ.opus"

That's a fair bit of domain expertise regarding file formats and conversion, and it's all right out there, in the open. It's also a two step process. This is a problem that can be fixed by a small shell script.

#!/bin/sh
set -eu

mkdir -p /tmp/ytdl.$$
cd /tmp/ytdl.$$
for url in "$@"; do
  youtube-dl -x "$url"
  for f in *; do
    ffmpeg -i "$f" -vn -c:a libopus "$f.opus"
    rm "$f"
    mv "$f.opus" /radio
  done
done
cd /
rm -rf /tmp/ytdl.$$
ls -1 /radio/*.opus > /radio/playlist.m3u

There is one new feature in this shell script: playlist management. After we've ingested new files into /radio, we rebuild a .m3u playlist. This is a simple file: one file path per line, read sequentially. Piping the ls output through sort ensures that we have a stable playlist order; we'll let other systems randomize playlist ordering if they wish.

We can copy that script (which I've unimaginatively named entrypoint) into our docker image via the Dockerfile:

FROM ubuntu:20.04
RUN apt-get update \
 && DEBIAN_FRONTEND=noninteractive apt-get install -y python3 python3-pip ffmpeg \
 && pip install youtube-dl

WORKDIR /radio

COPY entrypoint /usr/bin/entrypoint
RUN chmod 0755 /usr/bin/entrypoint
ENTRYPOINT ["entrypoint"]

With this change in place, we go back to invoking our container (after a fresh build!) with just the URL we want downloaded and transcoded:

$ docker run -v $PWD/radio:/radio youtube-dl https://youtu.be/D7hxDYBoHvQ?list=PLGDWYlG32yKoYAuREw9eSLFIdWJZ_3rW8
[youtube:tab] Downloading playlist PLGDWYlG32yKoYAuREw9eSLFIdWJZ_3rW8 - add --no-playlist to just download video D7hxDYBoHvQ
[youtube:tab] PLGDWYlG32yKoYAuREw9eSLFIdWJZ_3rW8: Downloading webpage
[youtube:tab] PLGDWYlG32yKoYAuREw9eSLFIdWJZ_3rW8: Downloading webpage
[download] Downloading playlist: Rent / Buy / Build Episodes
[youtube:tab] playlist Rent / Buy / Build Episodes: Downloading 1 videos
[download] Downloading video 1 of 1
[youtube] D7hxDYBoHvQ: Downloading webpage
[youtube] D7hxDYBoHvQ: Downloading MPD manifest
[download] Destination: Episode 0 - What's All This Then-D7hxDYBoHvQ.m4a
[download] 100% of 9.73MiB in 00:0123MiB/s ETA 00:005
[ffmpeg] Correcting container in "Episode 0 - What's All This Then-D7hxDYBoHvQ.m4a"
[ffmpeg] Post-process file Episode 0 - What's All This Then-D7hxDYBoHvQ.m4a exists, skipping
[download] Finished downloading playlist: Rent / Buy / Build Episodes
ffmpeg version 4.2.4-1ubuntu0.1 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-10ubuntu2)
  configuration: --prefix=/usr --extra-version=1ubuntu0.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'Episode 0 - What's All This Then-D7hxDYBoHvQ.m4a':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2mp41
    encoder         : Lavf58.29.100
  Duration: 00:10:30.10, start: 0.000000, bitrate: 129 kb/s
    Stream #0:0(eng): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 128 kb/s (default)
    Metadata:
      handler_name    : ISO Media file produced by Google Inc.
Stream mapping:
  Stream #0:0 -> #0:0 (aac (native) -> opus (libopus))
Press [q] to stop, [?] for help
[libopus @ 0x560757a9e280] No bit rate set. Defaulting to 96000 bps.
Output #0, opus, to 'Episode 0 - What's All This Then-D7hxDYBoHvQ.m4a.opus':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2mp41
    encoder         : Lavf58.29.100
    Stream #0:0(eng): Audio: opus (libopus), 48000 Hz, stereo, flt, 96 kb/s (default)
    Metadata:
      handler_name    : ISO Media file produced by Google Inc.
      encoder         : Lavc58.54.100 libopus
      major_brand     : isom
      minor_version   : 512
      compatible_brands: isomiso2mp41
size=    6139kB time=00:10:30.11 bitrate=  79.8kbits/s speed=72.5x
video:0kB audio:6085kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.876994%

$ ls -l radio
total 6144
-rw-r--r-- 1 jhunt staff 6286163 Mar 22 14:52 "Episode 0 - What's All This Then-D7hxDYBoHvQ.m4a.opus"
-rw-r--r-- 1 jhunt staff      61 Mar 22 14:52  playlist.m3u

We can even do multiple!

$ docker run -v $PWD/radio:/radio youtube-dl \
  ... \
  ... \
  ...

You're On The Air With Icecast

Time to turn up the transmitters and start broadcasting! For that, we need Icecast. As before, we'll start with an Ubuntu 20.04 base and add the packages we need.

FROM ubuntu:20.04
RUN apt-get update \
 && apt-get install -y icecast2

Icecast won't let just anybody stream music to it. That would be pure chaos! This is a broadcast radio station, not a peer-to-peer free-for-all. We chose the tunes. We pick the commercials. We write the jingles. To keep out the riff-raff, Icecast uses speakeasy authentication. Each source driver is required to provide the secret password before we'll accept its audio stream.

The Icecast core team definitely has a sense of humor, because in the default configuration, the source password is "hackme". We need to change that.

It's important to remember that Icecast was born when XML was all the rage and came of age before it was fashionable to configure things via environment variables. To change the source driver password, we need to modify the configuration file: /etc/icecast2/icecast.xml.

FROM ubuntu:20.04
ARG PASSWORD
RUN apt-get update \
 && apt-get install -y icecast2 \
 && sed -i -e "s/>hackme</>$PASSWORD</" /etc/icecast2/icecast.xml

This sed command finds all instances of ">hackme<" and replaces it with ">$PASSWORD<". The new ARG directive sets up a Docker build argument, which is a bit like an environment variable that only exists when the container image is built. We specify it thusly:

$ docker build --build-arg PASSWORD=sekrit -t icecast .

Let's take it for a spin and make sure everything is ship shape.

$ docker run icecast icecast2 -c /etc/icecast2/icecast.xml
[2021-03-22  19:04:15] WARN CONFIG/_parse_root Warning,  not configured, using default value "localhost". This will cause problems, e.g. with YP directory listings.
[2021-03-22  19:04:15] WARN CONFIG/_parse_root Warning,  not configured, using default value "Earth".
[2021-03-22  19:04:15] WARN CONFIG/_parse_root Warning,  contact not configured, using default value "icemaster@localhost".
[2021-03-22  19:04:15] WARN fserve/fserve_recheck_mime_types Cannot open mime types file /etc/mime.types
ERROR: You should not run icecast2 as root
Use the changeowner directive in the config file

Drat.

Icecast doesn't want to run as root, so we're going to need to provision a user account inside the container.

FROM ubuntu:20.04
ARG PASSWORD
RUN apt-get update \
 && apt-get install -y icecast2 \
 && sed -i -e "s/>hackme</>$PASSWORD</" /etc/icecast2/icecast.xml \
 && useradd radio \
 && chown -R radio:radio /etc/icecast2 /var/log/icecast2
USER radio
ENTRYPOINT ["icecast2"]
CMD ["-c", "/etc/icecast2/icecast.xml"]

The useradd command updates the files inside the container (notably, /etc/passwd), while the USER directive tells Docker that it should switch to that effective UID when the container starts up.

I also took the liberty of setting an entrypoint and some arguments. When you specify both of them, you get the ability to provide "default" arguments (CMD) to the base command of the docker image (ENTRYPOINT).

That's all there is to the Icecast image. Running it requires a little bit more effort, but not much. Notably, we need to provide a port so that listeners can find the radio station, and so that source drivers can stream in their audio. We do this by forwarding ports from the Docker host into the container, like this:

$ docker run -p 8000:8000 icecast
[2021-03-22  19:11:48] WARN CONFIG/_parse_root Warning,  not configured, using default value "localhost". This will cause problems, e.g. with YP directory listings.
[2021-03-22  19:11:48] WARN CONFIG/_parse_root Warning,  not configured, using default value "Earth".
[2021-03-22  19:11:48] WARN CONFIG/_parse_root Warning,  contact not configured, using default value "icemaster@localhost".
[2021-03-22  19:11:48] WARN fserve/fserve_recheck_mime_types Cannot open mime types file /etc/mime.types

Inside the container namespaces, Icecast is binding all interfaces on port 80. Outside, Docker (with a little help from iptables) is forwarding traffic bound for the host on port 8000 into the container, on port 8000. (Incidentally, this is usually done via a Linux bridge and two ends of a veth pair, but that's a topic for a different, more nerdy post.)

You can see the port assignments in both the docker ps output:

$ docker ps
CONTAINER ID   IMAGE     COMMAND                  CREATED        STATUS                  PORTS                  NAMES
52993e74a1b3   icecast   "icecast2 -c /etc/ic…"   1 second ago   Up Less than a second   0.0.0.0:8000->8000/tcp   friendly_albattani

and also via netstat or lsof (pick your favorite!):

# lsof -nP -i TCP
com.docke 29389 jhunt   41u     IPv6  0xb3ce8809472e12f         0t0                 TCP *:8000 (LISTEN)

Let's leave Icecast to its thoughts, and move onto setting up our audio source.

The Magic That Is LiquidSoap

What is LiquidSoap? MAGIC.

I truly, and fervently mean that. Their website is more modest:

LiquidSoap is a powerful and flexible language for describing audio and video streams. It offers a rich collection of operators that you can combine at will, giving you more power than you need for creating or transforming streams.

more power than you need – that about sums up what we're going to use it for. We have audio files that we ripped right out of some Internet videos, and we have a radio broadcasting engine just waiting to be fed some juicy audio streams. We need a way to hook those up. And we're going to use LiquidSoap.

Here's the humble .liq script we're going to use:

output.icecast(%opus,
  host = "10.128.0.56",
  port = 8000,
  password = "sekrit",
  mount = "pirate-radio.opus",
  playlist.safe(reload=120,"/radio/playlist.m3u"))

But before we get ahead of ourselves, let's build out our Docker image for running that little gem.

You know the drill; start with Ubuntu and add the packages you need. As we've done for every image so far, we're going to set an ENTRYPOINT so that we can pretend our container is the liquidsoap command.

FROM ubuntu:20.04
RUN apt-get update \
 && apt-get install -y liquidsoap
USER nobody
ENTRYPOINT ["liquidsoap"]

As with Icecast, we set a USER, so that Liquidsoap doesn't get angry about us being root and all. This time, however, we're just going to use the default nobody account that comes with Ubuntu.

Convinced that the image is fit and fine, we can revisit that .liq script. Here's a version with some more comments.

# radio.liq
#
# This script streams audio from files on-disk,
# in an m3u playlist, to our Icecast server in
# opus format.
#
output.icecast(%opus,
  host = "10.128.0.56",        # where is Icecast?
  port = 8000,                 # what port is it bound on?
  password = "sekrit",         # what's the secret pass phrase?
  mount = "pirate-radio.opus", # what should we call this stream?

  # create an infallible playlist (hence 'safe')
  # using the files on-disk, in /radio.
  #
  playlist.safe(reload=120,"/radio/playlist.m3u"))

(Remember, LiquidSoap is a fully-fledged programming language. You can do all sorts of crazy things with it.)

Notably, we're going to output a stream to our Icecast server. Because the container has it's own network stack (containers!) I've specified the IP of the Docker host itself as the Icecast host endpoint. This causes packets to route out of the container and eventually find their way through our port-forwarding rules and into the other container. In the next part of this series, when we bring Docker Compose into the picture, we'll sand that particular rough patch down sufficiently.

Remember that m3u playlist we made while we were ingesting new audio files via youtube-dl? That's what this LiquidSoap script is going to consume while it builds its audio stream. I want to point out something that may not be obvious at first: the paths specified in the playlist must be valid when our script goes to consume them. Keeping everything always mounted at /radio suffices, but it's not the only way.

This is it, the final piece of the puzzle. Let's spin up that docker container and see if it works.

$ docker run -d --rm --name icecast -p 8000:8000 icecast
$ docker run --name liquid -v $PWD/radio:/radio 'output.icecast(%opus, host=...'
...
2021/03/22 20:09:17 [playlist(dot)m3u:3] Loading playlist...
2021/03/22 20:09:17 [playlist(dot)m3u:3] No mime type specified, trying autodetection.
2021/03/22 20:09:17 [playlist(dot)m3u:3] Playlist treated as format application/x-mpegURL
2021/03/22 20:09:17 [decoder:3] Method "OGG" accepted "/radio/Episode 0 - What's All This Then-D7hxDYBoHvQ.m4a.opus".
2021/03/22 20:09:17 [playlist(dot)m3u:3] Successfully loaded a playlist of 1 tracks.
2021/03/22 20:09:17 [decoder:3] Method "OGG" accepted "/radio/Episode 0 - What's All This Then-D7hxDYBoHvQ.m4a.opus".
2021/03/22 20:09:17 [playlist(dot)m3u:3] Prepared "/radio/Episode 0 - What's All This Then-D7hxDYBoHvQ.m4a.opus" (RID 2).
2021/03/22 20:09:17 [pirate-radio(dot)opus:3] Connecting mount pirate-radio.opus for source@192.168.88.225...
2021/03/22 20:09:17 [pirate-radio(dot)opus:3] Connection setup was successful.
2021/03/22 20:09:17 [clock.wallclock_main:3] Streaming loop starts, synchronized with wallclock.

The logs for the LiquidSoap container clearly show that the source was able to connect and start streaming in audio. Now we can visit the web interface (http://:8000), which should look something like this:

Congratulations. You're officially an Internet radio DJ. Now, click that play button, enjoy the show, and go brainstorm a better radio handle.

In the next part of the series, we'll gather up all of the port forwarding rules, commands, and volume bind-mounts, and wrap it all up in a neat little package using Docker Compose. This will make our lives easier as we iterate on the whole system.

Implementing Non-Trivial Containerized Systems - Part 1: Picking Components

James Hunt — Thu, 06 May 2021 05:18:56 +0000

So, you want to start a radio station, eh?

This is the first part of a multi-part series on designing and building non-trivial containerized solutions. We're making a radio station using off-the-shelf components and some home-spun software, all on top of Docker, Docker Compose, and eventually, Kubernetes.

In this part, we're going to explore how the different parts of the system interface with one another, to set the stage for our next post, where we Dockerize everything!

I first met Icecast (https://icecast.org/) when I worked at a web-hosting startup around the turn of the millennium. One night, one of my co-workers and I had the crazy idea to load a bunch of audio files on the networked file server and stream them to our workstations. We could listen to music while we worked 90+ hours a week. Strange times. After realizing it wasn't as simple as exporting .ogg files over HTTP, we found Icecast (and its pal, Ices2) and built a rudimentary, local-network broadcast radio station.

That's sort of what Icecast does – it makes streams of audio available to clients (er, listeners) over HTTP. These streams are analogous to old-school AM/FM radio stations. The listener can't pick what gets played (not directly, at least), and they can't control where they are in the stream. Wherever they connect, that's what they listen to.

Icecast is actually more of a go-between. It will keep track of which streams exist, and connect listeners to those streams, but it is not responsible for what's in the streams. For that, you need a source.

A source can be just about anything; a physical audio input device (i.e. a microphone), a CD player, audio files, you name it. The go-to source driver for Icecast is Ices, but we're going to use something with a bit more functionality; a little something called LiquidSoap.

LiquidSoap (https://www.liquidsoap.info/) is really cool. Like, phenomenally powerful. We're going to use it to combine multiple audio files into a single stream and send that off to Icecast. This doesn't even begin to scratch the surface of what LiquidSoap is capable of. I hope it isn't insulted.

Finally, we need to talk about where we are going to get the actual audio that we want LiquidSoap to bundle up and stream to Icecast. If I were a good teacher, this paragraph would end with a link to an S3 bucket or a GitHub project or some internet endpoint where you can download test files to play with, and we could move on.

I'm not a good teacher. You're going to work for that audio.

We're actually going to build a means of acquiring audio into the system itself, using YouTube as our primary source. For that, we'll use a handy little tool called youtube-dl, in concert with ffmpeg, another sharp little tool that makes quick work of audio video files. Here's how it will work:

We start with a YouTube(dot)com video URL, which we will feed to youtube-dl. From that, we get a video file, on-disk. Since we're only interested in the audio track, we'll feed that video file into ffmpeg to rip out just the audio track into a more appropriate audio file format (like opus / ogg).

We have arrived at our final architecture.

Next up, we'll start putting things into containers, and see if our design will actually work.

Implementing Non-Trivial Containerized Systems

James Hunt — Sat, 24 Apr 2021 01:59:46 +0000

Your average blog post / tutorial / video about containerizing software goes a little something like this:

Trouble is, all that "simple Docker stuff" is where things can (and often do!) go horribly horribly wrong, and there is precious little material available out there on the Internet for helping you through those precarious bits. No more!

In this series, I'm going to take you through the complete containerization effort, using a nontrivial application composed of both off-the-shelf components and some home-spun software.

Together, we're going to build and containerize an Internet radio station. Let's meet the software we're going to be using "off the shelf" to do most of the heavy lifting:

Icecast2 - https://icecast.org/
Liquid Soap - https://www.liquidsoap.info/
ffmpeg - https://www.ffmpeg.org/
youtube-dl - https://youtube-dl.org/

These four components will allow us to implement an audio delivery system, over TCP, that randomizes tracks that we pull off of YouTube. I'm going to make episodes of the Rent / Buy / Build podcast available.

As we build out this solution, we're going to climb the abstraction tower. We'll start by building and running our containers by hand, using nothing but the docker command. This is the proof-of-concept phase: does the software work together, and can we make all the components play nice and do what we want?

This first stage is usually a very messy one, with lots of dead-ends, failed experiments and retries. I consider this a feature of the process, not a bug. Containers afford us the freedom to play and experiment, without risk of polluting our system with unwanted packages, unused processes, or broken configurations. If something doesn't work, all we have to do is shut down the container and delete the image.

After we've worked out precisely which components we need, and how they work together, we'll move onto the orchestration stage. Using Docker Compose, we'll make some infrastructure code that can be used to deploy all of the pieces and parts together, as a cohesive and contained unit. It is at this stage that we capture the relationships between the containerized components and their data stores, explicitly.

At this point, I like to take a step back and determine if the whole system is complete: does it lack features, user interfaces, or anything else? Anything that is missing usually doesn't exist on its own, and needs to be built. For this particular project, we're going to need a pretty web interface for interacting with our radio station, and driving new track ingestion.

I specifically wait until I've finished assembling components before I go off and build any of the missing bits. I find that this helps to (a) keep the scope of anything I build to a minimum, and (b) ensures that the main backbone of the system is solid and functional before I expend any substantial effort writing glue code. Custom software ain't cheap; not only does developing software take time from other projects, but it also places some non-trivial obligation on future-you's time, vis-a-vis maintenance work.

Once all the gaps are filled, and we're happy with the solution, we move onto the final stage: converting our docker compose recipe into a Kubernetes deployment or three, so that we can run and scale and survive node outages. While this stage builds heavily on the previous stages – Docker Compose and Kubernetes share a lot of the same concepts, after all – it also demands its own specialized bag of tricks.

When all is said and done, we'll have taken our idea – wouldn't it be cool to run an Internet radio station – from concept to production-grade implementation, using containers all the way.

Let's get started, shall we?

Join us next week for part 1, where we dive into assembling the off-the-shelf components into a working system, fit for containerization.