DEV Community: Maciej Mionskowski

How to pass environment variables to front-end container images

Maciej Mionskowski — Sun, 10 Apr 2022 16:19:07 +0000

Environment variables are a standard way to parametrize backend containers. For some reason, they haven't seen wide adoption on the frontend side, which just as much requires customization. Both React and Vue still recommend creating separate .env files for different environments, which is unwieldy at best if you want to containarize the application. In this tutorial, I will guide You through an opinionated way to create environment agnostic frontend images in React.

What are the advantages of environment agnostic frontend images?

Reduced CI pipeline time - single build pass means no need to create three different images for your development, staging, and production environments
Simplified environment promotion - deploy an image to staging environment and promote it to production once all tests pass
Mitigated risk of deploying improper image to production environment

How to add an API URL environment variable to frontend Docker images?

The most common use case for environment variables on the frontend side is to have a customizable backend url for dev, staging and production environments respectively.
This example is based on a React app created using create-react-app. But the examples can be easily ported to Vue or even Next with slight modifications.

Step 1: Create `/public/env.js` file

You should put values related to the local development environment there. You might decide to commit the file to the code repository assuming that all local environments will have the same configuration.

window.env = {}
window.env.API_HOST = 'http://localhost:10001' // local development API_HOST if applicable

Step 2: Create a `script` tag in `index.html`'s `<head>` section pointing to the file created previously.

It is important to load the file before loading any other javascript that will use the variables, thus <head> seems to be a good place.

<head>
    ...
    <script src="%PUBLIC_URL%/env.js"></script>
</head>

Step 3: Create a `docker` directory

This is where all image related files will live to reduce clutter in the project root.

Step 4: Create `50-substitute-env-variables.sh` under `/docker`

The 50-substitute-env-variables.sh script will be responsible for substituting environment variables in container runtime. It will utilize a built-in feature in the nginx image that runs scripts from /docker-entrypoint.d/ directory.

#!/usr/bin/env sh

set -o errexit
set -o nounset 
set -o pipefail

: "${API_HOST}" # ensure API_HOST exists and exit otherwise

cat <<EOF > /usr/share/nginx/html/env.js
window.env = {};
window.env.API_HOST = "$API_HOST";
EOF

Don't forget to make it executable by running chown +x 50-substitute-env-variables.sh

Step 5: Create `nginx.conf` under `/docker`

You might want to tweak the try_files directive based on the router you use. The configuration below will try to load a file if it exists and the index.html otherwise.

user nginx;

worker_processes    auto;

events { worker_connections 1024; }

http {
    server {
        server_tokens off;

        listen  80;
        root    /usr/share/nginx/html;
        include /etc/nginx/mime.types;

        location / {
            try_files $uri $uri/ index.html =404;
        }
    }
}

Step 6: Create a `Dockerfile` under `/docker`

We will use multi-stage Docker image to reduce the image size. Note that You should bind both node and nginx images to some version.

FROM node:current as build

WORKDIR /src

COPY package.json /src

RUN npm install

COPY . /src

RUN npm run build


FROM nginx:alpine

RUN rm -rf /usr/share/nginx/html/*
COPY --from=build /src/build /usr/share/nginx/html/
COPY /docker/nginx.conf /etc/nginx/nginx.conf
COPY /docker/50-substitute-env-variables.sh /docker-entrypoint.d/

At the end of this step the directory structure should look as follows.

/app
    /docker
        50-substitute-env-variables.sh
        Dockerfile
        nginx.conf

Step 7: Reference the environment variable in code

You can reference the API_HOST variable under window.env.API_HOST, for example:

function App() {
  const apiHost = window.env.API_HOST

  return (
    <div className="App">
      <p>
        API Host: {apiHost}
      </p>
    </div>
  );
}

Step 8: Build the image

From app's root directory execute:

docker build -f docker/Dockerfile -t docker.your-company.com/app:version .

After successful build, you can start the container by typing:

docker run --rm -e API_HOST=http://prod.company.com/ -p 8080:80 docker.your-company.com/app:version

In case you forget to specify the environment variable the container will exit with:

/docker-entrypoint.d/50-substitute-env-variables.sh: line 7: API_HOST: parameter not set

You can now access the container under 127.0.0.1:8080.

The full code is available on Github.

The journey of sharing a wired USB printer over the network

Maciej Mionskowski — Mon, 04 Apr 2022 17:17:03 +0000

I was in the market for a printer that was cheap to buy and cheap to run. I did not print in color, so I concluded that a ~~dot matrix~~ laser printer would be a good choice.
I looked up a couple of units and decided on Brother DCP-1510 as it was on sale for ~$100 with replacement toners running for $8 apiece. Not a bad deal. It had one caveat - no ethernet port, no WiFi support, and no Internet Printing Protocol.

I did not need all the features, but an ethernet port would be nice
Courtesy of XKCD

That did not put me off. I have never been printing much and could live with the cable. I also knew I could attach it to a print server such as CUPS and add networking capabilities to the network-impaired printer. I had a Raspberry Pi Zero sitting around, and I wouldn't hesitate to use it. This post is a story about the journey I experienced setting it up.

The driver support or lack thereof

I flashed Arch Linux ARM onto my RPi and went driver hunting. It turned out the manufacturer does not support the ARM architecture.

After a couple of DDG searches, I found brlaser - a community-driven Brother driver.
Perfect! I installed CUPS, compiled the driver, and shared the printer over the network.

Brlaser as seen in the driver selection prompt.

I clicked the Print button.

Why does it take 45 seconds to print a single page‽

My eyes turned towards the printer anticipating the first sheet of paper to appear quickly, but nothing happened. Not until I tried to ssh into the Pi and started debugging. After a couple of seconds I began to hear the brrr printer noise while it was spitting out the page. I tried to print a second one as I thought it needed some priming, but it also took well over half a minute.

I clicked Print again and began observing the CUPS interface. It took 30 s to get through the Processing Page state. Something was off.

As it turns out, Raspberry Pi Zero is not that powerful. The driver uses ghostscript which pegged the CPU usage to 100%. Back to the drawing board.

Can I share an arbitrary USB device over the network? Sure I can!

I wasn't aware of the RAW queue mode in CUPS at the time, but I heard about usbip, which sounded like a compelling solution to the problem, or so I thought.

I followed a great Arch Wiki tutorial on usbip and sure enough, I had my printer attached.

The need to load a kernel module was a bit unsettling, but I installed the drivers and configured the printer using CUPS successfully.

I hit Print and almost immediately had the page handy. Whoa, that was easier than expected!

It's not all roses 🌹

I rebooted my laptop the next day and tried to print again, a real document this time.

It wouldn't work - the device was gone - I couldn't detach the device, I couldn't attach a new one, I tried reloading the kernel module, and it wouldn't work either, nada.

At that point, I concluded I'm not in favor of running a kernel module that's misbehaving
I also had some security concerns around the whole architecture and therefore I uninstalled it.

Learning about RAW queues

I came back to the problem after a couple of days, but now I had a powerful tool under my belt: knowledge about CUPS RAW queues.

A RAW queue is

[...] a queue where the filtering system is not involved and the print job goes directly to a printer or another queue:
https://wiki.debian.org/CUPSPrintQueues"

This seemed promising for my use case. The idea was to set up a RAW queue on the Pi and then do the heavy lifting (filtering) on significantly more powerful user machines.

.----------------.         .--------------.        .---------.
|       PC       |         | Raspberry PI |        .         .
|   (filtering)  +-------->|  (raw queue) +------->| Printer |
| (brlaser + gs) |         |(no filtering)|        '         .
'----------------'         '--------------'        '---------'

I quickly compiled this setup and it worked out perfectly.

Note: CUPS plans to deprecate drivers and raw queues in the future because of how wide-spread IPP has become. I don't consider this to be a big issue, you can always pin the CUPS version. There's still a ton of old hardware out there, and that won't change quickly (and doesn't have to).

The SD card gives up

After a few weeks, the SD card running the CUPS server gave up.
I bought a new one and tried to quickly reinstall everything, but Arch Linux ARM abandoned armv6 architecture in the meantime. I decided to use Raspberry Pi OS and automate the setup.

Automating the build process

I like to have my infrastructure defined in code and I maintain a number of Ansible playbooks and Terraform workspaces to control my servers.

Packer seemed like the perfect tool for the job.
I have never used it before and wanted to get familiar with the tool.
It doesn't come with ARM support out of the box, but there are two community projects to fill that niche.

I tried the packer-builder-arm first, but it couldn't run my Ansible playbook due to a bug. I applied a patch but quickly ran into other issues.

At that point, I decided to use packer-plugin-arm-image instead. The setup did not work out of the box, but after a simple PR, it built my first empty image and proved it's possible to build an ARM image locally.

Defining goals

I wanted an end-to-end setup with the following functionality:

OS installation (Raspberry Pi OS)
WiFi setup
Autodiscovery via mDNS & DNSSD
CUPS installation
Printer configuration

I developed two Ansible playbooks to accomplish that. I will skip the technicalities and maybe write about the details another time.

Run it yourself!

The final solution with a comprehensive README can be found under the maciekmm/printer-rpi-image github repo.

Building, flashing, and running the image yourself is as simple as running:

# build the base image
WIFI_SSID=<SSID> WIFI_PASSWORD=<PASSWORD> SSH_PUBLIC_KEY=$(cat ~/.ssh/id_ed25519.pub) vagrant up
# flash the image onto the sd card
dd bs=4M if=./output-raspberry_pi_os.img of=<sdcarddevice> && sync
# NOTE: this runs on the live Pi. Connect it first with printers attached!
# configure the printer and firewalls
ansible-playbook -i hosts live.yaml

What's left is configuring the driver and discovering the printer locally. This can be done by installing CUPS locally and running through the wizard.

Closing thoughts

I accomplished several things:

I made my wired printer wireless,
I learned how to use Packer,
I learned a bit about CUPS queues,
I published an open source project for you to be able to do the same.

Overall, this was an interesting project and I'm happy to share this story.

How to detect breaking changes and lint Protobuf automatically using Gitlab CI and Buf

Maciej Mionskowski — Sat, 02 Apr 2022 10:33:54 +0000

Protocol Buffers or Protobufs are language-agnostic
mechanisms for serializing data.
Protobuf schemas are specified using Protocol Buffer language, which is among the most popular and widely adopted IDLs in the industry.

Protobufs are most commonly used in RPC services for inter-service communication. Their usage is also growing for public-facing interfaces, and recently they have been adopted in tools such as Apache Kafka.

Used correctly, they enable both forward and backward compatible message producing and consuming. Meaning that old consumers/clients (using old Protobuf schema) can consume messages from new producers/servers and vice-versa. For me, this is the most compelling point in the offerings of Protobufs.

Breaking changes detection

To maintain backward/forward compatibility in Protobufs, every change to the schema must be thoroughly code-reviewed and tested for compliance. Humans are prone to making errors. Therefore, several tools have emerged to help with the process, most notably Buf.

We’re working quickly to build a modern Protobuf ecosystem. Our first tool is the Buf CLI, built to help you create consistent Protobuf APIs that preserve compatibility and comply with design best practices. The tool is currently available on an open-source basis.

To check if the current working copy is compatible with a previous revision, you can invoke.

buf check --against 'reference-to-a-previous-revision'

Where reference-to-a-previous-revision may be either a git repository reference or an image built by Buf.

To check against a particular branch, you can execute

buf check --against '.git#branch=master'

For all other means of referencing particular code revision, you can consult Buf's excellent docs.

Ensuring adequate compatibility constraints

Not all systems use Protobufs equal, some will serialize the message to JSON down the line, others will solely rely on binary messages. Buf is flexible in terms of defining an adequate compatibility level for a project.

Buf's docs provide a great overview of supported rules.

Detecting breaking changes using Gitlab CI

When using a Merge Request based flow it is usually enough to check every merge request for compatibility breaking changes against the target branch.

The solution does not cover all situations, most notably when changes are introduced without using merge requests, but supporting all cases would require building and storing buf images.

Besides a static binary, Buf also provides a docker image. Using it will simplify the workflow.

To check every merge request introduce the following snippet to your repository's .gitlab-ci.yml:

stages:
  - ensure backwards compatibility

validate merge request:
  stage: ensure backwards compatibility
  image: 
    name: bufbuild/buf:0.41.0
    entrypoint: [""]
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
  script:
    - buf breaking --against "${CI_REPOSITORY_URL}#branch=${CI_MERGE_REQUEST_TARGET_BRANCH_NAME}"

CI_REPOSITORY_URL expands to a URL that can be supplied to git clone. It includes a token, therefore, access management is of no concern.

Detecting breaking changes using other CI solutions

Buf's repository contains exemplary workflow definitions for:

Travis CI
Github Actions
Circle CI

Code style checking

Linters help to ensure code style consistency across files. With Protobuf it is no different.

Linting with buf is effortless and can be done by running buf lint.

If you encounter errors or warnings from buf, do not try to fix them absent-mindedly. Things will break if the schema is used in a production environment. Instead, make small, incremental changes and check for incompatibilities. Should one arise, you can make an exclusion and fix your mistake in the future interface version.

Lint automatically using Gitlab CI

To check every commit on every branch, introduce the following snipped to .gitlab-ci.yml.

stages:
  - lint

lint:
  stage: lint
  image: 
    name: bufbuild/buf:0.41.0
    entrypoint: [""]
  script:
    - buf lint
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
    - if: '$CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS'
      when: never
    - if: '$CI_COMMIT_BRANCH'

The rules attached to the lint stage will prevent multiple pipelines running for a merge request.

Combining linting with breaking changes detection

You can easily combine examples outlined above in a single .gitlab-ci.yml:

image: 
  name: bufbuild/buf:0.41.0
  entrypoint: [""]

stages:
  - lint
  - ensure backwards compatibility

lint:
  stage: lint
  script:
    - buf lint
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
    - if: '$CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS'
      when: never
    - if: '$CI_COMMIT_BRANCH'

validate merge request:
  stage: ensure backwards compatibility
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
  script:
    - buf breaking --against "${CI_REPOSITORY_URL}#branch=${CI_MERGE_REQUEST_TARGET_BRANCH_NAME}"

I have created a sample repository under https://gitlab.com/mionskowski/protobuf-ci. Navigate to Merge Requests to see both passed and failed pipelines.