DEV Community: Santiago

Secret management for the layman

Santiago — Sat, 16 Nov 2024 08:17:47 +0000

For a long time, I've been debating myself how to properly handle secrets in a convenient, cheap and reproducible fashion.

I run a small website: reciperium.com (check it out!) and I don't want to depend too much on the cloud.

Most of the available tools have the following problems:

Hard to setup
Hard to integrate with your infra
Hard to communicate / share with teammates (you can give access but they still have to discover the secrets, and build the files)

I've developed a straightforward solution using age, and git.

Secrets are stored as .age files in the git repo
Supports multiple users

I call this workflow "tracked secrets".

Setup

First thing, prevent your private key from being leaked into git.

For that we add to our .gitignore the private keys:

*.key

If you use nix, you can set up a development environment with all the dependencies needed. In my case, I use just and age for this.

{
  description = "My Application";
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
  };å
  outputs =
    inputs@{ flake-parts, ... }:
    flake-parts.lib.mkFlake { inherit inputs; } {
      systems = [
        "x86_64-linux"
        "aarch64-darwin"
        "x86_64-darwin"
      ];
      perSystem =
        {
          config,
          self',
          inputs',
          pkgs,
          ...
        }:
        {
          devShells.default = pkgs.mkShell {
            name = "reciperium";
            buildInputs = [
              pkgs.age
              pkgs.just
            ];
            shellHook = ''
              just --list
            '';
          };
        };
    };
}

You can also configure direnv, to load the nix dependencies, as soon as you cd into the directory.
This has been a powerful new method in my development toolkit.

Interface

The generic commands we are gonna define are the following

# Create new key and add it to the recipients
new-key

# Decrypt ALL secrets
decrypt

# Encrypt ALL secrets
encrypt

These commands can be implemented in a Makefile, justfile, bash scripts, or any scripting mechanism.

Workflows

With those 3 commands we can support several different workflows:

Initializing my host

The commands to be executed for this workflow should be:

new-key

This should create a me.key and populate recipients.txt with the public key.

If working on an existing repository, then I should create a commit, push the changes to git and ask to a colleague to encrypt the secrets again (after doing a pull)

Encrypting all the secrets

The command to be executed:

encrypt

This command can either encrypt everything necessary or it can be split into multiple commands internally.
It should encrypt with all the recipients present in recipients.txt.

Decrypting all secrets

decrypt

It should decrypt everything in the right place. And the user is ready to work!

Creating a secret for a CI

One of the cool benefits of this approach, is that we can integrate it with our CI.

new-key github
encrypt

That should generate a github.key, and it should populate the recipients.txt. You can upload the github.key as a secret and your pipelines are ready to use it!

The second command encrypts everything again.

Implementation example

Let's say we have a repository with our infrastructure. It has 2 folders, one with terraform code. And another with a docker-compose, which gets deployed to the server using docker-swarm.

app
├── infra-tf
│   ├── main.tf
│   └── terraform.tfvars
└── stack
    ├── .env
    └── docker-compose.yaml

The secrets of our application are terraform.tfvars and .env.

As I said, I've been using just for managing the commands, but you can use shell scripts or make.

Let's take a look at the justfile:

# Create a new private key and store public in recipients.txt
sec__new_key name="me":
    test -f "{{name}}.key" || age-keygen -o "{{name}}.key" && age-keygen -y "{{name}}.key" >> recipients.txt

# Decrypt ALL secrets
sec__decrypt:
    age -d -i me.key stack/stack.env.tar.gz.age | tar xvz
    age -d -i me.key infra-tf/infra-tf.tar.gz.age | tar xvz

# Encrypt ALL secrets
sec__encrypt: _sec__encrypt__infra _sec__encrypt__stack

# Encrypt terraform.tfvars
_sec__encrypt__infra:
    tar cvz \
        infra-tf/terraform.tfvars | \
        age -R recipients.txt > infra-tf/infra-tf.tar.gz.age

# Encrypt the stack environment files
_sec__encrypt__stack:
    tar cvz \
        stack/.env \
        age -R recipients.txt > stack/stack.env.tar.gz.age

Okay, let's recap. For creating a private key for your current host:

just sec__new_key

The "new key" command is self-service, a user pulls the repo, runs the just sec__new_key,
and then they push. Another user, with access to the secrets, runs the encrypt again. And finally, the original
user pulls the latest commits, so they can use decrypt locally.

For encryption, we run:

just sec__encrypt

And to decrypt:

just sec__decrypt

Finally, we want to create a secret for our CI

just sec__encrypt github

We can also on-board other users, but it involves sending the private key to them, which can be cumbersome, the recommendation is to let people onboard themselves, and they are responsible for their private key.

just sec__encrypt lara
just sec__encrypt jon

What to do with the private key?

After running just sec__new_key, we have a me.key. You can store a copy in your secret manager.

I personally use gopass, but there are more options, e.g: 1password, lastpass or bitwarden.

Potential improvements

How can we handle different environments?

One option is to have a secrets folder, with the different environments there.

secrets/
├── dev/
├── preview/
└── prod/

And we could make just accept different parameters when calling the commands:

just sec__encrypt prod

Final comments

Even though, I've seen a lot of people share their nixos configuration publicly, including their encrypted secrets, I still wonder if it's safe, and if so, why is it not used widely?
I want to believe it's safe, and I know that if the secrets get leaked, I'll have to update them all.
The plus side, is that you know where they are, and you can probably write comments on how to retrieve or generate them again, becaues it's just files.

What do you think of this approach?

Thanks for reading

Digesting Make and Makefiles

Santiago — Sat, 12 Mar 2022 14:40:13 +0000

Make and its makefiles, are a fantastic tool to keep track of the commands needed to build or run an application.

Just create a Makefile at the root of your project, start adding commands, and done... right? Not really.

Let's explore a bit more what I mean, and then try to build a mental model that matches reality a bit better,
closing with some magic you can do with Makefiles.

Set up a practice field by running in the terminal:

mkdir /tmp/practice
cd /tmp/practice
touch index.html

And write a minimal Makefile with your favourite editor (E.g: vim Makefile)

build:
    mkdir dist
    cp index.html dist/index.html

Then execute in your terminal:

make build

This will create a folder dist, and copy index.html into the folder dist.

This is good. Particularly for me, it documents the commands used by the project, and helps future me.

Let's refactor a bit, to show some extra functionality:

dist:
    mkdir dist

build: dist
    cp index.html dist/index.html

Notice the build: dist, it means: call dist command before running build

make build

The functionality remains the same.

Recap

We created two commands: build and dist.

And dist is executed by the build before running itself.

This is ok, but it's not necessary a correct mental model.
In the Makefile world, commands are not commands... but files.

New mental model

In a nutshell, make builds a dependency graph of files and folders.

build and dist are actually target files (or folders).
doing build: dist means that build depends on dist existing first.

If we had a file called build , doing make build, wouldn't execute anything.

touch build
make build
$ make: `build' is up to date.

If we want to actually treat build as a command, we have to add .PHONY.

dist:
    mkdir dist

.PHONY: build
build: dist
    cp index.html dist/index.html

This way Make no longer sees build as a file, but instead as a recipe, and it will be
executed everytime.

Back to targets

Remember what we said about targets and dependency graph? No? Me neither.
Don't worry, I actually didn't say anything.

If you treat your targets as files, Make can keep track of the files that have
changed, and update only those.

By knowing this our previous example could be refactored into:

dist:
    mkdir dist

dist/index.html: dist
    cp index.html dist/index.html

And now we tell Make to create our target file:

make dist/index.html

If we run multiple times, we get:

make: `dist/index.html' is up to date.

We would expect that by modifying index.html, and running make dist/index.html, it would be rebuilt,
but we are getting the same message.

Dependencies

We need to tell Make that it depends on another file: index.html

dist:
    mkdir dist

dist/index.html: dist index.html
    cp index.html dist/index.html

This time, if index.html is modified, make dist/index.html will run again.
Thus, when dependecies are updated, target files are recreated.

Patterns

Now let's say we have many html files

touch about.html privacy.html docs.html

And we want to do the same for all the files, without creating many commands in Make.
For this case we use a pattern (%), and some Make variables:

$@: the target file
$<: the input file

dist:
    mkdir dist

dist/%.html: %.html dist
    cp $< $@

make dist/about.html

And we would be explicitly telling Make which file to create.

But what if we want to copy all the files at the same time?

Our current implementation, doesn't know about the available files.
And we are providing the about to the make dist/about.html.

We have to find a way to "know" all the possible targets.

Variables

We are gonna find the source files (*.html files which are not in the dist folder), store in a variable,
and then use that information to create the target html files.

SRC_HTMLS := $(shell find . -name '*.html' -depth 1)

TARGET_HTMLS := $(SRC_HTMLS:./%.html=dist/%.html)

.PHONY: build
build: $(TARGET_HTMLS)
    $(info Done)

dist:
    mkdir dist

dist/%.html: %.html dist
    cp $< $@

make build

What happens is:

build has all the TARGET_HTMLS files as dependency
Make also sees dist/%.html and the pattern will fit the criteria for each TARGET_HTMLS
In dist/%.html we have as dependency the %.html, so Make takes the pattern and checks if a %.html file exist
If the conditions are met, it runs the cp command for each file, unless they are already present and not updated

You can think of

build: $(TARGET_HTMLS)

build: dist/index.html dist/about.html dist/docs.html dist/privacy.html

Variables can be reference using $() or ${}.

We also use $(info Done) to send information messages to the user.
Make also provides $(warning text…) for warnings, and
$(error text...) to exit earlier with an error code different than 0.
See Make-Control-Functions

One more time

Let's start over by removing the dist folder, and see what happens

rm -rf dist

❯ make build
mkdir dist
cp index.html dist/index.html
cp about.html dist/about.html
cp docs.html dist/docs.html
cp privacy.html dist/privacy.html
Done

❯ make build
Done
make: Nothing to be done for `build'.

As you can see, files that have not changed, won't be rebuilt by Make.

Now, try using touch on the different html files and running make build to see what happens.

More functionality

Make is a powerful tool, and provides much more functionality, so far with the web stack
I haven't had the need for more complexity.

I usually wrap docker commands, and make use of different variables, but if the need arises,
make has extra functionality, life:

functions
if/conditions
change the shell in which the commands are executed
.delete_on_error

Notes

Make is usually a good way to keep track of a project's commands, as it's available in Unix systems,
and you can be up and running fast.
But by no means is perfect, it can sometimes be hard to read, or use.
It's not available on Windows. And because it was designed for the C,C++ era, it plays well with files,
but it doesn't mean it fits perfectly the web development paradigm, where you don't "transform" files much
and where docker is used a lot.

So depending on your situation, there are some popular alternatives:

just: modern approach to make written in rust
earthly.dev: repeatable builds based on docker
bazel: build tool from Google
scripts-to-rule-them-all: just use scripts, like starlette

Resources

Thanks for reading! 👋

If you are interested in what I write, follow me on twitter

The Layout Team

Santiago — Mon, 08 Nov 2021 22:14:16 +0000

For the last couple of months I've had this idea spinning in my head, which I'm calling:

The Layout Team

Is a work in progress and I'll try to update it when new things come to my mind. The topic can be discussed forever, I will try to formalize the idea while keeping it short.

I see how the frontend industry is led mostly by hype, and this time I'm not fond of the direction we are going, specifically with micro-frontends. This pattern, spite of its benefits, I don't think it can be implemented properly by most teams, and it's not an idea we should keep suggesting.

Instead, I'm going to propose an alternative, mostly in the middle. And as you probably guessed... it's "The Layout Team".

As far as I'm concerned, the ultimate goal of a frontend is to deliver a good user experience, and this includes being fast.

Micro-frontends, make this target hard to achieve.
If you pull parts from all around it will take longer than pulling from a single place. Of course some teams can accomplish this (out of the question), and they may need it, but most of the time, is not required, but... what do we do then?

The main issue to me, is that a frontend application has to be glued together at some point, or somewhere. Whether you use a micro-frontend architecture or a monorepo, the final user has to experience one cohesive app, this is different to backends, there's no UI there, mostly machines talk with APIs. Your frontend talks with the API, but the human interacts with the frontend.

Hence the introduction of "The Layout Team" (I'm giving it a formal name).

This team could have many different flavours. But ideally, it should be an independent team, holding ownership of the layout of the app.

Yes, there's nothing fancy here, and the title is self-explanatory.

The Layout Team maintains the layout, and checks that everyone operates inside the boundaries created by this team.

Its responsibilities include:

Monitor styles to prevent overlapping components or breaking issues
Review Pull Requests
Train other developers, whether through quarterly presentations or one-to-one coaching, but do it consistently over time. Not fire and forget.
Maintain some shared state (logged user or is_authenticated or any other herbs). But most of the times teams should be able to add and manage their own global state
Write tools to assist other teams, like linters to prevent CSS or JS
Identify CSS or JS code that may affect the whole app, and potentially code them into the linters. Example:
- Do not use fixed/absolute because... (unless approved to do so)
- Do not use negative margins because we found that no one knows who to...
Write tests for the layout

One easy way to do this, is by having a monorepo. The layout for the different pages is defined by "The Layout Team", and the rest of the teams write components, which can be later placed in the places designated by the layout team.

By doing this, it becomes very easy to produce a small bundle. Because the dependencies are shared. It's then potentially easier, to identify shared code and cache it in a separate bundle across the application.

One of the "benefits" of micro-frontend is supposed to be the freedom for teams to choose what framework to use, but you end up sending bigger assets to the end user.
This goes against optimizing for the best user experience. So ideally, stick to a single framework, and deliver it once.

If you are using React in your monorepo, it means everyone will stick to that version. If you have multiple repos, even if everyone uses the same framework, you may end up with different versions, or even the same and still delivering them as part of each apps bundle!

Finally, The Layout Team leverages the use of flex and grid heavily.
They shape the app over time. And create slots for each team.

Let's see an example:

<div class="box grid grid-cols-2">
  <div maintainer="teamA">
    <ComponentFromTeamA/>
  </div>
  <div>
    <div maintainer="teamB">
      <ComponentFromTeamB/>
    </div>
    <div maintainer="teamC">
      <ComponentFromTeamC/>
    </div>
  </div>
</div>

- Hey! It's almost the same example as a micro-frontend!

- Well... yes, what did you expect?

Each team now has a space to place their components, and there's full visibility over who maintains what.

It is very important, that the people, that are part of this team, understand flex and grid very well.

Useful layout resources

I would very much like your feedback.

What has been your experience with micro-frontends?
Do you think "the layout team" would work?
What do you think of this proposal?

Thanks for reading

Changelog generation released for commitizen

Santiago — Mon, 04 May 2020 18:44:40 +0000

After a lot of work, time and effort, we were able to release commitizen v1.19 which includes automatic changelog generation.

pip install -U commitizen

With this release the cycle is complete, commitizen now supports:

custom commit rules
automatic version bump based on the rules
automatic changelog generation
git hook to check that the commit is valid

By default the system is using conventional commits but you can come up with your own rules.

Usage

For starters the configuration is straightforward, first, start with:

cz init

It will prompt the user for information about the repo and based on that it will generate the toml configuration file.

Once that is done, after you've done some commits complying with the rules, simply run:

cz bump --changelog

That's it.

You can execute that command in your CI and then push the changes.

For more in-depth explanations, check the documentation.

I hope this tool helps you automate your systems and relief you of some repetitive tasks.

Check commmitizen's github for some github actions, the repo is auto publishing docs, releases to pypi, testing, etc.

Simplest React Hook component for PWA install button

Santiago — Tue, 24 Dec 2019 12:50:49 +0000

This is the simplest snippet I could come up with to add a button which will prompt the user for the install pop up of your PWA.
It doesn't use any redux, nor nothing.

import React, { useEffect, useState } from "react";


const InstallPWA = () => {
  const [supportsPWA, setSupportsPWA] = useState(false);
  const [promptInstall, setPromptInstall] = useState(null);

  useEffect(() => {
    const handler = e => {
      e.preventDefault();
      console.log("we are being triggered :D");
      setSupportsPWA(true);
      setPromptInstall(e);
    };
    window.addEventListener("beforeinstallprompt", handler);

    return () => window.removeEventListener("transitionend", handler);
  }, []);

  const onClick = evt => {
    evt.preventDefault();
    if (!promptInstall) {
      return;
    }
    promptInstall.prompt();
  };
  if (!supportsPWA) {
    return null;
  }
  return (
    <button
      className="link-button"
      id="setup_button"
      aria-label="Install app"
      title="Install app"
      onClick={onClick}
    >
      Install
    </button>
  );
};

export default InstallPWA;

Any feedback is welcome.

React folder structure

Santiago — Sun, 15 Dec 2019 11:03:14 +0000

Hello people, I wanted to share my current folder structure for my React projects, after some time this is it:

.
├── package.json
├── jsconfig.json
├── public/
└── src/
    ├── assets/
    ├── components/
    └── pages/

components: Any component that don't belong to the pages should live here.

pages: Components living in this folder should map to a url. This is useful because if you want to move to Next.JS, there's not much needed to be done. This folder can have sub folders, if you are using reach-router, it fits perfectly.

assets: This folder is optional, it may include things as images, or any other static file that you'd like to put there, depending on the size of the app, the static content can also live next to each component, so it's not that needed.

And the jsconfig.json can have this content:

{
  "compilerOptions": {
    "baseUrl": "./src",
    "paths": {
      "pages": ["./src/pages/*"],
      "cards": ["./src/cards/*"],
      "components": ["./src/components/*"]
    }
  },
  "exclude": ["node_modules", "build", "coverage", "dist", "lib"]
}

So my question is, what do you think? Would you change it to something different?

Cheers

Question: how to create sass/css packages

Santiago — Fri, 15 Nov 2019 09:03:45 +0000

Hello people! I'd like to create a sass packages and I'd like to hear which tools do you use to manage, publish, etc.

I've realized that for the creation of JS packages, there's ton of documentation, but I haven't found much info on sass, css.

I don't really want to maintain webpack scripts, so I'm looking for some already pre-cooked libraries or skeletons.

Some questions I can think of:

Which tools do you use to create bundles? (webpack, parcel?, etc)
How do you publish? is there something to keep in mind?

Thanks!

Libraries for functional webcomponents?

Santiago — Tue, 29 Oct 2019 08:52:08 +0000

Hello people, I would like to open a discussion over functional webcomponents, do you know any? I really like the idea of React Hooks, and I enjoy them a lot. The documentation of webcomponents uses mostly classes. For external reasons I have to use webcomponents, I was wondering if people know of any good functional webcomponent package.
So far I've found hybrids but I have no references about it.

Some questions I'm thinking:

What do you think of webcomponents?
Would you like functional webcomponents?
Do you know any functional wc package?
Have you used any of them?

Regards and happy coding!