DEV Community: Rob Galanakis

Makefile and Dotenv

Rob Galanakis — Tue, 09 Jun 2020 12:00:00 +0000

In this series on Makefiles, we've covered how to use wildcards and using application presets rather than having to pass in values.

There's one more technique we use for controlling Makefiles:

Dotenv (.env files)

We love love love using .env files to configure our applications. If you're not familiar with them, the gist is that they specify environment variable keys and values in a very simple file format. For example, to control the LOG_LEVEL var, you can use:

LOG_LEVEL=info

And then load that at application startup (most languages have some sort of dotenv package for this).

While our full configuration practices are a topic for another day, we have a few rules we use on all of our projects:

All configuration through environment variables (do not load files, do not talk to external stores)
Always use .env files.
Never check in .env files.
Only have one .env file (no cascading files based on environment).
.env takes precedence over values in the environment.

Some of these decisions are no doubt controversial, but the goal is that .env files provide unambiguous, canonical storage of local environment variables, and non-local environments use whatever native mechanisms for injecting environment variables before starting the process (Heroku Config Vars, AWS Parameter Store hooked up to ECS, etc).

Make and .env

This ".env is the canonical storage of local environment" is so easy and standard, we can even set up Make to use it!

We put this block at the top of Makefile- it looks for a .env file, and if present, it will set Make variables from it:

ifneq (,$(wildcard ./.env))
    include .env
    export
endif

Where is this useful? Well, let's say you want to connect to your local application's database. We can put that database URL into the .env file:

DATABASE_URL=postgres://appuser:pass@localhost:13005/myapp

(We always run our databases through docker-compose and use different ports for different projects- you won't ever go back to :5432 once you start doing it this way)

Now in our Makefile, we can set up a target to connect via psql. Note the cmd-exists-% target from last week

ifneq (,$(wildcard ./.env))
    include .env
    export
endif

cmd-exists-%:
    @hash $(*) > /dev/null 2>&1 || \
        (echo "ERROR: '$(*)' must be installed and available on your PATH."; exit 1)

psql: cmd-exists-psql
    psql "${DATABASE_URL}"

Now run it from the command line:

$ make psql
 psql "postgres://appuser:pass@localhost:13005/myapp"
 psql (12.2, server 11.4 (Debian 11.4-1.pgdg90+1))
 Type "help" for help.

Docker too!

One more nice thing for this super-flexible .env file! Docker and docker-compose can accept an --env-file param, and by keeping everything in a single .env, you can share all the same configuration between your local application, Make, and Docker build.

Here's an adjusted Makefile snippet, and an example Docker command:

ifneq (,$(wildcard ./.env))
    include .env
    export
    ENV_FILE_PARAM = --env-file .env
endif

docker-server:
    docker run --rm -it -p $(PORT):$(PORT) $(ENV_FILE_PARAM)

The $PORT, of course, is defined in the .env file :)

More next time

Later this week we'll talk about one more neat thing we do with Make to standardize even further. If you are writing any JavaScript in particular, it's life-changing.

Finally, we'll wrap up this series in one week by talking about why we are all-in on Make as a task runner (if we tried to sell you on it in the first post you probably would have ignored us).

Makefile Application Presets

Rob Galanakis — Tue, 26 May 2020 13:30:21 +0000

We saw in the last post how to use Makefile wildcards to write targets like this:

migrate-to-%:
    @bundle exec rake migrate[$(*)] 
guard-%:
    @if [-z '${${*}}']; then echo 'ERROR: variable $* not set' && exit 1; fi
logs: guard-STACK
    @awslogs get -w /ecs/$(STACK)_MyService

So that we can build CLIs like this:

$ make migrate-to-50
Migrating to version 50...

$ make logs
ERROR: variable STACK not set
$ STACK=qa make logs
...

(Note the @ prefix on commands in the Makefile, it avoids the line being echoed to stdout)

This is neat but it only works well for user-supplied values, like "50".There are cases where we want the user to supply an argument , but not the value.Say, for example, users want to specify ‘production’ to ‘staging’but we don’t want them to remember the URL to the server.

We can use wildcards to dynamically select a Make variable:

staging_url:=https://staging-api.lithic.tech
production_url:=https://api.lithic.tech

ping-%:
    curl "$($(*)_url)"

And we can use it as so:

$ make ping-staging
curl "https://staging-api.lithic.tech"

Okay, this example isn’t incredibly useful.But for some clients, we have multiple deployed versions of the same application,and we can use these variables to avoid having to remember where applications are deployed.

For example, let’s say we have 3 versions of a codebase deployed in Heroku: one staging and two production apps.In the Make snippet below, each _app variable refers to the name of a Heroku app.We can use that app name to get the database string using the Heroku CLI,and pass that to psql (Postgres CLI).

staging_app:=lithic-api-staging
production-pdx_app:=lithic-api-production
production-nyc_app:=lithic-api-production-nyc

psql-%:
    psql `heroku config:get DATABASE_URL --app=$($(*)_app)`

Now to connect to staging, it’s as simple as:

$ make psql-staging

If we use Heroku’s Review Apps, we should also support an environment-variable version of these sorts of commands,since the app names are ephemeral. Instead of a wildcard, we’ll require the APP environment variable is set:

psql-app: guard-APP
    psql `heroku config:get DATABASE_URL --app=${APP}`

Putting It All Together

The example above has a couple small error cases that may confuse users:if psql or heroku are not on the PATH,the command will error with a sort of cryptic error:

$ make psql-staging
psql `heroku config:get DATABASE_URL --app=lithic-api-staging`
/bin/sh: heroku: command not found
could not connect to server: No such file or directory

Yuck! Can we use something like our lovely guard-% target to declare our executable dependencies?

You bet we can!

cmd-exists-%:
    @hash $(*) > /dev/null 2>&1 || \
        (echo "ERROR: '$(*)' must be installed and available on your PATH."; exit 1)

guard-%:
    @if [-z '${${*}}']; then echo 'ERROR: environment variable $* not set' && exit 1; fi

psql-%: cmd-exists-heroku cmd-exists-psql
    psql `heroku config:get DATABASE_URL --app=$($(*)_app)`

psql-app: guard-APP cmd-exists-heroku cmd-exists-psql
    psql `heroku config:get DATABASE_URL --app=${APP}`

Now if you’re missing heroku or psql, you get a nice message:

$ make psql-staging
ERROR: 'heroku' must be installed and available on your PATH.

Who knew Make could be so fun to use?

Not done with Make yet

We’ll have one more blog post involving Make next week, along with a dump of a bunch of useful Makefile helpers.Stay tuned!

This was originally posted on my consultancy’s blog, at https://lithic.tech/blog/2020-05/makefile-apps. If you have any questions, please leave a comment here!

Beautiful Makefiles with Wildcards

Rob Galanakis — Fri, 22 May 2020 18:30:14 +0000

Every single project we build includes a Makefile as a task runner. Every. Single. One.

Why? Because it allows someone to jump into a codebase and start working with the same set of tools and commands as everyone else.

Want to know how to install, build, test, deploy, and see what else you can do?Just open Makefile.

This may not be so important if you spend all your time on one monolith, but if you’re jumping around to many services or clients, it’s a lifesaver.

It’s also so, so nice for open source projects, I’m surprised GitHub doesn’t suggest it.

In this post and future ones, we’ll go over some of the tricks we’ve learned building Makefiles in several dozen separate projects.

Today’s lesson: Wildcards (%)

Dynamic Arguments with Wildcards (%)

Many popular languages include some (Make-inspired) script runner, so most commands look something like this:

migrate:
    bundle exec rake migrate
install:
    yarn install
runserver:
    python ./manage.py runserver

That’s nice, but what about when you want to supply arguments to one of those CLI commands.For example, how can you run bundle exec rake migrate[50] to migrate to version 50, rather than latest?

Wildcards, that’s how! For migrations, we can add a new command to migrate to a specific version:

migrate-to-%:
    bundle exec rake migrate[$(*)]

Now if you want to migrate to a specific version, you can run:

$ make migrate-to-50
Migrating to version 50...

Well, we think that’s pretty cool, but what else can wildcards do?

Declarative Argument Dependencies

The other way to configure Makefiles is with environment variables.Let’s say we use a consistent STACK environment variable for working with deployed Cloudformation stacks (as we actually do).

It can be pretty annoying to debug the errors if you forget to set STACK:

logs:
    awslogs get -w /ecs/$(STACK)_MyService

That can be invoked (correctly) as awslogs get -w /ecs/development_MyService or (incorrectly) as awslogs get -w /ecs/_MyService. If it’s missing, your error will be about some unknown service or wrong file path, rather than about a missing parameter.

Well, we have a special command we’ve been using for years that we’ll share with you now:

guard-%:
    if [-z '${${*}}']; then echo 'ERROR: variable $* not set' && exit 1; fi

It’s maybe our favorite two lines of code in the world.

Now, you can rewrite the logs target like this, which is also more declarative and clearer about its dependencies:

guard-%:
    if [-z '${${*}}']; then echo 'ERROR: variable $* not set' && exit 1; fi
logs: guard-STACK
    awslogs get -w /ecs/$(STACK)_MyService

Finally, if you invoke logs without STACK set, you get a nice error:

$ make logs
ERROR: variable STACK not set
make: *** [guard-STACK] Error 1

Wait, there’s more!

There are a couple other wildcard-based tricks we still have in store, as well as some more Makefile goodness.

This was originally posted on my consultancy’s blog, at https://lithic.tech/blog/2020-05/makefile-wildcards. If you have any questions, please leave a comment or get in touch!

Dynamic JavaScript and React Configuration

Rob Galanakis — Mon, 18 May 2020 02:23:07 +0000

Most of the frontends we build at Lithic are written in React.Normally we deploy them as static apps in Netlify.This will write environment variables from process.env directly into the compiled JavaScript.

Sometimes, though, we need to dynamically configure the applications based on runtime,not build time, environment variables. The basic create-react-app workflow doesn’t work in this situation.

There are many approaches to runtime configuration of static JavaScript applications.Most of them involve writing out a config.js file at server boot time,and having the client load this before or alongside the main JS bundle.In some solutions, the JS is dynamically templated.

All of these approaches have downsides:

Loading config.js before your main bundle introduces latency before you can load your app.
Loading config.js alongside your main bundle means your app cannot synchronously access config,which introduces complexity.
Templating your main bundle on each request is nontrivially complex(we love nginx too but the fewer script callouts the better).

There is a solution, though: template config directly into index.html at server/container boot time.

Our Solution

We built runtime-js-envto handle dynamic JS config with no downsides.It’s a simple Go program that rewrites index.html to include a window._jsenv object with your configpulled from REACT_APP_, NODE_, and HEROKU_ environment variables.

You call it at container/server boot time,and the only change you need to make in your JS is change calls to process.envinto calls to window._jsenv || process.env. See the GitHub repo for some examplesof how we handle this.

Your index.html file will be modified to include a <script> tag inside your <head>.It’s safe to call multiple times, and uses Go’s HTML5 parser so should be valid for whatever you throw at it.

Usage

runtime-js-env is a Go program, so you’ll need Go installed.

$ go get github.com/lithictech/runtime-js-env
$ runtime-js-env --help
NAME:
   runtime-js-env - A new cli application

USAGE:
   runtime-js-env [global options] command [command options] [arguments...]

COMMANDS:
   help, h Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --index value, -i value Path to the index.html file. Default to index.html in pwd. (default: "index.html")
   --window-var-name value, -w value Attribute name for the config object. (default: "_jsenv")
   --env-prefixes value, -p value Environment variable prefixes to copy into the config object. (default: "REACT_APP_", "NODE_", "HEROKU_")
   --indent value, -t value Indentation for each line in the config script tag. (default: " ")
   --help, -h show help (default: false)
$ runtime-js-env -i public/index.html

Feedback

If you have any suggestions or feedback,please open an issueor get in touch directly.

This was originally posted on my consultancy’s blog, at https://lithic.tech/blog/2020-05/react-dynamic-config. If you have any questions, please leave a comment or get in touch!

Should I use a graph database?

Rob Galanakis — Wed, 13 May 2020 06:52:42 +0000

One of our clients asked us the other day:

I'm setting up a web app that stores information about devices and the interconnected dependencies between devices. The app would have a web front for a user to navigate through the graph of devices and be able to click through each node, find related/linked nodes, and dive deeper into a given node.We're trying to decide what database to use, and thinking about a graph database. Do you have any advice?

Graph Databases (like Neo4j, AWS Neptune, and many others) are pretty amazing,but it's usually only a good idea to use one if you fit one of two criteria:

You really want to use a graph database. If you want to learn something new, go for it!But our job as consultant normally isn't to use unfamiliar technology.
You have a "true graph problem."

It can sometimes be difficult to tell if you have a "true graph problem,"or just have a relational problem that looks like a graph problem.

"What devices are linked to other devices" just looks like a graph problem.It's pretty easily solved with just two tables:

devices(id, name, <other columns>) stores the device information (and would probably have foreign keys to/from other application tables)
devices_network(upstream_id, downstream_id) is a join table specifying the relationships between devices (you may want some to add some constraints).

You can then create a graph with some joins between those tables or multiple queries.This is going to be fast and efficient to get the network for any particular node,and maps really well to associations in Object Relational Mappers,wither eager loading/preloading.

If you're not using an ORM, it's still pretty simple to build an in-memory network structureof some set of the graph to answer questions about deeper relationships.

What is a "true graph problem"?

Graph problems are less about connections and nodes, and more about the edges between those nodes."What is connected to this node?" is an easy question to answer with a relational database, as we saw above.Some questions that are not easy to answer would be "what paths are available to travel between two nodes?"and "what are the costs/constraints between two nodes?"

You can think of air travel. If we're traveling between airports,we're not so concerned with "what airports are connected to JFK," which is a relational problem.We're rather asking "what is the fastest/cheapest/shortest path between JFK and SFO or nearby airports,"which can only be solved in terms of a graph,because information about the edges/connections are fundamental to the solution.

Put more abstractly: if the edges between your nodes have important properties,you probably have a graph problem. If the edges between nodes are just connections,you have a relational problem.

Advice

It's always much easier to run and develop against a relational database than a graph or any specialized database.Relational databases are a marvel of technology (especially PostgreSQL 😍),and you shouldn't throw that out lightly.Specialized tech means you’re going to have to do a lot of stuff from scratch, or with vastly more limited libraries.Far fewer tutorials and acquired knowledge. You’re just on your own for a whole lot more!

That said, there are times graph and other specialized databases are the way to go.It's just a good idea to make sure you know why you're choosing it.

This was originally posted on my consultancy's blog, at https://lithic.tech/blog/2020-05/should-I-use-a-graph-database. If you have any questions, please leave a comment or get in touch!