Infra as code, pipelines and hidden patterns

This article is archived and not maintained. It mostly contains notes and unorganized thought about the subject. Read it at your own risks.

I rarely find articles about implementation patterns and technicalities of infrastructure as code, and build and deployment pipelines other than basic tool usage. In fact, most articles I can find on the web about infra as code and devops are about the benefits of adopting such practices and rarely aimed at users of these practices.

This article aims at creating a record of patterns and/or practices that I often encourage when writing infrastructure as code and deployment pipelines.

If you read this and:

• know of other similar resources,
• have and encourage other similar practices,
• think something is missing, please contribute by adding a comment, I am interested to hear about these things.

Using stack outputs in other stacks

It is possible to reference the output of other deployment stacks and use it as input in another stack.

Example: Let's say we have a project split in 2: a front-end and a backend. The backend will create some resources that the front-end also needs to reference. What often do is creating secrets in the front-end build/deployment pipeline and referencing the value of these secrets in our code.
This is a manual step that can be removed. To do so:

• Define these values as output of the backend stack
• Reference the backend stack from the front-end stack
• Use the outputs of the backend stack in the front-end stack

How to do this with:

• Pulumi
• Terraform

Isolating the base infrastructure from the application specific infrastructure

Key principle: speed

There are multiple ways to organize infrastructure as code. A lot of projects keep the infra definition as close to the code as possible. It is a practice I endorse: related things should be as close as possible from each other as possible. However, this will result in a slow deployment time.

This slow deployment time is caused by the deployment pipeline always validating the whole infrastructure. Problem is: once it's been built, the infrastructure rarely changes.
To optimize deployment speed, isolate the base blocks of the infra and only build/deploy the pieces that need to be rebuilt/redeployed in the application part of your code.

Ex:

• In a front-end repository, only push file to an already existing S3 bucket.
• Keep Cognito, databases creation, API Gateway, etc. in an infrastructure stack. Only deploy your lambdas/containers from the backend repository

Automation of projects and privileges

You can keep track of your repositories/repository accesses with infrastructure as code. This is an easy way to centralize the resources your company uses, control your company code policies (example: at least one approval before you can merge to master) and control who can access what.

Artifact based deployments

Key principles: speed, reproducibility

Keep your build and deployment pipelines separate. Do not rebuild your docker image, React project on every deployment. Your build pipeline should build a docker image and store it in an image registry, or compile and bundle your JS files. Your deployment pipeline should use the resulting artifacts and deploy them with runtime variables to the runtime environments.

Interesting Resources

• Common Patterns of Infrastructure as Code Architecture