DEV Community: Flavius Dinu

Are you still searching for a Git and GitHub tutorial in 2025?

Flavius Dinu — Sun, 03 Aug 2025 07:43:45 +0000

Search no more.

TL/DR?

Also, if you like the video, don’t forget to like, subscribe, and ring the bell for more videos like this. It helps more than you think!

What Are Git and GitHub?

Before touching a single command, clear up the most common beginner confusion: Git and GitHub are not the same thing.

Git is a version control system installed on your computer. It tracks every change you make to your project, maintains a full history, and lets you experiment with confidence. Accidentally deleted something? Git lets you revert. Introduced a bug? Git helps you identify and undo the exact change. It’s your local safety net and experiment playground.
GitHub is a cloud-based hosting service for Git repositories. If Git is the tool that manages history locally, GitHub is where you push that history to collaborate, back up, review code, manage issues, and coordinate work across a team.

Simple tip: Git lives on your machine; GitHub lives on the web. Throughout this tutorial, you’ll see how they work together in the professional workflow developers use every day.

Setup: Your First Steps

Enough theory. Let’s get everything installed and ready.

Create a GitHub Account

Your GitHub profile is your identity in the developer world. Go to GitHub’s signup page, enter your email, choose a username (pick something professional if you intend to show it to employers), set a password, verify your email, and skip optional “personalization” steps for now. You’ll land on your dashboard, your new home base.

Install Git

Git isn’t usually preinstalled, so you need to add it to your system:

Windows: Download the installer from the official Git site. Run it and accept the defaults. If prompted to choose a default editor, pick something like VS Code if you have it. Git Bash will be installed — it’s a great shell for running Git.
Mac: Open Terminal and type git. If it’s not installed, macOS will prompt to install Xcode Command Line Tools. Say yes. Alternatively, if you use Homebrew:
brew install git
Linux: Use your distribution’s package manager, e.g.,
sudo apt install git

After installation, verify it with:

git --version

You should see a version number, congrats, Git is installed.

Configure Git

Tell Git who you are so commits are properly attributed:

git config --global user.name "Your Name"
git config --global user.email "your.email@example.com"

The --global flag makes this the default for all repositories on your machine.

Your First Repository

Now that tools are ready, let’s create a project space.

On GitHub

Log into GitHub.
Click the + in the upper-right and choose New repository.
Fill in:

Repository name: e.g., hello-world
Description: A short sentence explaining the project.
Visibility: Choose Public for now.
Initialize with README: Check this, every project should have a README.

Skip .gitignore and license for today; you can add them later. Click Create repository.

You now have your first repository, a hosted project with an initial README.

Core Workflow: Clone → Add → Commit → Push

This is the daily loop of Git.

Clone the Repo Locally

On your repo page, click the green Code button, copy the HTTPS URL, then in terminal:

git clone https://github.com/your-username/hello-world.git
cd hello-world

Now you have a full local copy, including history.

Make a Change

Edit README.md in your editor, and add something like:

This is my first local change.

Save the file.

Stage the Change

Staging tells Git what you intend to include in the next snapshot:

git add README.md

Or stage everything:

git add .

Commit the Change

A commit records the staged changes with a message:

git commit -m "Update README file with a new line"

Push to GitHub

Send your local commit to the remote repository:

git push origin main

origin is the default remote.
main is the primary branch.

After pushing, refresh your GitHub repo page and the change should appear. You’ve completed the full cycle: clone → edit → stage → commit → push. This is the foundation.

Understanding Branches: Parallel Universes

Branching is what makes Git transformative.

A branch is a separate line of development, a “parallel universe” of your project if you will. The main branch holds stable, production-ready code. To build a new feature safely, you branch off and experiment in isolation.

Let’s take a look on how to create a new branch:

git checkout -b new-feature

You’ve created and switched to new-feature. Create a file:

touch new-feature-file.txt
git add .
git commit -m "Add file for the new feature"

Switch back:

git checkout main

The new file disappears, it lives only on new-feature. Switch back again:

git checkout new-feature

The file returns. That isolation is what makes branching safe and powerful.

Pull Requests

Once your feature is ready, you want to merge it into main but in team environments, you do this through a Pull Request (PR).

Steps

Push your feature branch:

git push origin new-feature

On GitHub, you’ll see a prompt to Compare & pull request. Click it.

On the PR page:

Confirm you’re merging new-feature into main.
Give it a clear title (e.g., “Add New Feature”).
Write a descriptive body: what the change does, why it was made, and how to test it.

Next, create the PR and invite teammates to review. They can comment on specific lines, suggest changes, and approve.

Once approved, click Merge pull request. Optionally delete the old feature branch to keep things clean.

You’ve just followed a professional collaboration workflow: develop in isolation, request review, then merge.

Team Collaboration in Action

On a team, the repository is the single source of truth. Everyone clones it, works on their own branches, and integrates via PRs. Key practices:

Collaborators

Repository owners invite others under Settings → Collaborators , granting push access.

Branching Strategy (Feature Branch Workflow)

Never commit directly to main.
Create a new branch per task, e.g., feature/user-login or bugfix/fix-typo.
Work on that branch; commit as needed.
Open a PR to bring changes into main.
Require at least one reviewer’s approval.
Merge and delete the feature branch.

Stay Up to Date

Before starting new work:

git checkout main
git pull origin main

git pull fetches and integrates others’ changes so your base is current.

Going Deeper: Advanced Concepts

Now that you’ve mastered the essentials, here are a few things that separate beginners from effective collaborators.

Structured Branching (e.g., Git Flow)

Large teams sometimes use models like Git Flow , which introduces branches like:

develop for integration of completed features,
release branches for stabilization,
main for production releases.

You don’t need this immediately, but it’s useful to know such conventions exist.

Merge Conflicts

Conflicts arise when two branches edit the same lines of the same file. Git will pause the merge and mark the conflict in the file with:

<<<<<<< HEAD
Your changes
=======
Their changes
>>>>>>> branch-name

Resolve by editing the file to the desired state, then:

git add <file>
git commit

Modern editors often have visual conflict resolution tools built-in to simplify this.

GitHub for Project Management

GitHub isn’t just for code, it’s a project hub.

Issues: Track tasks, bugs, ideas. Assign them, label them, and discuss context.
Project Boards: Visualize progress (like a Kanban). Drag PRs and Issues across columns like “To Do,” “In Progress,” and “Done.”

Mastering these lets you manage the code and the project around it.

Conclusion

You started by asking: What even is Git?

Now you’ve:

Installed and configured Git.
Created a GitHub repo.
Done the core workflow: clone, add, commit, push.
Used branching to isolate work safely.
Opened and merged a pull request.
Seen how teams coordinate through collaborators, reviews, and synchronization.
Touched on advanced ideas like structured branching, merge conflicts, and using GitHub as a project platform.

You’ve gone from confused beginner to someone with a real, professional workflow. The next step is practice: build your own projects, contribute to open-source, and keep using this loop until it becomes second nature.

If this guide helped, save it, share it, or subscribe to updates for more practical deep dives. You’re no longer just learning — you’re building. Now go make something awesome.

Self-Host n8n with Docker: Should You Do It?

Flavius Dinu — Tue, 08 Jul 2025 06:52:08 +0000

Let’s be honest: Everyone is using n8n for automating their workflows. And I’m not going to tease you anymore, I’ll say that absolutely YES, you should self-host n8n on Docker.

However, if you’re unfamiliar with n8n, I've got you covered.

Think of n8n as the Swiss Army knife of workflow automation. Instead of wrestling with complex CI/CD tools for every little automation task, you get a visual, node-based interface that actually makes sense. Your team can build workflows without needing a PhD in DevOps. No more fighting with overcomplicated tools or reinventing the wheel every time you need to automate something.

In this guide, I’ll walk you through everything you need to know about getting n8n up and running with Docker. We’ll start simple with a local installation with Docker, and work our way up to a setup that deploys n8n on an EC2 instance in AWS using Terraform (you can use OpenTofu as well).

TL/DR? Check the video instead:

Prerequisites

For this tutorial you will need:

Docker & Docker Compose
Basic knowledge of the CLI and Git
An AWS account with permissions to create EC2 instances, security groups, and Route53 records
Terraform or OpenTofu

Local n8n installation with Docker-compose

Let’s start simple with a local installation with Docker-compose. For that you can use the Self-Hosted AI starter kit for that, build by the n8n team.

You will first need to clone the repository by using:

git clone git@github.com:n8n-io/self-hosted-ai-starter-kit.git

Next, go to the directory containing the self-hosted-ai-starter-kit repository, and cp the .env.example to a new file called .env. You will then need to modify this .env file with values that make sense for your environment.

Depending on your operating system and your GPU, you can deploy n8n in multiple ways:

Nvidia GPU

docker compose --profile gpu-nvidia up

AMD GPU

docker compose --profile gpu-amd up

Everyone else:

docker compose --profile cpu up

As I’m on a Mac with Apple Silicon I will use the latter command and in the end this is what the output looks like:

Attaching to n8n, n8n-import, ollama, ollama-pull-llama, qdrant, postgres-1
postgres-1 | 
postgres-1 | PostgreSQL Database directory appears to contain a database; Skipping initialization
postgres-1 | 
qdrant | _ _    
qdrant | _____ | |_ ____ _ _ __ | |_  
qdrant | / _` |/ _` | ' __/ _` | '_ \|__ | 
qdrant | | (_| | (_| | | | (_| | | | | |_  
qdrant | \ __, |\__ ,_|_| \ __,_|_| |_|\__ | 
qdrant | |_|                           
qdrant | 
qdrant | Version: 1.14.1, build: 530430fa
qdrant | Access web UI at http://localhost:6333/dashboard
qdrant | 
qdrant | 2025-07-07T18:56:16.373057Z INFO storage::content_manager::consensus::persistent: Loading raft state from ./storage/raft_state.json    
ollama | time=2025-07-07T18:56:16.383Z level=INFO source=routes.go:1235 msg="server config" env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:4096 OLLAMA_DEBUG:INFO OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://0.0.0.0:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/root/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]"
ollama | time=2025-07-07T18:56:16.386Z level=INFO source=images.go:476 msg="total blobs: 6"
ollama | time=2025-07-07T18:56:16.386Z level=INFO source=images.go:483 msg="total unused blobs removed: 0"
ollama | time=2025-07-07T18:56:16.388Z level=INFO source=routes.go:1288 msg="Listening on [::]:11434 (version 0.9.5)"
ollama | time=2025-07-07T18:56:16.388Z level=INFO source=gpu.go:217 msg="looking for compatible GPUs"
ollama | time=2025-07-07T18:56:16.394Z level=INFO source=gpu.go:377 msg="no compatible GPUs were discovered"
ollama | time=2025-07-07T18:56:16.395Z level=INFO source=types.go:130 msg="inference compute" id=0 library=cpu variant="" compute="" driver=0.0 name="" total="7.7 GiB" available="4.5 GiB"
postgres-1 | 2025-07-07 18:56:16.415 UTC [1] LOG: starting PostgreSQL 16.9 on aarch64-unknown-linux-musl, compiled by gcc (Alpine 14.2.0) 14.2.0, 64-bit
postgres-1 | 2025-07-07 18:56:16.415 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
postgres-1 | 2025-07-07 18:56:16.415 UTC [1] LOG: listening on IPv6 address "::", port 5432
postgres-1 | 2025-07-07 18:56:16.416 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres-1 | 2025-07-07 18:56:16.421 UTC [29] LOG: database system was shut down at 2025-07-06 11:59:00 UTC
postgres-1 | 2025-07-07 18:56:16.425 UTC [1] LOG: database system is ready to accept connections
qdrant | 2025-07-07T18:56:16.431182Z INFO qdrant: Distributed mode disabled    
qdrant | 2025-07-07T18:56:16.431604Z INFO qdrant: Telemetry reporting enabled, id: d33bc0ef-1661-4d58-8b03-f1a751fc0c2f    
qdrant | 2025-07-07T18:56:16.432002Z INFO qdrant: Inference service is not configured.    
qdrant | 2025-07-07T18:56:16.577110Z INFO qdrant::actix: TLS disabled for REST API    
qdrant | 2025-07-07T18:56:16.577167Z INFO qdrant::actix: Qdrant HTTP listening on 6333    
qdrant | 2025-07-07T18:56:16.577376Z INFO actix_server::builder: starting 5 workers
qdrant | 2025-07-07T18:56:16.577395Z INFO actix_server::server: Actix runtime found; starting in Actix runtime
qdrant | 2025-07-07T18:56:16.577407Z INFO actix_server::server: starting service: "actix-web-service-0.0.0.0:6333", workers: 5, listening on: 0.0.0.0:6333
qdrant | 2025-07-07T18:56:16.607417Z INFO qdrant::tonic: Qdrant gRPC listening on 6334    
qdrant | 2025-07-07T18:56:16.607430Z INFO qdrant::tonic: TLS disabled for gRPC API    
ollama | [GIN] 2025/07/07 - 18:56:19 | 200 | 802.291µs | 172.24.0.5 | HEAD "/"
ollama | [GIN] 2025/07/07 - 18:56:20 | 200 | 690.502584ms | 172.24.0.5 | POST "/api/pull"
pulling manifest 
ollama-pull-llama | pulling dde5aa3fc5ff: 100% ▕██████████████████▏ 2.0 GB                         
ollama-pull-llama | pulling 966de95ca8a6: 100% ▕██████████████████▏ 1.4 KB                         
ollama-pull-llama | pulling fcc5a6bec9da: 100% ▕██████████████████▏ 7.7 KB                         
ollama-pull-llama | pulling a70ff7e570d9: 100% ▕██████████████████▏ 6.0 KB                         
ollama-pull-llama | pulling 56bb8bd477a5: 100% ▕██████████████████▏ 96 B                         
ollama-pull-llama | pulling 34bb5ab01051: 100% ▕██████████████████▏ 561 B                         
ollama-pull-llama | verifying sha256 digest 
ollama-pull-llama | writing manifest 
ollama-pull-llama | success

Now, you can access n8n directly on http://localhost:5678.

Next, you can run the demo workflow that just chats with Ollama:

N8n makes it easy to import workflows directly by using json files, so what you can do is head out to this link, and choose one of the workflows.

I’ve chosen the first one in the list and then I clicked on Use for Free :

Then you get a couple of different options:

Select Copy template to clipboard [JSON] and save it to a file on your computer.

Next, go to your n8n instance and click on create workflow:

Now you can select the three dots in the top right corner and select import from file:

Select the file you’ve created before and the workflow gets imported automatically:

Using Terraform to deploy n8n on AWS EC2 with Docker

I’ve put together a simple Terraform configuration that lets you deploy n8n on AWS without much hassle, making it easy to have the server up and running non stop to run all the workflows you need.

You can find the code here, make sure you clone or fork it.

The configuration is pretty straight forward to use, you will just need to provide a couple of variables using terraform.tfvars, environment variables or a secrets management solution to ensure everything goes smooth:

public_key_path                
create_dns_record              
domain_name                    
key_name                       
basic_auth_password            
n8n_encryption_key             
n8n_user_management_jwt_secret

By setting the create_dns_record to true you will be able to access n8n directly from your domain (right now only through http).

Also, the public_key_path should be the path to your ssh public key so make sure you have access to the private key as well in order to connect to the instance via ssh (you don’t really need to connect via ssh to it, but I’ll show how to track something if you do).

I’m using by default a t2.micro instance, so in this case, the AI capabilities won’t work due to RAM limitations, but you can still import other workflows to n8n as I’ve shown above.

NOTE : The solution can be extended based on the demand to get closer to be production ready. You can use an infrastructure orchestration platform or a generic CI/CD to help you with the deployment. You should also implement remote state if you want to use it in production.

As soon as you populate the variables, you can run:

terraform init

Initializing the backend...
Initializing provider plugins...
- Finding latest version of hashicorp/aws...
- Installing hashicorp/aws v6.2.0...
- Installed hashicorp/aws v6.2.0 (signed by HashiCorp)

...

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

After you initialize the configuration you can run:

terraform plan

Plan: 4 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + n8n_dns_record = (known after apply)
  + n8n_instance_public_ip = (known after apply)

Four resources will be created: an EC2 instance, a security group, a key pair, and optionally a dns record. The n8n installation is handled in the cloud-init script that can be found here.

As soon as you’re happy with the plan you can run:

terraform apply

After a couple of minutes, your resources will be up and running, but you will still have to be patient for a couple more minutes for the cloud init script to fully deploy.

If you are impatient as I certainly am, you can check in real time what is happening with the cloud init script by ssh-ing to the instance and doing a:

tail -f /var/log/cloud-init-output.log

As soon as the script finishes you will be able to access n8n via:

ec2_public_ip:5678
your_record_name:5678 (if you set up a dns record)

You can run a terraform output command to see what are the defined outputs.

Key points

If you’re serious about automating your workflows without any limitations, self-hosting n8n with Docker is the way to go.

The process is smooth, scalable, and entirely under your control.

With this setup, you’re not only increasing flexibility, but you’re also building automation on your terms, which let’s be honest, it’s fun.

So yes, you should self-host n8n on Docker, and now, you know exactly how.

Lens Kubernetes IDE: How to Simplify K8s Management in 2025

Flavius Dinu — Mon, 23 Jun 2025 09:36:12 +0000

Kubernetes (K8s) has improved how we deploy and manage containers, but with this power comes a lot of complexity that can easily overwhelm even experienced engineers. If you are managing five production clusters across different environments, juggling kubectlcommands across multiple terminal windows, how fast could you actually debug an issue?

Visual cluster management has become essential for DevOps workflows, even though kubectl remains the backbone of Kubernetes management.

You need to remove the friction between memorizing debugging commands across multiple clusters, switching between them, and always remembering what are the resource relationships. This is where Lens comes in, your K8s IDE that shortens the gap between CLI and visual clarity.

In this post, we will walk through what Lens is, how to install it, how to connect it to a cluster (it’s way easier than you would expect), and we will also deploy a bunch of K8s resources inside it to view and troubleshoot them with Lens.

Check out my YouTube channel:

DevOps with Flavius

What is Lens?

Lens is your unified Kubernetes IDE that offers visual cluster management capabilities that complement kubectl. You should think of it as your mission control center for Kubernetes that gives you a single interface where you can monitor, debug, and manage multiple clusters without drowning in terminal commands.

One of the core values that Lens offers is multi-cluster visibility. You can stop switching between terminal contexts and memorizing cluster configurations. With Lens, you get real-time dashboards, integrated log streaming, and visual resource management across all your environments.

On top of that, you can view logs, connect easily to different pods, view configmaps and secrets information easily, port-forward to your services to see if they are working properly, real-time resource monitoring, and much more.

How to Install Lens

Installing Lens is an easy process. Just head out to https://k8slens.dev/download, choose your operating system, and then follow the process described for each OS:

After installing it, launch Lens and complete the initial setup, and create your own LensID. Lens will scan for existing kubeconfig files in your ~/.kube directory and will give you the option to import discovered clusters. This automatic discovery streamlines the onboarding process if you already have kubectlconfigured.

K0s clusters

I will assume that you don’t have any K8s clusters created, so let’s jump into creating one. If you already have a K8s cluster up and running, you don’t need to follow this, but at the same time, I will give you another option of creating lightweight clusters apart from minikube and kind by leveraging K0s.

So K0s is a lightweight Kubernetes distribution that delivers a production-like cluster from a single binary, making it ideal for learning, testing, and development workflows. Right now, it runs on Linux, with experimental Windows support.

Here are the steps you have to follow if you want to install it on your Linux machine:

$ curl -sSf https://get.k0s.sh | sudo sh
$ sudo k0s install controller --single
$ sudo k0s start # wait a minute
$ sudo k0s kubectl get nodes

Connect Lens to your cluster

As I’m on MacOS, I can’t use K0s right now, so I will stick to kind.

To create a kind cluster, you can simply run:

$ kind create cluster --name lens-cluster

Creating cluster "lens-cluster" ...
 ✓ Ensuring node image (kindest/node:v1.30.0) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
Set kubectl context to "kind-lens-cluster"

You can now use your cluster with:

kubectl cluster-info --context kind-lens-cluster

Not sure what to do next? 😅 Check out https://kind.sigs.k8s.io/docs/user/quick-start/

$ kubectl cluster-info --context kind-lens-cluster
Kubernetes control plane is running at https://127.0.0.1:63490
CoreDNS is running at https://127.0.0.1:63490/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

As soon as I’ve created my cluster, if I open Lens, I can see that it already connected to it:

Populate your cluster with different K8s Resources

To really explore Lens, you need to have K8s resources inside your cluster. So I’ve put together a couple of resources that you can deploy to observe what is happening:

First, I will start with a basic nginx deployment that shows pod lifecycle management:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.21
        ports:
        - containerPort: 80
        resources:
          requests:
            memory: "64Mi"
            cpu: "250m"
          limits:
            memory: "128Mi"
            cpu: "500m"

Apply this deployment using kubectl. I will use Lens’ built-in terminal:

kubectl apply -f nginx_deployment.yaml

Next, let’s create a service to expose the nginx deployment:

apiVersion: v1
kind: Service
metadata:
  name: nginx-service
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: ClusterIP

Let’s apply it:

kubectl apply -f nginx_service.yaml

Now, let’s create a config map that will change the nginx.conf file:

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-config
data:
  nginx.conf: |
    server {
        listen 80;
        location / {
            return 200 'Hello from Lens!\n';
            add_header Content-Type text/plain;
        } 

kubectl apply -f nginx_conf.yaml

Now that we’ve created a couple of resources, we are ready to explore Lens.

First Glance at Lens

Open your Lens application, and select the K8s cluster in which you’ve deployed the above resources.

You will see many options from which you can choose, and all of them are really important to understand:

Overview shows your cluster health assessment. This is where you get visibility into node status, resource utilization, and workload distribution

Nodes show you data about your cluster nodes

Workloads will let you explore your deployed resources

Config will show you data about your configmaps, secrets, resource quotas, limit ranges and more

In the Network you will see information about your services, ingresses, and others

And as you can see, there are other options present, so this would be a great time to stay a couple of minutes in the app, and explore all the things that you can do.

As soon as there are changes happening in your cluster, Lens picks them and propagates them immediately through the interface. Pod restarts, scaling operations, and configuration changes appear without manual refresh, providing live insight into cluster operations that static kubectl output cannot simply match.

Let’s now explore the resources we have deployed. Here are all the pods running:

By clicking on the 3 dots on the right side, you get a couple of options:

You can easily attach to a pod, open a shell, evict it, view the logs, edit it, and even delete it.

I will now press on Shell, to connect to one of my nginx pods:

Here is the ConfigMap:

And this is the service:

Check Out Logs with Lens

Log management through Lens makes K8s log management a breeze. Instead of juggling multiple terminal windows with kubectl logs commands, you get unified log streaming with advanced filtering and search capabilities.

As you saw, when we connected to the pod’s shell, we had an option to access logs. The interface automatically handles multi-container pods by providing separate log streams with clear container identification.

The search functionality helps a lot with debugging specific issues. You can use the search box to filter logs for error keywords, specific HTTP status codes, and others. Lens highlights matching terms and provides context around each match, making it easier to understand the sequence of events leading to problems.

You also have the ability to export logs. Select a time range or search filter, then export the results to a file for offline analysis or integration with external logging systems. This feature bridges the gap between interactive debugging and formal incident documentation.

Port-Forward to Nginx

Apart from everything that I’ve shown you until now, you also get an easy way to enable port forwarding through Lens.

Just go to your Network tab, select Services, and then choose your service:

You will see an option to Forward it, so let’s click on it:

You can choose a local port to forward it to, or leave it as Random, have the option to directly open in your browser, and also choose https. I will leave everything as default and click on Start.

As soon as I clicked on start, I will get redirected to my browser, and I can see my web application.

As you can see, what I’ve added in my configmap is not currently rendering in my web page, and that’s to be expected, as I didn’t specify anywhere in my deployment that I want to use the configmap.

So let’s update the deployment to use the configmap:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.21
        ports:
        - containerPort: 80
        resources:
          requests:
            memory: "64Mi"
            cpu: "250m"
          limits:
            memory: "128Mi"
            cpu: "500m"
        volumeMounts:
        - name: nginx-config-volume
          mountPath: /etc/nginx/nginx.conf
          subPath: nginx.conf
      volumes:
      - name: nginx-config-volume
        configMap:
          name: nginx-config

As soon as I did and I’ve reapplied the code, I see in Lens the yellow triangle with the exclamation point for my pod. This means that there is an issue there. Let’s check the logs and understand what is happening.

It seems like the “server” directive is not allowed in the /etc/nginx/nginx.conf, so this means that there is an issue with my configmap. Let’s fix it and come back:

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-config
data:
  nginx.conf: |
    events {}
    http {
        server {
            listen 80;
            location / {
                return 200 'Hello from Lens!\n';
                add_header Content-Type text/plain;
            }
        }
    }

After fixing it, we now correctly see the intended message from the configmap:

Security considerations remain important even with port forwarding. The tunnel only provides local access and doesn’t expose services to other machines on your network. Anyway, please be mindful of sensitive services like databases that contain production data, and terminate port forwards when they’re no longer needed.

You can always see a list of all the port forwards in the Network -> Port Forwarding:

If connection issues arise, Lens provides diagnostic information about tunnel status, including error messages and retry attempts. Common issues include local port conflicts and service endpoint problems.

Deploy ArgoCD and connect to It

ArgoCD implements GitOps principles that align perfectly with modern DevOps workflows. Deploying ArgoCD inside your K8s cluster and managing it with Lens demonstrates how visual cluster management integrates with continuous deployment tools.

If you want to learn more about ArgoCD check out this post.

Let’s install ArgoCD, but first let’s create a namespace:

kubectl create namespace argocd

kubectl apply -n argocd -f [https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml](https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml)

You can easily monitor the deployment progress through Lens by navigating to the argocd namespace in the Workloads section. You’ll see multiple ArgoCD components initializing, such as the API server, repository server, application controller, and Redis cache. The visual status indicators show pod startup progress and resource allocation for each component. This save us you a lot of time and frustration, as you don’t need to spam kubectl get commands.

ArgoCD requires initial password configuration for the admin user. You can easily get that from Lens by going to Config->Secrets and selecting the argocd-initial-admin-secret.

Secrets are Base64 encoded, but Lens gives you the ability to show them, by clicking the Show button. Save the password, and now let’s verify if all components were deployed successfully.

Seems like everything is running properly, so we are now ready to access the ArgoCD UI. We will use port forwarding once more. Navigate to your services, and look for the ArgoCD server service, and start the port forward:

This is what you will initially see in your Browser, so ensure you use the _visit this website _option.

The ArgoCD login page appears, requesting username and password credentials. Use admin as the username and the password you extracted from the secret:

The ArgoCD dashboard provides a clean interface for application management, showing deployed applications, sync status, and health indicators.

In Lens, you will see that a new Custom Resource Definition has appeared for the argoproj.io, making it easy to track all the Argo related resources in a single view:

I will now deploy a dummy ArgoCD application just to show you how you will see it in Lens:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: guestbook
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/argoproj/argocd-example-apps.git
    targetRevision: HEAD
    path: guestbook
  destination:
    server: https://kubernetes.default.svc
    namespace: default
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Save the above to a file and use kubectl apply to deploy it. This is how you see the App in Lens after deploying:

And in ArgoCD:

We won’t go through ArgoCD’s capabilities for this tutorial, I just wanted to show you how easy Lens makes the overall process of deployment, and also for debugging. If you have any ArgoCD apps that fail, you can use Lens to examine logs, resource status, and configuration details.

Deploy a Helm Chart

Let’s add a popular Helm repository to our local configuration:

helm repo add bitnami https://charts.bitnami.com/bitnami

As soon as I’ve done this, in Lens, I can go to Helm and select Charts, and I will see all the available from the repository.

I can easily search for a chart at the top of the Charts page:

Let’s deploy a WordPress application:

helm install my-wordpress bitnami/wordpress \
  --namespace wordpress \
  --create-namespace \
  --set wordpressUsername=admin \
  --set wordpressPassword=secretpassword \
  --set service.type=ClusterIP \
  --set persistence.enabled=false

Again, we can monitor the deployment process by navigating to the Wordpress namespace in the Workloads->Pods:

The Helm release appears in Lens under the Releases section, showing release status, chart version, and installation notes:

We can access the WordPress application by doing port forwarding, and similar to ArgoCD’s deployment, we should get the login credentials from the my-wordpresssecret (or well, this time we’ve actually hardcoded the password when we installed the helm chart; don’t do this).

Key Points

Lens is my go-to product when it comes to choosing a K8s IDE. You get all the debugging capabilities into one place, a built-in terminal, and the observability you need to understand what is happening in your K8s cluster.

Using Lens saves you a lot of time, because you don’t need to remember complicated kubectl commands for everything your process would require. It’s important to understand that Lens is not a 100% replacement for kubectl, and this is not what it promises. It’s a complement to kubectl that aims to make your life easier, and equip you with all the tools you need to make your cloud-native life easier.

Want to learn Terraform? Check out my Terraform series:

Terraform from 0 to Hero — 0. I like to start counting from 0, maybe I enjoy lists too much

IaCConf 2025: Why Infrastructure as Code deserves its own conference

Flavius Dinu — Tue, 06 May 2025 09:50:34 +0000

Infrastructure as Code (IaC) has come a long way since its beginnings. What started as spinning up cloud resources has now grown into a full-blown ecosystem with a very engaged community. IaC on its own can add a lot of complexity to your processes and cause more issues, but combined with governance, compliance, security, deployment strategies, self-service, it easily becomes the fuel your delivery engine needs.

This year, IaCConf debuts as a virtual event by bringing together this community, and it’s shaping up to be one of the most relevant and practical events in the space.

What makes IACConf stand out

Unlike broader DevOps/SRE or cloud-native events, IACConf is focused on the challenges and innovations around defining infrastructure as code. That means the content goes deeper, the conversations are more relevant, and the takeaways are directly applicable to your processes.

There will be sessions on:

Getting started with IaC
Managing IaC at scale across teams and environments (using OpenTofu, Terraform, Ansible, Crossplane, and others)
Platform Engineering and IaC
How IaC impacts your maturity
AI in IaC

And that’s just scratching the surface. Check out the full agenda here.

One of the things I appreciate most about this conference is that it’s not about product pitches or high-level fluff. Practitioners share what worked, what broke, and how they improved. Whether you are just starting out, are in the trenches, or shaping platform strategy, there’s something for you here.

On another note, I’m also excited to contribute to this year’s lineup with a session on Getting Started with IaC, together with my colleague Emin Alemdar. But honestly, I’m just as excited to learn and to connect with others working through the same challenges.

The event will be on Thu, May 15, 2025 @ 11:00am EDT, so don’t forget to register here.

This post was originally posted on Medium.

Deploying a static Website with Pulumi

Flavius Dinu — Thu, 03 Apr 2025 11:45:30 +0000

This is a submission for the Pulumi Deploy and Document Challenge: Fast Static Website Deployment

What I Built

For this challenge, I've built a simple static website based on Docusaurus for tutorials and blog posts. As I'm not too seasoned with Frontend development, I only made small changes to the template, and added some very simple blog posts and tutorials there.

To make this more interesting, I've decided to build the infrastructure in both Python and JavaScript using Pulumi, and deploy the websites to different S3 buckets and subdomains.

The Python Pulumi code is deployed with GitHub Actions. This leverages static credentials for AWS embedded as repository secrets. I have implemented two workflows:

One for the actual deployment of the Pulumi Python code and the application called pulumi-deploy. This one runs on push to the main branch, but it first does a pulumi preview and waits for manual approval to do a pulumi up
Another one for removing the application from S3 and destroying the infrastructure called pulumi-destroy. This one will run manually

The JavaScript Pulumi code is deployed with Spacelift and leverages dynamic credentials based on Spacelift's cloud integration for AWS.

Live Demo Link

Even though it is the same website, it is deployed differently as I've mentioned above, thus it is using different subdomains:

Project Repo

flavius-dinu / pulumi_dev_challenge

Deploying a static website with Pulumi

The aim of this project is to showcase how to deploy a static website built using Docusaurus by using:

Pulumi (python) and GitHub Actions
Pulumi (javascript) and Spacelift

Repo structure explained:

.github -- GitHub Actions workflows for deploying the Pulumi python code
docs -- Documentation website built with Docusaurus, that can easily host your blogs and tutorials
javascript-infrastructure -- Pulumi javascript infrastructure that will be used by Spacelift to deploy the infrastructure and the application
python-infrastructure -- Pulumi python infrastructure that will be used by Spacelift to deploy the infrastructure and the application
images -- Images used throughout the documentation

To use this project, feel free to fork the repository and follow the instructions below.

Prerequisites

An AWS Route53 hosted zone for your domain
An AWS S3 bucket for Pulumi state hosting
An AWS role for building dynamic credentials for Spacelift

How to create

…

View on GitHub

Ensure you check the README.md to understand how to use the automation.

My Journey

Initially, I started this challenge by thinking about what kind of static website I would implement. I was actually debating between making a newsletter or creating a blog/tutorials website, the latter ending up being the right choice for me.

As I'm by no means a frontend developer, I've started researching what would be the best framework to achieve this based on my limited experience, and it seemed like Docusaurus was the easiest choice for me. Again, this is very subjective, so I would encourage everyone to do their own research before building something like this, especially if this is not your day-to-day.

I followed Docusaurus documentation for how to initialize the project, and then I deployed the website locally to see how it looks like. After diving into the repository template, I found some of the things I wanted to change pretty easily:

How to add an author to website
How to add social links
Where to change the footer
How to add new articles

For some of the other things, I have done a lot of "grep -ri insert_whatever" inside the repository to view what are the locations in which something is used and I made changes to reflect my view.

Now that I was done with the website, I've decided what I will do for the static hosting. I've debated between using other Cloud providers than the ones I'm accustomed to (AWS, Azure, Oracle Cloud), but as I already had my domain registered in AWS Route53, I ended up choosing AWS.

Next, I had to decide what services I would use so I chose:

S3 bucket for static content storage
CloudFront distribution for content delivery
ACM certificate for HTTPS for DNS validation
Route53 DNS configuration: 1 record for ACM certificate validation and an A record pointing to the CloudFront distribution

Then I built the initial version of the infrastructure using Pulumi with Python, and I decided to deploy it via Spacelift. Then I thought to myself, wouldn't it be more interesting to explore a different way to build the infrastructure and to deploy it? So I ended up with two versions of the infrastructure, one implemented in Python that I decided to deploy, in the end, with GitHub Actions, and another one implemented in Javascript which I ended up deploying with Spacelift.

By having two deployment options, I could easily see Pulumi in action and how the process is different when using a generic CI/CD and an Infrastructure Orchestration platform. It took me a couple of hours to build the GitHub Actions CI/CD pipelines, but with Spacelift I was done in less than an hour.

I could've easily just keep a single version of the infrastructure, but I wanted the work to be more challenging, and that's why I went for multiple versions.

During this process, I faced a couple of issues, but the most important thing that I've learned is the fact that CloudFront only accepts ACM certificates from the us-east-1 region.

This made me waste some time, as I couldn't really understand why that was happening. So to make this clear for everyone, this occurs because even though CloudFront is a global service, its configuration and metadata are managed centrally in us-east-1. By requiring certificates in us-east-1, AWS avoids the complexity of doing certification syncs across regions. I'm sure this is in the study material for multiple AWS certifications, but as I'm not using CloudFront too much, it really slipped my mind.

Using Pulumi

I used Pulumi to define and deploy the necessary infrastructure for a static website in AWS. As Pulumi is versatile, it made it easy for me to build my infrastructure resources in the programming languages I like to use.

It was very easy to set up the boilerplates for both the Python and Javascript examples by running:

pulumi new python
pulumi new javascript

After following the prompts, I started the actual development of the Python project.

Pulumi helped me easily:

create an S3 bucket with website hosting enabled and tags
use a CloudFront distribution to serve content securely, ensure only CloudFront can access the S3 bucket using Origin Access Identity, and enable HTTPS using ACM-issued certificates
implement DNS Configuration with Route53 to retrieve the existing hosted zone and create the necessary DNS records

Why was Pulumi Beneficial

Multi-Language Support - I implemented the same infrastructure twice using Python and JavaScript which can be useful for teams that have different backgrounds
Resource Management with code - Pulumi lets you take advantage of programming capabilities (loops, conditionals) to dynamically configure resources; Using functions such as the apply function makes it easy to handle dependencies
Easy to use by developers - My background is in DevOps engineering, so I've seen firsthand the issues developers have with DevOps processes. Without self-service, developer velocity is affected, but by leveraging something that developers know, the impact is lower.
Stateful approach - Pulumi's stateful nature makes it easy to detect changes, manage dependencies, and ensure idempotency

Future developments

For this small project, I believe the infrastructure code is in good shape, but it would be beneficial to take advantage of some of the capabilities infrastructure orchestration platforms offer out of the box like:

Policy as code - Policies could be implemented to restrict the Cloudfront distribution's Price class, require multiple approvals for runs, and ensure you are notified in your Slack/MS teams about failed runs
Drift detection - Drift happens, so why let it ruin your application? Implementing drift detection and remediation can be a lifesaver when it comes to ensuring infrastructure reliability and resilience
Self-service - Developers should be able to deploy resources on demand while staying safe
Secrets management - Use a specialized secrets management solution such as HashiCorp Vault, OpenBao, or Pulumi ESC to ensure secrets are securely stored and encrypted
Write unit tests - Capture issues before applying the code
Integrate security vulnerability scanning - Ensure you are not deploying code with vulnerabilities: use Checkov, Kics, or other specialized solutions for that

In addition to this, we can explore other deployment options, such as Pulumi Cloud, use other programming languages to build the infrastructure, and even choose other cloud providers.

Key points

This was a very fun challenge that had real-world applicability. I believe Pulumi's flexibility makes it a powerful choice for managing cloud infrastructure, especially where teams prefer using programming languages.

Pulumi integrates with ease with many CI/CDs and Infrastructure Orchestration Platforms to automate deployments, minimize human errors and implement governance.

It was a great opportunity to use Pulumi in different contexts and leverage modern DevOps technologies to steamline cloud infrastructure management.

Keep building!

Running AI Models in Docker

Flavius Dinu — Fri, 28 Mar 2025 10:52:51 +0000

Do you know what is the easiest way to run AI models? It’s of course, using Docker.

Docker has added support for running AI models in Docker Desktop 4.40.0, and it’s sooo easy to get started — right now, this is only available on Mac with Apple Silicon. Soon, this will also be available on Windows.

Note: I got early access to the feature, and this will be soon GA.

How does the Model Runner work?

The Docker Model Runner doesn’t run in a container, and it uses a host-installed inference server (llama.cpp), that runs locally on your computer. In the future, Docker will support additional inference servers.

To break down how this works:

You have a host-level process that allows direct access to the hardware GPU acceleration
GPU acceleration is enabled via Apple’s Metal API during query processing.
Models are cached locally in your host machine’s storage and are dynamically loaded into memory by llama.cpp when needed This means that your data never leaves your infrastructure.

Models are stored as OCI artifacts in Docker Hub, ensuring compatibility with other registries, including internal ones. This approach enables faster deployments, reduces disk usage, and improves UX by avoiding unnecessary compression.

How can you use the Model Runner?

To get started, first go to Docker Desktop, and under Settings select Features in Development, and ensure the Enable Docker Model Runner is toggled on.

After you enable the feature, make sure you restart Docker Desktop.

To see if the model runner is up, you can simply run:

docker model status
Docker Model Runner is running

If you want to see what are the available models in Docker:

docker model list
{"object":"list","data":[]}

Of course, you haven’t pulled any model yet, the list is going to be empty, as it is in my case.

Right now, you can find a list of models here:

To pull a model, you can simply run:

docker model pull model_name

For this example, I will use ai/llama3.2.

docker model pull ai/llama3.2
Model ai/llama3.2 pulled successfully
We are now ready to run the model. You can do it either interactively, or just do a one-off command.

docker model run ai/llama3.2                                
Interactive chat mode started. Type '/bye' to exit.
> Hey how are you?
I'm just a language model, so I don't have feelings or emotions like 
humans do, but I'm functioning properly and ready to help you 
with any questions or tasks you may have! 
How about you? How's your day going?
docker model run ai/llama3.2 "Hello"
Hello! How are you today? Is there something I can help you with, or would you like to chat?

Well, this is cool, but you want to take it to the next level and use it in your apps, right?

You can connect to the model in three ways:

From the container by using the internal DNS name: http://model-runner.docker.internal/
From the host using the Docker Socket
From the host using TCP

Another great thing is that the API offers Open-AI compatible endpoints:

GET /engines/{backend}/v1/models
GET /engines/{backend}/v1/models/{namespace}/{name}
POST /engines/{backend}/v1/chat/completions
POST /engines/{backend}/v1/completions
POST /engines/{backend}/v1/embeddings

Wrap-up

Docker Model Runner makes running AI models ridiculously simple. With just a few commands, you can pull, run, and integrate models into your applications — all while keeping everything local and secure.

This is just the beginning — as Docker expands support to Windows and additional inference servers, the experience will only get better.

Stay tuned, and keep building 🐳

How to effectively use YAML variables in your Terraform code

Flavius Dinu — Fri, 21 Mar 2025 14:59:34 +0000

TL/DR:

In this post we will explore how to easily use YAML variables inside of your Terraform code.

The best part? You don't need any external tools for that and everything also applies to OpenTofu.

First, let's take a look at a very simple example that will create one ec2 instance.

provider "aws" {
  region = "eu-west-1"
}

resource "aws_instance" "this" {
  instance_type = "t2.micro"
  ami           = "ami_id"
}

Now let's build a very straightforward yaml configuration that will replace the hardcoded values:

region: "eu-west-1"
instance_type: "t2.micro"
ami: "ami_id"

Let's suppose we've saved this yaml configuration in the same directory where the Terraform configuration is kept, in a file called vars.yaml.

To make use of this file, we will need to use it in a locals block. Terraform has a built-in file function that let's you read the contents of your files. Because this is a yaml file, we will need to actually use something to make Terraform read from it properly. Luckily there is a function available for this too, called yamldecode.

The logic would be to first open the file, then decode it, then use the local variable in which we implement the logic to use the fields from inside of it.

This is how this will look like:

locals {
  yaml_vars = yamldecode(file("./vars.yaml"))
}

provider "aws" {
  region = local.yaml_vars.region
}

resource "aws_instance" "this" {
  instance_type = local.yaml_vars.instance_type
  ami           = local.yaml_vars.ami
}

Pretty straightforward, right? Now, you can take this up a notch and use it in a for_each as well.

Let's suppose you build this yaml file for your instances and vpcs:

instances:
  instance1:
    instance_type: t2.micro
    ami: ami1
    tags: {"env": "dev"}
  instance2:
    instance_type: t3.micro
    ami: ami2
    tags: {"env": "dev"}
  instance3:
    instance_type: t3.micro
    ami: ami2
    tags: {"env": "dev"}

vpcs: 
  vpc1:
    cidr_block: 10.0.1.0/24
  vpc2:
    cidr_block: 10.0.2.0/24

This is how your terraform code will look like:

locals {
  yaml_vars = yamldecode(file("./vars.yaml"))
}

provider "aws" {
  region = "eu-west-1"
}

resource "aws_instance" "this" {
  for_each      = local.yaml_vars.instances
  instance_type = each.value.instance_type
  ami           = each.value.ami
  tags          = merge({
    Name: each.key
  }, each.value.tags)
}

resource "aws_vpc" "this" {
  for_each = local.yaml_vars.vpcs
  cidr_block = each.value.cidr_block
}

Now you can quickly generate as many EC2 instances as you'd like, and as many VPCS as you'd like by easily modifying the yaml file.

Docker from 0 to Hero

Flavius Dinu — Sat, 15 Mar 2025 12:25:42 +0000

Docker has grown a lot in popularity in recent years due to the way it simplifies application packaging and deployment. In my view, understanding Docker and its core concepts is essential for software and DevOps engineers alike.

In this article we will cover:

Docker Intro
Docker Images and Containers
Docker Networks and Volumes
Docker Registries
Docker Compose
Docker Security
Docker Swarm and Kubernetes
Docker CI/CD
Docker Optimization and Beyond Docker

Check out my YouTube Channel!

1. What is Docker?

Docker is an open-source platform that allows you to automate your application lifecycle using container technologies. To put it simply, containers are just simple, lightweight, and isolated environments that run your applications.

Docker provides an abstraction layer on top of the host operating system, allowing applications to run consistently regardless of differences in the underlying infrastructure.

Why use Docker?

There are several reasons why Docker has become so adopted in all IT processes.

With Docker you can enable your developers to package their applications and all their dependencies into a single element (container) — ensuring consistency and reusability across different environments. This eliminates (or at least tries) the infamous “It works on my machine” problem, which let’s face it — it’s so annoying.
Docker provides a lightweight alternative to virtualization. The difference between containers and virtual machines is that containers share the host operating system kernel, resulting in reduced overhead and faster startup times compared virtual machines (VMs include a full guest operating system, meaning they come up with their own kernel, which runs on top of a hypervisor).
Docker allows for the easy scaling and deployment of applications across different environments, making it a great solution for cloud-native architectures.

Note: Interviewers will always ask you what’s the difference between containers and virtual machines, so make sure you understand and remember it.

Docker Key benefits

Here are some of the key benefits Docker offers:

Consistency and reusability — Ensures your application will run according to plan, independent of the environment it runs in
Portability — By keeping your Docker images as small as possible, you can easily share them accross different environments.
Efficiency — Being lightweight, Docker containers reduce the overhead when compared to virtual machines. It makes them faster to start and more efficient in terms of resource consumption.
Scalability — Scaling up or down application, is something that Docker can do easily. Combining it with Docker Compose, Docker Swarm or Kubernetes, takes scalability to the next level.
Isolation — Docker ensures that different applications that run on the same host don’t interfere with each other
Version control and rollbacks — By pairing your Docker images and Docker configurations with version control, you can easily have different versions of your Docker images, making rollbacks a piece of cake

Getting Started with Docker

Now that we have a fundamental understanding of Docker is, and what are the benefits of using Docker, let’s get started with the installation and setup.

Installing Docker

Installing Docker is simple, as Docker provides installation packages for different operating systems, including macOS, Linux distributions and Windows. Simply download the appropriate package for your system and follow the installation instructions provided by Docker.

For the majority of the operating systems, the easiest thing to do is installDocker Desktop.

Docker Architecture Overview

To effectively work with Docker, you need to first understand the Docker architecture. At a high level, Docker consists of three main components:

Docker daemon (dockerd)— The Docker daemon is responsible for building, running, and monitoring containers.
Docker client — The Docker client is a command-line tool that allows you to interact with the Docker daemon. It sends commands to the daemon and receives information from it.
Docker registry — The Docker registry is a centralized repository for Docker images. It allows you to pull images from the registry and push your own images.

Docker’s Core Components

In addition to the other architecture components, Docker also relies on others as well:

Image — read-only template that contains everything needed to run an application (code, runtime, libraries, and system tools). You can leverage images created by others, or you have the ability to create your own.
Container — instance of an image that you can run, start, stop, and delete. They are isolated from each other and the host system, ensuring that applications run in a consistent environment. You can easily create multiple containers from the same image.
Registry — centralized place for storing and distributing Docker images. These images can be easily pulled on the host system, and based on them, you can create Docker containers. Docker Hub is the default public registry, but you can also set up private registries or use other public registries as well. Don’t worry, we will cover this in detail in another post.
Docker CLI — primary way to interact with Docker. It provides a set of commands that allow you to manage containers, images, networks, and volumes. With the Docker CLI, you can create, start, stop, and remove containers. You can also build, tag, and push images, as well as manage networks and volumes. We will explore some of the most useful commands below.
Dockerfile — file that contains a set of instructions for building a Docker image. It specifies the base image, different configurations that you want to be made, and commands to be executed when the image is built. The Dockerfile allows you to automate the process of creating Docker images, making it easy to reproduce and share accross different environments.
Docker Compose — tool for defining and running multi-container Docker applications. It allows you to define a group of containers as a single service and configure their dependencies and network connections. With Docker Compose, you can easily define complex application architectures and manage their deployment and scaling.
Docker Swarm — native clustering and orchestration solution provided by Docker. It allows you to create and manage a swarm of Docker nodes, enabling high availability and scalability for your applications. With Docker Swarm, you can deploy and scale your applications across multiple Docker hosts, ensuring that they are always available and can handle increased workloads.

These core components work together to provide a powerful and flexible platform for building, deploying, and managing containerized applications with Docker.

Docker Commands

Now that we have covered the basics of Docker, let’s explore a list of Docker commands that will help you work with Docker more effectively.

Basic Docker Commands

Here are some of the basic Docker commands you’ll frequently use:

docker run: Run a command in a new container
docker build: Builds a Docker image
docker tag: Tags an image
docker start: Start one or more stopped containers
docker stop: Stop one or more running containers
docker rm: Remove one or more containers
docker ps: Gets details of all running Docker containers
docker ps -a: Gets details of all your Docker containers
docker cp: Copies entire files or folders between your local filesystem to your containers
docker logs: Gives you in-depth details into your containers

Docker Compose and Swarm commands

We will dive deep into Docker Compose and Swarm in the next part of this article. In the meantime, here are some commonly used Docker Compose and Swarm commands, just as a little teaser of what we will do:

docker-compose up: Create and start containers
docker-compose down: Stop and remove containers, networks, and volumes
docker-compose build: Build or rebuild services
docker swarm init: Initialize a swarm
docker swarm join: Join a swarm as a worker or manager
docker service create: Create a new service
docker node ls: List nodes in a swarm

This part was originally posted on Medium.

2. Docker images and Docker containers

What is a Docker image?

A Docker image is a template that will be used by your containers, in which you install all the packages required to run your applications.

Let’s get back to Docker images. Acting as blueprints for your Docker containers, a Docker image is composed of multiple read-only layers, stacked on top of each other. We will build a Docker image later on, I will show you the layers, so don’t worry.

Each of these layers is an instruction in your Dockerfile, and these layers may contain different information: you could have a layer that specifies the base image from which you are building your image, or you may have another layer that installs some dependencies that are required for your application, or a layer that simply copies some files from your local filesystem to your Docker image.

Regardless of the underlying infrastructure (having Docker installed on different operating systems), you can be 100% sure, that your image will run on it if Docker is installed (small caveat, your image architecture must match the host system’s architecture).

Building these images in layers means that Docker can reuse the layers to speed up the building process for the current image you are using and even reuse these layers across different images that you may be building. That’s why, to speed things up, you should avoid making changes to superior layers, as caching will be broken.

By default, caching is enabled, but this doesn’t mean that you cannot build an image without reusing the cached information. In some cases, you may want to do that, and there is an argument that comes in handy that lets you do so, and that is the --no-cache arg.

What is a Docker container?

A Docker container is an element created from a Docker image that fits the purpose you have set inside that particular image. Because Docker images are just blueprints, they don’t do anything on their own, and they need to be used inside a container to accomplish the task at hand.

Docker containers run in their own, isolated , environments, meaning that they are separated from each other and even the host system. Each of these containers has its filesystems, processes that are running, and network, while they are still sharing the operating system’s kernel. As mentioned in the previous article, this is one of the main differences between containers and virtual machines, so don’t forget to take note.

Containers are more lightweight when compared to their virtual machine counterparts, and they are designed to be ephemeral. You can easily start, stop, and destroy containers and all of their data may be lost if the data is not explicitly stored outside of it (in a Docker volume, or the relevant information may be pushed to different storing solutions, such as Amazon S3, or Azure Blob storage).

Docker containers are flexible and make them an ideal choice for microservices applications because you can scale them together or independently, or directly in container orchestration platforms such as Kubernetes or Docker Swarm.

Working with existing Docker images

As I’ve mentioned before you can easily use existing Docker images and create containers out of them.

I will just show you an easy example of how to do it, but we will talk more about this when we reach the part dedicated to registries. Docker has a neat way of checking if you have an image locally, otherwise it will pull it from the Docker Hub registry.

Pulling an image

To pull an image, we can use the docker pull image_name command. For the sake of this example, let’s pull the hello-world image and create a container from it:

docker pull hello-world
Using default tag: latest
latest: Pulling from library/hello-world
478afc919002: Pull complete 
Digest: sha256:53cc4d415d839c98be39331c948609b659ed725170ad2ca8eb36951288f81b75
Status: Downloaded newer image for hello-world:latest
docker.io/library/hello-world:latest

Now, let’s create the container:

docker run hello-world                                                        

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (arm64v8)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

Pushing an image

To push an image to a registry, you can run the docker push image_namecommand. As I’ve mentioned before, we will deep dive into a different part.

Listing your local images

To list your images, you can simply run the docker images command:

docker images         
REPOSITORY     TAG       IMAGE ID       CREATED          SIZE
kindest/node   <none>    c67167dbf296   3 months ago     980MB
hello-world    latest    ee301c921b8a   16 months ago    9.14kB

Removing a local image

To remove a local image, you can simply run the docker image rm image_nameor docker image rm first_few_letters_of_id

docker image rm ee
Untagged: hello-world:latest
Untagged: hello-world@sha256:53cc4d415d839c98be39331c948609b659ed725170ad2ca8eb36951288f81b75
Deleted: sha256:ee301c921b8aadc002973b2e0c3da17d701dcd994b606769a7e6eaa100b81d44
Deleted: sha256:12660636fe55438cc3ae7424da7ac56e845cdb52493ff9cf949c47a7f57f8b43
➜  episode2 docker images     
REPOSITORY     TAG       IMAGE ID       CREATED          SIZE
kindest/node   <none>    c67167dbf296   3 months ago     980MB

What is a Dockerfile?

A Dockerfile is a special document that contains instructions for building a Docker image. It is essentially a script of commands that Docker uses to automatically create a container image.

The Dockerfile supports the following instructions, and I’ll list them in the order of their importance (from my point of view, of course):

FROM — this is creating a new build stage, starting from a base image. This means that if you have two FROM instructions in the same Dockerfile, you will have two build stages
RUN — executes different commands inside the base image you are using. You can have multiple commands in a single RUN instruction, by using “&&”
CMD — the default command the docker container will run
ENTRYPOINT — specifies the default executable
EXPOSE — ports your app is listening on
ADD — adds files from the host system, URLs, and can also add files and extract them
COPY — adds files from the host system only
ARG — build-time variable that can be used in other instructions
ENV — sets env variables inside the docker container
WORKDIR — sets the working directory for any other commands that may run after it
VOLUME — creates a mount point with the specified path
SHELL — sets the shell
USER — sets the user
HEALTHCHECK — defines a command to test the health of the container
STOPSIGNAL — sets the system call signal for exiting a container
ONBUILD — this will set an instruction for the image when it is used as a base image for another one
LABEL — adds metadata to the image in a key-value format
MAINTAINER — deprecated in favor of LABEL, was used to specify the maintainer of the image

Note 1: You will be asked in interviews what is the difference between ADD and COPY. Keep in mind that COPY can be used only for copying files from the host filesystem, while ADD can also get files from a URL or unarchive files.

Note 2 : You will get another question in interviews that refers to the difference between CMD and ENTRYPOINT. CMD is used for providing default arguments for the container’s executable, while ENTRYPOINT defines the executable itself. If you set CMD, and not set ENTRYPOINT, what happens is CMD acts as the ENTRYPOINT.

Dockerfile tutorial

Nothing goes better in a tutorial than a real-life example.

If you want to watch a video instead, I have you covered:

To give you some chills from your agile process, let’s suppose you receive the following ticket:

Title : Create a Docker image that standardizes our team’s development process

As a : DevOps Engineer

I want : To build a standard development environment for my team

So that : We can ensure that everyone develops their code with the same versions

Description : Our image should have the following:

start from an Alpine image
latest open-source version of Terraform installed
OpenTofu 1.8.1
Python 3.12 and pip
kubectl 1.28
Ansible 2.15
Golang 1.21

Acceptance criteria:

A Dockerfile is created that builds the image with all the tools specified and their versions
The image is built successfully and tested to ensure all tools function properly
Documentation on how to use the image is provided

Ok, now that we’ve received the ticket, let’s start building the Dockerfile that solves it.

We will start with a specific version of Alpine, and you may wonder why we are not using the “latest” keyword. This happens because, in a new version, we may face some dependency issues or unexpected changes that may break your image build. It is a best practice to avoid using “latest” for anything that you are building because, in this way, you ensure consistency.

FROM alpine:3.20

So, in the above example, I’m specifying the base image as Alpine:3.20.

Now, we haven’t receive exact instructions about what Terraform version to use, but we know that we have to use the latest open-source version. After some research on their repository, we have found out that the latest open-source version is 1.5.7:

Now, we are ready to define our environment with all the versions we want to use. You may wonder why we are defining them inside an ENV block. Well, that’s because we want to be easily able to update the versions when this is required in the future.

Also, I have defined the pipx bin directory to something in our path. This will be required to easily install Ansible.

ENV TERRAFORM_VERSION=1.5.7 \
    OPENTOFU_VERSION=1.8.1 \
    PYTHON_VERSION=3.12.0 \
    KUBECTL_VERSION=1.28.0 \
    ANSIBLE_VERSION=2.15.0 \
    GOLANG_VERSION=1.21.0 \ 
    PIPX_BIN_DIR=/usr/local/bin

Next, let’s install some of the dependencies and some of the helpers you may require for a successful development environment:

RUN apk add --no-cache \
    curl \
    bash \
    git \
    wget \
    unzip \
    make \
    build-base \
    py3-pip \
    pipx \
    openssh-client \
    gnupg \
    libc6-compat

Now, let’s add the instructions that install Terraform:

RUN wget -O terraform.zip https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_linux_amd64.zip && \
    unzip terraform.zip && \
    mv terraform /usr/local/bin/ && \
    rm terraform.zip

We are first downloading the terraform archive, then we are unzipping it, next, we are moving the executable in our path, and in the end, we are removing the archive.

We will do the same process for OpenTofu:

RUN wget -O tofu.zip https://github.com/opentofu/opentofu/releases/download/v${OPENTOFU_VERSION}/tofu_${OPENTOFU_VERSION}_linux_amd64.zip && \
    unzip tofu.zip && \
    mv tofu /usr/local/bin/ && \
    rm tofu.zip

Next, let’s install kubectl:

RUN curl -LO "https://dl.k8s.io/release/v$KUBECTL_VERSION/bin/linux/amd64/kubectl" && \
    chmod +x kubectl && \
    mv kubectl /usr/local/bin/

This will download the kubectl binary, make it executable, and then move it to the path.

Now, let’s install Ansible:

RUN pipx install ansible-core==${ANSIBLE_VERSION}

We are finally ready to install the last tool, golang:

RUN wget https://golang.org/dl/go$GOLANG_VERSION.linux-amd64.tar.gz && \
    tar -C /usr/local -xzf go$GOLANG_VERSION.linux-amd64.tar.gz && \
    rm go$GOLANG_VERSION.linux-amd64.tar.gz && \
    ln -s /usr/local/go/bin/go /usr/local/bin/go && \
    ln -s /usr/local/go/bin/gofmt /usr/local/bin/gofmt

In the same way, we are downloading the archive, unarchiving it, but now we are creating some symlinks to ensure go is in our PATH.

Let’s also set a workdir. When we will run our container, this will be our starting directory:

WORKDIR /workspace

We should also add the command we want our Dockerfile to run:

CMD ["bash"]

In the end, your Dockerfile should look like this:

FROM alpine:3.20

ENV TERRAFORM_VERSION=1.5.7 \
    OPENTOFU_VERSION=1.8.1 \
    PYTHON_VERSION=3.12.0 \
    KUBECTL_VERSION=1.28.0 \
    ANSIBLE_VERSION=2.15.0 \
    GOLANG_VERSION=1.21.0 \ 
    PIPX_BIN_DIR=/usr/local/bin

RUN apk add --no-cache \
    curl \
    bash \
    git \
    wget \
    unzip \
    make \
    build-base \
    py3-pip \
    pipx \
    openssh-client \
    gnupg \
    libc6-compat

RUN wget -O terraform.zip https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_linux_amd64.zip && \
    unzip terraform.zip && \
    mv terraform /usr/local/bin/ && \
    rm terraform.zip

RUN wget -O tofu.zip https://github.com/opentofu/opentofu/releases/download/v${OPENTOFU_VERSION}/tofu_${OPENTOFU_VERSION}_linux_amd64.zip && \
    unzip tofu.zip && \
    mv tofu /usr/local/bin/ && \
    rm tofu.zip

RUN curl -LO "https://dl.k8s.io/release/v$KUBECTL_VERSION/bin/linux/amd64/kubectl" && \
    chmod +x kubectl && \
    mv kubectl /usr/local/bin/

RUN pipx install ansible-core==${ANSIBLE_VERSION}

RUN wget https://golang.org/dl/go$GOLANG_VERSION.linux-amd64.tar.gz && \
    tar -C /usr/local -xzf go$GOLANG_VERSION.linux-amd64.tar.gz && \
    rm go$GOLANG_VERSION.linux-amd64.tar.gz && \
    ln -s /usr/local/go/bin/go /usr/local/bin/go && \
    ln -s /usr/local/go/bin/gofmt /usr/local/bin/gofmt

WORKDIR /workspace

CMD ["bash"]

Now, let’s go to the directory that contains our Dockerfile and build the image:

docker build -t dev_image:1.0.0 .
[+] Building 38.2s (8/11)                                                                                                                          docker:desktop-linux
[+] Building 48.1s (12/12) FINISHED                                                                                                                docker:desktop-linux
 => [internal] load build definition from Dockerfile                                                                                                               0.0s
 => => transferring dockerfile: 1.46kB                                                                                                                             0.0s
 => [internal] load metadata for docker.io/library/alpine:3.20                                                                                                     0.5s
 => [internal] load .dockerignore                                                                                                                                  0.0s
 => => transferring context: 2B                                                                                                                                    0.0s
 => CACHED [1/8] FROM docker.io/library/alpine:3.20@sha256:0a4eaa0eecf5f8c050e5bba433f58c052be7587ee8af3e8b3910ef9ab5fbe9f5                                        0.0s
 => [2/8] RUN apk add --no-cache     curl     bash     git     wget     unzip     make     build-base     py3-pip     pipx     openssh-client     gnupg     libc  17.3s 
 => [3/8] RUN wget -O terraform.zip https://releases.hashicorp.com/terraform/1.5.7/terraform_1.5.7_linux_amd64.zip &&     unzip terraform.zip &&     mv terraform  2.7s 
 => [4/8] RUN wget -O tofu.zip https://github.com/opentofu/opentofu/releases/download/v1.8.1/tofu_1.8.1_linux_amd64.zip &&     unzip tofu.zip &&     mv tofu /usr  4.1s 
 => [5/8] RUN curl -LO "https://dl.k8s.io/release/v1.28.0/bin/linux/amd64/kubectl" &&     chmod +x kubectl &&     mv kubectl /usr/local/bin/                       5.8s 
 => [6/8] RUN pipx install ansible-core==2.15.0                                                                                                                    7.8s 
 => [7/8] RUN wget https://golang.org/dl/go1.21.0.linux-amd64.tar.gz &&     tar -C /usr/local -xzf go1.21.0.linux-amd64.tar.gz &&     rm go1.21.0.linux-amd64.tar  9.2s 
 => [8/8] WORKDIR /workspace                                                                                                                                       0.0s 
 => exporting to image                                                                                                                                             0.5s 
 => => exporting layers                                                                                                                                            0.5s 
 => => writing image sha256:23fe925c0eb2e0931bc86f592373bcd13916e6dbbb4ce74b18fff846fb8f2f4d                                                                       0.0s 
 => => naming to docker.io/library/dev_image:1.0.0                                                                                                                 0.0s 

What's next:
    View a summary of image vulnerabilities and recommendations → docker scout quickview

The “-t” flag of the docker build command, lets us specify the image :.

Let’s see our newly created image:

docker images
REPOSITORY     TAG       IMAGE ID       CREATED         SIZE
dev_image      1.0.0     23fe925c0eb2   7 minutes ago   783MB

Now, let’s create a container from our image and check all the image versions to see if we meet the acceptance criteria we have in our ticket:

docker run -ti dev_image:1.0.0
062c8343eef7:/workspace#

These two options combined (“-ti”), allow you to run a container interactively with a terminal attached. This is especially useful for running a shell inside the container, so you can execute commands directly inside the container, as we would like to do with this.

Let’s check out our tools versions:

Now, we have met two out of three things related to our acceptance criteria, and I guess that the documentation can be easily written from all the details provided above, so we can say that this ticket can be moved into review 😁

In the real world, you will most likely want to take advantage of an editor to edit your code and run it from the Docker container. You would also want to give a name to your container, so let’s do that. The easiest way to do this is when you are creating it like so:

docker run -ti --name dev_container2 -v ~/Workspace:/workspace dev_image:1.0.0

This command will bind the host directory Workspace from the home of my user to /workspace. If I create a container in this way, I will be redirected to my workspace where I have all my code. Pretty neat, right?

You may ask yourself, why I deleted the archives from the image when I was creating it. The reason is pretty simple, I wanted to make the image as small as possible to keep it portable.

Everything seems simple, right? Well, it’s not. Until I got all the dependencies right, I messed it up a million times, so don’t worry if you will also mess it up, it’s just part of the process.

Managing Docker containers

Listing containers

We can list containers with the docker ps command, but this alone will only show the containers that are running:

docker ps                                                                     
CONTAINER ID   IMAGE                  COMMAND                  CREATED       STATUS       PORTS                       NAMES
8315e81fd8e6   kindest/node:v1.30.0   "/usr/local/bin/entr…"   2 weeks ago   Up 9 hours   127.0.0.1:53925->6443/tcp   my-cluster-control-plane

If you want to see all your existing containers, you can run docker ps -a

docker ps -a
CONTAINER ID   IMAGE                  COMMAND                  CREATED          STATUS                     PORTS                       NAMES
d6ae4b01f9de   dev_image:1.0.0        "bash"                   5 minutes ago    Exited (0) 5 minutes ago                               dev_container2
ad38f4f944e8   dev_image:1.0.0        "bash"                   5 minutes ago    Exited (0) 5 minutes ago                               dev_container
062c8343eef7   dev_image:1.0.0        "bash"                   16 minutes ago   Exited (0) 7 minutes ago                               mystifying_bhaskara
8315e81fd8e6   kindest/node:v1.30.0   "/usr/local/bin/entr…"   2 weeks ago      Up 9 hours                 127.0.0.1:53925->6443/tcp   my-cluster-control-plane

Starting a container

This is a very simple process, you can simply run docker start container_name or docker start first_few_letters_of_id

episode2 docker start d6
d6
➜  episode2 docker ps      
CONTAINER ID   IMAGE                  COMMAND                  CREATED         STATUS         PORTS                       NAMES
d6ae4b01f9de   dev_image:1.0.0        "bash"                   7 minutes ago   Up 2 seconds                               dev_container2
8315e81fd8e6   kindest/node:v1.30.0   "/usr/local/bin/entr…"   2 weeks ago     Up 9 hours     127.0.0.1:53925->6443/tcp   my-cluster-control-plane

Attaching to a container

You’ve started your dev container, but how can you use it? Well, you need to attach to it. The command is docker attach container_name or docker attach first_few_letters_of_id

episode2 docker attach d6
d6ae4b01f9de:/workspace#

Stopping a container

Similar to starting, this is a very simple process, you can simply run docker stop container_name or docker stop first_few_letters_of_id

episode2 docker start d6 
d6
➜  episode2 docker ps       
CONTAINER ID   IMAGE                  COMMAND                  CREATED          STATUS         PORTS                       NAMES
d6ae4b01f9de   dev_image:1.0.0        "bash"                   11 minutes ago   Up 5 seconds                               dev_container2
8315e81fd8e6   kindest/node:v1.30.0   "/usr/local/bin/entr…"   2 weeks ago      Up 9 hours     127.0.0.1:53925->6443/tcp   my-cluster-control-plane
➜  episode2 docker stop d6 
d6
➜  episode2 docker ps     
CONTAINER ID   IMAGE                  COMMAND                  CREATED       STATUS       PORTS                       NAMES
8315e81fd8e6   kindest/node:v1.30.0   "/usr/local/bin/entr…"   2 weeks ago   Up 9 hours   127.0.0.1:53925->6443/tcp   my-cluster-control-plane

Removing containers

To remove a container, you will simply run docker rm container_name or docker rm first_few_letters_of_id

docker rm d6                                                                  
d6
➜  episode2 docker ps -a
CONTAINER ID   IMAGE                  COMMAND                  CREATED          STATUS                      PORTS                       NAMES
ad38f4f944e8   dev_image:1.0.0        "bash"                   14 minutes ago   Exited (0) 14 minutes ago                               dev_container
062c8343eef7   dev_image:1.0.0        "bash"                   25 minutes ago   Exited (0) 16 minutes ago                               mystifying_bhaskara
8315e81fd8e6   kindest/node:v1.30.0   "/usr/local/bin/entr…"   2 weeks ago      Up 9 hours                  127.0.0.1:53925->6443/tcp   my-cluster-control-plane

This was originally posted on Medium.

3. Docker Network and Volumes

Docker networks types

Docker offers several network types, each serving different purposes:

1. Bridge Network

The bridge network is the default network type in Docker. When you start a container without specifying a network, it’s automatically attached to the bridge network.

Key features :

Containers on the same bridge can communicate with each other
Uses Network Address Translation (NAT) for external communication

2. Host Network

The host network removes network isolation between the container and the Docker host.

Key features :

Container uses the host’s network stack directly
Useful for optimizing performance in specific scenarios

3. Overlay Network

Overlay networks are used in Docker Swarm mode to enable communication between containers across multiple Docker hosts.

Key features :

Enables multi-host networking
Essential for deploying swarm services

4. Macvlan Network

Macvlan networks allow you to assign a MAC address to a container, making it appear as a physical device on your network.

5. Ipvlan Network

Ipvlan is similar to Macvlan but works at the network layer instead of the data link layer.

6. None Network

The none network disables networking for a container. It’s useful when you want to completely isolate a container from the network

To be completely honest, I haven’t had a need in my use cases for #4 or #5, but I have worked in some of my use cases with the rest of them. Of course, bridge is the one I’ve used the most.

Docker network examples

Let’s list the existing networks:

docker network list
NETWORK ID     NAME      DRIVER    SCOPE
9a571665758c   bridge    bridge    local
ff5a515b4fd3   host      host      local
cdf9ca0775b3   kind      bridge    local
a62f6451cad1   none      null      local

Bridge network example

Now, let’s create a new bridge network:

docker network create --driver bridge my_new_bridge

You will see an output similar to this, which will actually show you the entire id of the network bridge that you have created.

06ca1c432576d5e865da9a0bf4d...

If we list the existing networks again, we will see something similar:

docker network list                                
NETWORK ID     NAME            DRIVER    SCOPE
9a571665758c   bridge          bridge    local
ff5a515b4fd3   host            host      local
cdf9ca0775b3   kind            bridge    local
06ca1c432576   my_new_bridge   bridge    local
a62f6451cad1   none            null      local

Now, let’s create two alpine containers in this bridge, and ping one from the other:

docker run -dit --name alpine1 --network my_new_bridge alpine
docker run -dit --name alpine2 --network my_new_bridge alpine

We are using the -doption, because we don’t want to be attached to the containers.

If we run docker ps we can see that both containers are up and running:

docker ps                                                    
CONTAINER ID   IMAGE                  COMMAND                  CREATED          STATUS          PORTS                       NAMES
a51f7134e7ea   alpine                 "/bin/sh"                28 seconds ago   Up 28 seconds                               alpine2
cacfa2c11cf1   alpine                 "/bin/sh"                34 seconds ago   Up 33 seconds                               alpine1

Now, let’s ping one from the other:

docker exec -it alpine1 ping -c 2 alpine2

PING alpine2 (172.19.0.3): 56 data bytes
64 bytes from 172.19.0.3: seq=0 ttl=64 time=0.079 ms
64 bytes from 172.19.0.3: seq=1 ttl=64 time=0.136 ms

--- alpine2 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.079/0.107/0.136 ms


docker exec -it alpine2 ping -c 2 alpine1

PING alpine1 (172.19.0.2): 56 data bytes
64 bytes from 172.19.0.2: seq=0 ttl=64 time=0.063 ms
64 bytes from 172.19.0.2: seq=1 ttl=64 time=0.141 ms

--- alpine1 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.063/0.102/0.141 ms

As you can see, ping works just fine between the two containers.

Bonus: If you want to find the ip address of a container you can use the docker inspect command:

docker inspect alpine1 | grep -i IPAddress
            "SecondaryIPAddresses": null,
            "IPAddress": "",
                    "IPAddress": "172.19.0.2",

And of course, ping will work on the ip address as well, not only on the hostname:

docker exec -it alpine2 ping -c 2 172.19.0.2 

PING 172.19.0.2 (172.19.0.2): 56 data bytes
64 bytes from 172.19.0.2: seq=0 ttl=64 time=0.104 ms
64 bytes from 172.19.0.2: seq=1 ttl=64 time=0.159 ms

--- 172.19.0.2 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.104/0.131/0.159 ms

Host network example

A good candidate for a host network example would be a network monitoring tool that needs direct access to the host system.

Note: This example only works on Linux.

For this example, I will use the Prometheus node exporter, which is a tool that exposes hardware and OS metrics to Prometheus:

docker run -d --name node_exporter --network host prom/node-exporter

Let’s see the container port:

docker inspect --format='{{.Config.ExposedPorts}}' node_exporter

map[9100/tcp:{}]

Tip : docker inspect returns a json, so you can leverage the --format option to get the relevant information about your containers. In the above case, I’m checking the values under Config, and because one of the values is ExposedPorts, I can easily check the value under it as well.

Note: Docker Desktop for Windows and MacOS uses a lightweight virtual machine to run containers. This means the --network host option doesn't work as it does on Linux systems. If you want to use this example on your operating system, you should leverage port mapping like this:

docker run -d --name node_exporter -p 9100:9100 prom/node-exporter

Docker volumes

Docker volumes allow you to persist data generated by containers. Whatever data is inside a Docker container is ephemeral (everything is deleted after the container is deleted), and Docker volumes are the ones that save data if your workloads require it.

Why use Docker volumes?

There are a couple of reasons for which you would use Docker volumes:

Data Persistence — ensures that data isn’t lost when a container stops or it is deleted
Data Sharing — you can share data between containers
Isolation — data is kept outside the container’s writable layer, meaning that there is a better separation of concerns

Docker volume types

There are three types of Docker volumes:

Named volumes — volumes managed by Docker and stored in Docker’s default location on the host.
Anonymous volumes — similar to named volumes but they are assigned a random name when they are created. This name is unique across the Docker host. Just like named volumes, anonymous volumes persist even if you remove the container that uses them, except if you use the --rmflag when creating the container, in which case the anonymous volume is destroyed.
Bind mounts — while bind mounts are not exactly volumes, I like to think of them as volumes. You’ve already seen this in the last part when I mounted an existing directory from the host system to a Docker container. In that example, I was passing my workspace directory (which had my source code), to a container to ensure that I ran my code with the specific versions I used in my developer environment image.

There are also some docker volume classes, but I’ll let you discover them yourselves if you are interested.

Named volume example

Let’s create a named volume:

docker volume create nginx                                        
nginx

Now, let’s create a nginx container that uses this volume:

docker run -d --name nginx_named -v nginx:/usr/share/nginx/html -p 8080:80 nginx:latest 
d96aea6e5cd85ca238150606b0555ead92274a8561c69c6364f45322276ad063

I’ve mapped the volume I’ve created in the /usr/share/nginx/html and also mapped the 8080 port to access nginx from my local machine.

Now, let’s modify the content of the index.html file from our volume:

docker exec nginx_named sh -c "echo 'Hello from the named volume!' > /usr/share/nginx/html/index.html"
curl http://localhost:8080                                                                            
Hello from the named volume!

Ok, so now let’s delete the container, and recreate a new one to easily see if the content is still there:

docker stop nginx_named                                                                               
nginx_named

docker rm nginx_named                                                                  
nginx_named

docker run -d --name nginx_named -v nginx:/usr/share/nginx/html -p 8080:80 nginx:latest           
e1204ca5dd9aec68bbefb97e8b39c5acbca284f569edf44420e79a2b6b8b6cf7

curl http://localhost:8080                                                                            
Hello from the named volume!

As you can see, the content is still there because our data persisted.

Let’s clean up before going to the next example:

docker stop nginx_named                                                                               
nginx_named

docker rm nginx_named                                                                  
nginx_named

docker volume rm nginx    
nginx

Anonymous volume example

An anonymous volume doesn’t have to be created before the container. It can be used directly with the -v option, when you are doing the actual creation of the volume:

docker run -d --name nginx_anonymous_volume -v /usr/share/nginx/html -p 8081:80 nginx:latest 
e739ee4fe20f8cd7f253105b45aa46a311979f56aa6cfce5c858617fddaec800

docker exec nginx_anonymous_volume sh -c "echo 'Hello from Anonymous Volume!' > /usr/share/nginx/html/index.html"

curl http://localhost:8081
Hello from Anonymous Volume!

Now, let’s mount this volume to another nginx container. To do that, we will first need to get the unique identifier associated with it. We can do that

docker inspect --format='{{.Mounts}}' nginx_anonymous_volume

[{volume 6895fe7f79404ec8a2b337f3e74de6291d03c869c89d18bc866a11ae64b66e18

Based on that long unique identifier we can mount this volume to another container like so:

docker run -d --name nginx_anonymous_volume2 -v 6895fe7f79404ec8a2b337f3e74de6291d03c869c89d18bc866a11ae64b66e18:/usr/share/nginx/html -p 8082:80 nginx:latest
c55a85c4dca0fa34d00b9d4010b44f02d17e303f7602c5618d32fd5c6b8f62ca

Let’s check if this container will return the same result when we access it:

curl http://localhost:8082
Hello from Anonymous Volume!

This part was originally posted on Medium.

4. Docker registries

Publishing an image to a Docker registry

For this example, we will use Docker Hub, as it is Docker’s own container registry. Head out to https://hub.docker.com/, and create an account.

After you create your account you should see something similar to this:

I already have two images in this account, but given the fact that you’ve just created an account from scratch, you shouldn’t have anything. Now, let’s create a repository:

You will need to provide a name to your image and a short description, and you can also choose if your image is public or private. Keep in mind that you can only have one private image on a free account. Take also a note on the suggested commands to push an image:

docker tag local-image:tagname new-repo:tagname


docker push new-repo:tagname

For this example, I will use the development environment image that I’ve built in the second part. Don’t worry if this is the first part you see, as I’ll show you the image again, we will also build it and then tag and push it.

Before showing the Dockerfile and going through the process, this is the info I’ll use.

I clicked on create and this is the result:

Now, let’s go through the process and push an image.

This is the Dockerfile I will use:

FROM alpine:3.20

ENV TERRAFORM_VERSION=1.5.7 \
    OPENTOFU_VERSION=1.8.1 \
    PYTHON_VERSION=3.12.0 \
    KUBECTL_VERSION=1.28.0 \
    ANSIBLE_VERSION=2.15.0 \
    GOLANG_VERSION=1.21.0 \ 
    PIPX_BIN_DIR=/usr/local/bin

RUN apk add --no-cache \
    curl \
    bash \
    git \
    wget \
    unzip \
    make \
    build-base \
    py3-pip \
    pipx \
    openssh-client \
    gnupg \
    libc6-compat

RUN wget -O terraform.zip https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_linux_amd64.zip && \
    unzip terraform.zip && \
    mv terraform /usr/local/bin/ && \
    rm terraform.zip

RUN wget -O tofu.zip https://github.com/opentofu/opentofu/releases/download/v${OPENTOFU_VERSION}/tofu_${OPENTOFU_VERSION}_linux_amd64.zip && \
    unzip tofu.zip && \
    mv tofu /usr/local/bin/ && \
    rm tofu.zip

RUN curl -LO "https://dl.k8s.io/release/v$KUBECTL_VERSION/bin/linux/amd64/kubectl" && \
    chmod +x kubectl && \
    mv kubectl /usr/local/bin/

RUN pipx install ansible-core==${ANSIBLE_VERSION}

RUN wget https://golang.org/dl/go$GOLANG_VERSION.linux-amd64.tar.gz && \
    tar -C /usr/local -xzf go$GOLANG_VERSION.linux-amd64.tar.gz && \
    rm go$GOLANG_VERSION.linux-amd64.tar.gz && \
    ln -s /usr/local/go/bin/go /usr/local/bin/go && \
    ln -s /usr/local/go/bin/gofmt /usr/local/bin/gofmt

WORKDIR /workspace

CMD ["bash"]

Now let’s build it:

docker build -t devops-dev-env:1.0.0 .

Let’s check if it’s available locally:

docker images                         
REPOSITORY           TAG       IMAGE ID       CREATED        SIZE
devops-dev-env       1.0.0     c3069a674574   2 days ago     351MB

We are now ready to tag it with our repository name (I could’ve done this from the build phase, but I wanted to show you that you don’t need to worry about it). Your repository name will be different than mine, so ensure you make the changes accordingly.

docker tag devops-dev-env:1.0.0 flaviuscdinu93/devops-dev-env:1.0.0


docker images                                                      
REPOSITORY                      TAG       IMAGE ID       CREATED        SIZE
flaviuscdinu93/devops-dev-env   1.0.0     c3069a674574   2 days ago     351MB
devops-dev-env                  1.0.0     c3069a674574   2 days ago     351MB

Before pushing the image, ensure you are signed in with your user in Docker Desktop.

Let’s push the image:

docker push flaviuscdinu93/devops-dev-env:1.0.0
The push refers to repository [docker.io/flaviuscdinu93/devops-dev-env]
e46b168236e7: Pushed 
9b08b92b54a9: Pushed 
f3e42b4e3f66: Pushed 
4b49e924d63f: Pushed 
190aeee88612: Pushed 
8fc9c443438f: Pushed 
9110f7b5208f: Pushed 
1.0.0: digest: sha256:ffa6b.... size: 1794

Now, if we head back to Docker Hub, we will see our image there:

We can click on public view, and see what others will see when they find our image:

If we want to pull our image from the registry, we can simply run:

docker pull flaviuscdinu93/devops-dev-env:1.0.0

Top 10 Docker Registries

In this section, I will present some of the most popular registries in my view, but I won’t show you how to push an image to all of them. Nevertheless, the process is pretty similar to the one I’ve presented above.

Without further ado, these are the most popular registries:

Docker Hub
GitHub Container Registry
AWS Elastic Container Registry (ECR)
Azure Container Registry (ACR)
Google Artifact Registry (GCR)
GitLab Container Registry
Oracle Container Registry
JFrog Artifactory
Harbor
RedHat Quay

1. Docker Hub

We’ve already seen how Docker Hub works and how it looks so I don’t think it needs much presentation.

It offers a massive repository of publicly available images, including official images for popular open-source projects, databases, and programming languages.

Key features:

Free tier with access to public images
Automated builds from your VCS
Integrates easily with Docker Desktop
Offers private repositories (1 for free, but there are paid plans available)

2. GitHub Container Registry

If you are already using GitHub for version control, the GitHub Container Registry seems like a very viable choice for storing and managing your Docker or even OCI images.

The GitHub Container registry is part of the GitHub Packages offering that has other popular registries as well such as npm, RubyGems, Apache or Gradle.

Key Features:

Integrated with GitHub Actions for CI/CD
Free tier available with public repositories
Private repositories included in GitHub Pro and GitHub Teams plans
Fine-grained permissions

3. AWS Elastic Container Registry (ECR)

Amazon ECR is a Docker container registry provided by AWS. It integrates seamlessly with other AWS services, making it an excellent choice for users already in the AWS ecosystem.

Key Features:

Integrates seamlessly with Amazon ECS, EKS, Fargate
It is highly scalable and secure
Supports both private and public repositories
It has a Pay-as-you-go pricing model

4. Azure Container Registry (ACR)

Azure’s Container Registry is a Docker container registry provided by Microsoft. If you are already in the Microsoft ecosystem and are heavily dependent inside your workflows on a container registry, this seems like a no-brainer.

Key Features:

Integrated with Azure Kubernetes Service (AKS)
Geo-replication for high availability
Supports Helm charts and OCI artifacts
Built-in tasks for image building and patching

5. Google Artifact Registry (GAR)

Google Artifact Registry is Google’s answer to Docker image management., It’s pretty similar to the GitHub Packages offering and has a strong integration with Google Cloud services. Of course, if you are using Google Cloud, using Google’s registry makes a lot of sense.

Key Features:

Integrated with Google Kubernetes Engine (GKE)
Global availability and high security
Integrated with Cloud IAM for fine-grained access control
Automatic vulnerability scanning

6. GitLab Container Registry

GitLab Container Registry is a part of the GitLab CI/CD platform. It allows users to build, test, and deploy Docker images directly from their GitLab repositories, and if you keep your code on GitLab, you can easily use it for hosting your Docker images.

Key Features:

Integrated with GitLab CI/CD for seamless workflows
Free for both public and private repositories
Fine-grained access control
Built-in monitoring and logging

7. Oracle Container Registry

Oracle Container Registry is Oracle Cloud Infrastructure’s offering for managing your Docker images. It is tailored for Oracle products and services, making it ideal for organizations using Oracle Cloud Infrastructure.

Key Features:

Offers pre-built images for Oracle products
Integrated with Oracle Cloud Infrastructure
Secure and compliant with Oracle standards
Supports both public and private repositories

8. JFrog Artifactory

JFrog Artifactory is a universal artifact repository manager that supports Docker images along with other build artifacts. Artifactory is known for its robust security features and extensive integration options.

Key Features:

Supports Docker, Helm, and other package formats
Advanced security features such as Xray for vulnerability scanning
High availability and scalability
Integration with various CI/CD tools

9. Harbor

Harbor is a CNCF-graduated open-source container registry that emphasizes security and compliance. It has a range of features to manage Docker images, including RBAC, vulnerability scanning, and image signing.

Key Features:

Role-based access control (RBAC)
Integrated with Clair for vulnerability scanning
Image replication across multiple registries
Content trust and image signing

10. RedHat Quay

If you are in the RedHat ecosystem, Quay is a viable choice for hosting your Docker images. It offers both a public and private image repository, along with automated security scanning.

Key Features:

Continuous security monitoring
Integrated with OpenShift for seamless Kubernetes deployment
Supports multiple image storage backends
Customizable notifications and webhooks

This part was originally posted on Medium.

5. Docker Compose

What is Docker Compose?

Docker Compose is a Docker native tool that helps you manage multi-container applications by defining their configuration into a yaml file.

You can add all details related to your containers in this file and also define their networks and volumes.

It offers the following benefits:

Keeps it simple — you can define your entire application in a single yamlfile
Reusability — you can easily share and leverage version control for your configurations
Consistency — ensures your environments are the same, regardless of the operating system you are using
Scalability — you can easily scale up or down with simple Docker-Compose commands

Docker Compose commands

Here are some of the most important arguments that can be used with the docker-compose command:

up — creates and starts containers
down — stops and removes containers
start — starts existing containers for a service
stop — stops running containers without removing them
restart — restarts all stopped and running services
ps — lists containers and their statuses
logs — displays the log outputs of the services
run — runs a one-time command on a service
cp — copies files or folders between a container and the machine that runs Docker Compose
pull — pulls images for services defined in the compose file
build — builds or rebuilds services defined in the compose file
exec — executes a command in a running container

There are many other commands as well that you can use, but these are the most used ones. You can check what other commands are available by running docker-compose --help.

Docker Compose example

Before jumping into an example, let’s clarify some things first. If you write your Docker Compose code into a docker-compose.yml file, you won’t need to do anything special when you are running a command.

However, you don’t need to name it like, and you could choose any name you want. The only difference is, that you will need to specify the compose file with a -f option.

Ok, now we are ready to build an example. For this example, I will combine multiple technologies to demonstrate the power of Docker Compose:

Redis — in-memory data store to persist the API hits of our application
Flask — processes requests, and interacts with our Redis DB
NodeJS — will present the Flask application, making API requests to it
Nginx — reverse proxy routing incoming requests to the Node.js fronted and managing traffic between the frontend and backend

First things, first, I will create a docker-compose.yml file in which I will define a network:

networks:
  my_bridge_network:
    driver: bridge

I will use this bridge network for all of the containers I will define. Next, I will define a volume that Redis will use, as I want to have this data saved, regardless of running docker-compose down.

volumes:
  redis_data:

Now, let’s start defining our services:

services:
  redis:
    image: redis:6.2.6
    container_name: redis_server
    networks:
      - my_bridge_network
    volumes:
      - redis_data:/data

In the above example, I’m using the redis:6.2.6 image, and I’ve named my container redis-server. I am connecting to the existing network, and also leveraging my named volume in the /data directory.

Next, let’s define the flask application:

backend:
    image: python:3.9.7-slim
    container_name: flask_backend
    working_dir: /app
    volumes:
      - ./backend:/app
    command: >
      sh -c "pip install --no-cache-dir -r requirements.txt &&
        python app.py"
    networks:
      - my_bridge_network
    depends_on:
      - redis

As you can see, in this case, I’ve defined a working directory called /app and I’m mounting an existing directory on my local machine called backend in this mount point.

Next, I’m running two commands, one that installs the prerequisites in requirements.txt and the other that runs my application. At this point, I’m selecting the network, and I’ve also defined the depends_on redis, to wait for redis to be available, before running my application.

By now, you are wondering, what I have in the backend directory, so let me show you:

app.py — this connects to Redis using the container name and saves the number of API hits in Redis

from flask import Flask, jsonify
import redis

app = Flask(name)
r = redis.Redis(host='redis_server', port=6379, decode_responses=True)

@app.route('/')
def hello():
count = r.incr('hits')
return jsonify(message="Hello from Flask!", hits=count)

if name == "main":
app.run(host='0.0.0.0', port=5000)
requirements.txt — this will contain the requirements of our application

Flask
redis

Next, let’s define the frontend service:

 frontend:
    image: node:14.17.6-alpine
    container_name: node_frontend
    working_dir: /app
    volumes:
      - ./frontend:/app
    command: npm start
    ports:
      - "3000:3000"
    networks:
      - my_bridge_network

I’m using a node:14 image, and I’m also mapping an existing directory called frontend to /app, which is set as the working directory. I’m also specifying a command that starts the frontend application, the port mapping, and the same network.

In the frontend directory, there are multiple folders and files, but I’ve created only one manually:

index.js — I’m connecting here to my backend using the container name and Flask’s default port which is 5000, and I’m using a get to find out how many hits I have in my backend

const express = require('express');
const axios = require('axios');
const app = express();
const port = 3000;

app.get('/', async (req, res) => {
try {
const response = await axios.get('http://flask_backend:5000/');
res.send(<h1>${response.data.message}</h1><p>API hits: ${response.data.hits}</p>);
} catch (error) {
res.send(<h1>Error connecting to the backend</h1>);
}
});

app.listen(port, () => {
console.log(Frontend listening at http://localhost:${port});
});

To generate the other files, we need to go to our frontend directory and run the following:

npm init -y
npm install axios
npm install express

This will generate a node_modules folder, a package.json file and a package-lock.json file. Go to the package.json file and add a start command to the scripts. In the end, it should look similar to this:

{
  "name": "frontend",
  "version": "1.0.0",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1",
    "start": "node index.js"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "description": "",
  "dependencies": {
    "axios": "^1.7.7",
    "express": "^4.19.2"
  }
}

Now, we are ready to add the last container in our docker-compose configuration: nginx.

nginx:
    image: nginx:1.21.3
    container_name: nginx_proxy
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf
    ports:
      - "80:80"
    networks:
      - my_bridge_network
    depends_on:
      - frontend

Its setup is pretty similar to what we’ve seen before, and we are mounting an nginx.conf file to the default configuration path in nginx.

This is the content of nginx.conf:

events { }

http {
    server {
        listen 80;

        location / {
            proxy_pass http://node_frontend:3000;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
        }
    }
}

As you can see, we’ve set the proxy_pass to our frontend.

This is how our docker-compose.yaml file will look like in the end:

networks:
  my_bridge_network:
    driver: bridge

volumes:
  redis_data:

services:
  redis:
    image: redis:6.2.6
    container_name: redis_server
    networks:
      - my_bridge_network
    volumes:
      - redis_data:/data

  backend:
    image: python:3.9.7-slim
    container_name: flask_backend
    working_dir: /app
    volumes:
      - ./backend:/app
    command: >
      sh -c "pip install --no-cache-dir -r requirements.txt &&
        python app.py"
    networks:
      - my_bridge_network
    depends_on:
      - redis

  frontend:
    image: node:14.17.6-alpine
    container_name: node_frontend
    working_dir: /app
    volumes:
      - ./frontend:/app
    command: npm start
    ports:
      - "3000:3000"
    networks:
      - my_bridge_network

  nginx:
    image: nginx:1.21.3
    container_name: nginx_proxy
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf
    ports:
      - "80:80"
    networks:
      - my_bridge_network
    depends_on:
      - frontend

And this is our directory structure:

.
├── backend
│   ├── app.py
│   └── requirements.txt
├── docker-compose.yml
├── frontend
│   ├── index.js
│   ├── package-lock.json
│   └── package.json
│   └── node_modules (a lot of files and folders here, omitted)
└── nginx.conf

Now, if we run docker-compose up this is what we’ll see:

And if you run docker-compose down and then docker-compose up again, you will see the count remains the same, because we saved the volume, but keep in mind that if you want to delete the volume you can do that by running docker-compose down -v.

I know this was a fairly complicate example, but these are the examples you will face in real-life.

As a DevOps engineer, you don’t need to know how to build these applications, as your developers will build them, but you should be able to follow their Python, Javascript, Golang, Java code, and help them if there are issues with making these containers communicate.

At the same time, as a developer, you don’t need to know Docker Compose and all the instructions you have to build, but you need to know how to containerize your application in a way it makes sense. In this example, we could’ve built custom images that had, for example, the requirements already installed.

This part was originally posted on Medium.

6. Securing your Docker Environment

What are the most common Docker security risks?

Before understanding how to improve security-related issues, we must first understand what are the most common ones:

Vulnerable base images — if you are using outdated base images they might contain known vulnerabilities. To solve these issues you need to constantly scan for vulnerabilities and update them as soon as you are facing these kinds of issues
Excessive privileges — you might find it easier to run containers as root, but in reality, even though your job might be easier initially, by running a container with root privileges you will increase the risk of your host machine if the container is compromised.
Container breakout — if an attacker gets access to one of your containers, they be able to gain control over the host system as well. This usually happens because of vulnerable images, misconfigurations, and excessive privilege.
Permissive networks — as you already know, by default, Docker uses a bridge network that allows containers to communicate with each other and external networks. If you need to ensure better isolation and reduce the chances of having breaches, you need to ensure that you do not expose services that should be kept internal.
Insecure image registries — there are many registries that are available online from which you can pull your container images. If you are using untrusted registries, this can lead to pulling compromised images that may contain malware or other malicious software
Secrets management — if you are not handling your secrets well, they could be easily exploited in case of attacks
Vulnerabilities in orchestrators (K8s, Swarm) — you need to ensure proper RBAC and network policies for your orchestrators to avoid the issues described above

Finding vulnerabilities in your Docker images

There are many tools you can use to find vulnerabilities in the Docker images you are using, and you will get information about each vulnerability, severity levels, descriptions, and also references to CVE databases.

Here are my top three:

I will not compare them in this post, but I will show you how to use Trivy to scan your docker images.

Using Trivy to scan Docker images

Go to Trivy’s GitHub repository and follow the instructions on how to install it on your operating system.

Now, let’s run the image scanner and see what are the vulnerabilities that we have in the image we’ve built in the second part:

trivy image flaviuscdinu93/devops-dev-env:1.0.0

2024-09-08T15:43:58+03:00 INFO [vuln] Vulnerability scanning is enabled
2024-09-08T15:43:58+03:00 INFO [secret] Secret scanning is enabled
2024-09-08T15:43:58+03:00 INFO [secret] If your scanning is slow, please try '--scanners vuln' to disable secret scanning
2024-09-08T15:43:58+03:00 INFO [secret] Please see also https://aquasecurity.github.io/trivy/v0.55/docs/scanner/secret#recommendation for faster secret detection
2024-09-08T15:43:58+03:00 INFO Detected OS family="alpine" version="3.20.2"
2024-09-08T15:43:58+03:00 INFO [alpine] Detecting vulnerabilities... os_version="3.20" repository="3.20" pkg_num=71
2024-09-08T15:43:58+03:00 INFO Number of language-specific files num=4
2024-09-08T15:43:58+03:00 INFO [gobinary] Detecting vulnerabilities...
2024-09-08T15:43:58+03:00 INFO [python-pkg] Detecting vulnerabilities...
2024-09-08T15:43:58+03:00 WARN Using severities from other vendors for some vulnerabilities. Read https://aquasecurity.github.io/trivy/v0.55/docs/scanner/vulnerability#severity-selection for details.

The output will be a breakdown based on the different components you have installed in the image:

Because in my image, I’m starting from alpine:3.20, I have a couple of issues with my image as you can see above. Every vulnerability has a code, and if you want to learn more about it, you can just copy and paste it into Google or whatever other search engine you are using and search for it.

Check out this link if you want to learn more about CVE-2024–45490.

Now, those are just the vulnerabilities related to Alpine, but in my image, I have vulnerabilities related to other tools as well:

I won’t show you all the vulnerabilities because I have a ton. But as you can see, in Trivy’s output, if you want to solve these issues, you need to upgrade to a version in which these problems are not present anymore.

You can also run Trivy and return a json with all of the data related to a scan like so:

trivy image -f json -o results.json flaviuscdinu93/devops-dev-env:1.0.0

My scan has over 4k lines, so I won’t be able to add it here, but you get the point, I have some work to do to fix these issues.

Building a GitHub Actions pipeline that scans for vulnerabilities

We will build a very simple repository that contains a Dockerfile (the same one that builds the development environment for your DevOps engineers, used above) and a GitHub Actions workflow that builds the image and checks for vulnerabilities using Trivy.

This workflow will run only when there are changes to the Dockerfile, because it doesn’t make too much sense to run it otherwise.

name: Docker Image Scan with Trivy

on:
  push:
    paths:
      - 'Dockerfile'
  pull_request:
    paths:
      - 'Dockerfile'

Above we’ve added a rule for this workflow to run only when there is a push, or a pull request made that changes the Dockerfile.

Next, we can start defining our job, and checking out the code:

jobs:
  scan:
    name: Scan Docker Image
    runs-on: ubuntu-latest
    steps:
      - name: Checkout code
        uses: actions/checkout@v3

Before being able to check the image for vulnerabilities, we need to first install Docker and build the image:

- name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2

- name: Build image
  run: |
    docker build -t devimage:latest .

Next, we need to install Trivy:

- name: Install Trivy
  run: |
    sudo apt-get install wget apt-transport-https gnupg lsb-release
    wget -qO - https://aquasecurity.github.io/trivy-repo/deb/public.key | sudo apt-key add -
    echo deb https://aquasecurity.github.io/trivy-repo/deb $(lsb_release -sc) main | sudo tee -a /etc/apt/sources.list.d/trivy.list
    sudo apt-get update
    sudo apt-get install trivy

Now we are ready to do the scan. In this example, I will force Trivy to exit with an error when the severity is critical, otherwise, the exit code will be 0:

- name: Run Trivy vulnerability scanner
  run: |
    echo "Scanning docker-image"
    trivy image --exit-code 1 --severity CRITICAL devimage:latest
    trivy image --exit-code 0 --severity HIGH,MEDIUM devimage:latest

We already know that our image has some critical vulnerabilities, so our pipeline will exit with an error:

Let’s change this behavior to make it return a 0 exit code for all cases, just to see if it is working properly:

By making this change, we can see that our pipeline finishes successfully, and if we check the logs, we can see all of our scan results.

If you want to leverage the code directly you can get it from here.

Docker security best practices

To keep all of your Docker environment secure, you will need to implement at least the following best practices:

Improve your image security

Constantly scan for vulnerabilities, and use minimal base images such as Alpline or distroless to reduce the attack surface. In addition to this, implement multi-stage builds (we will talk about this in detail in the next parts) to minimize the final image size and reduce vulnerabilities. Leverage CI/CDs to constantly check for issues related to your images.

Implement least privilege access

Whenever you can, run your images as a different user than root. This will ensure that if your container is compromised, your host machine won’t be affected as much.

Set resource limits

Ensure your enforce resource limits for CPU and memory, as this will prevent denial of service attacks to resource exhaustion.

Use read-only filesystems

In some cases, your Docker container won’t need to write anything to the filesystem, so in these cases, it will make much more sense to leverage read-only filesystems. In this way, attackers will have a hard time to install malware on your containers.

Create and use different networks

By creating and using different network, you ensure that isolation is kept as much as possible and only the services that really need to be exposed, will be exposed.

Enable logging and monitoring

Monitor your container logs to detect any anomalies or even malicious activity. This will help in identifying strange behavior in real time.

This part was originally posted on Medium.

7. Docker Swarm vs Kubernetes

What is Docker Swarm?

Docker Swarm is Docker’s native orchestrating tool. It is working seamlessly with Docker, offering a straightforward approach to container orchestration.

Key Features:

Easy Setup : Swarm mode is built into the Docker engine, making it simple to set up and use.
Familiar Docker CLI : If you’re already using Docker, the learning curve for Swarm is minimal.
Service Discovery : Automatic service discovery and load balancing.
Rolling Updates : Built-in support for rolling updates and rollbacks.
Scaling : Easy horizontal scaling of services.

Let’s deploy the same application we’ve built in part 5, using Docker compose, the repository is here.

For Swarm, I’ve created a folder in the repository called swarm, that contains a slightly changed docker-compose file.

You will see that the first difference is the fact that I’m using an overlay network:

networks:
  my_overlay_network:
    driver: overlay

As mentioned in previous parts, overlay networks are distributed across multiple Docker hosts and combined with Swarm, creates a real orchestration platform.

Also, now you can easily place your services accross multiple nodes:

deploy:
    replicas: 1
    placement:
      constraints:
        - node.role == manager

Before deploying the configuration, we need to first initialize or join a swarm. In my case, I will initialize a new swarm:

docker swarm init

To deploy the configuration from the swarm directory you should navigate to it and run:

docker stack deploy -c docker-compose.yml my_stack

If you want to see the services deployed from inside the stack if you’ve created, you can easily run:

docker service ls                                                 

ID             NAME                    MODE         REPLICAS   IMAGE                 PORTS
xp3djy1t0d21   my_stack_backend        replicated   1/1        python:3.9.7-slim     
kyrr4hlsk8bq   my_stack_frontend       replicated   1/1        node:14.17.6-alpine   *:3000->3000/tcp
xkr5we3ht4bl   my_stack_nginx          replicated   1/1        nginx:1.21.3          *:80->80/tcp
xeofhdsz7fn1   my_stack_redis-server   replicated   1/1        redis:6.2.6

If everything is working well you can access the application at localhost:80:

Otherwise, to view logs, you can easily run:

docker service logs my_stack_serviceName

E.G:

docker service logs my_stack_backend

my_stack_backend.1.t8h4m4pllk2h@docker-desktop    | WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
my_stack_backend.1.t8h4m4pllk2h@docker-desktop    |  * Running on all addresses (0.0.0.0)
my_stack_backend.1.t8h4m4pllk2h@docker-desktop    |  * Running on http://127.0.0.1:5000
my_stack_backend.1.t8h4m4pllk2h@docker-desktop    |  * Running on http://172.21.0.3:5000
my_stack_backend.1.t8h4m4pllk2h@docker-desktop    | Press CTRL+C to quit
my_stack_backend.1.t8h4m4pllk2h@docker-desktop    | 10.0.1.4 - - [10/Sep/2024 10:50:05] "GET / HTTP/1.1" 200 -
my_stack_backend.1.t8h4m4pllk2h@docker-desktop    | 10.0.1.4 - - [10/Sep/2024 10:50:06] "GET / HTTP/1.1" 200 -
my_stack_backend.1.t8h4m4pllk2h@docker-desktop    | 10.0.1.4 - - [10/Sep/2024 10:50:06] "GET / HTTP/1.1" 200 -
my_stack_backend.1.t8h4m4pllk2h@docker-desktop    | 10.0.1.4 - - [10/Sep/2024 10:50:07] "GET / HTTP/1.1" 200 -
my_stack_backend.1.t8h4m4pllk2h@docker-desktop    | 10.0.1.4 - - [10/Sep/2024 10:50:07] "GET / HTTP/1.1" 200 -
my_stack_backend.1.t8h4m4pllk2h@docker-desktop    | 10.0.1.4 - - [10/Sep/2024 10:50:07] "GET / HTTP/1.1" 200 -
my_stack_backend.1.t8h4m4pllk2h@docker-desktop    | 10.0.1.4 - - [10/Sep/2024 10:50:07] "GET / HTTP/1.1" 200 -
my_stack_backend.1.t8h4m4pllk2h@docker-desktop    | 10.0.1.4 - - [10/Sep/2024 10:50:07] "GET / HTTP/1.1" 200 -
my_stack_backend.1.t8h4m4pllk2h@docker-desktop    | 10.0.1.4 - - [10/Sep/2024 10:50:07] "GET / HTTP/1.1" 200 -
my_stack_backend.1.t8h4m4pllk2h@docker-desktop    | 10.0.1.4 - - [10/Sep/2024 10:50:07] "GET / HTTP/1.1" 200 -
my_stack_backend.1.t8h4m4pllk2h@docker-desktop    | 10.0.1.4 - - [10/Sep/2024 10:50:07] "GET / HTTP/1.1" 200 -
my_stack_backend.1.t8h4m4pllk2h@docker-desktop    | 10.0.1.4 - - [10/Sep/2024 10:50:07] "GET / HTTP/1.1" 200 -

Pros of using Swarm:

Simple to learn and use
Tight integration with Docker
Lightweight and fast

Cons of using Swarm:

Limited advanced features compared to Kubernetes
Not as widely adopted in enterprise environments

What is Kubernetes?

Kubernetes, often abbreviated as K8s, is an open-source container orchestration platform originally developed by Google, and now under the CNCF umbrella. It offers a more comprehensive set of features for complex, large-scale deployments.

Key Features:

Auto-scaling : Horizontal pod autoscaling based on CPU utilization or custom metrics.
Self-healing : Automatic replacement of failed containers.
Advanced Networking : Powerful networking capabilities with support for various plugins.
Storage Orchestration : Dynamic provisioning of storage.
Declarative Configuration : Define the desired state of your system using YAML files.

Let’s reuse the above example, but now define the Kubernetes configuration for it. Again, the repository is here.

For K8s, I have defined two Docker images, one for the backend and the other one for the frontend:

# Dockerfile-BE
FROM python:3.9.7-slim

WORKDIR /app

COPY backend/* ./
RUN pip install --no-cache-dir -r requirements.txt

CMD ["python", "app.py"]


# Dockerfile-FE
FROM node:14.17.6-alpine

WORKDIR /app

COPY frontend/* .

RUN npm install

CMD ["node", "index.js"]

The first step I’ve done is I’ve built the images, tagged them appropiately and pushed them to the Docker registry. If you want to build your images and host them in your own registries, make sure you are in the root of the repository and then run:

docker build -t image_name:image_tag -f kubernetes/docker_images/Dockerfile_BE .
docker build -t image_name:image_tag -f kubernetes/docker_images/Dockerfile_FE .

Then, you can easily push the images to the registry of your choice using the docker push commands.

Now, if you want to use my public images they are available here:

flaviuscdinu93/be:1.0.3
flaviuscdinu93/fe:1.0.1

Ok, so for every service that I’m using I will build at least a deployment and a service, with the exception of Redis (which needs a volume), and Nginx (which a need a configmap to hold the configuration).

All the files are defined in the repository under the kubernetes folder, but let’s walk through a couple of them:

---
apiVersion: v1
kind: Namespace
metadata:
  name: dev

I’ve defined a namespace for all the different component I will be deploying into K8s to better isolate my resources.

For the backend, I’ve defined the following deployment file:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: backend
  namespace: dev
spec:
  selector:
    matchLabels:
      app: backend
  template:
    metadata:
      labels:
        app: backend
    spec:
      containers:
      - name: backend
        image: flaviuscdinu93/be:1.0.3
        ports:
        - containerPort: 5000

This will create a container in a pod based on the backend image I previosly defined, and open the 5000 port.

Next, I’ve defined a service to the backend to expose this container:

---
apiVersion: v1
kind: Service
metadata:
  name: backend
  namespace: dev
spec:
  selector:
    app: backend
  ports:
    - port: 5000

This happens because the selector is set to get the backend app.

The same things are happening for the others as well, the only difference is that redis will mount a volume, and nginx will use some data from a configmap:

# nginx configmap
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-config
  namespace: dev
data:
  nginx.conf: |
    events { }

    http {
      upstream frontend {
          server frontend:3000;
      }

      server {
          listen 80;

          location / {
              proxy_pass http://frontend;
              proxy_set_header Host $host;
              proxy_set_header X-Real-IP $remote_addr;
              proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
              proxy_set_header X-Forwarded-Proto $scheme;
          }
      }
    }


# redis deployment using the volume
spec:
    containers:
    - name: redis
      image: redis:6.2.6
      volumeMounts:
      - name: redis-data
        mountPath: /data
    volumes:
    - name: redis-data
      persistentVolumeClaim:
        claimName: redis-data

Ok, now that everything is defined we can apply the code. To do that go to the kubernetes directory and run:

kubectl apply -f .

# To see everything that was created you can easily run:

kubectl get all -n dev   
NAME                            READY   STATUS    RESTARTS      AGE
pod/backend-5459857cff-w2z8f    1/1     Running   0             37m
pod/frontend-679cdc8dc7-ts6lz   1/1     Running   0             37m
pod/nginx-6d6b99d9ff-pvzr4      1/1     Running   3 (37m ago)   37m
pod/redis-8545d5dbbf-dfmwz      1/1     Running   0             37m

NAME                   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
service/backend        ClusterIP   10.96.185.238   <none>        5000/TCP       37m
service/frontend       ClusterIP   10.96.234.210   <none>        3000/TCP       37m
service/nginx          NodePort    10.96.254.83    <none>        80:30542/TCP   37m
service/redis-server   ClusterIP   10.96.169.17    <none>        6379/TCP       37m

NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/backend    1/1     1            1           37m
deployment.apps/frontend   1/1     1            1           37m
deployment.apps/nginx      1/1     1            1           37m
deployment.apps/redis      1/1     1            1           37m

NAME                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/backend-5459857cff    1         1         1       37m
replicaset.apps/frontend-679cdc8dc7   1         1         1       37m
replicaset.apps/nginx-6d6b99d9ff      1         1         1       37m
replicaset.apps/redis-8545d5dbbf      1         1         1       37m

To access the application from our nginx reverse proxy we can port-forward it for now, as we haven’t defined any loadbalancers:

kubectl port-forward svc/nginx 8080:80 --namespace dev

Forwarding from 127.0.0.1:8080 -> 80
Forwarding from [::1]:8080 -> 80

Now go to localhost:8080 and see if you can access the application:

You can see that the application is working properly.

Pros of using K8s:

Highly scalable and flexible
Large ecosystem and community support
Backed by major cloud providers — AWS, Azure, Google Cloud

Cons of using K8s:

Steeper learning curve
More complex setup and management

Head to Head

If you are heavily invested in the Docker ecosystem and your team is more familiar with Docker and doesn’t want a steep learning curve, Docker Swarm will be a better choice, especially for a small to mid-sized deployment.

On the other hand, if you working on complex applications, and you need a solution that’s widely adopted in enterprise environments and has dedicated services in major cloud providers, Kubernetes will be the way to go.

Even though Kubernetes is more complex, it offers unparalleled power and flexibility for large-scale deployments.

This part was originally posted on Medium.

8. CICD Pipeline for your Docker images

Repository structure

You can find the repository here.

This is what the repository structure looks like:

We have a GitHub Actions pipeline called docker-ci-cd.yml, and three images (yes the ones that I’ve started to play around since part 5), each of them with their own Dockerfiles.

Pipeline walkthrough

The worfklow will run only when there are changes to the images, on both pull-requests and merges to the main branch. We have set an environment variable called REGISTRY, in which we specify the GitHub Actions registry.

name: Docker CI/CD Pipeline example

on:
  push:
    branches: ["main"]
    paths:
      - 'images/**'
  pull_request:
    branches: ["main"]
    paths:
      - 'images/**'
env:
  REGISTRY: ghcr.io

In the pipeline, we have three jobs (the last one is just a placeholder that you can build however you’d like):

Detect Changes — This will detect if there are any changes to the images, their code, or even their tests
Build and Push — Builds the image, tests it, runs vulnerability scans, and pushes the images if the branch is main
Deploy — You can modify this job to deploy the containers using whatever orchestrator you want

Let’s start with the Detect Changes job:

detect-changes:
    runs-on: ubuntu-latest
    outputs:
      matrix: ${{ steps.set-matrix.outputs.matrix }}

This job will run on a ubuntu machine, and will declare an output that other jobs will use. This output will be built inside this job in a step called set-matrix.

steps:
    - uses: actions/checkout@v3
      with:
        fetch-depth: 0

First things first, we are checking out the repository, with a fetch-depth of 0, which means all git history is fetched, which is helpful for comparing multiple changes across commits and branches.

- id: set-matrix
  run: |
    if [ "${{ github.event_name }}" == "pull_request" ]; then
      # For pull requests, compare against the base branch
      BASE_SHA=$(git merge-base ${{ github.event.pull_request.base.sha }} ${{ github.sha }})
      CHANGED_DIRS=$(git diff --name-only $BASE_SHA ${{ github.sha }} | grep '^images/' | cut -d/ -f2 | sort -u | jq -R -s -c 'split("\n")[:-1]')
    elif [ "${{ github.event_name }}" == "push" ]; then
      # For pushes, compare against the previous commit
      CHANGED_DIRS=$(git diff --name-only ${{ github.event.before }} ${{ github.sha }} | grep '^images/' | cut -d/ -f2 | sort -u | jq -R -s -c 'split("\n")[:-1]')
    fi

We are checking if the event is either a pull_request or a push, and we will do different things based on the event type.

If the event is a pull_request, we are finding the common ancestor of the pull request’s base branch and the current commit. Based on it, we are running git diff to see all the files and folders that have changed, and we are filtering the results to find only the folders inside the images that have changed. Then with the cut command, we are extracting the subdirectory names within the images folder, and in the end, we are using jq to format the result as JSON.

If the event is a push, we compare the previous commit with the current one, and everything else stays pretty much the same.

If there are no changes, we skip the build with an exit 0 code. Also, at the end, we add our changed directory in the matrix variable and send it to the $GITHUB_OUTPUT variable. This ensures that all the other jobs can access the matrix.

if [ "$CHANGED_DIRS" == "[]" ] || [ -z "$CHANGED_DIRS" ]; then
  echo "No changes detected, skipping build."
  exit 0  # Exit without setting the matrix
fi

echo "matrix=${CHANGED_DIRS}" >> $GITHUB_OUTPUT

The Build and Push job only executes if the detect-changes job detects changes (the matrix output is not empty).

build-and-push:
    needs: detect-changes
    runs-on: ubuntu-latest
    if: ${{ needs.detect-changes.outputs.matrix != '[]' && needs.detect-changes.outputs.matrix != '' }}
    strategy:
      matrix:
        image: ${{fromJson(needs.detect-changes.outputs.matrix)}}
    permissions:
      contents: read
      packages: write
      security-events: write

In the first two steps, we are checking out the repository, and setting up Docker buildx which provides advanced features for building Docker images.

steps:
  - name: Checkout repository
    uses: actions/checkout@v3

  - name: Set up Docker Buildx
    uses: docker/setup-buildx-action@v2

In the next step, we log into the container registry, but only for push events. It uses the GitHub actor as the username and the GitHub token for authentication:

- name: Log in to the Container registry
  if: github.event_name == 'push'
  uses: docker/login-action@v2
  with:
    registry: ${{ env.REGISTRY }}
    username: ${{ github.actor }}
    password: ${{ secrets.GITHUB_TOKEN }}

Now we extract the metadata for the Docker image, including tags and labels — we are generating a tag based on the Git SHA and branch name.

- name: Extract metadata (tags, labels) for Docker
  id: meta
  uses: docker/metadata-action@v4
  with:
    images: ${{ env.REGISTRY }}/${{ github.repository }}/${{ matrix.image }}
    tags: |
      type=sha,prefix={{branch}}-

In the next step, we build the Docker image using the context from the specific image directory. It doesn’t push the image yet but loads it into the Docker daemon:

- name: Build Docker image
  uses: docker/build-push-action@v4
  with:
    context: ./images/${{ matrix.image }}
    push: false
    tags: ${{ steps.meta.outputs.tags }}
    labels: ${{ steps.meta.outputs.labels }}
    cache-from: type=gha
    cache-to: type=gha,mode=max
    load: true

We’ve also included a security vulnerability scanning process based on Trivy, and we are also uploading the security scan result to Security tab.

- name: Run Trivy vulnerability scanner
  uses: aquasecurity/trivy-action@master
  with:
    image-ref: ${{ steps.meta.outputs.tags }}
    format: 'sarif'
    output: 'trivy-results.sarif'
    ignore-unfixed: true
  continue-on-error: true


 - name: Upload Trivy scan results to GitHub Security tab
   uses: github/codeql-action/upload-sarif@v2
   if: always()
   with:
     sarif_file: 'trivy-results.sarif'

As we’ve created a couple of tests for our images, we want to run them before pushing the images, so we are using Google’s container structure tests for that:

- name: Run container structure tests
  run: |
    # Install container-structure-test
    curl -LO https://storage.googleapis.com/container-structure-test/latest/container-structure-test-linux-amd64
    chmod +x container-structure-test-linux-amd64
    sudo mv container-structure-test-linux-amd64 /usr/local/bin/container-structure-test

    # Set the image reference
    IMAGE_REF="${{ steps.meta.outputs.tags }}"

    # Run the tests
    container-structure-test test --image "$IMAGE_REF" --config ./images/${{ matrix.image }}/structure-tests.yaml

  shell: bash

- name: Push Docker image
  if: github.event_name == 'push'
  run: |
    echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u $ --password-stdin
    docker push ${{ steps.meta.outputs.tags }}

In case we have a push event, we are pushing the image to the registry.

The last job, as mentioned before, doesn’t do anything at this moment, but it’s a placeholder for your orchestrator deployment:

deploy:
    needs: build-and-push
    runs-on: ubuntu-latest
    if: github.event_name == 'push' && github.ref == 'refs/heads/main'

    steps:
      - name: Deploy to production
        run: |
          echo "Deploying to production..."

Example Pipeline Run:

Here are the images pushed to the registry:

Also, you can see all the vulnerabilities pushed under the security tab:

There are many vulnerabilities in these images, and I will solve a couple of them for the Python one, just to show you the number decreases:

This is how the Python image looked before:

FROM python:3.9.7-slim
WORKDIR /app
COPY src/* ./
RUN pip install --no-cache-dir -r requirements.txt
CMD ["python", "app.py"]

This is how it looks now:

FROM python:3.11-slim-bullseye
RUN useradd -m -s /bin/bash appuser
WORKDIR /app
COPY src/* ./
RUN pip install --no-cache-dir -r requirements.txt
USER appuser
CMD ["python", "app.py"]

As you can see, I’ve updated the image to use a newer python version, and made it use a non-root user:

The pipeline is running for it, and at the end, we will see that the number of vulnerabilities has decreased considerably:

This part was originally posted on Medium.

9. Docker Optimization and Beyond Docker

Dockerfile Optimization

To create lightweight containers there are a few strategies that you should know:

Use multi-stage builds — Separate build and runtime environments, and only copy necessary artifacts to the final image
Choose lightweight base images — Alpine Linux or Distroless images for even smaller sizes
Minimize the number of layers — Combine RUN commands using & &, and use .dockerignore to exclude unnecessary files
Cleanup step in your layer — Remove package manager caches and temporary files
Leverage build-cache — Order layers from least to most frequently changing
Use specific tags for base images — Avoid the latest tag when you are building containers

Let’s see this in action for two images:

The DevOps development environment image that we’ve built in the second part

flaviuscdinu93/devops-dev-env 1.0.0 c3069a674574 3 weeks ago 351MB

2. A Python image that runs an application:

flaviuscdinu93/python_app 1.0.0 7306a8fe9c7c 50 seconds ago 399MB

This is how the first image looks initially:

FROM alpine:3.20

ENV TERRAFORM_VERSION=1.5.7 \
    OPENTOFU_VERSION=1.8.1 \
    PYTHON_VERSION=3.12.0 \
    KUBECTL_VERSION=1.28.0 \
    ANSIBLE_VERSION=2.15.0 \
    GOLANG_VERSION=1.21.0 \ 
    PIPX_BIN_DIR=/usr/local/bin

RUN apk add --no-cache \
    curl \
    bash \
    git \
    wget \
    unzip \
    make \
    build-base \
    py3-pip \
    pipx \
    openssh-client \
    gnupg \
    libc6-compat

RUN wget -O terraform.zip https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_linux_amd64.zip && \
    unzip terraform.zip && \
    mv terraform /usr/local/bin/ && \
    rm terraform.zip

RUN wget -O tofu.zip https://github.com/opentofu/opentofu/releases/download/v${OPENTOFU_VERSION}/tofu_${OPENTOFU_VERSION}_linux_amd64.zip && \
    unzip tofu.zip && \
    mv tofu /usr/local/bin/ && \
    rm tofu.zip

RUN curl -LO "https://dl.k8s.io/release/v$KUBECTL_VERSION/bin/linux/amd64/kubectl" && \
    chmod +x kubectl && \
    mv kubectl /usr/local/bin/

RUN pipx install ansible-core==${ANSIBLE_VERSION}

WORKDIR /workspace

CMD ["bash"]

For this example, I won’t use multi-stage builds, as I’ll leverage them for the next image. Now, let’s combine all RUN instructions in a single RUN instruction, remove wget (use only curl), and remove the root cache.

FROM alpine:3.20

ENV TERRAFORM_VERSION=1.5.7 \
    OPENTOFU_VERSION=1.8.1 \
    KUBECTL_VERSION=1.28.0 \
    ANSIBLE_VERSION=2.15.0 \
    PIPX_BIN_DIR=/usr/local/bin

WORKDIR /workspace

RUN apk add --no-cache \
    bash \
    curl \
    git \
    unzip \
    openssh-client \
    py3-pip \
    pipx && \
    curl -L -o terraform.zip https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_linux_amd64.zip && \
    unzip terraform.zip && \
    mv terraform /usr/local/bin/ && \
    rm terraform.zip && \
    curl -L -o opentofu.zip https://github.com/opentofu/opentofu/releases/download/v${OPENTOFU_VERSION}/tofu_${OPENTOFU_VERSION}_linux_amd64.zip && \
    unzip opentofu.zip && \
    mv tofu /usr/local/bin/ && \
    rm opentofu.zip && \
    curl -LO "https://dl.k8s.io/release/v${KUBECTL_VERSION}/bin/linux/amd64/kubectl" && \
    chmod +x kubectl && \
    mv kubectl /usr/local/bin/ && \
    pipx install ansible-core==${ANSIBLE_VERSION} && \
    rm -rf /root/.cache && \

CMD ["bash"]

I’ve built the image and used a 1.0.1 tag:

flaviuscdinu93/devops-dev-env 1.0.1 8d1344dd324c 5 seconds ago 340MB

As you can see, the optimization isn’t that big, we’ve reduced just 11MB.

Now, let’s leverage a multi-stage build for the second image. The image initially looks like this:

FROM ubuntu:20.04

RUN apt-get update && \
    apt-get install -y python3 python3-pip

WORKDIR /app

COPY backend/ /app

RUN pip3 install -r requirements.txt

EXPOSE 5000

CMD ["python3", "app.py"]

I will switch the image from ubuntu to alpine and leverage a multi stage build:

FROM python:3.9-alpine as builder

WORKDIR /app

RUN apk add --no-cache build-base

COPY backend/requirements.txt .

RUN pip install --no-cache-dir --user -r requirements.txt

COPY backend/app.py .

FROM python:3.9-alpine

WORKDIR /app

COPY --from=builder /app/app.py .
COPY --from=builder /root/.local /root/.local

ENV PATH=/root/.local/bin:$PATH

EXPOSE 5000

CMD ["python", "app.py"]

After building the image, this is the result:

flaviuscdinu93/python_app 1.0.0-alpine 41243d14a523 1 second ago 59.6MB

We’ve reduced the initial image from almost 400MB to 60MB.

Docker deployment strategies

There are multiple deployment strategies that can be used to deploy your containers, and it all depends on what other tools you are using in your ecosystem.

1. Single-container deployment

In some special cases, deploying a single Docker container might be enough. What you’ll need to do is package your application and its dependencies into a Docker image and run it as a container on a machine.

Best Practices:

Use official base images — Start with minimal and secure base images from trusted sources
Optimize the Dockerfile — You should minimize the number of layers and remove unnecessary files to reduce the image size
Environment variables — Leverage environment variables for configuration to maintain portability across environments.

2. Multi-Container deployment with Docker Compose

Docker Compose can be easily used for defining and running multi-container Docker applications. As you already know, you can easily manage multiple services using a single docker-compose.yml file. If you haven’t seen the docker-compose article, take a look here.

Best Practices:

Service isolation — You should define each service in its own container to improve modularity
Networking — Use Docker network features to enable communication between containers (create the networks to fit your needs). Don’t rembember how? Take a look here.
Volumes — Remember the Nginx + NodeJS + Flask + Redis example built here? We’ve used volumes for data persistence to prevent data loss when recreating containers

3. Container orchestration with Kubernetes or Docker Swarm

For complex applications requiring scalability and high availability, leverage Kubernetes or Docker Swarm.

If you don’t remember what you can do with Kubernetes or Docker Swarm take a look at the K8s and Swarm part.

4. Use CI/CD pipeline

Integrating Docker into your CI/CD pipeline automates the building, testing, and deployment of applications. They will make your life easier. Check out an example of a GitHub Actions CI/CD pipeline here.

Best Practices:

Automated builds — Use CI/CD tools like Jenkins, GitLab CI/CD, or GitHub Actions to automate Docker image builds
Testing — Implement automated testing at each stage to catch issues early
Versioning and tagging — Tag Docker images with meaningful version numbers or commit hashes for traceability

5. Blue-Green Deployments

Blue-green deployment involves running two identical production environments. One of them (blue) is live and serves all production traffic, while the other one (green) is idle. When deploying a new version, you switch the traffic from blue to green after ensuring that everything works as planned. Then you can decide what you want to do with the blue environment (decommission or repurpose)

Advantages:

Zero downtime — Users experience no downtime during deployment
Easy rollbacks — You can revert traffic to the previous version quickly if there are any issues with your deployments

6. Canary Deployments

Canary deployment is a strategy to roll out the new version to a small subset of users before a full release. This minimizes risk by limiting exposure to potential issues.

Best Practices:

Monitoring — Closely monitor the performance and errors in the canary deployment
Gradual rollout — Increase the percentage of traffic to the new version incrementally

7. Rolling Updates

Rolling updates replace instances of your application one at a time with the new version. This ensures that a few cases are always serving requests, providing continuous availability.

Depending on what tools you are using in your ecosystem to deploy your Docker containers, you will most likely have to check out the best practices on how to implement the best deployment strategies.

Beyond Docker

Podman

Podman is a daemon-less container engine developed by Red Hat that allows you to run and manage containers without the need for a central daemon (unlike the Docker engine). It adheres to the Open Container Initiative (OCI) standards, ensuring compatibility with Docker images and container registries.

Key Features

Daemon-less architecture — You won’t need root-running daemon, enhancing security
Rootless containers — Enables running containers without root privileges
Docker-compatible CLI — Supports many Docker commands, making it easy to transition

Building Images with Podman

Building images with Podman is nearly identical to Docker, using the same Dockerfile syntax.

podman build -t myimage:latest -f Dockerfile .

LXC

LXC is an operating-system-level virtualization method for running multiple isolated Linux systems (containers) on a single host. It provides a lightweight alternative to full-machine virtualization.

Key Features

System containers — Offers containers that behave like complete Linux systems
Low overhead — Utilizes kernel features for isolation, ensuring minimal performance overhead
customization : You can easily customize the container environment

Building Images with LXC

LXC doesn’t use images in the same way Docker does, and it relies on templates to create containers.

Creating an LXC Container

sudo lxc-create -n mycontainer -t ubuntu

CRI-O

CRI-O is an open-source container runtime for Kubernetes that implements the Kubernetes Container Runtime Interface (CRI) using OCI-compliant runtimes. It aims to provide a lightweight alternative to Docker for Kubernetes environments.

Key Features

Kubernetes integration — It is specifically designed for K8s
Lightweight runtime — Reduces overhead by removing unnecessary components.
Security enhancements — Supports advanced security features like SELinux.

Building Images with CRI-O

CRI-O focuses solely on running containers and does not include image-building capabilities. To build images, you can use Buildah , another tool from the same ecosystem.

Building Images with Buildah

buildah bud -t myimage:latest .

buildah push myimage:latest docker://registry.example.com/myimage:latest

This part was originally posted on Medium.

Hope it helps!

Keep up building!

Terraform from 0 to Hero

Flavius Dinu — Tue, 04 Mar 2025 18:28:49 +0000

Terraform from 0 to Hero

This entire series was originally posted on Medium.

Table of contents:

1. What is Terraform?
2. Terraform Provider
3. Terraform Resources
4. Terraform Data Sources and Outputs
5. Terraform Variables and Locals
6. Terraform Provisioners and Null resources
7. Terraform Loops and Conditionals
8. Terraform CLI commands
9. Terraform functions
10. Working with files
11. Understanding Terraform state
12. Terraform depends_on and lifecycle block
13. Terraform Dynamic blocks
14. Terraform Modules
15. Best practices for modules I
16. Best practices for modules II
17. Bonus1: OpenTofu differences
18. Bonus2: Specialized Infrastructure Orchestration platforms

1. What is Terraform?

Once upon a time, if you worked in the IT industry, there was a big chance you faced different challenges when provisioning or managing infrastructure resources. It often felt like spinning plates, trying to keep everything running smoothly and making sure that all the resources are properly configured.

Then, Terraform came to the rescue and saved us from this daunting task that took a lot of time.

So what is Terraform? Terraform started as an open-source infrastructure as code (IaC) tool, developed by Hashicorp, that makes it easy to create and take care of your infrastructure resources. Now, it changed it license to BSL. If you want to learn more about the license change, and how it compares to OpenTofu checkout this article.

It’s built in Golang (Go), which gives it a lot of power to create different infrastructure pieces in parallel, making it reliable by taking advantage of Go’s strong type-checking and error-handling capabilities.

Terraform uses HCL (Hashicorp Configuration Language) code to define its resources, but even JSON can be used for this, if you, of course, hate your life for whatever reason. Let’s get back to HCL. It is a human-readable , high-level programming language that is designed to be easy to use and understand:

resource "resource_type" "resource_name" {
  param1 = "value1"
  param2 = "value2"
  param3 = "value3"
  param4 = "value4"
}

resource "cat" "british" {
  color           = "orange"
  name            = "Garfield"
  age             = 5
  food_preference = ["Tuna", "Chicken", "Beef"]
}

I don’t want to get into too much detail about what a resource is in this article, as I plan to build a series around this, but the above code, with a pseudo real-life example, is pretty much self-explanatory.

To make it as simple as possible for now to understand, when you are using HCL and declaring something, it will have a type (let’s suppose there is a resource type cat, for example) and a name on the first line. Inside the curly brackets, you are going to specify how you want to configure that type of “something”.

In our example, we will create a “cat”, that will be named inside of terraform as “british”. After that, we are configuring the cat’s real name, the one that everyone will know about, the color, the age, and what it likes to eat.

As you see, the language, at a first glance seems to be pretty close to English. There is more to it, of course, but you are going to see it in the next articles.

One of the main benefits of using Terraform is that it is platform-agnostic. This means that people that are coding in Terraform, don’t need to learn different programming languages to provision infrastructure resources in different cloud providers. However, this doesn’t mean that if you develop the code to provision a VM instance in AWS, you can use the same one for Azure or GCP.

Nevertheless, this can save a lot of time and effort, as engineers won’t need to constantly switch between a lot of tools (like going from Cloudformation for AWS to ARM templates for Azure).

A consistent experience is offered across all platforms.

Terraform is stateful. Is this a strength or is this a weakness? This topic is highly subjective and it depends on your use case.

One of the main benefits of statefulness in Terraform is that it allows it to make decisions about how to manage resources based on the current state of the infrastructure. This ensures that Terraform does not create unnecessary resources and helps to prevent errors and conflicts. This can save time and resources, make the provisioning process more efficient and also encourage collaboration between different teams.

Terraform keeps its state in a state file and uses it and your configuration, to determine the actions that need to be taken to reach the desired state of the resources.

Even though I’ve presented only strengths until now, being stateful has a relatively big weakness: the managing of the state file. This adds complexity to the system and also in the case that the state file gets corrupted or deleted, it can lead to conflicts and errors in the provisioning process.

We will tackle this subject in detail in the next articles.

You may now ask, how can I easily install it? Well, Hashicorp’s guide does a great job of helping you install it, just select your platform and you are good to go.

Ok, now you have a high-level idea about what Terraform is, how to install it, but it’s ok if you still have a lot of questions, as all the answers will come shortly.

Originally posted on Medium.

2. What is a Terraform provider?

In short, Terraform providers are plugins that allow Terraform to interact with specific infrastructure resources. They act as an interface between Terraform and the underlying infrastructure, translating the Terraform configuration into the appropriate API calls and allowing Terraform to manage resources across a wide variety of environments. Each provider has its own set of resources and data sources that can be managed and provisioned using Terraform.

One of the most common mistakes that people make when they are thinking about Terraform providers, is the fact that they assume that Terraform providers exist only for Cloud Vendors such as Amazon Web Services (AWS), Microsoft Azure, Google Cloud Platform (GCP), or Oracle Cloud Infrastructure (OCI). There are a lot of other providers that can be used that don’t belong to a Cloud Vendor as Template, Kubernetes, Helm, Spacelift, Artifactory, VSphere, and Aviatrix, to name a few.

Each provider has its own set of resources and data sources that can be managed and provisioned using Terraform. For example, the AWS provider has resources for managing EC2 instances, EBS volumes, and ELB load balancers.

Another great thing related to this feature is the fact that you can build your own Terraform provider. While it has an API, you can translate it to Terraform, so that’s just great. However, there are thousands of providers already available in the registry, so you don’t need to reinvent the wheel.

Before jumping in and showing you examples of how to use providers, let’s discuss about how to use the documentation.

After you are selecting your provider from the registry, you will be redirected to the provider’s page.

In the above view, click on documentation:

Usually, the first tab you are directed to will explain how to configure and use the provider and some simple examples of how to create some simple resources. Of course, this view is different from one provider to another, but usually, you will see these examples.

We will get back to how to use the documentation when we will talk about resources in the next article.

Even though AWS is the biggest player in the market when it comes to cloud vendors, in this article I will show an example provider with OCI and one with Azure.

You can choose your own, of course, by going to the registry and selecting whatever suits you.

OCI

You have multiple ways to connect to the OCI provider as stated in the documentation, but I will only discuss about the default one:

Based on your tenancy and your user will have to specify the following details:

tenancy_ocid
user_ocid
private_key or private_key_path
private_key_password (optional, only required if your password is encrypted)
fingerprint
region

  provider "oci" {
      tenancy_ocid     = "tenancy_ocid"
      user_ocid        = "user_ocid"
      fingerprint      = "fingerprint"
      private_key_path = "private_key_path"
      region           = "region"
    }

After you specify the correct values for all of the values mentioned above you can interact with your Oracle Cloud Infrastructure’s cloud account.

You will be able to define resources and datasources to create and configure different pieces of infrastructure.

AZURE

Similar to the OCI provider, the Azure provider, can be configured in multiple ways:

I will discuss the Azure CLI option which is the easiest way to authenticate by leveraging the az login command.

    provider "azurerm" {
      features {}
    }

This is the only configuration you have to do in the Terraform code and after running az login and following the prompt you are good to go.

Originally posted on Medium.

3. What is a Terraform Resource?

Resources in Terraform refer to the components of infrastructure that Terraform is able to manage, such as virtual machines, virtual networks, DNS entries, pods, and others. Each resource is defined by a type, such as “aws_instance” or “google_dns_record”, “kubernetes_pod”, “oci_core_vcn”, and has a set of configurable properties, such as the instance size, vcn cidr, etc. Remember the cat example from the first article in this series.

Terraform can be used to create, update, and delete resources, managing dependencies between them and ensuring they are created in the correct order. You can also create explicit dependencies between some of the resources if you would like to do that by using depends_on

Let’s go in-depth and try to understand how to create these resources and how can we leverage the documentation.

I will start with something simple, an aws_vpc

First things first, whenever you are creating a resource, you will need to go to the documentation. It is really important to understand what you can do for a particular resource and I believe that you should try to build a habit around this.

On the right-hand side, you have the On this page space, with 4 elements that you should know like you know to count:

Example Usage → this will show you a couple of examples of how to use the resource
Argument Reference → in this section you are going to see all the parameters that can be configured for a resource. Some parameters are mandatory and others are optional (these parameters will be signaled with an Optional placed between brackets)
Attributes Reference → here you will find out what a resource exposes and I will talk about this in more detail when I get to the Outputs article
Import → Until now, if you didn’t know anything about Terraform and you’ve just read my articles, you are possibly thinking that it’s possible to import different resources in the state and that is correct. In this section, you are going to find out how you can import that particular resource type

So let’s create a VPC. First, we need to define the provider as specified in the last article, the only caveat now, is that we are doing it for a different cloud provider. If you forgot how to do it just reference the previous article as it has two examples for Azure and OCI and you easily do it for AWS.

Nevertheless, I will show you an option for AWS too using the credentials file that is leveraged by aws cli also. The provider will automatically read the AWS_ACCESS_KEY and AWS_SECRET_ACCESS_KEY from the ~/.aws/credentials, so make sure you have that configured. An example can be found here.

We are then going to add the vpc configuration. Everything should be saved in a .tf file.

    provider "aws" {
      region = "us-east-1"
    }

    resource "aws_vpc" "example" {
      cidr_block = "10.0.0.0/16"
    }

As we see in the documentation, all the parameters for the vpc are optional, as AWS will assign everything that you don’t specify for you.

Now it is time to run the code. For this, we are going to learn some terraform essential commands and I’m going just to touch upon the basics of these commands:

terraform init → Initializes a working directory with Terraform configuration files. This should be the first command executed after creating a new Terraform configuration or cloning an existing one from version control. It also downloads the specified provider and modules if you are using any and saves them in a generated .terraform directory.
terraform plan → generates an execution plan, allowing you to preview the changes Terraform intends to make to your infrastructure.
terraform apply → executes the actions proposed in a Terraform plan. If you don’t provide it a plan file, it will generate an execution plan when you are running the command, as if terraform plan ran. This prompts for your input so don’t be scared to run it.
terraform destroy → is a simple way to delete all remote objects managed by a specific Terraform configuration. This prompts for your input so don’t be scared to run it.

We will get into more details when we are going to tackle all Terraform commands during this series.

Ok, now that you know the basics, let’s run the code.

Go to the directory where you’ve created your terraform file with the above configuration and run terraform init

_Initializing the backend...  

Initializing provider plugins...  
\- Finding latest version of hashicorp/aws...  
\- Installing hashicorp/aws v4.50.0...  
\- Installed hashicorp/aws v4.50.0 (signed by HashiCorp)  

Terraform has created a lock file .terraform.lock.hcl to record the provider  
selections it made above. Include this file in your version control repository  
so that Terraform can guarantee to make the same selections by default when  
you run "terraform init" in the future.  

Terraform has been successfully initialized!  

You may now begin working with Terraform. Try running "terraform plan" to see  
any changes that are required for your infrastructure. All Terraform commands  
should now work.  

If you ever set or change modules or backend configuration for Terraform,  
rerun this command to reinitialize your working directory. If you forget, other  
commands will detect it and remind you to do so if necessary._

After that, let’s run terraform plan to see what is going to happen:

_Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:  
\+ create  

Terraform will perform the following actions:  
_  
# aws __vpc.example will be created  
\+ resource "aws__vpc" "example" {  
\+ arn = (known after apply)  
\+ cidr __block = "10.0.0.0/16"  
\+ default__network __acl__ id = (known after apply)  
\+ default __route__ table __id = (known after apply)  
\+ default__security __group__ id = (known after apply)  
\+ dhcp __options__ id = (known after apply)  
\+ enable __classiclink = (known after apply)  
\+ enable__classiclink __dns__ support = (known after apply)  
\+ enable __dns__ hostnames = (known after apply)  
\+ enable __dns__ support = true  
\+ enable __network__ address __usage__ metrics = (known after apply)  
\+ id = (known after apply)  
\+ instance __tenancy = "default"  
\+ ipv6__association __id = (known after apply)  
\+ ipv6__cidr __block = (known after apply)  
\+ ipv6__cidr __block__ network __border__ group = (known after apply)  
\+ main __route__ table __id = (known after apply)  
\+ owner__id = (known after apply)  
\+ tags __all = (known after apply)  
}  

Plan: 1 to add, 0 to change, 0 to destroy.  

───────────────────────────────────────────────────────────────────  

Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now._

In the plan, we see that we are going to create one vpc with the above configuration. You can observe that the majority of the parameters will be known after apply, but the cidr block is the one that we’ve specified.

Let’s apply the code and create the vpc with terraform apply:

 _Terraform will perform the following actions:  

# aws_vpc.example will be created  
\+ resource "aws_vpc" "example" {  
\+ arn = (known after apply)  
\+ cidr_block = "10.0.0.0/16"  
\+ default_network_acl_id = (known after apply)  
\+ default_route_table_id = (known after apply)  
\+ default_security_group_id = (known after apply)  
\+ dhcp_options_id = (known after apply)  
\+ enable_classiclink = (known after apply)  
\+ enable_classiclink_dns_support = (known after apply)  
\+ enable_dns_hostnames = (known after apply)  
\+ enable_dns_support = true  
\+ enable_network_address_usage_metrics = (known after apply)  
\+ id = (known after apply)  
\+ instance_tenancy = "default"  
\+ ipv6_association_id = (known after apply)  
\+ ipv6_cidr_block = (known after apply)  
\+ ipv6_cidr_block_network_border_group = (known after apply)  
\+ main_route_table_id = (known after apply)  
\+ owner_id = (known after apply)  
\+ tags_all = (known after apply)  
}  

Plan: 1 to add, 0 to change, 0 to destroy.  

Do you want to perform these actions?  
Terraform will perform the actions described above.  
Only 'yes' will be accepted to approve.  

Enter a value:_

Enter yes at the prompt and you will be good to go.

 _Enter a value: yes  

aws_vpc.example: Creating...  
aws_vpc.example: Creation complete after 3s [id=vpc-some_id]  

Apply complete! Resources: 1 added, 0 changed, 0 destroyed._

Woohoo, we have created a vpc using Terraform. You can then go into the AWS console and see it in the region you’ve specified for your provider.

Very nice and easy, I would say, but before destroying the vpc, let’s see how we can create a resource that references this existing vpc.

An AWS internet gateway exists only inside of a vpc. So let’s go and check the documentation for the internet gateway.

From the documentation, we see in the example directly that we can reference a vpc id for creating the internet gateway:


    resource "aws_internet_gateway" "gw" {
      vpc_id = ""
    }

But the 100-point question is, how can we reference the above-created vpc? Well, that is not very hard if you remember the following:

type.name.attribute

All resources have a type, a name, and some attributes they expose. The exposed attributes are part of the Attributes reference section in the provider. Some providers will explicitly mention they are exposing everything from Argument reference + Attribute reference.

Let’s take our vpc as an example, its type is aws_vpc , its name is example and it exposes a bunch of things (remember, the documentation is your best friend).

So, as the internet gateway requires a vpc_id and we want to reference our existing one, our code will look like this in the end:


    provider "aws" {
      region = "us-east-1"
    }

    resource "aws_vpc" "example" {
      cidr_block = "10.0.0.0/16"
    }

    resource "aws_internet_gateway" "gw" {
      vpc_id = aws_vpc.example.id
    }

We can then reapply the code with terraform applyand terraform will simply compare what we have already created with what we have in our configuration and create only the internet gateway.

aws __vpc.example: Refreshing state... [id=vpc-some__ id]  

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:  
\+ create  

Terraform will perform the following actions:  

_# aws_internet_gateway.gw will be created  
\+ resource "aws_internet_gateway" "gw" {  
\+ arn = (known after apply)  
\+ id = (known after apply)  
\+ owner_id = (known after apply)  
\+ tags_all = (known after apply)  
\+ vpc_id = "vpc-some_id"  
}  

Plan: 1 to add, 0 to change, 0 to destroy.  

Do you want to perform these actions?  
Terraform will perform the actions described above.  
Only 'yes' will be accepted to approve.  

Enter a value: yes  

aws_internet_gateway.gw: Creating...  
aws_internet_gateway.gw: Creation complete after 2s [id=igw-some_igw]  

Apply complete! Resources: 1 added, 0 changed, 0 destroyed._

Pretty neat, right?

Once we are done with our infrastructure, we can destroy it, using terraform destroy

_aws_vpc.example: Refreshing state... [id=vpc-some_vpc]  
aws_internet_gateway.gw: Refreshing state... [id=igw-some_igw]  

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:  
\- destroy  

Terraform will perform the following actions:  

# aws_internet_gateway.gw will be destroyed  
\- resource "aws_internet_gateway" "gw" {  
\- all parameters are specified here  
}  

# aws_vpc.example will be destroyed  
\- resource "aws_vpc" "example" {  
\- all parameters are specified here  
}  

Plan: 0 to add, 0 to change, 2 to destroy.  

Do you really want to destroy all resources?  
Terraform will destroy all your managed infrastructure, as shown above.  
There is no undo. Only 'yes' will be accepted to confirm.  

Enter a value: yes  

aws_internet_gateway.gw: Destroying... [id=igw-some_igw]  
aws_internet_gateway.gw: Destruction complete after 2s  
aws_vpc.example: Destroying... [id=vpc-some_vpc]  
aws_vpc.example: Destruction complete after 0s  

Destroy complete! Resources: 2 destroyed._

It can be a little overwhelming, but bear with me and understand the key points:

A resource is a component that can be managed with Terraform (a VM, a Kubernetes Pod, etc)
Documentation is your best friend, understand how the use those 4 sections from it
There are 4 essential commands that help you provision and destroy your infrastructure: init/plan/apply/destroy
When referencing a resource from a configuration we are using type.name.attribute

Originally posted on Medium.

4. Data Sources and Outputs

Terraform resources are great and you can do a bunch of stuff with them. But did I tell you can use Data Sources and Outputs in conjunction with them to better implement your use case? Let’s jump into it.

To put it as simply as possible, A data source is a configuration object that retrieves data from an external source and can be used in resources as arguments when they are created or updated. When I am talking about an external source, I am referring to absolutely anything: manually created infrastructure, resources created from other terraform configurations, and others.

Data sources are defined in their respective providers and you can use them with a special block called data. The documentation of a data source is pretty similar to one of a resource, so if you’ve mastered how to use that one, this will be a piece of cake.

Let’s take an example of a data source that returns the most recent ami_id (image id) of an Ubuntu image in AWS.

    provider "aws" {
      region = "us-east-1"
    }

    data "aws_ami" "ubuntu" {
      filter {
        name   = "name"
        values = ["ubuntu-*"]
      }
      most_recent = true
    }

    output "ubuntu" {
      value = data.aws_ami.ubuntu.id
    }

I am putting a filter on the image name and I’m specifying all the image names that start with ubuntu-. I’m adding the most_recent = true to get only one image, as the aws_ami data source doesn’t support returning multiple image ids. So this data source will return only the most recent Ubuntu ami.

In the code example, there is also one reference to an output, but I haven’t exactly told you what an output is, did I?

An output is a way to easily view the value of a specific data source, resource, local, or variable after Terraform has finished applying changes to infrastructure. It can be defined in the Terraform configuration file and can be viewed using the terraform output command, but just to reiterate, only after a terraform apply happens. Outputs can be also used to expose different resources inside a module, but we will discuss this in another post.

Outputs don’t depend on a provider at all, they are a special kind of block that works independently from them.

The most important parameters an output supports are value , for which you specify what you want to see, and description (optional) in which you explain what that output wants to achieve.

Let’s go back to our example.

In the output, I have specified a reference to the above data source. When we are referencing a resource, we are using type.name.attribute, for data sources, it’s pretty much the same but we have to prefix it with data , so data.type.name.attribute will do the trick.

As I mentioned above, in order to see what this returns, you will first have to apply the code. You are not going to see the contents of a data source without an output, so I encourage you to use them at the beginning when you are trying to get familiar with them.

This is the output of a terraform apply:

_data.aws_ami.ubuntu: Reading...

data.aws_ami.ubuntu: Read complete after 3s [id=ami-0f388924d43083179]

Changes to Outputs:

+ ubuntu = "ami-0f388924d43083179"

You can apply this plan to save these new output values to the Terraform state, without changing any real infrastructure.

Do you want to perform these actions?

Terraform will perform the actions described above.

Only 'yes' will be accepted to approve.

Enter a value: yes

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

ubuntu = "ami-0f388924d43083179"_

In an apply, the first state a resource goes through is creating , a data source is going through a reading state, just to make the differences between them clearer.

Now, let’s use this image to create an ec2 instance:

provider "aws" {
  region = "us-east-1"
}

data "aws_ami" "ubuntu" {
  filter {
    name   = "name"
    values = ["ubuntu-*"]
  }
  most_recent = true
}

output "ubuntu" {
  value = data.aws_ami.ubuntu.id
}

resource "aws_instance" "web" {
  ami           = data.aws_ami.ubuntu.id
  instance_type = "t2.micro"
}

Just by referencing the ami id from our data source and an instance type, we are able to create an ec2 instance with a terraform apply

Don’t forget to delete your infrastructure if you are practicing, as everything you create will incur costs. Do that with a terraform destroy.

Originally posted on Medium.

5. Terraform Variables and Locals

Terraform variables and locals are used to better organize your code, easily change it, and make your configuration reusable.

Before jumping into variables and locals in Terraform, let’s first discuss their supported types.

Usually, in any programming language, when we are defining a variable or a constant, we are assigning it, or it infers a type.

Supported types in Terraform:

Primitive:

String
Number
Bool

Complex — These types are created from other types :

List
Set
Map
Object

Null — Usually represents absence, really useful in conditional expressions.

There is also the any type, in which you basically add whatever you want without caring about the type, but I really don’t recommend it as it will make your code harder to maintain.

Variables

Every variable will be declared with a variable block and we will always use it with var.variable_name. Let’s see this in action:

resource "aws_instance" "web" {
  ami           = data.aws_ami.ubuntu.id
  instance_type = var.instance_type
}

variable "instance_type" {
  description = "Instance Type of the variable"
  type        = string
  default     = "t2.micro"
}

I have declared a variable called instance_type and in it, I’ve added 3 fields, all of which are optional, but usually, it is a best practice to add these, or at least the type and description. Well, there are three other possible arguments (sensitive, validation, and nullable), but let’s not get too overwhelmed by this.

In the resource block above, I’m referencing the variable, with var.instance_type and due to the fact I’ve set the default value to t2.micro, my variable will get that particular value and I don’t need to do anything else. Cool, right?

Well, let’s suppose we are not providing any default value and we are not doing anything else and we run a terraform apply. As Terraform does not know the value of the variable, it will ask you to provide a value for it. Pretty neat, that means you can forget to assign it. This is not a best practice, though.

There are a couple of other ways you can assign values to variables. If you happen to specify a value for a variable in multiple ways, Terraform will use the last value it finds, by taking into consideration their precedence order. I’m going to present these to you now:

using a default → as in the example above, this will be overwritten by any other option
using a terraform.tfvarsfile → this is a special file in which you can add values to your variables

instance_type = "t2.micro"
using a *.auto.tfvars file → similar to the terraform.tfvars file, but will take precedence over it. The variables' values will be declared in the same way. The “*” is a placeholder for any name you want to use
using -var or -var-file when running terraform plan/apply/destroy. When you are using both of them in the same command, the value will be taken from the last option.

terraform apply -var="instance_type=t3.micro" -var-file="terraform.tfvars"→ This will take the value from the var file, but if we specify the -var option last, it will get the value from there.

Some other variables examples:

variable "my_number" {
  description = "Number example"
  type        = number
  default     = 10
}

variable "my_bool" {
  description = "Bool example"
  type        = bool
  default     = false
}

variable "my_list_of_strings" {
  description = "List of strings example"
  type        = list(string)
  default     = ["string1", "string2", "string3"]
}

variable "my_map_of_strings" {
  description = "Map of strings example"
  type        = map(string)
  default = {
    key1 = "value1"
    key2 = "value2"
    key3 = "value"
  }
}

variable "my_object" {
  description = "Object example"
  type = object({
    parameter1 = string
    parameter2 = number
    parameter3 = list(number)
  })
  default = {
    parameter1 = "value"
    parameter2 = 1
    parameter3 = [1, 2, 3]
  }
}

variable "my_map_of_objects" {
  description = "Map(object) example"
  type = map(object({
    parameter1 = string
    parameter2 = bool
    parameter3 = map(string)
  }))
  default = {
    elem1 = {
      parameter1 = "value"
      parameter2 = false
      parameter3 = {
        key1 = "value1"
      }
    }
    elem2 = {
      parameter1 = "another_value"
      parameter2 = true
      parameter3 = {
        key2 = "value2"
      }
    }
  }
}

variable "my_list_of_objects" {
  description = "List(object) example"
  type = list(object({
    parameter1 = string
    parameter2 = bool
  }))
  default = [
    {
      parameter1 = "value"
      parameter2 = false
    },
    {
      parameter1 = "value2"
      parameter2 = true
    }
  ]
}

In the above example, there are two variables of a simple type (number and bool), and a couple of complex types. As the simple ones are pretty easy to understand, let’s jump into the others.

list(string) — in this variable, you can declare how many strings you want inside the list. You are going to access an instance of the list by using var.my_list_of_strings[index]. Keep in mind that lists start from 0. var.my_list_of_strings[1] will return string2.

map(string) — in this variable, you can declare how many key:value pairs you want. You are going to access an instance of the map by using var.my_map_of_strings[key] where key is on the left-hand side from the equal sign. var.my_map_of_strings["key3"] will return value

object({}) — inside of an object, you are declaring parameters as you see fit. You can have simple types inside of it and even complex types and you can declare as many as you want. You can consider an object to be a map having more explicit types defined for the keys. You are going to access instances of an object, by using the same logic as you would for a map.

map(object({})) — I’ve specified this complex build, because this is something I am using a lot inside of my code because it works well with for_each (don’t worry, we will talk about this in another post). You are going to access a property of an object in the map by using var.my_map_of_objects["key"]["parameter"] and if there are any other complex parameters defined you will have to go deeper. var.my_map_of_objects["elem1"]["parameter1"] will return value. var.my_map_of_objects["elem1"]["parameter3"]["key1"] will return value1.

list(object({})) — This is something I’m using in dynamic blocks(again, we will discuss this in detail in another post). You are going to access a property of an object in the list, by using var.my_list_of_objects[index]["parameter"]. Again, if there are any parameters that are complex, you will have to go deeper. var.my_list_of_objects[0]["parameter1"] will return value.

One important thing to note is the fact that you cannot reference other resources or data sources inside a variable, so you cannot say that a variable is equal to a resource attribute by using the type.name.attribute.

Locals

On the other hand, a local variable assigns a name to an expression, making it easier for you to reference it, without having to write that expression a gazillion times. They are defined in a locals block, and you can have multiple local variables defined in a single local block. Let’s take a look:

locals {
  instance_type = "t2.micro"
  most_recent   = true
}

data "aws_ami" "ubuntu" {
  filter {
    name   = "name"
    values = ["ubuntu-*"]
  }
  most_recent = local.most_recent
}

resource "aws_instance" "web" {
  ami           = data.aws_ami.ubuntu.id
  instance_type = local.instance_type
}

As you see, we are defining inside the locals block two local variables and we are referencing them throughout our configuration with local.local_variable_name.

As opposed to variables, inside of a local, you can define whatever resource or data source attribute you want. We can even define more complex operations inside of them, but for now, let’s just let this sync in as we are going to experiment with these some more in the future.

Originally posted on Medium.

6. Terraform Provisioners and Null Resource

Terraform provisioners have nothing in common with providers. You can use provisioners to run different commands or scripts on your local machine or a remote machine, and also copy files from your local machine to a remote one. Provisioners, exist inside of a resource, so in order to use one, you will simply have to add a provisioner block in that particular resource.

One thing worth mentioning is the fact that a provisioner is not able to reference the parent resource by its name, but they can use the self object which actually represents that resource.

They are considered a last resort, as they are not a part of the Terraform declarative model.

There are 3 types of provisioners:

local-exec
file (should be used in conjunction with a connection block)
remote-exec (should be used in conjunction with a connection block)

All provisioners support two interesting options when and on_failure.

You can run provisioners either when the resource is created (which is, of course, the default option) or if your use case asks for it, run it when a resource is destroyed.

By default, on_failure is set to fail, which will fail the apply if the provisioner fails, which is expected Terraform behaviour, but you can set it to ignore a fail by setting it to continue.

From experience, I can tell you that sometimes provisioners fail for no reason, or they can even appear to be working and not doing what they are expected to. Still, I believe it is still very important to know how to use them, because, in some of your use cases, you may not have any alternatives.

Before jumping into each of the provisioners, let’s talk about null resources. A null resource is basically something that doesn’t create anything on its own, but you can use it to define provisioners blocks. They also have a “trigger” attribute, which can be used to recreate the resource, hence to rerun the provisioner block if the trigger is hit.

Local-Exec

As its name suggests, a local-exec block is going to run a script on your local machine. Nothing too fancy about it. Apart from the when and on_failure options, there are a couple of other options you can specify:

command — what to run; this is the only required argument.
working_dir — where to run it
interpreter — what interpreter to use (e.g /bin/bash), by default terraform will decide based on your system os
environment — key/value pairs that represent the environment

Let’s see this in action in a null resource and observe the output of a terraform apply

resource "null_resource" "this" {
  provisioner "local-exec" {
    command = "echo Hello World!"
  }
}

# null_resource.this: Creating...
# null_resource.this: Provisioning with 'local-exec'...
# null_resource.this (local-exec): Executing: ["/bin/sh" "-c" "echo Hello World!"]
# null_resource.this (local-exec): Hello World!
# null_resource.this: Creation complete after 0s [id=someid]

You can use this to run different scripts before or after an apply of a specific resource by using depends_on (we will talk about this in another article in more detail).

Connection Block

Before going into the other two provisioners, remote-exec and file, let’s take some time and understand the connection block. In order to run or copy something on a remote vm, you will first have to connect to it, right?

Connection blocks, support both ssh and winrm, so you can easily connect to both your Linux and Windows vms.

You even have the option to connect via a bastion host or a proxy, but I will just show you a simple connection block for a Linux VM.

connection {
    type        = "ssh"
    user        = "root"
    private_key = "private_key_contents"
    host        = "host"
}

File

The file provisioner is used to copy a file from your local vm to a remote vm. There are three arguments that are supported:

source (what file to copy)
content (the direct content to copy on the destination)
destination (where to put the file)

As mentioned before, file needs a connection block to make sure it works properly. Let’s see an example on an ec2 instance.

provider "aws" {
  region = "us-east-1"
}

locals {
  instance_type = "t2.micro"
  most_recent   = true
}

data "aws_ami" "ubuntu" {
  filter {
    name   = "name"
    values = ["ubuntu-*"]
  }
  most_recent = local.most_recent
}

resource "aws_key_pair" "this" {
  key_name   = "key"
  public_key = file("~/.ssh/id_rsa.pub")
}

resource "aws_instance" "web" {
  ami           = data.aws_ami.ubuntu.id
  instance_type = local.instance_type
  key_name      = aws_key_pair.this.key_name
}

resource "null_resource" "copy_file_on_vm" {
  depends_on = [
    aws_instance.web
  ]
  connection {
    type        = "ssh"
    user        = "ubuntu"
    private_key = file("~/.ssh/id_rsa")
    host        = aws_instance.web.public_dns
  }
  provisioner "file" {
    source      = "./file.yaml"
    destination = "./file.yaml"
  }
}

# null_resource.copy_file_on_vm: Creating...
# null_resource.copy_file_on_vm: Provisioning with 'file'...
# null_resource.copy_file_on_vm: Creation complete after 2s [id=someid]

Remote-Exec

Remote-Exec is used to run a command or a script on a remote-vm.

It supports the following arguments:

inline → list of commands that should run on the vm
script → a script that runs on the vm
scripts → multiple scripts to run on the vm

You have to provide only one of the above arguments as they are not going to work together.

Similar to file, you will need to add a connection block.

resource "null_resource" "remote_exec" {
  depends_on = [
    aws_instance.web
  ]
  connection {
    type        = "ssh"
    user        = "ubuntu"
    private_key = file("~/.ssh/id_rsa")
    host        = aws_instance.web.public_dns
  }
  provisioner "remote-exec" {
    inline = [
      "mkdir dir1"
    ]
  }
}

# null_resource.remote_exec: Creating...
# null_resource.remote_exec: Provisioning with 'remote-exec'...
# null_resource.remote_exec (remote-exec): Connecting to remote host via SSH...
# null_resource.remote_exec (remote-exec):   Host: somehost
# null_resource.remote_exec (remote-exec):   User: ubuntu
# null_resource.remote_exec (remote-exec):   Password: false
# null_resource.remote_exec (remote-exec):   Private key: true
# null_resource.remote_exec (remote-exec):   Certificate: false
# null_resource.remote_exec (remote-exec):   SSH Agent: true
# null_resource.remote_exec (remote-exec):   Checking Host Key: false
# null_resource.remote_exec (remote-exec):   Target Platform: unix
# null_resource.remote_exec (remote-exec): Connected!
# null_resource.remote_exec: Creation complete after 3s [id=someid]

In order to use the above code, just use the file example and change the copy_file_on_vm null resource with this one and you are good to go.

You have to make sure that you can connect to your vm, so make sure you have a security rule that permits ssh access in your security group.

Even though I don’t recommend provisioners, keep in mind they may be a necessary evil.

Originally posted on Medium.

7. Terraform Loops and Conditionals

In this post, we will talk about how to use Count, for_each, for loops, ifs, and ternary operators inside of Terraform. It will be a long journey, but this will help a lot when writing better Terraform code.

As I am planning to use the Kubernetes provider in this lesson, you can easily create your own Kubernetes cluster using Kind. More details here.

👉COUNT 👈

If I say I hate count, that would be an understatement. I get why people use it, but in my book, since for_each was released, I never looked back. This is my really unpopular opinion, so don’t take my word for it.

Let’s see what we can do with count. Using count we can, you guessed it, create multiple resources of the same type. Every terraform resource supports the count block. Count exposes a count.index object, which can be used in the same way you would use an iterator in any programming language.

resource "kubernetes_namespace" "this" {
  count = 5
  metadata {
    name = format("ns%d", count.index)
  }
}

The above block will create 5 namespaces in Kubernetes with the following names: ns1, ns2, ns3, ns4, ns5. All fun and games until now. If you change the count to 4, the last namespace, ns5 will be deleted if you re-apply the code.

In any case, when you are using count, you can address a particular index of your resource by using type.name[index]. In our case that means individual resources can be accessed with kubernetes_namespace.this[0] to kubernetes_namespace.this[4].

Let’s suppose you want to customize, a little bit the names of the namespaces. For that, we can you use a local or a variable in conjunction with a function. Don’t worry if you see a couple of functions now, we will discuss functions in a separate article in detail.

locals {
  namespaces = ["frontend", "backend", "database"]
}

resource "kubernetes_namespace" "this" {
  count = length(local.namespaces)
  metadata {
    name = local.namespaces[count.index]
  }
}

The above will create three namespaces called frontend, backend, and database. Let’s suppose for whatever reason, you want to remove the backend namespace and keep only the other two.

What is going to happen when you reapply the code?

Plan: 1 to add, 0 to change, 2 to destroy.

Let’s break this down. We are using a list with 3 elements, which means that our list has 3 indexes: 0, 1, and 2. If we remove the element from index 1, backend, the element from index 2, becomes the element from index 1.

To make it even more clear, initially, we have the following resources:

kubernetes_namespace.this[0] → frontend
kubernetes_namespace.this[1] → backend
kubernetes_namespace.this[2] → database

After we remove backend:

kubernetes_namespace.this[0] → frontend
kubernetes_namespace.this[1] → database

Due to the fact that database changes its terraform identity by moving from index 2 to index 1, it will be recreated, and imagine all the problems you will have when you are recreating a namespace with a ton of things inside.

Also, let’s suppose you are creating a hundred ec2 instances and you are using a list for that to better configure them. For some reason, let’s suppose you want to remove the instance with index 13 (see what I did there?), what’ going to happen with the ones from index 14 to 100? They will be recreated because all of them are going to change their index to what it was minus 1.

And that’s why I hate count. It doesn’t give you the flexibility to create very generic resources and in my book, that’s a hard pass.

👉FOR_EACH 👈

I am a big fan of using for_each on all of the resources, as you never know when you want to create multiple resources of the same kind. For_Each can be used with map and set variables, but I don’t remember a use case in which I used a set. So what I’m always doing is using for_each on maps, and more specifically on map(object). I’ll show you what that looks like in a bit.

For_each exposes one attribute called each. This attribute contains a key and value which can be used witheach.key and each.value.

With for_each, you will reference an instance of your resource with type.name[key].

Let’s use a variable this time to create the namespaces, and let’s configure some more information for them in order to see why for_each is superior from my point of view.

resource "kubernetes_namespace" "this" {
  for_each = var.namespaces
  metadata {
    name        = each.key
    annotations = each.value.annotations
    labels      = each.value.labels
  }
}

variable "namespaces" {
  type = map(object({
    annotations = optional(map(string), {})
    labels      = optional(map(string), {})
  }))
  default = {
    namespace1 = {}
    namespace2 = {
      labels = {
        color = "green"
      }
    }
    namespace3 = {
      annotations = {
        imageregistry = "https://hub.docker.com/"
      }
    }
    namespace4 = {
      labels = {
        color = "blue"
      }
      annotations = {
        imageregistry = "my_awesome_registry"
      }
    }
  }
}

We have defined a variable called namespaces and we are going to iterate through it on the kubernetes_namespace resource. This variable has a map(object) type and inside of it, we’ve defined two optional properties: annotations and labels.

Optional can be used on parameters inside object variables to give the possibility to omit that particular parameter and to provide a default value for it instead. As this feature is available from Terraform 1.3.0, I believe it will soon be embraced by the community as a best practice (for me, it is already). Inside this variable, we have added a default value, just for demo purposes, in the real world, you are going to anyway separate resources from variables in their own files, and you are going to provide default values as empty maps if you are using the above approach with optionals on your parameters, but that’s a completely different story.

Let’s go a little bit through our default value:

All namespaces will be our each.key
Everything after namespaceX = will be our each.value → Meaning, that if we want to reference labels or annotations, we are going to use each.value.labels or each.value.annotations

Due to the fact that both parameters inside of our variable have been defined with the optional block, we can omit them, meaning that namespace1 = {} is a valid configuration. This is happening because, in our resource, we’ve assigned the name in the metadata block to each.key.

The entire configuration translates into:

namespace1 → will have no labels and no annotations
namespace2 → will have only labels
namespace3 → will have only annotations
namespace4 → will have both labels and annotations

If I want to remove namespace2 for whatever reason, what is going to happen to namespace3 and namespace4? Absolutely nothing. Due to the fact that we are not using a list anymore and we are using a map, by removing an element of the map, there is going to be no change to the others (remember, Terraform identifies our resources with kubernetes_namespace.this["namespace1"] to kubernetes_namespace.this["namespace4"]).

And this is why I will always vouch for for_each instead of count.

**👉Ternary Operators 👈

I’ve seen plenty of people arguing that Terraform doesn’t have an if instruction. Well, it does, but you can use that only when you are building complex instructions in for loops (not for_each), which I will discuss soon. For any other type of condition, you have ternary operators and the syntax is:

condition ? val1 : val2

The above means if the condition is true, use val1, if the condition is false, use val2. Let’s see it in action:

locals {
  use_local_name = false
  name           = "namespace1"
}

resource "kubernetes_namespace" "this" {
  metadata {
    name = local.use_local_name ? local.name : "namespace2"
  }
}

In the above example, I’m checking if local.use_local_name is equal to true, and if it is, I’m going to provide to my namespace the name that is in local.name otherwise, I am going to provide it namespace2.

Of course, due to the fact that I’ve set use_local_name to false, this means that the name of my namespace will be namespace2.

The beautiful or the ugliest part of ternary operators (depending on who reads the code) is the fact that you can use nested conditionals.

Let’s build a local variable that uses nested conditionals:

locals {
  val1               = 1
  val2               = 2
  val3               = 3
  nested_conditional = local.val2 > local.val1 ? local.val3 > local.val2 ? local.val3 : local.val2 : local.val1
}

In the above operation, we are checking initially if val2 is greater than val1:

if it’s not, then nested_conditional will go to the last “:” and assign the value to local.val1
if it is, then we are checking if val3 is greater than val2 and:

- if val3 is greater than val2 the value of nested _conditional will be val3

- if val3 is less than val2 the value of nested_conditional will be val2

These nested conditionals can get pretty hard to understand and usually if you see something that goes more than 3 or 4 levels, there is almost always an error in judgment somewhere or you should do some changes to the variable or expression that you are using when you are building this as it will get almost impossible to maintain in the long run.

👉For loops and Ifs 👈

If you are familiar with Python, you are going to notice pretty easily that for loops and ifs in Terraform are pretty similar to Python’s list comprehensions.

Let me show you what I’m talking about:

locals {
  list_var = range(5)
  map_var = {
    cat1 = {
      color = "orange",
      name  = "Garfield"
    },
    cat2 = {
      color = "blue",
      name  = "Tom"
    }
  }
  for_list_list = [for i in local.list_var : i * 2]
  for_list_map  = { for i in local.list_var : format("Number_%s", i) => i }

  for_map_list = [for k, v in local.map_var : k]
  for_map_map  = { for k, v in local.map_var : format("Cat_%s", k) => v }

  for_list_list_if = [for i in local.list_var : i if i > 2]
  for_map_map_if   = { for k, v in local.map_var : k => v if v.color == "orange" }
}

We’ve defined a list variable that generates numbers from 0 to 4 list_varand a map variable with two elements map_var. As you can see, for the other 6 locals defined in the code snippet, you can build both lists and maps by using this type of loop. By starting the value with [ you are creating a list, and by starting the value with { you are creating a map.

The difference from a syntax standpoint is that when you are building a map you have to provide the => attribute. The sky is the limit when it comes to these expressions, you can nest them on how many levels you want depending on the structure you are iterating through, but this will become, again, very hard to maintain.

If you are cycling through a map variable, and you are using a single iterator, you will actually cycle only through the values of the map, by using two, you will cycle through both keys and variables (the first iterator will be the key, the second iterator will be the value).

for_list_list = [
  0,
  2,
  4,
  6,
  8,
]
for_list_map = {
  "Number_0" = 0
  "Number_1" = 1
  "Number_2" = 2
  "Number_3" = 3
  "Number_4" = 4
}
for_map_list = [
  "cat1",
  "cat2",
]
for_map_map = {
  "Cat_cat1" = {
    "color" = "orange"
    "name" = "Garfield"
  }
  "Cat_cat2" = {
    "color" = "blue"
    "name" = "Tom"
  }
}
for_list_list_if = [
  3,
  4,
]
for_map_map_if = {
  "cat1" = {
    "color" = "orange"
    "name" = "Garfield"
  }
}

Above are all the values of the locals defined with for loops and ifs. Let’s discuss the last two:

for_list_list_if = [for i in local.list_var : i if i > 2]

This is cycling through our initial list that holds the numbers from 0 to 4 and is creating a new list with only the elements that are greater than 2.

for_map_map_if = { for k, v in local.map_var : k => v if v.color == "orange" }

This one is cycling through our initial map variable and will recreate a new map with all the elements that have the color equal to orange.

There is another operator called splat(*) that can help with providing a more concise way to reference some common operations that you would usually do with a for. This operator works only on lists, sets, and tuples.

splat_list = [
    {
      name = "Mike"
      age  = 25
    },
    {
      name = "Bob"
      age  = 29
    }
  ]
  splat_list_names = local.splat_list[*].name

If for example, you would’ve had a list of maps in the above format, you can easily build a list of all the names or ages from it, by using the splat operator.

Originally posted on Medium here.

8. Terraform CLI Commands

Throughout my posts, I said I don’t want to repeat myself or reinvent the wheel, right?

Well, I am keeping my promise, so if you want to see all commands that you can use and even download an awesome cheatsheet with them, you can follow this article which really nails it: https://spacelift.io/blog/terraform-commands-cheat-sheet

9. Terraform Functions

Terraform functions are built-in, reusable code blocks that perform specific tasks within Terraform configurations. They make your code more dynamic and ensure your configuration is DRY. Functions allow you to perform various operations, such as converting expressions to different data types, calculating lengths, and building complex variables.

These functions are split into multiple categories:

String
Numeric
Collection
Date and Time
Crypto and Hash
Filesystem
IP Network
Encoding
Type Conversion

This split, however, can become overwhelming to someone who doesn’t have that much experience with Terraform. For example, the formatlist list function is considered to be a string function even though it modifies elements from a list. A list is a collection though; some may argue that this function should be considered a collection function, but still, at its core, it does changes to strings.

For that particular reason, I won’t specify the function type when I’ll describe them, but just go with what you can do with them. Of course, I will not go through all of the available functions, but through the ones, I am using throughout my configurations.

ToType Functions

ToType is not an actual function; rather, many functions can help you change the type of a variable to another type.

tonumber(argument) → With this function you can change a string to a number, anything else apart from another number and null will result in an error

tostring(argument) → Changes a number/bool/string/null to a string

tobool(argument) → Changes a string (only “true” or “false”)/bool/null to a bool

tolist(argument) → Changes a set to a list

toset(argument) → Changes a list to a set

tomap(argument) → Converts its argument to a map

In Terraform, you are rarely going to need to use these types of functions, but I still thought they are worth mentioning.

format(string_format, unformatted_string)

The format function is similar to the printf function in C and works by formatting a number of values according to a specification string. It can be used to build different strings that may be used in conjunction with other variables. Here is an example of how to use this function:

locals {
 string1       = "str1"
 string2       = "str2"
 int1          = 3
 apply_format  = format("This is %s", local.string1)
 apply_format2 = format("%s_%s_%d", local.string1, local.string2, local.int1)
}

output "apply_format" {
 value = local.apply_format
}
output "apply_format2" {
 value = local.apply_format2
}

# Result in:
apply_format  = "This is str1"
apply_format2 = "str1_str2_3"

formatlist(string_format, unformatted_list)

The formatlist function uses the same syntax as the format function but changes the elements in a list. Here is an example of how to use this function:

locals {
 format_list = formatlist("Hello, %s!", ["A", "B", "C"])
}

output "format_list" {
 value = local.format_list
}

# Result in:
format_list = tolist(["Hello, A!", "Hello, B!", "Hello, C!"])

length(list / string / map)

Returns the length of a string, list, or map.

locals {
 list_length   = length([10, 20, 30])
 string_length = length("abcdefghij")
}

output "lengths" {
 value = format("List length is %d. String length is %d", local.list_length, local.string_length)
}

# Result in:
lengths = "List length is 3. String length is 10"

join(separator, list)

Another useful function in Terraform is “join”. This function creates a string by concatenating together all elements of a list and a separator. For example, consider the following code:

locals {
 join_string = join(",", ["a", "b", "c"])
}

output "join_string" {
 value = local.join_string
}

# Result in:
The output of this code will be “a, b, c”.

try(value, fallback)

Sometimes, you may want to use a value if it is usable, but fall back to another value if the first one is unusable. This can be achieved using the “try” function. For example:

locals {
 map_var = {
   test = "this"
 }
 try1 = try(local.map_var.test2, "fallback")
}

output "try1" {
 value = local.try1
}

# Result:
The output of this code will be “fallback”, as the expression local.map_var.test2 is unusable.

can(expression)

A useful function for validating variables is “can”. It evaluates an expression and returns a boolean indicating if there is a problem with the expression. For example:

variable "a" {
 type = any
 validation {
   condition     = can(tonumber(var.a))
   error_message = format("This is not a number: %v", var.a)
 }
 default = "1"
}

# Result:
The validation in this code will give you an error: “This is not a number: 1”.

flatten(list)

In Terraform, you may work with complex data types to manage your infrastructure. In these cases, you may want to flatten a list of lists into a single list. This can be achieved using the “flatten” function, as in this example:

locals {
 unflatten_list = [[1, 2, 3], [4, 5], [6]]
 flatten_list   = flatten(local.unflatten_list)
}

output "flatten_list" {
 value = local.flatten_list
}

# Result:
The output of this code will be [1, 2, 3, 4, 5, 6].

keys(map) & values(map)

It may be useful to extract the keys or values from a map as a list. This can be achieved using the “keys” or “values” functions, respectively. For example:

locals {
 key_value_map = {
   "key1" : "value1",
   "key2" : "value2"
 }
 key_list   = keys(local.key_value_map)
 value_list = values(local.key_value_map)
}

output "key_list" {
 value = local.key_list
}

output "value_list" {
 value = local.value_list
}

# Result: 
key_list = ["key1", "key2"]
value_list = ["value1", "value2"]

slice(list, startindex, endindex)

Slice returns consecutive elements from a list from a startindex (inclusive) to an endindex (exclusive).

locals {
 slice_list = slice([1, 2, 3, 4], 2, 4)
}


output "slice_list" {
 value = local.slice_list
}

# Result:
slice_list = [3]

range

Creates a range of numbers:

one argument(limit)
two arguments(initial_value, limit)
three arguments(initial_value, limit, step)

`locals {
range_one_arg = range(3)
range_two_args = range(1, 3)
range_three_args = range(1, 13, 3)
}

output "ranges" {
value = format("Range one arg: %v. Range two args: %v. Range three args: %v", local.range_one_arg, local.range_two_args, local.range_three_args)
}

Result:

range = "Range one arg: [0, 1, 2]. Range two args: [1, 2]. Range three args: [1, 4, 7, 10]"`

lookup(map, key, fallback_value)

Retrieves a value from a map using its key. If the value is not found, it will return the default value instead

locals {
 a_map = {
   "key1" : "value1",
   "key2" : "value2"
 }
 lookup_in_a_map = lookup(local.a_map, "key1", "test")
}


output "lookup_in_a_map" {
 value = local.lookup_in_a_map
}

# Result:
This will return: lookup_in_a_map = "key1"

concat(lists)

Takes two or more lists and combines them in a single one

locals {
 concat_list = concat([1, 2, 3], [4, 5, 6])
}


output "concat_list" {
 value = local.concat_list
}


# Result:
concat_list = [1, 2, 3, 4, 5, 6]

merge(maps)

The merge function takes one or more maps and returns a single map that contains all of the elements from the input maps. The function can also take objects as input, but the output will always be a map.

Let’s take a look at an example:

locals {
 b_map = {
   "key1" : "value1",
   "key2" : "value2"
 }
 c_map = {
   "key3" : "value3",
   "key4" : "value4"
 }
 final_map = merge(local.b_map, local.c_map)
}


output "final_map" {
 value = local.final_map
}

# Result:
final_map = {
  "key1" = "value1"
  "key2" = "value2"
  "key3" = "value3"
  "key4" = "value4"
}

zipmap(key_list, value_list)

Constructs a map from a list of keys and a list of values

locals {
 key_zip    = ["a", "b", "c"]
 values_zip = [1, 2, 3]
 zip_map    = zipmap(local.key_zip, local.values_zip)
}

output "zip_map" {
 value = local.zip_map
}

# Result
zip_map = {
  "a" = 1
  "b" = 2
  "c" = 3
}

expanding function argument …

This special argument works only in function calls and expands a list into separate arguments. Useful when you want to merge all maps from a list of maps

locals {
 list_of_maps = [
   {
     "a" : "a"
     "d" : "d"
   },
   {
     "b" : "b"
     "e" : "e"
   },
   {
     "c" : "c"
     "f" : "f"
   },
 ]
 expanding_map = merge(local.list_of_maps...)
}

output "expanding_map" {
 value = local.expanding_map
}

# Result
expanding_map = {
  "a" = "a"
  "b" = "b"
  "c" = "c"
  "d" = "d"
  "e" = "e"
  "f" = "f"
}

file(path_to_file)

Reads the content of a file as a string and can be used in conjunction with other functions like jsondecode / yamldecode.

locals {
  a_file = file("./a_file.txt")
}

output "a_file" {
  value = local.a_file
}

# Result
The output would be the content of the file called a_file as a string.

templatefile(path, vars)

Reads the file from the specified path and changes the variables specified in the file between the interpolation syntax ${ … } with the ones from the vars map.

locals {
 a_template_file = templatefile("./file.yaml", { "change_me" : "awesome_value" })
}


output "a_template_file" {
 value = local.a_template_file
}

# Result
This will change the ${change_me} variable to awesome_value.

jsondecode(string)

Interprets a string as json.

locals {
 a_jsondecode = jsondecode("{\"hello\": \"world\"}")
}


output "a_jsondecode" {
 value = local.a_jsondecode
}

# Result
jsondecode = {
 "hello" = "world"
}

jsonencode(string)

Encodes a value to a string using json

locals {
 a_jsonencode = jsonencode({ "hello" = "world" })
}


output "a_jsonencode" {
 value = local.a_jsonencode
}

# Result
a_jsonencode = "{\"hello\":\"world\"}"

yamldecode(string)

Parses a string as a subset of YAML, and produces a representation of its value.

locals {
 a_yamldecode = yamldecode("hello: world")
}


output "a_yamldecode" {
 value = local.a_yamldecode
}

# Result:
a_yamldecode = {
 "hello" = "world"
}

yamlencode(value)

Encodes a given value to a string using YAML.

locals {
 a_yamlencode = yamlencode({ "a" : "b", "c" : "d" })
}


output "a_yamlencode" {
 value = local.a_yamlencode
}

# Result:
a_yamlencode = <<EOT
"a": "b"
"c": "d"

EOT

You can use Terraform functions to make your life easier and to write better code. Keeping your configuration as DRY as possible improves readability and makes updating it easier, so functions are a must-have in your configurations.

These are just the functions that I am using the most, but some honorable mentions are element, base64encode, base64decode, formatdate, uuid, and distinct.

Originally posted on Medium here.

10. Working with files

When it comes to Terraform, working with files is pretty straightforward. Apart from the .tf and .tfvars files, you can use json, yaml, or whatever other file types that support your needs.

In the last article, I mentioned a couple of functions that interact with files, like file & template_file, and right now I want to show you how easy it is to interact with a lot of files and use some of their content as variables for your configuration.

For the following examples, let’s suppose we are using this yaml file:

namespaces:
  ns1:
    annotations:
      imageregistry: "https://hub.docker.com/"
    labels:
      color: "green"
      size: "big"
  ns2:
    labels:
      color: "red"
      size: "small"
  ns3:
    annotations:
      imageregistry: "https://hub.docker.com/"

Reading a file only if it exists

locals {
  my_file = fileexists("./my_file.yaml") ? file("./my_file.yaml") : null
}

output "my_file" {
  value = local.my_file
}

We are using two functions for this and a ternary operator. The fileexistsfunction checks if the file exists and returns true or false based on the result. In our case above, if the file exists, my_file will be the content of my_file.yaml as a string, otherwise will be null.

Let’s take this up a notch and transform this into a real use case.

Reading the yaml file and using it in a configuration if it exists

locals {
  namespaces = fileexists("./my_file.yaml") ? yamldecode(file("./my_file.yaml")).namespaces : var.namespaces
}

output "my_file" {
  value = local.namespaces
}

resource "kubernetes_namespace" "this" {
  for_each = local.namespaces
  metadata {
    name        = each.key
    annotations = lookup(each.value, "annotations", {})
    labels      = lookup(each.value, "labels", {})
  }
}

variable "namespaces" {
  type = map(object({
    annotations = optional(map(string), {})
    labels      = optional(map(string), {})
  }))
  default = {
    ns1 = {}
    ns2 = {}
    ns3 = {}
  }
}

In the example above, we are checking if the file exists, and if it does we are going to load the namespaces from the yaml file, otherwise use a variable for that. As the file exists, we are using yamldecode on the loaded string to create a map variable and we are passing it to for_each. This can be extremely useful in some use cases.

Example Terraform plan:

# kubernetes_namespace.this["ns1"] will be created
  + resource "kubernetes_namespace" "this" {
      + id = (known after apply)

      + metadata {
          + annotations      = {
              + "imageregistry" = "https://hub.docker.com/"
            }
          + generation       = (known after apply)
          + labels           = {
              + "color" = "green"
              + "size"  = "big"
            }
          + name             = "ns1"
          + resource_version = (known after apply)
          + uid              = (known after apply)
        }
    }

  # kubernetes_namespace.this["ns2"] will be created
  + resource "kubernetes_namespace" "this" {
      + id = (known after apply)

      + metadata {
          + generation       = (known after apply)
          + labels           = {
              + "color" = "red"
              + "size"  = "small"
            }
          + name             = "ns2"
          + resource_version = (known after apply)
          + uid              = (known after apply)
        }
    }

  # kubernetes_namespace.this["ns3"] will be created
  + resource "kubernetes_namespace" "this" {
      + id = (known after apply)

      + metadata {
          + annotations      = {
              + "imageregistry" = "https://hub.docker.com/"
            }
          + generation       = (known after apply)
          + name             = "ns3"
          + resource_version = (known after apply)
          + uid              = (known after apply)
        }
    }

Using templatefile on a yaml file and use its configuration if it exists

We can go further on this example and add some variables that we will change with templatefile. To do that, we must first make some changes to our initial yaml file.

namespaces:
  ns1:
    annotations:
      imageregistry: ${image_registry_ns1}
    labels:
      color: "green"
      size: "big"
  ns2:
    labels:
      color: ${color_ns2}
      size: "small"
  ns3:
    annotations:
      imageregistry: "https://hub.docker.com/"

The only change that we will have to do for the above code will be a change to the locals:

locals {
  namespaces = fileexists("./my_file.yaml") ? yamldecode(templatefile("./my_file.yaml", { image_registry_ns1 = "ghcr.io", color_ns2 = "black" })).namespaces : var.namespaces
}

Those two variables defined inside the yaml file using the ${} syntax, will be changed with the ones from the templatefile functions.

Fileset

The fileset function helps with identifying all files inside a directory that respect a pattern.

locals {
  yaml_files = fileset(".", "*.yaml")
}

The above will show all the yaml files, inside the current directory. This output will be a list of all those files. Not very useful on its own, right?

Well, you can use the file function to load the content of these files as strings. You can take them one by one, using list indexes, but if you want to take it up a notch, what you can do is use a for loop and group them together in something that makes sense.

Let’s suppose you have 2 other yaml files that are similar to the one we used at the beginning of the post. We can group them all together in a single variable:

locals {
  namespaces = merge([for my_file in fileset(".", "*.yaml") : yamldecode(file(my_file))["namespaces"]]...)
}

output "namespaces" {
  value = local.namespaces
}

As the files are exactly the same, only the names of the namespaces are different, this will result in:

namespaces = {
  "ns1" = {
    "annotations" = {
      "imageregistry" = "https://hub.docker.com/"
    }
    "labels" = {
      "color" = "green"
      "size" = "big"
    }
  }
  "ns2" = {
    "labels" = {
      "color" = "red"
      "size" = "small"
    }
  }
  "ns3" = {
    "annotations" = {
      "imageregistry" = "https://hub.docker.com/"
    }
  }
  "ns4" = {
    "annotations" = {
      "imageregistry" = "https://hub.docker.com/"
    }
    "labels" = {
      "color" = "green"
      "size" = "big"
    }
  }
  "ns5" = {
    "labels" = {
      "color" = "red"
      "size" = "small"
    }
  }
  "ns6" = {
    "annotations" = {
      "imageregistry" = "https://hub.docker.com/"
    }
  }
  "ns7" = {
    "annotations" = {
      "imageregistry" = "https://hub.docker.com/"
    }
    "labels" = {
      "color" = "green"
      "size" = "big"
    }
  }
  "ns8" = {
    "labels" = {
      "color" = "red"
      "size" = "small"
    }
  }
  "ns9" = {
    "annotations" = {
      "imageregistry" = "https://hub.docker.com/"
    }
  }
}

You can check another cool example here.

Other Examples

You can get filesystem-related information using these key expressions:

path.module — This function returns the path of the current module being executed. This is useful for accessing files or directories that are relative to the module being executed.
path.root — This function returns the root directory of the current Terraform project. This is useful for accessing files or directories located at the project's root.
path.cwd — This function returns the current working directory where Terraform is being executed before any chdir operations happened. This is useful for accessing files or directories that are relative to the directory where Terraform is running from.

There are some other file functions that can be leveraged in order to accommodate some use cases, but to be honest I’ve used them only once or twice.

Still, I believe mentioning them, will bring some value.

basename — takes a path and returns everything apart from the last part of it

E.G: basename("/Users/user1/hello.txt") will return hello.txt.

dirname — behaves exactly opposite to basename, returns all the directories until the file

E.G: dirname("/Users/user1/hello.txt") will return /Users/user1

pathexpand — takes a path that starts with a ~ and expands this path adding the home of the logged in user. If the path, doesn’t use a ~ this function will not do anything

E.G: You are logged in as user1 on a Mac: pathexpand("~/hello.txt") will return /Users/user1/hello.txt

filebase64 — reads the content of a file and returns it as base64 encoded text.

abspath — takes a string containing a filesystem path and returns the absolute path

Originally posted on Medium here.

11. Understanding Terraform State

Terraform state is a critical component of Terraform that enables users to define, provision, and manage infrastructure resources using declarative code. In this blog post, we will explore the importance of Terraform state, how it works, and best practices for managing it.

It is a json file that tracks the state of infrastructure resources managed by Terraform. By default, the name of the file is terraform.tfstate and whenever you update the first state, a backup is generated called terraform.tfstate.backup.

This state file is stored locally by default, but can also be stored remotely using a remote backend such as Amazon S3, Azure Blob Storage, Google Cloud Storage, or HashiCorp Consul. The Terraform state file includes the current configuration of resources, their dependencies, and metadata such as resource IDs and resource types. There are a couple of products that help with managing state and provide a sophisticated workflow around Terraform like Spacelift or Terraform Cloud.

How does it work?

When Terraform is executed, it reads the configuration files and the current state file to determine the changes required to bring the infrastructure to the desired state. Terraform then creates an execution plan that outlines the changes to be made to the infrastructure. If the plan is accepted, Terraform applies the changes to the infrastructure and updates the state file with the new state of the resources.

You can use the terraform state command to manage your state.

terraform state list: This command lists all the resources that are currently tracked by Terraform state.
terraform state show: This command displays the details of a specific resource in the Terraform state. The output includes all the attributes of the resource.
terraform state pull: This command retrieves the current Terraform state from a remote backend and saves it to a local file. This command is useful when you want to make manual operations in a remote state.
terraform state push: This command uploads the local Terraform state file to the remote backend. This command is useful after you made manual changes to your remote state.
terraform state rm: This command removes a resource from the Terraform state. This doesn’t mean the resource will be destroyed, it won’t be managed by Terraform after you’ve removed it.
terraform state mv: This command renames a resource in the Terraform state.
terraform state replace-provider: This command replaces the provider configuration for a specific resource in the Terraform state. This command is useful when switching from one provider to another or upgrading to a new provider version.

Supported backends

Amazon S3 Backend

The Amazon S3 backend is a popular choice for remote state storage. To configure the Amazon S3 backend, you will need to create an S3 bucket and an IAM user with permissions to access the bucket. Here is an example of how to configure the Amazon S3 backend in Terraform:

terraform {
  backend "s3" {
    bucket         = "my-terraform-state-bucket"
    key            = "terraform.tfstate"
    region         = "us-west-2"
  }
}

The S3 backend supports locking, but to do that you will need also need a dynamodb table. The table must have a partition key named LockID as a string. If this is not configured, state locking will be disabled.

terraform {
  backend "s3" {
    bucket         = "my-terraform-state-bucket"
    key            = "terraform.tfstate"
    region         = "us-west-2"
    dynamodb_table = "terraform-state-lock"
  }
}

Azure Blob Storage Backend

The Azure Blob Storage backend is similar to the Amazon S3 backend and provides a remote storage solution for Terraform state files. To configure the Azure Blob Storage backend, you will need to create an Azure Storage Account and a container to store the Terraform state file. This backend supports locking by default, so you won’t need to configure anything else for locking.

Here is an example of how to configure the Azure Blob Storage backend in Terraform:

terraform {
  backend "azurerm" {
    storage_account_name = "mytfstateaccount"
    container_name       = "tfstate"
    key                  = "terraform.tfstate"
  }
}

Google Cloud Storage Backend

The Google Cloud Storage backend is a popular choice for remote state storage for users of Google Cloud Platform. To configure the Google Cloud Storage backend, you will need to create a bucket and a service account with access to the bucket. Again there is no need to do anything else to enable locking. Here is an example of how to configure the Google Cloud Storage backend in Terraform:

terraform {
  backend "gcs" {
    bucket       = "my-terraform-state-bucket"
    prefix       = "terraform/state"
    credentials  = "path/to/credentials.json"
  }
}

HTTP Backend

It is a simple backend that can be useful for development or testing, but it is not recommended for production use because it does not provide the same level of security and reliability as other backends.

To configure the HTTP backend, you will need to have access to a web server that can serve files over HTTP or HTTPS. Here is an example of how to configure the HTTP backend in Terraform:

terraform {
  backend "http" {
    address = "https://my-terraform-state-server.com/terraform.tfstate"
  }
}

In this example, the address parameter specifies the URL of the Terraform state file on the web server. You can use any web server that supports HTTP or HTTPS to host the state file, including popular web servers like Apache or Nginx.

When using the HTTP backend, it is important to ensure that the state file is protected with appropriate access controls and authentication mechanisms. Without proper security measures, the state file could be accessed or modified by unauthorized users, which could lead to security breaches or data loss.

Remote State Data Source

The Terraform Remote State Data Source, like any other data source, retrieves existing information. This special data source, doesn’t depend on any provider, this allows you to retrieve state data from a previously deployed Terraform infrastructure. This can be useful if you need to reference information from one infrastructure to another.

To use a remote state data source in Terraform, you first need to configure the remote state backend for the infrastructure you want to retrieve data from. This is done in the backend block in your Terraform configuration as specified above. After you create your infrastructure for the first configuration, you can reference it in the second one using the remote data source.

data "terraform_remote_state" "config1" {
  backend = "local"

  config = {
    path = "../config1"
  }
}

resource "null_resource" "this" {
  provisioner "local-exec" {
    command = format("echo %s", data.terraform_remote_state.config1.outputs.var1)
  }
}

In the above example, we are supposing that we have a configuration in the directory “../config1” that has some Terraform code up and running. In that code, we have declared a “var1” output, that we are referencing in our null resource.

As mentioned in some of the previous articles, Terraform documentation is your best friend when it comes to understanding what you can do with a resource or data source, or what it exposes.

This data source exposes:

Best Practices

There are several best practices for managing Terraform state, including:

Use a remote backend : Storing the Terraform state file remotely provides several benefits, including better collaboration, easier access control, and improved resilience. Remote backends such as Amazon S3 or HashiCorp Consul can be used to store the state file securely and reliably.
Use locking : When multiple users are working on the same Terraform project, locking is necessary to prevent conflicts. Locking ensures that only one user can modify the state file at a time, preventing conflicts and ensuring changes are applied correctly. As shown before, there are many backends that support locking.
Use versioning: Your configuration should always be versioned, as this will make it easier to achieve an older version of the infrastructure if something goes wrong with the changes you are making.
Isolate state:

Don’t add hundreds of resources in the same state : Making a mistake with one resource can potentially hurt all your infrastructure
Have a single state file per environment : When making changes, it is usually a best practice to first make the change on a lower environment and after that promote it to the higher one
Use Terraform workspaces : Terraform workspaces allow users to manage multiple environments, such as development, staging, and production, with a single Terraform configuration file. Each workspace has its own state file, allowing changes to be made to each of them

5. Use modules : Versioned modules will make it easier to make changes to your code, hence changes will be easier to promote across environments, making operations to the state less painful.

erraform state is a critical component that enables users to define, provision, and manage infrastructure resources using declarative code. Terraform state ensures that resources are created, updated, or destroyed only when necessary and in the correct order. Remote backends such as Amazon S3, Azure Blob Storage, Google Cloud Storage, or HashiCorp Consul can be used to store the state file securely and reliably, while state file locks can prevent conflicts when multiple users are working with the same Terraform configuration. Consistent naming conventions and the terraform state command can help to ensure that Terraform state files are easy to manage and understand.

By following these best practices for managing Terraform state, users can ensure that their infrastructure resources are managed effectively and efficiently using Terraform.

Originally posted on Medium here.

12. Terraform Depends On and Lifecycle Block

Understanding how to use depends_on and the lifecycle block can help you better manage complex infrastructure dependencies and handle resource updates and replacements. In this post, I will provide an overview of what these features are, how they work, and best practices for using them effectively in your Terraform code.

Depends_on

The depends_on meta-argument in Terraform is used to specify dependencies between resources. When Terraform creates your infrastructure, it automatically determines the order in which to create resources based on their dependencies. However, in some cases, you may need to manually specify the order in which resources are created, and that's where depends_on comes in.

depends_on works on all resources, data sources, and also modules. You are going to most likely use depends_on whenever you are using null_resourceswith provisioners to accomplish some use cases.

Let’s look into one simple example:

resource "kubernetes_namespace" "first" {
  metadata {
    name = "first"
  }
}

resource "kubernetes_namespace" "second" {
  depends_on = [
    kubernetes_namespace.first
  ]
  metadata {
    name = "second"
  }
}

Even though these namespaces resources are the same, the one named “second” will get created after the one named “first”.

As the depends_on argument uses a list, this means that your resources can depend on multiple things before they are getting created.

resource "kubernetes_namespace" "first" {
  metadata {
    name = "first"
  }
}

resource "kubernetes_namespace" "second" {
  depends_on = [
    kubernetes_namespace.first,
    kubernetes_namespace.third
  ]
  metadata {
    name = "second"
  }
}

resource "kubernetes_namespace" "third" {
  metadata {
    name = "third"
  }
}

# kubernetes_namespace.first: Creating...
# kubernetes_namespace.third: Creating...
# kubernetes_namespace.third: Creation complete after 0s [id=third]
# kubernetes_namespace.first: Creation complete after 0s [id=first]
# kubernetes_namespace.second: Creating...
# kubernetes_namespace.second: Creation complete after 0s [id=second]

As we’ve made the second namespace depend also on the third now, you can see from the above apply output that the first and third race to get created first, and after they finished, the second one started the creation process.

What about depends_on on modules? It works exactly the same:

module "null" {
  depends_on = [
    resource.null_resource.this
  ]
  source = "./null"
}

resource "null_resource" "this" {
  provisioner "local-exec" {
    command = "echo this resource"
  }
}

# null_resource.this: Creating...
# null_resource.this: Provisioning with 'local-exec'...
# null_resource.this (local-exec): Executing: ["/bin/sh" "-c" "echo this resource"]
# null_resource.this (local-exec): this resource
# null_resource.this: Creation complete after 0s [id=1259986187217330742]
# module.null.null_resource.this: Creating...
# module.null.null_resource.this: Provisioning with 'local-exec'...
# module.null.null_resource.this (local-exec): Executing: ["/bin/sh" "-c" "echo this module"]
# module.null.null_resource.this (local-exec): this module
# module.null.null_resource.this: Creation complete after 0s [id=3893065594330030689]

As you see, the null resource from the module gets created after the other one. And of course, modules can even depend on other modules, but you got the drill.

Do I use depends_on now? Not that much, but back in the day on Terraform 0.10, whenever there were problems related to dependencies this was used to fix them.

Lifecycle Block

In Terraform, a lifecycle block is used to define specific behaviors for a resource during its lifecycle. This block is used to manage the lifecycle of a resource in Terraform, including creating, updating, and deleting resources.

The lifecycle block can be added to a resource block and includes the following arguments:

create_before_destroy : When set to true, this argument ensures that a new resource is created before the old one is destroyed. This can help avoid downtime during a resource update.
prevent_destroy : When set to true, this argument prevents a resource from being destroyed. This can be useful when you want to protect important resources from being accidentally deleted.
ignore_changes : This argument specifies certain attributes of a resource that Terraform should ignore when checking for changes. This can be useful when you want to prevent Terraform from unnecessarily updating a resource.
replace_triggered_by: This is relatively new, came up in Terraform 1.2, and it is used to replace a resource if any attributes of that resource have changed, or even other resources have changed. Also, if you use count or for_each on the resource, you can even retrigger the recreation if there is a change to an instance of that resource (using count.index or each.key)

To be honest, I’ve used prevent_destroy only once or twice, ignore_changes,and create_before_destroy a couple of times, and I’ve just found out about replace_triggered_by as I was writing the article.

Let’s see a lifecycle block in action:

resource "kubernetes_namespace" "first" {
  metadata {
    name = "first"
    labels = {
      color = "green"
    }
  }
  lifecycle {
    ignore_changes = [
      metadata[0].labels
    ]
  }
}

output "namespace_labels" {
  value = kubernetes_namespace.first.metadata[0].labels
}


# kubernetes_namespace.first: Creating...
# kubernetes_namespace.first: Creation complete after 0s [id=first]

# Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

# Outputs:

# namespace_labels = tomap({
#   "color" = "green"
# })

I’ve applied the above code for the first time and created the namespace with the above labels. In the lifecycle block, I’ve added the necessary configuration to ignore changes related to labels.

Now, let’s suppose someone comes in and tries to make a change to the labels:

resource "kubernetes_namespace" "first" {
  metadata {
    name = "first"
    labels = {
      color = "blue"
    }
  }
  lifecycle {
    ignore_changes = [
      metadata[0].labels
    ]
  }
}

output "namespace_labels" {
  value = kubernetes_namespace.first.metadata[0].labels
}


# kubernetes_namespace.first: Refreshing state... [id=first]

# No changes. Your infrastructure matches the configuration.

# Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.

# Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

# Outputs:

# namespace_labels = tomap({
#   "color" = "green"
# })

I’ve tried to reapply the code, but Terraform detects there is no change required due to the fact that we have the lifecycle block in action.

Let’s add to the lifecycle block a create_before_destroy option. With this option enabled it doesn’t mean that you won’t ever be able to destroy the resource, but whenever there is a change that dictates the resource has to be recreated, it will first create the new resource and after that delete the existing one.

With the namespace resource already created, I’ve changed its name to induce a breaking change:

# kubernetes_namespace.first must be replaced
+/- resource "kubernetes_namespace" "first" {
      ~ id = "first" -> (known after apply)

      ~ metadata {
          - annotations      = {} -> null
          ~ generation       = 0 -> (known after apply)
          ~ name             = "first" -> "second" # forces replacement
          ~ resource_version = "816398" -> (known after apply)
          ~ uid              = "684f7401-1554-46fb-b21f-8e49329e76fa" -> (known after apply)
            # (1 unchanged attribute hidden)
        }
    }

Plan: 1 to add, 0 to change, 1 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

kubernetes_namespace.first: Creating...
kubernetes_namespace.first: Creation complete after 0s [id=second]
kubernetes_namespace.first (deposed object 725799ef): Destroying... [id=first]
kubernetes_namespace.first: Destruction complete after 6s

Apply complete! Resources: 1 added, 0 changed, 1 destroyed.

Outputs:

namespace_labels = tomap({
  "color" = "blue"
})

As you can see from the Terraform apply output, another resource was created first and after that, the old one was deleted.

Overall, depends_on and the lifecycle block can help you in some edge-case situations you may get in with your IaC. I don’t really use them frequently, but sometimes they are a necessary evil.

Orginally posted on Medium here.

13. Dynamic Blocks

Dynamic Blocks in Terraform let you repeat configuration blocks inside of a resource based on a variable/local/expression that you are using inside of them. They make your configuration DRY (Don’t Repeat Yourself).

Oh, I remember the days before they introduced this feature. I was working for Oracle at the time and was in charge of building reusable modules for our cloud components. Let’s get this straight, you CANNOT truly build reusable modules without dynamic blocks. It really is impossible, it’s like you would say that humans can breathe underwater without any equipment.

How Dynamic Blocks work

In a dynamic block, you can use the following parameters:

for_each (required) → iterates over the value you are providing
content (required) → block containing the body of each block that you are going to create
iterator (optional) → temporary variable used as an iterator
labels (optional) → list of strings that define the block labels. Never used them, tbh.

You can have nested dynamic blocks, or you can use dynamic blocks to avoid generating an optional block inside configurations.

Let’s take a look at the following example:

resource "helm_release" "example" {
  name       = "my-redis-release"
  repository = "https://charts.bitnami.com/bitnami"
  chart      = "redis"

  set {
    name  = "cluster.enabled"
    value = "true"
  }

  set {
    name  = "metrics.enabled"
    value = "true"
  }

  set {
    name  = "service.annotations.prometheus\\.io/port"
    value = "9127"
    type  = "string"
  }
}

The above block is just a simplified version taken out of Terraform’s documentation and as you can see it repeats the set block three times. This set block, for a helm release, is just used to add custom values to be merged with the yaml values file.

Let’s see how we can rewrite this using dynamic blocks:

locals {
  set = {
    "cluster.enabled" = {
      value = true
    }
    "metrics.enabled" = {
      value = true
    }
    "service.annotations.prometheus\\.io/port" = {
      value = "9127"
      type  = "string"
    }
  }
}

resource "helm_release" "example" {
  name       = "my-redis-release"
  repository = "https://charts.bitnami.com/bitnami"
  chart      = "redis"

  values = []

  dynamic "set" {
    for_each = local.set
    content {
      name  = set.key
      value = set.value.value
      type  = lookup(set.value, "type", null)
    }
  }
}

The above code accomplishes the same thing, as the one that was repeating the blocks three times. Since we are using the for_each on the local variable, we are going to create the block three times.

When you are not defining an iterator, your iterator name will be exactly the name of the block, in our case is set.

Let’s use an iterator to make this clear. The dynamic block will change to:

dynamic "set" {
    for_each = local.set
    iterator = i
    content {
      name  = i.key
      value = i.value.value
      type  = lookup(i.value, "type", null)
    }
  }

As mentioned before, you can use dynamic blocks to avoid generating blocks altogether, so to achieve this in our example, what we can do is just make the local variable empty:

locals {
  set = {}
}
resource "helm_release" "example" {
  name       = "my-redis-release"
  repository = "https://charts.bitnami.com/bitnami"
  chart      = "redis"

  values = []

  dynamic "set" {
    for_each = local.set
    iterator = i
    content {
      name  = i.key
      value = i.value.value
      type  = lookup(i.value, "type", null)
    }
  }
}

The OCI Security List Problem

A security list in Oracle Cloud Infrastructure is pretty similar to a network access control list in AWS. When I started building the initial network module for OCI, security groups were not available so the only way to restrict network-related rules was to use security lists. I was using Terraform 0.11 and in that version, there weren’t that many features available (no dynamic blocks, no for_each) so it was pretty hard to build a reusable module.

Because security list rules were embedded inside the security list resource as blocks, it was impossible to make something generic out of them.

So basically, there was no real module built, before dynamic blocks, but after they were released, this bad boy was created:

resource "oci_core_security_list" "this" {
  for_each       = var.sl_params
  compartment_id = oci_core_virtual_network.this[each.value.vcn_name].compartment_id
  vcn_id         = oci_core_virtual_network.this[each.value.vcn_name].id
  display_name   = each.value.display_name

  dynamic "egress_security_rules" {
    iterator = egress_rules
    for_each = each.value.egress_rules
    content {
      stateless   = egress_rules.value.stateless
      protocol    = egress_rules.value.protocol
      destination = egress_rules.value.destination
    }
  }

  dynamic "ingress_security_rules" {
    iterator = ingress_rules
    for_each = each.value.ingress_rules
    content {
      stateless   = ingress_rules.value.stateless
      protocol    = ingress_rules.value.protocol
      source      = ingress_rules.value.source
      source_type = ingress_rules.value.source_type

      dynamic "tcp_options" {
        iterator = tcp_options
        for_each = (lookup(ingress_rules.value, "tcp_options", null) != null) ? ingress_rules.value.tcp_options : []
        content {
          max = tcp_options.value.max
          min = tcp_options.value.min
        }
      }
      dynamic "udp_options" {
        iterator = udp_options
        for_each = (lookup(ingress_rules.value, "udp_options", null) != null) ? ingress_rules.value.udp_options : []
        content {
          max = udp_options.value.max
          min = udp_options.value.min
        }
      }
    }
  }
}

As you can see, in this one, I’m also using nested dynamics, but because I’m using lookups, you don’t even need to specify the “tcp_options” or “udp_options” inside the “ingress_rules” if you don’t want to specify them for one of your rules. This could’ve been done more elegantly using optionals.

Keep in mind this was developed more than 2.5 years ago, back in my Oracle days.

You can also check the OCI Adoption framework Thunder, for more Oracle Cloud modules, but again, this is more than 2.5 years old so it will need an update.

I’m also planning to revive it and update the modules to the latest features available, so send me a message if you want to take part in this revamp.

Dynamic blocks offer a powerful way to manage complex infrastructure resources in Terraform and have become a key feature of the tool since their introduction in version 0.12. By enabling you to generate blocks dynamically based on input data, dynamic blocks make it easier to automate infrastructure management and eliminate manual processes, making Terraform an even more powerful and flexible infrastructure-as-code tool.

Originally posted on Medium here.

14. Terraform Modules

Terraform modules are one of the most important features that Terraform has to offer. They make your code reusable, can be easily versioned and shared with others, and act as blueprints for your infrastructure.

In this post, I am just going to scratch the surface of Terraform modules, I want to go into more detail in the last two articles of this series.

I like to think of Terraform modules as you would think about an Object Oriented Programming Class. In OOP, you define a class and after that, you can create multiple objects out of it. The same goes for Terraform modules, you define the module, and after that, you can reuse it as many times as you want.

Why should you use modules?

There are several reasons to use Terraform modules in your IaC projects:

Code Reusability : Modules help you avoid duplicating code by allowing you to reuse configurations across multiple environments or projects. This makes your infrastructure code more maintainable and easier to update.
Separation of Concerns : Modules enable you to separate your infrastructure into smaller, more focused units. This results in cleaner, more organized code that is easier to understand and manage.
Versioning : I cannot stress the importance of versioning enough. Modules support versioning and can be shared across teams, making it easier to collaborate and maintain consistency across your organization’s infrastructure.
Simplified Configuration : By encapsulating complexity within modules, you can simplify your root configurations, making them easier to read and understand.
Faster Development : With modules, you can break down your infrastructure into smaller, reusable components. This modular approach accelerates development, as you can quickly build upon existing modules rather than starting from scratch for each new resource or environment.
Scalability : Modules enable you to build scalable infrastructure by allowing you to replicate resources or environments easily. By reusing modules, you can ensure that your infrastructure remains consistent and manageable even as it grows in size and complexity.

Minimal module structure

A typical module should contain the following files:

main.tf: Contains the core resource declarations and configurations for the module.
variables.tf: Defines input variables that allow users to customize the module's behavior.
outputs.tf: Specifies output values that the module returns to the caller, providing information about the created resources.
README.md: Offers documentation on how to use the module, including descriptions of input variables and outputs.

What I also like to do, when I’m building modules, is to create examples for those modules.

So in each module, what I typically do, is create an examples folder in which I define at least a main.tf in which I create an object for that module.

For the Readme file, I usually leverage terraform-docs to get the documentation automated, but nevertheless, I also explain what the module does, how to leverage the examples and deep dive into why I took some decisions related to the code.

Module Example

Let’s create a simple module for generating config maps in Kubernetes.

# Module main code

resource "kubernetes_namespace" "this" {
  for_each = var.namespaces
  metadata {
    name        = each.key
    labels      = each.value.labels
    annotations = each.value.annotations
  }
}

resource "kubernetes_config_map" "this" {
  for_each = var.config_maps
  metadata {
    name        = each.key
    namespace   = each.value.use_existing_namespace ? each.value.namespace : kubernetes_namespace.this[each.value.namespace].metadata[0].name
    labels      = each.value.labels
    annotations = each.value.annotations
  }

  data        = each.value.data
  binary_data = each.value.binary_data
}


# Module Variable code

variable "namespaces" {
  description = "Namespaces parameters"
  type = map(object({
    labels      = optional(map(string), {})
    annotations = optional(map(string), {})
  }))
  default = {}
}

variable "config_maps" {
  description = "Config map parameters"
  type = map(object({
    namespace              = string
    labels                 = optional(map(string), {})
    annotations            = optional(map(string), {})
    use_existing_namespace = optional(bool, false)
    data                   = optional(map(string), {})
    binary_data            = optional(map(string), {})
  }))
}


# Module outputs code

output "config_maps" {
  description = "Config map outputs"
  value       = { for cfm in kubernetes_config_map.this : cfm.metadata[0].name => { "namespace" : cfm.metadata[0].namespace, "data" : cfm.data } }
}

The above module code will create how many namespaces and config maps you want in your Kubernetes cluster. You can even create your config maps in existing namespaces, as you are not required to create namespaces if you don’t want to.

I’m taking advantage of optionals, to avoid passing parameters in some of the cases.

# example main.tf code

provider "kubernetes" {
  config_path = "~/.kube/config"
}

module "config_maps" {
  source = "../"
  namespaces = {
    ns1 = {
      labels = {
        color = "green"
      }
    }
  }

  config_maps = {
    cf1 = {
      namespace = "ns1"
      data = {
        api_host = "myhost:443"
        db_host  = "dbhost:5432"
      }
    }
  }
}

# example outputs.tf code
output "config_maps" {
  description = "Config map outputs"
  value       = module.config_maps.config_maps
}

In the above example, I am creating one namespace and one config map into that namespace and I am also outputting the config maps.

Example terraform apply

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

module.config __maps.kubernetes__ namespace.this["ns1"]: Creating...
module.config __maps.kubernetes__ namespace.this["ns1"]: Creation complete after 0s [id=ns1]
module.config __maps.kubernetes__ config __map.this["cf1"]: Creating...
module.config__ maps.kubernetes __config__ map.this["cf1"]: Creation complete after 0s [id=ns1/cf1]

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

Outputs:

config __maps = {
  "cf1" = {
    "data" = tomap({
      "api__ host" = "myhost:443"
      "db_host" = "dbhost:5432"
    })
    "namespace" = "ns1"
  }
}

You can check out the repository and also the generated Readme.md file here.

Publishing Modules

You can easily publish your modules to a Terraform registry. If you want to share your module with the community, you can easily leverage Terraform’s Public Registry.

However, if you have an account with a sophisticated deployment tool such as Spacelift, you can take advantage of the private registry it offers and take advantage of the built-in testing capabilities.

Minimal Best Practices

If you are just starting to build Terraform module, take into consideration the following best practices:

Keep modules focused : Each module should have a specific purpose and manage a single responsibility. Avoid creating overly complex modules that manage multiple unrelated resources.
Use descriptive naming : Choose clear and descriptive names for your modules, variables, and outputs. This makes it easier for others to understand and use your module.
Document your modules : Include a README.md file in each module that provides clear instructions on how to use the module, input variables, and output values. In addition to this, use comments in your Terraform code to explain complex or non-obvious code.
Version your modules : Use version tags to track changes to your modules and reference specific versions in your configurations. This ensures that you’re using a known and tested version of your module, and it makes it easier to roll back to a previous version if needed.
Test your modules : Write and maintain tests for your modules to ensure they work as expected. The terraform validate and terraform plancommands can help you identify configuration errors, while other tools like Spacelift’s built-in module testing or Terratest can help you write automated tests for your modules.

Terraform modules are a powerful way to streamline your infrastructure management, making your IaC more reusable, maintainable, and shareable. By following best practices and leveraging the power of modules, you can improve your DevOps workflow and accelerate your infrastructure deployments.

Originally posted on Medium here.

15. Best practices for modules I

In order to build reusable Terraform modules that can be easily leveraged to achieve almost any architecture, I believe that at least the following best practices should be put in place:

Each Terraform module should exist in its own repository
Use for_each and map variables
Use dynamic blocks
Use ternary operators and take advantage of terraform built-in functions, especially lookup, merge, try, can
Build outputs
Optional: Use pre-commit

Terraform module in its own repository

You may wonder why I am suggesting that every terraform module should be in its own repository.

Well, doing so, will most definitely help you, when you are developing new features to that module because you won’t need to do big changes in a lot of places.

In addition to this, creating/updating repositories with new features of that module will occur really fast, and you will still have the possibility to keep the older versions in place for other automation that you are building.

You can easily reference a git source/registry source when you want to use a module, but tagging is essential.

E.G. Referencing a Github Terraform Module:

module "example" {  
  source = "git@github.com:organisation(user)/repo.git?ref=tag/branch"
}

When it comes to tagging, my recommendation is to use the following tag format: vMajor.Minor.Patch.

Use for_each and map variables

I didn’t specify count, did I?

Using count on your resources will cause more problems than offering help in the end. I’m not talking about checking if a variable is set to true and if it is, set the count to 1, else set it to 0 (not a big fan of this approach, either), I’m talking about creating multiple resources based on a list variable.

Let’s suppose you want to create 3 ec2 instances in AWS using Terraform, each of them with different images, different types, and different azs, similar to the code below.

locals {
  vm_instances = [
    {
      ami           = "ami-1"
      az            = "eu-west-1a"
      instance_type = "t2.micro"
    },
    {
      ami           = "ami-2"
      az            = "eu-west-1b"
      instance_type = "t3.micro"
    },
    {
      ami           = "ami-3"
      az            = "eu-west-1c"
      instance_type = "t3.small"
    }
  ]
}resource "aws_instance" "this" {
  count             = length(local.vm_instances)
  ami               = local.vm_instances[count.index].ami
  availability_zone = local.vm_instances[count.index].az
  instance_type     = local.vm_instances[count.index].instance_type
}

The code is working properly, all of your 3 instances are created and everything is looking smooth after you apply it.

Let’s suppose, for some reason, you want to delete the second instance. What is going to happen to the third one? It will be recreated because that’s how a list variable works if you remember from the 7th article in this series. When you are removing the second instance (index 1), the third instance (index 2) will change its index to 1.

This is just a simple example, but how about a case in which you had 20 instances, and for whatever reason you needed the remove the first one in the list? All the other 19 instances would be recreated and that downtime could really affect you.

By leveraging for_each you can avoid this pain, and due to the fact that it exposes an each.key and each.value when you are using maps, this will greatly help you achieve almost any architecture you desire.

locals {
  vm_instances = {
    vm1 = {
      ami           = "ami-1"
      az            = "eu-west-1a"
      instance_type = "t2.micro"
    }
    vm2 = {
      ami           = "ami-2"
      az            = "eu-west-1b"
      instance_type = "t3.micro"
    }
    vm3 = {
      ami           = "ami-3"
      az            = "eu-west-1c"
      instance_type = "t3.small"
    }
  }
}resource "aws_instance" "this" {
  for_each          = local.vm_instances
  ami               = each.value.ami
  availability_zone = each.value.az
  instance_type     = each.value.instance_type
}

The code is pretty similar but it is now using a map variable instead of a list one, and by removing an instance now, the others are not going to be affected in any way. In the above example, the keys are vm1, vm2, and vm3 and the values are the ones that are after the equal sign.

By using the for_each approach, you have the possibility of creating the resources in any way you want, reaching a more generic state and accommodating many use cases.

Keep in mind that both examples from above are not an actual representation of modules, they are just used to point out the differences between for_each and count.

Use dynamic blocks

I remember when dynamic blocks weren’t a thing in Terraform, and I had to build a module for security lists in Oracle Cloud.

The problem was that security list rules were blocks inside of the security list resource itself, so whatever we would’ve done, that module, in the end, wasn’t very reusable.

One can argue that we could’ve prepared some jinja2 templates, had a script in place that would render those based on an input, but still, in my opinion, that isn’t a reusable terraform module.

Using dynamic blocks, you can easily repeat a block inside of a resource how many times you want based on the input variable, and that most certainly was a game changer at the time of the release.

The good part is that you can have a dynamic block in place, even when you don’t want to create that block at all. In some architectures, you will need a feature enabled many times, but in others, you won’t need that feature at all. Why build two separate modules for that when you can have only one, right?

Use Ternary operators and Terraform functions

Terraform functions complicate the code. That is most certainly a fact and if somebody doesn’t have that much experience with Terraform they will have a hard time reading the code.

Nevertheless, when you are building reusable modules, you will most likely encounter the need of having some conditions inside of your code. You will see there are some arguments that cannot exist with other ones, so for your module to accommodate both use cases, you will most likely need to leverage a ternary operator with a lookup on the variable.

Build Outputs

Inside of a module, an output is literally what that module is exposing for other modules to use.

Make sure that whenever you are building a module, you are exposing the component that can be reused by other components (e.g. subnet ids, because they may be used by vms).

By using for_each, exposing an output will become harder, but don’t worry, you will get the hang of it pretty fast.

output "kube_params" {
  value = { for kube in azurerm_kubernetes_cluster.this : kube.name => { "id" : kube.id, "fqdn" : kube.fqdn } }
}

In the above example, I am exposing some azure kubernetes cluster outputs, that I consider relevant. Due to the fact that my resource has a for_each on it, I have to cycle through all of the resources of that kind to be able to access the arguments. I am creating a map variable with a name key that underneath will have an id and fqdn as part of its values.

Terraform documentation helps you a lot in knowing what a resource can export so you will only need to go to the documentation page of that particular resource.

Using Pre-commit

Pre-commit can easily facilitate your terraform development and can make sure that your code respects the standard imposed by your team before you are pushing the code to the repository.

By using pre-commit, you can easily make sure that:

your terraform code is valid
linting was done properly
documentation for all your parameters has been written

There are other things that can be done, so you should check the documentation for the following:

In order to take advantage of pre-commit, you should define a file called .pre-commit-config.yaml inside of your repository with content similar to this:

repos:
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.3.0
    hooks:
      - id: end-of-file-fixer
      - id: trailing-whitespace
  - repo: https://github.com/antonbabenko/pre-commit-terraform
    rev: v1.72.2
    hooks:
      - id: terraform_fmt
      - id: terraform_tflint
      - id: terraform_docs
      - id: terraform_validate

Of course, many other hooks can be added, but in the example from above, we are using just the essentials.

You should have pre-commit installed locally because as its name states, you should run it prior to making a commit. When you are doing so, all of the problems related to your code will be mentioned and some of them will even be fixed.

Originally posted on Medium here.

16. Best practices for modules II

In this last post of this series, I am going to show you how I build reusable Terraform modules by respecting the best practices.

What I’m going to build is:

A Kubernetes Module for Azure (AKS)
A Helm Module that will be able to deploy helm charts inside of Kubernetes clusters
A repository that leverages the two modules in order to deploy a Helm chart inside of AKS

Prerequisites

In order to follow this tutorial yourself, you are going to need some experience with Terraform, Kubernetes, Helm, pre-commit, and git.

I would suggest installing all of them by following the instructions present in the tools documentation:

Terraform
Kubectl
Helm
Pre-commit
azure cli
A text editor of you choice (I’m using Visual Studio Code, but feel free to use any editor you like)

You should have an Azure account. In order to create a free one you can go here.

AKS Module

I will first start by leveraging my terraform-module-template and create a repository from it, by clicking on Use this template.

After this, I just cloned my repository, and now I’m prepared to start the development.

By using the above template, I have the following folder structure for the Terraform code:

.
├── README.md # Module documentation
├── example # One working example based on the module
│   ├── main.tf # Main code for the example
│   ├── outputs.tf # Example outputs
│   └── variables.tf # Example variables
├── main.tf # Main code for the module
├── outputs.tf # Outputs of the module
├── provider.tf # Required providers for the modules
└── variables.tf # Variables of the module

The starting point of the development will be the root main.tf file.

Let’s first understand what we have to build. To do that, Terraform documentation is our best friend.

I navigated to the resource documentation and from the start, I looked at an example to understand the minimum parameters I require in order to build the module.

As I just want to build a simple module for demo purposes and without too many fancy features, I will just get the absolute necessary parameters. In the real world , this is not going to happen, so make sure you give a thorough reading of the Argument Reference of the resource page.

resource "azurerm_kubernetes_cluster" "this" {
  for_each            = var.kube_params
  name                = each.value.name
  location            = each.value.rg_location
  resource_group_name = each.value.rg_name
  dns_prefix          = each.value.dns_prefix

I’ve started pretty small, just adding some essential parameters that are required to create a cluster.

A variable called kube_params was defined in order to add all of our cluster related parameters. All the parameters from above will be mandatory in our module, due to the fact that I’ve defined them with each.value. something, where something can be whatever you want (just add something that makes sense for that particular parameter).

I’ve continued to add some mandatory blocks inside of this resource:

default_node_pool {
    enable_auto_scaling = each.value.enable_auto_scaling
    max_count           = each.value.enable_auto_scaling == true ? each.value.max_count : null
    min_count           = each.value.enable_auto_scaling == true ? each.value.min_count : null
    node_count          = each.value.node_count
    vm_size             = each.value.vm_size
    name                = each.value.np_name
  }

  dynamic "service_principal" {
    for_each = each.value.service_principal
    content {
      client_id     = service_principal.value.client_id
      client_secret = service_principal.value.client_secret
    }
  }

  dynamic "identity" {
    for_each = each.value.identity
    content {
      type         = identity.value.type
      identity_ids = identity.value.identity_ids
    }
  }
  tags = merge(var.tags, each.value.tags)
}

In the above example, you are going to see many of the best practices that I’ve mentioned in my previous post: the use of dynamic blocks, ternary operators, and functions.

For the dynamic blocks, by default, I am not going to create any block if the parameter is not present.

For the tags, I’ve noticed that in many cases, companies want to be able to add some global tags to resources and, of course, individual tags for a particular resource. In order to accommodate that, I’m simply merging two map variables: one that exists inside of kube_params and another one that will exist in the tags var.

As some may want to export the kubeconfig of the cluster after it is created, I’ve provided that option also:

resource "local_file" "kube_config" {
  for_each = { for k, v in var.kube_params : k => v if v.export_kube_config == true }
  filename = each.value.kubeconfig_path
  content  = azurerm_kubernetes_cluster.this[each.key].kube_config_raw
}

To export the kubeconfig, you will simply need to have a parameter in kube_params called export_kube_config set to true.

This is how my main.tf file looks like in the end:

resource "azurerm_kubernetes_cluster" "this" {
  for_each            = var.kube_params
  name                = each.value.name
  location            = each.value.rg_location
  resource_group_name = each.value.rg_name
  dns_prefix          = each.value.dns_prefix

  default_node_pool {
    enable_auto_scaling = each.value.enable_auto_scaling
    max_count           = each.value.enable_auto_scaling == true ? each.value.max_count : null
    min_count           = each.value.enable_auto_scaling == true ? each.value.min_count : null
    node_count          = each.value.node_count
    vm_size             = each.value.vm_size
    name                = each.value.np_name
  }

  dynamic "service_principal" {
    for_each = each.value.service_principal
    content {
      client_id     = service_principal.value.client_id
      client_secret = service_principal.value.client_secret
    }
  }

  dynamic "identity" {
    for_each = each.value.identity
    content {
      type         = identity.value.type
      identity_ids = identity.value.identity_ids
    }
  }
  tags = merge(var.tags, each.value.tags)
}

resource "local_file" "kube_config" {
  for_each = { for k, v in var.kube_params : k => v if v.export_kube_config == true }
  filename = each.value.kubeconfig_path
  content  = azurerm_kubernetes_cluster.this[each.key].kube_config_raw
}

For the variables.tf file, I just defined the kube_params and tags variables as they are the only vars that I’ve used inside of my module.

variable "kube_params" {
  type = map(object({
    name                = string
    rg_name             = string
    rg_location         = string
    dns_prefix          = string
    client_id           = optional(string, null)
    client_secret       = optional(string, null)
    vm_size             = optional(string, "Standard_DS2_v2")
    enable_auto_scaling = optional(string, true)
    max_count           = optional(number, 1)
    min_count           = optional(number, 1)
    node_count          = optional(number, 1)
    np_name             = string
    service_principal = optional(list(object({
      client_id     = optional(string, null)
      client_secret = optional(string, null)
    })), [])
    identity = optional(list(object({
      type         = optional(string, "SystemAssigned")
      identity_ids = optional(list(string), [])
    })), [])
    kubeconfig_path = optional(string, "~./kube/config")
  }))
  description = "AKS params"
}

variable "tags" {
  type        = map(string)
  description = "Global tags to apply to the resources"
  default     = {}
}

With the release of optional, types, I’ve not used any type for the variables anymore, because right now we can omit some of the parameters by using the optional function at the variable level. More about that here.

output "kube_params" {
  value = { for kube in azurerm_kubernetes_cluster.this : kube.name => { "id" : kube.id, "fqdn" : kube.fqdn } }
}

output "kube_config" {
  value = { for kube in azurerm_kubernetes_cluster.this : kube.name => nonsensitive(kube.kube_config) }
}

output "kube_config_path" {
  value = { for k, v in local_file.kube_config : k => v.filename }
}

I’ve defined three outputs, one for the most relevant parameters of the Kubernetes cluster and two for the kubeconfig. The ones related to kubeconfig are giving users the possibility to login into the cluster by using the kubeconfig file or the Kubernetes user/password/certificates combination.

You can use functions inside of the outputs and you are encouraged to do so, to accommodate your use case.

The module code is now ready, but you should at least have an example inside of it to show users how can they run the code.

So in the examples folder, in the main.tf file, I’ve added the following code:

provider "azurerm" {
  features {}
}

module "aks" {
  source = "../"
  kube_params = {
    kube1 = {
      name                = "kube1"
      rg_name             = "rg1"
      rg_location         = "westeurope"
      dns_prefix          = "kube"
      identity            = [{}]
      enable_auto_scaling = false
      node_count          = 1
      np_name             = "kube1"
      export_kube_config  = true
    }
  }
}

I’ve added the provider configuration and called the module with some of the essential parameters. You can add a variable to the kube_params and provide values in the variable directly or in a tfvars file or even in an yaml file if you want.

In the above example, you are creating only one kubernetes cluster, if you want more, just copy & paste de kube1 block and add whatever parameters you want, based on the module similar to this:

provider "azurerm" {
  features {}
}

module "aks" {
  source = "../"
  kube_params = {
    kube1 = {
      name                = "kube1"
      rg_name             = "rg1"
      rg_location         = "westeurope"
      dns_prefix          = "kube"
      identity            = [{}]
      enable_auto_scaling = false
      node_count          = 1
      np_name             = "kube1"
      export_kube_config  = true
    }
    kube2 = {
      name                = "kube2"
      rg_name             = "rg1"
      rg_location         = "westeurope"
      dns_prefix          = "kube"
      identity            = [{}]
      enable_auto_scaling = false
      node_count          = 4
      np_name             = "kube2"
      export_kube_config  = false
    }
    kube3 = {
      name                = "kube3"
      rg_name             = "rg1"
      rg_location         = "westeurope"
      dns_prefix          = "kuber"
      identity            = [{}]
      enable_auto_scaling = true
      max_count           = 4
      min_count           = 2
      node_count          = 2
      np_name             = "kube3"
      export_kube_config  = false
    }
  }
}

I’ve added 3 just for reference, you have full control related to the number of clusters you want to create.

Ok, let’s get back to the initial example. Before running the code you should first login to Azure:

az login

You may think we are done, right? Well, not yet, because I like to run pre-commit with all the goodies on my code. I am using the same pre-commit file I mentioned in the previous post so before adding my code, I’m going to run:

pre-commit run --all-files

Before running this command, I made sure that in my README.md I had the following lines:

<!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
<!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->

Between these lines, tfdocs will populate the documentation based on your resources, variables, outputs, and provider.

This is what the README looks like after running the pre-commit command:

Now, this module is done, I’ve pushed all my changes and a new tag was created automatically in my repository (I will discuss about the pipelines I am using for that in another post).

Helm Release Module

All the steps are the same as for the other module, first we have to understand what we have to build by reading the documentation and understanding the parameters.

I’ve created a new repository using the same template and I’ve started the development per se. As I don’t want to repeat all the steps from above in this post, I will show you just the end result of the main.tf file for this module.

resource "helm_release" "this" {
  for_each         = var.helm
  name             = each.value.name
  chart            = each.value.chart
  repository       = each.value.repository
  version          = each.value.version
  namespace        = each.value.namespace
  create_namespace = each.value.create_namespace ? true : false

  values = [for yaml_file in each.value.values : file(yaml_file)]
  dynamic "set" {
    for_each = each.value.set
    content {
      name  = set.value.name
      value = set.value.value
      type  = set.value.type
    }
  }
  dynamic "set_sensitive" {
    for_each = each.value.set_sensitive
    content {
      name  = set_sensitive.value.name
      value = set_sensitive.value.value
      type  = set_sensitive.value.type
    }
  }
}

This module is also pushed, tagged, and good to go.

Automation based on the Two Modules

I’ve created a third repository, but this time from scratch in which I will use the two modules that were built throughout this post.

provider "azurerm" {
  features {}
}

module "aks" {
  source = "git@github.com:flavius-dinu/terraform-az-aks.git?ref=v1.0.3"
  kube_params = {
    kube1 = {
      name                = "kube1"
      rg_name             = "rg1"
      rg_location         = "westeurope"
      dns_prefix          = "kube"
      identity            = [{}]
      enable_auto_scaling = false
      node_count          = 1
      np_name             = "kube1"
      export_kube_config  = true
      kubeconfig_path     = "./config"
    }
  }
}

provider "helm" {
  kubernetes {
    config_path = module.aks.kube_config_path["kube1"]
  }
}

# Alternative way of declaring the provider

# provider "helm" {
#   kubernetes {
#     host                   = module.aks.kube_config["kube1"].0.host
#     username               = module.aks.kube_config["kube1"].0.username
#     password               = module.aks.kube_config["kube1"].0.password
#     client_certificate     = base64decode(module.aks.kube_config["kube1"].0.client_certificate)
#     client_key             = base64decode(module.aks.kube_config["kube1"].0.client_key)
#     cluster_ca_certificate = base64decode(module.aks.kube_config["kube1"].0.cluster_ca_certificate)
#   }
# }

module "helm" {
  source = "git@github.com:flavius-dinu/terraform-helm-release.git?ref=v1.0.0"
  helm = {
    argo = {
      name             = "argocd"
      repository       = "https://argoproj.github.io/argo-helm"
      chart            = "argo-cd"
      create_namespace = true
      namespace        = "argocd"
    }
  }
}

So, in my repository, I am creating one Kubernetes cluster, in which I am deploying an ArgoCD helm chart. Due to the fact that the helm provider in both cases (commented and uncommented code) has an explicit dependency on module.aks.something it will first wait for the cluster to be ready and after that, the helm chart will be deployed on it.

I’ve kept the commented part in the gist because I wanted to showcase the fact that you are able to connect to the cluster using two different outputs, and both solutions are working just fine.

You should use a remote state if you are collaborating or using the code in a production environment. I haven’t done that for this example, because it is just a simple demo.

Now the automation is done, so what we can do is:

terraform init
terraform plan
terraform apply

Both components are created successfully and if you login to your AKS cluster, you are going to see the ArgoCD related resources by running:

kubectl get all -n argocd

Repository Links

Useful Documentation

If you need some help to better understand some of the concepts that I’ve used throughout this post, I’ve put together a list of useful articles and component documentation:

For Each

https://www.terraform.io/language/meta-arguments/for_each

Dynamic blocks

Originally posted on Medium.

17. Bonus1 — OpenTofu features

In this part we will cover some OpenTofu features that Terraform doesn’t have.

So here are the standout features:

State encryption — built-in state encryption, ensuring enhanced security for sensitive infrastructure data. Unlike Terraform, which relies on third-party tools or external state backends for encryption, OpenTofu provides this feature natively, allowing users to secure their state files directly within the platform.
Support for.tofu Files — use of .tofu file extensions, which can override standard .tf files. This feature gives users greater flexibility in managing configurations, enabling easy application of specific overrides or conditional configurations. This functionality isn’t available in Terraform, which relies solely on .tf files.
for_each in Provider Configuration Blocks — making it possible to dynamically create multiple instances of a provider configuration. This feature allows each resource instance to select a different provider configuration based on a list or map, ideal for infrastructure duplication across regions. Terraform does not support this feature in provider configurations.
-exclude Planning Option —allows users to specify objects to skip during planning and application, providing finer control over infrastructure management. Unlike Terraform’s -target option, which only targets specific resources, the -exclude option allows selective exclusion of specified objects and their dependencies. This capability is particularly useful for managing large, complex setups.

18. Bonus2 — Specialized Infrastructure Orchestration Platform

Now that you know how to use Terraform and OpenTofu, there are many questions that will pass through your mind:

How can I manage it at scale?
How do I enable collaboration?
How can I enforce my infrastructure process?
How can I ensure that I stay safe all the way?
How can I enable self-service?
How can I easily combine Terraform/OpenTofu with other tools that I’m using such as Ansible or Kubernetes?
How can I include security vulnerability scanning tools in my workflow?
How can I ensure that I still deliver fast?
How can I protect myself from infrastructure drift?
How can I detect and remediate drift?
How can I take advantage of a private registry?

You may have other questions as well, and to all of them you can use the same answer — “by leveraging an infrastructure orchestration platform”.

Spacelift is an infrastructure management platform that helps you with all of the above, and more. To understand more how you can leverage it, check out this article.

Is this really the Minimum to become a DevOps Engineer?

Flavius Dinu — Tue, 05 Nov 2024 12:36:39 +0000

You’ve asked so I kind of delivered, I guess.

I’ve seen many social media posts saying that DevOps is not a job but a methodology in software development. DevOps started as a methodology and I 100% agree that it should be one, but the thing is that many companies are searching for DevOps engineers, and DevOps engineering roles are here to stay for a while.

To tell you the truth, I didn’t become a DevOps engineer by accident. I saw the methodology, and I thought, ok this seems like it will take off soon. I took a bet, and I’ve started reading about Cloud providers, CI/CD pipelines, configuration management, and infrastructure as code, and I’ve started becoming better in scripting to try and land a job in this area. Luckily the market was in a way better shape than it is today, and the competition wasn’t that big in the beginning, so I managed to transition to a position like this, less than a year into working.

One year ago, I wanted to write a series about what I consider it minimum takes to become a DevOps engineer, but I realized that to cover everything that I planned to cover I would have to write a book, which is something that I don’t really want to commit to — yet. So what I’ll do instead, is try to share as much as possible in this post.

I’ve been blessed (or cursed) to work with many clients that had very different architectures, maturity levels, and were using various tools to achieve their day-to-day DevOps engineering practices, so in this post, I will cover some of the common ground I’ve seen being used, show you some nice learning materials and give you some recommendations for next steps.

Before jumping into that, I want to clarify — you cannot really be a Junior DevOps engineer if the company you are working for is not going to be patient and give you all the time you need to shadow a Senior. So for more experienced guys, please be the senior you needed when you were a junior (I know it sounds like a cliche, but it’s not), as in this way we will make the industry a better place. Stop blaming and start sharing and let’s be truthful to ourselves — there were so few people born extraordinary in this industry, and for the rest of us this is a continuous grind.

Here is the list of skills the majority of DevOps engineers should have:

Linux
Scripting
Version Control (VCS)
Cloud Technologies
Infrastructure as Code (IaC)
Configuration Management
Continuous Integration & Continuous Delivery (CI/CD)
Containerization & Orchestration
Observability & Monitoring
Security and Governance

1. Linux

Is being a pro in the Windows OS wrong? Of course, it’s not, but judging from the fact that even on Microsoft Azure, the majority of VMs are using some sort of Linux distribution, it would be better to dedicate some time to learning Linux, rather than going all in on Windows:

When I was thinking about writing the series one year ago, I’ve built an article that has my take on what you should learn when it comes to Linux, so check it out here.

Continue reading on Medium