DEV Community: Felipe Arcaro

Can I scale my dockerized Flask solution with Kubernetes?

Felipe Arcaro — Sat, 06 Apr 2024 12:14:05 +0000

TL;DR

Yes - with some work.

Here are the steps:

Install Kompose - a conversion tool that allows you to convert your Docker Compose code to Kubernetes configuration files
- Run kompose convert in the same directory as your docker-compose.yml to generate the config files for your Kubernetes cluster
Install Minicube - a tool that allows us to spin up a Kubernetes cluster in a local machine
- Run minikube start to start your Kubernetes cluster
- Run minikube dashboard to spin up a web-based user interface that allows you to manage your Kubernetes cluster
Install kubectl - a CLI tool provided by Kubernetes for communicating with a Kubernetes cluster's control plane using the Kubernetes API
- Run kubectl apply -f <filename> to apply your services/deployments

Let's consider the following scenario - our application has gained popularity and is attracting a large number of users and, to keep up with that, we need a way of handling increased traffic.

We could vertically scale - that is, allocate more resources to our containers:

But instead, we want to horizontally scale it by creating multiple copies of the containers and distributing traffic among them:

And as you probably guessed by the title of this post, we're going to use Kubernetes for that.

Wait... isn't Kubernetes and Docker the same thing?

Not really. Docker focuses on creating and running containers, while Kubernetes is a container orchestration platform that manages the deployment, scaling, and operations of those containers in a cluster.

You can think of Docker as the chef who prepares individual dishes (containers) in the kitchen, ensuring they're well-packaged and portable, while Kubernetes serves as the restaurant manager who orchestrates the dining experience, seating guests (users) at tables (containers), coordinating servers (nodes), and managing the overall flow of the restaurant to ensure a delightful dining service..

What is Kubernetes?

Kubernetes, often referred to as k8s, is an open-source container orchestration engine designed to automate the deployment, scaling, and management of containerized applications. It aims to simplify application management, ensure high availability via replica sets, and facilitate scalability.

By abstracting the underlying infrastructure, Kubernetes offers a declarative approach to define the desired state of applications and their dependencies using YAML files.

Kubernetes' architecture

Deploying Kubernetes creates a cluster, consisting of worker nodes running containerized applications. Each cluster has at least one worker node, hosting Pods, which form the application workload. The control plane manages these nodes and Pods. In production, the control plane typically runs across multiple computers for fault-tolerance and high availability.

Let's take a quick look at Kubernetes' main components.

Control Plane:

kube-apiserver: Exposes the Kubernetes API for cluster management
etcd: Stores all cluster data reliably
kube-scheduler: Assigns nodes for newly created Pods
kube-controller-manager: Runs controller processes to manage cluster state.
cloud-controller-manager: Integrates cloud-specific control logic (if applicable).

Node:

kubelet Ensures containers described in PodSpecs are running and healthy.
kube-proxy: Maintains network rules for network communication to Pods.
Container runtime: Manages execution and lifecycle of containers within Kubernetes.

You can learn more about it on Kubernete's official page.

Getting to work

Enough theory, let's get to work.

For this experiment, we're going to need three tools:

Kompose - a conversion tool that allows us to convert our Docker Compose code to Kubernetes configuration files
Minicube - a tool that allows us to spin up a Kubernetes cluster in a local machine
kubectl - a CLI tool provided by Kubernetes for communicating with a Kubernetes cluster's control plane using the Kubernetes API

Converting our `docker-compose.yml` file

Once we have all of those installed, we will start converting our docker-compose file running kompose convert in the same directory as our docker-compose.yml file.

A few files were created in that directory:

env-configmap.yaml
mongo-db-claim0-persistentvolumeclaim.yaml
mongo-db-claim1-persistentvolumeclaim.yaml
mongo-db-service.yaml
mongo-db-deployment.yaml
app-service.yaml
app-deployment.yaml
schedule-service-claim0-persistentvolumeclaim.yaml
schedule-service-claim1-persistentvolumeclaim.yaml
schedule-service-service.yaml
schedule-service-deployment.yaml

Firing it all up

I am not super familiar with Helm but I think one of its features is to manage dependencies between those files, allowing us to run all of them in a docker-compose up style.

For this post, let's do everything manually and cover the details on the app files.

We'll start exposing our variables:

kubectl apply -f env-configmap.yaml

Then spinning up up our Mongo DB service running:

kubectl apply -f mongo-db-claim0-persistentvolumeclaim.yaml
kubectl apply -f mongo-db-claim1-persistentvolumeclaim.yaml
kubectl apply -f mongo-db-service.yaml
kubectl apply -f mongo-db-deployment.yaml

And now, we can spin up our Flask application:

kubectl apply -f app-service.yaml
kubectl apply -f app-deployment.yaml

Let's take a deeper looker into these two files files:

app-service.yaml:

apiVersion: v1
kind: Service
metadata:
  annotations:
    kompose.cmd: kompose convert
    kompose.version: 1.32.0 (HEAD)
  labels:
    io.kompose.service: app
  name: app
spec:
  ports:
    - name: '8001'
      port: 8001
      targetPort: 8001
  selector:
    io.kompose.service: app

app-deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    kompose.cmd: kompose convert
    kompose.version: 1.32.0 (HEAD)
  labels:
    io.kompose.service: app
  name: app
spec:
  replicas: 3
  selector:
    matchLabels:
      io.kompose.service: app
  template:
    metadata:
      annotations:
        kompose.cmd: kompose convert
        kompose.version: 1.32.0 (HEAD)
      labels:
        io.kompose.network/test-kompose-my-network: 'true'
        io.kompose.service: app
    spec:
      containers:
        - args:
            - flask
            - run
            - '--host'
            - 0.0.0.0
            - '--port'
            - '8001'
          env:
            - name: ENV
              valueFrom:
                configMapKeyRef:
                  key: ENV
                  name: env
            - name: FLASK_SECRET_KEY
              valueFrom:
                configMapKeyRef:
                  key: FLASK_SECRET_KEY
                  name: env
            - name: MONGO_INITDB_DATABASE
              valueFrom:
                configMapKeyRef:
                  key: MONGO_INITDB_DATABASE
                  name: env
            - name: MONGO_INITDB_ROOT_PASSWORD
              valueFrom:
                configMapKeyRef:
                  key: MONGO_INITDB_ROOT_PASSWORD
                  name: env
            - name: MONGO_INITDB_ROOT_USERNAME
              valueFrom:
                configMapKeyRef:
                  key: MONGO_INITDB_ROOT_USERNAME
                  name: env
            - name: MONGO_PASSWORD
              valueFrom:
                configMapKeyRef:
                  key: MONGO_PASSWORD
                  name: env
            - name: MONGO_USER
              valueFrom:
                configMapKeyRef:
                  key: MONGO_USER
                  name: env
            - name: QUOTES_APP_MONGO_CONN
              valueFrom:
                configMapKeyRef:
                  key: QUOTES_APP_MONGO_CONN
                  name: env
            - name: SENDGRID_API_KEY
              valueFrom:
                configMapKeyRef:
                  key: SENDGRID_API_KEY
                  name: env
          image: local-app-image
          imagePullPolicy: IfNotPresent
          name: my-app
          ports:
            - containerPort: 8001
              hostPort: 8001
              protocol: TCP
      restartPolicy: Always

Most parameters here are self-explanatory, but I want to highlight two of them:

imagePullPolicy - we need to set that to Never or IfNotPresent if we intend to run images we built locally. We also need to run eval $(minikube docker-env) in the same terminal I'm running kubectl commands as stated in the official documentation
replicas - we can choose the number of containers we want to spin up (nifty stuff)

Lastly, we spin up the schedule service:

kubectl apply -f schedule-service-claim0-persistentvolumeclaim.yaml
kubectl apply -f schedule-service-claim1-persistentvolumeclaim.yaml
kubectl apply -f schedule-service-service.yaml
kubectl apply -f schedule-service-deployment.yaml

Taking a look at our creation

To see what we just did, we could go old school with kubectl get pods (and some other commands) on the terminal.

But Minikube has a web-based user interface that allows us to manage our Kubernetes'cluster in the browser. We can simply run minikube dashboard and go to this link on any browser.

Here are some of the things we can do in that dashboard:

Monitor the health of our applications and cluster
Create, view, update, and delete Kubernetes resources
View logs of our applications
Access and manage secrets, config maps, and other configurations.
Debug applications directly from the dashboard

Final thoughts

Many folks find Kubernetes to be complex, and some even say that those who praise it might not actually use it regularly.

Although Kubernetes has become easier to use in some ways, setting it up still needs some previous knowledge, especially when it comes to configuring networks for load balancers to handle pods replicas correctly.

But it's pretty amazing how quickly you can get a Kubernetes environment running on your own computer. This is super helpful when you're ready to move our applications to platforms like AWS EKS - it makes testing and development easier and sets you up for a smoother deployment in the real world.

Accessing my home server from a different continent (the hard way)

Felipe Arcaro — Sat, 30 Mar 2024 20:54:27 +0000

TL;DR

I have a home server in Porto, Portugal. When I needed to travel to Brazil for 4 months, I wanted to access my home server from a different network. While there are several tools available for this purpose, I decided to hack my way through it because... why not?

So, I took the following steps:

Wrote a script to periodically send me emails informing me if my home server's external IP had changed
Configured the required port forwarding on my router
Set up SSH keys between my laptop and my home server
Implemented some monitoring

My home server's IP

If you've ever tinkered with your home network, you've probably noticed that each device within your house is assigned an internal IP address, allowing them to communicate with each other. However, all these devices share access to the internet through a single IP address provided by your Internet Service Provider (ISP). You can easily find this external IP address on websites like this, but it's important to note that this IP address might change over time.

While there are tools like DuckDNS available to manage dynamic IP changes, I decided to create my own hacky solution – I wrote a script to periodically check if my IP had changed and send me emails using SendGrid.

#!/bin/bash

# Constants
SENDGRID_API_KEY="MY_KEY"
TO_EMAIL="my_email@gmail.com"
FROM_EMAIL="my_email@gmail.com"
CURRENT_IP="my_ip"

# Get the external IP address using an external service
EXTERNAL_IP=$(curl -s http://api.ipify.org)

# Compose the email content
if [ "$EXTERNAL_IP" == "$CURRENT_IP" ]; then
        EMAIL_BODY="Yo, your home server IP address is still the same: $EXTERNAL_IP"
else
        EMAIL_BODY="Yo, it looks like your home server IP address has changed: $EXTERNAL_IP"
fi

EMAIL_SUBJECT="Home Server - Weekly IP check"

# Create a temporary JSON payload file
TMP_JSON=$(mktemp)
echo '{
  "personalizations": [
    {
      "to": [
        {
          "email": "'"$TO_EMAIL"'"
        }
      ],
      "subject": "'"$EMAIL_SUBJECT"'"
    }
  ],
  "from": {
    "email": "'"$FROM_EMAIL"'"
  },
  "content": [
    {
      "type": "text/plain",
      "value": "'"$EMAIL_BODY"'"
    }
  ]
}' > "$TMP_JSON"

# Send email using SendGrid API
curl -X POST "https://api.sendgrid.com/v3/mail/send" \
     -H "Authorization: Bearer $SENDGRID_API_KEY" \
     -H "Content-Type: application/json" \
     --data-binary @"$TMP_JSON"

# Clean up temporary JSON file
rm "$TMP_JSON"

I scheduled the execution of that script using a cronjob (Linux).

Setting up port forwarding

Remember I mentioned all our home devices' traffic exits through a single IP? Well, it works the other way around too – we can use that same IP to access our home devices, with a little help. Most routers come with a firewall that blocks incoming traffic for security reasons.

However, they also offer port forwarding, a feature that allows us to specify rules like "if someone tries to access my IP on port 123, redirect them to this internal IP (specific device) on the same port".

Setting up port forwarding involves accessing our router's settings through a web browser, logging in with your credentials, and configuring the port forwarding rules accordingly. SSH (Secure Shell) connections typically use port 22by default. I decided to use a less common port to avoid attacks targeting the default port.

With port forwarding in place, we are ready to access my home server using SSH with ssh my_user@my_external_ip and entering my password, right? Well, technically yes, but passwords are no fun (and not really secure). That's why it's time to set up SSH keys for extra security.

Configuring SSH keys

Creating keys on my laptop

First, I generated SSH keys on my laptop by running the following command in the terminal:

ssh-keygen -t rsa -b 4096 -C "email@example.com"

By default, both the public and private keys are stored in our home directory within a hidden folder named .ssh. The private key is typically named id_rsa, while the public key has the same name but with a .pub extension (id_rsa.pub).

To view and copy the newly created public key, we can use the command cat ~/.ssh/id_rsa.pub, which will display something like this: ssh-rsa <long-string> email@example.com

Adding the public key to my home server

Next, now on my home server, I log in with the desired user for SSH connections, and create the .ssh folder and authorized_keys file if they don't already exist:

mkdir -p /home/my_user/.ssh && touch /home/my_user/.ssh/authorized_keys

Then, I paste my public key on authorized_keys file using a text editor.

Finally, we ensure that the permissions on the ~/.ssh directory and the authorized_keys file are correctly set for SSH to use them:

chmod 700 /home/my_user/.ssh
chmod 600 /home/my_user/.ssh/authorized_keys

These permissions restrict access to the .ssh directory and its contents to only the user associated with it.

Removing password access

Now that we have properly set up the SSH keys, it's time to remove the password access. For that, we simply change set PasswordAuthentication no inside /etc/ssh/sshd_config and restart the SSH service with sudo service ssh restart.

Monitoring

Although this workflow seems fairly secure, it's always a good idea monitor SSH access. My home server runs on Ubuntu, so I SSH (and other) activity is logged on /var/log/auth.log file.

I created a simple script to check for any changes in that file:

#!/bin/bash

# Constants
SENDGRID_API_KEY="MY_KEY"
TO_EMAIL="my_email@gmail.com"
FROM_EMAIL="my_email@gmail.com"
CURRENT_IP="my_ip"

send_email_notification() {

        # Create a temporary JSON payload file
        TMP_JSON=$(mktemp)
        echo '{
          "personalizations": [
            {
              "to": [
                {
                  "email": "'"$TO_EMAIL"'"
                }
              ],
              "subject": "'"$EMAIL_SUBJECT"'"
            }
          ],
          "from": {
            "email": "'"$FROM_EMAIL"'"
          },
          "content": [
            {
              "type": "text/plain",
              "value": "'"$EMAIL_BODY"'"
            }
          ]
        }' > "$TMP_JSON"

        # Send email using SendGrid API
        curl -X POST "https://api.sendgrid.com/v3/mail/send" \
             -H "Authorization: Bearer $SENDGRID_API_KEY" \
             -H "Content-Type: application/json" \
             --data-binary @"$TMP_JSON"

        # Clean up temporary JSON file
        rm "$TMP_JSON"
}

# Define the critical file to monitor
ssh_logs_file="/var/log/auth.log"

# Store the initial checksums
initial_md5sum=$(md5sum "$ssh_logs_file")
echo "Initial md5sum: $initial_md5sum"

while true; do
    # Calculate the current checksums
    current_md5sum=$(md5sum "$ssh_logs_file")

    # Compare checksums to detect changes
    if [[ "$initial_md5sum" != "$current_md5sum" ]]; then
        send_email_notification
        echo "Oops, it seems like checksum is different, new logs were added to ssh auth.log"
        echo "Initial md5sum: $initial_md5sum"
        echo "Current md5sum: $current_md5sum"
        initial_md5sum="$current_md5sum"
    fi

    # Adjust the sleep interval
    sleep 300
done

Then I ran the script

 chmod +x check-ssh-logs.sh 
 nohup ./check-ssh-logs.sh &

nohup is a nifty command used to run a process immune to hangups, allowing it to continue running even after logging out

Then I got married, traveled back to Portugal, had to move to a different apartment and haven't turned on my home server ever since.

Dockerizing a Flask application

Felipe Arcaro — Sat, 30 Mar 2024 11:46:29 +0000

TL;DR

Here's the docker-compose.yml file to dockerize a Flask application, a Mongo database and a schedule service:

version: '3.8'

services:
  app:
    container_name: my_app_${ENV}
    build:
      context: . 
    ports:
      - 8001:8001
    depends_on:
      - mongo_db
    networks:
      - my_network
    command: ["flask", "run", "--host", "0.0.0.0", "--port", "8001"]
    env_file: .env

  schedule_service:
    container_name: schedule_service_${ENV}
    build:
      context: ./scheduler
    volumes:
      - ./common:/app/common
      - ./db:/app/db  
    networks:
      - my_networknetwork
    command: ["python", "-u", "scheduler.py"] 
    env_file: .env

  mongo_db:
    container_name: mongo_db_${ENV}
    image: mongo:4.4.18
    restart: always
    ports:
      - 27017:27017
    volumes:
      - ./DB-STORAGE/mongo-db-$ENV:/data/db
      - ./db/init-mongo.sh:/docker-entrypoint-initdb.d/init-mongo.sh
    networks:
      - my_network
    env_file: .env

networks:
  my_network:  
    driver: bridge

Run the following command from your project's root directory docker-compose up --build -d to fire it all up.

What is Docker?

Docker is a tool that helps us package up our apps so they can run smoothly no matter where they're deployed – it's like putting our app in a virtual box that contains everything it needs to work - code, settings, and libraries. These boxes are called "containers," and they're lightweight and easy to move around.

With Docker, there's no such a thing as "but it works on my machine..."

Getting started with Dockerfile and Docker Compose

To spin up a container, we can do it directly from the terminal or we can create a file called Dockerfile, which works like a recipe for our app's container. In the Dockerfile, we specify what our app needs to run, like the programming language and dependencies.

If we want to spin up more containers at once, we can use Docker Compose. Docker Compose reads a configuration file – usually called docker-compose.yml, that describes all the containers our app needs to run and how they should interact.

Why should we dockerize an application?

Dockerizing an application offers several benefits:

It enhances collaboration and streamlines development workflows – we can work in isolated environments without conflicts, speeding up development and making it easier to onboard new team members
It makes our app more portable – we can easily move it between different environments without worrying about compatibility issues which simplifies deployment and ensures consistency across different platforms
It improves scalability and resource management - we can easily start/stop containers instances to accommodate fluctuations in traffic

Dockerizing a Flask application

Let's say we've got a Flask application with four main components:

network: custom Docker network
app: our Flask application
mongo_db: a Mongo database
schedule: an email schedule service

Let's break the Docker Compose code down to understand these components at parameter level.

network

networks:
  quot-network:  
    driver: bridge

networks is used to define custom Docker networks in Docker Compose, it allows us to create separate networks for our services, and it helps facilitate communication between the containers running on those networks
my_network is the name of the custom network being defined
driver specifies the network driver to be used for the custom network

A few notes on Docker networks:
- The bridge driver is the default network driver in Docker and is suitable for most use cases. It enables communication between containers on the same host and provides automatic DNS resolution for service discovery. Each custom network with the "bridge" driver is isolated from the host's default bridge network and other custom bridge networks
- When using the bridge driver, containers on the same network can communicate with each other using their container names as hostnames (e.g., schedule)
- Using the host network instead of bridge allows a container to share the network namespace with the host system. This means the container shares the host's network stack, and network ports used by the container are directly mapped to the host

app

version: '3.8'

services:
  app:
    container_name: my_app_${ENV}
    build:
      context: . 
    ports:
      - 8001:8001
    depends_on:
      - mongo_db
    networks:
      - my_network
    command: ["flask", "run", "--host", "0.0.0.0", "--port", "8001"]
    env_file: .env

version: '3.8' specifies the version of the Docker Compose file syntax being used - in this case, it's version 3.8
services is the top-level key that defines the services/containers that will be managed by Docker Compose
app the name of the service/container being defined
container_name specifies the custom name for the container that will be created based on this service. The variable ${ENV} dynamically sets the suffix based on an environment variable. For example, if the value of ${ENV} is "production", the container name will be my_app_production.
- The .env file should be placed at the root of the project directory next to our docker-compose.yml
build indicates that the service will be built using a Dockerfile located in the current directory (denoted by .). The context parameter defines the build context, which means the current directory and its subdirectories will be used to find the necessary files for the build
ports exposes ports from the container to the host - it will map port 8001 from the host to port 8001 in the container, that means any traffic coming to port 8001 on the host will be forwarded to port 8001 of the container
depends_on specifies that this service depends on another service called mongo_db. It ensures that the mongo_db is up and running before this service starts
networks attaches the service to a pre-existing Docker network, this allows both the app service and other services connected to communicate with each other
command overrides the default command in the Dockerfile that would be executed when the container starts. The app will run with the following parameters: flask run --host 0.0.0.0 --port 8001, meaning Flask will listen on all available network interfaces (0.0.0.0) and port 8001
- When we define the command parameter in the Docker Compose file for a service, it takes precedence over the default CMD command specified in the Dockerfile
env_file specifies the file from which environment variables should be read and passed to the container

For reference, this is the service's Dockerfile:

FROM python:3.8-slim-buster as base 

# Create app directory
WORKDIR /app

# Install Python requirements first so it's cached
COPY ./requirements.txt .
RUN pip3 install -r requirements.txt

# Copy Flask project to container
COPY . .

# Set Flask configurations
ENV FLASK_APP=app.py
ENV FLASK_RUN_HOST=0.0.0.0

##############

FROM base as DEV

RUN pip3 install debugger
RUN pip3 install debugpy

# Define Flask environment (production is default)
ENV FLASK_ENV=development 

CMD ["python", "-m", "debugpy",\
    "--listen", "0.0.0.0:5678",\
    "--wait-for-client",\
    "-m", "flask", "run", "--host", "0.0.0.0", "--port", "8000"]

##############

FROM base as PROD

CMD ["flask", "run"]

We won't get into how to deploy it to production in this article but I wanted to quickly mention that using Gunicorn - a popular WSGI (Web Server Gateway Interface) HTTP server for Python web applications is probably a good idea:

# Run the app with Gunicorn 
CMD ["gunicorn", "--bind", "0.0.0.0:8001", "our_app:app"]

mongo db

  mongo_db:
    container_name: mongo_db_${ENV}
    image: mongo:4.4.18
    restart: always
    ports:
      - 27017:27017
    volumes:
      - ./DB-STORAGE/mongo-db-$ENV:/data/db
      - ./db/init-mongo.sh:/docker-entrypoint-initdb.d/init-mongo.sh 
    networks:
      - my_network
    env_file: .env

container_name well, we already know :)
image specifies the Docker image to be used for the container. In this case, it's using the official MongoDB image with version 4.4.18 from Docker Hub
restart indicates the restart policy for the container - always means the container will be automatically restarted if it exits, regardless of the exit status
ports maps port 27017 from the host to port 27017 in the container (default MongoDB port)
- Usually, it's best to keep our database private and only let other services (containers) access it. But for development purposes, we can expose it and use tools like MongoDB Compass
volumes mounts directories or files from the host into the container. This is used to persist data and configuration files
- ./DB-STORAGE/mongo-db-$ENV:/data/db will mount a host directory named ./DB-STORAGE/mongo-db-$ENV into the /data/db directory inside the container. This allows the MongoDB data to be stored persistently on the host filesystem
- ./db/init-mongo.sh:/docker-entrypoint-initdb.d/init-mongo.sh will mounts the init-mongo.sh file from the host into the /docker-entrypoint-initdb.d/init-mongo.sh path in the container which is the official MongoDB Docker image entry point.
networks attaches the service to a pre-existing Docker network called "my_network". This allows the MongoDB service and other services connected to "my_network" to communicate with each other.
env_file the environment variables from .env file will be used in the entrypoint script (init-mongo.sh) to configure MongoDB settings during startup

This is the init-mongo.sh file:

mongo -u "$MONGO_INITDB_ROOT_USERNAME" -p "$MONGO_INITDB_ROOT_PASSWORD" --authenticationDatabase admin <<EOF

use my_db;

db.createUser({
    user: "$MONGO_USER",
    pwd: "$MONGO_PASSWORD",
    roles: 
        [
            { 
                role: "readWrite", 
                db: "my_db" 
            },
            { 
                role: "dbAdmin", 
                db: "my_db" 
            }     
        ] 
});

If we want our MongoDB user to have both read/write access to an existing database and the ability to create collections and documents, we should grant it both the readWrite role and the dbAdmin role.

schedule

schedule_service:
    container_name: schedule_service_${ENV}
    build:
      context: ./schedule
    volumes:
      - ./common:/app/common
      - ./db:/app/db  
    networks:
      - my_network
    command: ["python", "-u", "schedule.py"] 
    env_file: .env

The context parameter defines the build context, which means the ./schedule directory and its subdirectories will be used to find the necessary files for the build

When we're only using a Dockerfile, we can specify the Dockerfile's location and the context (root directory) but when we're working with Docker Compose, we can only specify the context, which means we cannot go up in the directory from where the Dockerfile is

docker build -f $RelativePathToSourceCode -t $AppName . - in that case -f $RelativePathToSourceCode defines where our Dockerfile is but the . at the end defines the context (root directory)

In this case, we still want to use some common packages from the application, so we can map them using volumes
We already know that command overrides the default command that would be executed when the container starts. In this case, the script will be executed with the Python interpreter in unbuffered mode (-u flag) to ensure that the output is immediately displayed in the container logs

In summary, the schedule_service service in the Docker Compose file builds a container from the ./schedule directory, mounts specific directories from the host into the container, and runs the Python script schedule.py as the main command for the container. Additionally, it connects the container to the my_network for communication with other services on the same network and reads environment variables from the .env file.

Firing it all up

Now that we have everything set up, it's time to fire it all up.

Running docker-compose build --no-cache followed by docker-compose up is the best way to ensure that our Docker containers are built from scratch without using any cached layers from previous builds.

Alternatively, using docker-compose up --build will rebuild the images for all services, ignoring any previously cached layers. This ensures that we have the latest version of our application and all its dependencies.

The docker-compose down command both stops and removes containers associated with our Docker Compose project, but the exact behavior depends on the options used with the command.

By default, docker-compose down does the following:

Stops all the containers defined in our docker-compose.yml file that are currently running as part of the project. The containers are gracefully stopped, and their resources are released
After stopping the containers, docker-compose down also removes the containers

To clean it all up, we can use docker-compose down --rmi all.

Introdução ao Pelican

Felipe Arcaro — Sat, 28 Oct 2023 13:14:27 +0000

TL;DR

Para criar um site estático com Pelican, siga os passos a seguir:

Crie uma pasta do projeto
Instale o Pelican e os pacotes Markdown executando pip install "pelican[markdown]"
Execute pelican-quickstart e responda às perguntas que aparecerão
Crie um artigo chamado meu_artigo.md com o seguinte conteúdo:

Title: This is my first article
Date: 2010-12-03 10:20
Category: Cool

Cool content goes here.

Execute pelican -r -l, abra um navegador e acesse http://localhost:8000/ para ver o site funcionando localmente.

O que é Pelican?

O Pelican é um gerador de site estático escrito em Python.

Aqui estão algumas coisas que gosto sobre o Pelican:

Podemos escrever conteúdo nos formatos reStructuredText ou Markdown usando a ferramenta com a qual estamos familiarizados (eu uso o Obsidian, a propósito)
Ele inclui uma ferramenta de linha de comando simples para gerar nosso site e realizar algumas outras coisas interessantes
Ele gera um site estático que pode ser facilmente hospedado em qualquer lugar (até mesmo no AWS S3 (Simple Storage Service)!)
Possui temas e plugins de código aberto feitos pela comunidade

O que é um site estático?

Ok, mas o que é um site estático?

Um site estático é um tipo de site que não altera seu conteúdo - não se atualiza com base em interações do usuário ou dados em tempo real (sem bancos de dados!). Geralmente é composto por arquivos HTML, CSS e, às vezes, arquivos JavaScript.

Para pessoas não técnicas, um site estático pode ser entendido como um folheto impresso. É como uma imagem de informações que não muda a menos que alguém fisicamente altere as palavras ou imagens no papel (o programador, no caso).

Por ser tão simples, normalmente é mais rápido, mais confiável, seguro e com certeza divertido de construir.

Por que Pelican?

Sempre tive dificuldades para escolher a melhor ferramenta de escrita. Experimentei o MailChimp, o Medium, o Ulysses, o Evernote, o Notion, o Google Docs e outros, e muitas vezes me vi alternando entre eles - acabando por gastar mais tempo fazendo isso do que escrevendo efetivamente.

Outra coisa com a qual tive dificuldade foi decidir a melhor maneira de salvar e possivelmente controlar versões das minhas notas, porque, no final do dia, todas essas outras ferramentas ainda seriam responsáveis por armazenar meus dados.

Decidi escrever minhas notas e artigos como arquivos Markdown (uso o Obsidian para isso) e armazenar esses arquivos automaticamente com o iCloud no Mac.

E quando termino um artigo ou preciso editar um artigo que já publiquei, posso simplesmente usar o Pelican para (re)gerar as páginas do blog a partir dos meus arquivos Markdown.

Configurando o Pelican

Como mencionei anteriormente, o Pelican é construído em Python, portanto, precisamos apenas instalá-lo como um pacote no PyPI (Python Package Index) e executar alguns comandos de linha de comando (CLI). Aqui estão os passos que precisamos seguir:

Crie uma pasta de projeto
Instale os pacotes Pelican e Markdown (recomendado fazer isso em um ambiente virtual Python)

# Crie um ambiente virtual para o Python
python -m venv env-pelican

# Ative o ambiente virtual no Linux/Mac
source env-pelican/bin/activate

# Ative o ambiente virtual no Windows
./env-pelican/Scripts/activate

# Instale o pacote do Pelican
python -m pip install "pelican[markdown]"

Execute pelican-quickstart e responda às perguntas que aparecerão
Crie um artigo chamado meu_artigo.md com o seguinte conteúdo (você pode aprender mais sobre os metadados suportados pelo Pelican aqui)

Title: This is my first article
Date: 2010-12-03 10:20
Category: Cool

Cool content goes here.

Execute pelican -r -l, abra um navegador e acesse http://localhost:8000/ para ver o site em funcionamento localmente
- A opção -r reinicializa o Pelican sempre que uma modificação ocorre nos arquivos de conteúdo
- A opção -l serve os arquivos de conteúdo via HTTP na porta 8000

Você vai ver algo muito parecido com a imagem abaixo:

Contribuindo para projetos open source

Felipe Arcaro — Mon, 16 Oct 2023 19:26:41 +0000

TL;DR

Finalmente adentrei o mundo do open source

Airflow - Dynamic DAGs example to MWAA local

Sempre fui apaixonado pelo mundo do código aberto (open source). A maior parte do meu trabalho atual depende muito de ferramentas e pacotes open source. Na verdade, devemos reconhecer que uma parte significativa da internet funciona em cima do open source.

No entanto, devo dizer que nunca compreendi verdadeiramente porque as pessoas investiriam inúmeras horas na construção de software gratuito apenas pelo bem da criação - isso é, até eu construir meu primeiro site.

A sensação que tive ao construir algo do nada foi incrível, mesmo que fosse mal projetado e ninguém fosse usá-lo. Na verdade, às vezes acho o processo de lançar um produto com usuários reais tão divertido quanto criar algo que resolve exclusivamente um problema para mim e talvez para alguns dos meus amigos.

Nunca me considerei um engenheiro com todas as skills necessárias para contribuir para um projeto open source, no entanto, e até mesmo navegar pelos repositórios no GitHub era assustador.

Este mês, eu decidi mudar isso. Estou orgulhoso em apresentar minha primeira contribuição open source:

Airflow Dynamic DAGs example to MWAA local

Embora essa contribuição não seja uma proeza de genialidade em engenharia, e uma parte significativa do código já existisse em outro lugar, espero que isso ajude alguém por aí - ou, no mínimo, confunda alguns robôs espreitando nas profundezas da internet.

Creating an AWS IAM user with Terraform

Felipe Arcaro — Tue, 10 Oct 2023 10:06:46 +0000

TL;DR

Here's the Terraform code to create an IAM user with access to EC2 and RDS instances:

data "aws_iam_policy_document" "my_project_policy" {
    statement {
        resources = [aws_instance.my_project_ec2.arn]
        actions = ["ec2:*"]
        effect = "Allow" # Default is Allow
    }

    statement {
        resources = ["aws_db_instance.my_project_db.arn"]
        actions = ["rds:DescribeDBInstances",
                   "rds:DescribeDBClusters",
                   "rds:DescribeGlobalClusters"]
    }             
}

# Provides an IAM user
resource "aws_iam_user" "my_project_user" {
  name = "my-project-user"
}

Now that we already created an IAM role so the EC2 instance can access the RDS instance, it's time to create an IAM user with specific (and restricted) access to use/manage the infrastructure we just created.

Here's a quick recap on IAM:

Identity and Access Management (IAM) is a service that allows you to manage access to AWS resources securely
IAM enables you to create and manage AWS users and groups and control their level of access to AWS resources
We create groups with permissions and then add users to these groups
We create roles and assign them to AWS services so they communicate with each other

Creating an IAM user

Here are the things we'll need to add to our Terraform code:

A policy defining permissions
An IAM user
Combine the two above

Let's start creating the policy:


data "aws_iam_policy_document" "my_project_policy" {
    statement {
        resources = [aws_instance.my_project_ec2.arn]
        actions = ["ec2:*"]
        effect = "Allow" # Default is Allow
    }

    statement {
        resources = ["aws_db_instance.my_project_db.arn"]
        actions = ["rds:DescribeDBInstances",
                   "rds:DescribeDBClusters",
                   "rds:DescribeGlobalClusters"]
    }             
}

# Provides an IAM user
resource "aws_iam_user" "my_project_user" {
  name = "my-project-user"
}

# Provides an IAM policy attached to a user
resource "aws_iam_user_policy" "my_project_user_policy" {
    name = "my-project-policy"
    user = aws_iam_user.my_project_user.name
    policy = data.aws_iam_policy_document.my_project_user_policy.json
}

You probably noticed this is the first time we're using a data block instead of a resource block. The key difference here is that resource blocks are used to create/manage infrastructure resources, while data blocks are used to fetch and reference information from external services.

Now we create the user:

# Provides an IAM user
resource "aws_iam_user" "my_project_user" {
  name = "my-project-user"
}

And attach the policy to the user:

# Provides an IAM policy attached to a user
resource "aws_iam_user_policy" "my_project_user_policy" {
    name = "my-project-policy"
    user = aws_iam_user.my_project_user.name
    policy = data.aws_iam_policy_document.my_project_user_policy.json
}

Wrapping up

That's it, now we're really done with our little project. After going through all those steps, here's what we've got:

An EC2 instance where we can deploy our Python project
An RDS Postgres instance where that Python project can persist data
An IAM role that allows the EC2 instance to connect to the RDS Postgres instance
An IAM user with proper permissions to interact with the EC2 and RDS instances

Creating an AWS IAM role with Terraform

Felipe Arcaro — Tue, 03 Oct 2023 19:44:04 +0000

TL;DR

Here's the Terraform code to create an IAM role so an EC2 instance has proper access to an RDS instance:

# IAM role for EC2 instance
resource "aws_iam_role" "my_project_role" {
  name = "my-project-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Sid    = ""
        Principal = {
          Service = "ec2.amazonaws.com"
        }
      }
    ]
  })
}

# IAM policy to be attached to the EC2 role
resource "aws_iam_role_policy" "my_project_role_policy" {
  name = "my-project-role-policy"
  role = aws_iam_role.my_project_role.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = [
          "rds:*",
        ]
        Effect   = "Allow"
        Resource = aws_db_instance.my_project_db.arn
      },
    ]
  })
}

# Update EC2 resource created earlier
resource "aws_instance" "my_project_ec2" {
  instance_type = "t2.micro"
  ami = "ami-06e46074ae430fba6"
  tags = {
    Name = "my-python-project"
  }

  iam_instance_profile = aws_iam_role.my_project_role.arn
}

Nice, we're done with the goals we defined at the beginning of this article but we want more – actually, we need more. Now it's time to create an IAM user with specific (and restricted) access to use/manage the infrastructure we just created and an IAM role so the EC2 instance can access the RDS instance.

Identity and Access Management (IAM) is a service that allows you to manage access to AWS resources securely. IAM enables you to create and manage AWS users and groups and control their level of access to AWS resources.

As per IAM standards:

We create groups with permissions and then add users to these groups
We create roles and assign them to AWS services so they communicate with each other

For this example, we won't create any IAM groups but it is important to remember that we cannot assign a role to a user.

Creating an IAM role

Here are the things we'll need to add to our Terraform code:

A policy defining permissions
An IAM role
Update the EC2 code to use that role

Let's start creating an IAM role for our EC2 service:

# IAM role for EC2 instance
resource "aws_iam_role" "my_project_role" {
  name = "my-project-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Sid    = ""
        Principal = {
          Service = "ec2.amazonaws.com"
        }
      }
    ]
  })
}

And now we create a policy with all permissions to our RDS instance and reference the role we just created.

# IAM policy to be attached to the EC2 role
resource "aws_iam_role_policy" "my_project_role_policy" {
  name = "my-project-role-policy"
  role = aws_iam_role.my_project_role.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = [
          "rds:*",
        ]
        Effect   = "Allow"
        Resource = aws_db_instance.my_project_db.arn
      },
    ]
  })
}

Now we have to update our EC2 instance code to make sure it uses the role and permissions we just defined. For that, we just add iam_instance_profile = aws_iam_role.my_project_role.arn to it:

resource "aws_instance" "my_project_ec2" {
  instance_type = "t2.micro"
  ami = "ami-06e46074ae430fba6"
  tags = {
    Name = "my-python-project"
  }

  iam_instance_profile = aws_iam_role.my_project_role.arn
}

Creating an AWS EC2 instance with Terraform

Felipe Arcaro — Mon, 25 Sep 2023 16:48:14 +0000

TL;DR

Here's the Terraform code to create an EC2 instance:

resource "aws_instance" "my_project_ec2" {
  instance_type = "t2.micro"
  ami = "ami-06e46074ae430fba6"
  tags = {
    Name = "my-python-project"
  }
}

EC2 Instance (Elastic Compute Cloud)

Elastic Compute Cloud (EC2) is a virtual server provided by AWS that allows you to run applications and services in the cloud - it's just a regular computer but in the cloud.

EC2 is awesome because:

We don't need to invest in infrastructure upfront
It allows us to scale up/down (vertically) to handle different workloads
It offers different purchasing options (on demand, reserved, spot)

Terraform it up

We're past the basics now, so let's create an EC2 instance. Here's the code to do it:

resource "aws_instance" "my_project_ec2" {
  instance_type = "t2.micro"
  ami = "ami-06e46074ae430fba6"
  tags = {
    Name = "my-python-project"
  }
}

The resource name my-python-project that comes after the resource type aws_instance is not the same as the resource name that appears in the Admin Console. For that, I had to use tags parameter.

Once again, the infrastructure is updated after we run terraform plan and terraform apply and a new EC2 instance is created.i

Wait, what is AMI?

Amazon Machine Image (AMI) is a pre-configured package required to launch an EC2 instance that includes an operating system, software packages and other requirements. It's essentially a template that can be used to launch EC2 instances (Linux, Windows or Mac)

AMI is awesome because:

We don't need to manually install packages and requirements
We can launch several EC2 with the same configuration (load balancing)
When selecting the AMI, we can use the Advanced Details tab to run a bash script (e.g. to update yum and install httpd)

Ok. So far, we have a computer where we can run our project and a database in which we can persist and read data. Now it's time to give them proper access - in other words, we have to specify what AWS resources and who should be able to access them.

Creating an AWS RDS instance with Terraform

Felipe Arcaro — Mon, 18 Sep 2023 18:07:58 +0000

TL;DR

Here's the Terraform code to create an RDS Postgres instance:

# RDS instance
resource "aws_db_instance" "my_project_db" {
  engine               = "postgres"
  instance_class       = "db.t3.micro"
  username             = "sidewalk"
  password             = "${var.postgres_pw}"
  allocated_storage    = 20
  publicly_accessible = true
}

Creating an RDS instance

Amazon Relational Database Service (RDS) is a managed database service provided by AWS that simplifies the process of setting up, operating, and scaling a relational database in the cloud.

Here is a quick recap of some RDS notable features:

Multi-AZ
- Supports standby instance in another AZ that helps with failure recover
Read replicas
- Offload read requests
- Can be used in disaster recovery plan
Automatic backups
- Configurable time window
Configurable storage
IAM authentication
RDS Enhance monitoring

Starting in the Management Console (UI)

First, we create a free RDS Postgres instance through the Management Console. Since this was the first AWS service on this account, it automatically created a VPC ¹ and a Security Group ² to ensure the instance was securely isolated and protected by a firewall.

To see if the Postgres database is working, we can use PGAdmin, but we also need to go through the following steps:

Make the RDS instance publicly available
Add our personal IP to the Security Group's inbound rules used by the database so we can connect to it from outside the VPC (our personal computer):
- Although not recommended, we can all ports 0-655353 and all IPs 0.0.0.0/0 to the inbound rules so everyone can connect to our database

Everything seems to be working as expected.

Importing existing resources to Terraform code

Now, let's start with terraform import to import my database instance to my configuration file so I can manage it from there.

First, we create the most basic version of the aws_db_instance resource on our Terraform main.tf file:

resource "aws_db_instance" "my_project_db" {
  engine               = "postgres"
  instance_class       = "db.t2.micro"
  username             = "sidewalk"
  password             = "${var.postgres_pw}"
}

You probably noticed we're using a variable instead of writing the password in plain text. Here's the easiest way to do it:

Create a file called vars.tf in the same folder as the main.tf file and add the following code to it:

variable "super_secret_password" {
  default = "ThisPassword123!"
}

Now you can call it from your main terraform code like this ${var.super_secret_password}

Ok, now it's time to run terraform import aws_instance.my_project_db <db-instance-id> (you can find the <db-instance-id> in the Management Console) – it seems like the database was imported successfully.

There was a small problem though – the database we created through the Management Console had the parameter instance_class = db.t3.micro while our Terraform code had instance_class = db.t2.micro.

When we run terraform plan, it says that a particular resource must be replaced. Then, when we run terraform apply, the database created through the Management Console is deleted and we get an error saying there were some important parameters missing to create the new one – in this case, the allocated_storage parameter.

Key learnings on using `terraform import`:

To import an existing infrastructure from AWS, you must have the resource in the Terraform code with all required parameters
Make sure the configuration you have on your configuration file matches exactly what you have on AWS
terraform plan won't show if there are required parameters missing, only terraform apply

Creating resources from Terraform code

Now that we know how this works, let's go ahead and create the database directly from the Terraform code using AWS documentation on aws_db_instance:

resource "aws_db_instance" "my_project_db" {
  engine               = "postgres"
  instance_class       = "db.t3.micro"
  username             = "sidewalk"
  password             = "${var.postgres_pw}"
  allocated_storage    = 20
}

We can run terraform plan to make sure everything is looking good and terraform apply to create the database. After the database is created, we can type terraform show to see the infrastructure's current state – the VPC, the Security Group, etc.

Key learnings on creating infrastructure using Terraform code:

It automatically created it on the only VPC available on that AWS account
It automatically added the only Security Group available on that AWS account

And here's the updated Terraform code to make the RDS instance publicly accessible:

# RDS instance
resource "aws_db_instance" "my_project_db" {
  engine               = "postgres"
  instance_class       = "db.t3.micro"
  username             = "sidewalk"
  password             = "${var.postgres_pw}"
  allocated_storage    = 20
  publicly_accessible = true
}

VPC – Virtual Private Cloud is a virtual network infrastructure that allows you to create a logically isolated network environment. It is essentially a virtual data center in the cloud ↩
A Security Group is a virtual firewall that controls the inbound and outbound traffic for one or more instances. Every instance within a VPC in AWS must be associated with at least one security group. ↩

Introduction to Terraform

Felipe Arcaro — Tue, 12 Sep 2023 23:17:00 +0000

TL;DR

Terraform is a tool used in the field of infrastructure as code (IaC) to create and manage computer resources – servers, networks, databases, and so on. It allows us to define a desired infrastructure in a configuration file using a simple and declarative language.

Once a Terraform project is started, the state of the managed infrastructure and configuration needs to be stored somewhere - usually locally or on AWS S3. Then, that state file is used by Terraform essentially to map real-world resources to our configuration file and keep track of metadata.

Here are some important Terraform commands to know:

terraform plan: Generates an execution plan, showing changes to infrastructure resources that would be made when applied
terraform apply: Applies the changes defined in Terraform configuration to create or update infrastructure resources
terraform destroy: Destroys and removes all resources created by Terraform for a given configuration
terraform show: Displays the current state or details of Terraform-managed resources
terraform import: Imports existing infrastructure resources into Terraform's state for management

What is Terraform?

Before we jump into coding things out, let's go over some key concepts of Terraform.

Once we have defined our infrastructure configuration, Terraform takes care of provisioning and managing the resources. It interacts with cloud providers like Amazon Web Services (AWS) to create and update infrastructure based on the configuration provided in the file.

Using Terraform enables us to treat infrastructure as code – we can version control it, collaborate with others, and easily replicate or rebuild it in a consistent and repeatable manner.

And since we're going to be provisioning infrastructure in AWS, it's also important to understand the difference between a resource, an instance, and a service.

A resource is a logical entity that represents a piece of AWS infrastructure – an S3 bucket, an EC2 instance, an RDS database
An instance is a specific occurrence of a resource that is created from a particular template or configuration – an EC2 instance is a virtual server that is launched from an Amazon Machine Image (AMI). Each instance is a unique copy of the AMI, with its own IP address, storage, and other attributes
A service is a collection of related AWS resources and API operations that work together to perform a specific function – AWS offers over 200 different services, including compute, storage, database, analytics, machine learning, and more

Getting the environment ready

There are a few steps we need to go through in order to get our development environment ready:

Install AWS CLI
- Run aws configure on the terminal and provide the following parameters
  - AWS Access Key ID
  - AWS Secret Access Key
  - Default region name
  - Default output format
Install Terraform CLI

Project structure

Here's the project structure we are working with:

my-project/
└── terraform/
    └── main.tf

And here's the boilerplate code we're starting with:

# Specify AWS resource
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.0"
    }
  }
}

# Configure the AWS Provider
provider "aws" {
  region = "us-east-1"
}

# Define where the state will be stored
terraform {
  backend "local" {
    path = ".tfstate"
  }
}

To start the Terraform project, we navigate to the my-project/terraform/ directory and run terraform init which essentially downloads all necessary modules and defines the backend.

Wait, what is this backend thing?

To understand what the Terraform backend is, let's quickly cover what a Terraform state is first.

You probably saw some state-related code in the code snippet I provided earlier:

# Define where the state will be stored
terraform {
  backend "local" {
    path = ".tfstate"
  }
}

Once a Terraform project is started, the state of the managed infrastructure and configuration needs to be stored somewhere, and that state is used by Terraform essentially to map real-world resources to our configuration file and keep track of metadata.

Here are some key aspects of the Terraform state:

Its primary purpose is to store bindings between objects in a remote system (e.g. an EC2 instance) and resource instances declared in a configuration file
It's generally saved in a file called terraform.tfstate
It can be stored locally – recommended for developing purposes
It can be stored in the cloud (e.g S3) – recommended for deployment purposes, so it can be versioned, encrypted, and securely shared

The backend simply defines where Terraform stores its state.

How to "run" terraform code

We already have some Terraform code, can't we just run it? Kinda. Let's take a look at the following commands:

terraform plan
terraform apply
terraform destroy
terraform show
terraform import

Terraform plan

The terraform plan command creates a plan. It can be used to preview the planned changes to the infrastructure.

Under the hood, the command:

Reads the current state of any remote objects to make sure that the Terraform state is up-to-date
Compares the current configuration to the prior state and notes any differences
Proposes a set of change actions that should, if applied, make the remote objects match the configuration

Terraform apply

The terraform plan command applies the changes to the infrastructure to get to the state where the remote objects match the configuration. terraform apply can be run without running terraform plan first although it's not recommended.

Terraform destroy

The terraform destroy command is used to destroy all mapped objects. terraform plan -destroy can be used to see what destruction will be made beforehand.

It is also worth mentioning that if the infrastructure has already been provisioned with terraform apply, removing objects from the Terraform code and running terraform apply again will also destroy them.

Terraform show

The terraform show command can be used at any time to inspect the configuration of a Terraform state.

Terraform import

The terraform import command can be used for bringing existing resources under Terraform management.

With that in our toolkit, it's time to get our hands dirty.

Contributing to open source projects

Felipe Arcaro — Wed, 06 Sep 2023 21:11:54 +0000

TL;DR

I have made my initial foray into the realm of open-source contributions:

Airflow - Dynamic DAGs example to MWAA local

I have always been passionate about the open-source world. The majority of my current job relies heavily on open-source tools and packages. In fact, it is remarkable to acknowledge that a significant portion of the internet operates on open-source code.

However, I must say that I never truly comprehended why people would invest countless hours in building software merely for the sake of creation – that is, until I built my first website.

The high I got from building something out of thin air was amazing, even though it was poorly designed and nobody was going to use it. In fact, at times, I find the process of shipping a product with real users to be as fun as creating something that exclusively solves a problem for myself and perhaps for some of my friends.

I never really considered myself a good-enough engineer to contribute to an open-source project, though, and even browsing through open-source repositories was overwhelming and daunting.

However, this month, I decided to change that. I am proud to present my inaugural open-source contribution:

Airflow - Dynamic DAGs example to MWAA local

While it may not be a feat of engineering genius, and a significant portion of the code already existed elsewhere, hopefully it will help someone out there – or, at the very least, confuse a few robots lurking in the depths of the crazy interwebs.

Adding different Pelican themes to your static website

Felipe Arcaro — Fri, 01 Sep 2023 17:42:26 +0000

TL;DR

In order to add a different theme to your static website generated with Pelican, do the following:

Clone one of these community-managed theme repos
Run the following command on the terminal pelican content -t /path/to/theme -r -l

What is a Pelican theme?

When you create your first static website using Pelican, you'll notice the design looks like it hasn't been updated since the 90s:

That's exactly where Pelican themes come into play. There is a community-managed repository of Pelican Themes for people to share and use.

Can I create my own Pelican theme?

You can obviously create your own Pelican theme but if you want to get up and running fast with a not-so-bad-looking website, the easiest path is to pick a Pelican theme that already exists.

How to use a Pelican theme

Although it hasn't been updated in 7 years, I really like the simplicity/responsiveness of the Alchemy theme, so that's what we're using for this example.

Clone the repository

Navigate to the Alchemy Pelican theme repository and clone it. You'll normally find more information about a theme by looking at the README.md file in the repository.

Change your configs

There are two ways you can tell Pelican to use your new theme when creating your static website:

Reference it with the -t flag when running your pelican command

pelican content -t path/to/the/theme/pelican-alchemy/alchemy -r -l

Reference it in the pelicanconf.py file adding the following line to it

THEME = 'path/to/the/theme/pelican-alchemy/alchemy'

And then running the following command:

pelican content -s pelicanconf.py path/to/the/theme/pelican-alchemy/alchemy -r -l

Make sure you're specifying the right path – in this example, we need to specify pelican-alchemy/alchemy and not just pelican-alchemy)

Here's how your static website will look like with the new Alchemy theme:

DEV Community: Felipe Arcaro

Can I scale my dockerized Flask solution with Kubernetes?

TL;DR

Wait... isn't Kubernetes and Docker the same thing?

What is Kubernetes?

Kubernetes' architecture

Control Plane:

Node:

Getting to work

Converting our docker-compose.yml file

Firing it all up

Taking a look at our creation

Final thoughts

Accessing my home server from a different continent (the hard way)

TL;DR

My home server's IP

Setting up port forwarding

Configuring SSH keys

Creating keys on my laptop

Adding the public key to my home server

Removing password access

Monitoring

Dockerizing a Flask application

TL;DR

What is Docker?

Getting started with Dockerfile and Docker Compose

Why should we dockerize an application?

Dockerizing a Flask application

network

app

mongo db

schedule

Firing it all up

Introdução ao Pelican

TL;DR

O que é Pelican?

O que é um site estático?

Por que Pelican?

Configurando o Pelican

Contribuindo para projetos open source

TL;DR

Creating an AWS IAM user with Terraform

TL;DR

Creating an IAM user

Wrapping up

Creating an AWS IAM role with Terraform

TL;DR

Creating an IAM role

Creating an AWS EC2 instance with Terraform

TL;DR

EC2 Instance (Elastic Compute Cloud)

Terraform it up

Wait, what is AMI?

Creating an AWS RDS instance with Terraform

TL;DR

Creating an RDS instance

Starting in the Management Console (UI)

Importing existing resources to Terraform code

Key learnings on using terraform import:

Creating resources from Terraform code

Key learnings on creating infrastructure using Terraform code:

Introduction to Terraform

TL;DR

What is Terraform?

Getting the environment ready

Project structure

Wait, what is this backend thing?

How to "run" terraform code

Terraform plan

Terraform apply

Terraform destroy

Terraform show

Terraform import

Contributing to open source projects

TL;DR

Adding different Pelican themes to your static website

TL;DR

What is a Pelican theme?

Can I create my own Pelican theme?

How to use a Pelican theme

Converting our `docker-compose.yml` file

Key learnings on using `terraform import`: