DEV Community: Thiago (Zozô) Ozores

Building multiarch images with Buildah

Thiago (Zozô) Ozores — Fri, 22 Sep 2023 19:00:00 +0000

With the growth and popularization of platforms beyond amd64, especially the ARM platform, it becomes a concern for those maintaining container images to build them tailored for these platforms as well.

In this article, I will show you how to build these images using Buildah.

Buildah is an open-source project led by RedHat (but receives contributions from other companies) that aims to simplify the construction of container images that adhere to and are compatible with the OCI (Open Container Initiative) standard.

So let’s get to the steps.

Prerequisites

Installation of the Buildah package
Installation of the qemu-user-static package (Ubuntu/RedHat) or qemu-arch-extra package (Arch Linux);

sudo dnf install -y buildah qemu-user-static (Fedora and derivatives)

sudo apt-get install -y buildah qemu-user-static (Ubuntu/Debian and derivatives)

pacman -Sy buildah qemu-arch-extra (Arch Linux and derivatives)

Building the images

The workflow you need to follow to build the images is as follows:

Create the manifest

export MANIFEST="multiarch-image" # manifest name, can be any name
export IMAGE="registry/repository/image:tag" # image name, e.g., docker.io/ozorest/example:latest
buildah manifest create "$MANIFEST"

Run the build for each architecture

In the directory with your Dockerfile, you can run the following loop for each architecture:

for arch in amd64 arm64; do
 buildah build --arch $arch --tag "$IMAGE" --manifest "$MANIFEST" .
 # buildah build --arch $arch --tag "$IMAGE" --manifest "$MANIFEST" -f path_to_Dockerfile, in case the Dockerfile is not in the current directory
done

Push the manifest to the registry

buildah manifest push --all "$MANIFEST" "docker://$IMAGE"

And that’s it, folks! Stay tuned for more content!

Diagrams as Code

Thiago (Zozô) Ozores — Thu, 05 May 2022 00:00:00 +0000

Come, come, come and find out how to generate diagrams using Python code!!

Em Português 🇧🇷 🇵🇹

In my day-to-day as an instructor, one of the jobs is to create diagrams that clearly illustrate the topic being presented to students, there are many good graphic tools online like draw.io, LucidChart, among others.

But despite these tools being very intuitive and easy to use, when you need to scale the creation of diagrams, need to create diagrams bringing information from external tools or even create simple diagrams quickly, you end up running into problems with formatting options and lack of automation of these tools, which make it difficult to create a “factory” of diagrams.

Thinking about this scenario, Python has a package that can be used to represent and generate diagrams as code, facilitating the creation of this “factory”.

The package is called diagrams and it has a very interesting way of working, it makes use of Python’s operator overload to perform in a more intuitive way the connection that the nodes will have in diagram graphics. For example:

The >> operator represents a right-to-left binding
The << operator represents a left-to-right binding
The - operator represents a directionless binding
And it’s still possible to make bidirectional bindings using the Edge class

If you’re looking for a more programmatic way to generate diagrams, this package is worth checking out.

Below are some examples I developed, which can be tested on Google Colab

Github Actions: Sharing artifacts between jobs

Thiago (Zozô) Ozores — Sun, 28 Feb 2021 16:56:55 +0000

This will be a quick tip.

Some days ago, configuring a pipeline inside Github Actions I had the need to use a file generated by a job in another job, that are using distinct OS.

It's pretty straight forward do that, the only thing that you need is the upload-artifact and the download-artifact actions

Here is an example:

job1:

  runs-on: ubuntu-latest
  steps:
  - uses: actions/checkout@v1

  - run: mkdir -p dist

  - run: echo hello > dist/world.txt

  - uses: actions/upload-artifact@master
    with:
      name: hello-world-artifact
      path: dist/world.txt

job2:
  runs-on: macos-latest

  steps:
  - run: mkdir -p dist

  - uses: actions/download-artifact@master
    with:
      name: hello-world-artifact
      path: dist/world.txt

  - run: cat dist/world.txt

Source code and documentation for the actions:

BONUS TIP
I had the need to fetch a file from the Github Releases as well and to do that I used the third-party action dsaltares/fetch-gh-release-asset@master, but the drawback is that action only runs on Linux (because of that I have to copy an artifact from one job to another one ;-) )

Here is an example:

uses: dsaltares/fetch-gh-release-asset@master
with:
  repo: "your-user/your-repo"
  version: "latest"
  file: "package.zip"
  target: "dist/package.zip"
  token: ${{ secrets.YOUR_TOKEN }} # If your repo is private, you need to fill with the personal access token

Source code and documentation for the dsaltares/fetch-gh-release-asset action

That's all folks! Thank you and stay in tune for more tips.

[PT-BR] Github Actions: Compartilhando artefatos entre jobs

Thiago (Zozô) Ozores — Sun, 28 Feb 2021 16:56:40 +0000

Esta vai ser uma dica rápida.

Outro dia configurando um pipeline no Github Actions, eu tive a necessidade de usar um arquivo gerado em um job em um outro job, que estavam usando diferentes sistemas operacionais.

É bem simples fazer isso, tudo que você precisa é destas duas actions: upload-artifact e download-artifact

Aqui está um exemplo:

job1:

  runs-on: ubuntu-latest
  steps:
  - uses: actions/checkout@v1

  - run: mkdir -p dist

  - run: echo hello > dist/world.txt

  - uses: actions/upload-artifact@master
    with:
      name: hello-world-artifact
      path: dist/world.txt

job2:
  runs-on: macos-latest

  steps:
  - run: mkdir -p dist

  - uses: actions/download-artifact@master
    with:
      name: hello-world-artifact
      path: dist/world.txt

  - run: cat dist/world.txt

Código-fonte e documentação das actions:

DICA BÔNUS
Eu também tive a necessidade de fazer o download de um arquivo do Releases do Github, para isso eu usei esta action dsaltares/fetch-gh-release-asset@master de terceiros, mas a desvantagem é que esta action apenas roda em Linux (por isso que eu precisei copiar um artefato de um job em outro ;-) )

Aqui está um exemplo:

uses: dsaltares/fetch-gh-release-asset@master
with:
  repo: "your-user/your-repo"
  version: "latest"
  file: "package.zip"
  target: "dist/package.zip"
  token: ${{ secrets.YOUR_TOKEN }} # Se o seu repo é privado, você precisa do access token

Código-fonte e documentação da action dsaltares/fetch-gh-release-asset

Isso é tudo, pessoal! Obrigado e fique sintonia para mais dicas.

The Poor Man's Guide to Django deployment in the AWS

Thiago (Zozô) Ozores — Mon, 31 Aug 2020 18:26:16 +0000

In this post I will talk about how to deploy a Django application (but easy to replicate to any other Python framework like Flask, for example) in the AWS saving a lot of money (depending of the traffic of your application and what services are you intended to use inside AWS, it could be free...forever!)

AND THIS POST WILL BE A QUITE LONG, SORRY FOR THAT!

In July, I finished the development of my educational project for children using Django and my first question when I finished was how to deploy this application in the cloud in a cheapest way possible.

My first option was Heroku, because is quite easy to deploy and implement a continuous deployment process using them for free, but if I have to scale my application for any reason, Heroku could get a lot more expensive, so I decided to think better.

In that process, I found a project that was a game changer in my quest. The project is Zappa, below the description of the project taken from they readme at Github:

Zappa makes it super easy to build and deploy server-less, event-driven Python applications (including, but not limited to, WSGI web apps) on AWS Lambda + API Gateway. That means infinite scaling, zero downtime, zero maintenance - and at a fraction of the cost of your current deployments!

They accomplishes that, indeed, is quite easy deploy the app in the AWS, the hardest part of the deployment is configure the IAM roles and policies inside AWS (I think this is the hardest part for any deployment there :-) ). So, without further ado, let me show you how I managed to deploy my Django app using Zappa.

PS: It is out of scope of this article explain deeper how AWS services works, I'm assuming that you have some knowledge about it and I won't use the console to configure AWS services, I'll use the AWS CLI, so if you want to use it as well, check here how to setup CLI

PS 2: Setup AWS CLI to use some user with Administrator access but NEVER EVER NEVER USE ROOT ACCOUNT! And keep the Access and Secret keys safe, for example using a password manager

PS 3: All commands and examples were made using a Linux workstation

1. Creating the required AWS S3 Bucket

Zappa will use the S3 bucket to upload the Lambda-compatible archive generated by the deploy command which I'll show further
To create the bucket using the CLI:

aws s3api create-bucket --bucket name_of_the_bucket --region region_of_your_choose --create-bucket-configuration LocationConstraint=region_of_your_choose

The region parameter and the LocationConstraint configuration are required if you will create a bucket outside us-east-1 region, if you choose this region you can remove both from the command.

Remember that bucket names are unique globally, so if you receive an error like BucketAlreadyExists, you have to choose a new name.

If everything goes well, you should receive this return from the command:

{
    "Location": "http://name_of_your_bucket.s3.amazonaws.com/"
}

PS 4: If your application uses SQLite as database, I recommend you to create one more bucket for that and if your application uses static and media files, I recommend also you to create another bucket for that, we will see later how to manage them inside AWS

2. Configuring needed IAM policies, roles, group and user

For a better security (and also a best practice), we need to setup some IAM objects exclusive for Zappa to restrict which AWS resources it will have access.

2.1 Creating the role

Let's start creating the role that will be passed to the Lambda function which will be created by Zappa during the deployment process.

For create the role using the CLI, we need a JSON file that will be represent which AWS services will be allowed to call other AWS services in the Zappa user behalf (the Assume Role Policy Document).

Create the JSON file in some directory in your computer with this content:

So, let's go back to the CLI for create the role:

aws iam create-role --role-name my-role --assume-role-policy-document file:///tmp/zappa_assume_role.json

my-role could be any name that you want.
file:///tmp/zappa_assume_role.json must match with the path of the file that you created before.

This will be the return, if the command succeded:

{
    "Role": {
        "Path": "/",
        "RoleName": "my-role",
        "RoleId": "AROA3IDJ3HNEWIQZA5IQZ",
        "Arn": "arn:aws:iam::773316098889:role/my-role",
        "CreateDate": "2020-08-29T19:24:37Z",
        "AssumeRolePolicyDocument": {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Sid": "",
                    "Effect": "Allow",
                    "Principal": {
                        "Service": [
                            "apigateway.amazonaws.com",
                            "lambda.amazonaws.com",
                            "events.amazonaws.com"
                        ]
                    },
                    "Action": "sts:AssumeRole"
                }
            ]
        }
    }
}

Save the value returned by the Arn field in some place, we will need him afterwards.

2.2 Attaching the policy to the role

Now, it's time to join the policy with the role that we created before.

The AWS policy can also be a JSON document that represents the permissions that will be granted for use the services inside AWS, for the role that we will attach it.

For that again, you need create a JSON file in some directory in your computer with this content:

Again, let's go back to the CLI and execute the following command:

aws iam put-role-policy --role-name my-role --policy-name my-policy --policy-document file:///tmp/zappa_policy.json

my-role must match with the name of the role that we created before.
my-policy could be any name that you want.
file:///tmp/zappa_policy.json must match with the path of the file that you created before.

If the command succeded, nothing will be returned

But if you want to confirm if everything goes well, you can run this command:

aws iam get-role-policy --role-name my-role --policy-name my-policy | head -3

The return should be:

{
    "RoleName": "my-role",
    "PolicyName": "my-policy",

2.3 Creating the group

In the last 2 steps, we defined the role and policy that will be passed to the Lambda function which will be created by Zappa during the deployment.

Now, we have to define the group, user and the policies that will allow Zappa create the resources inside AWS (Ex.: the Lambda function, create the API gateway, store the package in S3, etc).

We will start creating the group, executing that following CLI command:

aws iam create-group --group-name my-group

This will be the return, if the command succeded:

{
    "Group": {
        "Path": "/",
        "GroupName": "my-group",
        "GroupId": "AGPA3IDJ3HNEYAXDIXQ5K",
        "Arn": "arn:aws:iam::773316098889:group/my-group",
        "CreateDate": "2020-08-29T20:30:14Z"
    }
}

2.4 Attaching the policy for Zappa general permissions to the group

We will do a similar job, that we did in the 2.2 step, but instead create just one JSON, we will create two JSON's, because we have to attach two policies for the group, one policy for general permissions and the other one specific for S3 permissions.

Let's start with the general one, create the JSON file in some directory in your computer with this content below, but this time we need do a small change in the JSON before using inside the CLI.

Find full_arn_from_created_role inside the content and replace with the Arn of the previous created role.

After that let's go to the CLI and run the command that will attach the policy to the group:

aws iam put-group-policy --group-name my-group --policy-document file:///tmp/zappa_general_policy.json --policy-name my-general-policy

my-group must match with the name of the group that we created before.
my-general-policy could be any name that you want, but cannot be the same from previous policy created in the step 2.2.
file:///tmp/zappa_general_policy.json must match with the path of the file that you created before.

If the command succeded, nothing will be returned

But if you want to confirm if everything goes well, you can run this command:

aws iam get-group-policy --group-name my-group --policy-name my-general-policy | head -3

The return should be:

{
    "GroupName": "my-group",
    "PolicyName": "my-general-policy",

2.5 Attaching the policy for Zappa specific S3 permissions to the group

Now, we will attach the S3 specific policy to the group, we will repeat the same steps of the previous one.
But the content of the JSON to apply will be different and again we have to do a small change in the JSON before using inside the CLI.

Find full_arn_from_s3_bucket inside the content and replace with the Arn of the S3 bucket created in the step 1.

The ARN of the S3 bucket follow this pattern: arn:aws:s3:::name_of_your_bucket

If you created more than one bucket in the step 1, you must add the ARN of those as well.

After that let's go repeat the same commands from the previous step:

aws iam put-group-policy --group-name my-group --policy-document file:///tmp/zappa_s3_policy.json --policy-name my-s3-policy

my-group must match with the name of the group that we created before.
my-s3-policy could be any name that you want, but cannot be the same from previous policies created in the steps 2.2 and 2.4.
file:///tmp/zappa_s3_policy.json must match with the path of the file that you created before.

If the command succeded, nothing will be returned

But if you want to confirm if everything goes well, you can run this command:

aws iam get-group-policy --group-name my-group --policy-name my-s3-policy | head -3

The return should be:

{
    "GroupName": "my-group",
    "PolicyName": "my-s3-policy",

2.6 Creating the user, attach it to the group and create the secret key

At last, we have to create the user that will be used exclusive by Zappa, attach it to the group created previously and generate the access key of the user.

Let's start creating the user:

aws iam create-user --user-name my-user

The return should be:

{
    "User": {
        "UserName": "my-user",
        "Path": "/",
        "CreateDate": "2013-06-08T03:20:41.270Z",
        "UserId": "AIDAIOSFODNN7EXAMPLE",
        "Arn": "arn:aws:iam::123456789012:user/Bob"
    }
}

Now, we will attach this user to the group:

aws iam add-user-to-group --user-name my-user --group-name my-group

If the command succeded, nothing will be returned

But if you want to confirm if everything goes well, you can run this command:

aws iam get-group --group-name my-group

The return should be:

{
    "Users": [
        {
            "Path": "/",
            "UserName": "my-user",
            "UserId": "AIDA3IDJ3HNEVBCYBTCVB",
            "Arn": "arn:aws:iam::773316098889:user/my-user",
            "CreateDate": "2020-08-29T21:55:40Z"
        }
    ],
    "Group": {
        "Path": "/",
        "GroupName": "my-group",
        "GroupId": "AGPA3IDJ3HNEYAXDIXQ5K",
        "Arn": "arn:aws:iam::773316098889:group/my-group",
        "CreateDate": "2020-08-29T20:30:14Z"
    }
}

Now, finally let's generate the user access key:

aws iam create-access-key --user-name my-user

The return should be:

{
    "AccessKey": {
        "UserName": "my-user",
        "Status": "Active",
        "CreateDate": "2015-03-09T18:39:23.411Z",
        "SecretAccessKey": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYzEXAMPLEKEY",
        "AccessKeyId": "AKIAIOSFODNN7EXAMPLE"
    }
}

Save the SecretAccessKey and AccessKeyId in a safe place (it is a best practice do that), we will need them further.

3. Considerations before install and configure Zappa

There are two things that Zappa won't help you inside AWS: how to handle your static and media files and how to connect to a database.

So, if you want to use AWS for that also, you will have to handle that inside your Python app.

In my deployment, to keep things simple, I chosen SQLite as database and store the database file in a S3 bucket and I also chosen store the static and media in another S3 bucket (because of that I warmed you to create some extra buckets in the step 1).

Other options are available, for example Aurora or RDS as database (but if you use them, you should have to update the IAM policies that we created before).

To setup Django to use S3 both for store the SQLite database and store the static/media files, we need to install two python packages that will be responsible for that.

Remember to activate your virtualenv, if you are using that or use pipenv

pip install django-s3-storage # Will handle the static/media files
pip install django-s3-sqlite # Will handle the SQLite database

Example of how to configure Django settings.py to manage the static/media files and the SQLite database using S3:

4. Installing and configuring Zappa

Now we will install and configure Zappa, for install we will use pip.

Remember to activate your virtualenv, if you are using that or use pipenv

pip install zappa

For configure Zappa, you need to create the file zappa_settings.json inside the root of your Django application repository (the same place of the manage.py), below follow an example:

dev is the name of the stage that will be created inside the API Gateway.

django_settings must be name_of_your_django_project.settings

s3_bucket must be the bucket that we created in the step 1

You can check my project repo at Github for more information.

5. Deploying the application

Finally, we got to what matters :-)

Before run the command to finally deploy the application inside AWS, we need to define the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables with the access key and secret key of the user that we created in the step 2.6

Remember to activate your virtualenv before, if you are using that or if you are using pipenv, run pipenv shell before

export AWS_ACCESS_KEY_ID=access_key_of_the_user_created_before
export AWS_SECRET_ACCESS_KEY=secret_key_of_the_user_created_before

Now, we can run the command to deploy our application for first time:

zappa deploy dev

dev must be the name of the stage that you defined in the zappa_settings.json

If everything goes well, the return should be something similar to:

...
Deploying API Gateway...
Deployment complete!: https://wf31r9h75a.execute-api.us-west-2.amazonaws.com/dev

And you can use the above URL to test if your application is running properly, if not you can use this command to check the logs and check what happened:

zappa tail dev

And to do some update in your application, you don't need run the deploy command again, just run:

zappa update dev

So, that's all folks, below I'm leaving some links with additional information:

Deploying a Flask App

Deploying using custom domain and SSL certificate

Deploying using Aurora as database

Thanks a lot for the patience!
And have a nice day!

[Fedora 32] How to solve docker internal network issue

Thiago (Zozô) Ozores — Fri, 12 Jun 2020 02:54:49 +0000

Recently I upgraded to Fedora 32 with a fresh install and started to face an issue using docker-compose and docker in general, containers weren't able to talk each other.

After some googling I found that default backend for firewalld was changed from iptables to nftables.

I tried to do the proposed fixes for Docker described in the link above, but without success, so the way to solve the issue for me was put back iptables as firewalld backend.

With those commands below, I was able to solve the issue.

sudo sed -i 's/FirewallBackend=nftables/FirewallBackend=iptables/g' /etc/firewalld/firewalld.conf

sudo systemctl restart firewalld docker

That's all folks, thanks in advance and stay in tune.

Mirroring repositories at Gitlab.com

Thiago (Zozô) Ozores — Tue, 09 Jun 2020 14:07:08 +0000

I'm migrating most part of my repositories to the Gitlab.com, because they ultimately have a nice set of features, but I'm not intended to left Github, because it has another nice set of features and also they have a certain charm ;-)

And with that on mind, I was wondering how to sync my repos from Github to Gitlab and also the reverse. I checked some options to do that, but the simplest option that I've found was to use an awesome feature available at Gitlab.com, called mirroring repositories.

It is possible to mirror any internet acessible git repository in the both ways (push and pull), so if you update in the Gitlab the other repo is updated as well and also the inverse.

So, I synchronized my github repos with gitlab in that way, follow the steps to configure it, below:

1) Go to the project page inside Gitlab

2) In the left sidebar, look for "Settings" option

3) That option will open a set of other options, click in the "Repository" option

4) Expand the "Mirroring repositories" option

5) Fill the "Git repository URL" with the address of your repo, if you use https, you must fill the URL together with the user. Ex.: https://user@github.com/user/repo.git

6) In the combo "Mirror direction" choose if you will do a Push or a Pull (you can configure both at the same time)

7) After choose the authentication method, password if you will use https, public key if you will use ssh

8) If you chose password, you must fill the textbox below with the password of your source repo, if you chose public key you must add a new ssh key.

9) If you want you can check "Keep divergent refs" or "Only mirror protected branches" (I usually don't do that)

10) At last, click in the beautiful green button called "Mirror repository" and that's it.

So, that's it for now.

Local NLB for Kubernetes

Thiago (Zozô) Ozores — Tue, 02 Jun 2020 18:02:45 +0000

I decided to build a Kubernetes infra from scratch using some virtual machines inside my laptop, because I think it's the better way to study for CKA and to know better how Kubernetes works (and since I also don't want to pay a few dollars to a cloud provider, because exchange rates have skyrocketed in Brazil :-) )

The baddest thing to use this kind of infra instead cloud providers is try to expose a service using the LoadBalancer type. Inside the cloud provider such thing is quite easy.

But since I discovered the MetalLB project I don't have such problem anymore.

MetalLB provides a network load balancer implementation for bare metal kubernetes clusters, using standard routing protocols (since Kubernetes doesn't offer one), in other words, I'm able to load balancing requests using the kubernetes way (the right way ;-) ), not using an external nginx to loadbalancing nodeport services.

Below the steps which I did in my environment.

Environment

4 libvirt VM's w/ CentOS 7 (provisioned with Vagrant)
1 master node
3 workers nodes
Kubernetes 1.18
MetalLB 0.9.3

Preparation
First I needed to check, as documentation of MetalLB asks for, if I need to enable the strict ARP mode for kube-proxy. Since Kubernetes 1.14.2 for kube-proxy in ipvs mode, we need to enable strict ARP.

To check it:

kubectl describe cm -n kube-system kube-proxy | grep -i strictarp

If you get this return:

  strictARP: false

You need to enable strictARP, to do that use those commands:

kubectl get cm kube-proxy -n kube-system -o json > strict.json && sed 's/strictARP: false/strictARP: true/g' strict.json && kubectl replace -f strict.json; rm -f strict.json

Installation
Now we are able to install, the documentation provide to us two ways to install, the manifest way and the kustomize way, I chose apply the manifests:

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/namespace.yaml
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.3/manifests/metallb.yaml

We need to generate a secret also (used to encrypt the communication between the speakers)

kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"

After that according the docs those components will be deployed:

This will deploy MetalLB to your cluster, under the metallb-system namespace. The components in the manifest are:

The metallb-system/controller deployment. This is the cluster-wide controller that handles IP address assignments.
The metallb-system/speaker daemonset. This is the component that speaks the protocol(s) of your choice to make the services reachable.
Service accounts for the controller and speaker, along with the RBAC permissions that the components need to function.

At last we need to configure MetalLB defining and deploying a configmap, I used the simplest mode (according the docs, but in my case was true), the Layer 2 mode:

metallb-config.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 192.168.122.2-192.168.122.10

kubectl apply -f metallb-config.yaml

After that MetalLB was able to allocate those IP's (they are not random, they are one part of my available IP's from the virtual network builded by libvirt) as a load balancer.

And now we can test.