DEV Community: Charles Uneze

Cost-Efficient ETL for Small Datasets using AWS Lambda, S3, Wrangler, and Glue

Charles Uneze — Mon, 08 Sep 2025 09:07:27 +0000

While studying the AWS Data Associate certification guide and designing a data pipeline, you will be exposed to data transformation concepts like transforming a file’s format or transforming the actual data.

As a refresher, the file format is often transformed to improve query speed. And columnar-supported file formats are preferred. Popular examples are Parquet and ORC (Optimized Row Columnar).

Transforming the actual data is about performing actions on the rows & columns; things like removing duplicate rows, for example.

The tricky thing about transforming a file format on the cloud is that it can get expensive. And sometimes you don't often need that expensive transformer when you aren’t frequently working with a huge dataset.

The AWS Glue ETL job uses Spark for transformation, and the minimum data processor allocates 4 vCPUs, 16 GB memory, 94GB disk. While studying, this can get expensive.

An alternative is to convert the file using a library. In my case, where I code in Python, the PyArrow library is a good choice. I also had to convert a CSV file, so I added the Pandas library too.

The solution works locally; it converts CSV to Parquet. However, the package installation size is too large to add to Lambda during a trigger. I also didn't want to package and deploy a container image to ECR and tell Lambda to pull it.

The best alternative I ended up figuring out was to use the AWS Wrangler package. It comes preinstalled with Pandas and Pyarrow and can be attached to Lambda’s layer.

How to install AWS Wrangler on Lambda

To install it as a Lambda layer, visit the AWS serverless repository.
Deploy it for free
Once it's completed, it should become available in your Lambda layer.
Now you can write your transformation logic in Python and run it as a Lambda function. You also need to reference the newly added Lambda layer.

Defining the other ETL logics

Create a Glue catalog, database, crawler, trigger, and workflow.
Create an isolated workgroup for your Athena query service based on Presto
Add your CSV dataset to S3
As previously described, your Lambda function reads the dataset on S3:
- Removes duplicate rows
- Converts the CSV into Parquet
- Saves the new file to an S3 path
- Then it triggers your Glue workflow
The Glue workflow immediately does the following using Glue:
- Crawls the Parquet file
- Creates a table
Then you can use Athena to query the data in S3

Demo

Architecture

I have defined this logic using Terraform so that you can play with it. Clone the following repository, update the backend configuration, bucket name, and region, then apply your configuration.

git clone https://github.com/network-charles/aws-etl-wrangler.git
cd aws-etl-wrangler
terraform init -backend-config=backend.conf
terraform apply --auto-approve

Final takeaway

If you are studying the AWS Data Associate using small datasets, once you understand how an ETL job works, you can transition to using AWS Wrangler in your pipeline instead.

Using GitOps to Manage a KWOK Deployment

Charles Uneze — Fri, 14 Mar 2025 15:49:39 +0000

Overview
Requirements
File Tree
Create a Cluster
Install Flux Controllers
Create a Secret for Authenticating to the GitHub Repository
Connect Flux to the Repository
Target the KWOK Manifests using Flux
Apply Custom Resources (CRs) of Stages
Target the Deployment and Node YAML Manifests using Flux
Install Prometheus and Grafana
Enable Metrics on Kube-Scheduler
Grab the Grafana Username and Password
Scale the Deployment to 30 and Modify Node Affinity
View Scheduler Metrics on Grafana Dashboard
Conclusion

Overview

So, I have been studying the GitOps associate exam outline, I decided to document my learning by implementing it on an open-source project I worked on during my past internship experience at DaoCloud.

DaoCloud has an open-source project called Kubernetes without Kubelet (KWOK) which helps developers who are building products on Kubernetes, test their Kubernetes control plane. They do this by creating a huge amount of pods or nodes to stress test the control plane. Since running such tests in a real environment is expensive, KWOK makes this cost-effective.

I haven’t found any resource on the internet showing users how to deploy this using Flux, a GitOps tool, and monitoring the Kubernetes scheduler using Prometheus, this article should help.

High-level architecture of a KWOK deployment using GitOps

It's also worth mentioning that I found a KWOK documentation bug and opened a PR that got merged while I was preparing for this article.

Requirements

File Tree

GitHub repository

├── cluster.yaml
├── gitops
│   ├── flux
│   │   ├── git.yaml
│   │   ├── kwok.yaml
│   │   └── others.yaml
│   │   └── prometheus-grafana.yaml
│   └── kustomization
│       ├── kwok
│       │   ├── base
│       │   │   ├── kustomization.yaml
│       │   │   └── namespace.yaml
│       │   └── overlays
│       │       └── kustomization.yaml
│       └── others
│           ├── base
│           │   ├── deployment.yaml
│           │   ├── kustomization.yaml
│           │   ├── namespace.yaml
│           │   └── node.yaml
│           └── overlays
│               └── modify-number-of-pods
│                   ├── deployment.yaml
│                   └── kustomization.yaml
└── secrets.yaml

We will be working with a few files, so this tree will guide us through it.

Breaking it into sections:

gitops/flux/ → Flux configurations
gitops/kustomization/kwok/ → KWOK setup using Kustomize
gitops/kustomization/others/ → Additional Kubernetes resources
secrets.yaml → Holds authentication credentials

Create a Cluster

The first step is to create a real Kubernetes cluster. KWOK will be deployed into the cluster. Here I will be using a Kubernetes in Docker (KIND) cluster.

kind create cluster --config cluster.yaml

cluster.yaml

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
    extraPortMappings:
      - containerPort: 30000 # The k8s service port for grafana
        hostPort: 3000 # The host port you want to expose the k8s service
        protocol: TCP
      - containerPort: 30909 # The k8s service port for prometheus
        hostPort: 9090 # The host port you want to expose the k8s service
        protocol: TCP

The cluster.yaml file configures port mappings that allow the host to access the Grafana and Prometheus services once they are deployed into the Kind cluster for observability.

Install Flux Controllers

Flux is a GitOps tool that takes the infrastructure code defined in a Git repository and automatically deploys it into a Kubernetes cluster. It also watches for changes in the repository and synchronizes those changes in the cluster.

All these interactions are handled by the controllers.

flux install

kubectl get pod -n flux-system

NAME                                       READY   STATUS    RESTARTS        AGE
helm-controller-5bb6849c4f-rbrrj           1/1     Running   1               5m
kustomize-controller-68597c4488-9vtnl      1/1     Running   1               5m
notification-controller-7d6f99878b-plvzl   1/1     Running   1               5m
source-controller-666dc49455-z4gs7         1/1     Running   1               5m

Create a Secret for Authenticating to the GitHub Repository

Flux needs to authenticate to your repository because this repository may be private.

echo -n "testing" | base64
echo -n "testing1234" | base64

Your username and password need to be converted to base64. The result will be specified as the value.

secrets.yaml

apiVersion: v1
kind: Secret
metadata:
  name: github-auth
  namespace: flux-system
type: Opaque
data:
  username: dGVzdGluZw==
  password: dGVzdGluZzEyMzQ=

kubectl apply -f secrets.yaml

Connect Flux to the Repository

├── gitops
├── flux
│   │   ├── git.yaml
│   │   ├── kwok.yaml
│   │   └── others.yaml

kubectl apply -f gitops/flux/git.yaml

flux/git.yaml

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: git
  namespace: flux-system
spec:
  interval: 1m # Syncs to the repo every 1 minutes
  url: https://github.com/network-charles/kwok-gitops.git
  ref:
    branch: main
  secretRef:
    name: github-auth

For real-world production workloads, a longer interval (e.g., 5-10 minutes) is recommended to reduce API rate limits and unnecessary sync attempts.

Target the KWOK Manifests using Flux

When working with Flux, you can use a tool called Kustomize to place your main configs in a base directory, and then modify any of these configs using a copy placed in an overlays directory. It’s preferable this way than messing with your main config.

├── gitops
│     ├── flux
│     │   │   └── kwok.yaml
│       └── kustomization
│       ├── kwok
│       │   ├── base
│       │   │   ├── kustomization.yaml
│       │   │   └── namespace.yaml
│       │   └── overlays
│       │       └── kustomization.yaml

kubectl apply -f gitops/flux/kwok.yaml

flux/kwok.yaml

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: kwok
  namespace: flux-system
spec:
  interval: 1m # Syncs to the repo every 1 minutes
  targetNamespace: kwok
  sourceRef:
    kind: GitRepository
    name: git
    namespace: flux-system
  path: "/gitops/kustomization/kwok/overlays"
  prune: true
  timeout: 1m # time it takes for the manifest to finish applying before timing out

Flux targets the directory/path containing the file I will use to deploy KWOK into my cluster.

The manifest below is the targeted file. I am attempting to modify the image version used to deploy KWOK into my cluster. Since I am already referencing the base directory via the resources field, Flux will create everything there first, afterward, it’d modify any image version changes I specify in the patches and images fields.

kustomization/overlays/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../base
patches:
  - target:
      kind: Kustomization
    patch: |-
      apiVersion: kustomize.config.k8s.io/v1beta1
      kind: Kustomization
      metadata:
        name: kwok
        namespace: kwok
      resources:
        - https://github.com/kubernetes-sigs/kwok/kustomize/kwok?ref=v0.5.1
images:
  - name: registry.k8s.io/kwok/kwok
    newTag: "v0.5.1"

Apply Custom Resources (CRs) of Stages

This makes it possible to simulate the state of a pod or node in the cluster with very little resources.

KWOK_REPO=kubernetes-sigs/kwok
KWOK_LATEST_RELEASE=$(curl "https://api.github.com/repos/${KWOK_REPO}/releases/latest" | jq -r '.tag_name')
kubectl apply -f "https://github.com/${KWOK_REPO}/releases/download/${KWOK_LATEST_RELEASE}/stage-fast.yaml"

Target the Deployment and Node YAML Manifests using Flux

├── gitops
│     ├── flux
│     │   │  └── others.yaml
│     └── kustomization
│       └── others
│           ├── base
│           │   ├── deployment.yaml
│           │   ├── kustomization.yaml
│           │   ├── namespace.yaml
│           │   └── node.yaml
│           └── overlays
│               └── modify-number-of-pods
│                   ├── deployment.yaml
│                   └── kustomization.yaml

kubectl apply -f gitops/flux/others.yaml

Now all manifests in this others directory will be managed by Flux.

gitops/flux/others.yaml

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: others
  namespace: flux-system
spec:
  interval: 1m # Syncs to the repo every 1 minutes
  sourceRef:
    kind: GitRepository
    name: git
    namespace: flux-system
  path: "/gitops/kustomization/other/overlays/modify-number-of-pods"
  prune: true
  timeout: 1m

Notice how the overlays directory is still been referenced and used to make changes to the main deployment.

Install Prometheus and Grafana

This deploys Prometheus and Grafana using Helm, but it's managed by Flux. I like this because I don’t need to manage Helm manually using the command line.

prometheus-grafana.yaml

apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: prometheus-community
  namespace: flux-system
spec:
  interval: 1m
  url: https://prometheus-community.github.io/helm-charts
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: prometheus-grafana
  namespace: flux-system
spec:
  targetNamespace: monitoring
  interval: 1m
  chart:
    spec:
      chart: kube-prometheus-stack
      version: "*"
      sourceRef:
        kind: HelmRepository
        name: prometheus-community
        namespace: flux-system
  install:
    createNamespace: true
  values:
    prometheus:
      service:
        nodePort: 30909
        type: NodePort
    grafana:
      service:
        type: NodePort
        nodePort: 30000

kubectl apply -f gitops/flux/prometheus-grafana.yaml

Enable Metrics on Kube-Scheduler

For security purposes, Kube-Scheduler is not accessible by default. So, metrics aren’t scraped, but if you are running tests and need to collect metrics from it, you need to bind the address and make it accessible to other pods like the Prometheus pod, etc.

docker exec -it kind-control-plane bash
apt update
apt install vim -y
vi /etc/kubernetes/manifests/kube-scheduler.yaml

spec:
  containers:
  - command:
    - kube-scheduler
    - --bind-address=0.0.0.0  # Allow listening on all interfaces

Now the Prometheus service monitor scraping metrics from the scheduler will be in an up state.

kube-scheduler service monitor

Grab the Grafana Username and Password

kubectl get secrets prometheus-grafana -o jsonpath='{.data.admin-password}' | base64 --decode ; echo
kubectl get secrets prometheus-grafana -o jsonpath='{.data.admin-user}' | base64 --decode ; echo

Scale the Deployment to 30 and Modify Node Affinity

Initially, I had one pod running in the deployment.

kubectl get deployments apps nginx-deployment

NAME               READY   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment   1/1     1            1           5s

Now, I’ll scale the deployment to 30. To do this, I will modify a copy of the deployment stored in the overlays directory.

├── gitops
│   └── kustomization
│       └── others
│           ├── base
│           └── overlays
│               └── modify-number-of-pods
│                   ├── deployment.yaml
│                   └── kustomization.yaml

I updated the replicas field and the values in the deployment file.

overlays/modify-number-of-pods/deployment.yaml

replicas: 30 # update this
...
affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
        - matchExpressions:
            - key: type
              operator: In
              values:
                - kwokkk # update this
...

The above manifest updates the deployment to 30 pods, it also uses the wrong label value, so the pods are unschedulable.

I intentionally made them unschedulable so we can see some scheduler metrics in action.

Now, push your config to GitHub and wait for 60s, so Flux pulls and updates your cluster.

git commit -am "update deployment"
git push
sleep 60
kubectl get deployments apps nginx-deployment

NAME               READY   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment   0/30    30           0           16m

View Scheduler Metrics on Grafana Dashboard

Log into Grafana by visiting localhost:3000. The username and password you received maybe this.

username = admin
password = prom-operator

Go to Menu > Dashboard > Kubernetes/Scheduler> Edit > Add > Visualization > Queries > Metrics

Add a metric to view all pending and unschedulable pods.

scheduler_pending_pods{queue="unschedulable"}

You should see a graph similar to this, telling you 30 pods are currently pending.

Prometheus metrics graph on Grafana dashboard

Conclusion

Now, we have learned how engineers and developers use GitOps to manage their KWOK deployment, collect metrics from the control-plane, and view them on Grafana. This method works on several other use cases. In a public cloud's managed Kubernetes clusters, the cluster’s control plane node is often restricted, however, the cloud provider exposes the Prometheus metrics to you. If you manage your control plane node, then this article should help you learn to expose yours for testing.

To find more control-plane use cases for KWOK, see this GitHub issue.
If you’d like an opportunity to intern in the Cloud Native Computing Foundation space, see the Linux Foundation Mentorship Program and Google Summer of Code.

Wikimedia Foundation GSOD 2024 Project Report

Charles Uneze — Fri, 01 Nov 2024 08:01:50 +0000

Project Overview
Key Challenges
Project Outcome
Key Learning
Best Practices Identified
Recommendations for Future Projects
Conclusion
More Resources

Project Overview

This is a report for a project I worked on with the Wikimedia Technical Documentation team during the Google Season of Docs 2024 program.

The "Complete Migration of MediaWiki Documentation" project aimed to migrate an 82-page document from an old official wiki called Meta to a new one called MediaWiki.

This wasn’t just about transferring text. A MediaWiki server had to be installed locally to test each feature described before documenting it again, and the new text had to be easier to read, navigate, and use. For maximum server uptime, I deployed the MediaWiki server on Amazon Web Services (AWS).

I also had to ensure existing content on Meta that is Wikimedia-specific is easier to navigate and understand.

Key Challenges

During the migration, we identified five main challenges:

Scattered Information: Some content about MediaWiki is located on Meta, making it hard for users to find everything in one place.
New Look and Feel: Since MediaWiki is a new software, it came with a new user interface in some places.
Rewriting for Clarity: We couldn’t just copy-paste text from Meta due to licensing issues.
Inconsistent Organization: Similar Wikimedia Foundation-specific pages on Meta weren’t well organized into a category.
Poor Navigation: Meta’s content navigation page which should provide easy access to most visited pages wasn’t properly structured.

Project Outcome

The project led to several major improvements:

Centralized Information: All key MediaWiki documentation is now on MediaWiki. This will save users from switching between platforms and help them find the latest information faster.
Updated Visuals: New images have been added to match the new interface, so users see the exact screens they’ll be working with, reducing confusion.
User-Friendly Writing: All migrated contents are now more readable and engaging, making the documentation more approachable for all users.
Better Organization: With improved categorization, users can now quickly find similar pages they need on Meta.
Enhanced Navigation: Meta's revamped content navigation page now leads users to the most frequently visited pages, so they can get to popular guides without wasting time. I used a wiki Pageviews Analysis tool to rank the most visited pages.

The progress of this outcome was first documented on a Google Sheet maintained by a Google Season of Docs (GSOD) volunteer named Okereke Chinweotito. A copy was then replicated on a Wikimedia Phabricator issue as a bi-weekly status update.

Key Learning

My knowledge improved in these areas:

When editing a wiki, users make use of a markup language called wikitext. This project helped me improve my knowledge of this language.
I learned how to navigate a wiki and perform operations like installing extensions, renaming a page, merging a page, adding permissions to a user, etc.
I also had to brush up on my HTML skills all over again.

Best Practices Identified

While working on this project, these were the best practices I identified that were missing:

Start with Contextual Introductions: Each page should begin with a brief introductory paragraph that explains what the feature does. This provides users with context before diving into instructions on how to use the feature.
Remove redundant lines: Between the end of a section or subsection and the start of another, always leave one empty line rather than two.
Add Visual Aids Where Relevant: When describing some operations that involve navigating the menu, it may be best to add a screenshot too.
Migrate technical pages to the manual namespace: Pages containing extensive code or mathematical equations are best suited in the Manual namespace. This helps users find more technical content quickly.
Test locally before documenting: As much as you can, it’s best to test the feature on a local MediaWiki server first before documenting it.

Recommendations for Future Projects

I am going to start by outlining the key factors that contributed to the success of this project:

This project was well documented before it began, with all issues properly specified, so contributors could work on it. Unlike a few other projects I saw that weren’t selected by Google.
Interested Technical Writers were instructed to outline their statement of interest on Wikimedia Phabricator rather than having to join the Zulip group chat.
The community members were very active and supportive, particularly experienced wiki administrators like Pppery.

For future projects, maintaining detailed documentation of the issues to be addressed when submitting a proposal to Google and promoting an engaged community with a smooth onboarding process will be key strategies to replicate this success.

Conclusion

By migrating MediaWiki content from Meta to the MediaWiki platform and organizing Wikimedia-specific help pages on Meta, I created a more accessible and user-friendly resource for the Wikimedia community. This project strengthened my technical documentation skills and highlighted the value of community collaboration in open source. I look forward to continuing to support Wikimedia’s mission by contributing further to its documentation.

More Resources

My Experience Working on the KWOK Project as an LFX Mentee

Charles Uneze — Tue, 27 Aug 2024 11:23:54 +0000

What’s the KWOK Project About?
How I Got Accepted
- Step 1: Use the Project
- Step 2: Analyze the Expected Outcome
New Concepts I Learned
What I Struggled With
My Contributions
Conclusion

In June 2024, I got the amazing notification that I’d been selected as an LFX mentee in Term 2, 2024. This was my second attempt to participate in the LFX mentorship program. I found out about the program from Divine Odazie in one of his Twitter spaces.

I was selected to work on the Kubernetes without Kubelet (KWOK) project with mentorship support from the DaoCloud software engineering team, Shiming Zhang and Zhenghao Zhu.

What’s the KWOK Project About?

Source: CNCF

KWOK is a Kubernetes sub-project, people often refer to these sub-projects as Kubernetes Special Interest Group (SIG) projects.

It allows developers to simulate large clusters with hundreds of nodes and pods that consume little resources. It uses real Kubernetes components but operates without a Kubelet.

How I Got Accepted

I never knew about the project in the past. However, I was studying the Certified Kubernetes Administrator (CKA) topics, so I understood how Kubernetes works from the perspective of an administrator. I just needed an extra push to understand the KWOK project.

Step 1: Use the Project

I spun up a virtual machine on AWS and installed the requirements needed to test the project in action. I tried various scenarios, as you would when you are learning a certification like the CKA. You can find the scenarios I tried in this repository.

Along the way, I noticed issues both in the project documentation and in the code outputs when interacting with the tool via the CLI. For each issue, I opened a PR, and when I got confused, I shared my questions on the Slack channel.

Step 2: Analyze the Expected Outcome

Each project specifies its expected outcome. While building my confidence, I made sure to know that I could achieve the outcome. So I reviewed each project outlined in the referenced GitHub issue. I read each pull request and blog post to learn how the project uses KWOK. When I was confident that I understood it to a great extent, I sent in my mentee application.

New Concepts I Learned

New cloud native projects: While working on the project, I had the opportunity to learn about other projects too. For example, Kubevirt, Kyverno, Clusterpedia, etc.
How to generate an image from a bash script: A tool called Asciiema allows developers to test their bash script in action while also recording the execution from their terminal. This is helpful because it validates that your script works. There were times when I assumed my documentation was accurate, however, when I tried the instructions using the tool, it failed. Thankfully, I got it fixed.
How other cloud-native projects utilize custom control planes: While studying the CKA topics, I only knew about the Kubernetes control plane. This mentorship exposed me to other cloud-native tools that use additional custom control planes for their specific needs.
How projects uniquely utilize etcd: I initially knew that configuration data was stored in etcd. However, I never knew that a project could utilize it to store other important information. For example, Kyverno uses it to store policy reports.

What I Struggled With

Git: I struggle with not knowing when to use a git rebase vs git merge. KWOK rolled out a new feature that needed to be integrated into my branch. I wanted to show only my commit history in my pull request, but I was confused about which command to use. I ended up using a git rebase because it produces a cleaner commit history. It integrates the new changes into my branch without including the commit history from the original branch in my pull request.
Time difference: My mentor is based in China, so he is 7 hours ahead of me. When resolving a GitHub suggestion, I often need to start addressing issues from 4 or 5 a.m. in my time zone.

My Contributions

Conclusion

The mentorship program was an amazing learning experience. I don't think the LFX mentorship is an easy program to get into. Most of the time, you need at least a CKA, CKAD, or CKS level of understanding to stand a chance. My tasks involved a lot of references to CKA topics.

This experience broadened my understanding of Kubernetes, and I’m excited to apply the skills and knowledge I have gained to new cloud-native projects and also to continue contributing to open-source software.

I’m also thankful to my mentors for giving me a chance to showcase and improve my skills. I encourage others who’d like to break into the cloud-native industry to leverage the LFX mentorship program to get hands-on experience, you won’t regret it.

Utilizing Coverage AI Agents for Better Unit Tests

Charles Uneze — Tue, 21 May 2024 05:02:52 +0000

Artificial Intelligence agents have improved developers’ workflow in the last few years and the release of this paper from Meta about their TestGen-LLM has been a game changer for unit testing code.
CodiumAI, an AI startup is at the forefront of the commercial application of the TestGen-LLM with their new open-source tool. What I love about this TestGen-LLM implementation is that it keeps iterating the code until coverage increases. See below.

Source: Automated Unit Test Improvement using Large Language Models at Meta

CodiumAI’s Cover-Agent

Pre-coverage AI agents, Python developers would use the Coverage python plugin or the Pytest-cov plugin, while Go developers use go test -coverprofile=... (See Go coverage profile), then they’ll have to manually write tests to improve coverage in the lines missed in their code.
Today, tools like Cover-Agent do this automatically.

Demo

Below is a simple script that deploys AWS services using Python’s Boto3 library.

Installation
Follow this guide on the CodiumAI Cover-Agent GitHub repository to install the necessary packages.
The following packages are also required:

Boto3
Moto



pip install boto3 moto

Code to test
A simple production script that creates an EC2 instance, a Dynamodb table, and puts an item into the table.

File: aws.py



import time
import boto3

dynamodb = boto3.resource('dynamodb')
ec2 = boto3.resource('ec2')

def create_ec2_instance():
    # Define parameters for the instance
    instance_params = {
        'ImageId': '12345',
        'InstanceType': 't2.micro',
        'KeyName': 'key',
        'MinCount': 1,
        'MaxCount': 1
    }

    # Create the EC2 instance
    instances = ec2.create_instances(**instance_params)

    # Wait for the instance to be in running state
    instance = instances[0]
    while instance.state['Name'] != 'running':
        time.sleep(5)  # Wait for 5 seconds before checking again
        instance.reload()  # Reload the instance object to get the latest state
 # Return the instance state
    return instance

def create_dynamodb_table_and_put_item():
    """
    Create a DynamoDB table and put an item in it.
    """

    # Create a DynamoDB table
    table = dynamodb.create_table(
        TableName='AMI_Table',
        KeySchema=[
            {'AttributeName': 'id', 'KeyType': 'HASH'}
        ],
        AttributeDefinitions=[
            {'AttributeName': 'id', 'AttributeType': 'N'}
        ],
        ProvisionedThroughput={
            'ReadCapacityUnits': 5,
            'WriteCapacityUnits': 5
        }
    )

    # Wait for the table to be created (this is important!)
    table.wait_until_exists()

    # Get the table resource
    get_table = dynamodb.Table('AMI_Table')

    # Put item in the table
    get_table.put_item(
        Item={
            'id': 1,
            'PK': 'Partition_Key',
            "REGION_AMI_ID": "ami-123",
            "AMI_State": "Create_AMI"
        }
    )

Below is another script that tests against the production code. This script has only one test, which tests the create_ec2_instance() function to see if the instance ends up in the running state.

File: test_aws.py



import boto3
from moto import mock_aws
from datetime import datetime
from aws import create_ec2_instance, create_dynamodb_table_and_put_item

@mock_aws
def test_create_ec2_instance():
    # Create the instance
    instance = create_ec2_instance()
    # Assert that the instance is in a running state
    assert instance.state['Name'] == 'running'

Check current coverage percentage
Below is how much coverage my current script has (see pytest-cov configuration references to understand what each argument means):



$ pytest --cov=aws --cov-report=xml --cov-report=term

Name     Stmts   Miss  Cover
----------------------------
aws.py      17      4    76%
----------------------------
TOTAL       17      4    76%
Coverage XML written to file coverage.xml

A 76% coverage.
While this is nice, it isn't great.

Check the missed line in the coverage.xml file

These lines belong to the imported function that was not executed during the initial test.



<line number="35" hits="0"/>
<line number="50" hits="0"/>
<line number="53" hits="0"/>
<line number="56" hits="0"/>

For a better visual, I love the coverage html command. Then I navigate to the htmlcov folder and open the aws_py.html file. See below:

File: htmlcov/aws_py.html

Now, you can go ahead and write tests for each line to increase the coverage, however, if this is a large codebase, it’s exhausting too.

Run Cover-Agent
I am adjusting the coverage from 76% to 77%, so the AI can increase it to 100% by itself. See below:



$ cover-agent \
  --source-file-path "aws.py" \
  --test-file-path "test_aws.py" \
  --code-coverage-report-path "./coverage.xml" \
  --test-command "pytest --cov=. --cov-report=xml --cov-report=term" \
  --test-command-dir "./" \
  --coverage-type "cobertura" \
  --desired-coverage 77 \
  --max-iterations 5

cover_agent.main - INFO - Current Coverage: 76.47%
cover_agent.main - INFO - Desired Coverage: 77%
cover_agent.UnitTestGenerator - INFO - Token count for LLM model gpt-4o: 1930

Streaming results from LLM model...

Test passed and coverage increased. Current coverage: 100.0%

Some texts from the output are removed to improve readability.

A part of the output says:
Test passed and coverage increased. Current coverage: 100.0%
So, I was impressed. But I need to be certain that it's not hallucinating.

View the new test that increased the coverage

Any new test(s) that increase the coverage are automatically added to the test file.

File: test_aws.py



@mock_aws
def test_create_dynamodb_table_wait_until_exists():
    """
    Test that the DynamoDB table waits until it exists before proceeding.
    This test ensures that the wait_until_exists method is called and the table is ready for operations.
    """
    # Create the DynamoDB table and put an item in it
    create_dynamodb_table_and_put_item()

    # Get the table resource
    dynamodb = boto3.resource('dynamodb', region_name='eu-west-2')
    table = dynamodb.Table('AMI_Table')

    # Assert that the table exists and is active
    table.wait_until_exists()
    assert table.table_status == 'ACTIVE'

Check if the test passed



$ pytest

collected 2 items                                                                                                                  

test_aws.py ..                            [100%]

============== 2 passed in 6.78s ===============

Some texts from the output are removed to improve readability.
And yes, it does.

See the coverage report again



$ pytest --cov=aws --cov-report=xml --cov-report=term

Name     Stmts   Miss  Cover
----------------------------
aws.py      17      0   100%
----------------------------
TOTAL       17      0   100%
Coverage XML written to file coverage.xml

Some texts from the output are removed to improve readability.

View the list of all test results

To see the full result view the test_results.html file in your browser.

The image appears in low quality due to how dev.to rendered it. View the high quality image here.

Some of the tests in the table failed because coverage did not increase, but it doesn’t mean this isn’t a suitable test case.

Why did only one function improve coverage?

Now, I expected four new tests for each line missed while checking the coverage initially. But this single new test, test_create_dynamodb_table_wait_until_exists(), does indeed contribute to coverage beyond just the lines it explicitly touches. Here's how:

Line 35: table = dynamodb.create_table( Although this was not directly executed in the test, the test indirectly verifies that the table exists and is active after creation. This implies that the creation process is happening correctly before the wait_until_exists()call. So, indirectly, the test ensures coverage for the table creation line as well.
Line 50: table.wait_until_exists() This line is directly covered by the test, as it's part of the test logic.
Line 53: get_table = dynamodb.Table('AMI_Table') Although the test doesn't include this line, it's indirectly tested because if this line failed, the subsequent put_item() call on get_table would fail, causing the test to fail.
Line 56: get_table.put_item( Like line 53, this line is indirectly covered because the test verifies that the table is active and ready for operations, including putting items into it.

Why should you generate coverage reports?

Generating coverage reports is essential for several reasons:

Code Quality Assessment: Coverage reports help in assessing the quality of your codebase by indicating which parts of your code are being exercised by tests and which are not. Higher code coverage generally indicates a more thorough test suite.
Identifying Untested Code: It helps identify areas of your code that are not covered by tests, which may indicate potential bugs or unhandled edge cases. Just like the demo above.
Code Reviews: Coverage reports can be useful during code reviews to ensure that new code changes are adequately tested and do not decrease overall coverage.

Overall, generating coverage reports is a valuable practice for maintaining and improving the quality of your codebase.

Docker ARP Analysis Guide

Charles Uneze — Tue, 16 Jan 2024 21:49:16 +0000

Introduction
Requirements
- Install Homebrew & Multipass
- Download an Ubuntu Server from Multipass
- Go to the Shell of the Ubuntu Server
- Install Docker
- Mount a Local Folder to an Ubuntu Directory
- Verify Ubuntu Server Installation
- Install Wireshark
Architecture Deployment Guide
- Enter into the Shell of the Ubuntu Server Again
- Create a New Network
- Verify Network
- Pull Alpine Image
- Listen for ARP Packets in each Bridge
- Create 2 Containers in the Default Bridge, Also Connect Them to the Custom Bridge
- Send Pings to the Internet From the First Interface
- Send Pings to the Internet From the Second Interface
- View Mac Addresses of Each Bridge Interface
- View Mac Addresses of Each Container Interface
- End ARP Packet Capture 1
- End ARP Packet Capture 2
- Move the Files to the Local Directory
View Packet Captures
- Docker0 Bridge Interface
- Docker1 Bridge Interface
Clean-Up
- Stop and Remove the Container
- Delete Custom Bridge
- Verify Network List
- Stop Ubuntu Server
Conclusion
Resources

Read this as technical documentation instead.

Introduction

I've been immersing myself in CI/CD pipeline studies lately, and now that I've gotten a good grasp of it, I can finally dedicate my full attention to understanding container technologies. To start, I decided to focus on networking, which is the stack I'm most familiar with.
In this simple how-to guide, I'll walk you through observing the Address Resolution Protocol (ARP) in action within a Docker environment.
An Address Resolution Protocol(ARP) allows a container to learn the MAC address of another device (in this lab, it’s the bridge and other containers) dynamically.
Without an ARP, a ping between containers, external networks, or even a web request between containers and other devices fails.

Requirements

I recently made the switch to a Mac and found using Multipass from Canonical simpler than spinning up a VM with tools like Virtualbox. However, when it comes to container networking labs, non-Linux systems are not recommended. While I downloaded Docker Desktop for Mac, I had trouble seeing all the Docker network interfaces.
For Windows users, I will advise you to use WSL2 instead, it’s easier to deploy and manage compared to having to use a virtual machine like Virtualbox.
Keep in mind that the Ubuntu installation requirement is only for Mac and Linux users. This guide will use three terminal tabs throughout.

Install Homebrew and Multipass

Download Homebrew



# Terminal-1 Mac/Linux
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Download Multipass



# Terminal-1 Mac/Linux
brew install multipass

Download an Ubuntu Server from Multipass

This Ubuntu server will be customized to run the same specifications as an AWS EC2-T2 Micro instance type.

1 CPU
1 GB of RAM
8 GB of disk
version 22.04

Simple and fast, right?



# Terminal-1 Mac/Linux
multipass launch jammy --name=ubuntu --cpus=1 --disk=8G --memory=1G

jammy is the image name for Ubuntu server version 22.04.

Go to the Shell of the Ubuntu Server



# Terminal-1 Mac/Linux
multipass shell ubuntu

Install Docker

Download docker on the Ubuntu server.



# Terminal-1 Ubuntu
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

# Close this Ubuntu shell.
# Sometimes you might need to use the exit command severally
# to successfully exit the shell.
exit

Mount a Local Folder to an Ubuntu Directory

Mount the Documents/ folder in your Mac, or other machine to the mnt/ directory in the Ubuntu server running inside Multipass.



# Terminal-1 Mac/Linux
multipass mount /Users/apple/Documents /mnt/

Verify Ubuntu Server Installation



apple@Charles-MBP ~ % multipass info ubuntu
Name:           ubuntu
State:          Running
Snapshots:      0
IPv4:           192.168.64.4
                172.17.0.1
Release:        Ubuntu 22.04.3 LTS
Image hash:     9dxa2awl28c8 (Ubuntu 22.04 LTS)
CPU(s):         1
Load:           0.00 0.00 0.00
Disk usage:     2.3GiB out of 7.7GiB
Memory usage:   201.4MiB out of 951.6MiB
Mounts:         /Users/apple/Documents => /mnt
                    UID map: 501:default
                    GID map: 20:default

Install Wireshark

Choose your preferred machine and download.

Architecture Deployment Guide

I have a simple architecture that deploys two docker containers in two different subnets. In docker, you can attach one container to several subnets. This is achieved by a new interface being created and assigned to that subnet.

Enter Into the Shell of the Ubuntu Server Again



# Terminal-1 Mac/Linux
multipass shell ubuntu

Create a New Network

I noticed that in docker, you only specify a subnet and mask. This makes sense because if you are deploying this on AWS, a VPC will be defined already, all you need to do is create a new subnet mask for your containerized environment.



# Terminal-1 Ubuntu
docker network create \
-o com.docker.network.bridge.name=docker1 \
--subnet=172.18.0.0/24 \
--gateway=172.18.0.253 \
custom_bridge

Verify Network



# Terminal-1 Ubuntu

# This is a simulated command and output
ubuntu@ubuntu:~$ docker network ls
NETWORK ID     NAME            DRIVER    SCOPE
6e5ffkvms8c3   bridge          bridge    local
**d3b0029faed7   custom_bridge   bridge    local**
e9f55dsdf605   host            host      local
e9708nj5179a   none            null      local

Pull Alpine Image

I love using Alpine Linux because it’s lightweight.



# Terminal-1 Ubuntu
docker pull alpine

Open a New Terminal

Execute this command to open a new tab. "⌘ + T"

Then enter the Ubuntu shell



# Terminal-2 Mac
multipass shell ubuntu

Listen for ARP Packets in Each Bridge

Now that the Ubuntu shell has been initialized, execute the below command to capture all packets.

docker0



# Terminal-2 Ubuntu
sudo tcpdump -i docker0 -w capture_docker_0.pcap

Open a third terminal tab "⌘ + T"

docker1

Execute another command to listen for all packets in the docker1 bridge interface.



# Terminal-3 Ubuntu
sudo tcpdump -i docker1 -w capture_docker_1.pcap

Create 2 Containers in the Default Bridge, Also Connect Them to the Custom Bridge



# Terminal-2 Ubuntu

# Create containers in the default bridge
docker run -itd \
--name=alpine1 \
--ip=172.17.0.2 \
alpine

docker run -itd \
--name=alpine2 \
--ip=172.17.0.4 \
alpine

# Connect new interfaces in the containers to another network.
docker network connect \
--ip=172.18.0.3 \
custom_bridge alpine1

docker network connect \
--ip=172.18.0.5 \
custom_bridge alpine2

Send Pings to the Internet From the First Interface



# Ping google.com four times in each container from 'bridge'
docker exec -it alpine1 ping -I 172.17.0.2 -c 2 google.com
docker exec -it alpine2 ping -I 172.17.0.4 -c 2 google.com

Send Pings to the Internet From the Second Interface



# Ping google.com four times in each container from 'bridge'
docker exec -it alpine1 ping -I 172.18.0.3 -c 2 google.com
docker exec -it alpine2 ping -I 172.18.0.5 -c 2 google.com

View MAC Addresses of Each Bridge Interface

The Organizationally Unique Identifier (OUI) of all Docker network adapters is 02:42. So expect all docker container MAC addresses to begin with that.



# Terminal-1 Ubuntu
ip --brief link | grep -E 'docker0|docker1' | awk '{print $1, $3}'

Output



docker0 02:42:28:a8:cb:f5
docker1 02:42:7c:61:6d:f0

View Mac Addresses of Each Container Interface

bridge



# Terminal-1 Ubuntu
docker network inspect bridge --format '{{range .Containers}}{{.Name}}: {{.MacAddress}}{{"\n"}}{{end}}'

Output-1



alpine1: 02:42:ac:11:00:02
alpine2: 02:42:ac:11:00:03

custom_bridge



# Terminal-2 Ubuntu
docker network inspect custom_bridge --format '{{range .Containers}}{{.Name}}: {{.MacAddress}}{{"\n"}}{{end}}'

Output-2



alpine1: 02:42:ac:12:00:03

alpine2: 02:42:ac:12:00:05

End ARP Packet Capture 1



# Terminal-2 Ubuntu

"control + c"

End ARP Packet Capture 2



# Terminal-3 Ubuntu

"control + c"

Move the Files to the Local Directory

The captured packet files will be moved to the local directory so they can be viewed and analyzed using Wireshark.



# Terminal-1, 2, or 3 Ubuntu

mv capture_docker_0.pcap /mnt

mv capture_docker_1.pcap /mnt

View Packet Captures

Docker0 Bridge Interface

Now I located and opened the packet capture.

We noticed something interesting: The default MAC addresses of alpine1: 02:42:ac:11:00:02, and alpine2: 02:42:ac:11:00:03, were requesting the MAC address for the bridge's interface, docker0: 02:42:28:a8:cb:f5, to send an Ethernet frame to google.com. The bridge's interface was used as the next hop.

We also see that the DNS name server lookup for google.com could be possible only after an ARP reply from docker0: 02:42:28:a8:cb:f5.

Docker1 Bridge Interface

The same also applies here. the custom MAC addresses of alpine1: 02:42:ac:12:00:03, and
alpine2: 02:42:ac:12:00:05 were requesting the mac-address for the bridge’s interface docker1: 02:42:7c:61:6d:f0 so it can send an ethernet frame destined to google.com. The bridge’s interface is used as the next hop.

Go to, View Mac Addresses of Each Bridge Interface and View Mac Addresses of Each Container Interface to confirm the mac-addresses of each device when analyzing the packet capture.

Clean-Up

Stop and Remove the Container



docker stop alpine1 alpine2

docker rm alpine1 alpine2

Delete Custom Bridge



docker network rm custom_bridge

Verify Network List



root@ubuntu:/home/ubuntu# docker network ls

NETWORK ID     NAME      DRIVER    SCOPE

2eabde4e866c   bridge    bridge    local

e9f54b21c605   host      host      local

e9708605179a   none      null      local

Stop Ubuntu Server



multipass stop ubuntu

Conclusion

Now, I understand that everyone can't stop talking about Kubernetes, but a lot of senior engineers have advised that it'd be best to learn docker before picking kubernetes up. Though I've played with Kubernetes severally, I struggled. However, each new day I keep spending with docker makes understanding kubernetes a piece of cake. This guide serves as a strong foundation for analyzing ARP packets in containers.

Resources

I enjoyed these two articles from Hank Preston, a principal engineer at Cisco. The last one is from me.

Exploring Default Docker Networking Part 1
Exploring Default Docker Networking Part 2
A Routers Intimacy With MAC Addresses (I wrote this a few years ago. It discusses ARP and ICMP in a Cisco environment.)

CodiumAI PR-Agent Dominates the Dev World with Versatility and Open-Source Power

Charles Uneze — Sun, 03 Dec 2023 13:09:12 +0000

In the fast-paced world of software development, where every line of code matters, effective collaboration is non-negotiable. Yet, the often-overlooked practice of thorough pull request documentation can hinder the clarity and efficiency crucial to a project’s success.

CodiumAI PR-agent (Git Plugin) is an automatic action tool that responds to pull request (PR) events in or to your repository; PR created, PR merged, Issues Created, and Labels created.

In this article, we explore how this solution empowers developers, streamlines workflows, and elevates the collaborative coding experience, in comparison to the GitHub Co-pilots PR solution.

Key Features

To describe some of these features, I’ll use a recently merged pull request I made to the CodiumAI PR agent project. A live scenario like that makes it easy to understand.

Auto Description (`/describe`) Prompt:

I made a PR to the CodiumAI PR agent project on GitHub. The maintainer wanted to properly format the PR message I made, so he queried the agent via the /describe prompt.

Prompt made by an Open Source Maintainer

This prompt generated a description of the PR. It contains the following: title, type, description, main files walkthrough, and labels. It also edited my initial comment.
1) Title
The Title of the PR is renamed when the prompt /describe is called. Below we can see the before and after effects of this.

2) Type
This is the label name. The “Refactoring” label name was used because, in the commit I made, I was restructuring the code to improve its readability
3) Description
This is a summary of the PR.
4) Main Files Walkthrough
This provides a closer explanation of what was changed in the file I committed in this PR.

Contributors' comments were edited due to the prompt

5) Labels
This labels the PR properly.

Question Answering (`/ask ...`) Prompt:

This allows a maintainer to question the commit more to be sure if it's worth a merge.

In this PR I opened, the maintainer asked more questions, and when he was satisfied, he merged the PR. This is a real-world implementation and it's amazing how the agent provided great feedback.

Update Changelog (`/update_changelog`) Prompt:

This prompt updates the change log file in the repository. A change log file, often named CHANGELOG.md in a repository, tracks and describes version-specific updates, like new features or bug fixes, aiding users and contributors.

Auto Review (`/review`) Prompt:

This is similar to the /describe prompt but with an extra description.

It reviews if relevant tests are added to the code. If it is a simple .md file used for documentation or a .txt file used for outlining dependencies to be installed, no tests are required.
It also scales how deep it needs to review a commit depending on whether it is documentation or actual code.
It reviews the code for security issues and recommends a fix if a flaw is found.

Code Suggestions (`/improve`):

This prompt suggests the best of what other developers have figured. It comes as a suggestion, and then you have choices:

Commit the suggestions you are given.
Address the suggestion by resolving it. Here the PR agent suggested I use a later version of the Python Request library.

In a real-world use case, it's important to ensure that a new version committed is compatible with the existing codebase and that it doesn't introduce any breaking changes or conflicts with other dependencies.

Generate Custom Labels (`/generate_labels`) Prompt:

The outcome of this prompt is similar to a part of the result in the /describe prompt; therefore, add labels.
In the below example, an enhancement label was added because I committed a suggestion made by the agent via the /improve prompt.

Add Documentation (`/add_docs`) Prompt:

This prompt suggests the necessary documentation for the commit.

I ended up using this suggestion.

Find Similar Issue (`/similar_issue`) Prompt:

This prompt suggests similar open issues in the repository.
In the below example, a user opened an issue about inconsistencies in the Bitbucket documentation section of the CodiumAI PR agent. To find similar issues in the repository, one of the maintainers issued a /similar_issue prompt. One similar documentation issue here is the “1. [GITLAB] Installation Guide for GitLab please?”

Comparative Analysis: CodiumAI vs. GitHub Copilot

In the dynamic landscape of collaborative coding, CodiumAI PR-Agent, and GitHub Copilot PR emerge as notable contenders, each with its unique strengths and characteristics.

CodiumAI PR-Agent:

Excels with a comprehensive set of commands, providing developers with enhanced control over pull requests.
Showcases platform flexibility, operating seamlessly across various environments beyond GitHub.
Boasts an open-source model, fostering transparency and community engagement.
It offers versatility by being available not only in Git but also in IDEs like VScode and IntelliJ IDEA.
The commands are free for developers.

GitHub Copilot:

Stands out for its strong focus on code generation within the GitHub environment.
Currently features a more limited set of commands; summarizing pull requests, but plans to expand in the future.
It is primarily confined to the GitHub ecosystem.
Operates with a closed-source model, potentially limiting visibility into the development process.
The commands are only free for 30 days; afterward, they are available in a paid plan.

Comparative Conclusion:
For developers seeking a robust, all-encompassing solution with a diverse command toolkit, platform flexibility, and transparency, CodiumAI PR-Agent proves to be a superior choice. Its open-source nature and compatibility with various environments provide a more versatile and collaborative coding experience. While GitHub Copilot excels in certain aspects, developers may find CodiumAI PR-Agent better suited for projects requiring broader functionality, flexibility, and community engagement. Ultimately, the choice between the two depends on specific project requirements and preferences in the collaborative coding journey.

Community Impact

How Open Source Projects Use It

CodiumAI PR-Agent’s influence extends deeply within open-source projects. An exemplary illustration is Kubescape, a Cloud Native Computing Foundation (CNCF) sandbox project. Since its adoption in August, Kubescape has been utilizing the PR-Agent service. They also recently had a public bug bounty collaboration with CodiumAI. This program added an extra layer of community-driven scrutiny, encouraging contributors to utilize simple commands like /describe for effective pull request messages.

Here the contributor wanted to better describe the PR, so he used the /describe prompt.

Challenges and Solutions: CodiumAI PR-Agent

Learning Curve:
- Challenge: Initial complexity in understanding diverse commands.
- Solution: Provide user-friendly documentation, tutorials, and community support.
Compatibility Issues:
- Challenge: Integrating with diverse repositories may pose compatibility challenges.
- Solution: Regularly update for compatibility, and encourage prompt issue reporting.
Command Understanding:
- Challenge: Users may find nuances in commands challenging. Like understanding when and where to apply certain commands based on the specific context of a pull request.
- Solution: Enhance command explanations, offer contextual help, and provide examples.
Community Engagement:
- Challenge: Ensuring sustained community involvement.
- Solution: Implement bug bounties, feature requests, and collaborative projects. Communicate updates and involve the community in decisions.
Security Concerns:
- Challenge: Addressing security concerns in an open-source environment.
- Solution: Conduct regular security audits, encourage responsible disclosure, and maintain transparent communication channels.
Limited Platform Awareness:
- Challenge: Users may not be fully aware of platform availability.
- Solution: Improve visibility through clear documentation, platform-specific guides, and marketing efforts.
Command Evolution:
- Challenge: Evolving commands without overwhelming users.
- Solution: Prioritize user feedback, conduct usability testing, and communicate updates.

Future Roadmap

Repository Control Configuration: The new feature builds upon GitHub Actions’ existing configurability, allowing users to refine and tailor the behavior of CodiumAI PR-Agent within their private repositories. This enhancement provides organizations with a more granular approach to managing collaborative coding processes while maintaining the flexibility to choose the level of automation that suits their workflow.

Conclusion: Why CodiumAI PR-Agent Dominates the Dev World with Versatility and Open-Source Power

In summary, CodiumAI PR-Agent is a versatile and open-source tool that empowers developers and enhances collaboration in software development. With features like auto description, question answering, update changelog, and code suggestions, it provides comprehensive support for pull requests. Compared to GitHub Copilot, CodiumAI PR-Agent offers platform flexibility and transparency. Despite challenges, it continues to evolve with a future roadmap focused on repository control configuration and advanced testing integration. Overall, CodiumAI PR-Agent is a dominant force in the development world.

Automating AWS Image Management using Serverless, Python, and NoSQL

Charles Uneze — Sat, 21 Oct 2023 08:03:31 +0000

AWS provides managed SSM documents for specific actions which you can add to your maintenance window tasks. However, there are cases where you need a custom action, such as this one.

This blog post will discuss a Python script that leverages AWS serverless and NoSQL database services to automate the creation and deletion of an AMI whose instance had just been patched using SSM. We'll explore how the script manages state sharing between functions and stores state information in a DynamoDB table, all while ensuring that your AMIs are maintained seamlessly.

Understanding State Sharing Generally

State sharing in Python allows multiple functions to access and modify shared data, enabling them to communicate and coordinate using a common state. Here's a concise Python example:

# Shared global variable
shared_counter = 0

# Function to increment
def increment_counter():
    global shared_counter
    shared_counter += 1

# Function to get the value
def get_counter_value():
    return shared_counter

# Example usage
increment_counter()
print(get_counter_value())  # Output: 1
increment_counter()
print(get_counter_value())  # Output: 2

In this code, shared_counter is a global variable, shared between functions for data sharing and coordination.

Understanding State Sharing in my Script

State sharing in the main script is achieved through a global variable called image_id. This variable is accessible and modifiable by multiple functions, allowing them to share and update the same state information, such as the currently active AMI. This facilitates the automation of AMI creation and deletion while maintaining a consistent state across functions.

The Script

The script starts by initializing AWS clients with Boto3. It creates clients for EC2 (ec2) and DynamoDB (dynamo) to handle AWS services. A DynamoDB table named AMI_Table is defined with a partition key structure for item identification.


import boto3
import datetime

# Initialize the AWS clients
ec2 = boto3.client('ec2')
dynamo = boto3.client('dynamodb')
table_name = 'AMI_Table'
key = {
    'PK': {
        'S': 'Partition_Key'
    }
}

The table format

Storing State in DynamoDB

State management is crucial in any automation process, and DynamoDB serves as an excellent choice for this task. In this script, state information, specifically the image_id, is stored in the DynamoDB table.

The script retrieves the image_id from the DynamoDB table, which acts as the identifier for the currently active AMI. This is the key to maintaining state information across multiple function invocations.

# Get ami_id value from dynamodb AMI_Table
get_ami_id = dynamo.get_item(TableName=table_name, Key=key)
# Pass the value to a new variable
image_id = get_ami_id['Item']['AMI_ID']['S']

AMI Creation

The script defines a function create_ami(InstanceId) for creating a new AMI. It takes an EC2 InstanceId as input, and here's how it works:

It creates a new AMI based on the provided InstanceId using the ec2.create_image function.
The current_time is used to generate a unique name for the new AMI.
The script updates the image_id global variable with the newly created AMI's ID.
It stores the image_id value in the DynamoDB table to maintain the state.

def create_ami(InstanceId):
    # Declare image_id as a global variable
    global image_id
    # Get the current time to use as the AMI name
    current_time = datetime.datetime.now().strftime('%Y-%m-   %d-%H-%M-%S')
    if InstanceId:
        create_image = ec2.create_image(
            InstanceId=InstanceId,
            # Use the current time as the image name
            Name=f'{current_time}_AMI'
        )
        # Update the global image_id variable
        image_id = create_image['ImageId']
        # Store the image_id value as a state in a dynamodb ami_table
        update_ami_id = dynamo.update_item(
            TableName=table_name,
            Key=key,
            #  (:) in :image_id is used to define a placeholder for 
            # attribute name or a value that will be set.
            UpdateExpression="SET AMI_ID = :image_id",
            # Use :image_id as a placeholder, its actual value is desired_state. 
            ExpressionAttributeValues={
                ':image_id': {
                    'S': image_id
                }
            }
        )
        # Return the value so when the function is passed to a variable, a value exists.
        return image_id
    else:
        return "InstanceId not found."

AMI Deletion

Another function, delete_ami(image_id), is responsible for deleting the existing AMI. It follows these steps:

Deregister the current AMI using ec2.deregister_image.
Retrieve the snapshot associated with the AMI and delete it.

def delete_ami(image_id):
    # Deregister the AMI
    deregister_ami = ec2.deregister_image(ImageId=image_id)
    # Get the snapshot ID associated with the AMI
    ami_info = ec2.describe_images(ImageIds=[image_id])
    snapshot_id = ami_info['Images'][0]['BlockDeviceMappings'][0]['Ebs']['SnapshotId']
    # Delete the associated snapshot
    delete_snapshot = ec2.delete_snapshot(SnapshotId=snapshot_id)

Updating State

The update_state(desired_state) function is used to update the state in the DynamoDB table. This function is essential for transitioning between creating and deleting AMIs.

def update_state(desired_state):
    get_ami_state = dynamo.get_item(TableName=table_name, Key=key)
    # Update the value of AMI_state
    dynamo.update_item(
        TableName=table_name,
        Key=key,
        #  (:) in :new_state is used to define a placeholder for 
        # attribute name or a value that will be set.
        UpdateExpression='SET AMI_State = :new_state',
        # Use :new_state as a placeholder, its actual value is desired_state. 
        ExpressionAttributeValues={':new_state': {'S': desired_state}}
    )

Lambda Function

Finally, the script provides a Lambda function called lambda_handler(event, context) that orchestrates the entire process. It reads the current state from DynamoDB, which will be 'Create_AMI' then it executes the code block for it. At the end of it, it updates the current NoSQL state to 'Delete_AMI' so during further execution, only the code block having this state value runs.

The script handles cases where the InstanceId is missing and ensures that the state is updated correctly.

def lambda_handler(event, context):
    try:
        # Read the dynamodb table
        db_read = dynamo.get_item(TableName=table_name, Key=key)
        # Get the AMI State from the dynamodb table
        ami_state = db_read['Item']['AMI_State']['S']

        if ami_state == 'Create_AMI':
            # Extract the InstanceId from the event dictionary
            InstanceId = event.get('InstanceId')
            # Check if InstanceId value exists
            if InstanceId:
                # Call the create_ami function and pass the InstanceId as a string.
                create_ami(InstanceId)
                # Update the value of AMI_state
                update_state('Delete_AMI')
                return f'Created new AMI: {image_id}'
            # When InstanceId value does not exist
            else:
                return 'InstanceId not found in event data.'

        elif ami_state == 'Delete_AMI':
            # Check if image_id value exists
            if image_id:
                # Store the image_id before deletion
                existing_ami_id = image_id
                # Delete the existing AMI
                delete_ami(existing_ami_id)
                # Extract the InstanceId from the event dictionary
                InstanceId = event.get('InstanceId')
                # Call the create_ami function
                new_ami_id = create_ami(InstanceId)
                return f'Deleted existing AMI: {existing_ami_id} and created a new AMI: {new_ami_id}'
            else:
                return 'image_id is not set, cannot delete AMI'
        else:
            return 'Invalid state'
    except Exception as e:
        return str(e)

Conclusion

By using Python, AWS services like EC2, Lambda, DynamoDB, and careful state management, this script automates the creation and deletion of AMIs seamlessly. This can be integrated into a larger infrastructure to ensure that your EC2 instances are always based on the latest and most reliable images. If you want to add several images, you can simply modify the NoSQL AMI_ID attribute in the AMI_Table item to use a list type for its values, and then tweak the rest of the code.

Using NoSQL databases like DynamoDB is a powerful choice for storing state information in such automation scripts. This script is a clear example of how state sharing between functions and external systems can be effectively managed to automate complex workflows.
Check out Lab 4 in this repo to find the full codes implementation.

How to Allow Custom-Routing-Traffic in AWS Global Accelerator with Terraform

Charles Uneze — Mon, 09 Oct 2023 14:15:16 +0000

By intelligently routing traffic, AWS Global Accelerator improves the availability and performance of applications. However, there is currently no Terraform AWS resource available to allow custom traffic routing to specific destinations in the subnet endpoint. I'll demonstrate a Terraform workaround in this post to enable this in AWS Custom Routing Global Accelerator.

Workaround Using Terraform

To allow custom routing traffic in AWS Global Accelerator, I'll leverage Terraform along with a null resource. Here's how you can do it:
1. Create a null resource in your Terraform code:

resource "null_resource" "allow_custom_routing_traffic" {

  triggers = {
    endpoint_group_arn = aws_globalaccelerator_custom_routing_endpoint_group.endpoint_group.id
    endpoint_id        = aws_subnet.Endpoint_Subnet.id
  }

  provisioner "local-exec" {
    command = "aws globalaccelerator allow-custom-routing-traffic --endpoint-group-arn ${self.triggers.endpoint_group_arn} --endpoint-id ${self.triggers.endpoint_id} --allow-all-traffic-to-endpoint --region us-west-2"
  }
}

2. Apply your configuration:
terraform apply -auto-approve

null_resource.allow_custom_routing_traffic: Creating...
null_resource.allow_custom_routing_traffic: Provisioning with 'local-exec'...
null_resource.allow_custom_routing_traffic (local-exec): Executing: ["cmd" "/C" "aws globalaccelerator --region us-west-2 allow-custom-routing-traffic --endpoint-group-arn arn:aws:globalaccelerator::1234:accelerator/1234/listener/f1234/endpoint-group/1234 --endpoint-id subnet-1234 --allow-all-traffic-to-endpoint"]
null_resource.allow_custom_routing_traffic: Creation complete after 8s [id=1234]

The null resource is used to apply the aws-cli config.
For the triggers, it's assumed you already have those resources defined in your code. Also, this trigger block ensures that the null_resource will be triggered and execute the provisioner whenever either the endpoint_group_arn or the endpoint_id changes.

Conclusion

While Terraform AWS Provider does not currently provide a direct Terraform AWS resource for allowing custom routing traffic, you can work around this limitation by using a Terraform null resource, as demonstrated in this blog post. This approach allows you to gain more control over how your traffic is routed in AWS Global Accelerator, enhancing the performance and availability of your applications.
s/o to Andres Montalban for assisting me in this.

Configuring a Transit Gateway Between 3 VPCs using Terraform

Charles Uneze — Sat, 11 Mar 2023 14:06:45 +0000

In this post, we will be configuring 3 intra-VPC communication using Terraform.
We will also be gathering packet data using flow-logs and Cloudwatch.

Algorithm of the Infrastructure

Create a first, second, and third VPC
Create first VPCs Internet Gateway
Create a first and second subnet in the first VPC
Specify the default NACL in the first VPC
Specify the default security group in the first VPC, and modify.
- Add an ingress rule for SSH and ICMP
- Add an egress rule for all protocols
Launch an instance in the first subnet, for the first VPC
Create a Transit gateway
Specify the default route table for the first VPC
- Add a route destined to all IP addresses, and a target to the Internet Gateway.
- Add a route destined to the second VPC, and a target to the Transit Gateway.
- Add a route destined to the third VPC, and a target to the Transit Gateway.
Create a transit gateway attachment to the two subnets associated to the first VPC.
Create a first and second subnet in the second VPC
Specify the default route table for the second VPC
- Add a route destined to the first VPC, and a target to the Transit Gateway.
- Add a route destined to the third VPC, and a target to the Transit Gateway.
Specify the default NACL in the second VPC
Specify the default security group in the second VPC, and modify.
- Add an ingress rule for SSH and ICMP
- Add an egress rule for all protocols
Launch an instance in the first subnet, for the second VPC
Create a transit gateway attachment to the two subnets associated to the second VPC.
Create a first and second subnet in the third VPC
Specify the default route table for the third VPC
- Add a route destined to the first VPC, and a target to the Transit Gateway.
- Add a route destined to the second VPC, and a target to the Transit Gateway.
Specify the default NACL in the third VPC
Specify the default security group in the third VPC, and modify.
- Add an ingress rule for SSH and ICMP
- Add an egress rule for all protocols
Launch an instance in the first subnet, for the third VPC
Create a transit gateway attachment to the two subnets associated to the third VPC.
Create a Transit gateway route table for the second and third VPC
- Add a route destined to all IP addresses, which uses the first VPCs transit gateway attachment as an ingress.
- Associate the first and second transit gateway attachment to the route table so they are both used individually as egresses.
Create a Transit gateway route table for the first VPC
- Add two routes destined to the second and third VPC, which uses the second and third VPCs transit gateway attachment as an ingress.
- Associate the first transit gateway attachment to the route table, so its used as an egress.
Create an IAM Role for flow logs
Create an IAM Role policy for flow logs
Create a Cloudwatch log group
Create a flow log for the first VPC

okay, this is a long algorithm, but this is basically how the infrastructure will be built.
Also, one thing I learnt about writing code for this infrastructure is the naming convention. I struggled with what to name my subnets. I decided that it was best it reflects the VPC name and its AZ, for example, public_subnet_1_for_VPC_A_AZ_2A

Create a first, second, and third VPC

resource "aws_vpc" "VPC_A" {
  cidr_block       = "192.168.0.0/16"
  instance_tenancy = "default"

  tags = {
    Name = "VPC_A"
  }
}

resource "aws_vpc" "VPC_B" {
  cidr_block       = "10.0.0.0/16"
  instance_tenancy = "default"

  tags = {
    Name = "VPC_B"
  }
}

resource "aws_vpc" "VPC_C" {
  cidr_block       = "10.1.0.0/16"
  instance_tenancy = "default"

  tags = {
    Name = "VPC_C"
  }
}

Create first VPCs Internet Gateway

resource "aws_internet_gateway" "VPC_A_IGW" {
  vpc_id = aws_vpc.VPC_A.id

  tags = {
    "name" = "VPC_A_IGW"
  }
}

Create a first and second subnet in the first VPC

resource "aws_subnet" "public_subnet_1_for_VPC_A_AZ_2A" {
  vpc_id                  = aws_vpc.VPC_A.id
  cidr_block              = "192.168.1.0/24"
  availability_zone       = "us-west-2a"
  map_public_ip_on_launch = "true"

  tags = {
    Name = "public_subnet_1_for_VPC_A_AZ_2A"
  }
}

resource "aws_subnet" "public_subnet_2_for_VPC_A_AZ_2B" {
  vpc_id            = aws_vpc.VPC_A.id
  cidr_block        = "192.168.2.0/24"
  availability_zone = "us-west-2b"
  tags = {
    Name = "public_subnet_2_for_VPC_A_AZ_2B"
  }
}

Specify the default NACL in the first VPC

resource "aws_default_network_acl" "Default_VPC_A_NACL" {
  default_network_acl_id = aws_vpc.VPC_A.default_network_acl_id

  ingress {
    protocol   = -1
    rule_no    = 100
    action     = "allow"
    cidr_block = "0.0.0.0/0"
    from_port  = 0
    to_port    = 0
  }

  egress {
    protocol   = -1
    rule_no    = 100
    action     = "allow"
    cidr_block = "0.0.0.0/0"
    from_port  = 0
    to_port    = 0
  }

  tags = {
    "name" = "Default_VPC_A_NACL"
  }
}

Specify the default security group in the first VPC, and modify.

Add an ingress rule for SSH and ICMP
Add an egress rule for all protocols

resource "aws_default_security_group" "SG_bastion" {
  vpc_id = aws_vpc.VPC_A.id

  ingress {
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    from_port   = -1
    to_port     = -1
    protocol    = "icmp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

Launch an instance in the first subnet, for the first VPC

resource "aws_instance" "Bastion" {
  ami               = "ami-0b029b1931b347543"
  instance_type     = "t2.micro"
  tenancy           = "default"
  availability_zone = "us-west-2a"
  key_name          = ""
  subnet_id         = aws_subnet.public_subnet_1_for_VPC_A_AZ_2A.id
  security_groups   = ["${aws_default_security_group.SG_bastion.id}"]

  tags = {
    "name" = "Bastion"
  }
}

Create a Transit gateway

resource "aws_ec2_transit_gateway" "TGW_Lab" {
  description                     = "the labs transit gateway"
  default_route_table_association = "disable"
  default_route_table_propagation = "disable"

  tags = {
    Name = "TGW_Lab"
  }
}

Specify the default route table for the first VPC

Add a route destined to all IP addresses, and a target to the Internet Gateway.
Add a route destined to the second VPC, and a target to the Transit Gateway.
Add a route destined to the third VPC, and a target to the Transit Gateway.

resource "aws_default_route_table" "VPC_A_RT" {
  default_route_table_id = aws_vpc.VPC_A.default_route_table_id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.VPC_A_IGW.id
  }

  route {
    cidr_block         = "10.0.0.0/16"
    transit_gateway_id = aws_ec2_transit_gateway.TGW_Lab.id
  }

  route {
    cidr_block         = "10.1.0.0/16"
    transit_gateway_id = aws_ec2_transit_gateway.TGW_Lab.id
  }

  tags = {
    Name = "VPC_A_RT"
  }
}

Create a transit gateway attachment to the two subnets associated to the first VPC.

resource "aws_ec2_transit_gateway_vpc_attachment" "TGA_VPC_A" {
  subnet_ids         = [aws_subnet.public_subnet_1_for_VPC_A_AZ_2A.id, aws_subnet.public_subnet_2_for_VPC_A_AZ_2B.id]
  transit_gateway_id = aws_ec2_transit_gateway.TGW_Lab.id
  vpc_id             = aws_vpc.VPC_A.id

  tags = {
    Name = "TGA_VPC_A"
  }
}

Create a first and second subnet in the second VPC

resource "aws_subnet" "public_subnet_1_for_VPC_B_AZ_2A" {
  vpc_id            = aws_vpc.VPC_B.id
  cidr_block        = "10.0.2.0/24"
  availability_zone = "us-west-2a"
  tags = {
    Name = "public_subnet_1_for_VPC_B_AZ_2A"
  }
}

resource "aws_subnet" "public_subnet_2_for_VPC_B_AZ_2B" {
  vpc_id            = aws_vpc.VPC_B.id
  cidr_block        = "10.0.3.0/24"
  availability_zone = "us-west-2b"
  tags = {
    Name = "public_subnet_2_for_VPC_B_AZ_2B"
  }
}

Specify the default route table for the second VPC

Add a route destined to the first VPC, and a target to the Transit Gateway.
Add a route destined to the third VPC, and a target to the Transit Gateway.

resource "aws_default_route_table" "VPC_B_RT" {
  default_route_table_id = aws_vpc.VPC_B.default_route_table_id

  route {
    cidr_block         = "192.168.0.0/16"
    transit_gateway_id = aws_ec2_transit_gateway.TGW_Lab.id
  }

  route {
    cidr_block         = "10.1.0.0/16"
    transit_gateway_id = aws_ec2_transit_gateway.TGW_Lab.id
  }

  tags = {
    Name = "VPC_B_RT"
  }
}

Specify the default NACL in the second VPC

resource "aws_default_network_acl" "Default_VPC_B_NACL" {
  default_network_acl_id = aws_vpc.VPC_B.default_network_acl_id

  ingress {
    protocol   = -1
    rule_no    = 100
    action     = "allow"
    cidr_block = "0.0.0.0/0"
    from_port  = 0
    to_port    = 0
  }

  egress {
    protocol   = -1
    rule_no    = 100
    action     = "allow"
    cidr_block = "0.0.0.0/0"
    from_port  = 0
    to_port    = 0
  }

  tags = {
    "name" = "Default_VPC_B_NACL"
  }
}

Specify the default security group in the second VPC, and modify.

Add an ingress rule for SSH and ICMP
Add an egress rule for all protocols

resource "aws_default_security_group" "DB_1" {
  vpc_id = aws_vpc.VPC_B.id

  ingress {
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    from_port   = -1
    to_port     = -1
    protocol    = "icmp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

Launch an instance in the first subnet for the second VPC.

resource "aws_instance" "DB_1" {
  ami               = "ami-0b029b1931b347543"
  instance_type     = "t2.micro"
  tenancy           = "default"
  availability_zone = "us-west-2a"
  key_name          = ""
  subnet_id         = aws_subnet.public_subnet_1_for_VPC_B_AZ_2A.id
  security_groups   = ["${aws_default_security_group.DB_1.id}"]

  tags = {
    "name" = "DB_1"
  }
}

Create a transit gateway attachment to the two subnets associated to the second VPC.

resource "aws_ec2_transit_gateway_vpc_attachment" "TGA_VPC_B" {
  subnet_ids         = [aws_subnet.public_subnet_1_for_VPC_B_AZ_2A.id, aws_subnet.public_subnet_2_for_VPC_B_AZ_2B.id]
  transit_gateway_id = aws_ec2_transit_gateway.TGW_Lab.id
  vpc_id             = aws_vpc.VPC_B.id

  tags = {
    Name = "TGA_VPC_A"
  }
}

Create a first and second subnet in the third VPC

resource "aws_subnet" "public_subnet_1_for_VPC_C_AZ_2A" {
  vpc_id            = aws_vpc.VPC_C.id
  cidr_block        = "10.1.2.0/24"
  availability_zone = "us-west-2a"
  tags = {
    Name = "public_subnet_1_for_VPC_C_AZ_2A"
  }
}

resource "aws_subnet" "public_subnet_2_for_VPC_C_AZ_2B" {
  vpc_id            = aws_vpc.VPC_C.id
  cidr_block        = "10.1.3.0/24"
  availability_zone = "us-west-2b"
  tags = {
    Name = "public_subnet_2_for_VPC_C_AZ_2B"
  }
}

Specify the default route table for the third VPC

Add a route destined to the first VPC, and a target to the Transit Gateway.
Add a route destined to the second VPC, and a target to the Transit Gateway.

resource "aws_default_route_table" "VPC_C_RT" {
  default_route_table_id = aws_vpc.VPC_C.default_route_table_id

  route {
    cidr_block         = "192.168.0.0/16"
    transit_gateway_id = aws_ec2_transit_gateway.TGW_Lab.id
  }

  route {
    cidr_block         = "10.0.0.0/16"
    transit_gateway_id = aws_ec2_transit_gateway.TGW_Lab.id
  }

  tags = {
    Name = "VPC_C_RT"
  }
}

Specify the default NACL in the third VPC

resource "aws_default_network_acl" "Default_VPC_C_NACL" {
  default_network_acl_id = aws_vpc.VPC_C.default_network_acl_id

  ingress {
    protocol   = -1
    rule_no    = 100
    action     = "allow"
    cidr_block = "0.0.0.0/0"
    from_port  = 0
    to_port    = 0
  }

  egress {
    protocol   = -1
    rule_no    = 100
    action     = "allow"
    cidr_block = "0.0.0.0/0"
    from_port  = 0
    to_port    = 0
  }

  tags = {
    "name" = "Default_VPC_C_NACL"
  }
}

Specify the default security group in the third VPC, and modify.

Add an ingress rule for SSH and ICMP
Add an egress rule for all protocols

resource "aws_default_security_group" "DB_2" {
  vpc_id = aws_vpc.VPC_C.id

  ingress {
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    from_port   = -1
    to_port     = -1
    protocol    = "icmp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

Launch an instance in the first subnet, for the third VPC

resource "aws_instance" "DB_2" {
  ami               = "ami-0b029b1931b347543"
  instance_type     = "t2.micro"
  tenancy           = "default"
  availability_zone = "us-west-2a"
  key_name          = ""
  subnet_id         = aws_subnet.public_subnet_1_for_VPC_C_AZ_2A.id
  security_groups   = ["${aws_default_security_group.DB_2.id}"]

  tags = {
    "name" = "DB_2"
  }
}

Create a transit gateway attachment to the two subnets associated to the third VPC.

resource "aws_ec2_transit_gateway_vpc_attachment" "TGA_VPC_C" {
  subnet_ids         = [aws_subnet.public_subnet_1_for_VPC_C_AZ_2A.id, aws_subnet.public_subnet_2_for_VPC_C_AZ_2B.id]
  transit_gateway_id = aws_ec2_transit_gateway.TGW_Lab.id
  vpc_id             = aws_vpc.VPC_C.id

  tags = {
    Name = "TGA_VPC_C"
  }
}

Create a Transit gateway route table for the second and third VPC

Add a route destined to all IP addresses, which uses the first VPCs transit gateway attachment as an ingress.
Associate the first and second transit gateway attachment to the route table so they are both used individually as egresses.

resource "aws_ec2_transit_gateway_route_table" "TGW_RTB_VPC_B_C" {
  transit_gateway_id = aws_ec2_transit_gateway.TGW_Lab.id

  tags = {
    "name" = "TGW_RTB_VPC_B_C"
  }
}

resource "aws_ec2_transit_gateway_route" "TGW_RTB_VPC_B_C_Route_1" {
  destination_cidr_block         = "0.0.0.0/0"
  transit_gateway_attachment_id  = aws_ec2_transit_gateway_vpc_attachment.TGA_VPC_A.id
  transit_gateway_route_table_id = aws_ec2_transit_gateway_route_table.TGW_RTB_VPC_B_C.id
}

resource "aws_ec2_transit_gateway_route_table_association" "TGW_RTB_VPC_B_C_Association_1" {
  transit_gateway_attachment_id  = aws_ec2_transit_gateway_vpc_attachment.TGA_VPC_B.id
  transit_gateway_route_table_id = aws_ec2_transit_gateway_route_table.TGW_RTB_VPC_B_C.id
}

resource "aws_ec2_transit_gateway_route_table_association" "TGW_RTB_VPC_B_C_Association_2" {
  transit_gateway_attachment_id  = aws_ec2_transit_gateway_vpc_attachment.TGA_VPC_C.id
  transit_gateway_route_table_id = aws_ec2_transit_gateway_route_table.TGW_RTB_VPC_B_C.id
}

Create a Transit gateway route table for the first VPC

Add two routes destined to the second and third VPC, which uses the second and third VPCs transit gateway attachment as an ingress.
Associate the first transit gateway attachment to the route table, so its used as an egress.

resource "aws_ec2_transit_gateway_route_table" "TGW_RTB_VPC_A" {
  transit_gateway_id = aws_ec2_transit_gateway.TGW_Lab.id

  tags = {
    "name" = "TGW_RTB_VPC_A"
  }
}

resource "aws_ec2_transit_gateway_route" "TGW_RTB_VPC_A_Route_1" {
  destination_cidr_block         = "10.0.0.0/16"
  transit_gateway_attachment_id  = aws_ec2_transit_gateway_vpc_attachment.TGA_VPC_B.id
  transit_gateway_route_table_id = aws_ec2_transit_gateway_route_table.TGW_RTB_VPC_A.id
}

resource "aws_ec2_transit_gateway_route" "TGW_RTB_VPC_A_Route_2" {
  destination_cidr_block         = "10.1.0.0/16"
  transit_gateway_attachment_id  = aws_ec2_transit_gateway_vpc_attachment.TGA_VPC_C.id
  transit_gateway_route_table_id = aws_ec2_transit_gateway_route_table.TGW_RTB_VPC_A.id
}

resource "aws_ec2_transit_gateway_route_table_association" "TGW_RTB_VPC_A_Association_1" {
  transit_gateway_attachment_id  = aws_ec2_transit_gateway_vpc_attachment.TGA_VPC_A.id
  transit_gateway_route_table_id = aws_ec2_transit_gateway_route_table.TGW_RTB_VPC_A.id
}

Create an IAM Role for flow logs

resource "aws_iam_role" "role_lab_flow_logs" {
  name = "role_lab_flow_logs"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "vpc-flow-logs.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF
}

Create an IAM Role policy for flow logs

resource "aws_iam_role_policy" "IAM_Role_Policy_for_Flow_Log" {
  name = "IAM_Role_Policy_for_Flow_Log"
  role = aws_iam_role.role_lab_flow_logs.id

  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents",
        "logs:DescribeLogGroups",
        "logs:DescribeLogStreams"
      ],
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}
EOF
}

Create a Cloudwatch log group

resource "aws_cloudwatch_log_group" "Transit_Gateway_Log_Group" {
  name              = "Transit_Gateway_Log_Group"
}

Create a flow log for the first VPC

resource "aws_flow_log" "flow_log_tgw_lab" {
  iam_role_arn    = aws_iam_role.role_lab_flow_logs.arn
  log_destination = aws_cloudwatch_log_group.Transit_Gateway_Log_Group.arn
  traffic_type    = "ALL"
  vpc_id          = aws_vpc.VPC_A.id

  tags = {
    Name = "flow_log_tgw_lab"
  }
}

This is usually the log format when reading a flow log data:
log_format = "${version} ${account-id} ${interface-id} ${srcaddr} ${dstaddr} ${srcport} ${dstport} ${protocol} ${packets} ${bytes} ${start} ${end} ${action} ${log-status}"

When you deploy the code and grab the public IP of the bastion host, then SSH into it, then do a ping to 8.8.8.8
When simply go to the Cloudwatch service via the GUI, specifically to "Log Groups," there you will find your flow log data streams for several elastic network interfaces (ENIs).
Do a terraform state show aws_instance.Bastion to get the ENI of your instance. Then go back to cloudwatch and select the ENI. Open the data, and simply search for 8.8.8.8 in the log stream, you will notice that an ICMP session was made.

Securing your VPC using Public & Private Subnets using Terraform

Charles Uneze — Fri, 10 Mar 2023 13:07:36 +0000

In this lab, two instances are created, one is launched in a first subnet which is public, while the other is launched the second subnet which is private.

The private instance must access the internet using a Network Address Translation (NAT) gateway deployed into the first subnet.

No default security group, NACL, or route table is used in this demonstration. This means any non-default being used must be explicitly associated with its resource. For example, a non-default route table must be explicitly associated with a subnet, etc.

Also, in this lab, we will explore the stateful nature of a Security Group, at Algorithm no 12.

Algorithm of the Infrastructure

Create a VPC
Create an Internet Gateway
Create a first subnet
Create a first NACL for the first subnet
Specify the first route table
- Add a route destined to all IP addresses, and a target to the Internet Gateway.
Create a security group in the VPC for the first subnet
- Add an ingress rule for SSH
- Add an egress rule for SSH
Create an instance for the first subnet
Create an elastic IP
Create a NAT Gateway in the first subnet and attach the elastic IP to it
Create a second subnet
Create a second NACL for the second subnet
Create a second security group
- Add an ingress rule for SSH
- Add an egress rule for all ICMP
Create a second route table
- Add a route destined for the NAT Gateway
Create a second instance

Create a VPC

CIDR = 10.0.0.0/16

resource "aws_vpc" "VPC_A" {
  cidr_block       = "10.0.0.0/16"
  instance_tenancy = "default"

  tags = {
    Name = "VPC_A"
  }
}

Create an Internet Gateway

resource "aws_internet_gateway" "VPC_A_IGW" {
  vpc_id = aws_vpc.VPC_A.id

  tags = {
    Name = "VPC_A_IGW"
  }
}

Create a first subnet

resource "aws_subnet" "VPC_A_Public_Subnet_A" {
  vpc_id                  = aws_vpc.VPC_A.id
  cidr_block              = "10.0.20.0/24"
  availability_zone       = "us-west-2a"
  map_public_ip_on_launch = "true"

  tags = {
    Name = "VPC_A_Public_Subnet_A"
  }
}

Create a first NACL for the first subnet

resource "aws_network_acl" "VPC_A_Public_NACL_A" {
vpc_id = aws_vpc.VPC_A.id
subnet_ids = [aws_subnet.VPC_A_Public_Subnet_A.id]

  ingress {
    protocol   = -1
    rule_no    = 100
    action     = "allow"
    cidr_block = "0.0.0.0/0"
    from_port  = 0
    to_port    = 0
  }

  egress {
    protocol   = -1
    rule_no    = 100
    action     = "allow"
    cidr_block = "0.0.0.0/0"
    from_port  = 0
    to_port    = 0
  }

  tags = {
    "name" = "VPC_A_Public_NACL_A"
  }
}

resource "aws_network_acl_association" "VPC_A_Public_NACL_A_Association" {
  network_acl_id = aws_network_acl.VPC_A_Public_NACL_A.id
  subnet_id      = aws_subnet.VPC_A_Public_Subnet_A.id
}

Specify the first route table

resource "aws_route_table" "VPC_A_Public_RT_A" {
  vpc_id = aws_vpc.VPC_A.id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.VPC_A_IGW.id
  }

  tags = {
    Name = "VPC_A_Public_RT_A"
  }
}

resource "aws_route_table_association" "VPC_A_Public_RT_A_Association_A" {
  subnet_id      = aws_subnet.VPC_A_Public_Subnet_A.id
  route_table_id = aws_route_table.VPC_A_Public_RT_A.id
}

Create a security group in the VPC for the first subnet

resource "aws_security_group" "SG_bastion" {
  vpc_id      = aws_vpc.VPC_A.id
  description = " SG for bastion host. SSH access only"

  ingress {
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

Create an instance for the first subnet

resource "aws_instance" "Bastion" {
  ami               = "ami-0b029b1931b347543"
  instance_type     = "t2.micro"
  availability_zone = "us-west-2a"
  key_name          = ""
  tenancy           = "default"
  subnet_id         = aws_subnet.VPC_A_Public_Subnet_A.id
  security_groups   = ["${aws_security_group.SG_bastion.id}"]
}

Create an elastic IP

resource "aws_eip" "EIP_for_NAT_GW" {
  vpc = true
}

Create a NAT Gateway in the second subnet and attach the elastic IP to it

resource "aws_nat_gateway" "NAT_GW" {
  subnet_id         = aws_subnet.VPC_A_Public_Subnet_A.id
  connectivity_type = "public"
  allocation_id     = aws_eip.EIP_for_NAT_GW.id

  tags = {
    "Name" = "NAT_GW"
  }
}

Create a second subnet

resource "aws_subnet" "VPC_A_Private_Subnet_A" {
  vpc_id                  = aws_vpc.VPC_A.id
  cidr_block              = "10.0.10.0/24"
  availability_zone       = "us-west-2a"
  map_public_ip_on_launch = "false"

  tags = {
    Name = "VPC_A_Private_Subnet_A"
  }
}

Create a second NACL for the second subnet

resource "aws_network_acl" "VPC_A_Private_NACL_A" {
vpc_id = aws_vpc.VPC_A.id
subnet_ids = [aws_subnet.VPC_A_Private_Subnet_A.id]

  ingress {
    protocol   = -1
    rule_no    = 100
    action     = "allow"
    cidr_block = "0.0.0.0/0"
    from_port  = 0
    to_port    = 0
  }

  egress {
    protocol   = -1
    rule_no    = 100
    action     = "allow"
    cidr_block = "0.0.0.0/0"
    from_port  = 0
    to_port    = 0
  }

  tags = {
    "name" = "VPC_A_Private_NACL_A"
  }
}

resource "aws_network_acl_association" "VPC_A_Private_NACL_A_Association" {
  network_acl_id = aws_network_acl.VPC_A_Private_NACL_A.id
  subnet_id      = aws_subnet.VPC_A_Private_Subnet_A.id
}

Create a second security group

Here we will explore the stateful nature of a security group

resource "aws_security_group" "SG_Private" {
  name        = "SG_Private"
  description = "Security group for private subnet instances."
  vpc_id      = aws_vpc.VPC_A.id

  ingress {
    description     = "Incoming & Outgoing SSH"
    from_port       = 22
    to_port         = 22
    protocol        = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    description = "Outgoing & Incoming ICMP"
    from_port   = -1
    to_port     = -1
    protocol    = "icmp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    "name" = "SG_Private"
  }
}

We can notice that the ingress rule is for an SSH protocol, while the egress rule is for the ICMP protocol.

On AWS, when a packet ingresses or is directed towards an instance, it can leave the instance even if the egress rule doesn't match. That is why I described it as “incoming & outgoing SSH”
The Bastion instance can SSH into the private instance. Since the Bastion security group has an egress SSH rule set. Also, since the Private security grphp has an ingress SSH rule set.

Also, the private instance can ping any address on the Internet.
ICMP uses an "echo request" and an "echo reply.”
Since the first outgoing “request” traffic was approved, its incoming “reply” traffic will be approved too. That is why I described it as an “outgoing & incoming ICMP”

Create a second route table

resource "aws_route_table" "VPC_A_Private_RT_A" {
  vpc_id = aws_vpc.VPC_A.id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_nat_gateway.NAT_GW.id
  }

  tags = {
    Name = "VPC_A_Private_RT_A"
  }
}

resource "aws_route_table_association" "VPC_A_Private_RT_A_Association" {
  subnet_id      = aws_subnet.VPC_A_Private_Subnet_A.id
  route_table_id = aws_route_table.VPC_A_Private_RT_A.id
}

Create a second instance

resource "aws_instance" "VPC_A_Private_Subnet_A_Instance_A" {
  ami               = "ami-0b029b1931b347543"
  instance_type     = "t2.micro"
  availability_zone = "us-west-2a"
  key_name          = ""
  tenancy           = "default"
  subnet_id         = aws_subnet.VPC_A_Private_Subnet_A.id
  security_groups   = ["${aws_security_group.SG_Private.id}"]

  tags = {
    "name" = "VPC_A_Private_Subnet_A_Instance_A"
  }
}

After deploying the configurations, configure SSH Agent forwarding on Putty, then SSH into your first instance, and then into the second instance. Lastly do a ping to any public URL from the second instance and make sure its successful.

Follow my AWS Advanced Networking Journey on Github

AWS Advanced Networking
https://github.com/Its-All-About-the-Journey/AWS-Advanced-Networking

Create an Instance with an Elastic IP Attached using Terraform

Charles Uneze — Mon, 06 Mar 2023 18:56:22 +0000

On my journey to explore the Kubernetes networking ecosystem, I decided to take my first step by completing the AWS Networking course outline using Terraform. I am attempting to gain an understanding of how things work at this layer.

Laying the Ground Work for My Code

The first thing I did was to install Terraform using this tutorial.

Defining the Backend for Storing Infrastructure Information

The next thing I did was to create a "Terraform.tf" file so I can set up the backend where all the information about this infrastructure will be stored. The information is mapped to the real infrastructure on AWS. I use a local backend, meaning the information is stored on my computer. This information is in JSON format.



terraform {
    backend "local" {
      path = ""
    }

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.0"
    }
  }
}

The above code in this file specifies that I am using a local backend, and also that I am using an “AWS provider.” The provider is a plugin for interacting with the AWS APIs so the infrastructure can be built.

Specifying the Cloud Provider

Next I created a "main.tf" file where I will specify the cloud provider and also the architecture as code.



# Configure the AWS Provider
provider "aws" {
  region = "us-west-2"
  access_key = ""
  secret_key = ""
}

Algorithm of the Architecture

Create a VPC
Specify the default NACL
Specify the default security group
Create an Internet Gateway
Specify the default route table, and add a new route to the Internet Gateway.
Create a subnet
Create an instance
Create an elastic IP address

Create a VPC

VPC = 10.0.0.0/16



# Create VPC
resource "aws_vpc" "VPC_A" {
  cidr_block       = "10.0.0.0/16"
  instance_tenancy = "default"

  tags = {
    name = "VPC_A"
  }
}

When a VPC is created, AWS uses an overlay network to offer multi-tenancy for implementation. I wrote some words about this on my substack.

Specify the default NACL



# Specify the default NACL, regardless if the VPC creates it on default
resource "aws_default_network_acl" "Default_VPC_A_NACL" {
  default_network_acl_id = aws_vpc.VPC_A.default_network_acl_id

  ingress {
    protocol   = -1
    rule_no    = 100
    action     = "allow"
    cidr_block = "0.0.0.0/0"
    from_port  = 0
    to_port    = 0
  }

  egress {
    protocol   = -1
    rule_no    = 100
    action     = "allow"
    cidr_block = "0.0.0.0/0"
    from_port  = 0
    to_port    = 0
  }

  tags = {
    "name" = "Default_VPC_A_NACL"
  }
}

Specify the default security group



# Specify the default SG, regardless if the VPC creates it on default
resource "aws_default_security_group" "Default_VPC_A_SG" {
  vpc_id = aws_vpc.VPC_A.id

  ingress {
    protocol  = -1
    from_port = 0
    to_port   = 0
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = -1
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    "name" = "Default_VPC_A_SG"
  }
}

Create an Internet Gateway



# Create the Internet Gateway
resource "aws_internet_gateway" "VPC_A_IGW" {
  vpc_id = aws_vpc.VPC_A.id

  tags = {
    name = "VPC_A_IGW"
  }
}

Specify the default route table, and add a new route to the Internet Gateway



# Create the default route table
resource "aws_default_route_table" "VPC_A_Default_RT" {
  default_route_table_id = aws_vpc.VPC_A.default_route_table_id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.VPC_A_IGW.id
  }

  tags = {
    name = "VPC_A_Default_RT"
  }
}

Create a subnet



# Create Subnet
resource "aws_subnet" "VPC_A_Subnet_A" {
  vpc_id                  = aws_vpc.VPC_A.id
  cidr_block              = "10.0.0.0/24"
  availability_zone       = "us-west-2a"

  tags = {
    name = "VPC_A_Subnet_A"
  }
}

Create an Instance



# Create AWS instance
resource "aws_instance" "VPC_A_Subnet_A_AWS_Linux" {
  ami               = "ami-0b029b1931b347543"
  instance_type     = "t2.micro"
  tenancy           = "default"
  availability_zone = "us-west-2a"
  subnet_id = aws_subnet.VPC_A_Subnet_A.id

  tags = {
    name = "VPC_A_Subnet_A_AWS_Linux"
  }
}

Create an elastic IP address



# Create an Elastic IP
resource "aws_eip" "VPC_A_EIP" {
  instance = aws_instance.VPC_A_Subnet_A_AWS_Linux.id
  vpc      = true
  tags = {
    name = "VPC_A_EIP"
  }
}

# Associate the Elastic IP to the instance
resource "aws_eip_association" "VPC_A_EIP-Association" {
  instance_id   = aws_instance.VPC_A_Subnet_A_AWS_Linux.id
  allocation_id = aws_eip.VPC_A_EIP.id
}

After applying the configuration, check for the public IP address of the script via the command below
terraform state show aws_eip.VPC_A_EIP
Then do a ping to the public IP address, make sure it is successful.

Do make a comment if you experience any issue in the code or article.

Follow my AWS Advanced Networking Journey on Github

AWS Advanced Networking
https://github.com/Its-All-About-the-Journey/AWS-Advanced-Networking

DEV Community: Charles Uneze

Cost-Efficient ETL for Small Datasets using AWS Lambda, S3, Wrangler, and Glue

How to install AWS Wrangler on Lambda

Defining the other ETL logics

Demo

Final takeaway

Using GitOps to Manage a KWOK Deployment

Table of Contents

Overview

Requirements

File Tree

Create a Cluster

Install Flux Controllers

Create a Secret for Authenticating to the GitHub Repository

Connect Flux to the Repository

Target the KWOK Manifests using Flux

Apply Custom Resources (CRs) of Stages

Target the Deployment and Node YAML Manifests using Flux

Install Prometheus and Grafana

Enable Metrics on Kube-Scheduler

Grab the Grafana Username and Password

Scale the Deployment to 30 and Modify Node Affinity

View Scheduler Metrics on Grafana Dashboard

Conclusion

Wikimedia Foundation GSOD 2024 Project Report

Table of Contents

Project Overview

Key Challenges

Project Outcome

Key Learning

Best Practices Identified

Recommendations for Future Projects

Conclusion

More Resources

My Experience Working on the KWOK Project as an LFX Mentee

Table of Contents

What’s the KWOK Project About?

How I Got Accepted

Step 1: Use the Project

Step 2: Analyze the Expected Outcome

New Concepts I Learned

What I Struggled With

My Contributions

Conclusion

Utilizing Coverage AI Agents for Better Unit Tests

CodiumAI’s Cover-Agent

Demo

Why should you generate coverage reports?

More Reading

Docker ARP Analysis Guide

Table of Contents

Introduction

Requirements

Install Homebrew and Multipass

Download an Ubuntu Server from Multipass

Go to the Shell of the Ubuntu Server

Install Docker

Mount a Local Folder to an Ubuntu Directory

Verify Ubuntu Server Installation

Install Wireshark

Architecture Deployment Guide

Enter Into the Shell of the Ubuntu Server Again

Create a New Network

Verify Network

Pull Alpine Image

Open a New Terminal

Listen for ARP Packets in Each Bridge

docker0

docker1

Create 2 Containers in the Default Bridge, Also Connect Them to the Custom Bridge

Send Pings to the Internet From the First Interface

Send Pings to the Internet From the Second Interface

View MAC Addresses of Each Bridge Interface

View Mac Addresses of Each Container Interface

bridge

custom_bridge

End ARP Packet Capture 1

End ARP Packet Capture 2

Move the Files to the Local Directory

View Packet Captures

Auto Description (`/describe`) Prompt:

Question Answering (`/ask ...`) Prompt:

Update Changelog (`/update_changelog`) Prompt:

Auto Review (`/review`) Prompt:

Code Suggestions (`/improve`):

Generate Custom Labels (`/generate_labels`) Prompt:

Add Documentation (`/add_docs`) Prompt:

Find Similar Issue (`/similar_issue`) Prompt: