DEV Community: Ali Ogun

Step by Step Troubleshooting WAFv2 - With Pictures

Ali Ogun — Mon, 03 Jun 2024 16:53:58 +0000

You suspect that the Web Application Firewall (WAF) may be obstructing the functionality of your application. This article provides guidance on diagnosing the issue and adjusting/removing WAF rules as necessary.

Open WEB ACLs tab under the WAF dashboard as shown in figure.

Make sure you choose the correct region. In this example my WAF is deployed in Ohio(us-east-2). And then click your WAF rule, here in this example the name is managed_rules.

In this dashboard, you can access comprehensive metrics related to your WAF. Take a moment to review the graphics and metrics to become acquainted with them. Make sure that Blocked button is selected only. However, the information we require is located at the bottom of the page, so please scroll down accordingly.

4.At the bottom, we can see at the Attack Types graph that our WAF detected 3 different attack types as SQL Injection, NoUserAgent and BadBots. We see those 3 because previously I tried to do an SQL injection to my application and also I made HTTPS requests without proper headers. In Requests terminated by managed rule groups you can see which rule groups are blocking your requests. In my case they are AWSManagedRulesSQLiRuleSet (for SQL injection) and AWSManagedRulesCommonRuleSet (for missing and bad agent header). Now that we have gathered some information, our focus needs to shift to the individual rules rather than the rule groups. Let's direct our attention to the graphic located at the bottom left Top 10 managed rule labels.

If we take a closer look at this graph, we can see exactly when each rule kicks in and stops your access. Just match up the times when you tried to access with the points on the graph. The pink line on the graph shows which rule is stopping you. Below the graph, you'll see a list of rules that were triggered, each with its own color. In my case, it looks like the rule that's causing the trouble is aws:core-rule-set:NoUserAgent_Header. So what does it mean?

The rule label is aws:core-rule-set:NoUserAgent_Header. After the last column, we see the name of the rule we want to exclude, which is NoUserAgent_Header. I actually sent a request with an empty user agent header, triggering this rule. Now, let's proceed and see how to exclude the rule programmatically.

The rule NoUserAgent_Header belongs to the core-rule-set, and we need to refer to the corresponding AWS documentation to verify the actual name of NoUserAgent_Header because it is case-sensitive. As shown in the image below, we find the corresponding LABEL and Role Name. So in our case, the rule name I need to use in the exclude list is NoUserAgent_HEADER, not NoUserAgent_Header, because it is case-sensitive. Please take a look at the table below to find corresponding AWS page for your label name that you see in CloudWatch.

Label from CloudWatch	Corresponding AWS Page
core-rule-set	AWSManagedRulesCommonRuleSet
sql-database	AWSManagedRulesSQLiRuleSet
windows-os	AWSManagedRulesWindowsRuleSet
known-bad-inputs	AWSManagedRulesKnownBadInputsRuleSet
amazon-ip-list	AWSManagedRulesAmazonIpReputationList

Now we can go to our root module and define our variables as shown below. We wanted to exclude NoUserAgent_HEADER and I place it under the rules_to_exclude_common list because this rule is part of the AWSManagedRulesCommonRuleSet. As you can see I also exclude as I wish other rules. Here I also excluded SQLi_QUERYARGUMENTS which is a rule blocks SQL injections and also I excluded UserAgent_BadBots_HEADER which inspects for common User-Agent header values that indicate that the request is a bad bot. You can see the code snippet at the bottom of the page.

But why did we place NoUserAgent_HEADER under the rules_to_exclude_commonblock? In the table below, you can identify the corresponding Terraform variable to utilize in your code by comparing it with the Label from the CloudWatch column.
| Label from CloudWatch | Corresponding TF Variable |
|-----------------------|--------------------------------|
| core-rule-set | rules_to_exclude_common |
| sql-database | rules_to_exclude_sql |
| windows-os | rules_to_exclude_windows |
| known-bad-inputs | rules_to_exclude_bad |
| amazon-ip-list | rules_to_exclude_reputation |

Now we can provision our infrastructure with that code (or whichever IaC Tool you use):



$ terraform apply

Now we can check whether our changes have been applied with going back to the dashboard from STEP 3. In that screen you need to click to Rules.

Here I will click common. Because I wanted to exclude NoUserAgent_HEADER , and I know that I declared it in exclusion list under the rules_to_exclude_common block and this rule is part of AWSManagedRulesCommonRuleSet.

Below, you can observe that NoUserAgent_HEADER is explicitly allowed now. Additionally, UserAgent_BadBots_HEADER is allowed, as both have been excluded. Remember that SQLi_QUERYARGUMENTS rule has also been excluded, which you can confirm under the sql section rather than common.

So finally, you can see that again I send a request with an empty user agent header, but this time I receive successful response rather than 403 Forbidden.

IDE Extension for AWS Application Composer

Ali Ogun — Tue, 19 Dec 2023 10:45:03 +0000

In re:Invent 2023, AWS announced IDE extension for AWS Application Composer. Let’s explore!

About a year ago, OpenAI unveiled ChatGPT, a Large Language Model(LLM) that drove a great attention, and the AI hype got rapidly integrated into our daily lives. In this wave of innovation, AWS — as a key player pioneer in tech advancements — has been leveraging AI in numerous ways, notably with their recent announcements in AWS re:Invent 2023, like AWS Q. As a Cloud Engineer, Infrastructure as Code (IaC) remains a cornerstone of my daily operations, crucial for maintaining sustainable IT infrastructure. However, amidst the buzz, an announcement seems to me that it has slipped under the radar of the community.

The AWS Application Composer and its IDE Extension, fortified with AI enhancements. While Terraform dominates my workload, I can see the potential significance of this release, especially for those relying heavily on CloudFormation. So, shall we?

What is AWS Application Composer?

In AWS re:Invent 2022 event, AWS announced that service and it’s available since March 2023. AWS Application Composer is a powerful tool that makes developing and deploying apps on the AWS cloud easier. It serves as a visual builder that lets developers quickly and effectively design, build, and coordinate different AWS services to create applications.

This tool makes it easier to integrate several AWS services into a coherent application architecture by offering a drag and drop interface. Using pre-built components, templates, and workflows, developers can shorten the time it takes to construct an application.

Without requiring complex manual configuration, customers can use Application Composer to visually sketch out the architecture of their apps, integrating different AWS services including Lambda functions, API Gateways, databases, and more. However, it’s currently available on the AWS website. AWS has elevated its capabilities with an IDE extension. While it’s only available with Visual Studio Code at the moment, I expect broader IDE support in the future. Let’s take a look at it.

What is IDE extension for AWS Application Composer?

Now, AWS Application Composer seamlessly integrates into your IDE, allowing you to visually construct modern applications and refine your IaC(infrastructure as code) templates using AWS CodeWhisperer (CodeWhisperer is a service like GitHub CoPilot, an AI-powered productivity tool for the IDE and command line that generates codesuggestions based on comments and existing code. ).

[How it looks like on IDE]

This IDE extension mirrors the familiar drag-and-drop experience found in the console, empowering you to swiftly prototype ideas and concentrate on your application code. Not only does it bring Application Composer to your IDE, but it also enables generative AI-powered code suggestions(CodeWhisperer) within the CloudFormation template, all while visualizing the application architecture in a split view. With synchronized visualization and CloudFormation template editing in the IDE, you can refine designs without switching between consoles, reducing manual coding and enhancing productivity. And it is available at no charge. Awesome, isn’t it?

*With Application Composer running in your IDE, you can also use the various tools available in your IDE. For example, you can seamlessly integrate IaC templates generated real-time by Application Composer with AWS Serverless Application Model (AWS SAM) to manage and deploy your serverless applications.*

Beside from the use cases mentioned in the announcement of the IDE extension for AWS Application Composer, I believe of that this will make it far easier to quickly validate what resources will be created by a CloudFormation template. This will be especially useful in CDK stacks where it will work as a quick way to verify that all the right components of an architecture are present without digging through the actual generated template.

It also integrates directly with StepFunctions workflow studio on Visual Studio Code.

Before proceeding, let’s install it together.

How to Install AWS Application Composer in Visual Studio Code?

We’ll install the latest AWS Toolkit for Visual Studio Code plugin. Don’t mind and go ahead if you already have the AWS Toolkit plugin installed.

In Visual Studio Code Extensions Marketplace, you can search for AWS Toolkit and click the blue install button. Or, just click here.

Now, on left hand side you can see on your Visual Studio Code that AWS and Q sections have been added. But as you can see, now we need to login oeither with AWS Builder ID or SSO(single sign-on). And that’s it. You have it.

In order to start using Application Composer, no need to authenticate into your AWS account. With Application Composer available on your IDE, you can open your existing AWS CloudFormation or AWS SAM templates.

Or simply you can create a new empty file, and right-click on the file, then select “Open with Application Composer” to start developing with drag and drop interface of Application Composer. In my case, I created a “try.yaml” file. You can see in the picture below.

Voila! You have it.

As you can see above, it provided me with a blank canvas which I can work on it. Here I have both code and visual editors at the same time. Any changes that I make on the canvas will also be reflected in real time on my IaC template.

As you can see below, my “try.yaml” file is empty for now.

For instance, I just drag and drop on canvas an API Gateway and a Lambda function. Then I connected them simply with drag and drop functionality again.

Do you remember my empty “try.yaml” file? Now let’s go back and look at my YAML file. You can see below in the image that it’s not empty anymore. Generative AI already wrote the IaC for us.

With available IaC templates in your local, it’ll be easier for you to manage the applications with AWS SAM CLI. You can easily create CI/CD with sam pipeline or deploy your stack with sam deploy.

One of the features which will accelerate your dev workflow is the built-in Sync feature that easily integrates with AWS SAM command sam sync. This feature syncs your local app changes to your AWS account, that’s helpful for you to do test and validation before you deploy your app into the prod.

With the new development of IaC templates with generative AI capabilities, you can instantly launch over 1000 resources in CloudFormation with generative AI code suggestions. This simplifies the integration of standard IaC resources and allows you to easily scale your architecture.

For example, suppose you have configured Amazon MQ, a standard IaC resource, and you want to use Application Composer to adjust some settings in your AWS CloudFormation resource. Adjust the values in the Resource Configuration section and select Generate. Application Composer provides code suggestions that seamlessly integrate with your IaC templates. This feature eliminates the need for context switching and streamlines the development process. Build modern applications with AWS Application Composer Canvas while leveraging tools like Amazon CodeWhisperer and AWS SAM to accelerate your development workflow.[*]

Some Possible Use Cases

Rapid Application Prototyping: In today’s Agile IT World, AWS Application Composer’s visual canvas within the IDE may allow for quick prototyping of application ideas. Developers can easily design, modify, and visualize architectures without relying on manual coding. This may accelerate the initial development phase, and enabling faster experimentation and validation of concepts.
Infrastructure as Code (IaC) Development: For CloudFormation users, the IDE extension powered by Gen-AI offers a game changing approach to IaC development. It generates AI code suggestions, which reducing the time spent old style coding CloudFormation templates. The split view feature combines visualization and template editing, streamlining the iteration process and booming productivity.
Enhanced Collaboration and Iteration: By eliminating the need for constant context switching between consoles, teams may collaboratively work on projects more efficiently. The synchronization of Application Composer’s visualization and CloudFormation template editing in the IDE helps easy collaboration, allowing for quicker iterations and better development cycles.
Code Efficiency and Resource Inclusion: The generative AI capabilities empower developers to easily incorporate CloudFormation’s extensive library of resources. This ensures effective coding by providing suggestions that align with specific resource configurations, enabling easy integration of standard IaC resources into architectural designs.
Streamlined Application Development Workflow: AWS Application Composer, in conjunction with Amazon CodeWhisperer and AWS SAM, facilitates a comprehensive and efficient application development workflow. Developers can harness visual design, AI-driven suggestions, and associated AWS tools to streamline the entire application development lifecycle.

If you have any questions, please don’t hesitate to contact me.

Resources

AWS ECS - Spot Instance Draining vs Fargate Spot

Ali Ogun — Mon, 20 Nov 2023 10:21:26 +0000

For most organisations, cutting costs on AWS is a constant struggle. The goal of this article is to assist you to understand key differences in between Spot Instance Draining versus Fargate Spot.

Amazon Web Services (AWS) offers different services for running containerized workloads efficiently. Two of them, ECS Spot Instance Draining and Fargate Spot, serve to different needs within the AWS ecosystem. Understanding their functionalities, benefits, and trade-offs is crucial for optimizing cost of containerised services.

I assume that you already know what the Spot Instance is and how unreliable they are due to the their nature.

AWS ECS Spot Instance Draining

Spot Instance provides us up to 90% off. ECS Spot Instance Draining enables the graceful handling of Spot Instance terminations. This feature allows ECS tasks to respond to termination notifications, ensuring applications complete tasks, preserve data integrity, and shut down gracefully before Spot Instances are reclaimed. But how?

Termination Notifications: You receive advance warnings of Spot Instance terminations, which enabling ECS tasks to prepare and complete ongoing processes.

Automated Spot Instance Draining will automatically place Spot instances in “DRAINING” state upon the receipt of two minute interruption notice. ECS tasks running on Spot instances will automatically be triggered for shutdown before the instance terminates and replacement tasks will be scheduled elsewhere on the cluster. No new ECS service tasks will be started on the instances once the termination process has begun. ECS takes over the coordination of termination of tasks with the termination of the underlying EC2 instance using the inherent instance “DRAINING” functionality. The managed termination, scheduling of replacement tasks and graceful termination of the LB connections reduces the probability of service interruptions. This makes it easier for customers to use Spot instances as part of their ECS cluster.
*

Fargate Spot

Fargate Spot offers a serverless computing experience by managing the underlying infrastructure, including Spot Instances, on behalf of the user. It provides up to a 70% discount off the regular Fargate price. This service combines the benefits of Fargate's serverless approach with the cost efficiency of Spot Instances, ensuring seamless scalability and cost savings. It comes with managed infrastructure advantage.

Managed Infrastructure: Eliminate the need for managing underlying instances, allowing users to focus solely on deploying and managing containerized applications.

Choosing the Right Service for Your Use Case

Management Overhead: Consider the trade-off between managing underlying infrastructure (ECS Spot Instance Draining) and adopting a fully managed service (Fargate Spot). If you don't need to access the underlying infrastructure go for Fargate.
Workload Characteristics: Evaluate workload requirements and make the decision in accordance. Because with Spot Instances pricing is per hour whereas Fargate pricing is per hour based on requested vCPU, memory, Operating Systems, CPU Architecture1, and storage resources for the Task or Pod.

I hope this quick recap was beneficial for you, providing a perspective that helps you to work along within the tools available within AWS for ECS-related operations.

AWS Secrets Manager vs. Systems Manager Parameter Store - Choosing the Right Solution for Your Needs

Ali Ogun — Thu, 21 Sep 2023 10:57:16 +0000

AWS offers two powerful tools, AWS Secrets Manager and Systems Manager Parameter Store. These services are essential for securely managing secrets and configuration data in your AWS environment.

Here, we'll compare AWS Secrets Manager and Systems Manager Parameter Store to help you make a choice for your specific requirements. Whether you're an experienced AWS user or just confused starting, this comparison will guide you in choosing the right solution for your cloud security and configuration management needs.

I. AWS Secrets Manager

AWS Secrets Manager is a robust service designed for the secure management of sensitive information within AWS environments. It provides a centralized repository for secrets, reducing the risk of accidental exposure and simplifies the task of storing, retrieving, and rotating credentials, API keys, and other secrets.

II. Systems Manager Parameter Store

Systems Manager Parameter Store is another valuable service provided by AWS, offering secure management of configuration data and secrets. Systems Manager Parameter Store is designed for storing and managing configuration data, secrets, and other information. It provides a secure and centralized location for these critical assets. And now you see how those two services sound similar. We'll take a look at the similarities and the differences down below. Let's dive in!

III. Comparing AWS Secrets Manager and Systems Manager Parameter Store

Let's dive into a detailed comparison of AWS Secrets Manager and Systems Manager Parameter Store to help you make an informed decision based on your specific needs. We'll explore various aspects of both services:

Ease of Use:
Both services offer similar UIs on which you can declare key-values pairs for your parameters and secrets.
Storage and Encyrption:
When it comes to data storage and encryption, both AWS Secrets Manager and Systems Manager Parameter Store offer robust capabilities, but with some notable differences.
- AWS Secrets Manager: It allows you to store secrets with a generous size limit of up to 64KB. These secrets can include sensitive information like passwords, API keys, and other confidential data. Encryption is seamlessly integrated with AWS Key Management Service (KMS), ensuring that your secrets remain secure. However, it's important to note that AWS Secrets Manager is primarily designed for secret management.
- Systems Manager Parameter Store: It provides flexibility in data storage. Standard Parameters can hold values of up to 4KB (4096 characters), while Advanced Parameters can accommodate larger entries, up to 8KB. Unlike AWS Secrets Manager, Parameter Store's primary use case extends beyond secrets. It is designed to manage various configuration variables, including URLs, database hostnames, custom settings, product keys, and more. While encryption is an option, it's not enforced by default. You can choose to enable encryption explicitly for added security. Encryption in Parameter Store is also handled via AWS KMS, requiring KMS Decrypt permissions for applications retrieving encrypted values.
Version Control:
- AWS Secrets Manager: It allows multiple versions to exist at the same time when you are performing a secret rotation. Secrets Manager distinguishes between different versions by the staging labels.
- Systems Manager Parameter Store: Parameter Store only allows one version of the parameter to be active at any given time.
Secret Rotation:
- AWS Secrets Manager: It offers the ability to switch secrets at any given time and can be configured to regularly rotate depending on your requirements.
- Systems Manager Parameter Store: In contrast, Systems Manager Parameter Store does not provide the same level of automated secret rotation as AWS Secrets Manager. While Parameter Store is a nice tool for managing configuration data and secrets, it does not offer built-in automation for secret rotation. If you rely on Parameter Store for secrets you may need to implement custom rotation processes to achieve the same level of security that AWS Secrets Manager provides.
Cross-Account Access:
- AWS Secrets Manager: It offers a convenient feature for cross-account access. You can configure secrets in AWS Secrets Manager to be accessed from another AWS account. This is particularly useful when you have secrets that are centrally managed in one AWS account but need to be accessed by applications or services running in different accounts. You can set up IAM roles or permissions that allow the necessary cross-account access, ensuring that your secrets remain protected while enabling secure sharing between AWS accounts.
- Systems Manager Parameter Store: It also provides options for cross-account access, although the process may require more manual configuration compared to AWS Secrets Manager. You can implement IAM roles and policies to grant access to Parameter Store parameters from another AWS account. While this approach allows for cross-account access, it may involve additional setup steps and management efforts.
Pricing
- AWS Secrets Manager: AWS Secrets Manager operates on a pay-as-you-go model, meaning you are billed based on your usage. While it offers advanced secret management features, these come at a cost from the very beginning.
- Systems Manager Parameter Store: One of the standout advantages of Systems Manager Parameter Store is that it offers a free tier, allowing you to store up to 10,000 parameters at no cost. This can be particularly cost-effective for organizations with a substantial need for parameter management. If you require more advanced functionality for your parameters, such as setting expiration dates or time-to-live, you can choose to use Advanced Parameters, which comes at an additional cost!

Secrets Manager is purpose-built for the secure management of sensitive information, and as such, it enforces encryption by default when you create a secret. This means that you can never store data in plaintext within Secrets Manager, ensuring that your confidential information remains protected. Additionally, Secrets Manager offers a convenient built-in password generator accessible through the AWS CLI. This feature proves invaluable when you need to create resources like an RDS (Relational Database Service) instance using a CloudFormation template. With the password generator, you can effortlessly generate a randomized, highly secure password and later reference it in your RDS configuration, enhancing the overall security of your resources.

Image-source

IV. How to Choose Between AWS Secrets Manager and Systems Manager Parameter Store

Choosing between AWS Secrets Manager and Systems Manager Parameter Store boils down to your specific needs:

AWS Secrets Manager: Opt for AWS Secrets Manager if you require robust secret management with features like automated rotation, audit logging, and seamless AWS service integration. It's ideal for securing confidential data.
Systems Manager Parameter Store: If you need a versatile solution for managing both secrets and non-secret configuration data, especially in larger quantities, Systems Manager Parameter Store, with its free tier and parameter policies, is a cost-effective choice.

Evaluate your security and cost requirements to make the right choice for your cloud environment.

In a nutshell, AWS Secrets Manager excels in securing confidential data, providing automated rotation, audit logging, and seamless integration with AWS services.

On the other hand, Systems Manager Parameter Store, on the other hand, offers versatility, cost-effectiveness, and a free tier for managing both secrets and non-secret configuration data.

The choice between these services hinges on your specific requirements, balancing security, scalability, and cost considerations. Evaluate your needs carefully to make the right selection for your AWS environment.

The DIY Cloud At Home

Ali Ogun — Sun, 10 Sep 2023 22:01:36 +0000

The DIY Cloud At Home

An Epic Journey with the Cloudfryer!

I’ve been a long-time follower of Metin Seylan, a Senior Software Developer known for sharing both entertaining and educational content. Today, I stumbled upon Metin Seylan’s tweet explaining his homegrown ‘Cloudfryer’ project. I was captivated by the insights he shared and felt compelled to spread the knowledge within the community.

In this article, we will find answers to these questions:

Why did he start this project?
What hardware did he use?
What kind of cloud experience did he create at home?
How much investment did he make in the hardware?
Was all this effort worth it for him to avoid using the cloud?

So, let’s start.

He started learning TensorFlow and realized that his primary need wasn’t a massive GPU but rather an efficient data pipeline.

So, what did he need to do?

According to him;

1) It involved:

Continuously scraping data from websites using bots.
Using Kafka Streams to preprocess this data, making it ideal for model training.

2) His hardware requirements included:

Approximately 36 CPU cores
Around 200GB of RAM
10Gbit network bandwidth
Space-saving

At first, he considered using a Raspberry Pi, but he found it to be:

Insufficient
Costly

His preference was the Lenovo Thinkcentre Tiny M720Q, which he described as a real beast.

3) So, why did he choose it? Let’s take a look

According to him, the Lenovo Thinkcentre Tiny M720Q had several appealing features:

All components are upgradeable.
It supports 8th and 9th generation CPUs, including the i9–9900T (8 Core 16 Thread).
It can handle up to 64GB of RAM.
It has a PCI-E slot, which is a significant advantage.
It offers a variety of storage options, including NVME and SATA slots.

4) To meet his needs, he made the following changes:

Upgraded the CPU to an i5–9500T, offering 6 cores and 6 threads.
Increased RAM to 64GB with Corsair Vengeance SODIMM.
Opted for a Samsung 980 NVME SSD for storage.
Enhanced networking with a 10GB Cluster Networking Intel X540-T2 NIC.

5)Now let’s talk about the software and the automation

All software components were automated using Ansible, eliminating manual installations. It’s entirely open source!
You can find the details and configurations on *his GitHub repository*.

He also mentioned that he used various software components, and you can learn more about which ones and why by checking out the repository.

6) Which software he used and why?

According to him, he used the following software components and the reasons behind their selection:

**Kubernetes** — A well-known choice for container orchestration.
**Cilium** — Selected for Kubernetes networking and bare-metal load balancing support.
**Helm** — Chosen as the Kubernetes package manager.
Prometheus and Grafana — Utilized for cluster observability.
**Longhorn**] — Referred to as “SSD for K8s,” it allows distributed use of all disks within the cluster and automates backup and snapshot processes.
**ArgoCD** — For GitOps.

These software choices were made based on their respective functionalities and suitability for the project’s requirements.

7) So, what is he programming and what are the dependencies?

For his web scraper bot, he is writing code in Kotlin and outlined his approach:

He’s using Kotlin for web scraping.
To build a lightweight and efficient native executable application, he relies on a combination of Spring Boot, Spring Native, and GraalVM.

For storage and pipeline management, he mentioned using:

Kafka for handling messaging.
Kafka Streams for stream processing.

These choices reflect his strategy for building an efficient and resource-friendly web scraper bot and pipeline.

8) The cost of the setup

Regarding the costs of the hardware components, here’s the breakdown:

Think Centre Tiny: Approximately 65 Euro
i5–9500T CPU: Ranging from 30 to 65 Euro
64GB RAM: Approximately 100 Euro
X540 T2 NIC: Ranging from 30 to 70 Euro
NVME: About 30 Euro

He highlights that it’s a budget-friendly setup, mainly consisting of second-hand purchases, making it a cost-effective solution. Roughly around 255 to 330 euro.

9) How about electric bill?

When it comes to electricity consumption, he shared:

At idle, it consumes only about ~66 watts.
Under heavy usage, it’s around ~250 watts or so.

10) Why just not cloud?

He explained why he opted for an on-premises setup instead of the cloud:

He doesn’t have critical, customer-focused work, so concerns about downtime and security are minimal.
His home has a parallel internet connection with 1GBit up/down speeds, which makes it feasible for him to run his own infrastructure effectively.

These factors contributed to his decision to choose an on-premises solution over a cloud-based one.

11) How much would it cost on the cloud?

He shared his estimation on Google Cloud Platform(GCP). By the way, it’s time to mention that he is a Google Cloud Developer Expert. Don’t forget to visit his personal website. And here is the GCP bill:

12) How about AWS?

As an AWS Community Builder, I want to share the cost of a similar infra on AWS(Amazon Web Services). If you go with an on-demand machine like the Metin did(I assume), the cost is:

And this would make around 2,694 USD. It’s slightly more expensive than GCP. But since we compare it with a home setup, there will be an initial investment. Hence, we can choose Reserve Instances with a 3-year commitment. And the price goes down to:

It’s monthly 2,403 USD. With proper adjustments, this price can go even more down. It’s just a matter of adjustment according to your use cases.

By making informed plan choices, cloud bills can be significantly reduced. This underscores the invaluable role of consulting with cloud professionals who can guide you toward optimizing your infrastructure and keeping costs in check.

As you can see with specific requirements, a meticulously crafted ‘Cloudfryer’ setup at home might just be the perfect recipe. However, the cloud offers unparalleled scalability, convenience, and cost-efficiency, making it an enticing choice for many. So, as you embark on your own journey of technological exploration, remember that the answer lies in aligning your choice with your goals, resources, and ambitions. Whichever path you choose, whether it’s the cozy corner of your home or the boundless skies of the cloud, let it serve as the canvas upon which you craft your digital masterpiece.

To see the original Twitter thread(in Turkish) click here.

Let’s Create a Cron Job

Ali Ogun — Sat, 09 Sep 2023 12:58:25 +0000

In this article we will write a bash script and set a schedule with crontab. Students can use this article for “born2beroot” project (from 42 Cursus). My greetings to especially 42-Heilbronn peers. Okay, let’s start real quick.

Bash Script

If you are already familiar with bash, you might skip this part and directly jump next section. (Setting Up Crontab)

Create script file under this directory as: /usr/local/bin/monitoring.sh

Let’s take a look at our script below and dive into it.

#!/bin/bash

arc=$(uname -a)
pcpu=$(grep “physical id” /proc/cpuinfo | sort | uniq | wc -l)
vcpu=$(grep “^processor” /proc/cpuinfo | wc -l)
fram=$(free -m | awk ‘$1 == “Mem:” {print $2}’) 
uram=$(free -m | awk ‘$1 == “Mem:” {print $3}’)
pram=$(free | awk ‘$1 == “Mem:” {printf(“%.2f”), $3/$2*100}’) 
fdisk=$(df -Bg | grep ‘^/dev/’ | grep -v ‘/boot$’ | awk ‘{ft += $2} END {print ft}’)

udisk=$(df -Bm | grep ‘^/dev/’ | grep -v ‘/boot$’ | awk ‘{ut += $3} END {print ut}’)

pdisk=$(df -Bm | grep ‘^/dev/’ | grep -v ‘/boot$’ | awk ‘{ut += $3} {ft+= $2} END {printf(“%d”), ut/ft*100}’)

cpul=$(top -bn1 | grep ‘^%Cpu’ | cut -c 9- | xargs | awk ‘{printf(“%.1f%%”), $1 + $3}’)

lb=$(who -b | awk ‘$1 == “system” {print $3 “ “ $4}’)lvmt=$(lsblk | grep “lvm” | wc -l)

lvmu=$(if [ $lvmt -eq 0 ]; then echo no; else echo yes; fi)
ctcp=$(cat /proc/net/sockstat{,6} | awk ‘$1 == “TCP:” {print $3}’)
ulog=$(users | wc -w)
ip=$(hostname -I)
mac=$(ip link show | awk ‘$1 == “link/ether” {print $2}’) 
cmds=$(journalctl _COMM=sudo | grep COMMAND | wc -l)

wall “ 
       #Architecture: $arc
       #CPU Physical: $pcpu
       #vCPU: $vcpu
       #Total and Used Amount of RAM: $uram/${fram}MB ($pram%)
       #Disk Amount and Use: $udisk/${fdisk}Gb ($pdisk%)
       #CPU Use Rate: $cpul
       #Last Restart Time: $lb
       #LVM State of Use: $lvmu
       #Active Number of Connection: $ctcp ESTABLISHED
       #Number of Users Using the Server: $ulog
       #IP and MAC Addresses: IP $ip ($mac)
       #Number of Used Sudo: $cmds cmd
     ”

So, let’s explain briefly what I wrote above:

arc=$(uname -a) → Shows the architecture and kernel version of the current operating system.
pcpu=$(grep “physical id” /proc/cpuinfo | sort | uniq | wc -l) → Returns the number of physical processors.
vcpu=$(grep “^processor” /proc/cpuinfo | wc -l) → Returns the number of virtual processors.
fram=$(free -m | awk ‘$1 == “Mem:” {print $2}’) → Returns the amount of available RAM of the server.
uram=$(free -m | awk ‘$1 == “Mem:” {print $3}’) → Returns the amount of RAM used.
pram=$(free | awk ‘$1 == “Mem:” {printf(“%.2f”), $3/$2*100}’) → printf(“%.2f”) returns 2 values after the comma and if it is $3/$2*100 then it returns percentage version.
fdisk=$(df -Bg | grep ‘^/dev/’ | grep -v ‘/boot$’ | awk ‘{ft += $2} END {print ft}’) → Returns the amount of available storage of the server.
udisk=$(df -Bm | grep ‘^/dev/’ | grep -v ‘/boot$’ | awk ‘{ut += $3} END {print ut}’) → Returns the server’s used storage space.
pdisk=$(df -Bm | grep ‘^/dev/’ | grep -v ‘/boot$’ | awk ‘{ut += $3} {ft+= $2} END {printf(“%d”), ut/ft*100}’) → (storage used / accessible space * 100) gives us percentage usage.
cpul=$(top -bn1 | grep ‘^%Cpu’ | cut -c 9- | xargs | awk ‘{printf(“%.1f%%”), $1 + $3}’) → Returns the CPU utilization rate as a percentage.
lb=$(who -b | awk ‘$1 == “system” {print $3 “ “ $4}’) → Returns the last reboot date and time.
lvmt=$(lsblk | grep “lvm” | wc -l) → Returns the information of disks configured with LVM.
lvmu=$(if [ $lvmt -eq 0 ]; then echo no; else echo yes; fi) → Indicates whether LVM is active in the system. P.S: You need to install the net-tools package for other items to run smoothly.
ctcp=$(cat /proc/net/sockstat{,6} | awk ‘$1 == “TCP:” {print $3}’) → Returns the current number of active connections.
ulog=$(users | wc -w) → Returns the number of users using the server.
ip=$(hostname -I) → Gives the server’S IP Address.
mac=$(ip link show | awk ‘$1 == “link/ether” {print $2}’) → Gives the server’s MAC Address.
cmds=$(journalctl _COMM=sudo | grep COMMAND | wc -l) → Returns the number of commands run with sudo. (If sudo is logged in, it is written with the number of sudo usage of other users; otherwise, it gives as much information as the sudo command used by the current user.)
wall “some message” → is a command-line utility that displays a message on the terminals of all logged-in users. The messages can be either typed on the terminal or the contents of a file.

Add the rule that script could be executed without sudo password

Open sudoers file:

$ sudo visudo

Add this line:

$ your_username ALL=(root) NOPASSWD: /usr/local/bin/monitoring.sh

And the reboot the system with :

$ sudo reboot

You can check if your script works well with this command:

$ sudo /usr/local/bin/monitoring.sh

Okay we completed the script. Now we gotta setup the crontab for scheduling it.

Setting Up Crontab

Open the crontab with:

$ sudo crontab -u root -e

At the end of the file add this rule as it shown in Figure-1:

$ */10 * * * * /usr/local/bin/monitoring.sh

And finally use chmod +x to make your script executable:

$ chmod +x monitoring.sh

And you are ready to go. You should have such an output every 10 minutes:

Well done!

Deploy React App to AWS S3 Bucket with GitLab Pipeline

Ali Ogun — Sat, 09 Sep 2023 12:56:07 +0000

How to deploy React App into AWS S3 Bucket?

In this article step by step I will try to explain how to deploy a basic React App on AWS S3 bucket and how to automate this process with GitLab CI/CD pipeline.

I have prepared a simple primitive React App. With Dockerfile, I dockerize my React App. I try to keep this docker image as small as possible with .dockerignore file. I deploy my React App on to AWS S3 bucket. In order to improve the efficiency, I built a CI/CD pipeline on GitLab.

Prerequisites

Static Web Page — I have a one single static page built with React App.
**AWS Account — **This little deployment will be in free-tier limits.
GitLab Account — We need this to build a CI/CD pipeline.

📦 Dockerize

In order to keep my docker image as small as possible, I’ve done 2 things:

I’ve used .dockerignore file to excluded unnecessary files
I’ve used node:slim as base image. As the word "slim" imply, this image is slim version of node image. I've used node image to be able to use npm package manager.

In my Docker image, I’ve used two main commands. First, I’ve build the artifact form the files with:

$ npm run build

Then, I’ve run the artifact with:

$ npm start

🚀 Deployment

I use a GitLab CI/CD pipeline to carry out the deployment which I will cover in upcoming section more detailed.

♾️ CI/CD Pipeline

For pipeline, I use GitLab CI/CD. In order to provision my pipeline, I use .gitlab-ci.yml file.

It looks like this:

IMPORTANT: Before use of this file, please remember to add those:

AWS ACCOUNT ID
AWS_ACCESS_KEY (I added as Masked Variables)
AWS_SECRET_KEY (I added as Masked Variables)

⚙️ Pipeline Architecture

I designed my pipeline on GitLab CI/CD with 4 stages and 4 jobs as seen in the figure:

For more information, you can click here. Anyway, my CI/CD pipeline’s stages are like down below:

Build — Here I build the artifact.
Test — Here I do tests with npm test command. One important thing is, npm test requires an interactive use. In order to turn this off, I've used CI=true parameter.
**Image-push — **In this stage I dockerize my React App. And then I push the image to the AWS ECR.
**Deploy — **In deploy job, I am deploying my React App to AWS S3. By this way I am hoping to access my React App and represent a demo. You can click here to check how it looks like.

Here is the link of my React App, if you would like to take a look at it:
*https://protein-bootcamp-prod.s3.eu-central-1.amazonaws.com/index.html*

If you are not willing to click the link, let me leave here an image for you to see how it works:)

☁️ Amazon Web Services

I use AWS as Cloud Provider in this projet. I tried to use Free-tier services. Anyway, it is always good practice to set an alert threshold in budget. Watch out your money!

Resources

Hosting a static website using Amazon S3
GitLab CI/CD | GitLab

Replace GitLab Shared Runners With an AWS EC2 Instance

Ali Ogun — Sat, 09 Sep 2023 12:46:48 +0000

No More GitLab Runner

We love GitLab. We love GitLab CI/CD Pipeline. Out of question. Free GitLab runners are perfect for personal small projects since they are free, available and fully managed. On the other hand it may seem short for the bigger project’s pipelines. And the reason is obvious. Because they are shared with other users.

Ain’t we tired of old and slow GitLab Runners? If the answer is yes, then this article is cut out exactly for you. We will use an AWS EC2 machine for our pipeline to run at. I will stay in free-tier during the article, so no worries.

How to do?

1. Create an AWS account and log in.

2. Create an EC2 machine:

I use t2.micro machine since it will be enough for my projet and it is free-tier eligible. Don’t forget to choose your PEM because we will use it in next step.

I’ve used Linux/GNU Debian distribution but you can choose any distro you want.

You can choose the computation power of the machine as the size of your CI/CD jobs grow. I’ve used default settings since I was deploying a very basic React app.

3. Connect to EC2 machine:

Open an SSH client. Locate your private key file.For instancein my case it is protein.pem. Then run the comment below. But replace the protein.pem with your own PEM file name and also change the PublicIPv4DNS address with your own EC2 machine's ip address:

$ ssh -i "privatekey.pem" ec2-user@PublicIPv4DNS

4. Take registration token from GitLab:

In the GitLab repository respectively click Settings > CI/CD > Runners. There you will see your registration token. We will use it to connect our pipeline to our EC2 machine.

5. Install Gitlab Runner to EC2:

Download the binary for your system:

$ sudo curl -L --output /usr/local/bin/gitlab-runner https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-linux-amd64
Give it permission to execute

$ sudo chmod +x /usr/local/bin/gitlab-runner
Create a GitLab Runner user

$ sudo useradd --comment 'GitLab Runner' --create-home gitlab-runner --shell /bin/bash
Install and run as a service

$ sudo gitlab-runner install --user=gitlab-runner --working-directory=/home/gitlab-runner $ sudo gitlab-runner start

6. Register Runner to EC2

$ sudo gitlab-runner register --url [https://gitlab.com/](https://gitlab.com/) --registration-token $REGISTRATION_TOKEN

Don’t forget to replace your registration token in the above script.

7. Activate Your Runner

Now you runners should be visible in Settings > CI/CD > Runners like this:

Click the button shown in the image in order to activate your runners.

7. Deactivate Shared Runners

As default GitLab use shared runners. So, stop shared runners to use your runners to carry out your pipe line executions.

Congratulations! You’re done. Now, you are running your pipeline on your own EC2 machine. As a result, your Settings > CI/CD > Runners screen should looks like this:

Configuration

For advance settingsou can change the behavior of GitLab Runner and of individual registered runners.

To do this, you modify a file called config.toml, which uses the TOML format.

You can find the config.toml file in: /etc/gitlab-runner/

Here as shown:

And the last but not the least; I leave my git repository link here where you can find my config.toml file. Here you can go and take a look at it.

Discover the key differences between AWS WAF, ACLs, and Security Groups to protect your cloud applications like a pro!

Ali Ogun — Sat, 09 Sep 2023 12:44:22 +0000

When it comes to securing your applications and infrastructure on the cloud, there are various tools and services available. AWS offers several security features that can help you protect your resources. I was recently asked by one of my colleagues about what the difference is between AWS WAF vs ACLs vs Security Groups. And here we are!

While these tools all serve the same purpose, there are differences between them that you need to understand to choose the right tool for your needs.

What is AWS WAF?

In the simplest way, AWS WAF (Web Application Firewall) is a Firewall service as the name implies.

It helps protect your web applications from common web exploits that could affect application availability, compromise security, or consume excessive resources. AWS WAF allows you to create rules that block common attack patterns, such as:

SQL injection
Cross-site scripting (XSS)
UDP floods
SYN floods
HTTP floods
Bad bots
Spoofing

With AWS WAF, you can also protect your APIs from bots and scrapers and can create custom rules or use pre-configured rules to protect your applications.

What is NetworkACL?

Network ACL (Access Control List) is used to filter traffic based on IP addresses or ranges.

ACLs can be applied to individual subnets or network interfaces, and they allow or deny traffic based on the rules you set up. ACLs are commonly used to control traffic in and out of a VPC (Virtual Private Cloud). Besides that, ACLs are stateless. It means they do not keep track of the traffic flow.

What are Security Groups?

Security Groups are also used to control traffic, but they operate at the instance level rather than the subnet level. It is the security level that you encounter after you pass through the NACLs.

Security Groups act as virtual firewalls for your instances, controlling inbound and outbound traffic. Security Groups are stateful, which means they keep track of the traffic flow. This makes them easier to use than ACLs, as you don’t have to worry about setting up separate rules for inbound and outbound traffic.

Differences between AWS WAF, ACLs, and Security Groups

The main difference between AWS WAF, ACLs, and Security Groups is the level at which they operate. AWS WAF is a service that operates at the application layer, protecting your web applications from common web exploits. ACLs operate at the subnet level, controlling traffic in and out of your VPC based on IP addresses or ranges. Security Groups operate at the instance level, acting as virtual firewalls for your instances and controlling inbound and outbound traffic.
Another key difference is the type of traffic they filter. AWS WAF is focused on protecting your web applications from common web exploits, such as SQL injection or cross-site scripting. ACLs and Security Groups, on the other hand, are focused on controlling traffic based on IP addresses or ranges.
Finally, there is a difference in the way they handle the traffic. ACLs are stateless, which means they don’t keep track of the traffic flow. This makes them more difficult to manage than Security Groups. Because Security Groups are stateful and can track the traffic flow. By the way, AWS WAF can also be used in conjunction with Security Groups to provide additional protection for your web applications.

Let me try to provide an analogy to make things clear. Imagine that you are an individual who places a high value on personal security and decides to hire a team of security personnel. For example, you may choose to hire three employees.

A private guard
An old security guy in front of your building
A security desk in front of your home

Your private WAF guardians remain ever-vigilant against potential hazards. They are highly professional and continuously monitor for security risks, always keeping an eye on the bigger picture.

Someone wants to enter the building to see you. They must pass our security personnel, but our ACL security personnel is elderly and cannot remember everyone who comes in or goes out. Also, as you can see, he does not have a desk to write down people’s names. Therefore, he must always check who you are whenever you enter or exit.

Finally, your visitor has arrived at the front door of the building, but there is one more security measure in place — Security Groups. As you can see, this security personnel works indoors and has a desk to keep track of information. Once you inform him of your identity upon entering, he will record it and allow you to exit without asking again. This is because he knows who you are, unlike the ACL security personnel who must check again when you exit the building.

So the image pretty much looks like this:

Summary

In summary, AWS WAF, ACLs, and Security Groups are all important tools for securing your applications and infrastructure on the cloud. While they all serve the same purpose, they operate at different levels and filter different types of traffic. Understanding the differences between these tools is important when choosing the right tool for your needs. By choosing the right tool, you can ensure that your applications and infrastructure are protected from common web exploits and unauthorized access.

I chose EC2 Storage over EBS despite it’s ephemeral — Here’s Why…

Ali Ogun — Sat, 09 Sep 2023 12:41:16 +0000

In the world of cloud infrastructure, selecting the right storage solution is essential to getting the best performance and reliability. I don’t say it as a AWS Storage Community Builder, I tell it also as a regular AWS customer. I started to think about writing this article when I had to choose between EC2 Storage and EBS (Elastic Block Store).

Contrary to popular belief, I decided on EC2 Storage despite its ephemeral nature. I’ll explain the motivations behind my unusual choice in this article, as well as the benefits that EC2 Storage offers.

First thing first;

1- EC2 Storage vs EBS:

Before diving into the logic behind my choice, let’s establish a foundational understanding of EC2 Storage and EBS. If you are already familiar with the services, you can simply skip this section. The local instance storage that comes with Amazon EC2 instances is referred to as “EC2 Storage”; it provides high-speed performance temporary block-level storage but lacks durability. EBS, on the other hand, offers robust and long-lasting block storage that is compatible with EC2 instances. While EBS is frequently the preferred option, I needed to go against it because of my client’s requirement, but I decided to investigate the unrealized potential of EC2 Storage. If you want to read more about other possible storage solutions on AWS, I recommend you to read my article.

Main reason is;

2- Superiority of Performance:

One of the main reasons and also the ultimate one, for choosing EC2 Storage was its exceptional performance. With EC2 Storage, you can simply experience lightning-fast data access, accelerated disk I/O, and remarkably low latency. Me personally in this use case, I had a database requires a block storage with the throughput to support several million transactions per second. If you look closer to the EBS; you can get more information about EBS types. Although let me give a quick answer; 350,000 IOPS. Well if you don’t know what IOPS is, Input/output operations per second **(IOPS*, pronounced *eye-ops) is an input/output performance measurement used to characterize computer storage devices like hard disk drives (HDD), solid state drives (SSD), and storage area networks (SAN). Like benchmarks, IOPS numbers published by storage device manufacturers do not directly relate to real-world application performance.

And unrevealed potential;

3- Flexible and Cost-Efficient:

EC2 Storage’s inherent flexibility and cost effectiveness were further strong arguments in its favor. As an ephemeral storage option, EC2 Storage can be dynamically provisioned and terminated as needed. This flexibility empowered me to design highly scalable and fault-tolerant architectures, leveraging auto-scaling capabilities and reducing unnecessary costs associated with persistent storage.

I chose a different route by embracing the ephemeral nature of EC2 Storage in a world where EBS is frequently seen as the de facto option for persistent storage. The client requirement for superior performance, flexibility, and cost effectiveness motivated me to architect this solution. I wanted to use this blog to show off EC2 Storage’s latent potential and encourage readers to choose a creative storage solution. Even while EC2 Storage might not be appropriate for all use cases, it has changed the game for me. We can open new doors and significantly improve our cloud infrastructure by questioning the established quo and looking at other solutions.

Remember, sometimes it’s worth venturing into uncharted territory to uncover hidden gems that can revolutionize our technological landscapes. I also encourage you to read this article to get more detailed about using AWS EC2 instance store vs EBS for MySQL.

Byzantine Generals Problem

Ali Ogun — Sat, 09 Sep 2023 12:37:12 +0000

Byzantine Generals Problem

A Byzantine fault is a state of a computer system, particularly distributed computing systems, where components may fail and there is imperfect information on whether a component has failed. It is also known as a Byzantine generals problem, interactive consistency, source congruency, error avalanche, Byzantine agreement problem, and Byzantine failure. The phrase gets its name from an allegory known as the “Byzantine generals problem”, which was created to explain a scenario in which the system’s participants must agree on a coordinated approach to prevent catastrophic system failure, yet some of these individuals are unreliable.

A server, for example, may inconsistently appear to failure-detection systems as both malfunctioning and working while exhibiting various symptoms to various observers in a Byzantine fault. It is challenging for the other components to declare it failed and block it from the network since they first need to agree on which component actually failed. A fault-tolerant computer system’s resistance to such circumstances is known as byzantine fault tolerance (BFT).

Consider a group of generals attacking a fort as an example of the flaw in its most basic form. The generals must determine whether to advance or retire as a group; some may favour advance while others favour retreat. The most crucial factor is that all generals come to an agreement on a strategy since a few generals’ haphazard attack would result in a rout and would be worse than either a coordinated attack or a coordinated retreat.

The issue is made more difficult by the existence of treacherous generals who might not only vote for a poor strategy, but also do so selectively. For instance, the ninth general may send a vote of attack to the remaining generals and a vote of retreat to the four generals who support attacking while the remaining four generals support retreating. While the remainder will attack (which might not go well for the attackers), those who obtained a vote to retreat from the ninth general will retreat. The generals’ physical separation from one another and requirement to convey their votes via messengers who might not deliver them or might fabricate votes adds to the difficulty of the situation.

Some Solutions

In 1982, Lamport, Shostak, and Pease described a number of pioneering solutions. They started off by noticing that the Generals’ Problem may be reduced to solving a “Commander and Lieutenants” problem, in which all faithful Lieutenants must act together and that their actions must match what the Commander ordered, assuming the Commander is loyal:

One method takes into account circumstances in which signals may be faked, but they would still be Byzantine-fault tolerant so long as there were less than one-third of the generals who were disloyal. It basically comes down to showing that the one Commander and two Lieutenants problem cannot be handled if the Commander is a traitor. Dealing with one-third or more traitors is impossible. To illustrate this, imagine that we have a traitorous Commander A and two Lieutenants, B and C. When A orders B to attack and C to retreat, and B and C communicate with one another by forwardeding A’s message, neither B nor C can determine who the traitor is because it may not necessarily be A — the other Lieutenant may have forged the message claiming to be from A. It can be shown that if n is the number of generals in total, and t is the number of traitors in that n, then there are solutions to the problem only when n > 3*t* and the communication is synchronous (bounded delay)
Unforgeable message signatures are required for the second solution. Digital signatures (in contemporary computer systems, this may be accomplished in practise via public-key cryptography) can offer Byzantine fault tolerance in the presence of an arbitrary number of betraying generals for security-critical systems.

You may hear the term “Byzantine Generals Problem” also in Blockchains. Blockchain is a public, distributed ledger that contains the records of all transactions. If all users of the Bitcoin network, known as nodes, could agree on which transactions occurred and in what order, they could verify the ownership and create a functioning, trustless money system without the need for a centralized authority. Due to its decentralized nature, blockchain relies heavily on a consensus technique to validate transactions. It is a peer-to-peer network that offers its users transparency as well as trust. Its distributed ledger is what sets it apart from other systems. Blockchain technology can be applied to any system that requires proper verification.[*]

In terms of distrbiuted systems; a source processor is assumed to broadcast its initial value to another processor in the system in accordance with the Byzantine agreement problem notion. To select a source processor, sequential ordering is used and a source processor is chosen at random.

All non-faulty processors must agree on the same value in order to obtain the byzantine agreement procedure. The source processor might not always be defective. Then, other processors do not use its starting value.

The goal of the Byzantine problem is to develop defences against failures in which system components fail in unpredictable ways, such as by processing requests erroneously, corrupting their local information, or generating inconsistent or wrong outputs. The Byzantine failure simulates real-world conditions in which computer and network hardware failures, network congestion, disconnection, and malicious attacks can cause computers and networks to behave in unexpected ways.

In an article published by MICHAEL J. FISCHER, NANCY A. LYNCH, and MICHAEL S. PATERSON; they have shown that a natural and important problem of fault-tolerant cooperative computing cannot be solved in a totally asynchronous model of computation. These results do not show that such problems cannot be “solved” in practice; rather, they point up the need for more refined models of distributed computing that better reflect realistic assumptions about processor and communication tim- ings, and for less stringent requirements on the solution to such problems.

To sum up, we find ourselves at the nexus of creativity and history as we reach the end of our adventure through the complex world of the Byzantine Generals Problem. This age-old puzzle, which was inspired by the generals of the Byzantine Empire, has survived the test of time and now paves the way for contemporary technologies.

The underlying ideas of blockchain technology and distributed systems are intimately resonant with the nature of the challenge, which is attaining consensus and order in the face of deceit and ambiguity. Just as it previously directed commanders seeking victory behind the walls of Byzantium, The Byzantine Generals Problem acts as a beacon, directing us through the maze-like intricacies of networked communication.

The Byzantine Generals Problem is an enduring challenge in the blockchain realm, necessitating creative solutions to guarantee that ledgers are immutable, trust is upheld, and digital transactions are secure. This emphasises the importance of the Byzantine Fault Tolerance (BFT) algorithms, consensus mechanisms, and cryptographic protocols that now underpin the decentralised currency and smart contract foundations.

The Byzantine Generals Problem is a signal for distributed systems engineers and architects to take up arms. It serves as a reminder that reaching consensus in a networked environment where components may malfunction or act maliciously is a riddle requiring creative solutions. It encourages us to create systems that, like those ancient generals, can cooperate and have faith in the face of uncertainty.

As we leave the Byzantine Generals to their ancient strategies, we step into the future, armed with the wisdom they imparted. Their challenge has evolved, but the quest for consensus and order endures. It’s a reminder that in the ever-expanding landscape of technology, the lessons of the past continue to shape the possibilities of tomorrow. So, with Byzantium as our guide, let us continue forging ahead, navigating the Byzantine labyrinth of the digital age, and seeking innovation amidst the echoes of history.

What Byzantine Generals Teach Us About AWS SQS

Ali Ogun — Sat, 09 Sep 2023 11:52:40 +0000

Ever heard of the Byzantine Generals? Click here and check it out. But in a nutshell: Generals trying to launch a coordinated attack on a city, but some of them are secretly plotting to confuse the troops. It's like trying to plan a surprise party when a few friends are determined to spill the beans.

Now, let's step into the tech world. We're talking about messages traveling through a web of computers and bouncing around. They need to arrive in the right order. This is where the message queues comes in. In AWS the service is called Amazon Simple Queue Service (SQS).

So, how come they are related you are wondering. Right? Stay still. Today, we'll uncover the connection between Byzantine Generals and SQS. It's a story of how ancient problems can teach us a lot about modern tech. So grab your coffee, and let's dive in!

AWS SQS and Message Ordering:

In AWS, SQS is the very first service that was published for public use in 2004. In cloud computing, where information flows like a river through Amazon Web Services (AWS). Among its many offerings, Amazon Simple Queue Service (SQS) stands as a message queue service for Asynchronous messaging. It's essential to create decoupled resilient applications.

Within AWS SQS, two distinct queue types emerge: Standard Queues and FIFO (First-In-First-Out) Queues.

Standard Queues : It offers a best-effort approach to message ordering. The reason is here standard queues prioritize scalability and high throughput, which may lead to messages being processed out of order. But why? We'll come to that in a moment.

FIFO Queues : These queues guarantee that messages are delivered in the exact same order. When message ordering is a crucial assurance for applications, such as processing financial transactions or completing e-commerce sales, this service comes in handy.

Okay, here that question comes to mind. What is best-effort and why my messages come out of order in the standard queue?

To make things easy to understand, we use often illustrations to grab that kind of abstract concept like SQS. You can see that it looks like a solid pipe. But is it? What is going on under the hood really? Let's take a look at the image below.

This image is from AWS reinvent event. You can see that it's represented as a solid pipe. But let's take a look at the next image.

In this image, messages seem randomly distributed but of course, sorted with best-effort. What is best-effort? Why order is not guaranteed? Let's zoom into the pipe itself.

If we look closer, we understand the reason why it's not a solid pipe. *Because it is distributed on Amazon SQS servers!

What do I mean? When you push a message into the queue, Amazon SQS redundantly stores a message in more than one availability zone (AZ). Message copies are stored in multiple AZs, no single computer, network, or AZ failure can make messages inaccessible. It's perfect right! A standard queue makes a best effort to preserve the order of messages, but more than one copy of a message might be delivered out of order.

Amazon SQS stores copies of your messages on multiple servers for redundancy and high availability.

On rare occasions, one of the servers that stores a copy of a message might be unavailable when you receive or delete a message. If this occurs, the copy of the message isn't deleted on that unavailable server, and you might get that message copy again when you receive messages. *

If you don't mind the order of the messages in your use case, it brings you some superiority in terms of scalability. As long as you keep pushing new messages, AWS can spin up more and more SQS servers and keep handling your messages. Well, here the price that you pay is an ordering issue.

Okay, but how does SQS FIFO Queue work its magic?

Here the AWS team put in hard work to guarantee and to provide exactly-once processing. The exact details of how AWS SQS maintains the order of messages in a FIFO queue are not publicly disclosed, as this is considered part of AWS's internal architecture and proprietary information.

It's a blend of intelligent mechanisms. Each message entering an SQS FIFO queue carries a unique Message Deduplication ID to prevent duplicates. Additionally, messages are organized into Message Groups defined by a Message Group ID. Messages within the same group are processed sequentially, respecting the order of entry. The AWS team here making an extra effort to make sure the order is maintained which comes with a cost. It's more expensive than Standard Queue and has a limited number of transactions per second (TPS).

The second thing is the maximum timeout for storing a given message deduplication ID is 5 minutes. A duplicate message would be accepted and continued processing in the SQS FIFO queue if it arrived more than five minutes after the original. If exactly-once were a crucial requirement, other distributed system components would have to be responsible for ensuring message uniqueness.

So, let's come back to our topic. How all those are related to the Byzantine Generals?

Understanding the Byzantine Generals Problem:

To grasp the connection between Byzantine Generals and the modern-day challenges of message ordering, let's take a closer look at the historical problem.

Imagine an ancient army preparing to lay siege to an enemy's city. The success of their mission depends on synchronized attacks from multiple fronts. But here's the catch: not all the generals can be trusted. Some of them might be traitors and they can sabotage the plan.

In the face of this problem, the loyal generals must somehow agree on a common strategy, even though they can't be sure if the orders they receive from their fellow commanders are genuine or deceitful. The challenge, known as the "Byzantine Generals Problem", is to achieve consensus in a distributed and potentially unreliable communication network.

The complexity of contemporary distributed systems corresponds to this historical issue in a world where integrity is of utmost importance. Our digital systems struggle with faulty parts, network issues, and the necessity of maintaining the right sequence of events, much like the devoted generals trying to plan their attack.

The Byzantine Generals Problem serves as a timeless parable of the difficulties inherent in distributed decision-making. It underscores the vital importance of achieving consensus and maintaining order, whether on the battlefield of ancient Byzantium or in today's cloud computing infrastructure.

Message Ordering Challenges in Distributed Systems such as SQS:

Maintaining the proper sequence of messages is comparable to orchestrating a difficult symphony in the interconnected world of distributed computing, where data passes across networks. Consider a situation where you must make sure a set of activities takes place exactly as planned.

Distributed systems(such as SQS) often involve multiple nodes(most probably different AZs) or components that communicate and collaborate to achieve a common goal. However, these systems operate in a volatile environment with uncertainty. Network delays, hardware failures, and asynchronous communication can introduce chaos into what should be an orderly sequence of events. This quest for precise message ordering is a fundamental challenge in distributed systems.

In the AWS world, AWS Simple Queue Service (SQS) comes to the rescue and AWS team manages this chaos for us. SQS offers a solution to the solve the message ordering problem. However, it's not the end of the story. SQS FIFO queues guarantee in-order and exactly-once processing only within the queue itself, not in the entire distributed system that would use the queue.

The Total Order Broadcast problem appears in the field of distributed computing. According to the Total sequence Broadcast, messages must be sent to every participant in the same sequence, regardless of who they are to. In a distributed system, FIFO delivery of messages is basically a special case of the issue with extra order restrictions. Understanding the constraints of potential FIFO delivery problem solutions is aided by research on the Total Order Broadcast problem. Simply put, the Total Order Broadcast problem is equivalent to the problem of achieving distributed consensus, i.e. having all participants in a distributed system agree on message delivery order.

Although a SQS FIFO queue is undoubtedly a very helpful tool in developing a solution to this issue, it is not a solution in and of itself. The task of guaranteeing exactly-once publication outside the queue typically falls to us developers.