DEV Community: Mohammad Quanit

Asteroids Game with Amazon Q: My Journey with AI-Assisted Game Development

Mohammad Quanit — Wed, 28 May 2025 10:27:53 +0000

Introduction

Recently, I participated in the Amazon Q challenge by creating a classic Asteroids game using Python and Pygame. What made this project unique was leveraging Amazon Q's AI capabilities to assist with the development process.

For the organizers and anyone who wants to play this game:

Clone this GitHub repository for the game codebase,
Go to folder aws-q-game-challenge
Run source venv/bin/activate in bash
Run python asteroids.py in bash.

In this blog post, I'll share how I set up Amazon Q
CLI, the prompts I used to create the game, and how I enhanced it with custom assets.

Setting Up Amazon Q CLI

Before diving into game development, I needed to set up Amazon Q CLI on my macOS system. Here's how I did it:

Download the AWS Q CLI (if not already installed) from this link: AWS Q CLI for MacOS
Start a chat session:

q chat

With Amazon Q CLI set up, I was ready to build my game.

The Development Process

Initial Game Creation

I started by asking Amazon Q to help me create a basic Asteroids game. My first prompt was:

"I want to create a game using pygame, named asteroid_game. Can you help me with the basic structure?"

Amazon Q provided me with a complete initial implementation that included:
• Game window setup
• Player spaceship controls
• Asteroid generation
• Collision detection
• Scoring system
• Game loop structure

The initial version used simple geometric shapes for the spaceship and asteroids, which worked well but lacked visual appeal.

Enhancing with Real Images

To make the game more visually appealing, I asked Amazon Q:

"I've created a game using pygame, named asteroid_game. I want to use real stone images and a real spaceship for shooting."

Amazon Q then:

Created an assets directory structure
Modified the code to load and use external images
Added fallback mechanisms if images weren't available
Provided guidance on where to find free game assets

Customizing the Spaceship

I wanted to make the spaceship more prominent, so I used this prompt:

"Make the spaceship a bit bigger and move it to more bottom side of the game screen."

Amazon Q adjusted the code to:
• Increase the spaceship size from 40x50 to 60x75 pixels
• Position the ship closer to the bottom of the screen
• Adjust the bullet spawn position to match the new ship size

Game Assets

For the visual elements of the game, I used the following assets:

Spaceship: A top-down view spaceship image (spaceship.png) placed in the assets folder
Asteroids: Three different asteroid images for different sizes: • asteroid_large.png • asteroid_medium.png • asteroid_small.png
Sound Effects: A laser.wav file for the shooting sound

I sourced these assets from free game asset websites like:
• OpenGameArt.org
• Kenney.nl (which has excellent space game assets)
• Itch.io (free assets section)

Key Features of the Game

The final game includes several features that make it engaging:

Progressive Difficulty: As your score increases, the game levels up, and asteroids appear more frequently
Asteroid Splitting: When shot, larger asteroids split into smaller ones
Lives System: Players have multiple lives before game over
Invulnerability Period: Brief invulnerability after being hit
Visual Effects: Stars moving in the background for a space atmosphere
Score Tracking: Points awarded based on asteroid size

Lessons Learned

Working with Amazon Q to develop this game taught me several valuable lessons:

AI-Assisted Development: Amazon Q can significantly speed up development by providing complete, working code examples.
Iterative Improvement: Starting with a basic version and enhancing it step by step worked well.
Asset Integration: Adding real images greatly improved the visual appeal with minimal code changes.
Prompt Engineering: Being specific in my requests to Amazon Q yielded better results.

Conclusion

Creating an Asteroids game with Amazon Q was a fascinating experience that demonstrated how AI can assist in game development. A combination of Amazon Q's code generation capabilities and my creative direction resulted in a fun, playable game that I'm proud to submit for the challenge.

The most impressive aspect was how quickly I could iterate on the game design, from basic shapes to a visually appealing game with real assets, all through conversational prompts with Amazon Q.

If you're interested in game development but find coding challenging, or if you're an experienced developer looking to speed up your workflow, I highly recommend giving Amazon Q a try. It's like having a knowledgeable programming partner who's always ready to help.

This project demonstrates the potential of AI-assisted development tools like Amazon Q to democratize game creation and make coding more accessible to everyone.

Email Verifier using Go

Mohammad Quanit — Sun, 12 Jan 2025 08:23:20 +0000

Hello everyone, it's been quite some time since I wrote a tech blog, so I thought I should share something that I've done in my company. There was a requirement where I had to do some verification checks on email and I was using Go on that project, so in this blog, I'll be sharing how I did that and also for you guys to know how email verification and its internals work.
I am sharing here it as a mini project so you can follow what I am doing and share feedback if you like.

Building an email verifier tool in Go involves several components and considerations.

I am not going to deep dive into all of that but covering some of them to get you an understanding of the email verification process.

The first thing that we need to verify is a domain that we were getting from some input e.g. google.com.

Steps to start your project in Go in the terminal/cmd:

1. go mod init github.com/username/email-verifier

2. touch main.go

Starting the implementation in our main.go file.

package main

import (
    "bufio"
    "log"
    "net"
    "os"
    "strings"
)

func main() {
    scanner := bufio.NewScanner(os.Stdin)
    for scanner.Scan() {
        verifyDomain(scanner.Text())
    }

    if err := scanner.Err(); err != nil {
        log.Fatal("Error: could not read from input %v\n", err)
    }
}

In the above code snippet, the NewScanner function will take the domain as input from the terminal or cmd from the user. Then the for loop will continuously scan the very next input after every time the verifyDomain function is invoked. And if there is some error it will be printed on the console as simple as that.

Now, the fun part will be covered in the verifyDomain function, where every call happens to the specific domain through HTTP.

func verifyDomain(domain string) {
   var hasMX, hasSPF, hasDMARC bool
   var spfRecord, dmarcRecord string
}

When we verify email through its internals, we need some components to check whether it is valid. Those components are records and protocols used to manage and secure email delivery. Here's a breakdown:

MX Record

An MX (mail exchange) record specifies mail servers responsible for receiving emails on behalf of a domain. When someone sends an email to user@example.com, the sender's mail server queries the DNS to find the MX records for example.com.

Here's how to look at MX records using Go,

      // MX record
    mxRecords, err := net.LookupMX(domain)
    if err != nil {
        log.Printf("Error: could not find MX record for %s due to %v\n", domain, err)
    }
    if len(mxRecords) > 0 {
        hasMX = true
    }

SPF

SPF (Sender Policy Framework) is an email authentication method that specifies which mail servers are authorized to send emails on behalf of a domain. When an email is received, the recipient's server checks the sender's IP against the domain's SPF record to verify the email is legitimate. Checking SPF can help identify spoofed emails.

SPF example:

v=spf1 ip4:192.0.2.0/24 include:_spf.google.com -all

v=spf1: Indicates the SPF version.
ip4:192.0.2.0/24: Specifies allowed IP ranges.
include:_spf.google.com: Includes Google's SPF records.
-all: Reject emails from unauthorized sources.

Here's how to look at SPF using Go,

 // SPF record
    txtRecords, err := net.LookupTXT("spf." + domain)
    if err != nil {
        log.Printf("Error: could not find SPF record for %s due to %v\n", domain, err)
    }

    for _, record := range txtRecords {
        if strings.HasPrefix(record, "v=spf1") {
            hasSPF = true
            spfRecord = record
            break
        }
    }

DMARC

DMARC (Domain-based Message Authentication, Reporting, and Conformance) builds on SPF and DKIM (DomainKeys Identified Mail) to provide additional email authentication and reporting capabilities. It specifies how to handle emails that fail SPF or DKIM checks (e.g., reject, quarantine, or do nothing). The domain owner publishes a DMARC policy as a DNS TXT record. DMARC helps assess the security posture of a domain.

DMARC. example:

v=DMARC1; p=reject; rua=mailto:dmarc-reports@example.com;

v=DMARC1: Indicates the DMARC version.
p=reject: Reject emails that fail authentication.
rua: Email address for aggregate reports.

Here's how to look at DMARC using Go,

// DMARC record
    dmarcRecords, err := net.LookupTXT("_dmarc." + domain)
    if err != nil {
        log.Printf("Error: could not find DMARC record for %s due to %v\n", domain, err)
    }

    for _, record := range dmarcRecords {
        if strings.HasPrefix(record, "v=DMARC1") {
            hasDMARC = true
            dmarcRecord = record
            break
        }
    }

Below is the complete code for verifying email in Go.

package main

import (
    "bufio"
    "log"
    "net"
    "os"
    "strings"
)

func verifyDomain(domain string) {
    var hasMX, hasSPF, hasDMARC bool
    var spfRecord, dmarcRecord string

    // MX record
    mxRecords, err := net.LookupMX(domain)
    if err != nil {
        log.Printf("Error: could not find MX record for %s due to %v\n", domain, err)
    }
    if len(mxRecords) > 0 {
        hasMX = true
    }

    // SPF record
    txtRecords, err := net.LookupTXT("spf." + domain)
    if err != nil {
        log.Printf("Error: could not find SPF record for %s due to %v\n", domain, err)
    }

    for _, record := range txtRecords {
        if strings.HasPrefix(record, "v=spf1") {
            hasSPF = true
            spfRecord = record
            break
        }
    }

    // DMARC record
    dmarcRecords, err := net.LookupTXT("_dmarc." + domain)
    if err != nil {
        log.Printf("Error: could not find DMARC record for %s due to %v\n", domain, err)
    }

    for _, record := range dmarcRecords {
        if strings.HasPrefix(record, "v=DMARC1") {
            hasDMARC = true
            dmarcRecord = record
            break
        }
    }

    log.Printf("Domain: %v,\n MX: %v,\n SPF:  %v,\n DMARC:  %v,\n SPF Rec: %v,\n DMARC Rec %v,\n\n", domain, hasMX, hasSPF, hasDMARC, spfRecord, dmarcRecord)
}

func main() {
    scanner := bufio.NewScanner(os.Stdin)
    for scanner.Scan() {
        verifyDomain(scanner.Text())
    }

    if err := scanner.Err(); err != nil {
        log.Fatalf("Error: could not read from input %v\n", err)
    }
}

When you run go run main.go in the terminal, you will provide a domain name e.g. google.com or resend.com, it will find the records using the HTTP library that we used and return the response.

Example Response:

go run main.go
google.com
2025/01/12 13:15:26 Error: could not find SPF record for google.com due to lookup spf.google.com on 192.168.1.1:53: no such host
2025/01/12 13:15:26 Domain: google.com,
 MX: true,
 SPF:  false,
 DMARC:  true,
 SPF Rec: ,
 DMARC Rec v=DMARC1; p=reject; rua=mailto:mailauth-reports@google.com,

if you like this article, please like and share it with your Gopher friends and follow me on Linkedin, Github, Twitter/X.

Peace ✌🏻

AWS Local Zones: Enabling Low Latency Infrastructure Workloads

Mohammad Quanit — Sat, 20 Jul 2024 13:15:36 +0000

Many cloud customers have applications that require single-digit millisecond latency for end users, such as real-time gaming, financial trading platforms, healthcare diagnostics, live media broadcasting, and AR/VR. When an AWS Region isn't close enough to meet these latency requirements, Customers must provision their infrastructure and use different APIs and tools to build their applications for low latency.

That is where AWS Local Zones comes into play. This article explores the concept of AWS Local Zones, their benefits, and real-world use cases that demonstrate their value in modern cloud architecture.

AWS Local Zone

AWS Local Zones is a specialized infrastructure deployment option offered by Amazon Web Services (AWS). These zones are strategically located closer to end-users, allowing for ultra-low latency and the ability to support workloads that require rapid response times. By placing core services such as Compute, Storage, Network, and other selected services in these geographically dispersed locations, AWS enables customers to deploy applications that demand single-digit millisecond latencies for seamless user experiences, as well as to support on-premises data centers where proximity is crucial.

You might be thinking what about the AWS Regions and Zones and how does it fit with AWS Local Zones? Well, AWS Local Zones extend AWS regions (parent region) and are located close to a large population, industry, and IT centers. It then connects to their parent region via Amazon’s redundant and high-bandwidth network giving applications running in AWS Local Zones access to the rest of AWS services.

Below is the Local Zone Workload visualization from official AWS documentation.

How Local Zones Work

A Local Zone is an extension of an AWS Region that is located close to your users geographically. Local Zones have their own internet connections and support AWS Direct Connect. This allows resources created in a Local Zone to serve applications that require low latency.

To use a Local Zone, you must first enable it. Next, you create a subnet in the Local Zone. Finally, you launch resources in the Local Zone subnet. For more detailed instructions, see Getting Started with AWS Local Zones.

The following diagram illustrates an account with a VPC in the AWS Region us-west-2 that is extended to the Local Zone us-west-2-lax-1. Each zone in the VPC has one subnet, and each subnet has one EC2 instance.

Benefits of AWS Local Zones

Low Latency: By bringing AWS services closer to the user, Local Zones reduce latency, providing faster response times and improved user experience.
Hybrid Cloud Flexibility: Local Zones support hybrid cloud architectures, enabling seamless integration with on-premises infrastructure.
Data Residency Compliance: Helps meet local data residency requirements by keeping data within specific geographic boundaries.
Disaster Recovery: Provides additional options for disaster recovery and business continuity planning.

AWS Local Zones Use-Cases

Media Production and Live Broadcasting
A media company can use AWS Local Zones to host live video streaming applications. By processing live video streams in Local Zones, the company can achieve lower latency, ensuring high-quality, real-time broadcasting for viewers.
Real-Time Multiplayer Gaming
A gaming company can deploy game servers in Local Zones to provide low-latency connections for players in specific geographic regions. This ensures a smooth and responsive gaming experience, crucial for maintaining player engagement and satisfaction.
Healthcare Imaging and Diagnostics
A healthcare provider can use Local Zones to process medical images (e.g., X-rays, MRIs) close to the point of generation. This reduces the time required for analysis and enables faster diagnosis, enhancing patient care.
Financial Trading Platforms
A financial services firm can deploy trading applications in Local Zones to minimize latency in transaction processing. This ensures faster trade execution and helps the firm meet regulatory requirements for data residency and security.
Augmented and Virtual Reality (AR/VR)
An AR/VR company can use Local Zones to host the backend infrastructure for immersive experiences. Low-latency processing is critical for AR/VR applications to ensure a seamless and responsive user experience.

Getting Started with Local Zone

To get started with AWS Local Zones, you must first enable a Local Zone through the Amazon EC2 console or the AWS CLI. Next, create a subnet in a VPC in the parent Region, specifying the Local Zone when you create it. Finally, create AWS resources in the Local Zone subnet.

Local Zone from EC2 in my AWS console

Enabling Local Zone in my region us-east-1 and it will be in US East Atlanta.

Now I am going to set up a Local Zone VPC subnet. When you add a subnet, you just need to specify a range of IP addresses for that subnet. You can also choose to include a different range of IP addresses specifically for IPv6 if that's something your network uses. You get to decide which Local Zone this subnet belongs to, and you can have several subnets all in the same Local Zone if you need to.

Below are the details for Subnet in our Local Zone. As you can see I selected Availability Zone to our newly enabled Atlanta Local Zone

After creating a subnet for your local zone, Create a resource in your Local Zone subnet. I'm going to deploy the EC2 server in that local zone within this newly created subnet named lz-subnet.

Now I am not going to show you how to create an EC2 as it is not the scope of this article but make sure to edit Network settings,

Select your VPC.
Select your Local Zone subnet.
Enable or disable Auto-assign public IP.
Create a security group or select an existing one.

After creating an EC2, you should see these configurations for your EC2.

Notice, that I chose the c6i.large instance type because to Atlanta local zone doesn't support t2 series machines. Read here about Atlanta Local Zone machine support.

Clean up

When you are finished doing hands-on with Local Zone, delete the resources in the Local Zone. Then contact AWS Support to disable it.

Summary

AWS Local Zones provide specialized infrastructure that brings AWS services closer to end-users, enabling low-latency and high-performance workloads. By leveraging Local Zones, businesses can enhance their applications' performance, comply with data residency requirements, and improve overall user experience. Whether it's media production, gaming, healthcare, financial services, or AR/VR, AWS Local Zones offers versatile solutions for various industry needs.

If you learn something new, share this blog with AWS fellas and network, and do follow me on my socials:
Linkedin, Github, Twitter.

Peace ✌🏻

Trace & Observe Modern Apps using AWS X-Ray

Mohammad Quanit — Fri, 08 Mar 2024 10:44:12 +0000

Hi fellas, In this blog I am going to share my experience using one of the coolest AWS services named AWS X-Ray. AWS X-Ray is a fully managed monitoring and observability service that helps you collect data about requests that the application serves. It provides tools that enable you to filter, view, and gain insights into the collected data, helping you identify optimization opportunities and issues.

AWS X-Ray is a service that can help you figure out what's going on across all your systems. It lets you see how requests are routed through different service touchpoints and gives you a good idea of how your applications are performing. You can use it to monitor performance, identify bottlenecks, and troubleshoot errors. With AWS X-Ray, you can keep an eye on your systems and make sure everything's running smoothly. It allows you to visualize complex and detailed service relationships within highly distributed applications. You can trace message pathways and call stacks at any scale.

Below is the AWS X-Ray workflow image from AWS's official blog.

AWS X-Ray has been designed to work seamlessly with distributed systems. Over the last decade or two, as complex distributed systems have emerged, debugging has changed and has taken on a new meaning. Engineers can now analyze and debug applications, audit their applications securely, and compile data from AWS resources to determine bottlenecks in cloud architecture and improve application performance.

Now let's see the implementation of AWS X-Ray with a nodejs application. AWS X-Ray provides an SDK that can be imported and utilized within your application.

const express = require('express');
const app = express();
const serviceName = "HELLO-MICROSERVICE"
const port = 8000;

// Require AWS X-Ray SDK
const AWSXRay = require('aws-xray-sdk');
AWSXRay.captureHTTPsGlobal(require('http'));

// Use AWS X-Ray middleware
app.use(AWSXRay.express.openSegment('MyApp'));

app.get('/hello', (req, res) => {
  const seg = xray.getSegment();
  seg.addAnnotation('hello-microservice', serviceName);
  seg.addMetadata("Request Meta", req);

  res.send('Hello AWS X-Ray!');
});

// Close the X-Ray segment for the current request
app.use(AWSXRay.express.closeSegment());

app.listen(port, () => {
  console.log(`App listening at http://localhost:${port}`);
});

Make sure to install aws-xray-sdk before, when setting up instrumentation in nodejs app.

X-Ray Daemon

The AWS X-Ray daemon is a core software application that listens for traffic on UDP port 2000. It gathers raw segment data and relays it to the AWS X-Ray API. The daemon needs to work in conjunction with the AWS X-Ray SDKs and must be running so that the data sent by the SDKs can reach the X-Ray service. The X-Ray daemon is an open-source project that you can follow on GitHub. See more details of the AWS X-Ray daemon here.

Segments

AWS X-Ray receives service data in segments. XRay then groups segments that have a common request into traces. The resources running your application logic send data about their work as segments. Technically, it's an object that contains some metadata about the request including,

The host - a hostname, alias, or IP address
The request – method, client address, path, user agent
The response – status, content
The work done – start and end times, subsegments
Issues that occur – errors, faults, and exceptions, including automatic capture of exception stacks.

Subsegments

To better monitor the work done by your application, you can use subsegments to break down the data into smaller pieces. Subsegments provide detailed timing information about downstream calls made by your application to complete the original request. They also contain additional details about calls to external services, such as AWS, HTTP APIs, or SQL databases. Furthermore, you can define custom subsegments to instrument-specific functions or lines of code within your application.

Traces

AWS X-Ray can help you trace requests as they move across various services. It captures important information about the path, duration, and performance of each request. A unique ID, called a Trace ID, is used to track the path of a request through your application. A trace is essentially a collection of all the segments generated by a single request. This request is usually an HTTP GET or POST request that passes through a load balancer, hits your application code, and generates downstream calls to other AWS services or external web APIs.

A trace ID and a sampling decision are added to HTTP requests in tracing headers named X-Amzn-Trace-Id. The first X-Ray-integrated service that receives the request adds the tracing header, which is then read by the X-Ray SDK and included in the response.

X-Amzn-Trace-Id: Root=1-5759e988-bd862e3fe1be46a994272793;Sampled=1

Service Graph

AWS X-Ray sends data about your app's services to create a graph that shows all the resources and services your app consists of. This graph is a JSON document that contains important information about your app's components. By using this graph, X-Ray can create a visual map of your app's pieces. This map helps you see how everything works together.

X-Ray Cost

AWS X-Ray doesn't charge upfront fees or commitment. you only pay for what you use based on the number of traces, and segments recorded and retrieved. For the Free tier,

The first 100,000 traces recorded each month are free.
The first 1,000,000 traces retrieved or scanned each month are free.

After the free tier or specific usage in the free tier, this is how this service charges you, which I think is not that expensive.

Traces recorded cost $5.00 per 1 million traces recorded ($0.000005 per trace)
Traces retrieved cost $0.50 per 1 million traces retrieved ($0.0000005 per trace).
Traces scanned cost $0.50 per 1 million traces scanned ($0.0000005 per trace).
X-Ray Insights traces stored costs $1.00 per million traces recorded ($0.000001 per trace).

AWS X-Ray enables customers to choose their sampling rate. Customers considering AWS X-Ray may want to estimate their costs for recorded traces by multiplying their request or API call rate by the chosen sampling rate. Read more here about the specific regions and their costs.

Earlier I mentioned that this service works smoothly with distributed systems or microservices in general. This is a microservices-based project on nodejs that is available on GitHub that is provided by CloudAcademy. You can clone and do hands-on with X-Ray in a real microservices-based application.

Summary

Let's recap what we have learned in this article. We learned what AWS X-Ray is and how we can use it with our applications irrespective of its architecture. AWS X-Ray is a service that helps you monitor and optimize your applications. It tracks requests, identifies issues, and provides insights into complex service relationships. You can trace message pathways and call stacks at scale, ensuring everything is running smoothly.

Resources

Here are the resources to get you started with AWS X-Ray.

AWS X-Ray hands-on workshop
AWS X-Ray with Nodejs
AWS X-Ray with Go
AWS X-Ray troubleshooting
AWS X-Ray Costing

if you like this article, please like and share it with your cloud friends and follow me on Linkedin, Github, Twitter.

Peace ✌🏻

Navigate Your Containerized Apps to Success with AWS Copilot

Mohammad Quanit — Mon, 05 Feb 2024 14:45:13 +0000

Hello Engineers, In this article I am going to explore a CLI tool that will be a game changer for containerized applications and make it easy to deploy on Amazon Web Services. Recently I talked about AWS Copilot at the AWS Hungary Conference and shared my insights on this technology developed by the AWS Containers team.

I will be doing hands-on on this, so you can practice within your environment as you are reading this.

What is AWS Copilot

AWS Copilot is a Command Line Interface (CLI) tool that simplifies the process of deploying and operating containerized applications on AWS. It automates the creation of necessary infrastructure for running applications, such as load balancers, Amazon Elastic Container Registry, Amazon Elastic Container Service, and IAM roles. Copilot also manages your applications' deployment and operational tasks, allowing you to focus on writing code. With AWS Copilot, you can easily create, deploy, and manage your containerized applications on AWS.

AWS Copilot handles infrastructure provisioning and management for you where you don't manage EC2 instances. You define your application using simple YAML manifest files Copilot automatically creates services, load balancers, pipelines, etc.

Why AWS Copilot

Before AWS Copilot, To deploy and work with containerized applications engineers used AWS Elastic Container Service (ECS) which is a highly scalable and efficient container management service provided by Amazon Web Services (AWS). It allows you to run, manage, and scale containerized applications on a cluster of virtual machines within the AWS ecosystem. You have full control over infrastructure along with managing EC2 instances and registering them with the ECS cluster. You define task definitions that describe your containerized applications and might need to create services, load balancers, etc manually. Copilot makes it super easy to set up and deploy your containers on AWS - but getting started is only the first step of the journey.

You now understand the AWS copilot and its benefits, let's make our hands dirty by doing hands-on

Install AWS Copilot

To start playing with AWS Copilot, you first need to install it on your local machine. It ships as a separate package or software to one command install.

Below are commands to install it on different platforms

Linux:

curl -Lo copilot https://github.com/aws/copilot-cli/releases/latest/download/copilot-linux

chmod +x copilot && sudo mv copilot /usr/local/bin/copilot && copilot --help

MacOS:

curl -Lo copilot https://github.com/aws/copilot-cli/releases/latest/download/copilot-darwin

chmod +x copilot && sudo mv copilot /usr/local/bin/copilot && copilot --help

or you can use the brew package manager to install:

brew install aws/tap/copilot-cli

Initialized AWS Copilot for the Project

You can deploy any project on ECS by the copilot. If you don't have any project right now, here's a repo where you can practice all the stuff we are about to learn. After Installing the AWS Copilot and cloning the project if you don't have one, Go to the project that you want to deploy, and within your code directory run:

copilot init

The next thing we will do is answer a few questions from Copilot. Copilot will use these questions to help us choose the best AWS infrastructure for your service.

Once Copilot finishes setting up the infrastructure to manage your app, you’ll be asked if you want to deploy your service to a test environment type yes.

Now we can wait a few minutes ⏳ while Copilot sets up all the resources needed to run your service. After all the infrastructure for your service is set up, Copilot will build your image push it to Amazon ECR, and start deploying to Amazon ECS on AWS Fargate.

After your deployment completes, your service will be up and running and Copilot will print a link to the URL 🎉!

Copilot will handle the deployment process, creating AWS resources, and configuring the necessary infrastructure. Under the hood, Copilot bootstraps these AWS resources when setting up your application.

ECS Cluster
ECS Service
Fargate Tasks
ECR Repo
VPC
Subnets
Security Groups
Load Balancer

After the deployment of our application using Copilot, it creates a cluster on ECS where you can see or manage it from the AWS Console.

This also creates a Repository on ECR with the image provided from the project's docker file.

When working with AWS Copilot, there are some concepts that we should be aware of. Let's look into this one by one.

Manifest Files

After setting up and deploying your copilot application, in your project, you will see a new folder will be created named copilot contains some manifest files for the application services and environments. This manifest.yml file contains metadata that describes a service's, a job's, or an environment's architecture as infrastructure-as-code. It is a file generated from copilot init, copilot svc init, copilot job init, or copilot env init that gets converted to an AWS CloudFormation template. Manifest files are always stored under copilot//manifest.yml.

Application

An application refers to a top-level group composed of services, environments, and pipelines that are related to one another. It enables users to organize their services into an application, irrespective of whether it is composed of a single service that performs all functions or a constellation of microservices. The tool categorizes the services and environments into which they can be deployed, thereby creating an efficient and structured approach to service management.

You can see your deployed app details using the command:

copilot app show

You will see all your app details on the terminal.

Below are additional Copilot App Commands that are self-explanatory:

Environments

When you work with applications, you might need to create different versions of a service for different environments. For example, you might want to create a development environment (dev) and a production environment (prod). To make this process easier, you can use the copilot init command. This command allows you to create a test environment that contains all the AWS resources needed to provision a secure network (VPC, subnets, security groups, etc.), as well as other resources that can be shared between multiple services. These resources include an Application Load Balancer or an ECS Cluster. When you deploy your service into your test environment, it will use the test environment's network and resources. Your application can have multiple environments, and each one will have its networking and shared resources infrastructure.

In the above image, you can see the environment for the application we deployed earlier named Stage. We can have multiple environments according to our scenarios.

Let's create another environment for our app and name it prod.

copilot env init

After running this command, you'll see logs something like the below and you can verify your env by checking into this folder copilot/environments/prod/manifest.yml.

Service

Service is your actual application code and infrastructure
needed to get it up and running on AWS. Remember when we got asked by Copilot after running copilot init about our service name and type where we selected Load Balanced Web Service? AWS has some pre-defined services to cater to almost every use case. As for our application, we wanted to deploy it on Amazon Load-Balancer so we selected that service type.

Public Facing Internet Service

If you want your service to serve public internet traffic, you have three options:

Request-Driven Web Service - will provision an AWS App Runner Service to run your service.
Static Site - will provision a dedicated CloudFront distribution and S3 bucket for your static website.
Load Balanced Web Service - will provision an Application Load Balancer, a Network Load Balancer, or both, along with security groups, and an ECS service on Fargate to run your service.

Backend Service

If you require a service that is only accessible from within your application and cannot be accessed externally, you can create a Backend Service. Copilot will set up an ECS Service that runs on AWS Fargate, but it won't configure any endpoints that are accessible from the internet.

Worker Service

Worker Service enables your application's microservices to communicate with each other asynchronously using a pub/sub architecture. By publishing events to Amazon SNS topics, your microservices can share information with a Worker Service that consumes these events. This allows for a more efficient and scalable application architecture, where each microservice can focus on its specific task without being burdened by the need to directly communicate with other services. With Worker Services, your application can achieve high performance and reliability while maintaining flexibility and modularity.

Run this command to see what services are we using for our app:

copilot svc show

Job

Amazon Elastic Container Service (ECS) provides Jobs, which are tasks triggered by an event. Copilot simplifies creating and deploying applications on ECS. Copilot supports only Scheduled Jobs, which run at a fixed rate or schedule.

Select the app and job type, then define the schedule or rate. Copilot automatically creates AWS resources and deploys the task definition to ECS.

Command to create a job:

copilot job init

or to list existing jobs:

copilot job ls

Currently, we don't have a job created with our environment or application, you might see a blank list.

Copilot Jobs offers logs that display the most recent activity of your job. You can track your logs in real time using the --follow flag. This will show logs from any new instances of your job after you execute the command.

copilot job logs

Sample Logs:

copilot/myjob/37236ed Doing some work
copilot/myjob/37236ed Did some work
copilot/myjob/37236ed Exited...
copilot/myjob/123e300 Doing some work
copilot/myjob/123e300 Did some work
copilot/myjob/123e300 Did some additional work
copilot/myjob/123e300 Exited

Now after doing all of the above stuff, I recommend you delete all of your resources in AWS so that you don't get surprised at the end of the month.

For that just run the command:

copilot app delete

It will remove all the resources that have been created by Copilot along with all the services, stacks, environments, applications, etc.

If you like this article, do like, comment, share, and let me know if you have created something with AWS Copilot. And don't forget to follow me on Linkedin, Github, Twitter.

Peace ✌🏻

Testing Approaches in Microservices Using Go

Mohammad Quanit — Sat, 20 Jan 2024 07:48:10 +0000

Every application needs quality testing strategies to function properly and ensure resiliency and reliability for its infrastructure. Microservices are typically small independent services, as a result, they need to be tested independently and specifically for the feature they implement.

To ensure that microservices written in Go are reliable and functional, a thorough testing strategy must be developed, including unit, integration, and end-to-end testing. It's also important to consider different aspects of microservices architecture.

By implementing testing strategies and best practices, we can improve the testing process and ensure that the microservices are working as well as they can. Let's discuss some testing strategies and best practices for microservices written in Go.

Unit testing

Testing a single unit or small piece of code is called Unit Test. As we know, microservices are small independent, and isolated services that are supposed to do one single operation, engineers must write test cases for those services. Ideally, a Unit test is quite voluminous and is internal to the microservice. It should be an automated process and depends on the development framework within the service.

Unlike monolith where a whole application is combined in a single unit, Microservices take one service as a feature or application and it is easier to test them in unit testing. Since there will be a separate microservice single business function, developers and quality engineers can achieve the utmost accuracy in software along with massive cost reduction compared to reworking and buggy applications.

Unit testing in Go

To ensure that your Go microservices are functioning correctly, start by creating unit tests for each function and method. The Go testing framework can be used to generate test cases and assertions. If a function interacts with external services or has dependencies, it's recommended to use mocking libraries or create mock objects to isolate the unit being tested. It's important to cover edge cases, error handling, and boundary conditions in the unit tests to ensure your microservices are robust.

Below are some popular libraries for testing and mocking in Go:

gomock - Mocking framework for the Go programming language.
assert - Basic Assertion Library used alongside native go testing, with building blocks for custom assertions.
gomega - Rspec like matcher/assertion library.
Testify - Toolkit for mocks, and assertions.
ginkgo - BDD (Behavior Driven Development) Testing Framework for Go.

Below is a simple example of unit testing in Go via the native testing package:

package main

import (
    "testing"
)

type example struct {
    flag    bool
    counter int
    pi      float64
}

func TestExampleStructCreation(t *testing.T) {
    s1 := example{
        flag:    true,
        counter: 10,
        pi:      3.141592,
    }

    if s1.flag != true {
        t.Error("Expected flag to be true")
    }

    if s1.counter != 10 {
        t.Errorf("Expected counter to be 10, but got %d", s1.counter)
    }

    if s1.pi != 3.141592 {
        t.Errorf("Expected pi to be 3.141592, but got %f", s1.pi)
    }
}

Integration testing

As a software developer, it's important to test your microservices to ensure that they function properly within your workflow. While unit testing is a helpful way to check the individual functionality of each microservice, it's not enough to guarantee that all services will work together seamlessly. Therefore, it's crucial to test the connected services to ensure a smooth working flow. Integration tests validate that independently developed services work smoothly when connected.

Integration testing in Go

Write integration tests to ensure proper communication and interaction between microservices through their APIs or service interfaces. Use test databases for microservices that interact with databases to avoid affecting production data during integration tests.

Below are some popular libraries for integration testing in Go:

gnomock - integration testing with real dependencies (database, cache, even Kubernetes or AWS) running in Docker, without mocks.
go-hit - Hit is an HTTP integration test framework written in Golang.
go-mysql-test-container - Golang MySQL test-container to help with MySQL integration testing.

End-to-end testing

End-to-end or E2E testing is one the most important strategies to test your microservices workflow from start to end to verify the user's journey. These tests can be automated and only done for ultra-critical flows. Unlike, unit or integration testing, it is quite hard to do testing E2E frequently because it requires all microservices to spin up which could be difficult to maintain and automate. As a result, it’s reserved for testing only critical interactions between specific microservices.

When it comes to microservices, problems can arise at various levels, making it a complex system. Even if each service has been thoroughly unit tested, if they cannot communicate with each other, it will not meet the user's expectations. To ensure realistic testing, it's important to create dedicated test environments that closely resemble the production environment. Additionally, implementing end-to-end tests that can be automated and added to your CI/CD pipeline will allow for continuous validation. The more you automate these test flows, the better it will be to manage and fix errors without shipping them into production.

End-to-end testing in Go

For thorough testing of the Go microservices stack, it is crucial to carry out end-to-end tests that cover all the interactions between the microservices. It is highly recommended to establish dedicated test environments that closely resemble the production environment to ensure accurate testing. These end-to-end tests should be automated and integrated into the CI/CD pipeline to facilitate continuous validation. By following these best practices, you can ensure the reliability and efficiency of your microservices stack.

Below are some popular libraries for integration testing in Go:

httpexpect - Concise, declarative, and easy-to-use end-to-end HTTP and REST API testing.
baloo - Expressive and versatile end-to-end HTTP API testing made easy.
endly - Declarative end-to-end functional testing.

Performance testing

Performance testing is a non-functional testing strategy to measure the speed, responsiveness, and stability of a system under a particular workload. In general, it focuses on infrastructure resources such as CPU, GPU, Memory, Network, etc. Performance testing is crucial when running an overwhelming amount of load on your services to find bottlenecks or inability. Most of the performance tests can be achieved by load/stress testing along with benchmarking.

Load or Stress Testing is a type of performance testing used to determine the system's behavior under normal and peak conditions. The goal is to make sure that the application works smoothly under heavy loads when many users access it at the same time. When it comes to testing services, focusing on a single one can be akin to testing the entire stack. While this can be a challenging endeavor that requires careful automation, it is ultimately worth the effort, particularly when dealing with resource-intensive or problematic services.

Performance testing in Go

In Go microservices, it is crucial to perform load testing to evaluate how well your microservices handle concurrent requests and high-traffic loads. In many program languages, a Benchmark is a commonly used tool for measuring the execution time of code. Use Go's built-in benchmarking tool provided by the testing package to measure the performance of critical parts of your code and identify bottlenecks. In the xx_test.go file, we just need to add the function name starting with Benchmark like func BenchmarkXX(). When running the command go test -bench=. , it will trigger the benchmark function.

Below are some popular tools for performance testing in Go:

mockingjay - Fake server, Consumer Driven Contracts, and help with testing performance.
Apache JMeter - Test performance both on static and dynamic resources, Web dynamic applications.
K6.io - A modern load-testing tool, using Go and JavaScript.

Security testing

Security is a crucial aspect of your infrastructure but most of the time engineers do not give real attention to it. The first step is to find the scope and boundaries when considering security testing within microservices. Microservice architecture has a lot of benefits such as scalability, resiliency, and flexibility but it also poses some challenges when it comes to security as each microservice may have its vulnerabilities, dependencies, and communication protocols. Developers and security engineers need to assess the risks and threats that microservices might contain. This requires security to be considered on every layer of your microservice application starting with infrastructure.

Threat Modeling is a process that defines the flow of the system or microservices and helps to identify all the points of attack that hackers could exploit. Engineers need to use tools like OWASP Threat Dragon, Microsoft Threat Modeling Tool, or NIST Cybersecurity Framework to conduct a systematic and structured risk assessment. OWASP's Top 10 security principles are a great way to assess vulnerabilities and risks within your microservices.

Security testing in Go

When working with microservices developed in Golang, it is no exception that we need to use security tools. We should use static analysis tools like GoSec to scan your codebase for security vulnerabilities. This tool inspects source code for security problems by scanning the Go AST. Security engineers also need to conduct penetration testing to identify vulnerabilities in your microservices, APIs, and endpoints. There are tools already available like Burp Suite, OWASP ZAP, or Nmap, to perform penetration testing, fuzzing, and injection attacks on the web application and its microservices

Below are some popular tools for security and testing in Go:

gosec - Security Checker that inspects source code for security problems by scanning the Go AST.
secret - Prevent your secrets from leaking into logs, std*, etc.
secureio - An keyexchanging+authenticating+encrypting wrapper and multiplexer for io.ReadWriteCloser based on XChaCha20-poly1305, ECDH and ED25519.
Coraza - Enterprise-ready, mod security, and OWASP CRS compatible WAF library

If you like my article, please like and share feedback and let me know in the comments. And don't forget to follow me on Linkedin, Github, Twitter.

Peace ✌🏻

Deployment approaches in Microservices.

Mohammad Quanit — Tue, 16 Jan 2024 08:55:23 +0000

Deploying monolith applications usually means running one or more servers of a usually large single chunk of application. The deployment of a monolith might not always be a straightforward process but it is much simpler than deploying microservices.

Microservice applications can consist of tens or hundreds of interconnected services written in a variety of different programming languages and frameworks. Each microservice is a mini-application with its own resources, scaling, and deployment and you need to run several instances of a single microservice to scale.

For example, Let's say you have an e-commerce application consisting of some microservices that could be Catalog, Cart, Search, Payment, etc. Now you need to deploy each of these services separately and each service can required to run on more than one instance to achieve scalability for that specific service.

Deployment of microservices written in Golang requires careful planning and consideration of various deployment strategies. These strategies help ensure that your microservices are reliable, scalable, and can be efficiently managed in a production environment. Here are some deployment strategies and practices for microservices with Golang:

Containerization

Containerization is a technique where you build, test, and deploy your application in an isolated manner without interfering with other services. Tools like Docker, LXD, and Podman are used for containerizing & deploying microservices. Each microservice is packaged as a lightweight container along with its dependencies, making it consistent and portable across different environments.

Docker is one of the most popular and widely used container engines that almost all organizations prefer to use. It is highly configurable and developer friendly which makes it an automatic choice for building containers. We are going to learn Docker or Containerization in general in detail in the next module.
Below is an example Dockerfile for Golang:

# syntax=docker/dockerfile:1

FROM golang:1.21

# Set destination for COPY
WORKDIR /app

# Download Go modules
COPY go.mod go.sum ./
RUN go mod download

# Copy the source code. Note the slash at the end, as explained in
# https://docs.docker.com/engine/reference/builder/#copy
COPY *.go ./

# Build
RUN CGO_ENABLED=0 GOOS=linux go build -o /web-server

# Optional:
# To bind to a TCP port, runtime parameters must be supplied to the docker command.
# But we can document in the Dockerfile what ports
# the application is going to listen on by default.
# https://docs.docker.com/engine/reference/builder/#expose
EXPOSE 8080

# Run
CMD ["/web-server"]

Orchestration

When working with containerization using docker or any other tool, we know that there are many microservices that we need to containerize and deploy. Still, the question is to deploy where and how to manage tens or hundreds of containers as we scale up. Orchestration is a technique that handles containers as many as you have in an automated way. Platforms like Kubernetes & Docker Swarm are widely used for managing containerized microservices.

Kubernetes or K8s in short is the most popular container orchestration tool created by Google that most organizations prefer. Docker Swarm is also used for managing containers that are introduced by Docker Inc. itself, lacks some of the features that Kubernetes has, but both these tools provide enough features like scaling, load balancing, service discovery, and rolling updates, etc.

Blue-Green Deployment

The Blue-Green deployments technique in general, is a pattern that applies to both monolith and microservices architecture that strategically involves running two identical environments for your infrastructure. The primary goal of this deployment is to minimize downtime when switching from "blue" (current) and "green" (new) infrastructure. When a new version of a microservice is ready, traffic is switched from blue to green, allowing for easy rollbacks if issues arise. The two environments need to be kept separate but still look as similar as possible. These environments can be made up of different hardware or virtual machines that could be on the same or different hardware.

Blue-Green Deployment improves high availability by keeping microservices available during development and deployment. There will be no downtime because there is already a similar version of your microservices running simultaneously along with the stable one that serves the incoming traffic, so if somehow the stable version crashes or gets unstable the other identical environment will handle traffic. Another benefit is that if the new version isn't working correctly, you can quickly roll back to the previous variation (the blue microservice). Microservices are constantly monitored to track if any issue arises, it should be reverted to the blue state. This technique is also known as Red Black Deployment, a newer term being used by Netflix, Istio, and other frameworks/platforms that support container orchestration This strategy or technique is subtly but powerfully different than Blue-Green Deployment.

The only difference between them is Blue-Green deployments, both versions can get incoming requests at the same time but in Red-Black deployments, only one of the versions is getting traffic at any point in time.

Canary Deployment

Canary Deployment is one of the most popular strategies for deploying the application infrastructure. Like Blue/Green deployment, this technique can also be used with monolith and microservice architectures. It is better to transition slowly from blue to green instead of doing it suddenly.
In canary deployment, engineers deploy the new features or changes gradually in stages and the goal is to show that new feature or change to a specific set of users. This includes releasing the new version of services to a small percentage of the load first and seeing if it works as expected. The canary deployment releases only a single microservice at a time and microservices with higher criticality and risks involved can be made available before others.
To ensure a microservice is thoroughly tested with real users before launch, engineers can use Canary Deployment. This method compares different service versions and reduces downtimes while improving availability. Detecting issues early on prevents critical microservices from being compromised and keeps the entire system safe.

Rolling Deployment

Rolling deployment updates microservices one at a time while keeping the others running. In this strategy, an application's new version periodically replaces the old version of the application & deployments happen over a while. Implementing rolling deployments in your application deployment process can significantly increase high availability and reduce the risk of service disruptions. With this approach, multiple environments are up almost all the time, ensuring that your users can access your application without any interruptions. As the newer version of the application takes up complete charge, the old version gets released, allowing for a seamless transition. This ensures that your users experience minimal downtime and can continue to use your application without any disruption.

Rolling deployments allow deployments incrementally which helps reduce the downtime and reliability of the application by reducing the risk of widespread failures. SREs and DevOps engineers gradually update the server and continuously monitor it, so if there is any issue arises it can be detected early and resolved before the whole system affected.
Rolling deployments are a useful way to simplify the process of fixing issues that may occur during deployment. This strategy updates the system incrementally, so if there are any problems, only the updated servers need to be rolled back instead of the entire system. This gives developers and administrators more control and flexibility to manage the system's integrity.

Serverless Deployment

Serverless unlike its name means application resources are deployed and hosted on some servers where engineers don't need to care about anything related to infrastructure. Platforms that the Cloud companies provide like AWS Lambda, Google Cloud Functions, and Azure Functions are serverless platforms that provide all the resources you need to run a microservice with a Pay-As-You-Go model. Serverless Microservices contain cloud functions that perform highly specific roles within an application. These cloud functions automatically scale based on demand with only paying as much as you use to run those functions.

Serverless Microservices contain serverless functions that are small blocks of code running in response to an incoming request to that microservice. We discussed that microservices are also small independent services that can be scaled and managed independently of one another, so how does serverless fit in this situation?

Just like we can run the microservices in a container platform like docker separately from each other, we can write a single function for each microservice that is running on some cloud vendor without having to manage any overhead. A single microservice can have multiple functions deployed at the same time. Cloud providers offer a valuable solution for developers by handling the infrastructure, allowing them to focus their efforts on coding. This enables a more efficient workflow and streamlined development process.

Automation & Security Considerations

Automation processes have become a crucial aspect of software delivery especially when building cloud-native applications. Automation in deployment is the process of automating the workflow from developing to testing & the deployment of every single microservice. This process is reliable and effective across SDLC (Software Development Lifecycle). The goal of making the deployment process automated is to eliminate the challenges of manual deployments and enhance the quality and pace of releasing microservices.

DevOps engineers usually manage all sorts of deployments in infrastructure and are responsible for fixing any issues that come up with deployments. There are tons of tools that help DevOps engineers set up automated processes using CI/CD (Continuous Integration/Continuous Deployment) pipelines to automate the deployment process. CI/CD tools like Jenkins, Travis CI, or GitLab CI/CD can help automate testing, building, and deploying microservices. Deployment automation can also help engineers get quick feedback because it is less error-prone and can be released at a higher frequency, so you get your feedback immediately.

When designing and working with microservices, security is another thing to consider on levels of the infrastructure. We discussed security testing in the last module, It is important to implement security best practices at every stage of deployment, including secure communication, access control, and vulnerability scanning. When considering secure communication, always try to use HTTPS for secure communication between services.

As an engineer who is responsible for the deployment, you should know about phishing and credential stuffing. But it's also important to be on the lookout for attacks that come from within your network. To ensure your network stays safe, it's a good idea to use HTTPS for your microservices architecture. When using dependencies create automated workflows to detect issues in dependencies that scan your codebase.

Snyk is one of the most popular tools to work with security stuff and helps you to find vulnerabilities in your not just codebase but infrastructure.

Career Switch from Software Engineering: (discussion)

Mohammad Quanit — Fri, 24 Nov 2023 11:14:34 +0000

Hello everyone, suppose someone has given many interviews for a software engineering job but has not been able to secure one. In such a situation, should they consider switching to other career options? If you have faced this situation, I would appreciate hearing what you did or what you think someone should do in such a scenario.

Crafting Microservices in Go: A Theoretical Guide

Mohammad Quanit — Thu, 09 Nov 2023 10:42:36 +0000

In this blog, we are covering the following topics:

Establish a system-design team
Define your architecture
Using an API gateway
Service discovery
Inter-Service communication
Implement data storage

One of the most important aspects of building microservices is designing the architecture. This involves deciding on the granularity of the services, defining their APIs, and implementing the services themselves. We will explore each of these steps in detail and provide examples of how they can be accomplished using Go.

Establish a system-design team

Creating a microservices system can involve many people and components, which can make it complicated. The final software is the result of everyone's input and decisions. However, making everything work together smoothly can be difficult. That's why it's crucial to have a team responsible for guiding the system's direction and behavior. This team is referred to as the system design team in this model. In this model, the system design team has three core responsibilities:

Design Team Structure
Establish Standards & Incentives
Continually Improve the System

Define your architecture

To design and build microservices, engineers or architects need to determine the boundaries and responsibilities of each microservice. Identify what communication protocols & APIs are going to be used within microservices or what data storage either open-source or commercial database solutions for each service. In the previous section, we learned that the Golang community has developed numerous frameworks and tools for microservices.

However, it is essential to ensure that the chosen library or framework addresses the current problem and does not create additional challenges in the future. Decide whether you want to use a web framework like Gin, Echo, or Buffalo, or if you prefer to build your microservices from scratch keep in mind frameworks can provide useful tools for handling HTTP requests, routing, and middleware, but they may add some overhead.

Using an API gateway

Most of the time when working with microservices, you need to decide how the application client will interact with microservices. With the monolithic application, there is just one set of endpoints that can be replicated with load balancing that distributes traffic among them. In a microservices architecture, however, each microservice exposes a set of what are typically fine-grained endpoints.

An API Gateway is an implementation that acts as a single entry point for your microservices. The API Gateway handles requests from clients and passes them to your backend services while sending the responses back to the client. It's a middle-level server that sits between the client/internet and your backend microservices. The API Gateway encapsulates the internal system architecture and provides an API that is tailored to each client. API Gateway can have many responsibilities such as authentication, authorization, security, monitoring, load-balancing, rate limiting, caching, request management, and static response handling. Below is the visualization of the API gateway in microservices.

The Golang community has provided many libraries and packages that help engineers write API Gateways without writing them from scratch. Some of them are part of complete microservices frameworks while some are introduced as standalone API Gateway services. Below are some of the API Gateway tools provided by Go Community.

Lura - Ultra-performant API Gateway framework with middlewares.
resgate - Realtime API Gateway for building REST, real-time, and RPC APIs, where all clients are synchronized seamlessly.
Janus - A lightweight API Gateway and Management Platform that enables you to control who accesses your API.
Echo middleware - Echo doesn't provide an API gateway specifically but its middleware does some of the work that an API Gateway intends to do.

The API Gateway is responsible for request routing and protocol translation. It provides each of the application’s clients with a custom API. The API Gateway can also mask failures in the backend services by returning cached or default data.

Service discovery

In a traditional monolith application, you could probably hardwire the locations, but in a modern micro-services architecture, finding the need for locations is a non-trivial problem. API Gateway needs to know the location (IP and Port) of each microservice to which it communicates. Application services have dynamically assigned locations because of autoscaling and upgrades.

Service Discovery in microservices is a way of locating other services on a network. Service discovery implementations include a central server and clients that connect to the central server. Microservices applications typically run on virtualized or containerized environments, and the instances can change their locations (IP and PORT) dynamically. So there should be a mechanism that determines which microservice to invoke when a request arrives and that's where Service Discovery comes into play. Service Discovery acts as a registry where all addresses of microservices are tracked and these instances have dynamically assigned network paths. Consequently, the API Gateway like any other service client in the system, needs to use the system's service discovery mechanism either server-side discovery or client-side discovery.

Client-Side discovery - In this discovery pattern, the client is responsible for figuring out the network locations on available instances and load-balancing requests across them. It queries the service registry (a database of available instances) and then uses a load-balancing algorithm to select one of the available service instances to request. Types of Client-Side Service discoveries are Netflix Eureka, Zookeeper, and Consul.
Server-Side discovery - In this discovery pattern, a dedicated load-balancer does all the job of load-balancing. Clients make requests via a router, which queries the service registry and forwards the request to an available instance. In fact, there's no need to write discovery logic separately for each language and framework that the Service Consumer uses. Types of Server-Side Service discoveries are NGNIX and AWS ELB

The Golang community provided tools for implementing service discovery when writing microservices. Here's how you can implement service discovery in Go:

Use Service-discovery tools - To simplify finding and managing services with Go, try using a service discovery platform like Consul, etcd, or Go-Kit. These tools ensure efficient and seamless operation of your Go apps in a distributed architecture.
Register services - When a Go service starts, it should register itself with the service discovery tool by providing its name, IP address, port, and any other relevant metadata. This registration typically happens during the service's initialization phase.
Discover services - When a Go service needs to communicate with another service, it queries the service discovery tool to obtain the location (IP address and port) of the target service and the service discovery tool returns this information to the requesting service.
Handle failures and retries - In your Go service, write logic to handle failures and retry mechanisms to handle cases where service discovery fails or returns errors.
Monitor & maintain - Implement health checks to ensure the reliability of services. Continuously monitor the performance of your service discovery solution.

Inter-Service communication

One of the biggest challenges when working with microservices are communication mechanism. Microservices are distributed by nature and they require service communication on the network level. Every microservice has an instance and process, therefore services interact using inter-service protocols such as HTTP, gRPC, and AMQP (message brokers).

Microservices has a couple of communication styles has determine the direction of interaction.

Synchronous - In synchronous communication, the client expects some response after sending a request from the server that might even be blocked while it waits. HTTP or gRPC are the communication protocols that follow the synchronous communication pattern.
Asynchronous - In asynchronous communication, the client sends a request but doesn't wait for a response it doesn't block while waiting for a response, and the response itself may not be sent immediately. AMQP (Advanced Message Queuing Protocol) is a popular communication protocol that follows an asynchronous communication pattern using a publisher/subscriber model.

Implement data storage

Data Storage is an integral part of storing your data permanently. There are several implementation strategies for data storage when designing microservices. The best strategy depends on a number of factors, such as the size and complexity of your application, the type of data you need to store, and your budget. Designing a database is one of the most challenging concerns for your microservices. There are two implementation strategies when designing the data storage solution.

Within microservices architectures, there are several options for implementing data storage patterns. To provide a better understanding, here are some examples of such patterns:

Data partitioning: This pattern divides the data into multiple partitions, each of which is stored in a separate database. This can be used to improve scalability and performance, as each microservice can access only the data that it needs.
Data replication: This pattern replicates the data across multiple databases. This can be used to improve availability and fault tolerance, as the data will still be available even if one of the databases fails.
Event sourcing: This pattern stores the history of all changes to the data. This can be used to reconstruct the data state at any point in time, and it can also be used to implement complex business logic.

Golang community provides a lot of databases, relational & non-relational database drivers, cache storage, query builders, and schema migration tools. Golang supports integration with cloud-native data stores such as Amazon RDS, AWS DynamoDB, Azure Cosmos DB, etc. Below are some community-driven data storage that can be utilized in Go microservices.

mongo-go-driver - Official MongoDB driver for the Go language.
redigo - Redigo is a Go client for the Redis database.
go-elasticsearch - Official Elasticsearch client for Go.
pq - Pure Go Postgres driver for SQL database.
dynago - Simplify working with AWS DynamoDB

If you like my article, please like and share feedback and let me know in the comments. And don't forget to follow me on Linkedin, Github, Twitter.

Peace ✌🏻

Understanding Cloud Native & its Architecture

Mohammad Quanit — Fri, 03 Nov 2023 13:33:33 +0000

Overview

The term Cloud-Native first appeared 10 years ago when Netflix discussed their web application architecture in AWS re: invent talk 2013. At that time, the meaning of Cloud-Native was likely different than it is today. However, there were no clear definitions of this term and it means different things to different people or organizations.

In this blog, we are going to understand, what Cloud-Native is & what are the promises of Cloud-Native.

Cloud-Native is a software architecture pattern, which means any kind of application architecture whether it is web, mobile, or desktop completely built on a Cloud platform natively that is more scalable, available & loosely coupled. Before the Cloud Computing era, organizations did their infrastructure on-premise. The IT or system admins have to do all the hardware setup and configure all the services that are required to run the application. The more hardware to manage, the greater will be the cost.

Note: Cloud-Native is a pattern or approach to building applications using Cloud platforms and services for high availability, scalability, and modern dynamic environments, making loosely coupled systems.

When the Cloud Computing Era boomed in the tech industry, Some notable organizations started adopting the cloud quickly. They designed their architecture on the cloud completely which promises increased agility to ship new features without compromising their availability making it quicker to respond to changing customer demands. All the big giant companies like Google, Amazon, and Microsoft have their own Cloud Platforms that empower organizations to build their application using their platform. So, investing time in Cloud Native will be a good idea.

When should we adopt the Cloud-Native approach?

The answer is It Depends. If the application is relatively small with monolith architecture a Cloud-Native approach may not be necessary. A simple deployment model may be sufficient. However, for larger and more complex applications, Cloud-Native can offer a wide range of benefits such as increased scalability, faster deployment cycles, and high availability. Ultimately, the decision to adopt a Cloud-Native strategy should be based on a careful evaluation of the application's requirements and the organization's resources.

You might have an understanding of Cloud-Native, we are going to cover, Cloud-Native architecture along with its benefits.

Cloud-Native architecture

Cloud-native technologies help organizations build and run scalable applications in dynamic environments such as public, private, and hybrid cloud. The cloud-native approach heavily relies on utilizing Cloud platforms, Containers, Microservices, DevOps, Immutable Infrastructure, and Service mesh.

In the past, it was very common to see applications poorly designed & utilized. Cloud-native architecture introduced much-needed wisdom on how to effectively design applications and infrastructure. The design patterns and practices created throughout the years of mistakes in evolution provided us with the solution of best practices that focus on Application availability, Cost management, Efficiency, and reliability. The approach in Cloud-Native must enable loosely coupled systems that are reliable, resilient, and highly available along with easier management and observability.

Cloud-native technology is built on 4 pillars that provide a strong foundation for its architecture.

Microservices
Container Orchestration
DevOps & Automation
CI/CD Pipelines

Microservices

Before Cloud-Native architecture design, traditional systems were developed as single centralized applications with tightly coupled services and components making them difficult to deploy. This tightly coupled behavior between different components can potentially affect the entire application even with a minor change.

Microservice Architecture is the fundamental pillar of Cloud-Native. They are small, focused & independent services which makes them easier to develop and deploy. When working with monolith applications, the application needs to be rebuilt and redeployed as a whole every time we do a single change which makes it less flexible. Services can't be developed with different technologies and due to its single source of truth, high availability & scalability are nearly impossible to achieve. However, tracking bugs and issues in a monolith is quite easy and communication is fairly simple. For large enterprises and complex systems, the monolith will never be a good option due to its limitations with flexibility.

That's where Microservices Architecture comes into play. It is a fine-grained architecture for your system where services are further divided or broken into smaller services as Task-level services. The philosophy that Microservice architecture follows is to Keep services small and decoupled and should do only one thing (Unix philosophy). That's why they are called Microservices.

Container orchestration

In Pre-containerization, apps were developed as single instances on a single machine that could consume resources of your hardware. Then Virtual Machines have been introduced to distribute the workload for different instances. Multiple instances of the application can run on a single computer by having multiple Virtual Machines. But VMs consume resources too, as they are full-fledged operating systems running on an existing operating system. To solve those issues, Containers have been introduced. They are lightweight processes that run the applications in an isolated environment irrespective of what version you have for that application runtime.

They enable developers to package their applications to ship, deploy, and run on any platform. Containers are the core of Cloud-Native. There are many container technologies or tools that exist right now but the most popular one right now is Docker. It is the most developer-friendly tool to work with your workloads. According to Stack Overflow’s 2023 Developer Survey docker has been the most-used tool in the developer's community.

Orchestration on the other hand is a technique used to manage multiple containers at once. Cloud Native follows the principle of microservice architecture, so when you work with containers for those services, it becomes really hard & less efficient to manage containers for these microservices manually. So to orchestrate or manage the containers we used tools like Kubernetes, Docker Swarm, Apache Mesos, etc. These tools help developers tackle the problem by automating the scalability, load balancing, availability, scheduling, deployment, and networking of containers.

Devops & automation

DevOps or Development Operations, is one of the go-to strategies to include in your cloud-native architecture. The term Dev and Ops combines practices that allow us to adopt a Cloud-Native environment to ensure that a company constantly delivers quality software. DevOps is a paradigm-shifting approach to creating software. It has in recent times become the defective method for building, testing, delivering, and managing software applications.

The core idea that DevOps embodies is that both development and operation teams should work closely together throughout the entire software life-cycle. From the initial development, right through to the installation, running, and maintenance of the application. This is an important cultural shift that needs to be embraced for successful DevOps adoption.

Cloud-Native emphasizes automating the workflows and by applying the principles of DevOps, organizations can achieve automation and ensure the quick delivery of the systems. DevOps practices promote collaboration between development and operations teams. Automation of tasks such as provisioning, configuration management, and monitoring accelerates the deployment process and enhances operational efficiency.

CI/CD pipelines

When implementing DevOps practices into your cloud-native applications, you create automated workflows to build, test, deliver, and deploy the system for your environment. To achieve those steps, DevOps engineers create CI/CD pipelines to automate workflows. CI stands for Continous Integration and CD stands for Continous Deployment. It connects the development and operations to automate building, testing, and deploying the application. This practice ensures rapid and reliable software delivery, fostering agility and innovation.

Conclusion

Cloud-native architecture is an approach or pattern to design, develop, and deploy your systems using the above pillars we discussed. These pillars collectively enable cloud-native applications to achieve qualities like scalability, resilience, flexibility, and speed. By adhering to these principles, organizations can take full advantage of cloud infrastructure, allowing them to innovate rapidly and deliver value to customers more efficiently.

If you like my article, please like and share feedback and let me know in the comments. And don't forget to follow me on Linkedin, Github, Twitter.

Peace ✌🏻

Docker Container Security - Practices to Consider

Mohammad Quanit — Thu, 13 Jul 2023 12:18:21 +0000

Hello Engineers, In this article, I am going to share what container security is, some practices and standards for container security, why you should care about container security, some tools that can help us to make our containers less vulnerable.

Although there is nothing like fully secured everything in I.T industry still engineers or security engineers are required to make things less vulnerable so that your app won't get hacked or some bad guy not try to access important information (that could be leaked from containers) and also because that's what they are paid for, right.

So before moving forward, we will be looking at a bit of docker architecture as we are learning security in the docker context.

Docker, the most popular open-source containerization tool also standard for most of the platforms of the container launched in 2013 by Docker Inc. Engineers can easily create, deploy, and run applications in a self-contained environment called a container. It quickly gained popularity among developers and system administrators because it simplified the process of deploying applications across different environments, such as development, testing, and production

Docker has a full-fledged Architecture that contains some specific components that are:

Docker Client: This is a command-line tool that allows users to interact with the Docker platform.
Docker Daemon: This is the background process that runs on the host machine and manages the containers.
Docker Images: An image is a lightweight, standalone, executable package that includes everything needed to run the software, including the code, libraries, and dependencies.
Docker Registry: This is a repository that stores Docker images.

The above details of Docker are essential to understand how docker works internally. But this article mostly covers related to security, so I won't go into further details of how docker actually works.

Container Security

If we talk about container security, whether we work on Docker, LXD, RKT, Apache Mesos or any other tools your organization uses, the principles will be the same for overall container security.

Container security refers to the practices and technologies used to protect containerized applications. It can be any kind of application either micro-services, SPAs (Single page applications), Utilities or API etc. Those practices are used to protect containers from unauthorized data access, malicious attacks, and other security threats.

Containers are a lightweight and portable way to package and deploy applications, but they also introduce new security risks that need to be addressed as containers share the host kernel and can be vulnerable to attacks if not properly secured.

Container Security Considerations

Host/Kernel Security: Containers share the host kernels, which means any vulnerabilities in kernel or host can affect all the containers running on that host. It is important to keep the host system secure by regularly updating security patches, using anti-malware software, implementing other host security measures, Run container security tools like docker-bench-security.

Access Control: Containers should be run with the least amount of privileges necessary to perform their tasks, and access to containers and their associated resources. Always run it as non-root user. It is best to create a new user to perform and access docker resources. Running your containers on rootless mode will verify that your application environment is safe.

Container Monitoring: Monitoring is important for container activity and log events to detect potential vulnerabilities and incidents, and to have processes in place for responding to security incidents and implementing remediation measures. Use tools like Docker Logging and Use Docker's health check feature to periodically check the status of containerized applications.

Image Vulnerability Scanning: Containers are created from images, which can contain software vulnerabilities, malware, or other security risks. It is important to use trusted and verified images, scan images for vulnerabilities, and follow best practices for image security. Periodic scanning allows you to keep your images updated and audit critical directories and files. Tools like Anchore, Clair, or Trivy can scan container images and provide vulnerability reports.

Docker Security Policies: Docker environment allows use to setup our security policies in order to make our containers secure. Only use Docker images from trusted and verified sources. Create policies to restrict image pulls to approved Docker registries and repositories. Enforce Container strict isolation to prevent malicious attacks. We can use Use container runtime options like --privileged=false and --cap-drop to limit container capabilities. Create policies to implement network segmentation and firewall rules to control container communication. We can use Use Docker's built-in network features or there are some other tools like cillium or docker-bench to isolate containers and define ingress/egress rules.

Conclusion

Container security is a critical consideration when using containerization technologies like Docker and Kubernetes. By implementing best practices for container security, organizations can reduce the risk of security incidents and protect their applications and data from unauthorized access and other security threats.

If you like my article, please like and share feedback and let me know in comments. And don't forget to follow me on Linkedin, Github, Twitter.

Peace ✌🏻

Test Driven Development & Go

Mohammad Quanit — Sun, 16 Apr 2023 11:33:11 +0000

In this article I'll be discussing about Test Driven Development, its approaches, best practices in the context of Golang with some code examples. Here's the github repo link from where I am going to show you some code examples

Test Driven Development in short (TDD) is a software development approach where you write test cases for a small piece of code before writing the actual code. The goal is to ensure that the code meets specific requirements and behaves correctly in various scenarios. By writing tests first, you can catch errors early in the development process, and ensure that your code is easy to test, maintain, and refactor.

TDD is a cycle of writing a test, seeing it fail, writing code to pass the test, and then refactoring the code to improve its quality.

Motivations for TDD

Helps catch errors early: By writing tests before writing the actual code, TDD can help catch errors early in the development process. This reduces the time spent on debugging and ensures that your code is more reliable.
Ensures code meets requirements: TDD ensures that your code meets specific requirements and behaves correctly in various scenarios. This makes it easier to maintain and refactor your code as your project evolves.
Improves code quality: TDD encourages developers to write modular, testable, and maintainable code. This improves the overall quality of the codebase and reduces technical debt.
Reduces the cost of change: Since TDD ensures that your code is well-tested and modular, it reduces the cost of making changes to your codebase. This is particularly useful in large projects where changes can have significant impacts on the system.
Enables faster development: TDD can actually help speed up development by reducing the time spent on debugging and ensuring that new code doesn't break existing functionality.
Facilitates collaboration: TDD provides a common language and framework for collaboration between developers and other stakeholders. This helps ensure that everyone is on the same page and that the project moves forward smoothly.
Boosts confidence: TDD gives developers confidence in their code by ensuring that it behaves correctly and meets specific requirements. This confidence can lead to better decisions, more creativity, and more efficient development.

Testing in Golang

Golang provides a built-in testing tool that automates the process of running tests. The tool, called "go test", is easy to use and can be run from the command line. The Go testing package provides a range of functions and methods for creating and running tests. It includes functions for comparing values, reporting errors, and running tests in parallel.

The testing package also provides helper functions for reporting errors, such as "Error," "Errorf," and "Fail." These functions can be used to report errors that occur during the test.

Here's an example of a Test Case written in Go:

func TestSum(t *testing.T) {
    t.Run("Sum of numbers in array", func(t *testing.T) {
        numbers := []int{5, 4, 3, 2, 1}

        got := Sum(numbers)
        want := 16

        if got != want {
            t.Errorf("got %d, want %d, given %d", got, want, numbers)
        }
    })
}

Points to remember when writing test case in golang.

The file name for the test case must end with _test.go i.e main_test.go so that Go tool can detect what file to execute when running go test command to run test cases.

Function for the test code, must be starts with Test keyword i.e TestSum. That's how go tool knows to run that test functions written in test files.

Go's testing tool can also generate a coverage report, which shows how much of your code is covered by tests. Command for getting test coverage is

go test -cover

Go's testing package also includes support for benchmarks, which can be used to measure the performance of your code. Benchmark functions have a specific signature and are executed multiple times to provide an accurate measurement of performance. Command to run benchmarks on test cases.

go test -bench=.

Table Driven Testing

Go's testing package also supports table-driven tests, which allow you to test a function with multiple inputs and expected outputs. This can be useful for testing edge cases and ensuring that your code is robust.

Here's an example of Table Driven Test:

func TestSum(t *testing.T) {
    cases := []struct {
        description string
        num1        int
        num2        int
        expected    int
    }{
        {
            description: "1 + 2",
            num1:        1,
            num2:        2,
            expected:    3,
        },
        {
            description: "3 + 4",
            num1:        3,
            num2:        4,
            expected:    7,
        },
        {
            description: "10 + 45",
            num1:        10,
            num2:        45,
            expected:    70,
        },
    }

    for _, tt := range cases {
        t.Run(tt.description, func(t *testing.T) {
            result := Sum(tt.num1, tt.num2)
            if result != tt.expected {
                t.Errorf("expected %d, but got %d", tt.expected, result)
            }
        })
    }
}

TDD Best Practices to follow

The fundamental principle of TDD is to write tests before writing the actual code.
Tests should be small and focused on a single piece of functionality. This allows you to easily pinpoint errors when they occur.
Use Go testing tool to simplify and speed up your testing workflow.
Table-driven tests allow you to test a function with multiple inputs and expected outputs. This can help you catch edge cases and ensure your code is robust.
When writing tests that rely on external dependencies, use mocks to simulate the behaviour of those dependencies. This allows you to test your code in isolation and avoid flaky tests.
Use this feedback to refactor your code and make it more modular, maintainable, and testable.
Make sure your tests are updated to reflect any changes you make to your code.
Code coverage tools like go test -cover can help you identify areas of your code that are not adequately covered by your tests.
Use a continuous integration (CI) tool to automate your testing workflow. This ensures that your tests are run regularly and that any errors are caught early in the development process.

Tools by Go Community

Go comes with its built-in testing tool, but there are some open source tools available as well created by Go community all around the world. Here are some of them that are mostly used in Go projects:

Gomega - Matcher/Assertion lib https://github.com/onsi/gomega
GoCheck - Featured rich testing lib https://github.com/go-check/check
Testify - Toolkit for mocks, assertions https://github.com/stretchr/testify
GoMock - A dedicated mocking framework https://github.com/golang/mock
Ginkgo - A BDD testing framework for expressive specs https://github.com/onsi/ginkgo

Conclusion

Test Driven Development is a mental model where you write test cases before actual production code. It required some practice if you haven't written any test case before. Either you're working on real-world projects or just starting out, you can start exploring and do hands on on it.

If you like my article, please like and share feedback and let me know in comments. And don't forget to follow me on Linkedin, Github, Twitter.

Peace ✌🏻