DEV Community: Mubbashir Mustafa

Python Essentials: A Speedy Introduction

Mubbashir Mustafa — Tue, 25 Jun 2024 14:51:02 +0000

Are you ready to dive into the exciting world of Artificial Intelligence and Machine Learning but need a quick introduction to Python first? This crash course is here to help! In this article, we'll cover the basics of Python programming to get you up to speed quickly. Whether you're new to programming or just need a refresher, this guide will provide the essential knowledge you need to start coding confidently. Let's get started!

Chapter 1: Python Syntax and Basics

In this chapter, we'll quickly cover Python's syntax and foundational concepts. This will serve as a concise refresher to familiarize you with Python's unique features.

Code Structure

Python uses indentation to define blocks of code, unlike languages like JavaScript which use braces. Consistent indentation is crucial as it directly impacts the program's flow. Here's a quick example:

def greet(name):
    if name:
        print(f"Hello, {name}!")
    else:
        print("Hello, world!")

Variables and Data Types

Python is dynamically typed, meaning you don't need to declare the type of a variable when you create one. The basic data types in Python include:

Strings: Defined with single ('...') or double ("...") quotes.
Integers: No special syntax required, e.g., 5.
Floats: Use a point to denote decimal, e.g., 5.0.
Booleans: Written as True or False.

Example

name = "Alice"
age = 30
is_registered = True #capitalized boolean in python

Collections

Lists: Ordered and mutable collection, e.g., numbers = [1, 2, 3].
Dictionaries: Key-value pairs, e.g., person = {"name": "Alice", "age": 30}.
Sets: Unordered collection of unique elements, e.g., unique_numbers = {1, 2, 3}.
Tuples: Ordered and immutable collection, e.g., point = (1, 2).

List Comprehensions

List comprehensions provide a concise way to create lists:

squares = [x**2 for x in range(10)]
print(squares)

[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

Functions

Defining functions in Python uses the def keyword. You can specify default values for parameters, which makes them optional during calls:

def greet(name, greeting="Hello"):
    print(f"{greeting}, {name}!")

greet("Alice")  # Uses the default greeting
greet("Bob", "Howdy")  # Uses a custom greeting

This chapter sets the foundation for understanding how Python code is structured and executed. Up next, we will explore control structures to manage the flow of your Python programs.

Chapter 2: Control Structures

Python's control structures allow you to direct the flow of your program's execution through conditional statements and loops. This section will refresh your understanding of these constructs.

If, Elif, and Else

Conditional statements in Python are straightforward. The if statement evaluates a condition and executes a block of code if the condition is true. You can extend this logic with elif (else if) and else:

age = 25
if age < 18:
    print("You are a minor.")
elif age < 65:
    print("You are an adult.")
else:
    print("You are a senior.")

Python evaluates conditions until one is true, then skips the rest, known as short-circuiting.

Loops

Python provides for and while loops for iterating over sequences or executing a block of code repeatedly under a condition.

For Loops: Used for iterating over a sequence (like a list, tuple, or string).

for number in range(5):  # range(5) generates numbers from 0 to 4
    print(number)

While Loops: Execute as long as a condition is true.

count = 0
while count < 5:
    print(count)
    count += 1

Control within loops can be managed with break (to exit the loop) and continue (to skip the current iteration and continue with the next one):

for number in range(10):
    if number == 3:
        continue  # Skip the print statement for number 3
    if number == 8:
        break  # Exit the loop when number is 8
    print(number)

Iterators and Generators

Python uses iterators for looping. You can create your own iterator by implementing iter() and next() methods or using generator functions.

Generators: Use yield to generate a sequence of values. Generators are useful when you want to iterate over a sequence without storing the entire sequence in memory.

def countdown(num):
    while num > 0:
        yield num
        num -= 1

for i in countdown(5):
    print(i)

This chapter covers the basic control structures in Python. You'll find these essential for creating conditional logic and managing looping in your Python programs. Next, we'll dive into more Pythonic techniques and features to help you write cleaner and more efficient code.

Chapter 3: Pythonic Techniques

This chapter delves into more sophisticated Pythonic techniques, enabling you to write cleaner, more efficient, and more Python-specific code. These techniques leverage Python's unique capabilities and idiomatic practices.

Unpacking

Unpacking allows you to assign values from a list or tuple to variables in a single statement, improving readability and conciseness.

a, b, c = [1, 2, 3]
print(a, b, c)  # Outputs: 1 2 3

# Star expressions for capturing excess items
first, *middle, last = [1, 2, 3, 4, 5]
print(first, middle, last)  # Outputs: 1 [2, 3, 4] 5

Lambda Functions

Lambda functions are small anonymous functions defined with the lambda keyword. They are best used for short, simple functions.

# A simple lambda to add two numbers
add = lambda x, y: x + y
print(add(5, 3))  # Outputs: 8

# Often used in functions like map() and filter()
squares = list(map(lambda x: x**2, range(10)))
print(squares)  # Outputs: [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

Map and Filter

The map and filter functions are functional programming tools that apply a function to each item in an iterable (like a list) and return an iterable.

map(): Applies a function to each item in an iterable.
filter(): Extracts elements from an iterable for which a function returns True.

# Using map to square numbers
nums = [1, 2, 3, 4, 5]
squared = list(map(lambda x: x**2, nums))

# Using filter to find even numbers
evens = list(filter(lambda x: x % 2 == 0, nums))

When compared to list comprehensions, map() and filter() can be preferable for their readability and expressive power with small lambda functions. However, list comprehensions are often clearer and more Pythonic when the operation is straightforward.

This chapter introduces you to writing Python in a way that is not just correct but also stylistically Pythonic, embracing the language's philosophy of readability and simplicity. By using these advanced features, you can make your code more modular, reusable, and expressive. Up next, we'll cover file handling and exception management to ensure your programs are robust and professional.

Chapter 4: File Handling and Exception Management

In Python, managing files and handling exceptions are critical for creating robust applications. This chapter covers the essentials of these areas, focusing on best practices and Pythonic approaches.

File Handling

Python simplifies file operations with built-in functions and methods. The with statement ensures that files are properly closed after their suite finishes, even if an exception is raised.

# Reading from a file
with open('example.txt', 'r') as file:
    content = file.read()
    print(content)

# Writing to a file
with open('output.txt', 'w') as file:
    file.write("Hello, Python!\n")

For reading and writing files, Python offers methods like read(), readline(), readlines(), write(), and writelines(), allowing you to handle different file sizes and content types efficiently.

Exception Handling

Proper error and exception handling is essential for writing reliable and user-friendly programs. Python uses try, except, else, and finally blocks to handle exceptions.

try:
    # Try to do something that might raise an exception
    result = 10 / 0
except ZeroDivisionError:
    # Handle specific exceptions
    print("Divided by zero!")
else:
    # Execute if no exceptions
    print("Division successful!", result)
finally:
    # Always executed
    print("Cleaning up, if needed.")

Use specific exception types in except blocks to catch and handle different error conditions appropriately. This not only helps in debugging but also allows the program to continue or fail gracefully.

Context Managers

For more complex resource management, Python provides context managers that allow you to allocate and release resources precisely when you want. The with statement is commonly used with file handling, as shown above, but can also be used with other resources like network connections or locking mechanisms.

from contextlib import contextmanager

@contextmanager
def managed_file(name):
    try:
        f = open(name, 'w')
        yield f
    finally:
        f.close()

# Usage
with managed_file('hello.txt') as f:
    f.write('hello, world!')

This chapter ensures that you are equipped to handle file I/O operations and exceptions in your Python applications effectively, contributing to their reliability and maintainability. Next, we will explore modules, packages, and Python's environment management tools.

Chapter 5: Modules and Packages

This chapter focuses on the organization and reusability of code in Python through modules and packages. Understanding how to create, import, and manage modules and packages is key for developing maintainable and scalable Python applications.

Importing Modules

Python's modules are simply Python files with .py extensions containing Python code that can be reused in other Python scripts.

# Importing a whole module
import math
print(math.sqrt(16))  # Outputs: 4.0

# Importing specific functions
from math import sqrt
print(sqrt(16))  # Outputs: 4.0

# Using alias for modules
import numpy as np
array = np.array([1, 2, 3])

When you import a module, Python executes all of the code in the module file.

Creating Modules

Any Python file can be a module. To create a module, simply save your Python code in a .py file. Other Python scripts can then import this file as a module.

# Example of a simple module, saved as calculator.py

def add(a, b):
    return a + b

def subtract(a, b):
    return a - b

You can then use this module in other Python scripts:

from calculator import add, subtract
print(add(5, 3))  # Outputs: 8

Packages

A package is a collection of Python modules under a common directory. For Python to recognize a directory as a package, it must contain a file named init.py. The file can be empty but it signifies that the directory contains Python modules.

# Assume the following directory structure:
# mypackage/
#   __init__.py
#   subpackage1/
#       __init__.py
#       module1.py
#   subpackage2/
#       __init__.py
#       module2.py

# Importing from a package
from mypackage.subpackage1 import module1
module1.some_function()

Virtual Environments

Using virtual environments in Python helps manage dependencies for different projects. By creating a virtual environment, you can keep dependencies required by different projects separate from each other.

# Creating a virtual environment
python -m venv myprojectenv

# Activating the virtual environment
# On Windows
myprojectenv\Scripts\activate

# On MacOS/Linux
source myprojectenv/bin/activate

This setup ensures that any Python packages installed while the virtual environment is active will only affect this particular environment, helping avoid conflicts between project dependencies.

This chapter provides the tools you need to effectively manage and modularize your Python code, essential for professional, clean, and efficient Python programming. Coming up next, we'll discuss best practices in logging and debugging to enhance the observability and maintainability of your Python applications.

Chapter 6: Logging and Debugging

Effective logging and debugging are crucial for developing, maintaining, and troubleshooting Python applications. This chapter covers how to utilize Python's built-in logging module and debugging tools to keep your applications running smoothly and efficiently.

Logging

Python's logging module provides a flexible framework for emitting log messages from Python programs. It is preferable to using print statements because it offers different severity levels and allows you to direct the logs to different outputs.

import logging

# Basic configuration to log to file
logging.basicConfig(filename='app.log', level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')

# Different levels of logs
logging.debug('This is a debug message')
logging.info('This is an info message')
logging.warning('This is a warning message')
logging.error('This is an error message')
logging.critical('This is a critical message')

Using Log Levels

Log levels provide a way to categorize the importance of the messages logged by the application:

DEBUG: Detailed information, typically of interest only when diagnosing problems.
**INFO: Confirmation that things are working as expected.
WARNING: An indication that something unexpected happened, or indicative of some problem in the near future.
ERROR: Due to a more serious problem, the software has not been able to perform some function.
CRITICAL: A serious error, indicating that the program itself may be unable to continue running.

Debugging Tools
When it comes to debugging, Python provides several tools to help identify issues in the code. The most commonly used tool is the Python Debugger (pdb), which allows interactive debugging.

import pdb

# Example usage
def div(a, b):
    pdb.set_trace()
    return a / b

print(div(4, 2))

Using pdb, you can step through your code, inspect variables, and evaluate expressions to diagnose and fix issues more effectively.

Commands: list (shows current position in the code), next (executes the next line), continue (continues execution until the next breakpoint), break (adds breakpoints), print (prints a variable), and quit (exits the debugger).

Tips for Effective Debugging

Start Small: Test small parts of your code as you write them.
Use Logs: Insert logging statements to report the occurrence of particular events.
Isolate Problems: When you encounter a bug, narrow down where it could be by using unit tests or dividing the code.

This chapter equips you with essential tools for logging and debugging, helping ensure your Python applications perform as intended and making it easier to maintain and troubleshoot them. In the next chapter, we will explore the broader Python ecosystem, including toolchains and additional libraries that can help streamline your Python development process.

Chapter 7: Toolchain and Additional Libraries

In this final chapter, we'll explore the broader Python ecosystem, focusing on toolchains for code style enforcement, testing, and some additional libraries that extend Python's capabilities. These tools and libraries can significantly enhance productivity and ensure high-quality software development.

PEP 8 and Linters

PEP 8 is the style guide for Python code. Adhering to PEP 8 helps ensure that your Python code is readable and consistent. Python's linters like flake8 or pylint help enforce coding standards and catch potential errors.

# Installing flake8
pip install flake8

# Using flake8
flake8 my_script.py

These tools provide feedback on style issues, complexity, and programmatic errors, making them indispensable for maintaining code quality.

Testing
Testing is critical in the development process to ensure your code behaves as expected.

unittest: Python’s built-in library that allows you to test small units of code independently.

import unittest

class TestMathOperations(unittest.TestCase):
    def test_add(self):
        self.assertEqual(1 + 1, 2)

    def test_subtract(self):
        self.assertEqual(2 - 1, 1)

if __name__ == '__main__':
    unittest.main()

pytest: A more powerful, third-party testing framework with a simpler syntax and more features than unittest.

# Installing pytest
pip install pytest

# Using pytest
pytest

Popular Python Libraries

Several Python libraries can help simplify complex tasks across different domains:

Requests: Simplifies making HTTP requests for web clients.

import requests
response = requests.get('https://api.example.com/data')

Pandas: Essential for data analysis and manipulation.

import pandas as pd
data = pd.read_csv('data.csv')

NumPy: Provides support for large, multi-dimensional arrays and matrices.

import numpy as np
array = np.array([1, 2, 3])

Scikit-Learn: Ideal for implementing machine learning algorithms.

from sklearn.ensemble import RandomForestClassifier
model = RandomForestClassifier()
model.fit(train_data, train_labels)

Matplotlib: A plotting library for creating static, interactive, and animated visualizations in Python

import matplotlib.pyplot as plt
plt.plot([1, 2, 3], [4, 5, 6])
plt.show()

Extra Chapter: Python and Object-Oriented Programming

Introduction to Classes and Objects

In Python, classes are created using the class keyword, and objects are instances of these classes. Here’s a simple example:

class Dog:
    def __init__(self, name, age):
        self.name = name
        self.age = age

    def speak(self, sound):
        return f"{self.name} says {sound}"

# Creating an instance of Dog
my_dog = Dog("Rex", 4)
print(my_dog.speak("Woof"))

Attributes and Methods

Attributes are variables associated with a class, and methods are functions defined within a class that operate on its attributes. The init method is a special method called a constructor, used for initializing an instance of the class.

Inheritance

Python supports inheritance, allowing multiple base classes and thus facilitating complex relationships between objects.

class Cat(Dog):  # Inherits from Dog
    def purr(self):
        return f"{self.name} purrs."

# Usage
my_cat = Cat("Whiskers", 3)
print(my_cat.speak("Meow"))  # Inherited method
print(my_cat.purr())         # New method

Encapsulation

Encapsulation is the bundling of data with the methods that operate on these data. It restricts direct access to some of the object’s components, which can prevent the accidental modification of data:

class Person:
    def __init__(self, name, age):
        self.name = name
        self._age = age  # Leading underscore suggests protected (by convention)

    def get_age(self):
        return self._age

# Direct access to _age is discouraged
john = Person("John", 28)
print(john.get_age())

Polymorphism

Polymorphism allows for the interchangeability of components through a common interface. In Python, it’s more loosely applied, allowing different classes to be used interchangeably if they implement the same methods.

def animal_sound(animal):
    print(animal.speak("Hi there"))

animal_sound(my_dog)
animal_sound(my_cat)

Congratulations on completing this quick crash course on Python basics! I hope this speedy overview has given you a solid foundation in Python programming, equipping you with the essential skills needed to embark on your AI and ML journey. While this guide covered the fundamentals, there's so much more to learn and explore when implementing practical solutions in the world of Artificial Intelligence and Machine Learning. As you continue to build on these basics, you'll discover the power and versatility of Python in solving complex problems and creating innovative solutions. Stay curious, keep coding, and get ready to transform your ideas into reality with Python!

Notebook version: https://github.com/mubbashir10/applied_ai/blob/main/Chapter%200%20-%20Python%20basics.ipynb

Enable Slack Notifications for AWS Amplify Deployments

Mubbashir Mustafa — Sun, 20 Mar 2022 08:09:39 +0000

AWS Amplify is a set of purpose-built tools and features that lets front-end web and mobile developers quickly and easily build full-stack applications on AWS, with the flexibility to leverage the breadth of AWS services as your use cases evolve.

At our company, we use AWS Amplify for hosting our frontend (React) apps. Although it's an amazing service and we love it, it lacks one important feature: integration with AWS Chat Bot (for sending pipeline notification to slack). It however allows you to send notifications to email addresses (which is not ideal, at least for us).

But there's a robust workaround for this problem, let me share that with you.

From AWS web console, go to AWS Amplify

Select the app against which you want to enable slack notifications

Select "Notifications" from the left sidebar

Enter an email address that you use and then click save

Now you need to create a lambda function that will receive messages from the AWS SNS topic (created by AWS Amplify) and forward that to slack

From AWS Console, go to AWS Lambda

Select "Create function"

Give a name to your function, choose runtime as node 14.x, and click create

Add the following code to your function

// deps in lambda are added as layers
const fetch = require('node-fetch')

// this should be your webhook URL (doc: https://api.slack.com/messaging/webhooks)
const integrationURL = ''

exports.handler = async (event) => {

  const message = event.Records[0].Sns.Message

  return await fetch(integrationURL, {
      method: 'POST',
      body: JSON.stringify({
        attachments: [{
          title: `AWS Amplifyy Notification!`,
          text: message,
        }]
      }),
      headers: { 'Content-Type': 'application/json' },
    })
    .then((data) => console.log('sent!'))
    .catch((e) => console.error(e.response.data))
}

Or grab it from github gist

The code uses the node-fetch package to make HTTP post request, you will need to add a dependency layer to your lambda function for it to work

To get integration URL you will need to create the slack app and enable incoming webhooks, check out the official tutorial.

Save your changes and click the "Deploy" button

Next, add a trigger to your AWS Lambda function

Select "SNS" as the trigger, and then select the SNS Topic that would be created by AWS Amplify (when you added email earlier to enable notifications)

To test things out, go back to AWS Amplify and from being within the project, click "Redeploy this version"

You should receive a notification in your slack

The email you added earlier is no longer needed (we only did that to expose SNS Topic) and you could remove it by visiting AWS Amplify->notifications.

Let's connect:

Linkedin: https://www.linkedin.com/in/mubbashir10/

Twitter: https://twitter.com/mubbashir100

AWS Copilot - launch and manage containerized applications quickly 🐳

Mubbashir Mustafa — Mon, 31 Jan 2022 01:10:33 +0000

AWS Copilot is an open-source CLI-based tool by the AWS team which lets you deploy containerized services easily by running a few simple commands. Behind the scenes, AWS takes care of all the infrastructure provisioning and configuration.

Before we begin, it's assumed that you have already containerized your application and it's ready to be deployed. If not, you can clone this sample repo and follow along.

Step 1

Currently, AWS Copilot doesn't work with root users and you need to create an IAM user with programmatic access. Go to IAM and create a new IAM account with programmatic access

Note: ℹ️ It's not mentioned in the documentation which policies are needed for AWS Copilot to work, see this github issue. For the sake of this tutorial you can give Administrator Access to your role.

Step 2

Install the AWS Copilot, if you are on a mac and use homebrew, you can simply run

brew install aws/tap/copilot-cli

For other OS, grab the relevant binary and install that

Verify the installation by running copilot -v

Step 3

Inside your app folder, run

copilot init

It will ask you to name your application, provide any name

Next, select the application type (for this tutorial I am selecting Load Balanced Web Service

Next, it will ask for the name of the service, provide any suitable name

If there are multiple Dockerfiles present, it will ask you to pick the desired one

It will begin the build process now

Once the build is ready, it will ask you if you would like to deploy your service to a test environment, select Yes

Wait for it to finish the process, once finished it will give you a URL to the service

Grab the URL and test it in a browser

Step 4

Now set up a pipeline for automated workflows (CI/CD)

Inside your app run

copilot pipeline init

git add copilot/pipeline.yml copilot/buildspec.yml copilot/.workspace && git commit -m "Adding pipeline artifacts" && git push

copilot pipeline update

Once the pipeline finishes, you can verify the changes by visiting the same URL again

Cleanup

To delete the app just run copilot app delete

AWS Copilot takes the hassle of provisioning and managing infrastructure and lets you deploy your containerized applications quickly. Behind the scenes, it creates everything for you (Cloud formation template, AWS ECR repo, AWS Code Pipeline, AWS Code build project, AWS ECS task definition, AWS ECS Cluster, AWS ECS Service, AWS Load balancer, Security groups, etc.
If you need more granular control and are interested in deploying your dockerized application on AWS ECS manually, you can follow the step-by-step tutorial here.

Let's connect:

Linkedin: https://www.linkedin.com/in/mubbashir10/

Twitter: https://twitter.com/mubbashir100

Deploy containerized functions on AWS Lambda

Mubbashir Mustafa — Sun, 31 Oct 2021 19:21:31 +0000

AWS Lambda lets you deploy containerized functions by packaging your AWS lambda function code and dependencies using Docker up to a size of 10GB. Here's a tutorial to demonstrate how you can containerize and deploy nodejs based lambda functions.

Prepare Container Image for AWS Lambda:

If you prefer you can clone the repo, otherwise follow along.

Create a file called functions.js inside your node project and add the following sample function to it.

// A sample function to demo containers deployment on aws lambda
exports.helloLambda = async (event) => {
  const response = {
    isBase64Encoded: false,
    statusCode: 200,
    headers: {
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      message: "Containers on lambda!🐳",
    }),
  };
  return response;
};

Create a Dockerfile with the following content

FROM amazon/aws-lambda-nodejs:12
COPY functions.js package*.json ./
# RUN npm install // uncomment if your functions has dependencies
CMD [ "functions.helloLambda" ]

Build, tag and push the image to ECR*

aws ecr get-login-password --region <region-name> | docker login --username AWS --password-stdin <ecr-repo-uri-without-tag>

docker build -t node-app .

docker tag node-app:latest <ecr-repo-uri-without-tag>/<repo-name>:latest

docker push <ecr-repo-uri-without-tag>/<repo-name>:latest

*Learn how to publish images on ECR

Deploy The Image on AWS Lambda:

From the AWS Lambda landing page, select "Create function"

Choose "Container image", give any name, add image URI (can be obtained from AWS ECR) and click 'Create function'

To test the function, add a trigger

Choose API Gateway as a trigger and create an HTTP API and leave the security to open (for simplicity)

Once the trigger has been created, copy the endpoint URL and paste it in the browser

It should show you the response content

For containers to work with AWS Lambda, you can either use the open-source base container images that AWS Provides or you can add lambda runtime interface clients to your base images. In the tutorial, we have used a pre-built images.

Let's connect:

Linkedin: https://www.linkedin.com/in/mubbashir10/

Twitter: https://twitter.com/mubbashir100

Run Containers on AWS Lightsail

Mubbashir Mustafa — Mon, 13 Sep 2021 14:22:08 +0000

AWS Lightsail is an easy-to-use service that offers all the features needed to host web applications of any scale. Besides VPS services, the AWS Lightsail also lets you deploy your containers.

From the AWS console, navigate to 'Lightsail'

From the Lightsail homepage, switch to the 'Containers' tab and then click 'Create container service'

Select the region where you want to deploy your containerized application

Select size and number of instances based on your usage

Click 'Set up deployment'

Choose 'Specify a custom deployment'
Enter any meaningful name as Container name
Enter your public image name with tag, example: nginx:latest (at the time of writing AWS Lightsail doesn't support private images for container deployment)
Click 'Add open ports' and add ports on which your application listens to
In the PUBLIC ENDPOINT, select the container name which is open to public

Give a name to your service and click 'Create container service'

Once the deployment is finished successfully, it will change the status to active

To access your service/containerized application, use the link given as a public domain

Let's connect:

Linkedin: https://www.linkedin.com/in/mubbashir10/

Twitter: https://twitter.com/mubbashir100

Deploy Microservices on AWS ECS with Fargate (Serverless)

Mubbashir Mustafa — Sun, 12 Sep 2021 12:16:04 +0000

Amazon ECS (Amazon Elastic Container Service) is a highly scalable and fully managed container management service that makes it a breeze to run, stop, and manage containers on a cluster. You can use it to deploy your containerized applications or your microservices.

Before starting, here's a summary of key concepts in AWS ECS:

Cluster

A logical grouping of your services or tasks

Task Definition

Specification about how your container(s) should be run on AWS ECS

Task

An instance of the Task Definition

Service

A task manager

Section 1: Create an AWS ECS Cluster

From the AWS console, navigate to AWS ECS

Click 'Get started'

Click 'Create cluster'

Select 'Network only' as cluster template, click Next step

Configure your cluster and click 'Create'

Cluster name
Give any meaningful name to your cluster

Create VPC (optional)
Enable to launch your cluster in a VPC (A VPC is an isolated portion of the AWS Cloud)

Tags (optional)
You can use them to help you organize/group your AWS resources

CloudWatch Container Insights
Check to enable container insights (failures, restarts, CPU utilization, memory usage, .etc) and view them inside AWS Cloud Watch

Once the cluster has been created, click 'View Cluster'

Section 2: Create an AWS ECS Task Definition

From the left sidebar (expand it by clicking the hamburger menu from the left top), click 'Task Definitions'

Click 'Create new Task Definition'

Select 'Fargate' and click 'Next step'

Give a meaningful name to your task definition, leave the Task Role field empty (or you can create a new role from IAM and select it here), and leave the 'Network mode' to awsvpc (when launching as the Fargate type, only awsvpc can be selected)

Select 'Create new role' (or select any other existing role that you have created, and configure vCPU and Memory size for this task (this determines what charges you will be paying for this task)

Click 'Add container'

Give a name to your container, in the image field paste the image URI of the image (If you are not familiar with AWS ECR then read this first), add port to which your container listens to (for example if your nodejs app is running on 3000, then you should enter 3000 in this field)

You can skip the Advanced container configuration but if you need to configure other aspects of your container (like health checks, volumes, networking, the environment variable, etc. then you can do that from this section).

ℹ️ Note: You can add multiple containers under a Task definition

Once the container configuration is completed, click 'Add'

You can skip the integrations with AWS Firelens and AWS App Mesh unless you need it. If you need to add volumes that should be accessible to every container within a task, do so from the volumes section. The Tags are optional and let you organize your AWS resources.

Click 'Add'

Once the task definition has been created, it will show you a success message

Section 3: Create an AWS ECS Service

Now go back to your cluster and click 'Deploy' (from the services section)

Leave everything to default

Application Type:
Service

Specify revision manually
Uncheck

Family
Select the Task Definition you created earlier

Revision
Latest

Service Name
Any meaningful name

Desired Tasks
Enter the number of Tasks (instances) you want to run

Load balancing is optional but recommended. Configure a load balancer to distribute incoming traffic across the tasks running in your service.

Click on 'Create a new load balancer'
Give any meaningful name
Enter in the port where you want AWS ELB (elastic load balancer) to listen on (this is not the container's port that you configured earlier in the task definition)
Select HTTP/HTTPS as protocol (for HTTPS you will need to have a valid SSL certificate)
Target group routes traffic from AWS ELB to tasks/instances; give any name and select the protocol as HTTP

In the Networking section:

Select the VPC in which your cluster and task definitions are (by default, it's always already selected)
Select subnets (within VPC) where you want to deploy the task (I have selected all available subnets under my VPC)
Select "Use an existing security group", and select a security group that allows inbound HTTP traffic - learn how to create
Every task can be associated with a Public IP address, disable it if you don't need a public IP

Add tags if you want to and then click Deploy

Once the service has deployed, it will change the status to "Active"

Section 4: Access the AWS ECS Service

To access the service you can go to AWS EC2-> Load Balancers->Your Load Balancer Name and then copy the A-Record

To access an individual task (provided you enabled the public IP earlier), go to AWS ECS->Services->Your Service->Logs->Your Task->Network->Public IP

Let's connect:

Linkedin: https://www.linkedin.com/in/mubbashir10/

Twitter: https://twitter.com/mubbashir100

Control inbound/outbound traffic to an AWS Resource using AWS Security Groups

Mubbashir Mustafa — Sun, 12 Sep 2021 10:40:50 +0000

AWS Security group is like a virtual firewall within a VPC that acts at the instance level and not at a subnet level. Security groups have a set of rules to allow/disallow incoming/outgoing traffic to an instance.

Creating a security group is a fairly simple task. From the EC2 Console, select "Security Groups" under the Network & Security section

Click 'Create security group'

Give any meaningful name and a description, enter in the ID of the VPC in which you want to create the security group

Click 'Add rule' within the Inbound rules section and add HTTP and HTTPS and set the source to 'Anywhere' in both

Leave the Outbound rules section as it is (unless you want to change something) and click 'Create security group'

Let's connect:

Linkedin: https://www.linkedin.com/in/mubbashir10/

Twitter: https://twitter.com/mubbashir100

Deploy containerized services on AWS App Runner

Mubbashir Mustafa — Sun, 05 Sep 2021 11:38:15 +0000

AWS App Runner is the easiest way to deploy your Containerized, Node or Python based services. You don't have to worry about managing infrastructure or scaling up and down the resources when the load increases or decreases.

ℹ️ Note: It's assumed that you are familiar with AWS ECR and your image has already been containerized and pushed. If you are new to AWS ECR then read this first.

To deploy your services on AWS App Runner, navigate to AWS App Runner from the AWS console

From there click on "Create an App Runner service"

Now you need to provide the source image - which AWS App Runner will deploy as a service. For this, select "Container Registry", "Amazon ECR" (select Public, if your image is not private) and paste the Container image URI (you can get it from AWS ECR)

Next configure the deployment settings for your service. Select "Automatic" (it will deploy a new version of your service as soon as the new image has been pushed), "Create new service role" and "Next"

Give any name to your backend service, select vCPU and memory as per the need (expected load/traffic), and enter environment variables (if needed) as key/value pairs

In the "Auto-scaling" settings, you can use the default one or if you need to adjust them then create a "Custom configuration"

Concurrency:
Max. number of concurrent requests/connections after which a new instance will be launched
Minimum size:
It's the minimum number of instances that will always be running regardless of load
Maximum Size:
Maximum number of instances you want AWS App runner to launch when the load increases (in this case more than 100 concurrent requests/connections)

The health check is a mechanism to know whether the service is performing and stable or not. You can leave the default settings or configure them as needed

By default, the Unhealthy threshold is set to 5. This essentially means that whenever 5 health check requests from the load balancer to an instance fails, it will consider it unhealthy and try to replace it.

Next, you can create a custom IAM role from the https://aws.amazon.com/iam/ that will give permissions to your container to communicate with other AWS services or you can leave it. To encrypt the stored image (if you chose ECR image ) or stored source code bundle (if you chose GitHub, earlier) you can use AWS-owned encryption key or customer-managed key CMK (can be created from https://aws.amazon.com/kms/.

By default, if you don't provide one, an AWS-owned encryption key is used.

Tags are optional key/value pairs to help you identify and group your AWS resources. Add tags if needed and then click "Next"

Review your configurations and click "Create & Deploy"

It will take some time to build your app, once the status changes from "Operation in progress" to "Completed", your changes are deployed

Once the service has deployed successfully, get the default domain link and see it in action

Let's make some changes to your code and push it (build the docker image and push to AWS ECR - you will notice that the build process triggers automatically and deploys the new version

Notes:
AWS App Runner offers the easiest and quickest way to run containerized application. If you don't want to use conainerized application(s), then you can also deploy your nodejs or python based services on AWS App Runner from Github.

Let's connect:

Linkedin: https://www.linkedin.com/in/mubbashir10/

Twitter: https://twitter.com/mubbashir100

Push/Publish Docker images to AWS ECR

Mubbashir Mustafa — Fri, 03 Sep 2021 17:44:43 +0000

Amazon Elastic Container Registry (AWS ECR) is a container registry service where you store, manage, share, and deploy your container images (like DockerHub). Consider it as npmjs of containers instead of JS packages. AWS ECR lets you publish private as well as public images.

Before you proceed, it's assumed that you already have a containerized application. If you don't have one, you can clone this sample nodejs application.

From the AWS ECR console, click 'Get Started'

Options

Visibility settings: select Private or Public
Repository name: give any meaningful name
Tag immutability: when enabled, it will prevent tags from being overwritten by new pushes (when pushed with the same tag)

Enable Image scan setting to enable vulnerability scan after the image is pushed

Enable to use AWS KMS for encryption

Once the repository is created, it will take you back to the repositories list. Select the newly created repository and then click on the "View push commands" button.

Now you need to use these commands one by one to build, tag, and push your container image. But before proceeding make sure you have the latest version of AWS CLI installed.

Use the following command to check that AWS CLI is correctly installed on your system.

aws --version

It would output something like this. Your version could be different, it depends when you are reading this article.

Next, you need to create 'Access Keys'. For that, go to "My Security Credentials" from the dropdown with your username.

Click on "Create New Access Key", it will create a new key for you.

Copy "Access Key ID" and "Secret Access Key"

Now go back to your terminal and enter the following command
aws configure

It will ask for "Access Key ID" and "Secret Access Key", "Default region", provide them one by one. You can skip the default output format.

Once the AWS CLI has been configured, you can run the push commands. Open up the terminal and make sure you are inside the containerized app's root directory. Run each command shown in the popup opened up by clicking the "View push commands" button. Please note that the commands in the screenshot could be different than what you see in your popup and it's totally fine.

*If your 'Dockerfile' is named something other than 'Dockerfile', you will have to specify it using -f flag (see below).

Let's connect:

Linkedin: https://www.linkedin.com/in/mubbashir10/

Twitter: https://twitter.com/mubbashir100

The Easiest way to deploy services on AWS using App Runner

Mubbashir Mustafa — Fri, 20 Aug 2021 05:34:26 +0000

AWS App Runner is newer service introduced by AWS. Using AWS App Runner you can run your production grade containerized services within few mins.

From AWS Console, navigate to AWS App Runner and click on "Create an App Runner Service".

ℹ️ You can either link your GitHub repository (and build your app) or use ECR (private or public images) with AWS App Runner. For this tutorial, however, we are going to use Github.

Select "Source code repository", and then click on "Add new".

It will open up a new tab and ask for GitHub authorization, click on "Authorize AWS Connector for Github"

Give a name to your AWS-Github connection and then click on "Install another". It will take you to GitHub and display your account/organization. Select the one where your repo (that you want to use with AWS App Runner) is hosted.

Select "All Repositories (or you can choose a specific one)" and click "Install".

After the installation is finished, it will redirect you back to AWS. You will notice that now the dropdown has the connector name selected; click "Next"

Select the connection you just created. Then select the repository where your app's source code (the one you want to deploy on AWS App Runner) is located. Select the branch from the repository you want to use. In the deployment settings, select "Automatic" and click "Next"

ℹ️ If you don't have an existing app, you can clone this sample Node JS application to use with AWS App Runner

Choose "Configure all settings here", choose "Node.js 12" as the runtime, provide the installation command (usually npm i or yarn), provide the command to start your application (e.g yarn start) and finally enter in the port on which your app listens.

Give any name to your backend service, select vCPU and memory as per the need (expected load/traffic), and enter environment variables (if needed) as key/value pairs

In the "Auto-scaling" settings, you can use the default one or if you need to adjust them then create a "Custom configuration"

Concurrency:
Max. number of concurrent requests/connections after which a new instance will be launched
Minimum size:
It's the minimum number of instances that will always be running regardless of load
Maximum Size:
Maximum number of instances you want AWS App runner to launch when the load increases (in this case more than 100 concurrent requests/connections)

The health check is a mechanism to know whether the service is performing and stable or not. You can leave the default settings or configure them as needed

By default, if you don't provide one, an AWS-owned encryption key is used.

Tags are optional key/value pairs to help you identify and group your AWS resources. Add tags if needed and then click "Next"

Review your configurations and click "Create & Deploy"

It will take some time to build your app, once the status changes from "Operation in progress" to "Completed", your changes are deployed

Once the service has deployed successfully, get the default domain link and see it in action

Let's make some changes to your code and push it - you will notice that the build process triggers automatically and deploys the new version

Notes:
AWS App Runner is an easy-to-use service. You can get started with it without any prior AWS knowledge. It can help you deploy your services (containerized, nodejs, or python) easily. Not exactly, but it can be considered as a toned-down version of AWS Elastic Beanstalk or Google's App Engine

If you want to deploy containerized applications/services on AWS App Runner [then read] this(https://dev.to/mubbashir10/deploy-containerized-services-on-aws-app-runner-2897)

Let's connect:

Linkedin: https://www.linkedin.com/in/mubbashir10/

Twitter: https://twitter.com/mubbashir100

DevOps Roadmap - From Technical Perspective

Mubbashir Mustafa — Sat, 02 Jan 2021 09:44:16 +0000

What is DevOps (In my words):

A cross-functional mindset that makes your team more agile
The practice/culture to remove the barrier between developers (who develop apps, write code, are responsible for implementing feature requests, bug fixing, etc.) and operators (who deploy those apps, manage IT infrastructure/servers, and are responsible for uptime, security, stability, scalability, etc.)
Collaboration between operators, system administrators, DB administrators, developers, testers, etc.
A movement that essentially prevents people from saying "but it works on my localhost"
Honestly, a philosophy, one practical implementation is SRE

DevOps Focus Areas:

🎯 Reduce Organization Silos
Collaborate, communicate, educate and share.

🚀 Push Gradual Changes
Roll out small changes so that if something goes wrong you can easily roll back to a previously (acceptable) state.

🐞Embrace Failure
Wherever a human element is involved, failure is inevitable. A good DevOps strategy is one that considers failure as a normal thing.

📊Monitor Everything
Monitor everything, generate meaningful data, analyze it and reduce toil over a period of time.

🤖Tooling & Automation
Leverage tools to implement IaC and CI/CD

How to become a DevOps guy - a roadmap? (from a technical standpoint):
ℹ️ I have compiled the list in an order that I would follow if I am just starting

Computer Programming (any language will do - primarily, you should be able to write scripts)
Version control (git is a de facto standard)
Linux/Unix/Posix Compliant OS (get comfortable with it)
Basics of Computer Networking
Containerization (Docker, Podman, CRI-o, etc. - try to go for OCI compliant one)
Container Orchestration (Docker Swarm, Kubernetes, Mesos, .etc)
Cloud Platform (AWS, Azure, GCP, etc. - one important point here is that these cloud providers have so many products and it is assumed that you should be good in the ones that are needed in your workflow. For example, it is good to know how AWS Lex works, but it's not needed to become a DevOps engineer 🤷🏽‍♂️)
Continuous Integration/Continous Deployment/Continous Delivery - CI/CD (CircleCI, Github Actions, Bitbucket Pipelines, Gitlab CI/CD, Jenkins, Bamboo, Chef, AWS Code Build, AWS Code Pipeline, GCP Cloud Build, .etc)
Automation/Infrastructure as a Code or IaC (Terraform, Pulumi, Ansible, Puppet, Saltstack, .etc)
Logging and Monitoring (Grafana & Prometheus, etc.)

Notes:
You don't necessarily need to learn all of the available tools/services in a given category. For example, in the Cloud Platform section, the three big cloud providers out there have the following managed services for Kubernetes:

AWS-> (Elastic Kubernetes Service) EKS
GCP-> (Google Kubernetes Engine) GKE
Azure-> (Azure Kubernetes Service) AKS

Knowing more than one is desirable but not a requirement. I believe if you can deploy on one platform you should be able to deploy on the other as well.

Similarly in IaC, I would suggest you pick the ones that you generally enjoy working with or your potential employer/client is using.

What's Next?:
Once you are done with the list your basics are completed and you are ready to take on DevOps engineer role. You will also discover new tools as well along the way, like:
Helm (Kubernetes manager), Containerd (container runtime just like Docker, CRI-o), Packer (machine image builder for multiple platforms), Vagrant (headless virtualization), Prometheus (monitoring), Grafana (data visualization like Tableau, AWS Quicksight, .etc), Envoy (proxy like HA, Nginx, etc.), meshes (AWS Mesh, Istio, .etc) ....and many more 🤯 (so just don't be afraid, once your basics are strong you can give any and every tool a try and see if you enjoy it and if it's improving your workflow or it's just hype)

Launch Your WordPress website with AWS Lightsail with few clicks only!

Mubbashir Mustafa — Sat, 26 Dec 2020 00:29:29 +0000

Lightsail is an amazing service by AWS. It's beginner-friendly and someone who is new to AWS should begin with it to get the hang of it. It offers Virtual servers, storage, databases, and networking for a low and easy-to-understand prices. Lightsail also supports Load balancers, CDN and containers now!

In this tutorial, I am going to show you how easy it is to launch your WordPress website on AWS Lightsail with SSL and your domain.

Step 1: Launch WordPress Instance

From the AWS console, go to Lightsail. Once on the AWS Lightsail homepage, click "Create Instance"

Select region (closest to your targeted audience)

Select "WordPress" as your instance image

Click "Change SSH key pair"

Make sure you have downloaded the SSH key (we will need this later on) before proceeding. If you don't like the default one you can create a new one using "Create New". Leave "Enable Automatic Snapshots" for now, we will cover snapshot later on

Give any name to uniquely identify your instance, leave the number of instances to 1, choose the pricing plan that's suitable for you, and click "Create Instance"

Once you launch the instance, it will take some time to boot. Once the instance is booted up, you can copy the IP address and paste it into a new browser tab to access your website

If you enter the IP address in the browser tab you will notice that it has installed WordPress with the latest default theme - twenty twenty one (at the time of writing)

🎉 Congratulations! You have launched your own WordPress website. Let's attach an SSL certificate and domain to it and get admin credentials.

Step 2: Attach domain

From the AWS Lightsail dashboard, click on the newly created instance name

Select "Networking"

Click "Create static IP"

From the "Attach to an instance" dropdown, select the newly created instance. GIve any Name and click "Create"

This will attach a static IP to your instance (WHich essentially means that your IP will be the same even after you restart your instance). Your website will only be accessible with the new IP now (not the previous one). After the static IP has been created and attached it will show you the IP details page. Click on the AWS Lightsail logo from the top left corner to go back to the AWS Lightsail dashboard

Now go to your domain registrar's control panel and add a new A-record against your domain. Set the value equal to the new (static, the one we created earlier) IP

I am using AWS Route 53, but the setup is pretty much the same for every domain registrar (GoDaddy, Namecheap, Dreamhost, Hostgator, etc.)

You should be able to access your website using your domain now. Please note that it can take up to 72 hours to complete DNS propagation, so don't worry if you are not able to access your website using the domain name right away.

Step 3: Generate an SSL certificate

The next step is to generate and attach an SSL certificate. For that connect to your website using SSH by the following command
ssh -i <path_to_your_key> bitnami@<your-domain-or-ip>
When prompted for adding to the known host list, type yes and press enter
If it says something about bad permissions then run the following command to change the key file permissions
sudo chmod 400 <path_to_your_key>

Once you have logged in to your instance (via ssh), run the following command
sudo /opt/bitnami/bncert-tool

It will ask you a series of questions. Enter in the domain name you are using (add www version too if you intend to use it). Type Y and press enter when it asks for "Enable HTTP to HTTPS redirection", and "Do you agree to these changes?". Enter in your email address, again type Y and press enter when it asks "Do you agree to the Let's Encrypt Subscriber Agreement?"

After that, you should be able to access your website with https

Step 4: Get admin credentials

Let's first remove the bitnami default page link from the bottom right corner

While being logged in to your instance (via ssh), run
sudo /opt/bitnami/apps/wordpress/bnconfig --disable_banner 1

To get the credentials, run the following command:
cat ~/bitnami_credentials

Now go to https://<your_domain_here>/wp-admin and you should be able to login to the WordPress dashboard using the user/password we got earlier

Extras:

We used Let's Encrypt Free SSL and Lego Client behind the scenes to generate a free SSL and then set-up a cron job to renew the SSL certificate automatically when it expires
In AWS Lightsail you can take snapshots (full-fledged cloned images of your instance) by going to the Instance page and then selecting the snapshot tab. Using snapshots you can take backups, keep older versions of your website or you can create larger size (upgraded) instances. The AWS Lightsail snapshots are pretty cheap, you can check the prices here
It's a simple option and works really well, I am using AWS Lightsail for hosting my small baby clothes store: 🍼 Toddos Club