DEV Community: Abdullah Paracha

Building a Scalable Data Pipeline on AWS

Abdullah Paracha — Sat, 04 Jan 2025 17:43:00 +0000

Building a scalable data pipeline using AWS services. This pipeline ingests data from an external source, processes it, and loads it into Amazon Redshift for analysis.

Step 1: Data Ingestion to S3
Use Python and the AWS SDK (boto3) to upload data to an S3 bucket.

import boto3

# Initialize S3 client
s3 = boto3.client('s3')

# Define bucket name and file details
bucket_name = 'my-data-lake'
file_name = 'data.csv'
file_path = '/path/to/data.csv'

# Upload file to S3
s3.upload_file(file_path, bucket_name, file_name)
print(f"Uploaded {file_name} to {bucket_name}")

Step 2: Data Processing with AWS Glue
Create an AWS Glue job to transform raw data into a structured format.

Here’s an example Glue script written in PySpark:

import sys
def main():
    from awsglue.transforms import *
    from awsglue.utils import getResolvedOptions
    from pyspark.context import SparkContext
    from awsglue.context import GlueContext
    from awsglue.job import Job

    args = getResolvedOptions(sys.argv, ['JOB_NAME'])
    sc = SparkContext()
    glueContext = GlueContext(sc)
    spark = glueContext.spark_session
    job = Job(glueContext)
    job.init(args['JOB_NAME'], args)

    # Load data from S3
    input_path = "s3://my-data-lake/data.csv"
    dynamic_frame = glueContext.create_dynamic_frame.from_options(
        connection_type="s3",
        connection_options={"paths": [input_path]},
        format="csv",
    )

    # Perform transformations
    transformed_frame = ApplyMapping.apply(
        frame=dynamic_frame,
        mappings=[("column1", "string", "col1", "string"),
                 ("column2", "int", "col2", "int")]
    )

    # Write transformed data back to S3
    output_path = "s3://my-data-lake/transformed/"
    glueContext.write_dynamic_frame.from_options(
        frame=transformed_frame,
        connection_type="s3",
        connection_options={"path": output_path},
        format="parquet"
    )

    job.commit()

if __name__ == "__main__":
    main()

Step 3: Load Data into Amazon Redshift
Copy the transformed data from S3 into Amazon Redshift.

COPY my_table
FROM 's3://my-data-lake/transformed/'
IAM_ROLE 'arn:aws:iam::123456789012:role/MyRedshiftRole'
FORMAT AS PARQUET;

Step 4: Real-Time Data Processing with Amazon Kinesis
Use Kinesis to ingest and process streaming data in real time. Below is an example of setting up a simple Python consumer for Kinesis Data Streams:

import boto3
import json

def process_record(record):
    data = json.loads(record['Data'])
    print("Processed Record:", data)

# Initialize Kinesis client
kinesis = boto3.client('kinesis')
stream_name = 'my-data-stream'

# Fetch records from the stream
response = kinesis.get_records(
    ShardIterator=kinesis.get_shard_iterator(
        StreamName=stream_name,
        ShardId='shardId-000000000000',
        ShardIteratorType='LATEST'
    )['ShardIterator'],
    Limit=10
)

# Process each record
for record in response['Records']:
    process_record(record)

Step 5: Query Data Using Amazon Athena
For ad-hoc queries, you can use Athena to query the data directly from S3.

SELECT col1, col2
FROM "my_data_lake_database"."transformed_data"
WHERE col2 > 100;

Step 6: Automating Workflows with AWS Data Pipeline
Use AWS Data Pipeline to schedule and automate tasks such as running an EMR job or triggering an S3-to-Redshift load.

{
  "objects": [
    {
      "id": "Default",
      "name": "Default",
      "fields": []
    },
    {
      "id": "S3ToRedshiftCopyActivity",
      "type": "CopyActivity",
      "schedule": {
        "ref": "Default"
      },
      "input": {
        "ref": "MyS3DataNode"
      },
      "output": {
        "ref": "MyRedshiftTable"
      }
    },
    {
      "id": "MyS3DataNode",
      "type": "S3DataNode",
      "directoryPath": "s3://my-data-lake/transformed/"
    },
    {
      "id": "MyRedshiftTable",
      "type": "RedshiftDataNode",
      "tableName": "my_table"
    }
  ]
}

Conclusion:
AWS provides an extensive ecosystem of services that make building data pipelines efficient and scalable. Whether you’re dealing with batch or real-time processing, the combination of S3, Glue, Redshift, Kinesis, Athena, EMR, and Data Pipeline enables you to design robust solutions tailored to your needs. By integrating these services, data engineers can focus on extracting insights and adding value rather than managing infrastructure.

Start building your AWS data pipelines today and unlock the full potential of your data!

Building Interactive Applications with Amazon Bedrock, Amazon S3 and Streamlit

Abdullah Paracha — Thu, 08 Aug 2024 12:34:35 +0000

👉🏻 This is a step-by-step guide on how to build an interactive web application with render interactive elements and integrate Amazon S3 and Amazon Bedrock with the Streamlit application. With custom-designed and interactive web UI, we are able to showcase a complete data exploration application along with generative AI capabilities.
(Note: to better focus on the deployment process, some pre-requisite steps such as Amazon EC2 configuration are not covered in the guide)

Use Case
AWS Architecture
Step-by-step guide on deployment
- Connect to virtual machine using EC2 instance connect
- Deploy the Streamlit Application to Amazon EC2
Application Codes Tour
- Build Streamlit Basic WebUI
- Build an interactive file upload webpage
- Build a generative AI image generator
Conclusion

Streamlit is an open-source Python library that makes it easy to create and share custom web apps. Streamlit lets you transform Python scripts into interactive web apps in minutes, instead of weeks.

There are a number of use cases by building with Streamlit, such as:

Building dashboards and data apps
generate reports from large documents
or create generative AI chatbots.

With the current spiking trends on LLM applications, Streamlit allows develop to deliver dynamic interactive apps with only a few lines of code.

Amazon Bedrock is a fully managed service that offers a choice of high-performing foundation models (FMs) through a single API, along with a broad set of capabilities you need to build generative AI applications with security, privacy, and responsible AI. [2]

Amazon Bedrock takes advantages of the latest generative AI innovations with easy access to a choice of high-performing foundation models FMs from leading AI companies, such as Meta, Mistral AI, Stability AI, and Amazon.

There are many different foundational models available in Amazon Bedrock including text, chat, and image. Model Evaluation on Amazon Bedrock allows you to use automatic and human evaluations to select FMs for a specific use case. To tailor to your own needs, you can go from generic models to ones that are specialized and customized for your business and use case. [3]

In this use case, we will utilize Amazon Titan models to build generative AI applications as well as Streamlit’s easy-to-deploy framework, we can easily develop our own data and AI products.

1. Use Case

In this use case, we are going to build an interactive web app which allows users to explore data, upload files and create generative AI photos.

We will embed Amazon Bedrock FMs to the application, deploy a Streamlit application to an Amazon EC2 instance, and allow users to begin interacting with the application.

2. AWS Architecture

In the development process, we will:

Deploy a Streamlit application to Amazon EC2
Render Streamlit elements such as chatbot function in a web application
Integrate Amazon S3 and Amazon Bedrock with a Streamlit application

3. Step-by-Step guide on deployment

There are a number of pre-requisite steps before deploying the actual Streamlit applications. Before starting below steps, you should have:

create an Amazon EC2 instance
setup an S3 bucket to store uploaded files
create a Github repository to hold Streamlit python codes

3.1 Connect to virtual machine using EC2 instance connect

In EC2, right click the pre-configured EC2 name, connect to an EC2 instance using EC2 Instance Connect and access a shell.

The following deployment process will be completed in the Shell environment.

3.2 Deploy the Streamlit Application to Amazon EC2

Note: there will be detailed walkthrough of the application code in section 4.

In the terminal, use the following set of commands to configure the AWS account credentials:

aws configure set aws_access_key_id <Your acess_key_id> &&
aws configure set aws_secret_access_key <Your acess_key> &&
aws configure set default.region <Your AWS region>

In the above command, provide your AWS account credentials to be configured.

The following command will display the Amazon S3 bucket name required by the Streamlit application:

echo $BUCKET_NAME

In this use case, the S3 bucket name is displayed:

Enter the following command to clone Github directory:

git clone https://github.com/<Your github directory>.git

You may setup different branches for your github repo to store application code.

To deploy the application, enter the following set of commands:

cd src/ && pip install -r requirements.txt

This command installs the required Python packages for the Streamlit application.

To start the Streamlit application with webUI (python code walkthrough will be in the next section):

streamlit run Basics.py

This command starts the Streamlit application on the EC2 instance.
The Basics.py file is the main application file that you will run. Streamlit will utilize the pages/ directory to render the additional pages in a sidebar.
Copy the external URL, and open a new tab to start the Streamlit application.

4. Application Codes Tour

You may store the Streamlit application code is a src/ directory.

To run the application, first install the necessary libraries and listed them in the requirements.txt file.

streamlit
boto3
pandas
streamlit_pdf_viewer

The streamlit library will be referenced in each of the application files. This library contains the UI elements that will render the various widgets and fields used in the application.
boto3 will be used to interact with the AWS services. To use this library, you must configure AWS credentials within the environment the Streamlit application is served.
The pandas library is used to present data within the application.
streamlit_pdf_viewer is a third-party, custom Streamlit component that allows you to control how PDF files are displayed.

4.1 Build Streamlit Basic WebUI

Refer to below Python code to create a simple basic WebUI that display

import streamlit as st
import pandas as pd
import random

# Streamlit page configuration
st.set_page_config(layout="wide", page_title="Streamlit Basics")
st.title("Streamlit Basics")

# Streamlit container
with st.container(border=True):
    # Tabs
    text, data, chat, markdown = st.tabs(["Text", "Data", "Chat", "Markdown Editor"])

    # Tab content

    # Distplaying text
    with text:
        st.title("Titles")
        st.divider()
        st.header("Headers")
        st.subheader("Subheaders")
        st.text("Normal text")
        st.markdown("***Markdown Text***")
        st.code("for i in range(8): print(i)")


    # Displaying data as a data frame
    with data:
        st.subheader("Display data using a data frame")
        df = pd.DataFrame(
            {
                "name": ["Spirited Away", "Princess Mononoke", "My Neighbor Totoro"],
                "url": ["https://m.imdb.com/title/tt0245429/", "https://m.imdb.com/title/tt0119698/", "https://m.imdb.com/title/tt0096283/"],
                "reviews": [random.randint(0, 1000) for _ in range(3)],
                "views_history": [[random.randint(0, 5000) for _ in range(30)] for _ in range(3)],
            }
        )
        st.dataframe(
            df,
            column_config={
                "name": "Title",
                "reviews": st.column_config.NumberColumn(
                    "Reviews",
                    help="Total number of reviews",
                    format="%d ⭐",
                ),
                "url": st.column_config.LinkColumn("IMDb page"),
                "views_history": st.column_config.LineChartColumn(
                    "Views (past 30 days)", y_min=0, y_max=5000
                ),
            },
            hide_index=True,
        )

    # Chat input and variables
    with chat:
        st.subheader("Enter in a prompt to the chat")
        prompt = st.chat_input("Say something")
        if prompt:
            st.write(f"You entered the following prompt: :blue[{prompt}]")

    # Displaying Markdown and editor
    with markdown:
        st.subheader("Edit and render Markdown")
        md = st.text_area('Type in your markdown string (without outer quotes)')
        with st.container():
            st.divider()
            st.subheader("Rendered Markdown")
            st.markdown(md)

💡 Containers (st.container()) can be inserted into an app to provide structure for elements you choose to place inside.

Tabs within the container are created using the st.tabs() call. This method accepts a list of tab names as an argument and outputs separate tab objects.

The st.dataframe element accepts the df object as an argument along with a column_config. This configuration dictates how the data frame is displayed on the page.

Strealit Basics Webpage:

Display data using a data frame:

A simple chatbot that allows user to enter prompt:

This Streamlit Basics webpage displays simple data and text information, as well as a chatbot function.

Further use cases could be extended to store data portfolios with an embedded chat features that allows users to ask questions about the data.

4.2 Build an interactive file upload webpage

Refer to below Python code to create an interactive webpage for user to upload file to S3 bucket:

import os
import boto3
import streamlit as st
from streamlit_pdf_viewer import pdf_viewer
from io import BytesIO

# Amazon S3 client
s3 = boto3.client('s3')
bucket_name = os.environ['BUCKET_NAME']

st.set_page_config(layout="wide")

# Streamlit columns
upload_s3, read_s3 = st.columns(2)

# Column 1: Upload to Amazon S3 using Boto3
with upload_s3:
    st.subheader("Upload to Amazon S3")
    obj = st.file_uploader(label=f"Uploading to: :green[{bucket_name}]")
    if obj is not None:
        s3.upload_fileobj(obj, bucket_name, obj.name)

# Column 2: Read from Amazon S3 using Boto3
with read_s3:
    st.subheader("Read from Amazon S3")
    response = s3.list_objects_v2(Bucket=bucket_name)
    object_list = []

    if 'Contents' in response:
        for obj in response['Contents']:
            if not obj['Key'].endswith('/'):
                object_list.append(obj['Key'])
    else:
        st.write(f"S3 bucket is empty")

    selected_obj = st.selectbox(f"Selecting from: :green[{bucket_name}]", object_list, index=None)
    st.caption(f"You selected: :blue[{selected_obj}]")

st.divider()

# Displaying the selected Amazon S3 object
if selected_obj is None:
    st.caption("Please select an object from S3 bucket")
else:
    response = s3.get_object(Bucket=bucket_name, Key=selected_obj)
    body = response['Body'].read()

    # Displaying the object based on the file type
    if selected_obj.endswith(".png") or selected_obj.endswith(".jpg"):
        st.image(BytesIO(body))
    elif selected_obj.endswith(".pdf"):
        pdf_viewer(body)
    else:
        st.write(body.decode('utf-8'))

💡 The boto3 library will create an s3 client that will interact with the S3 service. The custom streamlit_pdf_viewer component and the BytesIO module will aid in rendering selected S3 objects to the page.

Let’s test it with a file upload. In the left column (refer to column 1 in above code), we upload a PitchBook PDF document.

Then we go to S3 to verify if the PDF has been uploaded successfully.

4.3 Build a generative AI image generator

Refer to below Python code to create Amazon Bedrock Titan Image Generator UI:

import boto3
import json
import base64
import streamlit as st
from io import BytesIO

# Amazon Bedrock client
bedrock = boto3.client('bedrock-runtime')
bedrock_model_id = "amazon.titan-image-generator-v1"

# Convert image data to BytesIO object
def decode_image(image_data):
    image_bytes = base64.b64decode(image_data)
    return BytesIO(image_bytes)

# Invoke Bedrock image model to generate image
def generate_image(prompt):
    body = json.dumps(
        {
            "taskType": "TEXT_IMAGE",
            "textToImageParams": {
                "text":prompt
            },
            "imageGenerationConfig": {
                "numberOfImages": 1,
                "quality": "standard",
                "height": 768,
                "width": 768,
                "cfgScale": 8.0,
                "seed": 100             
            }
        }
    )

    response = bedrock.invoke_model(
                modelId="bedrock_model_id",
                accept="application/json", 
                contentType="application/json",
                body=body
            )

    response_body = json.loads(response["body"].read())
    image_data = response_body["images"][0]

    return decode_image(image_data)

# Streamlit UI

with st.container():
    st.header("Amazon Bedrock Titan Image Generator", anchor=False, divider="rainbow")

    input_column, result_column = st.columns(2)

    # Text Input
    with input_column:
        st.subheader("Describe an image", anchor=False)
        prompt_text = st.text_input("Example: Two dogs sharing a bowl of spaghetti", key="prompt")

        # Generate and Clear buttons

        # Clear field function accessing session state
        def clear_field(prompt):
            st.session_state.prompt = prompt

        generate, clear = st.columns(2, gap="small")

        with generate:
            generate_button = st.button("Generate", use_container_width=True)
        # Clear field callback 
        with clear:
            st.button('Clear', on_click=clear_field, args=[''], use_container_width=True)

    # Resulting image column
    with result_column:
        st.subheader("Generated image", anchor=False)
        st.caption('Your image will appear here.')
        if generate_button:
            # Displays spinner + message while executing the generate_image function
            with st.spinner("Generating image..."):
                image = generate_image(prompt_text)
            st.image(image, use_column_width=True)

💡 The generate_image function passes the imageGenerationConfig, taskType, and user prompt to the Bedrock invoke_model method. The method will return a JSON body that is parsed to retrieve the generated image.

The generated image is decoded using the decode_image function and returned.

The clear_field function interacts with the Streamlit session state. It accepts a prompt and updates the value of the st.session_state.prompt. Session state is used to store and persist state that can be manipulated with the use of callback functions. clear_field is a callback function that will get invoked when a user clicks the Clear button.

The result_column checks if the generate_button value is true and calls the st.spinner element to display a temporary message as the generate_image function works in the background.

The resulting image is then passed to a st.image element to render it on the page.

Let’s try the image generator to create an image with entered prompt:

5. Conclusion

In this article, we have

Introduce Streamlit python library to create data app and interactive app with simple minimal codes
Deployed the Streamlit application to an Amazon EC2 instance.
Interacted with each application webpage with its interactive features
Go through the application codes that integrate Amazon Bedrock and Amazon S3

Reference and further readings:

What is streamlit. https://github.com/streamlit/streamlit
What is Amazon Bedrock? https://aws.amazon.com/bedrock/
Amazon Bedrock Developer Experience. https://aws.amazon.com/bedrock/developer-experience/
Quickly build Generative AI applications with Amazon Bedrock https://community.aws/content/2ddby9SeCKALvSz0CWUtx4Q4fPX/amazon-bedrock-quick-start?lang=en

Real-Time User Behavior Insights with Amazon Kinesis and Apache Flink: A Guide to Sessionizing Clickstream Data

Abdullah Paracha — Fri, 19 Jul 2024 19:15:45 +0000

👉🏻 This walkthrough is to stimulate a real world use case of sessionizing the clickstream data using Amazon Managed Service for Apache Flink, and storing sessions in an Amazon DynamoDB Table with AWS Lambda.
Note: Some processes and resources have been pre-configured.

Use Case Scenario
AWS Tools and Resources
Process and Architecture
Step by Step Configuration and Walkthrough
- Starting a Managed Apache Flink Studio Notebook Application
- Connecting to the Virtual Machine using EC2 Instance Connect and Simulating a Real-Time Clickstream
- Sessionizing the Clickstream Data using Amazon Managed Service for Apache Flink
- Storing Sessions in an Amazon DynamoDB Table with AWS Lambda
Conclusion

Streaming data is data that is generated continuously by thousands of data sources, such as log files generated by customers using your mobile or web applications, ecommerce purchases, in-game player activity, information from social networks, etc.. [1]

Sessionizing clickstream data involves grouping user interactions into sessions, where each session represents a sequence of activities by a user within a specific time window. There are many use cases of sessionizing clickstream data, including user behavior analysis, personalized recommendations, performance measurement, and fraud detection, etc..

Sessionizing clickstream data allows for real-time processing and analysis, providing immediate insights into user behavior and enabling prompt actions based on these insights. This real-time capability is essential for applications that require up-to-the-minute data to enhance user experiences and operational efficiency.

In the following use case, we will be stimulating a real-time clickstream data, sessionizing clickstream data using Amazon Managed Service for Apache Flink, and storing sessions data in Amazon DynamoDB table with AWS Lambda.

1. Use Case Scenario:

XShop, an online retail company, wants to enhance its customer experience by gaining deeper insights into user behavior on its website. They aim to achieve this by analyzing clickstream data in real-time to understand user interactions, optimize marketing strategies, and improve site performance.

As the data engineering teamof Xshop, we are targeting to build a solution on AWS to collect clickstream data and store session data for downstream processing and analysis.

2. AWS tools and resources:

Amazon Managed Service for Apache Flink is a fully managed service that enables you to perform analysis using SQL and other tools on streaming data in real time. There are no servers and clusters to manage, and there is no compute and storage infrastructure to set up. You pay only for the resources you use. [2]

Use cases for Amazon Managed Service for Apache Flink include:

Streaming extract, transform, and load (ETL) jobs
Create real-time log analysis
Ad-tech and digital marketing analysis
Perform stateful processing

Amazon DynamoDB is a fully managed NoSQL database service that provides fast and predictable performance with seamless scalability. You can use Amazon DynamoDB to create a database table that can store and retrieve any amount of data, and serve any level of request traffic. [3]

AWS Lambda provides compute service that runs your code in response to events and automatically manages the compute resources. You can combine AWS Lambda with other AWS services, preprocess data before feeding data into machine learning mode, or execute code in response to trigger changes in data. [4]

In this use case, we will:

Use Amazon Kinesis and Amazon Managed Service for Apache Flink to analyze clickstream data
Create an AWS Lambda function that adds records to an Amazon DynamoDB table
Configure Amazon Kinesis to send results to your AWS Lambda function.

3. Process & Architecture:

As the data engineering team, we will use Amazon Kinesis and Amazon Managed Service for Apache Flink to sessionize sample clickstream data and output it to DynamoDB using an AWS Lambda function.

4. Step-by-Step configuration & walkthrough:

4.1 Starting a Managed Apache Flink Studio Notebook Application

Currently, we have 2 data streams in Kinesis:

(1) click-stream data, which will ingest incoming click events from a data source

(2) session-stream data, which will ingest sessionized click stream data to be consumed by an AWS Lambda function

In Managed Apache Flink, create a studio notebook to process the click events and sessionize them using Structured Query Language (SQL).

4.2 Connecting to the Virtual Machine using EC2 Instance Connect and Simulating a Real-Time Clickstream:

Connect to an EC2 instance

Create a JSON file for a click event in the shell



echo '{
  "user_id": "$USER_ID",
  "event_timestamp": "$EVENT_TIMESTAMP",
  "event_name": "$EVENT_NAME",
  "event_type": "click",
  "device_type": "desktop"
}' > click.json

Hint:
💡 - built-in Bash command echo: print a JSON template

Bash shell called redirection: redirects the output of the echo command to a file (creating it if doesn't exist) called click.json.

To put records into Kinesis and simulate a clickstream



DATA_STREAM="click-stream"
USER_IDS=(user1 user2 user3)
EVENTS=(checkout search category detail navigate)
for i in $(seq 1 5000); do
    echo "Iteration: ${i}"
    export USER_ID="${USER_IDS[RANDOM%${#USER_IDS[@]}]}";
    export EVENT_NAME="${EVENTS[RANDOM%${#EVENTS[@]}]}";
    export EVENT_TIMESTAMP=$(($(date +%s) * 1000))
    JSON=$(cat click.json | envsubst)
    echo $JSON
    aws kinesis put-record --stream-name $DATA_STREAM --data "${JSON}" --partition-key 1 --region us-west-2
    session_interval=15
    click_interval=2
    if ! (($i%60)); then
        echo "Sleeping for ${session_interval} seconds" && sleep ${session_interval}
    else
        echo "Sleeping for ${click_interval} second(s)" && sleep ${click_interval}
    fi
done

Hint:
💡 - Setup of sample user ids and event types at the beginning

A loop that will execute 5000 times and a sleep statement
Statements that randomly select a user id and an event type, and assign them along with the current timestamp to variables
A statement that uses the envsubst command to substitute defined environment variables in the JSON template
A statement invoking the AWS command-line interface tool, putting the templated JSON record into the Kinesis Data Stream
A condition at the end of the loop that either sleeps for a few seconds or, periodically for longer, simulating the end of a session

The templated JSON and also the JSON response from Kinesis for each record put into the Data Stream

Select the studio notebook, open in Apache Zeppelin and create a new notebook

create tables for the Kinesis Data Streams



%flink.ssql(type = update)
DROP TABLE IF EXISTS click_stream;
CREATE TABLE click_stream (
          user_id STRING,
          event_timestamp BIGINT,
          event_name STRING,
          event_type STRING,
          device_type STRING,
          event_time AS TO_TIMESTAMP(FROM_UNIXTIME(event_timestamp/1000)),
            WATERMARK FOR event_time AS event_time - INTERVAL '5' SECOND
) WITH (
  'connector' = 'kinesis',
  'stream' = 'click-stream',
  'aws.region' = 'us-west-2',
  'scan.startup.mode' = 'latest-offset',
  'format' = 'json'
);
DROP TABLE IF EXISTS session_stream;
CREATE TABLE session_stream (
          user_id STRING,
          session_timestamp BIGINT,
          session_time TIMESTAMP(3),
          session_id STRING,
          event_type STRING,
          device_type STRING
) PARTITIONED BY (user_id) WITH (
  'connector' = 'kinesis',
  'stream' = 'session-stream',
  'aws.region' = 'us-west-2',
  'format' = 'json'
);

Hint:
💡 - in WITH clause for each table, specify configuration options to configure a connector. In this case, the connector is kinesis denoting that these tables are to be backed with AWS Kinesis Data Streams. The value of the stream option for both match the names of the Kinesis Data Streams you observed earlier.

view data from the stream



%flink.ssql
SELECT * FROM click_stream LIMIT 10;

The data is being put into the click-stream Kinesis data stream by the bash code ran earlier on the Amazon EC2 instance.

4.3 Sessionizing the Clickstream Data using Amazon Managed Service for Apache Flink:

Amazon Managed Service for Apache Flink allows you to continuously run SQL on streaming data, processing the data in real-time, and sending the results to many different destinations.

We will use SQL to determine the beginning of a new session for a user in your simulated click stream.

Identify events that start a new session



%flink.ssql
SELECT 
  *, 
  CASE WHEN event_timestamp - LAG(event_timestamp) OVER (
    PARTITION BY user_id 
    ORDER BY 
      event_time
  ) >= (10 * 1000) THEN 1 ELSE 0 END as new_session 
FROM 
  click_stream;

💡 This SQL query uses the LAG SQL function to compare a record with a previous record. The PARTITION BY user_id clause means it will restrict comparisons to records where the user_id is the same.

The ORDER BY event_time clause means records will be compared in ascending order.

When the difference between the event_timestamp field of two records is greater than ten seconds the new_session field will be 1, denoting the start of a new session, When new_ession is 0, it denotes the event is a continuation of a session.

Notice that the comparison multiplies ten by a thousand. This is because the event_timestamp field is Unix time and includes milliseconds. So multiplying by a thousand is required to get ten seconds in milliseconds.

To get only the new session boundaries



%flink.ssql
SELECT 
  *, 
  user_id || '_' || CAST(
    SUM(new_session) OVER (
      PARTITION BY user_id 
      ORDER BY 
        event_time
    ) AS STRING
  ) AS session_id 
FROM 
  (
    SELECT 
      *, 
      CASE WHEN event_timestamp - LAG(event_timestamp) OVER (
        PARTITION BY user_id 
        ORDER BY 
          event_time
      ) >= (10 * 1000) THEN 1 ELSE 0 END as new_session 
    FROM 
      click_stream
  ) 
WHERE 
  new_session = 1

This query wraps the previous query with an insert statement that puts records into the session_stream table. Because this query produces no output and is expected to be long-running, you will see a Duration displayed at the bottom of the paragraph that shows how long the query has been running.

💡 Amazon Managed Service for Apache Flink can be customized using Java JAR files you provide yourself. This means you could use Java code to load or deliver data to and from any network-accessible system. As an example, data could be loaded from a legacy API that has a Java library available.

4.4 Storing Sessions in an Amazon DynamoDB Table with AWS Lambda:

In AWS Lambda dashboard, in source code session, update source code:



from __future__ import print_function
import boto3
import base64
from json import loads
dynamodb_client = boto3.client('dynamodb')
table_name = "Table_Name"   #change to your table name
def lambda_handler(event, context):
    records = event['Records']
    output = []
    success = 0
    failure = 0
    for record in records:
        try:
            payload = base64.b64decode(record['kinesis']['data'])
            data_item = loads(payload)
            ddb_item = {
                'session_id': {'S': data_item['session_id']},
                'session_time': {'S': data_item['session_time']},
                'user_id': {'S': data_item['user_id']}
            }
            dynamodb_client.put_item(TableName=table_name, Item=ddb_item)
            success += 1
            output.append({'recordId': record['eventID'], 'result': 'Ok'})
        except Exception:
            failure += 1
            output.append({'recordId': record['eventID'], 'result': 'DeliveryFailed'})
    print('Successfully delivered {0} records, failed to deliver {1} records'.format(success, failure))
    return {'records': output}

Configure the session data stream events as an event source for this function

Trigger configuration:

Select a source: Select Kinesis
Kinesis stream: Select session-stream

In DynamoDB, go to Explore Items section

We are processing the records in the session-stream Kinesis data stream with an AWS Lambda function and putting the records into a DynamoDB table.

In this lab, you started a Kinesis Data Analytics application, simulated a real-time click-stream, sessionized the click-stream using your Kinesis Data Analytics application, and finally, you processed the output of your Kinesis Data Analytics stream with a Lambda function that sends the data to a DynamoDB table.

5. Conclusion:

In this walkthrough, we have:

Started a Kinesis Data Analytics application
Simulated a real-time clickstream
Sessionized the clickstream using the Kinesis Data Analytics application
Processed the output of the Kinesis Data Analytics stream with a Lambda function
Sent the processed data to a DynamoDB table using the Lambda function

Reference and Further Readings:

[1] What is streaming data? https://aws.amazon.com/streaming-data/#:~:text=Streaming data includes a wide,devices or instrumentation in data

[2] Amazon Managed Service for Apache Flink https://aws.amazon.com/managed-service-apache-flink/

[3] Amazon DynamoDB https://aws.amazon.com/managed-service-apache-flink/

[4] AWS Lambda https://aws.amazon.com/pm/lambda/?gclid=Cj0KCQjw-uK0BhC0ARIsANQtgGP7fWsSMGCsLXcj3w6sDSsTJpmPc4m0SmMWTzgw_xFKWQcPuUdh1IMaAtKmEALw_wcB&trk=d87368f2-b0ac-4e30-804b-b10e2d25d291&sc_channel=ps&ef_id=Cj0KCQjw-uK0BhC0ARIsANQtgGP7fWsSMGCsLXcj3w6sDSsTJpmPc4m0SmMWTzgw_xFKWQcPuUdh1IMaAtKmEALw_wcB:G:s&s_kwcid=AL!4422!3!651612781100!e!!g!!aws lambda!19836398320!150095228874

AWS PartyRock Event Planner Assistant

Abdullah Paracha — Sat, 09 Mar 2024 15:29:46 +0000

Concept: Event Planner Assistant
The "Event Planner Assistant" is a generative AI application designed to help users plan their virtual parties. It uses AI to suggest party themes, music playlists, and interactive activities tailored to the host's preferences and the nature of the event.

Key Features
Theme Suggestion: Generates creative party themes based on the type of event and user preferences.
Playlist Creation: Recommends music playlists that match the party's theme and mood.
Activity Ideas: Proposes interactive games and activities suitable for the event.

Building the Application on AWS PartyRock
Let's imagine the steps and simple code snippets that might be involved in creating such an application on AWS PartyRock, using a mix of pseudocode and descriptions to illustrate the process.

Step 1: Set Up User Input Interface
First, we create a simple user interface where the host can input details about the party, such as the type of event, preferred music genres, and any specific themes or activities they're interested in.

InputForm:
- EventType: [Birthday, Graduation, Casual Get-together, ...]
- MusicPreferences: [Pop, Rock, Electronic, Jazz, ...]
- ThemePreferences: [Input Text]
- ActivityInterest: [Games, Quizzes, Dance, ...]

Step 2: Theme Suggestion Logic
Using a predefined AI model, the application generates a list of party themes based on the event type and theme preferences.

def generate_theme(event_type, theme_preferences):
    # Imagine calling an AI model here
    suggested_themes = AIModel.generate_themes(event_type, theme_preferences)
    return suggested_themes

Step 3: Playlist Creation Logic
The application uses another AI model to curate a playlist based on the party's theme and the host's music preferences.

def create_playlist(theme, music_preferences):
    # Imagine calling an AI model here
    playlist_links = AIModel.create_playlist(theme, music_preferences)
    return playlist_links

Step 4: Activity Ideas Generation
Finally, the application suggests interactive activities and games that match the chosen theme and the host's interest in activities.

def suggest_activities(theme, activity_interest):
    # Imagine calling an AI model here
    activities = AIModel.suggest_activities(theme, activity_interest)
    return activities

Step 5: Compile and Present Suggestions
The application compiles the suggestions from each step and presents them to the user in an interactive format, possibly with options to customize further or explore alternatives.

user_input = gather_user_input()
theme_suggestions = generate_theme(user_input.eventType, user_input.themePreferences)
playlist = create_playlist(theme_suggestions[0], user_input.musicPreferences)  # Assume first theme
activities = suggest_activities(theme_suggestions[0], user_input.activityInterest)

present_to_user(theme_suggestions, playlist, activities)

Conclusion
While this example uses pseudocode and assumes the existence of AI models and a platform like AWS PartyRock, it illustrates how one could conceptualize and design an AI-driven application for event planning. The real power of generative AI applications lies in their ability to personalize and enhance experiences, offering unique and tailored suggestions that cater to individual preferences and needs.

Building Serverless Systems - Serverlesspresso

Abdullah Paracha — Sat, 20 Jan 2024 13:36:44 +0000

In the ever-evolving world of cloud computing, serverless architecture has emerged as a game changer, offering a new paradigm for building and deploying applications. This blog delves into the concept of serverless systems, focusing on a unique approach we've named "Serverlesspresso."

What is Serverless Computing?
Serverless computing is a cloud-computing execution model where the cloud provider dynamically manages the allocation and provisioning of servers. Essentially, it allows developers to build and run applications and services without worrying about the underlying infrastructure.

Benefits of Serverless Computing
Cost-Effective: You only pay for the resources you use.
Scalability: Automatically scales with your application's needs.
Maintenance-Free: No need to manage servers or infrastructure.
Faster Time-to-Market: Quicker deployment and update processes.

Introducing Serverlesspresso
Serverlesspresso is our coined term for building serverless systems that are as swift and efficient as your morning espresso. It's about creating systems that are not only quick to deploy but also robust and scalable, much like a well-brewed cup of coffee.

Key Components of Serverlesspresso
Event-Driven Architecture: Responds to events in real-time, like a barista responding to an order.
Microservices-Based: Like different coffee blends, each service performs a specific function.
API Gateway: The counter where you place your coffee order, handling and routing requests.
FaaS (Function as a Service): The coffee machine, executing specific functions as needed.

Real-World Example: A Serverless Coffee Shop
Imagine a coffee shop where each part of your coffee-making process is a separate, serverless function: one for grinding beans, another for boiling water, and so on. This is Serverlesspresso in action, where each function collaborates to deliver your perfect cup of coffee, or in tech terms, a seamless user experience.

Advantages in this Scenario
Efficiency: Each part works independently and only when needed.
Scalability: Can handle one cup or a hundred without additional setup.
Cost-Effectiveness: Only utilizes resources for each specific task.
Challenges and Considerations
While serverless systems offer numerous benefits, they're not without challenges:

Cold Starts: The initial latency when a function is first invoked.
Debugging and Monitoring: More complex in a distributed environment.
Vendor Lock-in: Dependence on specific cloud providers.

The Future of Serverlesspresso
The serverless world is rapidly evolving, with increasing focus on reducing cold starts, enhancing monitoring tools, and mitigating vendor lock-in issues. The future looks promising, with serverless becoming more adaptable and suited for a broader range of applications.

Conclusion
Serverlesspresso symbolizes an efficient, scalable, and cost-effective approach to building serverless systems. It's an approach where the architecture is as streamlined and satisfying as your morning espresso. As we move forward, serverless will continue to revolutionize how we think about and deploy applications in the cloud.

Serverlesspresso isn't just a concept; it's the future of cloud computing, brewing one efficient, scalable solution at a time. Let's embrace this change and watch as our digital world becomes as efficient as our favorite coffee shop!

AWS Cost Management: Strategies and Examples

Abdullah Paracha — Sat, 20 Jan 2024 13:27:40 +0000

Managing costs effectively in Amazon Web Services (AWS) is crucial for maximizing the benefits of cloud computing while maintaining budget control. AWS offers a range of tools and features designed to help users monitor and optimize their spending. In this blog, we'll explore some of these tools and discuss strategies for cost management, including real-world examples and visual aids.

Understanding AWS Pricing
AWS follows a pay-as-you-go pricing model, meaning you pay only for the resources you use. This model offers flexibility but also requires careful monitoring and management to avoid unexpected costs.

Key Components of AWS Pricing:
Compute Resources: Charges for EC2 instances depend on the type, size, and region.
Storage Costs: S3 and other storage services charge based on the amount of data stored and accessed.
Data Transfer: AWS charges for data transfer in and out of their services, especially across regions.
Additional Services: Services like RDS, Lambda, and others have their unique pricing structures.
Example:
Consider a scenario where you're running an EC2 instance for web hosting, using S3 for storage, and RDS for database services. The costs will include EC2 instance hours, data storage, and database instance costs.

Cost Management Tools in AWS
AWS provides several tools to help manage and track your spending:

AWS Cost Explorer: Allows you to visualize and manage AWS spending and usage over time.

AWS Budgets: Enables setting custom budgets to track service costs and usage.

AWS Trusted Advisor: Offers recommendations for cost optimization.

Effective Cost Management Strategies
Right-Sizing: Regularly review and adjust your AWS resources to match your actual needs.

Reserved Instances and Savings Plans: Purchase reserved instances or savings plans for services you use continuously to get significant discounts.

Monitor and Optimize Usage: Use AWS Cost Explorer and Trusted Advisor to identify underutilized resources.

Implement Cost Allocation Tags: Use tags to organize and track costs by project, department, or environment.

Automate to Save: Utilize AWS Lambda and other automation tools to shut down idle resources.

Real-World Example: E-commerce Platform Cost Optimization
An e-commerce company initially faced high AWS costs due to over-provisioned resources and inefficient data storage. By implementing the following steps, they reduced their AWS bill by 30%:

Switched to Reserved Instances for their stable EC2 needs.
Employed Amazon S3 lifecycle policies to archive older data to less expensive storage classes.
Set up AWS Budgets to monitor and alert for overages.
This approach demonstrates the impact of strategic cost management in AWS.

Conclusion
Effective cost management in AWS is a dynamic and ongoing process. By leveraging AWS's cost management tools and adopting best practices, businesses can optimize their cloud spending without sacrificing performance or scalability. Regular reviews and adjustments, aligned with business needs, are key to maximizing the benefits of AWS while controlling costs.

AWS Pinpoint with Lambda Function

Abdullah Paracha — Tue, 07 Mar 2023 08:55:01 +0000

AWS Pinpoint is a cloud-based marketing automation service offered by Amazon Web Services (AWS). It enables businesses to engage with their customers across multiple channels including email, SMS, push notifications, and voice messages. With Pinpoint, businesses can create targeted campaigns, track user engagement, and analyze campaign performance.

Overall, AWS Pinpoint is a powerful marketing automation tool that can help businesses to engage with their customers effectively and drive business growth.

Pinpoint offers various features such as:

Audience segmentation: Pinpoint enables businesses to segment their audience based on various criteria such as demographics, location, behavior, and user attributes. This helps businesses to create targeted campaigns that are personalized and relevant to their audience.
Multi-channel messaging: Pinpoint supports multiple messaging channels such as email, SMS, push notifications, and voice messages. This allows businesses to reach their customers on their preferred channels.
Campaign management: Pinpoint provides a user-friendly interface for creating, scheduling, and managing campaigns. It also offers A/B testing and analytics to help businesses optimize their campaigns.
Analytics and reporting: Pinpoint provides detailed analytics and reporting on campaign performance, user engagement, and customer behavior. This helps businesses to understand their customers better and make data-driven decisions.
Integrations: Pinpoint integrates with other AWS services such as Amazon S3, Amazon Kinesis, and Amazon Redshift. It also supports third-party integrations with popular marketing automation tools such as Salesforce, Marketo, and Hubspot.

Set Up Email
Type an email address that you will use to send email. For example, you can use your personal email address, or your work email address. Click the Verify button and keep this page open

Ensure you have one verified email addresses by following the process in the Configure Email section
Download the sample csv file and open it in a text editor on your computer.
Add/edit users using the wildcard local parts notation you saw when you sent a test message.

Edit the Address column in the CSV file to reflect one of the email addresses you verified in the Configure Email section. Example: If you verified abdullah.qadoos@systemsltd.com

Import a Segment
To import a segment, you define the endpoints or user IDs that belong to your segment in a comma-separated values (CSV) or JSON file. Then you import the file into Amazon Pinpoint to create the segment.

Choose Create a segment
Choose Import a segment
Choose Upload files from your computer and drop the CSV file you just saved in the last step into the Drop files here area, or click Choose files and browse to your CSV import file.
Provide a descriptive name for your segment. Click Create segment to complete the import.
After a couple of seconds your segment will be imported. Click on the segment name to view its details.
To view the new endpoint created, you can either export the segment or navigate to the AWS CloudShell console and wait till the environment is created. Replace the Amazon Pinpoint application id from the command below then paste it in the AWS CloudShell terminal and press Enter. This should return you a response containing all the information about this endpoint

aws pinpoint get-endpoint --application-id <place Pinpoint application_id here> --endpoint-id 111

Create & execute an AWS Lambda Function
You can create or update an Amazon Pinpoint endpoint via the Amazon Pinpoint REST API, Amplify or one of the AWS SDKs. For this example you will use the AWS Python SDK Boto3 and AWS Lambda function to host and execute the code.

Create a new AWS Lambda Function
Select Author from scratch, give a Function name, for runtime select Python 3.8 and click on Create function
Code Source

import boto3

client = boto3.client('pinpoint')

def lambda_handler(event, context):

    application_id = event['application_id']
    first_name = event['first_name']
    email_address = event['email_address']
    endpoint_id = event['endpoint_id']
    user_id = event['user_id']  
    age = event['age']
    interests = event['interests']

    response = client.update_endpoint(
    ApplicationId=application_id,
    EndpointId= endpoint_id,
    EndpointRequest={
        'Address': email_address,
        'ChannelType': 'EMAIL',
        'Metrics': {
            'age': age
        },
        'OptOut': 'NONE',
        'User': {
            'UserAttributes': {
                'FirstName': [
                    first_name,
                ],
                'interests': [
                    interests
                ]
            },
            'UserId': user_id
        }
    }
    )

    return response

Now that the code is ready, you need to allow AWS Lambda Function to perform the update_endpoint operation to your Amazon Pinpoint Project. To do this, you will add an IAM policy to the AWS Lambda Function role. Navigate to the tab Configuration > Permissions and click on the Role name link. This will take you to the Identity and Access Management (IAM) console and specifically on the Summary page of that role
In the Permissions tab select Add permissions and Create inline policy
Navigate to the JSON tab and paste the JSON seen below. Replace the values for AWS-REGION with the AWS region that you have created the Amazon Pinpoint project and AWS-ACCOUNT-ID and select Review policy. To find your AWS-ACCOUNT-ID visit this page

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "UpdateEndpoint",
            "Effect": "Allow",
            "Action": [
                "mobiletargeting:UpdateEndpoint*"
             ],
            "Resource": "arn:aws:mobiletargeting:AWS-REGION:AWS-ACCOUNT-ID:*"
        }
        ]
}

Paste workshop_lambda_policy as the policy name and click Create policy. Once the new policy is created, you should be able to see it under the Permissions policies list in the IAM role summary page
Return to the AWS Lambda function Code tab select the dropwdown next to the orange Test button and choose Configure test event. Using this AWS Lambda feature you can create a dummy payload to test your Lambda function. Copy the JSON below and paste it under Event JSON section. Select hello-world as Event template and type test as Event name

Replace the --- for application_id with your Amazon Pinpoint project id, which you can find by navigating to the Amazon Pinpoint console under the column Project ID
Replace the --- for email_address with the verfied email address from the lab Configure email channel. If you verified abdullah.qadoos@systemsltd.com
Scroll down and select Save

{
    "application_id" : "---",
    "first_name" : "Jake",
    "email_address" : "---",
    "endpoint_id" : "222",
    "user_id" : "userid2",
    "age": 35,
    "interests": "shirts"
}

Return to the AWS Lambda function Code tab and select Deploy, which is located next to the Test button.
To execute the code, click on the Test button. Once the code is executed it will return the response from Amazon Pinpoint, which will contain information about the update_endpoint request.
To view the new endpoint created, navigate to the AWS CloudShell console and wait till the environment is created. Replace the Amazon Pinpoint application-id in the command below, copy & paste it in the AWS CloudShell terminal and press Enter using your keyboard. This should return you a response containing all the information about this endpoint

aws pinpoint get-endpoint --application-id <place Pinpoint application_id here> --endpoint-id 222

AWS 5G: Powering Next-Generation Applications

Abdullah Paracha — Thu, 02 Feb 2023 11:36:59 +0000

The arrival of 5G networks is revolutionizing the way we interact with technology, bringing faster speeds, lower latency, and greater connectivity to a wide range of use cases, from video and audio streaming to gaming and the Internet of Things (IoT). To support the growing demand for 5G services, AWS has introduced AWS Wavelength, a fully managed service that enables developers to build and deploy 5G applications with low latency and high throughput.

Key Features of AWS 5G

Low Latency: AWS Wavelength provides a high-speed, low-latency connection to the AWS cloud, enabling real-time data processing for use cases that require fast and responsive connections.

Integration with AWS Services: AWS Wavelength integrates with other AWS services, such as Amazon EC2 and Amazon S3, to provide a complete solution for 5G application development. This enables developers to build and deploy 5G applications with the benefits of the AWS cloud, including security, scalability, and reliability.

Support for 5G Use Cases: AWS Wavelength is designed to support a wide range of 5G use cases, including augmented and virtual reality, gaming, and autonomous vehicles.

Benefits of AWS 5G

Fast and Responsive Applications: With its low latency and high throughput, AWS 5G enables the development of fast and responsive applications that can deliver a seamless user experience.

Cost-Effective: AWS Wavelength is a fully managed service that runs on AWS infrastructure, providing a cost-effective solution for 5G application development.

Scalable and Reliable: AWS 5G is designed to be scalable and reliable, making it possible to build and deploy applications that can grow and change with the needs of the business.

In conclusion, AWS 5G is a powerful tool for developers and organizations looking to take advantage of the benefits of 5G networks and the cloud. With its low latency, high throughput, and integration with other AWS services, AWS Wavelength provides a complete solution for 5G application development, enabling organizations to quickly and easily build and deploy 5G applications.

Create and Assume Roles in AWS

Abdullah Paracha — Tue, 17 Jan 2023 13:19:31 +0000

In this blog the objective in the AWS environment, utilizing policies and roles in the IAM console to restrict access to AWS resources and conclude by assuming a role and ensuring our policies are correct and that we have completed all objectives.

Image: ACloudGuru

Create the Correct S3 Restricted Policies and Roles

Create the S3RestrictedPolicy IAM policy. Ensure only the appconfig buckets are accessible.
- Select the S3 service and all S3 actions
- Select all resources except bucket
- Add the appconfig bucket names to the policy
Create the S3RestrictedRole IAM role.
- Set the trusted entity to another AWS account
- Add your account ID
- For permissions, select the S3RestrictedPolicy
Revoke the AmazonS3FullAccess access policy from the developergroup.
Attach the S3RestrictedPolicy to the dev1 user.

Configure IAM So the dev3 User Can Assume the Role

Create the AssumeS3Policy IAM policy.
- Select the STS service
- Select AssumeRole under the write options
- Add the S3RestrictedRole
Attach the AssumeS3Policy to the dev3 user.
Assume the S3RestrictedRole as the dev3 user.
- Log in as the dev3 user
- Switch roles to the S3RestrictedRole
- Verify access in S3

Docker and Kubernetes

Abdullah Paracha — Wed, 27 Apr 2022 12:01:53 +0000

Docker is a technology for creating and running containers, while Kubernetes is a container orchestration technology. Let's explore how Docker and Kubernetes align and how they support cloud-native computing.

Docker

Docker is a containerization platform that is used to create and run software containers. A container is a collection of one or more processes, organized under a single name and identifier. A container is isolated from the other processes running within a computing environment, be it a physical computer or a virtual machine. Basically, it’s a toolkit that makes it easier, safer and faster for developers to build, deploy and manage containers.

• Docker Engine: The runtime environment that allows
developers to build and run containers.
• Docker File: A simple text file that defines everything needed to build a Docker container image, such as OS network specifications and file locations. It’s essentially a list of commands that Docker Engine will run in order to assemble the image.
• Docker Compose: A tool for defining and running multi-container applications. It creates a YAML file to specify which services are included in the application and can deploy and run containers with a single command via the Docker CLI.

Other Docker API features include the ability to automatically track and roll back container images, use existing containers as base images for building new containers and build containers based on application source code.

Kubernetes

Kubernetes is an open-source container orchestration platform for scheduling and automating the deployment, management and scaling of containerized applications. Containers operate in a multiple container architecture called a cluster. A Kubernetes cluster includes a container designated as a master node that schedules workloads for the rest of the containers or worker nodes in the cluster. The master node determines where to host applications (or Docker containers), decides how to put them together and manages their orchestration. By grouping containers that make up an application into clusters, Kubernetes facilitates service discovery and enables management of high volumes of containers throughout their lifecycles.
Key Kubernetes functions include the following:

• Deployment: Schedules and automates container deployment across multiple compute nodes, which can be VMs
• Service Discovery and Load Balancing: Exposes a container on the internet and employs load balancing when traffic spikes occur to maintain stability.
• Auto-scaling features: Automatically starts up new containers to handle heavy loads, whether based on CPU usage, memory thresholds or custom metrics.
• Self-healing capabilities: Restarts, replaces or reschedules containers when they fail or when nodes die, and kills containers that don’t respond to user-defined health checks.
• Automated rollouts and rollbacks: Roll out application changes and monitors application health for any issues, rolling back changes if something goes wrong.

Although Kubernetes and Docker are distinct technologies, they are highly complementary and make a powerful combination. Docker provides the containerization piece, enabling developers to easily package applications into small, isolated containers via the command line. Developers can then run those applications across their IT environment, without having to worry about compatibility issues. If an application runs on a single node during testing, it will run anywhere.
When demand surges, Kubernetes provides orchestration of Docker containers, scheduling and automatically deploying them across IT environments to ensure high availability. In addition to running containers, Kubernetes provides the benefits of load balancing, self-healing and automated rollouts and rollbacks. Plus, it has a graphical user interface for ease of use.