DEV Community: Amit Kayal

API Gateway as Websocket

Amit Kayal — Tue, 21 Jan 2025 07:49:42 +0000

API Gateway as websocket

API Gateway as WS Components

Websocket provides bidirectional session aware communication between caller and receiver and a crucial component for realtime application.

Setup API Gateway for WebSocket
- Create a WebSocket API in the Amazon API Gateway console or through IAC.
- Define the WebSocket API route selection expression. Routes here are simply like a bridge to connections e.g.,
  - $request.body.action.
  - Define the following WebSocket routes:
  - $connect: Triggered when a client establishes a connection.
  - $disconnect: Triggered when a client disconnects.
  - Custom routes, e.g., sendMessage, to handle specific actions.
Create an Integration with AWS Lambda
- For each route ($connect, $disconnect, custom routes), integrate a Lambda function to handle the respective logic.
- Use the Lambda function's handler to process:
  - $connect: Store the connection in DynamoDB.
  - $disconnect: Remove the connection from DynamoDB.
  - Custom routes: Process the message and forward it to SQS.
DynamoDB for Connection Management
- Create a DynamoDB table to store:
  - Connection ID (Primary Key).
  - Session ID or other metadata for grouping connections.
- This table allows tracking active WebSocket connections for broadcasting messages.
Configure SQS for Message Queue
- Use an SQS FIFO queue for guaranteed order and deduplication.
- Messages processed in Lambda (custom routes) are sent to SQS for downstream services.
IAM Roles and Permissions
- Assign an IAM role to the API Gateway to invoke the integrated Lambda functions.
- Grant Lambda permissions to read/write from DynamoDB and send messages to SQS.
Client Connection and Messaging
- Use WebSocket-compatible libraries (e.g., ws in Node.js or WebSocket API in browsers) to:
- Establish a WebSocket connection to the API Gateway endpoint.
- Send and receive messages using the WebSocket protocol.

Architecture of Websocket mechanism

WebSocket Client:
- Initiates WebSocket connection and communicates via send() and onmessage().
API Gateway (WebSocket API):
- Manages WebSocket connections and invokes Lambda functions for defined routes.
Route Integration (Lambda Functions):
Every route should have an integration. There are 3 types — Mock, HTTP and Lambda.
- $connect: Adds connection metadata to DynamoDB.
- $disconnect: Removes connection metadata from DynamoDB.
- $default route: selected when route cant be evaluated against message
- Custom Routes: Processes messages to invoke integration based on message content and forwards them to SQS.
DynamoDB:
- Maintains active connection records, including connectionId and associated metadata.
SQS FIFO Queue:
- Queues messages for downstream processing, ensuring delivery order and deduplication.
Downstream Services:
- Processes messages from SQS and performs actions like notifications, data updates, or storage.

Security

Authentication and Authorization

Custom Authorizer (Lambda Authorizer)
It can only be used for the $connect route.
- Create a Lambda Authorizer to validate custom tokens or headers sent during connection attempts.
- Example:
  - Validate a JWT token from an identity provider (e.g., Cognito, Auth0).
  - Check the token against allowed users or roles.
Amazon Cognito:
- Use Amazon Cognito for user authentication.
- Configure API Gateway to use Cognito to validate tokens in connection requests.
- Best suited for applications with user pools.

Secure WebSocket Connections

Always use the secure WebSocket protocol (wss://). API Gateway enforces HTTPS/TLS, ensuring encrypted communication.
Associate a custom domain with API Gateway WebSocket endpoint. We should AWS Certificate Manager (ACM) to manage SSL/TLS certificates.

IP Whitelisting and Blacklisting

IP Whitelisting and Blacklisting: We should Attach AWS WAF to API Gateway and Block/allow requests based on IP addresses or CIDR ranges. we should also use rate limit to protect from DDoS attack ### API Gateway Throttling
We can Set rate and burst limits on API Gateway routes to limit the number of connections per client.
We can create API keys and associate them with usage plan and then we Limit the number of allowed requests per API key

Environment-based Access Control:

We should always use distinct stages (e.g., dev, prod) and restrict connections to the production API through IP rules.

Tools to test

There are following tools which we can explore to test websocket.

Piesocket
Postman

S3 table & S3 Metadata table

Amit Kayal — Mon, 09 Dec 2024 18:26:23 +0000

Open table format and its architecture

OpenTable formats, such as Apache Iceberg, Apache Hudi, and Delta Lake, have gained popularity in the data analytics mainly because:

ACID Transactions: OpenTable formats (e.g., Apache Iceberg, Delta Lake) ensure reliable and consistent data updates, even with concurrent access.
Schema Evolution: They allow seamless updates to schemas without disrupting existing pipelines, simplifying data management. metadata tracks the changes to the dataset. The files held in the Data layer are captured by the metadata files held in the Metadata layer. As the files change, the metadata files attached to them track these changes.
Optimized Queries: Partitioning and indexing enable faster queries by scanning only relevant data, improving performance and cost-efficiency.
Time Travel: Users can access historical versions of data for debugging, compliance, or analytics.
Interoperability: These formats integrate seamlessly with big data tools like Spark, Flink, and Presto, making them versatile and widely adopted.

Open file format

S3 table

Key Features

Amazon S3 Table is optimized for analytics workloads. It is designed to continuously enhance query performance and reduce storage costs for tabular data. This solution looks very promising if you are working with LakeHouse architecture. It’s a new type of bucket that organizes tables as sub-resources.
A new bucket type s3 table has been introduced to support this. As liked any other aws resoyrce, it has ARN, can take resource policy and as an unique feature it has dedicated endpoint.

S3 Tables are intended explicitly for storing data in a tabular format, such as daily purchase transactions, streaming sensor data, or ad impressions. This data is organized into columns and rows like a database table.
Table buckets support storing tables in the Apache Iceberg format. You can query these tables using standard SQL in query engines that support Iceberg.
Read/write allowed on datafiles and metadata files. Delete and update not allowed to save data integrity.
Compatible query engines include Amazon Athena, Amazon Redshift, and Apache Spark.
S3 Table automatically performs maintenance tasks like compaction and snapshot management to optimize your tables for querying, including removing unreferenced files.
S3 Table offers access management for both table and bucket
Fully managed apache icebarg tables in S3
It supports automatic compaction of underlying files to improve query performance and tune then further for better latency.

S3 Table buckets namespace

Namespace logically groups related s3 table together and thus allowing us to have greater control based on namespace of s3 tables. It helps us for following:

logical segmentation of data and multi tenancy
- supporting of multi tenancy by having separate namespace. Supports compliance with data isolation requirements in regulated industries.
- separate tables based on application, project etc
prevent naming conflicts
- Each namespace acts like a "container," allowing tables with the same name in different namespaces without conflicts.
Better Access Control
- Policies can grant or restrict access to specific namespaces, ensuring data security and compliance. It also reduces the risk of unauthorized access to unrelated tables in the same bucket.
Easy data management
- Makes our life easier to query, update, or delete related tables in bulk.
- Makes easy metadata management for tables grouped under a namespace.
Advanced workflows based on namespace
- It helps to simplify automation for data pipelines or real-time analytics applications.

S3 table opertaion & management

Table Operation
They are quite similar to CRUD operation.

list tables
create tables
Get table metadata location
Update table metadata location
Delete Table

Table Management

Put Table Policy
Put Table Bucket Policy
Put Table Maintenance Config
Put Table Bucket Maintenance Config

Policies related to S3 table operation

Allow access to create and use table buckets

Here Action Lists the specific actions the policy allows.

These actions are S3 Tables-specific:

s3tables:CreateTableBucket: Grants permission to create a table bucket in S3 Tables.
s3tables:PutTableBucketPolicy: Allows setting or updating the bucket policy for a table bucket.
s3tables:GetTableBucketPolicy: Allows retrieving the bucket policy associated with a table bucket.
s3tables:ListTableBuckets: Allows listing all table buckets within the specified scope.
s3tables:GetTableBucket: Grants permission to access the metadata of a specific table bucket.
Resource Defines the scope of the resources these actions can apply to.
"arn:aws:s3tables:region:account_id:bucket/*": Specifies all table buckets in the account (account_id) and region (region).
The * after bucket/ indicates that permissions apply to all buckets under this account and region.

{
    "Version": "2012-10-17",
    "Statement": [{
        "Sid": "AllowBucketActions for user",
        "Effect": "Allow",
        "Action": [
            "s3tables:CreateTableBucket",
            "s3tables:PutTableBucketPolicy",
            "s3tables:GetTableBucketPolicy",
            "s3tables:ListTableBuckets",
            "s3tables:GetTableBucket"
        ],
        "Resource": "arn:aws:s3tables:region:account_id:bucket/*"
    }]
}

Allow access to create and use tables in a table bucket

Here Action Lists the specific actions allowed by the policy, related to S3 Tables. Please note that The first policy focused on creating and managing table buckets and associated metadata, but it did not include granular operations like managing tables within namespaces. The first policy did not include actions such as creating tables, querying data, or updating metadata at the table level. These are the operations where namespaces become relevant.

s3tables:CreateTable: Allows creating new tables in the specified table bucket.
s3tables:PutTableData: Grants permission to write data to tables within the table bucket.
s3tables:GetTableData: Allows reading data from tables in the bucket.
s3tables:GetTableMetadataLocation: Allows retrieving metadata location information for a table.
s3tables:UpdateTableMetadataLocation: Grants permission to update the metadata location of a table.
s3tables:GetNamespace: Allows retrieving namespace information associated with the table bucket.
s3tables:CreateNamespace: Grants permission to create namespaces for organizing table data.

Resource section specifies

Grants permissions on the bucket named amzn-s3-demo-table-bucket
Grants permissions on all tables within the amzn-s3-demo-table-bucket

{
     "Version": "2012-10-17",
     "Statement": [ 
         {
             "Sid": "AllowBucketActions",
             "Effect": "Allow",
             "Action": [
                 "s3tables:CreateTable",
                 "s3tables:PutTableData",
                 "s3tables:GetTableData",
                 "s3tables:GetTableMetadataLocation",
                 "s3tables:UpdateTableMetadataLocation",
                 "s3tables:GetNamespace",
                 "s3tables:CreateNamespace"
             ],

             "Resource": [
               "arn:aws:s3tables:region:account_id:bucket/amzn-s3-demo-table-bucket",
               "arn:aws:s3tables:region:account_id:bucket/amzn-s3-demo-table-bucket/table/*"
            ]
         }
     ]
 }

Table bucket policy to allows read access to the namespace

This policy allows to read s3 tables from a namespace. Here Action Lists the specific actions allowed by the policy, related to S3 Tables.

s3tables:GetTableData: Allows reading data from tables in the bucket.
s3tables:GetTableMetadataLocation: Allows retrieving metadata location information for a table. The resource section allows all s3 tables under bucket amzn-s3-demo-table-bucket1 but then s3tables:namespace restrict to only hr related s3 tables.

{
     "Version": "2012-10-17",
     "Statement": [ 
         {
             "Effect": "Allow",
             "Action": [
             "Principal": {
               "AWS": "arn:aws:iam::123456789012:user/Jane"
             },
             "Action": [
                  "s3tables:GetTableData", 
                  "s3tables:GetTableMetadataLocation"
             ],
             "Resource":{ "arn:aws:s3tables:region:account_id:bucket/amzn-s3-demo-table-bucket1/table/*”}
             "Condition": { 
                  "StringLike": { "s3tables:namespace": "hr" } 
             }
     ]
 }

S3 table automatic maintenance

It provides automated maintenance through configurations that help simplify table management, optimize performance, and reduce operational overhead.

Table Lifecycle Management
- we can add S3 Table configurations that includes lifecycle policies that automatically handle data expiration, transitions, or archival.
- automatic snapshot expiration can be configured easily.
Data Compaction
- S3 Tables automatically compact small files (often produced by incremental writes) into larger, optimized files. It helps to have faster query and reduce storage cost.
Schema Evolution
- Automated checks ensure compatibility between new and existing data.
Metadata Optimization
- Indexing of metadata for faster querying and retrieval of table details.

All these can be policy based configuration.

Policy for snapshot management

By configuring the maximumSnapshotAge, we can specify the retention period for table snapshots. The following example ensures S3 Table will automatically retain only the snapshots from the last 30 days

MinimumSnapshots: Ensures that at least one snapshot is always retained, regardless of age.
MaximumSnapshotAge: Specifies the maximum age (in hours) for snapshots to be retained.

aws s3tables put-table-maintenance-configuration \
    --table-arn arn:aws:s3tables:region:account_id:bucket/bucket_name/table/table_name \
    --maintenance-configuration '{
        "SnapshotManagement": {
            "MinimumSnapshots": 1,
            "MaximumSnapshotAge": 720
        }
    }

S3 Table Integration with AWS Analytics

S3 Tables integrate seamlessly with AWS analytics services to enable querying, processing and insight generation.

Amazon Athena - Run serverless SQL queries on S3 Tables

Use AWS Glue to create a Data Catalog for S3 Tables.
Query data directly using SQL in Athena.
Leverage table formats like Apache Iceberg or Parquet for optimized performance.

AWS Glue - Automate ETL processes for S3 Tables

Use Glue Crawlers to discover table metadata.
Create ETL jobs to transform and load data into S3 Tables or other destinations.

S3 Metadata table

It includes system metadata including object tags and user defined metadata
stored into s3 table
generated in near real time during data creation so that it can be used in mins during query

Use case for S3 metadata table

Real-Time Analytics
- efficient query execution on metadata to identify relevant data partitions.
Machine Learning Pipelines
- metadata tables to filter, select, and partition data for model training.
Governance and Compliance
- Track data retention and enforce lifecycle policies via metadata.
Multi-Tenant Data Applications
- Use namespaces within metadata tables to logically isolate tenant data.
Data Cataloging and Discovery
- Use metadata queries to identify datasets matching specific criteria.

Here is the sample python based function which uses metadata table query from athena.

def query_metadata_table(criteria):

    query = f"""
        SELECT *
        FROM {DATABASE}.{TABLE}
        WHERE {criteria}
    """

    print(f"Running query: {query}")

    # Start Athena query
    response = athena_client.start_query_execution(
        QueryString=query,
        QueryExecutionContext={'Database': DATABASE},
        ResultConfiguration={'OutputLocation': S3_OUTPUT}
    )

    query_execution_id = response['QueryExecutionId']

    # Wait for query completion
    print("Waiting for query to complete...")
    while True:
        status = athena_client.get_query_execution(QueryExecutionId=query_execution_id)
        state = status['QueryExecution']['Status']['State']
        if state in ['SUCCEEDED', 'FAILED', 'CANCELLED']:
            break
        time.sleep(2)

    if state != 'SUCCEEDED':
        raise Exception(f"Query failed with state: {state}")

    # Retrieve results
    results = athena_client.get_query_results(QueryExecutionId=query_execution_id)
    datasets = []
    for row in results['ResultSet']['Rows'][1:]:  # Skip the header row
        datasets.append([col['VarCharValue'] for col in row['Data']])

    print(f"Query returned {len(datasets)} datasets matching the criteria.")
    return datasets

Brief Notes on AWS CodeDeploy

Amit Kayal — Thu, 21 Mar 2024 19:04:04 +0000

Service that automates code deployments to any instance, including Amazon EC2 instances and instances running on-premises.

Supported Platforms/Deployment Types:

EC2/On-Premises: In-Place or Blue/Green Deployments
- Describes instances of physical servers that can be Amazon EC2 cloud instances, on-premises servers, or both. Applications created using the EC2/On-Premises compute platform can be composed of executable files, configuration files, images, and more. o - - Deployments that use the EC2/On-Premises compute platform manage the way in which traffic is directed to instances by using an in-place or blue/green deployment type.
AWS Lambda: Canary, Linear, All-At-Once Deployments
- Applications created using the AWS Lambda compute platform can manage the way in which traffic is directed to the updated Lambda function versions during a deployment by choosing a canary, linear, or all-at-once configuration.
Amazon ECS: Blue/Green Deployment
- Used to deploy an Amazon ECS containerized application as a task set.
- CodeDeploy performs a blue/green deployment by installing an updated version of the containerized application as a new replacement task set. CodeDeploy reroutes production traffic from the original application, or task set, to the replacement task set. The original task set is terminated after a successful deployment.

Deployment approach for EC2

Deploys a revision to a set of instances.
Deploys a new revision that consists of an application and AppSpec file. The AppSpec specifies how to deploy the application to the instances in a deployment group.

Deployment approach for Lambda

Deploys a new version of a serverless Lambda function on a high-availability compute infrastructure.
Shifts production traffic from one version of a Lambda function to a new version of the same function. The AppSpec file specifies which Lambda function version to deploy.

Deployment approach for ECS

Deploys an updated version of an Amazon ECS containerized application as a new, replacement task set. CodeDeploy reroutes production traffic from the task set with the original version to the new replacement task set with the updated version. When the deployment completes, the original task set is terminated.

App Spec File

The application specification file (AppSpec file) is a YAML-formatted or JSON-formatted file used by CodeDeploy to manage a deployment. Note: the name of the AppSpec file for an EC2/On-Premises deployment must be appspec.yml. The name of the AppSpec file for an Amazon ECS or AWS Lambda deployment must be appspec.yml.

For ECS

The container and port in replacement task set where your Application Load Balancer or Network Load Balancer reroutes traffic during a deployment. This is specified with the LoadBalancerInfo instruction in the AppSpec file.
Amazon ECS task definition file. This is specified with its ARN in the TaskDefinition instruction in the AppSpec file.

For Lambda

Lambda function version to deploy.
Lambda functions to use as validation tests.

For EC2

Which lifecycle event hooks to run in response to deployment lifecycle events.

Bedrock Agent & Tools - Tracing Best practises

Amit Kayal — Wed, 20 Mar 2024 17:58:52 +0000

I understand most of bedrock agent userss will have a use case where you have implemented multiple Lambda functions with a Bedrock Agent to perform different tasks and are looking for guidance in Debugging the API calls and responses from the Agent and lambda functions.

Here are some of the approaches that we have been using and found quite effective to track and trace agents and usage of their tools

Enable Tracing for the Agent: When invoking the agent, set the debug parameter to true. This will enable detailed tracing for the agent's execution, including the tools (Lambda functions) invoked and their responses. The trace will be printed to the console or returned as part of the agent's response, depending on how you invoke the agent. [1] Example (Python):

python result = agent.run(query, debug=True)

Log Within Lambda Functions: Within each of your Lambda functions (tools), add logging statements to capture relevant information and events. You can use AWS Lambda's built-in logging capabilities or integrate with a centralized logging service like Amazon CloudWatch Logs. [2] Example (Python):

python import logging logger = logging.getLogger(__name__) def lambda_handler(event, context): http://logger.info (f"Received event: {event}") # Your Lambda function's logic here http:// logger.info (f"Returning result: {result}") return result

Correlate Logs Using Request IDs or Tracing IDs: To correlate logs across multiple Lambda functions and the agent, you can use request IDs or tracing IDs. Pass a unique ID as part of the event or context to your Lambda functions and include it in your log statements. This will allow you to trace the flow of events across different components of your system.

import logging import uuid def lambda_handler(event, context): request_id = event.get("request_id", str(uuid.uuid4())) logger = logging.getLogger(__name__) logger = logging.LoggerAdapter(logger, {"request_id": request_id}) logger.info(f"Received event: {event}") logger.info(f"Returning result: {result}") return result

Use AWS X-Ray for Distributed Tracing: AWS X-Ray is a service that can help you analyze and debug distributed applications, including Lambda functions. By integrating X-Ray with your Bedrock application, you can trace requests as they travel through your Lambda functions and gain insights into their performance and potential issues. [3] - Enable X-Ray tracing for your Lambda functions by adding the necessary configuration. - Instrument your Lambda functions with X-Ray tracing code to capture relevant information and events. - Use the X-Ray console or integrate with other monitoring tools to analyze the traces and identify potential bottlenecks or issues.
Implement Advanced prompts : By using advanced prompts, you can enhance your agent's accuracy through modifying these prompt templates to provide detailed configurations. You can also provide hand-curated examples for few-shot prompting, in which you improve model performance by providing labeled examples for a specific task. [4] By combining the built-in tracing mechanism, custom logging within your Lambda functions, and distributed tracing with AWS X-Ray, you can gain better visibility into the API calls, events, and interactions happening within your Bedrock agent and its associated tools. This can help you debug issues more effectively and trace errors back to their source across multiple Lambda functions.
Reference

AWS DEV OPS Professional Exam short notes

Amit Kayal — Sun, 17 Mar 2024 05:55:58 +0000

Last few weeks I have been preparing for this exam and have summarized below key notes for further quick reference.

Key Notes

You can use CloudWatch Logs to monitor applications and systems using log data. For example, CloudWatch Logs can track the number of errors that occur in your application logs and send you a notification whenever the rate of errors exceeds a threshold you specify. CloudWatch Logs uses your log data for monitoring; so, no code changes are required. For more information on Cloudwatch logs , please refer to the below link: http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/WhatIsCloudWatchLogs.html The correct answer is: Install the CloudWatch Logs Agent on your AMI, and configure CloudWatch Logs Agent to stream your logs.
You can add another layer of protection by enabling MFA Delete on a versioned bucket. Once you do so, you must provide your AWS account’s access keys and a valid code from the account’s MFA device in order to permanently delete an object version or suspend or reactivate versioning on the bucket. For more information on MFA please refer to the below link: https://aws.amazon.com/blogs/security/securing-access-to-aws-using-mfa-part-3/ IAM roles are designed so that your applications can securely make API requests from your instances, without requiring you to manage the security credentials that the applications use. Instead of creating and distributing your AWS credentials, you can delegate permission to make API requests using IAM roles For more information on Roles for EC2 please refer to the below link: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html
As your infrastructure grows, common patterns can emerge in which you declare the same components in each of your templates. You can separate out these common components and create dedicated templates for them. That way, you can mix and match different templates but use nested stacks to create a single, unified stack. Nested stacks are stacks that create other stacks. To create nested stacks, use the AWS::CloudFormation::Stackresource in your template to reference other templates. For more information on best practices for Cloudformation please refer to the below link: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/best-practices.html The correct answer is: Separate the AWS CloudFormation template into a nested structure that has individual templates for the resources that are to be governed by different departments, and use the outputs from the networking and security stacks for the application template that you control.
You can use Amazon CloudWatch Logs to monitor, store, and access your log files from Amazon Elastic Compute Cloud (Amazon EC2) instances, AWS CloudTrail, and other sources. You can then retrieve the associated log data from CloudWatch Logs. For more information on Cloudwatch logs please refer to the below link: http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/WhatIsCloudWatchLogs.html You can the use Kinesis to process those logs For more information on Amazon Kinesis please refer to the below link: http://docs.aws.amazon.com/streams/latest/dev/introduction.html The correct answers are: Using AWS CloudFormation, create a CloudWatch Logs LogGroup and send the operating system and application logs of interest using the CloudWatch Logs Agent., Using configuration management, set up remote logging to send events to Amazon Kinesis and insert these into Amazon CloudSearch or Amazon Redshift, depending on available analytic tools.
IAM roles are designed so that your applications can securely make API requests from your instances, without requiring you to manage the security credentials that the applications use. Instead of creating and distributing your AWS credentials For more information on IAM Roles please refer to the below link: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html
The AWS Security Token Service (STS) is a web service that enables you to request temporary, limited-privilege credentials for AWS Identity and Access Management (IAM) users or for users that you authenticate (federated users). The token can then be used to grant access to the objects in S3. You can then provides access to the objects based on the key values generated via the user id
As your infrastructure grows, common patterns can emerge in which you declare the same components in each of your templates. You can separate out these common components and create dedicated templates for them. That way, you can mix and match different templates but use nested stacks to create a single, unified stack. Nested stacks are stacks that create other stacks. To create nested stacks, use the AWS::CloudFormation::Stackresource in your template to reference other templates. For more information on Cloudformation best practises please refer to the below link: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/best-practices.html The correct answer is: Create separate templates based on functionality, create nested stacks with CloudFormation.
The default autosclae termination policy is designed to help ensure that your network architecture spans Availability Zones evenly. When using the default termination policy, Auto Scaling selects an instance to terminate as follows: Auto Scaling determines whether there are instances in multiple Availability Zones. If so, it selects the Availability Zone with the most instances and at least one instance that is not protected from scale in. If there is more than one Availability Zone with this number of instances, Auto Scaling selects the Availability Zone with the instances that use the oldest launch configuration. For more information on Autoscaling instance termination please refer to the below link: http://docs.aws.amazon.com/autoscaling/latest/userguide/as-instance-termination.html The correct answer is: Auto Scaling will select the AZ with 4 EC2 instances and terminate an instance.
Amazon RDS Read Replicas provide enhanced performance and durability for database (DB) instances. This replication feature makes it easy to elastically scale out beyond the capacity constraints of a single DB Instance for read-heavy database workloads. You can create one or more replicas of a given source DB Instance and serve high-volume application read traffic from multiple copies of your data, thereby increasing aggregate read throughput. Sharding is a common concept to split data across multiple tables in a database. Shard your data set among multiple Amazon RDS DB instances.Amazon ElastiCache is a web service that makes it easy to deploy, operate, and scale an in-memory data store or cache in the cloud. The service improves the performance of web applications by allowing you to retrieve information from fast, managed, in-memory data stores, instead of relying entirely on slower disk-based databases.
Continuous Integration (CI) is a development practice that requires developers to integrate code into a shared repository several times a day. Each check-in is then verified by an automated build, allowing teams to detect problems early.
Elastic Beanstalk simplifies this process by managing the Amazon SQS queue and running a daemon process on each instance that reads from the queue for you. When the daemon pulls an item from the queue, it sends an HTTP POST request locally to http://localhost/ with the contents of the queue message in the body. All that your application needs to do is perform the long-running task in response to the POST. For more information Elastic Beanstalk managing worker environments, please visit the below URL: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features-managing-env-tiers.html
If you suspend AddToLoadBalancer, Auto Scaling launches the instances but does not add them to the load balancer or target group. If you resume the AddToLoadBalancer process, Auto Scaling resumes adding instances to the load balancer or target group when they are launched. However, Auto Scaling does not add the instances that were launched while this process was suspended. You must register those instances manually. For more information on the Suspension and Resumption process, please visit the below URL: http://docs.aws.amazon.com/autoscaling/latest/userguide/as-suspend-resume-processes.html
You can use the container_commands key of elastic beanstalk to execute commands that affect your application source code. Container commands run after the application and web server have been set up and the application version archive has been extracted, but before the application version is deployed. Non-container commands and other customization operations are performed prior to the application source code being extracted. You can use leader_only to only run the command on a single instance, or configure a test to only run the command when a test command evaluates to true. Leader-only container commands are only executed during environment creation and deployments, while other commands and server customization operations are performed every time an instance is provisioned or updated. Leader-only container commands are not executed due to launch configuration changes, such as a change in the AMI Id or instance type. For more information on customizing containers, please visit the below URL: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/customize-containers-ec2.html The correct answer is: Use a “Container command” within an Elastic Beanstalk configuration file to execute the script, ensuring that the “leader only” flag is set to true.
A Dockerrun.aws.json file is an Elastic Beanstalk–specific JSON file that describes how to deploy a set of Docker containers as an Elastic Beanstalk application. You can use aDockerrun.aws.json file for a multicontainer Docker environment. Dockerrun.aws.json describes the containers to deploy to each container instance in the environment as well as the data volumes to create on the host instance for the containers to mount. http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create_deploy_docker_v2config.html
Elastic Beanstalk supports the deployment of web applications from Docker containers. With Docker containers, you can define your own runtime environment. You can choose your own platform, programming language, and any application dependencies (such as package managers or tools), that aren’t supported by other platforms. Docker containers are self-contained and include all the configuration information and software your web application requires to run.
When you see Amazon Kinesis as an option, this becomes the ideal option to process data in real time. Amazon Kinesis makes it easy to collect, process, and analyze real-time, streaming data so you can get timely insights and react quickly to new information. Amazon Kinesis offers key capabilities to cost effectively process streaming data at any scale, along with the flexibility to choose the tools that best suit the requirements of your application. With Amazon Kinesis, you can ingest real-time data such as application logs, website clickstreams, IoT telemetry data, and more into your databases, data lakes and data warehouses, or build your own real-time applications using this data. For more information on Amazon Kinesis, please visit the below URL: https://aws.amazon.com/kinesis
You can use CloudWatch Logs to monitor applications and systems using log data CloudWatch Logs uses your log data for monitoring; so, no code changes are required. For example, you can monitor application logs for specific literal terms (such as “NullReferenceException”) or count the number of occurrences of a literal term at a particular position in log data (such as “404” status codes in an Apache access log). When the term you are searching for is found, CloudWatch Logs reports the data to a CloudWatch metric that you specify. Log data is encrypted while in transit and while it is at rest For more information on Cloudwatch logs please refer to the below link: http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/WhatIsCloudWatchLogs.html Amazon CloudWatch uses Amazon SNS to send email. First, create and subscribe to an SNS topic. When you create a CloudWatch alarm, you can add this SNS topic to send an email notification when the alarm changes state. For more information on SNS and Cloudwatch logs please refer to the below link: The correct answers are: Install a CloudWatch Logs Agent on your servers to stream web application logs to CloudWatch., Create a CloudWatch Logs group and define metric filters that capture 500 Internal Server Errors. Set a CloudWatch alarm on that metric., Use Amazon Simple Notification Service to notify an on-call engineer when a CloudWatch alarm is triggered
When you provision an Amazon EC2 instance in an AWS CloudFormation stack, you might specify additional actions to configure the instance, such as install software packages or bootstrap applications. Normally, CloudFormation proceeds with stack creation after the instance has been successfully created. However, you can use a CreationPolicy so that CloudFormation proceeds with stack creation only after your configuration actions are done. That way you’ll know your applications are ready to go after stack creation succeeds.
Auto Scaling periodically performs health checks on the instances in your Auto Scaling group and identifies any instances that are unhealthy. You can configure Auto Scaling to determine the health status of an instance using Amazon EC2 status checks, Elastic Load Balancing health checks, or custom health checks By default, Auto Scaling health checks use the results of the EC2 status checks to determine the health status of an instance. Auto Scaling marks an instance as unhealthy if its instance fails one or more of the status checks. For more information monitoring in Autoscaling , please visit the below URL: http://docs.aws.amazon.com/autoscaling/latest/userguide/as-monitoring-features.html
You need to have a custom health check which will evaluate the application functionality. Its not enough using the normal health checks. If the application functionality does not work and if you don’t have custom health checks , the instances will still be deemed as healthy. If you have custom health checks, you can send the information from your health checks to Auto Scaling so that Auto Scaling can use this information. For example, if you determine that an instance is not functioning as expected, you can set the health status of the instance to Unhealthy. The next time that Auto Scaling performs a health check on the instance, it will determine that the instance is unhealthy and then launch a replacement instance For more information on Autoscaling health checks , please refer to the below document link: from AWS http://docs.aws.amazon.com/autoscaling/latest/userguide/healthcheck.html
A blue group carries the production load while a green group is staged and deployed with the new code. When it’s time to deploy, you simply attach the green group to the existing load balancer to introduce traffic to the new environment. For HTTP/HTTPS listeners, the load balancer favors the green Auto Scaling group because it uses a least outstanding requests routing algorithm As you scale up the green Auto Scaling group, you can take blue Auto Scaling group instances out of service by either terminating them or putting them in Standby state, For more information on Blue Green Deployments , please refer to the below document link: from AWS https://d0.awsstatic.com/whitepapers/AWS_Blue_Green_Deployments.pdf
Ensure first that the cloudformation template is updated with the new instance type. The AWS::AutoScaling::AutoScalingGroup resource supports an UpdatePolicy attribute. This is used to define how an Auto Scaling group resource is updated when an update to the CloudFormation stack occurs. A common approach to updating an Auto Scaling group is to perform a rolling update, which is done by specifying the AutoScalingRollingUpdate policy. This retains the same Auto Scaling group and replaces old instances with new ones, according to the parameters specified
With web identity federation, you don’t need to create custom sign-in code or manage your own user identities. Instead, users of your app can sign in using a well-known identity provider (IdP) —such as Login with Amazon, Facebook, Google, or any other OpenID Connect (OIDC)-compatible IdP, receive an authentication token, and then exchange that token for temporary security credentials in AWS that map to an IAM role with permissions to use the resources in your AWS account. Using an IdP helps you keep your AWS account secure, because you don’t have to embed and distribute long-term security credentials with your application.
The optional Conditions section includes statements that define when a resource is created or when a property is defined. For example, you can compare whether a value is equal to another value. Based on the result of that condition, you can conditionally create resources. If you have multiple conditions, separate them with commas. You might use conditions when you want to reuse a template that can create resources in different contexts, such as a test environment versus a production environment. In your template, you can add an EnvironmentType input parameter, which accepts either prod or test as inputs. For the production environment, you might include Amazon EC2 instances with certain capabilities; however, for the test environment, you want to use reduced capabilities to save money. With conditions, you can define which resources are created and how they’re configured for each environment type. For more information on Cloudformation conditions please refer to the below link: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/conditions-section-structure.html
Elastic Beanstalk already has the facility to manage various versions and you don’t need to use S3 separately for this.AWS beanstalk is the perfect solution for developers to maintain application versions. With AWS Elastic Beanstalk, you can quickly deploy and manage applications in the AWS Cloud without worrying about the infrastructure that runs those applications. AWS Elastic Beanstalk reduces management complexity without restricting choice or control. You simply upload your application, and AWS Elastic Beanstalk automatically handles the details of capacity provisioning, load balancing, scaling, and application health monitoring.
The first step in using Elastic Beanstalk is to create an application, which represents your web application in AWS. In Elastic Beanstalk an application serves as a container for the environments that run your web app, and versions of your web app’s source code, saved configurations, logs and other artifacts that you create while using Elastic Beanstalk. For more information on Applications, please refer to the below link: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/applications.html Deploying a new version of your application to an environment is typically a fairly quick process. The new source bundle is deployed to an instance and extracted, and then the web container or application server picks up the new version and restarts if necessary. During deployment, your application might still become unavailable to users for a few seconds. You can prevent this by configuring your environment to use rolling deployments to deploy the new version to instances in batches. For more information on deployment, please refer to the below link: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features.deploy-existing-version.html
Weighted routing lets you associate multiple resources with a single domain name (example.com) or subdomain name (acme.example.com) and choose how much traffic is routed to each resource. This can be useful for a variety of purposes, including load balancing and testing new versions of software. For more information on the Routing policy please refer to the below link: http://docs.aws.amazon.com/Route53/latest/DeveloperGuide/routing-policy.html
Amazon Elasticsearch Service makes it easy to deploy, operate, and scale Elasticsearch for log analytics, full text search, application monitoring, and more. Amazon Elasticsearch Service is a fully managed service that delivers Elasticsearch’s easy-to-use APIs and real-time capabilities along with the availability, scalability, and security required by production workloads. The service offers built-in integrations with Kibana, Logstash, and AWS services including Amazon Kinesis Firehose, AWS Lambda, and Amazon CloudWatch so that you can go from raw data to actionable insights quickly.
You can use CloudWatch Logs to monitor applications and systems using log data. For example, CloudWatch Logs can track the number of errors that occur in your application logs and send you a notification whenever the rate of errors exceeds a threshold you specify. CloudWatch Logs uses your log data for monitoring; so, no code changes are required. For example, you can monitor application logs for specific literal terms (such as “NullReferenceException”) or count the number of occurrences of a literal term at a particular position in log data (such as “404” status codes in an Apache access log). When the term you are searching for is found, CloudWatch Logs reports the data to a CloudWatch metric that you specify. For more information on Cloudwatch Logs please refer to the below link: http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/WhatIsCloudWatchLogs.html Amazon CloudWatch uses Amazon SNS to send email. First, create and subscribe to an SNS topic. When you create a CloudWatch alarm, you can add this SNS topic to send an email notification when the alarm changes state. For more information on Cloudwatch and SNS please refer to the below link: http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/US_SetupSNS.html
AWS OpsWorks is a configuration management service that uses Chef, an automation platform that treats server configurations as code. OpsWorks uses Chef to automate how servers are configured, deployed, and managed across your Amazon Elastic Compute Cloud (Amazon EC2) instances or on-premises compute environments. OpsWorks has two offerings, AWS Opsworks for Chef Automate, and AWS OpsWorks Stacks. For more information on Opswork and SNS please refer to the below link: https://aws.amazon.com/opsworks/
You can use Kinesis Streams for rapid and continuous data intake and aggregation. The type of data used includes IT infrastructure log data, application logs, social media, market data feeds, and web clickstream data. Because the response time for the data intake and processing is in real time, the processing is typically lightweight. The following are typical scenarios for using Kinesis Streams: Accelerated log and data feed intake and processing – You can have producers push data directly into a stream. For example, push system and application logs and they’ll be available for processing in seconds. This prevents the log data from being lost if the front end or application server fails. Kinesis Streams provides accelerated data feed intake because you don’t batch the data on the servers before you submit it for intake. Real-time metrics and reporting – You can use data collected into Kinesis Streams for simple data analysis and reporting in real time. For example, your data-processing application can work on metrics and reporting for system and application logs as the data is streaming in, rather than wait to receive batches of data. For more information on Amazon Kinesis and SNS please refer to the below link: http://docs.aws.amazon.com/streams/latest/dev/introduction.html
With Elastic Beanstalk, you can quickly deploy and manage applications in the AWS Cloud without worrying about the infrastructure that runs those applications. AWS Elastic Beanstalk reduces management complexity without restricting choice or control. You simply upload your application, and Elastic Beanstalk automatically handles the details of capacity provisioning, load balancing, scaling, and application health monitoring For more information on Elastic beanstalk please refer to the below link: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/Welcome.html
You can use intrinsic functions, such as Fn::If, Fn::Equals, and Fn::Not, to conditionally create stack resources. These conditions are evaluated based on input parameters that you declare when you create or update a stack. After you define all your conditions, you can associate them with resources or resource properties in the Resources and Outputs sections of a template.
Amazon RDS Multi-AZ deployments provide enhanced availability and durability for Database (DB) Instances, making them a natural fit for production database workloads. When you provision a Multi-AZ DB Instance, Amazon RDS automatically creates a primary DB Instance and synchronously replicates the data to a standby instance in a different Availability Zone (AZ). Each AZ runs on its own physically distinct, independent infrastructure, and is engineered to be highly reliable. In case of an infrastructure failure, Amazon RDS performs an automatic failover to the standby (or to a read replica in the case of Amazon Aurora), so that you can resume database operations as soon
You can use AWS CloudTrail to get a history of AWS API calls and related events for your account. This history includes calls made with the AWS Management Console, AWS Command Line Interface, AWS SDKs, and other AWS services. For more information on Cloudtrail, please visit the below URL: http://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-user-guide.html Amazon CloudWatch Events delivers a near real-time stream of system events that describe changes in Amazon Web Services (AWS) resources. Using simple rules that you can quickly set up, you can match events and route them to one or more target functions or streams. CloudWatch Events becomes aware of operational changes as they occur. CloudWatch Events responds to these operational changes and takes corrective action as necessary, by sending messages to respond to the environment, activating functions, making changes, and capturing state information
By default, all AWS accounts are limited to 5 Elastic IP addresses per region, because public (IPv4) Internet addresses are a scarce public resource. We strongly encourage you to use an Elastic IP address primarily for the ability to remap the address to another instance in the case of instance failure, and to use DNS hostnames for all other inter-node communication
You can manage Amazon SQS messages with Amazon S3. This is especially useful for storing and consuming messages with a message size of up to 2 GB. To manage Amazon SQS messages with Amazon S3, use the Amazon SQS Extended Client Library for Java. Specifically, you use this library to: Specify whether messages are always stored in Amazon S3 or only when a message’s size exceeds 256 KB. Send a message that references a single message object stored in an Amazon S3 bucket. Get the corresponding message object from an Amazon S3 bucket. Delete the corresponding message object from an Amazon S3 bucket. For more information on processing large messages for SQS, please visit the below URL: http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-s3-messages.html
AWS CloudFormation provisions and configures resources by making calls to the AWS services that are described in your template. After all the resources have been created, AWS CloudFormation reports that your stack has been created. You can then start using the resources in your stack. If stack creation fails, AWS CloudFormation rolls back your changes by deleting the resources that it created. The below snapshot from Cloudformation shows what happens when there is an error in the stack creation. For more information on how CloudFormation works , please refer to the below link: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-whatis-howdoesitwork.html
Because Elastic Beanstalk performs an in-place update when you update your application versions, your application may become unavailable to users for a short period of time. It is possible to avoid this downtime by performing a blue/green deployment, where you deploy the new version to a separate environment, and then swap CNAMEs of the two environments to redirect traffic to the new version instantly. Blue/green deployments require that your environment runs independently of your production database, if your application uses one. If your environment has an Amazon RDS DB instance attached to it, the data will not transfer over to your second environment, and will be lost if you terminate the original environment. For more information on Blue Green deployments with Elastic beanstalk , please refer to the below link: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features.CNAMESwap.html
Amazon RDS Read Replicas provide enhanced performance and durability for database (DB) instances. This replication feature makes it easy to elastically scale out beyond the capacity constraints of a single DB Instance for read-heavy database workloads. You can create one or more replicas of a given source DB Instance and serve high-volume application read traffic from multiple copies of your data, thereby increasing aggregate read throughput. Read replicas can also be promoted when needed to become standalone DB instances.
Amazon Route 53 health checks monitor the health and performance of your web applications, web servers, and other resources.
If you use SSL termination, your servers will always get non-secure connections and will never know whether users used a more secure channel or not. If you are using Elastic beanstalk to configure the ELB, you can use the below article to ensure end to end encryption. http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/configuring-https-endtoend.html
Amazon Kinesis Firehose is the easiest way to load streaming data into AWS. It can capture, transform, and load streaming data into Amazon Kinesis Analytics, Amazon S3, Amazon Redshift, and Amazon Elasticsearch Service, enabling near real-time analytics with existing business intelligence tools and dashboards you’re already using today. It is a fully managed service that automatically scales to match the throughput of your data and requires no ongoing administration. It can also batch, compress, and encrypt the data before loading it, minimizing the amount of storage used at the destination and increasing security. For more information on Kinesis firehose, please visit the below URL: https://aws.amazon.com/kinesis/firehose/
Amazon Redshift is a fully managed, petabyte-scale data warehouse service in the cloud. You can start with just a few hundred gigabytes of data and scale to a petabyte or more. This enables you to use your data to acquire new insights for your business and customers.
Use Cloudfront distribution for distributing the heavy reads for your application. You can create a zone apex record to point to the Cloudfront distribution. You can control how long your objects stay in a CloudFront cache before CloudFront forwards another request to your origin. Reducing the duration allows you to serve dynamic content. Increasing the duration means your users get better performance because your objects are more likely to be served directly from the edge cache. A longer duration also reduces the load on your origin.
Amazon EBS encryption offers you a simple encryption solution for your EBS volumes without the need for you to build, maintain, and secure your own key management infrastructure. When you create an encrypted EBS volume and attach it to a supported instance type, the following types of data are encrypted: Data at rest inside the volume All data moving between the volume and the instance All snapshots created from the volume Snapshots that are taken from encrypted volumes are automatically encrypted. Volumes that are created from encrypted snapshots are also automatically encrypted.
A tag is a label that you or AWS assigns to an AWS resource. Each tag consists of a key and a value. A key can have more than one value. You can use tags to organize your resources, and cost allocation tags to track your AWS costs on a detailed level. After you activate cost allocation tags, AWS uses the cost allocation tags to organize your resource costs on your cost allocation report, to make it easier for you to categorize and track your AWS costs. AWS provides two types of cost allocation tags, an AWS-generated tag and user-defined tags. AWS defines, creates, and applies the AWS-generated tag for you, and you define, create, and apply user-defined tags. You must activate both types of tags separately before they can appear in Cost Explorer or on a cost allocation report.
You can monitor the progress of a stack update by viewing the stack’s events. The console’s Events tab displays each major step in the creation and update of the stack sorted by the time of each event with latest events on top. The start of the stack update process is marked with an UPDATE_IN_PROGRESS event for the stack For more information on Monitoring your stack, please visit the below URL: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-updating-stacks-monitor-stack.html
A placement group is a logical grouping of instances within a single Availability Zone. Placement groups are recommended for applications that benefit from low network latency, high network throughput, or both. To provide the lowest latency, and the highest packet-per-second network performance for your placement group, choose an instance type that supports enhanced networking. For more information on Placement Groups, please visit the below URL: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/placement-groups.html
AWS CloudTrail is an AWS service that helps you enable governance, compliance, and operational and risk auditing of your AWS account. Actions taken by a user, role, or an AWS service are recorded as events in CloudTrail. Events include actions taken in the AWS Management Console, AWS Command Line Interface, and AWS SDKs and APIs. Visibility into your AWS account activity is a key aspect of security and operational best practices. You can use CloudTrail to view, search, download, archive, analyze, and respond to account activity across your AWS infrastructure. You can identify who or what took which action, what resources were acted upon, when the event occurred, and other details to help you analyze and respond to activity in your AWS account. For more information on Cloudtrail, please visit the below URL: http://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-user-guide.html
Custom resources enable you to write custom provisioning logic in templates that AWS CloudFormation runs anytime you create, update (if you changed the custom resource), or delete stacks. For example, you might want to include resources that aren’t available as AWS CloudFormation resource types. You can include those resources by using custom resources. That way you can still manage all your related resources in a single stack. Use the AWS::CloudFormation::CustomResource or Custom::String resource type to define custom resources in your templates. Custom resources require one property: the service token, which specifies where AWS CloudFormation sends requests to, such as an Amazon SNS topic.
Failover routing lets you route traffic to a resource when the resource is healthy or to a different resource when the first resource is unhealthy. The primary and secondary resource record sets can route traffic to anything from an Amazon S3 bucket that is configured as a website to a complex tree of records. For more information on Route53 Failover Routing, please visit the below URL: http://docs.aws.amazon.com/Route53/latest/DeveloperGuide/routing-policy.html
Deployment Types:
- Single Target Deployment - small dev projects, legacy or non-HA infrastructure; outage occurs in case of failure, testing opportunity is limited.
- All-at-Once Deployment - deployment happens on multiple targets, requires Orchestration tools, suitable for non critical apps in 5-10 range.
- Minimum in-service Deployment - keeps min in-service targets and deploy in multiple stages, suitable for large environments, allow automated testing, no downtime
- Rolling Deployments - x targets per stage, happens in multiple stages, after completion of stage 1, next stage begins, orchestration and health check required, can be least efficient if x is smaller, allow automated testing, no downtime if x is not large to impact application, can be paused, allowing multi-version testing.
- Blue Green Deployment - Deploy to seperate Green environment, update the code on Green, extra cost due to duplicate env during deployment, Deployment is rapid, cutover and migration is clean(DNS Change), Rollback easy(DNS regression), can be fully automates using CFN etc. Binary, No Traffic Split, not used to feature test
- A/B Testing - distribution traffic between blue/green, allows gradual performance/stability/health analysis, allows new feature testing, rollback is quick, end goal of A/B testing is not migration, Uses Route 53 for DNS resolution, 2 records one pointing A, other pointing B, weighted/round robin.
Intrinsic & Conditional Functions
- Intrinsic Fn - inbuilt function provided by AWS to help manage, reference, and condtionally act upon resources, situation & inputs to a stack.
- Fn::Base64 - Base64 encoding for User Data
- Fn::FindInMap - Mapping lookup
- Fn::GetAtt - Advanced reference look up
- Fn::GetAZs - retrieve list of AZs in a region
- Fn::Join - construct complex strings; concatenate strings
- Fn::Select - value selection from list (0, 1)
- Ref - default value of resource
- Conditional Functions - Fn::And, Fn::Equals, Fn::If, Fn::Not, Fn::Or
CFN Resource Deletion Policies
- A policy/setting which is associated with each resource in a template; A way to control what happens to each resource when a stack is deleted.
- Policy value - Delete (Default), Retain, Snapshot
- Delete - Useful for testing environment, CI/CD/QA workflows,
- Presales, Short Lifecycle/Immutable env.
- Retain - live beyond lifcycle of stack; Windows Server Platform (AD), Servers with state, SQL, Exchange, File Servers,
- Non immutable architectures.
- Snapshot - restricted policy type only available for EBS volumes; takes snapshot before deleting for recovering data.
Immutable Architecture - Replace infra instead of upgrading or repairing faulty components, treat servers as unchangeable objects, don't diagnose and fix, throw away and re-create, Nothing bootstraped except AMI.
CFN Stack updates
- stack policy is checked, updates can be prevented; absence of
- stack policy allow all updates; stack policy cannot be deleted once applied. Once stack policy applied ALL objects are protected, Update is denied; to remove default DENY, explicit allow is required; can be applied to a single resource(id)/Wild card/NotResource; Has Principal and Action; Condition element (resource type) can also be used.
Stack updates: 4 Types - Update with No Interrupion, Some Interruption, Replacement, Delete

Sagemaker Model deployment and Integration

Amit Kayal — Wed, 18 Jan 2023 07:29:11 +0000

Sagemaker Model deployment and Integration

[TOC]

AWS Feature store

SageMaker Feature Store is a purpose-built solution for ML feature management. It helps data science teams reuse ML features across teams and models, serve features for model predictions at scale with low latency, and train and deploy new models more quickly and effectively.

Refer the notebook https://github.com/aws-samples/ml-lineage-helper/blob/main/examples/example.ipynb for more details.

Why is feature lineage important?

Imagine trying to manually track all of this for a large team, multiple teams, or even multiple business units. Lineage tracking and querying helps make this more manageable and helps organizations move to ML at scale. The following are four examples of how feature lineage helps scale the ML process:

Build confidence for reuse of existing features
Avoid reinventing features that are based on the same raw data as existing features
Troubleshoot and audit models and model predictions
Manage features proactively

AWS ML Lens and built-in models

Deployment Options

ML inference can be done in real time on individual records, such as with a REST API endpoint. Inference can also be done in batch mode as a processing job on a large dataset. While both approaches push data through a model, each has its own target goal when running inference at scale.

* Real Time*	Micro Batch	Batch
Execution Mode	Synchronous	Synchronous/Asynchronous	Asynchronous
Prediction Latency	Subsecond	Seconds to minutes	Indefinite
Data Bounds	Unbounded/stream	Bounded	Bounded
Execution Frequency	Variable	Variable	Variable/fixed
Invocation Mode	Continuous stream/API calls	Event-based	Event-based/scheduled
Examples	Real-time REST API endpoint	Data analyst running a SQL UDF	Scheduled inference job

Realtime deployment

Sagemaker real-time deployment has the following approach. Key point here is that we can have our inference pipeline coupled with autoscale.

Here are different ways, we can deploy real-time endpoint by sagemaker. You can see here multiple options from own model, own container to prebuilt container.

With sagemaker, prebuilt container and its own inference script, we can use this as shared below.

Quite a lot of time, we add our own inference script and this is quite simple as shown below.

It is not rare to have our own container and own trained model along with inference script. The architecture does not change for that and we still follow same architecture as shared below.

Autoscale

we can set autoscale policy for sagemaker endpoint to scale up and scale down automatically.

We have to set autoscale policy setup for endpoint. You can see here that ServiceNamespace is set to sgaemaker and resourceId is set to Endpoint name.

Multi Modal endpoint

SageMaker multi-model endpoints work with several frameworks, such as TensorFlow, PyTorch, MXNet, and sklearn, and you can build your own container with a multi-model server. Multi-model endpoints are also supported natively in the following popular SageMaker built-in algorithms: XGBoost, Linear Learner, Random Cut Forest (RCF), and K-Nearest Neighbors (KNN).

Refer the notebook https://github.com/aws-samples/sagemaker-multi-model-endpoint-tensorflow-computer-vision/blob/main/multi-model-endpoint-tensorflow-cv.ipynb to understand how we can deploy this/. Refer the blog https://aws.amazon.com/blogs/machine-learning/save-on-inference-costs-by-using-amazon-sagemaker-multi-model-endpoints/

All of the models that are hosted on a multi-modal endpoint must share the same serving container image.
Multi-model endpoints are an option that can improve endpoint utilization when your models are of similar size and share the same container image and have similar invocation latency requirements.
all the model needs to share same S3 bucket to host their weights

Cost advantages

This diagram demonstrates running 10 models on a multi-model endpoint versus using 10 separate endpoints. This results in savings of $3,000 per month, as shown in the following figure: Multi-model endpoints can easily scale to hundreds or thousands of models.

How to use?

To create a multi-model endpoint in Amazon SageMaker, choose the multi-model option, provide the inference serving container image path, and provide the Amazon S3 prefix in which the trained model artifacts are stored. You can organize your models in S3 any way you wish, so long as they all use the same prefix.

When you invoke the multi-model endpoint, you provide the relative path of a specific model with the new TargetModel parameter of InvokeEndpoint. To add models to the multi-model endpoint, simply store a newly trained model artifact in S3 under the prefix associated with the endpoint. The model will then be immediately available for invocations.

To update a model already in use, add the model to S3 with a new name and begin invoking the endpoint with the new model name. To stop using a model deployed on a multi-model endpoint, stop invoking the model and delete it from S3.

Instead of downloading all the models into the container from S3 when the endpoint is created, Amazon SageMaker multi-model endpoints dynamically load models from S3 when invoked. As a result, an initial invocation to a model might see higher inference latency than the subsequent inferences, which are completed with low latency. If the model is already loaded on the container when invoked, then the download step is skipped and the model returns the inferences with low latency.

Monitoring multi-model endpoints using Amazon CloudWatch metrics

To make price and performance tradeoffs, you will want to test multi-model endpoints with models and representative traffic from your own application. Amazon SageMaker provides additional metrics in CloudWatch for multi-model endpoints so you can determine the endpoint usage and the cache hit rate and optimize your endpoint. The metrics are as follows:

ModelLoadingWaitTime – The interval of time that an invocation request waits for the target model to be downloaded or loaded to perform the inference.
ModelUnloadingTime – The interval of time that it takes to unload the model through the container’s UnloadModel API call.
ModelDownloadingTime – The interval of time that it takes to download the model from S3.
ModelLoadingTime – The interval of time that it takes to load the model through the container’s LoadModel API call.
ModelCacheHit – The number of InvokeEndpoint requests sent to the endpoint where the model was already loaded. Taking the Average statistic shows the ratio of requests in which the model was already loaded.
LoadedModelCount – The number of models loaded in the containers in the endpoint. This metric is emitted per instance. The Average statistic with a period of 1 minute tells you the average number of models loaded per instance, and the Sum statistic tells you the total number of models loaded across all instances in the endpoint. The models that this metric tracks are not necessarily unique because you can load a model in multiple containers in the endpoint.

You can use CloudWatch charts to help make ongoing decisions on the optimal choice of instance type, instance count, and number of models that a given endpoint should host.

Inference Pipeline sagemaker

You can use trained models in an inference pipeline to make real-time predictions directly without performing external preprocessing. When you configure the pipeline, you can choose to use the built-in feature transformers already available in Amazon SageMaker. Or, you can implement your own transformation logic using just a few lines of scikit-learn or Spark code.

Refer https://sagemaker-examples.readthedocs.io/en/latest/sagemaker-python-sdk/scikit_learn_inference_pipeline/Inference%20Pipeline%20with%20Scikit-learn%20and%20Linear%20Learner.html / https://catalog.us-east-1.prod.workshops.aws/workshops/f238037c-8f0b-446e-9c15-ebcc4908901a/en-US/002-services/003-machine-learning/020-sagemaker for more details.

Inference pipeline allows you to host multiple models behind a single endpoint. But in this case, the models are sequential chain of models with the steps that are required for inference. This allows you to take your data transformation model, your predictor model, and your post-processing transformer, and host them so they can be sequentially run behind a single endpoint.
As you can see in this picture, the inference request comes into the endpoint, then the first model is invoked, and that model is your data transformation. The output of that model is then passed to the next step, which is actually your XGBoost model here, or your predictor model.
- That output is then passed to the next step, where ultimately in that final step in the pipeline, it provides the final response or the post-process response to that inference request.
- This allows you to couple your pre and post-processing code behind the same endpoint and helps ensure that your training and your inference code stay synchronized

Sagemaker Production Variant

Amazon SageMaker enables you to test multiple models or model versions behind the same endpoint using production variants. Each production variant identifies a machine learning (ML) model and the resources deployed for hosting the model. By using production variants, you can test ML models that have been trained using different datasets, trained using different algorithms and ML frameworks, or are deployed to different instance type, or any combination of all of these. You can distribute endpoint invocation requests across multiple production variants by providing the traffic distribution for each variant, or you can invoke a specific variant directly for each request. In this topic, we look at both methods for testing ML models.

Refer the notebook https://sagemaker-examples.readthedocs.io/en/latest/sagemaker_endpoints/a_b_testing/a_b_testing.html for implementation details.

Test models by specifying traffic distribution

Specify the percentage of the traffic that gets routed to each model by specifying the weight for each production variant in the endpoint configuration.

Test models by invoking specific variants

Specify the specific version of the model you want to invoke by providing a value for the TargetVariant parameter when you call InvokeEndpoint.

Amazon SageMaker Batch Transform: Batch Inference

We’ll use the Sagemaker Batch Transform Jobs and a trained machine learning model. It is assumed that we have already trained the model, pushed the Docker image to ECR, and registered the model in Sagemaker.

we need the identifier of the Sagemaker model we want to use and the location of the input data
either use a built-in container for your inference image or you can also bring your own.
Batch Transform partitions the Amazon S3 objects in the input by key and maps Amazon S3 objects to instances. When you have multiples files, one instance might process input1. csv , and another instance might process the file named input2. csv

In Batch Transform you provide your inference data as a S3 uri and SageMaker will care of downloading it, running the prediction and uploading the results afterwards to S3 again. You can find more documentation for Batch Transform here

If you trained a model using the Hugging Face Estimator, call the transformer() method to create a transform job for a model based on the training job (see here for more details): Refer https://huggingface.co/docs/sagemaker/inference

Note:

batch job has

instance count
instance type

transform job has

data location
content type

batch_job = huggingface_estimator.transformer(
    instance_count=1,
    instance_type='ml.p3.2xlarge',
    strategy='SingleRecord')


batch_job.transform(
    data='s3://s3-uri-to-batch-data',
    content_type='application/json',    
    split_type='Line')

If you want to run your batch transform job later or with a model from the 🤗 Hub, create a HuggingFaceModel instance and then call the transformer() method:

from sagemaker.huggingface.model import HuggingFaceModel

# Hub model configuration <https://huggingface.co/models>
hub = {
    'HF_MODEL_ID':'distilbert-base-uncased-finetuned-sst-2-english',
    'HF_TASK':'text-classification'
}

# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
   env=hub,                                                # configuration for loading model from Hub
   role=role,                                              # IAM role with permissions to create an endpoint
   transformers_version="4.6",                             # Transformers version used
   pytorch_version="1.7",                                  # PyTorch version used
   py_version='py36',                                      # Python version used
)

# create transformer to run a batch job
batch_job = huggingface_model.transformer(
    instance_count=1,
    instance_type='ml.p3.2xlarge',
    output_path=output_s3_path, # we are using the same s3 path to save the output with the input
    strategy='SingleRecord'
)

# starts batch transform job and uses S3 data as input
batch_job.transform(
    data='s3://sagemaker-s3-demo-test/samples/input.jsonl',
    content_type='application/json',    
    split_type='Line'
)

The input.jsonl looks like this:

import json
from sagemaker.s3 import S3Downloader
from ast import literal_eval
# creating s3 uri for result file -> input file + .out
output_file = f"{dataset_jsonl_file}.out"
output_path = s3_path_join(output_s3_path,output_file)

# download file
S3Downloader.download(output_path,'.')

batch_transform_result = []
with open(output_file) as f:
    for line in f:
        # converts jsonline array to normal array
        line = "[" + line.replace("[","").replace("]",",") + "]"
        batch_transform_result = literal_eval(line) 

# print results 
print(batch_transform_result[:3])

{"inputs":"this movie is terrible"}
{"inputs":"this movie is amazing"}
{"inputs":"SageMaker is pretty cool"}
{"inputs":"SageMaker is pretty cool"}
{"inputs":"this movie is terrible"}
{"inputs":"this movie is amazing"}

📓 Open the notebook for an example of how to run a batch transform job for inference.

Speeding up the processing

We have only one instance running, so processing the entire file may take some time. We can increase the number of instances using the instance_count parameter to speed it up. We can send multiple requests to the Docker container simultaneously, too. The configure concurrent transformations we must use the max_concurrent_transforms parameter.

Processing the output

In the end, we must get access to the output. We’ll find the output files in the location specified in the Transformer constructor. Every line contains the prediction and the input parameters. agemaker-notebook.ipynb) for an example of how to run a batch transform job for inference.

My journey into Sentence Transformer

Amit Kayal — Sun, 16 Oct 2022 16:49:41 +0000

Last few days I have been exploring sentence transformer and this page documents my notes/understanding. This note explains the basic of sentence transformer and deployment the same through sagemaker endopoint. The endpoint is then accessed from lambda. Terraform has been considered here for deployment.

What is Sentence Embedding?

I came across a nice example posted by Mathias about sentence comparison. Consider the following statements: “Nuclear power is dangerous!” and “Nuclear power is the future of energy!”

If we are talking about the topic, then definitely yes: both statements are opinions on nuclear power. So in that sense, they are very similar.
However, if we are talking about sentiment, then the answer is a resounding no. They are about as dissimilar in terms of sentiment as we can get.

What is Transformer and BERT?

BART is a denoising autoencoder for pretraining sequence-to-sequence models. It is trained by (1) corrupting text with an arbitrary noising function, and (2) learning a model to reconstruct the original text. It uses a standard Transformer-based neural machine translation architecture. It uses a standard seq2seq/NMT architecture with a bidirectional encoder (like BERT) and a left-to-right decoder (like GPT). This means the encoder's attention mask is fully visible, like BERT, and the decoder's attention mask is causal, like GPT2.

While BERT was trained by using a simple token masking technique, BART empowers the BERT encoder by using more challenging kinds of masking mechanisms in its pre-training. Once we get the token and sentence-level representation of an input text sequence, a decoder needs to interpret these to map with the output target.

Building task specific Transformer based solution

Green portion is pretrained one
other portion (purple) is custom head which we train further
QA head is predicting span of text which contains answer from context/text
Classification Head predicting binary value.

BERT for Sentence Similarity

Transformers work using word or token-level embeddings, not sentence-level embeddings.

Regular transformers produce sentence embeddings by performing some pooling operation such as the element-wise arithmetic mean on its token-level embeddings. A good pooling choice for BERT is CLS pooling. BERT has a special <CLS> token that is supposed to capture all the sequence information. It gets tuned on next-sentence prediction (NSP) during pre-training.

Before sentence transformers, the approach to calculating accurate sentence similarity with BERT was to use a cross-encoder structure. This meant that we would pass two sentences to BERT, add a classification head to the top of BERT — and use this to output a similarity score.

The BERT cross-encoder architecture consists of a BERT model which consumes sentences A and B. Both are processed in the same sequence, separated by a [SEP] token. All of this is followed by a feedforward NN classifier that outputs a similarity score.
The cross-encoder network does produce very accurate similarity scores (better than SBERT), but it’s not scalable. If we wanted to perform a similarity search through a small 100K sentence dataset, we would need to complete the cross-encoder inference computation 100K times.
To cluster sentences, we would need to compare all sentences in our 100K dataset, resulting in just under 500M comparisons — this is simply not realistic.
Ideally, we need to pre-compute sentence vectors that can be stored and then used whenever required. If these vector representations are good, all we need to do is calculate the cosine similarity between each.
With the original BERT (and other transformers), we can build a sentence embedding by averaging the values across all token embeddings output by BERT (if we input 512 tokens, we output 512 embeddings). Alternatively, we can use the output of the first [CLS] token (a BERT-specific token whose output embedding is used in classification tasks).
Using one of these two approaches gives us our sentence embeddings that can be stored and compared much faster, shifting search times from 65 hours to around 5 seconds (see below). However, the accuracy is not good, and is worse than using averaged GloVe embeddings (which were developed in 2014).

Sentence Transformer?

How Does it Work?

SentenceTransformers is a Python framework for state-of-the-art sentence, text, and image embeddings. Embeddings can be computed for 100+ languages and they can be easily used for common tasks like semantic text similarity, semantic search, and paraphrase mining.

The solution of the above lack of an accurate model with reasonable latency was designed by Nils Reimers and Iryna Gurevych in 2019 with the introduction of sentence-BERT (SBERT) and the sentence-transformers library.

SBERT produces sentence embeddings — so we do not need to perform a whole inference computation for every sentence-pair comparison.

SBERT is fine-tuned on sentence pairs using a siamese architecture. We can think of this as having two identical BERTs in parallel that share the exact same network weights.

In reality, we are using a single BERT model. However, because we process sentence A followed by sentence B as pairs during training, it is easier to think of this as two models with tied weights.

Use cases

Sentence Transformer Architecture Changes

SBERT uses a siamese architecture where it contains 2 BERT architectures that are essentially identical and share the same weights, and SBERT processes 2 sentences as pairs during training.

The training process of sentence transformers is especially designed with semantic similarity in mind.

Cross-encoders

A cross-encoder is thus trained by sentence-pairs along with a ground-truth label of how semantically similar they are.

Cross-encoders tend to perform very well on sentence-level tasks, they do suffer from a major drawback: cross-encoders do not produce sentence embeddings. **In the context of information retrieval, this implies that we **cannot pre-compute document embeddings and efficiently compare these to a query embedding. We are also not able to index document embeddings for efficient search.

Bi-encoders

In the context of information retrieval, this implies that we cannot pre-compute document embeddings and efficiently compare these to a query embedding. We are also not able to index document embeddings for efficient search.

we feed sentence A to BERT A and sentence B to BERT B in SBERT. Each BERT outputs pooled sentence embeddings. While the original research paper tried several pooling methods, they found mean-pooling was the best approach. Pooling is a technique for generalizing features in a network, and in this case, mean pooling works by averaging groups of features in the BERT.
After the pooling is done, we now have 2 embeddings: 1 for sentence A and 1 for sentence B. When the model is training, SBERT concatenates the 2 embeddings which will then run through a softmax classifier and be trained using a softmax-loss function.
At inference — or when the model actually begins predicting — the two embeddings are then compared using a cosine similarity function, which will output a similarity score for the two sentences. Here is a diagram for SBERT when it is fine-tuned and at inference.

How can you use SBERT for sagemaker endpoint?

BERT has its own Python library. Using it is as simple as using a model from the hugging face transformer library. Here, we have used multi-qa-MiniLM-L6-cos-v1 model for sentence similarity.

Here, I have shown how we can deploy this model as our sagemaker serverless endpoint.

# Choose transformer model for embeddings
from transformers import AutoTokenizer, AutoModel
import os
import sagemaker
import time
saved_model_dir = 'transformer'
os.makedirs(saved_model_dir, exist_ok=True)

tokenizer = AutoTokenizer.from_pretrained("sentence-transformers/multi-qa-MiniLM-L6-cos-v1")
model = AutoModel.from_pretrained("sentence-transformers/multi-qa-MiniLM-L6-cos-v1") 

tokenizer.save_pretrained(saved_model_dir)
model.save_pretrained(saved_model_dir)

#Defining default bucket for SageMaker pretrained model hosting
sagemaker_session = sagemaker.Session()
role = sagemaker.get_execution_role()

!cd transformer && tar czvf ../model.tar.gz *

model_data = sagemaker_session.upload_data(path='model.tar.gz', key_prefix='autofaiss-demo/huggingface-models')

from sagemaker.huggingface.model import HuggingFaceModel


# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
   model_data=model_data,       # path to your model and script
   entry_point = 'predict.py',
   source_dir = 'source_dir',
   role=role,                    # iam role with permissions to create an Endpoint
   transformers_version="4.12",  # transformers version used
   pytorch_version="1.9",        # pytorch version used
   py_version='py38',            # python version used
)

# deploy the endpoint endpoint
predictor = huggingface_model.deploy(
    initial_instance_count=1,
    instance_type="ml.m5.2xlarge"
    )

Here, I have created a lambda_handler function where the above endpoint is being called for similarity prediction.

import logging
import json
import boto3
import io
import os
import time
import logging
import sagemaker
from sagemaker.deserializers import JSONDeserializer
from sagemaker.serializers import IdentitySerializer

logger = logging.getLogger()
logger.setLevel(logging.INFO)
ENDPOINT_NAME = "huggingface-pytorch-inference-2022-10-14-20-02-16-258"
sagemaker_session = sagemaker.Session()

"""
FunctionName: invoke_endpoint
Input: transcript_item (sentence), label_map
    transcript_item type: string
    label_map type: dict
Output: Question
    type: string
"""
# @tracer.capture_method
def invoke_endpoint(payload, endpoint_name):
    runtime = boto3.client('runtime.sagemaker')
    response = runtime.invoke_endpoint(EndpointName=endpoint_name,
                                      ContentType="application/json",
                                      Body=json.dumps(payload))
    embeddings = json.loads((response["Body"].read()))
    return embeddings

# @tracer.capture_lambda_handler
def lambda_handler(event):
    start = time.time()
    similarity_scores = invoke_endpoint(event, ENDPOINT_NAME)
    end = time.time()
    logger.info(f"Profiling: \n Getting Embeddings: {1000*(end-start)} milliseconds")   
    return similarity_scores


json_event = {  
    "query_from_app" : "How many people live in London?",
    "actual_queries" : ["Around 9 Million people live in London", "London is known for its financial district"]
}

lambda_handler(json_event)

The sample output is as shown below...

{'Scores': [['Around 9 Million people live in London', 0.9156370759010315],
  ['London is known for its financial district', 0.49475768208503723]]}

Serverless deployment of Sentence Transformer

I have shared below our lambda code and terraform code for deployment.

Lambda Code

I have used terraform to deploy the lambda function and the endpoint is being defined here as environment variable.

from query_request import query
DDB_ENDPOINT = os.environ["ServiceConfiguration__DDB_ENDPOINT"]
REGION = os.environ["ServiceConfiguration__REGION"]

def invoke_endpoint(payload, endpoint_name):
    runtime = boto3.client('runtime.sagemaker')
    response = runtime.invoke_endpoint(EndpointName=endpoint_name,
                                      ContentType="application/json",
                                      Body=json.dumps(payload))
    embeddings = json.loads((response["Body"].read()))
    return embeddings

def lambda_handler(event, context):
    start = time.time()
    similarity_scores = invoke_endpoint(event, ENDPOINT_NAME)
    end = time.time()
    logger.info(f"Similarity scores: {similarity_scores}")
    logger.info(f"Profiling: \n Getting Embeddings: {1000*(end-start)} milliseconds")   
    return similarity_scores

Terraform

module "questn_similarity_classification" {
  source  = "terraform-module/lambda/aws"
  version = "2.12.6"

  function_name = "questn_similarity_classification"
  filename      = data.archive_file.questn_similarity_classification-zip.output_path
  source_code_hash = data.archive_file.questn_similarity_classification-zip.output_base64sha256
  description      = "questn_similarity_classification"
  handler        = "questn_similarity_classification.lambda_handler"
  runtime        = "python3.7"
  memory_size    = "1280"
  concurrency    = "25"
  lambda_timeout = "120"
  log_retention  = "30"
  publish        = true
  role_arn       = aws_iam_role.questn_similarity_classification_role.arn
  tracing_config = { mode = "Active" }
 # layers = [aws_lambda_layer_version.numpy_layer_37.arn, data.aws_lambda_layer_version.ml_faiss_layer_version.arn]

  vpc_config = {
    subnet_ids         = tolist(data.aws_subnet.efs_subnet.*.id)
    security_group_ids = [data.aws_security_group.default_sec_grp.id]
  }
  environment = {
    ServiceConfiguration__ENDPOINT_NAME    = var.ServiceConfiguration__ENDPOINT_NAME
    ServiceConfiguration__REGION = var.ServiceConfiguration__REGION
  }
  file_system_config = {
    # file_system_arn              = data.aws_efs_access_point.knn_efs.arn
    efs_access_point_arn = data.aws_efs_access_point.knn_efs.arn
    local_mount_path     = var.file_system_local_mount_path # Local mount path inside the lambda function. Must start with '/mnt/'.
    # file_system_local_mount_path = var.file_system_local_mount_path # Local mount path inside the lambda function. Must start with '/mnt/'.

  }

A learning journey with ELK stack

Amit Kayal — Wed, 01 Dec 2021 17:15:27 +0000

Kibana and Elastic Search

[TOC]

What is Kibana?

Kibana is a data visualization and exploration tool used for log and time-series analytics, application monitoring, and operational intelligence use cases. It offers powerful and easy-to-use features such as histograms, line graphs, pie charts, heat maps, and built-in geospatial support. Also, it provides tight integration with Elasticsearch, a popular analytics and search engine, which makes Kibana the default choice for visualizing data stored in Elasticsearch.Kibana works in sync with Elasticsearch and Logstash which together forms the so called ELK stack.

What is ELK Stack?

ELK stands for Elasticsearch, Logstash, and Kibana. ELK is one of the popular log management platform used worldwide for log analysis. In the ELK stack, Logstash extracts the logging data or other events from different input sources. It processes the events and later stores them in Elasticsearch.

Kibana is a visualization tool, which accesses the logs from Elasticsearch and is able to display to the user in the form of line graph, bar graph, pie charts etc. The basic flow of ELK Stack is shown in the image here −

Logstash is responsible to collect the data from all the remote sources where the logs are filed and pushes the same to Elasticsearch. Elasticsearch acts as a database where the data is collected and Kibana uses the data from Elasticsearch to represent the data to the user in the form of bargraphs, pie charts, heat maps as shown below −

For a small-sized development environment, the classic architecture will look as follows:

However, for handling more complex pipelines built for handling large amounts of data in production, additional components are likely to be added into your logging architecture, for resiliency (Kafka, RabbitMQ, Redis) and security (nginx):

This is of course a simplified diagram for the sake of illustration

What is Elasticsearch

It is distributed document stores which means once the document is stored then it can be retrieved from any node of the cluster.

default port for elasticsearch is 9201
default port for kibana is 5600

It shows the data on real time basis, for example, day-wise or hourly to the user. Kibana UI is user friendly and very easy for a beginner to understand. The following table gives a direct comparison between these terms−

Every row in RDBMS has an unique row identifier and similarly we have unique document id in elasticsearch for every document.

Elastichsarch built on top of Lucene. Every shard is simply a Lucene index. Lucene index, if simplified, is the inverted index. Every Elasticsearch index is a bunch of shards or Lucene indices. When you query for a document, Elasticsearch will subquery all shards, merge results and return it to you. When you index document to Elasticsearch, the Elasticsearch will calculate in which shard document should be written using the formula

shard = hash(routing) % number_of_primary_shards

Key Concepts

In Elasticsearch terms, Index = Database, Type = Table, Document = Row.

The key concepts of Elasticsearch are as follow

Node
- It refers to a single running instance of Elasticsearch.
Cluster
- It is a collection of one or more nodes. Cluster provides collective indexing and search capabilities across all the nodes for entire data.
Index
- It is a collection of different type of documents and their properties. Index also uses the concept of shards to improve the performance
Document
- It is a collection of fields in a specific manner defined in JSON format. Every document belongs to a type and resides inside an index. Every document is associated with a unique identifier called the UID.
Shard
- Indexes are horizontally subdivided into shards. This means each shard contains all the properties of document but contains less number of JSON objects than index. The horizontal separation makes shard an independent node, which can be store in any node.
Replicas
- Elasticsearch allows a user to create replicas of their indexes and shards.

Checking that Elasticsearch is running

You can test that your Elasticsearch node is running by sending an HTTP request to port 9200 on localhost:

curl -X GET "localhost:9200/?pretty"

response will be as similar to following.

{
  "name" : "BLR-AA202394",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "pYm2eSPlTh-EUo8hcVce2A",
  "version" : {
    "number" : "7.15.2",
    "build_flavor" : "default",
    "build_type" : "zip",
    "build_hash" : "93d5a7f6192e8a1a12e154a2b81bf6fa7309da0c",
    "build_date" : "2021-11-04T14:04:42.515624022Z",
    "build_snapshot" : false,
    "lucene_version" : "8.9.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

Rest API in EL

Every feature of Elasticsearch is exposed as a REST API:

Index API – Used to document the Index
Get API – Used to retrieve the document
Search API – Used to submit your query and get the result
Put Mapping API – Used to override default choices and define our own mapping

Elasticsearch has its own Query Domain Specific Language, where you specify the query in JSON format. Other queries can also be nested based on your need. Real projects require search on different fields by applying some conditions, different weights, recent documents, values of some predefined fields, and so on. All such complexity can be expressed through a single query.

Document analysis process by Elasticsearch

When a document request for indexing is received by elasticsearch, which in turn is handled by lucene, it converts the document in a stream of tokens. After tokens are generated, the same gets filtered by the configured filter. This entire process is called the analysis process, and is applied on every document that gets indexed.

One thing to learn from this is that the key to an efficient storage and retrieval process is the analysis process defined on the index, as per the application needs.

Sometime we also do

stemming and this means we convert the word into root word and store in index (Ex: such words are population, populations)
For synonym, we store single word rather than storing all the words (declined, reduced)

Remember that this same process of document normalization (during inverted index ) needs to be applied during document search query.

The analyzer can be custom or default one and this needs to be specified during Index creation. If the analyzer is not provided then EL uses default standard one.

Remember that, EL standard analyzer will not do do

stemming (Running to Run)- process of converting word to root word
stop word removal

Inverted Index

It uses a method called inverted index. Since we are talking about searching a term in a large collection of documents(aka collection of chapters in this case) we can use Inverted Indexes to solve this issue, and yes almost all books use these Inverted Indexes to make your life easier. Just like many other books "Team of Rivals" has inverted indexes at the end of the book as shown in this image.

An inverted index consists of a list of all the unique words that appear in any document, and for each word, a list of the documents in which it appears.

So after checking the Inverted indexes at the end of the book we know that "Baltimore" is mentioned on pages 629 and 630. So there are two parts in this searching for "Baltimore" in the lexicographically ordered Inverted Index list and fetching the pages based on the value of the index (here 629 and 630). The search time is very less for the term in the inverted index since in computing we actually use dictionaries(hash-based or search trees) to keep track of these terms

The purpose of an inverted index, is to store text in a structure that allows for very efficient and fast full-text searches. When performing full-text searches, we are actually querying an inverted index and not the JSON documents that we defined when indexing the documents.

An inverted index consists of all of the unique terms that appear in any document covered by the index.

Here is the diagram which shows a very simplified structure of an inverted index. Here we have separated stop words which are common ones and they also should be excluded from queries.

Post the analysis process, when the data is converted into tokens, these tokens are stored into an internal structure called inverted index. This structure maps each unique term in an index to a document. This data structure allows for faster data search and text analytics. All the attributes like term count, term position and other such attributes are associated with the term. Below is a sample visualization of how an inverted index may look like.

Post the tokens are mapped, document is stored on the disk. One can choose to store the original input of the document along with the analyzed document. The original input gets stored in a system field names "_source". Once can even choose to not analyze the input, and store the document without any analysis. The structure of the inverted index totally depends upon the analyzer chosen for indexing.

Another example:

Document 1: Elasticsearch is fast

Document 2: I want to learn Elasticsearch

Let’s take a peek into the Inverted Index and see the result of the Analysis and Indexing process:

As you can see, the terms are counted and mapped into document identifiers and its position in the document. The reason we don’t see the full document Elasticsearch is fast or I want to learn Elasticsearch is because they go through Analysis process, which is our main topic in this article.

Operation with Elasticsearch

Inserting a document in elasticsearch means Indexing a document

When you execute the PUT command for the fresh document, Then the document will be created. Such a case result attribute from the response will incite as “created”. When you execute the PUT command for the
second time for the same document, it will not create the new document again if the document already exists in the cluster. Instead, it will update the document. You can confirm that by the result from the
response. The result will indicate as “updated”. But when you execute the PUT command for the fresh document, Then the document will be created. If it's already available, then the document will be updated.

What is indexing

Index as a verb

The process of inserting a document into elasticsearch is called indexing. Indexing a document into elasticsearch is similar to insert the row in a database table. The only difference is, if the document already exists in the cluster, the indexing process will replace the old document.

Indexing a document from Kibana

Before indexing a document into kibana, first we need to decide where this document will be stored. In elasticsearch, a document belongs to a type and the type stays within an Index. An elasticearch cluster may contain multiple indexes and V7 allows only one type for per index. One index which belong to one type, may contain multiple documents and each document may have multiple fields.

Base command for an indexing a document is

{{host}}/:index/_alias/:alias

Here is a sample indexing which will create the index and have document added. I have also added following index pattern into elasticsearch.yml to enable auto indexing.

action.auto_create_index: .monitoring*,.watches,.triggered_watches,.watcher-history*,.ml*,.doc*,.json*,_json*,_doc*,*subscriber*,*product*,*event*

Note: Passing _doctype value (here it is json) is deprecated into current V7.

Indexing request:

Here intentionally i have used PUT request as this one requires ID value to be provided. My aim is here is to search by subscriberId and hence I am passing the ID value. Otherwise we can use POST command like “ POST /subscriber/json/” which will be auto generate id.

PUT /subscriber/json/5159683
{
    "Addresses": [
        {
            "City": "TX",
            "Country": "USA",
            "LineOne": "Parkway Suite Houston",
            "PostalCode": "77047",
            "State": "TX",
            "DefaultBilling": true,
            "DefaultHome": false,
            "DefaultPostal": true,
            "DefaultService": true,
            "DefaultShipping": false,
            "DefaultWork": false,
            "Id": 1412213,
            "Created": "2021-11-22T16:05:40.943Z",
            "Modified": "2021-11-22T16:05:40.943Z",
            "Name": "Parkway E Suite Houston",
            "ShipToName": "India",
            "Status": 1,
            "StatusName": "Active",
            "TaxPCode": null,
            "Verifiable": true,
            "Verified": true
        }
    ],
    "Subscriber": {
        "Category": 1,
        "CompanyName": "IH-163759714",
        "ConvergentBillerId": "40003771",
        "Created": "2021-11-22T16:05:40.890Z",
        "ExternalReference": "IH-EX163759714",
        "Id": 5159683,
        "Language": "en-GB",
        "Login": "aa@mallinator.com",
        "State": 0,
        "StateChangeDate": "2021-11-22T16:05:40.840Z",
        "Status": 1,
        "SubscriberTypeCode": 10121,
        "AdditionalProperties": [
            {
                "ExternalReference": "Invoice_Ref",
                "Id": 1886,
                "Name": "CustomerInvoiceRef",
                "Values": []
            },
            {
                "ExternalReference": "Bill_From_Address",
                "Id": 1888,
                "Name": "BillFromAddress",
                "Values": [
                    "add-sws"
                ]
            }
        ],
        "ContactPreferences": [
            {
                "ContactEventType": 12,
                "ContactMethod": 40,
                "Id": 2702216,
                "OptIn": true
            }
        ],
        "EffectiveStartDate": "2021-11-22T16:05:40.863Z",
        "HomeCountry": "USA",
        "InvoiceConfiguration": {
            "HideZeroAmount": false,
            "InvoiceDetailLevel": 1
        },
        "StateName": "Prospect",
        "StatusName": "Active",
        "SubscriberCurrency": "USD",
        "SubscriberTypeDetails": {
            "AccountingMethod": 2,
            "BillCycle": "40002163",
            "BillCycleDay": 1,
            "BillCycleName": "BillCycle_1",
            "IsReadOnly": false,
            "PaymentDueDays": 10,
            "PostpaidAccountNumber": "CAQ400037719"
        },
        "TermsAndConditionsAccepted": "2021-11-22T16:05:40.883Z"
    }
}

Indexing response:

here the type has been set with our input value
index has been set to our input subscriber
result value has set as created
sequence no has ben set as 3 and this is being set by elasticsearch

#! [types removal] Specifying types in document index requests is deprecated, use the typeless endpoints instead (/{index}/_doc/{id}, /{index}/_doc, or /{index}/_create/{id}).
{
  "_index" : "subscriber",
  "_type" : "json",
  "_id" : "5159684",
  "_version" : 1,
  "result" : "created",
  "_shards" : {
    "total" : 2,
    "successful" : 1,
    "failed" : 0
  },
  "_seq_no" : 3,
  "_primary_term" : 1
}

Here is the request when i passed on the input for an existing id and result value shows as updated which means the indexing has been updated. version has been set to 3 and this means the document has been updated 3 times. Every document in elasticsearch has version. So PUT request for existing document will update the version and for new request it will create

The _index mentions the index name for this document.

Any attribute starts with _ with document are called document metadata.

#! [types removal] Specifying types in document index requests is deprecated, use the typeless endpoints instead (/{index}/_doc/{id}, /{index}/_doc, or /{index}/_create/{id}).
{
  "_index" : "subscriber",
  "_type" : "json",
  "_id" : "5159683",
  "_version" : 3,
  "result" : "updated",
  "_shards" : {
    "total" : 2,
    "successful" : 1,
    "failed" : 0
  },
  "_seq_no" : 2,
  "_primary_term" : 1
}

Finding if a document available?

The HEAD command is being used to see if the document is available and this will respond us http 200 as shared below.

HEAD /subscriber/_doc/5159685

200 - OK

Retrieve an Index document

Here we are passing the Index name, document type and Id to retrieve the index.

GET /subscriber/json/5159685

and response will have the document along with its metadata. The found has value set to true which means the document has been found.

{
  "_index" : "subscriber",
  "_type" : "json",
  "_id" : "5159685",
  "_version" : 1,
  "_seq_no" : 7,
  "_primary_term" : 1,
  "found" : true,
  "_source" : {
    "Addresses" : [
      {
        "City" : "TX",
        "Country" : "USA",
        "LineOne" : "Parkway E Suite Houston",
        "PostalCode" : "77047",
        "State" : "TX",
        "DefaultBilling" : true,
        "DefaultHome" : false,
        "DefaultPostal" : true,
        "DefaultService" : true,
        "DefaultShipping" : false,
        "DefaultWork" : false,
        "Id" : 1412213,
        "Created" : "2021-11-22T16:05:40.943Z",
        "Modified" : "2021-11-22T16:05:40.943Z",
        "Name" : "Parkway E Suite 100 Houston",
        "ShipToName" : "MARLINK GROUP USA - HOUSTON",
        "Status" : 1,
        "StatusName" : "Active",
        "TaxPCode" : null,
        "Verifiable" : true,
        "Verified" : true
      }
    ],
    "Subscriber" : {
      "Category" : 1,
      "CompanyName" : "IH1-163759714",
      "ConvergentBillerId" : "400033771",
      "Created" : "2021-11-22T16:05:40.890Z",
      "ExternalReference" : "IH-EX163759714",
      "Id" : 5159685,
      "Language" : "en-GB",
      "Login" : "IH16375339714@aaar.com",
      "State" : 0,
      "StateChangeDate" : "2021-11-22T16:05:40.840Z",
      "Status" : 1,
      "SubscriberTypeCode" : 10121,
      "AdditionalProperties" : [
        {
          "ExternalReference" : "Invoice_Ref",
          "Id" : 1886,
          "Name" : "CustomerInvoiceRef",
          "Values" : [ ]
        },
        {
          "ExternalReference" : "Bill_From_Address",
          "Id" : 1888,
          "Name" : "BillFromAddress",
          "Values" : [
            "xxxx"
          ]
        }
      ],
      "ContactPreferences" : [
        {
          "ContactEventType" : 12,
          "ContactMethod" : 40,
          "Id" : 2702216,
          "OptIn" : true
        }
      ],
      "EffectiveStartDate" : "2021-11-22T16:05:40.863Z",
      "HomeCountry" : "USA",
      "InvoiceConfiguration" : {
        "HideZeroAmount" : false,
        "InvoiceDetailLevel" : 1
      },
      "StateName" : "Prospect",
      "StatusName" : "Active",
      "SubscriberCurrency" : "USD",
      "SubscriberTypeDetails" : {
        "AccountingMethod" : 2,
        "BillCycle" : "40002163",
        "BillCycleDay" : 1,
        "BillCycleName" : "BillCycle_1",
        "IsReadOnly" : false,
        "PaymentDueDays" : 10,
        "PostpaidAccountNumber" : "CAQ400037719"
      },
      "TermsAndConditionsAccepted" : "2021-11-22T16:05:40.883Z"
    }
  }
}

Updating an Index document

Having a PUT request for an ID will update if the document is already existing. The response will have result as updated.

Note that PUT request will update the whole document with new content and that means our new PUT request needs to contain all the elements. PUT request update the complete document.

If we want to update partially and don't want pass on all document elements then we should use POST request. POST request update the relevant element only.

Key point to note here are:

POST command has _update to let elasticsearch know about update
doc keyword has to be added as elastic search stores documents within doc
if we provide any new attribute into post request then it will be added into document.
Following logic gets triggered by elasticsearch after post request
- retrieves the existing document
- apply changes requested in post request
- removes the old document
- indexes new document (with update) in the place of old document

POST /subscriber/_update/5159683
{
  "doc":
  {
    "Addresses": [
        {
            "City": "Austin"

        }
    ]
  }

}

Deleting an document

Request is similar to get request

DELETE /subscriber/json/5159699

The response will have result element populated with deleted value.

{
  "_index" : "subscriber",
  "_type" : "json",
  "_id" : "5159699",
  "_version" : 2,
  "result" : "deleted",
  "_shards" : {
    "total" : 2,
    "successful" : 1,
    "failed" : 0
  },
  "_seq_no" : 33,
  "_primary_term" : 2
}

Bulk Insert

For bulk request, individual items are separated by newline characters (not commas) and th*ere are no square brackets at the end (ie the payload is a sequence of JSON documents*. The documents must be on a single line, no newlines are allowed within them. It seems bit unintuitive I know since resources don't have to be on a single line when you create them using PUT or POST for non-bulk operations

POST _bulk
{"index": {"_index": "subscriber", "_id": "5159199"}}
{"Subscriber": {"Category": 1, "CompanyName": "IH-163759714","ConvergentBillerId": "40003771", "Created": "2021-11-22T16:05:40.890Z","ExternalReference": "IH-EX163759714","Id": 5159199,"Language": "en-GB","Login": "1@1.com","State": 0, "StateChangeDate": "2021-10-22T16:05:40.840Z","Status": 1,"SubscriberTypeCode": 10121}}
{"index": {"_index": "subscriber", "_id": "5159099"}}
{"Subscriber": {"Category": 1, "CompanyName": "IH-163759714","ConvergentBillerId": "40003771", "Created": "2021-11-22T16:05:40.890Z","ExternalReference": "IH-EX163759714","Id": 5159099,"Language": "en-GB","Login": "1@1.com","State": 0, "StateChangeDate": "2021-10-22T16:05:40.840Z","Status": 2,"SubscriberTypeCode": 10122}}

Bulk insert response will return status of each document insert as shared below.

status code will be set to 201

{
  "took" : 20,
  "errors" : false,
  "items" : [
    {
      "index" : {
        "_index" : "subscriber",
        "_type" : "_doc",
        "_id" : "5159199",
        "_version" : 1,
        "result" : "created",
        "_shards" : {
          "total" : 2,
          "successful" : 1,
          "failed" : 0
        },
        "_seq_no" : 34,
        "_primary_term" : 2,
        "status" : 201
      }
    },
    {
      "index" : {
        "_index" : "subscriber",
        "_type" : "_doc",
        "_id" : "5159099",
        "_version" : 1,
        "result" : "created",
        "_shards" : {
          "total" : 2,
          "successful" : 1,
          "failed" : 0
        },
        "_seq_no" : 35,
        "_primary_term" : 2,
        "status" : 201
      }
    }
  ]
}

Bulk update

Here i am updating all the attributes of the documents.

POST _bulk
{"index": {"_index": "subscriber", "_id": "5159199"}}
{"Subscriber": {"Category": 1, "CompanyName": "IH-163759714","ConvergentBillerId": "40003771", "Created": "2021-11-22T16:05:40.890Z","ExternalReference": "IH-EX163759714","Id": 5159199,"Language": "en-GB","Login": "1@1.com","State": 1, "StateChangeDate": "2021-10-22T16:05:40.840Z","Status": 1,"SubscriberTypeCode": 10121}}
{"index": {"_index": "subscriber", "_id": "5159099"}}
{"Subscriber": {"Category": 1, "CompanyName": "IH-163759714","ConvergentBillerId": "40003771", "Created": "2021-11-22T16:05:40.890Z","ExternalReference": "IH-EX163759714","Id": 5159099,"Language": "en-GB","Login": "1@1.com","State": 1, "StateChangeDate": "2021-10-22T16:05:40.840Z","Status": 2,"SubscriberTypeCode": 10122}}

The response has status code set to 200 as shared below.

{
  "took" : 9,
  "errors" : false,
  "items" : [
    {
      "index" : {
        "_index" : "subscriber",
        "_type" : "_doc",
        "_id" : "5159199",
        "_version" : 2,
        "result" : "updated",
        "_shards" : {
          "total" : 2,
          "successful" : 1,
          "failed" : 0
        },
        "_seq_no" : 36,
        "_primary_term" : 2,
        "status" : 200
      }
    },
    {
      "index" : {
        "_index" : "subscriber",
        "_type" : "_doc",
        "_id" : "5159099",
        "_version" : 2,
        "result" : "updated",
        "_shards" : {
          "total" : 2,
          "successful" : 1,
          "failed" : 0
        },
        "_seq_no" : 37,
        "_primary_term" : 2,
        "status" : 200
      }
    }
  ]
}

Full text Search basics of Elasticsearch

Note: Whatever analyzer was using during Index creation, the same will be used during query operation. Every query term goes through analysis process.

English analyzer in EL does following

does tokenization based on white space
does stemming
removes stop words

Getting a document by Id

GET /subscriber/_doc/5159199

The response has found set to true as shared below which indicate that the id has been found. The original document is found under metadata "_source".

{
  "_index" : "subscriber",
  "_type" : "_doc",
  "_id" : "5159199",
  "_version" : 2,
  "_seq_no" : 36,
  "_primary_term" : 2,
  "found" : true,
  "_source" : {
    "Subscriber" : {
      "Category" : 1,
      "CompanyName" : "IH-163759714",
      "ConvergentBillerId" : "40003771",
      "Created" : "2021-11-22T16:05:40.890Z",
      "ExternalReference" : "IH-EX163759714",
      "Id" : 5159199,
      "Language" : "en-GB",
      "Login" : "1@1.com",
      "State" : 1,
      "StateChangeDate" : "2021-10-22T16:05:40.840Z",
      "Status" : 1,
      "SubscriberTypeCode" : 10121
    }
  }
}

Searching an Index

The following one will allow to search for all documents into the index.

GET /subscriber/_search

The search result will have following metadata and element hits.value.total will indicate how many documents been returned. Here timed_out set to false and this means request has not been timed out. the default timeout set to is 60 sec and we can specify this value in the request also to override the default value.

The value max_score equal to 1.0 means all the search results are relevant to search query.

{
    "took": 11,
    "timed_out": false,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 12,
            "relation": "eq"
        },
        "max_score": 1.0,
        "hits": [{
                "_index": "subscriber",
                "_type": "json",
                "_id": "5159698",
                "_score": 1.0,
                "_source": {
                ..........
                }
            }
        ]
    }
}

Searching by query string

we can pass on query string as shown below accompanied by q. Here elasticsearch is searching all the documents for the value 10122 irrespective of the field.

GET /subscriber/_search?q="10122"

The response has relation field value set to eq which means search happened with document string compare. The search result is ordered by _score and 1st one has _score se to 2/302585 as it had maximum time of query string 10122 match found.

{
  "took" : 429,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 6,
      "relation" : "eq"
    },
    "max_score" : 2.302585,
    "hits" : [
      {
        "_index" : "subscriber",
        "_type" : "json",
        "_id" : "49",
        "_score" : 2.302585,
        "_source" : {
          "Subscriber" : {
            "Category" : 1,
            "CompanyName" : "IH-10122",
            "ConvergentBillerId" : "10122",
            "Created" : "2021-11-22T16:05:40.890Z",
            "ExternalReference" : "IH-EX163759714",
            "Id" : 49,
            "Language" : "en-GB",
            "Login" : "1@1.com",
            "State" : 1,
            "StateChangeDate" : "2021-10-22T16:05:40.840Z",
            "Status" : 1,
            "SubscriberTypeCode" : 10121
          }
        }
      },
      {
        "_index" : "subscriber",
        "_type" : "json",
        "_id" : "79",
        "_score" : 1.7917595,
        "_source" : {
          "Subscriber" : {
            "Category" : 1,
            "CompanyName" : "IH-163759714",
            "ConvergentBillerId" : "10122",
            "Created" : "2021-11-22T16:05:40.890Z",
            "ExternalReference" : "IH-EX163759714",
            "Id" : 79,
            "Language" : "en-GB",
            "Login" : "1@1.com",
            "State" : 1,
            "StateChangeDate" : "2021-10-22T16:05:40.840Z",
            "Status" : 2,
            "SubscriberTypeCode" : 10122
          }
        }
      },
      {
        "_index" : "subscriber",
        "_type" : "json",
        "_id" : "5159099",
        "_score" : 1.0,
        "_source" : {
          "Subscriber" : {
            "Category" : 1,
            "CompanyName" : "IH-163759714",
            "ConvergentBillerId" : "40003771",
            "Created" : "2021-11-22T16:05:40.890Z",
            "ExternalReference" : "IH-EX163759714",
            "Id" : 5159099,
            "Language" : "en-GB",
            "Login" : "1@1.com",
            "State" : 1,
            "StateChangeDate" : "2021-10-22T16:05:40.840Z",
            "Status" : 2,
            "SubscriberTypeCode" : 10122
          }
        }
      },
      {
        "_index" : "subscriber",
        "_type" : "json",
        "_id" : "59099",
        "_score" : 1.0,
        "_source" : {
          "Subscriber" : {
            "Category" : 1,
            "CompanyName" : "IH-163759714",
            "ConvergentBillerId" : "40003771",
            "Created" : "2021-11-22T16:05:40.890Z",
            "ExternalReference" : "IH-EX163759714",
            "Id" : 59099,
            "Language" : "en-GB",
            "Login" : "1@1.com",
            "State" : 1,
            "StateChangeDate" : "2021-10-22T16:05:40.840Z",
            "Status" : 2,
            "SubscriberTypeCode" : 10122
          }
        }
      },
      {
        "_index" : "subscriber",
        "_type" : "json",
        "_id" : "599",
        "_score" : 1.0,
        "_source" : {
          "Subscriber" : {
            "Category" : 1,
            "CompanyName" : "IH-163759714",
            "ConvergentBillerId" : "40003771",
            "Created" : "2021-11-22T16:05:40.890Z",
            "ExternalReference" : "IH-EX163759714",
            "Id" : 599,
            "Language" : "en-GB",
            "Login" : "1@1.com",
            "State" : 1,
            "StateChangeDate" : "2021-10-22T16:05:40.840Z",
            "Status" : 2,
            "SubscriberTypeCode" : 10122
          }
        }
      },
      {
        "_index" : "subscriber",
        "_type" : "json",
        "_id" : "19",
        "_score" : 1.0,
        "_source" : {
          "Subscriber" : {
            "Category" : 1,
            "CompanyName" : "IH-163759714",
            "ConvergentBillerId" : "40003771",
            "Created" : "2021-11-22T16:05:40.890Z",
            "ExternalReference" : "IH-EX163759714",
            "Id" : 19,
            "Language" : "en-GB",
            "Login" : "1@1.com",
            "State" : 1,
            "StateChangeDate" : "2021-10-22T16:05:40.840Z",
            "Status" : 2,
            "SubscriberTypeCode" : 10122
          }
        }
      }
    ]
  }
}

Searching by specific element of document

We can pass document element name into query string and this will return all the documents matching with the value of the specific field.

GET /subscriber/_search?q=SubscriberTypeCode:"10121"

The response will have max_score value of 1 for all the returned elements as this is value compare by element name.

{
  "took" : 0,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 2,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "subscriber",
        "_type" : "json",
        "_id" : "149",
        "_score" : 1.0,
        "_source" : {
          "Category" : 1,
          "CompanyName" : "IH-10122",
          "ConvergentBillerId" : "12",
          "Created" : "2021-11-22T16:05:40.890Z",
          "ExternalReference" : "IH-EX163759714",
          "Id" : 149,
          "Language" : "en-GB",
          "Login" : "1@1.com",
          "State" : 1,
          "StateChangeDate" : "2021-10-22T16:05:40.840Z",
          "Status" : 1,
          "SubscriberTypeCode" : 10121
        }
      },
      {
        "_index" : "subscriber",
        "_type" : "json",
        "_id" : "139",
        "_score" : 1.0,
        "_source" : {
          "Category" : 1,
          "CompanyName" : "IH-10122",
          "ConvergentBillerId" : "12",
          "Created" : "2021-11-22T16:05:40.890Z",
          "ExternalReference" : "IH-EX163759714",
          "Id" : 139,
          "Language" : "en-GB",
          "Login" : "1@1.com",
          "State" : 1,
          "StateChangeDate" : "2021-10-22T16:05:40.840Z",
          "Status" : 1,
          "SubscriberTypeCode" : 10121
        }
      }
    ]
  }
}

Query DSL

If we see the above query curl command then it is as below and the curl command has the query in the url which can be exploited by hackers.

curl -XGET "http://localhost:9200/subscriber/_search?q=SubscriberTypeCode:"10121""

So instead of passing query into url, we will pass the query into the body which is more secured. All query of EL will be under query string and here is one example of match query where value is being compared.

GET /subscriber/_search
{
  "query":
  {
    "match": {
      "SubscriberTypeCode": "10122"
    }
  }
}

match

Now, lets see the corresponding curl command for this query dsl one and the url does not have any more id or field details. So query is specified in request body instead of url.

curl -XGET "http://localhost:9200/subscriber/_search" -H 'Content-Type: application/json' -d'
{
  "query":
  {
    "match": {
      "SubscriberTypeCode": "10122"
    }
  }
}'

Now, lets see the following query and its response below. The field value is being sent in lower case but still the query returned result. This is because during indexing, EL does the analysis process first which involves converting into lower case, stemming, root word etc.

GET /subscriber/_search
{
  "query":
  {
    "match": {
      "ExternalReference": "ih-EX163759714"
    }
  }
}

and the response of the above query

{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 4,
      "relation" : "eq"
    },
    "max_score" : 0.21072102,
    "hits" : [
      {
        "_index" : "subscriber",
        "_type" : "json",
        "_id" : "149",
        "_score" : 0.21072102,
        "_source" : {
          "Category" : 1,
          "CompanyName" : "IH-10122",
          "ConvergentBillerId" : "12",
          "Created" : "2021-11-22T16:05:40.890Z",
          "ExternalReference" : "IH-EX163759714",
          "Id" : 149,
          "Language" : "en-GB",
          "Login" : "1@1.com",
          "State" : 1,
          "StateChangeDate" : "2021-10-22T16:05:40.840Z",
          "Status" : 1,
          "SubscriberTypeCode" : 10121
        }
      },
      {
        "_index" : "subscriber",
        "_type" : "json",
        "_id" : "179",
        "_score" : 0.21072102,
        "_source" : {
          "Category" : 1,
          "CompanyName" : "IH-163759714",
          "ConvergentBillerId" : "16",
          "Created" : "2021-11-22T16:05:40.890Z",
          "ExternalReference" : "IH-EX163759714",
          "Id" : 179,
          "Language" : "en-GB",
          "Login" : "1@1.com",
          "State" : 1,
          "StateChangeDate" : "2021-10-22T16:05:40.840Z",
          "Status" : 2,
          "SubscriberTypeCode" : 10122
        }
      },
      {
        "_index" : "subscriber",
        "_type" : "json",
        "_id" : "139",
        "_score" : 0.21072102,
        "_source" : {
          "Category" : 1,
          "CompanyName" : "IH-10122",
          "ConvergentBillerId" : "12",
          "Created" : "2021-11-22T16:05:40.890Z",
          "ExternalReference" : "IH-EX163759714",
          "Id" : 139,
          "Language" : "en-GB",
          "Login" : "1@1.com",
          "State" : 1,
          "StateChangeDate" : "2021-10-22T16:05:40.840Z",
          "Status" : 1,
          "SubscriberTypeCode" : 10121
        }
      },
      {
        "_index" : "subscriber",
        "_type" : "json",
        "_id" : "169",
        "_score" : 0.21072102,
        "_source" : {
          "Category" : 1,
          "CompanyName" : "IH-163759714",
          "ConvergentBillerId" : "16",
          "Created" : "2021-11-22T16:05:40.890Z",
          "ExternalReference" : "IH-EX163759714",
          "Id" : 169,
          "Language" : "en-GB",
          "Login" : "1@1.com",
          "State" : 1,
          "StateChangeDate" : "2021-10-22T16:05:40.840Z",
          "Status" : 2,
          "SubscriberTypeCode" : 10122
        }
      }
    ]
  }
}

Limiting search result

Here, we are limiting result by size parameter.

GET /blogposts/_search
{
  "query": {
    "match": {
      "status": "draft"
    }
  },
  "size": 5

The following query shows how we can implement pagination in search query.

GET /blogposts/_search
{
  "query": {
    "match": {
      "status": "draft"
    }
  },
  "size": 5,
  "from": 10  
}

Limiting Number of elements/attrbutes

Sometime, the input document may have lot of elements and we may not need all the elements in search result. In elastic search, search result stays within source attribute and in following way we can reduce number of elements.

GET /blogposts/_search
{
  "query": {
    "match": {
      "status": "draft"
    }
  },
  "size": 5,
  "from": 5,
  "_source": ["title","status"]  
}

There can be scenarios, where we want to exclude only one field and in that scenario, we need to add excludes conditions.

GET /blogposts/_search
{
  "query": {
    "match": {
      "status": "draft"
    }
  },
  "size": 5,
  "from": 5,
  "_source": {
    "excludes": "tags"
  }

match_all

The following match_all allows elastic search to query all and return all documents from the index. Here within _source, we are specifying what are elements required from documents.

GET /blogposts/_search
{
  "query": {
    "match": {
      "status": "draft"
    }
  },
  "size": 5,
  "from": 5,
  "_source": ["title","status"]

multi_match

it allows to search a query string for two elements in a document as shared below. It does full text search.

GET /subscriber/_search
{
  "query": {
    "multi_match": {
      "query": "ih",
      "fields": ["ExternalReference","CompanyName"]
    }
  }
}

curl -XGET "http://localhost:9200/subscriber/_search" -H 'Content-Type: application/json' -d'
{
  "query": {
    "multi_match": {
      "query": "ih",
      "fields": ["ExternalReference","CompanyName"]
    }
  }
}'

post_filter

The post_filter is applied to the search hits at the very end of a search request, after aggregations have already been calculated. Its purpose is best explained by example:

Imagine that you are selling shirts that have the following properties:

PUT /shirts
{
    "mappings": {
        "_doc": {
            "properties": {
                "brand": { "type": "keyword"},
                "color": { "type": "keyword"},
                "model": { "type": "keyword"}
            }
        }
    }
}

PUT /shirts/_doc/1?refresh
{
    "brand": "gucci",
    "color": "red",
    "model": "slim"
}

Imagine a user has specified two filters:

color:red and brand:gucci. You only want to show them red shirts made by Gucci in the search results. Normally you would do this with a bool query:

GET /shirts/_search
{
  "query": {
    "bool": {
      "filter": [
        { "term": { "color": "red"   }},
        { "term": { "brand": "gucci" }}
      ]
    }
  }
}

However, you would also like to use faceted navigation to display a list of other options that the user could click on. Perhaps you have a model field that would allow the user to limit their search results to red Gucci t-shirts or dress-shirts.

This can be done with a terms aggregation:

GET /shirts/_search
{
  "query": {
    "bool": {
      "filter": [
        { "term": { "color": "red"   }},
        { "term": { "brand": "gucci" }}
      ]
    }
  },
  "aggs": {
    "models": {
      "terms": { "field": "model" } 
    }
  }
}

	Returns the most popular models of red shirts by Gucci.

But perhaps you would also like to tell the user how many Gucci shirts are available in other colors. If you just add a terms aggregation on the color field, you will only get back the color red, because your query returns only red shirts by Gucci.

Instead, you want to include shirts of all colors during aggregation, then apply the colors filter only to the search results. This is the purpose of the post_filter:

GET /shirts/_search
{
  "query": {
    "bool": {
      "filter": {
        "term": { "brand": "gucci" } 
      }
    }
  },
  "aggs": {
    "colors": {
      "terms": { "field": "color" } 
    },
    "color_red": {
      "filter": {
        "term": { "color": "red" } 
      },
      "aggs": {
        "models": {
          "terms": { "field": "model" } 
        }
      }
    }
  },
  "post_filter": { 
    "term": { "color": "red" }
  }
}

	The main query now finds all shirts by Gucci, regardless of color.
	The `colors` agg returns popular colors for shirts by Gucci.
	The `color_red` agg limits the `models` sub-aggregation to red Gucci shirts.
	Finally, the `post_filter` removes colors other than red from the search `hits`.

must and must_not condition

This condition allows us to match one condition and stops another condition. Ex: Our requirement is to filter all documents where subtypeid is not 10222 but externalreference is having IH then for the 1st clause we need to use must not and 2nd one shd have must.

GET /subscriber/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "match": {
            "ExternalReference": "ih"
          }
        }

      ]
      , "must_not": [
        {
          "match": {
            "SubscriberTypeCode": "10121"
          }

        }
      ]
    }
  }
}

sorting

We need to add sorting on the query result. Here you can see the sort is placed outside of query as the query result is passed to sorting.

Note: Text fields are analyzed and then tokenized. So it is not possible to sort the text fields.

GET /blogposts/_search
{
  "query": {
    "match": {
      "status": "draft"
    }
  },
  "size": 10,
  "from": 3,
  "_source": ["title","status"],
  "sort": [
    {
      "published_date": {
        "order": "desc"
      }
    }

If we have to sort text field then we need to use keyword fields as they are non-analyzed but that will be slow.

Full match condition

Here, we are doing full match which is identified by operator and and this is based on title field.

GET /blogposts/_search
{
  "query": {
    "match": {
      "title":{
        "query":"Introduction to elasticsearch"
        , "operator": "and"
      }
    }
  }

Minimum match condition

It is sometime necessary to match defined words at minimum level and we don't want full all words to be matched as per our search text.

GET /blogposts/_search
{
  "query": {
    "match": {
      "title":{
        "query":"Introduction to elasticsearch", 
        "minimum_should_match": 2
      }
    }
  }

}

we also can give % wise match rather than specifying perfect no of terms.

GET /blogposts/_search
{
  "query": {
    "match": {
      "title":{
        "query":"Introduction to elasticsearch", 
        "minimum_should_match": "25%"
      }
    }
  }

}

Term and match query

Term query does not go through analysis process. match query goes through analysis process. Lets understand through following example.

Lets see the following two example of match query returning us records. This is because the elasticsearch is storing Draft as draft due to lowercase in analysis process. Hence searching by Draft or draft will return us same set of query results.

GET /blogposts/_search
{
  "query": {
    "match": {
      "status": "draft"
    }
  }
}


GET /blogposts/_search
{
  "query": {
    "match": {
      "status": "Draft"
    }
  }
}

Now, lets look at the term query and since elasticsearch has lowercase filter here so all texts are stored in lower case,

1st query here will not return any data
2nd query here will return data

GET /blogposts/_search
{
  "query": {
    "term": {
      "status": {
        "value": "Draft"
      }
    }
  }
}

GET /blogposts/_search
{
  "query": {
    "term": {
      "status": {
        "value": "draft"
      }
    }
  }
}

terms filter

This is used when we want to add multiple term filter together. Here “tags” is the document element name and “elasticsearch” and “fast” are element value to compare. Note: This is or condition and any document having tags value of elasticsearch or fast will be returned in result.

GET /blogposts/_search
{
  "query": {
    "terms": {
      "tags": [
        "elasticsearch",
        "fast"
      ]
    }
  }
}

prefix query

Remember that, this query does not go through analysis process and hence exact string needs to be passed. It does not calculate score and all score remains as 1.

GET  /blogposts/_search
{
  "query": {
    "prefix": {
      "title": {
        "value": "intro"
      }
    }
  }
}

wildcard query

This is same as prefix query.

GET  /blogposts/_search
{
  "query": {
    "wildcard": {
      "title": {
        "value": "n*ro"
      }
    }
  }
}

Complex Queries

Lets take the following requirement and see how we can write the query.

blogpost must not be in draft status
blogpost should have elasticsearch
blogpost should have like > 10

Lets break this business problem into multiple phases.

Solution

First, we try to achieve the condition to remove draft blogposts from query.
1. We need query here for searching
2. Since there will be multiple conditions, so we need to use bool type of query
3. we need to filter blogposts which are in draft status and this is case insensitive search. So we i have used must_not and then match filter with field name status

GET  /blogposts/_search
{
  "query": {
    "bool": {
      "must_not": [
        {
          "match": {
            "status": "draft"
          }
        }
      ]
    }
  }
}

Now, we will need to allow all blogpost which are having elasticsearch into tag.
1. So we need to use must clause into bool query
2. we will then use match clause as we want to have case insensitive search

GET  /blogposts/_search
{
  "query": {
    "bool": {
      "must_not": [
        {
          "match": {
            "status": "draft"
          }
        }
      ],
      "must": [
        {
          "match": {
            "tags": "elasticsearch"
          }
        }
      ]
    }
  }
}

Now we need to add condition of greater than.
1. So we need filter condition as it will take search result out to further filter data as per where clause.
2. we need to add range condition here as our condition of >10 is based on range

So, our *final query will be *

GET  /blogposts/_search
{
  "query": {
    "bool": {
      "must_not": [
        {
          "match": {
            "status": "draft"
          }
        }
      ],
      "must": [
        {
          "match": {
            "tags": "elasticsearch"
          }
        }
      ],
      "filter": [
        {"range": {
          "no_of_likes": {
            "gte": 10
          }
        }}
      ]
    }
  }
}

Query Relevence

We can use the score value in the search result to understand how the document is relevant to original query. Relevancy score is represented by a positive floating number.

The scoring of a document is determined based on the field matches from the query specified and any additional configurations you apply to the search.

Refer the article How scoring works for more in-depth analysis on this.

Document matching happens in

binary sense (matching with exact values)
relevancy sense matching

Scoring Function

Uses following and this means more time a term appears in a document then the document is more relevant while more time the term appears across other documents then it is less relevant.

Term Frequency (TF)
Inverse document Frequency (IDF)

When a document matches the query, Lucene calculates the score by combining the score of each matching term. This scoring calculation is done by the practical scoring function.

score(q,d)  =  
           queryNorm(q)  
          · coord(q,d)    
          · ∑ (           
                tf(t in d)   
              · idf(t)²      
              · t.getBoost() 
              · norm(t,d)    
            ) (t in q)

score(q,d) is the relevance score of document d for query q.
queryNorm(q) is the query normalization factor.
coord(q,d) is the coordination factor.
The sum of the weights for each term t in the query q for document d.
- tf(t in d) is the term frequency for term t in document d.
- idf(t) is the inverse document frequency for term t.
- t.getBoost() is the boost that has been applied to the query.
- norm(t,d) is the field-length norm, combined with the index-time field-level boost, if any.

Boosting

The _boost field (document-level boost) was removed, but field-level boosts, and query boosts still work just fine. If we want to boost matches on field1:

Boosting can be of:

Index time boosting
- stores at index. boost value cant be changed in future other than re-indexing
query time boosting
- Allows to define at query time and can be changed anytime

"bool": {
    "should": [{
        "terms": {
            "field1": ["67", "93", "73", "78", "88", "77"],
            "boost": 2.0
        }
    }, {
        "terms": {
            "field2": ["68", "94", "72", "76", "82", "96", "70", "86", "81", "92", "97", "74", "91", "85"]
        }
    }, {
        "terms": {
            "category": ["cat2"]
        }
    }]
}

We can add also like this,,,

A class of boost 2 means it is twice as important compare to others
Boost figure is relative.
Tough to finalize correct boost value and this is more an iterative process to find out optimal relevency score

GET /blogposts/_search
{
  "query": {
    "bool": {
      "should": [
        {
          "match": {
            "title":{
              "query": "Elasticsearch",
              "boost": 2
            }
          }
        },
        {
          "match": {
            "content":{
              "query": "Elasticsearch"

            }
          }
        }
      ]
    }
  }

}

Filter Search basics of Elasticsearch

Filter condition allows us to add more condition into search criteria. Here we see first condition is must and this means it will retrieve all documents where ExternalReference has ih text. This is being set as must. Next condition is filter based one which will be triggered. here filter condition will be based on status greater than 1. Here since i need range bound one so have used range filter.

Always first filter context gets executed and then query context gets executed. This is to ensure filtered objects are available for document search by query objects. Score does not gets calculated for filter query results by elasticsearch. Score only gets calculated for query context.

GET /subscriber/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "match": {
            "ExternalReference": "ih"
          }
        }

      ]
      , "filter": [
        {
          "range": {
            "Status": {
              "gte": 1
            }
          }
        }
      ]
    }
  }
}

There can be filter based on term which is where we compare the value. term filter has been used here to compare the value. in case of date range, we need to use range filter.

GET /subscriber/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "match": {
            "ExternalReference": "ih"
          }
        }

      ]
      , "filter": [
        {
          "term": {
            "Status": "2"
          } 
        }
      ]
    }
  }
}

Cluster Node

Cluster has a master node and any node can become master node. Every node knows where each document leaves. When an index is sharded, a given document within that index will only be stored within one of the shards.

There are two main reasons why sharding is important, with the first one being that it allows you to split and thereby scale volumes of data. So if you have growing amounts of data, you will not face a bottleneck because you can always tweak the number of shards for a particular index. The other reason why sharding is important, is that operations can be distributed across multiple nodes and thereby parallelized. This results in increased performance, because multiple machines can potentially work on the same query. This is completely transparent to you as a user of Elasticsearch.

So how do you specify the number of shards an index has? You can optionally specify this at index creation time, but if you don’t, a default number of 5 will be used. This is sufficient in most cases, since it allows for a good amount of growth in data before you need to worry about adding additional shards.

Primary and Replica shard

There are two types of shards: primaries and replicas. Each document in an index belongs to one primary shard. ... The number of primary shards in an index is fixed at the time that an index is created, but the number of replica shards can be changed at any time, without interrupting indexing or query operations.

Elasticsearch uses the concept of the shard to subdivide the index into multiple pieces and allows us to make one or more copies of index shards called replicas. Please refer to this SO answer to get a detailed understanding of shards and replicas.

To set the number of shards and replicas as properties of index: Here it means number of primary shard will be 6 and for each primary shard there will be 2 replica shard available.

PUT /indexName

{
  "settings": {
    "index": {
      "number_of_shards": 6,
      "number_of_replicas": 2
    }
  }
}

Some important tips for choosing the number of shards and replicas:

The number of shards cannot be changed after an index is created. If you later find it necessary to change the number of shards, then you will have to reindex all the documents again.
To decide no of shards, you will have to choose a starting point and then try to find the optimal size through testing with your data and queries.
Replicas tend to improve search performance (not always). But, it is recommended to have at least 1 replica (so that data is preserved in case of hardware failure)
Refer this medium article, that states that number of nodes and number of shards (primary shard + replicas), should be proportional to each other. This is important for Elasticsearch to ensure proper load balancing.
As stated in this article it is recommended to keep the number of shards per node below 20 per GB heap it has configured.
According to this blog when you’re planning for capacity, try and allocate shards at a rate of 150% to 300% (or about double) the number nodes that you had when initially configuring your datasets

Replication serves two purposes, with the main one being to provide high availability in case nodes or shards fail. For replication to even be effective if something goes wrong, replica shards are never allocated to the same nodes as the primary shards, which you can also see on the above diagram. This means that even if an entire node fails, you will have at least one replica of any primary shards on that particular node. The other purpose of replication — or perhaps a side benefit — is increased performance for search queries. This is the case because searches can be executed on all replicas in parallel, meaning that replicas are actually part of the cluster’s searching capabilities.

Replication between Primary and Replica

So how does Elasticsearch keep everything in sync? Elasticsearch uses a model named primary-backup for its data replication.

Adding, updating, or removing documents — are sent to the primary shard. The primary shard is then responsible for validating the operations and ensuring that everything is good. This involves checking if the request is structurally invalid, such as trying to add a number to an object field or something like that. When the operation has been accepted by the primary shard, the operation will be performed locally on the primary shard itself. When the operation completes, the operation will be forwarded to each of the replica shards in the replica group. If the shard has multiple replicas, the operation will be performed in parallel on each of the replicas. When the operation has completed successfully on every replica and responded to the primary shard, the primary shard will respond to the client that the operation has completed successfully. This is all illustrated in the below diagram.

Search Operation and its combine

There are two phases of search operation in elasticsearch,

query all the matching documents from all the sherds
combine all the search records in master node from all the sherds for the respective documents

Distributed searching

A search request can be accepted by any node in the Elasticsearch cluster. Each of the nodes in the cluster is aware of all the other nodes. The catch being that, the node receiving the search request, by default is unaware of where the data that is to be queried resides. Hence, the node has no choice but to broadcast the request to all the shards and then gather their responses into a globally sorted result set that it can return to the client.

The process of determining which shard a particular document resides in is termed as routing. By default, routing is handled by Elasticsearch. The default routing scheme hashes the ID of a document and uses it as the partition key to find a shard. This includes both user-provided IDs and randomly generated IDs picked by Elasticsearch. Documents are assigned shards by hashing the document ID, dividing the hash by the number of shards and taking the remainder.

Index Components

Manually Index creation

We can create index manually and mainly we need to provide shard and replica related configuration.

PUT /blogposts/
{
  "settings": {
    "number_of_shards": 2
    , "number_of_replicas": 1
  }

}

Here, we can define index element datatype and it is better that we always define explicitly datatype rather than relying on Elasticsearch.

Key datatype are:

string
Float
Double
date
boolean
array (Can only hold one datatype data. It cant hold multiple data type like string and Boolean)
Keyword

PUT /blogposts/_mapping
{
  "properties":{
    "title":{"type":"text"},
    "content": {"type": "text"},
    "published_date":{"type":"date"}
  }
}

Response from Elasticsearch will be acknowledgement one.

{
  "acknowledged" : true
}

How index components are?

We can find out details of index by GET command with Index name.

GET /subscriber/

Elasticsearch defines index fields mapping and its data types during 1st document insert by default. So if later new datatype has been sent into any field then Elasticsearch will give error. So it is always good to define the index mapping explictly rather than allowing Elasticsearch to assume it by default.

The search result will have following 3 components

aliases
settings
- It will have following informations
- number_of_shards (Elasticsearch will set this by default to 1)
- provided_name
- index creation date
- uuid (Unique identifier of the index)
- number_of_replicas (Elasticsearch will set this by default to 1)

    "settings" : {
        "index" : {
          "routing" : {
            "allocation" : {
              "include" : {
                "_tier_preference" : "data_content"
              }
            }
          },
          "number_of_shards" : "1",
          "provided_name" : "subscriber",
          "creation_date" : "1637599846785",
          "number_of_replicas" : "1",
          "uuid" : "L4jp0hosRq2Pjqk3hHxP5w",
          "version" : {
            "created" : "7150299"
          }
        }
      }

mappings
- defines how the index should be distributed. Mapping is either generated automatically or we can define explicitly.
- It will have all element names for documents we have indexed.
- Elasticsearch detects field types from its values.

  "PostalCode" : {
                "type" : "text",
                "fields" : {
                  "keyword" : {
                    "type" : "keyword",
                    "ignore_above" : 256
                  }
                }
              }

{
  "subscriber" : {
    "aliases" : { },
    "mappings" : {
      "properties" : {
        "Addresses" : {
          "properties" : {
            "City" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "Country" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "Created" : {
              "type" : "date"
            },
            "DefaultBilling" : {
              "type" : "boolean"
            },
            "DefaultHome" : {
              "type" : "boolean"
            },
            "DefaultPostal" : {
              "type" : "boolean"
            },
            "DefaultService" : {
              "type" : "boolean"
            },
            "DefaultShipping" : {
              "type" : "boolean"
            },
            "DefaultWork" : {
              "type" : "boolean"
            },
            "Id" : {
              "type" : "long"
            },
            "LineOne" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "Modified" : {
              "type" : "date"
            },
            "Name" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "PostalCode" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "ShipToName" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "State" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "Status" : {
              "type" : "long"
            },
            "StatusName" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "Verifiable" : {
              "type" : "boolean"
            },
            "Verified" : {
              "type" : "boolean"
            }
          }
        },
        "Category" : {
          "type" : "long"
        },
        "CompanyName" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "ConvergentBillerId" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "Created" : {
          "type" : "date"
        },
        "ExternalReference" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "Id" : {
          "type" : "long"
        },
        "Language" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "Login" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "State" : {
          "type" : "long"
        },
        "StateChangeDate" : {
          "type" : "date"
        },
        "Status" : {
          "type" : "long"
        },
        "Subscriber" : {
          "properties" : {
            "AdditionalProperties" : {
              "properties" : {
                "ExternalReference" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "Id" : {
                  "type" : "long"
                },
                "Name" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "Values" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                }
              }
            },
            "Category" : {
              "type" : "long"
            },
            "CompanyName" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "ContactPreferences" : {
              "properties" : {
                "ContactEventType" : {
                  "type" : "long"
                },
                "ContactMethod" : {
                  "type" : "long"
                },
                "Id" : {
                  "type" : "long"
                },
                "OptIn" : {
                  "type" : "boolean"
                }
              }
            },
            "ConvergentBillerId" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "Created" : {
              "type" : "date"
            },
            "EffectiveStartDate" : {
              "type" : "date"
            },
            "ExternalReference" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "HomeCountry" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "Id" : {
              "type" : "long"
            },
            "InvoiceConfiguration" : {
              "properties" : {
                "HideZeroAmount" : {
                  "type" : "boolean"
                },
                "InvoiceDetailLevel" : {
                  "type" : "long"
                }
              }
            },
            "Language" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "Login" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "State" : {
              "type" : "long"
            },
            "StateChangeDate" : {
              "type" : "date"
            },
            "StateName" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "Status" : {
              "type" : "long"
            },
            "StatusName" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "SubscriberCurrency" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "SubscriberTypeCode" : {
              "type" : "long"
            },
            "SubscriberTypeDetails" : {
              "properties" : {
                "AccountingMethod" : {
                  "type" : "long"
                },
                "BillCycle" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "BillCycleDay" : {
                  "type" : "long"
                },
                "BillCycleName" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "IsReadOnly" : {
                  "type" : "boolean"
                },
                "PaymentDueDays" : {
                  "type" : "long"
                },
                "PostpaidAccountNumber" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                }
              }
            },
            "TermsAndConditionsAccepted" : {
              "type" : "date"
            }
          }
        },
        "SubscriberTypeCode" : {
          "type" : "long"
        },
        "TermsAndConditionsAccepted" : {
          "type" : "date"
        }
      }
    },
    "settings" : {
      "index" : {
        "routing" : {
          "allocation" : {
            "include" : {
              "_tier_preference" : "data_content"
            }
          }
        },
        "number_of_shards" : "1",
        "provided_name" : "subscriber",
        "creation_date" : "1637599846785",
        "number_of_replicas" : "1",
        "uuid" : "L4jp0hosRq2Pjqk3hHxP5w",
        "version" : {
          "created" : "7150299"
        }
      }
    }
  }
}

Analyzer

An analyzer — whether built-in or custom — is just a package which contains three lower-level building blocks: character filters, tokenizers, and token filters.

Character filters

Character filters are used to preprocess the stream of characters before it is passed to the tokenizer. An example is HTML stripc character filter that strips HTML elements from a text and replaces HTML entities with their decoded value (e.g, replaces & with &).

Tokenizers

A tokenizer receives a stream of characters, breaks it up into individual tokens (usually individual words), and outputs a stream of tokens.

Token filters

Token filters accept a stream of tokens from a tokenizer and can modify tokens (eg lowercasing), delete tokens (eg remove stopwords) or add tokens (eg synonyms). An example is a lowercase token filter that simply changes token text to lowercase. ASCII folding token filter converts alphabetic, numeric, and symbolic characters that are not in the Basic Latin Unicode block (first 127 ASCII characters) to their ASCII equivalent, if one exists. For example, the filter changes à to a.

There can be multiple token filters as shown and each token filter does different work. There can be different query time and Index time analyzer.

Characteristics of an analyzer**

An analyzer may have zero or more character filters, which are applied in order.
An analyzer must have exactly one tokenizer.
An analyzer may have zero or more token filters, which are applied in order.
Tokenizer generates tokens, which will be passed on to the token filter and then eventually become terms in the inverted index.
Certain tokenizers like ngram, edgengram can generate lots of tokens, which can cause higher disk usage.

The analyzer will affect how we search the text, but it won’t affect the content of the text itself. With this example, if we search for let, the Elasticsearch will still return the full text Let’s build an autocomplete! instead of only let.

Elasticsearch’s Analyzer has three components you can modify depending on your use case:

Character Filters
Tokenizer
Token Filter

Character Filter

The character filter has the ability to perform addition,removal or replacement actions on the input text given to them.

Here is a sample request

POST _analyze/?pretty
{
  "tokenizer": "standard",
  "char_filter": ["html_strip"],
  "text": "The <b> Auto-generation </b> is a success"

  }

curl -XPOST "http://localhost:9200/_analyze/?pretty" -H 'Content-Type: application/json' -d'
{
  "tokenizer": "standard",
  "char_filter": ["html_strip"],
  "text": "The <b> Auto-generation </b> is a success"

  }'

The response shows that, html tag has been removed and the resulting tokens are like below:

  "tokens" : [
    {
      "token" : "The",
      "start_offset" : 0,
      "end_offset" : 3,
      "type" : "<ALPHANUM>",
      "position" : 0
    },
    {
      "token" : "Auto",
      "start_offset" : 8,
      "end_offset" : 12,
      "type" : "<ALPHANUM>",
      "position" : 1
    },
    {
      "token" : "generation",
      "start_offset" : 13,
      "end_offset" : 23,
      "type" : "<ALPHANUM>",
      "position" : 2
    },
    {
      "token" : "is",
      "start_offset" : 29,
      "end_offset" : 31,
      "type" : "<ALPHANUM>",
      "position" : 3
    },
    {
      "token" : "a",
      "start_offset" : 32,
      "end_offset" : 33,
      "type" : "<ALPHANUM>",
      "position" : 4
    },
    {
      "token" : "success",
      "start_offset" : 34,
      "end_offset" : 41,
      "type" : "<ALPHANUM>",
      "position" : 5
    }
  ]
}

Tokenizer

The input text after its transformation from the Character filter is passed to the tokeniser. The tokenizer would split this input text into individual tokens (or terms) at specific characters. The default tokenizer in elasticsearch is the “standard tokeniser”, which uses the grammar based tokenisation technique.

The above command shows the resulting tokens are like below:

“The”,”Auto”,”generation”,”is”,”a”,”success”

The words are split whenever there is a white-space and also a hyphen (-).

Standard Tokenizer

default analyzer of the Elasticsearch
It uses grammar based Tokenization specified in https://unicode.org/reports/tr29/
Does following
- Tokenizer
- Standard Tokenizer
- Token Filters
- Lower Case Token Filter
- Stop Token Filter (disabled by default)

Simple Analyzer

The simple analyzer breaks text into tokens at any non-letter character, such as numbers, spaces, hyphens and apostrophes, discards non-letter characters, and changes uppercase to lowercase. It does not remove stop words.

POST _analyze
{
  "analyzer": "simple",
  "text": "The 2 QUICK Brown-Foxes jumped over the lazy dog's bone."
}

The simple analyzer parses the sentence and produces the following tokens:

[ the, quick, brown, foxes, jumped, over, the, lazy, dog, s, bone ]

Whitespace analyzer

The whitespace analyzer breaks text into terms whenever it encounters a whitespace character. The whitespace analyzer is not configurable. If you need to customize the whitespace analyzer then you need to recreate it as a custom analyzer and modify it, usually by adding token filters. This would recreate the built-in whitespace analyzer and you can use it as a starting point for further customization:

POST _analyze
{
  "analyzer": "whitespace",
  "text": "The 2 QUICK Brown-Foxes jumped over the lazy dog's bone."
}

The above sentence would produce the following terms:

[ The, 2, QUICK, Brown-Foxes, jumped, over, the, lazy, dog's, bone. ]

Stop analyzer

The stop analyzer is the same as the simple analyzer but adds support for removing stop words. It defaults to using the _english_ stop words.

POST _analyze
{
  "analyzer": "stop",
  "text": "The 2 QUICK Brown-Foxes jumped over the lazy dog's bone."
}

The above sentence would produce the following terms:

[ quick, brown, foxes, jumped, over, lazy, dog, s, bone ]

Keyword analyzer

The keyword analyzer is a “noop” analyzer which returns the entire input string as a single token.

POST _analyze
{
  "analyzer": "keyword",
  "text": "The 2 QUICK Brown-Foxes jumped over the lazy dog's bone."
}

The above sentence would produce the following single term:

[ The 2 QUICK Brown-Foxes jumped over the lazy dog's bone. ]

Fingerprint analyzer

The fingerprint analyzer implements a fingerprinting algorithm which is used by the OpenRefine project to assist in clustering.

Input text is lowercased, normalized to remove extended characters, sorted, deduplicated and concatenated into a single token. If a stop word list is configured, stop words will also be removed.

creates fingerprint which can be used for duplicate detection
input is lowercased, normalized to remove extended character, sorted, de-duplicated and concatenated

POST _analyze
{
  "analyzer": "fingerprint",
  "text": "Yes The customer is not honest. He is not reliable for company."
}

Output:

{
  "tokens" : [
    {
      "token" : "company customer for he honest is not reliable the yes",
      "start_offset" : 0,
      "end_offset" : 63,
      "type" : "fingerprint",
      "position" : 0
    }
  ]
}

Custom Analyzer

Here for custom analyzer, we have provided following

analyzer name which is being set as amit_custom_analyze
- analyzer type is set to custom
- char_filter is set to html stripping. we can have 0 or more character filter and hence it is array type
- tokenizer filter is set to standard one
- token filter is being set as lowercase to convert into lower case

This custom analyzer now can be associated with any of the index element.

PUT  /blogposts
{
  "settings": {
    "number_of_shards": 2,
    "number_of_replicas": 1,
    "analysis": {
      "analyzer": {
        "amit_custom_analyze":{
          "type": "custom",
          "char_filter": ["html_strip"],
          "tokenizer": "standard",
          "token_filters": ["lowercase"]

        }
      }
    }
  }
}

Response:

{
  "acknowledged" : true,
  "shards_acknowledged" : true,
  "index" : "blogposts"
}

Custom Index with Custom Analyzer

Here:

mapping provides each element types and its allowed data type
setting provides Index metadata attributes

PUT  /blogposts
{
   "settings": {
    "number_of_shards": 2,
    "number_of_replicas": 1,
    "analysis": {
      "analyzer": {
        "amit_custom_analyze":{
          "type": "custom",
          "char_filter": ["html_strip"],
          "tokenizer": "standard",
          "token_filters": ["lowercase"]

        }
      }
    }
  },
  "mappings": {
    "properties": {
      "title":{"type": "text", "analyzer": "amit_custom_analyze"},
      "content":{"type": "text", "analyzer": "amit_custom_analyze"},
      "published_date":{"type": "date"},
      "no_of_likes":{"type": "text"}
    }
  }

}

response:

{
  "acknowledged" : true,
  "shards_acknowledged" : true,
  "index" : "blogposts"
}

Here is one example of using custom analyzer during analysis process:

Example shows

html content removed
converted to lower case

POST  /blogposts/_analyze
{
  "analyzer": "amit_custom_analyze",
  "text": ["This is <html> kayal exploring elasticsearch"]
}

{
  "tokens" : [
    {
      "token" : "This",
      "start_offset" : 0,
      "end_offset" : 4,
      "type" : "<ALPHANUM>",
      "position" : 0
    },
    {
      "token" : "is",
      "start_offset" : 5,
      "end_offset" : 7,
      "type" : "<ALPHANUM>",
      "position" : 1
    },
    {
      "token" : "kayal",
      "start_offset" : 15,
      "end_offset" : 20,
      "type" : "<ALPHANUM>",
      "position" : 2
    },
    {
      "token" : "exploring",
      "start_offset" : 21,
      "end_offset" : 30,
      "type" : "<ALPHANUM>",
      "position" : 3
    },
    {
      "token" : "elasticsearch",
      "start_offset" : 31,
      "end_offset" : 44,
      "type" : "<ALPHANUM>",
      "position" : 4
    }
  ]
}

How to create customer tokenizer and filters

Here, I have shown an example of Index creation command where custom tokenizer and filters have been defined.

punctuation is the name of the tokenizer and similarly symbol is the name of the char_filter
tokenizer will look for pattern @# and tokenize
symbol will look for emoji :) and :( to convert them into happy and sad

PUT  /blogposts
{
   "settings": {
    "number_of_shards": 2,
    "number_of_replicas": 1,
    "analysis": {
      "analyzer": {
        "amit_custom_analyze":{
          "type": "custom",
          "tokenizer": "punctuation",
          "char_filter":"symbol"
        }
      },
      "tokenizer": {
        "punctuation":{
          "type":"pattern",
          "pattern":"@#"
        }
      },
      "char_filter": {
        "symbol":{
          "type":"mapping",
          "mappings": [":) ==> happy",":( ==> sad"]
        }
      }
    }
  },
  "mappings": {
      "properties": {
      "title":{"type": "text", "analyzer": "amit_custom_analyze"},
      "content":{"type": "text", "analyzer": "amit_custom_analyze"},
      "published_date":{"type": "date"},
      "no_of_likes":{"type": "text"}
    }
    }

  }

Aggregation

Every aggregation is combination of one or more buckets and metrices. Aggregation refers to the collection of documents or a set of documents obtained from a particular search query or filter. Aggregation forms the main concept to build the desired visualization in Kibana.

Aggregation always works on top of query and filter result. This is true unless we use post_filter.

Buckets

A bucket mainly consists of a key and a document. When the aggregation is executed, the documents are placed in the respective bucket.
Bucket aggregations categorize sets of documents as buckets. The type of bucket aggregation determines whether a given document falls into a bucket or not.
use bucket aggregations to implement faceted navigation (usually placed as a sidebar on a search result landing page) to help you’re users narrow down the results.
Buckets are similar to GROUP BY

Note: Here the tag is keyword field and not an analyzed one. If this is analyzed one then we need nested raw one.

We can have sub-aggregation within buckets also.

Metric aggregations—This aggregation helps in calculating matrices from the fields of aggregated document values.

Pipeline aggregations—As the name suggests, this aggregation takes input from the output results of other aggregations.

Matrix aggregations (still in the development phase)—These aggregations work on more than one field and provide statistical results based on the documents utilized by the used fields.

Steps are

Document ==> Filter ==> Query ==> Query Result ==> Aggregation ==> Aggregation Result

GET /blogposts/_search
{
  "aggs": {
    "tag_wise_stats": {
      "terms": {
        "field": "tags",
        "size": 10
      }
    }
  }
}

GET /shirts/_search
{
  "query": {
    "bool": {
      "filter": [
        { "term": { "color": "red"   }},
        { "term": { "brand": "gucci" }}
      ]
    }
  },
  "aggs": {
    "models": {
      "terms": { "field": "model" } 
    }
  }
}

Metrices

Metric Aggregation mainly refers to the maths calculation done on the documents present in the bucket. For example if you choose a number field the metric calculation you can do on it is COUNT, SUM, MIN, MAX, AVERAGE etc.

Logstash

Logstash is typically used as the “processing” engine for any log management solution (or systems that
deal with changing data streams).

These data can be structured, semi-structured, or unstructured, and can have many different schemas. To Logstash, all these data are “logs” containing “events”. Logstash can easily parse and filter out the data from these log events using one or more filtering plugins that come with it. Finally, it can send the filtered output to one or more destinations. Again, there are prebuilt output interfaces that make this task simple.

Logstash itself doesn’t access the source system and collect the data, it uses input plugins to ingest the data from various sources.

Note: There’s a multitude of input plugins available for Logstash such as various log files, relational databases, NoSQL databases, Kafka queues, HTTP endpoints, S3 files, CloudWatch Logs, log4j events or Twitter feed.

Once data is ingested, one or more filter plugins take care of the processing part in the filter stage.

Here is a sample logstash config file which allows to push data into elasticsearch host. This config file which allows logstash to takes data from STDIN and and pushes into blogposts index of elasticsearch running in localhost port 9200.

input { stdin {codec => json } }
output {
  elasticsearch { 
  hosts => ["localhost:9200"]
  index => "blogposts"
  document_type => _doc
  }
  stdout { }
}

Filters are an in-line processing mechanism that provide the flexibility to slice and dice your data to fit your needs. Let’s take a look at some filters in action. The following configuration file sets up the grok and date filters.

input { stdin { } }

filter {
  grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
  }
  date {
    match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
  }
}

output {
  elasticsearch { hosts => ["localhost:9200"] }
  stdout { codec => rubydebug }
}

Sample Filter Plugin

Reference Architecture

References

42 Elasticsearch Query Examples – Hands-on Tutorial

Demand Forecasting with AWS Forecast

Amit Kayal — Sat, 02 Oct 2021 12:24:08 +0000

What is forecasting?

A time series essentially is a series of quantitative values. These values are obtained over time, and often have equal time intervals between them. These intervals can be quite different and may consist of yearly, quarterly, monthly or hourly buckets for instance.

Time-series methods are:

Moving Average
Autoregression
Vector Autoregression
Autoregressive Integrated Moving Average
Autoregressive Moving Average

Components of Time Series

Key classifications of the components of the time series are:

Random or Irregular movements.
Cyclic Variations.
Seasonal Variations.
Trend.

Why Amazon Forecast

Fully managed service that uses machine learning to deliver highly accurate forecasts.
Based on the same machine learning forecasting technology used by Amazon.com.
Automated machine learning
- Includes AutoML capabilities that take care of the machine learning for you.
Works with any historical time series data to create accurate forecasts
- in a retail scenario, Amazon Forecast uses machine learning to process your time series data (such as price, promotions, and store traffic) and combines that with associated data (such as product features, floor placement, and store locations) to determine the complex relationships between them.
- So, it can combine combining time series data with additional variables for time series prediction

How Amazon Forecast works?

Here is the flow diagram taken from AWS site and key points here to note are

Historical data can be pushed into S3
Forecast can be triggered on data arrival in S3
Output of forecast can be pushed into S3 for further actions.

Forecasting automation with Amazon Forecast by applying MLOps

The following model architecture taken from AWS site allows us to build, train, and deploy a time-series forecasting model leveraging an MLOps pipeline encompassing Amazon Forecast, AWS Lambda, and AWS Step Functions. To visualize the generated forecast, we will use a combination of AWS serverless analytics services such as Amazon Athena and Amazon QuickSight.

Key components of this architecture are

dataset is uploaded to the Amazon S3 cloud storage under the /**train** directory (prefix).
uploaded file triggers Lambda, which initiates the MLOps pipeline built using a Step Functions state machine.
The state machine stitches together a series of Lambda functions to build, train, and deploy a ML model in Amazon Forecast.
Amazon CloudWatch, which captures Forecast metrics is being used for log analysis
SNS being used for Forecasting job status change notification
- final forecasts become available in the source Amazon S3 bucket in the **/forecast** directory.
- ML pipeline saves any old forecasts in the **/history** directory.

Forecast Workflow

The workflow to generating forecasts consists of the following steps.

Creating related datasets and a dataset group
Retrieving training data
Training predictors (trained model) using an algorithm or AutoML
Evaluating the predictor with metrics
Creating a forecast
Retrieving forecast for users

Amazon Forecast supports the following dataset domains:

RETAIL Domain – For retail demand forecasting
INVENTORY_PLANNING Domain – For supply chain and inventory planning
EC2 CAPACITY Domain – For forecasting Amazon Elastic Compute Cloud (Amazon EC2) capacity
WORK_FORCE Domain – For work force planning
WEB_TRAFFIC Domain – For estimating future web traffic
METRICS Domain – For forecasting metrics, such as revenue and cash flow
CUSTOM Domain – For all other types of time-series forecasting

Example 1: Dataset Types in the RETAIL Domain

If you are a retailer interested in forecasting demand for items, you might create the following datasets in the RETAIL domain:

Target time series is the required dataset of historical time-series demand (sales) data for each item (each product a retailer sells). In the RETAIL domain, this dataset type requires that the dataset includes the item_id, timestamp, and the demand fields. The demand field is the forecast target, and is typically the number of items sold by the retailer in a particular week or day.
Optionally, a dataset of the related time series type. In the RETAIL domain, this type can include optional, but suggested, time-series information such as price, inventory_onhand, and webpage_hits.
Optionally, a dataset of the item metadata type. In the RETAIL domain, Amazon Forecast suggests providing metadata information related to the items that you provided in target time series, such as brand, color, category, and genre.

A case study

I took the dataset from Kaggle Store Item Demand Forecasting Challenge which has given 5 years of store-item sales data, and asked to predict 3 months of sales for 50 different items at 10 different stores.

Here is the way I have used AWS Forecasting with minimal coding.

Import your data

Dataset Details

Following are the details needs to be provided.

Dataset name
Frequency of your data’
Data schema
- I have used here schema builder option which is more visual one. Another option is json schema which allows us to specify AttributeName and AttributeType in the JSON format.
- Forecast data schema has concept of domain to make our dataset creation much easier. I have selected retail domain option here and forecast has guided me to have following attributes.
- item_id (attribute type: string) - Mandatory by forecast
- timestamp(attribute type: timestamp and have selected format as yyyy-mm-dd) - Mandatory by forecast
- demand(attribute type float) - Mandatory by forecast
- store (attribute type: string) - Had to add this as my forecast has to be based on timestamp, store id and item id.
- It is essential that All attributes displayed must exist in your CSV file and must be ordered in the same order that they appear in your CSV file

I have used following python code to process my dataset from kaggle.

train_df = pd.read_csv("train.csv")
train_df["timestamp"]=pd.to_datetime(train_df['timestamp'])
train_df["timestamp"]=pd.to_datetime(train_df['timestamp'],
                                     format='%Y%m%d %H:%M:%S')
train_df_final = train_df_final[['item_id', 'timestamp', 'demand', 'store']]
train_df_final.to_csv("train_df.csv",
                      index=False,
                     date_format='%Y-%m-%d')

Upload dataset into AWS S3

Create an AWS S3 bucket, and upload the time-series data into the bucket.

s3.create_bucket(Bucket=bucketName)
s3.upload_file(Filename="data/item-demand-time.csv", Bucket=bucketName, Key=key)

Dataset import details

Following are the details needs to be provided for Import task.

Dataset import name
Select time zone Info
- My dataset does not have TZ as any variable and so I have selected the option of do
Data location Info
- This is input file path from my S3 bucket which needs to be provided
IAM Role Info
- Dataset groups require permissions from IAM to read your dataset files in S3.

Now it will give us option to import and once this is done then we should get Successfully imported your data.

Train a predictor

Train a predictor, a custom model with underlying infrastructure that Amazon Forecast trains on your datasets.
Following are the key parameters required here.

Additional configurations to be set during this phase include

How the training dataset is to be split into training and testing dataset ?
How the missing data is to be addressed ?
How the model validation is to be performed (i.e., back test window in the context of time-series analysis)
How many times the model validation is to be performed during the model training phase (i.e., number of back test windows in the context of time-series analysis)
What is the forecast horizon ?

Predictor settings

Forecast name
Forecast horizon
- This number tells Amazon Forecast how far into the future to predict your data at the specified forecast frequency.
Forecast frequency
- My data set has timestamp daily and hence i have set this as 1 day

Predictor details

Algorithm selection
- Here i have selected the option of AutoML which allows me to let Amazon Forecast choose the right algorithm for dataset.
Optimization metric
- I have selected default here.
- Amazon Forecast provides Root Mean Square Error (RMSE), Weighted Quantile Loss (wQL), Average Weighted Quantile Loss (Average wQL), Mean Absolute Scaled Error (MASE), Mean Absolute Percentage Error (MAPE), and Weighted Absolute Percentage Error (WAPE) metrics to evaluate your predictors.
Forecast dimensions
- Item id is used in training by default and that has been added as mandatory by Forecast. Additionally I have selected Store because my aim is to have forecast based on store and item id.
Forecast type
- Choose up to 5 quantiles between 0.01 and 0.99 (by increments of 0.01). AWS allows us to have by default 0.1,0.5 and 0.9.

Advanced Configuration

Here this is the default FeaturizationMethod being recommended by Amazon Forecast. Provides information about the method that featurizes (transforms) a dataset field.

Here this method is only being applied for my element demand which is specified by AttrubuteName.

[
    {
        "AttributeName": "demand",
        "FeaturizationPipeline": [
            {
                "FeaturizationMethodName": "filling",
                "FeaturizationMethodParameters": {
                    "aggregation": "sum",
                    "frontfill": "none",
                    "middlefill": "zero",
                    "backfill": "zero"
                }
            }
        ]
    }
]

Supplementary features

This is quite crucial information and can impact business problem sometime.

Weather info
- Amazon Forecast Weather Index combines multiple weather metrics from historical weather events and current forecasts at a given location to increase your demand forecast model accuracy.
- In retail inventory management use cases, day-to-day weather variation impacts foot traffic and product mix.
Holiday info

Create a Forecaster

Once the Predictor is trained, it is to be prepared to provide the forecasting.

create_forecast_response = forecast.create_forecast(
                           ForecastName=forecastName,
                           PredictorArn=predictorArn)

Following key inputs needs to be provided from console and then we can start the process of forecast.

Forecast name
Predictor (This is the one was created in earlier step)
Forecast types
- By default, Amazon Forecast will generate forecasts for 0.10, 0.50 and 0.90 quantiles.

Make Forecasts

Now we are ready to make forecasts. In our case, we are going to write the forecasted outputs back in S3 bucket.

Following inputs needs to be provided.

Start date Info
End date

I have given below a snapshot of the forecasts which I got using the Predictor that I trained.

References

Build a custom model for object and logo detection

Amit Kayal — Sat, 25 Sep 2021 13:30:58 +0000

Build a custom model for object and logo detection

Let’s have a quick look at AWS ML stack which offers almost all the ML aspects to solve our business problem.

What is Object detection?

It is a task to detect instance of certain classes from images or in video frame. Our aim is to find out what object is where in an image/video frame. It is the heart of other object dependent task like segmentation, image caption, object tracking etc.

It is quite important to understand the difference between Image recognition and Image detection. First one only cares about detecting the object while the later one aims to detecting the desired object location.

How does Object detection works?

Deep learning-based object detection models have two parts. An encoder which takes an image as input and runs it through a series of blocks(generally they are combination of convolution, BN and Max Pooling) and layers that learn to extract statistical features used to locate and label objects. Outputs from the encoder are then passed to a decoder, which predicts bounding boxes and labels for each object.

Object detectors output the location and label for each object, but how do we know how well the model is doing? For an object’s location, the most commonly-used metric is intersection-over-union (IOU). Given two bounding boxes, we compute the area of the intersection and divide by the area of the union. This value ranges from 0 (no interaction) to 1 (perfectly overlapping). For labels, a simple “percent correct” can be used. Here is an example which shows this.

Popular Models for Object Detection

Models uses either two stage or one stage object detection and it has been seen that one stage one is much faster. Two-stage detectors have high localization and object recognition accuracy, whereas the one-stage detectors achieve high inference speed.

I have shown here how from same input image, we can use deep learning models for classification, detection (one stage/two stage) or semantic segmentation ones.

Here is the list of most popular ones.

Why Rekognition

Object detection requires quite in-depth knowledge on Deep learning and network design. This is what AWS Rekognition does the heavy lifting and allows developer to build serverless application while AWS service is managing the deep learning aspects.

Amazon Rekognition Image offers APIs to detect objects and scenes, detect and analyze faces, recognize celebrities, detect inappropriate content, and search for similar faces in a collection of faces, along with APIs to manage resources.

Training for custom model

We can train a model by using the Amazon Rekognition Custom Labels console, or by the Amazon Rekognition Custom Labels API. You are charged for the amount of time that it takes to successfully train a model. Typically training takes from 30 minutes to 24 hours to complete.

Custom Labelling and why Rekognition

It is quite common request from customer to hear “Amazon Rekognition Labels is not specific enough for what I need for my business” and lets understand now how we AWS does the heavy lifting for such scenarios.

In order to handle above custom labelling request, we have two approach

DIY

Customization requires expertise and resources for managing

Deep Learning model training and fine-tune
Thousands of image to collect and labelling manually
- Error prone and subject which can lead to inconsistencies
Take weeks to complete labelling and model training

Amazon Rekognition Custom Labels

Rekognition Custom Labels builds off of Rekognition’s existing capabilities, which are already trained on tens of millions of images across many categories. Instead of thousands of images, we simply need to upload a small set of training images (typically a few hundred images or less) that are specific to our use case into our easy-to-use console.

Rekognition Custom Labels automatically loads and inspects the training data, selects the right machine learning algorithms, trains a model, and provides model performance metrics

It is Customized Image Analysis to easily detect objects and scenes you define as most relevant to our domain. Key advantages are

Guided Experience to create labeled images
Train and evaluate with no coding and no ML experience
Easy to use fully managed API

Amazon Rekognition Custom Labels provides a simple end-to-end experience where you start by labeling a dataset, and Amazon Rekognition Custom Labels builds a custom ML model for you by inspecting the data and selecting the right ML algorithm. After your model is trained, you can start using it immediately for image analysis. If you want to process images in batches (such as once a day or week, or at scheduled times during the day), you can provision your custom model at scheduled times.

Sample Serverless solution using Amazon Rekognition

Following architecture being taken from AWS github where it is being explain how we can build cost-optimal batch solution with Amazon Rekognition Custom Labels which provision custom model at scheduled times, process all our images, and then deprovision our resources to avoid incurring extra cost.

This application creates a serverless Amazon Rekognition Custom Label Detection workflow which runs on a pre-defined schedule (note that the schedule is enabled by default at deployment). It demonstrates the power of Step Functions to orchestrate Lambda functions and other AWS resources to form complex and robust workflows, coupled with event-driven development using Amazon EventBridge.

As an image is stored in Amazon S3 bucket, it triggers a message which gets stored in an Amazon SQS queue.
Amazon EventBridge is configured to trigger an AWS Step Function workflow at certain frequency (1 hour by default).
As the workflow runs it checks the number of items in the Amazon SQS queue. If there are no items to process in the queue, workflow ends. If there are items to process in the queue, workflow starts the Amazon Rekognition Custom Labels model and enables Amazon SQS integration with a Lambda function to process those images.
As integration between Amazon SQS queue and Lambda is enabled, Lambda start processing images using Amazon Rekognition Custom Labels.
Once all the images are processed, workflow stops the Amazon Rekognition Custom Labels model and disables integration between Amazon SQS queue and Lambda function

Key Points of Rekognition API

API operations don’t save any of the generated labels. Wecan save these labels by placing them in database, along with identifiers for the respective images.
Rekognition API for Video includes these features:
- Real-time analysis of streaming video;
- Person identification and pathing;
- Face recognition;
- Facial analysis;
- Objects, scenes and activities detection;
- Inappropriate video detection; and,
- Celebrity recognition.
Rekognition API for Images includes these features;
- Object and scene detection;
- Facial recognition;
- Facial analysis;
- Face comparison;
- Unsafe image detection;
- Celebrity recognition; and,
- Text in Image.

Code sample for Rekognition

This example displays the estimated age range and other attributes for detected faces, and lists the JSON for all detected facial attributes. Change the value of photo to the image file name. Change the value of bucket to the Amazon S3 bucket where the image is stored.

#Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
#PDX-License-Identifier: MIT-0 (For details, see https://github.com/awsdocs/amazon-rekognition-developer-guide/blob/master/LICENSE-SAMPLECODE.)

import boto3
import json

def detect_faces(photo, bucket):

    client=boto3.client('rekognition')

    response = client.detect_faces(Image={'S3Object':{'Bucket':bucket,'Name':photo}},Attributes=['ALL'])

    print('Detected faces for ' + photo)    
    for faceDetail in response['FaceDetails']:
        print('The detected face is between ' + str(faceDetail['AgeRange']['Low']) 
              + ' and ' + str(faceDetail['AgeRange']['High']) + ' years old')

        print('Here are the other attributes:')
        print(json.dumps(faceDetail, indent=4, sort_keys=True))

        # Access predictions for individual face details and print them
        print("Gender: " + str(faceDetail['Gender']))
        print("Smile: " + str(faceDetail['Smile']))
        print("Eyeglasses: " + str(faceDetail['Eyeglasses']))
        print("Emotions: " + str(faceDetail['Emotions'][0]))

    return len(response['FaceDetails'])
def main():
    photo='photo'
    bucket='bucket'
    face_count=detect_faces(photo, bucket)
    print("Faces detected: " + str(face_count))


if __name__ == "__main__":
    main()

Reference

AWS Rekognition Developer Guide

AWS messaging and events - Decoupling applications with an event bus

Amit Kayal — Sun, 07 Mar 2021 17:42:02 +0000

AWS messaging and events - Decoupling applications with an event bus

Few days back, I was listening to a session from James and this note is summary of my notes from his session. All credit goes to him for such an outstanding session.

What is Event Bus?

Standard Wikipedia definition says “An Eventbus is a mechanism that allows different components to communicate with each other without knowing about each other. ... Components can also listen to Events on an Eventbus, without knowing who sent the Events” .

Event driven architecture pattern is a distributed asynchronous architecture pattern which helps to create highly scalable reactive applications. The main idea is delivering and processing events asynchronously without depending on producers and consumers.

Advantages are:

Decoupling of producers and consumers
Filters and routes events

What is monolithic micro service Integration

Let's look into the Integration architecture. Good point is that here, we are using microservice approach but bad part is that it has become monolithic architecture approach.

Now, lets look more into the following and understand how fallout management will happen. Let’s say we have some error happened into Fulfilment microservice and now we need to communicate back to payment microservice to rollback/credit the payment and then report the same to Invoice, Forecasting microservice.

This architecture still have all our standard microservice architecture advantages

Loose Coupling
Scalability
Extensibility
Complexity reduction by multiple microservices
High Availability
Fault Tolerance

Integration Patterns with Event Bus

So, the above monolithic architecture, shown above can be more decoupled with following architecture where Order Service (Producer) and other services (consumers) are separated by Event Bus. So, all downstream services read from Event Bus and we have rules to filter accordingly. So, order service need to talk only to Event Bus and team of Order Service does not need to be bothered about other dependent services.

Integration pattern with AWS EventBridge

This is a revolutionary service which helps in serverless world to create decoupled services. This is a serverless event bus service which is managed by AWS. It can integrate with Our own native application through AWS SDK and AWS services.

It is always recommended to use SQS between EventBridge and Lambda to provide durability.

This service is originated from Amazon Cloudwatch events which processes loads of events from several AWS services.

Another interesting point is that, we have SQS, SNS and then EventBridge to Integrate with. So, confusion is which one to use and when to use.

SQS is a queue where producer pushes the event into queue and consumers reads from Queue. Messages in queue are durably persistently.
The above pattern is quite similar with SNS where publishers sent message with a topic and consumers read as per their selected topic.

Main difference between SNS and EventBridge are routing capability of service and type of consumers.

SNS is great for large set fan-out and having thousands of consumers
Whereas, EventBridge is more suitable when we need rich routing capability. It is also possible to combine with high set of consumer services coupled with this rich routing capability ones.
Also, EventBridge can have many(Producers) to many (Consumers) relationship which is quite unique.

EventBridge has retry capability also, if we have a downstream system which is not available for some reason then EventBridge will keep on retrying for 24 hrs. We then can have DLQ attached with EventBridge attached where message can be moved if the message is not delivered after multiple attempt.

Default Event Bus always comes with account and we need to create other type of Event Bus if required. Once we have these Bus define then we define rules and they can become very complex sometime depending on routing requirements. Rule can define, “Event with attribute of type Payment need to go to Payment Service”.

What is EventBridge Rules

In EventBridge, an event is simply a JSON structure. Events generated from AWS services always contain a number of descriptive fields and are identifiable by the source attribute prefix “aws”. A typical event structure for a custom application looks like this:

AWS SDK EventBridge PutEvent API sends custom events to Amazon EventBridge so that they can be matched to rules.

Request Syntax:

{
   "Entries": [ 
      { 
         "Detail": "string",
         "DetailType": "string",
         "EventBusName": "string",
         "Resources": [ "string" ],
         "Source": "string",
         "Time": number,
         "TraceHeader": "string"
      }
   ]
}

Response Structure from EventBridge

If the action is successful, the service sends back an HTTP 200 response.

{
   "Entries": [ 
      { 
         "ErrorCode": "string",
         "ErrorMessage": "string",
         "EventId": "string"
      }
   ],
   "FailedEntryCount": number
}

EventBridge examines the incoming event and compares it against this rule. The rule specifies a source value of custom.myown and, as this exists in the event, the pattern matches. It then routes the event to the rule’s targets:

The example above shows a static, exact match pattern – the attribute is either present, or it’s not.

Here is another architecture pattern which shows how multiple consumers can be triggered based on rules.

Scalable webhook

Following examples shows a scalable architecture, where API Endpoint is connected with EventBriddge. EventBridge is then routing specific events to SQS which is triggering lambda when message arrived in SQS. So even if Lambda is down due to some application code issue then SQS will be able to persistently retain message for a fixed time. So here even though API gateway has 29 sec sync time out, it is always a good practice to have send something from API gateway and get some response (may be a dummy one) and then in backend we do async processing. So here in this architecture, API gateway is getting the response from EventBridge and the behind the scene EventBridge is doing all the async work to get job done.

Responding to AWS Step Functions workflows

Let’s refer to above ECommerce application microservice application architecture and take Payment microservice. Payment service can have very complex logic sometime depending of payment type Cheque, Credit Card etc. Sometime Cheque payment can take several days to process due to Bank clearance etc and there may be requirement of reversing the payment when payment fails. Key point is that no pricing impact on AWS EventBridge as any events to EventBridge from AWS Services are free of charge to process.

Here, lambda function does something depending on state function result is successful or not.

Cloudtrail-integration

CloudTrail is enabled on your AWS account when you create the account. For an ongoing record of events in your AWS account, including events for EventBridge, create a trail. A trail enables CloudTrail to deliver log files to an Amazon S3 bucket.

Here, is the extension of above architecture pattern where cloudtrail will capture all the events and then pass-on to EventBridge. Here the lambda function is being triggered only when our source of event is “aws.s3” and bucketName are “mysales-Jan”, “mysales-Feb” and “mysales-March”.

Key point here is that our bucket names are hardcoded and this means whenever a new buckets are created and if we have to use same lambda then rule needs to be changed. So we can make a dynamic rule as shared below by using bucket prefix.

Multiple buckets with multiple Lambda functions

In the above standard S3 and Lambda integration, a single Lambda function can only be invoked by bucket name prefix in the rule. When you need to invoke multiple functions with the same or overlapping prefixes or suffixes, the EventBridge integration can handle this.

EventBridge allows up to five targets per rule, so you can specify up to five separate Lambda functions to receive the event. All five functions are invoked in parallel when the event pattern matches. To use this, add the targets in the rule – no change to the event pattern is required.

Following example shows, three buckets and three Lambda functions, all subscribing to the same event pattern.

Decoupling large no applications with EventBridge

Following architecture pattern uses events to decouple each service used to process incoming S3 objects. It can also use one or more buckets as event sources, which you can change dynamically as needed. Most importantly, it can be easier to introduce changes and new functionality, since the application is no longer deployed as a mono-repo. Metadata is being sent to Elasticsearch service for Indexing and searching ability.

References

All credit of this article goes to James Beswick.

Do You Have a Look-alike?

Amit Kayal — Sat, 06 Mar 2021 17:39:17 +0000

DoppelGanger tool

This project aims to find a doppelganger or look-alike to a given person within a celebrity dataset.

The repo after unzipping the data.zip should be organized as follow:

DoppelGanger
├── DoppelGanger.ipnyb
├── README.md
└── data
    ├── celeb_mapping.npy
    ├── encoding_celeb_mini.dat
    └── images
        └── celeb_mini
        │   ├── n00000001
        │   │   └── *.JPEG
        │   ├── n00000003
        │   │   └── *.JPEG
        │   ├── ...
        └── test-images
            └── *.jpg

What are Embeddings

To identify a human face, we need to identify the specific facial features. Examples would be the length of our nose, the distance between our eyes, the angle between nose and mouth, …
We humans have perfected recognizing this features and can identify subconsciously millions of them. However, to program that by hand would be impossible. Therefore, we are using cv2.dnn.blobFromImage network to find a good representation of those facial features and its combinations and project them into a lower dimensional space. This lower dimensional space which incorporates the high dimensional space of the facial feature information is called embedding.

How a Face Recognition System works?

Here is a sample implementation architecture which helps to identify face. So, we need a FaceDatabase which is nothing but Face Descriptors.

It is always essential for us to have frontal face detection to ensure that we get face feature descriptors properly.

How dlib face deetction does it work?

Our face has several features that can be identified, like our eyes, mouth, nose, etc. When we use DLib algorithms to detect these features we actually get a map of points that surround each feature. This map composed of 67 points (called landmark points) can identify the following features:

Jaw Points = 0–16
Right Brow Points = 17–21
Left Brow Points = 22–26
Nose Points = 27–35
Right Eye Points = 36–41
Left Eye Points = 42–47
Mouth Points = 48–60
Lips Points = 61–67

Face Compare Application

Enrolment/Encode the known faces

This sections will load all the test images and Persons’ images that are stored in DATA_PATH/images/faces folder will have subfolders. Each subfolder has images of one person. We will store this mapping of images and their corresponding labels to use it later in testing.

High level steps for enrolment are:

data is organized assuming following structure
- faces folder has subfolders.
- each subfolder has images of a person
read subfolders in folder "faces"
We, then define a dictionary named labels which will contain integer labels for corresponding image in imagePaths as shared in sample output below. So key here is the image path and value is the celebrity name indicator or the subfolder name.

dict_keys(['/content/celeb_mini/n00001446/n00001446_00000913.JPEG', '/content/celeb_mini/n00000739/n00000739_00000390.JPEG', '/content/celeb_mini/n00000739/n00000739_00000579.JPEG', '/content/celeb_mini/n00000739/n00000739_00000522.JPEG', '/content/celeb_mini/n00000739/n00000739_00000847.JPEG', '/content/celeb_mini/n00000739/n00000739_00000514.JPEG', '/content/celeb_mini/n00000163/n00000163_00000837.JPEG', '/content/celeb_mini/n00000611/n00000611_00000028.JPEG', '/content/celeb_mini/n00000611/n00000611_00000025.JPEG', '/content/celeb_mini/n00001020/n00001020_00000076.JPEG', '/content/celeb_mini/n00001020/n00001020_00000083.JPEG'])

After this the face descriptor generation process kicks off (we have assumed here each image has one face only which may not be valid sometime)
- Process enrollment images one by and one.
- Convert image from RGB to BGR, because Dlib uses BGR as default format.
- Detect faces in the image. For each face we will compute a face descriptor.
- For each face get facial landmarks.
- Compute face descriptor using facial landmarks. This is a 128 dimensional vector which represents a face. We will convert face descriptor from Dlib’s format to a NumPy array for ease of mathematical computations and read/write on disk.
- For each face descriptor we will also save the corresponding label. Order of face descriptor for each face in faceDescriptors NumPy array’s rows is same as the order of labels in variable index
- So now we have label and its corresponding face descriptors. Now save descriptors and the descriptor-label mapping to disk.
- A face descriptor same value will be as shared below.

  array([-0.12628451,  0.1418225 ,  0.12840877, -0.08114216, -0.15471405,
         -0.0478919 , -0.02732727, -0.13895543,  0.14248449, -0.06592183,
          0.17603964, -0.07321192, -0.2755293 , -0.03996106, -0.02778118,
          0.15708025, -0.18403991, -0.20414677, -0.04625163,  0.00582148,
          0.00072036,  0.04701969,  0.05542792,  0.09160887, -0.10294718,
         -0.32763368, -0.07201297, -0.01759379,  0.01539746, -0.11150647,
         -0.06122024,  0.07024163, -0.26702914, -0.05566515,  0.0628224 ,
          0.25546333, -0.12654914, -0.14244375,  0.22005849,  0.0302763 ,
         -0.17283939, -0.06293783,  0.10661078,  0.27153793,  0.28196251,
         -0.03696643, -0.03208613, -0.07497264,  0.12953417, -0.32792288,
          0.08953883,  0.20995598,  0.07063661,  0.1470409 ,  0.02476202,
         -0.19132754,  0.02425877,  0.13574196, -0.24825127,  0.01421646,
          0.09448326, -0.16684172, -0.01213282, -0.09918778,  0.26380458,
          0.14026739, -0.12445666, -0.16065101,  0.20945829, -0.17040111,
         -0.03908831,  0.0788178 , -0.11134427, -0.11543901, -0.25824887,
          0.01017199,  0.39593434,  0.10053834, -0.12500034,  0.02014588,
         -0.1384005 , -0.10397173, -0.10062244,  0.09627406,  0.00835954,
          0.00835303, -0.12891442,  0.07478838,  0.21302247, -0.03757607,
          0.02387221,  0.2324305 ,  0.01796461,  0.00087321,  0.08747191,
          0.03362498, -0.12506446, -0.02355893, -0.20592356, -0.02991434,
         -0.12417389, -0.15015683, -0.02568636,  0.10652462, -0.16659626,
          0.16554774,  0.04287212, -0.06953543, -0.06864648, -0.13304259,
         -0.11927435, -0.05156082,  0.1704922 , -0.32410818,  0.24830483,
          0.11175595,  0.05624811,  0.18281324,  0.01676171,  0.13893685,
          0.01264865, -0.07391733, -0.0738699 , -0.06425698,  0.01094038,
          0.00706462,  0.05539028,  0.09609816])

Compute and compare the similarities for Test Image

Once we have the encodings of our dataset, the goal is to compute the similarities between the encoding of a given image and the known encodings. The doppelganger or looka-like celebrity will then be the one with the more similarities.

Primarily dlib has been used here for face detection and shape detection. Dlib is a modern C++ toolkit containing machine learning algorithms and tools for creating complex software in C++ to solve real world problems. It is used in both industry and academia in a wide range of domains including robotics, embedded devices, mobile phones, and large high performance computing environments.

Key points are:

dlib.shape_predictor uses “shape_predictor_68_face_landmarks.dat” file
- This model file is designed for use with dlib's HOG face detector. That is, it expects the bounding boxes from the face detector to be aligned a certain way, the way dlib's HOG face detector does it. It won't work as well when used with a face detector that produces differently aligned boxes, such as the CNN based mmod_human_face_detector.dat face detector.
dlib.face_recognition_model_v1 uses “dlib_face_recognition_resnet_model_v1.dat”
- This model is a ResNet network with 29 conv layers. It's essentially a version of the ResNet-34 network from the paper Deep Residual Learning for Image Recognition by He, Zhang, Ren, and Sun with a few layers removed and the number of filters per layer reduced by half.
- The network was trained from scratch on a dataset of about 3 million faces.
The DLib function get_frontal_face_detector() will return a detector that is a function we can use to retrieve the faces information. Each face is an object that contains the points where the image can be found.
The DLib offers a function called shape_predictor() that will help in identifying face features but with a caveat, it needs a pre-trained model to work and that’s why we are using dlib_face_recognition_resnet_model_v1.dat

PREDICTOR_PATH = DATA_PATH + 'shape_predictor_68_face_landmarks.dat'
FACE_RECOGNITION_MODEL_PATH = DATA_PATH + 'dlib_face_recognition_resnet_model_v1.dat'

# landmarkDetector = dlib.shape_predictor(PREDICTOR_PATH)
# Initialize face detector, 
# facial landmarks detector and face recognizer
faceDetector = dlib.get_frontal_face_detector()
shapePredictor = dlib.shape_predictor(PREDICTOR_PATH)
faceRecognizer = dlib.face_recognition_model_v1(FACE_RECOGNITION_MODEL_PATH)

High level steps will be

Initialize Dlib’s Face Detector, Facial Landmark Detector and Face Recognition neural network objects
Read the query image and convert it from BGR to RGB. Because Dlib uses RGB as default format.
Detect faces in query image. For each face detect facial landmarks.
- For each face computer face descriptor using facial landmarks. Convert descriptor to a NumPy array.
- Calculate Euclidean distance between face descriptors in query images versus face descriptors of enrolled images. Find the enrolled face for which distance is minimum.
- Dlib specifies that in general, if two face descriptor vectors have a Euclidean distance between them less than 0.6 then they are from the same person, otherwise they are from different people.
- If minimum distance if less than threshold, find the name of person from index, else the person in query image is unknown.