DEV Community: Amit Kayal

Building a Hybrid AWS Microservices Platform with API Gateway, Lambda, ECS, and Load Balancers

Amit Kayal — Mon, 20 Apr 2026 18:41:39 +0000

Building a Hybrid AWS Microservices Platform with API Gateway, Lambda, ECS, and Load Balancers

Introduction

When teams start splitting a large backend into smaller services, the first infrastructure question is usually not "How do we build a microservice?" but "How do we expose many different services safely, consistently, and without creating a networking mess?"

Our architecture provides a practical answer to that problem using a hybrid AWS design:

API Gateway as the front door
Lambda for lightweight serverless capabilities and supporting workflows
ECS Fargate for containerized business services
Internal load balancers for private service routing
Terraform for repeatable, staged infrastructure delivery

The important architectural idea is separation of concerns. Public access, authentication, routing, container execution, and service discovery are all handled by different layers. That keeps the platform easier to scale and much easier to evolve as the number of services grows.

The Core Pattern

At a high level, the platform follows this flow:

A client sends an HTTPS request to API Gateway.
API Gateway applies request-level controls such as API key enforcement, CORS behavior, and route matching.
The request is sent either to a Lambda-backed endpoint or to a private containerized service.
For ECS services, traffic goes through a VPC Link into internal load balancing.
The load balancer forwards the request to the correct ECS service based on path rules.
ECS Fargate runs one or more healthy tasks for that service and returns the response.

This gives a single API surface to consumers while allowing the backend implementation to vary by use case.

Why Combine Lambda and ECS?

A platform like this benefits from using both compute models rather than forcing every workload into one.

Lambda is a strong fit for:

lightweight request handlers
event-driven tasks
simple orchestration
platform support functions
endpoints that do not need a full container lifecycle

ECS Fargate is a better fit for:

long-lived HTTP microservices
containerized frameworks and dependencies
services that need more predictable runtime behavior
APIs that benefit from load balancing, health checks, and horizontal scaling

In our architecture, the design supports both. Some APIs are routed to Lambda-based services, while others are routed to ECS services defined through service configuration. That hybrid model is useful in real organizations because all services do not have the same runtime needs.

A Three-Stage Infrastructure Model

One of the strongest ideas in our architecture is the staged Terraform layout. Instead of deploying everything together, the infrastructure is split into three layers.

Stage 1: Networking

The first stage establishes the network foundation:

VPC selection or creation
public and private subnet discovery or provisioning
internal Network Load Balancer
internal Application Load Balancer
VPC Link for API Gateway
ECS task security group
ALB log storage and network observability components

This stage is intentionally infrastructure-only. No application services are deployed here.

Stage 2: Compute

The second stage provisions the actual execution environment:

ECS cluster on Fargate
ECR repositories for service images
target groups per service
ALB listener and listener rules
ECS service definitions
CloudWatch log groups
Lambda functions used by the platform

This stage consumes outputs from the networking stage so the compute layer never hardcodes network assumptions in its own design.

Stage 3: API Gateways

The third stage exposes services through API Gateway:

a public API for internet-facing consumption
a private API for VPC-only access
route creation from service metadata
VPC Link integrations for containerized services
Lambda proxy integrations for Lambda-backed services
API keys, usage plans, and stage configuration

This split is operationally important. Teams can change routing without rebuilding networking, and they can add services without redesigning the entire platform.

The Request Path for ECS Services

For containerized microservices, the implementation follows a private ingress model.

The path is:

Client -> API Gateway -> VPC Link -> internal NLB -> internal ALB -> ECS service -> ECS task

That may look like one hop too many at first, but each layer has a purpose.

API Gateway

API Gateway is the public control plane. It handles:

TLS termination at the edge
route exposure
API key enforcement
request and header mapping
CORS handling
stage-based deployment

It gives consumers a stable API contract while keeping the backend private.

Why a VPC Link Is Used

ECS services are not exposed directly to the internet. Instead, API Gateway connects privately into the VPC using a VPC Link. That allows the public API layer to reach internal services without making the services themselves public.

This is a strong security pattern because the application runtime stays inside the VPC, but consumers still get a clean managed API endpoint.

Why the Repository Uses Both NLB and ALB

A useful implementation detail in our architecture is that the VPC Link targets an internal Network Load Balancer, and that NLB forwards to an internal Application Load Balancer.

This arrangement provides two separate benefits:

The NLB is used as the stable target for the API Gateway VPC Link.
The ALB performs path-based routing to the actual microservices.

The ALB is what makes many ECS services practical behind one internal entry point. Each service gets its own listener rule and target group, so the platform can route based on URL path rather than provisioning a separate load balancer per service.

How Load Balancing Works

The load-balancing model is service-oriented.

Each ECS microservice contributes:

a base API path
an ALB path pattern
a listener rule priority
a container port
a health check definition

From that metadata, Terraform creates:

one target group per service
one listener rule per service
one ECS service per service

This means the routing layer is not manually duplicated for every new microservice. The service declares its path and runtime settings, and the platform generates the infrastructure around it.

Target Groups

Each target group points to ECS tasks using IP targets. That is the correct choice for Fargate because tasks run with their own elastic networking interfaces rather than on shared EC2 hosts.

The target groups in this repository also use application-level health checks. A task is considered healthy only when its service endpoint responds successfully on the configured health path.

That matters because container startup is not the same as application readiness. A service may be running from ECS's perspective but still not ready to receive traffic.

Listener Rules

The ALB listener is configured once, and each service gets a path-based rule. For example, a service under a quoting path can be matched independently from a service under a product-pricing path.

This keeps the routing layer centralized and avoids deploying a dedicated ALB per service, which would become expensive and operationally noisy as the platform grows.

Health Checks and Traffic Protection

The repository uses health checks in multiple places:

API health endpoints at the application level
ALB target group health checks
ECS service health grace periods
container health checks inside the task definition

That layered approach improves resilience:

unhealthy tasks are removed from target groups
ECS replaces failed tasks
API Gateway continues to route through the same private entry point

The result is a platform that can recover from instance-level failures without changing the public API contract.

How ECS Is Structured

The ECS side of the platform is built for repeatability rather than one-off service definitions.

ECS Cluster

The platform provisions a shared ECS cluster per environment. That allows multiple microservices to run within the same operational boundary while still being isolated at the task and service level.

The cluster uses Fargate, which removes the need to manage EC2 worker nodes. This simplifies operations significantly:

no patching of container hosts
no cluster capacity management at the instance level
easier scaling by task count

Reusable ECS Service Module

Instead of defining each ECS service from scratch, the repository uses a reusable Terraform module for service deployment.

That module is responsible for:

task definition creation
container logging configuration
IAM role wiring
ECS service creation
target group attachment
subnet and security group placement
optional capacity provider strategy

This is a strong platform choice. It makes service onboarding consistent and reduces drift between services.

Task Definitions

Each service runs as a Fargate task with:

a named container image from ECR
CPU and memory settings
environment variables
a health check command
CloudWatch logging

The repository also includes support for an additional X-Ray sidecar container in the task definition pattern, which is useful for distributed tracing in a microservice environment.

Network Mode

Tasks run with awsvpc networking, which gives each task its own network interface and private IP. This is the standard model for ECS on Fargate and is what allows ALB target groups to use IP mode cleanly.

Subnet and Security Group Design

This repository supports both existing/default VPC usage and a more segmented custom VPC model.

That flexibility matters because many teams start in a default-VPC or dev-friendly setup and later move to stricter network isolation for staging and production.

Subnet Placement

The network layer discovers public and private subnets where available. In a custom VPC, the design supports proper private subnet deployment. In a simpler default VPC setup, the platform can fall back to available public subnets when private ones are not present.

This is an important operational nuance:

development environments often optimize for simplicity
higher environments usually optimize for stricter isolation

The repository is built to handle both.

Security Groups

The security model follows least-privilege intent:

ECS tasks accept application traffic from the internal load-balancing layer
services are not directly internet-facing
API Gateway reaches backend services through private network integration

This keeps the application tier out of direct public exposure while still allowing a public API facade.

Config-Driven Service Onboarding

One of the most scalable ideas in our architecture is that services are registered through configuration rather than by handcrafting infrastructure every time.

There is a master service registry that lists enabled services per environment, and each service provides its own deployment metadata, including:

service identity
container port
desired task count
CPU and memory
API base path
ALB path pattern
listener priority
health check behavior
logging retention
autoscaling preferences

This creates a platform model rather than a collection of unrelated microservices.

Adding a new service becomes a repeatable process:

Create the service.
Define its configuration.
Register it in the service catalog.
Build and publish the image.
Apply Terraform stages.

That is much easier to maintain than cloning infrastructure blocks over and over.

Container Delivery with ECR

For ECS workloads, the container supply chain is straightforward:

Build the service image.
Push it to an ECR repository.
Reference the tagged image in the ECS task definition.
Update the ECS service to roll out the new task definition.

Our platform provisions one ECR repository per service, with image scanning enabled. That is a good baseline for a microservices platform because it keeps artifacts separated by service while still following a common naming convention.

There is also an explicit deployment phase between infrastructure provisioning and API exposure where container images are built and pushed. That is a practical real-world step many diagrams omit, but it is essential because ECS cannot run a service until the image exists in the registry.

How Lambda Fits into the Platform

Lambda is used here as a first-class platform option, not as an afterthought.

There are two useful Lambda patterns in our architecture.

1. Lambda as an API Backend

Some services can be exposed through API Gateway using Lambda proxy integration. This is ideal for capabilities that are naturally event-driven, lightweight, or operationally simpler as functions than as always-on containers.

In this model:

API Gateway owns the route
Lambda executes the business logic
API Gateway returns the Lambda response directly

This avoids unnecessary load-balancer and container overhead for smaller workloads.

2. Lambda as a Platform Support Function

Our architecture also provisions Lambda functions that support the overall platform, such as authentication-related or onboarding-related workflows.

This is a smart use of Lambda in a hybrid platform because not every supporting concern needs to run inside ECS.

Authentication and API Protection

Our architecture clearly treats API protection as an API Gateway concern.

The current public API implementation enforces API key usage through API Gateway methods, API keys, and usage plans. The codebase also provisions a supporting API key validation Lambda function and related permissions, which shows the platform is designed to accommodate Lambda-based validation flows where needed.

From a blog perspective, the important architectural takeaway is this:

keep authentication and traffic governance at the gateway layer
keep service containers focused on business logic
keep private workloads private

That separation keeps the platform easier to secure and easier to reason about.

Public and Private API Models

Another strength of our architecture is that it supports both public and private APIs.

Public API

The public API is intended for internet-facing access. It handles:

external client access
API keys and usage plans
CORS behavior
Lambda and ECS route exposure

Private API

The private API is intended for internal or VPC-scoped access. It is useful when services should only be reachable from trusted network boundaries such as internal AWS workloads, integration environments, or enterprise connectivity paths.

This split is helpful when some capabilities should be public and others should remain internal even though they share the same service platform underneath.

Observability and Operations

A microservices platform is only as good as its operational visibility.

Our architecture includes observability at several levels:

CloudWatch log groups for ECS services
CloudWatch logs for Lambda functions
API Gateway stage logging
ALB logging support
VPC flow logging
X-Ray-friendly task patterns

That combination helps answer the most common production questions:

Did the request reach the gateway?
Was it routed to the right backend?
Was the target healthy?
Did the service fail or time out?
Was the problem in networking, routing, or application logic?

Without that layered visibility, hybrid platforms become difficult to troubleshoot.

Scaling Characteristics

This architecture scales well because each layer can evolve somewhat independently.

API Layer Scaling

API Gateway absorbs public traffic without requiring the backend to manage edge-facing concerns directly.

ECS Scaling

ECS services scale by task count. Each service can define:

desired count
minimum and maximum capacity
CPU and memory sizing
autoscaling thresholds

That means heavily used services can scale out without affecting lighter services.

Platform Growth

As more services are added, the platform does not need a new ingress pattern each time. The same path-based routing model continues to work as long as route definitions and listener priorities stay clean.

Alignment with AWS Well-Architected Best Practices

This architecture also aligns well with AWS best-practice design principles, especially the AWS Well-Architected mindset.

Operational Excellence

We have structured the platform so that it is operated as a system rather than as a collection of one-off deployments.

This is reflected in:

staged Terraform deployments for clearer ownership and safer changes
configuration-driven service onboarding
consistent ECS service patterns through reusable modules
standardized logging and deployment workflows

This reduces manual drift and makes operational changes more repeatable.

Security

Security is addressed through layered controls rather than a single protection point.

We have adhered to good AWS security practices by:

placing ECS services behind private networking rather than exposing them directly
using API Gateway as the controlled ingress layer
applying API-level protection at the gateway
using security groups to limit east-west traffic
supporting encrypted log and storage patterns
separating public access from internal service routing

This follows the AWS principle of strong boundaries, least privilege, and defense in depth.

Reliability

Reliability comes from designing for failure at the service and routing layers.

We have incorporated that through:

multi-AZ subnet placement
load balancer health checks
ECS task replacement behavior
target group isolation per service
decoupled gateway and backend layers
staged infrastructure dependencies with clear outputs between layers

This means a failing task or unhealthy target does not require the API surface itself to change.

Performance Efficiency

The architecture chooses the right compute model for the right workload.

That is an AWS best practice because it avoids treating all traffic the same.

Examples include:

Lambda for lighter, event-oriented, or supporting workflows
ECS Fargate for containerized services that need steady HTTP handling
ALB path-based routing for efficient multi-service consolidation
service-specific CPU, memory, and scaling settings

This lets us tune services independently instead of overprovisioning everything at the platform level.

Cost Optimization

Cost optimization is also visible in the design choices.

We are not multiplying infrastructure unnecessarily. Instead, the architecture encourages shared but controlled platform components:

one API layer for many services
one internal routing layer for many ECS workloads
shared ECS cluster patterns per environment
service-level scaling instead of blanket scaling
support for Fargate and optional capacity-provider strategies where appropriate

That is much closer to AWS best practice than provisioning separate ingress and compute stacks for every small service.

Sustainability and Maintainability

Even when sustainability is not called out directly, maintainable designs usually consume fewer engineering and infrastructure resources over time.

The architecture helps here by:

reducing duplicated infrastructure definitions
making service onboarding metadata-driven
encouraging reuse of shared platform components
keeping the public contract stable while backend services evolve

That leads to lower long-term complexity, which is a practical form of architectural efficiency.

Why This Pattern Works Well

This AWS pattern is effective because it balances standardization with flexibility.

It standardizes:

deployment stages
ingress architecture
service registration
load-balancer behavior
logging and health checks
ECS service creation

It stays flexible by allowing:

Lambda-backed endpoints
ECS-backed endpoints
public and private APIs
different service-level scaling and runtime settings
multiple environments with different networking strategies

That is exactly what a growing microservices platform needs.

Practical Implementation Advice

If you want to implement a similar architecture, a good sequence is:

Build the networking foundation first.
Keep all service backends private.
Put API Gateway in front of everything external.
Use ECS Fargate for containerized APIs that benefit from long-lived service behavior.
Use Lambda for support functions and lightweight endpoints.
Register services through metadata, not repetitive infrastructure definitions.
Use path-based ALB routing so many services can share one internal ingress layer.
Add strong health checks and centralized logs before traffic grows.

The key is not just choosing AWS services, but assigning each AWS service a clear responsibility.

Conclusion

Our architecture demonstrates a mature way to implement Lambda and ECS-based microservices through API Gateway without exposing backend services directly.

The architecture uses:

staged Terraform for separation of concerns
API Gateway as the public and private API facade
Lambda where serverless execution makes sense
ECS Fargate for containerized microservices
NLB and ALB together for private, path-aware routing
config-driven onboarding for scale

For teams building an enterprise microservices platform, this is a strong pattern because it supports security, operational clarity, and service growth without forcing every workload into the same runtime model.

Most importantly, it turns infrastructure into a reusable platform. Once that platform is in place, adding the next service becomes much easier than adding the first one.

Lessons Learned

Keeping API Gateway as the front door and backend services private makes the architecture easier to secure and easier to evolve.
Using both Lambda and ECS is more practical than forcing every use case into a single compute model.
Path-based routing through shared internal load balancing scales better than creating isolated ingress infrastructure for every service.
Service onboarding becomes significantly easier when routing, health checks, scaling, and runtime settings are driven by configuration.
Health checks, logging, and observability need to be designed from the beginning; adding them later is much harder in a distributed system.
A staged infrastructure model reduces operational risk because networking, compute, and API exposure can be changed independently.
Standardizing platform patterns early saves substantial effort as the number of microservices grows.

Building a Practical Lambda Capacity Provider Platform: Lessons Learned from Warm Pools, Version Hygiene, and CI/CD Reality

Amit Kayal — Mon, 20 Apr 2026 18:25:06 +0000

Building a Practical Lambda Capacity Provider Platform: Lessons Learned from Warm Pools, Version Hygiene, and CI/CD Reality

There is a big difference between a slide-deck architecture and an operating system you can trust on a Monday morning.

This implementation captures that difference well. On paper, the idea is simple: create a shared AWS Lambda Managed Instances capacity provider, run latency-sensitive workloads on ARM64, keep the pool warm with EventBridge, prune old Lambda versions before they become operational debt, and wrap the whole thing in a GitHub Actions plus CodeBuild delivery model. In practice, each of those choices changes how you think about performance, cost, blast radius, and developer discipline.

What follows is not a generic cloud post. It is the kind of write-up you produce after actually building and living with the system.

The Real Problem We Were Solving

Traditional Lambda is excellent when you want abstraction and convenience. It becomes less elegant when your workload is sensitive to startup time, carries heavier dependencies, or needs more predictable execution behavior under bursty load.

That is where a Lambda capacity provider changes the discussion.

In this implementation, the platform is built around a shared aws_lambda_capacity_provider that uses ARM64 Graviton instances and auto scaling. The core idea is straightforward: instead of leaving execution placement entirely to the default Lambda fleet, we deliberately provide a managed compute pool that multiple functions can share. That gives us more control over cost-performance characteristics and lets us design around cold-start pain rather than merely complain about it.

The choice is visible in the Terraform:

The provider runs on arm64
Allowed instance types are constrained to m6g.large, m6g.xlarge, m7g.large, and m7g.xlarge
Scaling is set to Auto
The maximum pool ceiling is set to 64 vCPU
The capacity provider is placed in the default VPC, with unsupported Availability Zones filtered out

That last point matters more than it first appears. The code explicitly excludes unsupported AZs such as us-east-1e, which is a good example of operational maturity: the happy path is not enough when the service itself has placement constraints.

How We Actually Created the Capacity Provider

One thing I wanted this platform to avoid was "concept architecture" with no implementation backbone. So the capacity provider here is not described abstractly. It is provisioned directly in Terraform and wired into the Lambda lifecycle in a fairly intentional way.

The build starts in terraform_file/agent_core_sync_cp.tf.

First, the capacity provider itself is created with aws_lambda_capacity_provider. The naming pattern ties it to the service and environment, which is the right instinct for multi-environment operation. The provider is tagged as shared compute for agent workloads, which matters later for discoverability and platform governance.

Second, the provider is placed inside the default VPC, but not blindly. In terraform_file/data.tf, the code:

discovers the default VPC
fetches the default subnets
inspects subnet Availability Zones one by one
excludes unsupported zones such as us-east-1e
optionally caps how many subnets are used

This is a subtle but important design choice. Lambda Managed Instances often create one placement footprint per subnet or AZ. If you do not control subnet spread, you can end up creating more infrastructure surface area than you intended.

Third, the provider uses a dedicated security group rather than inheriting something vague and accidental. The current implementation keeps outbound traffic fully open and allows inbound HTTPS. That is permissive, but it is at least explicit and repeatable. Early-stage platforms benefit from that kind of clarity.

Fourth, the capacity provider gets its own operator role through AWSLambdaManagedEC2ResourceOperator. That is a critical detail. Capacity providers are not just Lambda resources; they need AWS to manage the EC2-backed execution infrastructure on your behalf. If you miss that role, the platform does not really exist no matter how nice your Terraform looks.

Fifth, the instance requirements are opinionated. The code forces arm64 and narrows the fleet to supported Graviton M-family instance types. That is one of the better engineering decisions in this implementation because it converts an architectural preference into an enforceable runtime rule.

Finally, the Lambda function is attached to the capacity provider in terraform_file/lambda_clm_router_agent.tf through capacity_provider_config. That is where the abstraction becomes real. We are not just provisioning a pool and hoping someone uses it later. We are explicitly binding a published Lambda to that pool and tuning:

memory GiB per vCPU
max concurrency per execution environment
ARM64 runtime alignment
published versioning through Lambda aliases

That is the full loop: provision shared compute, constrain placement, grant AWS the operator role it needs, attach live functions to the pool, and then manage the resulting version sprawl with automation. That is what makes this feel like a platform artifact rather than a loose Terraform experiment.

Lesson 1: A Capacity Provider Is Not a Tuning Knob. It Is an Operating Model.

Teams often talk about capacity providers as if they are just a performance optimization. That framing is too shallow.

The moment you move Lambda onto managed instances, you are no longer only buying faster startup. You are adopting a new operating model with very clear implications:

You now care about instance family compatibility
You need to think about subnet strategy and AZ support
You have to reason about pool scaling ceilings, concurrency, and memory per vCPU
You are effectively blending serverless ergonomics with infrastructure accountability

This implementation shows that transition clearly. The CLM router Lambda is not just declared with a runtime and handler. It is attached to the shared capacity provider and explicitly tuned with:

execution_environment_memory_gib_per_vcpu
per_execution_environment_max_concurrency
publish = true
architectures = ["arm64"]

That is the tell. Once we start specifying how execution environments should behave, we are no longer simply "deploying a Lambda." We are shaping compute economics.

The practical lesson here is simple: if you adopt Lambda Managed Instances, treat it like platform engineering, not like a runtime checkbox.

Lesson 2: ARM64 Delivers Real Value, but Only if You Respect Service Constraints

One of the strongest decisions in this implementation is the bias toward Graviton. For Python-heavy agent workloads, ARM64 is usually the right default. The economics are better, and the performance-per-dollar story is often compelling.

But there is an important nuance that the Terraform comments correctly capture: not every EC2 family you might expect is supported in the way you assume. This implementation explicitly avoids unsupported combinations and narrows the fleet to supported M-family Graviton instances.

That is a good lesson in cloud architecture generally: cloud products market flexibility, but production systems survive on constraint management.

The teams that do well with modern AWS services are not the ones that assume every SKU works. They are the ones that encode the service's real boundaries in Terraform so no one has to rediscover them during an incident window.

Lesson 3: Warmup Is Not a Hack. It Is a Deliberate Control Loop.

There is a tendency in engineering circles to treat "warming" as a slightly embarrassing workaround. I think that is the wrong mindset.

This implementation schedules the CLM router Lambda every five minutes through EventBridge. The handler itself is intentionally lightweight and effectively acts as a keep-alive mechanism. That is not laziness. It is an explicit decision to keep the shared pool alive for latency-sensitive traffic.

More specifically, the warmer exists to reduce the probability that the capacity provider has to spin up fresh managed instance capacity for a new invocation path after a quiet period. That is the practical point of the EventBridge rule in terraform_file/eventbridge_cp_arm.tf. By invoking the Lambda on a steady rate(5 minutes) schedule, the platform keeps the execution path warm enough that the shared capacity provider is less likely to fall all the way back to a cold, scale-from-zero posture right before a real request arrives.

The important insight is this: once you care about cold-start predictability, you need a control loop.

That control loop can be:

Provisioned concurrency
Scheduled warmers
Request shaping
A shared managed instance pool

In this design, the team chose scheduled warm invocation plus a shared capacity provider. That is a sensible middle ground. It is cheaper and simpler than overcommitting always-on infrastructure, while still materially reducing the first-hit penalty.

In plain English: the EventBridge warmer is being used here so the capacity provider does not need to spin up a brand-new server footprint every time traffic reappears after idle time. For interactive or latency-sensitive agent workloads, that is a very practical optimization.

The strategic lesson is that warmup should be measured against business latency, not ideological purity. If a five-minute EventBridge schedule protects user experience and keeps cost acceptable, it is doing its job.

Lesson 4: Shared Pools Create Efficiency, but They Also Create Coupling

The capacity provider here is intentionally shared across platform agents and automation services. That is the right move early in a platform journey because it improves utilization and prevents every Lambda from inventing its own isolated infrastructure story.

But shared pools always introduce two forms of coupling:

Technical coupling, because multiple workloads compete for the same execution substrate
Organizational coupling, because one team's deployment patterns can affect another team's cost and performance envelope

That is why the concurrency controls here matter. The CLM router function uses a per-execution-environment concurrency setting, and the environment-specific .tfvars files pin that concurrency to 4. That is more than a performance number. It is a fairness policy.

If I were advising a platform team scaling this pattern, I would say this clearly: shared capacity providers are excellent, but they need quota thinking from day one. Otherwise the first successful workload becomes the first noisy neighbor.

Lesson 5: If You Publish Versions Aggressively, You Need Lifecycle Hygiene on Day One

This implementation makes another good call: the Lambda functions are published, aliased, and then cleaned up with an automated version pruner.

That matters because version sprawl is one of those quiet operational problems that teams ignore until it becomes annoying enough to disrupt deployments. Published versions accumulate quickly when CI/CD is active. If you do not manage them, you eventually pay in clutter, confusion, or hard service limits.

The lambda_version_pruner implementation is stronger than a simplistic cleanup script because it preserves what actually matters:

It scans all Lambda functions
It filters only functions associated with the target capacity provider
It lists all aliases and protects aliased versions
It keeps the latest N published versions
It deletes everything older that is neither current nor aliased

This is exactly the kind of automation mature teams invest in. Not glamorous. Very valuable.

There is also an understated platform principle here: rollback is not just about keeping artifacts. It is about keeping the right artifacts. By preserving aliased versions, the pruner respects deployment intent rather than blindly optimizing for tidiness.

There is also a more practical capacity-provider reason for doing this, and it deserves to be stated directly.

When you run a shared Lambda Managed Instances pool, you want the platform to spend its effort on the versions that are actually serving traffic, warming correctly, or remaining available for safe rollback. If old published versions keep accumulating forever, three unhealthy things tend to happen:

operators lose clarity on which versions are still meaningful
rollback and alias management become noisier than they should be
the shared platform carries more deployment residue than useful runtime intent

Strictly speaking, deleting old Lambda versions does not magically increase CPU on the capacity provider. What it does do is improve platform hygiene around the shared pool. It ensures that the versions attached to aliases, warmup patterns, and deployment workflows remain deliberate and limited. In other words, it improves capacity-provider utilization indirectly by reducing version sprawl around the workloads that consume that shared capacity.

That matters in real operations. The healthier the deployment surface is, the easier it is to reason about what is warming, what is active, what can be rolled back, and what should no longer influence the platform at all.

So the version pruner is not just a cleanup utility. It is part of making the shared capacity provider operationally efficient. Not by adding raw compute, but by reducing noise, protecting the versions that matter, and keeping the platform focused on live execution paths instead of historical leftovers.

Lesson 6: GitHub Actions Should Orchestrate. CodeBuild Should Execute.

Architecturally, the CI/CD model here is sensible.

GitHub Actions is used as the control plane:

branch-based triggering
security scanning
environment selection
AWS credential injection
build orchestration

AWS CodeBuild is used as the execution plane:

Terraform install
terraform init
terraform validate
terraform plan
terraform apply

I like this split. It keeps GitHub Actions lightweight and makes AWS the place where the actual infrastructure mutation happens. That usually gives better access control, cleaner auditability, and fewer surprises around long-running plan or apply steps.

The buildspecs pin Terraform 1.12.2, install the CLI explicitly, and then execute plan/apply flows with environment-specific variable files. That is exactly the kind of boring repeatability you want in infrastructure delivery.

This is one of the most practical lessons from the implementation: do not force GitHub Actions to be your full deployment runtime if AWS-native execution gives you better control.

Lesson 7: CI/CD Maturity Is Not About Having a Pipeline. It Is About Where the Gates Actually Are.

The implementation also reveals a harder truth: CI/CD design is won or lost not by YAML volume, but by trigger discipline.

There are some good instincts here:

Dev deployment is chained off a successful security workflow
Security scanning runs on push and PR for dev
PR security review is scoped only to actual code and infrastructure changes
Environment-specific secrets are used for AWS access

That said, the current implementation also shows the kinds of issues every fast-moving team encounters:

The dev deploy workflow is triggered by Security Checks (Push), not by a broader quality gate such as tests plus security plus static analysis
The QA workflow is currently triggered on pull_request to qa, yet it also includes an apply stage, which is a risky combination
The sanity workflow references a different CodeBuild project naming pattern, which looks like copy-forward drift from another implementation
One dev apply step mixes generic and environment-specific secrets in a way that deserves tightening

This is not a criticism of the team. It is actually the most authentic part of the system.

Real pipelines evolve through reuse, renaming, urgency, and partial migration. The useful engineering habit is not pretending they are pristine. It is recognizing that pipeline drift is itself a production concern.

My blunt lesson here is this: CI/CD is software. It needs the same review rigor as application code.

Lesson 8: Documentation Drift Is a Reliability Signal

The README here is ambitious and useful, but parts of it clearly describe a broader or earlier architecture than the exact files currently present. That mismatch is more important than most teams realize.

When documentation and implementation diverge, three things happen:

new engineers learn the wrong system
reviewers approve changes with outdated mental models
incidents take longer to resolve because operators trust stale diagrams

One of the best engineering habits is to treat documentation drift as an operational bug, not as a cosmetic issue.

This implementation makes that case well. The code is the source of truth. The docs are directionally strong, but some names, workflow descriptions, and file references have clearly moved over time. That is normal. What matters is catching it before the next engineer builds decisions on old assumptions.

Lesson 9: The Default VPC Is Fine for Speed, but It Should Be a Conscious Temporary Convenience

The Terraform intentionally uses the default VPC and default subnets, then layers in filtering and a custom security group. For early velocity, that is an acceptable choice. It removes friction and makes the first deployment much easier.

But teams should be honest about the tradeoff.

Using the default VPC accelerates setup. It does not provide the same clarity, segmentation, or policy hygiene that a dedicated workload VPC eventually should. The inbound HTTPS rule from 0.0.0.0/0 is another example of where a practical early-stage decision should later be revisited with a more opinionated security posture.

My view is simple: default VPC usage is fine when it is a speed decision. It becomes dangerous when it silently hardens into architecture.

Lesson 10: Least Privilege Usually Loses the First Battle. Do Not Let It Lose the War.

The Lambda IAM policy for the router function is broad. Very broad.

That is common when a platform team is trying to unblock integration work quickly across S3, SQS, SNS, DynamoDB, Bedrock, AppSync, logs, X-Ray, and secrets. The version pruner is noticeably tighter, which is encouraging. But the broader pattern remains familiar: the first version of a system usually over-grants.

The lesson is not "never do that." The lesson is "know when you are doing it, and schedule the hardening work while the platform is still comprehensible."

Security debt compounds. The longer a wide-open policy survives, the more invisible it becomes.

What This Repo Gets Right

If I strip away the drift and focus on the platform instincts, this implementation gets a lot right:

It treats capacity provider infrastructure as shared platform capability, not one-off function plumbing
It optimizes for ARM64 economics instead of defaulting to x86 out of habit
It acknowledges cold starts as a business problem and addresses them operationally
It preserves rollback safety with aliases while still pruning version sprawl
It separates orchestration from execution in CI/CD
It encodes AWS service constraints in Terraform comments and defaults, which reduces tribal knowledge

That is a strong foundation.

What I Would Improve Next

If I were turning this into the next version of a production-grade internal platform, I would prioritize the following:

Tighten naming consistency across the implementation.
The capacity provider name appears in slightly different forms across resources. That is how automation misses its target. Shared naming locals should eliminate this class of error.
Make QA and production promotion rules stricter.
A PR-triggered apply path should be removed. Plan on PR, apply on protected branch or approved environment gate is the cleaner model.
Run Terraform from a single explicit working directory.
The current layout places Terraform under terraform_file/, while some buildspec commands read like root-level execution. That ambiguity should be eliminated.
Move from broad IAM toward intent-based policies.
Especially for the router Lambda, policy scope should narrow as the workload stabilizes.
Revisit networking posture.
The default VPC is fine for speed; a dedicated VPC model is better for longevity, auditability, and controlled ingress.
Add stronger deployment quality gates.
Security review is useful, but infrastructure promotion should also hang off validation, tests, linting, and explicit approval where appropriate.
Add platform observability as code.
CloudWatch alarms, dashboarding, and cost visibility for the capacity provider should be treated as first-class Terraform resources, not follow-up tasks.

The Bigger Technical Lesson

The biggest takeaway from this implementation is not about Lambda specifically.

It is about how modern platform teams should build.

We should absolutely chase better cost-performance curves. We should use managed primitives aggressively. We should automate the boring work. But we also need the discipline to encode what we learn while the system is still small enough to reason about.

What makes this useful is that it shows both halves of real engineering:

the architectural intent
the implementation scars

That combination is where credible engineering judgment comes from.

Anyone can present a clean target state. The harder and more useful skill is building systems that survive contact with deployment friction, service constraints, naming drift, and operational reality.

That is what this implementation is doing. And that is why the lessons here matter.

Closing Thought

Capacity providers, warmers, version pruning, and GitHub-driven delivery are not separate topics. They are all answers to the same technical question:

How do we make cloud systems faster, cheaper, safer, and more repeatable without turning every application team into a specialized infrastructure group?

In this implementation, the answer was to centralize the hard platform decisions, automate the hygiene, keep the runtime warm where it matters, and stay honest about the places where the system still needs tightening.

That is not just good infrastructure work.

That is good engineering practice.

Lessons I learned building a memory-aware agent with Amazon Bedrock AgentCore Runtime

Amit Kayal — Mon, 20 Apr 2026 18:10:16 +0000

Lessons I learned building a memory-aware agent with Amazon Bedrock AgentCore Runtime

When I started building an agent with Amazon Bedrock AgentCore Runtime, I thought the difficult parts would be model selection, tool wiring, and deployment. Those certainly mattered, but the part that shaped the quality of the agent most was memory.

The first version of the agent could answer single prompts well enough, but it did not behave like a real multi-turn system. Follow-up questions were brittle. The agent lost short-range intent. Tool usage worked, but only within the narrow boundaries of the current prompt. As soon as the conversation depended on what happened one or two turns earlier, the system started to feel less like an agent and more like a stateless inference endpoint.

That experience changed how I approached the design. I stopped thinking about memory as a convenience feature and started treating it as part of the runtime architecture itself. This article is a distillation of the most important lessons I learned while building a short-term-memory-aware agent with Amazon Bedrock AgentCore Runtime and Strands.

Lesson 1: An agent is not really multi-turn until memory is part of the lifecycle

One of the first things I learned is that conversational continuity does not emerge automatically just because the application calls the same runtime repeatedly.

Without short-term memory, the agent only sees the current prompt unless the application keeps reconstructing and replaying history manually. That creates several problems:

previous instructions are easy to lose,
tool chains become fragile across turns,
users have to restate identifiers and intent,
the system becomes increasingly prompt-shaped rather than interaction-shaped.

What became clear to me is that short-term memory is not about storing everything forever. It is about preserving enough recent state for the current conversation to remain coherent.

That distinction matters. I was not trying to build a knowledge base or semantic fact store. I was trying to answer a simpler question: how do I help the agent remember what we were just doing?

Once I framed the problem that way, the architecture became much clearer.

Lesson 2: The cleanest pattern is explicit memory, not implicit transcript magic

Another lesson I learned quickly is that I did not want memory to be hidden behind vague runtime behavior. I wanted the agent code to make memory use explicit:

where memory comes from,
when it is read,
when it is written,
which user it belongs to,
which conversation it belongs to.

That led me to a pattern built around MemoryClient and hooks.

Instead of treating memory like a passive transcript that somehow appears at the edge of the request, I found it much more reliable to think about it as a lifecycle-managed dependency:

create a short-term memory resource,
pass the memory identity into the runtime,
read recent turns when the agent initializes,
write new messages as events when the conversation changes.

The important shift for me was this: memory worked best when it was part of the agent object model, not just part of request handling glue code.

Lesson 3: Hooks are where memory belongs

This was probably the biggest implementation insight.

Once I had a Strands-based agent running inside AgentCore Runtime, I needed to decide where the memory logic should live. I could have put everything directly into the entrypoint and manually stitched together request parsing, history retrieval, message persistence, and prompt injection. That would have worked, but it would have made the agent lifecycle harder to reason about.

What worked better was using hooks tied to the agent lifecycle itself:

AgentInitializedEvent
MessageAddedEvent

That structure gave me a much cleaner mental model.

On initialization, the agent needs context before it reasons. That is the right moment to retrieve the most recent turns from memory and inject them into prompt context.

When a new message is added, the conversation state has changed. That is the right moment to persist the latest user or assistant message back into memory.

The core interaction looks like this:

recent = memory_client.get_last_k_turns(
    memory_id=memory_id,
    actor_id=actor_id,
    session_id=session_id,
    k=5,
)

memory_client.create_event(
    memory_id=memory_id,
    actor_id=actor_id,
    session_id=session_id,
    messages=[(text, role)],
)

What I like about this model is that it is deterministic.

memory load happens before reasoning,
memory write happens when conversation state changes,
both operations use the same identity boundaries,
the entrypoint stays focused on request extraction rather than conversation orchestration.

That made the system easier to debug, easier to extend, and much easier to explain.

Lesson 4: Identity is the real memory boundary

Before building this, I thought of memory mostly as a storage problem. In practice, I learned it is just as much an identity problem.

The two identifiers that mattered most were:

actor_id
session_id

This separation ended up being foundational.

Why `actor_id` matters

actor_id is the user boundary. If that identifier is unstable, absent, or inconsistent, memory quality degrades immediately.

What I learned is that a memory system is only as good as the application identity you feed into it. If the same user appears under multiple IDs, the agent cannot retrieve a coherent conversational history. If two users are accidentally mapped to the same identity, memory becomes unsafe.

So one of my strongest takeaways is that actor_id should always come from a stable authenticated user identity, not from an incidental client-generated value.

Why `session_id` matters

session_id turned out to be just as important. A single user does not have just one conversation. They may have multiple active threads:

one troubleshooting flow,
one transcript analysis request,
one abandoned conversation from earlier,
one brand-new task.

Without a session boundary, all of that collapses into one memory stream. The agent might technically “remember,” but it remembers too much of the wrong thing.

That was a key lesson for me: useful memory is not just preserved memory. It is correctly scoped memory.

Lesson 5: The agent should be rebuilt per request, but memory should persist across requests

This was an architectural point that became clearer as I implemented the runtime flow.

The Strands agent instance itself is created per request. That makes sense because each invocation carries request-specific state:

the current user prompt,
the active user identity,
the active conversation session,
the active tool and runtime context.

But memory should not behave like request-local state. Memory has to outlive the agent instance and remain keyed to the same user and conversation across invocations.

That split was important for me to internalize:

agent instance lifecycle is short,
conversation memory lifecycle is longer,
the link between them is established through state and hooks.

Once I started thinking in those terms, the design felt much more natural.

Lesson 6: Deployment is part of the memory design

I originally thought of deployment as a separate concern from conversational behavior. Building this agent convinced me that the two are tightly connected.

The runtime needs to know which memory resource it should use, but I did not want that decision hardcoded in application logic. The better pattern was to resolve the correct memory resource during deployment and pass that identity into the runtime as configuration.

In practice, that meant the runtime received environment-specific values such as:

AGENT_NAME=<agent-name>
MEMORY_ID=<memory-id>

That gave me a few benefits immediately:

the same application code could move across environments,
memory resources stayed aligned with environment boundaries,
the runtime remained configurable without source changes,
the control plane remained the primary place where resource binding happened.

One of the clearest lessons here is that memory should be treated like any other environment-bound infrastructure dependency. If it is not part of deployment, it tends to become a hidden assumption.

Lesson 7: Short-term memory and long-term memory solve different problems

I found it helpful to stop using the word “memory” as if it meant one thing.

Short-term memory answered the question:

"What was happening in this conversation recently?"

Long-term memory answers a different question:

"What durable information should the system remember beyond this immediate interaction?"

For the agent I was building, the short-term problem came first. I needed:

recent-turn continuity,
bounded replay,
session-scoped context,
predictable event retention.

I did not need semantic fact retrieval in the first phase. I did not need vector search for historical knowledge. I needed the agent to remain coherent across adjacent turns.

That was an important design simplification. It kept the first version of the memory architecture focused on event continuity instead of overextending into knowledge retrieval prematurely.

Lesson 8: Recent-turn replay should be bounded

Once I had memory retrieval working, the next question was how much of it to inject back into the agent context.

My lesson here was simple: more memory is not always better memory.

If too much prior conversation is replayed:

prompt size grows,
token cost grows,
stale context starts competing with the current task,
reasoning quality can actually decline.

I found the most practical pattern was to retrieve the last few turns and inject them into prompt context in a compact representation. In this design, that replay window was bounded at five turns.

That gave me a good balance:

enough recent context for continuity,
small enough context for predictable prompt growth,
simple enough formatting to inspect and debug.

This also reinforced another lesson: short-term memory should be operationally understandable. I want to know what context the model saw, not just trust that some opaque memory layer handled it correctly.

Lesson 9: Memory becomes more valuable when tools are involved

The agent I built was not just a conversational shell. It had tools, including domain-specific behavior such as transcript retrieval and AWS interactions.

That is where the value of short-term memory became even more obvious.

In a tool-using workflow, the user often does not repeat the full context every turn. They say things like:

"use the same meeting"
"what did the second speaker say?"
"now summarize that"
"check the S3 output from before"

Without memory, the agent has to reconstruct working state from a single prompt. With memory, the agent has a much better chance of preserving:

the active object under discussion,
the prior user instruction,
the last tool result,
the intended next step.

One of my strongest takeaways is that memory is not just a conversational improvement. It is a workflow improvement. It makes tool orchestration across turns materially more coherent.

Lesson 10: Failure modes need to be designed, not discovered in production

Building this also made me think much more carefully about degraded behavior.

If memory resolution fails and the runtime cannot find a memory resource, the agent may still run. That sounds convenient, but it also means the system may silently shift from stateful to stateless behavior.

That taught me to treat the following as first-class operational conditions:

memory enabled,
memory disabled,
memory load succeeded,
memory write succeeded,
memory resolution failed,
identity inputs were missing or malformed.

The same thing applies to identity mistakes.

If actor_id is unstable, memory becomes fragmented.

If session_id is reused incorrectly, unrelated conversations bleed into each other.

If replay windows grow without discipline, prompt quality degrades.

These are not edge cases. They are part of the normal operating surface of a memory-aware agent.

Lesson 11: Retention, privacy, and compliance show up earlier than expected

Short-term memory sounds lightweight, but it is still stored interaction data.

That means retention policy is not just a platform setting. It is part of the product design. While building this, I became much more aware that memory decisions quickly intersect with:

data handling policy,
privacy expectations,
deletion and retention requirements,
security review,
production observability.

The technical implementation can be elegant, but if these operational questions are not addressed early, the design will be incomplete.

Lesson 12: AgentCore became more useful to me when I treated it as a runtime system, not just a hosting target

This may be the broadest lesson of all.

At first, I thought of AgentCore Runtime mainly as the place where the agent container would run. But while building with memory, I started appreciating it more as a runtime environment with clear operational boundaries:

the runtime executes the agent,
the framework manages reasoning and tools,
the memory plane manages event continuity,
the deployment workflow binds the right resources together.

That view helped me move beyond “deploy a model wrapper in a container” toward “operate an agent system with state, identity, and lifecycle.”

For me, that was the real shift.

The technical pattern I would reuse

If I were building the same class of agent again, I would reuse the same high-level pattern:

Create a dedicated short-term memory resource.
Resolve the correct memory resource during deployment.
Pass memory identity into the runtime explicitly.
Build the agent per request with user and session state.
Load recent turns during agent initialization.
Persist new messages when they are added.
Keep replay windows bounded.
Treat actor_id and session_id as core correctness boundaries.

I would also keep the same mental model:

short-term memory is for continuity,
long-term memory is for durable recall,
hooks are the right place for memory orchestration,
deployment is part of memory architecture,
observability should make degraded memory behavior visible.

Closing thought

The biggest lesson I learned while building with Amazon Bedrock AgentCore Runtime is that memory is not something you sprinkle onto an agent once the rest of the system works. Memory changes the shape of the system.

It affects:

request lifecycle,
identity boundaries,
prompt construction,
deployment,
observability,
privacy,
and tool coherence across turns.

Once I accepted that, the architecture became much more disciplined. The agent became easier to reason about, easier to operate, and much more capable in real multi-turn interactions.

That is the lesson I would carry into any future AgentCore build: if the experience is meant to feel conversational, memory has to be designed as a first-class runtime concern from the beginning.

API Gateway as Websocket

Amit Kayal — Tue, 21 Jan 2025 07:49:42 +0000

API Gateway as websocket

API Gateway as WS Components

Websocket provides bidirectional session aware communication between caller and receiver and a crucial component for realtime application.

Setup API Gateway for WebSocket
- Create a WebSocket API in the Amazon API Gateway console or through IAC.
- Define the WebSocket API route selection expression. Routes here are simply like a bridge to connections e.g.,
  - $request.body.action.
  - Define the following WebSocket routes:
  - $connect: Triggered when a client establishes a connection.
  - $disconnect: Triggered when a client disconnects.
  - Custom routes, e.g., sendMessage, to handle specific actions.
Create an Integration with AWS Lambda
- For each route ($connect, $disconnect, custom routes), integrate a Lambda function to handle the respective logic.
- Use the Lambda function's handler to process:
  - $connect: Store the connection in DynamoDB.
  - $disconnect: Remove the connection from DynamoDB.
  - Custom routes: Process the message and forward it to SQS.
DynamoDB for Connection Management
- Create a DynamoDB table to store:
  - Connection ID (Primary Key).
  - Session ID or other metadata for grouping connections.
- This table allows tracking active WebSocket connections for broadcasting messages.
Configure SQS for Message Queue
- Use an SQS FIFO queue for guaranteed order and deduplication.
- Messages processed in Lambda (custom routes) are sent to SQS for downstream services.
IAM Roles and Permissions
- Assign an IAM role to the API Gateway to invoke the integrated Lambda functions.
- Grant Lambda permissions to read/write from DynamoDB and send messages to SQS.
Client Connection and Messaging
- Use WebSocket-compatible libraries (e.g., ws in Node.js or WebSocket API in browsers) to:
- Establish a WebSocket connection to the API Gateway endpoint.
- Send and receive messages using the WebSocket protocol.

Architecture of Websocket mechanism

WebSocket Client:
- Initiates WebSocket connection and communicates via send() and onmessage().
API Gateway (WebSocket API):
- Manages WebSocket connections and invokes Lambda functions for defined routes.
Route Integration (Lambda Functions):
Every route should have an integration. There are 3 types — Mock, HTTP and Lambda.
- $connect: Adds connection metadata to DynamoDB.
- $disconnect: Removes connection metadata from DynamoDB.
- $default route: selected when route cant be evaluated against message
- Custom Routes: Processes messages to invoke integration based on message content and forwards them to SQS.
DynamoDB:
- Maintains active connection records, including connectionId and associated metadata.
SQS FIFO Queue:
- Queues messages for downstream processing, ensuring delivery order and deduplication.
Downstream Services:
- Processes messages from SQS and performs actions like notifications, data updates, or storage.

Security

Authentication and Authorization

Custom Authorizer (Lambda Authorizer)
It can only be used for the $connect route.
- Create a Lambda Authorizer to validate custom tokens or headers sent during connection attempts.
- Example:
  - Validate a JWT token from an identity provider (e.g., Cognito, Auth0).
  - Check the token against allowed users or roles.
Amazon Cognito:
- Use Amazon Cognito for user authentication.
- Configure API Gateway to use Cognito to validate tokens in connection requests.
- Best suited for applications with user pools.

Secure WebSocket Connections

Always use the secure WebSocket protocol (wss://). API Gateway enforces HTTPS/TLS, ensuring encrypted communication.
Associate a custom domain with API Gateway WebSocket endpoint. We should AWS Certificate Manager (ACM) to manage SSL/TLS certificates.

IP Whitelisting and Blacklisting

IP Whitelisting and Blacklisting: We should Attach AWS WAF to API Gateway and Block/allow requests based on IP addresses or CIDR ranges. we should also use rate limit to protect from DDoS attack ### API Gateway Throttling
We can Set rate and burst limits on API Gateway routes to limit the number of connections per client.
We can create API keys and associate them with usage plan and then we Limit the number of allowed requests per API key

Environment-based Access Control:

We should always use distinct stages (e.g., dev, prod) and restrict connections to the production API through IP rules.

Tools to test

There are following tools which we can explore to test websocket.

Piesocket
Postman

S3 table & S3 Metadata table

Amit Kayal — Mon, 09 Dec 2024 18:26:23 +0000

Open table format and its architecture

OpenTable formats, such as Apache Iceberg, Apache Hudi, and Delta Lake, have gained popularity in the data analytics mainly because:

ACID Transactions: OpenTable formats (e.g., Apache Iceberg, Delta Lake) ensure reliable and consistent data updates, even with concurrent access.
Schema Evolution: They allow seamless updates to schemas without disrupting existing pipelines, simplifying data management. metadata tracks the changes to the dataset. The files held in the Data layer are captured by the metadata files held in the Metadata layer. As the files change, the metadata files attached to them track these changes.
Optimized Queries: Partitioning and indexing enable faster queries by scanning only relevant data, improving performance and cost-efficiency.
Time Travel: Users can access historical versions of data for debugging, compliance, or analytics.
Interoperability: These formats integrate seamlessly with big data tools like Spark, Flink, and Presto, making them versatile and widely adopted.

Open file format

S3 table

Key Features

Amazon S3 Table is optimized for analytics workloads. It is designed to continuously enhance query performance and reduce storage costs for tabular data. This solution looks very promising if you are working with LakeHouse architecture. It’s a new type of bucket that organizes tables as sub-resources.
A new bucket type s3 table has been introduced to support this. As liked any other aws resoyrce, it has ARN, can take resource policy and as an unique feature it has dedicated endpoint.

S3 Tables are intended explicitly for storing data in a tabular format, such as daily purchase transactions, streaming sensor data, or ad impressions. This data is organized into columns and rows like a database table.
Table buckets support storing tables in the Apache Iceberg format. You can query these tables using standard SQL in query engines that support Iceberg.
Read/write allowed on datafiles and metadata files. Delete and update not allowed to save data integrity.
Compatible query engines include Amazon Athena, Amazon Redshift, and Apache Spark.
S3 Table automatically performs maintenance tasks like compaction and snapshot management to optimize your tables for querying, including removing unreferenced files.
S3 Table offers access management for both table and bucket
Fully managed apache icebarg tables in S3
It supports automatic compaction of underlying files to improve query performance and tune then further for better latency.

S3 Table buckets namespace

Namespace logically groups related s3 table together and thus allowing us to have greater control based on namespace of s3 tables. It helps us for following:

logical segmentation of data and multi tenancy
- supporting of multi tenancy by having separate namespace. Supports compliance with data isolation requirements in regulated industries.
- separate tables based on application, project etc
prevent naming conflicts
- Each namespace acts like a "container," allowing tables with the same name in different namespaces without conflicts.
Better Access Control
- Policies can grant or restrict access to specific namespaces, ensuring data security and compliance. It also reduces the risk of unauthorized access to unrelated tables in the same bucket.
Easy data management
- Makes our life easier to query, update, or delete related tables in bulk.
- Makes easy metadata management for tables grouped under a namespace.
Advanced workflows based on namespace
- It helps to simplify automation for data pipelines or real-time analytics applications.

S3 table opertaion & management

Table Operation
They are quite similar to CRUD operation.

list tables
create tables
Get table metadata location
Update table metadata location
Delete Table

Table Management

Put Table Policy
Put Table Bucket Policy
Put Table Maintenance Config
Put Table Bucket Maintenance Config

Policies related to S3 table operation

Allow access to create and use table buckets

Here Action Lists the specific actions the policy allows.

These actions are S3 Tables-specific:

s3tables:CreateTableBucket: Grants permission to create a table bucket in S3 Tables.
s3tables:PutTableBucketPolicy: Allows setting or updating the bucket policy for a table bucket.
s3tables:GetTableBucketPolicy: Allows retrieving the bucket policy associated with a table bucket.
s3tables:ListTableBuckets: Allows listing all table buckets within the specified scope.
s3tables:GetTableBucket: Grants permission to access the metadata of a specific table bucket.
Resource Defines the scope of the resources these actions can apply to.
"arn:aws:s3tables:region:account_id:bucket/*": Specifies all table buckets in the account (account_id) and region (region).
The * after bucket/ indicates that permissions apply to all buckets under this account and region.

{
    "Version": "2012-10-17",
    "Statement": [{
        "Sid": "AllowBucketActions for user",
        "Effect": "Allow",
        "Action": [
            "s3tables:CreateTableBucket",
            "s3tables:PutTableBucketPolicy",
            "s3tables:GetTableBucketPolicy",
            "s3tables:ListTableBuckets",
            "s3tables:GetTableBucket"
        ],
        "Resource": "arn:aws:s3tables:region:account_id:bucket/*"
    }]
}

Allow access to create and use tables in a table bucket

Here Action Lists the specific actions allowed by the policy, related to S3 Tables. Please note that The first policy focused on creating and managing table buckets and associated metadata, but it did not include granular operations like managing tables within namespaces. The first policy did not include actions such as creating tables, querying data, or updating metadata at the table level. These are the operations where namespaces become relevant.

s3tables:CreateTable: Allows creating new tables in the specified table bucket.
s3tables:PutTableData: Grants permission to write data to tables within the table bucket.
s3tables:GetTableData: Allows reading data from tables in the bucket.
s3tables:GetTableMetadataLocation: Allows retrieving metadata location information for a table.
s3tables:UpdateTableMetadataLocation: Grants permission to update the metadata location of a table.
s3tables:GetNamespace: Allows retrieving namespace information associated with the table bucket.
s3tables:CreateNamespace: Grants permission to create namespaces for organizing table data.

Resource section specifies

Grants permissions on the bucket named amzn-s3-demo-table-bucket
Grants permissions on all tables within the amzn-s3-demo-table-bucket

{
     "Version": "2012-10-17",
     "Statement": [ 
         {
             "Sid": "AllowBucketActions",
             "Effect": "Allow",
             "Action": [
                 "s3tables:CreateTable",
                 "s3tables:PutTableData",
                 "s3tables:GetTableData",
                 "s3tables:GetTableMetadataLocation",
                 "s3tables:UpdateTableMetadataLocation",
                 "s3tables:GetNamespace",
                 "s3tables:CreateNamespace"
             ],

             "Resource": [
               "arn:aws:s3tables:region:account_id:bucket/amzn-s3-demo-table-bucket",
               "arn:aws:s3tables:region:account_id:bucket/amzn-s3-demo-table-bucket/table/*"
            ]
         }
     ]
 }

Table bucket policy to allows read access to the namespace

This policy allows to read s3 tables from a namespace. Here Action Lists the specific actions allowed by the policy, related to S3 Tables.

s3tables:GetTableData: Allows reading data from tables in the bucket.
s3tables:GetTableMetadataLocation: Allows retrieving metadata location information for a table. The resource section allows all s3 tables under bucket amzn-s3-demo-table-bucket1 but then s3tables:namespace restrict to only hr related s3 tables.

{
     "Version": "2012-10-17",
     "Statement": [ 
         {
             "Effect": "Allow",
             "Action": [
             "Principal": {
               "AWS": "arn:aws:iam::123456789012:user/Jane"
             },
             "Action": [
                  "s3tables:GetTableData", 
                  "s3tables:GetTableMetadataLocation"
             ],
             "Resource":{ "arn:aws:s3tables:region:account_id:bucket/amzn-s3-demo-table-bucket1/table/*”}
             "Condition": { 
                  "StringLike": { "s3tables:namespace": "hr" } 
             }
     ]
 }

S3 table automatic maintenance

It provides automated maintenance through configurations that help simplify table management, optimize performance, and reduce operational overhead.

Table Lifecycle Management
- we can add S3 Table configurations that includes lifecycle policies that automatically handle data expiration, transitions, or archival.
- automatic snapshot expiration can be configured easily.
Data Compaction
- S3 Tables automatically compact small files (often produced by incremental writes) into larger, optimized files. It helps to have faster query and reduce storage cost.
Schema Evolution
- Automated checks ensure compatibility between new and existing data.
Metadata Optimization
- Indexing of metadata for faster querying and retrieval of table details.

All these can be policy based configuration.

Policy for snapshot management

By configuring the maximumSnapshotAge, we can specify the retention period for table snapshots. The following example ensures S3 Table will automatically retain only the snapshots from the last 30 days

MinimumSnapshots: Ensures that at least one snapshot is always retained, regardless of age.
MaximumSnapshotAge: Specifies the maximum age (in hours) for snapshots to be retained.

aws s3tables put-table-maintenance-configuration \
    --table-arn arn:aws:s3tables:region:account_id:bucket/bucket_name/table/table_name \
    --maintenance-configuration '{
        "SnapshotManagement": {
            "MinimumSnapshots": 1,
            "MaximumSnapshotAge": 720
        }
    }

S3 Table Integration with AWS Analytics

S3 Tables integrate seamlessly with AWS analytics services to enable querying, processing and insight generation.

Amazon Athena - Run serverless SQL queries on S3 Tables

Use AWS Glue to create a Data Catalog for S3 Tables.
Query data directly using SQL in Athena.
Leverage table formats like Apache Iceberg or Parquet for optimized performance.

AWS Glue - Automate ETL processes for S3 Tables

Use Glue Crawlers to discover table metadata.
Create ETL jobs to transform and load data into S3 Tables or other destinations.

S3 Metadata table

It includes system metadata including object tags and user defined metadata
stored into s3 table
generated in near real time during data creation so that it can be used in mins during query

Use case for S3 metadata table

Real-Time Analytics
- efficient query execution on metadata to identify relevant data partitions.
Machine Learning Pipelines
- metadata tables to filter, select, and partition data for model training.
Governance and Compliance
- Track data retention and enforce lifecycle policies via metadata.
Multi-Tenant Data Applications
- Use namespaces within metadata tables to logically isolate tenant data.
Data Cataloging and Discovery
- Use metadata queries to identify datasets matching specific criteria.

Here is the sample python based function which uses metadata table query from athena.

def query_metadata_table(criteria):

    query = f"""
        SELECT *
        FROM {DATABASE}.{TABLE}
        WHERE {criteria}
    """

    print(f"Running query: {query}")

    # Start Athena query
    response = athena_client.start_query_execution(
        QueryString=query,
        QueryExecutionContext={'Database': DATABASE},
        ResultConfiguration={'OutputLocation': S3_OUTPUT}
    )

    query_execution_id = response['QueryExecutionId']

    # Wait for query completion
    print("Waiting for query to complete...")
    while True:
        status = athena_client.get_query_execution(QueryExecutionId=query_execution_id)
        state = status['QueryExecution']['Status']['State']
        if state in ['SUCCEEDED', 'FAILED', 'CANCELLED']:
            break
        time.sleep(2)

    if state != 'SUCCEEDED':
        raise Exception(f"Query failed with state: {state}")

    # Retrieve results
    results = athena_client.get_query_results(QueryExecutionId=query_execution_id)
    datasets = []
    for row in results['ResultSet']['Rows'][1:]:  # Skip the header row
        datasets.append([col['VarCharValue'] for col in row['Data']])

    print(f"Query returned {len(datasets)} datasets matching the criteria.")
    return datasets

Brief Notes on AWS CodeDeploy

Amit Kayal — Thu, 21 Mar 2024 19:04:04 +0000

Service that automates code deployments to any instance, including Amazon EC2 instances and instances running on-premises.

Supported Platforms/Deployment Types:

EC2/On-Premises: In-Place or Blue/Green Deployments
- Describes instances of physical servers that can be Amazon EC2 cloud instances, on-premises servers, or both. Applications created using the EC2/On-Premises compute platform can be composed of executable files, configuration files, images, and more. o - - Deployments that use the EC2/On-Premises compute platform manage the way in which traffic is directed to instances by using an in-place or blue/green deployment type.
AWS Lambda: Canary, Linear, All-At-Once Deployments
- Applications created using the AWS Lambda compute platform can manage the way in which traffic is directed to the updated Lambda function versions during a deployment by choosing a canary, linear, or all-at-once configuration.
Amazon ECS: Blue/Green Deployment
- Used to deploy an Amazon ECS containerized application as a task set.
- CodeDeploy performs a blue/green deployment by installing an updated version of the containerized application as a new replacement task set. CodeDeploy reroutes production traffic from the original application, or task set, to the replacement task set. The original task set is terminated after a successful deployment.

Deployment approach for EC2

Deploys a revision to a set of instances.
Deploys a new revision that consists of an application and AppSpec file. The AppSpec specifies how to deploy the application to the instances in a deployment group.

Deployment approach for Lambda

Deploys a new version of a serverless Lambda function on a high-availability compute infrastructure.
Shifts production traffic from one version of a Lambda function to a new version of the same function. The AppSpec file specifies which Lambda function version to deploy.

Deployment approach for ECS

Deploys an updated version of an Amazon ECS containerized application as a new, replacement task set. CodeDeploy reroutes production traffic from the task set with the original version to the new replacement task set with the updated version. When the deployment completes, the original task set is terminated.

App Spec File

The application specification file (AppSpec file) is a YAML-formatted or JSON-formatted file used by CodeDeploy to manage a deployment. Note: the name of the AppSpec file for an EC2/On-Premises deployment must be appspec.yml. The name of the AppSpec file for an Amazon ECS or AWS Lambda deployment must be appspec.yml.

For ECS

The container and port in replacement task set where your Application Load Balancer or Network Load Balancer reroutes traffic during a deployment. This is specified with the LoadBalancerInfo instruction in the AppSpec file.
Amazon ECS task definition file. This is specified with its ARN in the TaskDefinition instruction in the AppSpec file.

For Lambda

Lambda function version to deploy.
Lambda functions to use as validation tests.

For EC2

Which lifecycle event hooks to run in response to deployment lifecycle events.

Bedrock Agent & Tools - Tracing Best practises

Amit Kayal — Wed, 20 Mar 2024 17:58:52 +0000

I understand most of bedrock agent userss will have a use case where you have implemented multiple Lambda functions with a Bedrock Agent to perform different tasks and are looking for guidance in Debugging the API calls and responses from the Agent and lambda functions.

Here are some of the approaches that we have been using and found quite effective to track and trace agents and usage of their tools

Enable Tracing for the Agent: When invoking the agent, set the debug parameter to true. This will enable detailed tracing for the agent's execution, including the tools (Lambda functions) invoked and their responses. The trace will be printed to the console or returned as part of the agent's response, depending on how you invoke the agent. [1] Example (Python):

python result = agent.run(query, debug=True)

Log Within Lambda Functions: Within each of your Lambda functions (tools), add logging statements to capture relevant information and events. You can use AWS Lambda's built-in logging capabilities or integrate with a centralized logging service like Amazon CloudWatch Logs. [2] Example (Python):

python import logging logger = logging.getLogger(__name__) def lambda_handler(event, context): http://logger.info (f"Received event: {event}") # Your Lambda function's logic here http:// logger.info (f"Returning result: {result}") return result

Correlate Logs Using Request IDs or Tracing IDs: To correlate logs across multiple Lambda functions and the agent, you can use request IDs or tracing IDs. Pass a unique ID as part of the event or context to your Lambda functions and include it in your log statements. This will allow you to trace the flow of events across different components of your system.

import logging import uuid def lambda_handler(event, context): request_id = event.get("request_id", str(uuid.uuid4())) logger = logging.getLogger(__name__) logger = logging.LoggerAdapter(logger, {"request_id": request_id}) logger.info(f"Received event: {event}") logger.info(f"Returning result: {result}") return result

Use AWS X-Ray for Distributed Tracing: AWS X-Ray is a service that can help you analyze and debug distributed applications, including Lambda functions. By integrating X-Ray with your Bedrock application, you can trace requests as they travel through your Lambda functions and gain insights into their performance and potential issues. [3] - Enable X-Ray tracing for your Lambda functions by adding the necessary configuration. - Instrument your Lambda functions with X-Ray tracing code to capture relevant information and events. - Use the X-Ray console or integrate with other monitoring tools to analyze the traces and identify potential bottlenecks or issues.
Implement Advanced prompts : By using advanced prompts, you can enhance your agent's accuracy through modifying these prompt templates to provide detailed configurations. You can also provide hand-curated examples for few-shot prompting, in which you improve model performance by providing labeled examples for a specific task. [4] By combining the built-in tracing mechanism, custom logging within your Lambda functions, and distributed tracing with AWS X-Ray, you can gain better visibility into the API calls, events, and interactions happening within your Bedrock agent and its associated tools. This can help you debug issues more effectively and trace errors back to their source across multiple Lambda functions.
Reference

AWS DEV OPS Professional Exam short notes

Amit Kayal — Sun, 17 Mar 2024 05:55:58 +0000

Last few weeks I have been preparing for this exam and have summarized below key notes for further quick reference.

Key Notes

You can use CloudWatch Logs to monitor applications and systems using log data. For example, CloudWatch Logs can track the number of errors that occur in your application logs and send you a notification whenever the rate of errors exceeds a threshold you specify. CloudWatch Logs uses your log data for monitoring; so, no code changes are required. For more information on Cloudwatch logs , please refer to the below link: http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/WhatIsCloudWatchLogs.html The correct answer is: Install the CloudWatch Logs Agent on your AMI, and configure CloudWatch Logs Agent to stream your logs.
You can add another layer of protection by enabling MFA Delete on a versioned bucket. Once you do so, you must provide your AWS account’s access keys and a valid code from the account’s MFA device in order to permanently delete an object version or suspend or reactivate versioning on the bucket. For more information on MFA please refer to the below link: https://aws.amazon.com/blogs/security/securing-access-to-aws-using-mfa-part-3/ IAM roles are designed so that your applications can securely make API requests from your instances, without requiring you to manage the security credentials that the applications use. Instead of creating and distributing your AWS credentials, you can delegate permission to make API requests using IAM roles For more information on Roles for EC2 please refer to the below link: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html
As your infrastructure grows, common patterns can emerge in which you declare the same components in each of your templates. You can separate out these common components and create dedicated templates for them. That way, you can mix and match different templates but use nested stacks to create a single, unified stack. Nested stacks are stacks that create other stacks. To create nested stacks, use the AWS::CloudFormation::Stackresource in your template to reference other templates. For more information on best practices for Cloudformation please refer to the below link: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/best-practices.html The correct answer is: Separate the AWS CloudFormation template into a nested structure that has individual templates for the resources that are to be governed by different departments, and use the outputs from the networking and security stacks for the application template that you control.
You can use Amazon CloudWatch Logs to monitor, store, and access your log files from Amazon Elastic Compute Cloud (Amazon EC2) instances, AWS CloudTrail, and other sources. You can then retrieve the associated log data from CloudWatch Logs. For more information on Cloudwatch logs please refer to the below link: http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/WhatIsCloudWatchLogs.html You can the use Kinesis to process those logs For more information on Amazon Kinesis please refer to the below link: http://docs.aws.amazon.com/streams/latest/dev/introduction.html The correct answers are: Using AWS CloudFormation, create a CloudWatch Logs LogGroup and send the operating system and application logs of interest using the CloudWatch Logs Agent., Using configuration management, set up remote logging to send events to Amazon Kinesis and insert these into Amazon CloudSearch or Amazon Redshift, depending on available analytic tools.
IAM roles are designed so that your applications can securely make API requests from your instances, without requiring you to manage the security credentials that the applications use. Instead of creating and distributing your AWS credentials For more information on IAM Roles please refer to the below link: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html
The AWS Security Token Service (STS) is a web service that enables you to request temporary, limited-privilege credentials for AWS Identity and Access Management (IAM) users or for users that you authenticate (federated users). The token can then be used to grant access to the objects in S3. You can then provides access to the objects based on the key values generated via the user id
As your infrastructure grows, common patterns can emerge in which you declare the same components in each of your templates. You can separate out these common components and create dedicated templates for them. That way, you can mix and match different templates but use nested stacks to create a single, unified stack. Nested stacks are stacks that create other stacks. To create nested stacks, use the AWS::CloudFormation::Stackresource in your template to reference other templates. For more information on Cloudformation best practises please refer to the below link: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/best-practices.html The correct answer is: Create separate templates based on functionality, create nested stacks with CloudFormation.
The default autosclae termination policy is designed to help ensure that your network architecture spans Availability Zones evenly. When using the default termination policy, Auto Scaling selects an instance to terminate as follows: Auto Scaling determines whether there are instances in multiple Availability Zones. If so, it selects the Availability Zone with the most instances and at least one instance that is not protected from scale in. If there is more than one Availability Zone with this number of instances, Auto Scaling selects the Availability Zone with the instances that use the oldest launch configuration. For more information on Autoscaling instance termination please refer to the below link: http://docs.aws.amazon.com/autoscaling/latest/userguide/as-instance-termination.html The correct answer is: Auto Scaling will select the AZ with 4 EC2 instances and terminate an instance.
Amazon RDS Read Replicas provide enhanced performance and durability for database (DB) instances. This replication feature makes it easy to elastically scale out beyond the capacity constraints of a single DB Instance for read-heavy database workloads. You can create one or more replicas of a given source DB Instance and serve high-volume application read traffic from multiple copies of your data, thereby increasing aggregate read throughput. Sharding is a common concept to split data across multiple tables in a database. Shard your data set among multiple Amazon RDS DB instances.Amazon ElastiCache is a web service that makes it easy to deploy, operate, and scale an in-memory data store or cache in the cloud. The service improves the performance of web applications by allowing you to retrieve information from fast, managed, in-memory data stores, instead of relying entirely on slower disk-based databases.
Continuous Integration (CI) is a development practice that requires developers to integrate code into a shared repository several times a day. Each check-in is then verified by an automated build, allowing teams to detect problems early.
Elastic Beanstalk simplifies this process by managing the Amazon SQS queue and running a daemon process on each instance that reads from the queue for you. When the daemon pulls an item from the queue, it sends an HTTP POST request locally to http://localhost/ with the contents of the queue message in the body. All that your application needs to do is perform the long-running task in response to the POST. For more information Elastic Beanstalk managing worker environments, please visit the below URL: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features-managing-env-tiers.html
If you suspend AddToLoadBalancer, Auto Scaling launches the instances but does not add them to the load balancer or target group. If you resume the AddToLoadBalancer process, Auto Scaling resumes adding instances to the load balancer or target group when they are launched. However, Auto Scaling does not add the instances that were launched while this process was suspended. You must register those instances manually. For more information on the Suspension and Resumption process, please visit the below URL: http://docs.aws.amazon.com/autoscaling/latest/userguide/as-suspend-resume-processes.html
You can use the container_commands key of elastic beanstalk to execute commands that affect your application source code. Container commands run after the application and web server have been set up and the application version archive has been extracted, but before the application version is deployed. Non-container commands and other customization operations are performed prior to the application source code being extracted. You can use leader_only to only run the command on a single instance, or configure a test to only run the command when a test command evaluates to true. Leader-only container commands are only executed during environment creation and deployments, while other commands and server customization operations are performed every time an instance is provisioned or updated. Leader-only container commands are not executed due to launch configuration changes, such as a change in the AMI Id or instance type. For more information on customizing containers, please visit the below URL: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/customize-containers-ec2.html The correct answer is: Use a “Container command” within an Elastic Beanstalk configuration file to execute the script, ensuring that the “leader only” flag is set to true.
A Dockerrun.aws.json file is an Elastic Beanstalk–specific JSON file that describes how to deploy a set of Docker containers as an Elastic Beanstalk application. You can use aDockerrun.aws.json file for a multicontainer Docker environment. Dockerrun.aws.json describes the containers to deploy to each container instance in the environment as well as the data volumes to create on the host instance for the containers to mount. http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create_deploy_docker_v2config.html
Elastic Beanstalk supports the deployment of web applications from Docker containers. With Docker containers, you can define your own runtime environment. You can choose your own platform, programming language, and any application dependencies (such as package managers or tools), that aren’t supported by other platforms. Docker containers are self-contained and include all the configuration information and software your web application requires to run.
When you see Amazon Kinesis as an option, this becomes the ideal option to process data in real time. Amazon Kinesis makes it easy to collect, process, and analyze real-time, streaming data so you can get timely insights and react quickly to new information. Amazon Kinesis offers key capabilities to cost effectively process streaming data at any scale, along with the flexibility to choose the tools that best suit the requirements of your application. With Amazon Kinesis, you can ingest real-time data such as application logs, website clickstreams, IoT telemetry data, and more into your databases, data lakes and data warehouses, or build your own real-time applications using this data. For more information on Amazon Kinesis, please visit the below URL: https://aws.amazon.com/kinesis
You can use CloudWatch Logs to monitor applications and systems using log data CloudWatch Logs uses your log data for monitoring; so, no code changes are required. For example, you can monitor application logs for specific literal terms (such as “NullReferenceException”) or count the number of occurrences of a literal term at a particular position in log data (such as “404” status codes in an Apache access log). When the term you are searching for is found, CloudWatch Logs reports the data to a CloudWatch metric that you specify. Log data is encrypted while in transit and while it is at rest For more information on Cloudwatch logs please refer to the below link: http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/WhatIsCloudWatchLogs.html Amazon CloudWatch uses Amazon SNS to send email. First, create and subscribe to an SNS topic. When you create a CloudWatch alarm, you can add this SNS topic to send an email notification when the alarm changes state. For more information on SNS and Cloudwatch logs please refer to the below link: The correct answers are: Install a CloudWatch Logs Agent on your servers to stream web application logs to CloudWatch., Create a CloudWatch Logs group and define metric filters that capture 500 Internal Server Errors. Set a CloudWatch alarm on that metric., Use Amazon Simple Notification Service to notify an on-call engineer when a CloudWatch alarm is triggered
When you provision an Amazon EC2 instance in an AWS CloudFormation stack, you might specify additional actions to configure the instance, such as install software packages or bootstrap applications. Normally, CloudFormation proceeds with stack creation after the instance has been successfully created. However, you can use a CreationPolicy so that CloudFormation proceeds with stack creation only after your configuration actions are done. That way you’ll know your applications are ready to go after stack creation succeeds.
Auto Scaling periodically performs health checks on the instances in your Auto Scaling group and identifies any instances that are unhealthy. You can configure Auto Scaling to determine the health status of an instance using Amazon EC2 status checks, Elastic Load Balancing health checks, or custom health checks By default, Auto Scaling health checks use the results of the EC2 status checks to determine the health status of an instance. Auto Scaling marks an instance as unhealthy if its instance fails one or more of the status checks. For more information monitoring in Autoscaling , please visit the below URL: http://docs.aws.amazon.com/autoscaling/latest/userguide/as-monitoring-features.html
You need to have a custom health check which will evaluate the application functionality. Its not enough using the normal health checks. If the application functionality does not work and if you don’t have custom health checks , the instances will still be deemed as healthy. If you have custom health checks, you can send the information from your health checks to Auto Scaling so that Auto Scaling can use this information. For example, if you determine that an instance is not functioning as expected, you can set the health status of the instance to Unhealthy. The next time that Auto Scaling performs a health check on the instance, it will determine that the instance is unhealthy and then launch a replacement instance For more information on Autoscaling health checks , please refer to the below document link: from AWS http://docs.aws.amazon.com/autoscaling/latest/userguide/healthcheck.html
A blue group carries the production load while a green group is staged and deployed with the new code. When it’s time to deploy, you simply attach the green group to the existing load balancer to introduce traffic to the new environment. For HTTP/HTTPS listeners, the load balancer favors the green Auto Scaling group because it uses a least outstanding requests routing algorithm As you scale up the green Auto Scaling group, you can take blue Auto Scaling group instances out of service by either terminating them or putting them in Standby state, For more information on Blue Green Deployments , please refer to the below document link: from AWS https://d0.awsstatic.com/whitepapers/AWS_Blue_Green_Deployments.pdf
Ensure first that the cloudformation template is updated with the new instance type. The AWS::AutoScaling::AutoScalingGroup resource supports an UpdatePolicy attribute. This is used to define how an Auto Scaling group resource is updated when an update to the CloudFormation stack occurs. A common approach to updating an Auto Scaling group is to perform a rolling update, which is done by specifying the AutoScalingRollingUpdate policy. This retains the same Auto Scaling group and replaces old instances with new ones, according to the parameters specified
With web identity federation, you don’t need to create custom sign-in code or manage your own user identities. Instead, users of your app can sign in using a well-known identity provider (IdP) —such as Login with Amazon, Facebook, Google, or any other OpenID Connect (OIDC)-compatible IdP, receive an authentication token, and then exchange that token for temporary security credentials in AWS that map to an IAM role with permissions to use the resources in your AWS account. Using an IdP helps you keep your AWS account secure, because you don’t have to embed and distribute long-term security credentials with your application.
The optional Conditions section includes statements that define when a resource is created or when a property is defined. For example, you can compare whether a value is equal to another value. Based on the result of that condition, you can conditionally create resources. If you have multiple conditions, separate them with commas. You might use conditions when you want to reuse a template that can create resources in different contexts, such as a test environment versus a production environment. In your template, you can add an EnvironmentType input parameter, which accepts either prod or test as inputs. For the production environment, you might include Amazon EC2 instances with certain capabilities; however, for the test environment, you want to use reduced capabilities to save money. With conditions, you can define which resources are created and how they’re configured for each environment type. For more information on Cloudformation conditions please refer to the below link: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/conditions-section-structure.html
Elastic Beanstalk already has the facility to manage various versions and you don’t need to use S3 separately for this.AWS beanstalk is the perfect solution for developers to maintain application versions. With AWS Elastic Beanstalk, you can quickly deploy and manage applications in the AWS Cloud without worrying about the infrastructure that runs those applications. AWS Elastic Beanstalk reduces management complexity without restricting choice or control. You simply upload your application, and AWS Elastic Beanstalk automatically handles the details of capacity provisioning, load balancing, scaling, and application health monitoring.
The first step in using Elastic Beanstalk is to create an application, which represents your web application in AWS. In Elastic Beanstalk an application serves as a container for the environments that run your web app, and versions of your web app’s source code, saved configurations, logs and other artifacts that you create while using Elastic Beanstalk. For more information on Applications, please refer to the below link: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/applications.html Deploying a new version of your application to an environment is typically a fairly quick process. The new source bundle is deployed to an instance and extracted, and then the web container or application server picks up the new version and restarts if necessary. During deployment, your application might still become unavailable to users for a few seconds. You can prevent this by configuring your environment to use rolling deployments to deploy the new version to instances in batches. For more information on deployment, please refer to the below link: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features.deploy-existing-version.html
Weighted routing lets you associate multiple resources with a single domain name (example.com) or subdomain name (acme.example.com) and choose how much traffic is routed to each resource. This can be useful for a variety of purposes, including load balancing and testing new versions of software. For more information on the Routing policy please refer to the below link: http://docs.aws.amazon.com/Route53/latest/DeveloperGuide/routing-policy.html
Amazon Elasticsearch Service makes it easy to deploy, operate, and scale Elasticsearch for log analytics, full text search, application monitoring, and more. Amazon Elasticsearch Service is a fully managed service that delivers Elasticsearch’s easy-to-use APIs and real-time capabilities along with the availability, scalability, and security required by production workloads. The service offers built-in integrations with Kibana, Logstash, and AWS services including Amazon Kinesis Firehose, AWS Lambda, and Amazon CloudWatch so that you can go from raw data to actionable insights quickly.
You can use CloudWatch Logs to monitor applications and systems using log data. For example, CloudWatch Logs can track the number of errors that occur in your application logs and send you a notification whenever the rate of errors exceeds a threshold you specify. CloudWatch Logs uses your log data for monitoring; so, no code changes are required. For example, you can monitor application logs for specific literal terms (such as “NullReferenceException”) or count the number of occurrences of a literal term at a particular position in log data (such as “404” status codes in an Apache access log). When the term you are searching for is found, CloudWatch Logs reports the data to a CloudWatch metric that you specify. For more information on Cloudwatch Logs please refer to the below link: http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/WhatIsCloudWatchLogs.html Amazon CloudWatch uses Amazon SNS to send email. First, create and subscribe to an SNS topic. When you create a CloudWatch alarm, you can add this SNS topic to send an email notification when the alarm changes state. For more information on Cloudwatch and SNS please refer to the below link: http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/US_SetupSNS.html
AWS OpsWorks is a configuration management service that uses Chef, an automation platform that treats server configurations as code. OpsWorks uses Chef to automate how servers are configured, deployed, and managed across your Amazon Elastic Compute Cloud (Amazon EC2) instances or on-premises compute environments. OpsWorks has two offerings, AWS Opsworks for Chef Automate, and AWS OpsWorks Stacks. For more information on Opswork and SNS please refer to the below link: https://aws.amazon.com/opsworks/
You can use Kinesis Streams for rapid and continuous data intake and aggregation. The type of data used includes IT infrastructure log data, application logs, social media, market data feeds, and web clickstream data. Because the response time for the data intake and processing is in real time, the processing is typically lightweight. The following are typical scenarios for using Kinesis Streams: Accelerated log and data feed intake and processing – You can have producers push data directly into a stream. For example, push system and application logs and they’ll be available for processing in seconds. This prevents the log data from being lost if the front end or application server fails. Kinesis Streams provides accelerated data feed intake because you don’t batch the data on the servers before you submit it for intake. Real-time metrics and reporting – You can use data collected into Kinesis Streams for simple data analysis and reporting in real time. For example, your data-processing application can work on metrics and reporting for system and application logs as the data is streaming in, rather than wait to receive batches of data. For more information on Amazon Kinesis and SNS please refer to the below link: http://docs.aws.amazon.com/streams/latest/dev/introduction.html
With Elastic Beanstalk, you can quickly deploy and manage applications in the AWS Cloud without worrying about the infrastructure that runs those applications. AWS Elastic Beanstalk reduces management complexity without restricting choice or control. You simply upload your application, and Elastic Beanstalk automatically handles the details of capacity provisioning, load balancing, scaling, and application health monitoring For more information on Elastic beanstalk please refer to the below link: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/Welcome.html
You can use intrinsic functions, such as Fn::If, Fn::Equals, and Fn::Not, to conditionally create stack resources. These conditions are evaluated based on input parameters that you declare when you create or update a stack. After you define all your conditions, you can associate them with resources or resource properties in the Resources and Outputs sections of a template.
Amazon RDS Multi-AZ deployments provide enhanced availability and durability for Database (DB) Instances, making them a natural fit for production database workloads. When you provision a Multi-AZ DB Instance, Amazon RDS automatically creates a primary DB Instance and synchronously replicates the data to a standby instance in a different Availability Zone (AZ). Each AZ runs on its own physically distinct, independent infrastructure, and is engineered to be highly reliable. In case of an infrastructure failure, Amazon RDS performs an automatic failover to the standby (or to a read replica in the case of Amazon Aurora), so that you can resume database operations as soon
You can use AWS CloudTrail to get a history of AWS API calls and related events for your account. This history includes calls made with the AWS Management Console, AWS Command Line Interface, AWS SDKs, and other AWS services. For more information on Cloudtrail, please visit the below URL: http://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-user-guide.html Amazon CloudWatch Events delivers a near real-time stream of system events that describe changes in Amazon Web Services (AWS) resources. Using simple rules that you can quickly set up, you can match events and route them to one or more target functions or streams. CloudWatch Events becomes aware of operational changes as they occur. CloudWatch Events responds to these operational changes and takes corrective action as necessary, by sending messages to respond to the environment, activating functions, making changes, and capturing state information
By default, all AWS accounts are limited to 5 Elastic IP addresses per region, because public (IPv4) Internet addresses are a scarce public resource. We strongly encourage you to use an Elastic IP address primarily for the ability to remap the address to another instance in the case of instance failure, and to use DNS hostnames for all other inter-node communication
You can manage Amazon SQS messages with Amazon S3. This is especially useful for storing and consuming messages with a message size of up to 2 GB. To manage Amazon SQS messages with Amazon S3, use the Amazon SQS Extended Client Library for Java. Specifically, you use this library to: Specify whether messages are always stored in Amazon S3 or only when a message’s size exceeds 256 KB. Send a message that references a single message object stored in an Amazon S3 bucket. Get the corresponding message object from an Amazon S3 bucket. Delete the corresponding message object from an Amazon S3 bucket. For more information on processing large messages for SQS, please visit the below URL: http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-s3-messages.html
AWS CloudFormation provisions and configures resources by making calls to the AWS services that are described in your template. After all the resources have been created, AWS CloudFormation reports that your stack has been created. You can then start using the resources in your stack. If stack creation fails, AWS CloudFormation rolls back your changes by deleting the resources that it created. The below snapshot from Cloudformation shows what happens when there is an error in the stack creation. For more information on how CloudFormation works , please refer to the below link: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-whatis-howdoesitwork.html
Because Elastic Beanstalk performs an in-place update when you update your application versions, your application may become unavailable to users for a short period of time. It is possible to avoid this downtime by performing a blue/green deployment, where you deploy the new version to a separate environment, and then swap CNAMEs of the two environments to redirect traffic to the new version instantly. Blue/green deployments require that your environment runs independently of your production database, if your application uses one. If your environment has an Amazon RDS DB instance attached to it, the data will not transfer over to your second environment, and will be lost if you terminate the original environment. For more information on Blue Green deployments with Elastic beanstalk , please refer to the below link: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features.CNAMESwap.html
Amazon RDS Read Replicas provide enhanced performance and durability for database (DB) instances. This replication feature makes it easy to elastically scale out beyond the capacity constraints of a single DB Instance for read-heavy database workloads. You can create one or more replicas of a given source DB Instance and serve high-volume application read traffic from multiple copies of your data, thereby increasing aggregate read throughput. Read replicas can also be promoted when needed to become standalone DB instances.
Amazon Route 53 health checks monitor the health and performance of your web applications, web servers, and other resources.
If you use SSL termination, your servers will always get non-secure connections and will never know whether users used a more secure channel or not. If you are using Elastic beanstalk to configure the ELB, you can use the below article to ensure end to end encryption. http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/configuring-https-endtoend.html
Amazon Kinesis Firehose is the easiest way to load streaming data into AWS. It can capture, transform, and load streaming data into Amazon Kinesis Analytics, Amazon S3, Amazon Redshift, and Amazon Elasticsearch Service, enabling near real-time analytics with existing business intelligence tools and dashboards you’re already using today. It is a fully managed service that automatically scales to match the throughput of your data and requires no ongoing administration. It can also batch, compress, and encrypt the data before loading it, minimizing the amount of storage used at the destination and increasing security. For more information on Kinesis firehose, please visit the below URL: https://aws.amazon.com/kinesis/firehose/
Amazon Redshift is a fully managed, petabyte-scale data warehouse service in the cloud. You can start with just a few hundred gigabytes of data and scale to a petabyte or more. This enables you to use your data to acquire new insights for your business and customers.
Use Cloudfront distribution for distributing the heavy reads for your application. You can create a zone apex record to point to the Cloudfront distribution. You can control how long your objects stay in a CloudFront cache before CloudFront forwards another request to your origin. Reducing the duration allows you to serve dynamic content. Increasing the duration means your users get better performance because your objects are more likely to be served directly from the edge cache. A longer duration also reduces the load on your origin.
Amazon EBS encryption offers you a simple encryption solution for your EBS volumes without the need for you to build, maintain, and secure your own key management infrastructure. When you create an encrypted EBS volume and attach it to a supported instance type, the following types of data are encrypted: Data at rest inside the volume All data moving between the volume and the instance All snapshots created from the volume Snapshots that are taken from encrypted volumes are automatically encrypted. Volumes that are created from encrypted snapshots are also automatically encrypted.
A tag is a label that you or AWS assigns to an AWS resource. Each tag consists of a key and a value. A key can have more than one value. You can use tags to organize your resources, and cost allocation tags to track your AWS costs on a detailed level. After you activate cost allocation tags, AWS uses the cost allocation tags to organize your resource costs on your cost allocation report, to make it easier for you to categorize and track your AWS costs. AWS provides two types of cost allocation tags, an AWS-generated tag and user-defined tags. AWS defines, creates, and applies the AWS-generated tag for you, and you define, create, and apply user-defined tags. You must activate both types of tags separately before they can appear in Cost Explorer or on a cost allocation report.
You can monitor the progress of a stack update by viewing the stack’s events. The console’s Events tab displays each major step in the creation and update of the stack sorted by the time of each event with latest events on top. The start of the stack update process is marked with an UPDATE_IN_PROGRESS event for the stack For more information on Monitoring your stack, please visit the below URL: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-updating-stacks-monitor-stack.html
A placement group is a logical grouping of instances within a single Availability Zone. Placement groups are recommended for applications that benefit from low network latency, high network throughput, or both. To provide the lowest latency, and the highest packet-per-second network performance for your placement group, choose an instance type that supports enhanced networking. For more information on Placement Groups, please visit the below URL: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/placement-groups.html
AWS CloudTrail is an AWS service that helps you enable governance, compliance, and operational and risk auditing of your AWS account. Actions taken by a user, role, or an AWS service are recorded as events in CloudTrail. Events include actions taken in the AWS Management Console, AWS Command Line Interface, and AWS SDKs and APIs. Visibility into your AWS account activity is a key aspect of security and operational best practices. You can use CloudTrail to view, search, download, archive, analyze, and respond to account activity across your AWS infrastructure. You can identify who or what took which action, what resources were acted upon, when the event occurred, and other details to help you analyze and respond to activity in your AWS account. For more information on Cloudtrail, please visit the below URL: http://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-user-guide.html
Custom resources enable you to write custom provisioning logic in templates that AWS CloudFormation runs anytime you create, update (if you changed the custom resource), or delete stacks. For example, you might want to include resources that aren’t available as AWS CloudFormation resource types. You can include those resources by using custom resources. That way you can still manage all your related resources in a single stack. Use the AWS::CloudFormation::CustomResource or Custom::String resource type to define custom resources in your templates. Custom resources require one property: the service token, which specifies where AWS CloudFormation sends requests to, such as an Amazon SNS topic.
Failover routing lets you route traffic to a resource when the resource is healthy or to a different resource when the first resource is unhealthy. The primary and secondary resource record sets can route traffic to anything from an Amazon S3 bucket that is configured as a website to a complex tree of records. For more information on Route53 Failover Routing, please visit the below URL: http://docs.aws.amazon.com/Route53/latest/DeveloperGuide/routing-policy.html
Deployment Types:
- Single Target Deployment - small dev projects, legacy or non-HA infrastructure; outage occurs in case of failure, testing opportunity is limited.
- All-at-Once Deployment - deployment happens on multiple targets, requires Orchestration tools, suitable for non critical apps in 5-10 range.
- Minimum in-service Deployment - keeps min in-service targets and deploy in multiple stages, suitable for large environments, allow automated testing, no downtime
- Rolling Deployments - x targets per stage, happens in multiple stages, after completion of stage 1, next stage begins, orchestration and health check required, can be least efficient if x is smaller, allow automated testing, no downtime if x is not large to impact application, can be paused, allowing multi-version testing.
- Blue Green Deployment - Deploy to seperate Green environment, update the code on Green, extra cost due to duplicate env during deployment, Deployment is rapid, cutover and migration is clean(DNS Change), Rollback easy(DNS regression), can be fully automates using CFN etc. Binary, No Traffic Split, not used to feature test
- A/B Testing - distribution traffic between blue/green, allows gradual performance/stability/health analysis, allows new feature testing, rollback is quick, end goal of A/B testing is not migration, Uses Route 53 for DNS resolution, 2 records one pointing A, other pointing B, weighted/round robin.
Intrinsic & Conditional Functions
- Intrinsic Fn - inbuilt function provided by AWS to help manage, reference, and condtionally act upon resources, situation & inputs to a stack.
- Fn::Base64 - Base64 encoding for User Data
- Fn::FindInMap - Mapping lookup
- Fn::GetAtt - Advanced reference look up
- Fn::GetAZs - retrieve list of AZs in a region
- Fn::Join - construct complex strings; concatenate strings
- Fn::Select - value selection from list (0, 1)
- Ref - default value of resource
- Conditional Functions - Fn::And, Fn::Equals, Fn::If, Fn::Not, Fn::Or
CFN Resource Deletion Policies
- A policy/setting which is associated with each resource in a template; A way to control what happens to each resource when a stack is deleted.
- Policy value - Delete (Default), Retain, Snapshot
- Delete - Useful for testing environment, CI/CD/QA workflows,
- Presales, Short Lifecycle/Immutable env.
- Retain - live beyond lifcycle of stack; Windows Server Platform (AD), Servers with state, SQL, Exchange, File Servers,
- Non immutable architectures.
- Snapshot - restricted policy type only available for EBS volumes; takes snapshot before deleting for recovering data.
Immutable Architecture - Replace infra instead of upgrading or repairing faulty components, treat servers as unchangeable objects, don't diagnose and fix, throw away and re-create, Nothing bootstraped except AMI.
CFN Stack updates
- stack policy is checked, updates can be prevented; absence of
- stack policy allow all updates; stack policy cannot be deleted once applied. Once stack policy applied ALL objects are protected, Update is denied; to remove default DENY, explicit allow is required; can be applied to a single resource(id)/Wild card/NotResource; Has Principal and Action; Condition element (resource type) can also be used.
Stack updates: 4 Types - Update with No Interrupion, Some Interruption, Replacement, Delete

Sagemaker Model deployment and Integration

Amit Kayal — Wed, 18 Jan 2023 07:29:11 +0000

Sagemaker Model deployment and Integration

[TOC]

AWS Feature store

SageMaker Feature Store is a purpose-built solution for ML feature management. It helps data science teams reuse ML features across teams and models, serve features for model predictions at scale with low latency, and train and deploy new models more quickly and effectively.

Refer the notebook https://github.com/aws-samples/ml-lineage-helper/blob/main/examples/example.ipynb for more details.

Why is feature lineage important?

Imagine trying to manually track all of this for a large team, multiple teams, or even multiple business units. Lineage tracking and querying helps make this more manageable and helps organizations move to ML at scale. The following are four examples of how feature lineage helps scale the ML process:

Build confidence for reuse of existing features
Avoid reinventing features that are based on the same raw data as existing features
Troubleshoot and audit models and model predictions
Manage features proactively

AWS ML Lens and built-in models

Deployment Options

ML inference can be done in real time on individual records, such as with a REST API endpoint. Inference can also be done in batch mode as a processing job on a large dataset. While both approaches push data through a model, each has its own target goal when running inference at scale.

* Real Time*	Micro Batch	Batch
Execution Mode	Synchronous	Synchronous/Asynchronous	Asynchronous
Prediction Latency	Subsecond	Seconds to minutes	Indefinite
Data Bounds	Unbounded/stream	Bounded	Bounded
Execution Frequency	Variable	Variable	Variable/fixed
Invocation Mode	Continuous stream/API calls	Event-based	Event-based/scheduled
Examples	Real-time REST API endpoint	Data analyst running a SQL UDF	Scheduled inference job

Realtime deployment

Sagemaker real-time deployment has the following approach. Key point here is that we can have our inference pipeline coupled with autoscale.

Here are different ways, we can deploy real-time endpoint by sagemaker. You can see here multiple options from own model, own container to prebuilt container.

With sagemaker, prebuilt container and its own inference script, we can use this as shared below.

Quite a lot of time, we add our own inference script and this is quite simple as shown below.

It is not rare to have our own container and own trained model along with inference script. The architecture does not change for that and we still follow same architecture as shared below.

Autoscale

we can set autoscale policy for sagemaker endpoint to scale up and scale down automatically.

We have to set autoscale policy setup for endpoint. You can see here that ServiceNamespace is set to sgaemaker and resourceId is set to Endpoint name.

Multi Modal endpoint

SageMaker multi-model endpoints work with several frameworks, such as TensorFlow, PyTorch, MXNet, and sklearn, and you can build your own container with a multi-model server. Multi-model endpoints are also supported natively in the following popular SageMaker built-in algorithms: XGBoost, Linear Learner, Random Cut Forest (RCF), and K-Nearest Neighbors (KNN).

Refer the notebook https://github.com/aws-samples/sagemaker-multi-model-endpoint-tensorflow-computer-vision/blob/main/multi-model-endpoint-tensorflow-cv.ipynb to understand how we can deploy this/. Refer the blog https://aws.amazon.com/blogs/machine-learning/save-on-inference-costs-by-using-amazon-sagemaker-multi-model-endpoints/

All of the models that are hosted on a multi-modal endpoint must share the same serving container image.
Multi-model endpoints are an option that can improve endpoint utilization when your models are of similar size and share the same container image and have similar invocation latency requirements.
all the model needs to share same S3 bucket to host their weights

Cost advantages

This diagram demonstrates running 10 models on a multi-model endpoint versus using 10 separate endpoints. This results in savings of $3,000 per month, as shown in the following figure: Multi-model endpoints can easily scale to hundreds or thousands of models.

How to use?

To create a multi-model endpoint in Amazon SageMaker, choose the multi-model option, provide the inference serving container image path, and provide the Amazon S3 prefix in which the trained model artifacts are stored. You can organize your models in S3 any way you wish, so long as they all use the same prefix.

When you invoke the multi-model endpoint, you provide the relative path of a specific model with the new TargetModel parameter of InvokeEndpoint. To add models to the multi-model endpoint, simply store a newly trained model artifact in S3 under the prefix associated with the endpoint. The model will then be immediately available for invocations.

To update a model already in use, add the model to S3 with a new name and begin invoking the endpoint with the new model name. To stop using a model deployed on a multi-model endpoint, stop invoking the model and delete it from S3.

Instead of downloading all the models into the container from S3 when the endpoint is created, Amazon SageMaker multi-model endpoints dynamically load models from S3 when invoked. As a result, an initial invocation to a model might see higher inference latency than the subsequent inferences, which are completed with low latency. If the model is already loaded on the container when invoked, then the download step is skipped and the model returns the inferences with low latency.

Monitoring multi-model endpoints using Amazon CloudWatch metrics

To make price and performance tradeoffs, you will want to test multi-model endpoints with models and representative traffic from your own application. Amazon SageMaker provides additional metrics in CloudWatch for multi-model endpoints so you can determine the endpoint usage and the cache hit rate and optimize your endpoint. The metrics are as follows:

ModelLoadingWaitTime – The interval of time that an invocation request waits for the target model to be downloaded or loaded to perform the inference.
ModelUnloadingTime – The interval of time that it takes to unload the model through the container’s UnloadModel API call.
ModelDownloadingTime – The interval of time that it takes to download the model from S3.
ModelLoadingTime – The interval of time that it takes to load the model through the container’s LoadModel API call.
ModelCacheHit – The number of InvokeEndpoint requests sent to the endpoint where the model was already loaded. Taking the Average statistic shows the ratio of requests in which the model was already loaded.
LoadedModelCount – The number of models loaded in the containers in the endpoint. This metric is emitted per instance. The Average statistic with a period of 1 minute tells you the average number of models loaded per instance, and the Sum statistic tells you the total number of models loaded across all instances in the endpoint. The models that this metric tracks are not necessarily unique because you can load a model in multiple containers in the endpoint.

You can use CloudWatch charts to help make ongoing decisions on the optimal choice of instance type, instance count, and number of models that a given endpoint should host.

Inference Pipeline sagemaker

You can use trained models in an inference pipeline to make real-time predictions directly without performing external preprocessing. When you configure the pipeline, you can choose to use the built-in feature transformers already available in Amazon SageMaker. Or, you can implement your own transformation logic using just a few lines of scikit-learn or Spark code.

Refer https://sagemaker-examples.readthedocs.io/en/latest/sagemaker-python-sdk/scikit_learn_inference_pipeline/Inference%20Pipeline%20with%20Scikit-learn%20and%20Linear%20Learner.html / https://catalog.us-east-1.prod.workshops.aws/workshops/f238037c-8f0b-446e-9c15-ebcc4908901a/en-US/002-services/003-machine-learning/020-sagemaker for more details.

Inference pipeline allows you to host multiple models behind a single endpoint. But in this case, the models are sequential chain of models with the steps that are required for inference. This allows you to take your data transformation model, your predictor model, and your post-processing transformer, and host them so they can be sequentially run behind a single endpoint.
As you can see in this picture, the inference request comes into the endpoint, then the first model is invoked, and that model is your data transformation. The output of that model is then passed to the next step, which is actually your XGBoost model here, or your predictor model.
- That output is then passed to the next step, where ultimately in that final step in the pipeline, it provides the final response or the post-process response to that inference request.
- This allows you to couple your pre and post-processing code behind the same endpoint and helps ensure that your training and your inference code stay synchronized

Sagemaker Production Variant

Amazon SageMaker enables you to test multiple models or model versions behind the same endpoint using production variants. Each production variant identifies a machine learning (ML) model and the resources deployed for hosting the model. By using production variants, you can test ML models that have been trained using different datasets, trained using different algorithms and ML frameworks, or are deployed to different instance type, or any combination of all of these. You can distribute endpoint invocation requests across multiple production variants by providing the traffic distribution for each variant, or you can invoke a specific variant directly for each request. In this topic, we look at both methods for testing ML models.

Refer the notebook https://sagemaker-examples.readthedocs.io/en/latest/sagemaker_endpoints/a_b_testing/a_b_testing.html for implementation details.

Test models by specifying traffic distribution

Specify the percentage of the traffic that gets routed to each model by specifying the weight for each production variant in the endpoint configuration.

Test models by invoking specific variants

Specify the specific version of the model you want to invoke by providing a value for the TargetVariant parameter when you call InvokeEndpoint.

Amazon SageMaker Batch Transform: Batch Inference

We’ll use the Sagemaker Batch Transform Jobs and a trained machine learning model. It is assumed that we have already trained the model, pushed the Docker image to ECR, and registered the model in Sagemaker.

we need the identifier of the Sagemaker model we want to use and the location of the input data
either use a built-in container for your inference image or you can also bring your own.
Batch Transform partitions the Amazon S3 objects in the input by key and maps Amazon S3 objects to instances. When you have multiples files, one instance might process input1. csv , and another instance might process the file named input2. csv

In Batch Transform you provide your inference data as a S3 uri and SageMaker will care of downloading it, running the prediction and uploading the results afterwards to S3 again. You can find more documentation for Batch Transform here

If you trained a model using the Hugging Face Estimator, call the transformer() method to create a transform job for a model based on the training job (see here for more details): Refer https://huggingface.co/docs/sagemaker/inference

Note:

batch job has

instance count
instance type

transform job has

data location
content type

batch_job = huggingface_estimator.transformer(
    instance_count=1,
    instance_type='ml.p3.2xlarge',
    strategy='SingleRecord')


batch_job.transform(
    data='s3://s3-uri-to-batch-data',
    content_type='application/json',    
    split_type='Line')

If you want to run your batch transform job later or with a model from the 🤗 Hub, create a HuggingFaceModel instance and then call the transformer() method:

from sagemaker.huggingface.model import HuggingFaceModel

# Hub model configuration <https://huggingface.co/models>
hub = {
    'HF_MODEL_ID':'distilbert-base-uncased-finetuned-sst-2-english',
    'HF_TASK':'text-classification'
}

# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
   env=hub,                                                # configuration for loading model from Hub
   role=role,                                              # IAM role with permissions to create an endpoint
   transformers_version="4.6",                             # Transformers version used
   pytorch_version="1.7",                                  # PyTorch version used
   py_version='py36',                                      # Python version used
)

# create transformer to run a batch job
batch_job = huggingface_model.transformer(
    instance_count=1,
    instance_type='ml.p3.2xlarge',
    output_path=output_s3_path, # we are using the same s3 path to save the output with the input
    strategy='SingleRecord'
)

# starts batch transform job and uses S3 data as input
batch_job.transform(
    data='s3://sagemaker-s3-demo-test/samples/input.jsonl',
    content_type='application/json',    
    split_type='Line'
)

The input.jsonl looks like this:

import json
from sagemaker.s3 import S3Downloader
from ast import literal_eval
# creating s3 uri for result file -> input file + .out
output_file = f"{dataset_jsonl_file}.out"
output_path = s3_path_join(output_s3_path,output_file)

# download file
S3Downloader.download(output_path,'.')

batch_transform_result = []
with open(output_file) as f:
    for line in f:
        # converts jsonline array to normal array
        line = "[" + line.replace("[","").replace("]",",") + "]"
        batch_transform_result = literal_eval(line) 

# print results 
print(batch_transform_result[:3])

{"inputs":"this movie is terrible"}
{"inputs":"this movie is amazing"}
{"inputs":"SageMaker is pretty cool"}
{"inputs":"SageMaker is pretty cool"}
{"inputs":"this movie is terrible"}
{"inputs":"this movie is amazing"}

📓 Open the notebook for an example of how to run a batch transform job for inference.

Speeding up the processing

We have only one instance running, so processing the entire file may take some time. We can increase the number of instances using the instance_count parameter to speed it up. We can send multiple requests to the Docker container simultaneously, too. The configure concurrent transformations we must use the max_concurrent_transforms parameter.

Processing the output

In the end, we must get access to the output. We’ll find the output files in the location specified in the Transformer constructor. Every line contains the prediction and the input parameters. agemaker-notebook.ipynb) for an example of how to run a batch transform job for inference.

My journey into Sentence Transformer

Amit Kayal — Sun, 16 Oct 2022 16:49:41 +0000

Last few days I have been exploring sentence transformer and this page documents my notes/understanding. This note explains the basic of sentence transformer and deployment the same through sagemaker endopoint. The endpoint is then accessed from lambda. Terraform has been considered here for deployment.

What is Sentence Embedding?

I came across a nice example posted by Mathias about sentence comparison. Consider the following statements: “Nuclear power is dangerous!” and “Nuclear power is the future of energy!”

If we are talking about the topic, then definitely yes: both statements are opinions on nuclear power. So in that sense, they are very similar.
However, if we are talking about sentiment, then the answer is a resounding no. They are about as dissimilar in terms of sentiment as we can get.

What is Transformer and BERT?

BART is a denoising autoencoder for pretraining sequence-to-sequence models. It is trained by (1) corrupting text with an arbitrary noising function, and (2) learning a model to reconstruct the original text. It uses a standard Transformer-based neural machine translation architecture. It uses a standard seq2seq/NMT architecture with a bidirectional encoder (like BERT) and a left-to-right decoder (like GPT). This means the encoder's attention mask is fully visible, like BERT, and the decoder's attention mask is causal, like GPT2.

While BERT was trained by using a simple token masking technique, BART empowers the BERT encoder by using more challenging kinds of masking mechanisms in its pre-training. Once we get the token and sentence-level representation of an input text sequence, a decoder needs to interpret these to map with the output target.

Building task specific Transformer based solution

Green portion is pretrained one
other portion (purple) is custom head which we train further
QA head is predicting span of text which contains answer from context/text
Classification Head predicting binary value.

BERT for Sentence Similarity

Transformers work using word or token-level embeddings, not sentence-level embeddings.

Regular transformers produce sentence embeddings by performing some pooling operation such as the element-wise arithmetic mean on its token-level embeddings. A good pooling choice for BERT is CLS pooling. BERT has a special <CLS> token that is supposed to capture all the sequence information. It gets tuned on next-sentence prediction (NSP) during pre-training.

Before sentence transformers, the approach to calculating accurate sentence similarity with BERT was to use a cross-encoder structure. This meant that we would pass two sentences to BERT, add a classification head to the top of BERT — and use this to output a similarity score.

The BERT cross-encoder architecture consists of a BERT model which consumes sentences A and B. Both are processed in the same sequence, separated by a [SEP] token. All of this is followed by a feedforward NN classifier that outputs a similarity score.
The cross-encoder network does produce very accurate similarity scores (better than SBERT), but it’s not scalable. If we wanted to perform a similarity search through a small 100K sentence dataset, we would need to complete the cross-encoder inference computation 100K times.
To cluster sentences, we would need to compare all sentences in our 100K dataset, resulting in just under 500M comparisons — this is simply not realistic.
Ideally, we need to pre-compute sentence vectors that can be stored and then used whenever required. If these vector representations are good, all we need to do is calculate the cosine similarity between each.
With the original BERT (and other transformers), we can build a sentence embedding by averaging the values across all token embeddings output by BERT (if we input 512 tokens, we output 512 embeddings). Alternatively, we can use the output of the first [CLS] token (a BERT-specific token whose output embedding is used in classification tasks).
Using one of these two approaches gives us our sentence embeddings that can be stored and compared much faster, shifting search times from 65 hours to around 5 seconds (see below). However, the accuracy is not good, and is worse than using averaged GloVe embeddings (which were developed in 2014).

Sentence Transformer?

How Does it Work?

SentenceTransformers is a Python framework for state-of-the-art sentence, text, and image embeddings. Embeddings can be computed for 100+ languages and they can be easily used for common tasks like semantic text similarity, semantic search, and paraphrase mining.

The solution of the above lack of an accurate model with reasonable latency was designed by Nils Reimers and Iryna Gurevych in 2019 with the introduction of sentence-BERT (SBERT) and the sentence-transformers library.

SBERT produces sentence embeddings — so we do not need to perform a whole inference computation for every sentence-pair comparison.

SBERT is fine-tuned on sentence pairs using a siamese architecture. We can think of this as having two identical BERTs in parallel that share the exact same network weights.

In reality, we are using a single BERT model. However, because we process sentence A followed by sentence B as pairs during training, it is easier to think of this as two models with tied weights.

Use cases

Sentence Transformer Architecture Changes

SBERT uses a siamese architecture where it contains 2 BERT architectures that are essentially identical and share the same weights, and SBERT processes 2 sentences as pairs during training.

The training process of sentence transformers is especially designed with semantic similarity in mind.

Cross-encoders

A cross-encoder is thus trained by sentence-pairs along with a ground-truth label of how semantically similar they are.

Cross-encoders tend to perform very well on sentence-level tasks, they do suffer from a major drawback: cross-encoders do not produce sentence embeddings. **In the context of information retrieval, this implies that we **cannot pre-compute document embeddings and efficiently compare these to a query embedding. We are also not able to index document embeddings for efficient search.

Bi-encoders

In the context of information retrieval, this implies that we cannot pre-compute document embeddings and efficiently compare these to a query embedding. We are also not able to index document embeddings for efficient search.

we feed sentence A to BERT A and sentence B to BERT B in SBERT. Each BERT outputs pooled sentence embeddings. While the original research paper tried several pooling methods, they found mean-pooling was the best approach. Pooling is a technique for generalizing features in a network, and in this case, mean pooling works by averaging groups of features in the BERT.
After the pooling is done, we now have 2 embeddings: 1 for sentence A and 1 for sentence B. When the model is training, SBERT concatenates the 2 embeddings which will then run through a softmax classifier and be trained using a softmax-loss function.
At inference — or when the model actually begins predicting — the two embeddings are then compared using a cosine similarity function, which will output a similarity score for the two sentences. Here is a diagram for SBERT when it is fine-tuned and at inference.

How can you use SBERT for sagemaker endpoint?

BERT has its own Python library. Using it is as simple as using a model from the hugging face transformer library. Here, we have used multi-qa-MiniLM-L6-cos-v1 model for sentence similarity.

Here, I have shown how we can deploy this model as our sagemaker serverless endpoint.

# Choose transformer model for embeddings
from transformers import AutoTokenizer, AutoModel
import os
import sagemaker
import time
saved_model_dir = 'transformer'
os.makedirs(saved_model_dir, exist_ok=True)

tokenizer = AutoTokenizer.from_pretrained("sentence-transformers/multi-qa-MiniLM-L6-cos-v1")
model = AutoModel.from_pretrained("sentence-transformers/multi-qa-MiniLM-L6-cos-v1") 

tokenizer.save_pretrained(saved_model_dir)
model.save_pretrained(saved_model_dir)

#Defining default bucket for SageMaker pretrained model hosting
sagemaker_session = sagemaker.Session()
role = sagemaker.get_execution_role()

!cd transformer && tar czvf ../model.tar.gz *

model_data = sagemaker_session.upload_data(path='model.tar.gz', key_prefix='autofaiss-demo/huggingface-models')

from sagemaker.huggingface.model import HuggingFaceModel


# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
   model_data=model_data,       # path to your model and script
   entry_point = 'predict.py',
   source_dir = 'source_dir',
   role=role,                    # iam role with permissions to create an Endpoint
   transformers_version="4.12",  # transformers version used
   pytorch_version="1.9",        # pytorch version used
   py_version='py38',            # python version used
)

# deploy the endpoint endpoint
predictor = huggingface_model.deploy(
    initial_instance_count=1,
    instance_type="ml.m5.2xlarge"
    )

Here, I have created a lambda_handler function where the above endpoint is being called for similarity prediction.

import logging
import json
import boto3
import io
import os
import time
import logging
import sagemaker
from sagemaker.deserializers import JSONDeserializer
from sagemaker.serializers import IdentitySerializer

logger = logging.getLogger()
logger.setLevel(logging.INFO)
ENDPOINT_NAME = "huggingface-pytorch-inference-2022-10-14-20-02-16-258"
sagemaker_session = sagemaker.Session()

"""
FunctionName: invoke_endpoint
Input: transcript_item (sentence), label_map
    transcript_item type: string
    label_map type: dict
Output: Question
    type: string
"""
# @tracer.capture_method
def invoke_endpoint(payload, endpoint_name):
    runtime = boto3.client('runtime.sagemaker')
    response = runtime.invoke_endpoint(EndpointName=endpoint_name,
                                      ContentType="application/json",
                                      Body=json.dumps(payload))
    embeddings = json.loads((response["Body"].read()))
    return embeddings

# @tracer.capture_lambda_handler
def lambda_handler(event):
    start = time.time()
    similarity_scores = invoke_endpoint(event, ENDPOINT_NAME)
    end = time.time()
    logger.info(f"Profiling: \n Getting Embeddings: {1000*(end-start)} milliseconds")   
    return similarity_scores


json_event = {  
    "query_from_app" : "How many people live in London?",
    "actual_queries" : ["Around 9 Million people live in London", "London is known for its financial district"]
}

lambda_handler(json_event)

The sample output is as shown below...

{'Scores': [['Around 9 Million people live in London', 0.9156370759010315],
  ['London is known for its financial district', 0.49475768208503723]]}

Serverless deployment of Sentence Transformer

I have shared below our lambda code and terraform code for deployment.

Lambda Code

I have used terraform to deploy the lambda function and the endpoint is being defined here as environment variable.

from query_request import query
DDB_ENDPOINT = os.environ["ServiceConfiguration__DDB_ENDPOINT"]
REGION = os.environ["ServiceConfiguration__REGION"]

def invoke_endpoint(payload, endpoint_name):
    runtime = boto3.client('runtime.sagemaker')
    response = runtime.invoke_endpoint(EndpointName=endpoint_name,
                                      ContentType="application/json",
                                      Body=json.dumps(payload))
    embeddings = json.loads((response["Body"].read()))
    return embeddings

def lambda_handler(event, context):
    start = time.time()
    similarity_scores = invoke_endpoint(event, ENDPOINT_NAME)
    end = time.time()
    logger.info(f"Similarity scores: {similarity_scores}")
    logger.info(f"Profiling: \n Getting Embeddings: {1000*(end-start)} milliseconds")   
    return similarity_scores

Terraform

module "questn_similarity_classification" {
  source  = "terraform-module/lambda/aws"
  version = "2.12.6"

  function_name = "questn_similarity_classification"
  filename      = data.archive_file.questn_similarity_classification-zip.output_path
  source_code_hash = data.archive_file.questn_similarity_classification-zip.output_base64sha256
  description      = "questn_similarity_classification"
  handler        = "questn_similarity_classification.lambda_handler"
  runtime        = "python3.7"
  memory_size    = "1280"
  concurrency    = "25"
  lambda_timeout = "120"
  log_retention  = "30"
  publish        = true
  role_arn       = aws_iam_role.questn_similarity_classification_role.arn
  tracing_config = { mode = "Active" }
 # layers = [aws_lambda_layer_version.numpy_layer_37.arn, data.aws_lambda_layer_version.ml_faiss_layer_version.arn]

  vpc_config = {
    subnet_ids         = tolist(data.aws_subnet.efs_subnet.*.id)
    security_group_ids = [data.aws_security_group.default_sec_grp.id]
  }
  environment = {
    ServiceConfiguration__ENDPOINT_NAME    = var.ServiceConfiguration__ENDPOINT_NAME
    ServiceConfiguration__REGION = var.ServiceConfiguration__REGION
  }
  file_system_config = {
    # file_system_arn              = data.aws_efs_access_point.knn_efs.arn
    efs_access_point_arn = data.aws_efs_access_point.knn_efs.arn
    local_mount_path     = var.file_system_local_mount_path # Local mount path inside the lambda function. Must start with '/mnt/'.
    # file_system_local_mount_path = var.file_system_local_mount_path # Local mount path inside the lambda function. Must start with '/mnt/'.

  }

A learning journey with ELK stack

Amit Kayal — Wed, 01 Dec 2021 17:15:27 +0000

Kibana and Elastic Search

[TOC]

What is Kibana?

Kibana is a data visualization and exploration tool used for log and time-series analytics, application monitoring, and operational intelligence use cases. It offers powerful and easy-to-use features such as histograms, line graphs, pie charts, heat maps, and built-in geospatial support. Also, it provides tight integration with Elasticsearch, a popular analytics and search engine, which makes Kibana the default choice for visualizing data stored in Elasticsearch.Kibana works in sync with Elasticsearch and Logstash which together forms the so called ELK stack.

What is ELK Stack?

ELK stands for Elasticsearch, Logstash, and Kibana. ELK is one of the popular log management platform used worldwide for log analysis. In the ELK stack, Logstash extracts the logging data or other events from different input sources. It processes the events and later stores them in Elasticsearch.

Kibana is a visualization tool, which accesses the logs from Elasticsearch and is able to display to the user in the form of line graph, bar graph, pie charts etc. The basic flow of ELK Stack is shown in the image here −

Logstash is responsible to collect the data from all the remote sources where the logs are filed and pushes the same to Elasticsearch. Elasticsearch acts as a database where the data is collected and Kibana uses the data from Elasticsearch to represent the data to the user in the form of bargraphs, pie charts, heat maps as shown below −

For a small-sized development environment, the classic architecture will look as follows:

However, for handling more complex pipelines built for handling large amounts of data in production, additional components are likely to be added into your logging architecture, for resiliency (Kafka, RabbitMQ, Redis) and security (nginx):

This is of course a simplified diagram for the sake of illustration

What is Elasticsearch

It is distributed document stores which means once the document is stored then it can be retrieved from any node of the cluster.

default port for elasticsearch is 9201
default port for kibana is 5600

It shows the data on real time basis, for example, day-wise or hourly to the user. Kibana UI is user friendly and very easy for a beginner to understand. The following table gives a direct comparison between these terms−

Every row in RDBMS has an unique row identifier and similarly we have unique document id in elasticsearch for every document.

Elastichsarch built on top of Lucene. Every shard is simply a Lucene index. Lucene index, if simplified, is the inverted index. Every Elasticsearch index is a bunch of shards or Lucene indices. When you query for a document, Elasticsearch will subquery all shards, merge results and return it to you. When you index document to Elasticsearch, the Elasticsearch will calculate in which shard document should be written using the formula

shard = hash(routing) % number_of_primary_shards

Key Concepts

In Elasticsearch terms, Index = Database, Type = Table, Document = Row.

The key concepts of Elasticsearch are as follow

Node
- It refers to a single running instance of Elasticsearch.
Cluster
- It is a collection of one or more nodes. Cluster provides collective indexing and search capabilities across all the nodes for entire data.
Index
- It is a collection of different type of documents and their properties. Index also uses the concept of shards to improve the performance
Document
- It is a collection of fields in a specific manner defined in JSON format. Every document belongs to a type and resides inside an index. Every document is associated with a unique identifier called the UID.
Shard
- Indexes are horizontally subdivided into shards. This means each shard contains all the properties of document but contains less number of JSON objects than index. The horizontal separation makes shard an independent node, which can be store in any node.
Replicas
- Elasticsearch allows a user to create replicas of their indexes and shards.

Checking that Elasticsearch is running

You can test that your Elasticsearch node is running by sending an HTTP request to port 9200 on localhost:

curl -X GET "localhost:9200/?pretty"

response will be as similar to following.

{
  "name" : "BLR-AA202394",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "pYm2eSPlTh-EUo8hcVce2A",
  "version" : {
    "number" : "7.15.2",
    "build_flavor" : "default",
    "build_type" : "zip",
    "build_hash" : "93d5a7f6192e8a1a12e154a2b81bf6fa7309da0c",
    "build_date" : "2021-11-04T14:04:42.515624022Z",
    "build_snapshot" : false,
    "lucene_version" : "8.9.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

Rest API in EL

Every feature of Elasticsearch is exposed as a REST API:

Index API – Used to document the Index
Get API – Used to retrieve the document
Search API – Used to submit your query and get the result
Put Mapping API – Used to override default choices and define our own mapping

Elasticsearch has its own Query Domain Specific Language, where you specify the query in JSON format. Other queries can also be nested based on your need. Real projects require search on different fields by applying some conditions, different weights, recent documents, values of some predefined fields, and so on. All such complexity can be expressed through a single query.

Document analysis process by Elasticsearch

When a document request for indexing is received by elasticsearch, which in turn is handled by lucene, it converts the document in a stream of tokens. After tokens are generated, the same gets filtered by the configured filter. This entire process is called the analysis process, and is applied on every document that gets indexed.

One thing to learn from this is that the key to an efficient storage and retrieval process is the analysis process defined on the index, as per the application needs.

Sometime we also do

stemming and this means we convert the word into root word and store in index (Ex: such words are population, populations)
For synonym, we store single word rather than storing all the words (declined, reduced)

Remember that this same process of document normalization (during inverted index ) needs to be applied during document search query.

The analyzer can be custom or default one and this needs to be specified during Index creation. If the analyzer is not provided then EL uses default standard one.

Remember that, EL standard analyzer will not do do

stemming (Running to Run)- process of converting word to root word
stop word removal

Inverted Index

It uses a method called inverted index. Since we are talking about searching a term in a large collection of documents(aka collection of chapters in this case) we can use Inverted Indexes to solve this issue, and yes almost all books use these Inverted Indexes to make your life easier. Just like many other books "Team of Rivals" has inverted indexes at the end of the book as shown in this image.

An inverted index consists of a list of all the unique words that appear in any document, and for each word, a list of the documents in which it appears.

So after checking the Inverted indexes at the end of the book we know that "Baltimore" is mentioned on pages 629 and 630. So there are two parts in this searching for "Baltimore" in the lexicographically ordered Inverted Index list and fetching the pages based on the value of the index (here 629 and 630). The search time is very less for the term in the inverted index since in computing we actually use dictionaries(hash-based or search trees) to keep track of these terms

The purpose of an inverted index, is to store text in a structure that allows for very efficient and fast full-text searches. When performing full-text searches, we are actually querying an inverted index and not the JSON documents that we defined when indexing the documents.

An inverted index consists of all of the unique terms that appear in any document covered by the index.

Here is the diagram which shows a very simplified structure of an inverted index. Here we have separated stop words which are common ones and they also should be excluded from queries.

Post the analysis process, when the data is converted into tokens, these tokens are stored into an internal structure called inverted index. This structure maps each unique term in an index to a document. This data structure allows for faster data search and text analytics. All the attributes like term count, term position and other such attributes are associated with the term. Below is a sample visualization of how an inverted index may look like.

Post the tokens are mapped, document is stored on the disk. One can choose to store the original input of the document along with the analyzed document. The original input gets stored in a system field names "_source". Once can even choose to not analyze the input, and store the document without any analysis. The structure of the inverted index totally depends upon the analyzer chosen for indexing.

Another example:

Document 1: Elasticsearch is fast

Document 2: I want to learn Elasticsearch

Let’s take a peek into the Inverted Index and see the result of the Analysis and Indexing process:

As you can see, the terms are counted and mapped into document identifiers and its position in the document. The reason we don’t see the full document Elasticsearch is fast or I want to learn Elasticsearch is because they go through Analysis process, which is our main topic in this article.

Operation with Elasticsearch

Inserting a document in elasticsearch means Indexing a document

When you execute the PUT command for the fresh document, Then the document will be created. Such a case result attribute from the response will incite as “created”. When you execute the PUT command for the
second time for the same document, it will not create the new document again if the document already exists in the cluster. Instead, it will update the document. You can confirm that by the result from the
response. The result will indicate as “updated”. But when you execute the PUT command for the fresh document, Then the document will be created. If it's already available, then the document will be updated.

What is indexing

Index as a verb

The process of inserting a document into elasticsearch is called indexing. Indexing a document into elasticsearch is similar to insert the row in a database table. The only difference is, if the document already exists in the cluster, the indexing process will replace the old document.

Indexing a document from Kibana

Before indexing a document into kibana, first we need to decide where this document will be stored. In elasticsearch, a document belongs to a type and the type stays within an Index. An elasticearch cluster may contain multiple indexes and V7 allows only one type for per index. One index which belong to one type, may contain multiple documents and each document may have multiple fields.

Base command for an indexing a document is

{{host}}/:index/_alias/:alias

Here is a sample indexing which will create the index and have document added. I have also added following index pattern into elasticsearch.yml to enable auto indexing.

action.auto_create_index: .monitoring*,.watches,.triggered_watches,.watcher-history*,.ml*,.doc*,.json*,_json*,_doc*,*subscriber*,*product*,*event*

Note: Passing _doctype value (here it is json) is deprecated into current V7.

Indexing request:

Here intentionally i have used PUT request as this one requires ID value to be provided. My aim is here is to search by subscriberId and hence I am passing the ID value. Otherwise we can use POST command like “ POST /subscriber/json/” which will be auto generate id.

PUT /subscriber/json/5159683
{
    "Addresses": [
        {
            "City": "TX",
            "Country": "USA",
            "LineOne": "Parkway Suite Houston",
            "PostalCode": "77047",
            "State": "TX",
            "DefaultBilling": true,
            "DefaultHome": false,
            "DefaultPostal": true,
            "DefaultService": true,
            "DefaultShipping": false,
            "DefaultWork": false,
            "Id": 1412213,
            "Created": "2021-11-22T16:05:40.943Z",
            "Modified": "2021-11-22T16:05:40.943Z",
            "Name": "Parkway E Suite Houston",
            "ShipToName": "India",
            "Status": 1,
            "StatusName": "Active",
            "TaxPCode": null,
            "Verifiable": true,
            "Verified": true
        }
    ],
    "Subscriber": {
        "Category": 1,
        "CompanyName": "IH-163759714",
        "ConvergentBillerId": "40003771",
        "Created": "2021-11-22T16:05:40.890Z",
        "ExternalReference": "IH-EX163759714",
        "Id": 5159683,
        "Language": "en-GB",
        "Login": "aa@mallinator.com",
        "State": 0,
        "StateChangeDate": "2021-11-22T16:05:40.840Z",
        "Status": 1,
        "SubscriberTypeCode": 10121,
        "AdditionalProperties": [
            {
                "ExternalReference": "Invoice_Ref",
                "Id": 1886,
                "Name": "CustomerInvoiceRef",
                "Values": []
            },
            {
                "ExternalReference": "Bill_From_Address",
                "Id": 1888,
                "Name": "BillFromAddress",
                "Values": [
                    "add-sws"
                ]
            }
        ],
        "ContactPreferences": [
            {
                "ContactEventType": 12,
                "ContactMethod": 40,
                "Id": 2702216,
                "OptIn": true
            }
        ],
        "EffectiveStartDate": "2021-11-22T16:05:40.863Z",
        "HomeCountry": "USA",
        "InvoiceConfiguration": {
            "HideZeroAmount": false,
            "InvoiceDetailLevel": 1
        },
        "StateName": "Prospect",
        "StatusName": "Active",
        "SubscriberCurrency": "USD",
        "SubscriberTypeDetails": {
            "AccountingMethod": 2,
            "BillCycle": "40002163",
            "BillCycleDay": 1,
            "BillCycleName": "BillCycle_1",
            "IsReadOnly": false,
            "PaymentDueDays": 10,
            "PostpaidAccountNumber": "CAQ400037719"
        },
        "TermsAndConditionsAccepted": "2021-11-22T16:05:40.883Z"
    }
}

Indexing response:

here the type has been set with our input value
index has been set to our input subscriber
result value has set as created
sequence no has ben set as 3 and this is being set by elasticsearch

#! [types removal] Specifying types in document index requests is deprecated, use the typeless endpoints instead (/{index}/_doc/{id}, /{index}/_doc, or /{index}/_create/{id}).
{
  "_index" : "subscriber",
  "_type" : "json",
  "_id" : "5159684",
  "_version" : 1,
  "result" : "created",
  "_shards" : {
    "total" : 2,
    "successful" : 1,
    "failed" : 0
  },
  "_seq_no" : 3,
  "_primary_term" : 1
}

Here is the request when i passed on the input for an existing id and result value shows as updated which means the indexing has been updated. version has been set to 3 and this means the document has been updated 3 times. Every document in elasticsearch has version. So PUT request for existing document will update the version and for new request it will create

The _index mentions the index name for this document.

Any attribute starts with _ with document are called document metadata.

#! [types removal] Specifying types in document index requests is deprecated, use the typeless endpoints instead (/{index}/_doc/{id}, /{index}/_doc, or /{index}/_create/{id}).
{
  "_index" : "subscriber",
  "_type" : "json",
  "_id" : "5159683",
  "_version" : 3,
  "result" : "updated",
  "_shards" : {
    "total" : 2,
    "successful" : 1,
    "failed" : 0
  },
  "_seq_no" : 2,
  "_primary_term" : 1
}

Finding if a document available?

The HEAD command is being used to see if the document is available and this will respond us http 200 as shared below.

HEAD /subscriber/_doc/5159685

200 - OK

Retrieve an Index document

Here we are passing the Index name, document type and Id to retrieve the index.

GET /subscriber/json/5159685

and response will have the document along with its metadata. The found has value set to true which means the document has been found.

{
  "_index" : "subscriber",
  "_type" : "json",
  "_id" : "5159685",
  "_version" : 1,
  "_seq_no" : 7,
  "_primary_term" : 1,
  "found" : true,
  "_source" : {
    "Addresses" : [
      {
        "City" : "TX",
        "Country" : "USA",
        "LineOne" : "Parkway E Suite Houston",
        "PostalCode" : "77047",
        "State" : "TX",
        "DefaultBilling" : true,
        "DefaultHome" : false,
        "DefaultPostal" : true,
        "DefaultService" : true,
        "DefaultShipping" : false,
        "DefaultWork" : false,
        "Id" : 1412213,
        "Created" : "2021-11-22T16:05:40.943Z",
        "Modified" : "2021-11-22T16:05:40.943Z",
        "Name" : "Parkway E Suite 100 Houston",
        "ShipToName" : "MARLINK GROUP USA - HOUSTON",
        "Status" : 1,
        "StatusName" : "Active",
        "TaxPCode" : null,
        "Verifiable" : true,
        "Verified" : true
      }
    ],
    "Subscriber" : {
      "Category" : 1,
      "CompanyName" : "IH1-163759714",
      "ConvergentBillerId" : "400033771",
      "Created" : "2021-11-22T16:05:40.890Z",
      "ExternalReference" : "IH-EX163759714",
      "Id" : 5159685,
      "Language" : "en-GB",
      "Login" : "IH16375339714@aaar.com",
      "State" : 0,
      "StateChangeDate" : "2021-11-22T16:05:40.840Z",
      "Status" : 1,
      "SubscriberTypeCode" : 10121,
      "AdditionalProperties" : [
        {
          "ExternalReference" : "Invoice_Ref",
          "Id" : 1886,
          "Name" : "CustomerInvoiceRef",
          "Values" : [ ]
        },
        {
          "ExternalReference" : "Bill_From_Address",
          "Id" : 1888,
          "Name" : "BillFromAddress",
          "Values" : [
            "xxxx"
          ]
        }
      ],
      "ContactPreferences" : [
        {
          "ContactEventType" : 12,
          "ContactMethod" : 40,
          "Id" : 2702216,
          "OptIn" : true
        }
      ],
      "EffectiveStartDate" : "2021-11-22T16:05:40.863Z",
      "HomeCountry" : "USA",
      "InvoiceConfiguration" : {
        "HideZeroAmount" : false,
        "InvoiceDetailLevel" : 1
      },
      "StateName" : "Prospect",
      "StatusName" : "Active",
      "SubscriberCurrency" : "USD",
      "SubscriberTypeDetails" : {
        "AccountingMethod" : 2,
        "BillCycle" : "40002163",
        "BillCycleDay" : 1,
        "BillCycleName" : "BillCycle_1",
        "IsReadOnly" : false,
        "PaymentDueDays" : 10,
        "PostpaidAccountNumber" : "CAQ400037719"
      },
      "TermsAndConditionsAccepted" : "2021-11-22T16:05:40.883Z"
    }
  }
}

Updating an Index document

Having a PUT request for an ID will update if the document is already existing. The response will have result as updated.

Note that PUT request will update the whole document with new content and that means our new PUT request needs to contain all the elements. PUT request update the complete document.

If we want to update partially and don't want pass on all document elements then we should use POST request. POST request update the relevant element only.

Key point to note here are:

POST command has _update to let elasticsearch know about update
doc keyword has to be added as elastic search stores documents within doc
if we provide any new attribute into post request then it will be added into document.
Following logic gets triggered by elasticsearch after post request
- retrieves the existing document
- apply changes requested in post request
- removes the old document
- indexes new document (with update) in the place of old document

POST /subscriber/_update/5159683
{
  "doc":
  {
    "Addresses": [
        {
            "City": "Austin"

        }
    ]
  }

}

Deleting an document

Request is similar to get request

DELETE /subscriber/json/5159699

The response will have result element populated with deleted value.

{
  "_index" : "subscriber",
  "_type" : "json",
  "_id" : "5159699",
  "_version" : 2,
  "result" : "deleted",
  "_shards" : {
    "total" : 2,
    "successful" : 1,
    "failed" : 0
  },
  "_seq_no" : 33,
  "_primary_term" : 2
}

Bulk Insert

For bulk request, individual items are separated by newline characters (not commas) and th*ere are no square brackets at the end (ie the payload is a sequence of JSON documents*. The documents must be on a single line, no newlines are allowed within them. It seems bit unintuitive I know since resources don't have to be on a single line when you create them using PUT or POST for non-bulk operations

POST _bulk
{"index": {"_index": "subscriber", "_id": "5159199"}}
{"Subscriber": {"Category": 1, "CompanyName": "IH-163759714","ConvergentBillerId": "40003771", "Created": "2021-11-22T16:05:40.890Z","ExternalReference": "IH-EX163759714","Id": 5159199,"Language": "en-GB","Login": "1@1.com","State": 0, "StateChangeDate": "2021-10-22T16:05:40.840Z","Status": 1,"SubscriberTypeCode": 10121}}
{"index": {"_index": "subscriber", "_id": "5159099"}}
{"Subscriber": {"Category": 1, "CompanyName": "IH-163759714","ConvergentBillerId": "40003771", "Created": "2021-11-22T16:05:40.890Z","ExternalReference": "IH-EX163759714","Id": 5159099,"Language": "en-GB","Login": "1@1.com","State": 0, "StateChangeDate": "2021-10-22T16:05:40.840Z","Status": 2,"SubscriberTypeCode": 10122}}

Bulk insert response will return status of each document insert as shared below.

status code will be set to 201

{
  "took" : 20,
  "errors" : false,
  "items" : [
    {
      "index" : {
        "_index" : "subscriber",
        "_type" : "_doc",
        "_id" : "5159199",
        "_version" : 1,
        "result" : "created",
        "_shards" : {
          "total" : 2,
          "successful" : 1,
          "failed" : 0
        },
        "_seq_no" : 34,
        "_primary_term" : 2,
        "status" : 201
      }
    },
    {
      "index" : {
        "_index" : "subscriber",
        "_type" : "_doc",
        "_id" : "5159099",
        "_version" : 1,
        "result" : "created",
        "_shards" : {
          "total" : 2,
          "successful" : 1,
          "failed" : 0
        },
        "_seq_no" : 35,
        "_primary_term" : 2,
        "status" : 201
      }
    }
  ]
}

Bulk update

Here i am updating all the attributes of the documents.

POST _bulk
{"index": {"_index": "subscriber", "_id": "5159199"}}
{"Subscriber": {"Category": 1, "CompanyName": "IH-163759714","ConvergentBillerId": "40003771", "Created": "2021-11-22T16:05:40.890Z","ExternalReference": "IH-EX163759714","Id": 5159199,"Language": "en-GB","Login": "1@1.com","State": 1, "StateChangeDate": "2021-10-22T16:05:40.840Z","Status": 1,"SubscriberTypeCode": 10121}}
{"index": {"_index": "subscriber", "_id": "5159099"}}
{"Subscriber": {"Category": 1, "CompanyName": "IH-163759714","ConvergentBillerId": "40003771", "Created": "2021-11-22T16:05:40.890Z","ExternalReference": "IH-EX163759714","Id": 5159099,"Language": "en-GB","Login": "1@1.com","State": 1, "StateChangeDate": "2021-10-22T16:05:40.840Z","Status": 2,"SubscriberTypeCode": 10122}}

The response has status code set to 200 as shared below.

{
  "took" : 9,
  "errors" : false,
  "items" : [
    {
      "index" : {
        "_index" : "subscriber",
        "_type" : "_doc",
        "_id" : "5159199",
        "_version" : 2,
        "result" : "updated",
        "_shards" : {
          "total" : 2,
          "successful" : 1,
          "failed" : 0
        },
        "_seq_no" : 36,
        "_primary_term" : 2,
        "status" : 200
      }
    },
    {
      "index" : {
        "_index" : "subscriber",
        "_type" : "_doc",
        "_id" : "5159099",
        "_version" : 2,
        "result" : "updated",
        "_shards" : {
          "total" : 2,
          "successful" : 1,
          "failed" : 0
        },
        "_seq_no" : 37,
        "_primary_term" : 2,
        "status" : 200
      }
    }
  ]
}

Full text Search basics of Elasticsearch

Note: Whatever analyzer was using during Index creation, the same will be used during query operation. Every query term goes through analysis process.

English analyzer in EL does following

does tokenization based on white space
does stemming
removes stop words

Getting a document by Id

GET /subscriber/_doc/5159199

The response has found set to true as shared below which indicate that the id has been found. The original document is found under metadata "_source".

{
  "_index" : "subscriber",
  "_type" : "_doc",
  "_id" : "5159199",
  "_version" : 2,
  "_seq_no" : 36,
  "_primary_term" : 2,
  "found" : true,
  "_source" : {
    "Subscriber" : {
      "Category" : 1,
      "CompanyName" : "IH-163759714",
      "ConvergentBillerId" : "40003771",
      "Created" : "2021-11-22T16:05:40.890Z",
      "ExternalReference" : "IH-EX163759714",
      "Id" : 5159199,
      "Language" : "en-GB",
      "Login" : "1@1.com",
      "State" : 1,
      "StateChangeDate" : "2021-10-22T16:05:40.840Z",
      "Status" : 1,
      "SubscriberTypeCode" : 10121
    }
  }
}

Searching an Index

The following one will allow to search for all documents into the index.

GET /subscriber/_search

The search result will have following metadata and element hits.value.total will indicate how many documents been returned. Here timed_out set to false and this means request has not been timed out. the default timeout set to is 60 sec and we can specify this value in the request also to override the default value.

The value max_score equal to 1.0 means all the search results are relevant to search query.

{
    "took": 11,
    "timed_out": false,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 12,
            "relation": "eq"
        },
        "max_score": 1.0,
        "hits": [{
                "_index": "subscriber",
                "_type": "json",
                "_id": "5159698",
                "_score": 1.0,
                "_source": {
                ..........
                }
            }
        ]
    }
}

Searching by query string

we can pass on query string as shown below accompanied by q. Here elasticsearch is searching all the documents for the value 10122 irrespective of the field.

GET /subscriber/_search?q="10122"

The response has relation field value set to eq which means search happened with document string compare. The search result is ordered by _score and 1st one has _score se to 2/302585 as it had maximum time of query string 10122 match found.

{
  "took" : 429,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 6,
      "relation" : "eq"
    },
    "max_score" : 2.302585,
    "hits" : [
      {
        "_index" : "subscriber",
        "_type" : "json",
        "_id" : "49",
        "_score" : 2.302585,
        "_source" : {
          "Subscriber" : {
            "Category" : 1,
            "CompanyName" : "IH-10122",
            "ConvergentBillerId" : "10122",
            "Created" : "2021-11-22T16:05:40.890Z",
            "ExternalReference" : "IH-EX163759714",
            "Id" : 49,
            "Language" : "en-GB",
            "Login" : "1@1.com",
            "State" : 1,
            "StateChangeDate" : "2021-10-22T16:05:40.840Z",
            "Status" : 1,
            "SubscriberTypeCode" : 10121
          }
        }
      },
      {
        "_index" : "subscriber",
        "_type" : "json",
        "_id" : "79",
        "_score" : 1.7917595,
        "_source" : {
          "Subscriber" : {
            "Category" : 1,
            "CompanyName" : "IH-163759714",
            "ConvergentBillerId" : "10122",
            "Created" : "2021-11-22T16:05:40.890Z",
            "ExternalReference" : "IH-EX163759714",
            "Id" : 79,
            "Language" : "en-GB",
            "Login" : "1@1.com",
            "State" : 1,
            "StateChangeDate" : "2021-10-22T16:05:40.840Z",
            "Status" : 2,
            "SubscriberTypeCode" : 10122
          }
        }
      },
      {
        "_index" : "subscriber",
        "_type" : "json",
        "_id" : "5159099",
        "_score" : 1.0,
        "_source" : {
          "Subscriber" : {
            "Category" : 1,
            "CompanyName" : "IH-163759714",
            "ConvergentBillerId" : "40003771",
            "Created" : "2021-11-22T16:05:40.890Z",
            "ExternalReference" : "IH-EX163759714",
            "Id" : 5159099,
            "Language" : "en-GB",
            "Login" : "1@1.com",
            "State" : 1,
            "StateChangeDate" : "2021-10-22T16:05:40.840Z",
            "Status" : 2,
            "SubscriberTypeCode" : 10122
          }
        }
      },
      {
        "_index" : "subscriber",
        "_type" : "json",
        "_id" : "59099",
        "_score" : 1.0,
        "_source" : {
          "Subscriber" : {
            "Category" : 1,
            "CompanyName" : "IH-163759714",
            "ConvergentBillerId" : "40003771",
            "Created" : "2021-11-22T16:05:40.890Z",
            "ExternalReference" : "IH-EX163759714",
            "Id" : 59099,
            "Language" : "en-GB",
            "Login" : "1@1.com",
            "State" : 1,
            "StateChangeDate" : "2021-10-22T16:05:40.840Z",
            "Status" : 2,
            "SubscriberTypeCode" : 10122
          }
        }
      },
      {
        "_index" : "subscriber",
        "_type" : "json",
        "_id" : "599",
        "_score" : 1.0,
        "_source" : {
          "Subscriber" : {
            "Category" : 1,
            "CompanyName" : "IH-163759714",
            "ConvergentBillerId" : "40003771",
            "Created" : "2021-11-22T16:05:40.890Z",
            "ExternalReference" : "IH-EX163759714",
            "Id" : 599,
            "Language" : "en-GB",
            "Login" : "1@1.com",
            "State" : 1,
            "StateChangeDate" : "2021-10-22T16:05:40.840Z",
            "Status" : 2,
            "SubscriberTypeCode" : 10122
          }
        }
      },
      {
        "_index" : "subscriber",
        "_type" : "json",
        "_id" : "19",
        "_score" : 1.0,
        "_source" : {
          "Subscriber" : {
            "Category" : 1,
            "CompanyName" : "IH-163759714",
            "ConvergentBillerId" : "40003771",
            "Created" : "2021-11-22T16:05:40.890Z",
            "ExternalReference" : "IH-EX163759714",
            "Id" : 19,
            "Language" : "en-GB",
            "Login" : "1@1.com",
            "State" : 1,
            "StateChangeDate" : "2021-10-22T16:05:40.840Z",
            "Status" : 2,
            "SubscriberTypeCode" : 10122
          }
        }
      }
    ]
  }
}

Searching by specific element of document

We can pass document element name into query string and this will return all the documents matching with the value of the specific field.

GET /subscriber/_search?q=SubscriberTypeCode:"10121"

The response will have max_score value of 1 for all the returned elements as this is value compare by element name.

{
  "took" : 0,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 2,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "subscriber",
        "_type" : "json",
        "_id" : "149",
        "_score" : 1.0,
        "_source" : {
          "Category" : 1,
          "CompanyName" : "IH-10122",
          "ConvergentBillerId" : "12",
          "Created" : "2021-11-22T16:05:40.890Z",
          "ExternalReference" : "IH-EX163759714",
          "Id" : 149,
          "Language" : "en-GB",
          "Login" : "1@1.com",
          "State" : 1,
          "StateChangeDate" : "2021-10-22T16:05:40.840Z",
          "Status" : 1,
          "SubscriberTypeCode" : 10121
        }
      },
      {
        "_index" : "subscriber",
        "_type" : "json",
        "_id" : "139",
        "_score" : 1.0,
        "_source" : {
          "Category" : 1,
          "CompanyName" : "IH-10122",
          "ConvergentBillerId" : "12",
          "Created" : "2021-11-22T16:05:40.890Z",
          "ExternalReference" : "IH-EX163759714",
          "Id" : 139,
          "Language" : "en-GB",
          "Login" : "1@1.com",
          "State" : 1,
          "StateChangeDate" : "2021-10-22T16:05:40.840Z",
          "Status" : 1,
          "SubscriberTypeCode" : 10121
        }
      }
    ]
  }
}

Query DSL

If we see the above query curl command then it is as below and the curl command has the query in the url which can be exploited by hackers.

curl -XGET "http://localhost:9200/subscriber/_search?q=SubscriberTypeCode:"10121""

So instead of passing query into url, we will pass the query into the body which is more secured. All query of EL will be under query string and here is one example of match query where value is being compared.

GET /subscriber/_search
{
  "query":
  {
    "match": {
      "SubscriberTypeCode": "10122"
    }
  }
}

match

Now, lets see the corresponding curl command for this query dsl one and the url does not have any more id or field details. So query is specified in request body instead of url.

curl -XGET "http://localhost:9200/subscriber/_search" -H 'Content-Type: application/json' -d'
{
  "query":
  {
    "match": {
      "SubscriberTypeCode": "10122"
    }
  }
}'

Now, lets see the following query and its response below. The field value is being sent in lower case but still the query returned result. This is because during indexing, EL does the analysis process first which involves converting into lower case, stemming, root word etc.

GET /subscriber/_search
{
  "query":
  {
    "match": {
      "ExternalReference": "ih-EX163759714"
    }
  }
}

and the response of the above query

{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 4,
      "relation" : "eq"
    },
    "max_score" : 0.21072102,
    "hits" : [
      {
        "_index" : "subscriber",
        "_type" : "json",
        "_id" : "149",
        "_score" : 0.21072102,
        "_source" : {
          "Category" : 1,
          "CompanyName" : "IH-10122",
          "ConvergentBillerId" : "12",
          "Created" : "2021-11-22T16:05:40.890Z",
          "ExternalReference" : "IH-EX163759714",
          "Id" : 149,
          "Language" : "en-GB",
          "Login" : "1@1.com",
          "State" : 1,
          "StateChangeDate" : "2021-10-22T16:05:40.840Z",
          "Status" : 1,
          "SubscriberTypeCode" : 10121
        }
      },
      {
        "_index" : "subscriber",
        "_type" : "json",
        "_id" : "179",
        "_score" : 0.21072102,
        "_source" : {
          "Category" : 1,
          "CompanyName" : "IH-163759714",
          "ConvergentBillerId" : "16",
          "Created" : "2021-11-22T16:05:40.890Z",
          "ExternalReference" : "IH-EX163759714",
          "Id" : 179,
          "Language" : "en-GB",
          "Login" : "1@1.com",
          "State" : 1,
          "StateChangeDate" : "2021-10-22T16:05:40.840Z",
          "Status" : 2,
          "SubscriberTypeCode" : 10122
        }
      },
      {
        "_index" : "subscriber",
        "_type" : "json",
        "_id" : "139",
        "_score" : 0.21072102,
        "_source" : {
          "Category" : 1,
          "CompanyName" : "IH-10122",
          "ConvergentBillerId" : "12",
          "Created" : "2021-11-22T16:05:40.890Z",
          "ExternalReference" : "IH-EX163759714",
          "Id" : 139,
          "Language" : "en-GB",
          "Login" : "1@1.com",
          "State" : 1,
          "StateChangeDate" : "2021-10-22T16:05:40.840Z",
          "Status" : 1,
          "SubscriberTypeCode" : 10121
        }
      },
      {
        "_index" : "subscriber",
        "_type" : "json",
        "_id" : "169",
        "_score" : 0.21072102,
        "_source" : {
          "Category" : 1,
          "CompanyName" : "IH-163759714",
          "ConvergentBillerId" : "16",
          "Created" : "2021-11-22T16:05:40.890Z",
          "ExternalReference" : "IH-EX163759714",
          "Id" : 169,
          "Language" : "en-GB",
          "Login" : "1@1.com",
          "State" : 1,
          "StateChangeDate" : "2021-10-22T16:05:40.840Z",
          "Status" : 2,
          "SubscriberTypeCode" : 10122
        }
      }
    ]
  }
}

Limiting search result

Here, we are limiting result by size parameter.

GET /blogposts/_search
{
  "query": {
    "match": {
      "status": "draft"
    }
  },
  "size": 5

The following query shows how we can implement pagination in search query.

GET /blogposts/_search
{
  "query": {
    "match": {
      "status": "draft"
    }
  },
  "size": 5,
  "from": 10  
}

Limiting Number of elements/attrbutes

Sometime, the input document may have lot of elements and we may not need all the elements in search result. In elastic search, search result stays within source attribute and in following way we can reduce number of elements.

GET /blogposts/_search
{
  "query": {
    "match": {
      "status": "draft"
    }
  },
  "size": 5,
  "from": 5,
  "_source": ["title","status"]  
}

There can be scenarios, where we want to exclude only one field and in that scenario, we need to add excludes conditions.

GET /blogposts/_search
{
  "query": {
    "match": {
      "status": "draft"
    }
  },
  "size": 5,
  "from": 5,
  "_source": {
    "excludes": "tags"
  }

match_all

The following match_all allows elastic search to query all and return all documents from the index. Here within _source, we are specifying what are elements required from documents.

GET /blogposts/_search
{
  "query": {
    "match": {
      "status": "draft"
    }
  },
  "size": 5,
  "from": 5,
  "_source": ["title","status"]

multi_match

it allows to search a query string for two elements in a document as shared below. It does full text search.

GET /subscriber/_search
{
  "query": {
    "multi_match": {
      "query": "ih",
      "fields": ["ExternalReference","CompanyName"]
    }
  }
}

curl -XGET "http://localhost:9200/subscriber/_search" -H 'Content-Type: application/json' -d'
{
  "query": {
    "multi_match": {
      "query": "ih",
      "fields": ["ExternalReference","CompanyName"]
    }
  }
}'

post_filter

The post_filter is applied to the search hits at the very end of a search request, after aggregations have already been calculated. Its purpose is best explained by example:

Imagine that you are selling shirts that have the following properties:

PUT /shirts
{
    "mappings": {
        "_doc": {
            "properties": {
                "brand": { "type": "keyword"},
                "color": { "type": "keyword"},
                "model": { "type": "keyword"}
            }
        }
    }
}

PUT /shirts/_doc/1?refresh
{
    "brand": "gucci",
    "color": "red",
    "model": "slim"
}

Imagine a user has specified two filters:

color:red and brand:gucci. You only want to show them red shirts made by Gucci in the search results. Normally you would do this with a bool query:

GET /shirts/_search
{
  "query": {
    "bool": {
      "filter": [
        { "term": { "color": "red"   }},
        { "term": { "brand": "gucci" }}
      ]
    }
  }
}

However, you would also like to use faceted navigation to display a list of other options that the user could click on. Perhaps you have a model field that would allow the user to limit their search results to red Gucci t-shirts or dress-shirts.

This can be done with a terms aggregation:

GET /shirts/_search
{
  "query": {
    "bool": {
      "filter": [
        { "term": { "color": "red"   }},
        { "term": { "brand": "gucci" }}
      ]
    }
  },
  "aggs": {
    "models": {
      "terms": { "field": "model" } 
    }
  }
}

	Returns the most popular models of red shirts by Gucci.

But perhaps you would also like to tell the user how many Gucci shirts are available in other colors. If you just add a terms aggregation on the color field, you will only get back the color red, because your query returns only red shirts by Gucci.

Instead, you want to include shirts of all colors during aggregation, then apply the colors filter only to the search results. This is the purpose of the post_filter:

GET /shirts/_search
{
  "query": {
    "bool": {
      "filter": {
        "term": { "brand": "gucci" } 
      }
    }
  },
  "aggs": {
    "colors": {
      "terms": { "field": "color" } 
    },
    "color_red": {
      "filter": {
        "term": { "color": "red" } 
      },
      "aggs": {
        "models": {
          "terms": { "field": "model" } 
        }
      }
    }
  },
  "post_filter": { 
    "term": { "color": "red" }
  }
}

	The main query now finds all shirts by Gucci, regardless of color.
	The `colors` agg returns popular colors for shirts by Gucci.
	The `color_red` agg limits the `models` sub-aggregation to red Gucci shirts.
	Finally, the `post_filter` removes colors other than red from the search `hits`.

must and must_not condition

This condition allows us to match one condition and stops another condition. Ex: Our requirement is to filter all documents where subtypeid is not 10222 but externalreference is having IH then for the 1st clause we need to use must not and 2nd one shd have must.

GET /subscriber/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "match": {
            "ExternalReference": "ih"
          }
        }

      ]
      , "must_not": [
        {
          "match": {
            "SubscriberTypeCode": "10121"
          }

        }
      ]
    }
  }
}

sorting

We need to add sorting on the query result. Here you can see the sort is placed outside of query as the query result is passed to sorting.

Note: Text fields are analyzed and then tokenized. So it is not possible to sort the text fields.

GET /blogposts/_search
{
  "query": {
    "match": {
      "status": "draft"
    }
  },
  "size": 10,
  "from": 3,
  "_source": ["title","status"],
  "sort": [
    {
      "published_date": {
        "order": "desc"
      }
    }

If we have to sort text field then we need to use keyword fields as they are non-analyzed but that will be slow.

Full match condition

Here, we are doing full match which is identified by operator and and this is based on title field.

GET /blogposts/_search
{
  "query": {
    "match": {
      "title":{
        "query":"Introduction to elasticsearch"
        , "operator": "and"
      }
    }
  }

Minimum match condition

It is sometime necessary to match defined words at minimum level and we don't want full all words to be matched as per our search text.

GET /blogposts/_search
{
  "query": {
    "match": {
      "title":{
        "query":"Introduction to elasticsearch", 
        "minimum_should_match": 2
      }
    }
  }

}

we also can give % wise match rather than specifying perfect no of terms.

GET /blogposts/_search
{
  "query": {
    "match": {
      "title":{
        "query":"Introduction to elasticsearch", 
        "minimum_should_match": "25%"
      }
    }
  }

}

Term and match query

Term query does not go through analysis process. match query goes through analysis process. Lets understand through following example.

Lets see the following two example of match query returning us records. This is because the elasticsearch is storing Draft as draft due to lowercase in analysis process. Hence searching by Draft or draft will return us same set of query results.

GET /blogposts/_search
{
  "query": {
    "match": {
      "status": "draft"
    }
  }
}


GET /blogposts/_search
{
  "query": {
    "match": {
      "status": "Draft"
    }
  }
}

Now, lets look at the term query and since elasticsearch has lowercase filter here so all texts are stored in lower case,

1st query here will not return any data
2nd query here will return data

GET /blogposts/_search
{
  "query": {
    "term": {
      "status": {
        "value": "Draft"
      }
    }
  }
}

GET /blogposts/_search
{
  "query": {
    "term": {
      "status": {
        "value": "draft"
      }
    }
  }
}

terms filter

This is used when we want to add multiple term filter together. Here “tags” is the document element name and “elasticsearch” and “fast” are element value to compare. Note: This is or condition and any document having tags value of elasticsearch or fast will be returned in result.

GET /blogposts/_search
{
  "query": {
    "terms": {
      "tags": [
        "elasticsearch",
        "fast"
      ]
    }
  }
}

prefix query

Remember that, this query does not go through analysis process and hence exact string needs to be passed. It does not calculate score and all score remains as 1.

GET  /blogposts/_search
{
  "query": {
    "prefix": {
      "title": {
        "value": "intro"
      }
    }
  }
}

wildcard query

This is same as prefix query.

GET  /blogposts/_search
{
  "query": {
    "wildcard": {
      "title": {
        "value": "n*ro"
      }
    }
  }
}

Complex Queries

Lets take the following requirement and see how we can write the query.

blogpost must not be in draft status
blogpost should have elasticsearch
blogpost should have like > 10

Lets break this business problem into multiple phases.

Solution

First, we try to achieve the condition to remove draft blogposts from query.
1. We need query here for searching
2. Since there will be multiple conditions, so we need to use bool type of query
3. we need to filter blogposts which are in draft status and this is case insensitive search. So we i have used must_not and then match filter with field name status

GET  /blogposts/_search
{
  "query": {
    "bool": {
      "must_not": [
        {
          "match": {
            "status": "draft"
          }
        }
      ]
    }
  }
}

Now, we will need to allow all blogpost which are having elasticsearch into tag.
1. So we need to use must clause into bool query
2. we will then use match clause as we want to have case insensitive search

GET  /blogposts/_search
{
  "query": {
    "bool": {
      "must_not": [
        {
          "match": {
            "status": "draft"
          }
        }
      ],
      "must": [
        {
          "match": {
            "tags": "elasticsearch"
          }
        }
      ]
    }
  }
}

Now we need to add condition of greater than.
1. So we need filter condition as it will take search result out to further filter data as per where clause.
2. we need to add range condition here as our condition of >10 is based on range

So, our *final query will be *

GET  /blogposts/_search
{
  "query": {
    "bool": {
      "must_not": [
        {
          "match": {
            "status": "draft"
          }
        }
      ],
      "must": [
        {
          "match": {
            "tags": "elasticsearch"
          }
        }
      ],
      "filter": [
        {"range": {
          "no_of_likes": {
            "gte": 10
          }
        }}
      ]
    }
  }
}

Query Relevence

We can use the score value in the search result to understand how the document is relevant to original query. Relevancy score is represented by a positive floating number.

The scoring of a document is determined based on the field matches from the query specified and any additional configurations you apply to the search.

Refer the article How scoring works for more in-depth analysis on this.

Document matching happens in

binary sense (matching with exact values)
relevancy sense matching

Scoring Function

Uses following and this means more time a term appears in a document then the document is more relevant while more time the term appears across other documents then it is less relevant.

Term Frequency (TF)
Inverse document Frequency (IDF)

When a document matches the query, Lucene calculates the score by combining the score of each matching term. This scoring calculation is done by the practical scoring function.

score(q,d)  =  
           queryNorm(q)  
          · coord(q,d)    
          · ∑ (           
                tf(t in d)   
              · idf(t)²      
              · t.getBoost() 
              · norm(t,d)    
            ) (t in q)

score(q,d) is the relevance score of document d for query q.
queryNorm(q) is the query normalization factor.
coord(q,d) is the coordination factor.
The sum of the weights for each term t in the query q for document d.
- tf(t in d) is the term frequency for term t in document d.
- idf(t) is the inverse document frequency for term t.
- t.getBoost() is the boost that has been applied to the query.
- norm(t,d) is the field-length norm, combined with the index-time field-level boost, if any.

Boosting

The _boost field (document-level boost) was removed, but field-level boosts, and query boosts still work just fine. If we want to boost matches on field1:

Boosting can be of:

Index time boosting
- stores at index. boost value cant be changed in future other than re-indexing
query time boosting
- Allows to define at query time and can be changed anytime

"bool": {
    "should": [{
        "terms": {
            "field1": ["67", "93", "73", "78", "88", "77"],
            "boost": 2.0
        }
    }, {
        "terms": {
            "field2": ["68", "94", "72", "76", "82", "96", "70", "86", "81", "92", "97", "74", "91", "85"]
        }
    }, {
        "terms": {
            "category": ["cat2"]
        }
    }]
}

We can add also like this,,,

A class of boost 2 means it is twice as important compare to others
Boost figure is relative.
Tough to finalize correct boost value and this is more an iterative process to find out optimal relevency score

GET /blogposts/_search
{
  "query": {
    "bool": {
      "should": [
        {
          "match": {
            "title":{
              "query": "Elasticsearch",
              "boost": 2
            }
          }
        },
        {
          "match": {
            "content":{
              "query": "Elasticsearch"

            }
          }
        }
      ]
    }
  }

}

Filter Search basics of Elasticsearch

Filter condition allows us to add more condition into search criteria. Here we see first condition is must and this means it will retrieve all documents where ExternalReference has ih text. This is being set as must. Next condition is filter based one which will be triggered. here filter condition will be based on status greater than 1. Here since i need range bound one so have used range filter.

Always first filter context gets executed and then query context gets executed. This is to ensure filtered objects are available for document search by query objects. Score does not gets calculated for filter query results by elasticsearch. Score only gets calculated for query context.

GET /subscriber/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "match": {
            "ExternalReference": "ih"
          }
        }

      ]
      , "filter": [
        {
          "range": {
            "Status": {
              "gte": 1
            }
          }
        }
      ]
    }
  }
}

There can be filter based on term which is where we compare the value. term filter has been used here to compare the value. in case of date range, we need to use range filter.

GET /subscriber/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "match": {
            "ExternalReference": "ih"
          }
        }

      ]
      , "filter": [
        {
          "term": {
            "Status": "2"
          } 
        }
      ]
    }
  }
}

Cluster Node

Cluster has a master node and any node can become master node. Every node knows where each document leaves. When an index is sharded, a given document within that index will only be stored within one of the shards.

There are two main reasons why sharding is important, with the first one being that it allows you to split and thereby scale volumes of data. So if you have growing amounts of data, you will not face a bottleneck because you can always tweak the number of shards for a particular index. The other reason why sharding is important, is that operations can be distributed across multiple nodes and thereby parallelized. This results in increased performance, because multiple machines can potentially work on the same query. This is completely transparent to you as a user of Elasticsearch.

So how do you specify the number of shards an index has? You can optionally specify this at index creation time, but if you don’t, a default number of 5 will be used. This is sufficient in most cases, since it allows for a good amount of growth in data before you need to worry about adding additional shards.

Primary and Replica shard

There are two types of shards: primaries and replicas. Each document in an index belongs to one primary shard. ... The number of primary shards in an index is fixed at the time that an index is created, but the number of replica shards can be changed at any time, without interrupting indexing or query operations.

Elasticsearch uses the concept of the shard to subdivide the index into multiple pieces and allows us to make one or more copies of index shards called replicas. Please refer to this SO answer to get a detailed understanding of shards and replicas.

To set the number of shards and replicas as properties of index: Here it means number of primary shard will be 6 and for each primary shard there will be 2 replica shard available.

PUT /indexName

{
  "settings": {
    "index": {
      "number_of_shards": 6,
      "number_of_replicas": 2
    }
  }
}

Some important tips for choosing the number of shards and replicas:

The number of shards cannot be changed after an index is created. If you later find it necessary to change the number of shards, then you will have to reindex all the documents again.
To decide no of shards, you will have to choose a starting point and then try to find the optimal size through testing with your data and queries.
Replicas tend to improve search performance (not always). But, it is recommended to have at least 1 replica (so that data is preserved in case of hardware failure)
Refer this medium article, that states that number of nodes and number of shards (primary shard + replicas), should be proportional to each other. This is important for Elasticsearch to ensure proper load balancing.
As stated in this article it is recommended to keep the number of shards per node below 20 per GB heap it has configured.
According to this blog when you’re planning for capacity, try and allocate shards at a rate of 150% to 300% (or about double) the number nodes that you had when initially configuring your datasets

Replication serves two purposes, with the main one being to provide high availability in case nodes or shards fail. For replication to even be effective if something goes wrong, replica shards are never allocated to the same nodes as the primary shards, which you can also see on the above diagram. This means that even if an entire node fails, you will have at least one replica of any primary shards on that particular node. The other purpose of replication — or perhaps a side benefit — is increased performance for search queries. This is the case because searches can be executed on all replicas in parallel, meaning that replicas are actually part of the cluster’s searching capabilities.

Replication between Primary and Replica

So how does Elasticsearch keep everything in sync? Elasticsearch uses a model named primary-backup for its data replication.

Adding, updating, or removing documents — are sent to the primary shard. The primary shard is then responsible for validating the operations and ensuring that everything is good. This involves checking if the request is structurally invalid, such as trying to add a number to an object field or something like that. When the operation has been accepted by the primary shard, the operation will be performed locally on the primary shard itself. When the operation completes, the operation will be forwarded to each of the replica shards in the replica group. If the shard has multiple replicas, the operation will be performed in parallel on each of the replicas. When the operation has completed successfully on every replica and responded to the primary shard, the primary shard will respond to the client that the operation has completed successfully. This is all illustrated in the below diagram.

Search Operation and its combine

There are two phases of search operation in elasticsearch,

query all the matching documents from all the sherds
combine all the search records in master node from all the sherds for the respective documents

Distributed searching

A search request can be accepted by any node in the Elasticsearch cluster. Each of the nodes in the cluster is aware of all the other nodes. The catch being that, the node receiving the search request, by default is unaware of where the data that is to be queried resides. Hence, the node has no choice but to broadcast the request to all the shards and then gather their responses into a globally sorted result set that it can return to the client.

The process of determining which shard a particular document resides in is termed as routing. By default, routing is handled by Elasticsearch. The default routing scheme hashes the ID of a document and uses it as the partition key to find a shard. This includes both user-provided IDs and randomly generated IDs picked by Elasticsearch. Documents are assigned shards by hashing the document ID, dividing the hash by the number of shards and taking the remainder.

Index Components

Manually Index creation

We can create index manually and mainly we need to provide shard and replica related configuration.

PUT /blogposts/
{
  "settings": {
    "number_of_shards": 2
    , "number_of_replicas": 1
  }

}

Here, we can define index element datatype and it is better that we always define explicitly datatype rather than relying on Elasticsearch.

Key datatype are:

string
Float
Double
date
boolean
array (Can only hold one datatype data. It cant hold multiple data type like string and Boolean)
Keyword

PUT /blogposts/_mapping
{
  "properties":{
    "title":{"type":"text"},
    "content": {"type": "text"},
    "published_date":{"type":"date"}
  }
}

Response from Elasticsearch will be acknowledgement one.

{
  "acknowledged" : true
}

How index components are?

We can find out details of index by GET command with Index name.

GET /subscriber/

Elasticsearch defines index fields mapping and its data types during 1st document insert by default. So if later new datatype has been sent into any field then Elasticsearch will give error. So it is always good to define the index mapping explictly rather than allowing Elasticsearch to assume it by default.

The search result will have following 3 components

aliases
settings
- It will have following informations
- number_of_shards (Elasticsearch will set this by default to 1)
- provided_name
- index creation date
- uuid (Unique identifier of the index)
- number_of_replicas (Elasticsearch will set this by default to 1)

    "settings" : {
        "index" : {
          "routing" : {
            "allocation" : {
              "include" : {
                "_tier_preference" : "data_content"
              }
            }
          },
          "number_of_shards" : "1",
          "provided_name" : "subscriber",
          "creation_date" : "1637599846785",
          "number_of_replicas" : "1",
          "uuid" : "L4jp0hosRq2Pjqk3hHxP5w",
          "version" : {
            "created" : "7150299"
          }
        }
      }

mappings
- defines how the index should be distributed. Mapping is either generated automatically or we can define explicitly.
- It will have all element names for documents we have indexed.
- Elasticsearch detects field types from its values.

  "PostalCode" : {
                "type" : "text",
                "fields" : {
                  "keyword" : {
                    "type" : "keyword",
                    "ignore_above" : 256
                  }
                }
              }

{
  "subscriber" : {
    "aliases" : { },
    "mappings" : {
      "properties" : {
        "Addresses" : {
          "properties" : {
            "City" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "Country" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "Created" : {
              "type" : "date"
            },
            "DefaultBilling" : {
              "type" : "boolean"
            },
            "DefaultHome" : {
              "type" : "boolean"
            },
            "DefaultPostal" : {
              "type" : "boolean"
            },
            "DefaultService" : {
              "type" : "boolean"
            },
            "DefaultShipping" : {
              "type" : "boolean"
            },
            "DefaultWork" : {
              "type" : "boolean"
            },
            "Id" : {
              "type" : "long"
            },
            "LineOne" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "Modified" : {
              "type" : "date"
            },
            "Name" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "PostalCode" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "ShipToName" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "State" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "Status" : {
              "type" : "long"
            },
            "StatusName" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "Verifiable" : {
              "type" : "boolean"
            },
            "Verified" : {
              "type" : "boolean"
            }
          }
        },
        "Category" : {
          "type" : "long"
        },
        "CompanyName" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "ConvergentBillerId" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "Created" : {
          "type" : "date"
        },
        "ExternalReference" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "Id" : {
          "type" : "long"
        },
        "Language" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "Login" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "State" : {
          "type" : "long"
        },
        "StateChangeDate" : {
          "type" : "date"
        },
        "Status" : {
          "type" : "long"
        },
        "Subscriber" : {
          "properties" : {
            "AdditionalProperties" : {
              "properties" : {
                "ExternalReference" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "Id" : {
                  "type" : "long"
                },
                "Name" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "Values" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                }
              }
            },
            "Category" : {
              "type" : "long"
            },
            "CompanyName" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "ContactPreferences" : {
              "properties" : {
                "ContactEventType" : {
                  "type" : "long"
                },
                "ContactMethod" : {
                  "type" : "long"
                },
                "Id" : {
                  "type" : "long"
                },
                "OptIn" : {
                  "type" : "boolean"
                }
              }
            },
            "ConvergentBillerId" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "Created" : {
              "type" : "date"
            },
            "EffectiveStartDate" : {
              "type" : "date"
            },
            "ExternalReference" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "HomeCountry" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "Id" : {
              "type" : "long"
            },
            "InvoiceConfiguration" : {
              "properties" : {
                "HideZeroAmount" : {
                  "type" : "boolean"
                },
                "InvoiceDetailLevel" : {
                  "type" : "long"
                }
              }
            },
            "Language" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "Login" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "State" : {
              "type" : "long"
            },
            "StateChangeDate" : {
              "type" : "date"
            },
            "StateName" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "Status" : {
              "type" : "long"
            },
            "StatusName" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "SubscriberCurrency" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "SubscriberTypeCode" : {
              "type" : "long"
            },
            "SubscriberTypeDetails" : {
              "properties" : {
                "AccountingMethod" : {
                  "type" : "long"
                },
                "BillCycle" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "BillCycleDay" : {
                  "type" : "long"
                },
                "BillCycleName" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "IsReadOnly" : {
                  "type" : "boolean"
                },
                "PaymentDueDays" : {
                  "type" : "long"
                },
                "PostpaidAccountNumber" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                }
              }
            },
            "TermsAndConditionsAccepted" : {
              "type" : "date"
            }
          }
        },
        "SubscriberTypeCode" : {
          "type" : "long"
        },
        "TermsAndConditionsAccepted" : {
          "type" : "date"
        }
      }
    },
    "settings" : {
      "index" : {
        "routing" : {
          "allocation" : {
            "include" : {
              "_tier_preference" : "data_content"
            }
          }
        },
        "number_of_shards" : "1",
        "provided_name" : "subscriber",
        "creation_date" : "1637599846785",
        "number_of_replicas" : "1",
        "uuid" : "L4jp0hosRq2Pjqk3hHxP5w",
        "version" : {
          "created" : "7150299"
        }
      }
    }
  }
}

Analyzer

An analyzer — whether built-in or custom — is just a package which contains three lower-level building blocks: character filters, tokenizers, and token filters.

Character filters

Character filters are used to preprocess the stream of characters before it is passed to the tokenizer. An example is HTML stripc character filter that strips HTML elements from a text and replaces HTML entities with their decoded value (e.g, replaces & with &).

Tokenizers

A tokenizer receives a stream of characters, breaks it up into individual tokens (usually individual words), and outputs a stream of tokens.

Token filters

Token filters accept a stream of tokens from a tokenizer and can modify tokens (eg lowercasing), delete tokens (eg remove stopwords) or add tokens (eg synonyms). An example is a lowercase token filter that simply changes token text to lowercase. ASCII folding token filter converts alphabetic, numeric, and symbolic characters that are not in the Basic Latin Unicode block (first 127 ASCII characters) to their ASCII equivalent, if one exists. For example, the filter changes à to a.

There can be multiple token filters as shown and each token filter does different work. There can be different query time and Index time analyzer.

Characteristics of an analyzer**

An analyzer may have zero or more character filters, which are applied in order.
An analyzer must have exactly one tokenizer.
An analyzer may have zero or more token filters, which are applied in order.
Tokenizer generates tokens, which will be passed on to the token filter and then eventually become terms in the inverted index.
Certain tokenizers like ngram, edgengram can generate lots of tokens, which can cause higher disk usage.

The analyzer will affect how we search the text, but it won’t affect the content of the text itself. With this example, if we search for let, the Elasticsearch will still return the full text Let’s build an autocomplete! instead of only let.

Elasticsearch’s Analyzer has three components you can modify depending on your use case:

Character Filters
Tokenizer
Token Filter

Character Filter

The character filter has the ability to perform addition,removal or replacement actions on the input text given to them.

Here is a sample request

POST _analyze/?pretty
{
  "tokenizer": "standard",
  "char_filter": ["html_strip"],
  "text": "The <b> Auto-generation </b> is a success"

  }

curl -XPOST "http://localhost:9200/_analyze/?pretty" -H 'Content-Type: application/json' -d'
{
  "tokenizer": "standard",
  "char_filter": ["html_strip"],
  "text": "The <b> Auto-generation </b> is a success"

  }'

The response shows that, html tag has been removed and the resulting tokens are like below:

  "tokens" : [
    {
      "token" : "The",
      "start_offset" : 0,
      "end_offset" : 3,
      "type" : "<ALPHANUM>",
      "position" : 0
    },
    {
      "token" : "Auto",
      "start_offset" : 8,
      "end_offset" : 12,
      "type" : "<ALPHANUM>",
      "position" : 1
    },
    {
      "token" : "generation",
      "start_offset" : 13,
      "end_offset" : 23,
      "type" : "<ALPHANUM>",
      "position" : 2
    },
    {
      "token" : "is",
      "start_offset" : 29,
      "end_offset" : 31,
      "type" : "<ALPHANUM>",
      "position" : 3
    },
    {
      "token" : "a",
      "start_offset" : 32,
      "end_offset" : 33,
      "type" : "<ALPHANUM>",
      "position" : 4
    },
    {
      "token" : "success",
      "start_offset" : 34,
      "end_offset" : 41,
      "type" : "<ALPHANUM>",
      "position" : 5
    }
  ]
}

Tokenizer

The input text after its transformation from the Character filter is passed to the tokeniser. The tokenizer would split this input text into individual tokens (or terms) at specific characters. The default tokenizer in elasticsearch is the “standard tokeniser”, which uses the grammar based tokenisation technique.

The above command shows the resulting tokens are like below:

“The”,”Auto”,”generation”,”is”,”a”,”success”

The words are split whenever there is a white-space and also a hyphen (-).

Standard Tokenizer

default analyzer of the Elasticsearch
It uses grammar based Tokenization specified in https://unicode.org/reports/tr29/
Does following
- Tokenizer
- Standard Tokenizer
- Token Filters
- Lower Case Token Filter
- Stop Token Filter (disabled by default)

Simple Analyzer

The simple analyzer breaks text into tokens at any non-letter character, such as numbers, spaces, hyphens and apostrophes, discards non-letter characters, and changes uppercase to lowercase. It does not remove stop words.

POST _analyze
{
  "analyzer": "simple",
  "text": "The 2 QUICK Brown-Foxes jumped over the lazy dog's bone."
}

The simple analyzer parses the sentence and produces the following tokens:

[ the, quick, brown, foxes, jumped, over, the, lazy, dog, s, bone ]

Whitespace analyzer

The whitespace analyzer breaks text into terms whenever it encounters a whitespace character. The whitespace analyzer is not configurable. If you need to customize the whitespace analyzer then you need to recreate it as a custom analyzer and modify it, usually by adding token filters. This would recreate the built-in whitespace analyzer and you can use it as a starting point for further customization:

POST _analyze
{
  "analyzer": "whitespace",
  "text": "The 2 QUICK Brown-Foxes jumped over the lazy dog's bone."
}

The above sentence would produce the following terms:

[ The, 2, QUICK, Brown-Foxes, jumped, over, the, lazy, dog's, bone. ]

Stop analyzer

The stop analyzer is the same as the simple analyzer but adds support for removing stop words. It defaults to using the _english_ stop words.

POST _analyze
{
  "analyzer": "stop",
  "text": "The 2 QUICK Brown-Foxes jumped over the lazy dog's bone."
}

The above sentence would produce the following terms:

[ quick, brown, foxes, jumped, over, lazy, dog, s, bone ]

Keyword analyzer

The keyword analyzer is a “noop” analyzer which returns the entire input string as a single token.

POST _analyze
{
  "analyzer": "keyword",
  "text": "The 2 QUICK Brown-Foxes jumped over the lazy dog's bone."
}

The above sentence would produce the following single term:

[ The 2 QUICK Brown-Foxes jumped over the lazy dog's bone. ]

Fingerprint analyzer

The fingerprint analyzer implements a fingerprinting algorithm which is used by the OpenRefine project to assist in clustering.

Input text is lowercased, normalized to remove extended characters, sorted, deduplicated and concatenated into a single token. If a stop word list is configured, stop words will also be removed.

creates fingerprint which can be used for duplicate detection
input is lowercased, normalized to remove extended character, sorted, de-duplicated and concatenated

POST _analyze
{
  "analyzer": "fingerprint",
  "text": "Yes The customer is not honest. He is not reliable for company."
}

Output:

{
  "tokens" : [
    {
      "token" : "company customer for he honest is not reliable the yes",
      "start_offset" : 0,
      "end_offset" : 63,
      "type" : "fingerprint",
      "position" : 0
    }
  ]
}

Custom Analyzer

Here for custom analyzer, we have provided following

analyzer name which is being set as amit_custom_analyze
- analyzer type is set to custom
- char_filter is set to html stripping. we can have 0 or more character filter and hence it is array type
- tokenizer filter is set to standard one
- token filter is being set as lowercase to convert into lower case

This custom analyzer now can be associated with any of the index element.

PUT  /blogposts
{
  "settings": {
    "number_of_shards": 2,
    "number_of_replicas": 1,
    "analysis": {
      "analyzer": {
        "amit_custom_analyze":{
          "type": "custom",
          "char_filter": ["html_strip"],
          "tokenizer": "standard",
          "token_filters": ["lowercase"]

        }
      }
    }
  }
}

Response:

{
  "acknowledged" : true,
  "shards_acknowledged" : true,
  "index" : "blogposts"
}

Custom Index with Custom Analyzer

Here:

mapping provides each element types and its allowed data type
setting provides Index metadata attributes

PUT  /blogposts
{
   "settings": {
    "number_of_shards": 2,
    "number_of_replicas": 1,
    "analysis": {
      "analyzer": {
        "amit_custom_analyze":{
          "type": "custom",
          "char_filter": ["html_strip"],
          "tokenizer": "standard",
          "token_filters": ["lowercase"]

        }
      }
    }
  },
  "mappings": {
    "properties": {
      "title":{"type": "text", "analyzer": "amit_custom_analyze"},
      "content":{"type": "text", "analyzer": "amit_custom_analyze"},
      "published_date":{"type": "date"},
      "no_of_likes":{"type": "text"}
    }
  }

}

response:

{
  "acknowledged" : true,
  "shards_acknowledged" : true,
  "index" : "blogposts"
}

Here is one example of using custom analyzer during analysis process:

Example shows

html content removed
converted to lower case

POST  /blogposts/_analyze
{
  "analyzer": "amit_custom_analyze",
  "text": ["This is <html> kayal exploring elasticsearch"]
}

{
  "tokens" : [
    {
      "token" : "This",
      "start_offset" : 0,
      "end_offset" : 4,
      "type" : "<ALPHANUM>",
      "position" : 0
    },
    {
      "token" : "is",
      "start_offset" : 5,
      "end_offset" : 7,
      "type" : "<ALPHANUM>",
      "position" : 1
    },
    {
      "token" : "kayal",
      "start_offset" : 15,
      "end_offset" : 20,
      "type" : "<ALPHANUM>",
      "position" : 2
    },
    {
      "token" : "exploring",
      "start_offset" : 21,
      "end_offset" : 30,
      "type" : "<ALPHANUM>",
      "position" : 3
    },
    {
      "token" : "elasticsearch",
      "start_offset" : 31,
      "end_offset" : 44,
      "type" : "<ALPHANUM>",
      "position" : 4
    }
  ]
}

How to create customer tokenizer and filters

Here, I have shown an example of Index creation command where custom tokenizer and filters have been defined.

punctuation is the name of the tokenizer and similarly symbol is the name of the char_filter
tokenizer will look for pattern @# and tokenize
symbol will look for emoji :) and :( to convert them into happy and sad

PUT  /blogposts
{
   "settings": {
    "number_of_shards": 2,
    "number_of_replicas": 1,
    "analysis": {
      "analyzer": {
        "amit_custom_analyze":{
          "type": "custom",
          "tokenizer": "punctuation",
          "char_filter":"symbol"
        }
      },
      "tokenizer": {
        "punctuation":{
          "type":"pattern",
          "pattern":"@#"
        }
      },
      "char_filter": {
        "symbol":{
          "type":"mapping",
          "mappings": [":) ==> happy",":( ==> sad"]
        }
      }
    }
  },
  "mappings": {
      "properties": {
      "title":{"type": "text", "analyzer": "amit_custom_analyze"},
      "content":{"type": "text", "analyzer": "amit_custom_analyze"},
      "published_date":{"type": "date"},
      "no_of_likes":{"type": "text"}
    }
    }

  }

Aggregation

Every aggregation is combination of one or more buckets and metrices. Aggregation refers to the collection of documents or a set of documents obtained from a particular search query or filter. Aggregation forms the main concept to build the desired visualization in Kibana.

Aggregation always works on top of query and filter result. This is true unless we use post_filter.

Buckets

A bucket mainly consists of a key and a document. When the aggregation is executed, the documents are placed in the respective bucket.
Bucket aggregations categorize sets of documents as buckets. The type of bucket aggregation determines whether a given document falls into a bucket or not.
use bucket aggregations to implement faceted navigation (usually placed as a sidebar on a search result landing page) to help you’re users narrow down the results.
Buckets are similar to GROUP BY

Note: Here the tag is keyword field and not an analyzed one. If this is analyzed one then we need nested raw one.

We can have sub-aggregation within buckets also.

Metric aggregations—This aggregation helps in calculating matrices from the fields of aggregated document values.

Pipeline aggregations—As the name suggests, this aggregation takes input from the output results of other aggregations.

Matrix aggregations (still in the development phase)—These aggregations work on more than one field and provide statistical results based on the documents utilized by the used fields.

Steps are

Document ==> Filter ==> Query ==> Query Result ==> Aggregation ==> Aggregation Result

GET /blogposts/_search
{
  "aggs": {
    "tag_wise_stats": {
      "terms": {
        "field": "tags",
        "size": 10
      }
    }
  }
}

GET /shirts/_search
{
  "query": {
    "bool": {
      "filter": [
        { "term": { "color": "red"   }},
        { "term": { "brand": "gucci" }}
      ]
    }
  },
  "aggs": {
    "models": {
      "terms": { "field": "model" } 
    }
  }
}

Metrices

Metric Aggregation mainly refers to the maths calculation done on the documents present in the bucket. For example if you choose a number field the metric calculation you can do on it is COUNT, SUM, MIN, MAX, AVERAGE etc.

Logstash

Logstash is typically used as the “processing” engine for any log management solution (or systems that
deal with changing data streams).

These data can be structured, semi-structured, or unstructured, and can have many different schemas. To Logstash, all these data are “logs” containing “events”. Logstash can easily parse and filter out the data from these log events using one or more filtering plugins that come with it. Finally, it can send the filtered output to one or more destinations. Again, there are prebuilt output interfaces that make this task simple.

Logstash itself doesn’t access the source system and collect the data, it uses input plugins to ingest the data from various sources.

Note: There’s a multitude of input plugins available for Logstash such as various log files, relational databases, NoSQL databases, Kafka queues, HTTP endpoints, S3 files, CloudWatch Logs, log4j events or Twitter feed.

Once data is ingested, one or more filter plugins take care of the processing part in the filter stage.

Here is a sample logstash config file which allows to push data into elasticsearch host. This config file which allows logstash to takes data from STDIN and and pushes into blogposts index of elasticsearch running in localhost port 9200.

input { stdin {codec => json } }
output {
  elasticsearch { 
  hosts => ["localhost:9200"]
  index => "blogposts"
  document_type => _doc
  }
  stdout { }
}

Filters are an in-line processing mechanism that provide the flexibility to slice and dice your data to fit your needs. Let’s take a look at some filters in action. The following configuration file sets up the grok and date filters.

input { stdin { } }

filter {
  grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
  }
  date {
    match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
  }
}

output {
  elasticsearch { hosts => ["localhost:9200"] }
  stdout { codec => rubydebug }
}

Sample Filter Plugin

Reference Architecture

References

42 Elasticsearch Query Examples – Hands-on Tutorial

Demand Forecasting with AWS Forecast

Amit Kayal — Sat, 02 Oct 2021 12:24:08 +0000

What is forecasting?

A time series essentially is a series of quantitative values. These values are obtained over time, and often have equal time intervals between them. These intervals can be quite different and may consist of yearly, quarterly, monthly or hourly buckets for instance.

Time-series methods are:

Moving Average
Autoregression
Vector Autoregression
Autoregressive Integrated Moving Average
Autoregressive Moving Average

Components of Time Series

Key classifications of the components of the time series are:

Random or Irregular movements.
Cyclic Variations.
Seasonal Variations.
Trend.

Why Amazon Forecast

Fully managed service that uses machine learning to deliver highly accurate forecasts.
Based on the same machine learning forecasting technology used by Amazon.com.
Automated machine learning
- Includes AutoML capabilities that take care of the machine learning for you.
Works with any historical time series data to create accurate forecasts
- in a retail scenario, Amazon Forecast uses machine learning to process your time series data (such as price, promotions, and store traffic) and combines that with associated data (such as product features, floor placement, and store locations) to determine the complex relationships between them.
- So, it can combine combining time series data with additional variables for time series prediction

How Amazon Forecast works?

Here is the flow diagram taken from AWS site and key points here to note are

Historical data can be pushed into S3
Forecast can be triggered on data arrival in S3
Output of forecast can be pushed into S3 for further actions.

Forecasting automation with Amazon Forecast by applying MLOps

The following model architecture taken from AWS site allows us to build, train, and deploy a time-series forecasting model leveraging an MLOps pipeline encompassing Amazon Forecast, AWS Lambda, and AWS Step Functions. To visualize the generated forecast, we will use a combination of AWS serverless analytics services such as Amazon Athena and Amazon QuickSight.

Key components of this architecture are

dataset is uploaded to the Amazon S3 cloud storage under the /**train** directory (prefix).
uploaded file triggers Lambda, which initiates the MLOps pipeline built using a Step Functions state machine.
The state machine stitches together a series of Lambda functions to build, train, and deploy a ML model in Amazon Forecast.
Amazon CloudWatch, which captures Forecast metrics is being used for log analysis
SNS being used for Forecasting job status change notification
- final forecasts become available in the source Amazon S3 bucket in the **/forecast** directory.
- ML pipeline saves any old forecasts in the **/history** directory.

Forecast Workflow

The workflow to generating forecasts consists of the following steps.

Creating related datasets and a dataset group
Retrieving training data
Training predictors (trained model) using an algorithm or AutoML
Evaluating the predictor with metrics
Creating a forecast
Retrieving forecast for users

Amazon Forecast supports the following dataset domains:

RETAIL Domain – For retail demand forecasting
INVENTORY_PLANNING Domain – For supply chain and inventory planning
EC2 CAPACITY Domain – For forecasting Amazon Elastic Compute Cloud (Amazon EC2) capacity
WORK_FORCE Domain – For work force planning
WEB_TRAFFIC Domain – For estimating future web traffic
METRICS Domain – For forecasting metrics, such as revenue and cash flow
CUSTOM Domain – For all other types of time-series forecasting

Example 1: Dataset Types in the RETAIL Domain

If you are a retailer interested in forecasting demand for items, you might create the following datasets in the RETAIL domain:

Target time series is the required dataset of historical time-series demand (sales) data for each item (each product a retailer sells). In the RETAIL domain, this dataset type requires that the dataset includes the item_id, timestamp, and the demand fields. The demand field is the forecast target, and is typically the number of items sold by the retailer in a particular week or day.
Optionally, a dataset of the related time series type. In the RETAIL domain, this type can include optional, but suggested, time-series information such as price, inventory_onhand, and webpage_hits.
Optionally, a dataset of the item metadata type. In the RETAIL domain, Amazon Forecast suggests providing metadata information related to the items that you provided in target time series, such as brand, color, category, and genre.

A case study

I took the dataset from Kaggle Store Item Demand Forecasting Challenge which has given 5 years of store-item sales data, and asked to predict 3 months of sales for 50 different items at 10 different stores.

Here is the way I have used AWS Forecasting with minimal coding.

Import your data

Dataset Details

Following are the details needs to be provided.

Dataset name
Frequency of your data’
Data schema
- I have used here schema builder option which is more visual one. Another option is json schema which allows us to specify AttributeName and AttributeType in the JSON format.
- Forecast data schema has concept of domain to make our dataset creation much easier. I have selected retail domain option here and forecast has guided me to have following attributes.
- item_id (attribute type: string) - Mandatory by forecast
- timestamp(attribute type: timestamp and have selected format as yyyy-mm-dd) - Mandatory by forecast
- demand(attribute type float) - Mandatory by forecast
- store (attribute type: string) - Had to add this as my forecast has to be based on timestamp, store id and item id.
- It is essential that All attributes displayed must exist in your CSV file and must be ordered in the same order that they appear in your CSV file

I have used following python code to process my dataset from kaggle.

train_df = pd.read_csv("train.csv")
train_df["timestamp"]=pd.to_datetime(train_df['timestamp'])
train_df["timestamp"]=pd.to_datetime(train_df['timestamp'],
                                     format='%Y%m%d %H:%M:%S')
train_df_final = train_df_final[['item_id', 'timestamp', 'demand', 'store']]
train_df_final.to_csv("train_df.csv",
                      index=False,
                     date_format='%Y-%m-%d')

Upload dataset into AWS S3

Create an AWS S3 bucket, and upload the time-series data into the bucket.

s3.create_bucket(Bucket=bucketName)
s3.upload_file(Filename="data/item-demand-time.csv", Bucket=bucketName, Key=key)

Dataset import details

Following are the details needs to be provided for Import task.

Dataset import name
Select time zone Info
- My dataset does not have TZ as any variable and so I have selected the option of do
Data location Info
- This is input file path from my S3 bucket which needs to be provided
IAM Role Info
- Dataset groups require permissions from IAM to read your dataset files in S3.

Now it will give us option to import and once this is done then we should get Successfully imported your data.

Train a predictor

Train a predictor, a custom model with underlying infrastructure that Amazon Forecast trains on your datasets.
Following are the key parameters required here.

Additional configurations to be set during this phase include

How the training dataset is to be split into training and testing dataset ?
How the missing data is to be addressed ?
How the model validation is to be performed (i.e., back test window in the context of time-series analysis)
How many times the model validation is to be performed during the model training phase (i.e., number of back test windows in the context of time-series analysis)
What is the forecast horizon ?

Predictor settings

Forecast name
Forecast horizon
- This number tells Amazon Forecast how far into the future to predict your data at the specified forecast frequency.
Forecast frequency
- My data set has timestamp daily and hence i have set this as 1 day

Predictor details

Algorithm selection
- Here i have selected the option of AutoML which allows me to let Amazon Forecast choose the right algorithm for dataset.
Optimization metric
- I have selected default here.
- Amazon Forecast provides Root Mean Square Error (RMSE), Weighted Quantile Loss (wQL), Average Weighted Quantile Loss (Average wQL), Mean Absolute Scaled Error (MASE), Mean Absolute Percentage Error (MAPE), and Weighted Absolute Percentage Error (WAPE) metrics to evaluate your predictors.
Forecast dimensions
- Item id is used in training by default and that has been added as mandatory by Forecast. Additionally I have selected Store because my aim is to have forecast based on store and item id.
Forecast type
- Choose up to 5 quantiles between 0.01 and 0.99 (by increments of 0.01). AWS allows us to have by default 0.1,0.5 and 0.9.

Advanced Configuration

Here this is the default FeaturizationMethod being recommended by Amazon Forecast. Provides information about the method that featurizes (transforms) a dataset field.

Here this method is only being applied for my element demand which is specified by AttrubuteName.

[
    {
        "AttributeName": "demand",
        "FeaturizationPipeline": [
            {
                "FeaturizationMethodName": "filling",
                "FeaturizationMethodParameters": {
                    "aggregation": "sum",
                    "frontfill": "none",
                    "middlefill": "zero",
                    "backfill": "zero"
                }
            }
        ]
    }
]

Supplementary features

This is quite crucial information and can impact business problem sometime.

Weather info
- Amazon Forecast Weather Index combines multiple weather metrics from historical weather events and current forecasts at a given location to increase your demand forecast model accuracy.
- In retail inventory management use cases, day-to-day weather variation impacts foot traffic and product mix.
Holiday info

Create a Forecaster

Once the Predictor is trained, it is to be prepared to provide the forecasting.

create_forecast_response = forecast.create_forecast(
                           ForecastName=forecastName,
                           PredictorArn=predictorArn)

Following key inputs needs to be provided from console and then we can start the process of forecast.

Forecast name
Predictor (This is the one was created in earlier step)
Forecast types
- By default, Amazon Forecast will generate forecasts for 0.10, 0.50 and 0.90 quantiles.

Make Forecasts

Now we are ready to make forecasts. In our case, we are going to write the forecasted outputs back in S3 bucket.

Following inputs needs to be provided.

Start date Info
End date

I have given below a snapshot of the forecasts which I got using the Predictor that I trained.

DEV Community: Amit Kayal

Building a Hybrid AWS Microservices Platform with API Gateway, Lambda, ECS, and Load Balancers

Building a Hybrid AWS Microservices Platform with API Gateway, Lambda, ECS, and Load Balancers

Introduction

The Core Pattern

Why Combine Lambda and ECS?

A Three-Stage Infrastructure Model

Stage 1: Networking

Stage 2: Compute

Stage 3: API Gateways

The Request Path for ECS Services

API Gateway

Why a VPC Link Is Used

Why the Repository Uses Both NLB and ALB

How Load Balancing Works

Target Groups

Listener Rules

Health Checks and Traffic Protection

How ECS Is Structured

ECS Cluster

Reusable ECS Service Module

Task Definitions

Network Mode

Subnet and Security Group Design

Subnet Placement

Security Groups

Config-Driven Service Onboarding

Container Delivery with ECR

How Lambda Fits into the Platform

1. Lambda as an API Backend

2. Lambda as a Platform Support Function

Authentication and API Protection

Public and Private API Models

Public API

Private API

Observability and Operations

Scaling Characteristics

API Layer Scaling

ECS Scaling

Platform Growth

Alignment with AWS Well-Architected Best Practices

Operational Excellence

Security

Reliability

Performance Efficiency

Cost Optimization

Sustainability and Maintainability

Why This Pattern Works Well

Practical Implementation Advice

Conclusion

Lessons Learned

Building a Practical Lambda Capacity Provider Platform: Lessons Learned from Warm Pools, Version Hygiene, and CI/CD Reality

Building a Practical Lambda Capacity Provider Platform: Lessons Learned from Warm Pools, Version Hygiene, and CI/CD Reality

The Real Problem We Were Solving

How We Actually Created the Capacity Provider

Lesson 1: A Capacity Provider Is Not a Tuning Knob. It Is an Operating Model.

Lesson 2: ARM64 Delivers Real Value, but Only if You Respect Service Constraints

Lesson 3: Warmup Is Not a Hack. It Is a Deliberate Control Loop.

Lesson 4: Shared Pools Create Efficiency, but They Also Create Coupling

Lesson 5: If You Publish Versions Aggressively, You Need Lifecycle Hygiene on Day One

Lesson 6: GitHub Actions Should Orchestrate. CodeBuild Should Execute.

Lesson 7: CI/CD Maturity Is Not About Having a Pipeline. It Is About Where the Gates Actually Are.

Lesson 8: Documentation Drift Is a Reliability Signal

Lesson 9: The Default VPC Is Fine for Speed, but It Should Be a Conscious Temporary Convenience

Lesson 10: Least Privilege Usually Loses the First Battle. Do Not Let It Lose the War.

What This Repo Gets Right

What I Would Improve Next

The Bigger Technical Lesson

Closing Thought

Lessons I learned building a memory-aware agent with Amazon Bedrock AgentCore Runtime

Lessons I learned building a memory-aware agent with Amazon Bedrock AgentCore Runtime

Lesson 1: An agent is not really multi-turn until memory is part of the lifecycle

Lesson 2: The cleanest pattern is explicit memory, not implicit transcript magic

Lesson 3: Hooks are where memory belongs

Lesson 4: Identity is the real memory boundary

Why actor_id matters

Why session_id matters

Lesson 5: The agent should be rebuilt per request, but memory should persist across requests

Lesson 6: Deployment is part of the memory design

Lesson 7: Short-term memory and long-term memory solve different problems

Why `actor_id` matters

Why `session_id` matters