Log Analytics (The "Observability") using AWS services

Log Analytics (The "Observability" Use Case) using AWS servics

The Problem: You have 1,000 ECS containers. One of them just crashed. Which one? Why?
The Data: Error logs, access logs, system logs.
The Requirement: You need to search through billions of lines of text to find a specific timestamp or error code.

Solution :
Resolve the problem by having this Data Flow Architecture :
ECS (App) → CloudWatch Logs → Lambda 1 (The Producer) → SQS Queue → Lambda 2 (The Consumer) → OpenSearch

Step 1: ECS to CloudWatch (The Configuration)
In your ECS Task Definition, you don't write code for this. You just set the logConfiguration:
code
JSON
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/ecs/my-app",
"awslogs-region": "us-east-1",
"awslogs-stream-prefix": "ecs"
}
}
Step 2: CloudWatch to Lambda 1 (The Trigger)
CloudWatch Logs use a Subscription Filter to trigger Lambda.
The Format: CloudWatch sends data in a Base64 encoded and Gzipped format.
Lambda 1 Logic: It must decode and decompress the data, then loop through the log events and send them to SQS.
The Event Format (What Lambda 1 receives):
code
JSON
{
"awslogs": {
"data": "H4sIAAAAAAAAA... (Compressed Data)"
}
}
Step 3: Lambda 1 to SQS (The Producer)
Lambda 1 sends a message to SQS. You should send the log as a JSON string.
The SQS Message Format (What you send to SQS):
code
JSON
{
"timestamp": "2023-10-27T10:00:00Z",
"message": "User logged in",
"logGroup": "/ecs/my-app",
"container_id": "abcd-1234"
}
Step 4: SQS to Lambda 2 (The Buffer Logic)
AWS makes this easy: You set SQS as a trigger for Lambda 2.
Batch Size: You can tell Lambda 2 to wait until there are 10 messages in SQS or 5 minutes have passed before it runs. This saves money and makes OpenSearch happy (writing in batches is faster than writing one-by-one).
The Event Format (What Lambda 2 receives from SQS):
code
JSON
{
"Records": [
{
"body": "{\"timestamp\": \"...\", \"message\": \"User logged in\"}",
"receiptHandle": "MessageReceiptHandle",
"eventSource": "aws:sqs"
}
]
}
Step 5: Lambda 2 to OpenSearch (The Final Step)
Lambda 2 takes the "body" from the SQS record and sends it to the OpenSearch Endpoint using an HTTP POST request.
The OpenSearch API Format:
code
Http
POST https://your-opensearch-endpoint.com/my-index/_doc/
Content-Type: application/json
{
"timestamp": "2023-10-27T10:00:00Z",
"message": "User logged in",
"source": "ecs-task-01"
}
Why this is the "Gold Standard" for Production:
Reliability: If OpenSearch is down, messages stay safely in SQS for up to 14 days.
Concurrency Control: You can limit Lambda 2 so it only runs, for example, 5 times at once. This prevents "DDoS-ing" your own OpenSearch database.
Cost: You only pay when logs are actually moving.

Checklist to try out above solution:

• Create an OpenSearch Domain (use a t3.small.search instance to keep it cheap).
• Create an SQS Standard Queue.
• Create Lambda 1: Add a CloudWatch Logs trigger. Give it IAM permission to sqs:SendMessage.
• Create Lambda 2: Add an SQS trigger. Give it IAM permission to es:ESHttpPost.
• Run a "Hello World" in ECS: Or just manually push a log to CloudWatch to see the chain reaction.

Summary :
The most important thing to remember is that CloudWatch data is compressed. If you send it straight from Lambda 1 to SQS without decompressing it, you will just have a bunch of "garbage text" in your queue that you can't search. Always decompress in the first Lambda.