DEV Community: Kadiri George

Saving AWS ECS CloudWatch Cost.

Kadiri George — Sat, 03 Jan 2026 14:02:22 +0000

As a DevOps Engineer who use a managed Docker runtime on AWS like Elastic Container Service to run containerize application the most easy to view logs is CloudWatch Logs.
AWS CloudWatch costs for ECS stem from data ingestion (logs/metrics), storage, analysis (Log Insights), and extra features like Container Insights, with ingestion at ~$0.50/GB (standard) and Log Insights at ~$0.005/GB scanned, while free tiers cover basic metrics and limited data, but high-volume container logs and detailed monitoring (like Container Insights) can significantly increase bills, requiring cost optimization via filtering, retention, and using the AWS Pricing Calculator. Container Insights Provides enhanced metrics for ECS/EKS, adding to costs but offering deep visibility. All these increasing CloudWatch cost may spike up Thousands of dollars monthly, which will make one look for alternate ways of streaming logs from applications.
FireLens is an AWS-provided log router for Amazon ECS/Fargate that uses Fluentd or Fluent Bit as a sidecar container to flexibly send container logs to various destinations like CloudWatch, S3, or third-party tools, simplifying complex log routing without changing application code. It works by adding a special logging configuration to your ECS task definition, allowing you to route logs from your main app container through the FireLens sidecar for processing and forwarding. Our goal here is to stream the applications logs to S3 bucket without changing the application code and save CloudWatch Logs cost.

Here is how it works:
⦁ ECS Task Definition: You configure your ECS task definition to use the FireLens log driver for your application container.
⦁ Sidecar Container: FireLens adds a sidecar container (running Fluent Bit or Fluentd) to your task.
⦁ Log Routing: Your application container sends logs to standard output (stdout/stderr), and FireLens intercepts these, processing them based on your configuration.
⦁ Pluggable Architecture: It uses plugins (like AWS for Fluent Bit) to send logs to destinations like CloudWatch, S3, or other endpoints that support JSON over HTTP, Fluentd Forward, or TCP.
The key benefits are
⦁ Route logs to multiple destinations for storage, analysis, or monitoring.
⦁ Efficiently handles log management at scale within ECS environments.
⦁ Route logs to multiple destinations for storage, analysis, or monitoring.
⦁ Easily manage logs without modifying application code or manually installing agents.
Steps for the configurations

Create an S3 Bucket First, ensure you have an S3 bucket ready for your logs.
Update IAM Task Execution Role Your ECS task execution role needs permissions to write to S3: { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:PutObject", "s3:PutObjectAcl" ], "Resource": "arn:aws:s3:::your-log-bucket/*" } ] }
Configure Your ECS Task Definition Here's an example task definition with FireLens configured for S3: { "family": "your-task-family", "networkMode": "awsvpc", "requiresCompatibilities": ["FARGATE"], "cpu": "256", "memory": "512", "executionRoleArn": "arn:aws:iam::account-id:role/ecsTaskExecutionRole", "taskRoleArn": "arn:aws:iam::account-id:role/ecsTaskRole", "containerDefinitions": [ { "name": "log_router", "image": "amazon/aws-for-fluent-bit:latest", "essential": true, "firelensConfiguration": { "type": "fluentbit", "options": { "enable-ecs-log-metadata": "true" } }, "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-group": "/ecs/firelens-container", "awslogs-region": "us-east-1", "awslogs-stream-prefix": "firelens" } } }, { "name": "app", "image": "your-app-image", "essential": true, "logConfiguration": { "logDriver": "awsfirelens", "options": { "Name": "s3", "region": "us-east-1", "bucket": "your-log-bucket", "total_file_size": "10M", "upload_timeout": "1m", "s3_key_format": "/logs/%Y/%m/%d/%H_%M_%S", "store_dir": "/tmp/fluent-bit/s3" } } } ] }

Key Configuration Options
S3 Output Plugin Options:

bucket: Your S3 bucket name
region: AWS region
total_file_size: Size of file before uploading (e.g., "10M")
upload_timeout: How often to upload (e.g., "1m")
s3_key_format: Path structure in S3 (supports time formatting and tags)
store_dir: Temporary storage location

Useful s3_key_format variables:
%Y/%m/%d: Date formatting
%H_%M_%S: Time formatting

Deploy Your Task Deploy the updated task definition to your ECS Fargate service. FireLens will automatically: Capture logs from the application container Buffer them in the log router container Stream them to S3 based on the configuration

After streaming the logs to S3 bucket, we now run into another issue which is how to view the logs to check for errors, 4xx and 5xx errors in the applications.

AWS gives us another service which is Amazon Athena. Amazon Athena is a serverless, interactive query service in AWS that lets you analyze large datasets directly in Amazon S3 using standard SQL, without needing to load data into a database. It's known for its simplicity (just point to data, define schema, and query), pay-per-query cost model (only pay for data scanned), and speed, making it ideal for ad-hoc analysis, log analysis, and exploring data lakes. We will use Amazon Athena to query the logs in the S3 bucket using standard query language.
Steps on how to use Athena to view Logs
Step 1
-- Create database
CREATE DATABASE IF NOT EXISTS ecs_logs_db;

-- Create table for JSON logs
CREATE EXTERNAL TABLE IF NOT EXISTS ecs_logs_db.log_table (
log STRING,
container_id STRING,
container_name STRING,
ecs_cluster STRING,
ecs_task_arn STRING,
ecs_task_definition STRING,
source STRING,
time STRING
)
ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'
WITH SERDEPROPERTIES (
'ignore.malformed.json' = 'true'
)
LOCATION 's3://ecs-logs-container/logs/'
TBLPROPERTIES ('has_encrypted_data'='false');

If your logs are plain text (not JSON):
-- Create database
CREATE DATABASE IF NOT EXISTS ecs_logs_db;

-- Create table for plain text logs
CREATE EXTERNAL TABLE IF NOT EXISTS ecs_logs_db.log_table (
log_line STRING
)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\n'
STORED AS TEXTFILE
LOCATION 's3://ecs-logs-container/logs/';

Step 2: Query the Logs
Basic Search for Errors:

SELECT *
FROM ecs_logs_db.log_table
WHERE log LIKE '%error%'
OR log LIKE '%ERROR%'
OR log LIKE '%exception%'
LIMIT 100;

Search with Time Filter (using S3 path partitions):

SELECT *
FROM ecs_logs_db.log_table
WHERE log LIKE '%error%'
AND "$path" LIKE '%2025/12/24%'
LIMIT 100;

Count Errors by Pattern:

SELECT
CASE
WHEN log LIKE '%404%' THEN '404 Error'
WHEN log LIKE '%500%' THEN '500 Error'
WHEN log LIKE '%exception%' THEN 'Exception'
ELSE 'Other Error'
END AS error_type,
COUNT(*) AS error_count
FROM ecs_logs_db.rexhub_logs
WHERE log LIKE '%error%' OR log LIKE '%exception%'
GROUP BY 1
ORDER BY error_count DESC;

Search for Specific Text:

SELECT *
FROM ecs_logs_db.log_table
WHERE log LIKE '%AccessDenied%'
LIMIT 50;

Get Recent Logs:
SELECT *
FROM ecs_logs_db.log_table
WHERE "$path" LIKE '%2025/12/24%'
ORDER BY "$path" DESC
LIMIT 100;

For better performance, partition the table by date:
-- Drop the old table
DROP TABLE IF EXISTS ecs_logs_db.log_table;

-- Create partitioned table
CREATE EXTERNAL TABLE IF NOT EXISTS ecs_logs_db.log_table (
log_line STRING
)
PARTITIONED BY (
year STRING,
month STRING,
day STRING
)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\n'
STORED AS TEXTFILE
LOCATION 's3://ecs-logs-container/logs/';

-- Add partitions
ALTER TABLE ecs_logs_db.log_table ADD
PARTITION (year='2025', month='12', day='24')
LOCATION 's3://ecs-logs-container/logs/2025/12/24/';

Then query with partitions:

SELECT *
FROM ecs_logs_db.log_table
WHERE year='2025'
AND month='12'
AND day='24'
AND log_line LIKE '%error%'
LIMIT 100;

I wrote this article because CloudWatch costs caught me off guard on a previous project. What started as a few dollars quickly grew into a noticeable line item on our AWS bill. After digging into the details and testing different approaches, I uncovered several strategies that significantly reduced our logging costs without sacrificing visibility into our ECS services.

I hope that sharing these lessons helps you save both time and money, whether you’re just setting up ECS monitoring or optimizing an existing setup.

I would love to hear your experience with CloudWatch costs on ECS. Have you found other optimization strategies that worked well? Feel free to drop your questions or insights in the comments— I am happy to discuss specific scenarios, and your input may help others facing similar challenges.

Jenkins: The Legacy CI that shaped Modern DevOps

Kadiri George — Sat, 06 Dec 2025 13:40:58 +0000

If you’ve been in DevOps for a while, chances are your CI/CD journey started with Jenkins. For me, I started with AWS CodePipeline which is propriety to AWS. It is very easy to use and setup but it lacks customization and integrations with third party tools.

When I started using Jenkins and I was able to customize my pipeline and integrate with other tools by downloading their plugin waoh - it felt like magic.

Jenkins earned its reputation as the go-to CI tool because it was open-source, flexible, and had a plugin for almost everything. Entire engineering teams built their automation practices on top of it, and for years, it was the backbone of software delivery pipelines. Jenkins might be legacy CI tool but it still shape today modern DevOps world.

But with that flexibility came challenges. Managing plugins, maintaining servers, and scaling Jenkins across teams often felt like a job of its own. Jenkins was designed before containers and cloud-native patterns became the norm, which makes it feel more “legacy” in today’s fast-moving DevOps world.

Now, with tools like GitHub Actions, GitLab CI/CD, and cloud-native solutions like ArgoCD, teams can build pipelines that are easier to manage, faster to scale, and better aligned with modern workflows.

That said, Jenkins still has a place in many organizations — especially where deep customization and legacy workloads exist. It’s hard to ignore the impact it’s had on DevOps as a practice.

I’m curious: is Jenkins still part of your CI/CD setup, or have you fully transitioned to newer tools?

AWS CLOUDWATCH VS UNIFIED CLOUDWATCH AGENT

Kadiri George — Sun, 05 Jan 2025 21:22:58 +0000

Amazon CloudWatch provides a reliable, scalable, and flexible monitoring solution that you can start using within minutes. You no longer need to set up, manage, and scale your own monitoring systems and infrastructure.

CloudWatch Metrics for EC2

AWS Provided Metric (AWS pushes them)

· Basic monitoring(default): Metrics are collected at 5 minutes interval.

· Detailed monitoring(paid): Metrics are collected at 1 minute interval.

Metrics includes CPU, Network, Disk and Status checks metrics.

Custom Metric (Yours to push)

· Basic Resolution: 1 minute resolution.

· High Resolution: all the way to 1 second resolution.

· Include RAM, application-level metrics.

· Make sure the IAM permission on the EC2 instance role are correct.

EC2 Included Metrics

· CPU: It include CPU utilization, credit usage and balance.

· Network: It include Network in/out of the instance

· Status check:

Instance status: It checks the metrics of the EC2 virtual machine.
System status: It checks the underlying hardware.

· Disk: It include Read/Write for Ops/Bytes (only for instance store)

Note: RAM is not included in the AWS EC2 metrics. To view metrics of EC2 RAM one need to use Unified CloudWatch Agent.

Unified CloudWatch Agent

It is used to collect addition system-level such as RAM, processes, used disk space, from EC2 instances, on premises server. The collected logs are sent to CloudWatch logs, by default no logs inside your EC2 instance will be sent to CloudWatch logs without using an agent.

Unified CloudWatch Agent – procstat plugin

It collects metrics and monitor system utilization of individual processes and support both Linux and Window servers. Examples amount of time the processes use CPU, amount of the memory the process uses….

Steps to collect metrics and logs from Amazon EC2 instances and on- premises server with CloudWatch agent.

A. Attach the appropriate role on your EC2 instance which are:

```
   CloudWatchAgentAdminPolicy
```
```
   CloudWatchAgentServerPolicy
```

B. Connect to your EC2 instance and run this command.

sudo yum install amazon-cloudwatch-agent and run the wizard to get started with this command sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard, follow the processes and select default choice in places you are not sure of what to select. Then specify your log file path e.g. /var/log/httpd/access_log and /var/log/httpd/error_log. I specify this path because I am running HTTPD on my server so yours may different if you are running others like apache and the likes on your server. You can also save the config file in AWS SSM Parameter Store for future use using this CloudWatchAgentAdminPolicy Role on your instance will allow put to SSM Parameter Store.

One can use this command to use the config stored in the SSM Parameter Store on a new instance.

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c ssm:AmazonCloudWatch-linux -s then check the CloudWatch log groups to find the access_log and error_log and start monitoring and also check CloudWatch metric to find CWAgent to see other metrics to monitor.

Thanks for reading.

WORKING WITH AWS CLOUD DEVELOPMENT KIT (CDK)

Kadiri George — Sun, 03 Nov 2024 13:43:54 +0000

AWS CDK is an open-source software development framework used to model and provision your cloud applications resources with familiar programming languages such as Python, Java, .NET, Go and Typescript.
Working with AWS CDK helps to deploy infrastructure as code with one favorite programming language instead of using CloudFormation which uses either JSON or YAML format.
When using Typescript for the CDK one needs to install Node.js on the PC.
CDK Building Block
cdk init - cdk init app – Language typescript. It will initialize the script with the prefer programming language in this case it is typescript.
cdk bootstrap – It is use only once for any deployment with CDK to the account of deployment.
cdk deploy – Deploy this stack to your default AWS account.
cdk destroy – Destroy this stack in your default AWS account.
cdk diff – Compare deployed stack with current stack.
cdk synth - Synthesizes and prints the CloudFormation template for one or more specified stacks.
With just a line of code a line of code I was to deploy an S3 bucket into my account.
const bucket = new s3.Bucket(this, 'MyFirstBucket'{bucketName:’myserverless-kadiri’});
CDK improve infrastructure and business logic. It automates AWS services provision with construct.
I was able to use AWS CDK to deploy an EC2 instance which runs python application, connect to my GITHUB repository and able to use it deploy with CodePipeline, CodeBuild and CodeDeploy. With all the resources I created, I was not bothered about the cost on AWS because with cdk destroy I can delete all the resources created. It can also be helpful to create a stagging services which can be deleted every night and started again every morning to save cost.
Useful link to get started with AWS CDK
https://lnkd.in/gTDaGcBi
https://lnkd.in/gCU23YgC
Thanks for reading.