DEV Community

Cover image for πŸ“Š Implementing Cloud Monitoring and Logging πŸ› οΈ
Madhurima Rawat
Madhurima Rawat

Posted on • Originally published at github.com

πŸ“Š Implementing Cloud Monitoring and Logging πŸ› οΈ

#cloud #tutorial #cloudcomputing #aws

πŸ“Š Hello Access Managers!!

From setting secure roles to watching them in action β€” let’s move from IAM to real-time monitoring and logging! πŸ”βž‘οΈπŸ“Š

πŸ“š All code, docs, and resources are available in my GitHub repository:

GitHub logo madhurimarawat / Cloud-Computing

This repository focuses on cloud computing and demonstrates how to set up virtual machines, S3, and other services using LocalStack. It provides a comprehensive guide to simulating AWS services locally for development and testing purposes.

Cloud-Computing

This repository focuses on cloud computing and demonstrates how to set up virtual machines, S3, and other services using LocalStack. It provides a comprehensive guide to simulating AWS services locally for development and testing purposes.

Repo Size GitHub Stars GitHub Forks GitHub Issues Closed Issues Open Pull Requests Closed Pull Requests GitHub Discussions GitHub Contributors Top Language License Last Commit Repository Age Workflow Status GitHub Watchers

Tools and Technologies βš™οΈπŸ’»

1. AWS CLI

AWS Command Line Interface (CLI) is a powerful tool that allows users to interact with AWS services directly from the terminal. It simplifies managing cloud resources by providing commands for a wide range of AWS services, enabling tasks such as provisioning, managing, and automating workflows with ease.

2. LocalStack

LocalStack is a fully functional, local testing environment for AWS services. It enables developers to simulate AWS services on their local machines, facilitating the development and testing of cloud-based applications without needing access to an actual AWS account.

3. Docker

Docker is a containerization platform that allows developers to build, share, and run applications in isolated environments called…


View on GitHub


In the last post,
madhurima_rawat

πŸ‘₯ Cloud Identity and Access Management (IAM) πŸ”“

Madhurima Rawat ・ May 17

#cloud #cloudcomputing #aws #iam

we explored Cloud Identity and Access Management (IAM) and how to secure your cloud infrastructure with fine-grained access control.

Today, we’re gonna dive into something equally vital,
πŸ“Š Cloud Monitoring and Logging!

First, I’ll explain what monitoring and logging actually mean, why they’re absolutely essential in any cloud setup, and share some real-life use cases that show their impact.
Then we’ll move ahead with the implementation.

What is AWS CloudWatch?

AWS CloudWatch is a monitoring and observability service that collects and tracks metrics, logs, and events to provide insights into cloud resources and applications.

πŸ”Ή Key Features of CloudWatch:

  • Metrics Monitoring πŸ“Š: Collects and analyzes performance metrics for AWS services.
  • Log Management πŸ“œ: Stores and processes logs for applications and infrastructure.
  • Alarms & Notifications 🚨: Triggers alerts based on threshold values.
  • Dashboards & Visualization πŸ“ˆ: Provides real-time monitoring dashboards.
  • Event-driven Automation βš™οΈ: Automates actions based on events.

Understanding Log Management

Log management is a crucial part of cloud monitoring and involves the collection, storage, and analysis of logs.

πŸ’‘ Six Components of Log Management:

  1. Log Collection πŸ—οΈ: Logs are gathered from various sources (servers, applications, cloud services).
  2. Log Aggregation πŸ—ƒοΈ: Logs are combined and stored in a centralized system like CloudWatch Logs.
  3. Log Storage 🏒: Logs are kept in CloudWatch Log Groups and Log Streams.
  4. Log Analysis πŸ”: Tools like AWS CloudWatch Insights analyze log data for patterns and trends.
  5. Log Monitoring & Alerts 🚨: CloudWatch can send alerts based on predefined conditions.
  6. Compliance & Retention βœ…: Logs are archived based on security and compliance requirements.

What Are Log Streams?

Log streams represent a sequence of log events from the same source (e.g., an application instance, EC2 instance, or microservice).

πŸ”Ή Example:

  • A web server might have separate log streams for each instance it runs on.
  • CloudWatch logs are organized in log groups, and each log group contains multiple log streams.

Real-life Use Case: CloudWatch Logs & Metrics

Scenario:

A company hosts its web application on AWS EC2 instances and wants to monitor CPU usage and analyze logs for errors.

Steps in AWS CloudWatch:

1️⃣ Enable CloudWatch Logs on EC2 instances to store application logs.

2️⃣ Create a log group /my/app/logs and define log streams for different instances.

3️⃣ Send logs & metrics (e.g., CPU usage, errors, response time) to CloudWatch.

4️⃣ Set up alarms to notify the DevOps team when CPU usage exceeds 80%.

5️⃣ Analyze logs using CloudWatch Insights to detect anomalies.

πŸ–ΌοΈ About the Cover Image:
It begins with a speedometer icon, representing metrics and performance monitoring β€” capturing real-time data like logs, usage stats, and resource health.
Next, a checkbox with a right tick symbolizes that all essential cloud services are being actively monitored and running smoothly βœ….
Following that, a warning icon appears to indicate that if anything goes wrong, the system will trigger alerts and notifications ⚠️.
Finally, a heart with a heartbeat line represents the idea of continuous, real-time monitoring β€” ensuring your cloud stays alive, healthy, and responsive πŸ’“πŸ“‘.

AWS CLI Commands for LocalStack (CloudWatch and Logs)

1. Create a Log Group 

aws --endpoint-url=http://localhost:4566 logs create-log-group
--log-group-name /my/app/logs
Enter fullscreen mode Exit fullscreen mode

Description:

  • Creates a log group named /my/app/logs in AWS CloudWatch Logs.

2. Create a Log Stream 

aws --endpoint-url=http://localhost:4566 logs create-log-stream
--log-group-name /my/app/logs --log-stream-name my-stream
Enter fullscreen mode Exit fullscreen mode

Description:

  • Creates a log stream named my-stream within the /my/app/logs log group.

3. Put Log Events 

aws --endpoint-url=http://localhost:4566 logs put-log-events --
log-group-name /my/app/logs --log-stream-name my-stream --log-
events "[{\"timestamp\":1741348140000,\"message\":\"Test log entry\"}]"
Enter fullscreen mode Exit fullscreen mode

Description:

  • Adds a log entry with a timestamp (1741348140000) and message (Test log entry) to the my-stream log stream.

Output:

-----------------------------------------------------------------------------------
|                                  PutLogEvents                                   |
+--------------------+------------------------------------------------------------+
|  nextSequenceToken |  00000000000000000000000000000000000000000000000000000001  |
+--------------------+------------------------------------------------------------+
||                             rejectedLogEventsInfo                             ||
|+-------------------------------------------------------------------+-----------+|
||  tooNewLogEventStartIndex                                         |  0        ||
|+-------------------------------------------------------------------+-----------+|
Enter fullscreen mode Exit fullscreen mode

4. Put Metric Data 

aws --endpoint-url=http://localhost:4566 cloudwatch put-metric-
data --namespace "MyApp" --metric-name "CPUUsage" --value 75
Enter fullscreen mode Exit fullscreen mode

Description:

  • Publishes a custom CloudWatch metric CPUUsage with a value of 75 in the MyApp namespace.

5. List CloudWatch Metrics 

aws --endpoint-url=http://localhost:4566 cloudwatch
list-metrics --namespace "MyApp"
Enter fullscreen mode Exit fullscreen mode

Description:

  • Lists all CloudWatch metrics under the MyApp namespace.

Output:

-------------------------------
|         ListMetrics         |
+-----------------------------+
||          Metrics          ||
|+-------------+-------------+|
|| MetricName  |  Namespace  ||
|+-------------+-------------+|
||  CPUUsage   |  MyApp      ||
|+-------------+-------------+|
Enter fullscreen mode Exit fullscreen mode

6. Describe Log Groups 

aws --endpoint-url=http://localhost:4566 logs describe-log-groups
Enter fullscreen mode Exit fullscreen mode

Description:

  • Retrieves details of all log groups available in CloudWatch Logs.

Output:

-----------------------------------------------------------------------------------------
|                                   DescribeLogGroups                                   |
+---------------------------------------------------------------------------------------+
||                                      logGroups                                      ||
|+-------------------+-----------------------------------------------------------------+|
||  arn              |  arn:aws:logs:us-east-1:000000000000:log-group:/my/app/logs:*   ||
||  creationTime     |  1741322075726                                                  ||
||  logGroupName     |  /my/app/logs                                                   ||
||  metricFilterCount|  0                                                              ||
||  storedBytes      |  0                                                              ||
|+-------------------+-----------------------------------------------------------------+|
Enter fullscreen mode Exit fullscreen mode

7. Retrieve Log Events 

aws --endpoint-url=http://localhost:4566 logs get-log-events
--log-group-name /my/app/logs --log-stream-name my-stream
Enter fullscreen mode Exit fullscreen mode

Description:

  • Fetches the log events from my-stream under /my/app/logs.

Output:

-------------------------------------------------------------------------------------
|                                   GetLogEvents                                    |
+--------------------+--------------------------------------------------------------+
|  nextBackwardToken |  b/00000000000000000000000000000000000000000000000000000000  |
|  nextForwardToken  |  f/00000000000000000000000000000000000000000000000000000000  |
+--------------------+--------------------------------------------------------------+
Enter fullscreen mode Exit fullscreen mode

Want to see how it all worked step by step? Check it out here:
πŸ”— Experiment 9 Output (PDF)

🧠 Curious about how each command runs and responds? See the detailed input-output flow here:
πŸ–₯️ Monitoring Setup Input-Output Flow (PDF)

πŸŽ‰ And that’s a wrap on Cloud Monitoring and Logging!

πŸ“š Found great monitoring tools, cheat sheets, or observability hacks? Drop them in the comments. I’d love to check them out and share with the community!

πŸš€ Stay tuned for the next article!
Up next, we’ll dive into Setting Up a Cloud-based CI/CD Pipeline β€” where you’ll learn how to automate deployment and streamline your cloud workflows like a pro. πŸ”„β˜οΈπŸ’»

Top comments (5)

Collapse
 
nathan_tarbert profile image
Nathan Tarbert •

Pretty cool seeing someone break it down like this - real hands-on beats reading docs any day.

Collapse
 
madhurima_rawat profile image
Madhurima Rawat •

Thanks! I also believe real hands-on is the best way to learn.

Collapse
 
surxjitds profile image
surxjit •

Hey

Collapse
 
madhurima_rawat profile image
Madhurima Rawat •

Hello, how was the article? Did you liked it?

Collapse
 
surxjitds profile image
surxjit •

Yup! Glad :)