In this post, I'm sharing how to extract meaningful data from your CloudWatch logs.
Logs are an essential part of troubleshooting application and infrastructure performance. Having good logging and monitoring infrastructure is crucial for making your application production-ready. With logs, you can track the events that have happened in your application, which is helpful when debugging issues. But can you do more with logs? Can you extract meaningful data from your CloudWatch logs? Read on!
Logs and Metrics
Amazon CloudWatch monitors your Amazon resources and the applications you run on AWS in real time. CloudWatch collects monitoring and operational data in the form of logs, metrics, and events. Amazon CloudWatch Logs is a centralized place for the logs from all of your applications, and AWS services that you use. When starting with CloudWatch it is important to understand these core concepts:
- Log Event. A log event is a record of some activity recorded by the application or resource being monitored.
- Log Stream. A log stream is a sequence of log events that share the same source; it represents the sequence of events coming from the application instance or resource being monitored. For example, a separate log stream is created for each instance of a Lambda function.
- Log Group. Log groups are groups of log streams that share the same retention, monitoring, and access control settings. An example would be, a log group for a Lambda function that contains all related log streams.
- Metric. A Metric is a time-ordered set of data points. Metrics are organized into namespaces. By default, many AWS services provide free metrics for resources by default, also you can publish your own application metrics to CloudWatch, known as Custom Metrics.
Metric Filters and Custom Metrics
You can search and filter the log events coming into CloudWatch Logs by creating one or more metric filters. Metric filters define the terms and patterns to look for in log data as it is sent to CloudWatch Logs. CloudWatch Logs uses these metric filters to turn log data into numerical CloudWatch metrics.
Metric Filters are created for a specific log group. When a metric filter matches a term, it increments the metric's count. For example, you can create a metric filter that counts how many times the word "Error" occurs in your log events, and an appropriate metric will be added to the CloudWatch metric namespace.
You can create very simple matching filters, like finding a word match and use complex pattern syntax for extracting values from JSON log events and space-delimited log events, you can even use values in log events to increment a metric's value. More details can be found in the AWS documentation.
Metric filters are applied to log events as new log events are created. But what if you need to do processing over multiple log events for a particular time period, like counting occurrences of unique values or correlating data from different log groups? In that case, you can publish your Custom Metrics to CloudWatch. With Custom Metrics, you can push your own metrics to CloudWatch using the AWS CLI or an API. By using CloudWatch API, you can filter and fetch log data, do the needed processing on the log data to calculate the metric value, and then push it to CloudWatch.
Metrics, graphs, and alarms
Amazon CloudWatch metrics can be used for search, graphing, and alarms.
To search your metrics you can use CloudWatch Metrics Insights. This service provides a powerful high-performance SQL query engine that you can use to query your metrics at scale and identify trends and patterns within all of your CloudWatch metrics in real-time.
Also, you can create graphs from your metrics, which makes it more efficient to see the metric activity over time.
Additionally, you can get notified when some metric reaches or goes below a threshold by setting up an AWS CloudWatch alarms.
Top comments (0)