DEV Community: ChrisCooney1

All the useful Cloudwatch CLI Commands

ChrisCooney1 — Tue, 13 Oct 2020 16:28:58 +0000

With its ability to provide a unified view of your application’s health, backed up by data in the form of logs, metrics and events, CloudWatch proves AWS’s leading position in cloud computing. That being said, CloudWatch has its pain points. Here are ten commands to help you through any rough spots. They run on the AWS CLI, which can be installed on Mac, Windows and Linux.

Logs

CloudWatch’s logging system has two fundamental units. Log streams are sequences of logging events that come from a single source. Log groups are convenient boxes to put bundles of log streams in.

create-log-group

This allows users to create log groups to put log streams in. When a log group is created, its default behaviour is that log events never expire.

Log group names can be between 1 and 512 characters in length and must consist entirely of valid characters. These are lowercase and capital letters a-z, the numbers 0-9, '_' (underscore), '-' (hyphen), '/' (forward slash), '.' (period), and '#' (number sign).

Additionally they need to be unique within a given AWS account’s geographic region. To explain what this entails, we need to understand the structure of an AWS account. When a user creates an account, its AWS resources are parcelled up into geographic areas called regions and each account can have multiple regions enabled.

When you create a log group for the region ‘Europe (Milan)’, for example, your account must have no other log groups of the same name in ‘Europe (Milan)’.

create-log-stream

This enables users to create a log stream for a particular log group. Users can decide what to call the log stream with the --log-stream-name option and they can use the --log-group-name option to attach the stream to a log group of their choosing. Valid log stream names in AWS have to conform to three criteria.

First, they need to have a length between 1 and 512 characters. Second, they must be unique within the log group that they are associated with. Third, they cannot contain colons or asterisks.

describe-log-groups

This command allows the user to see at a glance the log groups in their system. When the command returns results, they are ASCII-sorted by log group name.

This means that log group names are displayed in alphabetical order with the added property that names beginning with capital letters universally appear before names beginning with lower case characters. Names beginning with numbers are displayed before names beginning with letters.

The command comes with a suite of options allowing users to view as many or as few log groups as they wish, limit the results by prefix and specify page size. If pagination is not to your liking, you can disable it with the --no-paginate option.

describe-log-streams

This command allows the user to see at a glance the log streams in a given log group. Results are paginated by default to make extensive lists easier to browse, plus there are several command options that allow users to order log stream events by name or by the time they occurred, as well as viewing as many or as few events as appropriate.

filter-log-events

This command enables users to view the log events associated with a given log group. If no log group is specified, the command spits out as many log events as it can find, up to a maximum of 1MB worth.

An arsenal of versatile options enable users to filter log events through a range of different criteria. The --log-group-name option is useful to users who want to view log events from a specific log group.

If users want to filter events by log stream, they have a choice between --log-stream-names and --log-stream-name-prefix. The --log-stream-names option lets users view events from one or more particular log streams while the --log-stream-name-prefix option displays events from the log streams starting with a specific character string, e.g all the log streams that started with “abc123”.

The start-time and end-time options display events that occurred within a given time range.

get-log-events

This command allows users to quickly access the content of log streams and view a list of log events. The --log-group-name and --log-stream-name options enable users to specify the exact log stream or log group that their log is in.

The command is highly versatile, allowing users to specify how many events they want to view, the order in which they want to view them and the time range they are interested in looking at. The --log-group-name option allows log events to be filtered by log group.

get-log-record

Enables users to view the fields and values of a single log event. The --log-record-pointer option allows a user to pinpoint the exact log event they wish to view.

Seeing the Bigger Picture

Essential to DevOps is the ability to monitor the overall health of an application. DevOps engineers need to be able to catch any fires burning in their system before they reach their flashpoint. CloudWatch has two useful tools for this purpose, dashboards and metric statistics.

get-metric-statistics

This enables users to easily see the statistics for a specified metric. The AWS documentation defines statistics as “metric data aggregations over specified periods of time”.

A number of useful options allow users to display the exact metric statistics they want to see. The first two of these are --metric-name and --namespace. metric-name takes the name of the metric a user wants stats for but namespace is a little more complicated.

An AWS namespace is like a hermetically sealed box that metrics live in. Each metric knows only its home namespace and nothing about metrics in other namespaces. Because there is no default namespace, a user who wants information on a given metric must always specify the metric’s namespace.

Another interesting option is --dimensions. AWS metrics are complex data structures that can contain up to ten dimensions. The --dimensions option lets users list all the dimensions of a specified metric.

There are three options that control time ordering of metric statistics. The --start-time and --end-time options, as with other commands, define the beginning and end of a given time range that the user is interested in returning data for. The third option is called --period and controls the sampling frequency of returned data.

AWS provides six statistics for each metric.

Minimum shows the lowest value of a metric data sample within a specific time range while maximum shows the highest value.
Sum refers to the values of a particular metric added together and can be useful for calculating metric volume. SampleCount signifies the number of data points used in statistical analysis.
Average is a statistic defined by the expression Sum/SampleCount for a specified time range. This statistic can be extremely useful in combination with Maximum and Minimum and can greatly aid tasks such as resource allocation.
pNN.NN is the value of a given percentile. These statistics can be viewed in raw form using the --statistics option, or in percentile form with --extended-statistics.

There we are!

Cloudwatch isn't perfect and is lacking in many of the features that other monitoring, logging and observability tools have, but with these commands, you'll be able to easily traverse the often complex world of AWS monitoring.

Monitoring Heroku Apps

ChrisCooney1 — Wed, 12 Aug 2020 15:14:01 +0000

Heroku differentiates itself from other cloud providers, by offering a complete, cohesive environment. Where AWS and GCP present a decoupled toolkit, Heroku strives for a seamless, UI driven experience for the user. This philosophy is clearly embodied in their metrics functionality, which is often a single click or basic configuration file away. This is clear in its move from hobbyist toolkit to bedrock of huge, enterprise solutions.

Make use of Metrics

Heroku exposes metrics functionality via a clean tab on the UI. This allows users to track the behaviour of their system. If you’re running your software in a web dyno, you are already gathering metrics. These range from error statuses, such as 5XX and 4XX HTTP response codes, to latency times. These are basic raw metrics. For example, latencies can be viewed by percentile, to provide a clear indication of, not just the average of your application's response time, but also the variance. Access to this sort of statistical analysis greatly improves observability and speeds up troubleshooting, especially considering these measurements were gathered for you automatically. While these measurements are not a part of the Heroku free-tier, they are available on a hobbyist subscription, at $7 a month.

The Logging CLI

The Heroku CLI comes packed with a ton of great features. One of the basic, easy to use features is the Heroku logging toolkit. This is actually a basic type of log collection. When you have a handful of dynos, all running instances of your application, this CLI is able to give you a single view of all of those logs with a single command:

heroku logs --app=monitoring-test-application --tail

This is often a herculean task for organisations, requiring weeks of complex engineering effort and ongoing maintenance that only grows with the scope and scale of your system. Out of the box log collection is very consistent with the simple, intuitive mentality that Heroku applies.

Auditing

When we think of monitoring, we’re immediately taken to graphs, displaying HTTP latencies and error rates. We often overlook other sorts of monitoring and observability, such as the changes that are being applied to our system. In a bespoke set up, auditing is usually driven by an organisational requirement for security compliance and is inserted as an afterthought. Heroku have baked this into their UI, so that from the start, you can see the changes that are being applied to your deployment, as they happen.

Add-Ons

Add-Ons are Heroku’s method of adding modular functionality into your deployment. There is a vast array of add-ons available that extend the capability of your stack, and many of these support more sophisticated monitoring. As is often the case with this provider, they can rely on the opinionated underpinnings of the Heroku system, making for a seamless and straight forward experience.

They come with their own billing mechanism, meaning that you can choose to pay for a deluxe version of one add-on while only committing to a standard version of another. While this flexibility creates the opportunity for cost optimisation, these various small subscriptions can quickly add up. Add-ons are an essential part of the Heroku landscape, but keep track of the many different tools you will need. This modular billing system can become difficult to track, over time.

The best tool for the Job

To address the limitations of the add-on system within Heroku, you may scale to a point where it is necessary to export certain data into other platforms. While the monitoring solutions of Heroku are outstanding, they, much like the rest of the Heroku tooling, are opinionated. This means that for most use-cases, they will work, but as complexity grows and the need for more bespoke solutions arises, you may be forced to branch out.

This can, ironically, be solved by more add-ons. For example, if you outgrow the simple logging CLI that Heroku offers, you can install the Coralogix add-on for your application. Should you need to enhance your container scanning capability. If you need to do something incredibly bespoke, you can build your own add-on to export metrics, logs, and other data into your own systems.

This property, known as extensibility, means that when there is not an “off the shelf” solution available to you, you’re able to extend your deployment in new and unexpected directions. This is a key capability for any rapidly scaling architecture, that can’t afford to find itself handcuffed to a limited set of functionality by a service provider.

All in all

Heroku is a fantastic environment. The tools that are seamlessly presented to you will capture many of the typical use-cases found, during the lifetime of a software project. For larger or more complex projects, Heroku’s openness and extensibility creates constant avenues for innovation, while maintaining the opinionated consistency that underpins Heroku’s engineering experience. While caution must be exercised around the billing and cost optimisation of the various add-ons and servers, with some forethought, planning and the right 3rd party integrations, Heroku is capable of competing with even the mammoths of the cloud platform world.

Chris writes about DevOps and Monitoring for Coralogix.