DEV Community: Auke Noppe

Forward logs to Cloudwatch for an EC2 instance running a custom Linux AMI

Auke Noppe — Thu, 09 Jan 2025 12:00:34 +0000

I’m a big fan of using serverless components within AWS to build the systems I need: It’s an easy, low-threshold way to set up systems without the hassle of managing resources. Nonetheless, sometimes you need to spin up an old-fashioned EC2 virtual machine. You might even have to spin up an EC2 instance from a non-Amazon Linux AMI when, for example, the service you want to run is unavailable in any of the repositories provided by the distribution. However, non-Amazon Linux AMIs don’t have the Cloudwatch Agent installed and configured out of the box.

However, you want to have the logs of your EC2 instance available in Cloudwatch for easy and secure access to the logs and analysis of the behavior of your instance and services running on that instance. In this article, I explain how to manually install and configure the Cloudwatch Agent to get your instance’s logs available in Cloudwatch.

If you want to jump ahead: A working example can be found on GitHub.

Installing the Cloudwatch Agent

Whenever the Cloudwatch agent is unavailable as a package in the repository of the Linux distribution you are running, you can install it manually by leveraging the user data initialization script of the EC2 instance. The user data initialization script runs every time an EC2 instance is launched and can be used to execute commands and scripts to configure your instance. The Cloudwatch Agent is available for several distributions and other OSs. A complete overview of the available flavors can be found here. In this example, we will launch an EC2 instance with Debian 12 and we thus need the .deb package for easy manual installation of the Cloudwatch Agent. Let’s spin up some resources!

Launching an EC2 instance

Let’s set up a stack with AWS CDK that will deploy an EC2 instance based on Debian 12. The Debian distribution is a good candidate to illustrate this topic since it doesn’t have the Cloudwatch Agent package in its repository.

I’m using Python for these examples, and I’ve created a project using poetry and initialized an empty CDK stack:

from aws_cdk import Stack  
from constructs import Construct  

class Ec2CloudwatchAgentStack(Stack):  
    def __init__(self, scope: Construct, id: str, **kwargs) -> None:  
        super().__init__(scope, id, **kwargs)

I’ve chosen the Debian 12 AMI from the AMI catalog since this AMI doesn’t have the standard Amazon Linux tooling installed and configured, so suits perfectly to illustrate the topic at hand.

To configure an EC2 using a custom AMI, you need to look up the AMI identifier. The ID of the Debian 12 AMI is ami-0715d656023fe21b4.

Let’s add the EC2 configuration to our empty stack: We also need to define a VPC and to be able to log in after we create the EC2 instance, we add the EC2 instance to the public subnet, and we also need a key-pair to authenticate.

Configuring an EC2 instance inside a public subnet means that it is reachable from the internet. This is bad practice and should be avoided, especially when no security measures are in place like firewalls etc.

# VPC with private subnet  
vpc = ec2.Vpc(self, "VPC",  
              max_azs=2,  
              subnet_configuration=[ec2.SubnetConfiguration(  
                  name="public", cidr_mask=24, subnet_type=ec2.SubnetType.PUBLIC  
              )])  
# security group  
security_group = ec2.SecurityGroup(self, "SecurityGroup",  
                                   vpc=vpc,  
                                   description="Allow ssh access to ec2 instances",  
                                   allow_all_outbound=True)  
# open ssh to the ec2 instance  
security_group.add_ingress_rule(  
          ec2.Peer.any_ipv4(),  
          ec2.Port.tcp(22),  
          "Allow ssh access from the world"  
      )  

# import existing keypair key-pair-van-auke  
key_pair = ec2.KeyPair.from_key_pair_name(self, "MyKeyPair",  
                                          key_pair_name="key-pair-van-auke")  

# EC2 instance based on Debian 12  
instance = ec2.Instance(self, "MyEc2Instance",  
                        instance_type=ec2.InstanceType.of(  
                            ec2.InstanceClass.BURSTABLE2, ec2.InstanceSize.MICRO),  
                        machine_image=ec2.MachineImage.generic_linux({  
                            'eu-west-1': 'ami-0584590e5f0e97daa'  
                        }),  
                        vpc=vpc,  
                        vpc_subnets=ec2.SubnetSelection(subnet_type=ec2.SubnetType.PUBLIC),  
                        key_pair=key_pair,  
                        security_group=security_group)

This configuration can now be deployed to your AWS account using the cdk deploy command in your terminal. This will take a few minutes, and when it’s deployed we can test the configuration by trying to ssh into the instance:

$ ssh -i key-pair-van-auke.pem admin@54.73.44.74  
Linux ip-10-0-0-15 6.1.0-23-cloud-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.99-1 (2024-07-15) x86_64  

The programs included with the Debian GNU/Linux system are free software;  
the exact distribution terms for each program are described in the  
individual files in /usr/share/doc/*/copyright.  

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent  
permitted by applicable law.  
Last login: Mon Dec  2 10:56:14 2024 from 85.146.13.35  
admin@ip-10-0-0-15:~$

As you can see the EC2 instance is deployed successfully and we can log in.

EC2 User data

As you might know, EC2 instances have a way to run a script when the instance is created. The so-called user-data. To quote the documentation:

When you launch an Amazon EC2 instance, you can pass user data to the instance that is used to perform automated configuration tasks or to run scripts after the instance starts.

In this example, we will leverage this user data to install and configure the Amazon Cloudwatch Agent on the EC2 instance to forward specific logs to Amazon Cloudwatch for easy access and analysis.

We will start by extending the CDK configuration with the user-data instructions to download the .deb package of the Amazon Cloudwatch agent, install the package, and enable and start the service:

# edit user data  
instance.user_data.add_commands(  
    "apt-get update",  
    "apt-get install -y gpg",  
    "apt-get install -y wget",  
    "wget https://s3.amazonaws.com/amazoncloudwatch-agent/debian/amd64/latest/amazon-cloudwatch-agent.deb",  
    "dpkg -i -E ./amazon-cloudwatch-agent.deb",  
    "systemctl enable amazon-cloudwatch-agent",  
    "systemctl start amazon-cloudwatch-agent"  
)

If we then execute the cdk deploy command, the EC2 instance will be recreated and these instructions will be executed on the first boot of the instance.

Let’s check the service status:

admin@ip-10-0-0-130:~$ systemctl status amazon-cloudwatch-agent.service   
○ amazon-cloudwatch-agent.service - Amazon CloudWatch Agent  
     Loaded: loaded (/etc/systemd/system/amazon-cloudwatch-agent.service; enabled; preset: enabled)  
     Active: inactive (dead) since Fri 2024-12-20 07:24:07 UTC; 5min ago  
   Duration: 54ms  
    Process: 890 ExecStart=/opt/aws/amazon-cloudwatch-agent/bin/start-amazon-cloudwatch-agent (code=exited, status=0/SUCCESS)  
   Main PID: 890 (code=exited, status=0/SUCCESS)  
        CPU: 35ms  

Dec 20 07:24:07 ip-10-0-0-130 systemd[1]: Started amazon-cloudwatch-agent.service - Amazon CloudWatch Agent.  
Dec 20 07:24:07 ip-10-0-0-130 start-amazon-cloudwatch-agent[894]: D! [EC2] Found active network interface  
Dec 20 07:24:07 ip-10-0-0-130 start-amazon-cloudwatch-agent[894]: I! imds retry client will retry 1 timesI! Detected the instance is EC2  
Dec 20 07:24:07 ip-10-0-0-130 start-amazon-cloudwatch-agent[894]: 2024/12/20 07:24:07 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json ...  
Dec 20 07:24:07 ip-10-0-0-130 start-amazon-cloudwatch-agent[894]: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json does not exist or cannot read. Skipping it.  
Dec 20 07:24:07 ip-10-0-0-130 systemd[1]: amazon-cloudwatch-agent.service: Deactivated successfully.

Cool, the service is installed, started, however not running anymore: The last log line states that it’s missing the configuration file, and therefore shutdown. In order to instruct the Cloudwatch agent which log file(s) it needs to watch and forward to Amazon Cloudwatch, we need to supply a configuration file.

Configuring the Cloudwatch Agent

The Cloudwatch Agent needs a configuration file to know which data it needs to forward to the Amazon Cloudwatch service. There are multiple approaches possible: We can provide the configuration file in the user-data, or download it from S3.

In this situation, the Cloudwatch Agent configuration JSON file will have these contents:

{  
  "agent": {  
    "metrics_collection_interval": 60,  
    "logfile": "/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log"  
  },  
  "logs": {  
    "logs_collected": {  
      "files": {  
        "collect_list": [  
          {  
            "file_path": "/var/log/cloud-init-output.log",1️⃣  
            "log_group_name": "/aws/ec2-cloudwatch-agent",2️⃣  
            "log_stream_name": "{instance_id}/messages",3️⃣  
            "timestamp_format": "%b %d %H:%M:%S"4️⃣  
          }  
        ]  
      }  
    }  
  }  
}

The important part of the config file is the collect_list Array: Here you can define:

1️⃣ Which log files need to be watched and forwarded to Cloudwatch

2️⃣ The name of the log group to submit the logs to

3️⃣ The name of the log stream. You can see that I’ve added the instance_id as variable, so each instance will have its log stream.

4️⃣ The timestamp format of the messages

For more comprehensive documentation about the possibilities of the config file I want to redirect you to the corresponding documentation

Cloudwatch Configuration file via User-Data

To provide the configuration file to the Cloudwatch Agent via User-Data we can write the text we add to the user data into a file at a specific location:

# edit user data  
instance.user_data.add_commands(  
    "apt-get update",  
    "apt-get install -y gpg",  
    "apt-get install -y wget",  
    "wget https://s3.amazonaws.com/amazoncloudwatch-agent/debian/amd64/latest/amazon-cloudwatch-agent.deb",  
    "dpkg -i -E ./amazon-cloudwatch-agent.deb",  
    "systemctl enable amazon-cloudwatch-agent",  
    "systemctl start amazon-cloudwatch-agent",  
    "sudo mkdir -p /opt/aws/amazon-cloudwatch-agent/etc",  
    "sudo touch /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json",  
    "sudo cat <<EOF > /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json",  
# The content to write to the config file  
    "{",  
    "  \"agent\": {",  
    "    \"metrics_collection_interval\": 60,",  
    "    \"logfile\": \"/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log\"",  
    "  },",  
    "  \"logs\": {",  
    "    \"logs_collected\": {",  
    "      \"files\": {",  
    "        \"collect_list\": [",  
    "          {",  
    "            \"file_path\": \"/var/log/cloud-init-output.log\",",  
    "            \"log_group_name\": \"/aws/ec2-cloudwatch-agent\",",  
    "            \"log_stream_name\": \"{instance_id}/messages\",",  
    "            \"timestamp_format\": \"%b %d %H:%M:%S\"",  
    "          }",  
    "        ]",  
    "      }",  
    "    }",  
    "  }",  
    "}",  
    "EOF",  
)

This approach doesn’t necessarily produce the most readable CDK code, but it is easy to manage and it provides an easy way to make the configuration dynamic based on data that is available or produced by the stack this code runs in. We can for example inject tag values or data from Parameter Store that we have available in the stack to make a distinction between instances in Amazon Cloudwatch, for example.

When deploying this configuration, you’ll see that there is a log group created in Cloudwatch named /aws/ec2-cloudwatch-agentand it contains the log lines from /var/lo/cloud-init-output.log. If you want to forward other log files to Cloudwatch, you can extend the collect_list configuration array with the location of the file and the destination log group.

Conclusion

Sometimes when you need to fall back onto EC2 instances, you don’t want them to be a black box that you have to SSH into to be able to see what is happening inside that instance. I’ve shown you how to deploy an EC2 instance and install the Amazon Cloudwatch Agent on a Linux distribution that doesn’t contain the agent out of the box or isn’t available in the package manager either. I’ve also explained how to configure the agent to forward the contents of a specific log file to Amazon Cloudwatch so you can analyze the logs without having to SSH into the instance.

Silver Linings, No Silver Bullets: 3 Perspectives on Innovating with Cloud

Auke Noppe — Tue, 28 Feb 2023 13:48:44 +0000

Authors: Piet van Dongen and Auke Noppe

We’ve heard the phrase, seen the cynical stickers: “There is no cloud; it’s just someone else’s computer.” That might have been true 15 years ago, but the thing we call the cloud has evolved significantly since. We live in times of accelerating change; organizations that can leverage the cloud effectively will survive them, even thrive in them. How well prepared are you?

Taking great advantage of the cloud requires a shift in thinking: technology no longer belongs to the IT department (if it ever did). To become cloud natives, organizations need to learn to streamline their effort around everything the cloud offers.

Let’s look at this paradigm shift from several perspectives: operations, development, and leadership. Each of these perspectives starts with describing the ineffective way of working. We use the term cloud naive (as in: inexperienced or lacking knowledge). Or as the State of DevOps report classifies it: the way of the low performers. Then, we cross the chasm and look at it from the cloud-native perspective, as used by companies that use cloud technology to accelerate their business continuously.

Operations: embrace speed to reduce risk

Low performance: cloud naive operations

In simpler times, developers delivered complete software packages to operations people, who in turn installed and ran them on the appropriate infrastructure. Preferably not too often and with as little change needed as possible. A similar rule determined a snail’s pace of change for platforms: standardization before innovation. Why? Because every change introduced risk and operations were held responsible for it, often having to clean up the mess when things went wrong.

This model led operations departments to push back hard on changes and reduce the speed of change—developers adjusted by increasing release sizes and time between releases.

The result: slow change, little cooperation, suffering users.

Elite performance: cloud-native operations

The way out of this standoff is well-known by now but not always well understood or effectively implemented: DevOps. Making agile teams responsible for business outcomes forces down the classical barriers between the business, developers, and operations, leading to better customer outcomes. Cloud platforms have evolved by adding layer upon layer of abstraction onto the classic data center, leading to hundreds of managed services that teams can continuously leverage to decrease their time-to-value. Perhaps counterintuitively, this increased rate of change enables teams to manage risk better than ever.

Think about it: if time-to-value is the most critical metric for success, change size logically also gets smaller. Minor changes are easier to manage in case of failure, and deploying multiple times a day yields teams totally in control of their technological solutions.

This cloud-native way of doing operations yields precisely what these teams aimed for in the first place: less risk, more control.

Development: focus on outcomes, not technology

Low performance: cloud naive development

Before the emergence of the cloud, servers had to be installed by hand, as did the software running on them. Installing and testing releases was a time-consuming process, mostly done by other teams. So, developers made sure their sparsely released work contained as many features as possible. Any infrastructural requirements and other dependencies had to be carefully communicated and planned with IT operations, dictating long lead times between innovative ideas and customer value.

As a result, developers preferred working with stable, minimally variable environments simulated on their development machines. Their products were massive monoliths often written using a single programming language, running on single application container instances. Development feedback might have been fast; customer feedback certainly wasn’t. Developers were often kept totally in the dark about the impact of their changes in production until nightly calls from frustrated IT operations, who spoke in an entirely different IT dialect.

Again: slow change, little cooperation, suffering users.

Elite performance: cloud-native development

The step-by-step virtualization and containerization of servers, platforms, and runtime environments have enabled development teams to master what was previously the sole job of operations. And more importantly, the marriage of Dev and Ops has given birth to beautiful things like infrastructure as code, CI/CD, and SRE. Managing complex systems this way has opened up opportunities for creating massively scalable but well-observable systems. The old monoliths have been split into small, modular services or even single functions that typically communicate asynchronously using a mind-boggling number of events.

Now the customer wins. Changing, replacing, or radically rewriting a single, small module and testing and deploying it in the blink of an eye has become a reality. Experimenting — seeing if an idea for a new job to be done holds up in production — has become cheap as chips using the pay-as-you-go managed services as provided by cloud platforms like AWS, Azure, and GCP.

Leadership: get out of the way of innovation

Low performance: cloud naive leadership

During less volatile times, markets were predictable, transformations were carefully planned and budgeted, and users were okay with incremental changes every so often. Software, and the hardware it ran on, were complex and expensive beasts, mastered only by tech masochists stuffed away in cost centers called IT departments. Nothing got built until it was very meticulously specified in requirements documents. Even then, the delivered result was often wildly different from what the market specialists and innovators defined months or years before, if not already outdated.

Once more: slow change, little cooperation, suffering users.

Elite performance: cloud-native leadership

Software has been eating the world for a while now. Organizations need to adapt or accept a slow and painful decline into irrelevance. Users now expect to get what they need before realizing they do: a new Netflix series, a recommended product, or even an improved driving experience. Understanding users is key to market success and requires continuous experimentation and software development mastery. The cloud is a massive enabler: it enables organizations to focus on user needs by handing the management of undifferentiated technology over to the cloud provider.

But while technical skill development is essential, it is just one part of the innovation puzzle. Another is culture: leaders need to focus everyone’s attention on the importance of time-to-value and agility as the metrics for success. To create empowered, self-sufficient, and effective teams, people need to come together and shatter the walls previously separating them. Lastly, leaders need to take the long view and realize that short-term risk aversion can be a major innovation blocker.

To boldly go, together

Transforming an organization from cloud naiveness into cloud-native isn’t for the faint of heart, but very necessary nonetheless. Luckily, Luminis has traveled this path many times before and is a very effective guide on the path to cloud-native. For example, we helped OHRA, one of the largest Dutch digital insurers, migrate its complete data center to the cloud. Another example is the digital transformation of The Learning Network (TLN) we helped shape and implement.

Get in touch and let us help you kickstart your cloud-native transformation.