How to Monitor your AWS EC2/Workspace with Datadog

-1- Log in to your AWS and Datadog accounts. In this example, I will configure AWS Workspace. If you have an EC2 instance, you can follow these steps too. My OS is Ubuntu.

-2-. After deploying an EC2 instance or logging into an AWS Workspace, proceed to update the machine.

-3-. Next, navigate to Datadog → Integrations → Agent → Ubuntu and install the Datadog agent on your host machine. This agent will send metrics to Datadog and does not integrate directly with any AWS services.

-4-. Then click on 'Select API Key,' create a new key, and give it a name. Below that, you will see a command to install the Datadog agent on your system. Copy that command and run it on your server.

DD_API_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX DD_SITE="datadoghq.com"  bash -c "$(curl -L https://install.datadoghq.com/scripts/install_script_agent7.sh)"

-5-. The command will configure and install the Datadog agent on your machine.

sudo usermod -a -G docker dd-agent
systemctl status datadog-agent
datadog-agent version
hostname

You can run the above command. Since I have Docker installed on my system, I've added the Datadog agent to the Docker group to monitor Docker. After running the command, check the service status, Datadog version, and finally, the hostname. If you're using an EC2 instance, it will display the instance ID; for a Workspace, it will show the Workspace ID.

-6-. After installing the agent, navigate to your Datadog dashboard, go to 'Infrastructure,' and search for your instance ID or workspace ID. You will now see your host machine on the default dashboard.

-7-. If you notice, initially, you'll see only a few metrics. However, you can explore other options such as 'Host Info,' 'Containers,' 'Processes,' 'Network,' 'Logs,' and more. By default, you can view host info and metrics on the dashboard. If you want to see 'Processes' and 'Logs,' you'll need to enable them in the Datadog configuration file.

-8-. To enable viewing 'Processes' in the Datadog dashboard, you'll need to edit the configuration file.

vi /etc/datadog-agent/datadog.yaml

Add the following line to your configuration file to enable 'Processes' monitoring.

process_config:
  process_collection:
    enabled: true

-9-. After adding the line to the configuration file, restart the Datadog service. Then, navigate to Datadog, click on 'Processes,' and you'll now see the dashboard displaying Processes, PID, total CPU, and RSS memory.

sudo systemctl restart datadog-agent

-10-. Now that Processes are visible, you can expand the dashboard by clicking 'Open in Live Process' on the right-hand side for a larger view. Next, let's configure 'Logs.'

-11-. To configure 'Logs,' uncomment the line 'Logs: true' in the Datadog configuration file.

vi /etc/datadog-agent/datadog.yaml

-12-. Create a new directory for system logs:

sudo mkdir /etc/datadog-agent/conf.d/system_logs.d

-13-. Create the conf.yaml file in the new directory:

sudo vi /etc/datadog-agent/conf.d/system_logs.d/conf.yaml

-14-. Add the log collection configuration to the conf.yaml file:

logs:
  - type: file
    path: /var/log/syslog
    service: syslog
    source: syslog
    sourcecategory: system

  - type: file
    path: /var/log/auth.log
    service: auth
    source: auth
    sourcecategory: system

  - type: file
    path: /var/log/kern.log
    service: kernel
    source: kernel
    sourcecategory: system

  - type: file
    path: /var/log/messages
    service: messages
    source: messages
    sourcecategory: system

  # Add other log files as needed

-15-. After making the changes, save and exit the file. Typically, you can use 'wq!' to save and exit, but 'x' also works.

-16-. Ensure that the Datadog Agent user has the appropriate permissions to read the log files. Afterward, restart the agent.

sudo usermod -a -G adm dd-agent
sudo systemctl restart datadog-agent

-17-. Now go to Datadog again and go to "Logs" tab

-18-. Finally, with all these different metrics, create a new dashboard and consolidate all windows into one.

-19-. Of course, if you want to monitor additional services or metrics, feel free to explore the official Datadog website and refer to the documentation.

-20. In conclusion, this guide has demonstrated how to effectively monitor your AWS EC2 or Workspace using Datadog. We covered installing the Datadog agent, configuring metrics like Processes and Logs, and creating a unified dashboard. It's important to note that Datadog is a paid tool, so if you're practicing as a single user or student, be mindful of potential costs. However, Datadog offers a 14-day trial period, allowing you to explore its features for free during this time. Remember, depending on your specific use case, you can further explore Datadog to unlock additional metrics and services tailored to your needs. For more detailed options, visit the official Datadog website and consult their documentation.