DEV Community: Sajid Rashid

Introduction to Prometheus & Grafana for Observability

Sajid Rashid — Mon, 17 Mar 2025 16:36:17 +0000

What is Prometheus?

Prometheus is an open-source monitoring and alerting toolkit that is widely used in cloud-native ecosystems. It is the default monitoring solution for Kubernetes and supports metrics-based monitoring with a pull-based data collection model.

Official Documentation: Prometheus FAQ

Key Features of Prometheus:

Time-series Database: Stores metrics efficiently with labels for fast retrieval.
Powerful Querying (PromQL): Allows multi-dimensional queries to analyze data.
Push and Pull Metrics Collection: Supports both push (via Pushgateway) and pull-based monitoring.
Service Discovery: Auto-discovers services dynamically, making it ideal for cloud environments.
Alerting Mechanism: Works with Alertmanager to trigger alerts based on metric thresholds.

What is Grafana?

Grafana is an open-source data visualization and monitoring tool. It helps users create interactive dashboards with real-time data from various sources, including Prometheus, AWS CloudWatch, Loki, MySQL, and more.

Key Features of Grafana:

Customizable Dashboards: Interactive dashboards that combine multiple data sources in one view.
Plugins: A variety of plugins for data sources, visualizations, and apps.
Alerts: Built-in alerting to notify teams of potential issues.
Annotations: Overlay events on top of graphs to add context.
Data Source Support: Integrates with Prometheus, Loki (logs), MySQL, and more.

Observability Pillars: Logs, Metrics, Traces, and Profiles

A reliable system should collect and analyze different types of telemetry data:

Logs → Chronological records of events from applications and infrastructure.
Metrics → Numeric time-series data that help track system performance.
Traces → End-to-end request tracking for debugging microservices.
Profiles → Performance profiling to analyze CPU, memory, and execution times.

Each type of data is collected in specialized databases optimized for different observability needs.

Logs in Grafana (Loki)

Loki is a log aggregation system designed to work efficiently with Prometheus. Unlike traditional log systems, Loki does not index full log content, but instead indexes metadata (labels) for fast querying.

Advantages of Loki:

Efficient storage: Uses labels instead of indexing full logs.
Integration with Prometheus: Works with Alertmanager for alerting.
Supports structured and unstructured logs.

Example: How Loki Indexing Works

Timestamp: 2024-03-17 10:15:00  
Labels: {app="nginx", instance="1.1.1.1"}  
Log Content: "GET /about 200 OK"

Metrics in Prometheus

Prometheus stores time-series data with a label-based model. Instead of storing raw files, it organizes data using key-value pairs (labels).

Example Metric Labels in Prometheus:

server_type="web", region="us-east-1"

Querying with PromQL (Prometheus Query Language)

PromQL is a multi-dimensional query language used for analyzing metrics.

Example Query:

rate(http_requests_total[5m])

This query shows the rate of HTTP requests in the last 5 minutes.

Exemplars:
Exemplars are sample data points linked to traces, helping correlate logs and metrics with tracing data.

Traces in Tempo

Tempo is a distributed tracing backend used to trace individual transactions through a system. It works alongside Loki and Prometheus.

Use Cases:

Debugging microservices latency issues.
Identifying slow database queries.
End-to-end request tracking.

Example Query in TraceQL:

span.http.route = "/api/v1/user"

Profiles in Pyroscope

Pyroscope is a profiling database that collects application performance data over time. It uses FlameQL for querying.

Common Use Cases:

Detecting CPU bottlenecks.
Memory leak analysis.
Comparing performance across different software builds.

Example FlameQL Query:

topk(5, heap_alloc_bytes{app="payment-service"})

This retrieves the top 5 memory allocations in the payment service.

Data Collection Methods

Observability systems collect data through instrumentation and exporters.

Source Instrumentation: Uses SDKs, libraries, and agents to collect telemetry data during runtime.
Frontend Observability: Faro collects logs and traces from web applications.
Backend Monitoring: Uses OpenTelemetry to capture application traces and logs.

Example: OpenTelemetry Collector Pipeline

Application → OpenTelemetry Collector → Prometheus → Grafana

What to Do with the Data?

1. Visualization in Grafana
Use data sources to connect to Prometheus, Loki, and Tempo.
Configure panels to display metrics, logs, and traces.
2. Alerting in Grafana
Use OnCall for incident response and automated escalation via webhook, email, or Slack notifications.
You can also send alerts to BigPanda via webhook.

Summary

Prometheus = Metrics collection and monitoring.
Grafana = Data visualization and dashboards.
Loki = Log aggregation.
Tempo = Distributed tracing.
Pyroscope = Performance profiling.
OpenTelemetry = Unified telemetry collection.

Observability helps make our systems reliable!

Final Thoughts

Observability is essential for modern cloud infrastructure. By using Prometheus, Loki, Tempo, and Pyroscope, teams can collect, analyze, and visualize data efficiently in Grafana.

Follow me for more DevOps and Observability insights!

Setting Up an IAM Role for Terraform on EC2

Sajid Rashid — Mon, 17 Mar 2025 14:18:34 +0000

Introduction

When running Terraform on AWS EC2, it's best practice to use an IAM role instead of manually managing AWS access keys. An IAM role allows Terraform to securely authenticate and access AWS resources. This guide will walk you through the process of creating an IAM role, attaching it to an EC2 instance, and verifying the setup.

Step 1: Create an IAM Role with Permissions

1. Log in to AWS Management Console

Navigate to the AWS IAM Console.

2. Create a New IAM Role

In the IAM dashboard, click Roles in the left-hand menu.
Click Create role.

3. Select Use Case

Under Trusted entity type, choose AWS service.
Under Use case, select EC2 (since the role will be used by an EC2 instance).
Click Next.

4. Attach Policies (Permissions)

Choose the necessary permissions for Terraform to run. Some commonly used policies include:

AdministratorAccess (Full access to AWS services).
AmazonS3FullAccess (For managing S3 buckets).
AmazonEC2FullAccess (For full EC2 resource access).

Select the required policies and click Next.

5. Configure Tags (Optional)

Add tags if needed for identification or tracking.

Click Next.

6. Review and Create Role

Provide a name (e.g., Terraform-EC2-Role).
Review policies and click Create role.

Step 2: Attach the IAM Role to Your EC2 Instance

1. Go to the EC2 Dashboard

Navigate to the AWS EC2 Console.

2. Select Your EC2 Instance

Under Instances, find and select the EC2 instance where Terraform will run.

3. Modify the IAM Role

With the instance selected, click Actions.
Navigate to Security > Modify IAM Role.

4. Attach the IAM Role

Select the IAM role created earlier (Terraform-EC2-Role).
Click Update IAM Role.

Step 3: Verify IAM Role is Attached to the EC2 Instance

1. Check IAM Role in EC2 Dashboard

Select the instance and check the Description tab.
Ensure the IAM role field shows the assigned role (Terraform-EC2-Role).

2. Verify Permissions from EC2 Instance

SSH into your EC2 instance:

ssh ec2-user@<your-ec2-public-ip>

Run the following command:

aws sts get-caller-identity

This should return details about the IAM role attached to the instance.

Step 4: Run Terraform on EC2

With the IAM role attached, Terraform will automatically authenticate with AWS, eliminating the need for manual AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY configuration.

1. Run Terraform Commands:

terraform init
terraform plan
terraform apply

Terraform will now use the IAM role to authenticate with AWS and manage resources.

Follow me for more tips :)

What is Terraform? Everything You Need to Know

Sajid Rashid — Mon, 17 Mar 2025 13:46:19 +0000

Introduction

Terraform is an Infrastructure as Code (IaC) tool developed by HashiCorp that allows you to define, provision, and manage infrastructure using a declarative configuration language called HCL (HashiCorp Configuration Language). Terraform is widely used in cloud environments like AWS, Azure, and Google Cloud to automate infrastructure provisioning and ensure repeatability.

This article will provide an in-depth understanding of Terraform, its benefits, installation, key commands, and usage with cloud providers.

Why Use Terraform?

Terraform is a powerful tool that helps DevOps and infrastructure engineers achieve repeatability, automation, and version control in infrastructure provisioning. Some key benefits include:

Repeatability: Infrastructure can be defined and deployed consistently.
Auditability: Changes are tracked in a version control system (VCS) like Git.
Change Control: Any infrastructure change is introduced via code updates.
Collaboration: Teams can work together using GitHub, GitLab, or Bitbucket.

Key Terminology

IaC (Infrastructure as Code): Managing infrastructure through code.
CM (Configuration Management): Tools like Ansible, Puppet, SaltStack for managing configurations.
IaaS (Infrastructure as a Service): AWS, Azure, and other cloud providers.
PaaS (Platform as a Service): Platforms for hosting applications.
VCS (Version Control System): Git, GitHub, or GitLab for tracking changes.
CI/CD (Continuous Integration/Continuous Deployment): Automating deployment pipelines.
SDLC (Software Development Life Cycle): The development workflow from planning to maintenance.

Declarative vs. Imperative Approaches

Terraform follows the Declarative approach, meaning you describe the desired state of infrastructure, and Terraform figures out how to achieve it.

Declarative Example:

resource "aws_instance" "web" {
  ami           = "ami-12345678"
  instance_type = "t2.micro"
}

Imperative Approach:

Using CLI tools like AWS CLI or Python scripts to manually define the provisioning steps.

Understanding Terraform Components

1. Terraform Configuration Files

Terraform configuration consists of .tf files that define infrastructure resources.

2. State File

Terraform keeps track of deployed resources in a state file (terraform.tfstate). This file can be stored locally or in a remote backend like AWS S3 or Terraform Cloud.

3. Modules and Registry

Terraform modules allow code reusability and modularity, while the Terraform Registry provides pre-built modules for infrastructure components.

Installing Terraform

Using Docker:

docker pull hashicorp/terraform:latest
docker run --rm hashicorp/terraform --version

On CentOS:

sudo yum install -y yum-utils
sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
sudo yum -y install terraform

On Amazon Linux/EC2:

sudo yum install -y yum-utils shadow-utils
sudo yum-config-manager --add-repo https://rpm.releases.hashicorp.com/AmazonLinux/hashicorp.repo
sudo yum -y install terraform

Terraform Workflow

1. Initializing Terraform

terraform init

This prepares Terraform to manage infrastructure by downloading necessary providers.

2. Planning Infrastructure Changes

terraform plan

Compares the desired configuration with the current state and generates an execution plan.

3. Applying Changes

terraform apply

Executes the plan and provisions the infrastructure.

4. Destroying Resources

terraform destroy

Removes all resources managed by Terraform.

Terraform in a Version Control Environment

Terraform configurations should be managed using a Git repository to track changes.

Pushing Changes to GitHub
1- Clone the repository:

git clone https://github.com/your-repo/terraform-project.git

2- Initialize Terraform:

terraform init

3- Generate a plan:

terraform plan -out=tfplan.out
terraform show -no-color tfplan.out > tfplan.txt

4- Apply changes:

terraform apply tfplan.out

5- Push changes to GitHub:

git add .
git commit -m "Added Terraform configuration"
git push origin main

Creating an EC2 Instance with Terraform

Here’s how you can launch an EC2 instance using Terraform:

main.tf (Terraform configuration file)

provider "aws" {
  region = "eu-west-2"
}

resource "aws_instance" "web" {
  ami           = "ami-12345678"
  instance_type = "t2.micro"
  tags = {
    Name = "Terraform-EC2-Instance"
  }
}

Execution Steps:

terraform init
terraform plan -out=tfplan.out
terraform apply tfplan.out

Once applied, Terraform will output Instance ID, Public IP, and other details.

Summary

Terraform is a powerful, declarative IaC tool that simplifies infrastructure provisioning and management. With support for multiple cloud providers, modular code, and version control integration, it is an essential tool for DevOps engineers.

Start using Terraform today and automate your cloud infrastructure efficiently!

Follow me for more DevOps insights! :)