DEV Community: Thiago Rodrigues

Prometheus Architecture

Thiago Rodrigues — Fri, 19 Dec 2025 05:08:59 +0000

Prometheus Architecture

This is the first in a series of articles focused on the architecture of the main components of a modern monitoring stack.

Initially, I planned to start by comparing the different variants and tools in the Prometheus ecosystem (such as Mimir, Thanos, and Cortex). However, I realized it makes more sense to start with Prometheus itself; after all, it is the foundation and origin of all these solutions.

Most likely, at some point in your IT journey, you have heard of, seen, or used a metric exposed in Prometheus format for observability. Prometheus is an open-source project, graduated by the Cloud Native Computing Foundation (CNCF), being the second project to achieve this status, right after Kubernetes.

It works extremely well in Kubernetes environments, but also adapts perfectly to clouds and container-based environments in general.

Collection Model

Prometheus uses a pull-based approach to collect metrics. Unlike systems where agents actively send data, Prometheus goes to the source and "pulls" the data.

Figure 1: Prometheus data collection flow with pull and push mechanisms

The simplest way to get started is to run it as a container:

image: prom/prometheus:latest

To operate it, you need a .yml configuration file, where you define global parameters, scrape frequencies, and targets.

A basic startup configuration looks like this:

global:
  scrape_interval: 15s # Scrape frequency
  evaluation_interval: 15s # Rule evaluation frequency
  external_labels:
    cluster: 'demo-cluster'
    environment: 'dev'

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']
        labels:
          instance: 'prometheus-server'

The instance label (and others you add) allows you to filter and aggregate metrics later during queries, providing context to the data.

Core Architecture Components

A typical Prometheus deployment consists of several elements working together:

Figure 2: Complete Prometheus architecture with all core components

Prometheus Server — The brain of the operation, responsible for collection and storage.
Targets — The endpoints that your applications or servers expose with metrics.
Exporters — Agents that translate metrics from third-party systems to Prometheus format.
Time-Series Database (TSDB) — The internal database optimized for time series.
PromQL — The powerful language for querying and analyzing data.
Push Gateway — An auxiliary component for handling short-lived jobs.
Alert Manager — The system responsible for managing, grouping, and routing alert notifications.
Client Libraries — Libraries for instrumenting custom code directly in applications.

Let's detail each of them.

Prometheus Server

The server is the central component. It performs three main functions:

Scraping: Periodically connects to configured targets via HTTP to fetch metrics.
Storage: Writes collected data to its local time-series database (TSDB).
Evaluation and Querying: Evaluates alert rules and responds to queries from users or visualization systems (like Grafana) via PromQL API.

In practice, it is the heart of the operation, ensuring the continuous flow of data from source to storage.

Targets

"Targets" are the sources of your metrics. They can be practically anything: a Linux server, a Java application, an API endpoint, or Kubernetes pods.

By default, Prometheus fetches metrics from the /metrics HTTP endpoint of the target, although this path is configurable.

scrape_configs:
  - job_name: 'node-metrics'
    static_configs:
      - targets: ['instance-dev:9100']
        labels:
          instance: 'instance-dev'

Examples of common targets on an instance:

Port 9100: Where node_exporter exposes operating system metrics.
Port 8080: Where your application can expose custom business metrics.
Port 8081: Where cAdvisor exposes Docker container metrics.

Exporters

Not all software natively exposes metrics in Prometheus format (like a MySQL or Redis database, for example). That's where Exporters come in.

Exporters are small binaries that function as translators: they collect metrics from the original system (using its native APIs) and convert them to the readable text format that Prometheus understands, exposing them on an HTTP endpoint.

There is a vast range of exporters maintained by the community and officially:

node_exporter: Hardware and OS metrics (CPU, memory, disk).
blackbox_exporter: Probing of external endpoints via HTTP, DNS, TCP, ICMP.
mysqld_exporter / postgres_exporter / redis_exporter: Database-specific metrics.

You can check the complete list in the official documentation.

Metrics Format

When accessing an exporter's endpoint, you'll see data in plain text. Examples of what each exporter generates:

From node_exporter (System):

node_cpu_seconds_total{instance="instance-dev", cpu="0", mode="idle"} 145893.45
node_memory_MemAvailable_bytes{instance="instance-dev"} 4294967296

From cAdvisor (Containers):

container_cpu_usage_seconds_total{instance="instance-dev", name="my-app", image="nginx:latest"} 234.56

Application Metrics:

http_requests_total{instance="instance-dev", method="GET", status="200"} 1547
http_request_duration_seconds{instance="instance-dev", endpoint="/api/users"} 0.234

Time-Series Database (TSDB)

The data that Prometheus collects is, by definition, time series: numerical values that change over time, always associated with a timestamp.

To store this efficiently, Prometheus uses its own TSDB, optimized for this use case. It doesn't use a traditional SQL database.

How Prometheus Stores Data

Prometheus stores metrics on disk in structures called blocks.

Figure 3: Prometheus TSDB lifecycle and storage flow

Recent data is kept in memory for fast access and periodically flushed to disk.
Each block contains compressed samples, an index for fast lookups, and metadata.
Architecturally, the TSDB is focused on high write performance ("append-only").

Retention: By default, Prometheus keeps data locally for only 15 days. Older blocks are deleted to free up space. Prometheus was not natively designed to be a long-term storage solution, although the retention configuration can be changed.

PromQL

PromQL (Prometheus Query Language) is the integrated functional query language for retrieving and analyzing data. It's through it that you create dashboards in Grafana or define alerts.

The language allows you to select, filter, aggregate, and perform complex mathematical operations on time series.

Query Examples

# Simple metric selection (current value)
http_requests_total

# Filtering by labels
http_requests_total{instance="instance-dev", status="200"}

# Requests per second rate (average over the last 5 minutes)
rate(http_requests_total[5m])

# Global sum of request rate, aggregated by instance
sum(rate(http_requests_total[5m])) by (instance)

# Available memory percentage calculation
(node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100

# CPU usage (everything that's not 'idle')
100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

PromQL is vast and includes functions for working with percentiles (histogram_quantile), linear predictions (predict_linear), and temporal comparisons (offset).

Push Gateway

As mentioned, Prometheus uses a pull model. But what about short-lived jobs (like a batch backup script) that start and finish before Prometheus has a chance to scrape?

For this, there's the Push Gateway. It functions as an intermediate cache. The short-lived job "pushes" its metrics to the Push Gateway upon completion. Prometheus, in turn, scrapes the Push Gateway at its regular interval.

Example of sending via shell script

# Sends a metric indicating how long a job took
echo "job_duration_seconds 45.2" | curl --data-binary @- \
  http://pushgateway:9091/metrics/job/batch-job/instance/worker-1

Important Note

The Push Gateway is intended for very specific use cases. It should not be used to convert Prometheus into a push-based system. The pull model is preferable in most cases because it allows Prometheus to control the load, easily detect if a target is inactive (up/down), and simplifies service discovery.

Alert Manager

It's common to confuse the responsibilities here.

The Prometheus Server is the one that detects the problem (evaluating a PromQL rule) and fires the alert state.
The Alert Manager is the one that receives this firing and decides what to do with it.

Alert Manager is a separate component that handles the logistics of notifications. Its main functionalities are:

Grouping: If 100 services go down simultaneously because a network switch failed, you don't want to receive 100 emails. Alert Manager groups these similar alerts into a single notification.
Inhibition: If a critical alert of "Data Center down" is active, it can inhibit smaller alerts like "Server X unreachable".
Silencing: Allows "muting" alerts during planned maintenance windows.
Routing: Sends different alerts to different channels (e.g., critical alerts to PagerDuty, warnings to Slack).

Flow Example

In Prometheus (Detection Rule):

groups:
  - name: example
    rules:
      - alert: HighCPUUsage
        expr: rate(node_cpu_seconds_total{mode!="idle"}[5m]) > 0.8
        for: 5m # If the condition is true for 5 minutes...
        labels:
          severity: warning

Prometheus detects and sends to Alert Manager.

In Alert Manager (Routing Configuration):

route:
  # Default route
  receiver: 'team-slack'
  routes:
    # Specific route for critical cases
    - match:
        severity: critical
      receiver: 'pagerduty-oncall'

receivers:
  - name: 'team-slack'
    slack_configs:
      - channel: '#alerts-general'
  - name: 'pagerduty-oncall'
    pagerduty_configs:
      - service_key: '...'

Client Libraries

Besides using exporters for ready-made systems, the best practice is to instrument your own applications so they natively expose business and performance metrics.

Prometheus offers official client libraries for Go, Java/Scala, Python, and Ruby, in addition to several others maintained by the community (.NET, Node.js, Rust, etc.).

Example: Python Instrumentation

With just a few lines of code, your application starts serving a /metrics endpoint.

from prometheus_client import Counter, Histogram, start_http_server
import time

# 1. Define the metrics
requests_total = Counter(
    'http_requests_total',
    'Total HTTP requests received',
    ['method', 'endpoint', 'status'] # Labels for dimensionality
)

request_duration = Histogram(
    'http_request_duration_seconds',
    'Histogram of request duration',
    ['endpoint']
)

# 2. Use in application code (e.g., using decorators)
@request_duration.labels(endpoint='/api/users').time()
def handle_user_request():
    # Application logic...
    time.sleep(0.1)
    # Increment the counter at the end
    requests_total.labels(method='GET', endpoint='/api/users', status='200').inc()

if __name__ == '__main__':
    # 3. Start an HTTP server to expose the metrics
    start_http_server(8000)
    print("Metrics server running on port 8000...")
    # Main application loop...

The libraries already handle complex details like thread-safety and correct data formatting.

Why Prometheus?

Summarizing the strengths that made Prometheus the market standard:

Pull Model: Facilitates flow control, debugging, and failure detection in targets.
Multidimensional Data: The use of labels in time series allows incredibly flexible analyses.
PromQL: A query language designed specifically for the nature of monitoring data.
Operational Simplicity: A single static binary, easy to deploy, without complex external dependencies.
Service Discovery: Native and dynamic integration with Kubernetes, AWS, Azure, etc., to automatically discover new targets.
Open Ecosystem: Hundreds of exporters available for almost any technology.

Limitations

Despite being powerful, Prometheus' design (focused on the simplicity and reliability of a single node) brings inherent limitations:

Single Node Architecture: The Prometheus server was not designed for native horizontal scaling. If a server gets overloaded, the solution is usually to manually divide the load (sharding) among multiple servers.
Local and Ephemeral Storage: Data lives on the server's local disk. If the server dies and the disk is lost, the data is gone. There's no native data replication.
Long-Term Retention: It's not efficient to use Prometheus to store years of historical data.
Fragmented Global View: If you have multiple Kubernetes clusters, each with its own Prometheus, you don't have a unified view of metrics from all clusters in one place natively.

These architectural limitations were the motivators for creating tools that "embrace and extend" Prometheus, such as Thanos, Cortex, and Mimir.

Thank you for reading this far!

In the next articles, I intend to explore how to overcome the limitations mentioned above.

For this, I want to talk about:

Thanos: Adding long-term storage (Object Storage) and unified global view to Prometheus.
Cortex: The original solution for multi-tenant and horizontally scalable Prometheus.
Mimir: The evolution of Cortex, focused on massive scale and ease of operation.

Tags: #Prometheus #Monitoring #Observability #CNCF #DevOps #SRE

Grafana Agent Installation and Configuration

Thiago Rodrigues — Wed, 28 May 2025 14:55:44 +0000

Grafana Agent Installation and Configuration Documentation

Introduction to Grafana Agent

Grafana Agent is a single, lightweight binary that consolidates multiple observability tools into one solution. It replaces the need to run separately:

Prometheus for metrics collection
Promtail for log collection
Node Exporter for system metrics
cAdvisor for container metrics
Other exporters from the Prometheus ecosystem

Key Advantages

Single binary: Reduces operational complexity
Smaller footprint: Lower resource usage compared to multiple agents
Native Remote Write: Direct integration with Prometheus, Mimir, Cortex
Unified configuration: Single YAML for all functionalities
Flexibility: Enable/disable components as needed

Architecture and Components

┌─────────────────────────────────────────────────────────────┐
│                    Grafana Agent                            │
├─────────────────┬─────────────────┬─────────────────────────┤
│   Integrations  │     Metrics     │         Logs            │
│                 │                 │                         │
│ ┌─────────────┐ │ ┌─────────────┐ │ ┌─────────────────────┐ │
│ │Node Exporter│ │ │ Prometheus  │ │ │     Promtail        │ │
│ │             │ │ │   Scraper   │ │ │                     │ │
│ │ ┌─────────┐ │ │ │             │ │ │ ┌─────────────────┐ │ │
│ │ │cAdvisor │ │ │ │ ┌─────────┐ │ │ │ │ Docker Logs     │ │ │
│ │ └─────────┘ │ │ │ │ WAL     │ │ │ │ │                 │ │ │
│ │             │ │ │ └─────────┘ │ │ │ │ Syslog          │ │ │
│ │ ┌─────────┐ │ │ │             │ │ │ │                 │ │ │
│ │ │Custom   │ │ │ │Remote Write │ │ │ │ File Logs       │ │ │
│ │ │Exporters│ │ │ │             │ │ │ └─────────────────┘ │ │
│ │ └─────────┘ │ │ └─────────────┘ │ │                     │ │
│ └─────────────┘ │                 │ └─────────────────────┘ │
└─────────────────┴─────────────────┴─────────────────────────┘
                        │                        │
                        ▼                        ▼
              ┌─────────────────┐    ┌─────────────────┐
              │ Mimir/Prometheus │    │      Loki       │
              │     Backend      │    │    Backend      │
              └─────────────────┘    └─────────────────┘

Grafana Agent Installation

1. Download and Installation

# Define version
AGENT_VERSION="v0.40.3"
ARCH="amd64"  # or arm64 for ARM

# Download binary
wget https://github.com/grafana/agent/releases/download/${AGENT_VERSION}/grafana-agent-linux-${ARCH}.zip

# Extract and install
unzip grafana-agent-linux-${ARCH}.zip
sudo mv grafana-agent-linux-${ARCH} /usr/local/bin/grafana-agent
sudo chmod +x /usr/local/bin/grafana-agent

# Verify installation
grafana-agent --version

2. User and Directory Creation

# Create grafana-agent user
sudo useradd --system --no-create-home --shell /bin/false grafana-agent

# Create necessary directories
sudo mkdir -p /etc/grafana-agent
sudo mkdir -p /var/lib/grafana-agent
sudo mkdir -p /var/log/grafana-agent

# Set permissions
sudo chown -R grafana-agent:grafana-agent /var/lib/grafana-agent
sudo chown -R grafana-agent:grafana-agent /var/log/grafana-agent
sudo chown grafana-agent:grafana-agent /etc/grafana-agent

3. Systemd Configuration

Create the service file:

sudo tee /etc/systemd/system/grafana-agent.service > /dev/null <<EOF
[Unit]
Description=Grafana Agent
Documentation=https://grafana.com/docs/agent/
Wants=network-online.target
After=network-online.target
Requires=network.target

[Service]
Type=simple
User=grafana-agent
Group=grafana-agent
ExecStart=/usr/local/bin/grafana-agent --config.file=/etc/grafana-agent/config.yaml --storage.path=/var/lib/grafana-agent
Restart=always
RestartSec=5
StandardOutput=journal
StandardError=journal

# Resource limits
LimitNOFILE=65536
LimitNPROC=32768

# Security
NoNewPrivileges=true
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_BIND_SERVICE
PrivateTmp=true
PrivateDevices=true
ProtectHome=true
ProtectSystem=strict
ReadWritePaths=/var/lib/grafana-agent /var/log/grafana-agent /tmp

[Install]
WantedBy=multi-user.target
EOF

4. Enable and Start Service

# Reload systemd
sudo systemctl daemon-reload

# Enable for boot startup
sudo systemctl enable grafana-agent

# Don't start yet (we'll configure first)
# sudo systemctl start grafana-agent

Complete Configuration

Main Configuration File

Create the file /etc/grafana-agent/config.yaml:

# /etc/grafana-agent/config.yaml

# Server configurations
server:
  log_level: info
  log_format: logfmt
  http_listen_port: 9090
  grpc_listen_port: 9091

# Integrations (Built-in Exporters)
integrations:
  # Integrated Node Exporter
  node_exporter:
    enabled: true
    # Filesystem paths
    rootfs_path: /
    sysfs_path: /sys
    procfs_path: /proc

    # Enabled collectors
    set_collectors:
      - uname
      - cpu
      - loadavg
      - meminfo
      - filesystem
      - netdev
      - diskstats
      - cpufreq
      - os
      - time
      - xfs
      - cpu_guest_seconds_metric
      - boottime
      - systemd
      - processes
      - nvme
      - nfs
      - netstat
      - logind
      - stat
      - vmstat

    # Relabeling configurations
    relabel_configs:
      - action: replace
        replacement: '${INSTANCE_NAME}'
        target_label: instance
      - action: replace
        replacement: '${TENANT_ID}'
        target_label: tenant

  # Integrated cAdvisor (optional)
  cadvisor:
    enabled: true
    docker_only: true
    instance: '${INSTANCE_NAME}'

    relabel_configs:
      - action: replace
        replacement: 'cadvisor'
        target_label: job
      - action: replace
        replacement: '${TENANT_ID}'
        target_label: tenant

    # Remove unnecessary labels
    metric_relabel_configs:
      - action: labeldrop
        regex: 'container_label_com_docker_compose_.*'
      - action: labeldrop
        regex: 'container_label_org_.*'

  # Integrated Process Exporter (optional)
  process_exporter:
    enabled: false
    config:
      process_names:
        - name: "{{.Comm}}"
          cmdline:
          - '.+'

# Metrics Configuration
metrics:
  # Write-Ahead Log directory
  wal_directory: /var/lib/grafana-agent/wal

  # Global configurations
  global:
    scrape_interval: 30s
    scrape_timeout: 10s
    external_labels:
      cluster: '${CLUSTER_NAME}'
      region: '${AWS_REGION}'

    # Remote Write to Mimir/Prometheus
    remote_write:
      - url: https://mimir.${DOMAIN}/api/v1/push
        headers:
          X-Scope-OrgID: '${TENANT_ID}'

        # Queue configurations
        queue_config:
          capacity: 10000
          max_samples_per_send: 2000
          batch_send_deadline: 5s
          min_shards: 1
          max_shards: 200

        # Retry configurations
        write_relabel_configs:
          - source_labels: [__name__]
            regex: 'go_.*'
            action: drop

  # Scrape configurations
  configs:
    - name: default
      scrape_configs:
        # Self-scrape
        - job_name: 'grafana-agent'
          static_configs:
            - targets: ['127.0.0.1:9090']
          scrape_interval: 30s
          relabel_configs:
            - action: replace
              replacement: '${INSTANCE_NAME}'
              target_label: instance

        # Local applications scrape
        - job_name: 'local-apps'
          static_configs:
            - targets: ['127.0.0.1:8080', '127.0.0.1:3000']
          scrape_interval: 15s
          relabel_configs:
            - action: replace
              replacement: '${INSTANCE_NAME}'
              target_label: instance
            - action: replace
              replacement: '${TENANT_ID}'
              target_label: tenant

        # Service Discovery via file
        - job_name: 'file-sd'
          file_sd_configs:
            - files:
              - '/etc/grafana-agent/targets/*.json'
              refresh_interval: 30s
          relabel_configs:
            - action: replace
              replacement: '${INSTANCE_NAME}'
              target_label: instance

# Logs Configuration
logs:
  configs:
    - name: default
      # Client for Loki
      clients:
        - url: https://loki.${DOMAIN}/loki/api/v1/push
          headers:
            X-Scope-OrgID: '${TENANT_ID}'

          # Batching configurations
          batchwait: 1s
          batchsize: 1048576

          # Retry configurations
          backoff_config:
            min_period: 500ms
            max_period: 5m
            max_retries: 10

      # Positions file
      positions:
        filename: /var/lib/grafana-agent/positions.yaml

      # Log scrape configurations
      scrape_configs:
        # Docker logs
        - job_name: docker
          docker_sd_configs:
            - host: "unix:///var/run/docker.sock"
              refresh_interval: 30s

          relabel_configs:
            - source_labels: [__meta_docker_container_name]
              target_label: container
            - source_labels: [__meta_docker_container_name]
              target_label: service_name
            - source_labels: [__meta_docker_container_log_stream]
              target_label: stream
            - action: replace
              replacement: '${INSTANCE_NAME}'
              target_label: instance
            - action: replace
              replacement: '${TENANT_ID}'
              target_label: tenant

        # System logs via journald
        - job_name: systemd
          journal:
            json: false
            max_age: 12h
            path: /var/log/journal

          relabel_configs:
            - source_labels: [__journal__systemd_unit]
              target_label: unit
            - source_labels: [__journal__hostname]
              target_label: hostname
            - action: replace
              replacement: '${INSTANCE_NAME}'
              target_label: instance
            - action: replace
              replacement: '${TENANT_ID}'
              target_label: tenant

        # Specific file logs
        - job_name: syslog
          static_configs:
            - targets: [localhost]
              labels:
                job: syslog
                tenant: '${TENANT_ID}'
                __path__: /var/log/syslog

          relabel_configs:
            - action: replace
              target_label: instance
              replacement: '${INSTANCE_NAME}'

        # Custom application logs
        - job_name: app-logs
          static_configs:
            - targets: [localhost]
              labels:
                job: app-logs
                tenant: '${TENANT_ID}'
                __path__: /var/log/myapp/*.log

          # Processing pipeline
          pipeline_stages:
            - json:
                expressions:
                  timestamp: timestamp
                  level: level
                  message: message
                  module: module

            - timestamp:
                source: timestamp
                format: RFC3339Nano

            - labels:
                level:
                module:

# Traces Configuration (optional)
traces:
  configs:
    - name: default
      receivers:
        jaeger:
          protocols:
            thrift_http:
              endpoint: 0.0.0.0:14268
            grpc:
              endpoint: 0.0.0.0:14250

        otlp:
          protocols:
            grpc:
              endpoint: 0.0.0.0:4317
            http:
              endpoint: 0.0.0.0:4318

      remote_write:
        - endpoint: https://tempo.${DOMAIN}:443
          headers:
            X-Scope-OrgID: '${TENANT_ID}'

Environment Variables File

Create /etc/grafana-agent/environment:

# /etc/grafana-agent/environment

# Instance identification
INSTANCE_NAME="server-001"
TENANT_ID="company"

# Cluster and region
CLUSTER_NAME="production"
AWS_REGION="us-east-1"

# Base domain
DOMAIN="monitoring.company.com"

# Credentials (if needed)
# AWS_ACCESS_KEY_ID="your-access-key"
# AWS_SECRET_ACCESS_KEY="your-secret-key"

Configuration Script with Variable Substitution

Create /usr/local/bin/setup-grafana-agent.sh:

#!/bin/bash

# Script to configure Grafana Agent with variable substitution

set -e

# Load environment variables
source /etc/grafana-agent/environment

# Function to substitute variables in configuration file
substitute_variables() {
    local config_file="/etc/grafana-agent/config.yaml"
    local temp_file="/tmp/config.yaml.tmp"

    # Substitute variables
    envsubst < "${config_file}.template" > "${temp_file}"

    # Validate configuration
    if grafana-agent --config.file="${temp_file}" --config.validate; then
        mv "${temp_file}" "${config_file}"
        chown grafana-agent:grafana-agent "${config_file}"
        echo "Configuration updated successfully"
    else
        echo "Configuration validation error"
        rm -f "${temp_file}"
        exit 1
    fi
}

# Check if template exists
if [[ ! -f "/etc/grafana-agent/config.yaml.template" ]]; then
    echo "Configuration template not found"
    exit 1
fi

# Substitute variables
substitute_variables

# Reload service if running
if systemctl is-active --quiet grafana-agent; then
    systemctl reload grafana-agent
    echo "Service reloaded"
fi

Dynamic Service Discovery

File SD Configuration

Create directory for targets:

sudo mkdir -p /etc/grafana-agent/targets
sudo chown grafana-agent:grafana-agent /etc/grafana-agent/targets

Example targets file (/etc/grafana-agent/targets/web-servers.json):

[
  {
    "targets": ["192.168.1.10:9100", "192.168.1.11:9100"],
    "labels": {
      "job": "node-exporter",
      "env": "production",
      "team": "infrastructure"
    }
  },
  {
    "targets": ["192.168.1.20:8080", "192.168.1.21:8080"],
    "labels": {
      "job": "web-app",
      "env": "production",
      "team": "backend"
    }
  }
]

Management Scripts

Status Script

#!/bin/bash
# /usr/local/bin/grafana-agent-status.sh

echo "=== Grafana Agent Status ==="
systemctl status grafana-agent --no-pager

echo ""
echo "=== Recent logs ==="
journalctl -u grafana-agent --no-pager -n 20

echo ""
echo "=== Resource usage ==="
ps aux | grep grafana-agent | grep -v grep

echo ""
echo "=== Connectivity check ==="
curl -s http://localhost:9090/-/ready && echo "Agent ready" || echo "Agent not ready"

echo ""
echo "=== Status metrics ==="
curl -s http://localhost:9090/metrics | grep -E 'prometheus_agent_|agent_build_info'

Configuration Backup Script

#!/bin/bash
# /usr/local/bin/backup-agent-config.sh

BACKUP_DIR="/var/backups/grafana-agent"
DATE=$(date +%Y%m%d_%H%M%S)

mkdir -p "$BACKUP_DIR"

# Configuration backup
tar -czf "$BACKUP_DIR/grafana-agent-config-$DATE.tar.gz" \
    /etc/grafana-agent/ \
    /etc/systemd/system/grafana-agent.service

# Keep only last 10 backups
ls -t "$BACKUP_DIR"/grafana-agent-config-*.tar.gz | tail -n +11 | xargs -r rm

echo "Backup created: $BACKUP_DIR/grafana-agent-config-$DATE.tar.gz"

Initialization and Verification

First Startup

# 1. Create configuration from template
sudo cp /etc/grafana-agent/config.yaml /etc/grafana-agent/config.yaml.template

# 2. Run configuration script
sudo /usr/local/bin/setup-grafana-agent.sh

# 3. Validate configuration
sudo grafana-agent --config.file=/etc/grafana-agent/config.yaml --config.validate

# 4. Start service
sudo systemctl start grafana-agent

# 5. Check status
sudo systemctl status grafana-agent

# 6. Check logs
sudo journalctl -u grafana-agent -f

Verification Commands

# Service status
systemctl status grafana-agent

# Real-time logs
journalctl -u grafana-agent -f

# Check if listening on ports
ss -tulpn | grep grafana-agent

# Check agent metrics
curl http://localhost:9090/metrics

# Check readiness
curl http://localhost:9090/-/ready

# Check current configuration
curl http://localhost:9090/-/config

# Check discovered targets
curl http://localhost:9090/api/v1/targets

Monitoring and Troubleshooting

Key Agent Metrics

# Agent CPU usage
rate(process_cpu_seconds_total{job="grafana-agent"}[5m])

# Memory used
process_resident_memory_bytes{job="grafana-agent"}

# Samples sent via remote write
rate(prometheus_remote_storage_samples_total[5m])

# Remote write failures
rate(prometheus_remote_storage_samples_failed_total[5m])

# WAL size
prometheus_tsdb_wal_size_bytes

# Logs sent
rate(promtail_sent_entries_total[5m])

Common Issues and Solutions

1. Remote Write Failures

# Check connectivity
curl -I https://mimir.domain.com/api/v1/push

# Check certificates
openssl s_client -connect mimir.domain.com:443

# Check specific logs
journalctl -u grafana-agent | grep "remote_write"

2. High Memory Usage

# Adjust configurations in config.yaml
metrics:
  global:
    remote_write:
      - queue_config:
          capacity: 5000        # Reduce from 10000
          max_samples_per_send: 1000  # Reduce from 2000

3. Logs Not Being Collected

# Check permissions
ls -la /var/log/
ls -la /var/run/docker.sock

# Add user to docker group (for Docker logs)
sudo usermod -a -G docker grafana-agent
sudo systemctl restart grafana-agent

Advanced Configurations

Rate Limiting

metrics:
  configs:
    - name: default
      scrape_configs:
        - job_name: 'rate-limited-app'
          static_configs:
            - targets: ['app:8080']
          scrape_interval: 1m  # Less frequent for sensitive apps
          metrics_path: /metrics
          honor_labels: true

Metrics Filtering

metrics:
  global:
    remote_write:
      - url: https://mimir.domain.com/api/v1/push
        write_relabel_configs:
          # Drop specific metrics
          - source_labels: [__name__]
            regex: 'go_gc_.*|go_memstats_.*'
            action: drop

          # Keep only important metrics
          - source_labels: [__name__]
            regex: 'up|cpu_usage_.*|memory_usage_.*'
            action: keep

Multi-tenancy

metrics:
  global:
    remote_write:
      # Tenant A
      - url: https://mimir.domain.com/api/v1/push
        headers:
          X-Scope-OrgID: 'tenant-a'
        write_relabel_configs:
          - source_labels: [tenant]
            regex: 'tenant-a'
            action: keep

      # Tenant B  
      - url: https://mimir.domain.com/api/v1/push
        headers:
          X-Scope-OrgID: 'tenant-b'
        write_relabel_configs:
          - source_labels: [tenant]
            regex: 'tenant-b'
            action: keep

Conclusion

Grafana Agent provides a unified and efficient solution for observability, consolidating multiple tools into a single binary. With native remote write and embedded integrations, it significantly simplifies monitoring architecture, reducing operational complexity and resource overhead.

Implementation Benefits:

Simplicity: Single agent for metrics, logs, and traces
Efficiency: Lower resource usage than multiple agents
Flexibility: Modular configuration by need
Scalability: Native remote write for distributed backends
Maintenance: Centralized management via systemd

Grafana K8s Stack Implementation on Kubernetes Cluster

Thiago Rodrigues — Wed, 28 May 2025 14:55:29 +0000

Grafana K8s Stack Implementation on Kubernetes Cluster

Introduction

This article describes the complete implementation of an observability stack based on the Grafana ecosystem in a Kubernetes cluster. The stack includes:

Grafana Alloy: Metrics and logs collection agent
Loki: Log aggregation system
Mimir: Long-term metrics storage system
Grafana: Visualization interface and dashboards

Solution Architecture

The solution uses a distributed architecture where:

Alloy collects metrics and logs from the cluster
Loki stores and indexes logs with configurable retention
Mimir stores long-term metrics
Grafana provides the visualization interface
AWS Load Balancer Controller manages ingress

Prerequisites

Running Kubernetes cluster
Helm 3.x installed
AWS Load Balancer Controller configured
Valid TLS certificates
S3 buckets configured for storage

1. Grafana Alloy Configuration

Alloy is responsible for collecting metrics and logs from the Kubernetes cluster.

# values-alloy.yaml
alloy-logs:
  config:
    client:
      timeout: 30s
    resources:
      limits:
        memory: 256Mi
      requests:
        cpu: 100m
        memory: 128Mi
  enabled: true

alloy-metrics:
  config:
    global:
      external_labels:
        cluster: ${CLUSTER_NAME}
        source: alloy-agent
      scrape_interval: 3m
      scrape_timeout: 2m30s
    remote_write:
    - headers:
        X-Scope-OrgID: ${TENANT_ID}
      queue_config:
        batch_send_deadline: 30s
        max_backoff: 10s
        max_retries: 10
        max_samples_per_send: 1000
        min_backoff: 100ms
        retry_on_http_429: true
      send_exemplars: true
      url: http://mimir-gateway.mimir.svc.cluster.local/api/v1/push
  enabled: true
  resources:
    limits:
      memory: 768Mi
    requests:
      cpu: 250m
      memory: 384Mi

cluster:
  name: ${CLUSTER_NAME}

clusterEvents:
  enabled: true
  scrapeInterval: 3m

clusterMetrics:
  cadvisor:
    enabled: true
    metricRelabelings:
    - action: keep
      regex: container_cpu_.*|container_memory_.*|container_network_.*|container_fs_.*|machine_cpu_.*|container_spec_.*
      sourceLabels:
      - __name__
    relabelings:
    - action: labelmap
      regex: __meta_kubernetes_node_label_(.+)
  enabled: true
  kubelet:
    enabled: true
  nodeExporter:
    enabled: true

externalServices:
  loki:
    externalLabels:
      cluster: ${CLUSTER_NAME}
      tenant: ${TENANT_ID}
    host: http://loki-loki-distributed-gateway.loki.svc.cluster.local
    tenantId: ${TENANT_ID}
    writeEndpoint: /loki/api/v1/push
  prometheus:
    extraHeaders:
      X-Scope-OrgID: ${TENANT_ID}
    host: http://mimir-gateway.mimir.svc.cluster.local
    tenantId: ${TENANT_ID}
    writeEndpoint: /api/v1/push

extraScrapeConfigs: |-
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: pod
      - source_labels: [__meta_kubernetes_pod_container_name]
        action: replace
        target_label: container
    scrape_interval: 3m
    scrape_timeout: 2m30s

  - job_name: 'kubernetes-cadvisor'
    scheme: https
    tls_config:
      ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      insecure_skip_verify: true
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    kubernetes_sd_configs:
    - role: node
    relabel_configs:
    - action: labelmap
      regex: __meta_kubernetes_node_label_(.+)
    - target_label: __address__
      replacement: kubernetes.default.svc:443
    - source_labels: [__meta_kubernetes_node_name]
      regex: (.+)
      target_label: __metrics_path__
      replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
    metric_relabel_configs:
    - source_labels: [__name__]
      regex: 'container_cpu_.*|container_memory_.*|machine_cpu_.*|container_fs_.*|container_network_.*|container_spec_.*'
      action: keep
    scrape_interval: 3m
    scrape_timeout: 2m30s

kube-state-metrics:
  enabled: true
  metricLabelsAllowlist:
  - pods=[app,name,component]
  - deployments=[app,name,component]
  - statefulsets=[app,name,component]
  resources:
    limits:
      memory: 256Mi
    requests:
      cpu: 100m
      memory: 128Mi

logs:
  enabled: true
  extraClientConfig: |
    client:
      timeout: 10s
      batchwait: 1s 
      batchsize: 1048576
      max_retries: 5
      retry_on_status_codes: [429, 500, 502, 503, 504]
      backoff_config:
        min_period: 500ms
        max_period: 5m
        max_retries: 10
  pod_logs:
    discovery: all
    enabled: true
    namespaceAllowlist:
    - .*
    scrapeInterval: 3m
    stages:
    - json:
        expressions:
          level: level
          message: msg
          timestamp: time
    - timestamp:
        format: RFC3339Nano
        source: timestamp
    - labels:
        level: null

metrics:
  enabled: true
  scrapeInterval: 3m

opencost:
  enabled: false

prometheus-node-exporter:
  enabled: true

traces:
  enabled: false

Alloy Installation

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

# Create namespace
kubectl create namespace monitoring

# Install Alloy
helm upgrade --install alloy grafana/k8s-monitoring \
  --namespace monitoring \
  --values values-alloy.yaml \
  --set cluster.name=${CLUSTER_NAME} \
  --set externalServices.loki.tenantId=${TENANT_ID} \
  --set externalServices.prometheus.tenantId=${TENANT_ID}

2. Loki Configuration

Loki is responsible for log aggregation and storage.

# values-loki.yaml
nameOverride: "loki-distributed"

compactor:
  enabled: true
  persistence:
    enabled: false
  replicas: 1

distributor:
  maxUnavailable: 1
  replicas: 2

extraEnvVars:
- name: AWS_ACCESS_KEY_ID
  valueFrom:
    secretKeyRef:
      name: aws-credentials
      key: access_key_id
- name: AWS_SECRET_ACCESS_KEY
  valueFrom:
    secretKeyRef:
      name: aws-credentials
      key: secret_access_key

gateway:
  enabled: true
  ingress:
    annotations:
      alb.ingress.kubernetes.io/certificate-arn: ${TLS_CERTIFICATE_ARN}
      alb.ingress.kubernetes.io/group.name: ${ALB_GROUP_NAME}
      alb.ingress.kubernetes.io/healthcheck-path: /ready
      alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
      alb.ingress.kubernetes.io/scheme: internet-facing
      alb.ingress.kubernetes.io/ssl-redirect: "443"
      alb.ingress.kubernetes.io/target-type: ip
      kubernetes.io/ingress.class: alb
    enabled: true
    hosts:
    - host: loki.${DOMAIN}
      paths:
      - path: /
        pathType: Prefix
    ingressClassName: alb
  replicas: 1
  service:
    port: 80
    type: ClusterIP

ingester:
  maxUnavailable: 1
  persistence:
    enabled: false
  replicas: 2

loki:
  commonConfig:
    path_prefix: /var/loki
    replication_factor: 1
  structuredConfig:
    auth_enabled: true
    common:
      storage:
        filesystem:
          chunks_directory: /var/loki/chunks
          rules_directory: /var/loki/rules
    limits_config:
      ingestion_burst_size_mb: 6
      ingestion_rate_mb: 4
      max_global_streams_per_user: 15000
      max_query_series: 500
      reject_old_samples: false
      reject_old_samples_max_age: 168h
      retention_period: 24h
      volume_enabled: true

    compactor:
      working_directory: /var/loki/compactor
      shared_store: s3
      compaction_interval: 10m
      retention_enabled: true
      retention_delete_delay: 1h
      retention_delete_worker_count: 150

    memberlist:
      join_members:
        - loki-loki-distributed-memberlist.loki.svc.cluster.local:7946

    schema_config:
      configs:
        - from: "2020-10-24"
          index:
            period: 24h
            prefix: index_
          object_store: aws
          schema: v11
          store: boltdb-shipper

    server:
      grpc_listen_port: 9095
      http_listen_port: 3100

    storage_config:
      aws:
        access_key_id: ${AWS_ACCESS_KEY_ID}
        bucketnames: ${LOKI_S3_BUCKET}
        region: ${AWS_REGION}
        s3: s3://s3.amazonaws.com
        secret_access_key: ${AWS_SECRET_ACCESS_KEY}
      boltdb_shipper:
        active_index_directory: /var/loki/index
        cache_location: /var/loki/boltdb-cache
        shared_store: s3

minio:
  enabled: false

nginx:
  enabled: false

querier:
  replicas: 1

query_frontend:
  maxUnavailable: 0
  replicas: 1

resources:
  limits:
    cpu: 500m
    memory: 512Mi
  requests:
    cpu: 100m
    memory: 256Mi

ruler:
  enabled: false

singleBinary:
  enabled: false

Loki Installation

# Create secret for AWS credentials
kubectl create secret generic aws-credentials \
  --from-literal=access_key_id=${AWS_ACCESS_KEY_ID} \
  --from-literal=secret_access_key=${AWS_SECRET_ACCESS_KEY} \
  --namespace loki

# Install Loki
helm repo add grafana https://grafana.github.io/helm-charts
helm upgrade --install loki grafana/loki-distributed \
  --namespace loki \
  --create-namespace \
  --values values-loki.yaml

3. Mimir Configuration

Mimir is responsible for long-term metrics storage.

# values-mimir.yaml
alertmanager:
  enabled: false

compactor:
  persistentVolume:
    enabled: false
  replicas: 1

distributor:
  replicas: 2
  extraEnvVars:
  - name: AWS_ACCESS_KEY_ID
    valueFrom:
      secretKeyRef:
        name: aws-credentials
        key: access_key_id
  - name: AWS_SECRET_ACCESS_KEY
    valueFrom:
      secretKeyRef:
        name: aws-credentials
        key: secret_access_key

gateway:
  enabled: true
  enabledNonEnterprise: true
  ingress:
    annotations:
      alb.ingress.kubernetes.io/certificate-arn: ${TLS_CERTIFICATE_ARN}
      alb.ingress.kubernetes.io/group.name: ${ALB_GROUP_NAME}
      alb.ingress.kubernetes.io/healthcheck-path: /ready
      alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
      alb.ingress.kubernetes.io/scheme: internet-facing
      alb.ingress.kubernetes.io/ssl-redirect: "443"
      alb.ingress.kubernetes.io/target-type: ip
      kubernetes.io/ingress.class: alb
    enabled: true
    hosts:
      - host: mimir.${DOMAIN}
        paths:
          - path: /
            pathType: Prefix
    ingressClassName: alb
  replicas: 1
  service:
    port: 80
    type: ClusterIP

ingester:
  persistentVolume:
    enabled: false
  replicas: 2

mimir:
  structuredConfig:
    alertmanager_storage:
      backend: s3
      s3:
        bucket_name: ${MIMIR_ALERTS_BUCKET}
    blocks_storage:
      backend: s3
      bucket_store:
        sync_dir: /data/tsdb-sync
      s3:
        bucket_name: ${MIMIR_BLOCKS_BUCKET}
      tsdb:
        dir: /data/tsdb
    common:
      storage:
        backend: s3
        s3:
          access_key_id: ${AWS_ACCESS_KEY_ID}
          bucket_name: ${MIMIR_BLOCKS_BUCKET}
          endpoint: s3.amazonaws.com
          region: ${AWS_REGION}
          secret_access_key: ${AWS_SECRET_ACCESS_KEY}
    frontend:
      log_queries_longer_than: 10s
    limits:
      ingestion_burst_size: 150000
      ingestion_rate: 50000
      max_global_series_per_metric: 150000
      max_global_series_per_user: 2000000
    memberlist:
      abort_if_cluster_join_fails: false
      join_members:
        - mimir-gossip-ring.mimir.svc.cluster.local:7946
    multitenancy_enabled: true
    ruler_storage:
      backend: s3
      s3:
        bucket_name: ${MIMIR_RULES_BUCKET}
    server:
      grpc_listen_port: 9095
      http_listen_port: 8080

minio:
  enabled: false

mode: microservices

nginx:
  enabled: false

querier:
  replicas: 1

query_frontend:
  replicas: 1

query_scheduler:
  enabled: true
  replicas: 2
  resources:
    limits:
      cpu: 500m
      memory: 512Mi
    requests:
      cpu: 100m
      memory: 256Mi

ruler:
  enabled: false

store_gateway:
  persistentVolume:
    enabled: false
  replicas: 1

Mimir Installation

# Create secret for AWS credentials (if not exists)
kubectl create secret generic aws-credentials \
  --from-literal=access_key_id=${AWS_ACCESS_KEY_ID} \
  --from-literal=secret_access_key=${AWS_SECRET_ACCESS_KEY} \
  --namespace mimir

# Install Mimir
helm repo add grafana https://grafana.github.io/helm-charts
helm upgrade --install mimir grafana/mimir-distributed \
  --namespace mimir \
  --create-namespace \
  --values values-mimir.yaml

4. Grafana Configuration

# values-grafana.yaml
fullnameOverride: "grafana"

# Image configuration
image:
  repository: grafana/grafana
  tag: "12.0.0"

# Admin credentials using secret
admin:
  existingSecret: "grafana-admin-credentials"
  userKey: "admin-user"
  passwordKey: "admin-password"

# Service configuration
service:
  type: ClusterIP
  port: 80

# Persistence
persistence:
  enabled: true
  size: 5Gi

# Database configuration using secrets
env:
  GF_DATABASE_TYPE: postgres
  GF_DATABASE_HOST: ${DATABASE_HOST}
  GF_DATABASE_PORT: "5432"
  GF_DATABASE_NAME: grafana
  GF_DATABASE_USER: ${DATABASE_USER}
  GF_DATABASE_PASSWORD: ${DATABASE_PASSWORD}
  GF_DATABASE_SSL_MODE: disable
  GF_LOG_LEVEL: warn

# Grafana configuration
grafana.ini:
  server:
    domain: grafana.${DOMAIN}
    root_url: https://grafana.${DOMAIN}

# Ingress configuration
ingress:
  enabled: true
  ingressClassName: alb
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/group.name: ${ALB_GROUP_NAME}
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/healthcheck-path: /api/health
    alb.ingress.kubernetes.io/certificate-arn: ${TLS_CERTIFICATE_ARN}
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
    alb.ingress.kubernetes.io/ssl-redirect: '443'
  hosts:
    - grafana.${DOMAIN}
  path: /
  pathType: Prefix

Grafana Installation

# Create necessary secrets
kubectl create secret generic grafana-admin-credentials \
  --from-literal=admin-user=${GRAFANA_ADMIN_USER} \
  --from-literal=admin-password=${GRAFANA_ADMIN_PASSWORD} \
  --namespace grafana

# Install Grafana
helm repo add grafana https://grafana.github.io/helm-charts
helm upgrade --install grafana grafana/grafana \
  --namespace grafana \
  --create-namespace \
  --values values-grafana.yaml

5. IngressClass Configuration

# ingress-class.yaml
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: alb
spec:
  controller: ingress.k8s.aws/alb

kubectl apply -f ingress-class.yaml

6. Grafana Agent Configuration (Standalone)

For monitoring external resources outside the cluster:

# agent-config.yaml
server:
  log_level: debug

integrations:
  node_exporter:
    enabled: true
    rootfs_path: /
    sysfs_path: /sys
    procfs_path: /proc
    set_collectors:
    - uname
    - cpu
    - loadavg
    - meminfo
    - filesystem
    - netdev
    - diskstats
    - cpufreq
    - os
    - time
    - xfs
    - cpu_guest_seconds_metric
    - boottime
    - systemd
    - processes
    - nvme
    - nfs
    - netstat
    - logind
    - stat
    - vmstat
    relabel_configs:
    - action: replace
      replacement: '${INSTANCE_NAME}'
      target_label: instance
    - action: replace
      replacement: '${TENANT_ID}'
      target_label: tenant

metrics:
  wal_directory: /tmp/grafana-agent-wal
  global:
    scrape_interval: 30s
    remote_write:
    - url: https://mimir.${DOMAIN}/api/v1/push
      headers:
        X-Scope-OrgID: ${TENANT_ID}
  configs:
  - name: default
    scrape_configs:
    - job_name: agent
      static_configs:
      - targets: [ '127.0.0.1:9090' ]

logs:
  configs:
  - name: default
    clients:
    - url: https://loki.${DOMAIN}/loki/api/v1/push
      headers:
        X-Scope-OrgID: ${TENANT_ID}
    positions:
      filename: /tmp/positions.yaml
    scrape_configs:
    - job_name: docker
      docker_sd_configs:
      - host: "unix:///var/run/docker.sock"
        refresh_interval: 30s
      relabel_configs:
      - source_labels: [ '__meta_docker_container_name' ]
        target_label: 'container'
      - source_labels: [ '__meta_docker_container_name' ]
        target_label: 'service_name'
      - source_labels: [ '__meta_docker_container_log_stream' ]
        target_label: 'stream'
      - action: replace
        replacement: '${INSTANCE_NAME}'
        target_label: instance
      - action: replace
        replacement: '${TENANT_ID}'
        target_label: tenant
    - job_name: syslog
      static_configs:
      - targets: [ 'localhost' ]
        labels:
          job: syslog
          tenant: ${TENANT_ID}
      relabel_configs:
      - action: replace
        target_label: instance
        replacement: '${INSTANCE_NAME}'
      - action: replace
        target_label: tenant
        replacement: '${TENANT_ID}'

7. Complete Deployment Script

#!/bin/bash

# Environment variables
export CLUSTER_NAME="production-cluster"
export TENANT_ID="company"
export DOMAIN="monitoring.company.com"
export ALB_GROUP_NAME="monitoring-alb"
export TLS_CERTIFICATE_ARN="arn:aws:acm:region:account:certificate/cert-id"
export AWS_REGION="us-east-1"
export LOKI_S3_BUCKET="company-loki-data"
export MIMIR_BLOCKS_BUCKET="company-mimir-blocks"
export MIMIR_ALERTS_BUCKET="company-mimir-alerts"
export MIMIR_RULES_BUCKET="company-mimir-rules"
export DATABASE_HOST="postgres.company.internal"
export DATABASE_USER="grafana"
export DATABASE_PASSWORD="secure-password"
export GRAFANA_ADMIN_USER="admin"
export GRAFANA_ADMIN_PASSWORD="admin-password"

# Add Helm repositories
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

# Create namespaces
kubectl create namespace monitoring
kubectl create namespace loki
kubectl create namespace mimir
kubectl create namespace grafana

# Create secrets
kubectl create secret generic aws-credentials \
  --from-literal=access_key_id=${AWS_ACCESS_KEY_ID} \
  --from-literal=secret_access_key=${AWS_SECRET_ACCESS_KEY} \
  --namespace loki

kubectl create secret generic aws-credentials \
  --from-literal=access_key_id=${AWS_ACCESS_KEY_ID} \
  --from-literal=secret_access_key=${AWS_SECRET_ACCESS_KEY} \
  --namespace mimir

kubectl create secret generic grafana-admin-credentials \
  --from-literal=admin-user=${GRAFANA_ADMIN_USER} \
  --from-literal=admin-password=${GRAFANA_ADMIN_PASSWORD} \
  --namespace grafana

# Deploy Loki
helm upgrade --install loki grafana/loki-distributed \
  --namespace loki \
  --values values-loki.yaml

# Deploy Mimir
helm upgrade --install mimir grafana/mimir-distributed \
  --namespace mimir \
  --values values-mimir.yaml

# Deploy Grafana
helm upgrade --install grafana grafana/grafana \
  --namespace grafana \
  --values values-grafana.yaml

# Deploy Alloy
helm upgrade --install alloy grafana/k8s-monitoring \
  --namespace monitoring \
  --values values-alloy.yaml

echo "Deployment completed! Wait a few minutes for all pods to be ready."

8. Verification and Monitoring

Check pod status:

kubectl get pods -n monitoring
kubectl get pods -n loki  
kubectl get pods -n mimir
kubectl get pods -n grafana

Check ingress:

kubectl get ingress -A

Access logs:

kubectl logs -f deployment/alloy -n monitoring
kubectl logs -f deployment/loki-distributor -n loki
kubectl logs -f deployment/mimir-distributor -n mimir

9. Grafana Data Sources Configuration

After deployment, configure the following data sources in Grafana:

Loki Data Source:

URL: http://loki-loki-distributed-gateway.loki.svc.cluster.local
Headers: X-Scope-OrgID: ${TENANT_ID}

Mimir Data Source:

URL: http://mimir-gateway.mimir.svc.cluster.local
Headers: X-Scope-OrgID: ${TENANT_ID}

10. Production Considerations

Security:

Use Kubernetes secrets for sensitive credentials
Configure appropriate RBAC
Enable TLS for all communications

Performance:

Adjust resource limits based on data volume
Configure appropriate retention
Use suitable storage classes

Backup:

Configure S3 data backup
Document restore procedures
Regularly test backups

Monitoring:

Configure alerts for component failures
Monitor resource usage
Set up health check dashboards

Conclusion

This implementation provides a complete observability solution for Kubernetes environments, with high availability, scalability, and native AWS integration. The stack is suitable for production environments and can be customized according to specific needs.

Next Steps:

Configure custom dashboards
Implement critical alerts
Configure retention policies
Optimize performance based on real metrics

Instalação e Configuração do Grafana Agent

Thiago Rodrigues — Wed, 28 May 2025 14:54:58 +0000

Documentação de Instalação e Configuração do Grafana Agent

Introdução ao Grafana Agent

O Grafana Agent é um binário único e leve que consolida múltiplas ferramentas de observabilidade em uma única solução. Ele substitui a necessidade de executar separadamente:

Prometheus para coleta de métricas
Promtail para coleta de logs
Node Exporter para métricas de sistema
cAdvisor para métricas de containers
Outros exporters do ecossistema Prometheus

Principais Vantagens

Único binário: Reduz complexidade operacional
Menor footprint: Menor uso de recursos comparado a múltiplos agentes
Remote Write nativo: Integração direta com Prometheus, Mimir, Cortex
Configuração unificada: Single YAML para todas as funcionalidades
Flexibilidade: Habilita/desabilita componentes conforme necessário

Arquitetura e Componentes

┌─────────────────────────────────────────────────────────────┐
│                    Grafana Agent                            │
├─────────────────┬─────────────────┬─────────────────────────┤
│   Integrations  │     Metrics     │         Logs            │
│                 │                 │                         │
│ ┌─────────────┐ │ ┌─────────────┐ │ ┌─────────────────────┐ │
│ │Node Exporter│ │ │ Prometheus  │ │ │     Promtail        │ │
│ │             │ │ │   Scraper   │ │ │                     │ │
│ │ ┌─────────┐ │ │ │             │ │ │ ┌─────────────────┐ │ │
│ │ │cAdvisor │ │ │ │ ┌─────────┐ │ │ │ │ Docker Logs     │ │ │
│ │ └─────────┘ │ │ │ │ WAL     │ │ │ │ │                 │ │ │
│ │             │ │ │ └─────────┘ │ │ │ │ Syslog          │ │ │
│ │ ┌─────────┐ │ │ │             │ │ │ │                 │ │ │
│ │ │Custom   │ │ │ │Remote Write │ │ │ │ File Logs       │ │ │
│ │ │Exporters│ │ │ │             │ │ │ └─────────────────┘ │ │
│ │ └─────────┘ │ │ └─────────────┘ │ │                     │ │
│ └─────────────┘ │                 │ └─────────────────────┘ │
└─────────────────┴─────────────────┴─────────────────────────┘
                        │                        │
                        ▼                        ▼
              ┌─────────────────┐    ┌─────────────────┐
              │ Mimir/Prometheus │    │      Loki       │
              │     Backend      │    │    Backend      │
              └─────────────────┘    └─────────────────┘

Instalação do Grafana Agent

1. Download e Instalação

# Definir versão
AGENT_VERSION="v0.40.3"
ARCH="amd64"  # ou arm64 para ARM

# Download do binário
wget https://github.com/grafana/agent/releases/download/${AGENT_VERSION}/grafana-agent-linux-${ARCH}.zip

# Extrair e instalar
unzip grafana-agent-linux-${ARCH}.zip
sudo mv grafana-agent-linux-${ARCH} /usr/local/bin/grafana-agent
sudo chmod +x /usr/local/bin/grafana-agent

# Verificar instalação
grafana-agent --version

2. Criação do Usuário e Diretórios

# Criar usuário grafana-agent
sudo useradd --system --no-create-home --shell /bin/false grafana-agent

# Criar diretórios necessários
sudo mkdir -p /etc/grafana-agent
sudo mkdir -p /var/lib/grafana-agent
sudo mkdir -p /var/log/grafana-agent

# Definir permissões
sudo chown -R grafana-agent:grafana-agent /var/lib/grafana-agent
sudo chown -R grafana-agent:grafana-agent /var/log/grafana-agent
sudo chown grafana-agent:grafana-agent /etc/grafana-agent

3. Configuração do Systemd

Criar o arquivo de serviço:

sudo tee /etc/systemd/system/grafana-agent.service > /dev/null <<EOF
[Unit]
Description=Grafana Agent
Documentation=https://grafana.com/docs/agent/
Wants=network-online.target
After=network-online.target
Requires=network.target

[Service]
Type=simple
User=grafana-agent
Group=grafana-agent
ExecStart=/usr/local/bin/grafana-agent --config.file=/etc/grafana-agent/config.yaml --storage.path=/var/lib/grafana-agent
Restart=always
RestartSec=5
StandardOutput=journal
StandardError=journal

# Limites de recursos
LimitNOFILE=65536
LimitNPROC=32768

# Segurança
NoNewPrivileges=true
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_BIND_SERVICE
PrivateTmp=true
PrivateDevices=true
ProtectHome=true
ProtectSystem=strict
ReadWritePaths=/var/lib/grafana-agent /var/log/grafana-agent /tmp

[Install]
WantedBy=multi-user.target
EOF

4. Habilitar e Iniciar o Serviço

# Recarregar systemd
sudo systemctl daemon-reload

# Habilitar para iniciar no boot
sudo systemctl enable grafana-agent

# Não iniciar ainda (vamos configurar primeiro)
# sudo systemctl start grafana-agent

Configuração Completa

Arquivo de Configuração Principal

Criar o arquivo /etc/grafana-agent/config.yaml:

# /etc/grafana-agent/config.yaml

# Configurações do servidor
server:
  log_level: info
  log_format: logfmt
  http_listen_port: 9090
  grpc_listen_port: 9091

# Integrações (Exporters embutidos)
integrations:
  # Node Exporter integrado
  node_exporter:
    enabled: true
    # Caminhos para sistemas de arquivos
    rootfs_path: /
    sysfs_path: /sys
    procfs_path: /proc

    # Coletores habilitados
    set_collectors:
      - uname
      - cpu
      - loadavg
      - meminfo
      - filesystem
      - netdev
      - diskstats
      - cpufreq
      - os
      - time
      - xfs
      - cpu_guest_seconds_metric
      - boottime
      - systemd
      - processes
      - nvme
      - nfs
      - netstat
      - logind
      - stat
      - vmstat

    # Configurações de relabeling
    relabel_configs:
      - action: replace
        replacement: '${INSTANCE_NAME}'
        target_label: instance
      - action: replace
        replacement: '${TENANT_ID}'
        target_label: tenant

  # cAdvisor integrado (opcional)
  cadvisor:
    enabled: true
    docker_only: true
    instance: '${INSTANCE_NAME}'

    relabel_configs:
      - action: replace
        replacement: 'cadvisor'
        target_label: job
      - action: replace
        replacement: '${TENANT_ID}'
        target_label: tenant

    # Remover labels desnecessários
    metric_relabel_configs:
      - action: labeldrop
        regex: 'container_label_com_docker_compose_.*'
      - action: labeldrop
        regex: 'container_label_org_.*'

  # Process Exporter integrado (opcional)
  process_exporter:
    enabled: false
    config:
      process_names:
        - name: "{{.Comm}}"
          cmdline:
          - '.+'

# Configuração de Métricas
metrics:
  # Diretório para Write-Ahead Log
  wal_directory: /var/lib/grafana-agent/wal

  # Configurações globais
  global:
    scrape_interval: 30s
    scrape_timeout: 10s
    external_labels:
      cluster: '${CLUSTER_NAME}'
      region: '${AWS_REGION}'

    # Remote Write para Mimir/Prometheus
    remote_write:
      - url: https://mimir.${DOMAIN}/api/v1/push
        headers:
          X-Scope-OrgID: '${TENANT_ID}'

        # Configurações de queue
        queue_config:
          capacity: 10000
          max_samples_per_send: 2000
          batch_send_deadline: 5s
          min_shards: 1
          max_shards: 200

        # Configurações de retry
        write_relabel_configs:
          - source_labels: [__name__]
            regex: 'go_.*'
            action: drop

  # Configurações de scrape
  configs:
    - name: default
      scrape_configs:
        # Scrape do próprio agent
        - job_name: 'grafana-agent'
          static_configs:
            - targets: ['127.0.0.1:9090']
          scrape_interval: 30s
          relabel_configs:
            - action: replace
              replacement: '${INSTANCE_NAME}'
              target_label: instance

        # Scrape de aplicações locais
        - job_name: 'local-apps'
          static_configs:
            - targets: ['127.0.0.1:8080', '127.0.0.1:3000']
          scrape_interval: 15s
          relabel_configs:
            - action: replace
              replacement: '${INSTANCE_NAME}'
              target_label: instance
            - action: replace
              replacement: '${TENANT_ID}'
              target_label: tenant

        # Service Discovery via file
        - job_name: 'file-sd'
          file_sd_configs:
            - files:
              - '/etc/grafana-agent/targets/*.json'
              refresh_interval: 30s
          relabel_configs:
            - action: replace
              replacement: '${INSTANCE_NAME}'
              target_label: instance

# Configuração de Logs
logs:
  configs:
    - name: default
      # Cliente para Loki
      clients:
        - url: https://loki.${DOMAIN}/loki/api/v1/push
          headers:
            X-Scope-OrgID: '${TENANT_ID}'

          # Configurações de batching
          batchwait: 1s
          batchsize: 1048576

          # Configurações de retry
          backoff_config:
            min_period: 500ms
            max_period: 5m
            max_retries: 10

      # Arquivo de posições
      positions:
        filename: /var/lib/grafana-agent/positions.yaml

      # Configurações de scrape de logs
      scrape_configs:
        # Logs do Docker
        - job_name: docker
          docker_sd_configs:
            - host: "unix:///var/run/docker.sock"
              refresh_interval: 30s

          relabel_configs:
            - source_labels: [__meta_docker_container_name]
              target_label: container
            - source_labels: [__meta_docker_container_name]
              target_label: service_name
            - source_labels: [__meta_docker_container_log_stream]
              target_label: stream
            - action: replace
              replacement: '${INSTANCE_NAME}'
              target_label: instance
            - action: replace
              replacement: '${TENANT_ID}'
              target_label: tenant

        # Logs do sistema via journald
        - job_name: systemd
          journal:
            json: false
            max_age: 12h
            path: /var/log/journal

          relabel_configs:
            - source_labels: [__journal__systemd_unit]
              target_label: unit
            - source_labels: [__journal__hostname]
              target_label: hostname
            - action: replace
              replacement: '${INSTANCE_NAME}'
              target_label: instance
            - action: replace
              replacement: '${TENANT_ID}'
              target_label: tenant

        # Logs de arquivos específicos
        - job_name: syslog
          static_configs:
            - targets: [localhost]
              labels:
                job: syslog
                tenant: '${TENANT_ID}'
                __path__: /var/log/syslog

          relabel_configs:
            - action: replace
              target_label: instance
              replacement: '${INSTANCE_NAME}'

        # Logs de aplicações personalizadas
        - job_name: app-logs
          static_configs:
            - targets: [localhost]
              labels:
                job: app-logs
                tenant: '${TENANT_ID}'
                __path__: /var/log/myapp/*.log

          # Pipeline de processamento
          pipeline_stages:
            - json:
                expressions:
                  timestamp: timestamp
                  level: level
                  message: message
                  module: module

            - timestamp:
                source: timestamp
                format: RFC3339Nano

            - labels:
                level:
                module:

# Configuração de Traces (opcional)
traces:
  configs:
    - name: default
      receivers:
        jaeger:
          protocols:
            thrift_http:
              endpoint: 0.0.0.0:14268
            grpc:
              endpoint: 0.0.0.0:14250

        otlp:
          protocols:
            grpc:
              endpoint: 0.0.0.0:4317
            http:
              endpoint: 0.0.0.0:4318

      remote_write:
        - endpoint: https://tempo.${DOMAIN}:443
          headers:
            X-Scope-OrgID: '${TENANT_ID}'

Arquivo de Variáveis de Ambiente

Criar /etc/grafana-agent/environment:

# /etc/grafana-agent/environment

# Identificação da instância
INSTANCE_NAME="server-001"
TENANT_ID="company"

# Cluster e região
CLUSTER_NAME="production"
AWS_REGION="us-east-1"

# Domínio base
DOMAIN="monitoring.company.com"

# Credenciais (se necessário)
# AWS_ACCESS_KEY_ID="your-access-key"
# AWS_SECRET_ACCESS_KEY="your-secret-key"

Script de Configuração com Substituição de Variáveis

Criar /usr/local/bin/setup-grafana-agent.sh:

#!/bin/bash

# Script para configurar Grafana Agent com substituição de variáveis

set -e

# Carregar variáveis de ambiente
source /etc/grafana-agent/environment

# Função para substituir variáveis no arquivo de configuração
substitute_variables() {
    local config_file="/etc/grafana-agent/config.yaml"
    local temp_file="/tmp/config.yaml.tmp"

    # Substituir variáveis
    envsubst < "${config_file}.template" > "${temp_file}"

    # Validar configuração
    if grafana-agent --config.file="${temp_file}" --config.validate; then
        mv "${temp_file}" "${config_file}"
        chown grafana-agent:grafana-agent "${config_file}"
        echo "Configuração atualizada com sucesso"
    else
        echo "Erro na validação da configuração"
        rm -f "${temp_file}"
        exit 1
    fi
}

# Verificar se template existe
if [[ ! -f "/etc/grafana-agent/config.yaml.template" ]]; then
    echo "Template de configuração não encontrado"
    exit 1
fi

# Substituir variáveis
substitute_variables

# Recarregar serviço se estiver rodando
if systemctl is-active --quiet grafana-agent; then
    systemctl reload grafana-agent
    echo "Serviço recarregado"
fi

Service Discovery Dinâmico

Configuração via File SD

Criar diretório para targets:

sudo mkdir -p /etc/grafana-agent/targets
sudo chown grafana-agent:grafana-agent /etc/grafana-agent/targets

Exemplo de arquivo de targets (/etc/grafana-agent/targets/web-servers.json):

[
  {
    "targets": ["192.168.1.10:9100", "192.168.1.11:9100"],
    "labels": {
      "job": "node-exporter",
      "env": "production",
      "team": "infrastructure"
    }
  },
  {
    "targets": ["192.168.1.20:8080", "192.168.1.21:8080"],
    "labels": {
      "job": "web-app",
      "env": "production",
      "team": "backend"
    }
  }
]

Scripts de Gerenciamento

Script de Status

#!/bin/bash
# /usr/local/bin/grafana-agent-status.sh

echo "=== Grafana Agent Status ==="
systemctl status grafana-agent --no-pager

echo ""
echo "=== Últimos logs ==="
journalctl -u grafana-agent --no-pager -n 20

echo ""
echo "=== Uso de recursos ==="
ps aux | grep grafana-agent | grep -v grep

echo ""
echo "=== Verificação de conectividade ==="
curl -s http://localhost:9090/-/ready && echo "Agent ready" || echo "Agent not ready"

echo ""
echo "=== Métricas de status ==="
curl -s http://localhost:9090/metrics | grep -E 'prometheus_agent_|agent_build_info'

Script de Backup de Configuração

#!/bin/bash
# /usr/local/bin/backup-agent-config.sh

BACKUP_DIR="/var/backups/grafana-agent"
DATE=$(date +%Y%m%d_%H%M%S)

mkdir -p "$BACKUP_DIR"

# Backup da configuração
tar -czf "$BACKUP_DIR/grafana-agent-config-$DATE.tar.gz" \
    /etc/grafana-agent/ \
    /etc/systemd/system/grafana-agent.service

# Manter apenas últimos 10 backups
ls -t "$BACKUP_DIR"/grafana-agent-config-*.tar.gz | tail -n +11 | xargs -r rm

echo "Backup criado: $BACKUP_DIR/grafana-agent-config-$DATE.tar.gz"

Inicialização e Verificação

Primeira Inicialização

# 1. Criar configuração a partir do template
sudo cp /etc/grafana-agent/config.yaml /etc/grafana-agent/config.yaml.template

# 2. Executar script de configuração
sudo /usr/local/bin/setup-grafana-agent.sh

# 3. Validar configuração
sudo grafana-agent --config.file=/etc/grafana-agent/config.yaml --config.validate

# 4. Iniciar serviço
sudo systemctl start grafana-agent

# 5. Verificar status
sudo systemctl status grafana-agent

# 6. Verificar logs
sudo journalctl -u grafana-agent -f

Comandos de Verificação

# Status do serviço
systemctl status grafana-agent

# Logs em tempo real
journalctl -u grafana-agent -f

# Verificar se está escutando nas portas
ss -tulpn | grep grafana-agent

# Verificar métricas do agent
curl http://localhost:9090/metrics

# Verificar readiness
curl http://localhost:9090/-/ready

# Verificar configuração atual
curl http://localhost:9090/-/config

# Verificar targets descobertos
curl http://localhost:9090/api/v1/targets

Monitoramento e Troubleshooting

Principais Métricas do Agent

# CPU usage do agent
rate(process_cpu_seconds_total{job="grafana-agent"}[5m])

# Memória utilizada
process_resident_memory_bytes{job="grafana-agent"}

# Samples enviados via remote write
rate(prometheus_remote_storage_samples_total[5m])

# Falhas no remote write
rate(prometheus_remote_storage_samples_failed_total[5m])

# WAL size
prometheus_tsdb_wal_size_bytes

# Logs enviados
rate(promtail_sent_entries_total[5m])

Problemas Comuns e Soluções

1. Falhas no Remote Write

# Verificar conectividade
curl -I https://mimir.domain.com/api/v1/push

# Verificar certificados
openssl s_client -connect mimir.domain.com:443

# Verificar logs específicos
journalctl -u grafana-agent | grep "remote_write"

2. Alto Uso de Memória

# Ajustar configurações no config.yaml
metrics:
  global:
    remote_write:
      - queue_config:
          capacity: 5000        # Reduzir de 10000
          max_samples_per_send: 1000  # Reduzir de 2000

3. Logs Não Sendo Coletados

# Verificar permissões
ls -la /var/log/
ls -la /var/run/docker.sock

# Adicionar usuário ao grupo docker (para logs Docker)
sudo usermod -a -G docker grafana-agent
sudo systemctl restart grafana-agent

Configurações Avançadas

Rate Limiting

metrics:
  configs:
    - name: default
      scrape_configs:
        - job_name: 'rate-limited-app'
          static_configs:
            - targets: ['app:8080']
          scrape_interval: 1m  # Menos frequente para apps sensíveis
          metrics_path: /metrics
          honor_labels: true

Filtering de Métricas

metrics:
  global:
    remote_write:
      - url: https://mimir.domain.com/api/v1/push
        write_relabel_configs:
          # Drop métricas específicas
          - source_labels: [__name__]
            regex: 'go_gc_.*|go_memstats_.*'
            action: drop

          # Manter apenas métricas importantes
          - source_labels: [__name__]
            regex: 'up|cpu_usage_.*|memory_usage_.*'
            action: keep

Multi-tenancy

metrics:
  global:
    remote_write:
      # Tenant A
      - url: https://mimir.domain.com/api/v1/push
        headers:
          X-Scope-OrgID: 'tenant-a'
        write_relabel_configs:
          - source_labels: [tenant]
            regex: 'tenant-a'
            action: keep

      # Tenant B  
      - url: https://mimir.domain.com/api/v1/push
        headers:
          X-Scope-OrgID: 'tenant-b'
        write_relabel_configs:
          - source_labels: [tenant]
            regex: 'tenant-b'
            action: keep

Conclusão

O Grafana Agent oferece uma solução unificada e eficiente para observabilidade, consolidando múltiplas ferramentas em um único binário. Com remote write nativo e integrações embarcadas, simplifica significativamente a arquitetura de monitoramento, reduzindo a complexidade operacional e o overhead de recursos.

Benefícios da Implementação:

Simplicidade: Um único agente para métricas, logs e traces
Eficiência: Menor uso de recursos que múltiplos agentes
Flexibilidade: Configuração modular por necessidade
Escalabilidade: Remote write nativo para backends distribuídos
Manutenção: Gerenciamento centralizado via systemd

Implementação da Stack Grafana K8s em Cluster Kubernetes

Thiago Rodrigues — Wed, 28 May 2025 14:46:35 +0000

Implementação do Stack Grafana K8s em Cluster Kubernetes

Introdução

Este artigo descreve a implementação completa de um stack de observabilidade baseado no ecossistema Grafana em um cluster Kubernetes. O stack inclui:

Grafana Alloy: Agente de coleta de métricas e logs
Loki: Sistema de agregação de logs
Mimir: Sistema de métricas de longa duração
Grafana: Interface de visualização e dashboards

Arquitetura da Solução

A solução utiliza uma arquitetura distribuída onde:

Alloy coleta métricas e logs do cluster
Loki armazena e indexa logs com retenção configurável
Mimir armazena métricas de longa duração
Grafana fornece a interface de visualização
AWS Load Balancer Controller gerencia o ingress

Pré-requisitos

Cluster Kubernetes funcionando
Helm 3.x instalado
AWS Load Balancer Controller configurado
Certificados TLS válidos
Buckets S3 configurados para storage

1. Configuração do Grafana Alloy

O Alloy é responsável por coletar métricas e logs do cluster Kubernetes.

# values-alloy.yaml
alloy-logs:
  config:
    client:
      timeout: 30s
    resources:
      limits:
        memory: 256Mi
      requests:
        cpu: 100m
        memory: 128Mi
  enabled: true

alloy-metrics:
  config:
    global:
      external_labels:
        cluster: ${CLUSTER_NAME}
        source: alloy-agent
      scrape_interval: 3m
      scrape_timeout: 2m30s
    remote_write:
    - headers:
        X-Scope-OrgID: ${TENANT_ID}
      queue_config:
        batch_send_deadline: 30s
        max_backoff: 10s
        max_retries: 10
        max_samples_per_send: 1000
        min_backoff: 100ms
        retry_on_http_429: true
      send_exemplars: true
      url: http://mimir-gateway.mimir.svc.cluster.local/api/v1/push
  enabled: true
  resources:
    limits:
      memory: 768Mi
    requests:
      cpu: 250m
      memory: 384Mi

cluster:
  name: ${CLUSTER_NAME}

clusterEvents:
  enabled: true
  scrapeInterval: 3m

clusterMetrics:
  cadvisor:
    enabled: true
    metricRelabelings:
    - action: keep
      regex: container_cpu_.*|container_memory_.*|container_network_.*|container_fs_.*|machine_cpu_.*|container_spec_.*
      sourceLabels:
      - __name__
    relabelings:
    - action: labelmap
      regex: __meta_kubernetes_node_label_(.+)
  enabled: true
  kubelet:
    enabled: true
  nodeExporter:
    enabled: true

externalServices:
  loki:
    externalLabels:
      cluster: ${CLUSTER_NAME}
      tenant: ${TENANT_ID}
    host: http://loki-loki-distributed-gateway.loki.svc.cluster.local
    tenantId: ${TENANT_ID}
    writeEndpoint: /loki/api/v1/push
  prometheus:
    extraHeaders:
      X-Scope-OrgID: ${TENANT_ID}
    host: http://mimir-gateway.mimir.svc.cluster.local
    tenantId: ${TENANT_ID}
    writeEndpoint: /api/v1/push

extraScrapeConfigs: |-
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: pod
      - source_labels: [__meta_kubernetes_pod_container_name]
        action: replace
        target_label: container
    scrape_interval: 3m
    scrape_timeout: 2m30s

  - job_name: 'kubernetes-cadvisor'
    scheme: https
    tls_config:
      ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      insecure_skip_verify: true
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    kubernetes_sd_configs:
    - role: node
    relabel_configs:
    - action: labelmap
      regex: __meta_kubernetes_node_label_(.+)
    - target_label: __address__
      replacement: kubernetes.default.svc:443
    - source_labels: [__meta_kubernetes_node_name]
      regex: (.+)
      target_label: __metrics_path__
      replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
    metric_relabel_configs:
    - source_labels: [__name__]
      regex: 'container_cpu_.*|container_memory_.*|machine_cpu_.*|container_fs_.*|container_network_.*|container_spec_.*'
      action: keep
    scrape_interval: 3m
    scrape_timeout: 2m30s

kube-state-metrics:
  enabled: true
  metricLabelsAllowlist:
  - pods=[app,name,component]
  - deployments=[app,name,component]
  - statefulsets=[app,name,component]
  resources:
    limits:
      memory: 256Mi
    requests:
      cpu: 100m
      memory: 128Mi

logs:
  enabled: true
  extraClientConfig: |
    client:
      timeout: 10s
      batchwait: 1s 
      batchsize: 1048576
      max_retries: 5
      retry_on_status_codes: [429, 500, 502, 503, 504]
      backoff_config:
        min_period: 500ms
        max_period: 5m
        max_retries: 10
  pod_logs:
    discovery: all
    enabled: true
    namespaceAllowlist:
    - .*
    scrapeInterval: 3m
    stages:
    - json:
        expressions:
          level: level
          message: msg
          timestamp: time
    - timestamp:
        format: RFC3339Nano
        source: timestamp
    - labels:
        level: null

metrics:
  enabled: true
  scrapeInterval: 3m

opencost:
  enabled: false

prometheus-node-exporter:
  enabled: true

traces:
  enabled: false

Instalação do Alloy

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

# Criar namespace
kubectl create namespace monitoring

# Instalar Alloy
helm upgrade --install alloy grafana/k8s-monitoring \
  --namespace monitoring \
  --values values-alloy.yaml \
  --set cluster.name=${CLUSTER_NAME} \
  --set externalServices.loki.tenantId=${TENANT_ID} \
  --set externalServices.prometheus.tenantId=${TENANT_ID}

2. Configuração do Loki

Loki é responsável pela agregação e armazenamento de logs.

# values-loki.yaml
nameOverride: "loki-distributed"

compactor:
  enabled: true
  persistence:
    enabled: false
  replicas: 1

distributor:
  maxUnavailable: 1
  replicas: 2

extraEnvVars:
- name: AWS_ACCESS_KEY_ID
  valueFrom:
    secretKeyRef:
      name: aws-credentials
      key: access_key_id
- name: AWS_SECRET_ACCESS_KEY
  valueFrom:
    secretKeyRef:
      name: aws-credentials
      key: secret_access_key

gateway:
  enabled: true
  ingress:
    annotations:
      alb.ingress.kubernetes.io/certificate-arn: ${TLS_CERTIFICATE_ARN}
      alb.ingress.kubernetes.io/group.name: ${ALB_GROUP_NAME}
      alb.ingress.kubernetes.io/healthcheck-path: /ready
      alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
      alb.ingress.kubernetes.io/scheme: internet-facing
      alb.ingress.kubernetes.io/ssl-redirect: "443"
      alb.ingress.kubernetes.io/target-type: ip
      kubernetes.io/ingress.class: alb
    enabled: true
    hosts:
    - host: loki.${DOMAIN}
      paths:
      - path: /
        pathType: Prefix
    ingressClassName: alb
  replicas: 1
  service:
    port: 80
    type: ClusterIP

ingester:
  maxUnavailable: 1
  persistence:
    enabled: false
  replicas: 2

loki:
  commonConfig:
    path_prefix: /var/loki
    replication_factor: 1
  structuredConfig:
    auth_enabled: true
    common:
      storage:
        filesystem:
          chunks_directory: /var/loki/chunks
          rules_directory: /var/loki/rules
    limits_config:
      ingestion_burst_size_mb: 6
      ingestion_rate_mb: 4
      max_global_streams_per_user: 15000
      max_query_series: 500
      reject_old_samples: false
      reject_old_samples_max_age: 168h
      retention_period: 24h
      volume_enabled: true

    compactor:
      working_directory: /var/loki/compactor
      shared_store: s3
      compaction_interval: 10m
      retention_enabled: true
      retention_delete_delay: 1h
      retention_delete_worker_count: 150

    memberlist:
      join_members:
        - loki-loki-distributed-memberlist.loki.svc.cluster.local:7946

    schema_config:
      configs:
        - from: "2020-10-24"
          index:
            period: 24h
            prefix: index_
          object_store: aws
          schema: v11
          store: boltdb-shipper

    server:
      grpc_listen_port: 9095
      http_listen_port: 3100

    storage_config:
      aws:
        access_key_id: ${AWS_ACCESS_KEY_ID}
        bucketnames: ${LOKI_S3_BUCKET}
        region: ${AWS_REGION}
        s3: s3://s3.amazonaws.com
        secret_access_key: ${AWS_SECRET_ACCESS_KEY}
      boltdb_shipper:
        active_index_directory: /var/loki/index
        cache_location: /var/loki/boltdb-cache
        shared_store: s3

minio:
  enabled: false

nginx:
  enabled: false

querier:
  replicas: 1

query_frontend:
  maxUnavailable: 0
  replicas: 1

resources:
  limits:
    cpu: 500m
    memory: 512Mi
  requests:
    cpu: 100m
    memory: 256Mi

ruler:
  enabled: false

singleBinary:
  enabled: false

Instalação do Loki

# Criar secret para credenciais AWS
kubectl create secret generic aws-credentials \
  --from-literal=access_key_id=${AWS_ACCESS_KEY_ID} \
  --from-literal=secret_access_key=${AWS_SECRET_ACCESS_KEY} \
  --namespace loki

# Instalar Loki
helm repo add grafana https://grafana.github.io/helm-charts
helm upgrade --install loki grafana/loki-distributed \
  --namespace loki \
  --create-namespace \
  --values values-loki.yaml

3. Configuração do Mimir

Mimir é responsável pelo armazenamento de métricas de longa duração.

# values-mimir.yaml
alertmanager:
  enabled: false

compactor:
  persistentVolume:
    enabled: false
  replicas: 1

distributor:
  replicas: 2
  extraEnvVars:
  - name: AWS_ACCESS_KEY_ID
    valueFrom:
      secretKeyRef:
        name: aws-credentials
        key: access_key_id
  - name: AWS_SECRET_ACCESS_KEY
    valueFrom:
      secretKeyRef:
        name: aws-credentials
        key: secret_access_key

gateway:
  enabled: true
  enabledNonEnterprise: true
  ingress:
    annotations:
      alb.ingress.kubernetes.io/certificate-arn: ${TLS_CERTIFICATE_ARN}
      alb.ingress.kubernetes.io/group.name: ${ALB_GROUP_NAME}
      alb.ingress.kubernetes.io/healthcheck-path: /ready
      alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
      alb.ingress.kubernetes.io/scheme: internet-facing
      alb.ingress.kubernetes.io/ssl-redirect: "443"
      alb.ingress.kubernetes.io/target-type: ip
      kubernetes.io/ingress.class: alb
    enabled: true
    hosts:
      - host: mimir.${DOMAIN}
        paths:
          - path: /
            pathType: Prefix
    ingressClassName: alb
  replicas: 1
  service:
    port: 80
    type: ClusterIP

ingester:
  persistentVolume:
    enabled: false
  replicas: 2

mimir:
  structuredConfig:
    alertmanager_storage:
      backend: s3
      s3:
        bucket_name: ${MIMIR_ALERTS_BUCKET}
    blocks_storage:
      backend: s3
      bucket_store:
        sync_dir: /data/tsdb-sync
      s3:
        bucket_name: ${MIMIR_BLOCKS_BUCKET}
      tsdb:
        dir: /data/tsdb
    common:
      storage:
        backend: s3
        s3:
          access_key_id: ${AWS_ACCESS_KEY_ID}
          bucket_name: ${MIMIR_BLOCKS_BUCKET}
          endpoint: s3.amazonaws.com
          region: ${AWS_REGION}
          secret_access_key: ${AWS_SECRET_ACCESS_KEY}
    frontend:
      log_queries_longer_than: 10s
    limits:
      ingestion_burst_size: 150000
      ingestion_rate: 50000
      max_global_series_per_metric: 150000
      max_global_series_per_user: 2000000
    memberlist:
      abort_if_cluster_join_fails: false
      join_members:
        - mimir-gossip-ring.mimir.svc.cluster.local:7946
    multitenancy_enabled: true
    ruler_storage:
      backend: s3
      s3:
        bucket_name: ${MIMIR_RULES_BUCKET}
    server:
      grpc_listen_port: 9095
      http_listen_port: 8080

minio:
  enabled: false

mode: microservices

nginx:
  enabled: false

querier:
  replicas: 1

query_frontend:
  replicas: 1

query_scheduler:
  enabled: true
  replicas: 2
  resources:
    limits:
      cpu: 500m
      memory: 512Mi
    requests:
      cpu: 100m
      memory: 256Mi

ruler:
  enabled: false

store_gateway:
  persistentVolume:
    enabled: false
  replicas: 1

Instalação do Mimir

# Criar secret para credenciais AWS (se não existir)
kubectl create secret generic aws-credentials \
  --from-literal=access_key_id=${AWS_ACCESS_KEY_ID} \
  --from-literal=secret_access_key=${AWS_SECRET_ACCESS_KEY} \
  --namespace mimir

# Instalar Mimir
helm repo add grafana https://grafana.github.io/helm-charts
helm upgrade --install mimir grafana/mimir-distributed \
  --namespace mimir \
  --create-namespace \
  --values values-mimir.yaml

4. Configuração do Grafana

# values-grafana.yaml
fullnameOverride: "grafana"

# Image configuration
image:
  repository: grafana/grafana
  tag: "12.0.0"

# Admin credentials using secret
admin:
  existingSecret: "grafana-admin-credentials"
  userKey: "admin-user"
  passwordKey: "admin-password"

# Service configuration
service:
  type: ClusterIP
  port: 80

# Persistence
persistence:
  enabled: true
  size: 5Gi

# Database configuration using secrets
env:
  GF_DATABASE_TYPE: postgres
  GF_DATABASE_HOST: ${DATABASE_HOST}
  GF_DATABASE_PORT: "5432"
  GF_DATABASE_NAME: grafana
  GF_DATABASE_USER: ${DATABASE_USER}
  GF_DATABASE_PASSWORD: ${DATABASE_PASSWORD}
  GF_DATABASE_SSL_MODE: disable
  GF_LOG_LEVEL: warn

# Grafana configuration
grafana.ini:
  server:
    domain: grafana.${DOMAIN}
    root_url: https://grafana.${DOMAIN}

# Ingress configuration
ingress:
  enabled: true
  ingressClassName: alb
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/group.name: ${ALB_GROUP_NAME}
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/healthcheck-path: /api/health
    alb.ingress.kubernetes.io/certificate-arn: ${TLS_CERTIFICATE_ARN}
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
    alb.ingress.kubernetes.io/ssl-redirect: '443'
  hosts:
    - grafana.${DOMAIN}
  path: /
  pathType: Prefix

Instalação do Grafana

# Criar secrets necessários
kubectl create secret generic grafana-admin-credentials \
  --from-literal=admin-user=${GRAFANA_ADMIN_USER} \
  --from-literal=admin-password=${GRAFANA_ADMIN_PASSWORD} \
  --namespace grafana

# Instalar Grafana
helm repo add grafana https://grafana.github.io/helm-charts
helm upgrade --install grafana grafana/grafana \
  --namespace grafana \
  --create-namespace \
  --values values-grafana.yaml

5. Configuração do IngressClass

# ingress-class.yaml
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: alb
spec:
  controller: ingress.k8s.aws/alb

kubectl apply -f ingress-class.yaml

6. Configuração do Grafana Agent (Standalone)

Para monitoramento de recursos externos ao cluster:

# agent-config.yaml
server:
  log_level: debug

integrations:
  node_exporter:
    enabled: true
    rootfs_path: /
    sysfs_path: /sys
    procfs_path: /proc
    set_collectors:
    - uname
    - cpu
    - loadavg
    - meminfo
    - filesystem
    - netdev
    - diskstats
    - cpufreq
    - os
    - time
    - xfs
    - cpu_guest_seconds_metric
    - boottime
    - systemd
    - processes
    - nvme
    - nfs
    - netstat
    - logind
    - stat
    - vmstat
    relabel_configs:
    - action: replace
      replacement: '${INSTANCE_NAME}'
      target_label: instance
    - action: replace
      replacement: '${TENANT_ID}'
      target_label: tenant

metrics:
  wal_directory: /tmp/grafana-agent-wal
  global:
    scrape_interval: 30s
    remote_write:
    - url: https://mimir.${DOMAIN}/api/v1/push
      headers:
        X-Scope-OrgID: ${TENANT_ID}
  configs:
  - name: default
    scrape_configs:
    - job_name: agent
      static_configs:
      - targets: [ '127.0.0.1:9090' ]

logs:
  configs:
  - name: default
    clients:
    - url: https://loki.${DOMAIN}/loki/api/v1/push
      headers:
        X-Scope-OrgID: ${TENANT_ID}
    positions:
      filename: /tmp/positions.yaml
    scrape_configs:
    - job_name: docker
      docker_sd_configs:
      - host: "unix:///var/run/docker.sock"
        refresh_interval: 30s
      relabel_configs:
      - source_labels: [ '__meta_docker_container_name' ]
        target_label: 'container'
      - source_labels: [ '__meta_docker_container_name' ]
        target_label: 'service_name'
      - source_labels: [ '__meta_docker_container_log_stream' ]
        target_label: 'stream'
      - action: replace
        replacement: '${INSTANCE_NAME}'
        target_label: instance
      - action: replace
        replacement: '${TENANT_ID}'
        target_label: tenant
    - job_name: syslog
      static_configs:
      - targets: [ 'localhost' ]
        labels:
          job: syslog
          tenant: ${TENANT_ID}
      relabel_configs:
      - action: replace
        target_label: instance
        replacement: '${INSTANCE_NAME}'
      - action: replace
        target_label: tenant
        replacement: '${TENANT_ID}'

7. Script de Deploy Completo

#!/bin/bash

# Variáveis de ambiente
export CLUSTER_NAME="production-cluster"
export TENANT_ID="company"
export DOMAIN="monitoring.company.com"
export ALB_GROUP_NAME="monitoring-alb"
export TLS_CERTIFICATE_ARN="arn:aws:acm:region:account:certificate/cert-id"
export AWS_REGION="us-east-1"
export LOKI_S3_BUCKET="company-loki-data"
export MIMIR_BLOCKS_BUCKET="company-mimir-blocks"
export MIMIR_ALERTS_BUCKET="company-mimir-alerts"
export MIMIR_RULES_BUCKET="company-mimir-rules"
export DATABASE_HOST="postgres.company.internal"
export DATABASE_USER="grafana"
export DATABASE_PASSWORD="secure-password"
export GRAFANA_ADMIN_USER="admin"
export GRAFANA_ADMIN_PASSWORD="admin-password"

# Adicionar repositórios Helm
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

# Criar namespaces
kubectl create namespace monitoring
kubectl create namespace loki
kubectl create namespace mimir
kubectl create namespace grafana

# Criar secrets
kubectl create secret generic aws-credentials \
  --from-literal=access_key_id=${AWS_ACCESS_KEY_ID} \
  --from-literal=secret_access_key=${AWS_SECRET_ACCESS_KEY} \
  --namespace loki

kubectl create secret generic aws-credentials \
  --from-literal=access_key_id=${AWS_ACCESS_KEY_ID} \
  --from-literal=secret_access_key=${AWS_SECRET_ACCESS_KEY} \
  --namespace mimir

kubectl create secret generic grafana-admin-credentials \
  --from-literal=admin-user=${GRAFANA_ADMIN_USER} \
  --from-literal=admin-password=${GRAFANA_ADMIN_PASSWORD} \
  --namespace grafana

# Deploy Loki
helm upgrade --install loki grafana/loki-distributed \
  --namespace loki \
  --values values-loki.yaml

# Deploy Mimir
helm upgrade --install mimir grafana/mimir-distributed \
  --namespace mimir \
  --values values-mimir.yaml

# Deploy Grafana
helm upgrade --install grafana grafana/grafana \
  --namespace grafana \
  --values values-grafana.yaml

# Deploy Alloy
helm upgrade --install alloy grafana/k8s-monitoring \
  --namespace monitoring \
  --values values-alloy.yaml

echo "Deploy concluído! Aguarde alguns minutos para todos os pods ficarem prontos."

8. Verificação e Monitoramento

Verificar status dos pods:

kubectl get pods -n monitoring
kubectl get pods -n loki  
kubectl get pods -n mimir
kubectl get pods -n grafana

Verificar ingress:

kubectl get ingress -A

Acessar logs:

kubectl logs -f deployment/alloy -n monitoring
kubectl logs -f deployment/loki-distributor -n loki
kubectl logs -f deployment/mimir-distributor -n mimir

9. Configuração de Data Sources no Grafana

Após o deploy, configure as seguintes data sources no Grafana:

Loki Data Source:

URL: http://loki-loki-distributed-gateway.loki.svc.cluster.local
Headers: X-Scope-OrgID: ${TENANT_ID}

Mimir Data Source:

URL: http://mimir-gateway.mimir.svc.cluster.local
Headers: X-Scope-OrgID: ${TENANT_ID}

10. Considerações de Produção

Segurança:

Use secrets do Kubernetes para credenciais sensíveis
Configure RBAC apropriado
Habilite TLS em todas as comunicações

Performance:

Ajuste limites de recursos baseado no volume de dados
Configure retenção apropriada
Use storage classes adequados

Backup:

Configure backup dos dados no S3
Documente procedimentos de restore
Teste regularmente os backups

Monitoramento:

Configure alertas para falhas de componentes
Monitore uso de recursos
Configure dashboards de health check

Conclusão

Esta implementação fornece uma solução completa de observabilidade para ambientes Kubernetes, com alta disponibilidade, escalabilidade e integração nativa com AWS. O stack é adequado para ambientes de produção e pode ser customizado conforme necessidades específicas.