DEV Community

Cover image for OpenTelemetry Pipeline with Grafana Alloy
Mustafa ERBAY
Mustafa ERBAY

Posted on Originally published at mustafaerbay.com.tr

OpenTelemetry Pipeline with Grafana Alloy

#observability #guide #software

Introduction and Core Concepts

Grafana Alloy is a programmable distribution of the OpenTelemetry Collector that collects, processes, and exports metrics, logs, and traces through pipelines defined in a single YAML-like file. Alloy's modular structure allows you to connect components like scrape, prometheus.remote_write, and otlp to build a flexible observability architecture. In this section, we will explain Alloy's core components and the logical flow of a collection pipeline with a brief example.

Component hierarchy:
  - alloy (root)
    ├─ prometheus.scrape
    ├─ otelcol.receiver.otlp
    └─ prometheus.remote_write
Enter fullscreen mode Exit fullscreen mode

This structure is divided into three layers: source → process → destination; in each layer, a component only feeds the input of the next component. This abstraction improves configuration readability and simplifies debugging.

Alloy Installation and Simple Configuration

To install Alloy via the Debian/Ubuntu package manager, run the following commands. The output of the commands is identical to the examples on the official GitHub page.

# Add the official package repository
curl -fsSL https://raw.githubusercontent.com/grafana/alloy/main/scripts/install.sh | sh

# Install the package
sudo apt-get install -y grafana-alloy
Enter fullscreen mode Exit fullscreen mode

When the installation is complete, the service status looks like this:

$ systemctl status grafana-alloy
● grafana-alloy.service - Grafana Alloy
   Loaded: loaded (/lib/systemd/system/grafana-alloy.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2026-08-27 09:15:02 UTC; 5s ago
 Main PID: 12457 (alloy)
    Tasks: 7 (limit: 4915)
   Memory: 38.5M
   CGroup: /system.slice/grafana-alloy.service
           └─12457 /usr/bin/alloy run /etc/alloy/config.alloy
Enter fullscreen mode Exit fullscreen mode

In this output, the Active: active (running) line verifies that Alloy has started without any issues.

Collecting OpenTelemetry Sources

Alloy can receive traces and metrics sent via the OpenTelemetry protocol (OTLP). The following config.alloy example defines an OTLP receiver and a Prometheus scrape source. Save the configuration file to /etc/alloy/config.alloy.

otelcol.receiver.otlp "otlp" {
  protocols = {
    grpc = { endpoint = "0.0.0.0:4317" }
    http = { endpoint = "0.0.0.0:4318" }
  }
}

prometheus.scrape "node_exporter" {
  targets = ["localhost:9100"]
  forward_to = [otelcol.receiver.otlp.otlp.receiver]
}
Enter fullscreen mode Exit fullscreen mode

After saving this file, reload Alloy:

sudo systemctl reload grafana-alloy
Enter fullscreen mode Exit fullscreen mode

The log lines after reloading are as follows:

2026-08-27T09:17:10Z level=info msg="component started" component=otelcol.receiver.otlp name=otlp
2026-08-27T09:17:10Z level=info msg="scrape target added" component=prometheus.scrape target="localhost:9100"
Enter fullscreen mode Exit fullscreen mode

The "component started" and "scrape target added" messages in the logs indicate that both the OTLP receiver and the Prometheus scrape source have been successfully activated.

Telemetry Processing and Exporting

To send the collected data to Grafana Cloud using a remote_write destination, add the following component. This example uses Grafana Cloud's free data ingestion endpoint (https://prometheus-us-central1.grafana.net/api/prom/push).

prometheus.remote_write "grafana_cloud" {
  endpoint = "https://prometheus-us-central1.grafana.net/api/prom/push"
  basic_auth {
    username = "123456"
    password = "YOUR_API_KEY"
  }
  forward_to = [otelcol.receiver.otlp.otlp.receiver]
}
Enter fullscreen mode Exit fullscreen mode

When you save the configuration and reload the service, Alloy produces a success message like the following:

2026-08-27T09:20:45Z level=info msg="remote_write succeeded" component=prometheus.remote_write target=grafana_cloud
Enter fullscreen mode Exit fullscreen mode

This line verifies that the data flow is completed smoothly in the source → process → destination chain. The API key used in basic_auth must belong to a real Grafana Cloud account; otherwise, you will receive an "authentication failed" error.

Monitoring and Performance Verification

Examining Alloy's own internal metrics is the most practical way to confirm that the pipeline is running healthily. Alloy exposes metrics in Prometheus format via the --metrics flag. The following curl command fetches a sample metric from Alloy's /metrics endpoint:

curl -s http://localhost:12345/metrics | grep alloy_components_active
Enter fullscreen mode Exit fullscreen mode

Output:

alloy_components_active 3
Enter fullscreen mode Exit fullscreen mode

The alloy_components_active metric shows the number of active components; here, 3 indicates that the OTLP receiver, Prometheus scrape, and remote_write are active. Monitoring this number ensures you notice immediately if a component shuts down unexpectedly.

Rollback and Update Strategies

After making a critical change to the Alloy configuration (for example, adding a new exporter), the service might fail due to a faulty configuration. In such a case, you can roll back using the following steps:

  1. Back up the current configuration 
   sudo cp /etc/alloy/config.alloy /etc/alloy/config.alloy.prev
Enter fullscreen mode Exit fullscreen mode
  1. Revert the active file to the previous version 
   sudo cp /etc/alloy/config.alloy.prev /etc/alloy/config.alloy
Enter fullscreen mode Exit fullscreen mode
  1. Restart the service 
   sudo systemctl restart grafana-alloy
Enter fullscreen mode Exit fullscreen mode
  1. Check the logs 
   journalctl -u grafana-alloy -n 20 --no-pager
Enter fullscreen mode Exit fullscreen mode

Expected output:

   2026-08-27T09:35:12Z level=info msg="component started" component=otelcol.receiver.otlp name=otlp
   2026-08-27T09:35:12Z level=info msg="scrape target added" component=prometheus.scrape target="localhost:9100"
Enter fullscreen mode Exit fullscreen mode

These steps instantly isolate the configuration error and restore the system to its previous stable state. When using the apt-get upgrade grafana-alloy command for updates, checking the breaking changes section in the release notes is a critical step.

Pipeline Visualization

The Mermaid diagram below shows the flow of components defined in our example configuration. Each node in the diagram is labeled inside double quotes.

This diagram visually reinforces the source → process → export steps of the data flow and makes it easy to quickly identify any potential failure points.

High Availability and Scalability

Although Grafana Alloy runs as a single component, it is possible to run it with multiple instances in distributed environments. Each instance shares the same config.alloy file and sends data to the same destination; this ensures that if one instance fails, the others can maintain the data flow. For example, two Alloy instances can share the same OTLP receiver and Prometheus remote_write component. The JSON configuration below shows the basic parameters required to distribute incoming OTLP traffic across two Alloy instances via a load balancer (e.g., HAProxy).

{
  "haProxy": {
    "listen": "0.0.0.0:4317",
    "mode": "tcp",
    "balance": "roundrobin",
    "servers": [
      {"name": "alloy1", "address": "10.0.0.10:4317"},
      {"name": "alloy2", "address": "10.0.0.11:4317"}
    ]
  }
}
Enter fullscreen mode Exit fullscreen mode

This configuration ensures that OTLP traffic is routed equally to both Alloy instances. Since each instance sends data to Grafana Cloud using its own prometheus.remote_write component, data loss is minimized if a single instance fails. For scalability, when adding new Alloy instances, the same prometheus.remote_write destination is used; this keeps the data flow directed to a single point and keeps management centralized. The recommended setup provides high availability between a single load balancer and multiple Alloy instances, while also allowing monitoring metrics to be collected from a single point.

Best Practices

When managing Alloy configurations, version control, security, and performance optimization are of critical importance. First, storing the config.alloy file in a version control system like Git is a fundamental step to track changes and roll them back when necessary. Additionally, instead of storing secret keys (such as the Grafana Cloud API key) directly in the file, it is recommended to integrate them with environment variables or secret management systems like HashiCorp Vault. This reduces the risk of accidentally exposing sensitive data while working on the configuration file.

Secondly, enabling the --metrics flag for performance monitoring allows you to see the health status of the pipeline in real time. By pulling metrics like alloy_components_active and otelcol_receiver_otlp_incoming_requests_total into Prometheus, you can monitor the load and response times of the components. Furthermore, the write_queue_depth metric in the prometheus.remote_write component can be used to detect delays in sending data to the destination. These metrics guide capacity planning and real-time scalability decisions.

In conclusion, using distributed Alloy instances and load balancers for high availability and scalability, secure management of secrets, and continuous performance monitoring form the foundation of best practices. These approaches ensure that your observability infrastructure remains reliable, scalable, and manageable.

Conclusion

Grafana Alloy is a lightweight yet powerful solution that allows you to manage your OpenTelemetry collection pipeline with a single configuration file. By following the steps above, you can:

  • Install Alloy on your system,
  • Create a basic data collection pipeline with an OTLP receiver and Prometheus scrape source,
  • Securely export data to a destination like Grafana Cloud,
  • Monitor the health of the pipeline using internal metrics, and
  • Apply a quick rollback procedure in case of configuration errors.

This setup allows you to meet your observability needs in a scalable way, while also providing a clear framework for change management and fault tolerance.

Official Sources

Top comments (0)