AI-Driven Automated Network Monitoring & Anomaly Detection — Part 3: Storing and Querying Metrics in Prometheus

AI-Driven Automated Network Monitoring & Anomaly Detection — Part 3: Storing and Querying Metrics in Prometheus

In Part 1 we introduced the fundamentals of AIOps and set up a baseline environment for collecting network metrics. Part 2 walked through building lightweight exporters for our routers and switches, exposing metrics in the Prometheus exposition format. This chapter focuses on the next critical steps: persisting those metrics, scaling the storage layer, and querying the data for real‑time and historical insight.

Why Prometheus is Still the Go‑To for Network Monitoring in 2026

Marcel’s talk on “AIOps: Anomaly detection with Prometheus” (2026) and the Medium article by Shraman Padhalni both emphasize that Prometheus remains the de‑facto standard for real‑time metric collection in modern data centers. Its pull‑based model, expressive PromQL, and rich ecosystem of exporters make it an ideal backbone for any AI‑driven anomaly detection pipeline. Moreover, the 2026 PromCon EU conference highlighted the maturity of community tools such as Thanos, Cortex, and Grafana Labs’ Tempo for long‑term storage and observability‑as‑a‑service.

Below I’ll walk through the exact configuration steps you’ll need to turn a vanilla Prometheus deployment into a production‑grade, high‑availability, long‑term storage solution. I’ll also demonstrate how to query the data both programmatically and via the UI, and finally how to feed those metrics into an anomaly‑detection workflow using Spark or Netdata.

Prometheus Storage Model Recap

Prometheus stores each metric as a time‑series identified by a fully‑qualified name and a set of labels. Internally the data is written to a TSDB (time‑series database) that compresses data into chunks per minute, using a hybrid approach of LZ4 compression and delta‑encoding. The default retention period is 15 days, which is often insufficient for forensic analysis or trend detection.

To extend retention, you can either:

• Configure --storage.tsdb.retention.time for a longer local period (e.g., 90 days).
• Set up remote write to a long‑term store like Thanos, Cortex, or an object store (S3, GCS).
• Federate multiple Prometheus instances and pull data from child instances into a central “hub.”

For most AIOps pipelines, the remote write + Thanos combination offers the best balance between cost, scalability, and query performance.

Setting Up Prometheus with Thanos for Long‑Term Storage

Below is a minimal prometheus.yml that configures remote write to a Thanos sidecar. The sidecar will stream data to an S3 bucket (or any object store supported by Thanos). The example assumes you already have AWS credentials available in the container environment.

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: "router_exporter"
    static_configs:
      - targets: ["router1:9101", "router2:9101"]

remote_write:
  - url: "http://thanos-sidecar:19001/api/v1/receive"
    remote_timeout: 30s
    write_relabel_configs:
      - source_labels: [__name__]
        regex: ".*"
        action: keep

Key points:

• The remote_write block points to the Thanos sidecar that will push data to object storage.
• We use write_relabel_configs to filter which series are sent; in this example we keep everything.
• The sidecar must be running in the same pod or service as Prometheus to avoid network latency.

Deploying the Full Stack with Docker Compose

Below is a docker-compose.yml that brings up Prometheus, Thanos, a node exporter for host metrics, and Grafana for visualization. The example also includes a simple Python exporter for a hypothetical router that exposes a router_cpu_usage metric.

version: '3.8'

services:
  prometheus:
    image: prom/prometheus:v2.48.0
    container_name: prometheus
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    ports:
      - "9090:9090"
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.retention.time=90d' # local retention
    depends_on:
      - thanos-sidecar

  thanos-sidecar:
    image: thanosio/thanos:v0.32.0
    container_name: thanos-sidecar
    command:
      - sidecar
      - '--prometheus.url=http://prometheus:9090'
      - '--objstore.config-file=/etc/thanos/bucket.yaml'
      - '--tsdb.path=/prometheus'
    volumes:
      - prometheus_data:/prometheus
      - ./thanos:/etc/thanos
    ports:
      - "19001:19001" # remote write endpoint

  node-exporter:
    image: prom/node-exporter:v1.6.1
    container_name: node-exporter
    ports:
      - "9100:9100"

  router-exporter:
    build: ./router-exporter
    container_name: router-exporter
    ports:
      - "9101:9101"

  grafana:
    image: grafana/grafana-oss:10.0.0
    container_name: grafana
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
    ports:
      - "3000:3000"
    depends_on:
      - prometheus

volumes:
  prometheus_data:

The router-exporter service is built from a local Dockerfile that installs a simple Python script. Here’s that script (app.py):

from prometheus_client import start_http_server, Gauge
import random
import time

# Define a gauge metric for router CPU usage
router_cpu = Gauge('router_cpu_usage', 'CPU usage percentage of the router')

def collect_metrics():
    # In a real deployment you would query SNMP or a vendor API
    # For demo purposes we generate a random value
    router_cpu.set(random.uniform(10, 90))

if __name__ == '__main__':
    start_http_server(9101)
    while True:
        collect_metrics()
        time.sleep(15)

The Dockerfile for this exporter is trivial:

FROM python:3.11-slim
WORKDIR /app
COPY app.py /app/app.py
RUN pip install prometheus-client
CMD ["python", "app.py"]

Querying Metrics with PromQL

Once Prometheus is up and running, you can use the web UI (http://localhost:9090) to run PromQL queries. Below are some common queries for network monitoring.

Basic Time‑Series Query

# Current CPU usage per router
router_cpu_usage

Rate of Change Over 5 Minutes

# Rate of packet drops per second over the last 5 minutes
rate(router_packets_dropped_total[5m])

Histogram Quantile for Latency

# 95th percentile latency
histogram_quantile(0.95, sum(rate(router_latency_bucket[5m])) by (le))

Average Over a Day

# Daily average CPU usage
avg_over_time(router_cpu_usage[1d])

For programmatic access, you can hit the /api/v1/query endpoint. Below is a Python helper that fetches the last value of router_cpu_usage:

import requests

PROMETHEUS_URL = 'http://localhost:9090/api/v1/query'

def query(query):
    resp = requests.get(PROMETHEUS_URL, params={'query': query})
    return resp.json()

if __name__ == '__main__':
    result = query('router_cpu_usage')
    print(result)

Scaling Prometheus: Federation & High Availability

In a large network, you might have hundreds of routers and switches. Running a single Prometheus instance becomes a bottleneck. Two common strategies are:

• Federation: Child Prometheus instances scrape local exporters and expose a subset of metrics (usually rate() or sum()) to a central hub. The hub then aggregates data for cross‑domain queries.
• Thanos Query Layer: Thanos can query all child Prometheus instances as a single logical store, providing global downsampling and caching. This is the most common setup in 2026 when dealing with multi‑region deployments.

Here is a minimal federation config that exposes the router_cpu_usage series to the hub:

scrape_configs:
  - job_name: "federate"
    metrics_path: /federate
    params:
      'match[]':
        - '{__name__=~"router_cpu_usage"}'
    static_configs:
      - targets: ["localhost:9090"]

Integrating with AIOps & Anomaly Detection

Prometheus data is ideal for feeding into downstream analytics engines. Two popular patterns in 2026 are:

• Spark Streaming + PromQL Exporter: Use the prometheus-spark-connector to stream data into Spark, then apply MLlib or PySpark ML pipelines to detect outliers. Marcel’s talk demonstrated using Spark for trend extraction; the connector pulls data in micro‑batches and stores them in Delta Lake for long‑term analysis.
• Netdata Anomaly Detection: Netdata can consume Prometheus metrics and run statistical anomaly detection in real‑time, providing alerts via Webhooks or Prometheus Alertmanager. Netdata’s anomaly_detection plugin uses Gaussian Process regression and is fully open‑source.

Example: Spark Streaming Pipeline

# spark_prometheus_stream.py
from pyspark.sql import SparkSession
from pyspark.sql.functions import *
from pyspark.sql.types import *

spark = SparkSession.builder \
    .appName('PrometheusSparkStream') \
    .getOrCreate()

# Connect to Prometheus via the connector
df = spark.readStream \
    .format('prometheus') \
    .option('url', 'http://prometheus:9090') \
    .option('query', 'rate(router_packets_dropped_total[1m])') \
    .load()

# Simple anomaly detection: flag if rate > 100 packets/sec
anomalies = df.filter(col('value') > 100).withColumn('timestamp', current_timestamp())

# Write anomalies to a Delta table for persistence
query = anomalies.writeStream \
    .outputMode('append') \
    .format('delta') \
    .option('checkpointLocation', '/tmp/checkpoints') \
    .option('path', '/tmp/delta/anomalies') \
    .start()

query.awaitTermination()

Example: Netdata Anomaly Detection

Netdata can be configured to scrape the same Prometheus metrics and then apply its built‑in anomaly detection. In netdata.conf add:

[prometheus]
    enable = yes
    url = http://prometheus:9090/api/v1/query
    query = router_cpu_usage
    anomaly_detection = yes

Netdata will now display a red flag in the UI whenever CPU usage deviates more than 3 standard deviations from the mean over a 5‑minute window.

Performance Tuning Tips

• Label Cardinality: Avoid high cardinality labels (e.g., per‑host IPs in a large data center). Use instance sparingly and prefer job or region tags.
• Scrape Interval: 15 s is a good default for network devices. Increase to 30 s if the data is less volatile.
• Retention: Keep local retention to 30–90 days. Let Thanos handle the rest. This reduces disk usage on the Prometheus host.
• Chunk Size: Prometheus automatically splits data into 1‑minute chunks. If you observe high write latency, consider increasing storage.tsdb.max-block-duration to 1 hour.
• Alertmanager Integration: Configure Alertmanager to route alerts to Slack, PagerDuty, or email. Use for and annotations to reduce noise.

Putting It All Together: A Real‑World Workflow

• Network devices expose SNMP counters via a custom exporter.
• Prometheus scrapes these counters every 15 s.
• Local TSDB retains data for 90 days; remote write streams to Thanos, which stores 10 years of data in an S3 bucket.
• Netdata visualizes live metrics and runs anomaly detection, raising alerts.
• Prometheus queries feed a Spark streaming job that performs trend analysis and feeds results into a ML model hosted in SageMaker (or an on‑prem Jupyter cluster).
• Detected anomalies are written back to Prometheus as custom metrics, which in turn trigger Alertmanager alerts.

That completes the core of Part 3: you now have a resilient, scalable storage layer and a robust querying strategy that can be extended into any AIOps pipeline.

📚 References & Further Reading

• Prometheus – Official Documentation
• Thanos – Long‑Term Storage for Prometheus
• Netdata – Anomaly Detection on Prometheus Metrics
• Marcel’s AIOps Talk – 2026
• Real‑Time Anomaly Detection in Network Operations Using AIOps

Your Turn

What challenges have you encountered when scaling Prometheus for network‑wide monitoring? Share your experiences or questions below, and let’s discuss how to push the boundaries of AIOps together.

---

Originally published at https://artificial-inteligence.phptutorial.co.in