Milestone 4 (Part 1): Implementing OTLP HTTP Core in Heka Insights Agent (M4-1, M4-2)

Heka Insights Agent already had a canonical metrics pipeline from Milestone 3.

In this part of Milestone 4, I implemented the OTLP HTTP core in two focused steps:

• M4-1: Canonical metrics -> OTLP payload mapping layer
• M4-2: OTLP HTTP request sender and exporter wiring

This post covers only these two items. Auth headers, resource attributes, retry/compression controls are intentionally deferred to later M4 tasks.

Why This Split Matters

By separating mapping from transport, we get:

• stable internal metric model
• explicit OTLP payload construction
• transport logic that can evolve independently
• clean foundation for New Relic/Datadog-style OTLP integrations later

What Was Implemented

M4-1: OTLP Payload Mapping Layer

I added a dedicated mapper that converts canonical metric records into OTLP HTTP JSON payloads.

Core behavior:

• validates required canonical fields before send
• supports explicit type mapping:
• gauge -> OTLP gauge.dataPoints
• counter -> OTLP sum.dataPoints with cumulative temporality
• maps canonical labels to OTLP metric attributes
• maps timestamp_unix_ms to OTLP timeUnixNano
• rejects malformed metrics early with explicit errors

Result: malformed payloads are blocked before network transport.

M4-2: OTLP HTTP Sender + Exporter

I added OTLP HTTP sender/exporter flow and wired it into exporter selection.

Core behavior:

• EXPORTER_TYPE=otlp_http now creates OTLP exporter
• validates OTLP endpoint format (http/https absolute URL) at startup
• fails fast when endpoint is missing/invalid
• sends JSON payload via HTTP POST
• treats only 2xx responses as success
• raises explicit errors for HTTP failures and transport errors

Result: working end-to-end OTLP HTTP delivery with fail-fast startup safety.

Local Test Setup with OpenTelemetry Collector (Docker)

I used OTel Collector debug exporter to validate incoming metrics.

Collector config (otel-collector-config.yaml)

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318

exporters:
  debug:

service:
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [debug]

Run collector

docker run --rm \
  -p 4318:4318 \
  -v "$(pwd)/otel-collector-config.yaml:/etc/otelcol/config.yaml" \
  otel/opentelemetry-collector:latest \
  --config=/etc/otelcol/config.yaml

Agent .env for this test

LOG_LOCATION=./log/heka_agent.log
CPU_POLL_INTERVAL_SECONDS=10
EXPORTER_TYPE=otlp_http
OTLP_HTTP_ENDPOINT=http://localhost:4318/v1/metrics

Run agent

python src/main.py

Verification Signals

From runtime behavior:

• agent starts with exporter_type=otlp_http
• collector logs periodic metric batches every ~10 seconds
• no exporter exceptions during dispatch
• first cycle has fewer points due to CPU warm-up, then normalizes

Example collector signal:

• resource metrics: 1
• metrics: 24
• data points: 24

Tests Added

I added focused tests for M4-1/M4-2:

• payload mapping correctness (gauge/counter, labels, timestamps)
• validation failures for malformed canonical metrics
• HTTP sender request behavior and error handling
• exporter wiring and missing-endpoint startup failure

All tests pass:

PYTHONPATH=src python3 -m unittest discover -s tests -v

What Is Intentionally Not Included Yet

Deferred to later M4 items:

• auth headers (M4-3)
• resource attribute mapping (M4-4)
• timeout/compression/retry controls (M4-5)
• broader OTLP docs and expanded test matrix (M4-6, M4-7)

Closing

M4-1 and M4-2 establish the OTLP core path: canonical metrics are now mapped deterministically and sent over HTTP with fail-fast validation.

This gives a production-friendly base to layer auth, resource metadata, and resiliency controls next.

Repo URL: https://github.com/ronin1770/heka-insights-agent