Heka-Insights-Agent: Milestone 6 Complete: Datadog OTLP + Native Integration Paths

Milestone 6 is complete for the Heka Insights Agent.

Repository: https://github.com/ronin1770/heka-insights-agent

In this milestone, we implemented two Datadog-compatible delivery paths so teams can choose what fits their needs:

1. Datadog OTLP preset mode (EXPORTER_TYPE=datadog_otlp)
2. Datadog native API mode (EXPORTER_TYPE=datadog_native)

Milestone 6 Goal

Provide Datadog integration through both:

• OTLP-based delivery for portability
• Datadog-native delivery for backend-specific control

In common terms: this means your agent can now send metrics to Datadog in either a standards-oriented way (OTLP) or a Datadog-specific way (native API), without changing your collectors.

What We Implemented

1. Datadog OTLP preset mode (datadog_otlp)

We added a Datadog preset resolver that derives endpoint + auth headers from Datadog config.

• endpoint is derived from site: https://otlp.<DATADOG_SITE>/v1/metrics
• auth header is injected: dd-api-key: <DATADOG_API_KEY>
• optional Datadog hostname/tags are mapped into resource attributes

2. Datadog native exporter (datadog_native)

We implemented a native Datadog metrics exporter for POST /api/v1/series.

• endpoint is derived from site: https://api.<DATADOG_SITE>/api/v1/series
• canonical gauge -> Datadog gauge
• canonical counter -> Datadog count
• timestamp_unix_ms -> Unix seconds
• count.interval is derived from CPU_POLL_INTERVAL_SECONDS

3. Validation hardening (M6-3)

We fail fast on invalid/missing Datadog config:

• DATADOG_SITE must be an allowed full domain
• DATADOG_API_KEY must be non-empty
• DATADOG_TAGS must be strict key:value format

4. Deterministic mapping rules (M6-4)

We made mapping predictable:

• DATADOG_HOSTNAME overrides label-derived host
• tag conflicts are deterministic (DATADOG_TAGS override label tags by key)
• metric prefixing is idempotent (won't double-prefix)

5. OTLP vs native comparison docs (M6-5)

We added side-by-side docs so teams can quickly decide mode based on:

• transport style
• portability
• mapping behavior
• counter interval semantics

6. Docker-based milestone-6 integration tests (M6-6)

We added dedicated tests under:

• tests/milestone-6/test_datadog_live_integration.py

These are intentionally gated and only run when explicitly enabled.

Key Files Added/Updated

• src/config/runtime.py
• src/config/__init__.py
• src/exporters/factory.py
• src/exporters/datadog_native.py
• tests/test_config_otlp_env.py
• tests/test_datadog_exporters.py
• tests/milestone-6/test_datadog_live_integration.py
• docs/configuration.md
• docs/architecture.md
• docs/development.md
• README.md

Implementation Strategy

We followed the existing exporter architecture:

1. Keep collectors unchanged
2. Extend runtime config with strict validation and preset resolution
3. Reuse OTLP HTTP exporter for Datadog OTLP mode
4. Add a dedicated native exporter for Datadog API v1 series
5. Add deterministic mapping rules + comprehensive tests
6. Add Docker-backed live tests in a milestone-specific folder

Test Commands and Outputs

Local Datadog-focused test suite

PYTHONPATH=src python3 -m unittest -v \
  tests.test_config_otlp_env \
  tests.test_datadog_exporters

Example output:

Ran 21 tests in 0.008s

OK

Full local suite

PYTHONPATH=src python3 -m unittest discover -s tests -v

Example output:

Ran 42 tests in 0.014s

OK (skipped=1)

Docker live Datadog integration tests (milestone-6)

docker compose -f docker-compose.test.yml run --rm \
  -e RUN_OTLP_INTEGRATION=1 \
  -e RUN_DATADOG_LIVE_INTEGRATION=1 \
  -e DATADOG_SITE=us5.datadoghq.com \
  -e DATADOG_API_KEY=<REDACTED_DATADOG_API_KEY> \
  test-runner \
  pytest -vv -s -rs tests/milestone-6/test_datadog_live_integration.py

Observed output:

collected 2 items

tests/milestone-6/test_datadog_live_integration.py::DatadogLiveIntegrationTests::test_datadog_native_exports_gauge_and_count_metrics PASSED
tests/milestone-6/test_datadog_live_integration.py::DatadogLiveIntegrationTests::test_datadog_otlp_preset_exports_gauge_metric PASSED

2 passed in 2.63s

Why This Matters

With Milestone 6 complete, Datadog users can now choose the integration path that matches their operations model:

• choose OTLP preset for standards-aligned portability
• choose native mode for Datadog-specific control and semantics

Both paths are now validated, tested, and documented.

Contribute

If you're interested in observability agents, OTLP pipelines, or exporter design, contributions are welcome:

• https://github.com/ronin1770/heka-insights-agent

Useful contribution areas:

• additional Datadog live integration coverage
• CI automation for gated integration scenarios
• docs/examples for production deployments
• future milestone reliability and operability improvements