Milestone M3-3: Refactoring Console Output Into an Exporter Pathway

On April 24, 2026, I completed M3-3 for heka-insights-agent: moving console output from direct printing in the main loop to a proper exporter lifecycle.

This was part of Milestone #3 (Transport And Exporter Foundation) and specifically targeted:

• M3-3: Refactor console output to use exporter interface

The result is a cleaner delivery architecture where collectors remain untouched and output transport is now pluggable.

Why M3-3 mattered

Before this change, the runtime loop handled collection, formatting, and printing in one place. That worked for a console-only stage, but it tightly coupled runtime behavior to one output path.

For Milestone #3, the architecture needs to support:

• a canonical metric model
• reusable formatters
• exporter lifecycle boundaries
• future transport adapters

M3-3 was the bridge from “it prints metrics” to “it exports metrics through a contract.”

What the code looked like before

src/main.py previously:

1. collected CPU/memory/disk payloads
2. formatted them with PrometheusFormatter.format(...)
3. called print(prometheus_output, end="", flush=True) directly

That meant no exporter-owned lifecycle (initialize, shutdown), and no central place to swap output behavior.

Design goals for this refactor

The implementation targeted four practical goals:

1. Keep collectors unchanged.
2. Remove direct console printing from main.py.
3. Route output through Exporter.initialize(), Exporter.export(...), and Exporter.shutdown().
4. Preserve current Prometheus console output shape for backward compatibility.

What was implemented

1) Console exporter implementation

Created src/exporters/console.py with ConsoleExporter(Exporter).

Responsibilities:

• initialize(): prepare exporter state
• export(metrics): format and emit canonical metrics to stdout
• shutdown(): close lifecycle cleanly

ConsoleExporter now owns console writes. main.py no longer writes metrics directly.

2) Exporter factory wiring

Created src/exporters/factory.py with:

• create_exporter(exporter_type, logger=...) -> Exporter

Current behavior:

• returns ConsoleExporter for console
• falls back to ConsoleExporter for unimplemented exporter types with a warning

This keeps runtime deterministic while other exporters are still pending.

3) Canonical metric normalization pipeline

Created:

• src/pipeline/canonical_metrics.py
• src/pipeline/__init__.py

Added build_canonical_metrics(payloads, timestamp_unix_ms=...) to map collector payloads into canonical records.

Canonical record fields:

• name
• description
• type
• unit
• value
• labels
• optional timestamp_unix_ms

This separated normalization from transport and moved us closer to the milestone’s delivery pipeline model.

4) Formatter support for canonical metrics

Extended src/formatters/prometheus.py with:

• format_canonical(metrics)

This lets formatters operate on canonical data (not collector-specific payload dictionaries) while preserving the existing Prometheus text exposition output.

5) Main loop refactor to exporter lifecycle

Updated src/main.py so startup and runtime flow is now:

1. resolve EXPORTER_TYPE
2. create_exporter(...)
3. exporter.initialize()
4. collect payloads each cycle
5. normalize to canonical metrics
6. exporter.export(canonical_metrics)
7. exporter.shutdown() in finally

Direct print(...) of telemetry output was removed from main.py.

6) Python 3.10 typing compatibility fix

src/exporters/base.py originally used typing.NotRequired, which is not available in Python 3.10’s typing module.

The CanonicalMetric TypedDict was adjusted to a Python-3.10-safe form using:

• required base TypedDict
• total=False extension for optional timestamp

This kept type intent intact without requiring runtime upgrades.

Validation and runtime output

Validation performed:

• python3 -m compileall src
• smoke execution path for exporter creation + canonical conversion + console export

Observed runtime behavior after refactor:

• collector logs still emitted
• Prometheus lines still emitted each cycle
• timestamps populated in Unix milliseconds
• disk metrics emitted as aggregate and per-device series

In other words: behavior stayed stable, but ownership moved to exporter architecture.

What M3-3 achieved (and what it didn’t)

Completed in M3-3:

• console output now runs through exporter interface
• runtime flow is wired to exporter lifecycle
• normalization and formatting layers are separated from collection

Not part of M3-3 (handled in later items):

• strict fail-fast invalid exporter value handling (M3-4)
• additional backend exporters (Datadog, OTLP HTTP, New Relic)
• retry/buffering semantics beyond base hooks

Key lesson

This milestone was less about adding features and more about installing architectural seams.

The code now has explicit boundaries:

• collectors collect
• pipeline normalizes
• formatter renders
• exporter delivers

That separation is what will let future backend integrations land without rewriting collectors.

Next up

The immediate next step is M3-4: enforce strict startup validation for unsupported EXPORTER_TYPE values so runtime fails fast with explicit errors instead of warning + fallback.