DEV Community: Farhan Munir

Milestone 4 Complete — OTLP HTTP Exporter for Heka Insights Agent

Farhan Munir — Wed, 29 Apr 2026 06:49:27 +0000

In this milestone, I completed the OTLP HTTP exporter implementation for Heka Insights Agent and validated it with both unit tests and Docker-backed integration tests.

Repo: https://github.com/ronin1770/heka-insights-agent

What was implemented in Milestone 4

Milestone 4 includes:

OTLP HTTP exporter wiring
OTLP metric payload mapping
OTLP auth header support
OTLP resource attribute mapping
Retry/backoff behavior for transient failures
Unit test coverage for config + sender + exporter + mapping
Docker integration tests against a real OpenTelemetry Collector

Configuration added

OTLP configuration is environment-driven through .env:

LOG_LOCATION=./log/heka_agent.log
CPU_POLL_INTERVAL_SECONDS=10
EXPORTER_TYPE=otlp_http
OTLP_HTTP_ENDPOINT=http://localhost:4318/v1/metrics
OTLP_HTTP_HEADERS=key=Bearer abcd1234
OTLP_RESOURCE_ATTRIBUTES=service.name=heka-insights-agent,host.name=localhost
OTLP_HTTP_TIMEOUT_SECONDS=10
OTLP_HTTP_RETRY_MAX_ATTEMPTS=5
OTLP_HTTP_RETRY_INITIAL_BACKOFF_SECONDS=1
OTLP_HTTP_RETRY_MAX_BACKOFF_SECONDS=5

Notes

OTLP_HTTP_HEADERS format: key=value,key2=value2
OTLP_RESOURCE_ATTRIBUTES format: key=value,key2=value2
Retryable failures:
- transport errors
- HTTP 408, 429, and 5xx
Non-retryable failures:
- HTTP 400, 401, 403, 404, and similar client-side errors

Unit tests

Unit tests cover:

OTLP env parsing and validation
OTLP payload mapping
OTLP HTTP sender behavior
Retry/backoff behavior
Exporter initialization and wiring

Run unit tests:

PYTHONPATH=src python3 -m unittest discover -s tests -v

Unit test output

Ran 27 tests in 0.008s

OK

Integration tests (Docker + real collector)

Integration tests validate:

auth success (200)
auth reject path (401) and no retry on non-retryable status
no-auth collector path

Run OTLP integration tests:

RUN_OTLP_INTEGRATION=1 PYTHONPATH=src python3 -m unittest -v tests.test_otlp_http_integration

Integration test output

test_auth_rejected_without_retry_for_401 ... ok
test_auth_success_with_bearer_header ... ok
test_no_auth_collector_accepts_without_headers ... ok

----------------------------------------------------------------------
Ran 3 tests in 2.417s

OK

Collector config used in tests

The project includes collector fixtures under tests/fixtures/otlp/, including auth-required and no-auth variants for scenario testing.

Final result

Milestone 4 now has:

working OTLP HTTP exporter
production-style config controls
retry/backoff logic for transient failures
unit + integration test coverage
updated docs and run instructions

Repo again: https://github.com/ronin1770/heka-insights-agent

Milestone 4 (Part 1): Implementing OTLP HTTP Core in Heka Insights Agent (M4-1, M4-2)

Farhan Munir — Mon, 27 Apr 2026 09:57:30 +0000

Milestone 4 (Part 1): Implementing OTLP HTTP Core in Heka Insights Agent (M4-1, M4-2)

Heka Insights Agent already had a canonical metrics pipeline from Milestone 3.

In this part of Milestone 4, I implemented the OTLP HTTP core in two focused steps:

M4-1: Canonical metrics -> OTLP payload mapping layer
M4-2: OTLP HTTP request sender and exporter wiring

This post covers only these two items. Auth headers, resource attributes, retry/compression controls are intentionally deferred to later M4 tasks.

Why This Split Matters

By separating mapping from transport, we get:

stable internal metric model
explicit OTLP payload construction
transport logic that can evolve independently
clean foundation for New Relic/Datadog-style OTLP integrations later

What Was Implemented

M4-1: OTLP Payload Mapping Layer

I added a dedicated mapper that converts canonical metric records into OTLP HTTP JSON payloads.

Core behavior:

validates required canonical fields before send
supports explicit type mapping:
gauge -> OTLP gauge.dataPoints
counter -> OTLP sum.dataPoints with cumulative temporality
maps canonical labels to OTLP metric attributes
maps timestamp_unix_ms to OTLP timeUnixNano
rejects malformed metrics early with explicit errors

Result: malformed payloads are blocked before network transport.

M4-2: OTLP HTTP Sender + Exporter

I added OTLP HTTP sender/exporter flow and wired it into exporter selection.

Core behavior:

EXPORTER_TYPE=otlp_http now creates OTLP exporter
validates OTLP endpoint format (http/https absolute URL) at startup
fails fast when endpoint is missing/invalid
sends JSON payload via HTTP POST
treats only 2xx responses as success
raises explicit errors for HTTP failures and transport errors

Result: working end-to-end OTLP HTTP delivery with fail-fast startup safety.

Local Test Setup with OpenTelemetry Collector (Docker)

I used OTel Collector debug exporter to validate incoming metrics.

Collector config (`otel-collector-config.yaml`)

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318

exporters:
  debug:

service:
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [debug]

Run collector

docker run --rm \
  -p 4318:4318 \
  -v "$(pwd)/otel-collector-config.yaml:/etc/otelcol/config.yaml" \
  otel/opentelemetry-collector:latest \
  --config=/etc/otelcol/config.yaml

Agent `.env` for this test

LOG_LOCATION=./log/heka_agent.log
CPU_POLL_INTERVAL_SECONDS=10
EXPORTER_TYPE=otlp_http
OTLP_HTTP_ENDPOINT=http://localhost:4318/v1/metrics

Run agent

python src/main.py

Verification Signals

From runtime behavior:

agent starts with exporter_type=otlp_http
collector logs periodic metric batches every ~10 seconds
no exporter exceptions during dispatch
first cycle has fewer points due to CPU warm-up, then normalizes

Example collector signal:

resource metrics: 1
metrics: 24
data points: 24

Tests Added

I added focused tests for M4-1/M4-2:

payload mapping correctness (gauge/counter, labels, timestamps)
validation failures for malformed canonical metrics
HTTP sender request behavior and error handling
exporter wiring and missing-endpoint startup failure

All tests pass:

PYTHONPATH=src python3 -m unittest discover -s tests -v

What Is Intentionally Not Included Yet

Deferred to later M4 items:

auth headers (M4-3)
resource attribute mapping (M4-4)
timeout/compression/retry controls (M4-5)
broader OTLP docs and expanded test matrix (M4-6, M4-7)

Closing

M4-1 and M4-2 establish the OTLP core path: canonical metrics are now mapped deterministically and sent over HTTP with fail-fast validation.

This gives a production-friendly base to layer auth, resource metadata, and resiliency controls next.

Repo URL: https://github.com/ronin1770/heka-insights-agent

Heka-Insights-Agent: Milestone M3-4: Fail-Fast Exporter Validation at Startup

Farhan Munir — Fri, 24 Apr 2026 06:30:06 +0000

Milestone M3-4: Fail-Fast Exporter Validation at Startup

On April 24, 2026, I completed M3-4 in heka-insights-agent:

M3-4: Add configuration validation for exporter settings

This milestone closed a critical gap in the exporter foundation by removing silent fallback behavior and enforcing explicit startup failures for invalid exporter configuration.

Context

In Milestone M3-3, we routed output through the exporter interface and lifecycle:

initialize()
export(metrics)
shutdown()

That established the architecture.

M3-4 focused on correctness and operational safety: startup must fail early when exporter configuration is invalid or points to an unimplemented adapter.

The Problem Before M3-4

Prior behavior was permissive:

invalid EXPORTER_TYPE values defaulted back to console
unimplemented exporter types also downgraded to console

Why this was risky:

misconfiguration could go unnoticed in production
users could think they were exporting to one backend while actually exporting to console
behavior violated milestone acceptance criteria requiring explicit errors

M3-4 Goals

The implementation targeted three concrete outcomes:

invalid EXPORTER_TYPE must fail fast with a clear startup error
configured but unimplemented exporter adapters must fail fast with a clear startup error
docs must reflect strict validation (remove pre-M3-4 fallback messaging)

Implementation Summary

1) Strict validation in runtime config

Updated src/config/runtime.py in get_exporter_type(...).

Behavior now:

missing EXPORTER_TYPE still defaults to console
unsupported value now raises RuntimeError with explicit supported values list
optional logger records an error message before raising

This changed exporter selection from “best-effort fallback” to deterministic validation.

2) Removed fallback from exporter factory

Updated src/exporters/factory.py in create_exporter(...).

Behavior now:

console returns ConsoleExporter
other configured values currently raise RuntimeError because adapters are not implemented yet

This is intentional. It prevents false confidence and makes readiness of each exporter explicit.

3) Documentation updated to M3-4 semantics

Updated:

docs/configuration.md
README.md

Both now state that:

unsupported exporter values fail fast
unimplemented exporters fail fast
only missing value defaults to console

This removed the “fallback to console with warning” wording from the pre-M3-4 behavior.

Validation Performed

Validation included compile checks and runtime behavior checks.

Compile validation

python3 -m compileall src passed

Behavior validation

Tested startup resolution paths for:

missing EXPORTER_TYPE
EXPORTER_TYPE=console
EXPORTER_TYPE=invalid_value
EXPORTER_TYPE=otlp_http (declared but not implemented adapter)

Observed results:

missing value -> resolves to console
console -> exporter creation succeeds
invalid_value -> immediate RuntimeError with supported-values message
otlp_http -> immediate RuntimeError stating exporter not implemented

These outcomes align with M3-4 requirements.

Why this change matters operationally

Fail-fast startup validation improves reliability in real deployments:

misconfigurations are caught immediately
no hidden routing to fallback output
deployment behavior is explicit and auditable
future exporter rollout can be gated by implementation readiness

This is especially important when teams automate deployments and rely on env-based configuration.

Architectural Impact

After M3-4, exporter behavior is now strictly layered:

config validates selector
factory enforces implementation availability
runtime starts only on valid+implemented exporter path

That gives us a clean foundation for adding real transports (otlp_http, datadog_native, newrelic_otlp) without changing collector logic.

What’s Next

The natural next step after M3-4 is M3-5:

document exporter lifecycle and responsibilities in architecture docs
include startup validation expectations as part of operational guidance
capture adapter implementation contract for future backend integrations

M3-4 makes sure the system fails loudly when exporter config is wrong, which is exactly what a transport foundation should do before adding real outbound integrations.

Milestone M3-3: Refactoring Console Output Into an Exporter Pathway

Farhan Munir — Fri, 24 Apr 2026 05:49:23 +0000

Milestone M3-3: Refactoring Console Output Into an Exporter Pathway

On April 24, 2026, I completed M3-3 for heka-insights-agent: moving console output from direct printing in the main loop to a proper exporter lifecycle.

This was part of Milestone #3 (Transport And Exporter Foundation) and specifically targeted:

M3-3: Refactor console output to use exporter interface

The result is a cleaner delivery architecture where collectors remain untouched and output transport is now pluggable.

Why M3-3 mattered

Before this change, the runtime loop handled collection, formatting, and printing in one place. That worked for a console-only stage, but it tightly coupled runtime behavior to one output path.

For Milestone #3, the architecture needs to support:

a canonical metric model
reusable formatters
exporter lifecycle boundaries
future transport adapters

M3-3 was the bridge from “it prints metrics” to “it exports metrics through a contract.”

What the code looked like before

src/main.py previously:

collected CPU/memory/disk payloads
formatted them with PrometheusFormatter.format(...)
called print(prometheus_output, end="", flush=True) directly

That meant no exporter-owned lifecycle (initialize, shutdown), and no central place to swap output behavior.

Design goals for this refactor

The implementation targeted four practical goals:

Keep collectors unchanged.
Remove direct console printing from main.py.
Route output through Exporter.initialize(), Exporter.export(...), and Exporter.shutdown().
Preserve current Prometheus console output shape for backward compatibility.

What was implemented

1) Console exporter implementation

Created src/exporters/console.py with ConsoleExporter(Exporter).

Responsibilities:

initialize(): prepare exporter state
export(metrics): format and emit canonical metrics to stdout
shutdown(): close lifecycle cleanly

ConsoleExporter now owns console writes. main.py no longer writes metrics directly.

2) Exporter factory wiring

Created src/exporters/factory.py with:

create_exporter(exporter_type, logger=...) -> Exporter

Current behavior:

returns ConsoleExporter for console
falls back to ConsoleExporter for unimplemented exporter types with a warning

This keeps runtime deterministic while other exporters are still pending.

3) Canonical metric normalization pipeline

Created:

src/pipeline/canonical_metrics.py
src/pipeline/__init__.py

Added build_canonical_metrics(payloads, timestamp_unix_ms=...) to map collector payloads into canonical records.

Canonical record fields:

name
description
type
unit
value
labels
optional timestamp_unix_ms

This separated normalization from transport and moved us closer to the milestone’s delivery pipeline model.

4) Formatter support for canonical metrics

Extended src/formatters/prometheus.py with:

format_canonical(metrics)

This lets formatters operate on canonical data (not collector-specific payload dictionaries) while preserving the existing Prometheus text exposition output.

5) Main loop refactor to exporter lifecycle

Updated src/main.py so startup and runtime flow is now:

resolve EXPORTER_TYPE
create_exporter(...)
exporter.initialize()
collect payloads each cycle
normalize to canonical metrics
exporter.export(canonical_metrics)
exporter.shutdown() in finally

Direct print(...) of telemetry output was removed from main.py.

6) Python 3.10 typing compatibility fix

src/exporters/base.py originally used typing.NotRequired, which is not available in Python 3.10’s typing module.

The CanonicalMetric TypedDict was adjusted to a Python-3.10-safe form using:

required base TypedDict
total=False extension for optional timestamp

This kept type intent intact without requiring runtime upgrades.

Validation and runtime output

Validation performed:

python3 -m compileall src
smoke execution path for exporter creation + canonical conversion + console export

Observed runtime behavior after refactor:

collector logs still emitted
Prometheus lines still emitted each cycle
timestamps populated in Unix milliseconds
disk metrics emitted as aggregate and per-device series

In other words: behavior stayed stable, but ownership moved to exporter architecture.

What M3-3 achieved (and what it didn’t)

Completed in M3-3:

console output now runs through exporter interface
runtime flow is wired to exporter lifecycle
normalization and formatting layers are separated from collection

Not part of M3-3 (handled in later items):

strict fail-fast invalid exporter value handling (M3-4)
additional backend exporters (Datadog, OTLP HTTP, New Relic)
retry/buffering semantics beyond base hooks

Key lesson

This milestone was less about adding features and more about installing architectural seams.

The code now has explicit boundaries:

collectors collect
pipeline normalizes
formatter renders
exporter delivers

That separation is what will let future backend integrations land without rewriting collectors.

Next up

The immediate next step is M3-4: enforce strict startup validation for unsupported EXPORTER_TYPE values so runtime fails fast with explicit errors instead of warning + fallback.

Next Milestones for Heka Insights Agent: From Console Output to Real Telemetry Delivery

Farhan Munir — Tue, 21 Apr 2026 08:48:33 +0000

We’re moving beyond console-only output and into real telemetry delivery.

Over the next few milestones, heka-insights-agent will introduce a proper exporter layer, OTLP HTTP support, and first-class integrations for New Relic and Datadog. The focus is to keep the agent vendor-agnostic at its core while making it easier to route system metrics into modern observability platforms using clean, configurable dispatch patterns.

Project board: https://github.com/users/ronin1770/projects/3/views/1

python #observability #opentelemetry #devops #monitoring #prometheus #datadog #newrelic #opensource

✅ Milestone Completed: Prometheus Data Format Integration

Farhan Munir — Sun, 19 Apr 2026 07:36:58 +0000

✅ Milestone Completed: Prometheus Data Format Integration

This sprint focused on making telemetry output fully Prometheus-compatible while preserving our existing collector pipeline for CPU, memory, and disk I/O.

What I implemented

Added Prometheus text exposition format (v0.0.4) output
Included # HELP and # TYPE metadata
Supported gauge and counter metric families
Added label-based dimensions (CPU modes, disk devices)
Kept output deterministic and scrape-ready for a /metrics style endpoint

Metrics now covered

CPU usage and per-mode CPU time distribution
Virtual memory: used, available, total
Swap memory: used, total
Disk I/O bytes: read/write (aggregate + per-device)
Disk I/O operations: read/write (aggregate + per-device)

Validation results

Output verified against Prometheus exposition rules
No syntax violations found
Metrics are parseable and scrape-ready

Prior milestone context (OpenMetrics)

Before this, I shipped OpenMetrics-aligned output for the same core Linux host metrics with:

# HELP, # TYPE, # UNIT metadata
# EOF termination
Valid CPU, memory, and disk payload output

Next improvements

Evaluate ratio-based CPU values (0–1) vs percentage
Extend optional unit handling for stronger tooling interoperability
Continue OpenMetrics alignment where needed

Repo: https://github.com/ronin1770/heka-insights-agent

Milestone 2: Standardizing Telemetry Output with JSON, Prometheus, and OpenMetrics

Farhan Munir — Thu, 16 Apr 2026 15:25:23 +0000

Milestone 2: Standardizing Telemetry Output with JSON, Prometheus, and OpenMetrics

In this milestone, we are focusing on one thing only: data format standardization.

The Heka Insights Agent already collects CPU, memory, and disk telemetry.

Now the goal is to emit the same logical metrics in three standard output formats:

JSON
Prometheus text exposition
OpenMetrics text format

Why This Milestone Matters

If an agent has no clear format strategy, every downstream integration becomes custom work.

That slows down adoption and increases maintenance cost.

By standardizing format early, we get:

stable contracts for integrations
easier validation and testing
portability across observability stacks
clearer boundaries between collection and export

Milestone Scope (Only Data Format)

This milestone does not include transports, retry logic, or backend adapters.

It only covers how telemetry is represented and serialized.

Included:

canonical internal metric model
naming/type/unit rules
serializers for json, prometheus, openmetrics
deterministic output behavior
contract tests with golden files

Out of scope:

Datadog/New Relic senders
batching/compression/persistence
new collector domains

Canonical Metric Contract

Every metric will be representable through one shared contract:

name (string)
description (string)
type (gauge or counter)
unit (e.g. bytes, seconds, percent, count)
value (number)
labels (map of string to string; empty allowed)
timestamp_unix_ms (optional integer)

This contract is the core design decision in Milestone 2.

Serializers consume this model and render format-specific output without changing metric meaning.

Naming and Semantics Rules

To keep the output stable and machine-friendly:

metric names are lowercase snake_case
all names are prefixed with heka_
counters end in _total
unit suffixes are explicit (_bytes, _seconds, _percent)
label keys are lowercase snake_case
metric identity must stay consistent across formats

Current Metric Mapping

Initial canonical mapping includes:

heka_cpu_usage_percent (gauge)
heka_cpu_time_percent (gauge with mode=<field>)
heka_memory_virtual_used_bytes (gauge)
heka_memory_virtual_available_bytes (gauge)
heka_memory_virtual_total_bytes (gauge)
heka_memory_swap_used_bytes (gauge)
heka_memory_swap_total_bytes (gauge)
heka_disk_read_bytes_total (counter)
heka_disk_write_bytes_total (counter)
heka_disk_reads_total (counter)
heka_disk_writes_total (counter)

Format-Specific Requirements

JSON

UTF-8 JSON object
includes schema_version (starting at v1)
includes generated_at (RFC3339 UTC)
includes top-level metrics array

Prometheus

Prometheus text exposition format (0.0.4)
include # HELP and # TYPE lines
deterministic label ordering
no OpenMetrics-only directives

OpenMetrics

OpenMetrics text format
include # HELP, # TYPE, and # UNIT when known
terminate payload with # EOF
metric names and labels remain aligned with Prometheus mode

Configuration Contract

One selector controls serialization:

OUTPUT_FORMAT=json|prometheus|openmetrics

default: json
invalid values: fail fast with a clear startup error

Acceptance Criteria

Milestone 2 is done when:

same logical metric set is emitted in all three formats
names/types/units are consistent
Prometheus and OpenMetrics outputs validate
JSON includes schema metadata and metrics array
output order is deterministic
golden-file tests exist for each format

GitHub Milestone Breakdown

Work is tracked through:

M2-1 canonical metric model
M2-2 collector-to-canonical mapping
M2-3 JSON serializer
M2-4 Prometheus serializer
M2-5 OpenMetrics serializer
M2-6 output format config + validation
M2-7 fixture/contract tests
M2-8 docs update

Repo

Heka Insights Agent Update: Architecture + Configuration Docs Now Reflect Runtime Behavior

Farhan Munir — Thu, 16 Apr 2026 05:37:17 +0000

Build Update (April 16, 2026)

This week I focused on documentation quality and operational clarity for heka-insights-agent.

The goal was simple: make docs match the code exactly, so contributors and operators can reason about behavior without reading every module first.

What I updated

Rewrote docs/architecture.md from scratch
Rewrote docs/configuration.md from scratch
Expanded README.md with project context, setup, and environment guidance

Architecture documentation improvements

docs/architecture.md now documents:

the actual runtime topology and control loop in src/main.py
collector boundaries and behavior (CPUCollector, MemoryCollector, DiskCollector)
logging subsystem behavior in src/logger/config.py
current payload shapes emitted by collectors
known gaps (no sender layer yet, no tests yet, no schema versioning yet)
practical extension points for next phases

This gives a real “as-implemented” architecture baseline instead of aspirational text.

Configuration documentation improvements

docs/configuration.md now includes exact behavior for the two active runtime settings:

LOG_LOCATION
CPU_POLL_INTERVAL_SECONDS

It also documents:

source/precedence rules
defaults and validation behavior
failure modes
local setup and production recommendations

Important behavior clarified

Current config loading is split across two env files:

root .env is used for LOG_LOCATION
src/.env is used for CPU_POLL_INTERVAL_SECONDS

That split is now explicitly documented to reduce startup/debug confusion.

Why this matters

For an agent project, docs are part of reliability.

Operators need to know what can fail at startup, where config is read from, and what telemetry shape to expect downstream.

This update makes onboarding and future refactors safer.

Next steps

add transport/sender layer (backend adapters)
add collector-focused tests
consolidate config loading into a single source
define schema/versioning strategy for emitted payloads

Repo: https://github.com/ronin1770/heka-insights-agent

Reel Quick - Added Docker Support

Farhan Munir — Wed, 15 Apr 2026 14:17:25 +0000

Date: 2026-04-15

Project: Reel Quick (FastAPI + Next.js + ARQ + Mongo + Redis + optional GPU workers)

Context

We containerized the stack and tried to run it in production mode with Docker Compose.

Initial startup failed for both frontend build and backend runtime.

Issues Found

Docker Compose path mismatches:
Wrong env_file paths (docker/env/*.env expected but files were in docker/).
Wrong nginx config mount path (./docker/nginx/nginx.conf while actual file was docker/nginx.conf).
Build context was incorrect for a compose file located inside docker/.
Frontend TypeScript build failure:
location field type mismatch in frontend/app/create_video/page.tsx.
Value inferred as string | undefined but state expects string | null.
Backend container crash on startup:
ModuleNotFoundError: No module named 'db'.
backend/main.py used non-package imports like from db import ....

Root Causes

Relative paths in compose were not aligned with actual file layout.
Optional API response property (file_location?) was used directly inside state update.
Backend entrypoint (uvicorn backend.main:app) requires package-safe imports (backend.*).

Fixes Applied

Docker and Compose

Updated docker/docker-compose.yml:
- build.context changed from . to ...
- env_file paths corrected to backend.env and mongo.env.
- nginx bind mount fixed to ./nginx.conf.
Updated docker/backend.env:
- Added:
- UPLOAD_FILES_LOCATION=/app/video_files
- INPUT_FILES_LOCATION=/app/video_files
Added repo-root .dockerignore (Docker uses ignore file from build context root).
Synced docker/dockerignore entries.
Updated docker/README-docker-prod.md run commands.

Frontend

Fixed type narrowing in frontend/app/create_video/page.tsx:
- Captured file_location into uploadedLocation.
- Guarded before setFiles(...).
- Used guaranteed string value in state update.

Backend

Converted backend imports to package imports in backend/main.py:
- from db import ... -> from backend.db import ...
- Similar conversion for logger, models, objects, workers.
Updated backend/objects/sound_prompt_preset.py import to backend.objects....

Commands Used for Deploy

# Stop all running containers (host-wide)
docker ps -q | xargs -r docker stop

# Start Reel Quick with GPU workers
cd /home/farhan/reel-quick/docker
docker compose --profile gpu up -d --build

# Verify
docker compose ps
docker compose logs -f api --tail=200

Validation Checklist

docker compose ps shows api, frontend, nginx, mongo, redis, workers as running.
api logs no longer show ModuleNotFoundError: No module named 'db'.
Frontend image builds successfully (npm run build passes in container build stage).
Upload endpoint works (POST /uploads).
Workers/control panel endpoints return expected data.

Key Takeaways

Keep compose file paths consistent with its directory and build context.
Use package-qualified imports for Python app modules in containerized runtimes.
Narrow optional API fields before state updates in strict TypeScript projects.
Add .dockerignore at the actual build context root to avoid bloated builds.

Build Log: Implementing Full Text Overlay Feature in Reel Quick (with Accurate Live Preview)

Farhan Munir — Fri, 10 Apr 2026 05:46:36 +0000

Build Log: Implementing Full Text Overlay Feature in Reel Quick (with Accurate Live Preview)

In this build, I implemented the complete text overlay workflow in Reel Quick: from UI controls to background processing, plus a preview system that better matches final output.

Repo: https://github.com/ronin1770/reel-quick

Why this feature mattered

The earlier flow allowed adding overlay text, but styling control was limited and preview confidence was low.

Users needed to:

control text appearance (size, color)
control placement (top/center/bottom)
preview changes instantly before processing
avoid trial-and-error renders

The goal was to make text overlays practical for real reel production, not just a placeholder UI.

What the feature now includes

1. Overlay content + timing

Users can create overlays with:

text
start time
end time

2. Style controls

Added styling inputs in the dialog:

Font size range: 40–200
Text color:
- HTML5 color picker (input type="color")
- HEX input (synced with picker)
Position selector:
- top
- center
- bottom

3. Live preview (client-side only)

The overlay updates instantly while editing:

no backend call
no queue call
no video reprocessing

This lets users iterate quickly before clicking process.

4. Dialog UX improvements

The modal was redesigned to be usable at production scale:

controls on the left
preview on the right
scrollable modal for smaller viewports
action buttons always reachable

The key technical challenge: preview/output mismatch

A big issue was that selected preview size didn’t visually match rendered output.

Root cause

Frontend preview text used raw CSS px in a display container, while backend renders text on actual output video resolution.

Same number, different render context.

Fix strategy

I changed preview scaling logic to account for real video dimensions:

Read intrinsic source dimensions from video metadata (videoWidth, videoHeight)
Measure actual preview frame dimensions in the modal
Compute scale factor:
- scale = min(previewWidth/sourceWidth, previewHeight/sourceHeight)
Render preview text as:
- previewFontSize = selectedFontSize * scale
Preserve source aspect ratio in preview container
Mirror vertical placement behavior (top/center/bottom with scaled edge padding)

Result: preview size and placement now feel much closer to final rendered video.

API + backend integration

Good news: backend already supported style fields, so no backend API redesign was needed.

The frontend sends per-overlay payload with:

style.font_size
style.text_color
position.preset

Then it follows the existing pipeline:

Save overlays POST /videos/{video_id}/text-overlays
Enqueue processing POST /enqueue/text-overlay
ARQ worker picks job and runs MoviePy text overlay composition

Data flow (end to end)

User opens text overlay dialog
Configures text + timing + style + position
Verifies in live preview
Saves overlay config
Enqueues job
Worker validates and renders final video
Processed text-overlay video becomes available for download

Validation and guardrails

Font size is clamped to configured range
HEX color is normalized/validated
Overlay timing is validated before processing
Position options are constrained to supported presets
Preview updates remain instant and local

What changed from earlier behavior

Old frontend default style used a very small fixed size
New feature provides interactive style control + accurate preview scaling
Modal UX is now horizontal and production-friendly

What’s next

Potential follow-ups:

font family selection
stroke/shadow controls in UI
drag-and-drop custom placement
multiple overlay tracks/timeline editing

If you want, I can also generate:

a shorter Dev.to teaser version
an accompanying screenshot checklist for the post
a “before vs after” section with technical diff notes

Build Log: Shipping a Lean Python Telemetry Agent (CPU, Memory, Disk)

Farhan Munir — Wed, 08 Apr 2026 09:29:51 +0000

Build Log (April 8, 2026)

Today I implemented the first production-ready telemetry collectors for heka-insights-agent and wired them into the main polling loop.

What I built

Added an optimized CPUCollector in src/collectors/cpu.py
Added a MemoryCollector in src/collectors/memory.py
Added a DiskCollector in src/collectors/disk.py
Wired all collectors into src/main.py with a shared loop
Added environment-based poll interval support via CPU_POLL_INTERVAL_SECONDS
Added python-dotenv in requirements.txt

CPU collector design

I built CPU collection around psutil.cpu_times(...) snapshots and delta math (single source), instead of calling both cpu_percent and cpu_times_percent per cycle.

Key design points:

No thread offloading (to_thread) for this workload
First cycle is warm-up by design
Supports basic and detailed output modes
Optional per-core output
Uses MonotonicTicker to keep fixed cadence without drift

Memory collector design

Memory collection is intentionally lightweight:

One call each to psutil.virtual_memory() and psutil.swap_memory()
basic mode returns compact key fields
detailed mode returns full psutil fields
Raw byte values are preserved (server-side compute handles transformations)

Disk collector design

For disk, I chose cumulative I/O counters (not rates) because central compute is done server-side.

Uses psutil.disk_io_counters(perdisk=True)
Returns aggregate and per-disk counters
Filters to physical devices only
Excludes partitions from per-disk payload
Added device-name cache with periodic refresh to reduce repeated filtering overhead

Main loop wiring

src/main.py now runs:

CPU collector
Memory collector
Disk collector

All on the same interval, with separate log lines per collector.

Poll interval is loaded from .env via:

CPU_POLL_INTERVAL_SECONDS

Invalid values fall back safely to default 5.0s.

Profiling notes

I profiled a 120-second run and reviewed both process stats and cProfile output.

Key findings:

Agent CPU cost is very low (near-idle for this polling interval)
Max RSS is about 15 MB
Runtime is dominated by intentional sleep (expected)
Collector costs are small; disk collection is the heaviest of the three

What changed after profiling

Based on profile output, I optimized disk collection further:

Added cached physical-device list to avoid filtering every cycle
Kept output shape unchanged (disk_io + disk_io_perdisk)

Current status

The agent now has a clean baseline telemetry pipeline with low overhead and clear extension points for transport/shipping.

Next planned work:

Add payload shipping to backend endpoint
Add bounded retry/backoff
Add collector-focused tests

Repo URL

ronin1770 / heka-insights-agent

A lightweight agent for collecting essential Linux system telemetry and shipping it to a configurable backend.

heka-insights-agent

A lightweight agent for collecting essential Linux system telemetry and shipping it to a configurable backend.

Test

View on GitHub

Build Log: End-to-End Text Overlay Workflow for Video Processing

Farhan Munir — Thu, 02 Apr 2026 16:04:23 +0000

Build Log: End-to-End Text Overlay Workflow for Video Processing

Today I completed the core text overlay workflow across the frontend, backend, and worker pipeline. The main goal was to make text overlays behave like a real production feature instead of a partial UI action. That meant saving overlays, enqueueing processing jobs, running worker execution, tracking status, and making sure users can download the final processed file when rendering is complete.

What changed

The biggest update was implementing the full text overlay processing lifecycle from the UI down to the background worker.

A user can now create overlays for a video, save them through the API, enqueue a rendering job, and return to the videos page while processing continues in the background. Once the job is finished, the videos listing reflects that state and download actions point to the processed overlay output instead of the original file.

I also finalized one important behavior change in overlay saving. Instead of merging new overlays with existing ones, each save call now replaces the stored overlays for that video. This prevents stale overlay entries from lingering and causing recurring overlap conflicts.

Backend work completed

On the backend, I added the API and queue flow required to support the feature end to end.

The following routes are now in place:

POST /videos/{video_id}/text-overlays
POST /enqueue/text-overlay
GET /text-overlay-jobs
GET /videos/{video_id}/text-overlays/download

The save endpoint now stores the submitted overlays for a given video. The enqueue endpoint pushes a text overlay processing job into the queue and returns quickly, so the frontend does not need to wait on worker readiness before moving forward.

To support job visibility, I introduced a text_overlay_jobs collection for tracking processing records. This collection is now used for queued, pending, finished, and failure state tracking. I also added the required database initialization and an index on text_overlay_jobs.video_id so lookup remains efficient.

On the worker side, I added process_text_overlay_job and wired a dedicated text_overlay_worker into the queue system using TEXT_OVERLAY_QUEUE_NAME. That worker is also registered in the control panel worker setup so it is managed consistently with the rest of the background processing stack.

Another important backend fix was changing overlay persistence behavior from merge to replace. Earlier behavior allowed stale overlays to remain in the record, which could trigger overlap validation conflicts even after the user thought they had corrected the layout. Replacing overlays on save solved that problem cleanly.

Frontend updates

On the frontend, the CreateTextOverlayPage is now connected to the real processing flow.

The Done / Process Video action now performs the expected sequence:

Save overlays
Enqueue the text overlay job
Redirect the user back to /videos

I also added a loading and disabled state to prevent duplicate submissions while the request is in progress. This gives the action more predictable UX and reduces accidental repeat clicks.

Error handling was improved as well. Instead of showing vague failures, the frontend now parses backend validation messages more clearly, including field paths where possible. That makes it much easier to debug malformed overlay payloads during testing.

Because some UI controls are still not finalized, I added temporary default values for fields that are not yet exposed in the interface. Current defaults are:

position: top/center
style: font size 16, weight normal

These defaults let the feature move forward without blocking on the final design of overlay controls.

Videos page behavior

I also updated the /videos listing page so the new workflow feels integrated instead of isolated.

The Video ID column was removed to simplify the table and give more room to user-facing actions.

The Text Overlay column now reflects job state more meaningfully. If the overlay job for a video is finished, the action shows Done. Otherwise, it shows Create.

Download behavior was also updated. When an overlay job is finished successfully, the output and error download actions now point to the processed overlay video instead of the original file. If overlay processing has not completed, downloads continue to use the regular video file.

That small change is important because it makes the final output feel connected to the workflow the user just triggered.

Why the replace-on-save decision mattered

One of the more important logic decisions today was changing overlay saving from merge semantics to replace semantics.

Merging sounds convenient at first, but in practice it created stale state problems. Old overlays could remain attached to the video even after the user thought they were working from a clean set. That caused overlap validation issues to reappear and made the experience feel inconsistent.

Replacing overlays on each save call makes the system much more predictable. The saved overlays now reflect exactly what the user submitted in the latest request, which is the safer behavior for this kind of editor flow.

Current workflow

At this point, the text overlay feature supports the following flow:

User opens the text overlay page for a video
User adds or edits overlays
Frontend saves the overlays
Frontend enqueues a processing job
Worker renders the text overlay output in the background
Job status is tracked in text_overlay_jobs
Videos page reflects completion state
Download action serves the overlay-rendered file when available

That means the feature is now functional across storage, processing, status tracking, and output retrieval.

What is still temporary

A few parts are still intentionally temporary or transitional.

The overlay editor is still using fallback defaults for position and style where dedicated UI controls have not been built yet. The current implementation is enough to keep the workflow functional, but the page still needs the final interactive form controls for layout and style management.

The videos page is also currently showing a simplified status signal. This works for now, but it could later become more expressive with states like queued, processing, failed, and finished.

Final result

This was a solid progress day because the feature moved from partial UI scaffolding into a real end-to-end system. The frontend now triggers meaningful backend work, the worker actually processes jobs, job state is persisted, and the final output can be downloaded through the app.

Most importantly, the workflow now behaves in a predictable way. Saving overlays replaces prior state, queueing does not block on hard worker health checks, and completed processing is reflected directly in the videos list.

The next step is to improve the overlay editor UI itself, but the underlying pipeline is now in place.

DEV Community: Farhan Munir

Milestone 4 Complete — OTLP HTTP Exporter for Heka Insights Agent

What was implemented in Milestone 4

Configuration added

Notes

Unit tests

Unit test output

Integration tests (Docker + real collector)

Integration test output

Collector config used in tests

Final result

Milestone 4 (Part 1): Implementing OTLP HTTP Core in Heka Insights Agent (M4-1, M4-2)

Milestone 4 (Part 1): Implementing OTLP HTTP Core in Heka Insights Agent (M4-1, M4-2)

Why This Split Matters

What Was Implemented

M4-1: OTLP Payload Mapping Layer

M4-2: OTLP HTTP Sender + Exporter

Local Test Setup with OpenTelemetry Collector (Docker)

Collector config (otel-collector-config.yaml)

Run collector

Agent .env for this test

Run agent

Verification Signals

Tests Added

What Is Intentionally Not Included Yet

Closing

Heka-Insights-Agent: Milestone M3-4: Fail-Fast Exporter Validation at Startup

Milestone M3-4: Fail-Fast Exporter Validation at Startup

Context

The Problem Before M3-4

M3-4 Goals

Implementation Summary

1) Strict validation in runtime config

2) Removed fallback from exporter factory

3) Documentation updated to M3-4 semantics

Validation Performed

Compile validation

Behavior validation

Why this change matters operationally

Architectural Impact

What’s Next

Milestone M3-3: Refactoring Console Output Into an Exporter Pathway

Milestone M3-3: Refactoring Console Output Into an Exporter Pathway

Why M3-3 mattered

What the code looked like before

Design goals for this refactor

What was implemented

1) Console exporter implementation

2) Exporter factory wiring

3) Canonical metric normalization pipeline

4) Formatter support for canonical metrics

5) Main loop refactor to exporter lifecycle

6) Python 3.10 typing compatibility fix

Validation and runtime output

What M3-3 achieved (and what it didn’t)

Key lesson

Next up

Next Milestones for Heka Insights Agent: From Console Output to Real Telemetry Delivery

python #observability #opentelemetry #devops #monitoring #prometheus #datadog #newrelic #opensource

✅ Milestone Completed: Prometheus Data Format Integration

✅ Milestone Completed: Prometheus Data Format Integration

What I implemented

Metrics now covered

Validation results

Prior milestone context (OpenMetrics)

Next improvements

Milestone 2: Standardizing Telemetry Output with JSON, Prometheus, and OpenMetrics

Milestone 2: Standardizing Telemetry Output with JSON, Prometheus, and OpenMetrics

Why This Milestone Matters

Milestone Scope (Only Data Format)

Canonical Metric Contract

Naming and Semantics Rules

Current Metric Mapping

Format-Specific Requirements

JSON

Prometheus

OpenMetrics

Configuration Contract

Acceptance Criteria

GitHub Milestone Breakdown

Collector config (`otel-collector-config.yaml`)

Agent `.env` for this test