Farhan Munir

Posted on Jul 25

Implementing HTTP 429 Retry-After Handling in Heka Insights Agent

#python #opensource #testing #opentelemetry

We opened a focused reliability issue in the Heka Insights Agent to improve how the OTLP HTTP exporter behaves under backend rate limiting.

Issue URL: https://github.com/ronin1770/heka-insights-agent/issues/22

Repository: https://github.com/ronin1770/heka-insights-agent

The problem is simple: if an ingestion gateway returns 429 Too Many Requests, the agent should not immediately keep retrying on a short generic backoff loop until all attempts are exhausted.

If the server says:

429 Too Many Requests
Retry-After: 20

then the agent should respect that instruction, wait, and retry the same metrics batch at the correct time.

Why This Matters

In telemetry pipelines, rate limiting is normal. Gateways protect themselves, vendors enforce quotas, and burst traffic happens.

What matters is how the agent behaves when that happens.

A weak retry strategy looks like this:

request fails with 429
exporter retries too quickly
all configured attempts are consumed before the rate-limit window resets
the batch fails even though the backend gave a valid recovery signal

That is not just inefficient. It also creates noisy logs, unnecessary traffic, and lower delivery reliability.

For an agent that is expected to stay running continuously, this is the wrong operational behavior.

The Goal of Issue #22

Issue #22 defines a more correct path for 429 Too Many Requests handling in the OTLP HTTP exporter.

The required behavior is:

detect 429 explicitly instead of treating it like a generic transient HTTP error
read the Retry-After header case-insensitively
support integer-second delay values
use a default delay when the header is missing or invalid
cap excessive values with configuration
retry the same OTLP payload
reuse the same idempotency key
avoid stacking exponential backoff on top of server-provided delay
keep the main agent process and collectors running
add logging and timing instrumentation around the retry path

This is a narrow but important reliability improvement.

The Core Design Choice

The most important part of the issue is not just "sleep and retry."

It is retrying the same batch with the same idempotency key after honoring the delay requested by the server.

That protects correctness in a few ways:

the exporter does not replace the pending batch with newer metrics
retry semantics stay tied to the original request
the backend sees the same logical delivery attempt after the rate-limit window
clients do not accidentally create duplicate semantics by regenerating request identity

This is especially important for observability delivery paths, where retries need to be operationally safe, not just convenient.

Configuration Rules

The issue introduces explicit control for Retry-After behavior:

OTLP_RETRY_AFTER_DEFAULT_SECONDS=5
OTLP_RETRY_AFTER_MAX_SECONDS=300
OTLP_MAX_EXPORT_ATTEMPTS=5

That gives the exporter deterministic behavior across common cases:

missing header -> use default
empty header -> use default
invalid value -> use default
negative value -> use default
zero -> retry without additional delay
value above max -> cap it

This is the kind of behavior that operators can reason about quickly.

Async Sleep Matters

One of the stronger requirements in the issue is that the wait must be asynchronous:

await asyncio.sleep(retry_after_seconds)

That detail matters because the agent should continue running normally while one export batch is waiting for its retry window.

The wait should not:

block the event loop
freeze collectors
terminate the process
force a broad retry redesign for unrelated HTTP failures

This keeps the scope disciplined while still improving real runtime behavior.

Logging and Timing

The issue also calls for better observability around exporter retries themselves.

That includes logging:

endpoint
HTTP status
current attempt
maximum attempts
parsed Retry-After value
applied delay
whether defaulting or capping occurred
masked idempotency-key reference
retry outcome

It also asks for timing data such as:

retry_after_parse_ms
retry_after_sleep_seconds
retry_request_ms
total_batch_export_ms

This is important because retry logic without timing visibility becomes difficult to debug in production.

Testing Scope

Issue #22 is also explicit about testing expectations.

The test coverage should verify:

header parsing for valid and invalid cases
case-insensitive header handling
same payload reuse on retry
same idempotency key reuse on retry
correct attempt accounting
no extra exponential backoff layered onto Retry-After
successful retry exit behavior
max-attempt failure without crashing the agent
clean shutdown during the wait
collector continuity while the exporter is waiting
no logging of secrets or raw OTLP payloads

There is also a clear integration expectation: return 429, wait the correct amount of time, retry the same batch, then succeed and exit the retry loop cleanly.

Why I Like This Issue

This is the kind of issue that improves a system in a practical way.

It does not try to redesign all retry behavior in one pass. It focuses on one real failure mode, defines the expected runtime behavior precisely, and forces the implementation to think about: