TL;DR: Treat a malformed metrics-query response as an explicit unknown result, never as zero failures. For a marketplace's nightly data pipeline, attach tenant, pipeline stage, and run identifiers before logs enter shared storage; measure stored bytes by those dimensions; retain compact audit records for every query; and alert separately when the monitoring response cannot be proved valid. The least complex implementation is a strict Go decoder at the worker boundary plus a small, bounded quarantine record containing a digest and a truncated payload.
This ordering matters because the monitoring bill is usually discussed after the parser breaks, although the bill determines what evidence will still exist when it does. The useful quantity is not an advertised unit price. It is attributable stored volume:
daily bytes = sum(encoded event bytes × copies × retention path)
Measure that term first. A worker that silently converts a bad response into an empty result can suppress the failure alert and also assign the query's retrieval cost to an unclassified bucket. That is a correctness defect with an accounting symptom.
What is the bill actually made of?
For this workload, separate ingestion, retained storage, query scanning, and duplicated copies. CloudWatch's public pricing page, for example, describes log ingestion in per-GB terms; the durable lesson is that bytes crossing the ingestion boundary are a cost driver, while the exact commercial rate is deliberately outside this design. Compression, indexing, regional replication, and query execution may add different meters in a particular system, so exporting one total invoice number into a dashboard does not create cost attribution.
Start with an inventory measured from the encoded records, not from an estimate of source rows. The following program accepts an illustrative nightly volume, average encoded size, copy count, and retention days. The sample inputs are an arithmetic demonstration, not a benchmark: 8,000,000 records, 420 bytes each, two retained copies, and 14 days.
package main
import "fmt"
func gib(bytes uint64) float64 {
return float64(bytes) / (1024 * 1024 * 1024)
}
func main() {
const records uint64 = 8_000_000
const encodedBytes uint64 = 420
const retainedCopies uint64 = 2
const retentionDays uint64 = 14
daily := records * encodedBytes * retainedCopies
retained := daily * retentionDays
fmt.Printf("daily_ingested_gib=%.2f retained_gib_days=%.2f\n", gib(daily), gib(retained))
}
The multiplication exposes the dominant term without pretending every backend bills identically. Run it again with measurements from each pipeline stage. If catalog_export emits ten times the bytes of settlement_reconcile, optimize the former first even when the latter has the more alarming event count.
Cost attribution must be present at emission time. A useful minimum is marketplace_account, pipeline_stage, run_id, event_class, and outcome. Keep bounded values; a raw error message or order identifier used as a label can create an uncontrolled number of index terms. The high-cardinality detail belongs in the log body, while the stable dimensions belong in the fields used for allocation and filtering.
Do not infer ownership later from free-form text. It fails audit review because the mapping can change after the charge was incurred.
How should an alert worker defensively parse a malformed metrics query JSON response?
The response boundary needs three distinct checks: syntactic JSON validity, the expected envelope shape, and semantic validity of the values used by the alert. HTTP success alone proves none of them. A body containing valid JSON such as {"status":"ok"} is still unusable if the worker's contract requires a numeric failure count and a query identifier.
Model the outcome as valid, invalid, or transport_error. Two states are insufficient. In particular, do not use a zero value as the fallback for a missing number: zero means the pipeline was observed and no failures matched, whereas invalid means the observation could not be completed.
This complete program uses only the Go standard library. UseNumber delays numeric conversion, DisallowUnknownFields detects contract drift rather than ignoring it, the second decode rejects trailing JSON values, and validation refuses blank identifiers, negative counts, or a run mismatch.
package main
import (
"bytes"
"encoding/json"
"errors"
"fmt"
"io"
)
type QueryResult struct {
QueryID string `json:"query_id"`
RunID string `json:"run_id"`
FailureCount json.Number `json:"failure_count"`
}
func parseResult(body []byte, expectedRun string) (QueryResult, error) {
var result QueryResult
decoder := json.NewDecoder(bytes.NewReader(body))
decoder.UseNumber()
decoder.DisallowUnknownFields()
if err := decoder.Decode(&result); err != nil {
return result, fmt.Errorf("decode response: %w", err)
}
if err := decoder.Decode(&struct{}{}); !errors.Is(err, io.EOF) {
if err == nil {
return result, errors.New("response contains a trailing JSON value")
}
return result, fmt.Errorf("inspect response suffix: %w", err)
}
if result.QueryID == "" || result.RunID == "" || result.FailureCount == "" {
return result, errors.New("response omits a required field")
}
if result.RunID != expectedRun {
return result, errors.New("response run_id does not match the requested run")
}
count, err := result.FailureCount.Int64()
if err != nil || count < 0 {
return result, errors.New("failure_count must be a non-negative integer")
}
return result, nil
}
func main() {
tests := [][]byte{
[]byte(`{"query_id":"q-41","run_id":"run-2026-10-08","failure_count":3}`),
[]byte(`{"query_id":"q-42","run_id":"run-2026-10-08","failure_count":`),
[]byte(`{"query_id":"q-43","run_id":"run-2026-10-08"}`),
[]byte(`{"query_id":"q-44","run_id":"wrong-run","failure_count":0}`),
}
for _, body := range tests {
result, err := parseResult(body, "run-2026-10-08")
fmt.Printf("query=%q count=%q valid=%t error=%v\n",
result.QueryID, result.FailureCount, err == nil, err)
}
}
Strict unknown-field rejection is a conscious trade-off. It turns an additive upstream field into a visible compatibility event, which is appropriate when this narrow envelope is an internal contract and deployment is coordinated. For a public or independently evolving API, decode a versioned envelope and tolerate documented additive fields instead. The invariant is narrower: the worker must prove the fields that drive the alert, bind them to the requested run, and reject ambiguous values.
Preserve evidence without retaining every successful payload
Parsing is only half the incident record. For every attempt, write an immutable audit event with the request ID, run ID, pipeline stage, account allocation key, response class, byte length, content digest, parser version, and final decision. Store the raw body only in a bounded quarantine path when the class is invalid; cap its length before persistence and apply access controls appropriate to the data it may contain.
The digest answers a narrow but important question: did two workers receive the same bytes? It does not establish what the remote system intended to send, and it must not be used as a substitute for the response body during diagnosis. Likewise, retries must reuse an idempotency key derived from the logical query and pipeline run, while each attempt receives its own attempt number. The logical decision record is written once; attempt records remain append-only.
That is the exactly-once mindset in practical form. Network delivery is allowed to repeat. The alert decision is not.
A parser failure should open a monitoring-integrity alert whose fingerprint is based on stable dimensions such as operation, schema version, and parser error class, rather than the raw body or request ID. Sentry documents how event grouping and custom fingerprints influence which events become one issue; the general lesson applies without requiring a particular incident tool. Stable grouping prevents every nightly run from becoming a separate incident while retaining run IDs in the event details for reconstruction.
Make the alert state machine explicit
After parsing, the decision table is small enough to review with both operations and finance:
| Observation | Pipeline alert | Integrity alert | Cost allocation |
|---|---|---|---|
| Valid count is zero | Closed | Closed | Requested account and stage |
| Valid count is positive | Open | Closed | Requested account and stage |
| JSON or schema is invalid | Unknown, never closed | Open | Requested account and stage plus invalid_response
|
| Transport fails after retry budget | Unknown, never closed | Open | Requested account and stage plus transport_error
|
The distinction between open and unknown is operationally useful. Reusing the pipeline-failure alert for parser defects obscures whether the marketplace job failed or its observer failed. Closing an existing failure alert on unknown evidence is worse: it converts loss of visibility into apparent recovery.
Deploy the contract parser behind recorded fixtures for truncated input, HTML accidentally returned as a body, valid JSON with missing fields, decimal counts, negative counts, duplicate logical responses, and a mismatched run ID. The HTML fixture should fail at decoding; no special content sniffing is required for correctness. Also test the retry boundary with a fake transport so that two deliveries produce two attempt records but one logical decision.
These tests serve auditability as much as reliability. A reviewer can trace the decision from requested run, through exact response classification, to the alert transition and its allocation key.
Retention is a decision about future evidence
The change that moves the dominant storage term is usually selective retention, not parser micro-optimization. Keep compact audit events for all queries, full invalid responses for a short and explicitly governed quarantine period, and aggregated counts for trend analysis. Stop keeping full successful query payloads once the audit event and verified aggregate have been committed.
This deliberately removes the ability to replay an old successful response byte for byte after its retention window. If a later parser defect is discovered, the team can reconstruct which parser version made the decision and compare the stored count and digest, but it cannot reinterpret a body that no longer exists. That loss should be accepted in a retention review, with compliance and incident-reconstruction requirements setting the window; it should not emerge accidentally from a storage cleanup.
The final control is reconciliation. Compare the count of scheduled pipeline runs with valid decisions, invalid decisions, and exhausted transports for the same allocation period. Those categories must account for every run exactly once. Any remainder is a missing observation, and any excess is a duplicate logical decision, even if every individual worker log line looks healthy.
Top comments (0)