DEV Community

GregorSterling9652
GregorSterling9652

Posted on

Malformed Metrics Query Response JSON — Rollback-Safe Worker Input Validation

#node #observability #typescript

Reject a malformed metrics query response before its JSON can influence paging or rollback. TL;DR: parse the body once, validate the exact fields used by the decision, preserve a bounded diagnostic record, and turn invalid input into an unknown evaluation rather than healthy or failed. For a nightly structured-log pipeline, that distinction keeps a broken monitoring response from approving a bad run or waking someone for data that never existed.

The deciding constraint is rollback safety. A basic JSON.parse() plus optional chaining looks compact, but it merges transport success, JSON syntax, response shape, and query meaning into one optimistic path. The experiment here uses a stricter boundary: malformed or incomplete evidence cannot vote.

How should an alert worker parse a malformed metrics query response?

A response can be syntactically valid JSON and still be unusable. An HTML error page fails immediately, which is easy. Harder cases include {"status":"ok"}, a numeric field encoded as a string, a missing bucket, null, or a finite-looking envelope whose count becomes NaN after careless coercion. None proves that the nightly pipeline produced zero failures.

Zero and unknown are different operational facts. Treating both as falsy creates a dangerous failure mode: the alert worker may mark a run healthy because the evidence was absent. Treating every parser error as a pipeline failure is safer for the data but noisy for the operator. I would rather hold the previous deployment state, emit a separate monitor-health signal, and wait for a valid observation. That is a deliberate trade-off: rollback may be delayed, but a corrupt control input cannot trigger it.

Bad evidence gets no vote.

The useful unit of validation is the smallest claim the rollback rule consumes: a query identifier, a closed time window, and a finite non-negative failure count. Extra fields can change without breaking the worker. Required decision fields cannot.

Put one narrow boundary in front of the alert rule

The parser accepts text because the HTTP boundary is where syntax failures occur. It returns a discriminated union, so callers must handle invalid evidence explicitly. It also avoids coercion: Number("") is 0, but an empty string is not a measured count.

type QueryEvidence = {
  queryId: string;
  windowEnd: string;
  failureCount: number;
};

type ParseResult =
  | { ok: true; value: QueryEvidence }
  | { ok: false; reason: string };

function isRecord(value: unknown): value is Record<string, unknown> {
  return typeof value === "object" && value !== null && !Array.isArray(value);
}

function parseEvidence(body: string): ParseResult {
  let raw: unknown;
  try {
    raw = JSON.parse(body);
  } catch {
    return { ok: false, reason: "invalid_json" };
  }

  if (!isRecord(raw)) return { ok: false, reason: "invalid_root" };
  const { queryId, windowEnd, failureCount } = raw;

  if (typeof queryId !== "string" || queryId.length === 0) {
    return { ok: false, reason: "invalid_query_id" };
  }
  if (typeof windowEnd !== "string" || Number.isNaN(Date.parse(windowEnd))) {
    return { ok: false, reason: "invalid_window_end" };
  }
  if (typeof failureCount !== "number" ||
      !Number.isFinite(failureCount) ||
      failureCount < 0 || !Number.isInteger(failureCount)) {
    return { ok: false, reason: "invalid_failure_count" };
  }

  return { ok: true, value: { queryId, windowEnd, failureCount } };
}
Enter fullscreen mode Exit fullscreen mode

Keep the envelope small. A general-purpose schema layer can be justified when many workers share contracts, but this worker only needs three fields. Adding a dependency buys consistency and richer error details; it also expands upgrades, bundle size, and attachment to a validation API. The hand-written guard is dull, visible, and cheap to replace.

One subtle gap remains: Date.parse() establishes parseability, not that the window belongs to the run being evaluated. The decision layer must compare the returned timestamp with the expected nightly window. Parsing establishes shape. Context establishes relevance.

Make unknown a real state

A boolean cannot represent the control problem. Use three outcomes and keep notification policy separate from evidence parsing.

type Evaluation =
  | { state: "healthy"; count: number }
  | { state: "failed"; count: number }
  | { state: "unknown"; reason: string };

function evaluate(
  parsed: ParseResult,
  expectedQueryId: string,
  expectedWindowEnd: string,
): Evaluation {
  if (!parsed.ok) return { state: "unknown", reason: parsed.reason };

  const value = parsed.value;
  if (value.queryId !== expectedQueryId) {
    return { state: "unknown", reason: "query_mismatch" };
  }
  if (value.windowEnd !== expectedWindowEnd) {
    return { state: "unknown", reason: "window_mismatch" };
  }

  return value.failureCount > 0
    ? { state: "failed", count: value.failureCount }
    : { state: "healthy", count: 0 };
}
Enter fullscreen mode Exit fullscreen mode

Now the policies are plain: failed can stop promotion or invoke the established rollback path; healthy can allow the pipeline to advance; and unknown freezes the decision while reporting that the monitor lacks trustworthy evidence. No guesswork.

Do not page on every malformed poll. Group monitor-health events by stable causes such as invalid_json, window_mismatch, and invalid_failure_count, then page on persistence or exhaustion of a bounded retry policy. Event grouping systems commonly use stack traces, exception details, and fingerprints; a low-cardinality reason code gives the same class of failure a stable identity without embedding raw payloads. Preserve a request correlation identifier, response content type, byte length, and a redacted body sample under a strict size cap. Structured logs may contain secrets, so the full response should not become an error message by default.

Test the evidence path, not only the happy path

The focused experiment is a table of inputs and permitted state transitions. These cases matter more than a large snapshot of one successful response.

Input Parser result Rollback vote Diagnostic
Expected fields, count 0 valid healthy normal evaluation
Expected fields, count 4 valid failed normal evaluation
Truncated JSON invalid none invalid_json
JSON array invalid none invalid_root
Count "0" invalid none invalid_failure_count
Correct shape, stale window valid shape none window_mismatch
Correct shape, wrong query valid shape none query_mismatch

Property-based fuzzing is useful at this boundary, but a short deterministic corpus should stay in the repository. Include empty input, deeply nested irrelevant objects, huge numeric values, null, arrays, and strings where numbers belong. Cap response bytes before parsing so a valid but oversized document cannot monopolize worker memory. Set an HTTP deadline as well; defensive JSON handling does nothing for a connection that never completes.

Test the transition after a parser failure too. A later valid response must clear monitor-health degradation without erasing the earlier diagnostic, and a stale successful response must remain unknown. This is where rollback safety tends to leak: the parser is correct, while state retained from the previous poll silently casts a new vote.

Measure before adopting this boundary

Measure four things during a shadow run: invalid responses by reason, age of the newest valid window, time spent in unknown, and disagreement between the old boolean path and the three-state path. Also track payload bytes and retry volume. Log ingestion is commonly billed by data volume, so storing every raw malformed response is both a security liability and a cost that scales with failure frequency.

The acceptance criterion is not zero parser errors. Upstream systems and networks fail. The useful result is that no malformed, stale, or mismatched response changes deployment state; valid failures still trigger the established action; and repeated monitor-health failures become visible without creating one incident per poll.

Rollback automation should fail closed on evidence, not fail silent on observability. For a nightly log search, that means holding state until the worker can prove what window and query its count describes.

References

Top comments (0)