Treat a 400 from health-check log ingestion as two separate questions: was the body valid JSON, and did the decoded event match the accepted schema? Preserve a small evidence envelope before either check runs. For a marketplace, that envelope should identify the seller account, probe, deployment, and usage bucket without retaining credentials or the full response body. This makes the incident reconstructable and keeps cost attribution attached to rejected work.
Short answer: capture byte count, a digest, receipt time, content type, and attribution keys at the ingestion edge; parse once; validate the resulting value; then return a stable rejection class. Do not repeatedly stringify and parse the event while debugging. That destroys the distinction between bad wire bytes and a valid object with the wrong fields.
The deciding constraint is evidence, not log volume. A rejected uptime result still represents a probe execution and an ingest attempt. If its seller and usage bucket disappear with the payload, the later cost report cannot explain the work.
How should malformed JSON health check log ingest failures be separated?
A log line can look convincing in a console and still fail at different boundaries. The sender may have produced invalid JSON bytes. Middleware may already have decoded the body, after which application code serializes a wrapper instead of the event. Or the JSON can be syntactically valid while timestamp, level, or service has the wrong type or representation for the receiver's schema.
Those are different failures. Keep them different.
No object exists yet.
Record a rejection stage: transport, json_parse, or schema. A parse result says nothing about missing fields because no trustworthy object exists yet. A schema result means parsing succeeded, so retain a compact list of field violations rather than another payload copy.
type Attribution = {
sellerAccountId: string;
probeId: string;
deploymentId: string;
usageBucket: string;
};
type HealthResult = {
timestamp: string;
level: "info" | "warn" | "error";
service: string;
status: "up" | "down";
latencyMs: number;
};
This is an application contract, not a universal logging schema. The health fields answer operational questions; the marketplace keys answer who caused the work and where it should be counted. Otherwise a schema cleanup can remove the only join key used by incident and usage records.
Put evidence capture ahead of decoding
The focused experiment uses one known-good event, one malformed body, and one valid object with an invalid level. The failed design sends all three through a generic logger after parsing. It can say something was rejected, but the malformed case has no dependable object and the schema case becomes a wall of validation output.
The chosen design computes bounded metadata from incoming bytes, then parses and validates once:
import { createHash } from "node:crypto";
type Result = { ok: true; value: HealthResult } |
{ ok: false; violations: Array<{ path: string; code: string }> };
export async function ingest(
body: Buffer,
attribution: Attribution,
validate: (value: unknown) => Result,
write: (event: Record<string, unknown>) => Promise<void>,
): Promise<{ status: 202 | 400; code: string }> {
const evidence = {
...attribution,
receivedAt: new Date().toISOString(),
byteLength: body.byteLength,
bodyDigest: createHash("sha256").update(body).digest("hex"),
};
let decoded: unknown;
try {
decoded = JSON.parse(body.toString("utf8"));
} catch {
await write({ ...evidence, stage: "json_parse" });
return { status: 400, code: "INVALID_JSON" };
}
const result = validate(decoded);
if (!result.ok) {
await write({ ...evidence, stage: "schema", violations: result.violations });
return { status: 400, code: "INVALID_HEALTH_RESULT" };
}
await write({ ...evidence, stage: "accepted", event: result.value });
return { status: 202, code: "ACCEPTED" };
}
The digest supports correlation without making the rejected body the primary evidence record. It does not replace a security and retention review. Probe responses can contain tokens, customer URLs, or snippets that do not belong in ordinary logs.
The evidence write is awaited. Accepting or rejecting before evidence is durable creates an unexplained gap. A deployment may instead use a durable internal queue, but the decision should reflect whether that handoff succeeded. This adds latency. It buys a clearer boundary.
Make schema errors useful without echoing the payload
Record paths such as level, bounded machine-readable codes, and the schema revision. Avoid rejected values by default. Use an allowlist for diagnostic fields, cap violations, and count truncation. Stable codes also keep retry behavior independent of prose.
For schema evolution, deploy readers before writers when adding an optional field. Fowler's feature-toggle guidance separates release decisions from deployment and describes toggle-management costs. A narrow toggle can compare the next validator's classification without changing responses; remove it after migration.
Do not retry identical malformed bytes. Retry only a failure class explicitly defined as transient; keep parse and schema rejections terminal until the payload changes.
Testing the boundary matters just as much as choosing it.
Exercise truncated JSON, trailing content, arrays where an object is required, missing attribution, invalid timestamps, unknown levels, negative latency, and oversized bodies. Assert response code, rejection stage, digest stability, and every attribution key, not a giant formatted error string.
Then use known synthetic probe IDs in deployment checks. Accepted, malformed, and schema-invalid fixtures should reach their respective streams with the same seller and usage bucket. Custom log outputs have lifecycle and error-handling responsibilities, as the Logback appender documentation illustrates. Although this implementation is Node.js, test initialization, writes, shutdown, and sink failure.
Rollback deserves a test. Disabling the new validator should restore the prior classification without losing the schema revision behind each decision.
What should be measured before copying this design?
Measure rejections by stage, schema revision, deployment, seller account, and usage bucket. Track evidence-write failure and latency separately from validation. Keep digests and probe IDs in traceable records rather than high-cardinality metric labels.
Compare three counts: probe executions, ingest attempts, and accepted results. Their gaps describe different work. Cost attribution should follow execution and attempt records, not only accepted logs, or malformed traffic disappears from the ledger.
This design has a real limitation: awaiting evidence storage adds latency and makes sink availability part of the ingest path. It is not suitable when probe throughput matters more than reconstructing each rejection. In that case, use bounded counters and sampled diagnostics, accept the missing per-attempt trail, and keep cost attribution at the probe scheduler instead. The trade-off is explicit: less storage pressure and weaker incident evidence.
Use the synchronous architecture when rejected events matter enough to customer support, billing review, or incident reconstruction to justify that evidence handoff. For a marketplace dispute, defend the attribution envelope. The full raw body usually is not worth the exposure.
Top comments (0)