Short answer: repair a background job queue with malformed JSON, parse errors, or a 256kb payload limit by migrating its producer and consumer contract in stages. Add a versioned, reference-sized envelope; deploy dual-reading consumers; observe old traffic; then switch producers. For a marketplace draining a rate-limited worker pool, that order preserves delivery guarantees while bad messages are classified outside scarce business-work capacity.
| Migration path | Delivery posture | Operational fit | Catch |
|---|---|---|---|
| Stop-the-world replacement | One contract exists at a time | A queue can be fully paused and drained | Pausing may violate freshness targets |
| Dual-read, single-write | Old deliveries remain readable during the switch | Producers and consumers deploy independently | Compatibility code lives for one migration window |
| Consumer-only repair | Existing producers keep publishing unchanged | Producer ownership is unavailable | Oversized work still reaches queue admission |
Use dual-read, single-write here. Delivery continuity beats having the fewest branches. The runner-up is a full pause: choose it when the marketplace can stop producers, drain every old delivery, verify zero in-flight work, and start cleanly on the new contract. Consumer-only repair is a containment step, not the destination, because it can't prevent a message-too-large rejection before delivery.
The migration unit is an envelope version
Don't begin by raising a byte limit. Begin by naming the old and new contracts.
The old marketplace job may contain a whole listing snapshot. The new job carries only version, marketplaceId, listingId, and an immutable sourceKey. The source data moves to storage whose lifetime covers retry and replay. This change separates transport admission from business input size, but it introduces a real obligation: the object referenced by sourceKey must not silently change between duplicate deliveries.
Versioning is what makes that obligation deployable. A consumer can explicitly accept v1 and v2 instead of guessing a shape from whichever fields happen to exist. An unknown version is a permanent contract outcome. Invalid JSON is also permanent for those exact bytes. HTTP 429 from the downstream marketplace is different; time can change that result, so it belongs on a bounded delayed-retry path.
Those distinctions protect the rate-limited pool. Parsing, size checks, and schema validation happen before acquiring a business-work permit. Only a valid job may spend the permit on fetching its source and calling the marketplace. Acknowledgement comes after the durable business transition or idempotent completion marker, not after parsing. RabbitMQ's documentation describes consumer acknowledgements as confirmation that a delivery was handled and negative acknowledgements as a way to requeue or discard it. The transport controls vary elsewhere, but the ordering requirement remains useful.
How can a background job queue migrate malformed 256kb JSON payloads safely?
First, give both sides the same byte rule without giving them the same deployment clock. In this system, 256kb is the configured message budget; it isn't presented as a universal broker maximum. The code reserves 4 KB for transport metadata and measures the serialized UTF-8 bytes once.
const MESSAGE_BUDGET_BYTES = 256 * 1024;
const TRANSPORT_RESERVE_BYTES = 4 * 1024;
const BODY_LIMIT_BYTES = MESSAGE_BUDGET_BYTES - TRANSPORT_RESERVE_BYTES;
type JobV2 = {
version: 2;
marketplaceId: string;
listingId: string;
sourceKey: string;
};
function encodeV2(job: JobV2): Buffer {
const body = Buffer.from(JSON.stringify(job), "utf8");
if (body.byteLength > BODY_LIMIT_BYTES) {
throw new Error("payload_too_large");
}
return body;
}
That's the entire producer admission path.
The consumer needs a decoder that returns data or a stable reason, never a half-trusted object. A real v1 validator depends on the existing contract; the example models it as an injected function rather than inventing fields that aren't known here. The v2 path is concrete.
type JobV1 = { version: 1 } & Record<string, unknown>;
type AcceptedJob = JobV1 | JobV2;
type DecodeResult =
| { ok: true; job: AcceptedJob }
| {
ok: false;
reason:
| "payload_too_large"
| "invalid_json"
| "unsupported_version"
| "invalid_schema";
};
function isJobV2(value: unknown): value is JobV2 {
if (typeof value !== "object" || value === null) return false;
const job = value as Record<string, unknown>;
return (
job.version === 2 &&
typeof job.marketplaceId === "string" &&
job.marketplaceId.length > 0 &&
typeof job.listingId === "string" &&
job.listingId.length > 0 &&
typeof job.sourceKey === "string" &&
job.sourceKey.length > 0
);
}
function decodeJob(
body: Buffer,
isJobV1: (value: unknown) => value is JobV1,
): DecodeResult {
if (body.byteLength > BODY_LIMIT_BYTES) {
return { ok: false, reason: "payload_too_large" };
}
let value: unknown;
try {
value = JSON.parse(body.toString("utf8"));
} catch {
return { ok: false, reason: "invalid_json" };
}
if (isJobV1(value) || isJobV2(value)) return { ok: true, job: value };
const version =
typeof value === "object" && value !== null
? (value as Record<string, unknown>).version
: undefined;
return version === 1 || version === 2
? { ok: false, reason: "invalid_schema" }
: { ok: false, reason: "unsupported_version" };
}
I benchmark this boundary by branches and calls: one byte measurement, one parse, one version dispatch, then zero business calls for rejected input. That makes time-to-diagnosis much better than a generic exception handler. It also keeps config bloat in check; the byte budget and accepted versions should be shared contract values under source control, not four unrelated settings in a producer, consumer, replay CLI, and dashboard.
How does Phase A observe both contracts before writers change?
Deploy the dual reader first. Keep producers on v1. For every received delivery, record its byte band, decoded version, validation outcome, and disposition, but never the full marketplace payload. Reason codes such as invalid_json, invalid_schema, unsupported_version, and payload_too_large are enough for aggregation. Copying listing or seller data into diagnostic logs creates another retention surface and makes erasure work under GDPR Article 17 harder to reason about.
Shadow validation needs an expiry condition. Run it until the observation window includes the slowest expected retry and any replay procedure the team actually uses. I'm not sure a fixed number of hours is defensible without those local timings. What matters is seeing all ingress paths: each producer version, the replay CLI, delayed delivery, and an operator-triggered redrive.
The fixture suite should mirror those paths in painful detail. Send valid v1, valid v2, malformed bytes, an unknown version, a v2 object missing sourceKey, a multibyte value, a body exactly at BODY_LIMIT_BYTES, and a body one byte beyond it. Feed each case to producer admission where applicable and to the consumer decoder. Next, deliver the same valid identity twice; then repeat with the first business call complete but its acknowledgement absent. Assert one durable marketplace transition. Finally, return HTTP 429, assert that the identity isn't marked complete, advance the test clock to its next eligible time, and deliver again. The result table must say admission rejection, quarantine, delayed retry, or completion. “Failed” is useless.
Short version: observe every entrance.
During this phase, a permanent contract failure is recorded in quarantine metadata and acknowledged so it cannot cycle through the worker pool. A transient rate-limit response is delayed without a completion marker. Track oldest ready age, oldest retry-wait age, invalid input by reason and version, quarantine age, in-flight duration, and completed identities. Counts explain volume. Ages reveal a stuck drain.
Phase B moves writers only after readers
Once every live consumer accepts v2, switch one producer cohort to v2. The source object must be written before its queue reference is published, and its retention must cover the entire retry and replay window. Keep a stable business identity across redelivery so the consumer can detect a completed transition before calling the marketplace again.
Wait before expanding the cohort. Compare v1 and v2 completion age, quarantine reasons, source-fetch outcomes, and downstream 429 rates. This is a delivery comparison, not a speed contest: v2 wins only if it keeps valid work replayable and prevents malformed or oversized input from consuming permits. A smaller message is nice. It isn't proof.
Then move the remaining producers. Continue dual reading until no v1 message can remain in ready, delayed, in-flight, quarantine-replay, or operator-held paths. Deleting v1 support when the visible queue reaches zero is risky because visible depth doesn't account for every delivery state.
The reference design is not suitable when the input cannot live outside the queue for the required window, or when an added storage fetch would violate the job's latency budget. Stick with inline jobs when the encoded record remains comfortably inside the enforced budget and queue retention is the desired data lifetime. Choose chunking only when partitions are independently idempotent; splitting one indivisible mutation creates partial-completion semantics that the business must own.
Rollback preserves old deliveries
Rollback is the reason for single-write. If v2 signals breach an abort threshold, return producers to v1 while consumers continue to read both. No queued v2 delivery becomes unreadable. Investigate by reason code and job identity, fix the contract or its deployment configuration, then restart the cohort.
Do not roll back by disabling validation. That converts explicit quarantine into unpredictable business behavior and can let poison work retake the rate-limited pool. The safe rollback changes what is emitted while preserving what can be consumed.
The migration ends only when producers emit v2, all possible v1 deliveries have aged out or completed, quarantine has a tested replay path, duplicate delivery produces one durable transition, and operators can explain the oldest ready and retry-wait jobs without opening payload bodies. Remove v1 code after those conditions hold. Until then, a few extra decoder branches are cheaper than a broken delivery guarantee.
Top comments (0)