DEV Community: Joshua Hernandez

The HTTP 429 That Turned Seven Minutes Into Zero Work

Joshua Hernandez — Sun, 23 Aug 2026 07:53:42 +0000

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

ARGUS is a data-catalog governance swarm. Its agents use structured model calls to draft descriptions, classify sensitive fields, and review repairs before anything can be written.

During a 45-entity sweep, the system spent seven minutes producing zero proposals. The report called the result review failed.

The endpoint was reachable. Health probes returned 200. The logs showed only:

429 Too Many Requests

The retry loop was working exactly as written. That was the bug.

Two different failures shared one status code

The model provider enforced both a short burst limit and a daily token quota. Both failures arrived as HTTP 429 with the same useful headers.

Their remedies are opposites:

Condition	Correct response
Per-minute burst limit	Wait, then retry
Daily token quota exhausted	Stop using that endpoint for this sweep

The only reliable distinction was in the response body. The provider's daily-quota response explained:

on tokens per day (TPD): Limit 100000, Used 99299, Requested 1139

ARGUS never logged that text. The OpenAI-compatible client raised an httpx status exception whose default message contained the status and URL, but not the response body.

The information needed to fix the failure was present on the wire and discarded at the error boundary.

Why the health probe lied

The account was close to its cap, not necessarily at exactly zero remaining tokens. A tiny probe could still fit under the allowance and return 200.

The real structured request included a schema, column context, lineage context, and reserved output. It needed more than the remaining budget and failed.

That created a misleading picture:

connectivity looked healthy
authentication looked healthy
small calls looked healthy
every useful call failed

A probe only proves that the probe can run. It does not prove that the real workload fits inside the remaining quota.

The retry loop amplified the outage

ARGUS treated every 429 as temporary. Each model call could wait through six backoffs. A sweep could need roughly one hundred calls.

For a rate limit that clears in seconds, this is patient behavior. For a daily quota that resets tomorrow, it turns one deterministic failure into hours of waiting.

The retry policy had no concept of recoverability. It knew that a request was throttled, but not whether time inside the current operation could change the result.

Fix one: preserve the provider's explanation

The HTTP boundary now reads and carries a bounded copy of the response body when it raises an error.

Instead of this:

429 Too Many Requests

the log can say which limit was hit, how much was used, and what the failed request needed.

That single change identified the root cause on the next run.

The broader lesson is simple: an error abstraction should hide irrelevant transport details, but it must not erase the only field that distinguishes two operational states.

Fix two: model exhaustion as its own outcome

I added a BudgetExhausted exception rather than routing a daily quota through the ordinary retry path.

When the response identifies a long-lived token or request quota:

The call fails immediately.
The endpoint is retired for the current sweep.
A breaker prevents the remaining agents from repeating the same doomed call.
The breaker resets at the start of the next sweep.
The CLI labels the totals as partial.

The reset matters. A process-wide flag that survives its cause would make tomorrow's healthy quota look permanently dead.

Fix three: do not turn an outage into a verdict

The most important change was not in the HTTP client. It was in the Arbiter.

ARGUS already had a deliberate no-model mode. If the operator starts a run without a key, mechanical grounding becomes the declared review gate. A proposal that passes that gate can be approved.

Budget exhaustion halfway through a modeled run is different. Falling into the same branch would silently remove the review gate at the exact moment it stopped working.

The new outcome is:

not reviewed: model budget exhausted

It is not rejected, because no reviewer made a judgment. It is not approved, because the expected review never happened. Unreviewed proposals are never written.

This distinction protects both the data catalog and the truthfulness of the report.

Fix four: pace and rotate

The provider reports remaining tokens and reset timing on successful responses. ARGUS now uses those headers to pace before hitting the wall instead of discovering the wall by repeatedly crashing into it.

I also reduced reserved output from 4096 tokens to 900. The structured responses are short descriptions, tag lists, and verdicts. On free tiers, reserving 4096 tokens can consume rate-limit capacity even when the reply uses far less.

When multiple endpoints are configured, the retry and rotation loops are now separate:

retry when the same endpoint may answer next time
rotate when the endpoint will not answer during this sweep
do not rotate on unrelated errors such as malformed output or HTTP 500
trip the breaker only when every configured endpoint is unavailable

That last negative test matters. Rotating on every error would burn the entire provider chain because of one bad prompt.

Duration strings were another hidden bug

The original reset parser stripped a trailing s and called float().

Real providers return values such as:

1h30m43.2s
205ms

Both values failed the old parser and silently fell back to guessed delays. The replacement parser handles hours, minutes, seconds, milliseconds, decimals, and compound durations.

Before and after

Behavior	Before	After
Daily quota	Six backoffs per call	One classified failure
Provider explanation	Discarded	Preserved and logged
Remaining calls	Repeated the same failure	Short-circuited or rotated
Proposal outcome	`review failed`	`not reviewed`
Writes after exhaustion	Ambiguous fallback risk	Never written
Sweep totals	Looked complete	Explicitly partial
Reset header parsing	Simple float only	Compound duration support
Provider fallback	Stopped on exhausted endpoint	Rotates after bounded retries

The real exhausted endpoint that previously took seven minutes now produced a clear partial-run report in seconds and committed nothing.

The regression suite grew dedicated coverage for:

daily quota versus temporary rate limit
no retry after confirmed exhaustion
breaker reset between sweeps
no automatic approval after mid-run exhaustion
unreviewed proposals never reaching a write
duration parsing for 1h30m43.2s and 205ms
rotation after persistent unnamed throttling
no rotation on non-throttle failures
stopping clearly when every endpoint is spent

What I learned

Status codes are categories, not diagnoses

HTTP 429 says the request cannot run now. It does not say whether waiting inside the current job can help.

Observability must preserve decision-making evidence

Logging more bytes is not automatically useful. Logging the one bounded provider field that distinguishes retryable from terminal failure is.

Degraded mode and outage mode are different contracts

Starting without a reviewer is an explicit operating mode. Losing the reviewer halfway through is a failed assumption. They should not share a fallback branch merely because both lack a model response.

A partial success must call itself partial

The sweep produced real numbers for fewer entities than requested. Presenting those totals without qualification would overstate coverage even if every individual number were correct.

The best part of this fix is that it did not merely reduce latency. It stopped an infrastructure outage from being misreported as an AI judgment and preserved the review boundary when the budget disappeared.

AI assistance was used during code and draft iteration. Joshua Hernandez is the sole contest entrant and is responsible for the submitted implementation and evidence.

When a Safety Reviewer Rejected Everything and Still Passed Its Test

Joshua Hernandez — Sun, 23 Aug 2026 07:50:12 +0000

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

ARGUS is a data-catalog governance swarm. Specialist agents find missing descriptions, untagged sensitive columns, broken lineage assumptions, and other metadata defects. An Arbiter reviews each proposed repair before anything can be written.

That last reviewer is supposed to be the safety boundary. It must reject unsupported claims without blocking repairs that the evidence actually supports.

One live sweep exposed a failure that looked exactly like diligence.

The Arbiter rejected 86 of 112 proposals. Among the rejected repairs were classifications such as:

cust_first_name is PII
billing_zipcode is PII
shipping_address_line1 is PII

PII governance finished at 0.0 percent. The flagship capability had produced nothing, but the report was full of confident explanations about why each repair was unsafe.

The reviewer had not become safer. It had learned to refuse responsibility.

Why the prompt caused the failure

The original prompt explained what the Arbiter must reject. It said that schema and recorded lineage were established evidence and warned against inventing ownership, row counts, refresh cadence, business meaning, and downstream consumers.

What it did not explain was the difference between interpreting evidence and claiming a new fact about the world.

A strong model filled that gap with common sense. A weaker model followed the instructions literally and demanded outside corroboration that a column named cust_first_name contained a first name.

That meant the same prompt appeared to work until the swarm rotated to another model family. The model change did not create the ambiguity. It revealed ambiguity that had always been present.

The test rewarded the failure

The more embarrassing bug was in my regression test.

The live adversarial check had four cases:

Expected decision	Cases
Approve	1
Reject	3

A reviewer that rejected every proposal scored 3 out of 4, or 75 percent.

That number looked close enough to healthy during a quick review. The test designed to catch a broken reviewer was giving partial credit for the exact failure mode that broke it.

This is a common problem in safety-oriented systems. If negative cases dominate the benchmark, a component can look accurate by always choosing the conservative label. The aggregate score hides whether errors are false approvals or false rejections, even though those failures have opposite causes and opposite remedies.

Fix one: define three kinds of claims

I rewrote the Arbiter instructions around three explicit categories.

1. Established by the evidence

The schema establishes which columns exist and their types. Recorded lineage establishes the listed upstream and downstream relationships. A description that restates those facts can be approved.

2. Interpretation of the evidence

This is the judgment the Arbiter is being asked to make.

A column named cust_first_name can reasonably be classified as personal data. billing_zipcode is a postal code associated with a customer record. order_id identifies an order.

The reviewer does not need an external document to agree that a clear name means what it says. It should reject the interpretation only when the name is genuinely ambiguous. For example, region is not automatically PII, and account_number on a warehouse table may not identify a person.

3. Claims about the world

Ownership, refresh cadence, row counts, trustworthiness, and an unrecorded downstream consumer are not established by a schema. Those claims still require evidence and should be rejected when it is absent.

The new prompt also states the cost of both error directions:

Approving a fabrication puts a lie into a system people rely on. Rejecting a correct repair leaves the catalog broken and a sensitive column ungoverned.

Rejection is no longer described as the safe default. It is a decision with consequences.

Fix two: make the test discriminate

I replaced the four-case check with nine balanced cases.

Five must be approved:

A description grounded in the schema and real lineage
A description that only restates the columns
Tag cust_first_name as PII
Tag billing_zipcode as PII
Tag email as PII

Four must be rejected:

Tag the generic region column as PII
Invent a refresh cadence and row count
Invent ownership and data quality
Invent a downstream consumer absent from lineage

The false-positive trap matters. A prompt that approves every PII proposal is just as broken as one that rejects every proposal.

The harness now reports three outcomes separately:

FALSE APPROVAL
Fabrication would reach the catalog.

FALSE REJECTION
A correct repair was refused.

ERROR
No judgment was produced.

It no longer compresses opposite failure modes into one flattering percentage.

Before and after

Behavior	Before	After
Clear schema-derived description	Often rejected	Approved
Obvious PII column	Often rejected	Approved
Generic `region` column tagged as PII	Rejected	Rejected
Invented ownership or cadence	Rejected	Rejected
Test mix	1 approve, 3 reject	5 approve, 4 reject
Error reporting	One aggregate score	False approvals and false rejections separated

After the change, the nine-case harness scored 9 out of 9 on two model families that had not seen the revised prompt: gpt-4o-mini through GitHub Models and gemma-4-26b through OpenRouter.

Most importantly, both models discriminated in both directions. They approved the grounded PII repairs, rejected the fabricated world claims, and preserved the region false-positive trap.

What I learned

A conservative classifier can still be unsafe

Inaction has a cost. Refusing to tag an obvious sensitive column leaves the catalog ungoverned. A safety system must model the harm of false negatives, not only the harm of false positives.

Benchmarks encode incentives

A reject-heavy test suite rewarded rejection. Balancing labels helped, but separating error types was the real fix because it made the failure direction visible.

Model rotation is an adversarial test

Changing providers exposed an underspecified instruction that a stronger model had been silently repairing. Cross-model evaluation is useful even when the production system normally uses one provider.

A green report is not proof of useful work

The broken sweep completed, produced explanations, and looked cautious. The business outcome, 0.0 percent PII governance, was the signal that mattered.

What I am proud of

The fix was not "use a stronger model." It made the contract clearer and the regression test harder to game. That improved the behavior across two different model families and turned a vague reviewer into a component with measurable discrimination.

The most valuable debugging question was not "why did it reject this proposal?" It was "how did rejecting everything still look like success?"

AI assistance was used during code and draft iteration. Joshua Hernandez is the sole contest entrant and is responsible for the submitted implementation and evidence.

How to Add a Human Review Gate to an n8n Lead Intake Workflow

Joshua Hernandez — Sun, 23 Aug 2026 07:43:42 +0000

Most lead intake automations work on the happy path. A form arrives, the workflow sends an email, and the lead appears in a CRM.

The expensive failures happen at the edges:

the same webhook is delivered twice
a required field is missing
a high-risk message is routed automatically
an API accepts the request but drops a field
a downstream system fails after the workflow has already sent a confirmation

A reliable workflow needs more than connected nodes. It needs explicit states, a review boundary, and tests that prove what happens when something goes wrong.

The workflow shape

A practical review-gated intake flow has seven stages:

Receive the event
Normalize the fields
Create a deterministic duplicate key
Validate and classify the record
Pause for review when the record is risky or incomplete
Send only after approval
Record the final state and any failure details

The important design choice is that "received" and "approved" are different states. Receiving a lead should never imply that a person has accepted it or that an appointment exists.

A normalized record can look like this:

{
  "lead_id": "form|alex@example.com|2026-08-23",
  "source": "website_form",
  "name": "Alex",
  "email": "alex@example.com",
  "request_type": "service",
  "urgency": "routine",
  "review_status": "pending",
  "delivery_status": "not_sent"
}

This gives every later node a predictable contract.

Normalize before branching

Do not scatter cleanup logic across five nodes. Normalize once, near the start.

An n8n Code node can create a simple deterministic duplicate key:

const email = String($json.email || "")
  .trim()
  .toLowerCase();

const receivedAt = new Date($json.received_at);
const day = receivedAt.toISOString().slice(0, 10);

return [{
  json: {
    ...$json,
    email,
    dedupe_key: `${$json.source || "unknown"}|${email}|${day}`,
    review_status: "pending",
    delivery_status: "not_sent"
  }
}];

The exact key depends on the business. A contact form might use source, normalized email, and day. An order system should normally use the provider's event or order ID.

The rule is simple: retries for the same logical event must produce the same key.

Make the review gate explicit

A review gate does not need a complicated dashboard.

One useful n8n pattern is:

store the normalized record
route risky or incomplete records to a Wait node
send a reviewer a one-time decision link or internal form
resume the workflow with an explicit decision
branch on approved, rejected, or needs_changes

Keep the decision payload small:

{
  "lead_id": "form|alex@example.com|2026-08-23",
  "decision": "approved",
  "reviewed_by": "operations",
  "reviewed_at": "2026-08-23T18:14:00Z"
}

Do not let arbitrary fields from the review request overwrite the stored lead. Load the stored record by lead_id, validate the allowed decision values, and update only the review fields.

Send after approval, not before

The delivery branch should check both conditions:

review_status == approved
delivery_status == not_sent

After the downstream API or email succeeds, write the provider message ID and set delivery_status to sent.

If the workflow retries, the second run sees sent and does not send again. This is the small piece of state that prevents duplicate customer messages.

Log failures as states

A generic "workflow failed" alert is rarely enough to repair the problem.

Capture:

lead ID
workflow execution ID
stage that failed
normalized error category
retryable or permanent
downstream status code
provider request ID
timestamp

Avoid putting API keys, full authorization headers, or unnecessary customer data in logs.

A retryable timeout and a permanent validation error should not share the same path. Timeouts can enter a bounded retry queue. Invalid records should return to review with a specific reason.

Five tests worth running

Before connecting the workflow to production data, test these cases with synthetic records:

A valid approved lead is sent exactly once.
A duplicate event does not create a second send.
A missing required field enters review and does not send.
A rejected record never reaches the delivery node.
A downstream timeout creates a retryable failure record with no secret data.

Also test the recovery path. Fix the simulated downstream failure and confirm that the record can resume without duplicating earlier side effects.

A working reference

I published a tested n8n lead-intake implementation with validation, duplicate protection, human review, failure paths, and automated checks:

View the workflow and tests on GitHub

If you want to adapt the pattern to an existing form, webhook, email tool, or CRM, Blaz Algo Systems offers fixed-scope automation and reliability work.

There is also a downloadable Python API Reliability Kit for teams that want reusable validation and debugging material.