Burst Now, Starve Later: A Dual-Bucket Gate for Free-Tier Rate Limits

Your first hundred requests return 200. Then the next twenty all return 429. What actually changed?

Usually nothing in your code. You hit a second limit you never modeled. Free tiers rarely have one limit. They have at least two.

This post separates them, shows a small gate you can copy, and lists the claims worth testing on your own endpoint.

Two limits, not one

A rate limit governs speed. Requests per second, or per minute, with some burst capacity.

A budget limit governs volume. Requests per day, per month, or per account. It does not refill in seconds.

They fail the same way on the wire: HTTP 429. They need opposite responses.

• Rate exhausted: wait briefly, then continue.
• Budget exhausted: stop, cache, or reschedule for the next window.

Treating both as "retry with backoff" burns your budget while you sleep.

This distinction matters when you prototype on MonkeyCode's free model access and its free server option. You then have two independent limit surfaces behind one client.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Four claims worth checking

Claim 1: "A 429 means I sent too many requests per second"

Not necessarily. The status code is shared. RFC 9110 defines 429 as "Too Many Requests" without saying which dimension you exceeded. See RFC 9110 §15.5.29.

The corrected model: 429 means some counter is full. Your job is to find which one.

Claim 2: "Retry-After tells me when to retry"

It tells you a minimum delay, not a success guarantee. RFC 9110 allows either delay-seconds or an HTTP-date (§10.2.3).

Two consequences bite people:

1. A header value like 3600 usually points at a budget, not a burst.
2. A date value means your clock skew now matters. Parse both forms.

Claim 3: "Sleep one second and the window resets"

That is true for fixed windows and false for token buckets. Most real limiters are closer to a token bucket: tokens refill continuously, up to a cap.

So a burst of N requests can pass instantly, then nothing passes for a while. Your log looks like a cliff, not a ramp.

Claim 4: "My client-side rate setting matches the server"

Your client config is a guess until a response confirms it. Response headers, observed 429 timing, and the size of the burst that succeeded are the evidence.

If you cannot observe those, assume burst capacity is 1 and pace strictly.

Decision table

Symptom	Likely limit	Wrong fix	Reasonable fix
429 with Retry-After: 1, recovers	short-window rate	raise timeout	pace requests, honor the delay
429 with Retry-After: 3600 or a date	daily budget	exponential backoff	stop, cache, reschedule
Quiet 200s, then a cliff of 429	burst capacity	add retries	lower concurrency, add jitter
Slow 200s, no 429	queueing, not limiting	retry harder	measure in-flight count

The last row is a different problem. Slow responses without 429 usually mean queueing, which retries make worse.

The artifact: a two-counter gate

The gate below tracks a refilling token bucket and a fixed daily budget. It returns why it refused, because the two reasons need different caller behavior.

Illustrative code. It was not executed or benchmarked for this article.

# Illustrative only — not executed, not benchmarked.
import time
from dataclasses import dataclass, field

RATE_LIMITED = "rate"
BUDGET_GONE = "budget"

@dataclass
class DualGate:
    rate_per_s: float            # sustained refill; verify against provider docs
    burst: float                 # bucket capacity; assume 1.0 if unknown
    daily_budget: int            # your own hard cap, counted locally
    tokens: float = 0.0
    spent_today: int = 0
    last: float = field(default_factory=time.monotonic)

    def _refill(self, now: float) -> None:
        elapsed = now - self.last
        self.tokens = min(self.burst, self.tokens + elapsed * self.rate_per_s)
        self.last = now

    def acquire(self, cost: int = 1) -> tuple[bool, str | None, float]:
        """Return (allowed, reason, suggested_wait_seconds)."""
        now = time.monotonic()
        self._refill(now)

        if self.spent_today + cost > self.daily_budget:
            return False, BUDGET_GONE, float("inf")   # do not retry today

        if self.tokens < cost:
            wait = (cost - self.tokens) / self.rate_per_s if self.rate_per_s else float("inf")
            return False, RATE_LIMITED, wait

        self.tokens -= cost
        self.spent_today += cost
        return True, None, 0.0

The float("inf") sentinel is deliberate. It forces the caller to branch instead of sleeping forever.

Caller loop

# Illustrative only — not executed.
def guarded_call(gate: DualGate, send, max_wait: float = 2.0):
    while True:
        allowed, reason, wait = gate.acquire()
        if allowed:
            return send()
        if reason == BUDGET_GONE or wait > max_wait:
            raise RuntimeError(f"gate closed: {reason}, wait={wait}")
        time.sleep(wait)

Two properties matter here:

• A budget refusal becomes a fast, loud failure, not a slow one.
• A rate refusal becomes a short sleep, which is cheap.

The failure case to design for

If you run several processes, one in-memory gate per process multiplies your effective rate. Your aggregate burst becomes burst × processes.

Options: move the counters to shared storage with an atomic decrement, or divide rate_per_s and burst by the process count and accept the drift.

No gate can enforce a limit it does not own. This gate protects you from yourself.

How to verify on your endpoint

This is a plan, not a result. Run it, keep the log.

1. Send a burst ladder: 1, 2, 4, 8, 16, 32 parallel requests.
2. Record status code, Retry-After, and elapsed time per request.
3. Find the knee: the first ladder step where 429 appears.
4. Read the Retry-After units. Seconds or a date?
5. Repeat after a short pause. A refilled bucket recovers; a spent budget does not.
6. Log which of the two counters your gate thinks it hit, and compare.

If step 5 recovers instantly, you are rate limited. If it stays refused for hours, you found a budget.

Keep the ladder in CI as a manual job. Do not run it on a schedule against someone else's free tier.

Limitations and who should skip this

• Unknown limits. If the provider publishes nothing and returns no headers, every number here is a guess. Prefer conservative pacing.
• Multi-region or multi-key clients. The gate must key the counters by whatever the provider counts. Wrong key, wrong accounting.
• Latency-sensitive paths. A gate that sleeps adds tail latency. Reject fast and let the caller decide.
• Paid plans with negotiated limits. Contract terms beat client-side guesses. Do not build a gate around a documented SLA.
• Anything needing hard guarantees. A client-side gate is cooperative. It cannot promise fairness across other users.

The pattern is worth it when you have two limit shapes and one budget. It is overhead when you have one documented limit.

The corrected mental model

A 429 is a symptom. A budget is an exhaustion. A rate is a speed.

One counter cannot model both. So track two, name the refusal reason, and let the caller choose between waiting and stopping.

If you run the burst ladder on your own endpoint, keep the log. The shape of the knee tells you more than any header.

Which counter did your last 429 really come from?