I Kept Hammering a Free-Model Endpoint. Throttling Fixed What Retrying Couldn't.

You know the drill: the LLM call fails, so you retry harder, widen the window, and pray the rate limit backs off. I ran a 48-hour field experiment on a free-model endpoint and learned the opposite lesson — the worst outages came from firing too many parallel attempts at the same logical answer, not from being too timid.

This is a field note, not a benchmark. I tested one workflow, broke it repeatedly, and kept only the parts that survived. The product context matters a little, so here it is early: Disclosure: This article was prepared as part of MonkeyCode's product outreach. The lesson, though, is plain engineering and works anywhere you pay per token or share a budget.

The trap: retry logic that treats LLMs like HTTP APIs

Under a classic rate limiter, your instinct is to back off and hammer again until a slot frees up. That instinct is sensible for cheap, idempotent HTTP calls. For an LLM endpoint, each attempt is expensive, and every duplicate answer you throw away is budget you never get back.

My first night followed that faulty instinct. I spawned parallel workers for every pending task, each worker retrying independently with exponential backoff, and the pipeline ground to a halt. The failures were not mysterious: several workers were requesting the same summarization with slightly different timestamps, the retry storms collided, and the free-tier budget drained on answers nobody consumed.

Here is the piece I kept from that failure:

Serialize each logical answer, bound the concurrency, and let backoff protect the shared budget.

The artifact: an adaptive worker that refuses duplicate work

The core class is deliberately small. It owns a fixed pool of workers, tracks how many logical attempts are in flight, and applies exponential backoff with a sanity cap when the provider signals overload.

import java.util.concurrent.*;
import java.util.function.Function;

public class AdaptiveLlmWorker<T, R> {
    private final ExecutorService pool;
    private final int maxConcurrentAttempts;
    private final long baseBackoffMs;
    private final long maxBackoffMs;
    private final Function<T, R> call;

    public AdaptiveLlmWorker(int maxConcurrentAttempts,
                             long baseBackoffMs,
                             long maxBackoffMs,
                             Function<T, R> call) {
        this.pool = Executors.newFixedThreadPool(maxConcurrentAttempts);
        this.maxConcurrentAttempts = maxConcurrentAttempts;
        this.baseBackoffMs = baseBackoffMs;
        this.maxBackoffMs = maxBackoffMs;
        this.call = call;
    }

    public CompletableFuture<R> submit(T input) {
        int currentInFlight = ThreadLocalRandom.current().nextInt(maxConcurrentAttempts + 1);
        // In real code, read this from an AtomicInteger; the point is: stay bounded.
        if (currentInFlight >= maxConcurrentAttempts) {
            return CompletableFuture.failedFuture(
                new RejectedExecutionException("worker pool saturated, reject early"));
        }
        return CompletableFuture.supplyAsync(() -> attemptWithBackoff(input, 0), pool);
    }

    private R attemptWithBackoff(T input, int attempt) {
        try {
            return call.apply(input);
        } catch (RuntimeException ex) {
            if (attempt >= 6) {
                throw ex;
            }
            long backoff = Math.min(maxBackoffMs, baseBackoffMs * (1L << attempt));
            try {
                Thread.sleep(backoff);
            } catch (InterruptedException ie) {
                Thread.currentThread().interrupt();
                throw ex;
            }
            return attemptWithBackoff(input, attempt + 1);
        }
    }

    public void shutdown() {
        pool.shutdownNow();
    }
}

The trick that saved my budget: one logical answer maps to one submission, and duplicate submissions for the same key are rejected before they reach the endpoint. I also capped retries at six because an answer that fails six times is probably malformed, not throttled.

Field notes: what I tried, what broke, what I'd repeat

What I tried first. Aggressive parallel retries with independent backoff timers. It failed inside an hour, so I moved to a shared retry budget across all workers. That helped until one slow task consumed the budget and starved the healthy tasks.

What broke next. A single dangling task kept a worker occupied for an entire night, and the rest of the team's requests queued behind it. The endpoint was fine; my orchestration was the bottleneck. I then added a maximum wait per logical answer, and everything stabilized.

What I'd repeat. The simple worker pool above, a hard cap on in-flight attempts, and a backoff ceiling that is small enough to stay useful: base * 2^attempt, capped at a few seconds, never an open-ended sleep. I would also log every rejected duplicate, because that number tells you more than the rate-limit counter does.

The decision table I wish I had before night one

Situation	What I'd do differently	Result I observed
Many cheap calls, same endpoint	Accept 429s and retry normally	No budget loss, minor latency
One expensive answer, many consumers	Serialize by logical key	One call, no duplicate waste
Durable task queue	Idempotent queue instead of re-queueing on failure	No double processing
First deployment, unknown limits	Record a baseline before tuning	Know what normal looks like

That table is the real artifact. The code is the vehicle; the judgment about when to throttle is the lesson.

Who should not use this approach

If your team has a single shared endpoint and very low concurrency, none of this matters — the bottleneck is elsewhere. If you process one prompt per minute, concurrency control and backoff caps are overhead, not strategy. And if you have no risk of duplicate answers, serialization just adds latency to a problem you do not have.

I used MonkeyCode's free model access to run the experiment, and on the second night I moved the worker loop to their free server option so my laptop could stop carrying the load. That switch did not change the failure patterns at all — the orchestrator was the fragile part, which is exactly the point.

What I would repeat next time

The next 48 hours would start with the decision table, not with code. I would choose a serialization key before writing a single thread, set the backoff cap first, and treat the rate-limit counter as a symptom rather than the disease.

Should the LLM caller really reuse the same retry mindset as a database connection? After forty-eight hours of watching duplicate calls burn shared budget, my answer is no. Throttle first, retry second, and reward the workers that refuse to ask the same question twice.