When Not to Let a Free Model Tune Outbound Caps

A reconstructed incident starts on a quiet Tuesday night. The partner email arrived at 02:14 UTC sharp. An internal agent had issued a long GET storm.

The limiter lived inside a prompt, not config. A free model had raised burst during a refactor. The shared partner key sat close to revocation.

This field guide states when not to take that path. Outbound quotas stay under reviewed human-owned policy files. Limiter state also stays off fragile free servers.

The failure mode

Agent tool calls closely resemble ordinary HTTP clients. They still are not ordinary interactive user traffic. One prompt can fan out into hundreds of retries.

Free models often guess numbers that look reasonable. They do not hold the signed partner contract. They also miss last week's 429 error graph.

A reconstructed chain looks like the list below.

1. A developer asks a model to relax limits.
2. The model raises burst from ten to two hundred.
3. Staging stays quiet with tiny local fixtures.
4. Production traffic hits a shared partner tenant.
5. The partner replies with a key-revocation warning.

The HTTP client is not the core bug. Ownership of the quota ledger is the bug.

Red flags

The free lane stops when any item holds.

• The number is a contractual RPS or daily cap.
• The window maps onto a billed partner plan.
• Burst size protects a shared tenant or key.
• Retry storms can multiply one user action.
• Limiter state must survive process restarts.
• A wrong raise can revoke a production credential.
• The policy feeds a compliance or audit log.
• Several services share one outbound identity.

Those items are hard stops for runtime values. Drafting review comments remains acceptable in pull requests. Shipping the numeric policy is not acceptable here.

Better alternatives

Teams pin the policy in versioned config files. They review that file like a schema change. They derive the rate-limit key from stable facts.

A store must outlive a single virtual machine. The client fails closed when that store is down. Retries sit on a separate and smaller budget.

A practical split looks like this list.

• Policy file: human-owned RPS, burst, and window.
• Key material: caller id, route, and partner name.
• State store: Redis or another durable bucket.
• Drafting lane: comments, tests, and staging fixtures.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode provides free model access and a free server option.

Those lanes help draft tests and rehearse stub clients. They do not own the production ledger here.

Artifact: a pinned quota policy

The TypeScript below is a small reproducible sketch. It is not a vendor benchmark or load run. Teams must load real caps from their own contracts.

Policy file

{
  "partner": "maps-api",
  "identity": "prod-outbound-maps",
  "windowSeconds": 60,
  "maxRequestsPerWindow": 30,
  "burst": 5,
  "failMode": "closed",
  "owner": "platform-api",
  "reviewers": ["platform-api", "security"],
  "source": "contract-2026-q3"
}

source must name a human document. It must not name a chat transcript.

Limiter module

import { createHash } from "node:crypto";

export type QuotaPolicy = {
  partner: string;
  identity: string;
  windowSeconds: number;
  maxRequestsPerWindow: number;
  burst: number;
  failMode: "closed" | "open";
  owner: string;
  source: string;
};

export type Clock = { now(): number };

export function assertPinnedPolicy(p: QuotaPolicy): void {
  const ok =
    p.source.startsWith("contract-") || p.source.startsWith("runbook-");
  if (!ok) {
    throw new Error("policy source must be a human document id");
  }
  if (p.maxRequestsPerWindow <= 0 || p.burst <= 0) {
    throw new Error("quota numbers must be positive");
  }
  if (p.burst > p.maxRequestsPerWindow) {
    throw new Error("burst cannot exceed the window cap");
  }
  if (p.failMode !== "closed") {
    throw new Error("partner caps must fail closed");
  }
  if (p.source.includes("prompt") || p.source.includes("model")) {
    throw new Error("model output cannot own the quota policy");
  }
}

export function rateLimitKey(input: {
  identity: string;
  route: string;
  caller: string;
}): string {
  const material = `${input.identity}|${input.route}|${input.caller}`;
  return createHash("sha256").update(material).digest("hex");
}

export async function takeToken(opts: {
  store: Map<string, { count: number; resetAt: number }>;
  policy: QuotaPolicy;
  key: string;
  clock: Clock;
}): Promise<"allow" | "deny"> {
  assertPinnedPolicy(opts.policy);
  const now = opts.clock.now();
  const current = opts.store.get(opts.key);

  if (!current || now >= current.resetAt) {
    opts.store.set(opts.key, {
      count: 1,
      resetAt: now + opts.policy.windowSeconds * 1000,
    });
    return "allow";
  }

  if (current.count >= opts.policy.maxRequestsPerWindow) {
    return "deny";
  }

  current.count += 1;
  return "allow";
}

The in-memory Map is for tests only. Production code needs a shared durable store.

Tests that reject a model-shaped patch

import assert from "node:assert/strict";
import {
  assertPinnedPolicy,
  rateLimitKey,
  takeToken,
  type QuotaPolicy,
} from "./rate-limit.ts";

const base: QuotaPolicy = {
  partner: "maps-api",
  identity: "prod-outbound-maps",
  windowSeconds: 60,
  maxRequestsPerWindow: 30,
  burst: 5,
  failMode: "closed",
  owner: "platform-api",
  source: "contract-2026-q3",
};

function clockAt(now: number): { now(): number } {
  return { now: () => now };
}

const modelShaped: QuotaPolicy = {
  ...base,
  maxRequestsPerWindow: 5000,
  burst: 2000,
  source: "prompt-session-88",
};

assert.throws(() => assertPinnedPolicy(modelShaped));

const failOpen: QuotaPolicy = { ...base, failMode: "open" };
assert.throws(() => assertPinnedPolicy(failOpen));

const store = new Map();
const key = rateLimitKey({
  identity: base.identity,
  route: "GET /geocode",
  caller: "agent-worker-3",
});

for (let i = 0; i < 30; i++) {
  const r = await takeToken({
    store,
    policy: base,
    key,
    clock: clockAt(1_000),
  });
  assert.equal(r, "allow");
}

const denied = await takeToken({
  store,
  policy: base,
  key,
  clock: clockAt(1_000),
});
assert.equal(denied, "deny");

Run the sketch with Node 20 or later.

node --experimental-strip-types --test rate-limit.test.ts

The test does one primary job well. It proves the ledger ignores prompt-shaped policy sources.

Retries multiply a bad cap

A single user turn may call one tool. The client then retries on 502 three times. Four HTTP calls then hit the partner quota.

Free models often add retries for extra resilience. They rarely subtract that factor from the cap. The ledger then lies about the remaining budget.

Teams pin retry count beside the quota file. They count worst-case calls per user turn. The window cap must survive that product.

{
  "maxRetries": 2,
  "retryOn": ["502", "503"],
  "retryBudgetSource": "runbook-2026-09"
}

The retry file follows the same source rule. Prompt-owned retry counts are also rejected outright.

A tiny checker belongs in CI as well.

export function worstCaseCalls(opts: {
  toolsPerTurn: number;
  maxRetries: number;
}): number {
  return opts.toolsPerTurn * (1 + opts.maxRetries);
}

export function capSurvivesTurn(opts: {
  maxRequestsPerWindow: number;
  toolsPerTurn: number;
  maxRetries: number;
}): boolean {
  return worstCaseCalls(opts) <= opts.maxRequestsPerWindow;
}

A turn with two tools and two retries costs six calls. A window of thirty still has room. A window of four does not.

Decision table

The table below separates drafting work from shipping. Draft means text inside a review only. Ship means a value loaded at process start.

Signal	Free model may draft	Free model may ship	Free server may host state
Comment on a PR	Yes	No	n/a
Unit test fixtures	Yes	Yes, in CI copies	Yes
Window length	No	No	No
Max RPS / burst	No	No	No
Retry count	No	No	No
Fail-open vs fail-closed	No	No	No
Production bucket store	No	No	No
Staging rehearsal client	Yes	Yes, with fake caps	Yes, if data is fake

When a free server is the wrong host

A free server is useful for client rehearsal. It is a poor home for the live bucket.

Process death resets an in-memory limiter fast. The next burst looks like a fresh window. Partners still count every call from the old burst.

A free server is acceptable only when all hold.

• Traffic is synthetic or fully stubbed.
• Credentials are non-production copies only.
• Payloads contain no customer fields.
• Lost limiter state cannot page a vendor.

Teams move the ledger before real keys appear. The first production partner key is the deadline.

Debugging workflow

On-call work follows a short fixed workflow. The steps below avoid another prompt-owned policy patch.

1. Export last-hour outbound 429s grouped by partner.
2. Diff the quota file against the last tagged release.
3. Reject any source field that names a prompt.
4. Confirm CODEOWNERS covers the quota path.
5. Bounce a canary and confirm bucket keys remain.
6. Only then reopen the partner ticket.

Useful commands look like this pair.

git diff v2026.09.15 -- config/outbound-quotas.json
git blame -L 1,40 config/outbound-quotas.json

A blame line pointing at a chat export is a stop. Restore the last reviewed contract values first.

Exit criteria

Teams leave the free lane when one criterion trips.

1. A partner contract now states a numeric cap.
2. Two services share one outbound API key.
3. Production 429s appear in a rolling hour.
4. A model patch changed burst, window, or retries.
5. Limiter state vanished across a process restart.
6. Legal, security, or the vendor asks for an audit trail.

The exit path is mechanical and deliberately boring. Policy source swaps to a reviewed contract file. State moves into a durable shared store.

/config/outbound-quotas.json  @platform-api @security
/config/outbound-retries.json @platform-api @security

Teams should not ask a model to migrate numbers. They copy figures from the signed contract instead. Then they run the tests shown above.

Who should not use this approach

This sketch fits small agent backends with few partners. It is not a global edge rate limiter.

Skip this sketch in the cases below.

• Quotas are negotiated per customer in billing code.
• Traffic needs a clustered algorithm with coordinated clocks.
• The team cannot review JSON inside pull requests.
• Fail-closed would break a life-critical path without fallback.

Other teams need a dedicated gateway product instead. They still must not let a free model edit caps.

Limitations

The Map store does not cluster across nodes. The sketch counts requests, not token cost. Burst is not modeled apart from the window cap.

Clocks are assumed monotonic inside a single process. Partner caps change and files rot without reminders.

These tests do not prove real vendor behavior. They only refuse prompt-owned policy at load time.

Closing rule

If a number can revoke a partner key, a person owns it. Models may comment on the surrounding client code. The quota ledger itself stays pinned in review.