Start with an event order that can burn a 10M-token grant in nine hours. A batch job sees the free tier and assumes the grant is a throughput budget, so it fires three hundred concurrent requests at the gateway. The rate limiter answers with 429s, and the retry loop re-runs every side effect without an idempotency key. By the time the queue drains, the usage ledger shows roughly double the actual consumption, and the free server has restarted twice under memory pressure. The invariant that fails is simple: a token grant is a budget with a rate ceiling. A retry without a key converts a rate limit into a cost multiplier.
What I'm reviewing
MonkeyCode is an open-source project that currently offers free model access and a free server option, and its current terms advertise a 10M-token grant. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I'm reviewing the architecture from the outside, so treat every number here as operator-supplied as of 2026-08-26, and verify the current terms before you build anything on them.
Let me declare the assumptions before the analysis. Free model access means shared capacity with rate limiting and best-effort latency; the free server is a small process with finite memory and no uptime guarantee; and the grant is a usage budget, not a throughput promise. If any of those assumptions is wrong, the review still holds, but the thresholds shift. The interesting question is not whether the tier is generous; it's whether the gateway can keep one tenant's runaway prompt from starving the shared endpoint, and whether the ledger can survive a restart without duplicating work.
The data flow and where it breaks
The data flow I'll model has four hops, and each hop is a failure domain in disguise:
client ──> gateway ──> rate limit + grant check ──> queue ──> model endpoint
▲ │
└────────── usage ledger ◄── stream close ─────┘
- The client authenticates and submits a request to the gateway, which checks the tenant's rate limit and grant balance.
- The gateway either queues the request, rejects it with 429, or forwards it to the model endpoint.
- The model streams tokens back through the gateway, and the usage ledger records consumption when the stream closes.
- The free server runs the client, the gateway, or both, depending on the deployment, and a restart at any hop changes the replay behavior.
The rate limiter is the first failure domain, because a per-IP limiter lets one user's retry storm exhaust the shared budget for everyone. The queue is the second, because a slow upstream makes it grow, and on a free server the queue usually lives in process memory, so a long tail becomes an out-of-memory kill. The ledger is the third, because recording consumption at stream close makes in-flight requests invisible to the grant check, and a batch can overspend its own budget. The restart is the fourth, because a stateless client reconnects and replays the batch, and without idempotency keys the replay duplicates every side effect.
A minimal gateway state machine
Here's a minimal model of the gateway I'd want, written as a state machine rather than production code:
# gateway_model.py — minimal model, not production code
# States: IDLE, RATE_LIMITED, QUEUED, IN_FLIGHT, DONE, FAILED, REJECTED
from collections import deque
class Gateway:
def __init__(self, grant_tokens, rate_per_min, queue_depth):
self.balance = grant_tokens # reserved on send, not on completion
self.bucket = rate_per_min # per-tenant token bucket
self.queue = deque(maxlen=queue_depth)
self.seen_keys = set() # idempotency keys, persisted externally
def submit(self, req):
if req.key in self.seen_keys:
return replay(req.key) # return stored result, never re-run
if self.balance < req.est_tokens:
return reject(402, "grant exhausted")
if not self.bucket.take(1):
return reject(429, "rate limited")
self.balance -= req.est_tokens # reserve before send
self.seen_keys.add(req.key)
self.queue.append(req)
return accept(req.id)
The explicit denominator matters here: the grant is consumed as requests × estimated tokens per request, while the rate limiter is denominated in requests per minute. A batch of a thousand requests at ten thousand tokens each consumes the full grant, and the queue depth is the third denominator, because it caps memory on the free server.
Testable properties and injected failures
Two properties are invariants rather than performance goals, and both are testable with failure injection. First, for any retry sequence, a side effect runs at most once, because the gateway checks the idempotency key before it touches the model. Second, the grant can never be overspent by more than the in-flight margin, because tokens are reserved when the request leaves the queue, not when the response lands.
Inject a 429 storm and the bucket drains, the client backs off, and the queue depth caps memory; the acceptance rule is that one tenant's burst cannot push the process past its memory ceiling. Inject an upstream slowdown and the queue fills, new requests are rejected instead of queued, and the ledger still shows the reserved consumption. Inject a restart and the queue is lost, but the idempotency keys survive if the ledger is external, so the client's replay is deduplicated rather than duplicated.
Tradeoff table
| Decision | What you gain | What you pay |
|---|---|---|
| Reserve tokens before send | The grant cannot be overspent | A small latency cost per request |
| Per-tenant token bucket | One runaway prompt cannot starve the tier | More gateway state to shard |
| Idempotency keys at the gateway | Retries stop multiplying side effects | Key storage and lookup cost |
| External queue and ledger | A restart becomes a cold start, not a replay | More moving parts than a free server implies |
What I'd change next
If I owned this gateway, here is the order of changes I'd make, and each one is cheap to test in isolation.
- Move the idempotency check from the client to the gateway, and make the key mandatory for any request that can produce a side effect.
- Reserve tokens when the request leaves the queue, so the grant check always sees in-flight work.
- Give every tenant a token bucket with a small burst allowance, and make the queue depth a hard cap rather than a soft hint.
- Add a circuit breaker to the upstream model endpoint, with a half-open probe that lets one request through after a cooldown.
- Push the queue and the ledger into a sidecar or an external store, so the free server stays stateless and a restart is a cold start instead of a replay.
Limitations and who should not use this
I'm reviewing from the outside, so I haven't seen the real gateway's telemetry, and the state machine above is a model, not a benchmark. The 10M-token figure and the free server terms are operator-supplied as of 2026-08-26, and they will change, so check the current documentation before you size a workload against them. If your workload has a real SLA, regulated data, or a hard requirement that a batch finishes in one attempt, do not build it on a free tier; use the tier for experiments, evaluation harnesses, and load-model validation, and keep production traffic off it.
The counterexample to test
Here's the event order I'd want you to test before you trust any of this. A long-running agent starts at 23:59 UTC, the grant refreshes at 00:00, and the agent's in-flight request crosses the boundary while the bucket resets, so the gateway sees a fresh balance and a request it has already reserved. Should the gateway reject the request, replay it with the same key, or compensate by charging the new grant? If you can't answer that with a state transition, the free tier will answer it for you, and the answer will be a duplicate email.
Top comments (0)