Ponsubash Raj R

Posted on Jul 26

Free-Tier AI Made Me Build a Traffic Cop for Tokens

#ai #redis #systemdesign #backend

My Project Brief

CourseFlow is my attempt to solve a very practical problem: long YouTube courses are useful, but consuming them is slow, messy, and usually trapped inside video timelines.

So I built a system that takes a course playlist and turns it into structured learning material:

transcripts
cleaned lesson notes
Anki flashcards
course-level exports
optional diagrams

PROJECT REPOSITORY: Open

The fun part is that this is not a toy “summarize one video” app. It has to process entire playlists, sometimes 50 or 60 videos long. That means many transcript chunks, many LLM calls, many retries, and many ways to discover that free-tier AI is not free from consequences.

The first version worked locally. Then I started asking the real question:

Can this process a full course without wasting API quota, crashing halfway, or pretending 429 errors are a lifestyle?

That is where the quota-aware scheduler came in.

The Local Architecture

For this blog, I am focusing only on the local architecture.

The local setup uses a normal distributed application shape, just running on one machine:

The frontend starts course processing. The backend stores the course, videos, jobs, notes, and usage records. Celery workers handle the slow work. Redis handles fast coordination. PostgreSQL stores durable truth.

That split matters.

Redis is fast, but temporary. PostgreSQL is slower, but trustworthy.

The AI provider is external, rate-limited, and has opinions.

Celery workers are concurrent and mildly chaotic, as workers usually are when left unsupervised.

So the core design rule became simple:

Redis decides who may go now. PostgreSQL remembers what actually happened.

That one sentence saved the system from a surprising amount of pain.

AI APIs Are Shared Distributed Resources

At first glance, an LLM API looks like a function call:

response = client.chat.completions.create(...)

Cute.

In reality, it behaves more like a shared distributed resource with multiple constraints:

requests per minute/requests per day
tokens per minute/tokens per day
audio seconds per hour
model-specific limits
organization-level limits
retry windows
partial failures

So if five workers call the API at the same time, they are not five independent workers anymore. They are five people drinking from the same bottle and acting shocked when it becomes empty.

This is why local “sleep for 10 seconds” throttling was not enough. It works only when one worker exists, the moon is aligned, and no request uses more tokens than expected.

CourseFlow needed an organization-wide scheduler.

The Useful Part: Groq Gives Rate Limit Headers

Groq exposes rate limit information through response headers, including:

x-ratelimit-limit-requests
x-ratelimit-remaining-requests
x-ratelimit-reset-requests
x-ratelimit-limit-tokens
x-ratelimit-remaining-tokens
x-ratelimit-reset-tokens
retry-after for 429 responses

Reference: Groq rate limit documentation

That is extremely useful because the server is the source of truth. Local counters are helpful, but the provider knows the real remaining quota.

So the scheduler treats provider headers as authoritative whenever available.

A simplified version of the header parsing looks like:

def parse_groq_headers(headers):
    return {
        "rpd_remaining": int(headers.get("x-ratelimit-remaining-requests", 0)),
        "rpd_reset": parse_duration(headers.get("x-ratelimit-reset-requests")),
        "tpm_remaining": int(headers.get("x-ratelimit-remaining-tokens", 0)),
        "tpm_reset": parse_duration(headers.get("x-ratelimit-reset-tokens")),
        "retry_after": parse_seconds(headers.get("retry-after")),
    }

Redis Atomic Reservations: The Bouncer at the Door

The biggest risk with concurrent workers is double-spending quota.

Imagine Redis says there are 2 requests left. Three workers check at the same time:

Worker A: sees 2 left
Worker B: sees 2 left
Worker C: sees 2 left

All three proceed.

Congratulations, you have now duplicated the source of spending.

To prevent that, CourseFlow uses atomic Redis reservations. The decision and the counter update must happen together.

Conceptually, the reservation looks like this:

-- Pseudocode
if remaining_requests < requested_requests then
  return "blocked"
end

if remaining_tokens < estimated_tokens then
  return "blocked"
end

remaining_requests = remaining_requests - requested_requests
remaining_tokens = remaining_tokens - estimated_tokens

return "reserved"

Redis Lua scripts execute atomically, so no other worker can sneak in halfway through. Redis Lua scripting.

This turns quota into something workers reserve before calling the API.

Not after.

After is too late. After is when you are writing an incident report to yourself.

PostgreSQL Durable Usage Ledger: The Memory That Survives Restart

Redis is great for fast coordination, but if the local machine restarts, Redis may lose volatile state depending on configuration.

PostgreSQL is where CourseFlow records durable usage:

CREATE TABLE groq_usage_ledger (
    id UUID PRIMARY KEY,
    model TEXT NOT NULL,
    usage_type TEXT NOT NULL,
    request_id TEXT,
    requests_used INTEGER NOT NULL,
    tokens_used INTEGER DEFAULT 0,
    audio_seconds_used NUMERIC DEFAULT 0,
    created_at TIMESTAMPTZ NOT NULL
);

The actual schema can evolve, but the idea is stable:

record successful requests
record token usage
record audio seconds
record model and request type
allow quota state to be reconstructed

PostgreSQL gives the durable database foundation here.

This matters because workers can crash. Redis can be rebuilt. But completed usage should not become expired information.

Short Limits vs Daily Exhaustion

Not all rate limits mean the same thing.

Some are short-term limits:

requests per minute
tokens per minute
audio seconds per hour

For these, the correct behavior is:

Wait, then retry.

The job should be marked as rate_limited, scheduled after the reset time, and resumed later.

Daily exhaustion is different:

requests per day
tokens per day
daily provider allowance

For these, retrying in 30 seconds is just performance art.

The correct behavior is:

Defer until the daily window resets.

CourseFlow maps short waits to rate_limited and daily exhaustion to deferred.

That distinction is important for UX too. A user should know whether the system is temporarily waiting or done for the day.

Whisper Is Not “Just Another API Call”

Whisper transcription adds another dimension: audio duration.

Groq’s speech-to-text API has its own constraints. Reference: Groq speech-to-text documentation.

For long videos, CourseFlow chunks audio and sends each chunk separately. But each chunk needs quota reservation too.

The scheduler reserves:

one request
estimated billable audio duration
model-specific Whisper capacity

The tricky bit is minimum billable duration.

If a provider bills short audio chunks using a minimum duration, your scheduler must account for that. Otherwise, you will underestimate usage.

Conceptually:

billable_seconds = max(actual_chunk_seconds, 10)

That tiny max() carries real engineering weight. Without it, 100 tiny chunks can look cheap locally while the provider counts them differently.

Why 429s Should Not Consume Quota

This one is subtle and important.

A 429 Too Many Requests means the provider rejected the request because a limit was exceeded.

So the local reservation should be released.

If the scheduler reserves quota, sends the request, receives a 429, and still counts the reservation as consumed, the system punishes itself twice:

Provider: "No."
Local scheduler: "Understood. I will now reduce my own local quota too."

Completely wrong.

The better behavior is:

Reserve estimated capacity.
Send request.
If success, reconcile with actual usage and headers.
If 429, release the reservation.
Set a shared blocked_until gate using retry-after or reset headers.

Simplified:

try:
    response = call_groq()
    scheduler.commit_success(reservation, response.headers, response.usage)
except RateLimitError as exc:
    scheduler.release(reservation)
    scheduler.block_until(parse_retry_time(exc.headers))
    raise

Basically, a rejected request should influence scheduling, not usage accounting.

That is how the system avoids slowly starving itself after temporary rate limits.

Resumability: The Quiet Superpower

The scheduler is not just about avoiding 429s. It also makes long course processing resumable.

A course can take a while. During that time:

a worker can crash
the machine can restart
Redis can be cleared
the API can rate limit
Whisper can fail on one chunk
one lesson can be deferred while others continue

CourseFlow handles this by making every meaningful unit durable:

videos have statuses
notes are stored
transcript chunks are tracked
Whisper chunks are stored
usage is recorded
retries have timestamps
completed work is skipped on restart

So if chunk 17 of a video hits a rate limit, the system does not redo chunks 1-16 like it has short-term memory issues.

It retries the blocked chunk later.

That is the difference between a demo and a system.

The Final Local Design

The final flow looks like this:

The important part is not one specific model or provider.

The important part is the pattern:

Estimate before request.
Reserve atomically.
Call provider.
Reconcile from headers.
Persist durable usage.
Retry only the failed unit.

That pattern works for LLMs, Whisper, image generation, and basically any paid or limited API where “oops” has a quota cost.

Takeaway

Free-tier AI sounds like a pricing detail.

It is not.

When you build a real application on top of free-tier or quota-limited APIs, you are suddenly dealing with distributed systems problems:

shared state
concurrent workers
partial failure
durable accounting
retry classification
idempotency
backpressure

In summary:

I wanted video summaries. I got consensus-adjacent quota coordination.

That is the engineering lesson CourseFlow taught me.

And honestly, it was a good one.

DEV Community