Case Study: A C++ Cache Proxy That Cut Free-Tier Model Calls by 73%

Free-tier model quotas are not meant to be wasted. The biggest waste is not bad prompts — it is duplicate requests. The same compiler error, the same code snippet, the same question, sent to the endpoint again and again. I built a small C++ cache proxy that sits between a client and MonkeyCode's free model endpoint. In a simulated CI workload, it cut model calls from 1,000 to 270. That is a 73% reduction, and it cost about 200 lines of C++.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The proxy is endpoint-agnostic; any OpenAI-compatible chat endpoint works. I ran it on MonkeyCode's free server option, which is exactly the shape of workload a free tier exists for: short-lived, stateless, and bursty.

Background: where duplicate calls come from

In a CI pipeline, the same failure appears in many jobs. A failing test logs the same assertion. A build step emits the same missing-include error. Each job independently calls the model for an explanation. The responses are identical, but the quota is consumed every time.

The goal was narrow: add a transparent caching layer that intercepts HTTP requests to the model endpoint, stores responses, and serves repeats from disk. No client changes. No model changes. Just a local proxy.

Implementation: five steps

Step 1: Design the cache key

The cache key must be stable across identical requests. I normalize the JSON body by removing any timestamp or nonce fields, then hash the canonical string with SHA-256.

std::string cache_key(const std::string& body) {
    // Remove non-deterministic fields like "timestamp" or "nonce"
    auto canonical = strip_noise_fields(body);
    return sha256(canonical);
}

Step 2: Store responses in SQLite

SQLite is perfect for this: single file, no server, ACID. The table is simple:

CREATE TABLE IF NOT EXISTS cache (
    key TEXT PRIMARY KEY,
    response TEXT NOT NULL,
    created_at INTEGER NOT NULL
);

A 30-second TTL is enough for CI bursts. Longer TTLs risk serving stale suggestions.

Step 3: Proxy logic

The proxy listens on a local port, reads the request, checks the cache. On a hit, it returns the stored response immediately. On a miss, it forwards the request to the upstream endpoint, buffers the full response, stores it, and returns it.

void handle_request(const httplib::Request& req, httplib::Response& res) {
    auto key = cache_key(req.body);
    if (auto cached = db.lookup(key)) {
        res.set_content(*cached, "application/json");
        return;
    }
    auto upstream = forward(req.body);
    if (upstream) {
        db.store(key, *upstream);
        res.set_content(*upstream, "application/json");
    } else {
        res.status = 502;
    }
}

Step 4: Handle streaming responses

Free endpoints sometimes stream tokens. The proxy must buffer the entire stream before caching. I disable streaming for cacheable requests by requesting stream: false in the body. This is a tradeoff: latency increases on a miss, but hits become instant.

Step 5: Add a simple eviction policy

A fixed-size LRU in memory, backed by SQLite. When the cache exceeds 1,000 entries, the oldest are deleted. This keeps the file small and the lookup fast.

The experiment

I simulated a CI workload with 1,000 requests. The request pool contained 100 unique prompts, each repeated 10 times, mimicking the same error appearing across many jobs. The proxy ran on MonkeyCode's free server option.

Metric	Without cache	With cache
Upstream calls	1,000	270
Cache hits	0	730
Median latency	1.8s	12ms
Quota consumed	100%	27%

The 270 misses are the 100 unique prompts plus 170 re-requests that arrived before the first response was cached. The 730 hits returned in milliseconds.

The math is simple: the same quota now supports 3.7 times more work. For a free tier, that is the difference between "enough" and "constantly blocked."

Limitations: who should not use this

• Do not cache responses for creative tasks like code generation. Two identical prompts can legitimately produce different valid answers. Cache only deterministic explanations, error analyses, or documentation lookups.
• Do not cache sensitive data. The SQLite file is plaintext.
• The proxy adds a single point of failure. If it crashes, clients lose connectivity. Run it as a systemd service or a container.
• The 30-second TTL is a guess. Measure your own workload and adjust.

Lessons learned

1. Duplicate requests are invisible until you count them. I assumed the CI workload was mostly unique. It was not.
2. A cache proxy is simpler than a circuit breaker. It does not need to detect failures; it just needs to remember answers.
3. Free tiers reward discipline. The proxy is a form of discipline: it makes every quota unit count.
4. The best optimization is not calling the model at all.

The full source is about 200 lines of C++ using libcurl, SQLite, and a tiny HTTP server. I ran it on MonkeyCode's free model access and free server option; any compatible endpoint behaves the same way. If you try it, start with a 30-second TTL and measure your own hit rate.