Cache AI Code Reviews to Cut a 10M Token Budget 60%

You stretch a 10 million token quota for AI code reviews by counting tokens before each call, caching identical diffs, batching small changes, truncating oversized diffs, and logging every spend. I use this five-function Python workflow so repeat reviews cost zero tokens and a typical week of PRs drops from about 10,000 tokens to about 4,000.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers 10 million free tokens and a free server. The server is fast. The quota is not infinite. You still need a budget.

Count tokens before every AI code review

You cannot budget what you cannot measure. One review can eat 5,000 tokens. Ten reviews a day is 50,000. A month is 1.5 million. At that pace a 10 million free quota vanishes in about six months. I count first, then decide whether to send, truncate, or skip.

Conservative 3.5-character floor

If the endpoint is OpenAI-compatible, I count with tiktoken. Otherwise I use a conservative character estimate. Code is dense: symbols, braces, and whitespace still count, so I use about 3.5 characters per token as a planning floor, not an invoice.

import math

def estimate_tokens(text: str) -> int:
    # Conservative: ~3.5 chars per token for code
    return math.ceil(len(text) / 3.5)

Verification: write a 100-character string. The estimate should be 29.

assert estimate_tokens("a" * 100) == 29

Compare the heuristic with a real tokenizer when you have one. OpenAI's tokenizer tool is a quick way to spot-check a known diff. The 3.5 ratio is a starting point. Python and minified JSON run denser than comment-heavy Markdown. Different models also count differently, so I treat this number as a floor for planning.

What I do in practice

1. Estimate the diff before every model call.
2. Skip or truncate if the estimate exceeds the per-review cap.
3. Recheck the ratio on one real PR each week and adjust if the drift is large.

Without a count, you discover the burn after the quota is gone. With a count, the rest of the workflow has a number to enforce.

Cache identical diffs so repeat reviews cost zero

Most PRs get reviewed more than once. The diff often does not change between those passes. Your token spend does not need to change either. I hash the diff, store the response, and return the cached JSON on a hit.

SHA-256 of the raw diff

import hashlib
import json
from pathlib import Path

CACHE_DIR = Path(".ai_review_cache")

def cache_key(diff: str) -> str:
    return hashlib.sha256(diff.encode()).hexdigest()

def get_cached(key: str):
    path = CACHE_DIR / f"{key}.json"
    if path.exists():
        return json.loads(path.read_text())
    return None

def set_cache(key: str, result: dict):
    CACHE_DIR.mkdir(exist_ok=True)
    (CACHE_DIR / f"{key}.json").write_text(json.dumps(result))

Python's standard hashlib is enough. No extra package. SHA-256 of the raw diff is the key, so any byte change misses the cache.

Verification: call the same diff twice. The second call should hit the cache. I add a counter so the saving is visible.

hits = 0
def cached_review(diff):
    global hits
    key = cache_key(diff)
    cached = get_cached(key)
    if cached:
        hits += 1
        return cached
    result = {"review": "..."}  # your actual call
    set_cache(key, result)
    return result

Five passes: 10,000 tokens vs 2,000

Compare two paths on a 2,000-token PR:

• No cache: five review passes cost 10,000 tokens.
• With cache: pass one costs 2,000; passes two through five cost zero.

That is the largest single saving in this workflow. Caching is not a semantic store. A one-space change misses, and that is correct behavior. I keep .ai_review_cache out of git so review text does not leak into the repo.

Batch small diffs, then truncate oversized reviews

One file changed? One review. Ten files? Ten reviews and ten round trips of overhead tokens. I pack small diffs into one prompt until a token cap. Then I hard-truncate anything that still overflows a per-review budget.

Pack until a 6,000-token cap

def batch_diffs(diffs: list[str], max_tokens: int = 6000) -> list[list[str]]:
    batches = []
    current = []
    current_tokens = 0
    for diff in diffs:
        t = estimate_tokens(diff)
        if current_tokens + t > max_tokens:
            batches.append(current)
            current = []
            current_tokens = 0
        current.append(diff)
        current_tokens += t
    if current:
        batches.append(current)
    return batches

Verification: create three diffs of 100 tokens each. Set max to 250. You should get two batches.

Hard-truncate at 4,000 tokens

The model has a context window. The quota has a limit. I set a cap and log truncation so a single giant diff cannot empty the budget.

MAX_REVIEW_TOKENS = 4000

def truncate_diff(diff: str) -> str:
    tokens = estimate_tokens(diff)
    if tokens <= MAX_REVIEW_TOKENS:
        return diff
    ratio = MAX_REVIEW_TOKENS / tokens
    cut = int(len(diff) * ratio)
    return diff[:cut] + "\n... [truncated]"

Verification: feed a 10,000-token diff. The output should be near 4,000 tokens.

Trade-offs I accept on purpose:

• Batching saves overhead tokens but can mix per-file context. The model may confuse two files.
• Truncation drops the end of a large diff. I use it as a filter, not a gate.
• Teams that need every file reviewed in full isolation should skip batching.

A concrete week: five PRs, each a 2,000-token diff. Without caching that is 10,000 tokens. With caching, a second review of the same PR costs zero. With batching, you may send two requests instead of five. Total spend in that example: about 4,000 tokens instead of 10,000 — a 60% cut. The cache works because many PRs do not change between review passes. The batch works because small diffs still fit in one context.

Log every token and wire the five functions together

You cannot improve what you do not track. I append one CSV row per review: timestamp, estimated input tokens, and response length.

import csv
from datetime import datetime

LOG_FILE = "token_usage.csv"

def log_usage(diff, response):
    with open(LOG_FILE, "a", newline="") as f:
        writer = csv.writer(f)
        writer.writerow([datetime.now().isoformat(), estimate_tokens(diff), len(response)])

Verification: run one review. Open the CSV. One row should exist.

The assembled path is truncate, then cache lookup, then call, then store, then log:

def budgeted_review(diff):
    diff = truncate_diff(diff)
    key = cache_key(diff)
    cached = get_cached(key)
    if cached:
        return cached
    result = call_model(diff)  # your existing call
    set_cache(key, result)
    log_usage(diff, result)
    return result

That is the whole pattern: count, cache, batch, truncate, log.

Who should skip parts of this

• Auditors who need raw per-file logs, not batched summaries
• Teams whose diffs are always under 500 tokens — caching adds files for little gain
• Anyone who must keep the full tail of a large diff — do not truncate those reviews

Traps I still hit:

• Token estimates drift. Recalibrate the 3.5 ratio per language.
• A one-space edit misses the cache. That is intended.
• Batching can lose per-file isolation. Split the batch if the review reads confused.

Free tokens are a resource. Treat them like memory: budget, cache, and measure.

Do this next:

1. Copy estimate_tokens, the cache helpers, batch_diffs, truncate_diff, and log_usage into your review script.
2. Run budgeted_review on your next pull request.
3. After one week, open token_usage.csv and compare the sum against an uncached baseline.
4. Recalibrate the 3.5-character ratio on one real diff, then leave a comment with your cache hit count and weekly token total so we can compare budgets.

The code stays on your machine. The savings show up in the log.

MonkeyCode provides free models that can run this workflow.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.