I Kept the Allocation Graph

The stopwatch lied again. I kept the allocation graph instead.

An AI patch halved a tight Python loop. Resident memory still climbed like floodwater. Do you still trust wall clock after that?

Models love pretty reductions. They hoist lists and cache entire dicts. They flatten generators into giant arrays. The CPU graph smiles. The heap does not.

This is a local lab walkthrough. I label it that way on purpose. No production SLA lives in this note. You can copy the harness tonight.

I asked a coding model for a faster aggregator. The prompt stayed boring on purpose. Sum scores by user from JSONL. That was the entire job.

The first draft looked clever. It built a giant dict of lists. Then it mapped a mean across each list. It read clean and ran fast on ten thousand lines.

What happens at two million lines? Guess.

I used MonkeyCode's free models on a free server. I drafted alternate patches in that scratch box. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I treat the product as a compiler, not a profiler. It does not own the merge.

MonkeyCode is an open source coding project. Free model access and a free server option exist for draft-and-discard work. I still measure on my own machine.

The free server helps when local fans already scream. I generate candidates there. I judge them on a quiet core. Different rooms. Different jobs.

I froze two functions on one fixture. Same file. Same seed. One grouped scores in lists. One kept a running mean in two tiny dicts.

Here is the fixture builder. Run it once. Keep the file.

# make_scores.py — lab fixture, not production data
import json
import random
import pathlib

random.seed(7)
path = pathlib.Path("scores.jsonl")
users = [f"u{i:04d}" for i in range(500)]

with path.open("w", encoding="utf-8") as f:
    for n in range(2_000_000):
        rec = {"user": users[n % 500], "score": random.random() * 100.0}
        f.write(json.dumps(rec) + "\n")

Two million tiny records make allocation visible. You still do not need a cluster. This is a microscope slide. Treat it like one.

Now the two aggregators. I kept the names ugly on purpose.

# agg.py
from __future__ import annotations
import json
from collections import defaultdict

def agg_lists(path: str) -> dict[str, float]:
    buckets: dict[str, list[float]] = defaultdict(list)
    with open(path, encoding="utf-8") as f:
        for line in f:
            rec = json.loads(line)
            buckets[rec["user"]].append(float(rec["score"]))
    return {u: sum(v) / len(v) for u, v in buckets.items()}

def agg_running(path: str) -> dict[str, float]:
    n: dict[str, int] = defaultdict(int)
    s: dict[str, float] = defaultdict(float)
    with open(path, encoding="utf-8") as f:
        for line in f:
            rec = json.loads(line)
            u = rec["user"]
            n[u] += 1
            s[u] += float(rec["score"])
    return {u: s[u] / n[u] for u in n}

The model preferred agg_lists. Of course it did. Lists feel like data science. Running totals feel like 1998. Which one is the adult here?

I do not time a single call first. I snapshot the heap. Then I time. Order matters a lot. Timing itself allocates objects. You knew that, right?

# harness.py — lab only
from __future__ import annotations
import csv
import gc
import pathlib
import time
import tracemalloc
from agg import agg_lists, agg_running

OUT = pathlib.Path("alloc_graph.csv")

def peak_for(fn, path: str) -> dict:
    gc.collect()
    tracemalloc.start()
    t0 = time.perf_counter()
    out = fn(path)
    elapsed = time.perf_counter() - t0
    current, peak = tracemalloc.get_traced_memory()
    tracemalloc.stop()
    row = {
        "fn": fn.__name__,
        "n_keys": len(out),
        "sec": round(elapsed, 3),
        "peak_mib": round(peak / 1024 / 1024, 2),
        "current_mib": round(current / 1024 / 1024, 2),
    }
    new = not OUT.exists()
    with OUT.open("a", newline="", encoding="utf-8") as f:
        w = csv.DictWriter(f, fieldnames=row.keys())
        if new:
            w.writeheader()
        w.writerow(row)
    return row

if __name__ == "__main__":
    path = "scores.jsonl"
    print(peak_for(agg_lists, path))
    print(peak_for(agg_running, path))

I ran both functions three times. I threw away the first lap. Cold page cache lies with a straight face. Sound familiar?

I will not invent a sacred nanosecond for you. On my laptop the list peak jumped as N grew. The running mean barely moved. Wall clock stayed in the same neighborhood. That is the whole plot.

Read your own CSV like a crime scene. Ignore the sec column on the first pass. Then stare at the peak_mib column hard. If that column tracks N, you bought a leak in slow motion. Does your win still look like a win?

I varied N on purpose for the slope. I rebuilt the fixture at 50k, 200k, and 2M. The list line steepened on every larger N. The running line stayed a flat whisper throughout. I kept a slope rather than a screenshot.

A CPU sampler will not save you here. It sees json.loads and the loop body. It still does not see the backpack. Allocation is a count, not a duration. That is different physics and a different instrument.

Some patches hide the churn behind a C extension. Then tracemalloc can look calm and wrong. RSS still climbs in the time output. That is why I keep /usr/bin/time in the pocket. I want two instruments and one clear veto.

The CSV is the graph I kept. I do not need a dashboard for a veto. Later I plot peak_mib against N. I commit that CSV beside the patch.

What shape should you expect? Two lines on a napkin. One climbs with every extra score. One sits on a table of five hundred users. Could you explain that to an intern fast?

I used to paste CPU flamegraphs into reviews. People cheered at the thinner bar. Then a box died from an OOM. The flamegraph never showed those lists. Why would it even try? CPU was fine. The janitor was drowning.

Think of a sprinter with a backpack. You timed the legs only. Bricks kept falling in during the race. Funny result, right?

Commands I actually type sit below. Nothing fancy.

python make_scores.py
python harness.py
python harness.py
python harness.py
cat alloc_graph.csv

On Linux I also ask /usr/bin/time for RSS.

/usr/bin/time -v python -c "from agg import agg_lists; agg_lists('scores.jsonl')"
/usr/bin/time -v python -c "from agg import agg_running; agg_running('scores.jsonl')"

Read Maximum resident set size. That line often carries the veto. Optional picture for a skeptic:

pip install memray
python -m memray run -o lists.bin -c "from agg import agg_lists; agg_lists('scores.jsonl')"
python -m memray flamegraph lists.bin

If memray feels heavy, skip it. tracemalloc already tells the story. I keep the CSV as the committed artifact. I paste a flamegraph only when someone argues.

Cheap generated patches create a new kind of debt. The code looks smaller. The objects do not. Have you watched a “cleanup” allocate more than the mess?

Why do models do this so often? They trained on tidy notebooks. Notebooks load everything into RAM. They celebrate vectorization as virtue. They rarely celebrate constant memory.

Ask for faster code and you often get more RAM. Ask for fewer allocations and the patch changes. Did you ask the right question this time?

I now put a budget in the prompt. Peak extra memory must stay small. Stream the file. Do not collect per-event lists. I still verify the dump. Always verify the dump.

A prompt I actually reuse looks like this.

Rewrite agg_lists as a single-pass aggregator.
Do not store per-event lists.
Peak extra Python memory should stay roughly O(users).
Stream scores.jsonl. Return mean score per user.
Then I will profile both functions with tracemalloc.
Do not claim a speedup. I will measure.

Notice the last two lines. They are not decoration. Models love to narrate a win. Your job is to starve that habit. Make the graph the only speech.

Free model access makes retries cheap enough. That is why it belongs in this workflow. Generate three shapes. Profile all three. Keep one graph. Throw two patches away without guilt.

Is that wasteful? Compared with an OOM in prod? Please.

I do not ship a template in the PR. I paste four short lines. Wall clock delta was mixed, so I ignored it. tracemalloc peak climbed with N on the list version. The running version stayed flat. I merged the boring function.

Reviewers argue with vibes. Graphs end those arguments. Have you noticed that yet?

This harness lies in several ways. tracemalloc misses some C-level buffers. JSON parsing already allocates like a fountain. Two million lines is still a microbenchmark. GC timing will jitter on a noisy laptop.

Do not use this as a production profiler. Use it to veto a patch. Different tool. Different arrogance.

Skip this approach if your hot path is not Python. Skip it if you cannot reproduce the input. Skip it if you never measured a baseline. A graph without a baseline is fan fiction.

Also skip it if the allocator is not your problem. Network waits need waterfalls. Lock convoys need contention profiles. This lab will not smell those. It only hunts object churn.

Free servers are shared and bursty. I would not treat a remote giant run as gospel. I generate code there. I measure here. If your quiet core is compiling a browser, wait.

If you want a scratch box for those alternate aggregators, the free model access and free server option is how I obtained candidates. I still kept the graph.

I stopped asking if the loop was faster. I ask what we allocated instead. I keep the CSV. I keep the veto.

The interesting work moved again. It is not the loop body. It is the objects the loop births. You can feel clever and still drown.

Would I merge the pretty version for a tiny cron? Maybe. Would I merge it for a firehose? Not after that graph.

The heap said no. I listened.