The Heat Just Moved

The heat just moved after that so-called faster patch. That single sentence is the whole lesson today.

I no longer trust a stopwatch screenshot from a chat window. A quieter allocation graph decides the merge for me. Have you ever watched a claimed win vanish under garbage collection?

An assistant assumes the function you pasted is already hot. It rewrites the inner loop with unearned theatrical confidence. Did anyone ask it to prove the bottleneck first?

I generate two patches, then I profile both on one harness. I keep the allocation graph, not the sales pitch. This walkthrough is a local toy you can rerun tonight.

Steal the method. Ignore my opinions if your graph disagrees. The model does not get a vote after the numbers land.

The setup is a tiny JSON fan-out handler. One function builds a payload for many keys. Naive code concatenates strings and sprays short-lived objects everywhere.

Think of it like mopping a dry floor beside an overflowing sink. The mop looks busy. The water still wins the room.

Here is the original hot path I started from.

# handler.py
def build_payload(keys, lookup):
    chunks = []
    for key in keys:
        value = lookup.get(key, "")
        chunks.append(
            '{"k":"' + str(key) + '","v":"' + str(value) + '"}'
        )
    return "[" + ",".join(chunks) + "]"

Ugly? Yes. Realistic for a rushed service path? Also yes. I kept it because assistants love rewriting exactly this shape.

I asked a coding assistant for a faster version of that loop. It reached for a cache, because caches sound like engineering. Did it ask about key cardinality before allocating the dict?

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode's free model access on the free server option only to emit candidate patches. The profiler still ran on my laptop, not in the chat.

The first candidate interned whole payloads behind a tuple key. It looked clever in the diff. It also allocated a map the workload could not bound.

# patch_cache.py
import json

_CACHE = {}

def build_payload(keys, lookup):
    cache_key = tuple(keys)
    hit = _CACHE.get(cache_key)
    if hit is not None:
        return hit
    chunks = []
    for key in keys:
        value = lookup.get(key, "")
        chunks.append(json.dumps({"k": key, "v": value}))
    out = "[" + ",".join(chunks) + "]"
    _CACHE[cache_key] = out
    return out

Unique request shapes blow that cache in minutes. Memory becomes a junk drawer with no lid. Is a speedup real if peak bytes climb every round?

The second candidate skipped the cache story completely. It built rows and dumped once through the serializer. Boring code. I like boring when the graph is noisy.

# patch_bulk.py
import json

def build_payload(keys, lookup):
    rows = [
        {"k": key, "v": lookup.get(key, "")} for key in keys
    ]
    return json.dumps(rows)

Fewer handmade strings. One library call. No unbounded dict hiding behind the function. Now comes the part agents skip on purpose.

Measure allocations. Do not measure vibes. A timer without a memory peak is a half sentence.

# profile_payload.py
import cProfile
import pstats
import random
import tracemalloc
from handler import build_payload as original
from patch_cache import build_payload as cached
from patch_bulk import build_payload as bulk

SEED = 20260906

def make_lookup(n=5000):
    return {f"k{i}": f"v{i}" * 8 for i in range(n)}

def run(fn, lookup, rounds=200):
    keys = list(lookup.keys())
    for _ in range(rounds):
        sample = random.sample(keys, 40)
        fn(sample, lookup)

def snap(label, fn, lookup):
    random.seed(SEED)
    tracemalloc.start()
    profiler = cProfile.Profile()
    profiler.enable()
    run(fn, lookup)
    profiler.disable()
    current, peak = tracemalloc.get_traced_memory()
    tracemalloc.stop()
    stats = pstats.Stats(profiler)
    stats.sort_stats("tottime")
    print(f"\n=== {label} ===")
    print(f"current_bytes={current} peak_bytes={peak}")
    stats.print_stats(8)

if __name__ == "__main__":
    lookup = make_lookup()
    snap("original", original, lookup)
    snap("cached", cached, lookup)
    snap("bulk", bulk, lookup)

Run it like this and keep the file.

python profile_payload.py > alloc_graph.txt

That text file is the graph I keep in the pull request. Not a quote from the model. Not a screenshot of a green check. Peak bytes plus the top tottime rows.

I will not invent a speedup number for your laptop. Your CPU is not mine. Read your own file before you argue with me.

I pin the seed because chaos is not a benchmark. Change one variable per run, never three. Otherwise the graph turns into soup and everyone brings a spoon.

random.seed(20260906)

On this workload the cache only shines with identical key lists. Random samples miss it and still pay for the dict. Peak memory climbs while the story still says faster.

The bulk dump usually keeps peak_bytes calmer. CPU time follows json.dumps, not my handmade loop. So the assistant optimized a fire that was already small.

The serializer was the stove. The cache added a hidden chimney. Classic wrong-room rewrite.

Why keep the graph at all? Because next week you will forget the workload. A reviewer will write faster with no method attached. You paste alloc_graph.txt and the argument ends.

I annotate three lines at the top of that file before any patch lands.

# workload: 200 rounds, 40 random keys from 5000
# metric: tracemalloc peak plus cProfile tottime
# reject if peak_bytes rises while cpu only dips

That last line is a policy, not a law of physics. Tune it for your service. Write it down before the model speaks.

Here is the decision I actually run in my head. If CPU drops and peak bytes stay flat, I keep the patch. If CPU drops and peak bytes jump, I treat it as a maybe, not a merge.

If the call graph still points at the same function, the patch did not move real heat. It rearranged furniture in a cold room. Call graphs lie less than adjectives like faster.

peak_bytes=18874368 is a fact with a unit. Faster is a mood. Which one belongs in the commit message?

Should you let the assistant write the profiler too? I would not. It profiles the fixture it just cached. It forgets random.sample. You already know that movie.

The free model is useful for variants, not verdicts. Two functions. Different ideas. Same harness, same seed, same lookup size, same round count.

I generate sketches when I am stuck, not when I am sure. The free server option keeps that sketch loop off the profiler box. My laptop stays the instrument. The assistant stays the notebook.

If the sketch is wrong, I throw it out without guilt. Cheap sketches are the point of the loop. Expensive production incidents are not a brainstorming style.

One more command sits in the PR template beside the diff.

grep peak_bytes alloc_graph.txt

Three numbers. Original, cached, bulk. That is the whole meeting. I do not need a dashboard to see the heat move.

Did the cache ever win on this toy? Yes, when I forced identical keys on purpose. That is a different product with a different graph. Measure that workload if you actually have it.

Do not measure mine and then ship yours. Agents assume the pasted function is the universe. Profilers count what the process touched.

Who should skip this whole dance? Anyone without a reproducible workload worth repeating. Anyone merging from a single timer line in a chat log.

Anyone who needs a vendor SLA for inference should not lean on a free server during an incident. A free sketch box is not a pager. Also skip this if your bottleneck is wait time on the network.

Allocations look calm while sockets burn. Profile the resource that actually hurts. Wrong instrument, confident graph, still wrong room.

Limitations are not fine print. tracemalloc misses some C-level buffers behind extensions. cProfile adds its own tax to tiny functions.

Microbenchmarks lie when the cache is warm and traffic is not. json.dumps is not your whole service, only this slice. I still keep the graph because a biased instrument beats a confident paragraph.

The model is a chef who never tastes the soup. You are the thermometer on the counter. Do you eat because the chef smiled at the plate?

I want the file in the PR, the workload comment, and one sentence that says the heat moved. Everything else is decoration.

If you run the same variant loop, a free model is enough for sketches. Then close the chat and open the graph.