DEV Community: Alan West

Why your quantized LLM loses its MTP heads and how to keep them

Alan West — Wed, 27 May 2026 16:00:08 +0000

The frustrating problem

Last month a teammate pinged me with a classic head-scratcher. He'd taken a base model with multi-token prediction (MTP) heads, ran it through a standard quantization pipeline to ship a smaller GGUF for edge inference, and the latency numbers came back worse than expected. The model still generated coherent text, but the speculative decoding speedup he'd built his benchmarks around was gone.

We poked around for an hour before the penny dropped. The MTP heads had silently been dropped on the floor during conversion. The base weights survived. The extra prediction heads — the whole reason MTP exists — did not.

If you've worked with models that ship MTP layers (the technique popularized by DeepSeek-V3, where the model predicts the next N tokens in parallel as draft tokens), you might have already run into this. The conversion toolchain assumes anything that isn't a vanilla transformer block is dead weight and trims it. Here's why it happens and how to stop it.

What MTP heads actually are

Quick refresher so we're on the same page. MTP (multi-token prediction) adds auxiliary heads on top of the base model that each predict a future token at offset +1, +2, +3, etc. At inference time you can use them as a built-in draft model for speculative decoding, which gives you a real throughput win without needing a separate small model.

The key thing: these heads are architecturally distinct from the regular lm_head. They live in their own module tree, often named something like model.mtp.layers.0, model.mtp.layers.1 and so on. They reference shared embeddings but have their own normalization, attention, and projection weights.

That naming convention is exactly what trips up the tooling.

Root cause: conversion scripts have an opinionated allowlist

Most quantization toolchains weren't designed with MTP in mind. They walk the state dict and apply transformations based on regex matches against expected layer names. Anything that doesn't match is either:

Silently dropped (worst case)
Left in fp16/fp32 in the output (works but bloats the file)
Renamed in a way the loader can't recover (subtle breakage)

When I dug into the llama.cpp conversion script for the project, the relevant logic was essentially this pattern:

# Simplified version of what most converters do
KNOWN_PREFIXES = ("model.layers.", "model.embed_tokens.", "model.norm.", "lm_head.")

for name, tensor in state_dict.items():
    if not name.startswith(KNOWN_PREFIXES):
        # MTP heads land here and get skipped
        logger.debug(f"skipping unknown tensor: {name}")
        continue
    write_quantized(name, tensor)

The logger.debug is the killer. Unless you run conversion with debug logging on, you never see the skip messages. The file converts "successfully" and you walk away thinking everything's fine.

GPTQ-style quantizers have a related but different failure mode. They calibrate against forward passes through the model, and if your calibration code only exercises the main lm_head path, the MTP heads never see calibration data. Even if the weights are preserved, the resulting quantized heads are essentially random.

Step-by-step solution

Here's the workflow I now use whenever I touch a model with MTP heads.

Step 1: Inventory the heads before you touch anything

Before any conversion, dump the full state dict and grep for MTP-related modules. This sets your baseline.

from safetensors import safe_open

mtp_tensors = []
with safe_open("model.safetensors", framework="pt") as f:
    for key in f.keys():
        # Adjust prefix to whatever your model uses
        if "mtp" in key.lower() or "multi_token" in key.lower():
            mtp_tensors.append((key, f.get_slice(key).get_shape()))

for name, shape in mtp_tensors:
    print(f"{name}: {shape}")

print(f"\nTotal MTP tensors: {len(mtp_tensors)}")

Save this output. You'll diff against it after every conversion step.

Step 2: Patch the converter's allowlist

For llama.cpp style converters, you need to extend the known prefix list and add a mapping rule for the MTP heads. The clean way is to subclass or monkey-patch rather than editing the upstream script directly:

from convert_hf_to_gguf import Model

class MTPAwareModel(Model):
    def map_tensor_name(self, name: str) -> str:
        # Handle MTP heads explicitly before falling through
        if name.startswith("model.mtp."):
            # Preserve the layer index and submodule path
            # Output name needs to match what your loader expects
            return name.replace("model.mtp.", "mtp.")
        return super().map_tensor_name(name)

    def modify_tensors(self, data, name, bid):
        # Skip the parent class's filter for MTP layers
        if "mtp" in name:
            return [(self.map_tensor_name(name), data)]
        return super().modify_tensors(data, name, bid)

The critical bit is overriding modify_tensors — the default implementation has the silent skip we saw earlier.

Step 3: For GPTQ, calibrate through the MTP path

If you're using GPTQ-style quantization, your calibration loop needs to actually hit the MTP heads. The default model(input_ids) forward pass only routes through the main LM head. You need to force the MTP heads to see activations:

def calibration_forward(model, batch):
    # Standard forward populates main path activations
    outputs = model(**batch, output_hidden_states=True)

    # Manually invoke MTP heads using the final hidden state
    # This ensures each head gets calibration statistics
    hidden = outputs.hidden_states[-1]
    for i, head in enumerate(model.mtp.layers):
        # Shift input so head i predicts token at position +i+1
        shifted = hidden[:, :-(i + 1), :]
        _ = head(shifted)

    return outputs

Without this, your MTP heads quantize to garbage even though the file looks complete.

Step 4: Verify post-conversion

Re-run the inventory script against the converted file. The tensor count should match. If you went GGUF, you can also dump metadata:

# llama.cpp ships a metadata inspection tool
./gguf-dump model-quantized.gguf | grep -i mtp

Then run a quick speculative decoding sanity check. If the MTP heads are intact and properly calibrated, you should see your tokens-per-second numbers match (or get very close to) the unquantized baseline's speedup ratio.

Prevention tips

A few habits that have saved me repeated pain:

Always run converters with debug logging enabled. The skip messages are the single most useful signal you'll get, and they're hidden by default.
Tensor-count diff as part of CI. If your pipeline converts models automatically, fail the build when the output has fewer tensors than the input minus a known allowlist of intentionally-dropped weights.
Test speculative decoding throughput, not just generation quality. A model can produce fluent text with broken MTP heads — your end-to-end latency benchmark is the only thing that will catch the regression.
Pin your converter version. Upstream conversion scripts change their tensor-name handling more often than you'd think. A model that converted cleanly six months ago might silently break today.

MTP is one of those features where the failure mode is invisible until you measure the thing the feature was supposed to improve. Treat the conversion pipeline as untrusted by default, and you'll avoid burning an afternoon on it like we did.

How to build reliable geo-restrictions that actually hold up in production

Alan West — Wed, 27 May 2026 01:41:07 +0000

Last week I saw another platform get blocked in a European market because their geo-restriction setup was, charitably, optimistic. A single header check. No IP verification. Nothing to handle VPNs or the weird middle-ground of corporate proxies. The result? Regulators noticed users in restricted regions were still getting through, and the whole product got pulled.

I've shipped jurisdiction-based access controls for a fintech and a streaming-adjacent product, and I'll tell you up front: this stuff is harder than it looks. The problem isn't "detect the country" — that part has been solved for years. The problem is doing it reliably enough that compliance won't yell at you, without breaking your legitimate users in the process.

Let's walk through what actually goes wrong and how to build something that holds up.

Why naive geo-blocking fails

Most teams start with something like this:

// Don't do this in production
app.use((req, res, next) => {
  const country = req.headers['cf-ipcountry']; // or similar
  if (BLOCKED_COUNTRIES.includes(country)) {
    return res.status(451).send('Not available in your region');
  }
  next();
});

Looks fine. Ships in five minutes. And it falls apart roughly the moment a real user hits it. Here's what I've seen break:

IP geolocation is probabilistic, not deterministic. Free databases are accurate maybe 95-97% at the country level. The remaining few percent is users hitting your block page in the wrong country, or the right country thinking they're somewhere else.
Mobile carriers route weirdly. A user in Madrid on a mobile network might appear to come from a carrier hub in Frankfurt. I've seen this trip up half a dozen geo checks.
CDN headers can be cached. If you're caching responses upstream of your geo check, you can serve a Spanish user a German-cached page. Whoops.
VPN and residential proxy traffic. This is the big one. If you only check IP, you're trivially bypassed.

The root cause across all of these is that we're trying to make a binary decision (allow/deny) based on a fundamentally fuzzy signal (where is this packet coming from). The fix is to stop pretending the signal is clean.

A layered approach that actually works

The pattern I've landed on after migrating three projects is: multiple independent signals, scored, with explicit handling for ambiguous cases. Not a single check.

Here's the rough shape of the middleware I use. Adapt to your stack — this is Node, but the logic ports cleanly:

const geoip = require('geoip-lite'); // open-source, MaxMind GeoLite2 data

async function evaluateJurisdiction(req) {
  const signals = {
    ipCountry: null,
    headerCountry: null,
    declaredCountry: null,
    vpnLikely: false,
  };

  // Signal 1: server-side IP lookup (don't trust client headers alone)
  const ip = getClientIp(req); // see notes below on extracting this safely
  const lookup = geoip.lookup(ip);
  signals.ipCountry = lookup?.country || null;

  // Signal 2: edge/CDN-provided country header, if you have one
  // Useful as a cross-check, NOT as your only source
  signals.headerCountry = req.headers['x-edge-country'] || null;

  // Signal 3: account-declared jurisdiction from signup/KYC
  // This is the most reliable for authenticated users
  signals.declaredCountry = req.user?.declaredCountry || null;

  // Signal 4: VPN/proxy heuristics — ASN-based, since residential proxies
  // are detectable by the ASNs they route through
  signals.vpnLikely = await checkAsnAgainstVpnList(ip);

  return signals;
}

The key idea: collect signals first, decide second. The decision logic then looks something like:

function shouldBlock(signals, restrictedCountries) {
  // Authenticated user with a declared country we restrict? Always block.
  // This is the strongest signal — they told us themselves.
  if (signals.declaredCountry && restrictedCountries.includes(signals.declaredCountry)) {
    return { block: true, reason: 'declared_jurisdiction' };
  }

  // Two independent network signals agree on a restricted country? Block.
  const networkAgrees =
    signals.ipCountry === signals.headerCountry &&
    restrictedCountries.includes(signals.ipCountry);
  if (networkAgrees) {
    return { block: true, reason: 'ip_and_edge_agree' };
  }

  // VPN detected from a country we restrict, OR VPN with no other signal?
  // Force re-verification rather than silently allowing or blocking.
  if (signals.vpnLikely) {
    return { block: false, requireVerification: true, reason: 'vpn_detected' };
  }

  // IP says restricted but no corroborating signal — soft block with appeal path
  if (restrictedCountries.includes(signals.ipCountry)) {
    return { block: true, reason: 'ip_only', appealable: true };
  }

  return { block: false };
}

Notice what this does that the naive version doesn't:

It distinguishes between "definitely restricted" and "probably restricted"
It gives users a path to appeal a false positive
It treats VPN traffic as a signal to verify, not silently allow
It uses the user's own declared jurisdiction as the strongest signal when available

Getting the client IP right

One footgun worth calling out: extracting the actual client IP. If you naively use req.ip behind a load balancer or CDN, you might get the proxy's IP. And if you blindly trust X-Forwarded-For, anyone can spoof it.

The pattern:

function getClientIp(req) {
  // X-Forwarded-For is a chain: client, proxy1, proxy2, ...
  // Trust only the rightmost N entries where N = number of YOUR trusted proxies
  const forwarded = req.headers['x-forwarded-for'];
  if (!forwarded) return req.socket.remoteAddress;

  const chain = forwarded.split(',').map(s => s.trim());
  const trustedProxyCount = 1; // however many proxies sit in front of you
  const clientIndex = chain.length - 1 - trustedProxyCount;
  return chain[Math.max(0, clientIndex)] || req.socket.remoteAddress;
}

If you're on Express, use app.set('trust proxy', N) with the right number — the docs at expressjs.com cover this in more detail than I will here. The mistake I see most often is trust proxy: true, which trusts everything in the chain and is exploitable.

Prevention: things to bake in from day one

A few things I wish I'd done sooner:

Log every blocked request with the signal breakdown. When compliance asks "how do you verify users aren't in X?", you want auditable logs, not just "we have a check."
Build the appeal flow before you need it. Some percentage of your blocks will be wrong. Users hate hitting a wall with no recourse. A simple "verify with ID" path solves the majority of false positives.
Keep your GeoIP database current. MaxMind's GeoLite2 ships updates twice a week. If yours is six months stale, your accuracy is degrading silently.
Test with VPNs as part of CI. I have a smoke test that hits the staging environment through a few known VPN exit nodes. Catches regressions in the VPN detection logic faster than waiting for a user report.

The broader lesson: any time you're making a regulatory decision based on a network signal, you should assume the signal is partially wrong, partially gameable, and occasionally cached. Build for that, not against it. Your future self — and your compliance team — will thank you.

Why your VPS might be part of a botnet — and how to find out

Alan West — Tue, 26 May 2026 03:02:13 +0000

Last week I got a 3am email from my hosting provider. Subject: "Abuse report — your IP is participating in a DDoS." My first reaction was disbelief. My second was opening a laptop in bed and SSH'ing in like a chump.

This happens more often than people admit. A server you set up six months ago, forgot about, and never patched becomes someone else's attack tool. Recent law enforcement actions against bulletproof hosting operations have made one thing clear — a lot of compromised infrastructure being used in attacks is just neglected developer boxes. Old WordPress installs, exposed Redis, weak SSH keys.

Let's walk through how to actually figure out if your box is one of them, and what to do about it.

The symptoms are quieter than you'd expect

The Hollywood version of a compromised server is CPU pegged at 100% and obvious malware. Real life is duller. Modern attack toolkits are tuned to stay under the radar so the operator can keep using the box.

What you'll actually see:

Slightly elevated baseline CPU (5-15% from nothing)
Unexplained outbound traffic to weird ports
New cron entries you didn't write
SSH login attempts spiking from your own server's outbound logs (it's scanning for the next victim)
Mysterious processes with names like kdevtmpfsi, xmrig, or randomly-generated 6-char names

Step 1: Look at outbound connections first

Ingress filtering is easy. Egress is where most teams have nothing in place, and it's exactly where a compromised box gives itself away.

Start with ss — it's faster than netstat and ships with iproute2 on most modern distros:

# Show all established outbound TCP connections with the owning process
ss -tnpo state established

# Group by remote IP to spot fan-out patterns
ss -tn state established | awk 'NR>1 {print $5}' | cut -d: -f1 | sort | uniq -c | sort -rn | head

If you see hundreds of connections to a single IP, or a wide spray to random IPs on port 22, 23, 80, or 445, that's scanner behavior. Legitimate apps usually talk to a small set of known endpoints.

Next, pull the live process tree. I like pstree -p because it shows parentage — a lot of malware spawns from cron or from a web server worker, and the parent process is the giveaway:

pstree -panu

Look for processes whose parent is cron, sh, or your web server but whose command line is something opaque like a long base64 string or a binary in /tmp or /dev/shm.

Step 2: Check the usual hiding spots

Attackers are creatures of habit. Here's the rapid sweep I run:

# World-writable temp dirs are the #1 dropzone
ls -lat /tmp /var/tmp /dev/shm 2>/dev/null | head -30

# Recently modified files in system bin dirs
find /usr/bin /usr/sbin /usr/local/bin -mtime -30 -ls 2>/dev/null

# Cron entries for every user (not just root)
for u in $(cut -d: -f1 /etc/passwd); do
  echo "--- $u ---"
  crontab -u "$u" -l 2>/dev/null
done

# Systemd timers and services added recently
systemctl list-timers --all
find /etc/systemd/system /lib/systemd/system -mtime -60 -ls

The /dev/shm trick catches a lot of cryptominers — they drop the binary into shared memory because it's tmpfs (no disk writes, less forensic evidence) and runs from RAM.

If you find a suspicious binary, before you delete it, get a hash and check it against VirusTotal:

sha256sum /tmp/.suspicious_binary
# Then upload the hash (not the file) to virustotal.com

Step 3: Trace the entry point with auditd

This is the part most tutorials skip, and it's the most important. Cleaning the malware without knowing how it got in means you'll be doing this exact same dance next week.

Install auditd if you don't already have it, and set up rules to watch the obvious vectors:

# Watch execve syscalls — every program execution
auditctl -a always,exit -F arch=b64 -S execve -k exec_trace

# Watch writes to common malware dropzones
auditctl -w /tmp -p wa -k tmp_writes
auditctl -w /dev/shm -p wa -k shm_writes

# Watch SSH key file changes (very common persistence trick)
auditctl -w /root/.ssh/authorized_keys -p wa -k ssh_keys

Then search the existing logs for evidence:

# Look for shell execution chains from web server users
ausearch -k exec_trace --start recent | grep -E 'uid=(33|www-data|nginx|apache)'

In my 3am incident, this is what cracked it — a PHP file in an abandoned WordPress install was being POSTed to, spawning /bin/sh, which pulled down a payload via curl. Classic webshell chain. The site hadn't been touched in two years.

Step 4: Fix it properly

Don't just kill -9 and move on. The malware will respawn from whatever persistence hook it installed. Order of operations:

Snapshot first. If your provider supports it, take a disk snapshot before you change anything. You may need it for forensics or to satisfy an abuse report.
Cut network egress before killing processes. A common mistake is killing the miner first, which triggers a watchdog reinstall. Block outbound first:

# Drop all outbound traffic except SSH from your management IP
nft add table inet filter
nft add chain inet filter output { type filter hook output priority 0 \; policy drop \; }
nft add rule inet filter output ct state established,related accept
nft add rule inet filter output oifname lo accept
# Add your specific allowlist rules here

Kill the processes and remove persistence. Cron, systemd units, ~/.bashrc lines, /etc/ld.so.preload, modified SSH authorized_keys. Check all of them.
Rotate every credential that touched the box. SSH keys, API tokens in env files, database creds, cloud provider credentials. Assume they're all in someone's wallet now.
Reinstall, don't clean. I know. It's annoying. But a rootkit you didn't find will outlive your cleanup. If the box held anything sensitive, the right move is destroy and rebuild from a known-good config.

Prevention: stop being an easy target

Most compromises I've cleaned up came from a small handful of root causes:

Unattended upgrades disabled. Turn them on. unattended-upgrades on Debian/Ubuntu, dnf-automatic on RHEL-likes. Yes it's a small risk. Running months-old kernels is a much bigger one.
SSH password auth enabled. Set PasswordAuthentication no and use keys. Add fail2ban for the noise.
No egress filtering. Default-deny outbound is annoying to set up but it's the single thing that would have prevented every botnet enrollment I've seen. Even a basic rule blocking outbound to common scan ports (22, 23, 445, 3389) from anywhere but a specific allowlist would catch most of them.
Forgotten services. That staging box. That old CMS. That Redis you exposed for "five minutes" to test something. Run nmap against your own external IPs once a quarter. You will be surprised.

The boring stuff genuinely works. I've stopped chasing exotic hardening guides and just kept a checklist of these basics for every new box. The 3am abuse emails have gone to zero.

How to Fix Tool-Use Loops in Autonomous Coding Agents

Alan West — Tue, 26 May 2026 01:38:16 +0000

Last month I was helping a friend debug their autonomous coding agent. It had been "working" on a task for 47 minutes, burned through roughly twelve bucks in API costs, and somehow ended up exactly where it started. The logs showed it had called read_file on the same five files 23 times.

If you've built or experimented with AI coding agents, you've probably seen something like this. It's not a fun bug to debug — the agent isn't crashing, it isn't erroring, it just... never finishes.

The Problem: Why Agents Loop Forever

Tool-use loops are the most expensive failure mode in agent design. From the outside, the agent looks busy. It's reading files, calling tools, generating thoughts, producing output. But it's not making progress toward the goal.

The shape is almost always the same:

Agent reads file A
Agent realizes it needs context from file B
Reads file B, gets confused by something unexpected
Goes back to file A "to double-check"
Reads file B again because file A didn't have what it needed
Repeat until your wallet cries

I've now seen this in three different agent setups across two side projects and one client engagement. The symptoms are identical every time.

Root Cause: Stateless Decision-Making

The fundamental issue is that the agent's working state looks nearly identical at step N and step N+5. Same task description in the system prompt, same files implicitly available, same general feel of the conversation. So the model — given essentially the same inputs — makes essentially the same decision.

There are three concrete causes worth separating:

No explicit action history. The agent has called read_file("config.yaml") four times, but each turn the model mostly "sees" the latest tool result, not the pattern of what it's already tried.
No reflection step. Nothing in the loop ever asks "am I actually making progress?"
Errors get summarized away. A tool failure gets compressed into a vague "the previous call had an issue" and the model retries with the same broken inputs.

Let's walk through fixing each one.

Step 1: Track Tool Calls Explicitly

Don't rely on the conversation history to encode what's been tried. Build a structured log the model can actually reason about.

from collections import Counter
from dataclasses import dataclass, field
from typing import Any

@dataclass
class ToolCallLog:
    # Counts repeated (tool_name, args) pairs so we can detect loops
    calls: Counter = field(default_factory=Counter)
    history: list = field(default_factory=list)

    def record(self, name: str, args: dict[str, Any], result: str):
        key = (name, _hash_args(args))  # stable hash of args
        self.calls[key] += 1
        self.history.append({"name": name, "args": args, "result_preview": result[:200]})

    def summary_for_model(self) -> str:
        # Surface repeated calls so the model SEES the loop forming
        repeated = [(k, n) for k, n in self.calls.items() if n > 1]
        if not repeated:
            return "No repeated tool calls so far."
        lines = [f"- {name}{args} called {n}x" for (name, args), n in repeated]
        return "Repeated calls detected:\n" + "\n".join(lines)

Then inject log.summary_for_model() into the system prompt every turn. Suddenly the model can see that it's about to call read_file("config.yaml") for the fifth time, and most modern models will course-correct on their own.

Step 2: Add a Loop Detector

Don't trust the model to always notice. Add a circuit breaker:

MAX_IDENTICAL_CALLS = 3
MAX_TOTAL_STEPS = 40

def should_force_reflection(log: ToolCallLog) -> str | None:
    # Return a reflection prompt if we detect a loop, else None
    for key, count in log.calls.items():
        if count >= MAX_IDENTICAL_CALLS:
            name, args = key
            return (
                f"You've called {name} with the same args {count} times. "
                "This is a loop. Stop and explain in one sentence what you "
                "actually need, then choose a different strategy."
            )
    if len(log.history) >= MAX_TOTAL_STEPS:
        return (
            "You've taken many steps without finishing. Summarize what you "
            "know, what you still need, and propose a single next action."
        )
    return None

When this triggers, inject the returned string as a user message before the next model call. I've found this single change cuts wasted tokens by something like half on the workflows I've tested. Your mileage will vary, but the direction is consistent.

Step 3: Force Reflection on a Schedule

Even without a detected loop, models drift on long tasks. A periodic forced reflection helps. The cadence I've landed on is every 8–10 tool calls:

REFLECTION_INTERVAL = 8

def maybe_reflect(step: int, task: str) -> str | None:
    if step > 0 and step % REFLECTION_INTERVAL == 0:
        return (
            f"Pause. Original task: {task}\n"
            "In 3 short bullets, answer:\n"
            "1. What have I actually accomplished?\n"
            "2. What is still blocking completion?\n"
            "3. Is my current approach working, or should I change it?"
        )
    return None

This is borrowed from human pair programming — "hey, where are we?" every so often is healthy.

Step 4: Make Errors Loud

The last fix is the most boring but probably the most important. When a tool fails, don't soften the error message:

def format_tool_error(name: str, args: dict, exc: Exception) -> str:
    # Be specific about what failed. Generic errors invite retries.
    return (
        f"TOOL ERROR: {name} failed with {type(exc).__name__}: {exc}.\n"
        f"Inputs were: {args}.\n"
        "Do NOT retry with identical arguments. Either fix the inputs "
        "or choose a different tool."
    )

The "Do NOT retry with identical arguments" line sounds silly but actually moves the needle. I tested with and without it on the same task three times — without it, the agent retried failing calls about 60% of the time. With it, closer to 10%. Tiny sample size, but the effect was obvious.

Prevention: Design Choices That Help

A few patterns I now reach for by default when building agents:

Cap context per tool. Truncate read_file results to the relevant section instead of dumping whole files. Less noise, more signal.
Use scratchpad files. Give the agent a notes.md it can write to. Externalized memory is cheaper than re-deriving state from chat history.
Separate planning from execution. A small "planner" call that emits a 5-step plan, followed by an executor that follows it, loops far less than a single agent doing both.
Log everything during development. You cannot debug what you cannot see. Persist full tool histories to disk for the first few weeks of any new agent.

None of this is novel — the broader agent research community has been writing about reflection, planning, and memory for a while. But it's easy to skip these when you're hacking together a prototype and assume "the model will figure it out." It won't. Not reliably.

Wrapping Up

Tool-use loops are not a model problem so much as a harness problem. The model is doing exactly what you'd expect given identical inputs every turn. Your job, as the person building the loop around the model, is to make sure the inputs aren't identical — that the agent can see its own history, get nudged when it's stuck, and feel the weight of its errors.

Fix those four things and most of your runaway agent costs go away. The rest is just tuning.

How to Work Around MySQL's View Subquery Limitation (Bug #11472)

Alan West — Tue, 26 May 2026 01:01:45 +0000

That Moment Your VIEW Refuses to Compile

You're refactoring a reporting query. It's gnarly — three layers of aggregation, a couple of LEFT JOINs, the works. You wrap it in a subquery in the FROM clause, run it, get the right numbers. Nice. Now you decide to encapsulate the whole thing in a VIEW so the analytics folks don't have to copy-paste it.

You run CREATE VIEW. MySQL throws this in your face:

ERROR 1349 (HY000): View's SELECT contains a subquery in the FROM clause

If you've worked with MySQL long enough, this error has shown up at least once. It's the symptom of one of MySQL's oldest documented limitations — tracked as Bug #11472 — and developers have been finding workarounds for it for two decades.

I want to walk through why this happens, how to work around it cleanly with the tools that exist today, and what the reportedly-landed fix might change for your codebase.

What's Actually Going On

MySQL's VIEW implementation resolves views in one of two modes: MERGE and TEMPTABLE. MERGE rewrites the view's definition into the calling query. TEMPTABLE materializes the view into a temporary table first.

The catch: a subquery in the FROM clause — what the SQL spec calls a derived table — historically couldn't sit inside a view definition. The parser would let you create some shapes, but as soon as it hit a FROM (SELECT ...) AS something inside CREATE VIEW, you'd get error 1349.

The official VIEW docs describe the restriction. According to community discussion, a fix has reportedly landed — I haven't tested it against the official changelog myself, so I'm hedging there. Either way, the workarounds below are what most production codebases have been using.

The Classic Workaround: Nested Views

The most common trick is to split the view into two. The inner subquery becomes its own view, and the outer view selects from it.

-- Step 1: the inner view does the heavy lifting
CREATE VIEW order_totals_per_customer AS
SELECT
    customer_id,
    SUM(amount) AS total_spent,
    COUNT(*)    AS order_count
FROM orders
WHERE status = 'completed'
GROUP BY customer_id;

-- Step 2: the outer view joins to the inner one
-- (instead of inlining a derived table)
CREATE VIEW customer_revenue_report AS
SELECT
    c.id,
    c.name,
    COALESCE(ot.total_spent, 0) AS lifetime_value,
    COALESCE(ot.order_count, 0) AS orders
FROM customers c
LEFT JOIN order_totals_per_customer ot
    ON ot.customer_id = c.id;

This works. It's also a pain — you now have two views to maintain, and the names tend to multiply when the query is more complex. I've seen schemas with 40+ "helper" views that exist solely to dodge this restriction.

A Cleaner Approach: Common Table Expressions

If you're on MySQL 8.0 or later, CTEs are your friend. They were added in 8.0 (see the WITH syntax docs) and they're allowed inside view definitions.

CREATE VIEW customer_revenue_report AS
WITH order_totals AS (
    SELECT
        customer_id,
        SUM(amount) AS total_spent,
        COUNT(*)    AS order_count
    FROM orders
    WHERE status = 'completed'
    GROUP BY customer_id
)
SELECT
    c.id,
    c.name,
    COALESCE(ot.total_spent, 0) AS lifetime_value,
    COALESCE(ot.order_count, 0) AS orders
FROM customers c
LEFT JOIN order_totals ot ON ot.customer_id = c.id;

Same logic, one view, no extra schema noise. I migrated a reporting pipeline from the nested-view approach to CTEs last year — went from 18 supporting views to four. Made the on-call rotation much happier.

A couple of caveats from that migration:

CTEs in MySQL aren't always inlined the way you'd expect. The optimizer can choose to materialize them, and that may or may not be what you want for a given workload.
If you need recursion (parent/child trees, for example), WITH RECURSIVE works inside views too — just watch the cte_max_recursion_depth setting.

Debugging When You Still Hit Error 1349

If you've wrapped your query in a CTE and you're still hitting the error, there are a few usual suspects:

You're on MySQL 5.7 or earlier. CTEs are 8.0+. There's no backport. Upgrade or stick with the nested-view workaround.
The view definer lacks permission on the underlying tables. This isn't error 1349 specifically, but it shows up at view-creation time and the message can be misleading.
You've got a derived table hiding inside the CTE. A correlated subquery in the SELECT list is fine; a FROM (SELECT ...) inside the CTE body can still trip the historic restriction.

To check which algorithm MySQL picked for your view, query information_schema.VIEWS:

SELECT
    table_name,
    is_updatable,
    -- 'MERGE', 'TEMPTABLE', or 'UNDEFINED'
    algorithm
FROM information_schema.views
WHERE table_schema = DATABASE();

If you see TEMPTABLE where you expected MERGE, that's a hint that something in the view forced materialization — non-deterministic functions, aggregates, DISTINCT, or until recently, a derived table.

What the Fix Might Change

Based on what's circulating in the bug tracker thread, the fix reportedly relaxes the derived-table restriction inside view definitions. I'll be honest — I haven't tested it thoroughly yet, and the exact MySQL version that ships the fix matters a lot for whether you can adopt it.

A few things I'd want to verify before refactoring production code:

Which exact MySQL release contains the change (check the official changelog, not blog posts)
Whether ALGORITHM = MERGE works with derived tables, or if everything falls back to TEMPTABLE
How the optimizer handles the inlined derived table compared to an equivalent CTE

Until you can confirm those on your own deployment, the CTE pattern above is the safer bet. It's been in 8.0 since launch and the behavior is well-documented.

Prevention: Habits That Save Future-You

A few rules I've ended up living by after too many "why doesn't this view work" debugging sessions:

Write the query as a plain SELECT first, get it returning the right rows, then wrap it in a view. Don't debug DDL and query logic at the same time.
Prefer CTEs over nested views when you're on 8.0+. They're easier to read and they don't pollute the schema.
Check algorithm in information_schema. If TEMPTABLE shows up where you didn't expect it, your view won't be updatable and may not perform the way the planner makes it look.
Don't bury joins inside derived tables when a CTE would do the same job. It reads better and the optimizer has more to work with.

Twenty years is a long time for a bug to sit open. The workarounds are well-worn, the tooling around CTEs is solid, and most of the pain this caused has been quietly absorbed into MySQL muscle memory at this point. Worth knowing both the old patterns and the new ones — you'll run into both in any sufficiently old codebase.

Why your browser multitrack audio drifts out of sync (and how to fix it)

Alan West — Tue, 26 May 2026 00:08:02 +0000

If you've ever tried to build anything more ambitious than a single <audio> tag in the browser, you've probably hit this wall: you start two or three audio tracks at "the same time", and within 30 seconds they sound like a drunk wedding band. Drums leading the bass by 80ms, vocals lagging the guitar, the whole thing falling apart.

I hit this exact problem on a project last month — building a small multitrack practice tool for a friend who teaches guitar. First version used <audio> elements and play() calls in a loop. It worked beautifully for about 4 seconds before the tracks started smearing.

Let's dig into why this happens, and how to actually solve it.

The root cause: two clocks, no coordination

The core problem is that the browser has multiple timing systems and they do not agree with each other. When you call audio.play() on an HTMLAudioElement, you're at the mercy of:

The main thread event loop (which can stall during GC, layout, anything)
The media element's internal scheduler (which buffers independently per element)
Date.now() / performance.now() (wall clock, not audio clock)

Each <audio> element schedules itself against its own internal clock. There is no shared timebase. So when you do this:

// This is the trap. Looks correct. Is not.
track1.play();
track2.play();
track3.play();

The three play() calls return immediately, but each element starts producing samples whenever its decoder feels ready. On Chrome the gap might be 5ms. On Firefox under load, 40ms. And the elements keep drifting because they aren't slaved to the same sample clock.

A 1ms drift on a kick drum is audible. A 10ms drift is unusable.

The fix: schedule against the AudioContext clock

The Web Audio API was designed specifically to solve this. The AudioContext exposes a single, sample-accurate clock (context.currentTime), and every source node you create is scheduled against that clock with sub-millisecond precision.

The trick is to decode your audio into AudioBuffer objects up front, then schedule playback at a single future timestamp.

// One context = one shared clock for everything
const ctx = new AudioContext();

async function loadTrack(url) {
  const res = await fetch(url);
  const arrayBuffer = await res.arrayBuffer();
  // decodeAudioData returns a fully decoded buffer in memory
  return await ctx.decodeAudioData(arrayBuffer);
}

const [drums, bass, guitar] = await Promise.all([
  loadTrack('/drums.wav'),
  loadTrack('/bass.wav'),
  loadTrack('/guitar.wav'),
]);

Now here's the part most tutorials get wrong. You don't start the sources at ctx.currentTime. You schedule them slightly in the future, so all three start() calls definitely land before the playback time arrives:

function playSynced(buffers) {
  // 100ms lookahead. Long enough to absorb main-thread jitter,
  // short enough that the user does not perceive delay.
  const startAt = ctx.currentTime + 0.1;

  const sources = buffers.map(buf => {
    const src = ctx.createBufferSource();
    src.buffer = buf;
    src.connect(ctx.destination);
    // All three sources are armed against the SAME future timestamp
    src.start(startAt);
    return src;
  });

  return sources;
}

The magic is in that shared startAt value. The audio thread (which runs at high priority, separate from the main thread) sees three sources all scheduled for the same sample, and starts them in lockstep. No drift. Ever.

Per-track gain, mute, and solo

Once you have the basics, you almost always want per-track volume control. Insert a GainNode between each source and the destination:

function createTrack(buffer) {
  const gain = ctx.createGain();
  gain.connect(ctx.destination);

  return {
    buffer,
    gain,
    // Set volume with a tiny ramp to avoid clicks on sudden changes
    setVolume(value) {
      gain.gain.setTargetAtTime(value, ctx.currentTime, 0.01);
    },
    mute() { this.setVolume(0); },
    unmute() { this.setVolume(1); },
  };
}

Note setTargetAtTime instead of assigning gain.gain.value directly. Direct assignment causes a zipper-noise click because the value jumps between sample frames. The exponential ramp smooths it out over ~10ms.

The gotcha nobody mentions: source nodes are one-shot

This tripped me up for an embarrassing amount of time. An AudioBufferSourceNode can only be start()ed once. Ever. If you stop playback and want to play again, you have to create a new source node:

class Track {
  constructor(ctx, buffer) {
    this.ctx = ctx;
    this.buffer = buffer;
    this.gain = ctx.createGain();
    this.gain.connect(ctx.destination);
    this.source = null;
  }

  play(startAt) {
    // Always create a fresh source — they are disposable
    this.source = this.ctx.createBufferSource();
    this.source.buffer = this.buffer;
    this.source.connect(this.gain);
    this.source.start(startAt);
  }

  stop() {
    if (this.source) {
      this.source.stop();
      this.source.disconnect();
      this.source = null;
    }
  }
}

The AudioBuffer is the expensive thing — decoded PCM data sitting in memory. The source node is cheap, ephemeral plumbing. Treat them differently.

Prevention: a checklist

After shipping a couple of multitrack browser tools, here's what I check before any audio code goes anywhere near production:

One AudioContext per app. Creating multiple contexts means multiple clocks, which defeats the entire point.
Decode once, play many. Hold AudioBuffer objects in memory; never re-decode on each play.
Always schedule with lookahead. Never start(ctx.currentTime). Use at least ctx.currentTime + 0.05 to absorb jitter.
Resume the context on user gesture. Browsers ship AudioContext in suspended state. Call ctx.resume() inside a click handler or playback silently fails.
Use setTargetAtTime for parameter changes. Direct assignment to .value clicks. Ramps don't.
Watch your buffer sizes. A decoded stereo 44.1kHz minute is about 10MB in memory. Long sessions with many tracks add up fast.

The Web Audio API has a steep initial learning curve because it inverts how you usually think about timing — you describe when things should happen, then let the audio thread execute it, instead of imperatively saying do it now. Once that clicks, the drift problems disappear entirely.

For a deeper dive into scheduling patterns, Chris Wilson's A Tale of Two Clocks is still the canonical reference and worth bookmarking. It's the article that finally made all of this make sense to me.

Why LLM Coding Agents Drift on Long Back End Tasks (and How to Fix It)

Alan West — Mon, 25 May 2026 23:42:42 +0000

Last month I spent three days debugging a Django service where the AI agent had written... mostly correct code. The endpoints worked. The tests passed. But somewhere around the fourth file, it had quietly dropped a database transaction wrapper around a multi-step write. By file seven, it had forgotten that one of the models required tenant scoping.

This is constraint decay. And once you start watching for it, you see it everywhere.

What constraint decay actually is

When you hand an LLM agent a backend task, you give it a pile of constraints. Some are explicit (use this ORM, scope by tenant_id, wrap writes in transactions). Some are implicit (auth middleware applies to all routes, errors map to specific status codes). Early in the task, those constraints are fresh in context and the agent honors them.

As the task drags on, something predictable happens. The agent generates more code. That generated code pushes the original constraints further from the attention window. By the time it's writing the eighth function, the original instructions are competing with thousands of tokens of its own output for attention weight. Constraints fade. Output drifts.

I should say upfront: I haven't read every paper on this in detail, and recent work like the Constraint Decay preprint on arXiv is still being discussed. But the phenomenon itself is reproducible at home. Build a long enough agent loop with enough constraints and you'll watch it happen on your own machine.

Root cause: it's not memory, it's signal-to-noise

The first instinct when you see drift is "well, just put it in the context window." Modern models have huge context windows. But window size isn't really the issue.

The issue is that attention is a softmax over the entire context. When your system prompt is 200 tokens and the surrounding generated code is 8000 tokens of similar-looking function names, types, and patterns, the relative weight on the constraint shrinks. The constraint is present. It's just not salient.

You can verify this with a quick experiment. Give an agent a constraint like "every database write must go through audit_log()." Have it write five files. By file four, direct writes will often sneak in. Re-prompting with just the original constraint restores compliance immediately. The constraint never left the model — the model just stopped weighting it.

Step-by-step fix

Here's the pattern I've landed on after maybe a dozen agent-driven projects this year. It's not perfect. It does cut drift significantly.

1. Externalize constraints as checked artifacts

Don't rely on the agent remembering. Make the constraint a thing you can mechanically verify.

# constraints.py — source of truth for cross-cutting rules
INVARIANTS = {
    "tenant_scoping": {
        # any query on a multi-tenant model must include tenant_id
        "applies_to": ["User", "Invoice", "Subscription"],
        "check": "every .filter() / .get() includes tenant_id",
    },
    "audit_log": {
        # mutations to sensitive tables must be logged
        "applies_to": ["billing", "auth"],
        "check": "calls audit_log() before commit",
    },
}

Then write a small AST-walking linter that checks these. Now the constraint has a teeth-having enforcer that doesn't decay.

2. Chunk the work, refresh between chunks

Long single-shot generation is where decay is worst. Break the task into chunks, and between chunks, replay the relevant constraints.

def run_agent_task(task, constraints):
    chunks = decompose_task(task)
    results = []
    for chunk in chunks:
        # Constraints go near the top, fresh, every chunk
        prompt = build_prompt(
            constraints=constraints,
            prior_summary=summarize(results),  # summary, not full output
            current_chunk=chunk,
        )
        result = agent.run(prompt)
        results.append(result)
    return results

The key move is summarize(results) instead of dumping all prior code. A summary preserves the architectural decisions without crowding the constraint with thousands of code tokens.

3. Use a separate constraint-check pass

After every chunk, run a separate, narrow LLM call whose only job is to check the new code against the constraints. Single responsibility, fresh context.

def check_chunk(generated_code, constraints):
    prompt = (
        "Check this code against the constraints below. "
        "For each constraint, answer PASS or FAIL with one line of evidence.\n\n"
        f"CONSTRAINTS:\n{format_constraints(constraints)}\n\n"
        f"CODE:\n{generated_code}"
    )
    return narrow_model.run(prompt)  # smaller, cheaper model is fine

This is much more reliable than asking the main agent to self-check, because the checker isn't carrying the cognitive load of generation. Its context is short, its attention is undivided.

4. Make violations fail loud

When the checker finds a violation, don't try to "patch" the offending file. Roll back the chunk and regenerate with the violated constraint pinned at the very top — sometimes repeated. Repetition is ugly but it works. Models weight constraints that appear multiple times more heavily.

def regenerate_with_emphasis(chunk, violations):
    emphasized = "\n\n".join(
        f"CRITICAL CONSTRAINT (do not violate): {v}" for v in violations
    )
    # Yes, we repeat. Yes, it helps.
    return agent.run(emphasized + "\n\n" + chunk.prompt + "\n\n" + emphasized)

5. Keep human review on the constraint surface, not the code

This is the part people skip. You don't need to review every line the agent writes. You need to review the constraint set and the checker. If those two are correct, drift is bounded.

I have a habit now of starting every agent project by writing the constraints file first. Before any code. It feels weird because you're writing rules against code that doesn't exist yet, but it forces you to articulate the invariants up front, while you're still thinking clearly.

Prevention: design for bounded drift

A few patterns that keep the problem small in the first place:

Prefer narrow tasks. "Add a new endpoint" is bounded. "Build the whole admin panel" is not. Decay scales with task length, so shorter tasks decay less.
Use typed interfaces aggressively. When the agent has to satisfy a type signature, the type acts as a local, always-visible constraint. Tools like mypy or TypeScript catch a surprising amount of drift for free.
Lean on existing scaffolding. If your codebase has a BaseRepository that already enforces tenant scoping, the agent inherits the constraint by inheritance. The framework remembers what the agent forgets.
Don't trust passing tests. An agent that wrote both the code and the tests has aligned them to each other. Run the actual app. Hit the endpoints with curl. Check the database directly.

The honest summary: LLM agents are fantastic at the first hour of a task and progressively worse at the fourth. If you architect your workflow around that reality — short chunks, external constraints, mechanical checking — the drift becomes manageable. If you treat the agent like a junior dev who remembers everything you said, you'll be debugging silent constraint violations for days.

Three days, in my case. I've structured my projects differently since.

How to Fix Context Loss in Multi-Step AI Agent Workflows

Alan West — Mon, 25 May 2026 19:56:13 +0000

I spent last weekend debugging an agent that kept forgetting what it was doing. It would happily call three tools in sequence, then on the fourth one... blank stare. Wrong arguments. Hallucinated file paths. The classic "who am I, where am I, what was I doing" moment.

If you've built anything that chains together more than a couple of tool calls, you've probably seen this. The agent starts strong, makes a plan, and then somewhere around step three or four it starts looking like it took a nap and woke up in someone else's session.

Let's talk about why this happens and how to actually fix it.

The Problem: Your Agent Has Amnesia

Here's the symptom. You build an agent skill that's supposed to:

Read a config file
Validate it against a schema
Apply a transformation
Write the result back

Steps 1 and 2 work fine. By step 3, the agent has lost track of the variable names it pulled out in step 1. By step 4, it's writing to the wrong path entirely. You add logging. You add retries. You add a stern system prompt that says "DO NOT FORGET THE FILENAME." Nothing helps.

I ran into this on a code review agent I was building. It would read a file, identify three issues, then when asked to apply fixes, it would invent issues that didn't exist. The original analysis just... evaporated.

The Root Cause: Tool Calls Are Stateless

Here's the thing nobody tells you when you start building agents: the model itself has no memory between tool calls. The only "memory" is the message history you keep sending back.

When the agent calls a tool, here's what happens:

The model sees the entire message history
It picks a tool and arguments based on what it sees
The tool runs, returns a result
The result gets appended to the history
The model sees the updated history and picks the next move

So if step 1 returned a giant blob of JSON, step 2 returned another blob, step 3's context window is now mostly old tool output. The model's attention is spread thin. Important details — like the filename you extracted in step 1 — get drowned in noise.

This isn't a bug. It's how the architecture works. Each turn is a fresh inference over a growing message log.

Step-by-Step Solution

Step 1: Externalize State Into a Scratchpad

The single most effective fix I've found is to give the agent an explicit "scratchpad" — a small structured object the agent reads from and writes to between steps. Don't rely on the model to remember things. Make it write them down.

Here's a minimal version in Python:

from dataclasses import dataclass, field, asdict
import json

@dataclass
class AgentScratchpad:
    # Persistent across all tool calls in this run
    goal: str = ""
    facts: dict = field(default_factory=dict)
    decisions: list = field(default_factory=list)

    def to_prompt(self) -> str:
        # Inject this back into the system prompt every turn
        return f"CURRENT STATE:\n{json.dumps(asdict(self), indent=2)}"

# Two tools the agent uses to manage its own memory
def remember(scratchpad, key: str, value: str):
    scratchpad.facts[key] = value
    return f"Saved {key}"

def decide(scratchpad, decision: str):
    scratchpad.decisions.append(decision)
    return "Decision logged"

The key move: you expose remember and decide as tools the agent can call. After step 1, it calls remember("config_path", "/etc/app.yml"). By step 4, that fact is still right there in the system prompt, front and center.

Step 2: Compress Old Tool Output

The second problem is sheer volume. A read_file call on a 500-line config returns 500 lines. Three of those calls and the context is mostly raw file dumps.

Fix it by replacing old tool results with summaries once they're no longer the active focus:

def compress_history(messages, keep_recent=3):
    # Keep system message and recent N turns verbatim
    # Summarize everything in between
    if len(messages) <= keep_recent + 1:
        return messages

    head = messages[0:1]  # system prompt
    tail = messages[-keep_recent:]
    middle = messages[1:-keep_recent]

    # Replace verbose tool results with one-line summaries
    summary = summarize_turns(middle)  # your own summarizer
    return head + [{"role": "user", "content": summary}] + tail

I usually run this every 5-6 turns. The agent still has access to the scratchpad for facts that matter, and the recent turns for the immediate context. The middle gets squashed.

Step 3: Validate State Before Every Critical Tool Call

This is the prevention piece. For any tool that mutates something — writes a file, hits an API, runs a migration — wrap it in a guard that checks the scratchpad first:

def write_file_safe(scratchpad, path: str, content: str):
    # Refuse to write to a path the agent never recorded reading
    if path not in scratchpad.facts.get("known_paths", []):
        return (
            f"ERROR: {path} was never read in this session. "
            f"Read the file first or update scratchpad."
        )
    return do_write(path, content)

This catches the hallucination case cleanly. If the agent invents a path, the tool rejects it and tells the agent why. Nine times out of ten, the next turn the agent corrects itself and reads the right file.

A Pattern I've Started Using

After migrating three agents to this approach, I converged on a structure that looks roughly like:

System prompt holds the goal and tool definitions (static)
Scratchpad holds extracted facts and decisions (mutable, re-injected each turn)
Recent messages hold the last few turns verbatim (rolling window)
Summary replaces older middle turns (compressed)

The agent never needs to scan 50 turns of history to find a filename. It looks at the scratchpad, sees the fact, and acts on it.

Prevention Tips

A few things I wish I'd done from the start:

Design state-aware tools first. Every tool should either read from or write to the scratchpad, not just return data into the void.
Test long chains. Most agent bugs only show up after 5+ tool calls. Write integration tests that force the agent through a long workflow and assert on final state.
Log token counts per turn. When you see context creeping past 60-70% of the window, that's where attention starts to degrade. Compress earlier than you think you need to.
Don't trust the model to remember. If a fact matters in step 5, write it to the scratchpad in step 1. Treat the model like a brilliant intern with severe short-term memory loss.

The broader lesson here: agent reliability isn't really about prompt engineering. It's about state management. The model is doing inference; you're doing the bookkeeping. Get the bookkeeping right and the agent stops forgetting things.

I haven't tested the scratchpad pattern on workflows longer than about 20 tool calls, so your mileage may vary at extreme scale. But for the typical 3-10 step agent workflow, this fixed every context-loss bug I had.

How to do partial page updates without shipping a framework

Alan West — Mon, 25 May 2026 19:01:36 +0000

The problem: swapping part of a page is harder than it should be

You click a "Load more" button. Behind the scenes, your app fetches some HTML and shoves it into a <div>. Simple, right?

Except it never stays simple. You hit edge cases with orphaned event listeners, focus jumping around, scripts that mysteriously don't execute, and weird flashes when you replace a chunk of DOM.

I ran into this again last month while refactoring a dashboard. The team had jQuery doing partial swaps for the better part of a decade and finally wanted out. The "modern" alternatives all looked great in demos. Each one came with its own quirks.

Why the web makes this awkward

HTML, at its core, doesn't have a primitive for "replace this region with the contents of that response." You can navigate the whole page (<a href>, <form action>), or you can call fetch() and write the result into innerHTML yourself. There's no middle ground built into the platform.

That gap is why entire categories of tools exist:

HTMX turns attributes into AJAX swaps
Turbo (Hotwire) wraps regions in <turbo-frame> and intercepts navigation
React/Vue/Svelte rebuild the page from a virtual representation
Unpoly and friends do progressive enhancement around forms

Each of them is solving a problem the browser itself never tried to solve.

A quick tour of the pain

Here's the typical "just fetch and replace" approach:

// Replace #content with HTML from the server
async function loadFragment(url) {
  const res = await fetch(url);
  const html = await res.text();
  document.querySelector('#content').innerHTML = html;
}

Looks fine. But:

Inline <script> tags in html won't run (a long-standing quirk of innerHTML).
Any event listeners attached to the old #content children are gone.
If the user had focus inside the replaced region, it's lost.
Custom elements may re-run constructors in surprising ways.

After getting bitten by all of these in production, most teams reach for a library.

How HTMX solves it today

HTMX handles a lot of this for you with declarative attributes. The swap target and strategy live in the markup:

<button
  hx-get="/api/comments?page=2"
  hx-target="#comments"
  hx-swap="beforeend"
>
  Load more
</button>

<ul id="comments">
  <!-- existing items -->
</ul>

It handles script execution, optional out-of-band swaps, focus preservation via hx-preserve, and history integration. Worth reading the official docs if you haven't.

Turbo Frames, briefly

If you're in the Rails/Hotwire world, <turbo-frame> does similar work by wrapping regions and intercepting links inside them:

<turbo-frame id="comments" src="/posts/42/comments">
  Loading…
</turbo-frame>

Both approaches work. Both require shipping a library. Both reinvent things the browser arguably should handle itself.

What Chrome is reportedly exploring

According to the Chrome developers blog post on declarative partial updates, there's an early-stage proposal to give the platform a native way to express "replace this region with what comes back from the server." I haven't shipped anything against the proposal yet — at the time of writing it reads as an explainer, not a stable API — but the direction is interesting.

The general idea, as I read it, is that you describe the swap declaratively in markup and the browser does the fetch and DOM update for you. Think of it as the platform absorbing patterns that libraries like HTMX have demonstrated work well.

If you want to follow along officially, the right places to watch are:

The Chrome status entries for the proposal
The WHATWG repos where standardization discussion happens
The Web Platform Tests repo once an implementation lands

I would resist building anything load-bearing on a proposal-stage API. The shape will almost certainly shift.

What to do in the meantime

If you're feeling the pain today, here's the order I'd try things in:

Use the simplest tool that fits. If you only need partial updates in two places, hand-rolled fetch plus replaceChildren is fine.
Reach for HTMX or Turbo if you're doing this everywhere. Don't build your own framework. I've watched two teams try; both ended up with a worse HTMX.
Keep server-rendered HTML as the source of truth. Returning JSON and reconstructing markup on the client is what got us into the SPA mess in the first place.

Here's a safer vanilla pattern I lean on when a library would be overkill:

async function swapFragment(targetSelector, url) {
  const res = await fetch(url, { headers: { Accept: 'text/html' } });
  if (!res.ok) throw new Error(`Fragment fetch failed: ${res.status}`);

  const html = await res.text();
  const template = document.createElement('template');
  template.innerHTML = html; // parsed in an inert context, not executed

  const target = document.querySelector(targetSelector);
  // replaceChildren preserves more state than reassigning innerHTML
  target.replaceChildren(...template.content.childNodes);
}

A few notes on why this is less awful than the naive version:

<template> parses HTML in an inert context, so custom elements don't upgrade twice.
replaceChildren is generally kinder to scroll position and selection than overwriting innerHTML.
We still need to manually re-run any inline scripts the server sends. Honestly, I just don't.

Prevention tips so this doesn't haunt you again

A handful of habits that have saved me grief:

Pick one swap strategy per project. Mixing HTMX, Turbo, and hand-rolled fetches makes debugging miserable.
Test focus, scroll, and selection as part of normal QA — not just "did the right content appear."
Treat the response shape as a contract. If one endpoint returns JSON for some callers and HTML for others, you've doubled your test surface.
Watch the standards work. If a declarative API does land in browsers, your migration story will be much smoother if your current solution is markup-first rather than imperative JS scattered across the codebase.

Wrapping up

Partial page updates are one of those problems that look tiny from the outside and turn into a tar pit on the inside. Libraries have papered over the gap for years and done a respectable job. If browsers eventually expose a native primitive for this, a lot of glue code will go away. Until then: pick a sane tool, keep markup as the source of truth, and don't roll your own. I've tried. It's not worth it.

Migrating off Google Analytics: Umami vs Plausible vs Fathom

Alan West — Mon, 25 May 2026 18:39:21 +0000

I've been on a slow march away from Google Analytics for about two years now. Three client projects, one personal blog, and a small SaaS — all moved over to privacy-focused alternatives. The decision usually starts with a cookie banner complaint and ends with someone asking "wait, why are we sending visitor data to an ad company again?"

If you're thinking about making the switch, this post walks through the three options I've actually shipped to production: Umami, Plausible, and Fathom. I'll cover what each does well, where they stumble, and what the migration actually looks like.

Why bother migrating at all?

The usual reasons, in roughly the order clients bring them up:

GDPR/ePrivacy compliance without the cookie banner gymnastics
Page weight — the GA4 script weighs in around 50KB+ gzipped; the alternatives here are all under 3KB
Dashboard sanity — GA4's UI is, charitably, an acquired taste
Data ownership — especially relevant if you self-host

None of these are reasons to migrate a high-traffic e-commerce site overnight. But for content sites, marketing pages, and most SaaS dashboards, the tradeoff is usually worth it.

The contenders

Here's the honest side-by-side. I've used all three in production within the last 18 months.

Feature	Umami	Plausible	Fathom
License	MIT	AGPL-3.0	Proprietary (Fathom Lite is MIT)
Self-host	Yes (free)	Yes (Community Edition)	No (hosted only)
Hosted option	Yes (Umami Cloud)	Yes	Yes (only option)
Script size	~2KB	<1KB	~2KB
Cookies	None	None	None
Database	MySQL or PostgreSQL	PostgreSQL + ClickHouse	N/A (managed)
Built with	Next.js / Node	Elixir / Phoenix	Closed source

A few clarifications since this stuff changes:

Fathom Analytics (the paid product) is closed source. There's an older project called Fathom Lite that's MIT licensed, but it hasn't been actively developed in years. Don't confuse them.
Plausible Community Edition is the self-host option. The hosted version has extra features that aren't in CE — check their docs before assuming parity.
Umami is the most permissive license-wise, which matters if you want to embed it in a commercial product.

What the tracking code looks like

This is roughly what you replace your GA4 snippet with. All three are a single script tag, which is part of the point.

Umami (self-hosted or cloud):

<!-- Drop this in your <head>, replace with your own website ID -->
<script
  defer
  src="https://your-umami-instance.com/script.js"
  data-website-id="abc123-your-id-here"
></script>

Plausible:

<script
  defer
  data-domain="yourdomain.com"
  src="https://plausible.io/js/script.js"
></script>

Fathom:

<script
  src="https://cdn.usefathom.com/script.js"
  data-site="ABCDEFGH"
  defer
></script>

Notice none of them ask you to configure consent mode, set up data streams, or wire up tag manager. That's mostly the appeal.

Tracking a custom event

This is where the APIs diverge a bit. Here's the same "button clicked" event in each:

// Umami — global function attached to window
window.umami.track('signup-clicked', { plan: 'pro' });

// Plausible — same pattern, slightly different shape
window.plausible('Signup', { props: { plan: 'pro' } });

// Fathom — uses trackEvent
window.fathom.trackEvent('signup clicked');

Fathom's custom event API is the most limited of the three — no arbitrary properties on events in the same way. If you need rich event metadata, Umami and Plausible are better fits.

Self-hosting Umami in about 10 minutes

This is the workflow I've used for the last two client deployments. Assumes Docker is installed.

# Grab the official compose file
curl -o docker-compose.yml \
  https://raw.githubusercontent.com/umami-software/umami/master/docker-compose.yml

# Generate a hash salt for sessions
openssl rand -base64 32

# Edit docker-compose.yml: set DATABASE_URL and APP_SECRET
# Then bring it up
docker compose up -d

The default admin login is admin / umami — change that immediately. Then point a reverse proxy at port 3000 and you're done. I've run this on a $6/month VPS handling a few hundred thousand pageviews per month without trouble.

Plausible CE is similar but heavier — it needs ClickHouse, which is a bigger commitment if you're not already running it. For small/medium sites, Umami is the easier self-host story.

Migration steps (the boring but important part)

Here's the rough order I follow when moving a site off GA4:

Stand up the new analytics tool and verify it's collecting data
Run both in parallel for 2-4 weeks — you want to see how the numbers compare before you cut over
Export your historical GA data (BigQuery export if you have GA4; the UI export is rough)
Document the metric differences for stakeholders — privacy-focused tools count visitors differently, and your numbers WILL drop
Remove the GA snippet and any GTM containers that only existed to feed it

That last step about numbers dropping — be ready for it. Without cross-site cookies, returning visitors get counted as new more often. Bot filtering is also different. I've seen drops of 15-30% in reported sessions, and it usually has nothing to do with actual traffic.

So which one should you pick?

My actual recommendations after shipping all three:

Pick Umami if you want to self-host on a budget, value the MIT license, or need richer custom events. It's my default recommendation for technical teams.
Pick Plausible if you want a polished hosted experience and don't mind paying, or if you're already running ClickHouse and want a more battle-tested backend. The Plausible team also publishes a lot of useful research on web analytics.
Pick Fathom if you want zero infrastructure and a simple dashboard, and don't need self-hosting. It's the most "set it and forget it" of the three.

One caveat about hedging your bets

I ran Plausible and Umami side-by-side on a client site for a month last year. The visitor numbers were within ~3% of each other, which was reassuring. They're both counting things in roughly the same way — the differences come down to how each handles edge cases like prerendering and ad blockers.

If you want to verify the methodology, both projects publish their counting logic. Umami's docs are at umami.is/docs and Plausible's data policy is at plausible.io/data-policy. Worth reading before you commit to either.

The TL;DR: any of these will be a meaningful upgrade over GA4 for most sites. The migration is genuinely not that hard. The hardest part is convincing whoever owns the marketing dashboard that the numbers are different but not wrong.

Why Your PyTorch Training Crawls on a Beefy GPU (And How to Fix It)

Alan West — Sun, 24 May 2026 22:36:09 +0000

Last month I was helping a friend debug a training loop that was running at maybe 15% GPU utilization on an A100. Fifteen percent. On a card that costs more than my first car. He'd already tried bumping the batch size, swapping the optimizer, and rewriting the data loader — nothing moved the needle.

This is one of those frustrating problems where the obvious knobs do nothing, because the obvious knobs aren't where the bottleneck lives. So let's actually walk through how to figure out why your model is slow, instead of just throwing batch sizes at the wall.

The three regimes nobody tells you about

When a deep learning workload is slow, it's almost always slow for one of three reasons. Horace He laid this out really clearly in his "Making Deep Learning Go Brrrr From First Principles" post back in 2022, and the framing has stuck with me ever since:

Compute-bound — you're actually saturating the matmul units. Rare. Usually only happens with huge dense layers.
Memory-bandwidth-bound — the GPU is mostly waiting on data to move between HBM and the SMs. Way more common than people realize.
Overhead-bound — Python, the framework dispatcher, or kernel launch latency is dominating. Death by a thousand papercuts.

The punchline: most "my model is slow" problems are not compute-bound, even though that's where everyone instinctively looks first. If you're running a transformer with a bunch of small ops between the big matmuls, you're probably stuck in regime two or three.

Step 1: Figure out which regime you're in

Don't guess. Profile. PyTorch's built-in profiler will tell you most of what you need:

import torch
from torch.profiler import profile, ProfilerActivity

model = MyModel().cuda()
x = torch.randn(32, 3, 224, 224, device='cuda')

# Warm up — first iterations include cudnn autotuning and allocator setup
for _ in range(5):
    model(x)

with profile(
    activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA],
    record_shapes=True,
) as prof:
    for _ in range(10):
        out = model(x)
        out.sum().backward()

# Sort by self CUDA time to see what's actually burning GPU cycles
print(prof.key_averages().table(sort_by='self_cuda_time_total', row_limit=20))

What you're looking for:

If a few big gemm / conv kernels dominate → likely compute-bound, and you should be happy.
If you see a sea of tiny kernels (add, mul, relu, layer_norm components, etc.) eating real time → memory-bandwidth-bound.
If CPU time is way higher than CUDA time, or kernel launches are spaced out with gaps → overhead-bound.

For the friend's model, the profile showed hundreds of tiny pointwise kernels per step. Classic memory bandwidth problem.

Step 2: The arithmetic intensity check

Here's the back-of-the-envelope check that explains why small ops are murder. For each operation, ask: how many FLOPs am I doing per byte of memory I touch?

A modern GPU like an A100 does roughly 312 TFLOPs of fp16 matmul but only has about 2 TB/s of HBM bandwidth. That's a ratio of ~150 FLOPs per byte. If your operation does fewer FLOPs per byte than that, you're memory-bound — full stop. No amount of bigger batches will help if the math isn't there.

A pointwise relu on an fp32 tensor? You read 4 bytes, write 4 bytes, do 1 FLOP. That's 0.125 FLOPs per byte. You are wildly memory-bound. The GPU spends 99% of its time waiting on memory and 1% doing the actual work.

A dense matmul on big enough matrices? Hundreds of FLOPs per byte. Now you're cooking.

Step 3: Fuse the small stuff

The fix for memory-bandwidth-bound code is almost always operator fusion. Instead of running ten separate kernels that each round-trip through HBM, you run one kernel that keeps intermediate values in registers or shared memory.

The easiest win in modern PyTorch is torch.compile:

import torch

model = MyModel().cuda()

# mode='reduce-overhead' uses CUDA graphs to also chip away at launch overhead
# mode='max-autotune' spends more time compiling but can fuse more aggressively
compiled = torch.compile(model, mode='reduce-overhead')

# First call is slow — it's tracing and compiling
_ = compiled(x)

# Subsequent calls hit the cached compiled graph
for _ in range(100):
    out = compiled(x)

I've seen this give 1.5x–3x speedups on transformer-ish workloads with basically zero code changes. Your mileage varies a lot based on how dynamic your shapes are — if every batch has a different sequence length, you'll trigger recompiles and lose most of the win. See the torch.compile docs for the dynamic-shape options.

If you need more control, you can write fused kernels yourself in Triton. For a pointwise chain, it's usually not worth it — torch.compile will fuse those for you. For attention or other patterns with cross-element communication, hand-written kernels (or things like FlashAttention) are still where the big wins live.

Step 4: Crush overhead with CUDA graphs

If your profile shows lots of small gaps between kernels on the GPU timeline, you're overhead-bound. Each kernel launch has fixed CPU-side cost — Python, the dispatcher, CUDA itself. With small kernels, that overhead can be bigger than the kernel runtime.

CUDA graphs let you record a sequence of kernel launches once and replay them as a single submission:

import torch

model = MyModel().cuda()
static_input = torch.randn(32, 3, 224, 224, device='cuda')

# Warm up on a side stream before capture
s = torch.cuda.Stream()
s.wait_stream(torch.cuda.current_stream())
with torch.cuda.stream(s):
    for _ in range(3):
        static_output = model(static_input)
torch.cuda.current_stream().wait_stream(s)

# Capture the graph
g = torch.cuda.CUDAGraph()
with torch.cuda.graph(g):
    static_output = model(static_input)

# To run: copy new data into static_input in place, then replay
for batch in dataloader:
    static_input.copy_(batch)  # in-place copy, same buffer
    g.replay()
    # static_output now holds the result

The big gotcha: input tensors have to live at the same memory addresses each call. You're reusing buffers, not allocating new ones. That's why we copy_ instead of reassigning. torch.compile(mode='reduce-overhead') does basically this for you under the hood, which is why I usually reach for that first.

Prevention tips

A few habits that have saved me a lot of grief:

Profile before optimizing. Always. I've wasted entire afternoons "optimizing" things that were 2% of the runtime.
Watch your shapes. Dynamic shapes break torch.compile caches and CUDA graphs. If you can pad to a few bucket sizes, do it.
Stop sprinkling .cpu() and .item() calls. Each one forces a sync and stalls the pipeline. If you're doing it inside the training loop for logging, batch it up.
Check nvidia-smi while training. If utilization is below ~70%, something's wrong. That's your signal to break out the profiler.
Read the assembly when it really matters. For hot kernels, Triton lets you dump the PTX and see what actually got generated. Sometimes the autoscheduler does something silly.

The meta-lesson here is that GPU performance is a first-principles problem, not a vibes-based one. Once you know whether you're starved for FLOPs, bandwidth, or launches, the fix usually becomes obvious. The frustrating part is just resisting the urge to skip the profiling step.

How to sandbox AI coding agents without crippling them

Alan West — Sun, 24 May 2026 20:05:55 +0000

The Problem: Your AI Agent Has Root

A few months back I was helping a team set up a self-hosted AI coding agent. Standard setup — an LLM with tool access, running on a shared dev server, able to read files, execute commands, hit APIs. The usual.

Then someone ran a prompt that included pasted output from an untrusted webpage. The agent dutifully interpreted some embedded instructions and started rm -rf'ing a directory it had no business touching.

Nothing critical was lost. But it could have been.

This is the dirty secret of running agents that execute code — by default, they run with whatever permissions your process has. If that process is your dev environment, your agent has access to your SSH keys, your cloud credentials, your git history. Everything.

Let me walk through how to actually sandbox these things properly.

Why "Just Use Docker" Isn't Enough

The obvious answer is to stick the agent in a container. And yes, that's a start. But naive Docker setups still have:

Root inside the container by default (escapable through several known paths)
Full network access to your internal services
Bind mounts you didn't think hard enough about
No syscall filtering — kernel exploits exist

I've seen "sandboxed" setups where docker.sock was mounted in for convenience. That's not a sandbox. That's a hot tub.

The Layered Approach

The way I've come around to thinking about this: defense in depth. Each layer assumes the previous one was bypassed.

Layer 1: Drop root

Containers should not run as root. Basic, but skipped constantly.

FROM ubuntu:22.04

# Dedicated user, no sudo, no shell escalation
RUN useradd -m -s /bin/bash agent

# Switch BEFORE any app setup so caches/files are owned correctly
USER agent
WORKDIR /home/agent

COPY --chown=agent:agent ./app /home/agent/app

Layer 2: User namespaces

Even as non-root inside the container, you want the container's UIDs remapped on the host. So even if the agent somehow becomes root in the container, it's an unprivileged UID on the outside.

Configure it in /etc/docker/daemon.json:

{
  "userns-remap": "default"
}

After restarting the daemon, container UIDs get shifted to a high host range. A "root" process inside has zero privileges against the host filesystem. See the Docker user namespace docs for the full setup.

Layer 3: Seccomp filtering

This is the layer most people skip. seccomp lets you whitelist syscalls — so even if the agent compromises the container, it can't make syscalls you haven't allowed.

Docker ships a default seccomp profile that blocks around 40 dangerous syscalls. For agent workloads I tighten it further:

{
  "defaultAction": "SCMP_ACT_ERRNO",
  "syscalls": [
    {
      "names": [
        "read", "write", "open", "openat", "close",
        "stat", "fstat", "lstat", "mmap", "brk",
        "rt_sigaction", "execve", "exit", "exit_group",
        "futex", "clone", "fork", "wait4"
      ],
      "action": "SCMP_ACT_ALLOW"
    }
  ]
}

Run it like this:

docker run \
  --security-opt seccomp=./agent-seccomp.json \
  --security-opt no-new-privileges \
  --cap-drop=ALL \
  agent-image

--cap-drop=ALL strips every Linux capability. --no-new-privileges blocks setuid binaries from elevating. Together they shrink the attack surface inside the container down to almost nothing.

Layer 4: Network egress control

Agents need to make HTTP calls. They do not need to scan your internal network.

The cleanest pattern I've found is routing the container through a proxy that whitelists destinations:

# docker-compose.yml
services:
  agent:
    image: agent-image
    # Agent shares the proxy's network namespace — no direct egress
    network_mode: "service:proxy"

  proxy:
    image: nginx:alpine
    volumes:
      - ./proxy.conf:/etc/nginx/nginx.conf:ro
    networks:
      - egress

networks:
  egress:
    driver: bridge

The proxy allows only the endpoints the agent legitimately needs. The agent has no network interface of its own — every packet has to go through nginx, which has to recognize the host.

Layer 5: Filesystem isolation

Mount points are where I see the most mistakes. The agent needs to work on code, but the principle is: mount exactly what's needed, read-only where possible, and never anything sensitive.

docker run \
  --read-only \                              # Root FS is immutable
  --tmpfs /tmp:size=100M \                   # Scratch space, capped
  -v "$PROJECT_DIR:/workspace:rw" \          # The actual work dir
  -v "$PROMPT_FILE:/input/prompt:ro" \       # Read-only inputs
  agent-image

Notice what is NOT mounted: no ~/.ssh, no ~/.aws, no docker.sock, no parent directories that happen to contain a .env file.

Handling Multi-Session Workloads

If multiple developers share the agent infrastructure, isolation between sessions becomes its own problem. The fix is straightforward — one container per session, lifecycle tied to the session.

import uuid
import subprocess

def start_agent_session(user_id: str, project_path: str) -> str:
    session_id = str(uuid.uuid4())
    container_name = f"agent-{user_id}-{session_id}"

    subprocess.run([
        "docker", "run", "-d",
        "--name", container_name,
        "--rm",                              # Auto-cleanup on stop
        "--memory", "2g",                    # Hard memory cap
        "--cpus", "1.0",                     # CPU quota
        "--pids-limit", "100",               # Prevent fork bombs
        "-v", f"{project_path}:/workspace",
        "agent-image",
    ], check=True)

    return session_id

The cgroup limits (--memory, --cpus, --pids-limit) are the unsung heroes. Without them, one runaway agent can take down the host. I learned this one the hard way after an agent got stuck in a loop spawning subprocesses.

Prevention Tips

A few things I've learned that weren't obvious to me at first:

Treat the agent's environment as untrusted. Anything in its filesystem or env vars can be exfiltrated via prompt injection.
Audit your mounts every single time. Bind mounts are the #1 source of escapes I've actually witnessed.
Log every command the agent runs. When something goes wrong, you'll want the trail.
Set timeouts on everything. Agents that should take 30 seconds sometimes try to run for 30 hours. Kill them.
Use ephemeral containers. Reusing the same container across sessions invites state pollution and credential leakage between users.

The mental shift that helped me — stop thinking of the agent as "code I trust running on infra I trust." Think of it as a stranger you handed a terminal. Then design accordingly.

The layers above won't make an agent invulnerable. But they'll turn a single bad prompt from a catastrophe into a footnote.