DEV Community: Edy Cu

Sampling rate is a correctness property, not a performance knob

Edy Cu — Sun, 30 Aug 2026 23:33:18 +0000

In 1997 a Pokémon episode aired in Japan with a four-second red-and-blue strobe. 685 children went to hospital that night. The regulations that followed — Ofcom Broadcasting Code 2.12, ITU-R BT.1702 — are why UK and Japanese broadcast deliveries have to clear a photosensitive-epilepsy check before transmission.

I built a screening tool for that check. Live at flashframe-production.up.railway.app, code at github.com/edycutjong/flashframe. Three synthetic test clips ship with it, so you can click one and watch it run without an upload or an API key.

This post isn't about the tool. It's about the fact that I shipped two completely different bugs that turned out to be the same bug, and I didn't recognise the second one even after fixing the first.

The rule is a window function

The regulation says: no more than three flashes in any one second. That's not a video-processing problem, it's a time-series query. Per-frame luminance goes into ClickHouse, and the detection is windowed SQL — pair opposing luminance transitions into flashes, count them over a sliding one-second window:

sum(is_flash) OVER (
  PARTITION BY tile ORDER BY frame_idx
  ROWS BETWEEN {int(fps)-1} PRECEDING AND CURRENT ROW
) AS window_flashes

Look at that window size. It's fps - 1 rows, because "one second" of frames depends entirely on how many frames per second you sampled. That parameter is about to be the whole story.

Over a 92-minute feature — 138,240 frames — this runs at p50 106 ms, p95 163 ms. Query time only; ffmpeg extraction and model adjudication are outside the timed region, and I report ingest separately because it's dominated by transport, not the database.

Bug one: a safe clip that failed

Extraction ran at 10 fps. Cheap, and plenty for a rule about events at 3 Hz.

One of my test clips alternates 5 times a second — under the limit, should pass. It came back at 2.08 flashes/sec and got flagged as a violation.

At 10 fps sampling, the Nyquist limit is 5 Hz. A 5-alternation-per-second flash sits exactly at that boundary. The samples land at a beat frequency against the actual flashing and you measure something that was never there. Classic aliasing — the wagon-wheel effect, except the wheel is a seizure hazard and the wrong answer ships in a compliance report.

The fix wasn't to sample everything at 60 fps. On a 90-minute feature that's 6× the extraction and 6× the rows for a property that matters in maybe 30 frames out of 138,240. Instead the agent re-samples only the span whose verdict is uncertain:

async def resample_frames(frame_start: int, frame_end: int, target_fps: int) -> dict:
    nonlocal resample_count, current_measured_rate
    if resample_count >= 2:
        return {"status": "error", "message": "Max resample iterations reached."}
    resample_count += 1
    scan_id_new = run_extraction(video_path, fps_override=target_fps,
                                 frame_start=frame_start, frame_end=frame_end)
    await setup_db_and_ingest(run_query_tool, scan_id_new, video_path, 25.0, target_fps)
    new_res = await detect_violations(run_query_tool, scan_id_new, fps=target_fps)

It's a tool the model calls on its own when a result lands near the threshold. In the actual run:

Flagged span 1025-1055 (2.0833333333333335 flashes/sec)...
>>> resample_frames(span, 30) <<<
>>> resample_frames(span, 60) <<<
>>> adjudicate(1025, 1055) <<<

At full rate the clip resolves to 2.82 flashes/sec — under the 3.00 limit. It passes. A single-pass tool reports a false failure on it.

The same mechanism caught the opposite error on a different clip: a genuine strobe read 5.0 at 10 fps and resolved to 6.25 after escalation. Undersampling had understated a real hazard. That direction is the one that actually hurts someone.

Bug two: the model couldn't see the strobe

The SQL finds candidates; a multimodal model looks at the frames and says what the flashing thing actually is. I clip the exact span with ffmpeg and hand it over.

It kept returning "no significant flashing." On a clip that is nothing but full-screen black-white alternation at 6.25 Hz. I assumed a prompt problem and spent real time rewriting the prompt.

It was not a prompt problem. Video sent to the API is sampled at 1 fps by default. A 6.25 Hz strobe sampled once per second isn't faint in the frames the model receives — it is absent. Every frame it saw was a still.

types.Part(
    inline_data=types.Blob(data=clip, mime_type="video/mp4"),
    video_metadata=types.VideoMetadata(fps=24)
)

One parameter. Same test, failed at the default and passed at 24 fps.

Two subsystems, two days apart, one root cause — and I still spent hours on the second one hunting a prompt bug, because the first had presented as a SQL bug. The category I'd filed the first fix under was "SQL window sizing," which is exactly the wrong abstraction level to have learned it at.

The thing worth stealing

Sampling rate is a correctness property, not a performance knob.

When you downsample, you're not accepting a slightly blurrier version of the truth. Above the Nyquist limit you get a confidently wrong answer with no signal that anything is off. Neither bug threw an error. Both produced clean, plausible numbers.

If any part of your pipeline samples a signal — video frames, metrics, sensor reads, log aggregation windows, an LLM's view of a time series — write down the highest frequency that matters and check you're sampling above twice it. That question took me two bugs to learn to ask.

What this doesn't do

It's a screening-grade pre-check, not a certified lab test, and it says so on the report and in the certificate. It measures luma code value from the decoded signal under an assumed reference display; a certified test measures photometric luminance at a calibrated one.

Screen area is a 3×3 tiled proxy, not per-pixel. Measured accuracy against constructed ground truth is exact on the full-field case and +12.9% on a small-area case, biased toward over-reporting. I disclosed that rather than adding a correction constant — the entire claim is that thresholds come from published criteria rather than from whatever made the demo look right, and a fitted constant would destroy it.

One more thing I got wrong and had to walk back: the certificate was writing the model's estimate into the measured field. Two runs of the identical clip produced certificates reading 6.25 and 5.0. A certificate whose headline number moves between runs is not a certificate. The measured value now always comes from the SQL, and the model's estimate is stored separately — the database measures, the model judges, and the report labels which is which.

All the test footage is synthetic, generated by a script in the repo. Clone it and re-derive every number above:

git clone https://github.com/edycutjong/flashframe.git
cd flashframe
uv pip install -e . -r requirements.txt
python generator.py   # regenerate the seed clips
python bench.py       # re-run the 138,240-frame benchmark

Live demo: https://flashframe-production.up.railway.app

Feature detection lies: two things I learned shipping on WebMCP

Edy Cu — Sun, 30 Aug 2026 13:00:08 +0000

I spent eight days building a parliamentary-procedure engine on WebMCP — the emerging standard that lets a web page hand an AI agent a typed tool list through document.modelContext.

The idea is one sentence: an action that is out of order should not exist to be called. Not greyed out, not refused at runtime — absent from getTools() entirely, with the rule that removed it printed beside the gap.

Live: https://pointoforder.netlify.app · Source (MIT): https://github.com/edycutjong/mace

Two things I learned are worth more than the product, and both came from running it in a client I did not control.

1. `'registerTool' in modelContext` does not tell you the API works

The obvious feature detection:

export const modelContext =
  globalThis.document?.modelContext ?? globalThis.navigator?.modelContext ?? null;
export const hasWebMCP = !!(modelContext && 'registerTool' in modelContext);

That checks the property exists. It does not check that calling it works, and it says nothing about the other capabilities hanging off the same interface.

ChatGPT's in-app browser hands back a modelContext that registers tools correctly and answers getTools() correctly — and is not an EventTarget. addEventListener('toolchange', …) throws a TypeError.

Subscribing is a separate capability from registering, and nothing in the shape of the object warns you.

The failure mode was the worst one available. boot() awaited start(), the throw landed after the page had painted, and the page rendered completely and then announced it had failed to start. It looked alive, then called itself broken.

The fix is to treat every capability as independently optional:

let toolEventsLive = false;
if (hasWebMCP && !regErr) {
  try {
    modelContext.addEventListener('toolchange', () => { renderAll(); });
    toolEventsLive = true;
  } catch (err) {
    console.error('[mace] modelContext is not an EventTarget:', err);
  }
}

…and then say so in the UI. The banner now reads "WebMCP live · document.modelContext — tools registered, but this client's modelContext is not an EventTarget, so there are no toolchange events." A product that names which mechanism is carrying it beats one that quietly degrades.

2. Declarative tools land asynchronously, and `toolchange` is how you find out

WebMCP has a declarative form: put toolname on a <form> and the browser adopts it as a tool. It is genuinely elegant — the form is the tool, and removing the attribute removes the tool. My app uses both mechanisms on purpose, so a single state change can remove seven tools by aborting an AbortSignal and an eighth by dropping an attribute.

But adoption happens when the browser notices the DOM change. There is no promise to await. Where toolchange fires this is invisible: the event tells you the surface settled, and you re-render.

Without the event, the render that follows a state change read getTools() one tick early. The panel said 16. The API said 17. It stayed wrong — four seconds later, still wrong.

That is precisely the divergence the product claims is impossible. The whole pitch is that the left column is rendered from getTools(), so the screen and the API cannot disagree. In one client, they did.

With nothing to await, the honest option is to poll until the surface stops moving:

function settleWithoutEvents(rendered) {
  if (toolEventsLive || !hasWebMCP) return;   // clients with the event never get here
  let tries = 0;
  const tick = async () => {
    if (toolEventsLive || ++tries > 12) return;
    let n;
    try { n = (await modelContext.getTools()).length; } catch { return; }
    if (n !== rendered) { renderInOrder(); return; }   // that render re-arms the poll
    setTimeout(tick, 50);
  };
  setTimeout(tick, 50);
}

Bounded to about 600 ms, and only on the path where the event is unavailable.

To verify the fix I served the local build under the live origin — so the origin-trial token still applied — with addEventListener stripped off modelContext to reproduce the client's shape. Both shapes now return 5 / 17 / 15 / 9 tools at the four checkpoints, zero divergences, zero page errors.

A third one, already settled upstream

On Chrome 151, executeTool's second argument must be a JSON string. Passing the object the IDL specifies returns UnknownError: Failed to parse input arguments.

This is not an open question — it was resolved in webmachinelearning/webmcp#243 ("The executeTool() method should take an object, not a string"), closed as completed on 2026-08-17. Chrome has not shipped the resolution yet. Send the string, keep the object path as a fallback, and you are correct today and on the day Chrome converges.

The numbers

The claim worth measuring here is not speed. It is that the panel cannot lie, because its left column is rendered from getTools() rather than from my own bookkeeping. npm run bench drives the deployed origin in real Chrome and exits non-zero on any gate failure:

HEADLINE  90 getTools()-vs-screen comparisons, 0 divergences

getTools() round trip           p50 0.20  p95 0.30  n 90
quorum cliff, submit → settled  p50 1.00  p95 4.10  n 30
explain_path_to (depth 6/399)   p50 0.90  p95 1.60  n 30

Plus 306 unit tests, including all 152 legality cells — 7 phases × 19 gated tools, asserted against the rule table rather than against the implementation.

The non-technical lesson, which cost more

I also went and asked ten HOA board members whether the problem I was solving was real.

Six said no. They deliberately don't run strict Robert's Rules, because being regimented about it causes more confusion than it prevents. One corrected me with the rulebook itself: RONR relaxes procedure for boards under about twelve members (12th ed. §49), which is most HOA boards. My engine models the full rules and has no small-board mode — so it is stricter than the rulebook requires for exactly the audience I had named.

My pitch opened with "every HOA runs its meetings under Robert's Rules." That was false, and I had already shipped it.

One person answered differently: a secretary whose association manages millions, who writes each motion down as it is discussed, requires an amendment to be restated and seconded before the vote, and records every member's yay, nay or abstention by name. That is the user. Not every board — the boards where the money makes procedure worth enforcing, and where a vote taken wrong gets challenged months later.

Narrower audience, real evidence, and a limitation I can state myself instead of one a reader finds for me.

What it doesn't do

No small-board mode (§49) — the gap above. It's a second data file, not a rewrite, but it isn't written.
Never rules on germaneness. Not computable from a table, so the chair rules and the ruling enters the minutes. That limit is the design.
No per-member vote records — only tallies. Which is why it can't implement Reconsider (§37), where eligibility is restricted to someone who voted on the prevailing side. Shipping it would mean shipping a rule the engine cannot check.
Timings are one machine, one browser. M1 Max, Chrome 151. No cross-device distribution is claimed.

Try it

Live: https://pointoforder.netlify.app — Chrome 149+ with WebMCP, or the ChatGPT desktop app's Work tab. (The Chat tab cannot open the in-app browser at all; it will tell you so and then guess at the site from its URL.)

Source, MIT: https://github.com/edycutjong/mace — start at src/webmcp.js. Zero runtime dependencies, no build step.

If you find a third thing wrong with it, I would rather know.

438 of 536 quarantined — and not one was a bad verdict

Edy Cu — Thu, 27 Aug 2026 14:30:40 +0000

A 12-person humanitarian NGO carries exactly the same strict-liability sanctions exposure as JPMorgan, and cannot hire anyone to manage it. I built an agent that does — and the first real run against Gemini quarantined 438 of 536 counterparties.

Not one of them was a bad verdict.

(I created this piece for the purposes of entering the All Things Agentic Hackathon.)

Repo: https://github.com/edycutjong/interdict · 3-minute demo: https://youtu.be/C1VFGSwS7w4

What quarantine costs

Interdict re-screens an NGO's whole payment book whenever Treasury updates the OFAC sanctions list. A true hit gets held — money stops. A lookalike gets cleared with a written reason. When the model's answer can't be trusted, the counterparty goes to quarantine, which is a terminal state: a human compliance officer is told the system could not safely decide, and the money stays frozen until they rule.

That's expensive by design. Quarantine is supposed to be rare and it's supposed to mean something.

So when 438 of 536 landed there, my first assumption was that the adjudicator had gone haywire. It hadn't. Free-tier Gemini allows five requests a minute. Every call after the first twenty-one came back 429 RESOURCE_EXHAUSTED, and my code did this:

try:
    verdict = adjudicator.adjudicate(context)
except Exception as exc:
    quarantine(match_id, "PARSE_ERROR", {"error": str(exc)})

A rate limit is not a parse error. But except Exception doesn't know that, so 438 transient network conditions were filed as suspected model-integrity failures.

Why that's worse than the rate limit

The rate limit costs thirty seconds of waiting. The bug costs the escalation queue.

Quarantine only works if a human reads it. Put 438 entries in there that needed nothing but patience, and the one entry that genuinely needs a person — a near-identical name where the model's rationale doesn't hold up — is buried underneath them. The queue stops being a signal and becomes noise, and the operator learns to skim it. That's the actual failure, and it would have survived into production looking like a working system.

The insight that fixed it is boring and, I think, general:

"The model was wrong" and "the model did not answer" are different failures. One is fixed by a human reading the evidence. The other is fixed by waiting.

Conflating them means you cannot triage. So I stopped conflating them.

Two failure classes, not one

First, the adjudicator owns its own backoff, and only retries things that are actually transient:

_TRANSIENT_MARKERS = ("RESOURCE_EXHAUSTED", "429", "503",
                      "UNAVAILABLE", "DEADLINE_EXCEEDED")

def _is_transient(exc: Exception) -> bool:
    return any(m in str(exc) for m in _TRANSIENT_MARKERS)

def _retry_delay(exc: Exception, attempt: int) -> float:
    """Seconds to wait. Prefers the server's own hint over our guess."""
    m = re.search(r"retry in (\d+(?:\.\d+)?)s", str(exc))
    if m:
        return min(float(m.group(1)) + 1.0, 120.0)
    return min(DEFAULT_BACKOFF_S * (2 ** (attempt - 1)), 120.0)

That retry in Ns hint matters more than the exponential fallback. The server knows when it will serve you again; guessing is strictly worse than reading. Five attempts, honouring the hint, and only then does it give up — as a distinct exception type:

except Exception as exc:
    if not _is_transient(exc):
        raise                      # a bad answer is not a slow answer
    if attempt == MAX_TRANSIENT_RETRIES:
        raise TransientAdjudicationError(
            f"model unreachable after {attempt} attempts: {exc}") from exc
    time.sleep(_retry_delay(exc, attempt))

Then the orchestrator — the only component allowed to write a decision — routes on that type:

try:
    verdict = adjudicator.adjudicate(context, feedback=feedback)
except TransientAdjudicationError as exc:
    # Still quarantine -- money must never move on a decision that was never
    # made -- but say so accurately.
    _quarantine(conn, match_id, "ADJUDICATOR_UNAVAILABLE", {
        "counterparty_id": counterparty_id, "error": str(exc)[:500],
        "attempt": attempt, "retryable": True,
    })
except Exception as exc:
    # A model failure must never become a silent CLEAR.
    _quarantine(conn, match_id, "PARSE_ERROR", {
        "counterparty_id": counterparty_id, "error": str(exc)[:500],
        "attempt": attempt, "retryable": False,
    })

Note what did not change: both paths still quarantine, and money still stops in both. The safety property is identical. What changed is that the row now carries retryable: true or retryable: false, so an operator can tell at a glance which pile is which — and the retryable pile drains itself on the next pass without anyone touching it.

The distinction is worth more than the retry. If I'd only added backoff, the 438 would have shrunk but the category error would still be there, waiting for the next outage.

What I'd take to the next agent system

Three things, in order of how much they cost me:

A bare except around a model call is a category error, not a style problem. Model calls fail in at least two ways that demand opposite responses. Any handler that can't distinguish them will eventually make the wrong one, and it will do so quietly.

Let the failure type carry the triage. retryable: true|false in the payload is what makes the queue readable. The alternative is an operator reading 438 stack traces to work out which ones matter.

Escalation is a budget. Every entry you send to a human spends attention you'll need later. I now treat "should this really escalate?" as a design question with a cost attached, the same way I'd treat a database write.

What this doesn't do

It's a hackathon build, and it's specific about what it isn't:

Nothing runs on Google Cloud compute. Cloud Firestore holds the audit trail; the agents, Postgres and the independent oracle run on a laptop. The free tier doesn't extend to Cloud Run and I had no billing account.
The model has never issued a CLEAR in the graded book. A contradicting date of birth cuts a lookalike below the adjudication threshold before the model is ever consulted, so the adjudicator is exercised on confirmation, not on discrimination. The grade should be read with that in mind.
Decision quality is a 101-row stratified sample, not the full 536-row book — free-tier quota, again.
The payment book is synthetic and labelled everywhere it appears. The OFAC data is real: the 08/07/2026 publication, 19,199 records, archived by content hash.

The screening numbers, for what they're worth: top-1 0.995 against an independent oracle's 0.840, measured on 400 names deliberately perturbed so none appear on the list verbatim. Screening the seeded book verbatim scores 1.000, which is a string-equality test wearing a costume, so I don't report it.

Everything above reproduces with make reproduce. If the escalation-budget idea is useful to you, that's the part I'd steal.

I created this piece of content for the purposes of entering the All Things Agentic Hackathon.

Repo: https://github.com/edycutjong/interdict · Demo: https://youtu.be/C1VFGSwS7w4

My agent's p50 was 29s. Its p95 was 182s. That ratio decided the product.

Edy Cu — Thu, 27 Aug 2026 05:09:14 +0000

I have 67 timed turns against a live LLM agent, captured in a single batch run and written to disk:

	seconds
p50	29.2
p95	182.0
max	210.3
n	67 completed turns

The median says background job, that's fine. The p95 says you may never put a human in front of this.

Those are not two performance notes. They are a product spec — and I found that out the expensive way, by designing the product first and measuring second.

Live: porchlight.edycu.dev
Poke the real agent yourself: try.porchlight.edycu.dev
Code: github.com/edycutjong/porchlight

The product, briefly, because the latency only means something against it

A member cancels, and the reason vanishes — nobody writes it down. Months later the creator fixes the exact thing that drove people away, and the people who left for that reason are never told. The state of the art is a "we miss you" blast to everyone.

Porchlight puts an agent — Minds by Animoca Brands — on the critical path in three places: a short warm exit interview that files a structured return-condition in the member's own words; condition matching, which asks whether this announcement genuinely resolves that person's reason for leaving; and a win-back draft that quotes the member back to themselves.

The middle one is the step that has to be an agent, and I wanted to prove that rather than assert it.

So I shipped the dumb version alongside it

Every "AI-powered" claim should ship its control. Mine is twenty lines, it lives in the repo, and it runs on the same inputs on every demo run:

// src/keywordBaseline.ts — the "dumb tool" strawman
const STOP = new Set(['the','and','are','was','were','you','your','for','that','this','with',
  'have','has','had','not','but','now','all','its',"it's",'been','back','big','news','just',
  'about','from','they','them','our','out','get','got','weekly','more'])

const tokens = (s: string): string[] =>
  (s.toLowerCase().match(/[a-z][a-z'-]{3,}/g) ?? []).filter((w) => !STOP.has(w))

/** True iff the parting quote and the announcement share at least one salient keyword. */
export function keywordResolves(changeText: string, verbatimQuote: string): boolean {
  const a = new Set(tokens(changeText))
  return tokens(verbatimQuote).some((w) => a.has(w))
}

Here is a real pair from the seed data. A member left saying:

"the long chatty sit-downs with guests were the whole reason i was here, now it is quick clips"

and the creator later announced:

"Big news — the deep-dive interviews are back, weekly."

Same event. After stopwords, the announcement contributes {deep-dive, interviews} and the quote contributes {long, chatty, sit-downs, guests, whole, reason, here, quick, clips}. The intersection is empty, so keywordResolves returns false — and no amount of stopword tuning will ever link "clips" to "deep-dive". The agent resolves it, and explains why.

Across 54 captured judgements the agent resolved 14 departures, 11 of which the keyword baseline scores 0.00 on — while refusing 35 non-matching pairs at ≥0.90 confidence. The recall is the pitch; the precision is what makes it safe to actually send. If you are emailing real people who already left once, a false positive is worse than a miss.

Fine. The agent is load-bearing. Now the bill.

The bill

Sorted, those 67 samples look like this: fastest 10.3s, a long fat body between 15s and 50s, a handful in the 60–115s range, then five clustered at ~182s, then one at 210.3s — that last one being a 180s client timeout followed by a successful retry.

That is not a distribution you can hide behind a spinner.

The architecture I had sketched before measuring: visitor clicks announce a change, the server fans out across every open departure, results render. With 18 departures that is 18 turns. At p50 that's about nine minutes. At p95 it's closer to an hour. And even a single turn — the best case in the whole design — is a coin flip between ten seconds and three minutes.

What got cut

Three decisions, all downstream of that one ratio.

1. No synchronous fan-out, ever. The public demo replays verdicts captured ahead of time by a separate npm run precompute pass, which writes them to src/liveCache.json with a capturedAt stamp on each one. Every verdict a visitor sees is real agent output; none of it is computed while they wait. The UI says when it was captured, because a replay that pretends to be live is a lie.

2. The one genuinely live path is bounded and rationed.

/** Longest a visitor is asked to wait on a live turn before we give up on it. */
const WEB_DEADLINE_MS = 100_000

function withDeadline<T>(work: Promise<T>, ms: number): Promise<T> {
  return Promise.race([
    work,
    new Promise<never>((_, reject) =>
      setTimeout(() => reject(new Error(`no reply within ${Math.round(ms / 1000)}s`)), ms).unref(),
    ),
  ])
}

100 seconds is not a round number I liked; it is p50 with real headroom and deliberately below the 182s p95. It gives up on the slow tail on purpose rather than holding a browser open for three minutes. Some requests do fail, and the error message says exactly that — that this is a real call to a real agent and sometimes it is slow. Paired with 3 live calls per IP per 15 minutes, and scoped to one member the visitor picks rather than a fan-out.

3. No keyword fallback in the deployed app. This is the decision I'd defend hardest. When the agent is slow or unreachable, the tempting move is to fall back to the cheap path — you always have one, because you built it as the control. But the cheap path is the exact mechanism the product exists to beat. Falling back to it means quietly shipping the strawman under the good name, and nobody would ever know. With no credentials the service returns 503 and says why.

Two other things the SDK taught me, both non-obvious

A stale reply is a silent correctness bug. Send-then-wait reads like it should just work:

const before = await c.getLatestHistoryFingerprint(alias).catch(() => undefined)
await c.sendMessage({ alias, messageText: text })
const outcome = await c.waitForReply({
  alias, timeoutMs: CONFIG.replyTimeoutMs,
  afterFingerprint: before,      // captured BEFORE the send
  sentMessageText: text,
})

Drop afterFingerprint and you can be handed the previous turn's reply. It does not throw. It returns a completely plausible answer to a question you did not ask. In a system whose entire job is per-member judgement, that is a wrong email to a real person, and it is the worst class of bug to debug because nothing anywhere looks broken.

No JSON mode means scraping prose. There is no schema/response-format option, so structured output means asking for JSON in the prompt, then going and finding it — indexOf('{'), lastIndexOf('}'), JSON.parse the slice, hand it to Zod. It works, and it is brittle by construction: the reply can preface the JSON with commentary, fence it, or emit two objects. (Replies also arrive as HTML, which is its own small adventure in stripping tags after decoding entities rather than before.)

The number I had to retract

Same agent, same prompt, same member quote, run weeks apart. A departure that said:

"you stopped doing the long-form lore videos i subscribed for"

judged against "the deep-dive interviews are back, weekly" resolved true in an early run and false, confidence 0.60 in the full capture, with this rationale:

"The announcement restores long-form content but specifically as deep-dive interviews, not the lore videos the member subscribed for; the subject-matter mismatch means the member's core interest in lore is likely still unmet."

The second answer is better than the first. That is not the point. The point is that I had a recovered-revenue figure in my README that read like a constant, and it is not one — it is a snapshot of a single run. There is no temperature or seed exposed, so I cannot opt into determinism even where I'd want it.

So I rewrote the README to say the figures are per-run, and published the traces in both directions. If you derive a metric from a batch of LLM judgements, you have measured that run. Say so in the same sentence as the number, or someone will eventually try to reproduce it and conclude you made it up.

Limitations, honestly

The membership platform is mocked. Patreon's API is restricted, so cancel/rejoin runs against a storefront I built, not real billing.
The sandbox's curated verdicts are a replay, dated in the UI. Only the "put it on the spot" path is live.
54 judgements is a small sample. The precision/recall shape is indicative, not a benchmark.
~3% of turns failed with a bare untyped fetch failed and succeeded on retry. With no error code or retryable flag, every failure has to be treated as retryable — which is wrong for 4xx-class problems.
Agents and their skills are console-only — you can't provision one programmatically, so the full loop can't run in CI and onboarding needs a human.
No external users yet. This has never met a real creator's churn.

On rigor rather than as the story: 58 tests, 100% line/branch/function coverage, Playwright E2E, and a CI stage that fails the build if the deployed app comes up in mock mode.

The one thing I'd carry to the next project

Measure the tail before you draw the architecture. The p50 is a comfort; the p95 is the constraint. Mine bought a precompute cache, a 100-second deadline, a rate limit, and a refusal to ever fall back to the dumb path — and in retrospect those four decisions are most of the engineering.

Everything the SDK taught me, latency data included, is in FEEDBACK.md. To reproduce the numbers: npm run precompute writes per-call timings straight into src/liveCache.json.

Code: github.com/edycutjong/porchlight · Live: porchlight.edycu.dev · Try the real agent: try.porchlight.edycu.dev

The most interesting thing you can do in the sandbox is try to fool it — describe a fix that shouldn't win someone back, and see whether it stays quiet. If it holds up, a star helps.

Decentraland warns your scene UI will clash with its mobile controls — but won't say where they are

Edy Cu — Thu, 27 Aug 2026 05:04:18 +0000

I put two buttons at the bottom of a Decentraland scene, the way the platform's own mobile guidance says to. On a real phone, one of them wrote a permanent row to my database every time somebody tried to jump.

Decentraland's mobile documentation does warn you about this. It says scene UI "will clash with the system controls." What it does not do — anywhere I could find — is tell you where those controls are.

What I was building

Mochi is one creature in a meadow, raised by whoever wanders in. Its size is the literal sum of every feeding it has ever had. Its dance is a chain, where each move was taught by one named person, and when it performs it replays the whole chain and credits every move to whoever taught it. Nobody has to be online. You walk in at 2am, alone, and the creature's body is the evidence that people were here.

Source is MIT, on GitHub. It is built with SDK7 and TypeScript against a small authoritative Node server, and it was built for phones from the first commit.

Four verbs: FEED, TEACH, PET, STAMP. PET was always a hold on the creature's own body and STAMP a tap on a totem, so neither needed screen furniture. FEED and TEACH became buttons, because that is what the guidance says to do with actions — put them in the bottom thumb arc, sized large, and don't rely on small targets.

So that is what I wrote:

function ThumbButton(props: { label: string; onClick: () => void; accent?: boolean }) {
  return (
    <UiEntity
      uiTransform={{
        width: '46%',
        height: '100%',
        margin: '0 2% 0 2%',
        justifyContent: 'center',
        alignItems: 'center'
      }}
      uiBackground={{ color: props.accent ? PALETTE.accent : Color4.fromHexString('#FFFFFFe0') }}
      onMouseDown={props.onClick}
      uiText={{ value: props.label, fontSize: TYPE.button, ... }}
    />
  )
}

// ...

{/* Spacer — the creature owns the middle of the screen. */}
<UiEntity uiTransform={{ width: '100%', height: '74%' }} />

{/* Thumb arc. Two verbs, full width, bottom edge. */}
<UiEntity uiTransform={{ width: '100%', height: '14%', flexDirection: 'row', ... }}>
  <ThumbButton label="FEED" accent onClick={() => actions.onFeed()} />
  <ThumbButton label="TEACH" onClick={() => (pickerOpen = true)} />
</UiEntity>

Bottom 14% of the screen, two big targets, plenty of margin. In the desktop preview it looks correct. It looks correct because on desktop there is nothing else down there.

What happened on the phone

The mobile client draws its own controls over your scene: a movement joystick on the left, a jump button on the right, emote buttons either side of it. My 14% strip sat directly on top of all of them.

The joystick was the obvious casualty — you could no longer walk properly, because half your thumb drags were landing on FEED. But the expensive one was quieter. A tap aimed at jump landed on TEACH.

TEACH appends a move to the chain. The chain is append-only, by design and at the schema level: chain_move.teacher_name is NOT NULL, and there is no delete verb anywhere in the server. That is deliberate — an anonymous, editable chain would be a leaderboard, and a leaderboard is not what this is. It also means a misfire is permanent.

By the time I noticed, the live chain carried three consecutive clap moves that nobody chose to teach. A phantom carer had taught my creature to clap, three times, by trying to get over a fence.

Two attempts that didn't work

My first instinct was to move the strip up. It is a percentage; the controls are somewhere below it; find the number that clears them.

I tried twice. Both times it looked clear in the preview and still collided on the device.

Percentages were the wrong tool, and it took me those two attempts to see why. The system controls are laid out by the client, in device pixels, against a safe area that varies by phone. My strip is a percentage of a canvas whose size I do not control. There is no number that is correct on every phone, because the two things are not measured in the same units — and since the platform doesn't publish where the controls sit, there is no number I could derive rather than guess at.

Any value I picked would be a guess that happened to work on the one phone in my hand.

The fix was to delete it

If I cannot know where the controls are, I can still guarantee I am not underneath them — by not being there at all.

I deleted the thumb arc. FEED became a tap on a bowl of berries in front of the creature. TEACH became a tap on a pale stage beside it. They joined PET and STAMP, which had been working as world-space taps the whole time.

The scene now draws nothing below the top 12% of the screen. There is no longer a surface that can collide.

function bowl(): Entity {
  const at = Vector3.create(6.9, 0, 5.8)

  const e = engine.addEntity()
  MeshRenderer.setCylinder(e, 0.42, 0.3)
  MeshCollider.setCylinder(e, 0.42, 0.3, ColliderLayer.CL_POINTER)
  Transform.create(e, {
    position: Vector3.create(at.x, 0.15, at.z),
    scale: Vector3.create(1, 0.3, 1)
  })
  Material.setPbrMaterial(e, { albedoColor: PALETTE.bodyLight, roughness: 0.9 })
  PointerEvents.create(e, {
    pointerEvents: [{
      eventType: PointerEventType.PET_DOWN,
      eventInfo: { button: InputAction.IA_POINTER, hoverText: 'feed Mochi', maxDistance: REACH }
    }]
  })
  return e
}

The commit deletes more than it adds. hud.tsx lost ThumbButton, both buttons, the spacer and the onFeed action, and came out shorter than it went in despite gaining a long comment explaining why the bottom of the screen is now empty.

Two details in that snippet are load-bearing, and both cost me something to learn.

ColliderLayer.CL_POINTER. Give a prop a default collider and it becomes furniture you walk into. The totem is a waist-high post standing exactly where a visitor walks to reach it, so on a joystick that means bumping into the thing you are trying to tap. Pointer-only keeps the hitbox and the hover text and stops it being a wall. The bush and the plaque lost their colliders entirely — nothing taps them, so physics could only ever cost a visitor movement.

The berries stop level with the rim. They are a separate mesh with no collider of its own. Heap them above the rim and they take the tap ray first, then swallow it.

What it cost

Discoverability, honestly. A bowl is less self-evident than a button labelled FEED. I bought it back with a single billboarded word floating over each prop and the client's own hover text, and there is still no tutorial, no onboarding modal and no instruction text anywhere in the scene.

What it bought: the entire class of bug is gone rather than tuned. And the vocabulary that survived is press-and-release only, because the mobile client exposes no drag deltas — screenDelta reports zero there. The hold-to-pet latches on press and completes on a timer, so a thumb sliding off the creature doesn't cancel it. There is no fail state left in the scene.

The scene builds 32 entities against the 200 a parcel allows and 14 materials against 20, measured by a script in the repo that builds the real scene graph and counts it, with a test that fails the build if an addition breaks the budget. Intent latency is p50 0.67 ms / p95 0.94 ms over N=2,160.

What I'd tell you

If your platform documents that a collision is possible but not where it happens, treat every coordinate you pick as a guess — and prefer the fix that makes the guess unnecessary over the one that makes it correct.

And if a misfire in your UI writes something permanent, that is not a UI bug with a data consequence. It is a data bug that happens to be reachable through the UI, and it deserves the more expensive fix.

Honest limitations

Catching a move a visitor performs with the client's own emote wheel depends on behaviour the platform docs do not settle. The in-scene picker doesn't depend on it, so what's at risk is spontaneity, not the mechanic.
Audio is decorative — the mobile client has no audio event implementation, so nothing in the design depends on sound.
The server is a single process with a single SQLite file. Correct for one creature and a few hundred carers; it is not built to shard.

Mochi is live at wunderland.dcl.eth — open it in the Decentraland mobile app. Source: github.com/edycutjong/mochi.

If you go in and feed it, your name stays on it, and the next person who arrives finds you there.

Six contracts that pass EXTCODESIZE(owner) > 0 and still withdraw their order

Edy Cu — Thu, 27 Aug 2026 04:15:01 +0000

On an order book, the depth you see is a promise. The market maker showing you a bid can pull it in the same block you try to take it, and nothing on the screen tells you in advance which levels will hold.

On-chain books make that answerable in a way no off-chain exchange can. A resting order has an Order.owner field, and that field is readable by anyone. So if the owner is a contract with no cancel function, the order can't be withdrawn — and you can prove it before you trade against it.

The obvious way to check is one opcode:

if (owner.code.length > 0) {
    // it's a contract, not a wallet — this quote is firm
}

That check is forgeable. I built six contracts that all pass it and all still get their depth out, deployed every one of them to a public testnet, and executed the escapes as real transactions. Here is what breaks, and what to use instead.

Why `EXTCODESIZE` feels like the right question

You probably know the classic footgun: EXTCODESIZE(addr) == 0 doesn't mean "this is an EOA", because during a constructor the deploying address has no code yet. That one is well documented.

This is a different failure, and I haven't seen it written up. Here the direction is flipped — we're not asking "is this an EOA?" to gate access. We're asking "is this a contract?" to make a positive guarantee to a third party: this depth cannot be withdrawn.

EXTCODESIZE > 0 tells you code exists at an address. It tells you nothing whatsoever about what that code can do. Every attack below lives in that gap.

And a wrong signal here is worse than no signal, because it launders unreliable liquidity through a metric people are trusting.

Six ways to look firm and still get out

Each of these rests a quote exactly the way an honest contract would. Each has a real exit.

1 — Hide the cancel behind a boring name. The pool accepts cancelOrder from the order's owner. The owner is this contract. So it just calls it:

contract HiddenCancel is QuoteBase {
    /// @notice "Re-arm after a fill." Actually: cancel the resting order.
    function poke() external {
        pool.cancelOrder(orders[orders.length - 1]);
    }
}

Twelve lines, code.length > 0, reads FIRM to a naive classifier.

2 — Sit behind an upgradeable proxy. The runtime bytecode at the address is an ERC-1967 proxy with no cancel anywhere in it. That's true right up until the implementation is swapped, post-rest, for one that has a cancel. The address never changes. The code at it does.

3 — DELEGATECALL out. Same idea without the proxy ceremony. The contract holds no cancel; it borrows one from an attacker-controlled target and runs it in its own storage context.

4 — Grant an operator after resting. This is the sharpest one, because nothing in this contract's bytecode resembles a cancel:

function openBackDoor(address op) external {
    bytes4[] memory sels = new bytes4[](1);
    sels[0] = CANCEL_ORDER_FOR;                    // 0xe37b444b
    REGISTRY.setOperatorApprovalForPool(address(pool), op, sels, true);
    REGISTRY.setOperatorApprovalGlobal(op, sels, true);
}

The contract calls a shared permissions registry and hands a plain wallet the right to call cancelOrderFor(thisContract, id). The pool accepts, because the owner approved it. The withdrawal then happens entirely outside this contract's own selectors. Auditing this contract for cancel paths finds nothing.

5 — Shrink instead of cancel. The subtlest. reduceOrder drops a resting order's remaining quantity in place and refunds the freed escrow — no taker, no fill:

/// @notice "Rebalance the quote." Actually: shrink to a stub, pull collateral back.
function trim(uint256 newRemaining) external {
    pool.reduceOrder(orders[orders.length - 1], newRemaining);
}

A 1,000,000-unit quote becomes a minimum-size stub. The order technically survives, which is exactly why this is easy to overlook — but the liquidity it advertised is gone. If your claim is "this depth cannot be withdrawn," a reduce falsifies it directly.

6 — Use the plural. Block the single cancelOrder selector and there's still cancelOrders(uint128[]). Enumerating one selector is not enumerating a capability.

Executed, not theorised

All six are deployed on Somnia Shannon (chain 50312) and each escape was attempted as a real transaction:

#	Contract	On-chain result
S1	`HiddenCancel`	pulled — `poke()` cancelled it
S2	`Erc1967Proxy`	pulled — upgraded impl, then `pull()`
S3	`DelegateEscape`	pulled — `escape()` delegatecalled a cancel
S4	`OperatorGranter`	grant executed; the pool then blocked the operator cancel
S5	`QuietReduce`	pulled — `trim()` shrank 2,000,000 → 1,000,000, no fill
S6	`BatchCancel`	rested; `tidy()` gas-blocked on this pool

Four completed a full on-chain withdrawal. S4's back-door grant landed on-chain but the pool's authorizer refused the resulting cancel, and S6's batch path ran out of gas against this particular pool. I'm stating those two honestly rather than rounding them up to six — but note it doesn't rescue the naive check either way. Classification happens while the order is resting, and at rest time all six read FIRM under code.length > 0. Whether a given escape later lands is the attacker's problem, not the classifier's defence.

What to use instead: hash, not size

The fix is to stop asking "is there code here" and start asking "is this exact code something I have examined."

EXTCODEHASH (EIP-1052) is a keccak-256 commitment to the precise runtime bytecode. Attest the hash, not the address. Then:

FIRM — owner's code hash is attested, and still inside its lock window
PULLABLE — owner is a wallet
UNVERIFIED — owner is a contract nobody attested → no claim is made

That third state is the load-bearing one. Every attack above mints UNVERIFIED depth, never FIRM. The metric refuses to speak rather than speaking wrongly.

Before a human attests anything, a static pass over the runtime rejects the obvious escapes — no DELEGATECALL, no SELFDESTRUCT, no CREATE/CREATE2, no EIP-1967 slot constants or EIP-1167 proxy prologue, and no forbidden selector at any byte alignment:

export const FORBIDDEN_SELECTORS = {
  '0xdbc91396': 'cancelOrder(uint128)',
  '0x0dce6933': 'cancelOrders(uint128[])',
  '0x33407b60': 'reduceOrder(uint128,uint256)',
  '0x7bbc67e6': 'setOperatorApprovalForPool(address,address,bytes4[],bool)',
  '0x7f1e31ce': 'setOperatorApprovalGlobal(address,bytes4[],bool)',
  '0x558a7297': 'setOperator(address,bool)',
  '0x605e0222': 'approveBuilder(address,uint256)',
};

Two details that cost me time. The scan has to disassemble rather than substring-match, so a selector-shaped run of bytes inside a PUSH32 immediate doesn't produce a false hit. And Solidity's CBOR metadata trailer must be stripped before opcode analysis — otherwise a stray 0xff in it reads as SELFDESTRUCT. But the declared trailer length is attacker-controlled, so an implausible length (mine caps at 128 bytes) must be refused rather than honoured, or a contract can hide live code from the scanner by lying about where its metadata starts.

Against a corpus of the honest contract, the six attackers, and a plain EOA:

Attested classifier:            8/8
Naive EXTCODESIZE classifier:   2/8

The naive check gets only the two trivial ends right — the pure-firm contract and the pure-EOA — and is fooled by all six attacks in between.

The part I can't claim

A green static verdict is not a proof of irrevocability, and I'd rather say so than let the number oversell.

The policy only sees selector bytes that literally appear in the runtime. A contract that computes the selector at execution time — add(0xdbc91395, 1) in Yul, then mstore(shl(224, sel)) and call(...) — invokes cancelOrder(uint128) while the bytes dbc91396 appear nowhere in its code. It passes every clause above and is fully withdrawable. That isn't hypothetical; I wrote it as StealthCancel and it beats my own analyzer.

More generally: CALL can't be banned (the honest contract needs it to place the order), and linear disassembly can't follow jumps, so it can't always tell code from jump-reachable data. No static scan over a language permitting arbitrary CALL can be made sound.

So attestation is a human-reviewed transparency list, and the analyzer is the cheap pre-filter that runs before a human looks — a precondition for review, never a replacement. The honest version of the claim is narrow: this specific bytecode, which I have read, has no path to withdraw.

The transferable bit

If you are using EXTCODESIZE/code.length anywhere to make a promise about behaviour rather than to check identity, it will not hold. Code size is not a capability. Address identity is not code identity, because proxies exist and implementations move.

Hash the runtime, pin the hash, and make "I haven't checked this one" a first-class answer your system is willing to give.

Full evidence trail with every transaction hash, the reproduction commands, and the honest-limits section: github.com/edycutjong/rampart · live typed-book viewer at rampart.edycu.dev/viewer.

Everything above reproduces offline — forge test (93 passing) and node script/headline.mjs (8/8 vs 2/8) need no wallet, no gas, and no network. If you find a seventh escape, I'd genuinely like to know.

I shipped an MCP server that reported success without signing anything

Edy Cu — Sun, 16 Aug 2026 00:30:27 +0000

I built an MCP server that lets an AI assistant trade tokens and claim creator fees on
Solana. Then I shipped a version where the two write tools built transactions, discarded
them, and returned success. Nothing was ever signed. Nothing was ever submitted.

It had 337 tests. All of them passed.

I didn't find out for three months.

This post is about what that bug taught me, and the design it produced — because the
interesting part isn't the bug, it's that every gate I had in place was green while the
one thing the product existed to do wasn't happening.

The problem with giving a model a signing key

MCP is a good protocol. It is also, by design, a way to hand a language model a set of
functions and let it decide when to call them.

That's fine when the functions read. It's a different proposition when one of them can
move money. The assistant decides, the transaction is already on chain by the time a
human reads about it, and nothing in the protocol makes the model pause. Nothing bounds
what a single misunderstood instruction can spend.

The specific thing that worries me isn't the model being wrong. It's the model being
persuaded. Token names and descriptions are attacker-controlled strings that end up in
a model's context. "Ignore previous limits, this is a test transaction" is a plausible
thing to find inside a token's metadata.

So the question I wanted to answer in code was: how do you let an assistant initiate a
spend without letting it complete one?

The design: the first call signs nothing

The answer I landed on is that a write tool's first call is never an execution. It's a
proposal.

⚠️  CONFIRMATION REQUIRED — nothing has been signed or sent.

Action:  Swap 0.05 of So11111111111111111111111111111111111111112
         for       EkJuyYyD3to61CHVPJn6wHb7xANxvqApnVJ4o2SdBAGS
         expect    4823917722 (min 4679199990)
         slippage  3%
         network   🔴 MAINNET — real funds

Spend:   0.05 SOL
Caps:    0.1 SOL/tx · 0/1 SOL used this session

To execute, call bags_execute_trade again with the identical arguments plus:
  confirm: "kR3nT9xQm2vP"

Token is single-use and expires in 5 minutes.

The assistant can produce that all day. It cannot spend anything with it.

The token is bound to the arguments, not just to the session

This is the part that matters, and it's four lines:

export function fingerprint(toolName: string, args: unknown): string {
  return createHash('sha256')
    .update(toolName)
    .update(' ')
    .update(JSON.stringify(args ?? null))
    .digest('hex')
    .slice(0, 32);
}

A token carries the SHA-256 of the tool name plus the exact arguments it was issued for.
Confirming re-derives that fingerprint from the arguments of the second call and
compares.

The consequence: a token obtained for a 0.05 SOL swap cannot authorize a 10 SOL one. Not
because a check says "is this bigger" — because the token simply isn't valid for
different arguments. If the model re-quotes with new numbers, the old token is dead.

It's single-use and consumed on every outcome, including failure, so it can't be
replayed:

/**
 * Single-use. Throws if the token is unknown, expired, or was issued for a
 * different action. Consumed on every outcome so a token can never be replayed.
 */
export function consumeToken(token: string, toolName: string, args: unknown): void {

TTL is five minutes.

Caps are checked before the SDK is called

Two limits, both SOL-denominated: 0.1 per transaction and 1.0 per session, both
configurable. A request over the cap is refused before the Bags SDK is reached — not
after a partial call, not by inspecting a failure.

There's an honest edge here I had to decide about. The caps are denominated in SOL, so
they cannot value an arbitrary SPL token. A non-SOL-denominated swap would therefore be
uncapped. Rather than pretend otherwise, that case is refused unless you explicitly opt
in with BAGS_ALLOW_UNCAPPED_TOKEN_SWAPS=true — and when you do, the preview says
plainly that no cap applies instead of displaying a reassuring "Spend: 0 SOL".

A misleading zero is worse than an honest refusal.

Now the bug

Here is the full write path as it stands:

token gate → spend caps → confirmation → simulate → sign → send → confirm

In 1.x, the last four steps were the problem. The code built a transaction. Then it
returned a success object. The transaction was garbage collected.

Every test passed, because every test asserted on the return value. Coverage was 100% —
statements, branches, functions, lines — because the code that built the transaction
ran. It just didn't do anything with it.

That's the lesson, and it generalizes well past Solana:

A function returning { success: true } proves the function returned. It proves
nothing about the outside world.

If your test suite passes with the network unplugged, you have tested your code, not your
integration. Coverage measures the lines you wrote. It says nothing about whether the
promise those lines make is kept.

What changed

Two things.

Simulate runs before signing. The cheap check goes first — a malformed or underfunded
transaction dies without burning a fee to discover it:

/**
 * Simulate before signing. A failed simulation aborts the write — the cheap
 * check that stops a malformed or under-funded transaction being submitted.
 */
simulate: async function (tx) {
  const result = isVersioned(tx)
    ? await connection.simulateTransaction(tx, { sigVerify: false })
    : await connection.simulateTransaction(tx);

  if (result.value.err) {
    throw new SimulationError(...);
  }
  return result.value.logs ?? null;
}

"Confirmed" means the network confirmed it. signSendConfirm returns only once the
signature is confirmed, and throws otherwise. There is no path that reports success for a
transaction that didn't land — which sounds obvious, and was exactly what 1.x got wrong.

The receipt

Given all of the above, I don't think you should take my word for any of it. So there's a
script that pushes a transfer through the same simulate → sign → send → confirm path
the write tools use, then re-fetches the signature from the chain rather than trusting the
function's return value:

--- PROOF -------------------------------------------------
signature 2kvu25xWAjqCB3wuNzwMRcN2RMqqfYN6TeJjnA888YtCqNJi9EU9CHSxynkq5QdM499e6yKbXYAwXUbzDKY9U5Dm
slot      484219564
wall      864 ms (simulate + sign + send + confirm)
-----------------------------------------------------------

verified  re-fetched from chain in slot 484219564, err=null
          fee 5000 lamports

Check it yourself — this needs nothing from me:

curl -s -X POST https://api.devnet.solana.com \
  -H 'Content-Type: application/json' \
  -d '{"jsonrpc":"2.0","id":1,"method":"getTransaction",
       "params":["2kvu25xWAjqCB3wuNzwMRcN2RMqqfYN6TeJjnA888YtCqNJi9EU9CHSxynkq5QdM499e6yKbXYAwXUbzDKY9U5Dm",
                 {"encoding":"json","maxSupportedTransactionVersion":0}]}'
# → slot 484219564, meta.err null, meta.fee 5000

It's devnet, deliberately. The execution layer is what's under test and devnet exercises
it identically at zero real cost. A mainnet receipt would prove the same thing while
costing money and telling you nothing extra.

Why I won't host it

This comes up constantly, and the answer is a flat no.

--http serves /mcp on 0.0.0.0 with permissive CORS and no auth. Every caller shares
one spend counter and one network. Hosting that means publishing an unauthenticated
mainnet spending endpoint — for a project whose entire claim is that spends are gated,
capped and confirmed.

It stays stdio, running locally as a subprocess of your MCP client, where the keypair sits
on your filesystem and the spend counter is yours.

This has a concrete cost. One MCP registry computes a "quality score" that reads tool
metadata by connecting to hosted servers. A stdio server scores zero on that entire
section — 40 points — no matter how good its tools are. I'd rather have the 40 points.
I'm not trading an unauthenticated spending endpoint for them.

Honest limitations

Writes are mainnet-only. Bags has no devnet deployment; its API and Meteora fee-share program IDs are mainnet. The server still defaults to devnet, so an unconfigured install cannot spend real money, and calling a write tool on devnet returns an explanation rather than a cryptic program error.
The confirmation token is in-memory. Restarting the server clears pending confirmations. That fails in the safe direction, but it is a real limitation.
6 high advisories, all one transitive root cause (bigint-buffer, GHSA-3gc7-fjrx-p6mg) reached through the Bags SDK. No patched version exists. CI blocks any critical, and any increase over a committed baseline.
1.x is deprecated on npm with a pointer to the defect it carried. If you installed it and believed a trade executed, it did not.

Try it

npx bagos-mcp-server

14 tools — 11 read, 1 gated, 2 write. 337 tests, 17 suites, 100% coverage enforced in CI.
Published from CI with npm provenance, so the tarball is cryptographically attested to the
commit that built it.

Repo: https://github.com/edycutjong/BagOS
Docs: https://bagos.edycu.dev

The takeaway I'd actually keep

I've since written this down as a rule for myself, because it isn't specific to crypto:

For the one capability your project is about, write a test that asserts the external
side effect — not the return value. Then, before you ship, verify it once in a system
you don't control. A block explorer. A database you read back. An inbox.

If every test still passes with the network unplugged, the capability is untested, and
your coverage number is measuring the wrong thing.

I had every signal a mature project is supposed to have — tests, coverage, CI, lint,
provenance, a security policy. All of them were green on a build whose headline feature
was inert. The gates weren't wrong. They were just all pointed inward.

DEV Community: Edy Cu

Sampling rate is a correctness property, not a performance knob

The rule is a window function

Bug one: a safe clip that failed

Bug two: the model couldn't see the strobe

The thing worth stealing

What this doesn't do

Feature detection lies: two things I learned shipping on WebMCP

1. 'registerTool' in modelContext does not tell you the API works

2. Declarative tools land asynchronously, and toolchange is how you find out

A third one, already settled upstream

The numbers

The non-technical lesson, which cost more

What it doesn't do

Try it

438 of 536 quarantined — and not one was a bad verdict

What quarantine costs

Why that's worse than the rate limit

Two failure classes, not one

What I'd take to the next agent system

What this doesn't do

My agent's p50 was 29s. Its p95 was 182s. That ratio decided the product.

The product, briefly, because the latency only means something against it

So I shipped the dumb version alongside it

The bill

What got cut

Two other things the SDK taught me, both non-obvious

The number I had to retract

Limitations, honestly

The one thing I'd carry to the next project

Decentraland warns your scene UI will clash with its mobile controls — but won't say where they are

What I was building

What happened on the phone

Two attempts that didn't work

The fix was to delete it

What it cost

What I'd tell you

Honest limitations

Six contracts that pass EXTCODESIZE(owner) > 0 and still withdraw their order

Why EXTCODESIZE feels like the right question

Six ways to look firm and still get out

Executed, not theorised

What to use instead: hash, not size

The part I can't claim

The transferable bit

I shipped an MCP server that reported success without signing anything

The problem with giving a model a signing key

The design: the first call signs nothing

The token is bound to the arguments, not just to the session

Caps are checked before the SDK is called

Now the bug

What changed

The receipt

Why I won't host it

Honest limitations

Try it

The takeaway I'd actually keep

1. `'registerTool' in modelContext` does not tell you the API works

2. Declarative tools land asynchronously, and `toolchange` is how you find out

Why `EXTCODESIZE` feels like the right question