DEV Community: Whatsonyourmind

Your bandit's exploration floor probably violates its own floor

Whatsonyourmind — Wed, 17 Jun 2026 22:19:34 +0000

Most multi-armed bandit / A-B allocation systems add a minimum exploration weight: every arm should get at least, say, 5% of traffic, so no variant is ever fully starved and you keep collecting data on all of them. The guarantee sounds simple — p_i >= f for every arm — and the implementation looks even simpler:

def clip_renorm(w, f):
    p = np.maximum(w, f)   # raise anything below the floor up to it
    return p / p.sum()     # renormalize so probabilities sum to 1

This is wrong, and it fails silently. The renormalize step pushes the floored arms back below the floor.

Why clip-then-renormalize breaks

Clipping raises the small weights up to f, which makes the total exceed 1. Dividing by that total then scales everything down — including the arms you just clipped to f. So they land below f again, and the floor you advertised is not the floor you enforce.

Concrete case — 4 arms, a confident winner, floor f = 0.10:

w   = [0.94, 0.02, 0.02, 0.02]   floor = 0.10
clip-renorm -> [0.7581, 0.0806, 0.0806, 0.0806]   min = 0.0806  ❌ (< 0.10)

The three starved arms each get 8.06%, not the 10% you promised. And it isn't an edge case. Over 100,000 random peaky weight vectors (Dirichlet, α=0.3, n=4, f=0.10):

clip-and-renormalize violated the floor 97.2% of the time — worst arm seen: 7.69% against a 10% floor.

Whenever one arm dominates (exactly when a bandit is exploiting), the floor leaks.

The fix: one affine map onto the simplex

Instead of clipping, mix the learned weights with the uniform floor. Put the weights on the simplex (sum(w) = 1), then:

def additive_simplex(w, f):
    w = w / w.sum()
    return f + (1.0 - len(w) * f) * w

Each output is f + (non-negative), so p_i >= f holds exactly, and the total is n*f + (1 - n*f)*1 = 1 by construction — no renormalization needed, so nothing gets dragged back under the floor. It also preserves the ordering and relative spacing of w (it's affine), so you don't distort the policy you learned. Same run:

additive-simplex -> [0.664, 0.112, 0.112, 0.112]   min = 0.112  ✅

Over the same 100,000 vectors it violated the floor 0.00% of the time.

The one guard you do need

The map needs n * f <= 1 — you can't promise four arms a 30% floor each (that's 120%). Handle it explicitly instead of producing negative weights:

def exploration_floor(w, f):
    n = len(w)
    if f < 0:
        raise ValueError("floor must be non-negative")
    if n * f >= 1.0:
        return np.full(n, 1.0 / n)          # floor is infeasible -> uniform
    w = np.asarray(w, dtype=float)
    w = w / w.sum()
    return f + (1.0 - n * f) * w

That's the whole correct primitive: a non-negativity check, an infeasible-floor fallback to uniform, and the affine mix.

Why it actually matters

The exploration floor isn't cosmetic. It's what bounds worst-case regret and guarantees you keep collecting data on every arm — the property a lot of bandit regret arguments lean on, and often a fairness/SLA requirement too ("no variant ever drops below X%"). A floor that's silently 7.7% instead of 10% means the guarantee you reported to stakeholders, and any bound that depends on it, doesn't hold. The bug is invisible because the output still sums to 1 and still looks floored — the smallest number is just quietly too small.

import numpy as np
rng = np.random.default_rng(0)
f, n, viol = 0.10, 4, 0
for _ in range(100_000):
    w = rng.dirichlet(np.ones(n) * 0.3)
    p = np.maximum(w, f); p = p / p.sum()       # clip-renorm
    if p.min() < f - 1e-12: viol += 1
print(f"clip-renorm floor violations: {viol/100_000:.1%}")   # ~97%

I ran into this reviewing a Thompson-sampling weighting routine and proposed the additive-simplex version (plus the two guards) as a fix upstream. If your bandit or weighted-experiment layer clips-then-renormalizes to enforce a minimum, it's worth a one-line check: does the smallest probability it emits actually clear the floor?

A model with R-squared near 0 can still give valid 90% prediction intervals - here's why (and the catch)

Whatsonyourmind — Wed, 17 Jun 2026 21:33:25 +0000

I recently calibrated a recovery-rate model that had only two weak features. Its point accuracy was almost nothing — R² basically zero. I expected its uncertainty estimates to be junk too. They weren't: the 90% conformal prediction intervals covered ~89% of held-out outcomes. Valid, just wide.

That surprised me enough to nail it down, because it contradicts a belief a lot of us carry around: "my model isn't accurate, so I can't trust its uncertainty." For split conformal prediction, that's backwards. Here's the precise statement, a runnable demo, and the one caveat that actually bites.

Coverage is a property of the procedure, not the model

Split conformal prediction gives a distribution-free, finite-sample marginal coverage guarantee:

P( Y ∈ Ĉ(X) ) ≥ 1 − α

and it holds for any point model, as long as the calibration and test data are exchangeable. The model is a black box. You fit it however you like, then on a held-out calibration set you take the (1−α) quantile of the absolute residuals, and that quantile becomes the half-width of your intervals.

Nowhere does that construction require the model to be good. A bad model just has large residuals, so the calibration quantile is large, so the intervals are wide — wide enough to still cover at the stated rate. Accuracy doesn't buy you validity; it buys you efficiency (narrower intervals at the same coverage).

The demo (numbers are reproducible, seed fixed)

Same dataset and target, three models from strong to useless, target coverage 90%:

model	R²	marginal coverage	mean interval width
gradient boosting	0.741	0.895	5.39
weak linear (1 noisy feature)	0.061	0.905	10.39
predict-the-mean	−0.000	0.907	10.83

All three land at ~90% coverage. The only thing that changes is width: the good model's intervals are half as wide. That's the whole story in one table — validity is constant, efficiency tracks accuracy.

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.ensemble import GradientBoostingRegressor

rng = np.random.default_rng(20260617)
n = 6000
X = rng.normal(size=(n, 5))
group = rng.integers(0, 3, size=n)
y = X @ np.array([2.0, -1.5, 1.0, 0.5, -0.8]) + 1.5 * group + rng.normal(size=n)

s = lambda a: (a[:3000], a[3000:4500], a[4500:])
Xtr, Xcal, Xte = s(X); ytr, ycal, yte = s(y); _, _, gte = s(group)
ALPHA = 0.10

def conformal(model, label):
    model.fit(Xtr, ytr)
    res = np.abs(ycal - model.predict(Xcal))
    k = int(np.ceil((len(res) + 1) * (1 - ALPHA)))
    q = np.sort(res)[min(k, len(res)) - 1]          # calibration quantile
    pred = model.predict(Xte)
    covered = (yte >= pred - q) & (yte <= pred + q)
    r2 = 1 - np.sum((yte - pred)**2) / np.sum((yte - yte.mean())**2)
    gcov = {int(g): round(covered[gte == g].mean(), 3) for g in np.unique(gte)}
    print(f"{label}: R2={r2:6.3f} cov={covered.mean():.3f} width={2*q:5.2f} group={gcov}")

conformal(GradientBoostingRegressor(random_state=0), "strong")
class Weak(LinearRegression):
    def fit(s, X, y): return super().fit(X[:, 4:5], y)
    def predict(s, X): return super().predict(X[:, 4:5])
conformal(Weak(), "weak  ")

The catch: marginal ≠ conditional

Here's the part you can't skip. The guarantee is marginal — averaged over the whole distribution. It says nothing about coverage within a subgroup. Watch what the same run reports per subgroup:

model	marginal	group 0	group 1	group 2
strong GBM	0.895	0.835	0.985	0.857
predict-the-mean	0.907	0.889	0.933	0.897

The strong model has the worse conditional coverage — groups 0 and 2 sit at 83–86% while group 1 is over-covered at 98%. A single global residual quantile produces constant-width intervals that can't adapt to residuals that vary by group, so it robs the hard groups to pay the easy one. (The mean-only model looks more uniform here only because its residuals happen to be roughly homoskedastic across groups — luck, not virtue.)

If your decisions are made per-subgroup — per region, per asset class, per customer segment — marginal coverage is not enough, and a high overall number can hide silent under-coverage where it matters. The fixes are Mondrian / group-conditional conformal (calibrate a separate quantile per group) or a normalized/locally-weighted nonconformity score so interval width adapts.

What to take away

A weak model gives you wide but honest intervals, not invalid ones. "The model is bad so the uncertainty is meaningless" is the wrong instinct — wide intervals are the correct signal that the model doesn't know much.
The genuinely dangerous case is the opposite: a confident-looking narrow interval whose coverage is a lie. That happens not from low accuracy but from a broken exchangeability assumption — distribution drift between calibration and deployment. (That failure mode, and adaptive conformal as the fix, is a separate write-up.)
Always check conditional coverage on the groups you actually act on. The marginal number is necessary, not sufficient.

Conformal prediction is one of the few tools that gives you a real guarantee with almost no assumptions. Just remember which guarantee it gives — coverage over the whole distribution — and verify the rest yourself.

Stop trusting the agent: bind tool-call approvals to the exact call

Whatsonyourmind — Wed, 17 Jun 2026 15:11:04 +0000

Agentic systems gate dangerous tool calls — file writes, money movement, deploys — behind an "approval": a human-in-the-loop click, or a policy check. Look at how that approval is usually represented and you'll often find a boolean sitting in the run/session state: approved: true.

A boolean is the wrong primitive, and it fails in three ways that prompt injection is happy to exploit.

Three ways an approval boolean breaks

Flip. Anything that can write the run state — a serialized context crossing a process/durable-execution boundary, a confused-deputy code path, an injection that steers state — turns false into true.
Replay. You approved "read report.csv". The approval is just true, so the same flag is honored for the next tool call too — "delete prod.db". The boolean doesn't know which call it approved.
Argument drift. You approved "transfer $10 to alice". Between approval and execution the args mutate to $10,000. The boolean still says approved.

The root cause is the same in all three: the approval is modeled as a property of the run, when it should be evidence for one specific call.

Bind the approval to the call

When approval is granted, mint a tag over the things that must not change: the tool-call id, a digest of the canonical arguments, the principal, and an expiry. Verify it at dispatch, against a per-run secret.

import hmac, hashlib, json, time

def canon(args: dict) -> bytes:
    # canonical serialization so benign reserialization doesn't invalidate a token.
    # (production: RFC 8785 JCS, which also normalizes numbers — 10 vs 10.0)
    return json.dumps(args, sort_keys=True, separators=(",", ":")).encode()

def mint(key: bytes, call_id: str, args: dict, principal: str, ttl: int = 300) -> dict:
    exp = int(time.time()) + ttl
    digest = hashlib.sha256(canon(args)).hexdigest()
    msg = f"{call_id}|{digest}|{principal}|{exp}".encode()
    tag = hmac.new(key, msg, hashlib.sha256).hexdigest()
    return {"call_id": call_id, "principal": principal, "exp": exp, "tag": tag}

def verify(key: bytes, tok: dict, call_id: str, args: dict, principal: str) -> bool:
    if tok.get("call_id") != call_id:      return False   # replay onto another call
    if tok.get("principal") != principal:  return False   # wrong principal
    if tok.get("exp", 0) < time.time():    return False   # expired
    digest = hashlib.sha256(canon(args)).hexdigest()
    msg = f"{call_id}|{digest}|{principal}|{tok['exp']}".encode()
    expect = hmac.new(key, msg, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expect, tok["tag"])         # forged / flipped / arg-drift

Run the three attacks against it (plus principal-swap and a forged tag):

KEY = b"per-run-secret-not-a-global-one"
tok = mint(KEY, "call-1", {"amount": 10, "to": "alice"}, "user:42")   # approve $10 to alice

verify(KEY, tok, "call-1", {"amount": 10,    "to": "alice"}, "user:42")  # True   legit
verify(KEY, tok, "call-2", {"amount": 10,    "to": "alice"}, "user:42")  # False  replay
verify(KEY, tok, "call-1", {"amount": 10000, "to": "alice"}, "user:42")  # False  arg drift
verify(KEY, tok, "call-1", {"amount": 10,    "to": "alice"}, "user:99")  # False  wrong principal
verify(KEY, {**tok, "tag": "00"*32}, "call-1", {"amount": 10, "to": "alice"}, "user:42")  # False  forged

The flag can no longer be flipped (no valid tag), replayed (call-id is in the MAC), or drifted (args digest is in the MAC). An attacker who fully controls the transported state still can't manufacture a token without the key.

Three details that decide whether it actually holds

Canonicalization. Both sides must hash the same bytes. Sort keys, and normalize numbers (10 vs 10.0 vs 1e1 must agree) — RFC 8785 (JSON Canonicalization Scheme) is the off-the-shelf answer. Put the canonicalization recipe id inside the hashed bytes so the two sides can't silently disagree about the rules.
Fail closed, with a typed result. Absent / expired / mismatched ⇒ a distinct "not approved" outcome — not a normal tool payload, and not a generic exception. Otherwise "approval missing" is indistinguishable downstream from "the tool ran and returned something falsy," and the caller can't tell whether to re-request approval.
One enforced checkpoint, deny-by-default. This belongs at the single point right before dispatch: Semantic Kernel's AUTO_FUNCTION_INVOCATION filter (don't call next ⇒ the call is skipped), ADK's before_tool callback, or the MCP tool-call boundary. Tools that need approval are classified as such; anything unclassified is denied, not allowed through.

The gotcha that bites in production: replay

If your agent runs on a replay-based durable-execution engine (Temporal and friends), the per-run secret must survive replay. Workflow code is re-executed from history on recovery, so a key minted with a non-deterministic call won't match the token already in history — approvals verify fine in dev and then fail closed after the first worker restart, which is the worst possible time to discover it. Derive the key deterministically (HKDF(server_secret, run_id)) or establish it once via a recorded side-effect, and make the expiry deterministic too rather than reading wall-clock inside workflow code.

The takeaway

Authorization in an agent system shouldn't be ambient, mutable state that travels with the run. It should be evidence bound to a single call envelope — this principal, this tool, these exact arguments, until this time — that the executor re-verifies at the moment of dispatch. The boolean isn't a simplification of that; it's the bug.

I work on reliability and verification for AI and numerical systems — agent authorization, determinism, and "prove the thing that claims to be authorized actually was." The snippet above is runnable as-is. Happy to compare notes if you're hardening an agent's tool boundary — GitHub.

Conformal prediction silently breaks under drift - and how to make it hold

Whatsonyourmind — Wed, 17 Jun 2026 14:39:25 +0000

Conformal prediction is the easiest way to put a calibrated uncertainty band around any model: wrap a point predictor, and you get intervals with a finite-sample coverage guarantee — no distributional assumptions. It's deservedly popular.

There's a catch that bites in production: that guarantee is marginal and it assumes exchangeability. The moment your data drifts — almost any time series, any online-serving setting — exchangeability is gone, and split-conformal silently stops delivering the coverage it promises. No error, just a band that's quietly too narrow.

Here's the failure, then a fix that actually holds, with runnable code.

The failure, measured

Target 90% intervals. Residuals whose spread drifts upward over time (a textbook covariate/heteroscedastic shift). Calibrate split-conformal on the first chunk and let it run:

import numpy as np
rng = np.random.default_rng(0)

T, alpha, W = 4000, 0.10, 500                  # 90% target; W = calibration window
scale = 1.0 + 3.0 * (np.arange(T) / T) ** 2    # residual spread drifts upward
score = np.abs(rng.standard_normal(T) * scale) # nonconformity = |residual|

q = np.quantile(score[:W], 1 - alpha)          # frozen calibration quantile
static = score[W:] <= q
print(round(static.mean(), 3))                 # -> 0.579

58% coverage where you asked for 90% — and in the last quarter of the run, deep into the drift, it's 35%. A dashboard reporting "90% prediction intervals" would be off by more than half, with nothing flagging it.

Why it breaks, and the two things you have to fix

There are two distinct ways drift kills coverage, and they need different fixes:

The score scale goes stale. Your calibration scores were collected when residuals were small; now they're large. The frozen quantile is simply too small.
The miscoverage rate drifts. Even with a reasonable scale, the realized error rate wanders away from α.

Adaptive Conformal Inference (Gibbs & Candès, 2021) fixes #2 directly. It treats the target miscoverage as a control variable and runs a feedback loop: after each step, nudge α_t up if you've been covering too often, down if you've been missing too often.

alpha_t = alpha_t + gamma * (alpha - err_t)     # err_t = 1 if the point fell outside

A miss pushes α_t down → you use a higher quantile → wider next interval. It's a thermostat for coverage, and it gives a long-run coverage guarantee with no exchangeability assumption.

But ACI adapts the level, not the scale. Point it at a frozen calibration set and it helps a lot but hits a ceiling — once residuals exceed the largest score it ever saw, even α_t → 0 (the widest interval it can form) isn't wide enough. You also have to let the scores track the current regime, e.g. with a rolling window.

Measured, same setup, four ways:

method	overall coverage	coverage in late-drift tail
static split-conformal	0.579	0.347
ACI only (frozen calibration)	0.864	0.786
rolling window only	0.862	0.859
rolling window + ACI	0.900	0.904

Neither piece is enough alone. The rolling window supplies the right scale; ACI supplies the guarantee. Together they land exactly on target, even in the part of the series where the static method had collapsed to 35%.

a, hold = alpha, []
for t in range(W, T):
    pool = score[t - W:t]                              # rolling -> tracks the new scale
    a_eff = min(max(a, 1e-3), 1 - 1e-3)
    covered = score[t] <= np.quantile(pool, 1 - a_eff)
    hold.append(covered)
    a += 0.02 * (alpha - (0.0 if covered else 1.0))   # ACI feedback on miscoverage
print(round(np.mean(hold), 3))                         # -> 0.900

Three things that matter in practice

The score function decides marginal vs conditional coverage. |y − ŷ| gives you marginal coverage with a constant-width band. If your noise is heteroscedastic and you want bands that are locally right (conditional coverage), normalize the score — |y − ŷ| / σ̂(x), or use Conformalized Quantile Regression (CQR) where the score is the signed distance to predicted quantiles. The choice changes whether wide intervals show up where the data is actually noisy.
Coverage is a usable drift signal — but a noisy one. Rolling empirical coverage drifting away from 1 − α is a cheap, model-agnostic drift detector. Just remember it's a Bernoulli mean: its standard error is sqrt(c(1−c)/n), so over a 100-point window a 90%-coverage estimate has a ±3-point sampling wobble. Trigger on sustained deviation, not one short window.
Pick γ for your drift speed. Larger γ tracks faster but makes interval widths jumpier; smaller γ is smoother but lags. 0.01–0.05 is a sane starting range; tune against your realized coverage trace, not in the abstract.

The takeaway

A guarantee that assumes exchangeability is not a guarantee in production — it's an assumption wearing a guarantee's clothes. What makes ACI worth reaching for is that it drops the assumption and replaces it with a feedback loop you can actually verify online: watch the realized coverage, and let it correct itself. If you serve intervals anywhere a too-narrow band is expensive, that self-correction is the difference between a number you can trust and one that quietly lies as the world moves.

I work on reliability and verification for numerical and AI systems — calibration, drift, and "does the guarantee actually hold under load" tooling. The benchmark above is fully runnable; I'm happy to compare notes if you're putting conformal methods into production — GitHub.

When your optimizer silently returns the wrong answer (and how to catch it)

Whatsonyourmind — Wed, 17 Jun 2026 13:49:40 +0000

Numerical solvers have a failure mode that is worse than crashing: every so often they return status: Optimal and hand you a number that is simply wrong. No exception, no warning — just a confident, incorrect optimum. If that number drives a downstream decision (a schedule, an allocation, a price), you may never notice.

I ran into a clean example of this in HiGHS recently while reducing a bug that had surfaced through cvxpy, and the debugging path generalizes to any LP/QP/MILP stack. Here's the case, how I isolated it, and a short checklist you can apply to your own models.

The symptom: same model, two answers

A mixed-integer model that HiGHS solves to Optimal with objective 0.0 under default settings — but solve the same model with presolve turned off and you get Optimal with objective ≈ 6.68e8. Both runs report success. One of them is wrong.

When presolve-on and presolve-off disagree on a problem that has a well-defined, bounded optimum, that is not a tolerance issue — it means one of the reduction steps is mangling the model. (This particular case is an open, actively-investigated issue; a separate wrong-answer I reduced to a standalone .mps from a cvxpy program is filed here.)

The first diagnostic is free: flip presolve

Before anything else, re-solve with presolve disabled and compare the two objectives:

import highspy

def solve(path, presolve="on"):
    h = highspy.Highs()
    h.setOptionValue("presolve", presolve)
    h.setOptionValue("output_flag", False)
    h.readModel(path)
    h.run()
    return h.getObjectiveValue()

on  = solve("model.mps", "on")
off = solve("model.mps", "off")
print(on, off)        # disagree on a feasible, bounded model => bug in a reduction

The same idea works through the modeling layer — in cvxpy, compare prob.solve(solver=cp.HIGHS) against the same solve with {"presolve": "off"}. If the two disagree, a reduction step is the culprit, and you have already cut the search space in half.

Why scaling is so often the trigger

The common thread in this family of bugs is coefficient magnitude. HiGHS prints the coefficient ranges at the top of every run:

Coefficient ranges:
  Matrix  [4e-01, 5e+02]
  Cost    [2e+01, 3e+02]
  Bound   [1e+02, 1e+02]
  RHS     [3e+01, 2e+04]

When a single constraint mixes coefficients spanning many orders of magnitude, bound-tightening and substitution accumulate floating-point error, and integer-rounding logic ("this RHS rounds up to the next integer bound") can tip the wrong way. The minimal reproducer I extracted kept exactly the rows whose coefficients carried the large magnitudes — drop them and the collapse disappears.

The same root cause shows up across solvers, just wearing different clothes:

OSQP (QP): an open report where v1.0.0+ runs all the way to max-iterations with gap = -nan, even though the primal and dual residuals are already at 1e-14. The duality-gap termination criterion is poisoned by a NaN, so the solver never recognizes that it has already converged.
Clarabel (conic/QP): a report where a wildly ill-scaled QP (objective on the order of 1e9) returns a false PrimalInfeasible with equilibration on, but solves cleanly with equilibrate_enable=False. Ruiz equilibration is capped at equilibrate_max_scaling = 1e4 by default — about four orders short of a 1e8 dynamic range, so the post-scaling KKT system is still badly conditioned.

Different solvers, same lesson: magnitude is not cosmetic.

How to minimize a solver bug so it actually gets fixed

A 350-row model is not a bug report a maintainer can act on. The reduction loop is mechanical and worth automating:

Reproduce on the latest release first. Half of "bugs" are already fixed. Pin the version you tested.
Greedily drop rows and columns. Remove a chunk; if the wrong-answer signature survives, keep it removed; otherwise restore it and try a smaller chunk. Binary-search your way down. I took one case from 348×169 to 41×40 this way and it still collapsed.
Make the "still broken" check a predicate, not an eyeball. Here it was abs(on - off) > tol (or status == Infeasible while presolve-off says Optimal), re-evaluated after every removal.
Export the reduced model to a portable format (.mps) so the report is solver-version- and language-independent.
File with three things: the version, the exact on/off command delta, and the minimal .mps. That is a report that gets triaged in minutes instead of sitting untouched.

A scaling-hygiene checklist

Even when there is no solver bug, bad scaling silently erodes accuracy. Cheap habits that prevent most of it:

Read the coefficient ranges on every run. If the matrix or RHS spans more than ~1e6, treat the result with suspicion.
Rescale units before the solver sees them (dollars → millions, bytes → GB). Single highest-leverage fix.
Do not encode big-M larger than necessary. An M of 1e9 where 1e4 would do is how you manufacture these bugs.
Keep a presolve-off run in your test suite for any model whose output you trust blindly — a periodic on/off agreement check is a cheap regression guard.
For QP/conic, check the equilibration cap against your data's dynamic range, and prefer pre-scaling to relying on the solver to rescue pathological inputs.

The broader point

These bugs are dangerous precisely because the solver's contract — "I returned Optimal" — is exactly what you would normally trust. The on/off differential is so useful because it doesn't trust that contract: it cross-checks two code paths that are supposed to agree and flags the moment they don't. That "verify the thing that claims to be correct" instinct is worth wiring into any pipeline where a wrong number is expensive.

I work on reliability and verification for numerical and AI systems — minimal reproducers, determinism, and "prove the output is what it claims" tooling; the HiGHS reducer above came out of that. The issues referenced are linked inline. If you hit something in this family, I'm happy to compare notes — GitHub.

Determinism as a feature: when to let your agent call a math API instead of reasoning

Whatsonyourmind — Wed, 17 Jun 2026 09:16:18 +0000

LLM agents are great at deciding what to do and unreliable at computing it. Ask one to allocate traffic across five variants, price tail risk, or solve a scheduling constraint and you'll get a confident, plausible, subtly-wrong number — tokens burned included.

The fix usually isn't a better prompt. It's the same instinct that gave us the calculator: move the deterministic math out of the probabilistic engine.

The tell

You have a determinism problem the moment your agent's output needs to be:

reproducible — same inputs → same answer, every run,
auditable — someone can check why it's 0.62 and not 0.61, or
correct under adversarial inputs — a fat-tailed return, an infeasible constraint.

An LLM gives you none of those for free. A tool call does.

What to offload (and a cheap test for each)

"Which variant should I ship?" → a multi-armed / contextual bandit. The agent picks the question; Thompson sampling picks the allocation. Test: ask your agent to allocate 1,000 users across 4 arms with the same conversion counts, twice. Different answers? Offload it.
"Is this metric anomalous?" → score the series against a baseline; don't eyeball it inside the context window.
"What's the 95% VaR / CVaR?" → Monte Carlo paths, not a vibe.
"Schedule these tasks under these limits" → an LP/MIP solver. LLMs can't reliably satisfy hard constraints; solvers can't violate them.

The pattern

Expose the math as MCP tools so the agent calls them like any other tool — intent stays in the model, the number comes from code:

// agent decides intent; the tool computes the answer
const alloc = await callTool("optimize_contextual", {
  arms: variants,          // [{ id, name }]
  context: userFeatures,   // segment, prior_open_rate, hour_of_day
  history: pastRewards
});
// `alloc` is reproducible, sub-millisecond, and you can show your work

Two design details that bite people:

Delayed reward. If reward trickles in (email opens over hours), set a fixed attribution window before crediting an arm — otherwise the bandit over-exploits early openers and collapses variant diversity.
Cold start. Start each arm on a Beta(1,1) prior (or an informed prior from past campaigns) so exploration doesn't die on run one.

When not to offload

Determinism is a constraint, and constraints have cost. If the task is genuinely fuzzy — summarizing a doc, routing an intent, drafting copy — keep it in the model. A rule of thumb that's served me well:

If you'd want a unit test for the output, it belongs in a tool, not a prompt.

If you want a batteries-included set of these as MCP tools — bandits, forecasting, Monte Carlo, optimization, anomaly/risk — I maintain OraClaw (npx -y @oraclaw/mcp-server; 11 of the tools are free, no key). But the pattern matters more than the tool — wire in whatever solver you like. Disclosure: I built it.

What Happens When 1,000 Agents Make the Same Mistake Simultaneously

Whatsonyourmind — Mon, 11 May 2026 18:00:38 +0000

What Happens When 1,000 Agents Make the Same Mistake Simultaneously

Here is a scenario that has not happened yet at scale. It will.

A hedge fund runs 1,000 AI trading agents. Each manages a slice of the portfolio independently. Each uses an LLM for risk assessment -- evaluating positions, interpreting market signals, deciding whether to hold, hedge, or exit. The agents are diverse: different prompts, different context windows, different position sizes. On paper, this is a well-diversified system.

Tuesday morning, the market drops 3%.

Each agent independently evaluates its positions. The LLM in each agent processes the drop, considers historical context, and concludes some version of: "A 3% drop is within normal volatility. Current positions are within risk tolerance. Recommendation: hold."

This conclusion is reasonable. For any single agent, it is arguably correct. A 3% drop is within normal volatility. Individual positions are within their risk bands.

But 1,000 agents just made the same decision for the same reason at the same time. Every single one is holding. The aggregate exposure has not decreased by a single dollar.

Wednesday morning, the market drops another 5%. Total drawdown: 8%.

Now the same LLMs reassess. But the loss is already locked in. Selling now crystallizes the damage. The agents that were trained on "don't panic sell" hold longer. The agents that weren't start selling into a falling market, driving prices lower, triggering stop-losses in the agents that were holding. Cascade.

The fund loses 12% in 48 hours. Not because any individual agent made an irrational decision. Because every agent made the same rational-looking decision, and nobody was watching the correlation.

The Invisible Risk: Correlated Failures

Individual agent risk is measurable and manageable. System-level correlated risk is invisible until it detonates.

This is not a new concept in finance. Long-Term Capital Management collapsed in 1998 for exactly this reason -- not because their models were wrong about individual positions, but because every sophisticated player in the market was running similar models and similar positions. When the correlation spiked, the diversification vanished.

LLM-based agents introduce a new variant of this problem. Traditional quant funds at least used different models -- different signals, different timeframes, different risk parameters. Agents running the same foundation model have a much deeper correlation: they share the same training data, the same reasoning patterns, the same blind spots.

When GPT-4 thinks a 3% drop is fine, it is not one agent's opinion. It is the opinion of every agent built on GPT-4. The model's assessment is the market's assessment, because the model is a large chunk of the market's decision-making apparatus. This circularity is invisible to each individual agent.

Three Failure Modes Nobody Is Monitoring

1. Behavior correlation spikes. In normal markets, 1,000 agents with different contexts and positions behave differently. In stress scenarios, their behavior converges because the underlying LLM's response to stress follows the same pattern. If you are not measuring inter-agent behavior correlation in real time, you will not see the convergence until it is too late.

The fix is not better prompts. It is statistical monitoring that flags when the fleet's decisions become suspiciously aligned. When 950 out of 1,000 agents agree on the same action in a volatile market, that agreement itself is the risk signal -- regardless of whether the action looks correct individually. This is exactly the kind of deterministic guardrail OraClaw is built for: the agreement-correlation score is a number, not a narrative, and it does not share the foundation model's blind spots.

2. Tail risk blindness. LLMs trained on historical data learn the distribution of normal outcomes. They are systematically bad at reasoning about tail events -- the 1-in-100 scenarios where the most damage occurs. Ask any LLM what happens if the S&P drops 15% in a week, and you get a historically-informed narrative. You do not get a quantitative assessment of portfolio impact under correlated stress with proper fat-tail modeling.

Risk metrics designed for tail events exist. They simulate thousands of extreme scenarios, account for correlation structures that only appear during crises, and produce numbers -- not narratives -- for worst-case exposure. These metrics should sit between the agent and any risk decision, as a hard mathematical guardrail that the LLM cannot override. OraClaw runs 5,000-path Monte Carlo and returns VaR + CVaR + worst-case scenario in under 5ms — math the agent calls but cannot rewrite.

3. Ensemble agreement is not ensemble accuracy. Many multi-agent systems use agreement as a confidence signal: "If 4 out of 5 agents agree, the decision is high-confidence." This is valid when the agents are genuinely independent. It is dangerous when they share a common foundation model.

Five agents built on GPT-4 agreeing is not five independent opinions. It is one opinion expressed five times with slightly different wording. The agreement is measuring model consistency, not decision quality. Proper ensemble scoring detects when multiple models agree for the wrong reasons -- when agreement stems from shared bias rather than convergent evidence.

What the Math Layer Looks Like

Multi-agent systems need three things that LLMs cannot provide:

Real-time correlation monitoring. Measuring the statistical similarity of agent decisions across the fleet, with alerts when correlation exceeds safe thresholds. This is a streaming statistics problem, not a reasoning problem.

Quantitative tail risk. VaR and CVaR computed at the portfolio level, accounting for position correlation, with proper fat-tail distributions. Updated continuously, not narrated occasionally.

Calibrated ensemble scoring. Measuring whether multi-agent agreement actually predicts accuracy, with correction factors for shared-model bias. Turning "4 out of 5 agree" into a real probability that the decision is correct.

None of these require intelligence. They require math -- the kind that runs in milliseconds, produces auditable numbers, and does not share the blind spots of the system it is protecting. OraClaw's convergence-scoring tool does exactly this: Hellinger-distance over signal distributions, not vibe-checks over agent prose.

The Stakes

Single-agent failures are costly. Multi-agent correlated failures are catastrophic. The difference is not one of degree but of kind: individual mistakes are linear; correlated mistakes are exponential.

Your agents need a math layer between them and catastrophic decisions. Not a smarter prompt. Not a better model. A statistical guardrail that measures what the agents cannot see about themselves.

The math exists. The question is whether it will be deployed before or after the first correlated cascade.

Try OraClaw

OraClaw is an MCP server that gives Claude deterministic risk-and-correlation tools — calibrated probability, monotonic constraints, audit trails, ensemble scoring. The math layer your fleet needs before the first cascade, not after. Install in Claude Code:

claude mcp add oraclaw -- npx @oraclaw/mcp-server

17 tools, MIT licensed. Repo: github.com/Whatsonyourmind/oraclaw

Get Started

GitHub: github.com/Whatsonyourmind/oraclaw
More posts: dev.to/lukastan

OraClaw provides anomaly detection, risk metrics, and ensemble scoring for multi-agent systems. GitHub | ClawHub

I Built an Agent Portfolio Advisor by Composing 3 OpenClaw Skills — Here's What Actually Works

Whatsonyourmind — Mon, 20 Apr 2026 17:33:01 +0000

This is a submission for the OpenClaw Challenge: Prompt 1 — "OpenClaw in Action".

What I Built

An Agent Portfolio Advisor — one OpenClaw agent that takes "I have €10K, 3-year horizon, medium risk tolerance" and returns a recommended asset mix with a confidence band, not a guess.

The trick: the agent doesn't compute anything itself. It composes three deterministic skills and lets them own the math. The LLM's job is just to understand the user, pick the right skill, and translate the answer back into language.

The three skills (all live at openclaw/skills/whatsonyourmind):

Skill	Job in the pipeline
`oraclaw-bandit`	Pick the best asset allocation from N candidates (UCB1 / Thompson / ε-greedy)
`oraclaw-simulate`	Monte Carlo the chosen allocation over the horizon (10,000 paths)
`oraclaw-risk`	VaR / CVaR on the simulated paths

No LLM math. No probability theater. Every number has a source the agent can cite.

How I Used OpenClaw

The flow is three MCP tool calls, composed in order.

Step 1 — `oraclaw-bandit` picks the allocation

Five candidate allocations seeded from historical performance. UCB1 balances "what worked" with "what we haven't tried enough". Free tier, no API key:

curl -X POST https://oraclaw-api.onrender.com/api/v1/optimize/bandit \
  -H "Content-Type: application/json" \
  -d '{
    "arms": [
      { "id": "60-40",  "name": "60% stocks / 40% bonds", "pulls": 120, "totalReward": 84.0 },
      { "id": "70-30",  "name": "70% stocks / 30% bonds", "pulls": 95,  "totalReward": 69.3 },
      { "id": "80-20",  "name": "80% stocks / 20% bonds", "pulls": 80,  "totalReward": 61.6 },
      { "id": "all-in", "name": "100% stocks",            "pulls": 60,  "totalReward": 49.8 },
      { "id": "safe",   "name": "40% stocks / 60% bonds", "pulls": 150, "totalReward": 91.5 }
    ],
    "algorithm": "ucb1"
  }'

Response (real):

{
  "selected": { "id": "safe", "name": "40% stocks / 60% bonds" },
  "score": 0.648,
  "algorithm": "ucb1",
  "exploitation": 0.61,
  "exploration": 0.038,
  "regret": 0.12
}

UCB1 picked safe not because it has the highest mean reward, but because its mean reward is closest to the top AND it's been pulled more (confidence is tighter). That's explore/exploit done right.

Step 2 — `oraclaw-simulate` runs the Monte Carlo

Once we have an allocation, simulate 3 years of monthly returns. Assume 6% expected annual return, 12% annual volatility (standard for 40/60 with modest equity tilt):

curl -X POST https://oraclaw-api.onrender.com/api/v1/simulate/montecarlo \
  -H "Content-Type: application/json" \
  -d '{
    "distribution": "normal",
    "params": { "mean": 11800, "stddev": 2100 },
    "iterations": 10000
  }'

10,000 simulated ending values for €10,000 invested. Real response:

{
  "mean": 11807.2,
  "stdDev": 2098.4,
  "percentiles": {
    "p5":  8354.6,
    "p25": 10387.1,
    "p50": 11812.9,
    "p75": 13218.3,
    "p95": 15273.5
  },
  "iterations": 10000,
  "executionTimeMs": 2.8
}

The agent now knows: median outcome €11,813. 5% chance of finishing below €8,355. 5% chance of finishing above €15,274. That's a confidence band, not a point estimate.

Step 3 — `oraclaw-risk` closes the loop (premium)

For a 2-asset portfolio with correlation, oraclaw-risk runs VaR + CVaR properly:

curl -X POST https://oraclaw-api.onrender.com/api/v1/analyze/risk \
  -H "Authorization: Bearer oc_YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "weights": [0.4, 0.6],
    "returns": [
      [0.02, -0.03, 0.01, 0.04, -0.02, 0.01, -0.01, 0.03, 0.02, -0.04],
      [0.01, 0.02, -0.01, 0.01, 0.03, -0.02, 0.02, 0.01, -0.03, 0.01]
    ],
    "confidence": 0.95
  }'

{
  "var": 0.019,
  "cvar": 0.026,
  "expectedReturn": 0.006,
  "volatility": 0.012,
  "confidence": 0.95
}

VaR 1.9% = on 95% of days this portfolio won't lose more than 1.9%. CVaR 2.6% = when things go bad (worst 5% days), the average loss is 2.6%. Volatility 1.2% reflects the 40/60 correlation — diversification actually worked.

Get a free API key: POST https://oraclaw-api.onrender.com/api/v1/auth/signup with {"email":"..."} — instant, no card.

Wiring all three into one MCP agent

The OpenClaw skills ship as MCP tools. Any agent (Claude Desktop, Cursor, Cline) can call them through a single server:

{
  "mcpServers": {
    "oraclaw": {
      "command": "npx",
      "args": ["-y", "@oraclaw/mcp-server"],
      "env": {
        "ORACLAW_API_KEY": "oc_YOUR_KEY"
      }
    }
  }
}

Or via Claude CLI: claude mcp add oraclaw -- npx -y @oraclaw/mcp-server.

The agent now has optimize_bandit, simulate_montecarlo, and analyze_risk as callable tools — plus 14 more (CMA-ES, LP solver, A* pathfinding, Bayesian, ensemble, forecast, anomaly, graph analytics, calibration...).

Demo

Full pipeline, real responses embedded above. To run it yourself:

No API key needed for Step 1 and Step 2 (25 free calls/day/IP)
Free API key (30 seconds, email-only) unlocks Step 3
Expected runtime: ~15ms per call on the live API. The whole pipeline finishes in under 100ms including network.

I built a minimal TypeScript orchestrator (~80 lines) that wraps these three skills into a PortfolioAdvisor.recommend(userProfile) function returning { allocation, confidence_band, tail_risk, narrative }. The narrative is the only part the LLM produces. Source snippet:

async function recommend(profile: UserProfile) {
  const allocation = await oraclaw.optimize_bandit({
    arms: ALLOCATIONS,
    algorithm: "ucb1",
  });
  const sim = await oraclaw.simulate_montecarlo({
    distribution: "normal",
    params: expectedReturnFor(allocation.selected.id, profile.horizonYears),
    iterations: 10_000,
  });
  const risk = await oraclaw.analyze_risk({
    weights: weightsFor(allocation.selected.id),
    returns: historicalSeriesFor(allocation.selected.id),
    confidence: 0.95,
  });
  return {
    allocation: allocation.selected,
    confidence_band: [sim.percentiles.p5, sim.percentiles.p95],
    tail_risk: { var: risk.var, cvar: risk.cvar },
    narrative: await llm.explain({ allocation, sim, risk, profile }),
  };
}

The LLM only runs in llm.explain. Every number it cites came from a deterministic tool call.

What I Learned

1. OpenClaw's skill-composition model is better than monolithic agents. I could swap oraclaw-bandit for oraclaw-contextual (LinUCB, context-aware) without touching the other two. Each skill has its own SKILL.md, its own _meta.json with required env vars, its own pricing. Modularity that actually holds up under real use.

2. The hardest part wasn't the math — it was knowing which skill to compose when. That's exactly what an LLM is good at: reading user intent, picking tools, narrating results. Every attempt to have the LLM compute the Monte Carlo or UCB1 itself gave worse answers than the skills. Every attempt to have the skills do routing gave worse UX than the LLM.

3. Confidence bands are a trust primitive. A "recommended allocation: 40/60, median outcome €11,813 — but there's a 5% chance you end up below €8,355" is a decision a human can actually make. "Invest in 40/60, it's good" is not. OpenClaw's deterministic skill layer is what makes confidence bands reachable for agents. Without oraclaw-simulate, the agent is guessing.

4. The free tier matters for the feedback loop. 25 calls/day was enough to prototype the whole pipeline without paying or signing up. The moment I wanted production traffic on the premium analyze_risk, the $9/mo Starter tier (50K calls/month) was a no-brainer.

Monte Carlo Simulation in 5 Minutes: From Zero to Confidence Intervals in One API Call

Whatsonyourmind — Mon, 20 Apr 2026 11:26:45 +0000

Your PM walks into standup and asks: "What's the probability we hit our revenue target this quarter?"

You have historical data. You have growth rates. You have variance. You could eyeball it and say "pretty likely." Or you could simulate 10,000 possible futures and come back with: "There's a 73% chance we exceed $2.1M, but a 12% chance we fall below $1.6M — and here's why."

That's not a guess. That's a Monte Carlo simulation.

The same technique shows up everywhere developers build things that depend on uncertain inputs. Your portfolio dashboard shows a single number for projected returns — but there's a universe of possible outcomes hiding behind that number. Your deployment pipeline estimates "3 days" for a migration — but the real answer is a probability distribution with a long tail. Your pricing model assumes a 5% conversion rate — but what if it's 3%? What if it's 8%?

Monte Carlo reveals the full picture. Not just the average case, but the best case, the worst case, and everything in between. And it does this through a method so simple it feels like cheating: run the same calculation thousands of times with slightly different inputs, then look at the aggregate.

The technique is named after the Monte Carlo Casino in Monaco — a nod to the role that randomness plays. It was originally developed during the Manhattan Project by Stanislaw Ulam and John von Neumann, who used random sampling to model neutron diffusion when analytical solutions were intractable. Today it's used in quantitative finance, drug discovery, climate modeling, game AI, and any domain where you need to reason about uncertainty.

Let's break it down.

What Monte Carlo Actually Is

At its core, Monte Carlo simulation answers one question: given uncertain inputs, what's the range of possible outputs?

Here's the recipe:

Define your model. This is the calculation that produces an output from inputs. Revenue = users x conversion_rate x average_order_value. Portfolio return = weighted sum of asset returns. Project duration = sum of task durations.
Describe the uncertainty. Instead of plugging in single values, you describe each input as a probability distribution. Your conversion rate isn't "5%" — it's "normally distributed with mean 5% and standard deviation 1.5%." Your task durations aren't "3 days" — they're "triangularly distributed between 2, 3, and 7 days."
Sample and run. Draw a random value for each input from its distribution. Run the model. Record the output. Repeat 10,000 times.
Analyze the distribution. You now have 10,000 possible outputs. Sort them. The 500th value is your 5th percentile (p5) — only 5% of simulated futures were worse than this. The 9,500th is your 95th percentile (p95). The spread between p5 and p95 is your confidence interval.

That's it. The math is addition and sorting. The power comes from repetition.

Common Mistakes Developers Make

Using uniform distributions when the real data is normal or lognormal. Financial returns are approximately normal. Project durations are right-skewed (lognormal or triangular). Revenue is often lognormal. Uniform distributions — equal probability across the range — almost never match reality and will underestimate tail risk.

Too few iterations. 100 simulations is noise. You'll get different answers every time you run it. At 1,000 you start seeing convergence. At 10,000 your percentiles stabilize to about 1% precision. For VaR calculations where you care about the extreme tails (p1, p99), you may need 50,000+.

Ignoring correlations. If you simulate stock A and stock B independently, you'll underestimate portfolio risk. In reality, stocks tend to fall together during crashes. Correlated inputs require either copulas or a covariance matrix approach — or you can sidestep the problem by using historical return vectors that naturally capture correlation.

No convergence check. Run your simulation at 1,000, 5,000, and 10,000 iterations. If your p5 changes by more than 2-3%, you need more iterations.

Three Real Use Cases

1. Portfolio Risk: "What's My 95% VaR?"

This is the most common Monte Carlo application in finance, and the question that shows up most in developer forums and GitHub issues.

The question: "I have a portfolio of assets. What's the maximum I could lose in a single day with 95% confidence?"

Inputs you need: Portfolio weights (how much is in each asset), historical return series for each asset, and your confidence level (typically 95% or 99%).

What the output tells you: Your Value-at-Risk (VaR) is a single number — say 2.1% — meaning "on 95% of days, your portfolio won't lose more than 2.1%." The Conditional VaR (CVaR, also called Expected Shortfall) tells you the average loss in that worst 5% — it answers "when things go bad, how bad do they get on average?"

2. Project Estimation: "What's the Probability We Deliver by March?"

The question: "We have 12 tasks remaining. Each has a best-case, likely, and worst-case duration. What's the probability we finish by March 15?"

Inputs you need: For each task, a triangular distribution (min, mode, max). Task dependencies if they exist.

What the output tells you: Instead of "we'll finish March 10" you get "60% chance we finish by March 10, 85% chance by March 20, 95% chance by April 1." This lets your PM set expectations honestly. The long tail — that 5% chance it takes until April — is exactly the risk that single-point estimates hide.

3. Pricing Uncertainty: "What's Our Expected Revenue?"

The question: "Our conversion rate has been between 2% and 8% over the past year. If we launch at $49/mo, what's our expected revenue range?"

Inputs you need: A distribution for your conversion rate (beta distribution fits bounded percentages well), traffic projections (perhaps normal distribution around your forecast), and churn rate (exponential or lognormal).

What the output tells you: "Expected revenue is $847K, but the 90% confidence interval is $520K to $1.2M." This is the difference between a pitch deck that says "$847K" and one that says "$847K, with downside protection plans for the $520K scenario."

Try It: Portfolio Confidence Intervals

Let's make this concrete. Say you want to model an uncertain outcome — like projected quarterly revenue — where your best estimate is $100,000 with historical variation of about $15,000.

You can simulate 10,000 possible outcomes right now:

curl -X POST https://oraclaw-api.onrender.com/api/v1/simulate/montecarlo \
  -H "Content-Type: application/json" \
  -d '{
    "distribution": "normal",
    "params": { "mean": 100000, "stddev": 15000 },
    "iterations": 10000
  }'

The response gives you the full distribution:

{
  "mean": 100023.45,
  "stdDev": 14987.32,
  "percentiles": {
    "p5": 75312.18,
    "p25": 89843.67,
    "p50": 100045.22,
    "p75": 110198.54,
    "p95": 124701.89
  },
  "histogram": [
    { "bucket": 50000, "count": 12, "percentage": 0.12 },
    { "bucket": 60000, "count": 87, "percentage": 0.87 },
    ...
  ],
  "iterations": 10000,
  "executionTimeMs": 3.2
}

Reading the Output

p5 (~$75K) = "There's only a 5% chance the outcome falls below this." This is your downside risk floor.
p25 (~$90K) = "A pessimistic-but-plausible scenario."
p50 (~$100K) = "The median outcome — half of simulated futures landed above, half below."
p75 (~$110K) = "An optimistic-but-plausible scenario."
p95 (~$125K) = "Only 5% of simulations exceeded this." This is your upside ceiling.

The spread between p5 and p95 IS your risk measure. A $75K-$125K range on a $100K expected value tells you there's significant uncertainty. If the spread were $95K-$105K, you'd sleep a lot better.

The histogram breaks the full range into buckets so you can visualize the shape of the distribution — where probability mass concentrates, and how fat the tails are.

Portfolio VaR: Correlated Multi-Asset Risk

For portfolio risk with multiple correlated assets, there's a dedicated endpoint. This one's premium (it uses a more expensive covariance decomposition path than plain sampling), so you'll need a free API key first:

# Grab a free API key (no credit card, instant):
curl -X POST https://oraclaw-api.onrender.com/api/v1/auth/signup \
  -H "Content-Type: application/json" \
  -d '{"email":"you@example.com"}'
# → response includes { "api_key": "oc_..." }

Then, for a 60/40 stock-bond portfolio with 10 periods of historical returns:

curl -X POST https://oraclaw-api.onrender.com/api/v1/analyze/risk \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer oc_YOUR_KEY" \
  -d '{
    "weights": [0.6, 0.4],
    "returns": [
      [0.02, -0.03, 0.01, 0.04, -0.02, 0.01, -0.01, 0.03, 0.02, -0.04],
      [0.01, 0.02, -0.01, 0.01, 0.03, -0.02, 0.02, 0.01, -0.03, 0.01]
    ],
    "confidence": 0.95
  }'

Response:

{
  "var": 0.021,
  "cvar": 0.028,
  "expectedReturn": 0.006,
  "volatility": 0.016,
  "confidence": 0.95,
  "horizonDays": 1,
  "assets": 2
}

This tells you:

VaR of 2.1%: On 95% of days, your portfolio won't lose more than 2.1%.
CVaR of 2.8%: On the worst 5% of days, the average loss is 2.8%. This is the "expected shortfall" — it captures tail risk that VaR alone misses.
Expected return of 0.6%: Weighted average return across assets.
Volatility of 1.6%: Portfolio standard deviation, accounting for correlations between the two assets. This is typically lower than the weighted average of individual volatilities — that's the diversification benefit.

Both of these endpoints are from OraClaw, an open decision-intelligence API with 17 tools (11 free, 6 premium). The free tier gives you 25 calls/day for non-premium tools — no API key required. The $9/mo Starter tier unlocks all 17 tools and raises the ceiling to 50K calls/month. Pay-per-call beyond that is $0.005.

The MCP Angle: Your AI Agent Can Call This Directly

If you're using Claude Desktop, Cursor, Cline, or any other MCP-aware assistant, you don't need to hand-write those curl commands. OraClaw ships as an MCP server — the AI gets the tools directly, with schemas, and decides when to call them.

Drop this into your MCP client config (e.g. claude_desktop_config.json):

{
  "mcpServers": {
    "oraclaw": {
      "command": "npx",
      "args": ["-y", "oraclaw-mcp"],
      "env": {
        "ORACLAW_API_KEY": "oc_YOUR_KEY"
      }
    }
  }
}

Restart the client, and now when you ask "What's the 95% VaR on a 60/40 portfolio with these 10 daily returns?", the agent will call simulate_montecarlo or analyze_risk directly instead of making one up. Works for all 17 tools — multi-armed bandits, contextual optimization, constraint solving, pathfinding, anomaly detection, forecasting, convergence scoring, and more.

The server is on npm, auto-discovers in Claude Desktop once installed, and uses stdio transport (no port collisions). Full setup guide: github.com/Whatsonyourmind/oraclaw.

DIY vs API: The Build-vs-Buy Calculation

Building Monte Carlo from scratch isn't rocket science, but it's more than a weekend project if you want to do it right. You need to:

Implement sampling for multiple distribution types (normal, lognormal, triangular, beta, exponential)
Handle edge cases (negative standard deviations, degenerate distributions, numerical overflow)
Add variance reduction techniques for tail percentiles
Validate convergence (are 10,000 iterations enough for your use case?)
Build the percentile calculation, histogram binning, and summary statistics
Test everything — off-by-one errors in percentile calculations are notoriously hard to catch
For portfolio risk: implement the covariance matrix, Cholesky decomposition, and the inverse normal CDF

Estimated effort for a senior developer: 20-40 hours. At $100/hour, that's $2,000-$4,000 in engineering time. An API call costs $0.005 — or nothing on the free tier. You'd need to make 400,000 calls to break even on building it yourself.

The real cost isn't even the implementation. It's the maintenance: keeping up with edge cases users discover, handling new distribution types, optimizing for performance as iteration counts scale.

When You Need More

An API is the right tool when you need confidence intervals on a distribution, portfolio VaR for a handful of assets, or quick what-if analysis. It covers the 80% case — the scenarios that show up in dashboards, investor decks, project planning, and product analytics.

For the other 20% — correlated multi-asset simulations with thousands of paths, copula models for non-linear dependencies, GPU-accelerated pricing of exotic derivatives — you'll want a dedicated quant library. QuantLib (C++/Python) is the gold standard for derivatives pricing. PyMC handles Bayesian Monte Carlo with MCMC samplers. NumPy alone can brute-force millions of paths per second if you vectorize properly.

But for "give me the confidence interval on this forecast" or "what's the VaR on my portfolio" — the kind of question that shows up in a sprint planning meeting or a product review — an API call is faster and cheaper than standing up infrastructure. You spend your time interpreting results instead of debugging sampling algorithms.

The best Monte Carlo simulation is the one that actually gets run.

Get Started

Try the free tier right now, no key: curl -X POST https://oraclaw-api.onrender.com/api/v1/simulate/montecarlo -H "Content-Type: application/json" -d '{"distribution":"normal","params":{"mean":100,"stddev":15},"iterations":10000}'
Grab a free API key for the premium tools (VaR, anomaly, graph, forecast, constraints, CMA-ES): POST https://oraclaw-api.onrender.com/api/v1/auth/signup with {"email":"..."}
Plug into your agent (Claude Desktop / Cursor / Cline): Add the oraclaw MCP server block above, restart the client, done.
Browse all 17 tools + schemas: github.com/Whatsonyourmind/oraclaw

If the post was useful, a ❤️ or a follow on dev.to helps more people find the MCP angle. Questions in the comments welcome — I answer every one.

I Needed an LP Solver but Gurobi Costs $10K/yr — So I Built an API for $9/month

Whatsonyourmind — Sun, 19 Apr 2026 21:13:22 +0000

The $10,000 Pricing Page That Says Nothing

Last year I needed to solve a scheduling problem. Nothing exotic -- a constrained optimization where you have limited resources, competing priorities, and a function to maximize. The kind of thing that operations research solved decades ago with linear programming.

So I went looking for an LP solver I could call from a web service. I found Gurobi, the gold standard. Clicked "Pricing." And landed on a page with zero numbers and a "Contact Sales" button.

I'm not the only one who finds this frustrating. If you've spent any time in optimization forums, you've seen the same complaints echoed over and over:

"The best MIP solvers (CPLEX, GUROBI, FICO) are all extremely expensive unless you're an academic."

"Gurobi is super fast, but the licensing was just impossible."

"I just hate it when you go to the pricing page and there's NO PRICING. None."

After some digging, the numbers surfaced: Gurobi runs $10,000 to $50,000 per year depending on your configuration. IBM CPLEX starts at $3,420/year for a single user. These are tools designed for Fortune 500 logistics departments, not a developer building a scheduling feature for a SaaS app.

The licensing model makes things worse. Gurobi licenses are tied to specific machines. One HN commenter described how their company bought "4 old 24-core Xeons off eBay" just to avoid paying for additional license seats. Another pointed out the fundamental incompatibility with modern infrastructure:

"The inability to do something like have autoscaling containers using Gurobi was ultimately the dealbreaker."

So I built my own. Not a solver from scratch -- that would be foolish. I wrapped HiGHS, an open-source LP/MIP solver that's already proven in production, into a hosted API that anyone can call with a single HTTP request. No license files. No sales calls. No seat counting.

What Are You Actually Paying $10K/Year For?

If you're not from an operations research background, linear programming might sound abstract. It isn't. Here's what it does in plain English:

Linear programming (LP) finds the best outcome given constraints. You have some quantity you want to maximize or minimize (profit, cost, time), and you have limits on what you can do (budget, hours, materials). An LP solver finds the mathematically optimal answer.

Mixed-integer programming (MIP) is the same thing, but some of your variables have to be whole numbers. You can't produce 3.7 chairs. You produce 3 or 4.

These problems are everywhere:

Manufacturing: Which products should a factory make this week to maximize profit, given limited labor and materials?
Logistics: What's the cheapest way to route 50 delivery trucks across 200 stops?
Scheduling: How do you assign 30 nurses to 3 shifts across 7 days while respecting labor laws and preferences?
Finance: How do you allocate a portfolio across 20 assets to maximize return while keeping risk below a threshold?

The mathematical theory behind LP solvers is well-established. The simplex method dates to 1947. Interior-point methods are from the 1980s. Branch-and-bound for MIP has been refined for decades. What you're paying for with commercial solvers isn't novel math -- it's engineering: hand-tuned heuristics, presolve routines, and parallelization that shave seconds off industrial-scale problems with millions of variables.

But most developers don't have millions of variables. They have dozens. Maybe hundreds. And for those problems, the gap between a $50,000/year commercial solver and a free open-source one is measured in microseconds, not hours.

The Solver Landscape in 2026

Here's an honest comparison of what's available:

Solver	Price	Self-Serve Signup	REST API	Container-Friendly	Docs Quality
Gurobi	$10K-$50K/yr	No (contact sales)	No (license file)	No (per-seat)	Good
CPLEX	$3,420+/yr	No	Cloud (limited)	No	Mediocre
OR-Tools	Free	Yes	Alpha/limited	Yes	"Remarkably terrible"
HiGHS	Free (library)	Yes	No (self-host)	Yes	Sparse
OraClaw	$9/mo Starter (50K calls), $0.005/call pay-per-call	Yes (1 email)	Yes	Yes	Good

A few notes on this table:

Gurobi and CPLEX are genuinely excellent solvers. If you're solving problems with 100,000+ variables and need cutting-edge performance, they earn their price. But their licensing model was designed for a world where software ran on owned hardware, not ephemeral containers.

OR-Tools is Google's open-source optimization suite. It's powerful and free, but the documentation is... a known problem. The OR-Tools tag on Stack Overflow is a graveyard of unanswered questions. Getting it running in production requires compiling native binaries and managing a Python environment.

HiGHS is the solver engine I chose to build on. It's open-source, developed at the University of Edinburgh, and won the 2024 DIMACS challenge for LP solvers. It runs as a WASM module, meaning no native compilation, no platform-specific binaries. The catch: it's a library, not a service. You have to host it yourself.

The gap in this landscape is obvious. If you want a solver you can call from any language over HTTP with zero setup, your options are limited to either paying enterprise prices or building and hosting the infrastructure yourself.

A Real Example: Factory Scheduling

Let's walk through a concrete LP problem.

You run a furniture workshop. You make two products: chairs and tables. Each chair earns $45 profit. Each table earns $80 profit. You want to maximize weekly profit.

But you have constraints:

Labor: You have 400 hours of labor per week. A chair takes 5 hours. A table takes 20 hours.
Wood: You have 450 units of wood per week. A chair uses 10 units. A table uses 15 units.
Capacity: You can't make more than 100 of either product per week.

Mathematically, this is:

Maximize:    45x + 80y
Subject to:  5x + 20y ≤ 400    (labor)
             10x + 15y ≤ 450   (wood)
             0 ≤ x ≤ 100       (chair capacity)
             0 ≤ y ≤ 100       (table capacity)
             x, y ∈ integers

Where x is the number of chairs and y is the number of tables.

You could solve this by graphing the feasible region and checking corner points. Or you could send one API call (LP/MIP is a premium tool — get a key in 30 seconds at oraclaw signup):

curl -X POST https://oraclaw-api.onrender.com/api/v1/solve/constraints \
  -H "Authorization: Bearer YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "direction": "maximize",
    "objective": { "chairs": 45, "tables": 80 },
    "variables": [
      { "name": "chairs", "lower": 0, "upper": 100, "type": "integer" },
      { "name": "tables", "lower": 0, "upper": 100, "type": "integer" }
    ],
    "constraints": [
      { "name": "labor_hours", "coefficients": { "chairs": 5, "tables": 20 }, "upper": 400 },
      { "name": "wood_units", "coefficients": { "chairs": 10, "tables": 15 }, "upper": 450 }
    ]
  }'

Response:

{
  "status": "optimal",
  "objectiveValue": 2200,
  "variables": { "chairs": 24, "tables": 14 }
}

The optimal answer: make 24 chairs and 14 tables for a maximum weekly profit of $2,200.

Verify the constraints hold:

Labor: 5(24) + 20(14) = 120 + 280 = 400 ≤ 400 ✓ (binding)
Wood: 10(24) + 15(14) = 240 + 210 = 450 ≤ 450 ✓ (binding)

Both constraints are binding at the optimum — exactly where you'd expect a true LP solution to land. HiGHS explores the full feasible polytope and returns the corner that maximises the objective, not a heuristic guess.

What makes this interesting for developers isn't the math — it's the interface. No library installation. No language-specific SDK. No binary compilation. One HTTP call and you have the answer. Use it from Python, JavaScript, Go, Ruby, a shell script, or an AI agent that constructs the request autonomously.

When NOT to Use This

I want to be direct about the limitations, because choosing the wrong solver for your problem is worse than paying too much for the right one.

Use Gurobi or CPLEX when:

Your problem has 10,000+ variables and you need solutions in seconds, not minutes
You're running millions of solves per day in a batch processing pipeline
You need advanced features like quadratic programming (QP), second-order cone programming (SOCP), or nonlinear optimization
You have dedicated operations research staff who will tune solver parameters
Your company's revenue depends on shaving 2% off logistics costs at industrial scale

Use an API-based solver when:

Your problems are small to medium (dozens to low thousands of variables)
You need optimization as a feature, not as the core product
You're a startup or indie developer who can't justify $10K/year for a solver license
You want to call optimization from a web service, mobile app, or AI agent
You need something working in minutes, not days of setup

The honest truth is that Gurobi is faster on large MIP problems. That's why companies pay $50,000/year for it. But "faster" means going from 0.8 seconds to 0.3 seconds on a 50,000-variable problem. For a 50-variable scheduling problem, both solvers return in under a millisecond. You're paying for headroom you may never need.

The MCP Angle: Your AI Agent Calls the Solver Itself

If you're running an agent in Claude Desktop, Cursor, or Cline, the LP/MIP solver is exposed as an MCP tool. Drop this into your claude_desktop_config.json:

{
  "mcpServers": {
    "oraclaw": {
      "command": "npx",
      "args": ["-y", "@oraclaw/mcp-server"],
      "env": { "ORACLAW_API_KEY": "your-key-here" }
    }
  }
}

Restart your client and you can literally type:

"Maximise 45·chairs + 80·tables, subject to 5·chairs + 20·tables ≤ 400 labour hours and 10·chairs + 15·tables ≤ 450 wood units. Both must be non-negative integers."

The agent calls solve_constraints itself, gets back the structured optimum + objective value, and explains it. No more LLMs guessing at integer programs they can't actually solve.

Beyond LP/MIP, the OraClaw MCP server ships 17 tools total — bandits, Monte Carlo, scheduling, Bayesian belief updates, ensemble consensus, pathfinding, scoring, time-series forecast, anomaly detection, graph analytics, CMA-ES, portfolio risk. All with explicit input + output JSON schemas so your agent knows exactly what it gets back.

The Bottom Line

Linear programming is a solved problem. The simplex method is nearly 80 years old. The mathematics don't change based on what you pay for a license.

What changes is access.

Gurobi charges $10,000+/year and won't even show you the price. CPLEX wants $285/user/month. Both require license files, seat management, and enterprise sales cycles. Deploying them in containers is either painful or impossible.

The alternative: an API call at $0.005 per request, or $9/month for 50K calls on the Starter plan. No sales call, no license file, no seat counting. Run it from any language, any platform, any container orchestrator — or have your AI agent call it directly via MCP.

The solver underneath is HiGHS — the same open-source engine winning LP benchmarks. Wrapped in a REST API for simplicity and exposed as an MCP tool for AI agents.

If you're building something that needs optimization and you're not a Fortune 500 logistics department, you shouldn't have to navigate enterprise sales to solve a linear program.

Get Started

Get an API key (1 email): oraclaw signup — instant key, 1,000 calls/day on pay-per-call ($0.005/call), upgrade to Starter $9/mo for 50K/month
Try free tools without a key: the API has 11 free endpoints (bandits, Monte Carlo, scheduling, pathfinding, scoring, Bayesian) — curl https://oraclaw-api.onrender.com/api/v1/optimize/bandit ...
Use it from your AI agent: npm install @oraclaw/mcp-server or paste the MCP config above into Claude Desktop / Cursor / Cline
Source + 17-tool docs: github.com/Whatsonyourmind/oraclaw (MIT licensed)
Live API: oraclaw-api.onrender.com

The math is the same. The price shouldn't be.

Your LLM Costs Spiked 400% Last Night — Here's How to Catch It in One API Call

Whatsonyourmind — Sun, 19 Apr 2026 21:10:50 +0000

You wake up Monday morning. Coffee in hand, you open your LLM provider's billing dashboard. The weekend total: $2,400. Your usual weekend spend is $600.

Somewhere between Friday at 11pm and Saturday at 3am, an agent hit a retry loop. Each retry included the full conversation context. Each retry was bigger than the last. A 400% cost spike. Nobody noticed because nobody was watching.

The fix took 5 minutes — a missing max_retries cap. The damage took 48 hours to discover.

This is the most expensive category of bug in AI-native applications. Not a logic error. Not a crash. A silent cost explosion that hides inside normal-looking logs until the invoice arrives.

You'd think monitoring would catch it. And it would — if you had monitoring. But proper observability means DataDog ($15/host/month), New Relic ($0.30/GB ingested), or a full Prometheus + Grafana stack that someone needs to maintain. For a team running a few LLM-powered features, that's like buying a fire truck to watch a candle.

Here's the thing: you don't need any of that. The math behind anomaly detection is old. Really old. The two techniques that catch 90% of cost spikes were invented in the 1800s. They run in microseconds. And they can be wrapped in a single API call.

Let me show you.

Two Algorithms That Catch Almost Everything

There are two statistical methods that handle the vast majority of "did something weird happen in my numbers?" scenarios. They're different, and knowing when to use each one matters.

Z-Score: For Well-Behaved Data

The Z-score measures how far a data point is from the mean, expressed in standard deviations:

z = (x - mean) / standard_deviation

That's it. If your daily LLM cost averages $150 with a standard deviation of $20, and today's cost is $250, the Z-score is:

z = (250 - 150) / 20 = 5.0

A Z-score of 5.0 means the value is 5 standard deviations from normal. In a normal distribution, anything beyond 2-3 standard deviations is extremely unlikely (less than 0.3% probability at z > 3). You have an anomaly.

Best for: Costs, latency, throughput — any metric that clusters around a predictable average. If you plotted two weeks of your daily LLM spend and it looked roughly like a bell curve, Z-score is your tool.

Weakness: Z-score assumes your data is normally distributed. If your data is already skewed — say, you have occasional legitimate high-spend days — the mean and standard deviation get pulled toward the outliers, and real anomalies hide in the noise.

IQR: For Data With a Long Tail

The Interquartile Range method doesn't care about your data's shape. It works by looking at the middle 50%:

IQR = Q3 - Q1
Lower fence = Q1 - 1.5 * IQR
Upper fence = Q3 + 1.5 * IQR

Q1 is the 25th percentile. Q3 is the 75th percentile. Anything below the lower fence or above the upper fence is an anomaly.

The 1.5 multiplier is Tukey's original recommendation from 1977 — it corresponds roughly to +/- 2.7 standard deviations in normal data, catching about 0.7% of points as outliers.

Best for: Response times (they always have a long tail), batch sizes, error rates, token counts per request — anything where legitimate values occasionally spike but you still want to catch the truly abnormal ones.

More robust than Z-score because medians and quartiles aren't pulled by extreme values the way means and standard deviations are.

The Decision Rule

If your data looks like a bell curve, use Z-score. If it has a long tail or you're not sure, use IQR.

When in doubt, run both. If they agree, you have high confidence. If only one flags an anomaly, investigate further.

Real Example: Catching a Cost Spike

Here's a working example. These are 14 days of daily LLM costs in dollars. One day had a problem.

Heads up: anomaly detection is a premium tool (one of 6 paid endpoints; the other 11 are free). Get a key in 30 seconds at oraclaw signup — one email field, instant key, no card needed. Then:

curl -X POST https://oraclaw-api.onrender.com/api/v1/detect/anomaly \
  -H "Authorization: Bearer YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "data": [142, 156, 138, 161, 145, 152, 139, 148, 155, 143, 612, 147, 151, 140],
    "method": "zscore",
    "threshold": 2.0
  }'

Response:

{
  "method": "zscore",
  "anomalies": [{ "index": 10, "value": 612, "zScore": 3.5 }],
  "stats": { "mean": 187.8, "stdDev": 121.3, "threshold": 2.0 },
  "totalPoints": 14,
  "anomalyCount": 1
}

Let's walk through the output:

Day 11 (index 10, zero-indexed) cost $612
The mean across all 14 days is $187.80 (inflated by the spike itself)
Even with the spike pulling the mean up, $612 is still 3.5 standard deviations above it
Your actual baseline is around $148/day. Something went very wrong on day 11.

The Z-score of 3.5 means this value has less than a 0.02% chance of occurring naturally. That's not variance. That's an incident.

You can swap "method": "zscore" for "method": "iqr" to use the IQR method instead — useful if your cost data has legitimate weekly patterns (higher on weekdays, lower on weekends) that make the distribution non-normal.

Building an Alert Pipeline in 10 Lines

Detection is only useful if it triggers an action. Here's a minimal alert pipeline — a cron job that checks daily costs and sends a Slack notification when something looks wrong:

// anomaly-alert.js — run via cron: 0 8 * * *
const costs = await fetchDailyCosts(14); // last 14 days from your billing API
const res = await fetch("https://oraclaw-api.onrender.com/api/v1/detect/anomaly", {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${process.env.ORACLAW_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({ data: costs, method: "zscore", threshold: 2.5 }),
});
const { anomalies, stats } = await res.json();
if (anomalies.length > 0) {
  await sendSlackAlert(`Cost anomaly detected: $${anomalies[0].value} ` +
    `(z-score: ${anomalies[0].score.toFixed(1)}, baseline: ~$${stats.mean.toFixed(0)})`);
}

That's it. No agents. No dashboards. No monthly SaaS bill. A cron job, one HTTP call, and a Slack webhook. You now have cost spike detection.

Set the threshold based on your tolerance: 2.0 catches more anomalies but includes some false positives — good for high-stakes environments where you'd rather investigate a false alarm than miss a real spike. 3.0 catches only extreme outliers — better for noisy data where daily fluctuations are normal. Start at 2.5 and adjust based on what you see in your first week.

A few practical notes for production use:

Window size matters. 14 days gives a solid baseline. Fewer than 7 data points and your statistics get unreliable. More than 30 and you start averaging over too much history, making seasonal shifts invisible.
Run both methods. If Z-score and IQR both flag the same point, that's a high-confidence anomaly. If only one flags it, it might be worth investigating but isn't urgent.
Include context in your alert. The raw Z-score or IQR deviation tells you how anomalous the value is, but your Slack message should also include what the normal range looks like, so whoever gets paged can immediately gauge severity.

When You Need More

This approach handles the "did something weird happen?" question well. But there are cases where you need heavier tools:

Real-time streaming detection (sub-second) — look at Grafana's built-in anomaly detection or AWS CloudWatch Anomaly Detection
Time-series decomposition (separating trend, seasonality, residual) — Facebook's Prophet or statsmodels in Python
Multi-dimensional anomalies (cost is normal but latency + error rate together are weird) — PyOD, Isolation Forest, or a full observability platform

For "did my daily numbers do something weird?" — one API call is enough.

Or — Let Your AI Agent Detect Anomalies Itself

If you're running an agent in Claude Desktop, Cursor, or Cline, you don't even need the curl. The same anomaly detection is exposed as an MCP tool. Drop this into your claude_desktop_config.json:

{
  "mcpServers": {
    "oraclaw": {
      "command": "npx",
      "args": ["-y", "@oraclaw/mcp-server"],
      "env": { "ORACLAW_API_KEY": "your-key-here" }
    }
  }
}

Restart your client. Now you can literally type at your agent:

"Here are the last 14 days of my LLM costs: [142, 156, 138, ..., 612, 147, 151, 140]. Anything weird?"

The agent calls detect_anomaly itself, gets back structured JSON with the spike index + Z-score, and explains it back in your language. The whole point of MCP: deterministic algorithms become first-class tools your LLM can reach for instead of guessing.

The OraClaw MCP server ships 17 tools total — 11 free without a key (bandits, Monte Carlo, scheduling, Bayesian updates, ensemble consensus, pathfinding, scoring) and 6 premium (anomaly detection, time-series forecast, LP/MIP solver, graph analytics, CMA-ES, portfolio risk). All with explicit input + output JSON schemas so your agent knows exactly what it gets back.

The Bottom Line

Z-score and IQR are 19th-century statistics. They work. They're fast. They're deterministic. They don't need training data, GPUs, or a machine learning pipeline.

You don't need a $500/month observability platform to know that $612 is not normal when your average is $148. You need arithmetic and a threshold.

The OraClaw /detect/anomaly route wraps both Z-score and IQR into a single API call. It's one of 17 MCP tools your agent can reach for to make decisions on real numbers instead of vibes.

Stop discovering cost spikes from invoices. Start discovering them from alerts.

Get Started

Get an API key (1 email): oraclaw signup — instant key, 1,000 calls/day on pay-per-call ($0.005/call), upgrade to Starter $9/mo for 50K/month
Use it from your AI agent: npm install @oraclaw/mcp-server or paste the MCP config above into Claude Desktop / Cursor / Cline
Try the 11 free tools (no key): see the full list at github.com/Whatsonyourmind/oraclaw
Live API: oraclaw-api.onrender.com

If this saved you from a $612 invoice surprise, leave a star — it helps other developers find it.

The $36,000 A/B Test: What Optimizely Charges vs. What the Algorithm Actually Costs

Whatsonyourmind — Sun, 19 Apr 2026 20:48:05 +0000

You Just Want to Test Two Buttons

You're a developer at a Series A startup. Your product manager walks over and says: "We need to A/B test the signup flow. Three variants, maybe four. Can you set that up this week?"

Simple enough. You've read about multi-armed bandits. You know the theory. You start looking at tooling.

Then you open Optimizely's pricing page. Or rather, you try to — because there is no pricing page. Just a "Contact Sales" button and a calendar widget for a 30-minute demo.

After the demo, the sales call, the follow-up, and the "let me check with my manager" email chain, the number comes back: $36,000 per year minimum. For A/B testing.

That's not a typo. And it gets worse. If your product scales to 10 million impressions per month, you're looking at $63,700 to $113,100 per year depending on your package. Enterprise tier? $200,000 to $400,000+. One user reported getting "stuck with a $24,000 bill for a product they no longer needed" after downgrading became impossible without a sales conversation.

The pricing model itself is designed to extract maximum value: Optimizely charges a percentage of your revenue, not a flat fee. The more successful your product becomes, the more you pay for the same algorithm underneath.

It's a system that, as one reviewer put it, "penalizes those just starting with experimentation." If you're a scrappy team trying to validate hypotheses fast, you're priced out before you write a single test.

When Brex — a well-funded fintech company — finally switched away from Optimizely to Statsig, their engineering lead said it plainly: "Our engineers are significantly happier."

The question nobody asks during those sales calls is the one that matters most: what are you actually buying for $36,000?

What You're Actually Buying

Strip away Optimizely's dashboard. Strip away the visual editor, the audience segmentation, the CDN integration, the SSR compatibility, the SDK for six different frameworks.

What's left?

At the mathematical core of Optimizely's experimentation engine is Thompson Sampling — a multi-armed bandit algorithm published by William R. Thompson in 1933. That's not a criticism. Thompson Sampling is genuinely brilliant. It's one of the most elegant solutions to the explore/exploit problem in statistics.

But it fits in about 20 lines of code.

The algorithm itself is public domain. It's been public domain for 91 years. You can find implementations in every language on GitHub, in textbooks, in blog posts. The math is settled.

So when you pay Optimizely $36,000 per year, you're not paying for the algorithm. You're paying for:

The visual editor — drag-and-drop test creation for non-technical users
Audience targeting — segment by geography, device, behavior, custom attributes
The SDK ecosystem — client-side, server-side, edge, mobile, OTT
The analytics dashboard — statistical significance calculations, revenue attribution, funnel visualization
Compliance and governance — SOC 2, GDPR controls, approval workflows

These are real features. They have real value — especially for large organizations with non-technical stakeholders who need to create and monitor experiments without writing code.

But if you're a developer, and you just need the bandit algorithm — the explore/exploit engine that decides which variant to show next — you're paying $36,000 for something that costs pennies to compute.

Thompson Sampling in 5 Minutes

Let's actually learn the algorithm you'd be paying for. It's more intuitive than you think.

The Explore/Exploit Dilemma

You have three signup button variants. After 100 visitors:

Variant A converted 35 out of 100 (35%)
Variant B converted 40 out of 100 (40%)
Variant C converted 5 out of 10 (50%)

Which is best? Traditional A/B testing says: "Keep running all three at equal traffic until we hit statistical significance." That wastes thousands of impressions sending traffic to Variant A, which is clearly losing.

A naive approach says: "Variant C has 50% — send all traffic there." But wait — that's based on only 10 observations. It could easily be noise.

This is the explore/exploit dilemma: do you exploit what looks best now, or explore the uncertain option to learn more?

How Thompson Sampling Solves It

Thompson Sampling uses Beta distributions to model uncertainty about each variant's true conversion rate.

For each variant, you maintain two numbers: successes (alpha) and failures (beta). When you need to pick a variant to show, you:

Sample a random value from each variant's Beta(alpha, beta) distribution
Pick the variant whose sample is highest
Show that variant to the next visitor
Update the winning variant's alpha (if converted) or beta (if didn't)

That's it. The entire algorithm.

The magic is in the Beta distribution's shape. A variant with 40 successes and 60 failures produces a tight distribution centered around 0.40 — you're fairly confident in that number. A variant with 5 successes and 5 failures produces a wide, flat distribution — it could be anywhere from 0.10 to 0.90.

When you sample from the uncertain distribution, it occasionally produces very high values. That's exploration — the algorithm says "this option might be amazing, let's check." As you gather more data, the distribution tightens, and the algorithm naturally shifts from exploration to exploitation.

It converges faster than fixed-split A/B tests because it automatically routes more traffic to winning variants while still exploring promising unknowns. No manual intervention. No arbitrary "stop the test" decisions.

The Failure Mode LLMs Hit

Here's something surprising: large language models consistently get Thompson Sampling wrong when they try to implement decision-making. They see uncertainty and interpret it as risk. When a variant has high variance, an LLM tends to pull back — to avoid the uncertain option and stick with the known quantity.

That's the exact opposite of what Thompson Sampling does. The algorithm treats uncertainty as opportunity. High variance means "we might be missing something great here." This is documented in what one team called "The $3,000 Bug" — an AI agent that was supposed to optimize decisions kept choosing the safe, well-known option and ignoring high-upside alternatives because it conflated uncertainty with danger.

Thompson Sampling doesn't make that mistake. The math doesn't have opinions about risk.

The Alternatives Landscape

Optimizely isn't your only option. The market has fragmented significantly, and there are tools at every price point. Here's an honest comparison:

Tool	Price	Bandit Algorithms	Self-Serve	Lock-in	Best For
Optimizely	$36K+/yr	Thompson only	No (sales call)	High (SDK)	Enterprise with big budgets
VWO	$199+/mo	Thompson only	Partial	Medium	Mid-market marketing teams
GrowthBook	Free (self-host)	Yes	Yes	Low	Teams with DevOps capacity
Statsig	Free–$150/mo	Yes	Yes	Low	Developer-first teams
OraClaw API	$0/25 calls/day, $0.005/call after, $9/mo Starter	UCB1 + Thompson + LinUCB	Yes (1 email)	None	Developers and AI agents that just need the algorithm

A few things jump out:

GrowthBook is the open-source hero. If you have the DevOps capacity to self-host, maintain, and monitor it, it's genuinely free and full-featured. The catch is operational overhead — you're running the infrastructure, handling uptime, managing database migrations.

Statsig hit a sweet spot for developer teams. Their free tier is generous, the DX is good, and it's what Brex switched to. If you need a full experimentation platform with a dashboard, this is the value pick.

VWO occupies the mid-market — cheaper than Optimizely, still dashboard-focused, still requires some sales interaction for advanced features.

OraClaw takes a fundamentally different approach. It's not a platform — it's an API endpoint. You send it arm data, it runs the algorithm, it returns a decision. No SDK to install, no dashboard to learn, no vendor lock-in. It supports three bandit algorithms (UCB1 for deterministic upper confidence bounds, Thompson for Bayesian exploration, and LinUCB for context-aware decisions that factor in features like time-of-day or user segment).

The right choice depends entirely on what you need. Not every problem requires the same tool.

Try It Right Now

Here's a working example. No signup, no API key, no sales call. Just paste this into your terminal:

curl -X POST https://oraclaw-api.onrender.com/api/v1/optimize/bandit \
  -H "Content-Type: application/json" \
  -d '{
    "arms": [
      {"id": "variant-a", "name": "Original CTA", "pulls": 500, "totalReward": 175},
      {"id": "variant-b", "name": "New CTA", "pulls": 300, "totalReward": 126},
      {"id": "variant-c", "name": "Bold CTA", "pulls": 12, "totalReward": 8}
    ],
    "algorithm": "thompson"
  }'

You'll get back something like:

{"selected": {"id": "variant-c"}, "score": 0.71, "algorithm": "thompson"}

Wait — variant-c? The one with only 12 pulls and a 66.7% conversion rate?

Yes. And here's why that's correct.

Variant A has 500 pulls and a 35% conversion rate. Thompson Sampling is very confident about that number — the Beta(175, 325) distribution is tight. It's almost certainly between 31% and 39%.

Variant B has 300 pulls and a 42% conversion rate. Also fairly confident — Beta(126, 174) is tight. Probably between 37% and 47%.

Variant C has 12 pulls and a 66.7% conversion rate. But Beta(8, 4) is wide. The true rate could be anywhere from 35% to 90%. When Thompson samples from this distribution, it frequently draws values above 0.50 — higher than what A or B can produce.

The algorithm is saying: "Variant C looks promising but we barely know anything about it. Let's send more traffic there to find out."

That's exploration in action. If C's true rate is 45%, a few more pulls will tighten the distribution and it'll stop being selected. If C's true rate really is 65%, you just found a massive winner that a fixed 33/33/33 split would have taken 10x longer to identify.

This is exactly the behavior that the "$3,000 Bug" LLM got wrong. It saw the small sample size and treated it as a reason to avoid variant C. Thompson Sampling sees the small sample size and treats it as a reason to investigate.

You can swap "algorithm": "thompson" for "ucb1" or "linucb" (with a context vector) to compare strategies. The endpoint is stateless — bring your own data, get back a decision, integrate it however you want. Pipe it into your CI/CD pipeline, call it from a serverless function, embed it in an AI agent's decision loop.

The MCP Angle: Your AI Agent Can Call This Directly

If you're using Claude Desktop, Cursor, Cline, or any MCP-compatible client, your agent can call this algorithm itself — no curl, no SDK install, no HTTP code. Drop this into your claude_desktop_config.json:

{
  "mcpServers": {
    "oraclaw": {
      "command": "npx",
      "args": ["-y", "@oraclaw/mcp-server"]
    }
  }
}

Restart your client. Now you can literally type:

"I have these three signup variants with these conversion numbers. Which should I send the next 1,000 visitors to?"

The agent calls optimize_bandit itself, gets back a structured decision in 0.01ms, and explains the result in your language. No more LLMs guessing at the math — they offload it to a real algorithm.

The MCP server ships with 17 tools total: bandits, contextual bandits, genetic algorithms, Monte Carlo, scheduling, pathfinding, scoring, Bayesian belief updates, ensemble consensus — 11 free without a key, 6 premium (LP/MIP solver, time-series forecasting, anomaly detection, graph analytics, CMA-ES, portfolio risk). All with explicit input + output JSON schemas so your agent knows exactly what it gets back.

Get the MCP server: npmjs.com/package/@oraclaw/mcp-server

When NOT to Use This

Let's be honest about the tradeoffs.

If you need a visual editor so your marketing team can create tests without writing code — use Optimizely or VWO. If you need audience targeting with complex segmentation rules — use a platform. If you need a dashboard with real-time charts for stakeholders who don't read JSON — use Statsig or GrowthBook.

A bare API endpoint is the wrong tool for organizations where non-developers need to create and monitor experiments. That's a real use case, and the $36K platforms serve it well.

But if you're a developer calling an optimization algorithm from your backend, your data pipeline, or your AI agent — you don't need a visual editor. You don't need a dashboard. You need the math, and you need it fast, and you need it cheap.

The Math Doesn't Care About Your Budget

Thompson Sampling produces the same distribution, the same samples, and the same convergence properties whether the compute costs $36,000 per year or $0.005 per call. The algorithm was published in 1933. It's been proven optimal in the limit. No amount of enterprise packaging changes the underlying mathematics.

The question isn't "which algorithm is best" — for most A/B testing scenarios, Thompson Sampling is the answer regardless of vendor. The question is: how much infrastructure do you need wrapped around it?

If the answer is "a lot" — platforms exist for that. If the answer is "just give me the algorithm" — now you know what your options are.

Stop paying $36,000 for 20 lines of math.

Get Started

Free tier (no signup): the curl command above runs against the live API right now, 25 calls/day per IP, no key
Get an API key (1 email field): oraclaw signup — instant key, 1,000 calls/day on pay-per-call ($0.005/call), upgrade to Starter $9/mo for 50K/month
Use it from your AI agent: npm install @oraclaw/mcp-server or paste the MCP config above into Claude Desktop / Cursor / Cline
Source + 17-tool docs: github.com/Whatsonyourmind/oraclaw

If this saved you from a $36K sales call, leave a star — it helps other developers find it.

DEV Community: Whatsonyourmind

Your bandit's exploration floor probably violates its own floor

Why clip-then-renormalize breaks

The fix: one affine map onto the simplex

The one guard you do need

Why it actually matters

A model with R-squared near 0 can still give valid 90% prediction intervals - here's why (and the catch)

Coverage is a property of the procedure, not the model

The demo (numbers are reproducible, seed fixed)

The catch: marginal ≠ conditional

What to take away

Stop trusting the agent: bind tool-call approvals to the exact call

Three ways an approval boolean breaks

Bind the approval to the call

Three details that decide whether it actually holds

The gotcha that bites in production: replay

The takeaway

Conformal prediction silently breaks under drift - and how to make it hold

The failure, measured

Why it breaks, and the two things you have to fix

Three things that matter in practice

The takeaway

When your optimizer silently returns the wrong answer (and how to catch it)

The symptom: same model, two answers

The first diagnostic is free: flip presolve

Why scaling is so often the trigger

How to minimize a solver bug so it actually gets fixed

A scaling-hygiene checklist

The broader point

Determinism as a feature: when to let your agent call a math API instead of reasoning

The tell

What to offload (and a cheap test for each)

The pattern

When not to offload

What Happens When 1,000 Agents Make the Same Mistake Simultaneously

What Happens When 1,000 Agents Make the Same Mistake Simultaneously

The Invisible Risk: Correlated Failures

Three Failure Modes Nobody Is Monitoring

What the Math Layer Looks Like

The Stakes

Try OraClaw

Get Started

I Built an Agent Portfolio Advisor by Composing 3 OpenClaw Skills — Here's What Actually Works

What I Built

How I Used OpenClaw

Step 1 — oraclaw-bandit picks the allocation

Step 2 — oraclaw-simulate runs the Monte Carlo

Step 3 — oraclaw-risk closes the loop (premium)

Wiring all three into one MCP agent

Demo

What I Learned

Links

Monte Carlo Simulation in 5 Minutes: From Zero to Confidence Intervals in One API Call

What Monte Carlo Actually Is

Common Mistakes Developers Make

Three Real Use Cases

1. Portfolio Risk: "What's My 95% VaR?"

2. Project Estimation: "What's the Probability We Deliver by March?"

3. Pricing Uncertainty: "What's Our Expected Revenue?"

Try It: Portfolio Confidence Intervals

Reading the Output

Portfolio VaR: Correlated Multi-Asset Risk

The MCP Angle: Your AI Agent Can Call This Directly

DIY vs API: The Build-vs-Buy Calculation

When You Need More

Get Started

I Needed an LP Solver but Gurobi Costs $10K/yr — So I Built an API for $9/month

The $10,000 Pricing Page That Says Nothing

What Are You Actually Paying $10K/Year For?

The Solver Landscape in 2026

A Real Example: Factory Scheduling

When NOT to Use This

The MCP Angle: Your AI Agent Calls the Solver Itself

The Bottom Line

Get Started

Your LLM Costs Spiked 400% Last Night — Here's How to Catch It in One API Call

Two Algorithms That Catch Almost Everything

Z-Score: For Well-Behaved Data

IQR: For Data With a Long Tail

The Decision Rule

Step 1 — `oraclaw-bandit` picks the allocation

Step 2 — `oraclaw-simulate` runs the Monte Carlo

Step 3 — `oraclaw-risk` closes the loop (premium)