DEV Community: sichi chen

How do you score a pelican on a bicycle? Turning a viral SVG test into an evaluation you can audit

sichi chen — Tue, 27 Oct 2026 08:36:05 +0000

Picture three SVG pelicans on bicycles. The first is beautiful (layered feathers, shiny wheels, a sunset behind it), but both feet hang above the pedals. The second is a rough sketch, but the frame connects, the body sits on the saddle, and the feet are actually on the pedals. The third looks fine as a still image. Press play, though, and one foot drifts off on its own path.

Which one should score highest? If you pick the prettiest, you're scoring taste. If you pick the reliable one, you're scoring whether the task got done. And if you only look at screenshots, you may never notice what's wrong with the third.

Here are the main ideas from our longer write-up on turning the "pelican on a bicycle" test into something you can re-check.

1. Write the task down before you argue about scoring

"Draw a pelican riding a bicycle in SVG" tests whether a model can turn a written request into code that renders the right picture. It doesn't prove anything about the model's internal spatial understanding. Don't add requirements at scoring time that weren't in the prompt. Split it into two kinds of test:

Open task: keep it loose, and only penalize requirements that are clear and necessary.
Constrained task: state the viewpoint, the visible relations (each foot touching its pedal), and the output format (one self-contained SVG, no external images or scripts) before anyone runs it.

Also, render-look-revise loops measure a workflow, not one-shot generation. Keep those results on separate leaderboards.

2. Don't rush to a total score; break "riding" apart

"Looks good, 8/10" is the easiest score to give and the hardest to re-check. Score the subject, the bicycle structure, the riding relation, and the presentation separately, and decide in advance what each one does not include. For example, feather detail shouldn't count toward "is it a pelican."

3. Rules, geometry, and visual judges each have a job

The SVG DOM helps, but it's not a structural diagram that comes with the answer. Static rules work best for constraints that can actually be settled (self-contained file, no scripts).
A geometric check has to define what it measures before it measures anything.
Before a visual LLM judge takes the job, give the judge an exam. Ask it concrete questions ("is the left foot touching the left pedal?") rather than for a verdict on taste, and don't calibrate it on a single overall-agreement number.

4. Animation checks whether relations hold over time

It's not "a few extra screenshots." The question is whether the foot stays on the pedal while the pedal turns.

5. The scorer can be wrong too

A changed judge or rubric can make it look as if a model suddenly got worse. Items do go stale, but rewriting the test every day isn't more scientific. Change the constraint, not just the animal's name, and don't present a change in item difficulty as model progress.

What's actually worth automating is a procedure that can be audited later, and a leaderboard worth reading is one you can keep asking questions of.

The full article includes a sample constrained prompt, an acceptance table, and an example judge output record:
https://folkbench.com/en/blog/how-to-score-a-cycling-pelican?utm_source=luntan&utm_campaign=dev

Folkbench can be browsed without registering. Rankings and blog posts are open to everyone:
https://folkbench.com/?utm_source=luntan&utm_campaign=dev

How do you score open-ended generation tasks like this? I'd like to hear in the comments.

Stop trusting a 99% uptime claim: the LLM API relay failures we hit, and how we test stability

sichi chen — Wed, 14 Oct 2026 18:32:19 +0000

*Originally published on the Folkbench blog: https://folkbench.com/en/blog/ai-api-relay-stability-guide?utm_source=luntan&utm_campaign=dev *

A relay site may promise "99.9% fast and stable" and a "dedicated high-concurrency channel". Once you connect, things look different. This is how we went from buying on instinct to running automated checks before we pay for or integrate a relay.

1. The failures that forced the method

Fake 200 OK and a quiet truncation

Our early check was a Python script that POSTed to the relay every 5 minutes. HTTP 200 with some text = healthy.

Less than two days into production, users reported the model stopping mid-sentence. The logs showed that some relays still returned 200 OK when the upstream pool was exhausted (429 / 401), when the gateway timed out (504), or when a node crashed. Mid-stream they would quietly insert data: [DONE], or put {"error": "rate limit exceeded"} inside the JSON as if it were the model's answer. A status-code monitor was fully fooled.

The nighttime stall

Daytime average response stayed near 2 seconds. At the night peak, timeouts and abandonment climbed. The worst case still finished the whole answer in about 4 seconds, but the first token took 12 seconds. Users stared at an empty box and left.

Lesson: one ping, or two casual requests, is not a test of a streaming AI API.

2. Three versions of the test system

1.0 – blind trials: a few prompts in a web UI or 10 hand-written requests. Tiny, noisy sample; rare disconnects only show up in production.
2.0 – concurrency + status codes: Postman / Locust, status-code mix, average latency, success rate. Sees the outer gateway, misses broken streams, forged errors and stalls between tokens.
3.0 – a stream state machine: treat the relay like a long-connection network middlebox (think realtime audio/video or market-data feeds, not web requests). Multi-region probes split connection setup, TLS handshake, first-token arrival, token throughput and SSE parsing.

3. The three dimensions that decide production experience

Latency – separate waiting from generating

TTFT (time to first token): request → first SSE packet. Our production bar is TTFT < 1.5 s.
TPS: generated tokens / duration after the first token.
P95 / P99: ignore the flattering average.

Reliability – look through the fake success

Parse success: the official SDK must parse the payload, JSON must be complete, no relay-injected error inside the answer.
Retry-to-success: with exponential backoff, does a 429/5xx recover within 3 retries?

Stream quality – catch the silent failure

Stream interruption: share of calls that drop or go silent before data: [DONE].
Inter-token jitter: standard deviation of gaps between SSE packets. High jitter = unstable path or stacked proxy buffers.

4. References, not invented in a closed room

Google SRE four golden signals: latency, traffic, errors, saturation — the relay is treated as an untrusted external service.
OpenTelemetry + W3C Server-Sent Events: record TCP handshake, TLS, headers, first chunk and [DONE].
Wilson score interval: 10/10 (100%) is not automatically more trustworthy than 9,900/10,000 (99%). We rank by the lower bound:

Score = (p̂ + z²/(2n) − z·√(p̂(1−p̂)/n + z²/(4n²))) / (1 + z²/n)

p̂ = observed success rate, n = sample size, z = normal quantile for 95% confidence. Small samples get penalized.

5. How the probe is built

Multi-region probes (AWS US West, GCP East Asia – Hong Kong/Tokyo, Europe) separate a stuck relay from an upstream swing.
Socket state machine: CONNECTING (t0) → FIRST_BYTE (TTFT = t1 − t0) → STREAMING (gaps, TPS, jitter) → COMPLETED ([DONE] present, valid JSON).
Automatic attribution:
- 502/504 → the relay's own gateway node failed.
- 429/401 → concurrency control failed or upstream key balance ran out.
- 200 + mid-stream drop or parse failure → the relay's billing/filter middleware overflowed.

6. Three rules before you pick a relay

Don't trust one test. Sample day and night for at least 3–7 days; watch P95 and success rate in the 21:00–01:00 peak.
Test streaming. Non-streaming hides first-token delay, buffering and mid-stream deaths.
A price far too low is not a gift. It usually means a shared account pool, a poor proxy, or a cheap open model presented as a premium one.

Folkbench publishes these measurements (P95 latency, availability and per-token price per relay) with no sign-up needed: https://folkbench.com/?utm_source=luntan&utm_campaign=dev

Same model, different relay: a 5-metric checklist (with Python) before you pick an LLM API provider

sichi chen — Sun, 11 Oct 2026 18:25:02 +0000

If you buy access to the "same" model through different API relays/resellers, you quickly learn they are not the same. One times out at peak hours, another is fast at the median but terrible at P95, a third silently returns errors in a 200 body.

Here is the small checklist I use before committing to one, plus a Python script you can run yourself.

1. Compare like with like

Only compare providers serving the same model name. Mixing models turns a reliability question into a quality question.

2. Measure availability, not vibes

Count a request as failed if it times out, returns non-2xx, or returns an error payload. Report success rate with a sample size: "19/20" is not the same evidence as "950/1000".

3. Look at P95, not just the average

Users feel the slow tail. The mean hides it.

4. Write down when you measured

A relay that was fine at 03:00 can be broken at 21:00. Every number needs a timestamp.

5. Missing data stays missing

If you could not test a provider, show "no data". Do not fill the gap with a guess.

A minimal probe

import time, statistics, requests, datetime as dt

def probe(base_url, key, model, n=20, timeout=60):
    lat, ok = [], 0
    for _ in range(n):
        t0 = time.perf_counter()
        try:
            r = requests.post(f"{base_url}/v1/chat/completions",
                headers={"Authorization": f"Bearer {key}"},
                json={"model": model, "max_tokens": 16,
                      "messages": [{"role": "user", "content": "Reply with OK"}]},
                timeout=timeout)
            body = r.json()
            if r.ok and "choices" in body:
                ok += 1
                lat.append(time.perf_counter() - t0)
        except Exception:
            pass
    p95 = statistics.quantiles(lat, n=20)[18] if len(lat) >= 2 else None
    return {"checked_at": dt.datetime.utcnow().isoformat() + "Z",
            "success": f"{ok}/{n}", "p95_s": p95}

Only probe endpoints you are allowed to test, and keep n small so you do not hammer anyone.

Don't want to run it yourself?

I've been using Folkbench for exactly this: it ranks relays for the same model using the same published metrics (availability, response time, price), stamps each result with its check time, keeps missing data visible, and keeps commercial placements separate from the ranking order.

How do you choose between API relays? Curious what other metrics people track.

Before You Call an LLM Endpoint "Nerfed": A Small Statistics Checklist in Python

sichi chen — Wed, 07 Oct 2026 09:37:48 +0000

Every few weeks someone posts "provider X is serving a watered-down model" with a handful of screenshots, and every few weeks the replies split into "same here" and "works fine for me." Both camps are usually arguing from data that can't settle the question.

My last post argued that a single output tells you almost nothing. A commenter on it made a sharper point that I want to build on: even with repeated runs, the way you pick what to compare can manufacture a difference. Comparing two pre-chosen providers at 16/20 vs 8/20 is meaningful (Fisher p ≈ 0.02). Scanning a table of ten providers and quoting the best and worst is not, because with ten identical providers a gap that large shows up by luck about 30% of the time. Their suggested fix — compare against a pooled rate or a reference endpoint chosen in advance — is the backbone of this post.

So here's the checklist I now run before believing that an endpoint regressed. Everything assumes the simplest useful setup: one fixed probe (for example, "draw a pelican riding a bicycle as SVG"), a fixed pass/fail rubric, and a count of passes out of N attempts. All numbers below are illustrative — they come from the code shown, not from any real provider. Every snippet runs as-is with scipy and numpy.

1. Put an interval on every rate

A pass rate without an interval is a vibe. For binomial counts, the Wilson interval behaves well at small N:

from scipy.stats import binomtest

# Illustrative numbers, not real measurements
for label, k, n in [("last week", 34, 40), ("this week", 25, 40)]:
    ci = binomtest(k, n).proportion_ci(confidence_level=0.95, method="wilson")
    print(f"{label}: {k}/{n} = {k/n:.1%}   95% Wilson CI {ci.low:.1%} to {ci.high:.1%}")

last week: 34/40 = 85.0%   95% Wilson CI 70.9% to 92.9%
this week: 25/40 = 62.5%   95% Wilson CI 47.0% to 75.8%

The intervals barely overlap. That alone doesn't settle it (overlapping intervals are not a significance test, and non-overlap isn't required for one), but it tells you how loosely 40 runs pin down each rate. If you only had 10 runs, both intervals would span roughly half the scale.

2. One pre-chosen comparison: Fisher's exact test

If you decided before running anything that you would compare "last week" against "this week" on this probe, a 2×2 exact test is the honest tool:

from scipy.stats import fisher_exact

#          pass  fail
table = [[34,    6],   # last week (illustrative)
         [25,   15]]   # this week (illustrative)

two_sided = fisher_exact(table, alternative="two-sided").pvalue
one_sided = fisher_exact(table, alternative="greater").pvalue  # only if "drop" was pre-registered
print(f"two-sided p = {two_sided:.4f}")
print(f"one-sided p = {one_sided:.4f}")

two-sided p = 0.0406
one-sided p = 0.0203

Two caveats. The one-sided p-value is only legitimate if "this week is worse" was the hypothesis you wrote down in advance; choosing the direction after seeing the data is just halving your p-value for free. And p = 0.04 means "data this lopsided would be unusual if nothing changed," not "96% chance the provider swapped models."

3. Count how many comparisons you could have made

This is the commenter's point, and it deserves a simulation. Give every provider the exact same true pass rate (60%), run each 20 times, and look at the gap between the best and worst:

import numpy as np

rng = np.random.default_rng(42)
n_runs, p_true, sims = 20, 0.6, 200_000   # every provider is identical: 60% pass rate

for k in (2, 5, 10, 20):
    passes = rng.binomial(n_runs, p_true, size=(sims, k))
    gap = passes.max(axis=1) - passes.min(axis=1)
    print(f"{k:>2} providers: P(best-worst gap >= 8) = {(gap >= 8).mean():5.1%}   "
          f"95th percentile of gap = {np.percentile(gap, 95):.0f}")

 2 providers: P(best-worst gap >= 8) =  1.4%   95th percentile of gap = 6
 5 providers: P(best-worst gap >= 8) = 10.4%   95th percentile of gap = 8
10 providers: P(best-worst gap >= 8) = 29.9%   95th percentile of gap = 10
20 providers: P(best-worst gap >= 8) = 62.4%   95th percentile of gap = 11

With two pre-chosen providers, an 8-pass gap is rare. With ten, it happens about 30% of the time with nothing wrong — matching the commenter's estimate. With twenty, it's the norm. The 95th-percentile column is the gap you'd need just to clear the noise floor of "best vs worst," and it keeps rising as the table grows.

If your claim comes from eyeballing a leaderboard, the number of comparisons is the number of rows, not two.

4. Compare each endpoint to the pool, then correct for multiplicity

Instead of best-vs-worst, ask a question that doesn't depend on which rows happen to land at the extremes: is any endpoint off from the others by more than chance allows for this many endpoints?

import numpy as np
from scipy.stats import chi2_contingency, fisher_exact

# Illustrative pass counts for 8 endpoints serving the "same" model, 30 runs each
passes = {"A": 22, "B": 20, "C": 23, "D": 19, "E": 21, "F": 12, "G": 22, "H": 20}
n = 30

table = np.array([[k, n - k] for k in passes.values()])
chi2, p_omni, dof, _ = chi2_contingency(table)
print(f"omnibus chi2 = {chi2:.2f}, dof = {dof}, p = {p_omni:.4f}")

# Each endpoint vs the pooled pass rate of all the OTHER endpoints
raw = {}
for name, k in passes.items():
    others_pass = sum(passes.values()) - k
    others_fail = n * (len(passes) - 1) - others_pass
    raw[name] = fisher_exact([[k, n - k], [others_pass, others_fail]]).pvalue

# Holm step-down correction for 8 tests
m = len(raw)
running_max = 0.0
for i, (name, p) in enumerate(sorted(raw.items(), key=lambda kv: kv[1])):
    running_max = max(running_max, min(1.0, (m - i) * p))
    print(f"{name}: {passes[name]}/{n}  raw p = {p:.4f}  Holm p = {running_max:.4f}")

omnibus chi2 = 12.35, dof = 7, p = 0.0895
F: 12/30  raw p = 0.0018  Holm p = 0.0144
C: 23/30  raw p = 0.2221  Holm p = 1.0000
A: 22/30  raw p = 0.4178  Holm p = 1.0000
G: 22/30  raw p = 0.4178  Holm p = 1.0000
E: 21/30  raw p = 0.6862  Holm p = 1.0000
D: 19/30  raw p = 0.8367  Holm p = 1.0000
B: 20/30  raw p = 1.0000  Holm p = 1.0000
H: 20/30  raw p = 1.0000  Holm p = 1.0000

Two things worth noticing:

The omnibus chi-square doesn't reach 0.05. It spreads its attention over all 7 degrees of freedom, so one clear outlier among mostly similar endpoints can slip under it.
The per-endpoint test against the pooled rate of the others flags F, and it survives Holm correction across all 8 tests (adjusted p ≈ 0.014). Everyone else is unremarkable — including C at 23/30, which a "top of the table" screenshot would happily crown.

Holm is a reasonable default: it controls the chance of any false flag across the family and is never less powerful than plain Bonferroni. The leave-one-out pool matters too; if F were included in its own baseline, it would drag the baseline toward itself.

5. Pre-register the plan (it takes 10 lines)

"Pre-registration" sounds academic. In practice it means writing down the comparison before you collect data and making it hard to quietly edit later:

import hashlib, json

plan = {
    "question": "Has endpoint X's first-attempt pass rate dropped vs reference R?",
    "endpoints": {"test": "provider-x/model-y", "reference": "official-api/model-y"},
    "probe": "pelican-static-v1",
    "rubric": "pass = all required checks pass; timeouts count as fail",
    "runs_per_endpoint": 80,
    "schedule": "4 blocks, interleaved, random order within block",
    "primary_test": "Fisher exact, one-sided (X < R), alpha = 0.05",
    "stopping_rule": "no interim looks",
}
blob = json.dumps(plan, sort_keys=True).encode()
print(hashlib.sha256(blob).hexdigest())

1e240df48b2c46c2cbd1782eecd2fe87c69f1ef6b469b3a97f3625e803ab884e

Post the hash (in a commit, an issue, a chat message) before the first run, and publish the plan alongside the results. Note what goes in it: a reference endpoint fixed in advance, the sample size, the schedule, how timeouts are counted, the primary test, and the fact that there are no interim looks. If you later add a second probe or a third provider, that's a new plan, not an edit.

The reference endpoint matters most for "did it regress over time?" questions. Re-run the reference in the same window as the endpoint you suspect. If both dropped together, the culprit is more likely your probe, your judge, or something shared upstream — not a secret model swap at one provider.

6. Check that you can actually detect the drop you care about

Twenty runs per side feels like a lot when you're doing it by hand. It isn't, if the drop you're worried about is moderate. Here's a simulated power check for a real 80% → 60% drop:

import numpy as np
from scipy.stats import fisher_exact

rng = np.random.default_rng(7)
p_before, p_after, sims = 0.80, 0.60, 2000   # a real 20-point drop

for n in (20, 40, 80, 120):
    a = rng.binomial(n, p_before, sims)
    b = rng.binomial(n, p_after, sims)
    hits = sum(
        fisher_exact([[x, n - x], [y, n - y]], alternative="greater").pvalue < 0.05
        for x, y in zip(a, b)
    )
    print(f"n = {n:>3} per arm: power ~ {hits / sims:.0%}")

n =  20 per arm: power ~ 30%
n =  40 per arm: power ~ 56%
n =  80 per arm: power ~ 82%
n = 120 per arm: power ~ 95%

At 20 runs per arm you'd miss a genuine 20-point drop about 70% of the time. Around 80 per arm gets you to the conventional 80% power. Two consequences:

"I ran it 20 times and saw no difference" is weak evidence that nothing changed. Absence of significance is not equivalence.
Decide the drop size you care about first, then the N. Not the other way round.

7. Don't peek and stop when it looks good

The most natural thing in the world is to check the numbers after every batch and post as soon as the gap looks significant. It quietly breaks the test:

import numpy as np
from scipy.stats import fisher_exact

rng = np.random.default_rng(1)
p_true, sims, looks = 0.7, 2000, range(10, 61, 5)   # no real difference at all

false_alarms = 0
for _ in range(sims):
    a = rng.random(60) < p_true
    b = rng.random(60) < p_true
    for n in looks:
        x, y = a[:n].sum(), b[:n].sum()
        if fisher_exact([[x, n - x], [y, n - y]]).pvalue < 0.05:
            false_alarms += 1
            break

fixed = 0
for _ in range(sims):
    x, y = rng.binomial(60, p_true, 2)
    fixed += fisher_exact([[x, 60 - x], [y, 60 - y]]).pvalue < 0.05

print(f"peek every 5 runs, stop at p<0.05: false alarm rate ~ {false_alarms / sims:.1%}")
print(f"single test at n=60:              false alarm rate ~ {fixed / sims:.1%}")

peek every 5 runs, stop at p<0.05: false alarm rate ~ 8.7%
single test at n=60:              false alarm rate ~ 3.4%

Here both arms are identical. Peeking every 5 runs and stopping at the first p < 0.05 more than doubles the false-alarm rate compared with one test at the planned N. (Fisher's test is conservative at these sizes, which is why the fixed-N rate sits below 5%.) If you genuinely need to monitor continuously, use a method designed for it, such as sequential or always-valid tests. Otherwise, fix N and look once.

The checklist, condensed

Report pass rates with intervals, not bare percentages.
For one pre-chosen pair, use an exact test; pick one- vs two-sided in advance.
Count every comparison you could have made, not the one you ended up quoting.
Compare each endpoint to a pooled or pre-registered reference, and correct for multiplicity.
Write the plan down (and hash it) before the first run.
Size N for the smallest drop you care about.
Fix N and look once — or use a sequential method on purpose.

None of this tells you why an endpoint got worse. Timeouts, truncated outputs, a different default reasoning effort, and an actual model change can all lower a pass rate. Statistics only tells you whether there's a difference worth explaining. Keep timeouts and wrong answers as separate categories in your logs, so that when something does show up you can tell which kind of failure moved.

Where the tooling fits

The statistics above fit in a page. What actually costs time is the bookkeeping around it: same prompt, same parameters, timestamps on every run, a reference endpoint re-tested in the same window, and failures kept instead of dropped.

That bookkeeping is the problem I work on at Folkbench. It's a public leaderboard that compares different services (official APIs and third-party relays) serving the same model, using one published methodology, with the test time shown on each result and missing data left blank rather than guessed. There's also a pelican-on-a-bicycle gallery, which the site itself labels as a fun side-by-side that does not feed into the rankings and should not be read as a capability score (given everything above, I think that is the right call). If you'd rather start from existing results than build the harness yourself, it's at folkbench.com. Apply the same checklist to it: ask what was tested, when, and how many runs are behind each number.

If you run this kind of comparison yourself, I'd like to hear how you choose the reference endpoint. That's the decision I'm least sure about.

Disclosure: I work on Folkbench. Drafted with AI assistance; code and numbers checked by me. All numbers in this post are illustrative outputs of the code shown, not measurements of any real provider.

Is the model actually getting dumber, or are we just reading tea leaves from single samples?

sichi chen — Mon, 05 Oct 2026 13:10:12 +0000

Disclosure: I'm involved in building Folkbench, which I mention near the end of this post.

I've been chewing on a question that's been discussed to death but never really settled: when we say a model "got dumber" or "got nerfed," how much of that is real, and how much is just the illusion of a single sample?

Lately a lot of people I know have been throwing the pelican test at models (have it draw an SVG of a pelican riding a bicycle). It's a genuinely brutal test — it's not just about writing code, it's about spatial reasoning: beak and handlebars, feet and pedals, frame and wheels. If the model's spatial understanding is even slightly off, you get something pretty abstract.

But after running it a few dozen times, I think the biggest trap with the pelican test is judging by a single output.

An LLM is a probabilistic sampler with randomness baked in. Same prompt: on one run the spatial awareness is maxed out — frame, cranks, foot placement all correct; run it again later and suddenly it's postmodern abstract art. If provider A gets a basically sensible pose in 16 out of 20 runs and provider B only passes 8 out of 20, then the difference means something statistically. Declaring "A is the full model, B is watered down" based on one random screenshot is basically flipping a coin.

To actually compare anything, you need a shared baseline — fix the model version, reasoning effort, prompt and time window, set a reference point first, and then look only at relative differences.

That leads to a few really painful engineering details:

How do you score the pelican? Pure human review doesn't scale; pure LLM-as-a-judge tends to heavily favor its own output. For now it looks like you have to split it into four dimensions — Pelican (completeness), Bicycle (geometric soundness), Riding (how the bird actually connects to the bike spatially) and Animation (motion and clipping) — and cross-check with a vision judge plus static rules on the SVG DOM.
The lifecycle of test prompts. Once any "killer prompt" spreads, sooner or later it gets scraped into training data or targeted with fine-tuning. Long term you can't rely on a single question; you need a dynamic pool of probes covering spatial relations, instruction following and structural reasoning.

I'd been running these tests by hand for a while, and the biggest pain was the cost of record-keeping: repeated runs, saving SVGs, logging parameters, aligning timestamps — after a few dozen batches you're numb. What eats the time usually isn't the test itself but all the tedious logging and organizing around it. That said, if you just want a quick read on how different models perform, you don't have to do it all yourself — there's a ready-made leaderboard on Folkbench (https://folkbench.com/?utm_source=luntan&utm_campaign=dev) that can save you the effort.

If you're also playing with this test, I'd love to hear how you run it — or see your most cursed results.