DEV Community: Devin

18 Games, 617 Assertions: What a One-Person Static Arcade Taught Me About Testing

Devin — Sun, 08 Nov 2026 20:07:27 +0000

I wanted games that open the moment you click. No install, no sign-in, no loading screen, no cookie banner. Tap a link on a mediocre phone and you are already playing. That sounds like a small constraint. It ended up deciding almost every technical choice in the project.

The site now has 18 games — board games, puzzles, card games. All free, no login, each one a single file, no external dependencies, no trackers. Plain HTML, CSS and JavaScript. No framework, no build step. The boards and pieces are inline SVG I drew by hand.

This is not a story about clever architecture. It is a story about how I convinced myself the games actually work, and about two measurement mistakes that nearly made me ship broken pages.

Why static, and why hand-drawn

A static site is the only shape that satisfies the constraints. Every game is a file you can open from disk or put on any static host. There is no server that can be down, no database to migrate, nothing to keep warm. On a weak connection the whole game is one HTML document, loaded once.

No framework because a board game is a state machine. A position, a list of legal moves, and an event handler. Vanilla JavaScript expresses that directly. A framework would add a payload larger than the game itself, plus a build step and a dependency tree that has to stay current. I would rather spend that budget on tests.

Hand-drawn SVG because it keeps the promise of a single file. Raster images mean extra requests and resolution decisions; an SVG board scales to any screen and stays sharp. Inline SVG also gives one source of truth for a piece: the same shape can be filled, highlighted and animated. And there is no image licensing to audit.

One rule follows from all of this: no game may depend on another game, a shared runtime, or a CDN. If one page breaks, the other 17 are unaffected, and the broken page is still readable as source.

The hard part: testing without looking

Testing a game is not like testing a form. You cannot look at a board and decide whether it is correct. A position can look plausible and be wrong two moves deeper. So most of the suite is built around things you can count.

The core tool is perft: walk the move generator to a fixed depth and count leaf nodes. The counts are exact. If one edge case in move generation is wrong, the number diverges immediately.

For Chinese chess (xiangqi), perft at depths 1, 2 and 3 gives 44, 1,920 and 79,666. Those match the standard published values.

For international chess, perft(1/2/3) is 20, 400 and 8,902. On top of that, 37 targeted tests cover the rules that perft alone does not isolate: 11 for castling, 7 for en passant, 7 for promotion, 3 for checkmate and 5 for stalemate.

Go scoring is checked against three fixed boards with known results: 45:36, 42:39 and 40.5:40.5. The half-point case is the useful one. It forces the scoring code to keep fractional territory correct — a mistake that is invisible when you eyeball a board.

The 24 game, where you make 24 from four numbers, has a different failure mode: a deal can be generated correctly and still be unsolvable, and a solver can be too eager and claim a solution that does not exist. So the test deals 1,000 hands in a row and independently verifies every one is solvable, covering 669 distinct combinations. That catches both duplicate deals and a solver that overreports.

Dou Dizhu is tested by self-play: 200 AI-versus-AI games, 0 illegal plays, landlord win rate 63.0%. The first number is a rules-engine regression test. The win rate is a sanity check that the AI is not reaching its result by breaking the rules. I am not claiming the AI is strong — only that it obeys the rules and wins at a plausible rate.

Connect Four gets a relative check: the harder AI plays the simple AI 20 times and wins 20. That says the stronger setting is actually stronger. It says nothing about how strong it is against a human.

All of it runs with one command:

node games/tests/run-all.js

17 test files, 617 assertions. The single command matters more than the count. If running the tests is a chore, they rot. If it is one command that prints pass or fail, I run it after every change.

There is one more check that unit tests cannot see: a crawl of the whole site. It currently covers 108 pages and 112 internal links, and finds 0 404s. That is where a broken relative path gets caught — not in the game logic, but in the way the pages point at each other.

A note on what these tests do not prove. Perft proves that move generation matches a known count at a shallow depth. It does not prove the UI renders the move correctly, that the AI is pleasant to play against, or that the game is fun. Those still need a person. The tests exist to keep the parts that a person cannot check by eye from silently drifting.

Multiplayer: where the server earns its place

Multiplayer runs on Cloudflare Workers. A Durable Object per room handles signaling; voice is WebRTC mesh, capped at 4 players.

The Durable Object is there because signaling is stateful. A room has a membership list, join and leave events, and offer/answer routing between peers. In a stateless function you would have to invent a store and handle concurrent updates yourself. A Durable Object is one object per room, and requests to it are serialized. That removes a class of race conditions by construction.

For three of the board games the server is authoritative: the Durable Object validates moves, so a modified client cannot make an illegal move. In single-player that is a nicety. In multiplayer it is the difference between a game and a free-for-all, because the other players cannot see your screen.

The mesh is a deliberate ceiling. With 4 players, each peer sends audio to 3 others — 6 connections in total. At 10 players it would be 45, and uplink becomes the bottleneck. So 4 is a hard cap, not an arbitrary number.

Two mistakes that lied to me

The first was the 404 fallback. Static hosts often serve the index page for unknown paths, so a typo returns 200 with the home page attached. My first link checker asked, "did this URL return 404?" It reported no problems while links were broken. The fix was to stop trusting the status code and ask whether the page that came back is the page that was requested — compare title and content, not the status.

The second was "both deployments succeeded". The site ships to Cloudflare Pages and GitHub Pages. A successful deployment does not mean identical bytes: build order, caching and line endings can all differ. The only claim I trust is an md5 comparison of the same path fetched from both hosts. Right now every page is byte-identical. That is a much stronger statement than two green checkmarks, and it is the one I put in the README.

Both mistakes share a shape: I measured a proxy for correctness instead of correctness itself. A status code is not a page. A green deploy is not the same artifact.

Limits I would rather state

The mesh voice only makes sense at 4 players. There are no accounts and no match history. Some of the AIs are heuristics, not search — they play legally and reasonably, and they will not beat a strong human. The site has no analytics, which also means I cannot tell you which game people like most.

That is the honest shape of a one-person static project: a small surface, a lot of counting, and a few places where the only reliable test is a byte comparison.

If you want to try it, the games are here: https://hhfa008.github.io/games/

Two consulting slots this week: one hour, one iOS problem

Devin — Sun, 08 Nov 2026 06:40:40 +0000

I'm taking two consulting slots this week — one-hour, focused sessions on a single concrete iOS/macOS problem.

What I'm good at (13 years native Apple platforms):

A crash you can't reproduce, or a frame-rate drop that only shows up on device
Memory growth in long sessions
WebRTC / screen-sharing pipelines: capture, encode, transport, decode
App Review rejections (2.1, 4.2, 4.3, 5.1) — evidence gathering, not just appeals
Inherited codebases: where to start, what to leave alone

Recent work: merged two parallel live-streaming iOS SDKs into one 2M-line codebase (~90% less maintenance, zero high-severity incidents); led client work for a global sports event with 30+ engineers; rebuilt the view layer of a 100M+ DAU app.

How it works: $15 for 60 minutes, booked here → https://devinspire734.gumroad.com/l/60-minute-ios-debugging-session
You send the symptom, the stack, and what you already tried; I reply within 24h with two time slots (GMT+8, EU afternoon / US morning overlap). If I can't reproduce or explain the issue on the call, it's free.

If you'd rather just read: I also wrote up one of these debugging sessions as a case study — 0% packet loss and still 35 freezes → https://dev.to/devinios/zero-packet-loss-35-freezes-a-screen-sharing-performance-case-study-3hf8

Two slots, first come first served.

Vì sao chia sẻ màn hình vẫn giật dù mất gói 0%?

Devin — Sun, 08 Nov 2026 04:42:25 +0000

Chúng tôi đo một phiên chia sẻ màn hình trên macOS: độ phân giải 3024x1964, H.264 qua VideoToolbox, fps trung vị 29, thời gian encode 12.3 ms, dropped=0 ở phía chia sẻ.

Mạng thì hoàn hảo: RTT 0–1 ms, mất gói 0%.

Nhưng ở phía người xem, freezes vẫn cộng dồn tới 35 lần, có đoạn 2 giây chỉ còn 3 fps.

Kết luận đầu tiên: nút thắt không nằm ở mạng.

Vì sao

Ba thứ thường bị bỏ qua khi đo:

Encode không phải nguyên nhân. 12.3 ms/frame trên ngân sách 33 ms ở 30 fps — mới dùng 37%.
Adaptive quality không hề kích hoạt. 501/501 mẫu đều limit=none, tức engine chưa từng giảm chất lượng dù fps rơi xuống 3.
Thứ tự giảm chất lượng bị ngược. Với screen sharing, WebRTC xu hướng giữ nguyên độ phân giải và hy sinh frame rate. Kết quả: hình vẫn nét nhưng bị giật.

Nên làm gì

Đặt degradationPreference tường minh, đừng để mặc định.
Tự dựng thang độ phân giải (100% → 66% → 50%) và hạ độ phân giải trước khi fps xuống dưới 15.
Giới hạn độ phân giải ngay ở capture. Khung 3024x1964 là ~5.9 megapixel; hạ về 1920x1200 giảm 4 lần khối lượng.
Ghi log riêng dropped, freezes, thời gian decode — không có thì chỉ đoán.

Công cụ miễn phí

Mình làm một máy tính ngân sách latency/bandwidth cho chia sẻ màn hình:

https://hhfa008.github.io/tools/screen-share-calc/vi/

Bộ checklist 18 điểm + script đo đầy đủ (bản trả phí): https://devinspire734.gumroad.com/l/xejxsp/FIRST50

Kenapa Screen Sharing Tetap Lag Padahal Packet Loss 0%?

Devin — Sun, 08 Nov 2026 04:42:24 +0000

Kami mengukur satu sesi screen sharing di macOS: resolusi 3024x1964, H.264 lewat VideoToolbox, frame rate median 29 fps, waktu encoding 12.3 ms, dan dropped=0 di sisi pengirim.

Jaringan? Sempurna. RTT 0–1 ms, packet loss 0%.

Tapi di sisi penonton, freezes tetap terakumulasi sampai 35 kali, dengan satu segmen 2 detik yang hanya 3 fps.

Kesimpulan pertama: bottleneck-nya bukan jaringan.

Kenapa bisa begitu

Ada tiga hal yang biasanya luput diukur:

Encoding bukan penyebabnya. 12.3 ms per frame dari anggaran 33 ms di 30 fps — masih 37% terpakai.
Adaptive quality tidak pernah aktif. Dari 501 sampel, limit=none di semua sampel. Artinya engine tidak pernah menurunkan kualitas, meski frame rate jatuh ke 3.
Preferensi degradasi salah arah. Untuk screen sharing, WebRTC cenderung mempertahankan resolusi dan mengorbankan frame rate. Jadi saat jaringan menurun, yang Anda lihat adalah gambar tetap tajam tapi tersendat.

Yang bisa dilakukan

Set degradationPreference secara eksplisit, jangan biarkan default.
Buat tangga resolusi sendiri (100% → 66% → 50%) dan turunkan resolusi sebelum frame rate jatuh di bawah 15.
Batasi resolusi di sisi capture, bukan di encoder. Frame 3024x1964 adalah ~5.9 megapiksel; turun ke 1920x1200 memotong kerja 4x.
Catat dropped, freezes, dan waktu decode secara terpisah — tanpa itu Anda hanya bisa menebak.

Alat gratis

Saya membuat kalkulator budget latency & bandwidth untuk screen sharing: masukkan resolusi, fps, codec, dan bandwidth, lalu lihat di tahap mana budget Anda pecah.

https://hhfa008.github.io/tools/screen-share-calc/id/

Kalau butuh checklist lengkap 18 poin plus skrip pengukuran, ada di sini (versi berbayar): https://devinspire734.gumroad.com/l/xejxsp/FIRST50

Estimating screen-sharing bitrate: what bpp actually means and when it lies

Devin — Sun, 08 Nov 2026 03:01:13 +0000

Every screen-sharing feature hits the same meeting: someone asks how much bandwidth a share needs, and someone answers with a formula.

The formula is usually this:

bitrate ≈ width × height × fps × bpp

It looks authoritative: it has units, and it produces a number with "Mbps" after it. That is exactly why it is dangerous — it fails precisely on the content that matters most.

This post explains what bpp is, gives the ranges people use (as rules of thumb, not measurements), shows a measured control case where the formula holds, and lists three cases where it lies.

What bpp is, and what it is not

bpp — bits per pixel — is not a codec constant; no standard defines it. It is a budgeting convention: take one second of video, count its pixels, and ask how many bits to spend on each.

For a 1920×1080 stream at 30 fps, one second contains 62,208,000 pixels. At 0.08 bpp, that is about 4.98 Mbps. The arithmetic is trivial; the entire difficulty is choosing bpp.

The value folds in everything the codec does: motion compensation, quantization, entropy coding, and how much detail the encoder throws away. So bpp is useful as a planning number and useless as a guarantee. It answers "what budget should I reserve", not "what will the encoder produce".

Reference ranges (empirical, not measured here)

The numbers below are rules of thumb from common practice. I did not measure them for this post, and you should not quote them as if I did.

Codec / content	bpp (rule of thumb)
H.264, screen content	0.05 – 0.15
HEVC, same content and quality target	roughly 0.6 – 0.7 × the H.264 bitrate
AV1	lower again; treat the multiplier as a hypothesis, not a fact

Two warnings: the ranges are wide on purpose, and they describe typical mixed screen content — windows, text, images, some motion. Change the content and the range moves more than the codec does.

Remember what the formula multiplies. If you feed it a nominal 30 fps but the share emits 12 because the screen is static, you have overestimated by 2.5×. Use frames sent, not frames desired.

A control case where the estimate holds

We ran a share at 3024×1964, H.264 hardware encoding, median 29 fps. Per-frame encode time was 12.3 ms, and the encoder dropped 0 frames.

At 12.3 ms per frame the encoder ceiling is about 81 fps. At 29 fps, encoding occupies roughly 36% of the budget: 12.3 × 29 ≈ 357 ms out of every 1000 ms. The encoder is not saturated, and dropped stayed at 0.

That matters because the formula assumes the encoder can deliver the requested frames at the requested size. When it can, the estimate describes a real operating point. At 0.08 bpp this stream implies:

3024 × 1964 × 29 × 0.08 ≈ 13.8 Mbps

The number is a budget, not a measurement. What the test establishes is narrower: nothing on the encode side prevents the stream from spending it, so a bitrate estimate is the right tool.

Flip the condition and the formula fails. If per-frame encode time were 40 ms, the ceiling would be 25 fps, and asking for 29 fps would drop frames. Actual bits per second would land below the estimate — not because the network refused, but because the encoder never produced those frames.

Rule: check encode time × fps before trusting any bitrate estimate. Above roughly 70–80% of the frame budget, stop estimating and start measuring.

Where bpp lies

Three content patterns break the formula in three different directions.

Static screens: the estimate is too high. Inter-frame prediction makes unchanged regions almost free. A desktop that has not moved for ten seconds can encode to a few kilobits per second, not the 13.8 Mbps the formula predicts. Sizing a link from the estimate reserves capacity nobody uses. Worse, a bpp-driven health check will read a static share as suspiciously cheap.

Pure text: the estimate is too high, and quality is the thing to watch. Text on a flat background compresses extremely well, especially with screen-content tools, so actual bitrate can sit far below the estimate. The failure mode is not bandwidth but legibility: at a low bitrate the encoder can preserve the rectangles and destroy the strokes. Bits per second looks fine; the reader cannot read the code.

Video playback inside the share: the estimate is too low. A video window is high-entropy, fast-changing content with little screen-like redundancy. The encoder must spend real bits; capping it at the formula's budget does not make the content cheaper — it makes it blurrier. Measured bitrate will hit the ceiling and stay there, while quality collapses.

bpp estimates a typical frame, and screen content is rarely typical. It is a prior, not an answer.

The correction loop that actually works

Treat 0.08 bpp as a starting point and let the transport correct it. Measure bytesSent on the outbound RTP stream, convert to bits per second, and back out the bpp actually used:

async function measureBpp(pc, width, height, windowMs = 10000) {
  const snap = async () => {
    const stats = await pc.getStats();
    let r;
    stats.forEach(s => {
      if (s.type === 'outbound-rtp' && s.kind === 'video') r = s;
    });
    return { bytes: r.bytesSent, frames: r.framesSent, ts: r.timestamp };
  };
  const a = await snap();
  await new Promise(res => setTimeout(res, windowMs));
  const b = await snap();
  const seconds = (b.ts - a.ts) / 1000;
  return ((b.bytes - a.bytes) * 8) / seconds / (width * height * ((b.frames - a.frames) / seconds));
}

Use frames actually sent, not the configured rate. Read bytes at the transport, not the encoder, so RTP overhead and retransmissions are included. Use a window long enough to cross content states: ten seconds is a default.

Then close the loop: measure a representative minute, compare to the estimate, and correct the next estimate by the ratio. If measured bpp is 0.04 for your content mix at acceptable quality, plan with 0.04 — do not keep quoting 0.08 because a blog post said so. Including this one.

The short version

bitrate ≈ width × height × fps × bpp is a budgeting formula, not a codec guarantee.
Start at 0.08 bpp for H.264 screen content, then measure and correct.
Empirical ranges: H.264 0.05–0.15 bpp; HEVC roughly 0.6–0.7×; AV1 lower. These are rules of thumb, not measurements.
Before trusting the estimate, check that encode time × fps leaves headroom. In our 3024×1964 / 29 fps / 12.3 ms / 0-dropped run, it did.
Static screens and text will come in below the estimate; video playback will come in above it.
Use frames sent, at the transport layer, over a 10-second window.
The number you should trust is the one you measured on your own content.

To run the arithmetic and the trade-offs without a spreadsheet, there is a free calculator here: https://hhfa008.github.io/tools/screen-share-calc/. It shows every formula it uses and labels every output as an estimate.

Why your screen sharing drops to 5 fps instead of lowering resolution

Devin — Sat, 07 Nov 2026 19:53:52 +0000

When a screen share gets choppy, most people assume the network is bad. Sometimes it is. But there is a quieter failure mode where the network is fine, the encoder is fine, and the picture still collapses to single-digit frame rates while resolution never moves. The user ends up with the worst of both worlds: a soft frame that also stutters.

The reason is a default you probably never set: which dimension the pipeline is allowed to sacrifice under pressure.

The default is not "lower resolution"

WebRTC exposes degradationPreference on a sender, with three values: maintain-resolution, maintain-framerate, and balanced. The names suggest a symmetric trade-off. The behavior is not symmetric in practice.

For screen content, many stacks lean toward preserving resolution, because text and tables are unreadable if you halve the dimensions. That reasoning is sound for a still image. Video is not a still image. When the encoder is told to keep 3024×1964 and the bitrate budget tightens, it cannot keep quality by magic. It keeps the pixel count, spends fewer bits per frame, and the result is visibly soft. At the same time, the pacing logic has to shed something, and what it sheds is frame rate.

So "maintain resolution" does not actually protect legibility. It protects a number. What the viewer sees is a soft, low-frame-rate stream that is neither sharp nor fluid.

What our measurement showed

We ran an end-to-end capture of a 3024×1964 window, H.264 through the hardware encoder (VideoToolbox), sender and viewer on the same subnet.

Median frame rate: 29 fps. Minimum: 3 fps.
Single-frame encode time: 12.3 ms.
Dropped frames reported by the encoder: 0.
limit queried from the capture side: none in all 501 of 501 samples.

Read those together. The encoder is not struggling — 12.3 ms per frame leaves headroom. No frames were dropped at the encoder. The bandwidth estimator never once asked the pipeline to constrain itself. And yet the delivered frame rate bottomed out at 3.

That is the important part: adaptation never engaged. The 3 fps was not the controller choosing a lower frame rate under a bandwidth limit. It was the absence of any decision. Resolution stayed at full size the entire run because nothing touched it, and frame rate fell for reasons upstream of the decision the controller makes.

If you only watch average frame rate, this looks like "bad network." It is not. It is a pipeline with no owner for the resolution-versus-frame-rate choice.

Why the frame rate is the first thing to go

Three mechanisms conspire:

Capture is uneven. Screen content alternates between still and scrolling. A capture loop that emits at a fixed rate builds queue depth during scrolls. The frame rate you measure is an average over wildly varying demand.
Encode time is roughly fixed. 12.3 ms per frame is a throughput ceiling that does not move when content gets harder; the demand does. Throughput flat, demand spiky: queueing is the only possible result.
The jitter buffer is built for jitter, not for stops. If the sender pauses for 200 ms, the buffer underruns and the viewer freezes, while every network counter stays perfect.

None of these are solved by a preference flag. A preference says what to do if the engine decides to act. It does not make the engine act, and it does not remove capture-side load.

Three things you can actually control

1. Set `degradationPreference` explicitly

Do not inherit it. Decide, per session, what the product needs:

const sender = pc.getSenders().find(s => s.track?.kind === 'video');
const params = sender.getParameters();
params.degradationPreference = 'maintain-framerate';
await sender.setParameters(params);

For screen sharing where motion and responsiveness matter, maintain-framerate is often the better default — a soft frame that keeps updating is easier to follow than a sharp frame that freezes. For a document being read line by line, the opposite may be true. Either way, the choice is a product decision, not something to leave to the engine.

2. Maintain your own resolution ladder

Instead of waiting for adaptation, step resolution yourself on a schedule tied to observed conditions:

3024×1964 → 1920×1247 → 1280×831 → 960×623

Call applyConstraints with a reduced width/height, or use scaleResolutionDownBy on the sender. The point is that each step is a real, bounded reduction that lowers encode cost and queue depth — not a preference the engine may ignore.

3. Limit resolution at capture

The cheapest fix is upstream: do not capture at native size and hope the encoder manages it. Cap the capture dimensions to what the viewer can actually resolve. A 3024-pixel-wide frame on a 1440-pixel viewer is mostly wasted bits, and it makes every downstream stage more expensive. The pipeline gets easier at every stage when the source is already the size you intend to send.

The decision you are not making

The uncomfortable summary: when the engine keeps resolution and drops frames, it is making a choice you did not make. That choice is defensible for a photograph and wrong for a live screen. A number that preserves legibility at 3 fps is an illusion — a sharp frame no one can read because it never changes.

Instrument the frame-rate distribution, not the average. Watch whether limit ever leaves none. If it never does, adaptation is not saving you, and the resolution ladder is yours to build.

I also made a free calculator for this: Screen Sharing Latency & Bandwidth Calculator — plug in resolution, frame rate, codec, and available bandwidth to see where the budget breaks.

Zero packet loss, 35 freezes: a screen-sharing performance case study

Devin — Sat, 07 Nov 2026 18:56:24 +0000

Zero packet loss. 0–1 ms round-trip time. A hardware encoder that dropped nothing. And a viewer that froze 35 times, bottoming out at 3 frames per second.

That was the result of an end-to-end measurement of a macOS screen-sharing stack. The network was not the problem — and also not irrelevant. Understanding how both are true changes what you instrument and what you optimize.

The setup and the symptom

One Mac sharing a 3024×1964 window, H.264 through the hardware encoder, sender and viewer on the same subnet. Over 501 samples, RTT never left the 0–1 ms band and packet loss stayed at 0%. The encoder reported zero dropped frames.

The viewer told a different story: 35 cumulative freezes. Frame rate had a median of 29 fps, a minimum of 3, and a maximum of 48 — including a two-second stretch that delivered three frames. Single-frame encode time was 12.3 ms, roughly a third of a core at 29 fps. Sender CPU averaged 27.9% (41.2% peak); viewer CPU averaged 20.4%. Resident memory stayed inside a stable band on both sides, with no monotonic growth. The bandwidth limit queried from the capture side came back none in all 501 samples.

So: a perfect channel, an idle-looking endpoint, and a picture that periodically stops.

How to measure "stutter"

The word stutter is useless until you define it. I split observation into three planes and required all three to share one timeline.

Process and resources. CPU mean and peak, RSS range, and whether memory trends upward. This plane exists to kill the most over-used explanation first: a leak. There was no leak.
Capture and encode. Frame rate as a distribution — median, minimum, maximum — not an average. Per-frame encode time, dropped-frame count, capture queue behavior. An average of 29 fps with a minimum of 3 is a localized-stall signature that the mean erases.
Transport. RTT, loss, the actual ICE candidate pair type, DTLS-SRTP completion time, and the limit value asked of the capture side at intervals — "what am I being constrained to right now?"

Then the definition that matters: a freeze means the viewer's picture did not update for N consecutive frames. It does not mean a packet was lost. Capture stalls, encoder queueing, renderer drops, and jitter-buffer underruns all land in one bucket. That is why the first reaction to a freeze count should be "check the pipeline," not "check the network."

Three counterintuitive findings

1. A good network is exactly why adaptation never engages

Five hundred and one samples. Five hundred and one limit=none answers.

A bandwidth estimator assumes bandwidth is scarce. At 0–1 ms RTT and 0% loss, it concludes that available capacity is far above the current bitrate and reports no constraint. That is not a bug; it is the design. Adaptation is bandwidth-driven, not quality-driven. Its inputs are channel observations; its outputs can only constrain encoder parameters. When the bottleneck is not in the channel, the controller has no error signal.

The paradox follows: the better the network and the busier the endpoint — same-subnet casting, LAN collaboration, local demos — the less likely adaptation is to act, because the bottleneck sits in capture and encode queueing, upstream of the channel.

It also explains "all green in the lab, users still report stutter": a test matrix that covers only weak networks makes adaptation look excellent, and never covers strong network plus endpoint bottleneck — not a rare corner case.

2. Thirty-five freezes and 0% loss describe the same system

Zero loss proves packets arrived. It does not prove they arrived on time and in a group, and it does not prove the decoder can put them on screen.

Three stages produce "every packet arrived and the picture still will not move." First, capture does not emit evenly: 3 to 48 fps is a 16× spread, because screen content alternates between still and scrolling. A fixed-rate capture piles up queue depth during scrolls. Second, encode time is fixed while queue length is not. 12.3 ms per frame gives a theoretical ceiling near 81 fps, but it never varies with content while demand spikes when content changes. Demand up, throughput flat: queueing is the only possible outcome. Third, a jitter buffer is built for arrival-time jitter, not for the sender simply stopping. If the sender stalls for 200 ms, the buffer underruns and the player freezes while every network counter stays perfect.

Those 35 freezes were not 35 network incidents. They were 35 pipeline-rhythm incidents.

3. The problem was not the degradation order — it was that nobody made the choice

For screen sharing, the right degradation order is frame rate first, resolution second: text and tables depend on resolution for legibility, not frame rate.

But nothing degraded in that order. Resolution stayed at 3024×1964 for the entire run, encode time stayed at 12.3 ms, and limit stayed at none. Three frames per second was a passive outcome, not an active decision: resolution survived because nothing touched it, not because something protected it.

The trap is conflating a preference with a constraint. A preference flag says "if you must choose, keep resolution." It neither prevents frame rate from falling to 3 nor reduces capture load. Expecting a preference to act as a guardrail is expecting a recommendation to enforce itself.

Eight fixes that ship

Make capture parameters explicit. Cap width and height, set minimumFrameInterval to 1/15–1/20 s, and keep queueDepth around 3–5. Native resolution at 3 fps buys readability nothing and costs pipeline pressure.
Skip unchanged frames. Combine contentHint with frame-difference detection and send only 1–5 fps heartbeat frames while the screen is still. This usually moves freeze counts more than encoder tuning, because it removes uneven frame spacing at the source.
Make the frame-rate floor a separate constraint. "Maintain resolution" is a preference. Add an independent never-below-10-fps guardrail that lowers capture resolution when triggered instead of letting frames drop.
Evaluate HEVC and screen-content coding. Hardware HEVC saves bitrate at equal subjective quality, and screen-content extensions are worth testing. Keep per-frame encode time in monitoring; it is the pipeline's throughput ceiling and does not move with frame rate.
Report receiver-side metrics separately. Freeze count alone is not enough: report duration distribution, and distinguish "no new frame arrived" from "a frame arrived but was never rendered." Users call both stutter; they are different chains.
Test the TURN path explicitly. A 0–1 ms direct-path result does not generalize to a relay. Force the relay candidate and rerun the suite.
Look at audio alongside video. If the microphone is open, echo and noise are perceived before video problems. Enable AEC and noise suppression conditionally on the mic, not globally disabled to save CPU.
Shrink the black-screen window at join time. Connect and first-frame time leave a gap where the user stares at nothing. Send a low-resolution keyframe to light up the picture, then step quality up. Users tolerate blurry-but-immediate better than sharp-but-late.

Measurement beats optimization

The most valuable part was not the eight fixes, but that three individually normal readings only exposed the problem when read together:

27.9% CPU says the endpoint is idle.
0% loss and 0–1 ms RTT say the network is great.
501 of 501 limit=none says adaptation is working.

Stack all three and "35 freezes, minimum 3 fps" becomes visible.

Screen-sharing stutter hides in the frame-rate distribution, not the mean. It hides in frame spacing, not loss. It hides in negative evidence — the constraint that was never set. Get the measurement right first; optimization is one-time, measurement compounds.

I also built a free calculator for this: Screen Sharing Latency & Bandwidth Calculator — plug in resolution, fps, codec and bandwidth, and it shows where your latency budget breaks.

If you want the full checklist and scripts, here's the pack ($9).

Disclosure: I sell a paid pack that expands on this case study.

Items Saturate; Judges Do Not: What a Leaderboard Means When a Model Does the Scoring

Devin — Tue, 27 Oct 2026 23:36:27 +0000

This is part three of a short series on reading AI evaluations. Part one, Same Model, Same Scaffold, 10 Runs, measured repeats: one configuration, ten runs, a 2.2–6.0 point window. Part two, 62% vs 58% on the Same Benchmark, measured task counts. Both assumed a stable instrument. This part asks what happens when the instrument is a model.

A 0–5 leaderboard puts model A at 4.1 and B at 3.7. Four tenths of a point — enough for a tweet that A leads, enough for B's team to rewrite prompts. Now learn that the minimum detectable difference (MDD) for such leaderboards at native item counts is 0.41 to 1.24 points, and that on the paper's general rubric, 0.4 does not clear it. What do you say then?

Atul Anand's Ask Which, Not How Good: Sizing Benchmarks Scored by an LLM (arXiv:2609.27787, v1, preprint, August 17, 2026) decomposes 373,019 judge calls across 4 item sets, a 9-judge panel, and 42 systems. Its conclusion: when a model scores, neither item count nor run count is the variable that matters most.

The key is which question, not how good

A pointwise question asks what score an answer deserves; a pairwise question asks which of two answers is better. Most leaderboards use the first because it ranks cleanly, but they are not equally hard: "which is better" is far better conditioned, while "what score" folds the judge's scale habits, severity, and wording into the error.

Generalizability theory splits score variance into system, item, judge, and interactions such as system × judge, then computes Eρ², the share that is signal: rerun and the ranking holds, from 0 (noise) to 1 (repeatable). It is not accuracy; a judge can be stable and consistently wrong.

Why the ceiling does not rise with item count

This is the counterintuitive part:

"under a single judge, generalizability asymptotes to σ²s/(σ²s+σ²sj) regardless of item count, because the system-by-judge term carries no n_i. Items saturate; judges do not."

With one judge, the ceiling is σ²s/(σ²s+σ²sj) however many items you add: system × judge enters the error as σ²sj/nj, with judges in the denominator. Items shrink only the item-sampling term. What remains is "this judge is generous to A and harsh to B," which never cancels: one judge scores every item.

Two numbers pin it down. On MT-Bench at 0.75, 2 judges × 30 items roughly equals 1 judge × 77 items. At 0.80 nothing suffices: 0.80 sits 0.006 below the single-judge ceiling, and approaching it needs a divergent item count.

The advice splits in three. More items do not help within a fixed pool. More judges only approach the ceiling. Raising it needs a different judge universe, a different protocol, or item selection by discriminative power — the only item-side lever that raises it, to a 16.6% system variance share against Arena-Hard's 19.2%. Reading "items don't help" as "item design doesn't matter" inverts the point.

What 0.986 is, and what 8.6 points are

0.986 is often misread as 98.6% accurate. It is the reliability ceiling one judge reaches under a pairwise protocol that judges each pair in both orders and treats position as a factor. σ²sj falls two orders of magnitude below σ²s and the ceiling rises from 0.600–0.798 to 0.986 (bootstrap [0.934, 1.000], 11 systems); one judge reaches Eρ² = 0.93 at 24 items, so "one judge suffices."

But 0.986 means "rerun and the ranking holds," not "the judgment is right." On SummEval, reliability rose 0.711 → 0.959 (+0.248) while Kendall τ against a three-expert consensus moved only 0.596 → 0.710 (+0.114). Reliability is necessary for validity, not sufficient.

Then the 8.6 points, a new problem pairwise buys:

"A system placed in the first slot wins 8.6 percentage points more often than the same system, judged by the same judges on the same items, placed second."

First slot is presentation order, not first place: same system, same judges, same items, yet going first alone lifts the win rate 8.6 points. The bias varies 7.3 points across systems, reaches 25 at worst, and does not average away. Across 53 published win-rate comparisons the median claimed gain is 7.0 points; position bias is 1.23 times that, so an uncontrolled pairwise evaluation imports a bias larger than the median effect it hunts. In the same data, 70% of win-rate claims fall below pairwise's floor at native item counts.

An audit of 628 papers

The audit covers 628 arXiv papers from 2024–2026: five preregistered queries, 320 sampled by fixed seed, 312 coded, 227 reporting judge comparisons, κ = 0.73. Fewer than a quarter say whether they ran more than once (16.7% human, 20.0% automated); 46–67% report uncertainty; and all 17 checkable MT-Bench gains fall below its own floor (0.63 at 30 items, 0.54 at the native 80), which barely drops with more items because items saturate. This does not allege fabrication: the differences sit below the instrument's resolution, and the papers omit what a reader needs to notice.

What to do when you run a judge yourself

Report the judge universe: models, versions, count; otherwise reliability claims float.
Use the benchmark's native protocol and name it. Under MT-Bench's own template, 1–10 scale, and reference answers, the system variance share rises from 4.8% to 8.4% and the judges needed for 0.8 fall from 18 to 3.
Spend on judges before items. For 0.8 under a general 0–5 rubric: MT-Bench 25 judges, AlpacaEval 2 seven, the selected set and Arena-Hard two each.
Report MDD beside delta: the floor is 0.41–1.24 on a 0–5 scale, while 60 recovered pointwise gains have a median of 0.28; below it is not "leading."
When building a benchmark, select items by discriminative power.
Disclose repeat runs and response caching. Caching makes repeats fake.

With pairwise, run both orders and report prompt-level agreement: one judge rerun unchanged at temperature 0 moved the total about 1 point while 21.2% of items flipped win/tie/loss (arXiv:2610.30512).

What this paper cannot show

Six things, at least.

First, it is a single-author preprint v1, not peer reviewed.

Second, data and code are available from the author, not a public artifact: tables check out, reproduction is not cheap.

Third, it covers only 4 item sets, one pointwise 0–5 rubric, one scale; the paper concedes the general rubric is the conservative case.

Fourth, the Arena-Hard cell holds 17 items (others 32–43), making its σ²s estimate least precise; the pairwise arm uses the authors' own items, judge, and fixed baseline, not Arena-Hard's or AlpacaEval 2's pipeline.

Fifth, the audit samples rather than censuses: 628 framed, 312 coded, 227 reporting comparisons, κ between 0.45 and 1.00. It documents missing disclosure, not incorrect conclusions.

Sixth, reliability is not validity. A stable judge can stably measure the wrong thing, as SummEval shows.

The ruler

Back to 0.4 points: by this ruler it sits below the noise floor of most judge-scored leaderboards, and "who is stronger" is not answerable yet. Keep, not "0.4 points doesn't count," but "Items saturate; judges do not."

Adding items writes more marks on the ruler; changing judges makes the marks finer; changing the protocol swaps the ruler. Most people only do the first.

Disclosure: drafted with AI assistance and edited by the author.

62% vs 58% on the Same Benchmark: How Many Tasks Before the Gap Means Anything?

Devin — Tue, 27 Oct 2026 21:06:29 +0000

Two teams run the same agent benchmark on the same model with the same scaffold. Team A runs 100 tasks and reports 62%. Team B runs 250 and reports 58%. Both publish their task lists and pass rates. Neither is cheating.

Is the four-point gap real, or did the two teams stop at different points?

Wei-Jung Huang's paper (arXiv:2607.12338, 14 July 2026, KDD 2026 Agentic AI Evaluation and Trustworthiness workshop) answers this without running any new model. It shuffles the task order of records that three public agent benchmarks already published, then asks whether the decision after the first 15%, 25%, 50%... of tasks matches the decision after all of them.

Replay, not rerun

No new data, only a change in how much you see:

SWE-bench Lite: 552 ordered comparisons (276 unordered pairs), median 300 shared tasks
SWE-bench Verified: 156 ordered (78 unordered), median 496
AppWorld: 306 ordered (153 unordered), 585 tasks, 36 packages (18 systems × 2 splits)
tau-bench: 12 ordered (6 unordered), 278 entries per system over airline, retail, and telecom, with cost fields

Each pair gets 500 reveal orders and 200 bootstraps (tau-bench: 2,000 and 500). A real evaluation runs once in one order; replay estimates how often a comparison stalls if you stop early.

Three numbers that wear a percent sign

0 pp threshold: A wins if strictly above B on the completed task set, so one task can flip the whole decision. The strictest setting, no margin.

5 pp budget grid: a step size, not an effect threshold — 5% to 95% in 5-point increments. The paper separately tests 5 pp and 10 pp effect thresholds; don't merge them.

Unresolved rate: comparisons where the policy picks neither side and asks for more tasks or declines to conclude. Three gates hold together: at most 5% conditional decision error among decided comparisons, at most 5% coverage failure, at most 25% unresolved. That last is 25%, not 5%: being right when you decide doesn't help if you rarely decide.

Four benchmarks, four answers

At 0 pp on the 5 pp grid:

AppWorld: 15% — about 88 of 585 tasks, ~21% unresolved; still 15% at 5 and 10 pp
tau-bench: 25% — about 70 of 278 entries; 30% at 5 pp, 35% at 10 pp
SWE-bench Verified: 90% — about 446 of 496, 24.22% unresolved
SWE-bench Lite: 95% is still not enough. At 25%, error and coverage are in range, yet 55.92% of comparisons stay unresolved; at 95% it is 27.25%, above the ceiling.

What differs is the distribution of unresolved comparisons and coverage gaps: Lite decides accurately at 25% but leaves over half undecided; Verified only reaches 24.22% unresolved at 90%.

AppWorld arrives early because a coverage-aware bootstrap under difficulty grouping covers the required groups at 15%. A forced stratified policy needs 50%; forced uniform random fails coverage 99.96% of the time at 25%. On tau-bench, cheapest tasks first spends 11.51% of cost at 25% with 100% coverage failure.

Why the answers move

Five levers, none intrinsic to a benchmark: Lite has no sufficient budget at 0 pp but 55% at 5 pp and 35% at 10 pp; one-task slack on a coarse grid cuts its 0 pp budget to 75%, while repository × difficulty leaves more unresolved; low error often means few decisions rather than few mistakes; keeping one direction per unordered pair leaves 275 positives against 1 negative at 0 pp, so the paper targets ordered comparisons; and AppWorld's 15% moves to 10% or 25% as the truncation parameter a goes 0.10, 0.05, 0.02. A proportion is not a count: Lite's 95% of 300 is about 285 tasks, Verified's 90% of 496 about 446.

What this means when you read a paper

"We ran 100 tasks and A beat B by 0.5 points" is not a result. The paper's checklist asks five things:

What is the threshold, in percentage points?
How were tasks selected — random, stratified, cheapest first?
What is the coverage rule, and how many tasks per group?
What is the decision rule — bootstrap tail, paired normal, McNemar?
Is there a cap on unresolved comparisons, and is the actual rate reported?

Missing any one, "0.5 points higher" is "no conclusion yet": the same records go from no sufficient budget to 55% by changing the threshold.

What this paper cannot show

First, replay is not a new experiment: "It does not estimate performance on future tasks, because the replay orders permute a fixed completed record." It tells you how much of this batch was needed, not whether 15% survives a fresh batch.

Second, every conclusion depends on the threshold, coverage rule, grouping, and policy parameters. Move 0 pp to 5 or 10 pp and Lite goes from nothing to 55% or 35%; one-task slack on a coarse grid brings its 0 pp budget to 75%; AppWorld's 15% drifts from 10% to 25% with the truncation parameter. These are design choices, so a budget must travel with its direction and coverage rule.

Third, only three benchmark families are covered. OSWorld, WebArena, AgentDojo, AndroidWorld, Terminal-Bench, AgentBench, and MiniWoB++ were not replayed, and tau-bench's grid holds only 12 ordered comparisons (6 pairs). Don't extrapolate "AppWorld is 15%" to other interactive benchmarks.

Fourth, the public records are filtered leaderboard entries: SWE-bench keeps only near-complete entries covering at least 95% of a split, and AppWorld uses 36 public packages. Voluntary submissions, not random samples, so composition changes move the estimate.

Fifth, the cost findings hold only on tau-bench. Neither SWE-bench nor AppWorld publishes a cost field, so "cheap tasks first" is checkable on tau-bench alone.

Sixth, 5% decision error, 5% coverage failure, and 25% unresolved are the paper's shared targets, not an industry standard. Change the targets and the budgets change: a framework for computing a budget, not a universal "15% is enough".

One rule you can use now

When you see "we ran X tasks and A beat B by 0.5 points", check three things: is the decision threshold stated, were tasks selected and covered by repository, difficulty, or domain, and is the unresolved share reported. If any is missing, file it as "no conclusion yet". A task proportion is not a decision rule; without one it is just a cost figure.

More of my writing on AI evaluation: hhfa008.github.io/ai/writing

This is part two of a short series on reading AI evaluations. Part one: Same Model, Same Scaffold, 10 Runs.

Disclosure: drafted with AI assistance and edited by the author.

Same Model, Same Scaffold, 10 Runs: Agent Benchmark Scores Still Move by Up to 6 Points

Devin — Tue, 27 Oct 2026 20:56:08 +0000

Two teams run the same model, scaffold, and temperature on SWE-bench Verified, once each. Team A reports 32.4%; Team B reports 28.8%. Both post their logs. Neither is lying.

The gap comes from one distribution. Table 1 of On Randomness in Agentic Evals has this row: DeepSWE-preview on nano-agent, temperature 1.0, ten independent runs, mean 31.4% ± 1.0%, minimum 28.8%, maximum 32.4%. Team A landed at the top of the window; Team B landed at the bottom. They were comparing two samples, not two systems.

The paper, by Bjarni Haukur Bjarnason, André Silva, and Martin Monperrus, appeared on arXiv as 2602.07150 on February 6, 2026 (v3: March 25) and is a preprint.

What the experiment did

Most papers run each configuration once. This one ran three models (Qwen3-32B, DeepSWE-preview, Devstral-2) × two temperatures × two scaffolds (nano-agent, R2E-Gym), ten times each, on 500 SWE-bench Verified tasks: 60,000 trajectories, 25.58B tokens, 1.88M tool calls.

One definition matters. The headline range of 2.2 to 6.0 percentage points is the spread between the minimum and maximum single-run score within the same configuration. It is not the gap between two models. As the paper puts it: "single-run pass@1 estimates vary by 2.2 to 6.0 percentage points depending on which run is selected." The number measures repeatability, not capability.

Three sources of variance

The first is sampling. Above temperature zero, every token is drawn from a distribution. One different token changes the context for the next one, and the run can diverge from there.

The second is the scaffold. It changes both the level and the spread. DeepSWE-preview scores 31.4% ± 1.0% on nano-agent at temperature 1.0, but 34.4% ± 1.5% on R2E-Gym. Devstral-2 scores 63.5% ± 1.1% on nano-agent at temperature 0.2, but 34.9% ± 1.5% on R2E-Gym. The model did not change; the wrapper did.

The third is the runtime, and it only becomes visible at temperature zero.

Temperature zero is not a determinism switch

Greedy decoding sounds deterministic: always pick the most likely token. The paper shows it is not. Floating-point addition is not associative, and parallel reduction order, hardware-specific kernels, and batching can produce tiny differences in the same logits. Autoregressive decoding amplifies them.

At temperature 0, the standard deviations shrink but never reach zero. Qwen3-32B on nano-agent is 16.4% ± 1.2%; DeepSWE-preview on nano-agent is 20.4% ± 1.0%; Qwen3-32B on R2E-Gym is 22.3% ± 1.8%. The largest standard deviation at temperature 0 in the table is 1.8 points.

Lower temperature does not reliably shrink variance either. Qwen3-32B on nano-agent has a standard deviation of 0.7% at temperature 0.6 and 1.2% at temperature 0. Temperature zero delays divergence; it does not remove it.

So when a paper says temperature=0 makes its result reproducible, ask: over how many runs, with what standard deviation, on which inference engine and hardware?

Trajectories split early

For DeepSWE-preview on nano-agent at temperature 1.0, the median position of the first differing token between two runs was token 5, about 0.5% of the trajectory. At temperature 0 it moved to token 56. The paper's conclusion is that the first divergence almost always occurs within the first 1% of a trajectory. The next few hundred steps are the consequence of that one token.

The paper gives a concrete case: django__django-9296, Qwen3-32B, nano-agent, temperature 0. Two runs matched for 93 tokens. At token 94, one searched for the Paginator class; the other read the Django source. One patch landed in the wrong place; the other passed.

How to read a leaderboard

Before the ranking, ask five questions:

How many runs? Is this the mean of n independent runs, or the best single run?
Is there a standard deviation or interval? Without one, the score is a point, not an estimate.
What temperature? Zero is not deterministic; higher temperatures usually widen the spread.
What scaffold? The same model can move by tens of points when the harness changes.
Is this pass@1 or pass@k? DeepSWE-preview on R2E-Gym scores 34.4% pass@1, 52.9% pass@5, and 15.5% pass^5. All three say "pass"; they answer different questions.

The paper supplies a contrast. DeepSWE-preview on nano-agent scores 31.4% ± 1.0% at temperature 1.0 and 20.4% ± 1.0% at temperature 0, an 11.0-point difference the authors treat as real. Devstral-2 on the same scaffold scores 63.5% ± 1.1% at temperature 0.2 and 63.8% ± 1.6% at temperature 0, a 0.3-point difference they treat as noise. The deciding factor is the gap relative to the standard deviation.

How many runs you need

The paper uses a power calculation. It assumes single-run scores are approximately normal; Shapiro-Wilk tests passed for 11 of 12 configurations at p ≥ 0.05. Using the observed standard deviations of 0.7% (minimum), 1.5% (median), and 1.8% (maximum):

Detecting a 2-point improvement at p < 0.05 takes 2, 9, or 13 runs.
Detecting a 1-point improvement takes 8, 36, or 51 runs.
Detecting a 5-point improvement takes 2 runs at the median variance; 10 points takes 1 run at any of the three.

The paper states: "Detecting a 2% improvement at p < 0.05 with 80% power requires approximately 9 runs per agent under test, while detecting a 1% improvement requires 36 runs." The number depends on the effect you need to detect and the variance of your own setup.

What this study cannot show

First, it is a preprint: v1 on February 6, 2026, v3 on March 25, 2026, with no peer review.

Second, it covers one benchmark, three models, two scaffolds, and two temperatures. SWE-bench Verified has long software-engineering trajectories; the 2.2–6.0-point window may not transfer to short tasks or much longer ones.

Third, the paper does not link a code or data artifact. Reproducing the distribution means building the scaffolds and running 60,000 trajectories yourself.

Fourth, there is no calendar window. The paper reports ten runs per configuration but not the dates when the data was collected.

Fifth, variance at temperature 0 depends on the inference stack. The finding is that temperature 0 does not guarantee determinism, not that every temperature-0 evaluation has a standard deviation of at least 1.5 points.

Sixth, run-to-run variance and task-sampling variance are separate sources. Too few tasks and too few runs produce different errors, and they should be budgeted separately. This paper addresses only the second.

The cost of a single run

An evaluation costs more than the price of running it once. It also costs the price of believing the result. 25.58B tokens is expensive, but it buys a measurement of how unstable a single score is, which is usually cheaper than making a decision on one sample.

The next time you see "this model improved by 3 points," ask whether that is a difference between two ten-run means, or two single runs that landed at opposite ends of the same window.

This piece is part of a series on how to read AI evaluations. More writing is indexed at https://hhfa008.github.io/ai/writing/.

Disclosure: drafted with AI assistance and edited by the author.

The Median Paid Newsletter Converts 0.62% of Free Readers. Do the Math Before You Launch

Devin — Tue, 27 Oct 2026 11:27:29 +0000

The most useful number in beehiiv's 2026 paid-newsletter report is not a success story. It is the median.

The report, published on June 22, 2026 and based on first-party data from thousands of publications on beehiiv, puts the median free-to-paid conversion rate at 0.62%. That works out to roughly six paying subscribers for every 1,000 free readers. The median paid plan is $10 a month or $100 a year, and the typical creator in the dataset launched a paid tier about 45 days after starting the newsletter.

Those numbers are a baseline, not a verdict. But they change the arithmetic of launching a paid newsletter, and they are worth doing before you write another issue.

What 0.62% actually implies

At 0.62%, 100 paying subscribers require about 16,100 free readers. A straightforward rounding gives roughly 16,000. At 1%, the same 100 subscribers need 10,000 free readers. At 5% — a rate that appears in a lot of creator advice — you would need 2,000.

The gap between those scenarios is not a rounding error. It is the difference between a list you can build in a few months and a list you may spend years building. The report is explicit that 0.62% is a median, not an average, which means the middle publication is doing less well than the headline success stories suggest.

This is my arithmetic, not the report's: divide 100 by the conversion rate. The report supplies the rate and the pricing; the list-size examples are derived.

Pricing is more standardized than the advice

The median paid plan is $10 a month or $100 a year across nearly every industry and list size, according to the report, and it has not moved since 2024. That is a useful anchor. It means a creator does not need to invent a pricing strategy from scratch to be in the middle of the market; $10/$100 is the middle.

It also means the revenue math is easy to state. A newsletter with 1,000 paying subscribers at $10 a month generates $120,000 a year before platform fees, payment processing, and tax. At a 10% platform cut, $12,000 leaves the business every year. The report uses that example to make a point that is easy to miss: a four-figure subscriber count can still be a small business after the costs that sit between gross revenue and take-home cash.

Niche sets the ceiling

The report's sector breakdown is where the median stops being abstract. Sports newsletters have the highest median conversion rate at 1.93%, just over three times the platform average. Finance has a median of 0.78%, but the top 10% reaches 20.00% — a gap the report describes as nearly 26x. The same vertical, the same audience type, very different execution.

The value per subscriber varies just as much. A 1,000-subscriber investing newsletter can earn more than $2,700 a year, while the median 1,000-subscriber travel newsletter earns about $252. Those figures are not a guarantee; they are a reminder that "newsletter" is not one market.

The pattern behind the numbers is consistent with what operators already say: subjects tied to a decision, a professional use, or a tax-deductible expense convert and retain better than general-interest topics. A reader who can connect the subscription to a raise, a client, or a cost is easier to sell than a reader who simply finds the topic interesting.

Churn is the other half of the equation

The report's monthly churn rates range from 5.06% in Food & Drink to 16.67% in Money. News sits at 5.47%, sports at 7.76%, and AI at 13.33%. Estimated subscriber lifetime runs from about six months in Money to nearly 20 months in Food & Drink. Median lifetime value ranges from $83 in Community to $230 in Investing.

Those differences compound. A newsletter with 10% monthly churn loses most of a cohort in less than a year; a newsletter with 5% churn keeps it much longer. The report's conclusion is blunt: retention is where the real money is. Conversion gets the subscriber; churn decides whether the subscription becomes a business. One methodological caveat: the report's lifetime figures are steady-state estimates (1 ÷ monthly churn), not cohort survival curves.

What the report does not tell you

The data come only from beehiiv publications. They do not necessarily describe Substack, Ghost, Kit, or any other platform. Beehiiv has a commercial interest in showing that paid newsletters can work, and the report also says the company generated $19 million in subscription revenue in 2025, up 138% from 2024, with $35 million projected for 2026.

The report does not disclose refund rates, sample sizes by industry, or a comparable Substack conversion figure. It reports medians, not averages, so the top decile is hidden inside a single number. Treat 0.62% as a starting assumption for a beehiiv-style publication, not a law of the newsletter market.

Four things to do before you launch

Run the list-size math at three conversion rates. At 0.62%, 1%, and 3%, how many free readers do you need for 100 paying subscribers? If the answer at the median is larger than the audience you can realistically build, change the plan before you change the price.
Pick a niche tied to a decision or a budget. The sports and finance spreads are not just pricing differences; they reflect whether the reader can justify the cost. "Interesting" is a weak buying reason.
Launch paid around the six-week mark, not after the archive is perfect. The median creator in the dataset launched a paid tier 45 days after starting. A clear promise and a small archive beat a perfect archive with no offer.
Model churn, not just MRR. Run your revenue at 5%, 10%, and 16% monthly churn. If the business only works at the low end, you have a retention problem to solve before you have a growth problem.

The median is not a ceiling. The top 10% in finance shows that execution can move conversion by an order of magnitude. But the median is the right place to start, because it is the number most creators will actually experience — and it is much cheaper to plan for it than to discover it after launch.

Redis Changed Its License in 2024. The Fork Is Still Shipping.

Devin — Tue, 27 Oct 2026 09:51:20 +0000

Reconstructed from public reporting; not firsthand, no invented people or dialogue.

On March 20, 2024, Redis changed its license. Starting with Redis 7.4, new releases would ship under RSALv2 and SSPLv1 instead of BSD-3. The company did not hide the reason. Most of its commercial sales, it wrote, ran through the largest cloud providers, and those providers "commoditize Redis' investments and its open source community."

Two days later, on March 22, a new repository appeared on GitHub. It was called Valkey. The license was BSD-3. The code was a fork of the last BSD release.

That is the whole story in three sentences. The interesting part is what happened next, because both sides kept receipts.

The fork did not fade

Most forks of a popular project die in a few months. This one did not.

As of October 27, 2026, the Valkey repository has 27,528 stars and 1,431 forks. Redis has 77,130 stars. The fork is smaller, and it is also still here.

More useful than stars: release activity. Valkey 9.1.3 shipped on October 26, 2026. In the same 48 hours, the project also published maintenance releases on four older lines: 9.0.7, 8.1.11, 8.0.12, and 7.2.15. Four live branches is not a museum project. It is a project with users who cannot all upgrade at once.

The 9.x line also carries real engineering claims. The project's site says 9.1 does 2.1 million requests per second on a single server, saves up to 20% memory on small strings, and moves Lua out of the core into an optional module. Treat vendor benchmarks as marketing until you run them yourself. But "Lua is now a module" is the kind of change you only make when you are optimizing a codebase you expect to maintain for years.

Also worth noting: Valkey is a Linux Foundation project. That matters less for code quality than for governance. A foundation cannot unilaterally relicense the project the way a company can.

The reversal, in the company's own words

On May 1, 2025, Redis added AGPLv3 as a licensing option, starting with Redis 8. It kept SSPL. In the announcement, CEO Rowan Trollope wrote the sentence that makes this story worth telling:

"This achieved our goal—AWS and Google now maintain their own fork—but the change hurt our relationship with the Redis community."

Read that twice. The company says the license change worked as intended and still cost it something it cared about. It is rare to get both halves in one sentence from the person who signed the change.

The same post notes that Salvatore Sanfilippo, the original author, rejoined Redis in November 2024 as a developer evangelist. Redis 8 also folded Redis Stack into the core: JSON, time series, probabilistic types, and the query engine now ship together under AGPL. The company claims more than 30 performance improvements, commands up to 87% faster, and twice the throughput. Those are vendor numbers. The date and the license are facts.

What AGPL does and does not fix

Here is the part that gets flattened in arguments. AGPL is OSI-approved. That makes Redis open source again in the formal sense. It does not make Redis BSD again.

AGPL keeps the copyleft trigger, and extends it to the network. If you modify the software and let other people use it over a network, Section 13 requires you to offer them the corresponding source. For most teams running an unmodified cache, nothing changes. For anyone shipping a modified Redis as part of a hosted product, the compliance work is real, and it did not exist under BSD.

One detail gets lost in the argument: Redis 8 is dual-licensed. AGPLv3 is an option, not a replacement for SSPL. If your legal team already approved SSPL terms, nothing forces a change. If they rejected SSPL, AGPL is the path back. That distinction matters in procurement, where the real question is not "is it open source" but "which license will our scanner see on the artifact we deploy."

So the scoreboard is not "open source won." It is narrower. A company traded a permissive license for commercial protection, lost community trust, watched a foundation-backed fork keep shipping, and then added a copyleft license that is open source but stricter than what it removed. Everyone got something. Nobody got 2023 back.

What I would do this week

Audit by version, not by project. Your dependency scanner probably reports "Redis" once. The license depends on which release you actually deploy: 7.2 and earlier are BSD, 7.4 and later are RSALv2/SSPL, and 8.x can be AGPL. Pin the version and record the license next to it.
Ask your managed cache provider two questions: which engine, and which major version. "Redis-compatible" is not an answer. The engine determines your upgrade path if a license or price changes again.
If you host a modified version as a service, read AGPL Section 13 before the next upgrade, not after. The obligation is to offer source to network users, and "we only run it internally" is a different case from "our customers call it."
Keep the exit cheap. The core command surface is largely compatible, but modules and managed features are where migrations get expensive. A thin client abstraction and one tested fallback image cost little now and buy you options later.

None of this requires picking a side. It requires knowing which license is running under your connection string.

The part that generalizes

License changes are cheap to announce and expensive to undo. Redis added AGPL thirteen months after leaving BSD, and the fork from March 2024 is still publishing releases every week.

If you maintain a dependency that a company controls, the lesson is not "avoid company-controlled projects." The lesson is that the license is a versioned artifact, like the code. Treat it that way: record it, diff it, and know what you would do if it changed on a Tuesday.

The fork is what that plan looks like when you did not have one. It works. It is also more expensive than a one-line entry in your dependency policy.

Sources: Redis announcements of 2024-03-20 and 2025-05-01; valkey.io; GitHub repository and release metadata. Figures are as of 2026-10-27.

DEV Community: Devin

18 Games, 617 Assertions: What a One-Person Static Arcade Taught Me About Testing

Why static, and why hand-drawn

The hard part: testing without looking

Multiplayer: where the server earns its place

Two mistakes that lied to me

Limits I would rather state

Two consulting slots this week: one hour, one iOS problem

Vì sao chia sẻ màn hình vẫn giật dù mất gói 0%?

Vì sao

Nên làm gì

Công cụ miễn phí

Kenapa Screen Sharing Tetap Lag Padahal Packet Loss 0%?

Kenapa bisa begitu

Yang bisa dilakukan

Alat gratis

Estimating screen-sharing bitrate: what bpp actually means and when it lies

What bpp is, and what it is not

Reference ranges (empirical, not measured here)

A control case where the estimate holds

Where bpp lies

The correction loop that actually works

The short version

Why your screen sharing drops to 5 fps instead of lowering resolution

The default is not "lower resolution"

What our measurement showed

Why the frame rate is the first thing to go

Three things you can actually control

1. Set degradationPreference explicitly

2. Maintain your own resolution ladder

3. Limit resolution at capture

The decision you are not making

Zero packet loss, 35 freezes: a screen-sharing performance case study

The setup and the symptom

How to measure "stutter"

Three counterintuitive findings

1. A good network is exactly why adaptation never engages

2. Thirty-five freezes and 0% loss describe the same system

3. The problem was not the degradation order — it was that nobody made the choice

Eight fixes that ship

Measurement beats optimization

Items Saturate; Judges Do Not: What a Leaderboard Means When a Model Does the Scoring

The key is which question, not how good

Why the ceiling does not rise with item count

What 0.986 is, and what 8.6 points are

An audit of 628 papers

What to do when you run a judge yourself

What this paper cannot show

The ruler

62% vs 58% on the Same Benchmark: How Many Tasks Before the Gap Means Anything?

Replay, not rerun

Three numbers that wear a percent sign

Four benchmarks, four answers

Why the answers move

What this means when you read a paper

What this paper cannot show

One rule you can use now

Same Model, Same Scaffold, 10 Runs: Agent Benchmark Scores Still Move by Up to 6 Points

What the experiment did

Three sources of variance

Temperature zero is not a determinism switch

Trajectories split early

How to read a leaderboard

How many runs you need

What this study cannot show

The cost of a single run

The Median Paid Newsletter Converts 0.62% of Free Readers. Do the Math Before You Launch

What 0.62% actually implies

Pricing is more standardized than the advice

Niche sets the ceiling

Churn is the other half of the equation

What the report does not tell you

Four things to do before you launch

Redis Changed Its License in 2024. The Fork Is Still Shipping.

The fork did not fade

The reversal, in the company's own words

What AGPL does and does not fix

What I would do this week

The part that generalizes

1. Set `degradationPreference` explicitly