Testing the claim: a degraded-link matrix as a required CI gate

Bobby Larson — Wed, 19 Aug 2026 21:04:51 +0000

This is a writeup of building a required CI gate for degraded-network behavior. The system under test is a robotics fleet substrate, but the finding applies to anyone shaping networks in CI.

Ganglion exists to reach robots on networks nobody controls. Warehouse Wi-Fi, carrier CGNAT, a hospital VLAN, a customer firewall that was configured once in 2019 and has not been touched since.

Until this week that claim was a sentence on a website. CI ran on clean loopback, everything was green, and the failure modes that actually matter in the field were the exact ones the test suite could never produce.

That is now a required gate. Every push to main runs the full deploy, invoke and verify round trip over the relay against five shaped network profiles, and all five have to pass before anything merges.

I build Ganglion, so treat the enthusiasm accordingly. The part worth your time is not that it went green. It is what I got wrong on the way there.

The five profiles

clean: baseline, no shaping. If this one fails, something else is broken.
lossy: packet loss with light reordering.
high-latency: 250ms round trip.
asymmetric: plentiful downlink, starved uplink. This is the one nobody tests and the one teleop actually dies on, because control acknowledgements go the starved direction.
nat-relay: endpoints with no route to each other at all, forcing hole punching to fail and relay fallback to carry the session.

The last two are the ones I care about. Loss and latency are what people imagine a bad network is. Asymmetry and no-direct-route are what a bad network usually is.

What I got wrong

The original design assumed you can pin netem's seed and get a repeatable lossy run. Two profiles: a pinned-seed one that gates the build, and a nastier randomized one that runs nightly and is allowed to fail.

You cannot pin netem's seed. Its loss and jitter draw from the kernel RNG and there is no seed parameter to set. A "deterministic lossy netem profile" is not a thing that exists.

This matters more than a wrong detail usually would, because the whole design rested on it. A gate that fails randomly is worse than no gate at all. It does not catch regressions, it teaches everyone to re-run the job until it passes, and after a month nobody reads red as meaning anything.

What actually reproduces

The fix was to build the gate out of only the mechanisms that reproduce exactly, and to be honest that this is a smaller set than netem advertises:

Fixed netem delay. Constant delay, zero jitter. Deterministic.
tbf rate caps. Bandwidth ceilings are a token bucket, not a distribution. Deterministic.
iptables -m statistic --mode nth for loss. This drops precisely every Nth packet rather than N percent on average. Every 33rd packet is roughly three percent loss, and critically it is the same three percent every run.
Route blocking for the nat-relay case. Either a route exists or it does not.

Randomized netem still runs. It just runs nightly, in a separate non-blocking job, and a failure opens an issue instead of stopping a merge.

The nightly job generates its parameters deterministically from a recorded seed, so a failing run can be replayed. Worth being precise about what that buys, because it is easy to oversell: replaying reproduces the impairment distribution, not the packet-level draw. You get the same shape of bad network, not the same individual dropped packets. That is documented in the README rather than glossed over, and it is exactly why chaos never blocks a merge.

Riding the harness that already existed

The obvious implementation is a fresh veth pair or a pair of network namespaces built specifically for shaping. I did not do that, and I think the reason generalizes.

There was already an end-to-end dispatch harness running the real deploy, invoke and verify round trip through the relay. Building a second rig would have given me a network test that did not exercise the actual product path, which is the failure mode where CI is green and the thing still breaks.

So the matrix rides the existing harness. Shaping is applied inside the robot and operator containers before the agent starts. netem inside Docker was already proven green in CI by an existing mobile-CGNAT scenario, so the risky part was already de-risked. One rig, one new axis.

Record what you shaped

Every run writes a JSON artifact: mode, seed, the exact shaping commands issued, duration, and result.

This came out of a conversation on ROS Discourse where someone building replay tooling for robot fleets made the point better than I would have. Injected faults that are not recorded produce failures you can see and cannot get back. The fault has to be part of the run's record, or a genuine bug becomes a flaky ghost and gets closed as unreproducible.

Shape at the qdisc, record what you shaped. The two compose.

Keeping CI cheap

A five-profile matrix is an easy way to quadruple your CI bill. What kept it small:

One required-gate job, not five parallel ones. The profiles run sequentially inside it.
One build, reused across all five. Compiling five times to test networking is pure waste.
Main-push only. Not every push to every branch.
Early exit on docs-only pushes.

That lands around 15 to 25 minutes per push. The nightly chaos run is scheduled separately and blocks nothing.

The de-flaking pass, which is the part everyone skips

First full run was not clean, and the failure was not in the shaping. The nat-relay profile had a DNS race: containers with no route between them were resolving each other inconsistently at startup, and the test failed in a way that looked like a shaping bug. Fixed with static compose IPs.

The gate profiles also retry once and attach compose logs on failure. That is a deliberate concession. A gate that is right in principle and flaky in practice gets disabled within two weeks, and then you have neither.

Where it landed

Five of five deterministic profiles passing on main. Clean, lossy, high-latency, asymmetric, nat-relay.

The honest framing is not that Ganglion now works on bad networks. It is that a specific set of bad-network behaviors is now regression-tested, and if I break relay fallback under an asymmetric link, the build tells me instead of a customer telling me.

That is a smaller claim than the marketing sentence. It is also the first version of it I can point at.

Ganglion is Apache-2.0. The harness lives in test-harness/degraded-link/, including the determinism contract and the replay command: https://github.com/RobotDen/ganglion

Originally published at automaton.run. Ganglion is built for ROS 2 robot fleets, but the substrate underneath is transport-agnostic: outbound-only reachability, signed WASM tooling, and default-deny policy work the same for any distributed system on networks you don't control.

Tafy, A Meal Planner and Recipe Generator for Busy Devs!

Bobby Larson — Mon, 15 Apr 2024 06:23:11 +0000

This is a submission for the Cloudflare AI Challenge.

What I Built

Tafy is an intuitive meal planner that takes all of the pain out of planning your meals for the week by generating recipes.

Demo

You can demo the application here --> https://tafy.recipes

My Code

You can peruse at your leisure here: tafy-demo

Journey

This project has been something that I've been stewing on for years (pun intended)! I decided to pursue it as a business, then shortly after ChatGPT came out and I thought I was done for ... but then I realized that it would just make it that much better!

I decided to do this as soon as I heard about the challenge (about 5 days ago!) so I got to work, and decided to beg my family for forgiveness later - sorry, honey!

In my work, I've been doing freelance with LLMs and generative AI and so I've been following the space heavily, and learned recently about CopilotKit in an article on dev.to. So I had to try it!

With that, I had to hack it a bit to get the tooling to work with Cloudflare since a lot of text generation models don't support calling tools. Luckily, @hf/nousresearch/hermes-2-pro-mistral-7b hosted in Cloudflare Workers AI was able to rescue me, but the model did break my search integration.

I also decided to go with the @cf/bytedance/stable-diffusion-xl model for generating images, and so far I've been really impressed!

Finally, I'm using OpenAI's text-to-speech model for verbal descriptions of the meal plans. Though I was hoping to turn this into a means of getting a recipe read off to you while you make it, I actually like the smooth-jazz DJ AI voice-over while I'm looking at the sweet pics of the food that it comes up with!

I'm proud of what I've been able to accomplish what I did in such a short time, but I couldn't have done it without the help of folks in chat - thank you!!!

Check out my README on GitHub for a list of things that I'll be working on next, and maybe someday, if everything works out, I'll make (this website)[https://tafy.io] live!

Multiple Models and/or Triple Task Types

I used multiple models and tasks:

Models

@hf/nousresearch/hermes-2-pro-mistral-7b
@cf/bytedance/stable-diffusion-xl
OpenAI text-to-speech

Tasks

I'm not sure what constitutes as a task, so I'll take a shot here...

[x] The LLM calls tools which equate to "research," each a text generation task in its own right: search, curate, write, critique, revise.
[x] The recipe's summary is sent to OpenAI's text-to-speech API.
[x] Cloudflare Worker - text to image conversion.
[x] Web search (partial, this works with GPT-4, but not Hermes because I can't get it to output JSON).

Things I didn't get to, but may be partially setup:

[] Vectorize store with recipes deposited and used to supplement.
[] Preferences stored on disk, in a DB, or in object storage.

DEV Community: Bobby Larson