This field test was against Kubernetes.
The issue was about a large-scale latency and data-size difference between LIST and WatchList:
https://github.com/kubernetes/kubernetes/issues/138670
The issue described a strange behavior at scale.
At around 150K pods, WatchList was sending much more data than LIST for a semantically related operation. The discussion suggested that LIST appeared to be compressed, while WatchList did not appear to use the same compression behavior.
This was not a simple “fix this line” problem.
It was a transport boundary problem.
Kubernetes has two related paths for delivering initial object data:
LIST returns objects through the normal object/list response path.
WatchList delivers initial objects through the watch stream path.
Those paths are semantically related, but Scarab Diagnostic Suite helped identify that they do not share the same visible transport/encoding behavior.
A focused local proof confirmed the boundary.
Using the existing Kubernetes apiserver endpoint test harness:
- LIST with Accept-Encoding: gzip returned Content-Encoding: gzip
- WatchList with the same Accept-Encoding: gzip header returned no Content-Encoding
Same test server.
Same storage.
Same object-size threshold.
Same endpoint family.
The source path matched the result:
LIST goes through the normal object response writer, where gzip negotiation happens.
WatchList goes through the watch stream writer, where events are framed/chunked and the same gzip negotiation path is not used.
I did not propose a fix.
Changing transport behavior for watch streams is design-sensitive. It could affect long-running watches, flushing behavior, CPU/network tradeoffs, proxies, HTTP/2 behavior, and client expectations.
Instead, I left a diagnostic comment for the Kubernetes maintainers with the proof and asked whether a focused test-only contribution would be useful.
Field Test #007
Project: Kubernetes
Issue type: scale / transport behavior
Boundary: object/list response path vs watch stream response path
Proof: matching gzip request conditions produce different response encoding behavior
Outcome: diagnostic proof shared with maintainers
Takeaway: Scarab can identify and prove hidden transport boundaries inside major infrastructure software without jumping straight to repair.
This is the kind of result Scarab is built for.
Not every serious issue starts with a patch.
Sometimes the valuable contribution is identifying the boundary that explains why the system behaves differently than expected.
Top comments (0)