What a real user's USB capture found that my synthetic tests didn't: two bugs in usb-rea

For weeks every test of usb-rea ran on captures I generated myself, on protocols documented by liquidctl and rivalcfg, and on a few public Wireshark samples. All of that passed. Then a real user shared a real capture, and within a day it had found two bugs that none of my tests could have found.

This is the fifth post about usb-rea, an open-source tool I built that diffs USB captures per labelled click in a vendor app and writes a pyusb driver. The first one covered commands, the second one naming what the device reports back, the third one handshakes and session tokens, the fourth one video streams to case LCDs. This one is about what happened when the tool met hardware I don't own.

The capture

On liquidctl issue 936 the owner of an ASUS TUF Gaming LC III AIO (the one with an LCD on the pump) shared a USBPcap capture from Windows while changing the screen brightness in the vendor app. The full capture is big: about 76,000 packets, nearly all of them a 320x320 JPEG video the app streams to the screen at around 70 frames per second, plus four tiny brightness writes hidden in between.

That capture is what pushed me to add image stream detection in v0.7.4 (the fourth post). After that, the full capture gave the right answer: brightness is byte 4 of a 440-byte message on interrupt EP 0x01, the video goes to EP 0x03 with a 08 nn 00 80 header, and the device acks each frame with 08 81 on EP 0x84. That matched what the owner had worked out by hand.

Then I asked if I could keep a trimmed copy as a regression test. They said yes, and turning it into a test is where the two bugs showed up.

Making a fixture you can publish

A raw capture is not something to commit to a public repo. It has other devices on the same bus, and the video frames are literally whatever was on the screen. So the stored copy:

• keeps only this device's packets: the setup, the first 8 frames and two frames on each side of each brightness change (589 packets, 6.5 KB gzipped)
• zeroes the image data in every frame, keeping only the JPEG headers and the end marker, so nothing that was on the screen is in the file
• shifts the timestamps to a fixed date and replaces the IRP ids with counters

The test in tests/test_user_capture_lc3.py then checks the VID:PID, the brightness command and the stream end to end. The first run failed twice.

Bug 1: USBPcap puts the control reply in the "complete" record

usb-rea finds the vendor and product id from the GET_DESCRIPTOR reply at the start of a capture. On this file it said there wasn't one. Here is what v0.7.5 printed at the top of the report:

Device: `unknown (no GET_DESCRIPTOR in capture)`

But the reply is right there. This is the real output of usb-rea transfers on the fixture (first two lines):

     1 dev   2 ep 0x80 control   IN  8006000100001200 |
     2 dev   2 ep 0x80 control   IN  12 01 10 01 00 00 00 40 05 0b 7b 1c 51 00 01 02 03 01

USBPcap logs a control transfer in stages. Frame 1 is the setup stage (80 06 00 01 ..., GET_DESCRIPTOR for the device descriptor). The descriptor itself (12 01 ..., with 05 0b 7b 1c being 0B05:1C7B little-endian) arrives in a separate record for the "complete" stage, stage 3. My parser only took control IN data from the stage it expected, so on captures where the data comes in stage 3 the descriptor was simply dropped. My own synthetic USBPcap files never did that, which is exactly why the tests never caught it.

v0.7.6 reads control IN data from the complete record too. Same file, new report:

Device: `0b05:1c7b`

Bug 2: counting the ack over the whole capture

The second failure was quieter. v0.7.5 found the video stream and the brightness command, but no frame ack. Worse, the 08 81 acks didn't just disappear: they ended up in the report as a "reply to brightness", because a couple of them happened to arrive a few milliseconds after a brightness write.

The old rule was: a fixed device-to-host message is the ack if it shows up about once per frame over the whole capture (between 0.5x and 1.2x the number of frames). On the full capture that held. On the trimmed one it didn't: the fixture keeps the acks for frames I cut out, so it has 66 acks for 24 frames. That's an artefact of trimming, but the same thing happens on a real capture whenever the app keeps polling while it pauses the video, and a global ratio can't tell the difference.

v0.7.6 counts the ack per frame instead: a fixed device-to-host message between one frame's start and the next (and no more than 1 s after the frame's last packet). It's the ack when at least half of the frames have one and most of those have exactly one. The new frames_acked field says how many frames it was seen for. The report now has this line in the stream section (real output on the fixture):

- Ack: `08 81 00 00 00 00 00 00 00 00 00 00 00 00 00 00` on EP 0x84, 66x (frames 30, 52, 94, 116, 138)

And because the acks are set aside with the stream, the bogus "Reply to brightness" section is gone. Summarised from the usb-rea infer JSON of both versions on the fixture, same labels:

0.7.5  vid/pid None None   ack None                     responses 1
0.7.6  vid/pid 2821 7291   ack EP 0x84, frames_acked 18 responses 0

(2821 and 7291 are 0x0B05 and 0x1C7B.)

What the trimmed copy can't tell you

Some numbers in the fixture report are about the trimming, not the device, and I'd rather say so:

• The frame rate shows as about 0.2 frames per second, because most frames were cut out. The full capture is around 70.
• The brightness command comes out at 60% confidence from two clean samples (brightness 0 and brightness 50); the third action is ignored because it sent a differently shaped transaction. On the full capture it's 100%.
• The image content is zeros, so the fixture can't test anything that depends on the pixels.

What it does test is everything the bugs were about: the device descriptor in a USBPcap complete record, the stream headers, the per-frame ack, and the brightness byte on its own endpoint.

Try it

pip install -U usb-rea          # 0.7.6
usb-rea demo --device lcd

To run it on the same fixture from a clone of the repo:

gunzip -c tests/data/asus-tuf-lc3-lcd.pcap.gz > lc3.pcap
usb-rea report lc3.pcap -l "brightness 0;brightness 50;brightness 100" -o lc3.md

The lesson for me: synthetic captures test the inference, real captures test the assumptions about how capture tools write files. If you have a USB device whose app only runs on Windows, one USBPcap capture while you change one setting is the most useful thing you can send. Device requests are open: https://github.com/emirmrkaljevic/usb-rea

Thanks to the owner of the LC III who shared the capture and agreed to the trimmed copy being kept as a test.

I built usb-rea with help from AI coding tools; the inference itself is deterministic and every claim in its reports points at captured frames.