Telling a video stream to a case LCD apart from the commands: JPEG and raw RGB565 in USB captures

Capture the USB traffic of an AIO cooler with a screen on the pump cap, change the brightness four times in the vendor app, and diff the clicks. What you expect is one small command with one byte that follows the slider. What you get is tens of thousands of 1 KB packets, because the app never stops sending the picture to the screen while you click.

This is the fourth post about usb-rea, an open-source tool I built that diffs USB captures per labelled click and writes a pyusb driver. The first one covered commands, the second one naming what the device reports back, the third one handshakes and session tokens. This one is about screens.

The weakness a real capture showed

A user on liquidctl issue 936 shared a public USBPcap capture of an ASUS TUF LC III LCD while changing the brightness. It was the first real capture of that kind I had run usb-rea on, and the old version got it wrong in an instructive way: the video packets landed inside the brightness actions, and the "brightness" usb-rea reported was made of image bytes.

The diff wasn't broken. It did exactly what it was told: find the bytes that change with the label. In a capture full of video, video bytes change with everything.

So since v0.7.4 usb-rea sets image streams aside before anything else runs. On that capture the stream turned out to be a 320x320 JPEG video on its own endpoint, and once it was out of the way the brightness command was easy: byte 4 on EP 0x01 is the percentage.

Encoded images: look for the magic bytes

JPEG, PNG and GIF are the easy case because every image starts with a known marker (FF D8 FF for JPEG). The rule:

• On one host-to-device channel, at least 3 packets start an image at the same offset. That offset is the per-packet header length.
• A frame runs from its start packet to the image's end marker (JPEG FF D9, counting embedded thumbnails so an EXIF preview doesn't end the frame early), the next start, or a pause.
• Each header byte is constant, nn (number of packets in the frame) or ii (index of the packet in the frame).
• A fixed device-to-host message sent about once per frame is the ack.

Here is the synthetic LCD from the demo, which streams a 240x240 JPEG video next to its brightness command. This is the real output of usb-rea demo --device lcd, with the 64-byte brightness line trimmed:

captured 5 labelled actions, device 1234:abcd

brightness   100%  0x12 | 0x01 | 0x00 | 0x80 | round(value * 1 + 0) | 0x00 | 0x00 | ...

image stream, kept out of the commands: jpeg stream on interrupt EP 0x03: 240x240, 159 frames in 950 packets of 512 bytes, header `08 nn 00 80` then `08 ii 00 00`; ack 08 81 00 00 00 00 00 00 00 00 00 00 00 00 00 00 on EP 0x84 (driver: send_image())

and the matching part of report.md:

- Frames: 159 (3 to 8 packets each, 950 packets in all, 2 of them from a frame cut off by the capture)
- Image size: 240x240, 1507 to 3985 bytes
- Rate: about 20 frames per second
- Chunk: 512 bytes per packet, the last one padded
- Header (4 bytes): first packet `08 nn 00 80`, then `08 ii 00 00`
- Ack: `08 81 00 00 ...` on EP 0x84, 160x (frames 8, 18, 28, 42, 54)

The image size comes from the JPEG SOF marker, so you also learn the panel resolution for free.

Raw pixels: no magic, so look at the size

Plenty of cheap case screens don't take JPEG at all. They take plain pixels: RGB565 (2 bytes per pixel) or RGB888 (3 bytes), usually behind a short "draw" command. Nothing in those bytes says "image". This is now on main (not released yet), and the trick is arithmetic:

• Group packets on one channel into runs: large packets with no pause between them, all but the last the same length.
• A run is a frame when its payload is exactly width x height x bytes per pixel for a common screen size (80x160 up to 1920x480, both orientations).
• Or, better, when the short packet sent just before the run names a region and the payload is exactly that region. usb-rea tries x, y, width, height and x0, y0, x1, y1 as 16-bit fields in either byte order, plus four 10-bit fields packed into 5 bytes.
• Three or more frames make a stream. The draw commands go with the pixels; other short commands on the same endpoint stay commands.

The region version matters because these screens rarely redraw everything. A clock or a temperature readout is a small rectangle sent every second, and without the region those frames would look like random large commands.

Output of usb-rea demo --device lcd --raw rgb565, where the synthetic screen is 160x128 and every 4th frame only redraws a 48x16 clock:

image stream, kept out of the commands: raw rgb565 little-endian stream on bulk EP 0x02: 160x128, 79 frames (20 redraw part of the screen) in 4937 packets of 512 bytes, each frame after a command `55 aa 02 00 xx xx yy yy ww ww hh hh`; ack 55 aa 82 00 on EP 0x82 (driver: send_image())

and from the report:

- Screen: 160x128 RGB565, 40960 bytes (2 per pixel) per full frame
- Pixel byte order: little-endian (neighbouring pixels are smooth that way)
- Region: the command gives x, y, width, height as 16-bit little-endian fields from byte 4; 20 of 79 frames redraw only part of the screen

The byte order line is my favourite small heuristic in this release. RGB565 can be sent little or big endian, and the capture doesn't say which. But real pictures are smooth: neighbouring pixels have similar colours. So usb-rea decodes a frame both ways and keeps the order where neighbours differ less. On a flat colour it says it can't tell instead of guessing.

As a ground truth I wrote a test fixture from the public description of the Turing Smart Screen rev A protocol: 6-byte commands with x, y, end x and end y packed as 10-bit fields, DISPLAY_BITMAP = 197, then RGB565 little-endian pixels. usb-rea finds the 320x480 stream, the draw command rr rr rr rr rr c5 (rr = packed region), the partial redraws, and leaves the hello, orientation and brightness commands on the same endpoint alone.

16-bit header fields

Raw frames are big. A 128x160 RGB888 frame in 64-byte packets is 1024 packets, which doesn't fit a one-byte counter, so devices use 16-bit ones. The header description now reads a count or index above 255 as a u16 and shows it doubled, with its byte order: 00 00 nn nn, then ii ii nn nn. The driver fills both bytes in.

The driver side

The generated driver gets send_image(). For a JPEG stream you pass JPEG bytes of the captured size. For a raw stream you pass a Pillow image (converted to RGB565 in the captured byte order) or bytes already in the stream's format, and if the app used region commands you can redraw just part of the screen:

from PIL import Image
from lcd_driver import Device

d = Device()
d.send_image(Image.open("wallpaper.png").resize((160, 128)))
d.send_image(clock_pixels, x=100, y=8, size=(48, 16))   # just the clock

Pillow is optional; the driver doesn't import it.

What it can't do (yet)

• A screen size that isn't in the list and has no region command is missed.
• The same byte count can fit several shapes: 320x480 RGB565 is also 480x320, and 320x320 RGB888. The report lists the alternatives so you can check on the device.
• Region fields are only looked for in the separate draw command, and width is expected before height.
• RGB888 channel order (RGB or BGR) is not worked out, and streams split over several endpoints are not detected.
• A frame with a pause longer than 30 ms inside it gets split.

Try it

pip install -U usb-rea          # 0.7.5
usb-rea demo --device lcd
usb-rea demo --device lcd --raw rgb565

It's also an MCP server (usb-rea mcp), and the streams field in its output carries all of the above. Sending anything to real hardware still needs explicit confirmation from a human.

If you own a case or AIO screen whose app only runs on Windows, a capture while you change one setting would help a lot, especially raw-pixel ones. Issues are open: https://github.com/emirmrkaljevic/usb-rea

I built usb-rea with help from AI coding tools; the inference itself is deterministic and every claim in its reports points at captured frames.