Alignment Delay is Everywhere

if you have eyes to see

I've run into this one from time to time throughout my career. You're measuring end to end latency of a system and it's way higher than expected.. and then it's pretty close to what you expected... and then it's somewhere in between...

But there's no load on the system. You're testing locally. You're on a really stable connection. You're the only user. Why is this happening? And then you increase the camera's frame rate and you notice the latencies decrease roughly proportionally to the frame rate's increase.

Oh yeah, the inputs are waiting for the next frame from the camera.

Alignment delay (or sampling latency or quantization delay) happens all the time and is easy to miss when looking at overall system latency. Any time async events get quantized onto a periodic boundary you're likely to encounter this. This could be commands being mapped to frames, or cache TTL boundaries, or video players polling for playlist updates, or Vsync. You will end up with somewhere between 0 and T (where T is the period length) extra latency, and T/2 on average.

Alignment delays can also be additive in a system. If you have multiple parts of the system that quantize to a periodic boundary you can add ∑T_i/2 average latency to the system, if the periodic phases are independent. Let's say we have a simple live video system that adds timed metadata to the headers of video segments that are 2 seconds long, and we have players polling for new segments at that same rate but with their own phase. In this case, excluding other sources of latency, our timed metadata messages will take on average 2 seconds to be received by players with a worst case of 4 seconds. The metadata packet arrives and then must wait up to 2 seconds to be consumed by the segmenter, then the segment that was produced has to wait up to 2 seconds to be downloaded by a player. Sometimes we'll get lucky and latency will be very low, but on average we'll have 2 seconds of additional latency overall, one second for each boundary.

In cases where the periodic boundary is more than one event you can end up with a higher latency floor if the things that bound the period are not in phase. In the chart above we are looking at a real inference system running Fast-WAM. The inputs in this case are two camera frames and a command. The latency is measured from a command being sent to a response being received. On the left you can see that with the cameras aligned we are able to reproduce the full range of possible latencies by delaying commands. On the right our cameras that are running at 10fps are 50ms out of phase. Since a submitted command has to wait for the next two frames the minimum latency added is 50ms while the maximum remains 100ms for commands sent immediately after a frame is captured. In these cases instead of the latency range being [0,T] they are [Δφ, T] where Δφ represents the phase offset.

Detecting alignment delay can be challenging if you haven't instrumented your system with it in mind. You can look at arrival times of events and measure how long it takes for them to move through milestones. For example you can look at the difference between the time processing started on an event and the time it initially arrived. If you're aware of periodicity in your system you can look out for things that are delayed by an average of T/2 (or somewhere nearby). You can also add up the latencies you are measuring directly (e.g. encoder time, packager time, inference time, etc.) and look for delay that is not accounted for in these measurements.

Addressing alignment delay in your system is going to depend a lot on the requirements of the system. In some cases this sort of delay is entirely optional and self imposed. Does the timed metadata packet really need to be inserted at the start of a segment, or can it be inserted at any point in the segment’s lifecycle? The player polling is less optional. You don’t want players spamming your service with many needless requests. You could move the timed metadata to an out of band push channel, but that makes the “timed” part harder. Maybe you add a timestamp that lets players migrate to be more in phase with the system. If you are aligning things at frame boundaries on a server maybe you make it easier for commands to be synchronized with frames at the client.

In any case this is, for people working in domains like video systems, a source of latency that you will encounter from time to time and having it mind when building systems can help you keep unexpected latencies in check.