Low Latency and “Fast Enough”

What is low latency?
Why is it different from domain to domain?
And what is “fast enough”?

Everybody wants their application to have low latency. This is obvious. The real question is: how low?

Latency is the time between an event and the result we care about. For a user-facing application, it might be the time between a click and a visible response. In a data-processing system, it might be the time between receiving an item and producing a result. In an electronic trading system, it might be the time between receiving market data and sending an order.

The goal is not always to achieve the lowest theoretically possible latency. The goal is to be fast enough for the business case.

To explore this topic, let's divide latency-sensitive flows into three groups. This is a simplified division intended to make the contrast clearer.

1. User experience

If we care about user experience, 100–200 ms is usually acceptable for an ordinary interaction. It is better to be in the 50–100 ms range, though. Once the complete interaction is below roughly 50 ms, reducing it further will usually not make a meaningful difference to most users.

Here we are talking about the time between the user's action and the visible result. It usually includes:

• frontend processing;
• network time between the frontend and backend;
• backend processing;
• network time between the backend and its data storage;
• processing inside the data storage;
• and the same network path in the reverse direction.

This is the simplest flow. In reality the backend might make requests to other backend services, which introduces more processing and network steps.

Just as a reminder:

1 second = 1,000 milliseconds (ms)
1 ms     = 1,000 microseconds (µs)
1 µs     = 1,000 nanoseconds (ns)

Therefore:

1 second = 1,000,000 µs
1 second = 1,000,000,000 ns

For this kind of system, being fast enough usually means that users do not feel that the application is making them wait. Optimizing a 150 ms interaction may improve the experience. Optimizing a 20 ms interaction to 10 ms probably will not change the business result.

2. Backend processes

There are also cases where user experience is not the direct goal. We may have a large distributed system with different kinds of communication, data storage, messaging, synchronous and asynchronous processing, scheduling, batching, and so on.

Here we usually look first at whether the system can continuously keep up with its workload.

For example, imagine a permanent data stream flowing into the system. Our total sustained processing capacity must be higher than the sustained incoming rate. To simplify: this means processing each item more quickly than the interval between incoming items.

Otherwise, we are in trouble.

Of course, we may have a buffer for temporary out-of-control situations. But no buffer can save a system whose processing capacity is constantly below the throughput of the incoming stream. The buffer can only postpone the problem. Eventually it fills and we may end up with system failure and data loss.

Here being fast enough means processing the expected workload continuously without creating a permanently growing queue.

3. Competing systems

The third case is different. Your system competes with other systems, and the business result depends directly on its performance.

Here latency has to be as low as practically possible. You may care about nanoseconds in certain parts of the system, microseconds in others and every avoidable millisecond across the complete flow.

You are fast enough if you are the fastest or at least one of the fastest systems competing for the same opportunity.

This is mostly relevant to certain areas of FinTech, such as latency-sensitive electronic trading and market making. If another system reacts to the same market event before yours, it may take the opportunity before your order arrives. In this case lower latency does not merely make the system feel better. It can directly change the business result.

So, what is low latency?

Low latency is not a specific number.

For one system, 100 ms may be excellent. For another, 5 ms may be dangerously slow. A third system may need removing 100 ns from a critical operation.

The domain changes the meaning of being late:

• in a user-facing system, you are late if the user feels the delay
• in a backend process, you are late if the system cannot keep up with the workload
• in a competing system, you are late if someone else reaches the opportunity first

That is why the real goal is not simply to have “low latency”. The goal is to understand the deadline created by the business case and to be "fast enough" for it.

This is the relationship between low latency and being fast enough.

There is one more important point: a real system does not complete every operation in exactly the same amount of time. Therefore latency is not only a number. It is a distribution. That deserves a separate discussion.