Using Little's Law for Cloud Infrastructure Planning

I thought I'd share a bit about Little's Law and how it can be useful for infrastructure planning purposes.

Little's Law is a mathematical theorem from the science of queuing theory. It's very simple but very powerful for modeling everything from airport security lines to bus routes. Before showing an example of how to use it for cloud resource planning, let me explain the law itself:

Little's Law states that the (average) number of units in a process at a given time (L) is equal to the (average) arrival rate (λ) times the average amount of time each unit spends in the process (W).

Thus:
L = λ * W

Using an example from AWS Lambda, it may be used to estimate how many concurrent lambda functions might be required to support a certain process if we can estimate the request rate and the average duration of each invocation. Since the duration of each lambda invocation is a metric available in the AWS console, you need only estimate the number of requests per second to derive the number of concurrent lambda functions you should expect, on average:

request rate (λ): 100 (requests per second)
duration per invocation (W): 1 (second)
number of units in process at a time (L): L = λ * W

so:
100 requests/sec x 1 sec = 100 requests in progress at any given time

Thus we can determine that a process with these assumptions will require an average of 100 concurrent invocations of the lambda function at any given time. This process will not likely ever hit AWS's default soft limit of 1000 concurrent instances.

If the process expected 600 requests per second and each request had an average duration of 2 seconds, then we would be worried about the limit of 1000 concurrent instances, because we would expect 600 x 2 = 1200 requests to be in-process at any given time.

Thus, even though lambda is a service that scales automatically, there is a concurrency limit, and we ought to proactively check whether there is a risk of hitting it. Little's Law is a quick way to check.

Note that the formula can be manipulated to solve for other variables too: