The Courteous Consumer

As a developer on the DealerOn Reporting Team, I've gone through a lot of the pain points with importing data from other services. We automatically download data daily for thousands of accounts from some of the largest providers. The classic mistake that someone makes when designing a system like this is to build out a Task for each report/account and Task.WhenAll. Eventually you realize that too many requests could flood the provider so you add in a semaphore and set it to something like 20. Done and done, right?

This is dangerous because for the most part it will work. You'll get a lot of data and think you're safe until you need to backfill a single account for more than a few days. You'll watch the logs and after a few days worth you'll notice that your requests time out or have errors that don't make sense. What's wrong?

Most of the large providers implement rate protections on their APIs that will warn a consumer when they're getting close to a limit and outright block them when they cross it. If you don't know how to recognize and heed these warnings, you'll just end up cursing the provider's name in vain. In this article, we'll examine how Facebook, Microsoft, and Google tell you when it's time to slow down so that you can be a courteous consumer.

Facebook

Facebook's Graph API has many different throttling limits based on what you're accessing. Depending on when you're reading this, Facebook is also phasing in a change to their rate limiting called Business Use Case Usage. The basic premise of the old version and the new version is that every response to a call you make will have a header that tells you the current status of the account you are accessing.

x-ad-account-usage: {
  'acc_id_util_pct':9.67     //Percentage of calls made for this ad account.
}

When that percentage hits 100, further calls will result in an error. We decided to pause at 90% because we could have multiple requests running at once. This older version will reset to 0% after 5 minutes. The new response header:

x-business-use-case-usage: {
  '{business-id}':  [{
     'call_count': 100,       //Percentage of calls made for this business ad account.
     'total_cputime': 16,     //Percentage of the total cpu time has been used.
     'total_time': 45,       //Percentage of the total time has been used.
     'type': 'ads_insights',                      //Type of rate limit logic being applied.
     'estimated_time_to_regain_access': 10       //Time in minutes to resume calls.
  }]
}

If any of the first 3 numbers hit 100%, the account is restricted. Facebook API v3.3 will use either response depending on what endpoint you are accessing. Version 4.0 will just be the Business Use Case Usage style.

We check every response and use a Dictionary of SemaphoreSlims so that we can lock individual accounts until they are ready to resume. Any other calls that we build for that account will check _accountLocks and find it already in use. The method below only handles the old style and will need to be updated for the new version (including incorporating the nice estimated_time_to_regain_access field).

public static async Task CheckResponse(this HttpResponseMessage httpResponse, 
           BaseFacebookResponse dataResponse, string url, string accountId)
{
  var accountLimit = JsonConvert.DeserializeObject<FacebookAccountLimit> 
                     (httpResponse.GetHeaderValue("x-ad-account-usage") ?? "");
  var appLimit = JsonConvert.DeserializeObject<FacebookAccountLimit> 
                 (httpResponse.GetHeaderValue("x-app-usage") ?? "");

  if (appLimit != null && appLimit.Usage > MAX_USAGE_PERCENT)
  {
    await _appLock.WaitAsync();
    await Task.Delay(APP_SLEEP_MINUTES);
    _appLock.Release();
  }

  if (accountLimit != null 
   && accountLimit.Usage > WARNING_USAGE_PERCENT 
   && accountLimit.Usage <= MAX_USAGE_PERCENT)
  {
    await Task.Delay(ACCOUNT_SLOWDOWN);
  }

  if (accountLimit != null 
   && accountLimit.Usage > MAX_USAGE_PERCENT)
  {
    await _accountLocks[accountId].WaitAsync();
    await Task.Delay(ACCOUNT_SLEEP_MINUTES);
    _accountLocks[accountId].Release();
  }

  if (dataResponse.Error != null)
  {
    throw new Exception($@"Facebook API Error: {dataResponse.Error.Message}
                           URL: {url}
                           Full Error: 
                           {JsonConvert.SerializeObject(dataResponse.Error)}");
  }
}

Google Adwords

Adwords' rate limiting is based on watching for the RateExceededError in the response.

We're using the googleads-dotnet-lib nuget package which has an example in the source on how to handle RateExceededError:

try
{
  //make your request
}
catch (AdWordsApiException e)
{
  // Handle API errors.
  var innerException = e.ApiException as ApiException;
  if (innerException == null)
  {
    throw new Exception(
      "Failed to retrieve ApiError. See inner exception for more details.", e);
  }

  foreach (ApiError apiError in innerException.errors)
  {
    if (!(apiError is RateExceededError))
    {
      // Rethrow any errors other than RateExceededError.
      throw;
    }

    // Handle rate exceeded errors.
    var rateExceededError = (RateExceededError)apiError;
    //lock out other requests for this account
    await _accountLocks[accountId].WaitAsync();
    await Task.Delay(rateExceededError.retryAfterSeconds * 1000);
    _accountLocks[accountId].Release();
    //we should retry this request now
  }
}

On the other hand, the best offense is a good defense and we typically avoid the RateExceededError by using a semaphore to limit ourselves to 10 open Adwords report calls at any one time.

Google Analytics

This product is made by the same company as Adwords so the APIs must be similar, right? Once we're all done laughing, take a look at the documentation.

Each account has a hard daily limit as well as a real time Queries Per Second limit. You can detect the daily or QPS limit getting exceeded by the status code 403 or 429 but even better would be limiting your requests so that you don't hit it.

public async Task<T> MakeRateLimitedRequest<T>(Func<T> action)
{
  await _semaphore.WaitAsync();
  Task.Delay(_timeSpan).ContinueWith(_ => _semaphore.Release());
  return action();
}

If your semaphore has 10 slots and the timespan is 1 second, you shouldn't be able to make more than 10 requests per second. If you're handling multiple accounts, you'll want a dictionary of semaphores so that each can run at it's maximum allotment.

Bing Ads

Microsoft has probably the simplest method to detect and handle throttled requests:

To ensure resources for everyone, the API limits the number of requests a customer ID may make per minute. The limit is not documented and is subject to change. If you exceed the request per minute limit, the API returns HTTP status code 429. When you receive status code 429, you must wait 60 seconds before resubmitting the request.

A modified version of our Facebook method will handle this easily enough.

Conclusion

Something that might have come through in each section is that the documentation usually will explain how the provider will limit access to their servers. This is necessary to prevent a malicious user or someone that accidentally wrote an infinite loop from crashing everything. No matter what third party data you are accessing, make sure you fully test your code by running it until your rate limiters kick in so you can see them work. By following some of the suggestions here you'll soon learn to implement rate limiting before you notice a problem in production.