Patiently - an API rate limit handler for (Node) JS. A legal way to wait for API rate limit resets.

Hi, I am Daniel and I want to present to you one of my little minions I developed within the context of a side project of mine, the Quip Exporter. The Quip Exporter is a bulk exporter for the popular web-based editor Quip.

See also my other minion called "logmailer". You can find more info about logmailer here and here.

I would be so happy for a Github star on my repo. 😃 😍 I am looking very forward for discussions and comments. Thank you!

Short description

Sometimes there is no other way to be patient and to wait for the API rate limit to reset. Just think of a personalized API that allows you to perform actions in a specific user context. You want to do like 10.000 reqs as fast as possible. What if there are rate limits like 50 reqs per minute and 750 reqs per hour? And what if those limits are not bound to an IP or a host but to your user? In this case there is no other way but to wait for a limit reset. Of course, you can also politely ask the API owner to increase the limits. But even then you need patiently. :)

Where to find

patiently @ github

patiently @ npm

The "Why" and the "How"

I developed patiently for another tool which allows a bulk export of all documents including all images from a Quip account. Quip is a web based collaborative document and spreadsheet editor. I used Quip mainly for taking notes until I eventually switched to a Markdown based note taking system. You can find the exporter on my github page mindactuate.github.io/quip-exporter.

The challenge I had while developing Quip Exporter was the rate limit bound to a personal API token. Even if I switch IPs regularly (think of a proxy scraper), there is no chance to get around the API rate limits. That's where patiently comes in.

Patiently implements a "waiter function" where you can wrap in another function like your API call.

• Each function you give to the waiter as a parameter is queued in an array (queue)

  
  F -> E D C B A
  

• The longest waiting function is first

  
  F E D C B -> A
  

• Before the function is called, it is checked whether we have to wait first or not

  
  F E D C B -> A (wait?)
  

• If yes the queue processing is paused and when the waiting time elapsed, function A will be called

  
  F E D C B -> A (call)
  

Installing

Using npm:

$ npm install patiently

How to use

Use by setting limits manually

Perhaps you already know about the limits (maybe from the API docs).

     import patiently from "patiently";
     // let patiently = require("patiently");

     let options = {
       startWaitingCallback: function(info){console.log(info)}, // default is function(){}, calls a function if waiting necessary
       endWaitingCallback: function(info){console.log(info)}, // default is function(){}, calls a function after waiting
       waitingTickCallback: function(info){console.log(info)}, // default is function(){}, calls a function every tick
       msBetweenTwoCalls: 1000, // default is 0 milliseconds (no waiting time between two calls)
       minutelyLimit: 50, // default is Infinity (no minutely limit set)
       hourlyLimit: 750, // default is Infinity (no hourly limit set)
       test: false // default is false (if true, max waiting time is 5 secs)
     }

     var waiter = new patiently.LimitWaiter(options);

     let myApiCallFunction = async (url, callback) => {
         waiter.wait(function(){
           // your api call
           axios.get(url)
             .then(res => {
                callback(res.data);
             })
             .error(err => {
                callback(null, err);
             })
         })
     }

     // you can call myApiCallFunction as often you want
     // patiently can handle asynchronous api calls :)
     let url = "https://www.npmjs.com/package/patiently";
     myApiCallFunction(url);
     myApiCallFunction(url);
     myApiCallFunction(url);
     myApiCallFunction(url);
     // ...

License

MIT