What is an API?
Have you ever wondered how you send a message on WhatsApp, and then someone somewhere in the world received that exact message, and you know someone they communicated via a server of some sort? This was made possible via APIs. APIs serve as an intermediary between two software applications and allow them to share to transfer information.
API stands for Application Programming Interface. Each time you use an app like Facebook, send an instant message or check the weather on your phone, you’re using an API of some sort.
When working on APIs, we are often limited to a small number of requests we can make but then asked to pay some money to get unlimited access. Usually, we try to optimize our project to make fewer API calls, so you don't hit paywalls sooner, and other times, we pull out our cards to pay for the service, but we often don't want to pay and start looking for cheaper (free) alternatives. But there's a trick to bypass API rate limits. This won't work for all cases, but it'll significantly get you if the API provider limits IP addresses and not API keys.
1 - Making use of Proxies
A proxy server acts as a gateway between you and the internet. It’s an intermediary server separating end users from the websites they browse. While using a proxy server, internet traffic flows through the proxy server on its way to the address you requested. The request then comes back through that same proxy server (there are exceptions to this rule), and then the proxy server forwards the data received from the website to you.
What this all means is that, If my IP address is 123.456.789 and I make use of proxy server 234.567.890 to access https://www.example.com, My request will go from my IP address to the proxy server, then from my proxy server to the target domain, then from my target domain back to my proxy server and back to my PC. It's more or less a technique to hide your IP address.
You can use this to bypass APIs that track their rate limits using IP addresses. By sending your requests using different IP addresses, you bump your rate limit to X * Y, where X is the rate limit, andY` is the number of proxies you have. E.g., 300 requests per day having ten proxies become 3000 requests per day. You can get proxies from https://free-proxy-list.net/ http://free-proxy.cz/en/ https://hidemy.name/en/proxy-list/ which you can use in your scripts, e.g., the Python requests module has full support for proxies.
`python
import requests
api_url = "https://example.com"
proxies = {
"http": http_proxy,
"https": https_proxy,
"ftp": ftp_proxy
}
r = requests.get(api_url, proxies=proxies)
`
2 - Creating multiple API keys
This is useful in cases where the API provider uses API key issues by their service to track rate limits. All you have to do to bypass their limitations is create multiple accounts and get numerous API keys. You can also change your IP address after every request in case they track IP addresses too.
3 - Making API requests client-side with JavaScript
This is the most exciting method and can prove quite helpful a lot of times. I'm going to give a personal experience using this method to bypass rate limits.
I was building an application that allowed users to get information from a search box provided. For analytics purposes, I wanted to store the IP address of every user along with their country, region, and a few other information that you can extract from an IP address.
The API I was using to get IP address data was limited at 10K requests a month, and I didn't want to take chances "if users will use this 10K times or not" I decided to:
- make the API request with client-side JavaScript
- converted the response to a string
- added it to the form on the page with an input that had a type of hidden
- then used JavaScript to submit the form
- then used my backend to get this data and convert to JSON then extracted my desired data
This would no longer be My limits are 10K per month but then My limits are X * 10000 limits where X = number of users
javascript
fetch("https://example.com")
.then(data=>{
return data.text()
})
.then(res=>{
hiddenFormField.value = res;
pageForm.submit()
})
Summary
- We learned what APIs are
- We learned three ways to bypass API rate limits
- Proxy servers
- Multiple API keys
- Client-side JavaScript
- We learned what proxies are
I hope you’ve found this post to be helpful. If you know any other methods or want to suggest something I didn't include, feel free to leave a comment! Happy Hacking!
Top comments (3)
Nice ideas 👍 I've been using luminati.io with dedicated DC IPs for proxy. It cost me about $25 one off but never hit any reliability issues with proxies that I get with other solutions out there (namely free).
Thank you for this recommendation, I'll check them out 😊
The third option look like pretty promising and interesting to me.