I'm a developer and happiest when I have code in front of me 👨💻. Databricks Tech Champion. Long term sufferer of imposter syndrome. Always looking to learn something new.
A bit of Python which asynchronously downloads data from an API. But which also uses queues and worker process functions to throttle the query rate so as to not overload their systems (and so I don’t end up with a load of 429 responses). At the end it parses all the XML response data into csv. Used it to download 2.5 years worth of data (can only query one day at a time) to unblock a colleague.
I’ve used the aiohttp library before a lot, but this is the first time I’ve been happy with the throttling process as well.
I'm a developer and happiest when I have code in front of me 👨💻. Databricks Tech Champion. Long term sufferer of imposter syndrome. Always looking to learn something new.
A bit of Python which asynchronously downloads data from an API. But which also uses queues and worker process functions to throttle the query rate so as to not overload their systems (and so I don’t end up with a load of 429 responses). At the end it parses all the XML response data into csv. Used it to download 2.5 years worth of data (can only query one day at a time) to unblock a colleague.
I’ve used the aiohttp library before a lot, but this is the first time I’ve been happy with the throttling process as well.
All your data are belong to us ;-) Congratulations!
Oh, and it’s the first time I’ve blogged in a few years, so happy to be trying to get back into writing more content.
aizoo.info/post/bringing-azure-dat...