DEV Community

Discussion on: Ingest Millions of Messages in the Queue using Serverless

nachiketdhamankar profile image
Nachiket Dhamankar

Hi Hassan,
Great read. Using a blob store and using functions is a good idea. I was wondering how big the response would be from the external API? Would it be paginated? If the response is large, would a single function be able to chunk and write to a blob store?

mrhassanmurtaza profile image
Hassan Murtaza Author

Hi Nachiket,
Thank you for the feedback.

In my case, the response was just the ids so it wasn't big and it had no pagination. But even if we have pagination, we'll need to use something to hold the current state of invocations so that we can scale based on the number of pages/per invocations, etc.

Also, if the response is too large to be processed by a single function, we can for sure break it further as needed since the strength of functions lies in multiple small invocations :).