DEV Community

Discussion on: I wrote a crawler for the first time.

functional_js profile image
Functional Javascript

You could do that in certain cases. But there are only 4 of them. Plus by doing it serially to the same domain, you're not overwhelming the one server endpoint in this case.

Actually, as a robustness strategy, with multiple calls to the same domain, you typically want to insert manual delays to prevent a server rejections. I'll actually place a "sleep" between calls. It's not a race to get the calls done as fast as possible. The goal is robustness.

Also, this is a background job, so saving a second or two is not a critical performance criteria here. In this case one request isn't dependent on another so it's fine, but if it was you'll want to run them serially.

Also with Promise.all, if one fails, they all fail. It's less robust. With the serial approach each request is atomic and can succeed on it's own. I.E. Getting 3 out of 4 successful results is better than getting 0 out of 4 successes, even though some of them succeeded.

Also you now have an array of resolved promises that you still have to loop through and process. With the serial approach, it was done in 4 lines. Much easier to grok. Much easier to read. Much easier to debug.

If I had to do dozens of requests that had no order dependency on each other, and they all went to various domains, and 100% of them had to succeed otherwise all fail, then Promise.all is certainly the way to go. If you have 2 or 3 or 4 requests, there's really no compelling benefit. Default to simple.

So there are pros and cons to each approach to consider.
Thanks for the input!

Thread Thread
hasnaindev profile image
Muhammad Hasnain

Ahan, thank you for presenting the case in such detail. These things never occurred to me and now I know better. :)

Thread Thread
robloche profile image

The point about robustness is valid but for the sake of it, I'll mention that the potential issue with Promise.all can be avoided by using Promise.allSettled instead.

Some comments have been hidden by the post's author - find out more