DEV Community

Cover image for Javascript fetch, retry upon failure.
YCM Jason
YCM Jason

Posted on

Javascript fetch, retry upon failure.

#javascript #fetch #recursion #node

This post is republished since the previous one was not showing up on my profile for some reason.

So recently, I bumped into a situation where the network works/fails randomly. As it affects the consistency of my test results, I decided to implement a fetch_retry function which retries fetch upon failure up to n times.

Introduction

Fetch in Javascript is great. I hope you will agree that it provides a simple yet robust enough interface to do our AJAX requests.

However, network doesn't always work like we want it to, and it might fail randomly. To catch this issue, let's implement a function fetch_retry(url, options, n) which does fetch(url, options) but retries up to n times upon failure. And hence increasing the chance of success.

Let's think

Retrying things sounds like a loop. Why don't we write a for/while loop to do that? Something like the following, perhaps?

function fetch_retry(url, options, n) {
    for(let i = 0; i < n; i++){
        fetch(url, options);
        if(succeed) return result;
    }
}
Enter fullscreen mode Exit fullscreen mode

NO! Fetch is an asynchronous function, meaning that the program would not wait for result before continuing! n fetches will be called at the same time (kind of), regardless of whether the previous calls succeed!

This is not what we want. This is not retry upon failure, this is fetching n times simultaneously! (That being said, if written correctly, it could also increase the chance of success. Perhaps with something like Promsie.any? Although I am not a big fan of bluebird. I think native Promise is good enough.)

If you don't know about asynchronous functions and Promise in Javascript, watch this amazing video here, made by Jessica Kerr, before reading on!

Briefly about fetch

So fetch returns a Promise. We usually call it like this.

fetch(url, { method: 'GET' }).then(res => console.log('done'));
console.log('fetching...');
Enter fullscreen mode Exit fullscreen mode

If you understand Promise correctly, you should expect the result to be:

fetching...
done
Enter fullscreen mode Exit fullscreen mode

And if the network fails for some reason, the Promise rejects and we could catch the error as follows:

fetch(url, { method: 'GET' }).catch(err => /* ... */);
Enter fullscreen mode Exit fullscreen mode

So how to implement?

What does fetch_retry do?

We start off with thinking what do we want the function fetch_retry do. We know it has to call fetch somehow, so let's write that down.

function fetch_retry(url, options, n) {
    fetch(url, options)
        .then(function(result) {
            /* on success */
        }).catch(function(error) {
            /* on failure */
        })
}
Enter fullscreen mode Exit fullscreen mode

Now obviously fetch_retry has to be an asynchronous function, since we can't really define a synchronous function out of an asynchronous one. (or could we? Enlighten me.)

Definition: So this means fetch_retry should return a Promise that resolves if any attempt out of n attempts succeed, and rejects if all n attempts failed.

So let's return a Promise now.

function fetch_retry(url, options, n) {
    return new Promise(function(resolve, reject) { // <--- we know it is asynchronous, so just return a promise first!
        fetch(url, options)
            .then(function(result) {
                /* on success */
            }).catch(function(error) {
                /* on failure */
            })
    });
}
Enter fullscreen mode Exit fullscreen mode

What if fetch succeeds?

So if the fetch succeed, we obviously can just resolve the promise we are returning, by calling the resolve function. So the code become:

function fetch_retry(url, options, n) {
    return new Promise(function(resolve, reject) {
        fetch(url, options)
            .then(function(result) {
                /* on success */
                resolve(result); // <--- yeah! we are done!
            }).catch(function(error) {
                /* on failure */
            })
    });
}
Enter fullscreen mode Exit fullscreen mode

What if fetch fails?

What should we do on failure? Doing for/while loop here wouldn't really help, due to the asynchronous property we discussed previously. But there is one thing that we could do what for/while loop does. Does it ring a bell? Yes! Recursion!

My two rules of thumb when doing recursion:

  1. Do not think recursively. Don't try to follow your code recursively.
  2. Leap of faith, assume the recursive function you are defining works.

These two points are fundamentally the same! If you have the leap of faith, you wouldn't be thinking recursively into the code.

Ok, so let's try to take the leap of faith and assume fetch_retry will just work, magically.

If it works, then in on failure, what will happen if we call fetch_retry(url, options, n - 1)?

function fetch_retry(url, options, n) {
    return new Promise(function(resolve, reject) {
        fetch(url, options)
            .then(function(result) {
                /* on success */
                resolve(result);
            })
            .catch(function(error) {
                /* on failure */
                fetch_retry(url, options, n - 1) // <--- leap of faith, this will just work magically! Don't worry!
                    .then(/* one of the remaining (n - 1) fetch succeed */)
                    .catch(/* remaining (n - 1) fetch failed */);
            })
    });
}
Enter fullscreen mode Exit fullscreen mode

fetch_retry(url, options, n - 1) will just work magically by the leap of faith and would return a Promise which, by the definition we discussed previously, resolves if any attempt (out of n - 1 attempts) succeed, and rejects if all n - 1 attempts failed.

So now, what do we do after the recursive call? Notice that since fetch_retry(url, options, n - 1) would work magically, this means we have done all n fetching at this point. In the on failure case, simply resolves if fetch_retry(url, options, n - 1) resolves, and rejects if it rejects.

function fetch_retry(url, options, n) {
    return new Promise(function(resolve, reject) {
        fetch(url, options)
            .then(function(result) {
                /* on success */
                resolve(result);
            })
            .catch(function(error) {
                fetch_retry(url, options, n - 1)
                    .then(resolve)  // <--- simply resolve
                    .catch(reject); // <--- simply reject
            })
    });
}
Enter fullscreen mode Exit fullscreen mode

Great! We are almost there! We know we need a base case for this recursive call. When thinking about base case, we look at the function arguments, and decide at what situation we could tell the result immediately.

The answer is when n === 1 and the fetch fails. In this case, we could simply reject with the error from fetch, without calling fetch_retry recursively.

function fetch_retry(url, options, n) {
    return new Promise(function(resolve, reject) {
        fetch(url, options)
            .then(function(result) {
                /* on success */
                resolve(result);
            })
            .catch(function(error) {
                if (n === 1) return reject(error); // <--- base case!
                fetch_retry(url, options, n - 1)
                    .then(resolve)
                    .catch(reject);
            })
    });
}
Enter fullscreen mode Exit fullscreen mode

Clean things up

Redundant function

In our "on success" function, we are simply calling resolve(result). So this function instance is redundant, we could simply use resolve as the "on success" function. So the code would become:

function fetch_retry(url, options, n) {
    return new Promise(function(resolve, reject) {
        fetch(url, options).then(resolve) // <--- Much cleaner!
            .catch(function(error) {
                if (n === 1) return reject(error);
                fetch_retry(url, options, n - 1)
                    .then(resolve)
                    .catch(reject);
            })
    });
}
Enter fullscreen mode Exit fullscreen mode

Redundant promise

Now another stupid thing we are doing here is this line:

fetch_retry(url, options, n - 1).then(resolve).catch(reject)
Enter fullscreen mode Exit fullscreen mode

Do you see what is the problem?

Let me put this in context, we are essentially doing this:

new Promise(function(resolve, reject) {
    fetch_retry(url, options, n - 1).then(resolve).catch(reject)
});
Enter fullscreen mode Exit fullscreen mode

So this new promise is redundant in this case, because it is resolving if fetch_retry resolves, and rejecting if fetch_retry rejects. So basically it behaves exactly the same as how fetch_retry behaves!

So the above code is basically semantically the same as just fetch_retry by itself.

fetch_retry(url, options, n - 1)
// sementically the same thing as the following
new Promise(function(resolve, reject) {
    fetch_retry(url, options, n - 1).then(resolve).catch(reject)
});
Enter fullscreen mode Exit fullscreen mode

It require one more knowledge in order to clean up the code. We could chain promise.thens in the following way. Because promise.then returns a promise as well!

Promise.resolve(3).then(function(i) {
    return i * 2;
}).then(function(i) {
    return i === 6; // this will be true
});
Enter fullscreen mode Exit fullscreen mode

As you can see, we could pass the processed value onward to the next then and so on. If the value is a Promise, then the next then would receive whatever the returned Promise resolves. See below:

Promise.resolve(3).then(function(i) {
    return i * 2;
}).then(function(i) {
    return Promise.resolve(i * 2); // also work!
}).then(function(i) {
    return i === 12; // this is true! i is not a Promise!
};
Enter fullscreen mode Exit fullscreen mode

The same idea could be applied to catch as well! Thanks to Corentin for the shout out! So this means that we could even resolve a promise when it rejects. Here is an example:

Promise.resolve(3).then(function(i) {
    throw "something's not right";
}).catch(function(i) {
    return i
}).then(function(i) {
    return i === "something's not right";
};
Enter fullscreen mode Exit fullscreen mode

So how could we clean up with these knowledge? The code we have seem to be more complicated.

function fetch_retry(url, options, n) {
    return new Promise(function(resolve, reject) {
        fetch(url, options).then(resolve)
            .catch(function(error) {
                if (n === 1) return reject(error);
                fetch_retry(url, options, n - 1)
                    .then(resolve)  // <--- we try to remove this
                    .catch(reject); // <--- and this
            })
    });
}
Enter fullscreen mode Exit fullscreen mode

Well, we could resolve the returning promise with the promise returned by fetch_retry! Instead of fetch_retry(...).then(resolve).catch(reject). We could do resolve(fetch_retry(...))! So the code becomes:

function fetch_retry(url, options, n) {
    return new Promise(function(resolve, reject) {
        fetch(url, options).then(resolve)
            .catch(function(error) {
                if (n === 1) return reject(error);
                resolve(fetch_retry(url, options, n - 1)); // <--- clean, isn't it?
            })
    });
}
Enter fullscreen mode Exit fullscreen mode

Now we could go even further by removing the explicit creation of the Promise by resolving the promise in catch.

function fetch_retry(url, options, n) {
    return fetch(url, options).catch(function(error) {
        if (n === 1) throw error;
        return fetch_retry(url, options, n - 1);
    });
}
Enter fullscreen mode Exit fullscreen mode

Quoting from MDN with some words tweaked for more layman terms:

The Promise returned by catch(handler) is rejected if handler throws an error or returns a Promise which is itself rejected; otherwise, it is resolved.

ES6

I can predict some JS gurus would be hating me for not using arrow functions. I didn't use arrow functions for people who are not comfortable with it. Here is the ES6 version written with arrow functions, I wouldn't explain much.

const fetch_retry = (url, options, n) => fetch(url, options).catch(function(error) {
    if (n === 1) throw error;
    return fetch_retry(url, options, n - 1);
});
Enter fullscreen mode Exit fullscreen mode

Happy?

ES7

Yeah yeah, Promise is becoming lagacy soon once ES7 async/await hits. So here is an async/await version:

const fetch_retry = async (url, options, n) => {
    try {
        return await fetch(url, options)
    } catch(err) {
        if (n === 1) throw err;
        return await fetch_retry(url, options, n - 1);
    }
};
Enter fullscreen mode Exit fullscreen mode

Which looks a lot neater right?

In fact, we don't have to use recursion with ES7, we could use simple for loop to define this.

const fetch_retry = async (url, options, n) => {
    let error;
    for (let i = 0; i < n; i++) {
        try {
            return await fetch(url, options);
        } catch (err) {
            error = err;
        }
    }
    throw error;
};

// or (tell me which one u like better, I can't decide.)

const fetch_retry = async (url, options, n) => {
    for (let i = 0; i < n; i++) {
        try {
            return await fetch(url, options);
        } catch (err) {
            const isLastAttempt = i + 1 === n;
            if (isLastAttempt) throw err;
        }
    }
};
Enter fullscreen mode Exit fullscreen mode

Conclusion

To conclude, we have looked at 4 different versions of the same function. Three of them are recursive just written in different style and taste. And the last one with for loop. Let's recap:

Primitive version 

function fetch_retry(url, options, n) {
    return fetch(url, options).catch(function(error) {
        if (n === 1) throw error;
        return fetch_retry(url, options, n - 1);
    });
}
Enter fullscreen mode Exit fullscreen mode

ES6 

const fetch_retry = (url, options, n) => fetch(url, options).catch(function(error) {
    if (n === 1) throw error;
    return fetch_retry(url, options, n - 1);
});
Enter fullscreen mode Exit fullscreen mode

ES7 async/await recursive

This is my favorite.

const fetch_retry = async (url, options, n) => {
    try {
        return await fetch(url, options)
    } catch(err) {
        if (n === 1) throw err;
        return await fetch_retry(url, options, n - 1);
    }
};
Enter fullscreen mode Exit fullscreen mode

ES7 async/await for-loop 

const fetch_retry = async (url, options, n) => {
    let error;
    for (let i = 0; i < n; i++) {
        try {
            return await fetch(url, options);
        } catch (err) {
            error = err;
        }
    }
    throw error;
};

// or (tell me which one u like better, I can't decide.)

const fetch_retry = async (url, options, n) => {
    for (let i = 0; i < n; i++) {
        try {
            return await fetch(url, options);
        } catch (err) {
            const isLastAttempt = i + 1 === n;
            if (isLastAttempt) throw err;
        }
    }
};
Enter fullscreen mode Exit fullscreen mode

Tell me your feedback in the comments! :D

Top comments (15)

Collapse
 
kevinfilteau profile image
Kevin Filteau • • Edited

Liquid syntax error: 'raw' tag was never closed

Collapse
 
johnnyshrewd profile image
Johnny Shrewd •

Perhaps add a check for a value smaller than 1 to avoid the infinite cycle:

try {
 ....
} catch (err) {
     if (retry === 1 || retry < 1) throw err;
          return await this.johnny(arg, retry - 1);
     }
}
Enter fullscreen mode Exit fullscreen mode
Collapse
 
lonniebiz profile image
Lonnie Best •

Thanks! Your fetch_retry function saved the day on a bug I was encountering in deno:
github.com/denoland/deno/issues/13...

Collapse
 
webarter profile image
webarter •

If someone in 2021 trying the Primitive version or ES6 version, for some reason the execution never gets to catch unless you add the processing of the success before it: so instead of

return fetch(url, options).catch(function(error)

you absolutely necessarily need to do
return fetch(url, options).then(res => res.json()).catch(function(error)

or whatever processing of response you want to do in that .then()
you can still do next .then down the line from what fetch_retry returns you afterwards

But without this then fetch_retry doesn't actually attempt any retries as it never gets to catch, even though it got 500 server error in my case.

Collapse
 
geekykidstuff profile image
Christian Pasquel •

This is a very useful and perfectly written post. It not only provides an end result but also teaches you by showing the steps one would follow to write this function and improve it after some iterations, so you learn a way of thinking that can be applied to other functions you may want to write. Thank you for the post!

Collapse
 
norfeldtabtion profile image
Lasse Norfeldt • • Edited

I gave it a swing with deno and came up with this:

fetchWithRetry.ts

type Parameters = {
  url: string
  numberOfRetries?: number
  print?: boolean
  dev?: boolean
}

export async function fetchWithRetry({
  url,
  numberOfRetries = 3,
  print,
  dev,
}: Parameters): Promise<Response> {
  const retriesLeft = numberOfRetries - 1
  print && console.log(`Retries left: ${retriesLeft}, url: ${url}`)

  if (retriesLeft == 0) return Promise.reject('Too many attemps without luck')

  return await fetch(url).catch(async (error) => {
    if (numberOfRetries !== 0) {
      await fetchWithRetry({
        url: !dev ? url : url.slice(0, -1),
        numberOfRetries: retriesLeft,
        print,
        dev,
      })
    }
    return error
  })
}

if (import.meta.main) {
  console.log('main')

  const successUrl = 'https://www.google.com'
  const failureUrl = 'https://www.googsdsdsdsdle.com'
  const partialFailingUrl = 'https://www.google.com1'

  console.log(successUrl)
  await fetchWithRetry({ url: successUrl, print: true })

  console.log(failureUrl)
  await fetchWithRetry({ url: failureUrl, print: true }).catch(() => {})

  console.log(partialFailingUrl)
  await fetchWithRetry({ url: partialFailingUrl, print: true, dev: true })
}
Enter fullscreen mode Exit fullscreen mode 
deno run --allow-net fetchWithRetry.ts
Enter fullscreen mode Exit fullscreen mode

The short version of it

type Parameters = {
  url: string
  numberOfRetries?: number
}

export async function fetchWithRetry({
  url,
  numberOfRetries = 3
}: Parameters): Promise<Response> {
  const retriesLeft = numberOfRetries - 1
  if (retriesLeft == 0) return Promise.reject('Too many attemps without luck')

  return await fetch(url).catch(async (error) => {
    if (numberOfRetries !== 0) {
      await fetchWithRetry({
        url,
        numberOfRetries: retriesLeft,
      })
    }
    return error
  })
}
Enter fullscreen mode Exit fullscreen mode
Collapse
 
benarambide profile image
Benjamin Arambide •

So usefull, thanks to share this

Collapse
 
gabrielmmsestrem profile image
Gabriel Mauricio Melgarejo Sestrem •

Thanks Jason. That's my version based on your example:

  const fetchWithRetry = (url) => {
    return new Promise((resolve, reject) => {
      let attempts = 1;
      const fetch_retry = (url, n) => {
        return fetch(url).then(resolve).catch(function (error) {
        if (n === 1) reject(error)
        else
        setTimeout(() => {
          attempts ++
          fetch_retry(url, n - 1);
        }, attempts * 3000)
      });
    }
      return fetch_retry(url, 5);
    });
  }

Then I can call:

fetchWithRetry(api/data/${guid}).then(requestProcess)

Collapse
 
gabrielmmsestrem profile image
Gabriel Mauricio Melgarejo Sestrem •

Hi, can I add the retries to the options?

fetch(url, {
retries: 3,
retryDelay: 1000
})
.then(function(response) {
return response.json();
})
.then(function(json) {
// do something with the result
console.log(json);
});

Collapse
 
enoch1017 profile image
enoch1017 •

Hello, I am a beginner of javascript . After watching your blog about promise ,especially this part-->
.....
.catch(function(error) {
fetch_retry(url, options, n - 1)
.then(resolve) // <--- simply resolve
.catch(reject); // <--- simply reject
})

I don't understand why resolve & reject can be included in then & catch functions directly... Is anything omitted?
Could you tell me why or where I can reference about this issue?
Sorry for my poor English.
I truely appreciate for your time.

Collapse
 
ycmjason profile image
YCM Jason •

Thanks for your comments! I am not sure if I understand your question correctly.

But it sounds like you are confused with the idea of "functions as first-class citizens", which is a fancy way of saying you can pass functions as a values around.

Are you familiar with setTimeout? Have a read about it here.

Consider

setTimeout(function() {
  print('hello')
}, 3000);

We are passing a function function () { print('hello'); } to setTimeout as the first argument. So if we call that function f, then we can equivilantly pass f to setTimeout.

const f = function () {
  print('hello');
}
setTimeout(f, 3000);

So in another words, my code:

.catch(function(error) {
  fetch_retry(url, options, n - 1)
    .then(resolve)
    .catch(reject);
})

Is basically equivalent to:

.catch(function(error) {
  fetch_retry(url, options, n - 1)
    .then(() => resolve())
    .catch(() => reject());
})

It is also important to note that the following is incorrect (or not the same):

.catch(function(error) {
  fetch_retry(url, options, n - 1)
    .then(resolve())
    .catch(reject());
})

Because Javascript will first evaluate resolve() and use the value it returns to pass on to then.

I hope I did help a little.

View full discussion (15 comments)

Read next

victoriadeveloper profile image

Software Engineering Postmortem: Real-time Chat Microservice

Victoria -

shafayeat profile image

Advanced JavaScript Game Development Techniques for Modern Game Devs

Shafayet Hossain -

thinkthroo profile image

Biome.js, a toolchain to format and lint your web project

thinkThroo -

prince_beec5ccde00b7c6c73 profile image

Logo's designing effect in html css

Prince -