YCM Jason

Posted on Mar 27, 2018

Javascript fetch, retry upon failure.

#javascript #fetch #recursion #node

This post is republished since the previous one was not showing up on my profile for some reason.

So recently, I bumped into a situation where the network works/fails randomly. As it affects the consistency of my test results, I decided to implement a fetch_retry function which retries fetch upon failure up to n times.

Introduction

Fetch in Javascript is great. I hope you will agree that it provides a simple yet robust enough interface to do our AJAX requests.

However, network doesn't always work like we want it to, and it might fail randomly. To catch this issue, let's implement a function fetch_retry(url, options, n) which does fetch(url, options) but retries up to n times upon failure. And hence increasing the chance of success.

Let's think

Retrying things sounds like a loop. Why don't we write a for/while loop to do that? Something like the following, perhaps?

function fetch_retry(url, options, n) {
    for(let i = 0; i < n; i++){
        fetch(url, options);
        if(succeed) return result;
    }
}

NO! Fetch is an asynchronous function, meaning that the program would not wait for result before continuing! n fetches will be called at the same time (kind of), regardless of whether the previous calls succeed!

This is not what we want. This is not retry upon failure, this is fetching n times simultaneously! (That being said, if written correctly, it could also increase the chance of success. Perhaps with something like Promsie.any? Although I am not a big fan of bluebird. I think native Promise is good enough.)

If you don't know about asynchronous functions and Promise in Javascript, watch this amazing video here, made by Jessica Kerr, before reading on!

Briefly about fetch

So fetch returns a Promise. We usually call it like this.

fetch(url, { method: 'GET' }).then(res => console.log('done'));
console.log('fetching...');

If you understand Promise correctly, you should expect the result to be:

fetching...
done

And if the network fails for some reason, the Promise rejects and we could catch the error as follows:

fetch(url, { method: 'GET' }).catch(err => /* ... */);

So how to implement?

What does `fetch_retry` do?

We start off with thinking what do we want the function fetch_retry do. We know it has to call fetch somehow, so let's write that down.

function fetch_retry(url, options, n) {
    fetch(url, options)
        .then(function(result) {
            /* on success */
        }).catch(function(error) {
            /* on failure */
        })
}

Now obviously fetch_retry has to be an asynchronous function, since we can't really define a synchronous function out of an asynchronous one. (or could we? Enlighten me.)

Definition: So this means fetch_retry should return a Promise that resolves if any attempt out of n attempts succeed, and rejects if all n attempts failed.

So let's return a Promise now.

function fetch_retry(url, options, n) {
    return new Promise(function(resolve, reject) { // <--- we know it is asynchronous, so just return a promise first!
        fetch(url, options)
            .then(function(result) {
                /* on success */
            }).catch(function(error) {
                /* on failure */
            })
    });
}

What if `fetch` succeeds?

So if the fetch succeed, we obviously can just resolve the promise we are returning, by calling the resolve function. So the code become:

function fetch_retry(url, options, n) {
    return new Promise(function(resolve, reject) {
        fetch(url, options)
            .then(function(result) {
                /* on success */
                resolve(result); // <--- yeah! we are done!
            }).catch(function(error) {
                /* on failure */
            })
    });
}

What if `fetch` fails?

What should we do on failure? Doing for/while loop here wouldn't really help, due to the asynchronous property we discussed previously. But there is one thing that we could do what for/while loop does. Does it ring a bell? Yes! Recursion!

My two rules of thumb when doing recursion:

Do not think recursively. Don't try to follow your code recursively.
Leap of faith, assume the recursive function you are defining works.

These two points are fundamentally the same! If you have the leap of faith, you wouldn't be thinking recursively into the code.

Ok, so let's try to take the leap of faith and assume fetch_retry will just work, magically.

If it works, then in on failure, what will happen if we call fetch_retry(url, options, n - 1)?

function fetch_retry(url, options, n) {
    return new Promise(function(resolve, reject) {
        fetch(url, options)
            .then(function(result) {
                /* on success */
                resolve(result);
            })
            .catch(function(error) {
                /* on failure */
                fetch_retry(url, options, n - 1) // <--- leap of faith, this will just work magically! Don't worry!
                    .then(/* one of the remaining (n - 1) fetch succeed */)
                    .catch(/* remaining (n - 1) fetch failed */);
            })
    });
}

fetch_retry(url, options, n - 1) will just work magically by the leap of faith and would return a Promise which, by the definition we discussed previously, resolves if any attempt (out of n - 1 attempts) succeed, and rejects if all n - 1 attempts failed.

So now, what do we do after the recursive call? Notice that since fetch_retry(url, options, n - 1) would work magically, this means we have done all n fetching at this point. In the on failure case, simply resolves if fetch_retry(url, options, n - 1) resolves, and rejects if it rejects.

function fetch_retry(url, options, n) {
    return new Promise(function(resolve, reject) {
        fetch(url, options)
            .then(function(result) {
                /* on success */
                resolve(result);
            })
            .catch(function(error) {
                fetch_retry(url, options, n - 1)
                    .then(resolve)  // <--- simply resolve
                    .catch(reject); // <--- simply reject
            })
    });
}

Great! We are almost there! We know we need a base case for this recursive call. When thinking about base case, we look at the function arguments, and decide at what situation we could tell the result immediately.

The answer is when n === 1 and the fetch fails. In this case, we could simply reject with the error from fetch, without calling fetch_retry recursively.

function fetch_retry(url, options, n) {
    return new Promise(function(resolve, reject) {
        fetch(url, options)
            .then(function(result) {
                /* on success */
                resolve(result);
            })
            .catch(function(error) {
                if (n === 1) return reject(error); // <--- base case!
                fetch_retry(url, options, n - 1)
                    .then(resolve)
                    .catch(reject);
            })
    });
}

Clean things up

Redundant function

In our "on success" function, we are simply calling resolve(result). So this function instance is redundant, we could simply use resolve as the "on success" function. So the code would become:

function fetch_retry(url, options, n) {
    return new Promise(function(resolve, reject) {
        fetch(url, options).then(resolve) // <--- Much cleaner!
            .catch(function(error) {
                if (n === 1) return reject(error);
                fetch_retry(url, options, n - 1)
                    .then(resolve)
                    .catch(reject);
            })
    });
}

Redundant promise

Now another stupid thing we are doing here is this line:

fetch_retry(url, options, n - 1).then(resolve).catch(reject)

Do you see what is the problem?

Let me put this in context, we are essentially doing this:

new Promise(function(resolve, reject) {
    fetch_retry(url, options, n - 1).then(resolve).catch(reject)
});

So this new promise is redundant in this case, because it is resolving if fetch_retry resolves, and rejecting if fetch_retry rejects. So basically it behaves exactly the same as how fetch_retry behaves!

So the above code is basically semantically the same as just fetch_retry by itself.

fetch_retry(url, options, n - 1)
// sementically the same thing as the following
new Promise(function(resolve, reject) {
    fetch_retry(url, options, n - 1).then(resolve).catch(reject)
});

It require one more knowledge in order to clean up the code. We could chain promise.thens in the following way. Because promise.then returns a promise as well!

Promise.resolve(3).then(function(i) {
    return i * 2;
}).then(function(i) {
    return i === 6; // this will be true
});

As you can see, we could pass the processed value onward to the next then and so on. If the value is a Promise, then the next then would receive whatever the returned Promise resolves. See below:

Promise.resolve(3).then(function(i) {
    return i * 2;
}).then(function(i) {
    return Promise.resolve(i * 2); // also work!
}).then(function(i) {
    return i === 12; // this is true! i is not a Promise!
};

The same idea could be applied to catch as well! Thanks to Corentin for the shout out! So this means that we could even resolve a promise when it rejects. Here is an example:

Promise.resolve(3).then(function(i) {
    throw "something's not right";
}).catch(function(i) {
    return i
}).then(function(i) {
    return i === "something's not right";
};

So how could we clean up with these knowledge? The code we have seem to be more complicated.

function fetch_retry(url, options, n) {
    return new Promise(function(resolve, reject) {
        fetch(url, options).then(resolve)
            .catch(function(error) {
                if (n === 1) return reject(error);
                fetch_retry(url, options, n - 1)
                    .then(resolve)  // <--- we try to remove this
                    .catch(reject); // <--- and this
            })
    });
}

Well, we could resolve the returning promise with the promise returned by fetch_retry! Instead of fetch_retry(...).then(resolve).catch(reject). We could do resolve(fetch_retry(...))! So the code becomes:

function fetch_retry(url, options, n) {
    return new Promise(function(resolve, reject) {
        fetch(url, options).then(resolve)
            .catch(function(error) {
                if (n === 1) return reject(error);
                resolve(fetch_retry(url, options, n - 1)); // <--- clean, isn't it?
            })
    });
}

Now we could go even further by removing the explicit creation of the Promise by resolving the promise in catch.

function fetch_retry(url, options, n) {
    return fetch(url, options).catch(function(error) {
        if (n === 1) throw error;
        return fetch_retry(url, options, n - 1);
    });
}

Quoting from MDN with some words tweaked for more layman terms:

The Promise returned by catch(handler) is rejected if handler throws an error or returns a Promise which is itself rejected; otherwise, it is resolved.

ES6

I can predict some JS gurus would be hating me for not using arrow functions. I didn't use arrow functions for people who are not comfortable with it. Here is the ES6 version written with arrow functions, I wouldn't explain much.

const fetch_retry = (url, options, n) => fetch(url, options).catch(function(error) {
    if (n === 1) throw error;
    return fetch_retry(url, options, n - 1);
});

Happy?

ES7

Yeah yeah, Promise is becoming lagacy soon once ES7 async/await hits. So here is an async/await version:

const fetch_retry = async (url, options, n) => {
    try {
        return await fetch(url, options)
    } catch(err) {
        if (n === 1) throw err;
        return await fetch_retry(url, options, n - 1);
    }
};

Which looks a lot neater right?

In fact, we don't have to use recursion with ES7, we could use simple for loop to define this.

const fetch_retry = async (url, options, n) => {
    let error;
    for (let i = 0; i < n; i++) {
        try {
            return await fetch(url, options);
        } catch (err) {
            error = err;
        }
    }
    throw error;
};

// or (tell me which one u like better, I can't decide.)

const fetch_retry = async (url, options, n) => {
    for (let i = 0; i < n; i++) {
        try {
            return await fetch(url, options);
        } catch (err) {
            const isLastAttempt = i + 1 === n;
            if (isLastAttempt) throw err;
        }
    }
};

Conclusion

To conclude, we have looked at 4 different versions of the same function. Three of them are recursive just written in different style and taste. And the last one with for loop. Let's recap:

Primitive version

function fetch_retry(url, options, n) {
    return fetch(url, options).catch(function(error) {
        if (n === 1) throw error;
        return fetch_retry(url, options, n - 1);
    });
}

ES6

const fetch_retry = (url, options, n) => fetch(url, options).catch(function(error) {
    if (n === 1) throw error;
    return fetch_retry(url, options, n - 1);
});

ES7 async/await recursive

This is my favorite.

const fetch_retry = async (url, options, n) => {
    try {
        return await fetch(url, options)
    } catch(err) {
        if (n === 1) throw err;
        return await fetch_retry(url, options, n - 1);
    }
};

ES7 async/await for-loop

const fetch_retry = async (url, options, n) => {
    let error;
    for (let i = 0; i < n; i++) {
        try {
            return await fetch(url, options);
        } catch (err) {
            error = err;
        }
    }
    throw error;
};

// or (tell me which one u like better, I can't decide.)

const fetch_retry = async (url, options, n) => {
    for (let i = 0; i < n; i++) {
        try {
            return await fetch(url, options);
        } catch (err) {
            const isLastAttempt = i + 1 === n;
            if (isLastAttempt) throw err;
        }
    }
};

Tell me your feedback in the comments! :D

Top comments (15)

Kevin Filteau • Aug 29 '19 • Edited

Liquid syntax error: 'raw' tag was never closed

Johnny Shrewd • Apr 23 '19

Perhaps add a check for a value smaller than 1 to avoid the infinite cycle:

try {
 ....
} catch (err) {
     if (retry === 1 || retry < 1) throw err;
          return await this.johnny(arg, retry - 1);
     }
}

Lonnie Best • Jun 13 '22

Thanks! Your fetch_retry function saved the day on a bug I was encountering in deno:
github.com/denoland/deno/issues/13...

webarter • Jun 3 '21

If someone in 2021 trying the Primitive version or ES6 version, for some reason the execution never gets to catch unless you add the processing of the success before it: so instead of

return fetch(url, options).catch(function(error)

you absolutely necessarily need to do
return fetch(url, options).then(res => res.json()).catch(function(error)

or whatever processing of response you want to do in that .then()
you can still do next .then down the line from what fetch_retry returns you afterwards

But without this then fetch_retry doesn't actually attempt any retries as it never gets to catch, even though it got 500 server error in my case.

Christian Pasquel • May 27 '19

This is a very useful and perfectly written post. It not only provides an end result but also teaches you by showing the steps one would follow to write this function and improve it after some iterations, so you learn a way of thinking that can be applied to other functions you may want to write. Thank you for the post!

Lasse Norfeldt • Oct 22 '21 • Edited

I gave it a swing with deno and came up with this:

fetchWithRetry.ts

type Parameters = {
  url: string
  numberOfRetries?: number
  print?: boolean
  dev?: boolean
}

export async function fetchWithRetry({
  url,
  numberOfRetries = 3,
  print,
  dev,
}: Parameters): Promise<Response> {
  const retriesLeft = numberOfRetries - 1
  print && console.log(`Retries left: ${retriesLeft}, url: ${url}`)

  if (retriesLeft == 0) return Promise.reject('Too many attemps without luck')

  return await fetch(url).catch(async (error) => {
    if (numberOfRetries !== 0) {
      await fetchWithRetry({
        url: !dev ? url : url.slice(0, -1),
        numberOfRetries: retriesLeft,
        print,
        dev,
      })
    }
    return error
  })
}

if (import.meta.main) {
  console.log('main')

  const successUrl = 'https://www.google.com'
  const failureUrl = 'https://www.googsdsdsdsdle.com'
  const partialFailingUrl = 'https://www.google.com1'

  console.log(successUrl)
  await fetchWithRetry({ url: successUrl, print: true })

  console.log(failureUrl)
  await fetchWithRetry({ url: failureUrl, print: true }).catch(() => {})

  console.log(partialFailingUrl)
  await fetchWithRetry({ url: partialFailingUrl, print: true, dev: true })
}

deno run --allow-net fetchWithRetry.ts

The short version of it

type Parameters = {
  url: string
  numberOfRetries?: number
}

export async function fetchWithRetry({
  url,
  numberOfRetries = 3
}: Parameters): Promise<Response> {
  const retriesLeft = numberOfRetries - 1
  if (retriesLeft == 0) return Promise.reject('Too many attemps without luck')

  return await fetch(url).catch(async (error) => {
    if (numberOfRetries !== 0) {
      await fetchWithRetry({
        url,
        numberOfRetries: retriesLeft,
      })
    }
    return error
  })
}

Josue Benjamin Arambide Quispe • Nov 15 '18

So usefull, thanks to share this

Gabriel Mauricio Melgarejo Sestrem • Apr 2 '19

Thanks Jason. That's my version based on your example:

  const fetchWithRetry = (url) => {
    return new Promise((resolve, reject) => {
      let attempts = 1;
      const fetch_retry = (url, n) => {
        return fetch(url).then(resolve).catch(function (error) {
        if (n === 1) reject(error)
        else
        setTimeout(() => {
          attempts ++
          fetch_retry(url, n - 1);
        }, attempts * 3000)
      });
    }
      return fetch_retry(url, 5);
    });
  }

Then I can call:

fetchWithRetry(api/data/${guid}).then(requestProcess)

Gabriel Mauricio Melgarejo Sestrem • Mar 31 '19

Hi, can I add the retries to the options?

fetch(url, {
retries: 3,
retryDelay: 1000
})
.then(function(response) {
return response.json();
})
.then(function(json) {
// do something with the result
console.log(json);
});

enoch1017 • Jan 8 '19

Hello, I am a beginner of javascript . After watching your blog about promise ,especially this part-->
.....
.catch(function(error) {
fetch_retry(url, options, n - 1)
.then(resolve) // <--- simply resolve
.catch(reject); // <--- simply reject
})

I don't understand why resolve & reject can be included in then & catch functions directly... Is anything omitted?
Could you tell me why or where I can reference about this issue?
Sorry for my poor English.
I truely appreciate for your time.

YCM Jason • Jan 9 '19

Thanks for your comments! I am not sure if I understand your question correctly.

But it sounds like you are confused with the idea of "functions as first-class citizens", which is a fancy way of saying you can pass functions as a values around.

Are you familiar with setTimeout? Have a read about it here.

Consider

setTimeout(function() {
  print('hello')
}, 3000);

We are passing a function function () { print('hello'); } to setTimeout as the first argument. So if we call that function f, then we can equivilantly pass f to setTimeout.

const f = function () {
  print('hello');
}
setTimeout(f, 3000);

So in another words, my code:

.catch(function(error) {
  fetch_retry(url, options, n - 1)
    .then(resolve)
    .catch(reject);
})

Is basically equivalent to:

.catch(function(error) {
  fetch_retry(url, options, n - 1)
    .then(() => resolve())
    .catch(() => reject());
})

It is also important to note that the following is incorrect (or not the same):

.catch(function(error) {
  fetch_retry(url, options, n - 1)
    .then(resolve())
    .catch(reject());
})

Because Javascript will first evaluate resolve() and use the value it returns to pass on to then.

I hope I did help a little.

Ranesh • Dec 23 '19 • Edited

In the last line, do we really need to return await ...? i.e.

return await fetch_retry(url, options, n - 1);

?

Can we simply

return fetch_retry(url, options, n - 1);

?

YCM Jason • Dec 23 '19

You can omit the await. I like putting await because it reminds me that I am dealing with async functions and direct values instead of Promises. Although async/await are just playing with promises at the end of the day.

Ben Halpern • Mar 28 '18

Sorry about the issue. We know all about what the problem was. Slighly less about what the solution is yet. We'll have it fixed soon, promise.

YCM Jason • Mar 28 '18

It's alright, copy and pasting isn't very hard. Only thing is that I lost my comments and likes. I have set my previous post to unpublished in case you guys fixed it and two articles show on my profile. haha

DEV Community

Javascript fetch, retry upon failure.

Introduction

Let's think

Briefly about fetch

So how to implement?

What does `fetch_retry` do?

What if `fetch` succeeds?

What if `fetch` fails?

Clean things up

Redundant function

Redundant promise

ES6

ES7

Conclusion

Primitive version

ES6

ES7 async/await recursive

ES7 async/await for-loop

Top comments (15)

Read next

Build a spreadsheet app with an AI-copilot (Next.js, gpt4, LangChain, & CopilotKit)

Deploy NextJS App on Firebase 🔥

React: About highly customizable components - P1

Document Object Model (DOM) : One Byte Explainer

Introduction

Let's think

Briefly about fetch

So how to implement?

What does fetch_retry do?

What if fetch succeeds?

What if fetch fails?

Clean things up

Redundant function

Redundant promise

ES6

ES7

Conclusion

Primitive version

ES6

ES7 async/await recursive

ES7 async/await for-loop

Read next

Build a spreadsheet app with an AI-copilot (Next.js, gpt4, LangChain, & CopilotKit)

Deploy NextJS App on Firebase 🔥

React: About highly customizable components - P1

Document Object Model (DOM) : One Byte Explainer

What does `fetch_retry` do?

What if `fetch` succeeds?

What if `fetch` fails?