Fetch - from simple to scalable implementation

Hey there! 👋

I was bored and wanted to write something. I've ended up with this, a step-by-step guide on how I approach a task, from the most basic to the best fitting implementation for the needs.

What will I build?

Piece of code for fetching data. It will be fetching a jokes API, which returns either a list of jokes or a random joke.

I will then try to improve it step-by-step, until I have a solid and scalable base.

Context

Nothing is built except for the API, the task is to create a way of fetching the jokes so that the UI team can start doing their job.

Initial implementation

Most simple scenario would be to create some sort of function that fetches all the jokes, and one that fetches a random one. Easy enough, let's see how that works:

function fetchAllJokes() {
    return fetch('https://my-api.com/jokes')
        .then(response => response.json());
}

function fetchRandomJoke() {
    return fetch('https://my-api.com/jokes/random')
        .then(response => response.json());
}

As you can see, this would immediately work, and let the UI team do their job right now. But it's not very scalable, let's see how to improve on this without breaking anything for the UI team.

Iteration 1

We know that for now, we can only get jokes, but we also know that most likely this API will expand in the future. And we will need to implement other stuff, like creating/updating jokes. Fetching other resources, etc...

One thing I try to remind myself before I start to build or design a feature is:

"can I make it so I don't have to touch this code again?"

Most times the answer is yes, by kind of using the open-close principle, which states, that a function/method/class should be open for extension but closed to modification.

Another rule I try to apply to myself is, work yourself upwards. What I mean is, start from the most simple, low-level functionality, and then start building on top of that.

In this case the lowest level functionality is executing fetch, with a set of options. So I start by defining a custom function around fetch:

function fetcher(url, options = {}) {
    return fetch(url, {
        method: HttpMethods.GET,
        ...options,
    });
}

It's mostly the same as calling fetch directly, but, with a difference:

• It centralizes where fetch is called, instead of calling fetch directly in several places in the app, we only use it in the fetcher function.

• It's easier to change/modify in case fetch API changes or we want to do something before or after each fetch request. Though I would resist it if it can be avoided as you will see later on in the post.

Now that we have this base, we can start building on top of it. Let's make it possible to use the most common HTTP methods, like POST, PUT, GET, DELETE.

function fetcherPost(url, options = {}) {
    return fetcher(url, {
        ...options,
        method: HttpMethods.POST,
    });
}

function fetcherPut(url, options = {}) {
    return fetcher(url, {
        ...options,
        method: HttpMethods.PUT,
    });
}
// ...

I think you get the gist of it. We create a function for each method.

We would use it as follows:

function fetchAllJokes() {
    return fetcherGet('https://my-api.com/jokes')
        .then(response => response.json());
}

function fetchRandomJoke() {
    return fetcherGet('https://my-api.com/jokes/random')
        .then(response => response.json());
}

This is ok, but we can do better.

Iteration 2

The API uri will probably be the same in all requests, and maybe other ones too. So let's store that in an env variable:

function fetchAllJokes() {
    return fetcherGet(`${env.API_URL}/jokes`)
        .then(response => response.json());
}

Better, now you can see that converting response to JSON is also being repeated. How could we improve this?

First, let's see how NOT TO DO IT, which would be to just add it to the fetcher function, in the end, all requests pass through it, right?

function fetcher(url, options = {}) {
    return fetch(url, {
        method: HttpMethods.GET,
        ...options,
    })
    .then(response => response.json());
}

function fetchAllJokes() {
    return fetcherGet(`${env.API_URL}/jokes`);
}

Yes, we get rid of it in the fetchAllJokes function, but what if a request does not return JSON?

We would need to then remove it from the fetcher, and add it again to only those requests that return JSON. Wasting time changing stuff we have already done, and remember the rule "can I make it so I don't have to touch the code I write again?".

Now let's see HOW TO DO IT:

I want to express that there are always many correct ways to solve a problem, this is one of them. It's in no way the only or best solution

One option would be to extract the functionality into a function, for example:

function jsonResponse(response) {
    return response.json();
}

// Then we could use it as follows
function fetchAllJokes() {
    return fetcherGet(`${env.API_URL}/jokes`).then(jsonResponse);
}

// And if we receive other format
function fetchAllJokes() {
    return fetcherGet(`${env.API_URL}/jokes`).then(xmlResponse);
}

This is a good approach, as it lets us process the response afterward, depending on the data returned.

We could even extend the fetcher function, for each data format:

function jsonFetcher(url, options = {}) {
    return fetcher(url, options).then(jsonResponse);
}

function xmlFetcher(url, options = {}) {
    return fetcher(url, options).then(xmlResponse);
}

This approach is even better in some senses, as we can check things like headers, body, etc on each request...

For example, we want to ensure that, with json requests, a header of type 'application/json' is sent.

function jsonFetcher(url, options = {}) {
    const isPost = options.method === HttpMethods.POST;
    const hasHeaders = options.headers != null;

    if (!hasHeaders) options.headers = {};

    if (isPost) {
        options.headers['Content-Type'] = 'application/json';
    }

    return fetcher(url, options).then(jsonResponse);
}

Now, any time a post request is made with jsonFetcher, the content-type header is always set to 'application/json'.

BUT and a big BUT, with this approach, you might have spotted a problem. We now have to create new functions for each method (fetcherGet, fetcherPost), for each fetcher...

Iteration 3

This could be improved by rethinking how we create fetchers, instead of overriding the fetcher function, we could return an object, containing all methods for that specific fetcher.

One solution to this problem would be to create a function, which receives a fetcher, and returns an object with all methods attached:

function crudForFetcher(fetcher) {
    return {
        get(url, options = {}) {
            return fetcher(url, {
                ...options,
                method: HttpMethods.GET,
            })
        },
        post(url, options = {}) {
            return fetcher(url, {
                ...options,
                method: HttpMethods.POST,
            })
        },
        // ...more methods ...
    }
}

// Create fetch for each fetcher type
const fetchDefault = crudForFetcher(fetcher);
const fetchJson = crudForFetcher(jsonFetcher);
const fetchXml = crudForFetcher(xmlFetcher);

fetchJson.get('my-api.com/hello');

There's still a thing that's bugging me a bit, it is that we need to pass the full API URI in each request, now it's really simple to add this functionality as we have it all broke down.

What we can do is improve the crudForFetcher function a bit more, by making it receive some options:

function crudForFetcher(fetcher, options = { uri: '', root: '' }) {
    const { uri, root } = options;

    return {
        get(path, options = {}) {
            return fetcher(path.join(uri, root, path), {
                ...options,
                method: HttpMethods.GET,
            })
        },
        // ... more methods ...
    }
}

const jokesFetcher = crudForFetcher(
    jsonFetcher, 
    { 
        uri: env.API_URL, 
        root: `jokes` 
    }
);

What this change does, is, merges the URI, root, and path of a specific request, into a single URI.

In the case of jokesFetcher, the URI for the requests will always start with https://my-api.com/jokes.

We can now safely replace our original functions, without the UI team needing to change anything, but we now have a lot more power and ready to scale, yay!!!

function fetchAllJokes() {
    return jokesFetcher.get(); // `https://my-api.com/jokes`
}

function fetchRandomJoke() {
    return jokesFetcher.get('/random'); // `https://my-api.com/jokes/random`
}

As you can see, we have not modified anything we've built, except for crudForFetcher.

Everything put together

function fetcher(url, options = {}) {
    return fetch(url, {
        method: HttpMethods.GET,
        ...options,
    });
}

function jsonResponse(response) {
    return response.json();
}

function jsonFetcher(url, options = {}) {
    return fetcher(url, options).then(jsonResponse);
}

function crudForFetcher(fetcher, options = { uri: '', root: '' }) {
    const { uri, root } = options;

    return {
        get(path, options = {}) {
            return fetcher(path.join(uri, root, path), {
                ...options,
                method: HttpMethods.GET,
            })
        },
        post(path, options = {}) {
            return fetcher(path.join(uri, root, path), {
                ...options,
                method: HttpMethods.POST,
            })
        },
    }
}

// Exposed API
const fetchJokes = crudForFetcher(
    jsonFetcher, 
    { 
        uri: env.API_URL, 
        root: `jokes` 
    }
);
function fetchAllJokes() {
    return jokesFetcher.get(); 
}

function fetchRandomJoke() {
    return jokesFetcher.get('/random');
}

Summary

We've taken a simple implementation, and bit by bit, building up until we have something that will scale quite well, without breaking anything along the way (with a bit more refinement work of course).

I've been using this approach for the last couple our years, in a variety of projects, frameworks, languages, etc... and it's working out pretty well for me.

It's also been really productive, in the sense that it has reduced the amount of work I need to do significantly.

And just to reiterate, this is one approach of many that could work in this scenario. I might post a different approach using oop.

What to take out of this:

• Understand the task at hand
• Look at the forest, not just the trees (don't just implement the feature, think about it, and the stuff around it)
• Build things progressively, but not recklessly
• Make functions/methods as closed as possible
• Keep things simple

---

I really enjoyed writing this, and I hope you like the read as well!

If you did, consider supporting me by reacting to the post, following me here or over on GitHub, or commenting! ❤️

Comments

[artydev]

Thank you, very instructive and usefull
For pure information, here is an article that you could be interested in

daily-dev-tips.com/posts/javascrip...

[tq-bit]

A solid article, well done. I find it interesting that you favored nested functions over classes. Did you do that for the sake of ease or did you have some other motivation?

Also, perhaps as an additional interation, have you considered adding some kind of internal progress indicator? You can use the Response interface of the fetch API to achieve such behavior.

developer.mozilla.org/en-US/docs/W...

[Keff]

Thanks, happy you liked it!

Good question, actually it came out that way. I started by implementing the simplest function to fetch data, and then descended down that way. Though I think it's also easier to follow than with OOP. I also don't do much functional programming so it's always nice to refresh on it a bit :)

I will also be doing an article, mostly the same concept but with oop. And I might try including the progress indicator, I've not used it much with fetch though as I usually use other libraries around it for real projects.

Cheers!

[Peter Roto]

Thanks for the article, it was interesting.

As a newer Dev, I have a question that probably any one could answer, but could you have made fetcher a constructor function so thoughout the code you could simply call like fetcher.get(arguments, options, etc)?

If so, would you ever do this? If not, why wouldn't it work?

[Keff]

Hey, peter! I'm glad you liked it

I don't know If I understand the question correctly, I will answer what I think you meant, but please let me know if it wasn't this, I'm happy to resolve any questions you have!

Do you mean creating a fetcher object using a constructor function (class) instead, something like this:

class Fetcher {
   get(uri, options) {
     ...
   }
}

const fetcher = new Fetcher();
fetcher.get();

If so, the answer is yes, you can do it in many ways, depending on many factors, like the codebase you already have, do you follow any practice that forces you to do it one way or another, etc...

I'm currently working on a post similar to this one but in an object-oriented approach. In this example, I stuck to functions to make it clearer, as OOP has more code and is a bit more difficult to follow!

I will let you know when I post it so you can see the different approach.

[Peter Roto]

This answered my question. Thank you so much for your reply.

[John McElreavey]

Good article. Nice to see someone practicing some of the solid principles. Would love to see an OOP article too :)

[Keff]

Thanks, I'm glad you enjoyed it!

I will be writing one soon about oop I think. It's the approach I use on most of my projects, so it will be a bit more detailed