Where to Put Response Metadata - Envelope or HTTP Headers?

Thomas Werner — Sat, 15 Dec 2018 00:28:18 +0000

Here's some nitpicky head-scratcher I'm overthinking at the moment. Let's assume we have a RESTful API with resource collection endpoints that allow us to search and get paginated results.

Where to put response metadata such as "next page" URL or next page cursor token?

There are two ways to do that.

HTTP Response Headers

Here's my currently preferred way of adding meta information. It can be added as HTTP headers to the response, while leaving the actual response data untouched:

MyAPI-Next-Page: /users?next=ABCDEFG123

[
    {"id": "user-1"},
    {"id": "user-2"}
]

Response Envelope

However, many APIs out there (such as Facebook or Twitter) prefer to wrap the response in a data field of an envelope object, and add other meta data to that wrapper:

{
    "pagination": {
        "next": "/users?next=ABCDEFG123"
    },
    "data": [
        {"id": "user-1"},
        {"id": "user-2"}
    ]
}

It's even touted as the way to go on the following website that proposes a standard for JSON APIs: https://jsonapi.org/

Why I don't like Envelopes

To be honest I started to dislike them fairly recently. A couple of years ago I designed and implemented an ecommerce web API. Inspired by common JSON API design practices out there I actually did wrap search results in envelopes.

These were the three issues in our case:

1. The meta data was not really needed by the client apps after all

They use infini-scrollers and simply request the next page until the API responds with an empty result set.

Also total amount of result items or amount of pages is basically ignored. We didn't need to update the UI with that information, and we did not implement an old-school navigation bar.

2. Response data is a bit awkward to handle in JavaScript code

axios.get('/users').then(response => {
    //                              +-- really?
    //                              v
    let resultList = response.data.data; 
    // ...
});

Sure, this is really just a cosmetic thing that can be easily fixed by renaming the data property to something else such as results. But that would also mean breaking compatibility with that JSON:API standard I mentioned earlier (just kidding, I don't really care about that standard :-D).

3. It repeatedly introduced the same kind of bug in the client apps

The extra jump to the actual result set response.data.data was often overlooked, which repeatedly caused bugs that were only detected while testing. And some of them remained hidden for a while.

The added confusion probably also stemmed from the fact that individual resources were not wrapped in an envelope and therefor directly accessible through response.data.

(EDIT) 4. It's also kinda redundant

As someone over at Stackoverflow pointed out: HTTP is already the envelope. There really shouldn't be a need for wrapping result sets in an envelope data structure.

So why am I still confused?

I assume that very smart people with a ton of hands on practical experience work at Facebook, Twitter and the likes. I also assume they might have run into issues with using HTTP response headers for adding meta data to responses.

Does anyone know about such issues? I believe it might be problems with proxies that wipe out or alter headers? Or is it JavaScript in browsers that doesn't give full access to all headers? Or are there weird HTTP client libraries out there that don't know how to parse headers?

What is your experience or opinion?

Trying to update dependencies

Thomas Werner — Wed, 10 Oct 2018 23:28:12 +0000

DEV Community: Thomas Werner