Where to Put Response Metadata - Envelope or HTTP Headers?

Here's some nitpicky head-scratcher I'm overthinking at the moment. Let's assume we have a RESTful API with resource collection endpoints that allow us to search and get paginated results.

Where to put response metadata such as "next page" URL or next page cursor token?

There are two ways to do that.

HTTP Response Headers

Here's my currently preferred way of adding meta information. It can be added as HTTP headers to the response, while leaving the actual response data untouched:

MyAPI-Next-Page: /users?next=ABCDEFG123

[
    {"id": "user-1"},
    {"id": "user-2"}
]

Response Envelope

However, many APIs out there (such as Facebook or Twitter) prefer to wrap the response in a data field of an envelope object, and add other meta data to that wrapper:

{
    "pagination": {
        "next": "/users?next=ABCDEFG123"
    },
    "data": [
        {"id": "user-1"},
        {"id": "user-2"}
    ]
}

It's even touted as the way to go on the following website that proposes a standard for JSON APIs: https://jsonapi.org/

Why I don't like Envelopes

To be honest I started to dislike them fairly recently. A couple of years ago I designed and implemented an ecommerce web API. Inspired by common JSON API design practices out there I actually did wrap search results in envelopes.

These were the three issues in our case:

1. The meta data was not really needed by the client apps after all

They use infini-scrollers and simply request the next page until the API responds with an empty result set.

Also total amount of result items or amount of pages is basically ignored. We didn't need to update the UI with that information, and we did not implement an old-school navigation bar.

2. Response data is a bit awkward to handle in JavaScript code

axios.get('/users').then(response => {
    //                              +-- really?
    //                              v
    let resultList = response.data.data; 
    // ...
});

Sure, this is really just a cosmetic thing that can be easily fixed by renaming the data property to something else such as results. But that would also mean breaking compatibility with that JSON:API standard I mentioned earlier (just kidding, I don't really care about that standard :-D).

3. It repeatedly introduced the same kind of bug in the client apps

The extra jump to the actual result set response.data.data was often overlooked, which repeatedly caused bugs that were only detected while testing. And some of them remained hidden for a while.

The added confusion probably also stemmed from the fact that individual resources were not wrapped in an envelope and therefor directly accessible through response.data.

(EDIT) 4. It's also kinda redundant

As someone over at Stackoverflow pointed out: HTTP is already the envelope. There really shouldn't be a need for wrapping result sets in an envelope data structure.

So why am I still confused?

I assume that very smart people with a ton of hands on practical experience work at Facebook, Twitter and the likes. I also assume they might have run into issues with using HTTP response headers for adding meta data to responses.

Does anyone know about such issues? I believe it might be problems with proxies that wipe out or alter headers? Or is it JavaScript in browsers that doesn't give full access to all headers? Or are there weird HTTP client libraries out there that don't know how to parse headers?

What is your experience or opinion?

Comments

[Akash Kava]

Headers should only be used by underlying protocol implementation, not business information. Even if client does not use paging, doesn't mean any other implementation will not. Tomorrow if you wish to go old school navigation with pager, then rewriting api will be another pain.

Another important use case is, what if we may want to change protocol from HTTP to something else, we might want to use some other channel, switch from http to web sockets, migration should be easy in fact logic information should be independent of underlying transport.

[Thomas Werner]

In practice HTTP headers are used for adding meta information beyond the needs for just the transfer layer. We add authorization information, signatures, nonces, timestamps for triggering higher level application logic on the back-end side. And as Michiel Hendriks pointed out, there is even RFC 8288 that explicitly allows us to add web links as HTTP response headers. Reading through the specs, even the one about deprecating X- headers I do not get the impression that we were not supposed to add custom meta information to HTTP response (or request) headers.

Let's take the HTTP status code as an example. It is part of the transfer protocol but expresses success or failure or something else about the result of applying some business logic (object created, object moved, you are not authorized to do that etc.)? Should that also (always) go into an additional payload envelope? (I know there are scenarios where additional error / success information is appropriate in response bodies, and often we like to add the HTTP status code there, too)

Or what about the use of HTTP verbs like GET / POST / PATCH / PUT / DELETE that are part of HTTP and all pertain to business logic. Should that also go into a "client request envelope" just because we might be migrating in the future to a different transport?

[Akash Kava]

Status code is useless, back in 80s where error codes were used to indicate error and you had look up errors in table with matching description. We live in world of exceptions where we need to know error message and stacktrace !! We no longer simply display an error message either, we display relevant useful help links to user to troubleshoot further.

Again, authorization is independent of logic and authorization is transport specific. In HTTP you authorize every request, in WebSockets or traditional sockets you only authorize only first request. Also authorization may change based on form of authentication, still it isn't part of logic of individual operation. Transport will either execute logic if it is authorize or it will not at all execute the logic.

And see where users are moving, GraphQL, Firebase, all are additional abstraction over HTTP because HTTP wasn't sufficient. For end user it is more important to open a session, query, get results or exception irrespective of inspecting further headers and trying to investigate underlying implementation. Problem comes when you go mobile and you have very limited control over underlying transport.

[SondreB]

HTTP status codes are very useful and should be used. It does not take long time to learn the most basic, like 404 (Not Found) and 200 (OK). I recently had an incident where an API returned status 200, but within the content was an error message. Not using proper HTTP status codes makes consuming and debugging APIs much harder.

[panta82]

I would use the envelope. Don't tie your data format to a transport protocol implementation details. If you suddenly decide you want to store and replay data payloads or put it through queue or something like that, you'll be in trouble.

If a particular client doesn't need the envelope, it can easily strip it out as soon as the response comes in from the backend.

[Thomas Werner]

Devil's advocate incoming :-D

What would you do in case the payload format doesn't allow for wrapping in an envelope because it cannot be extended like that, such as a paginated slice of a CSV data dump? Or a chunk of a binary data stream?

[panta82]

This is now veering away from web api design, where you would use json/xml over http in 99% of cases. The answer is, it depends.

For csv, I would probably not include metadata at all, but require recipients to obtain it using a separate method. For binary data, you would probably get metadata as part of the binary format itself. But yes, maybe also as headers in the transport protocol.

It depends.

[Adrian B.G.]

X- headers are deprecated since 2012, and I dont think that result logic should exists in the headers anyway, so my bet would be in the response.

[Michiel Hendriks]

RFC 6648: Deprecating the "X-" Prefix and Similar Constructs in Application Protocols

The deprecation of X- is mainly concerning possible standardization of these items. But:

When it is extremely unlikely that some parameters will ever be standardized. In this case, implementation-specific and private-use parameters could at least incorporate the organization's name (e.g., "ExampleInc-foo" or, consistent with [RFC4288], "VND.ExampleInc.foo") or primary domain name (e.g., "com.example.foo" or a Uniform Resource Identifier [RFC3986] such as "example.com/foo"). In rare cases, truly experimental parameters could be given meaningless names such as nonsense words, the output of a hash function, or Universally Unique Identifiers (UUIDs) [RFC4122].

I do think headers are the place to put this information, it always has. There is just the rather time consuming part of figuring out the proper header usage. To simply put an X- in front of the header name is a bad practice (see above RFC for the reasoning.)

What you need to do

1. Find an existing, or work in progress, standard; and use it.
2. Think really really hard if your custom entry is standard worthy.
3. Only then use a vendor prefix. e.g. DevTo-Gizmo

As for the example of this article, there is a standard RFC 5988 to provide referential information in the HTTP headers. Note that they are working on RFC 8288 which will replace it. So it is probably best to follow RFC 8288. (And regularly check back for changes).

[Thomas Werner]

Great explanation, thank you. I will also keep an eye on RFC 8288.

[Thomas Werner]

Thanks for pointing out my mistake! I corrected it. Unfortunately I'm not sure if I can follow why pagination information or follow-up links should not be part of response headers. We already have a lot of meta information in HTTP response headers that describes the content such as content type, language, or content range which is similar to pagination... can you explain why that kind of information doesn't belong there?

[Adrian B.G.]

I see the next page links as other resource identifiers, as business logic and related to that specific request. I see them oposite to metadata like language and encoding, which are to describe the response and API as overall.

Also based on the logic that one client dont use it (infinite scroll) is not valid, you did not remove it just moved the information to another section of the HTTP response.

As for the data naming and common implementation mistakes I say that the API server and client implementations are details that should be automatically generated, I use Open API/swagger generators for that.

[rhymes]

I'd say headers.

This is my favorite implementation, GitHub API V3, using the Link header.

[Erle Carrara]

One issue is that the type of values passed through HTTP headers is always a string.

[Adrian B.G.]

The body is also a string, of a HTTP request and response.

[Erle Carrara]

Yes, body is a raw data that you can serialize to represent values with types (usually according to Content-Type header).

See RFC 723. tools.ietf.org/html/rfc7230#sectio...

Historically, HTTP has allowed field content with text in the ISO-8859-1 charset [ISO-8859-1], supporting other charsets only through use of [RFC2047] encoding. In practice, most HTTP header field values use only a subset of the US-ASCII charset [USASCII]. Newly defined header fields SHOULD limit their field values to US-ASCII octets. A recipient SHOULD treat other octets in field content (obs-text) as opaque data.

The body of http message can be anything you like... But headers not.

[Michiel Hendriks]

Not really. Content-Length for example is an integer.

[Joe Flateau]

It's still a string, a string representation of an integer, but a string none the less.

[Michiel Hendriks]

So is JSON and XML. It's one big string of characters.

[إبراهيم بن عبدالله المقرن]

It's not secure to return a JSON response without wrapping the it in an object (see the update section at the end of the blog) incompleteness.me/blog/2007/03/05/...

For even more haacked.com/archive/2008/11/20/ana...