Where to Put Response Metadata - Envelope or HTTP Headers?

Thomas Iguchi on December 15, 2018

Here's some nitpicky head-scratcher I'm overthinking at the moment. Let's assume we have a RESTful API with resource collection endpoints that al... [Read Full]
markdown guide
 

Headers should only be used by underlying protocol implementation, not business information. Even if client does not use paging, doesn't mean any other implementation will not. Tomorrow if you wish to go old school navigation with pager, then rewriting api will be another pain.

Another important use case is, what if we may want to change protocol from HTTP to something else, we might want to use some other channel, switch from http to web sockets, migration should be easy in fact logic information should be independent of underlying transport.

 

In practice HTTP headers are used for adding meta information beyond the needs for just the transfer layer. We add authorization information, signatures, nonces, timestamps for triggering higher level application logic on the back-end side. And as Michiel Hendriks pointed out, there is even RFC 8288 that explicitly allows us to add web links as HTTP response headers. Reading through the specs, even the one about deprecating X- headers I do not get the impression that we were not supposed to add custom meta information to HTTP response (or request) headers.

Let's take the HTTP status code as an example. It is part of the transfer protocol but expresses success or failure or something else about the result of applying some business logic (object created, object moved, you are not authorized to do that etc.)? Should that also (always) go into an additional payload envelope? (I know there are scenarios where additional error / success information is appropriate in response bodies, and often we like to add the HTTP status code there, too)

Or what about the use of HTTP verbs like GET / POST / PATCH / PUT / DELETE that are part of HTTP and all pertain to business logic. Should that also go into a "client request envelope" just because we might be migrating in the future to a different transport?

 

Status code is useless, back in 80s where error codes were used to indicate error and you had look up errors in table with matching description. We live in world of exceptions where we need to know error message and stacktrace !! We no longer simply display an error message either, we display relevant useful help links to user to troubleshoot further.

Again, authorization is independent of logic and authorization is transport specific. In HTTP you authorize every request, in WebSockets or traditional sockets you only authorize only first request. Also authorization may change based on form of authentication, still it isn't part of logic of individual operation. Transport will either execute logic if it is authorize or it will not at all execute the logic.

And see where users are moving, GraphQL, Firebase, all are additional abstraction over HTTP because HTTP wasn't sufficient. For end user it is more important to open a session, query, get results or exception irrespective of inspecting further headers and trying to investigate underlying implementation. Problem comes when you go mobile and you have very limited control over underlying transport.

 

I would use the envelope. Don't tie your data format to a transport protocol implementation details. If you suddenly decide you want to store and replay data payloads or put it through queue or something like that, you'll be in trouble.

If a particular client doesn't need the envelope, it can easily strip it out as soon as the response comes in from the backend.

 

Devil's advocate incoming :-D

What would you do in case the payload format doesn't allow for wrapping in an envelope because it cannot be extended like that, such as a paginated slice of a CSV data dump? Or a chunk of a binary data stream?

 

This is now veering away from web api design, where you would use json/xml over http in 99% of cases. The answer is, it depends.

For csv, I would probably not include metadata at all, but require recipients to obtain it using a separate method. For binary data, you would probably get metadata as part of the binary format itself. But yes, maybe also as headers in the transport protocol.

It depends.

 

X- headers are deprecated since 2012, and I dont think that result logic should exists in the headers anyway, so my bet would be in the response.

 

RFC 6648: Deprecating the "X-" Prefix and Similar Constructs in Application Protocols

The deprecation of X- is mainly concerning possible standardization of these items. But:

When it is extremely unlikely that some parameters will ever be standardized. In this case, implementation-specific and private-use parameters could at least incorporate the organization's name (e.g., "ExampleInc-foo" or, consistent with [RFC4288], "VND.ExampleInc.foo") or primary domain name (e.g., "com.example.foo" or a Uniform Resource Identifier [RFC3986] such as "example.com/foo"). In rare cases, truly experimental parameters could be given meaningless names such as nonsense words, the output of a hash function, or Universally Unique Identifiers (UUIDs) [RFC4122].

I do think headers are the place to put this information, it always has. There is just the rather time consuming part of figuring out the proper header usage. To simply put an X- in front of the header name is a bad practice (see above RFC for the reasoning.)

What you need to do

  1. Find an existing, or work in progress, standard; and use it.
  2. Think really really hard if your custom entry is standard worthy.
  3. Only then use a vendor prefix. e.g. DevTo-Gizmo

As for the example of this article, there is a standard RFC 5988 to provide referential information in the HTTP headers. Note that they are working on RFC 8288 which will replace it. So it is probably best to follow RFC 8288. (And regularly check back for changes).

 

Great explanation, thank you. I will also keep an eye on RFC 8288.

 

Thanks for pointing out my mistake! I corrected it. Unfortunately I'm not sure if I can follow why pagination information or follow-up links should not be part of response headers. We already have a lot of meta information in HTTP response headers that describes the content such as content type, language, or content range which is similar to pagination... can you explain why that kind of information doesn't belong there?

 

I see the next page links as other resource identifiers, as business logic and related to that specific request. I see them oposite to metadata like language and encoding, which are to describe the response and API as overall.

Also based on the logic that one client dont use it (infinite scroll) is not valid, you did not remove it just moved the information to another section of the HTTP response.

As for the data naming and common implementation mistakes I say that the API server and client implementations are details that should be automatically generated, I use Open API/swagger generators for that.

 

One issue is that the type of values passed through HTTP headers is always a string.

 
 

It's still a string, a string representation of an integer, but a string none the less.

 

The body is also a string, of a HTTP request and response.

 

Yes, body is a raw data that you can serialize to represent values with types (usually according to Content-Type header).

See RFC 723. tools.ietf.org/html/rfc7230#sectio...

Historically, HTTP has allowed field content with text in the ISO-8859-1 charset [ISO-8859-1], supporting other charsets only through use of [RFC2047] encoding. In practice, most HTTP header field values use only a subset of the US-ASCII charset [USASCII]. Newly defined header fields SHOULD limit their field values to US-ASCII octets. A recipient SHOULD treat other octets in field content (obs-text) as opaque data.

The body of http message can be anything you like... But headers not.

 

It's not secure to return a JSON response without wrapping the it in an object (see the update section at the end of the blog) incompleteness.me/blog/2007/03/05/...

For even more haacked.com/archive/2008/11/20/ana...

 

I'd say headers.

This is my favorite implementation, GitHub API V3, using the Link header.

code of conduct - report abuse