Headers should only be used by underlying protocol implementation, not business information. Even if client does not use paging, doesn't mean any other implementation will not. Tomorrow if you wish to go old school navigation with pager, then rewriting api will be another pain.
Another important use case is, what if we may want to change protocol from HTTP to something else, we might want to use some other channel, switch from http to web sockets, migration should be easy in fact logic information should be independent of underlying transport.
In practice HTTP headers are used for adding meta information beyond the needs for just the transfer layer. We add authorization information, signatures, nonces, timestamps for triggering higher level application logic on the back-end side. And as Michiel Hendriks pointed out, there is even RFC 8288 that explicitly allows us to add web links as HTTP response headers. Reading through the specs, even the one about deprecating X- headers I do not get the impression that we were not supposed to add custom meta information to HTTP response (or request) headers.
Let's take the HTTP status code as an example. It is part of the transfer protocol but expresses success or failure or something else about the result of applying some business logic (object created, object moved, you are not authorized to do that etc.)? Should that also (always) go into an additional payload envelope? (I know there are scenarios where additional error / success information is appropriate in response bodies, and often we like to add the HTTP status code there, too)
Or what about the use of HTTP verbs like GET / POST / PATCH / PUT / DELETE that are part of HTTP and all pertain to business logic. Should that also go into a "client request envelope" just because we might be migrating in the future to a different transport?
Status code is useless, back in 80s where error codes were used to indicate error and you had look up errors in table with matching description. We live in world of exceptions where we need to know error message and stacktrace !! We no longer simply display an error message either, we display relevant useful help links to user to troubleshoot further.
Again, authorization is independent of logic and authorization is transport specific. In HTTP you authorize every request, in WebSockets or traditional sockets you only authorize only first request. Also authorization may change based on form of authentication, still it isn't part of logic of individual operation. Transport will either execute logic if it is authorize or it will not at all execute the logic.
And see where users are moving, GraphQL, Firebase, all are additional abstraction over HTTP because HTTP wasn't sufficient. For end user it is more important to open a session, query, get results or exception irrespective of inspecting further headers and trying to investigate underlying implementation. Problem comes when you go mobile and you have very limited control over underlying transport.
We're a place where coders share, stay up-to-date and grow their careers.
We strive for transparency and don't collect excess data.