DEV Community: Evert Pot

Discovering features via HTTP OPTIONS

Evert Pot — Wed, 16 Oct 2024 13:44:00 +0000

Say you have an API, and you want to communicate what sort of things a user can do on a specific endpoint. You can use external description formats like OpenAPI or JSON Schema, but sometimes it’s nice to also dynamically communicate this on the API itself.

OPTIONS is the method used for that. You may know this HTTP method fromCORS, but it’s general purpose is for clients to passively find out ‘What can I do here?’.

All HTTP clients typically support making OPTIONS request. For example withfetch():

const response = await fetch(
  'https://example.org',
  {method: 'OPTIONS'}
);

A basic OPTIONS response might might look like this:

HTTP/1.1 204 No Content
Date: Mon, 23 Sep 2024 02:57:38 GMT
Server: KKachel/1.2
Allow: GET, PUT, POST, DELETE, OPTIONS

Based on the Allow header you can quickly tell which HTTP methods are available at a given endpoint. Many web frameworks emit this automatically and generate the list of methods dynamically per route, so chances are that you get this one for free.

To find out if your server does, try running the command below (with your URL!):

curl -X OPTIONS http://localhost:3000/some/endpoint/

One nice thing you could do with the Allow header, is that you could also communicate access-control information on a very basic level. For example, you could only include DELETE and PUT if a user has write access to a resource.

Accept and Accept-Encoding

There’s server other standard headers for discovery. Here’s an example showing a few at once:

HTTP/1.1 204 No Content
Date: Mon, 23 Sep 2024 02:57:38 GMT
Server: KKachel/1.2
Allow: GET, PUT, POST, DELETE, OPTIONS
Accept: application/vnd.my-company-api+json, application/json, text/html
Accept-Encoding: gzip,brotli,identity

You may already be familiar with Accept and Accept-Encoding from HTTP requests, but they can also appear in responses. Accept in a response lets you tell the client which kind of mimetypes are available at an endpoint. I like adding text/html to every JSON api endpoint and making sure that API urls can be opened in browsers and shared between devs for easy debugging.

The Accept-Encoding lets a client know in this case that they can compress their request bodies with either gzip or brotli (identity means no compression).

Patching, posting and querying

3 other headers that can be used are Accept-Patch, Accept-Postand Accept-Query. These three headers are used to tell a client what content-types are available for the PATCH, POST andQUERY http methods respectively.

For all of these headers, their values effectively dictate what valid values are for the Content-Type header when making the request.

HTTP/1.1 204 No Content
Date: Mon, 23 Sep 2024 02:57:38 GMT
Server: KKachel/1.2
Allow: OPTIONS, QUERY, POST, PATCH
Accept-Patch: application/json-patch+json, application/merge-patch+json
Accept-Query: application/graphql
Accept-Post: multipart/form-data, application/vnd.custom.rpc+json

In the above response, the server indicates it supports both JSON Patchand JSON Merge Patch content-types in PATCH requests. It also suggests that GraphQL can be used via the QUERY method, and for POST it supports both standard file uploads and some custom JSON-based format.

Typically you wouldn’t find all of these at the same endpoint, but I wanted to show a few examples together.

Where’s PUT?

Oddly, there’s no specific header for PUT requests. Arguably you could say that GET and PUT are symmetrical, so perhaps the Accept header kind of extends to both. But the spec is not clear on this.

I think the actual reality is that Accept-Patch was the first header in this category that really clearly defined this as a means of feature discovery on OPTIONS. Accept-Post and Accept-Query followed suit. I thinkAccept-Patch in OPTIONS was modelled after in-the-wild usage of Acceptin OPTIONS, even though the HTTP specific doesn’t super clearly define this.

If I’m wrong with my interpretation here, I would love to know!

Aside: If you’re wondering about DELETE, DELETE should never have a body,
so all a user would need to know is can they delete, which you can see
in the Allow header.
If this is new to you to, read my other article about GET request
bodies. Most of the information there is applicable to DELETE as well.

Linking to documentation

The OPTIONS response is also a great place to tell users where to find additional documentation. In the below example, I included both a machine-readable link to a documentation site, a link to an OpenAPI definition, and a message intended for humans in the response body:

HTTP/1.1 200 OK
Date: Mon, 23 Sep 2024 04:45:38 GMT
Allow: GET, QUERY, OPTIONS
Link: <https://docs.example.org/api/some-endpoint>; rel="service-doc"
Link: <https://api.example.org/openapi.yml>; rel="service-desc" type="application/openapi+yaml"
Content-Type: text/plain

Hey there!

Thanks for checking out this API. You can find the docs for this
specific endpoint at: https://docs.example.org/api/some-endpoint

Cheers,
The dev team

I recommend keeping the response body as mostly informal and minimal any real information should probably just live on its own URL and be linked to.

I used the service-doc and service-desc link relationships here, but you can of course use any of the IANA link relationship types here or a custom one. Also see the Web linking spec for more info.

Obscure uses

WebDAV usage

WebDAV, CalDAV and CardDAV also use OPTIONS for feature discovery. For example:

HTTP/1.1 204 No Content
Date: Mon, 23 Sep 2024 05:01:50 GMT
Allow: GET, PROPFIND, ACL, PROPPATCH, MKCOL, LOCK, UNLOCK
DAV: 1, 2, 3, access-control, addressbook, calendar-access

The server-wide asterisk request

Normally HTTP requests are made to a path on the server, and the first line looks a bit like the following in HTTP/1.1:

GET /path HTTP/1.1

But, there are a few other “request line” formats that are rarely used. One of them lets you discover features available on an entire server, using the asterisk:

OPTIONS * HTTP/1.1

The asterisk here is not a path. Normally asterisks aren’t even allowed in URIs. Many HTTP clients (including fetch()) don’t even support this request.

Classic webservers like Apache and Nginx should support this. To try it out, use CURL

curl -vX OPTIONS --request-target '*' http://example.org

Final notes

If you have a reason to allow clients to discover features on an endpoint, consider using OPTIONS instead of a proprietary approach! As you can see in many of these examples, it’s especially useful if you use mimetypeswell.

New Structured Fields RFC out, and so is my Javascript package

Evert Pot — Sat, 05 Oct 2024 01:55:09 +0000

A new RFC was released for Structured Fields: RFC9651.

What is it?

HTTP headers have been a bit of a free-for all in terms of how complex values
are encoded, with many headers requiring their own mini-parser.

A while back an effort was started to fix this for headers going forward, named 'Structured Fields'. They're called Fields and not 'Headers' because HTTP has both Headers and Trailers!

Structured fields let you encode things like lists, dictionaries, strings, numbers, booleans and binary data. The original RFC from 2021 is pretty successful and although many existing headers can't be retrofitted to this format, a lot of new standards are taking advantage.

Some examples:

// Parsed an ASCII string
Header: "foo"

// A simple string, called a 'Token' in the spec
Header: foo

// Parsed as number
Header: 5
Header: -10
Header: 5.01415

// Parsed into boolean
Header: ?1
Header: ?0

// Binaries are base64 encoded
Header: :RE0gbWUgZm9yIGEgZnJlZSBjb29raWU=:

// Items can have parameters
Header: "Hello world"; a="5"

// A simple list
Header: 5, "foo", bar, ?1

# Each element can have parameters
Header: sometoken; param1; param2=hi, 42

// A list can also contain lists itself. These are called 'inner lists' and
// use parenthesis
Header: sometoken, (innerlistitem1 innerlistitem2), (anotherlist)

// A simple dictionary
Header: fn="evert", ln="pot", coffee=?1

// Each item may have parameters too
Header: foo=123; q=1, bar=123, q=0.5

// A dictionary value may be an inner list again
Header: foo=(1 2 3)

The new RFC published last week adds 2 new data types: Dates and
'Display strings', which is a Unicode serialization that fits in the HTTP header (and trailer) format.

// Parsed into a Date object

Header: @1686634251

// A Unicode string, called a 'Display String' in the spec. They use

// percent encoding, but encode a different set of characters than

// URLs.

Header %"Frysl%C3%A2n"

Why should you care?

If you encounter these headers in the wild, it's a really good idea to use a standard parser. One of the reasons is that with using structured-fields, there's a built-in extension mechanism. You'll want to make sure that when a new parameter appears your application doesn't suddenly break.

You may also want to define and use your own HTTP headers. The structured fields format is a very good 'default choice' that removes decisions such as 'How should I encode a key value object' or 'how do I encode a UTF-8 string'.

With parsers popping up for every language, you don't have to worry about writing your own one-off formats.

Javascript package

I'm the maintainer of a Javascript library for Structured Fields, called "structured headers", which I've also updated for this new RFC. I wish I picked the name "structured-fields", but I picked the name before the original standard changed it's name.

I've just released v2 of this library supporting these new types, and also added ES Modules support.

Comments?

Reply to one of these:

[Mastodon post][4]
[Bluesky post][5]

Taking a look at Mastodon

Evert Pot — Tue, 01 Nov 2022 21:28:20 +0000

I've been a Twitter user and fan since 2007. With Twitter's future looking a bit grim, I started looking around if there's another place to go.

Twitter can't really be replaced with anything else, because everyone's Twitter experience is unique to them and their community. For me, it's the main way I stay in touch with my Open Source / HTTP / API / Hypermedia bubble and some friends. Losing that would suck! Unfortunately, there's no way that group would all flock to another platform.

But for the ones that do try something else, the talk of the town seems to be Mastodon.

Mastodon is interesting. On the surface it might just seem like a Twitter clone, but it's based on a federated protocol called 'ActivityPub'. What this means in practice is that there's no central server. There's many instances. Each of these instances is managed by different people, and many of them focus on specific interests.

With email, it doesn't matter which provider you go with Thanks to universal SMTP standards that every server uses, you can exchange messages with everyone else. This is the same with Mastadon. You're not siloed into a single instance,
and you can follow people from any other instance. Unlike email, it appears that with Mastadon you can actually migrate to different instances if you don't like your current one.

This has some interesting side effects too. I joined the
IndieWeb instance, which is a community I already loved. And even though I'm not siloed in, I get access to a local feed of like-minded people from that community. Everything feels new and more intimate.

Also, instead of one central authority that you have to trust to make the right moderation decisions, you can join one of many that aligns with your values, and you can block entire instances that don't.

So should you join? If you use Twitter to stay on top of news and follow high profile people then probably not. If you're like me, you might be able to find a community that fits your interest.

Will I stick to this? Who knows... but Twitter, like everything before, will fall out of favor one day and I'm enjoying Mastadon's ad-free, open source, developer-friendly experience. Reminds me of early Twitter or mid-2000's blogging culture.

Links

If you're just getting started, check out https://joinmastodon.org/ and find a server.
Follow me!
To find your Twitter friends on Mastadon try Twitodon and Fedifinder.
To automatically cross-post everything from Twitter to Mastodon and vice versa, use Moa.

Lastly, one of the interesting results of Mastodon building on open protocols, is that it allows alternative implementations.

The Microblog.pub project lets you self-host a single-user instance. Instead of joining some large instance, you deploy an instance on your own domain that's just for you. Can't get more control than that, and this might be something I'll consider in the future.

I don't see why this blog couldn't one day also be a 'microblog' and part of the fediverse.

Porting Curveball to Bun

Evert Pot — Tue, 13 Sep 2022 17:02:46 +0000

Bun is the hot new server-side Javascript runtime, in the same category as Node and Deno. Bun uses the JavascriptCore engine from Webkit, unlike Node and Deno which use V8. A big selling point is that
it's coming out faster in a many benchmarks, however the things I'm personally excited about is some of it's quality of life features:

It parses Typescript and JSX by default (but doesn't type check), which means there's no need for a separate 'dist' directory, or a separate tool like ts-node.
It loads .env files by default.
It's compatible with NPM, package.json, and many built-in Node modules.

I also like that it's 'Hello world HTTP server' is as simple as writing this file:

// http.ts
export default {
  port: 3000,
  fetch(request: Request): Promise<Response> {
    return new Response("Hello world!");
  },
};

And then running it with:

bun run http.ts

Bun will recognize that an object with a fetch function was default-exported, and start a server on port 3000. As you can see here, this uses the standard Request and Response objects you use in a browser, and can use async/await.

These are all things that didn't exist when Node and Express were first created, but seem like pretty good ideas for something built today. I don't think using Request and Response are good for more complex use-cases (streaming
responses, 1xx responses, trailers, upgrading to other protocols, getting tcp connection metadata like remoteAddr are some that come to mind), because these objects are designed for clients first.

But in many cases people are just building simple endpoints, and for that it's great.

Bun supports a ton of the standard Node modules, but it's also missing some such as support for server-side websockets and the node http/https/https packages, which for now makes it incompatible with popular frameworks like Express.

Porting Curveball

Curveball is a Typescript micro-framework we’ve been developing since mid-2018 as a modern replacement for Express and Koa. A key difference between
Curveball and these two frameworks is that it fully abstracts and encapsulates the core 'Request' and 'Response' objects Node provides.

This made it very easy to create a lambda integration in the past; instead of mapping to Node's Request and Response types, All I needed was simple mapping function for Lambdas idea of what a request and response looks like.

To get Express to run on AWS Lambda the Node http stack needs to be emulated, or a full-blown HTTP/TCP server needs to be started and proxied to. Each of these workarounds require a ton of code from libraries like serverless-express.

So with Bun up and coming, either the same work would need to be done to emulate Node's APIs, or Bun would would need to add full compability for the Node http module (which is eventually coming).

But because Curveball abstractions, it was relatively easy to get up and running. Most of the work was moving the Node-specific code into a new package.

Here's the Curveball 'hello world' on Bun:

import { Application } from '@curveball/kernel';

const app = new Application();

// Add all your middlewares here! This should look familiar to Koa users.
app.use( ctx => {
  ctx.response.body = {msg: 'hello world!'}; 
});

export default {
  port: 3000,
  fetch: app.fetch.bind(app)
};

It's still a bit experimental, but the following middlewares are tested:

And because JSX also just works in Bun, it's relatively easy to use it to generate HTML:

import { Application } from '@curveball/kernel';
import reactMw from '@curveball/react';
import React from 'react';

const app = new Application();

app.use(reactMw());

app.use( ctx => {

  ctx.response.type = 'text/html; charset=utf-8';
  ctx.response.body = <html>
    <body>
      <div>
        <h1>Hello world!</h1>
      </div>
    </body>
  </html>;

});

export default {
  port: 3000,
  fetch: app.fetch.bind(app)
};

This is not a full-blown hydration/server-rendering solution, but JSX has replaced template engines like EJS and Handlebars for us at Bad Gateway. JSX lets you do the same things, but you get the advantage of type checking, it's harder to create XSS bugs and full Javascript access.

Final thoughts on Bun

Compared to Deno, It was considerably easier to port Curveball to Bun.
Deno was a much greater challenge, due to it having such a radically different idea about packages and module loading. This was over a year ago, so it's worth giving a shot again.

So, I'm curious where all of this goes! Perhaps Bun is the future, or perhaps we'll see Node get parity with Bun and making it obsolete. Either way competition is good.

I feel Bun has a better chance than Deno, because it delivers many of its advantages and features while mostly staying inside with the Node ecosystem.

Although as it turns out Deno also has changed their tune and also made it a new goal to be compatible. I can't help wondering if this was inspired by Bun's recent success as well.

A new OAuth2 client for Javascript

Evert Pot — Mon, 20 Jun 2022 20:14:21 +0000

Frustrated with the lack of well maintained, minimal OAuth2 libraries, I wrote my own. This new OAuth2 library is only 3KB gzipped, mainly because it has 0 dependencies and relies on modern APIs like fetch() and Web Crypto which are built in Node 18 (but it works with Polyfills on Node 14 and 16).

It has support for key features such as:

authorization_code with PKCE support.
password and client_credentials grants.
a fetch() wrapper that automatically adds Bearer tokens and refreshes them.
OAuth2 endpoint discovery via the Server metadata document (RFC8414).
OAuth2 Token Introspection (RFC7662).

If your server does support the meta-data document, here's how simple the process can be:

client_credentials example

const { OAuth2Client } from '@badgateway/oauth2-client';

const client = new Client({
  clientId: '..',
  clientSecret: '..',
  server: 'https://my-auth-server.example'
});

const tokens = await client.clientCredentials();

Without the meta-data document, you will need to specify settings such as the tokenEndpoint and possibly the authorizationEndpoint depending on which flow you are using.

authorization_code example

The authorization_code flow is a multi-step process, so a bit more involved.
The library gives you direct access to the primitives, allowing you to integrate in your own frameworks and applications.

import { OAuth2Client, generateCodeVerifier } from '@badgateway/oauth2-client';

const client = new OAuth2Client({
  server: 'https://authserver.example/',
  clientId: '...',
});

// Part of PCKE
const codeVerifier = await generateCodeVerifier();

// In a browser this might work as follows:
document.location = await client.authorizationCode.getAuthorizeUri({
  redirectUri: 'https://my-app.example/',
  state: 'some-string',
  codeVerifier,
  scope: ['scope1', 'scope2'],
});

Handling the redirect back

const oauth2Token = await client.authorizationCode.getTokenFromCodeRedirect(
  document.location,
  {
    redirectUri: 'https://my-app.example/',
    state: 'some-string',
    codeVerifier,
  }
);

const oauth2Token = await authorizationCode.getToken(codeResponse);

Docs and download

Github
npmjs

Request bodies in GET requests

Evert Pot — Sat, 29 Jan 2022 19:14:00 +0000

12 years ago I asked on Stack Overflow: Are HTTP GET requests allowed to have request bodies?. This got a 2626 upvotes and a whopping 1.6 million views, so clearly it’s something lots of people are still curious about, and in some cases disagree with the accepted answer.

Because it keeps popping up in my Stack Overflow notifications (and I compulsively visit the site), the question has lived in my head rent-free. I keep adding context in my head, and I’ve been meaning to write some of this down for a few years now and hopefully evict it.

Anyway, if you’re just looking for a quick answer, it’s ‘No, you shouldn’t do this.’, you should probably use QUERY.

Undefined behavior

A number of people (most famously ElasticSearch) have gotten this wrong, but why? I think it’s because of this sentence in the HTTP Spec:

A payload within a GET request message has no defined semantics

That sentence could easily suggest that there’s no specific behavior associated to request bodies with GET requests, and that the behavior is left up to the implementor.

The reality is that this is more like Undefined behavior from languages like C/C++. My understanding is that leaving certain aspects of the C language undefined (instead of for example requiring an error to be thrown) leaves room for compiler implementations to make certain optimizations. Some compilers also have fun with this; GCC hilariously starts a video game in a specific case of undefined behavior which really brings home this point.

If you were to write a C program that relies on how a compiler dealt with specific undefined behavior, it means your program is no longer a portable C program, but it’s written in variant of C that only works on some compilers.

The same applies for HTTP as well. It’s true that undefined behavior means that you as a server developer can define it, but you are not an island!

When working with HTTP, there’s servers but also load balancers, proxies, browsers and other clients that all need to work together. The behavior isn’t just undefined server-side, a load balancer might choose to silently drop bodies or throw errors. There’s many real-world examples of this. fetch()for example will throw an error.

This hasn’t stopped people from doing this anyway. OpenAPI removedsupport for describing GET request bodies in version 3.0 (and DELETE, which has the same issue!), but was quitely added back in 3.1 to not prevent people from documenting their arguably broken APIs.

Why it’s not defined

The best source I have is this quote from Roy Fielding in 2007. Roy Fielding coined REST is and is one of the main authors of the HTTP/1.1 RFCs.

Yes. In other words, any HTTP request message is allowed to contain a message body, and thus must parse messages with that in mind. Server semantics for GET, however, are restricted such that a body, if any, has no semantic meaning to the request. The requirements on parsing are separate from the requirements on method semantics. So, yes, you can send a body with GET, and no, it is never useful to do so. This is part of the layered design of HTTP/1.1 that will become clear again once the spec is partitioned (work in progress).

….Roy

(His message was originally sent to the now-dead rest-discuss
group on Yahoo Groups, but I found an archive in JSON format)

However, I always found this answer unsatisfying. I understand that you might want a low-level protocol design that just passes messages containing headers, bodies, urls, methods and statuses and not concern itself with method-specific behavior.

That design goal doesn’t preclude GET from specifically rejecting request bodies though, and it also doesn’t preclude the spec from mentioning that request bodies should not be emitted.

So, unless there’s a different reason for this I’m going to have to assume that this was well-intentioned but just not written clearly enough.

Despite this explanation from one of the authors of the HTTP spec itself, I noticed some people will still claim a different interpretation, making a sort of a ‘death of the author’ argument. It’s a bit funny to think about ‘death of the author’ in technical specs, but I think they have a point. I believe that the definition of a standard such as HTTP is not the written RFC, it’s how everyone actually implements it.

The goal of the spec is to accurately describe the standard, not to define it. If what everyone is doing is different from the author’s intention, the spec has failed to accurately describe the standard, not vice versa.

The team behind the HTTP specs believes this too, which is why we’ve seen new releases of HTTP/1.1 without increasing the version. In each iteration major steps are taken to capture real-world usage, sometimes even in conflict with the original RFC2616.

However, in this case it’s irrelevant. Many popular HTTP implementations will throw errors when seeing GET bodies, such as fetch(). Using GETbodies will give you such poor interopability that it’s worth continuing to not do this.

In 2019 I’ve opened a ticket to request to fix this, and they listened. The upcoming HTTP/1.1 is going to include the following text:

Although request message framing is independent of the method used, content received in a GET request has no generally defined semantics, cannot alter the meaning or target of the request, and might lead some implementations to reject the request and close the connection because of its potential as a request smuggling attack (Section 11.2 of HTTP/1.1).

A client SHOULD NOT generate content in a GET request unless it is made directly to an origin server that has previously indicated, in or out of band, that such a request has a purpose and will be adequately supported.>An origin server SHOULD NOT rely on private agreements to receive content, since participants in HTTP communication are often unaware of intermediaries along the request chain.

Source.

Why should `GET` have a body?

I think we feel this way, because we’ve been told over and over again that it’s a good idea to use protocols in a ‘semantically correct’ way.

But why should be care about semantics? 2 reasons:

If you do things in a semantically correct way, other systems can make assumptions about how it should behave. For example: A PUT request may be automatically repeated if it failed the first time due to a network failure or 5xx error. A GET request may be cached if it contained aCache-Control header.
It’s self-documenting behavior. If you see a GET request it tells a developer something is being retrieved.

So while #1 doesn’t really apply here (there’s no technical advantages, because it’s undefined behavior), but #2 makes sense.

So if we decided to use GET for our complex search, what can we do? Body is obviously not allowed; our only options are headers and the URL. There’s a number of issues with this: Encoding UTF-8 is unclear, no real support for documents/mine-types, length limitations. It’s a bit of a mess!

Why was it designed not to support bodies?

Most developers working with HTTP and learning about it is in the context of APIs, but this is not the original use-case.

It all starts with HTML and browsers and URLS. The world wide web wasdesigned as a distributed hypertext document system, where every document has a global unique identifier called a URL.

To retrieve a hypertext document, you would do a GET on this url, to replace or create the document a PUT and to remove the document DELETE. (Note that it took longer to get a writable web, but I wanted to
illustrate what I think the intended design is without being encumbered
by facts).

Taken from this perspective, the idea of ‘parameters’ makes less sense.GET doesn’t just mean “Do a safe request and read something from the server”, it means: ‘Give me the document with this name’.

This specific feature is in part why the web was so successful. Given that the URL comprehensively describes to a client exactly how to connect to a server and retrieve a document, it made the <a> HTML work. It also let people make bookmarks, share links via email/chat and paint it on store-fronts.

The URL is the web’s superstar and killer feature. HTTP just let you find documents associated with the URL. HTTP/0.9 didn’t even have headers and can be read during a bathroom break.

Our needs have evolved, has HTTP?

I still maintain that even if you’re building a REST api, it’s good to think of your endpoints as documents, and the GET, PUT and DELETE as the most important HTTP methods to manipulate these, using the URL as the primary key.

But, sometimes we need to do something more complicated and it would be nice if HTTP had a prescribed way to handle cases where we want to describe a search query as a document.

The answer traditionally was to just use POST. POST is kind of your no-guarantees catch all method. Often also used to tunnel other protocols that don’t really speak HTTP well, such as SOAP, XMLRPC and more recently GraphQL.

But there’s a new contender. Now I think your best option is QUERY. This is a new method, that’s not quite standard yet but any compliant HTTP client and server should already support it. The goal of QUERY is specifically to solve this use-case.

QUERY solves the following problems:

It’s basically a ‘safe’ POST. This means the implication is that it’s for reading and safe to repeat.
It also is self-descriptive. It tells a developer that this is a API that’s focused on reading.
The spec also makes it cachable.

I think you should still continue the use GET for the majority of cases, but there’s a clear answer now for what to do when that doesn’t fit.

Anyway…

That’s a lot of writing for what seemingly is a simple question. It’s just one of those things that’s been bouncing around my head for a bit too long.

I hope it’s out of my system!

Big thank you to everyone who originally answered my SO question.

JWT should not be your default for sessions

Evert Pot — Fri, 14 May 2021 01:30:27 +0000

Cookies

When designing web applications, (especially the traditional HTML kind), you will at one point have to figure out how to log a user in and keep them logged in between requests.

The core mechanism we use to achieve this are cookies. Cookies are small strings sent by a server to a client. After a client receives this string, it will repeat this in subsequent requests. We could store a 'user id' in a cookie, and for any future requests we'll know what user_id the client was.

Cookie: USER_ID=123

But this is very insecure. The information lives in the browser, which means users can change USER_ID and be identified as a different user.

Sessions

The traditional way to solve this is what's known as a 'session'. I don't know what the earliest usage of sessions is, but it's in every web framework, and has been since web frameworks are a thing.

Often, sessions and cookies are described as 2 different things, but they're really not. A session needs a cookie to work.

Cookie: MY_SESSION_ID=WW91IGdvdCBtZS4gRE0gbWUgb24gdHdpdHRlciBmb3IgYSBmcmVlIGNvb2tpZQ

Instead of a predictable user id, we're sending the client a completely random session id that is impossibly hard to guess. The ID has no further meaning, and doesn't decode to anything. This is sometimes called an opaque token.

When a client repeats this session id back to the server, the server will look up the id in (for example) a database, which links it back to the user id. When a user wants to log out, the session id is removed from the data storage, which means the cookie is no longer associated with a user.

Where is the session data stored?

Languages like PHP have a storage system for this built in, and will by default store data in the local filesystem. In the Node.js ecosystem, by default this data will be in 'memory' and disappear after the server restarts.

These approaches make sense on developer machines, or when sites were hosted on long-lived bare-metal servers, but these days a deploy typically means a completely fresh 'system', so this information needs to be stored in a place that outlives the server. An easy choice is a database, but it's common for sites to use systems like Redis and Memcached, which works for tiny sites, but still works at massive scales.

Encrypted token

Over 10 years ago, I started working a bit more with OAuth v1 and similar authentication systems, and I wondered if we could just store all the information in the cookie and cryptographically sign it:

Despite getting some good answers, I didn't go through with it as I didn't feel confident enough in making this secure, and I felt it required a better understanding in crypto than I did.

A few years later, we got JWT, and it's hot shit! JWT itself is a standard for encrypting/signing JSON objects and it's used a LOT for authentication. Instead of an opaque token in a cookie, we actually embed the user_id again, but we include a signature. The signature can only be generated by the server, and it's calculated using a 'secret' and the actual data in the cookie.

This means that if the data is tampered with (the user_id was changed), the signature no longer matches.

So why is this useful? The best answer I have for this, is that it's not needed to have a system for session data, like Redis or a database. All the information is contained in the JWT, it means your infrastructure is in theory simpler. You're potentially making fewer calls to a data-store on a per-request basis.

Drawbacks

There are major drawbacks to using JWT.

First, it's a complicated standard and users are prone to get the settings wrong. If the settings are wrong, in the worst case it could mean that anyone can generate valid JWTs and impersonate anyone else. This is not a beginners-level problem either, last year Auth0 had a bug in one of
their products that had this very problem.

Auth0 is(or was? they just got acquired) a major vendor for security products, and ironically sponsor the jwt.io website. If they're not safe, what chance does the general (developer) public have?

However, this issue is part of a larger reason why many security experts dislike JWT: it has a ton of features and a very large scope, which gives it a large surface area for potential mistakes, by either library authors or users of those libraries. (alternative stateless tokens to JWT exists, and some of them do solve this.)

A second issue is 'logging out'. With traditional sessions, you can just remove the session token from your session storage, which is effectively enough to 'invalidate' the session.

With JWT and other stateless token this is not possible. We can't remove the token, because it's self-contained and there's no central authority that can invalidate them.

This is typically solved in three ways:

The tokens are made very short lived. For example, 5 minutes. Before the 5 minutes are over, we generate a new one. (often using a separate refresh token).
Maintain a system that has a list of recently expired tokens.
There is no server-driven log out, and the assumption is that the client can delete their own tokens.

Good systems will typically use the first two. An important thing to point out is, in order to support logout, you'll likely still need a centralized storage mechanism (for refresh tokens, revocation lists or both), which is the very
thing that JWT were supposed to 'solve'.

Sidenote: some people like JWT because it's fewer systems to hit per request, but that contradicts with being able to revoke tokens before they expire.

My favourite solution to this is keep a global list of JWTs that have been revoked before they expired (and remove the tokens after expiry). Instead of letting webservers hit a server to get this list, push the list to each server using a pub/sub mechanism.

Revoking tokens is important for security, but rare. Realistically this list is small and easily fits into memory. This largely solves the logout issue.

A last issue with JWT is that they are relatively big, and when used in cookies it adds a lot of per-request overhead.

All in all, that's a lot of drawbacks just to avoid a central session store. It's not my opinion that JWT are universally a bad idea or without benefits, but there is a lot to consider.

Why are they popular?

One thing that's surprised me when reading tech blogs, is that there is a lot of chatter around JWT. Especially on Medium and subreddits like /r/node I see intros to JWT on an extremely regular basis.

I realize that that doesn't mean that 'JWTs are more popular than session tokens', for the same reason that GraphQL isn't more popular than REST, or NoSQL than relational databases: it's just not that interesting to write about the technology that's been tried and tested for a decade or more (See: Appeal to novelty). In addition, subject experts that write about new solutions are likely going to have different problems & scale than the majority of their own readers.

However, these new technologies create a lot more buzz than their simpler counterparts, and if enough people keep talking about the hot thing, eventually this can translate to actual adoption, despite it being a sub-optimal choice for the majority of simple use-cases.

This is similar to many newer developers learning how to build SPAs with React before server-rendered HTML. Experienced devs would likely feel that server-rendered HTML should probably be your default choice, and building an SPA when needed, but this is not what new developers are typically taught.

Adopting complex systems before the simple option is considered is something I see happening more, but it surprised me for JWT.

As an exercise, I looked up the top most popular (by votes) posts on /r/node that mention JWT. I was going to go over the first 100, but got bored after the top 12.

From these 12 articles and Github repos:

1 mentions using a revocation list, 3 mention refesh tokens. The remaining articles and github repositories simply have no means of logging out.
1 article mentions that it might be better to use a standard session storage instead.
1 article uses both a standard session storage and JWT, making JWT unneeded.
1 github repository ships with pre-generated private keys. (yup)
Most of posts use expiry times of weeks or months and 3 posts never expire their JWTs.

Except 1, the quality of these highly upvoted posts was extremely low, the authors were likely not qualified to write about this and can potentially cause real-world harm.

All this at least confirmed my bias that JWT for security tokens is hard to get right.

On JWT and scale

Going through tons of Reddit posts and comments also got me a more refined idea of why people think JWTs are better. The top reason everywhere is: "It's more scalable", but it's not obvious at what scale people think they'll start to have issues. I believe the point where issues start to appear is likely much higher than is assumed.

Most of us aren't Facebook, but even at 'millions of active sessions', a distributed key->value store is unlikely going to buckle.

Statistically, most of us are building applications that won't make a Raspberry Pi break a sweat.

Conclusion

Using JWTs for tokens add some neat properties and make it possible in some cases for your services to be stateless, which can be desirable property in some architectures.

Adopting them comes with drawbacks. You either forego revocation, or you need to have infrastructure in place that be way more complex than simply adopting a session store and opaque tokens.

My point in all this is not to discourage the use of JWT in general, but be deliberate and careful when you do. Be aware of both the security and functionality trade-offs and pitfalls. Keep it out of your 'boilerplates' and templates, and don't make it the default choice.

Acknowledgements

Thanks to Nick Chang-Fong and Dominik Zogg for providing feedback and suggestions to this article!

Ketting 7 released

Evert Pot — Mon, 12 Apr 2021 16:30:00 +0000

We just released version 7 of Ketting. Ketting is a generic HATEOAS client for Javascript.

A whole bunch of features have been added since September. We've been testing Ketting 7 in beta since January, so I'm excited to get this out the door.

I've been working on this project since 2016. Normally, I would expect a project like this to get a little stale. For me personally, the opposite has been true and using Ketting (especially with React) is starting to feel a bit like a paradigm shift.

Read on to see what's new!

What is Ketting?

In short: Ketting is a generic REST client for Javascript. You can use it for pushing JSON objects via HTTP, but the richer your API is in terms of best practices and standard formats, the more it can automatically do for you.

It has support for Hypermedia formats such as HAL, Siren, Collection+JSON, JSON:API and can even understand and follow links from HTML.

In the past it was not uncommon to hear that HATEOAS is lacking a good generic client. If you are a Javascript/Typescript user this is no longer true.

More information can be found on the Github page.

What's new?

Better HAL-Forms support

HAL-Forms is an extension of HAL, and adds support for 'actions' or 'forms', similar to what the <form> tag is to HTML.

Since the start of the year HAL-Forms has seen major updates, which was a collaborative effort by several people from projects in the HAL community (including Spring HATEOAS and yours truly) and lead by it's author Mike Amudsen.

Spring HATEOAS released its HAL-Forms updates in version 1.3 M2 (unclear if this is a stable release or not), and Ketting follows today.

Major new features in HAL-Forms include:

Support for lookups
- Example use-case is rendering dropdowns/comboboxes.
- The list of possible options can either be provided in-line or via an external resource.
- JSON and text/csv is support for external resources.
Support for most of the HTML5 input types, such as checkbox, color, tel, email, textarea, etc.
Support for many of the field attributes that also exist in HTML5 form fields, such as placeholder, min, max, step, cols, rows, and others.
Support for a form target. Previously this could also be supplied via the URI.
Support for multiple forms per document.

React bindings: `<RequireLogin>`

(note: all of the new react-ketting features were backported to Ketting 6)

react-ketting now has a RequireLogin component, that can be use to handle the OAuth2 authorization_code flow in React applications.

Example usage:

function MyApp() {

  return <RequireLogin
    clientId="my-oauth2-client-id"
    authorizeEndpoint="https://..."
    tokenEndpoint="https://..">

    You are now logged in!
  </RequireLogin>;

}

React-bindings: `useCollection`

A useCollection hook was added to easily render a collection on a server.

Example usage:

import { useCollection, useResource } from 'react-ketting';
import { Resource } from 'resource';

type Article = {
  title: string;
  body: string;
}

function ArticleList() {

  const { loading, items } = useCollection<Article>('/articles');

  if (loading) return <div>Loading...</div>;

  return <section>
    <h1>Articles!</h1>

    {items.map( item => <ArticleItem key={item.uri} resource={item} /> ) }
  </section>;

}

function ArticleItem({resource}: { resource: Resource<Article> }) {

  const { loading, data } = useResource(resource);

  if (loading) return <div>Loading...</div>;

  return <div>
    <h2>{data.title}</h2>
    <p>{data.body}
  </div>

}

React-bindings: `refreshOnStale`

Both useResource and useCollection got a refreshOnStale flag, that will cause Ketting to automatically ask the server for a new resource state if the cache is marked stale for a resource.

This can have a pretty magical effect when you (for example) use a POST request on a collection to add a new member.

If you also used a useCollection hook on the same page to show the collection, that collection will automatically refresh it's own list.

The first fetch of useCollection will include a Prefer-Transclude HTTP header, telling the user to (ideally) embed every item of the collection in the response, but subsequent refreshes will not.

This means the first time we only need 1 HTTP request, but for refreshes only the collection itself (and not it's members) need to be returned.

If your 'create' operation also returned Content-Location, you can remove one more HTTP request from that list.

Similarly this can be used for DELETE requests of members of the collection, as long as your response includes Link: </parent-collection>; rel="invalidates", the collection will also automatically re-render with the deleted item removed.

For one application we took this a step further, and used Websockets to emit 'stale' events from the server. With virtually no modifications to the frontend, we were able to turn a single-user web application into an application that
could reflect changes in real-time from other users who were using the application at the same time. This really felt like an emerging property of a well designed system (standing on the shoulders of decades of Hypermedia, HTTP and REST research).

Personally I'm very hyped about this feature, and I can't wait to demo this on a meetups or a conferences (if my talk proposals ever get accepted!?).

Deprecation warnings

Ketting 7 will now emit warnings when it encounters a Deprecation or Sunset header, or when a link contains the status: "deprecated" hint.

For more information about this feature, read my previous article about this feature.

Removed support for Prefer-Push

HTTP/2 Push support in browsers is effectively dead. To reduce drag, I've removed the Prefer-Push feature from Ketting.

Smarter caching of newly created resoures.

If you use Ketting to create a new resource (for example with POST), and the server returns a Content-Location header in its response, it will store the response body with the new URI in it's cache.

This can potentially reduce roundtrips. Content-Location is a way for a server to say: 'The response body is the representation of the resource this URI'.

This is another great example of a HTTP caching feature in Ketting that goes beyond what Web Browsers typically do.

Other examples of this is being able to tease apart transcluded/embedded responses, allowing servers to invalidate caches for entries with an invalidates link and exposing cache-related events to the user.

`State` objects now have a `.follow()` and `.followAll()` function.

A State object is returned when you (for example) call resource.get(), resource.patch(), etc.

This object represents an 'Entity' or 'State' returned from the server, which really is a fancy way of saying 'the body' + headers that directly pertain to the body.

It also provides direct access to hypermedia features such as links and actions. The new addition lets you follow hypermedia links straight from any HTTP response.

const response = await myResource.post({
  data: {
     param1: 'value1'
  }
});

// If response contained a HAL, Siren, HTML link or HTTP Link header,
// we can follow it!
const newResource = response.follow('more-info');

Upgrading

Upgrading should be relatively pain-free for most users, except if you extend Ketting with custom format parsers.

If you run:

npm i ketting@7
npm i react-ketting@2 # React users only

And typescript doesn't complain, chances are that everything will work just
as before.

Ketting 7 has been in development and used by us in production since January.

Future plans

Long-term Ketting plans include

Better documentation and educational resources.
More React support, including a library of components for automatically rendering Hypermedia Forms/Actions and automatic paging of collections.
Bindings to other frontend-frameworks.
An official Websocket add-on, to enable real-time multiuser collaboration and bi-direction state updates.

A generic middleware pattern in Typescript

Evert Pot — Thu, 17 Dec 2020 08:50:21 +0000

I just realized this is the third time I'm writing an async middleware invoker, I thought I would share the generic pattern for the benefit of others.

I'm not sure if this is interesting enough for a NPM package, so I'll leave it here for inspiration.

The specific middleware pattern I am implementing is also used Curveball. (the one here is just a bit simpler).

We're working off a context, and we are running a chain of middlewares in order with this context as an argument.

We're also passing an next function. If this next function is called, the next middleware in the list will be called. If not, the chain will bebroken.

Furthermore, (unlike Express, but like Koa) middlewares can be async function or return a promise. If it is, we want to await it.

The setup

Lets start with the setup, describing the middleware:

/**
 * 'next' function, passed to a middleware
 */
type Next = () => void | Promise<void>;

/**
 * A middleware
 */
type Middleware<T> =
  (context: T, next: Next) => Promise<void> | void;

Middleware is the actual async/non-async middleware function. I made a
type for Next so I don't need to write it out more than once.

How we want to use it

This would be the 'getting started' section of the documentation.

The idea here is that we have an 'app', a set of middlewares and a context
we want to operate on.

The following code would be written by the user of this framework:

/**
 * The context type of the application.
 *
 * In 'koa' this object would hold a reference to the 'request' and 'response'
 * But our context just has a single property.
 */
type MyContext = {
  a: number;
}

/**
 * Creating the application object
 */
const app = new MwDispatcher<MyContext>();

/**
 * A middleware
 */
app.use((context: MyContext, next: Next) => {

  context.a += 1;
  return next();

});

/**
 * An async middleware
 */
app.use(async (context: MyContext, next: Next) => {

  // wait 2 seconds
  await new Promise(res => setTimeout(res, 2000));
  context.a += 2;
  return next();

});

Running this application

const context: MyContext = {
  a: 0,
}

await app.dispatch(context);
console.log(context.a); // should emit 3

The implementation

Making this all work is surprisingly terse:

/**
 * A middleware container and invoker
 */ 
class MwDispatcher<T> {

  middlewares: Middleware<T>[];

  constructor() {
    this.middlewares = [];
  }

  /**
   * Add a middleware function.
   */
  use(...mw: Middleware<T>[]): void {

    this.middlewares.push(...mw);

  }

  /**
   * Execute the chain of middlewares, in the order they were added on a
   * given Context. 
   */
  dispatch(context: T): Promise<void> {
     return invokeMiddlewares(context, this.middlewares)
  }

}

/**
 * Helper function for invoking a chain of middlewares on a context.
 */
async function invokeMiddlewares<T>(context: T, middlewares: Middleware<T>[]): Promise<void> {

  if (!middlewares.length) return;

  const mw = middlewares[0];

  return mw(context, async () => {
    await invokeMiddlewares(context, middlewares.slice(1));
  })

}

HTTP/2 Push is dead

Evert Pot — Thu, 12 Nov 2020 21:05:31 +0000

One of the hot features that came with HTTP/2 was PUSH frames. The main idea is that if the server can predict what requests a client might want to make, the server can preemptively send request/response pairs to the client and warm it's cache.

This is a feature I've been very interested in for a long time. I feel that it can be incredibly useful for APIs to invalidate & warm caches, remove the need for compound requests (which I feel is a hack, although sometimes a
necessary one).

To help advance this idea, I've worked on a Internet Draft to let API clients tell the server what resources they would like to have pushed, I built a Node.js framework with first-class, deeply integrated Push support, and added support for Prefer-Push to Ketting.

Unfortunately HTTP/2 push always felt like feature that wasn't quite there yet. It's usefulness was stunted due to Cache-Digest for HTTP/2 being killed off, and no browser APIs to hook into push events.

The Chrome team has considered removing Push support since at least 2018. The main reason being that they hadn't really seen the performance benefit for pushing static assets. At the time, I tried to defend the feature for the API use-case.

Yesterday, the Chrome team announced to remove the feature from their HTTP/2 and HTTP/3 protocol implementations.

I'm a little sad that it never got to its full potential, but with this step, I no longer think it's worthwhile to invest in this feature. So I'm ceasing my work on Prefer-Push, and will also remove support from the next major Ketting version.

On a positive note, I spent a lot of time thinking and working on this, but it is sometimes nice to just be able to close a chapter, rather than to wait and let it sizzle out. It's not an ideal conclusion, but it's a conclusion
nonetheless.

Current alternatives

Lacking server-driver push, we're back to many small HTTP request, or compound requests. This means you either have the N+1 problem, or (in the case of compound requests) poor integration with HTTP caches. More on this here.

If you're going the 'compound request' route, there is a draft of a similar header as Prefer-Push; Prefer: transclude, which Ketting also supports.

A feature that I hoped would work well in the future with Server Push was server-initiated cache invalidation. We never quite got that. To work around this, we use Websockets and will keep doing this for the forseeable future.

To reduce general latency of fetching many things, the 103 Early Hints can help, although this is not supported yet in Chrome, and this is also only really useful for speeding up delivery of assets like images, css and
Javascript as there's no way to hook into 1xx responses programmatically in a browser.

Apollo-like hooks for REST APIs with React and Ketting

Evert Pot — Sat, 03 Oct 2020 20:46:09 +0000

A bit ago we released Ketting 6. This is the accumulation of about a year of learning on how to better integrate REST APIs with frontend frameworks, in particular React.

It's packed with new features such as local state management, new caching strategies, (client-side) middleware support and change events. It's also the first release that has some larger BC breaks to make this all work.

Releasing Ketting 6 is a big personal milestone for me, and I'm really excited to unleash it to the world and see what people do with. A big thank you to all the people that beta tested this in the last several months!

What's Ketting?

It has support for Hypermedia formats such as HAL, Siren, Collection+JSON, JSON:API and can even understand and follow links from HTML.

It's often said that REST (and Hypermedia APIs) is lacking a good generic client. GraphQL has a lot of advantages, but a major one is tooling. Ketting aims to close that gap.

More information can be found on Github.

React support in Ketting 6

Ketting now has a separate react-ketting package that provides React bindings to Ketting. These bindings should look very familiar if you've used Apollo Client in the past.

Lets dive into an example:

Lets assume you have a REST api that has an 'article' endpoint. This returns something like:

{
  "title": "Hello world",
  "body": "..."
}

This article is retrieved with GET, and updated with PUT, this is how you would display it:

import { useResource } from 'react-ketting';


export function Article() {

  const { loading, error, data } = useResource('https://api.example/article/5');

  if (loading) {
    return <div>Loading...</div>;
  }

  if (error) {
    return <div class="error">{error.message}</div>;
  }

  return <article>
    <h1>{data.title}</h1>
    <p>{data.body}</p>
  </article>;

}

But what about mutations? Unlike GraphQL, mutations in REST APIs often have the same format for GET and PUT, so sending an updated resource to a server often just means mutating your 'data' and sending it back.

The following example would allow a user to edit an article and send it back:

import { useResource } from 'react-ketting';


export function Article() {

  const { loading, error, data, setData, submit } = 
    useResource('https://api.example/article/5');

  if (loading) {
    return <div>Loading...</div>;
  }

  if (error) {
    return <div class="error">{error.message}</div>;
  }

  const changeTitle = (title) => {
    data.title = title;
    setData(data);
  };
  const changeBody = (body) => {
    data.body = body;
    setData(data);
  };

  return <form onsubmit={() => submit}>
    <input type="text" value={data.title} onChange={ev => changeTitle(ev.target.value)  />
    <textarea onChange={ev => changeBody(ev.target.value)}>{data.body}</textarea>
    <button type="submit">Save!</button>
  </form>;

}

Whenever setData is called, the internal Ketting cache is updated with the new resource state. This is globally stored, based on the uri of the resource.

This means that if multiple components use useResource on that same uri, every component will see that update and triggere a re-render.

This is similar to Apollo's local state, except it's still bound to a single resource uri and can eventually be saved.

When submit() is called, the data is re-serialized and sent in a PUT request.

Non-exhaustive list of other changes

Multiple cache strategies, such as forever, short and never.
Support for fetch-middlewares. OAuth2 is reimplemented as such a plugin. These plugins can be added globally, or per-origin.
get() now returns a State object, and functions such as put() will require one as an argument.
put() now automatically updates the state cache.
Support for HEAD requests and following links from HEAD response headers.
PKCE support for OAuth2.
Links can now be mutated and sent back to the server.
Nested transcluded items/embeds.
A separate post() and postFollow() method.
Better support for binary responses and text/* responses.
Experimental: Support for Siren actions, HAL forms and submitting HTML forms.

Future plans

The next two things we are working on are:

Support for more hooks/components for common React/REST API patterns (tell us what you want!).
Deeper support for forms and actions from HAL Forms, Siren and HTML.
Websocket support for live server-initiated state pushes.

ECMAScript 4: The missing version

Evert Pot — Thu, 28 May 2020 09:41:38 +0000

In your build tools, you may have noticed that you have an ECMAScript target, and 5 and up, but never a 4. Why is that?

I thought it would be fun to dive into ECMAScript 4 a bit and see what we didn't get.

A brief history

According to Wikipedia, the first draft of ECMAScript 4 was dated February 1999. The original target for completion was August 2008.

ECMAScript 4 was very ambitious, and added a ton of features that were
perceived as important and missing from ECMAScript 3. It also 'fixed' a
number of things in the previous version, making it backwards incompatible in various ways.

ES4 was met with a bunch of controversies, and lacked sufficient support from browser vendors to be released and was ultimately abandoned.

In 2008 the standard was pronounced dead, and ES3.1 was renamed to ES5, which was a much more conservative and incremental update to ECMAScript.

The closest thing we had for ES4, was probably Flash ActionScript 3. There was a point during the release of AS3 that some of us thought that Flash and the Web was eventually going to converge.

For more details on politics and history of ES4, check out this great article on the auth0 blog.

What could have been?

Classes

Classes eventually landed in ES6, but here's how it might have looked like earlier:

class C {

 var val
 var number = 500;

 const pi = 3.14

 // A function
 function f(n) { return n+val*2 }

 // Getters and setters
 function set foo(n) {
   val = n;
 }

 function get foo() {
   return val;
 }
}

The syntax here is pretty different, but another notable is that these classes had properties and constants. Field declarations are currently 'experimental', so we almost caught up here.

Another surprising thing is that there is no this. Instead of variables being global by default, ES4 would first look in the class scope before checking higher scopes.

ES4 also had the following keywords for class members:

static
final
private, protected, public.
prototype, to define class members on its prototype. Not sure what the use-case is, but it's there.

Interfaces

ES4 introduced interfaces, which is something we don't have today (unless you use Typescript):

interface MyInterface {
  function foo();
}

Strict typing

ES4 introduced strict typing:

function add(a: int, b:int): int {
  return a + b;
}

It also had the type keyword similar to Typescript and union types. A typescript union like the following:

let a : number | string;

Is written as follows in ES4:

var a: (number, string)

ES4 also had generics:

class Wrapper<T> {
  inner: T
}

Like 👍

By default types in ES4 had to be exact types, and not a superset. Using the like keyword you can make this less restrictive:

function getCell(coords: like { x: int, y: int }) {

}

This probably exists because in ES4 types were Nominal and not Structural like
Typescript.

New types

In current ES we have booleans, objects, arrays, number, BigInt, but ES4 was going to introduce:

byte
int
unit
double
decimal

Of those, only the decimal type is in the planning today, and it will eventually probably look like:

const allowance = 1.50m

This m suffix also existed in ES4, and stands for "money".

triple-quoted strings.

To encode a string like: Hello my name is "Evert" in ES4, you could use triple-quotes:

const hi = """Hello my name is "Evert"""";

Packages

Packages are a bit like what we have now with modules. Packages can be
imported, but unlike ES6 modules, namespaces are more like a global naming system.

If a class is defined as :

package com.evertpot {

  // Private
  internal const foo = 5;

  class MyClass {

  }

}

Then you could use this class as follows:

const myObj = com.evertpot.MyClass;

Or:

import * from com.evertpot;
const myObj = MyClass;

As far as I know the standard doesn't define a relationship between namespaces and where the files can be loaded from.

Generic functions

Generic functions are not parameterized functions, they resemble
"Overloaded functions" in typescript a bit, but they're not quite the same and much more powerful.

Example:

class Foo {


  generic function addItem(x);

  function addItem(x: int) {
  }

  function addItem(x: number) {
  }

}

In the above example, I can call addItem with either a int or number, and the correct implementation will be picked at run-time.

E4X

While E4X was technically an extension to ES4, I think it deserves a mention.

E4X stands for ECMAScript for XML, and while that might not sound very exciting, take a look at a code snippet:

const myClass = 'welcome';
const name = 'Evert';
const foo = <div class={myClass}>{"Hello " + name }</div>;

Looks familiar?

Although not quite the same as JSX, it's becoming clear that this might have been a part of JSX's origin story.

While ES4 never landed, E4X actually worked in Firefox, until it was removed in Firefox 10.

More features

let const as a syntax for block-level constants. In ES5 and up const is already block-scope.
Generators (yield).
Tail-calls
Namespaced properties, classes and everything to avoid collisions, much like XML namespaces.

How would you load it?

Because Ecmascript 4 would break backwards compatibility, it would be important to tell a browser to interpret a script as ES4:

<script type="text/javascript;version=4" src="..."></script>

This is not unlike what we do with modules today:

<script type="module" src="..."></script>

Afterword

I hope this was an interesting view in the Javascript that could have been. Although we are slowly catching up with newer ECMAScript versions and tools like Typescript and JSX preprocessors, we're still not quite at 2007's vision for ECMAScript.

Perhaps if ES4 landed, fewer people would need complex build tools like Babel, Webpack and Typescript.

DEV Community: Evert Pot

Discovering features via HTTP OPTIONS

Accept and Accept-Encoding

Patching, posting and querying

Where’s PUT?

Linking to documentation

Obscure uses

WebDAV usage

The server-wide asterisk request

Final notes

New Structured Fields RFC out, and so is my Javascript package

What is it?

Why should you care?

Javascript package

Comments?

Taking a look at Mastodon

Links

Porting Curveball to Bun

Porting Curveball

Final thoughts on Bun

A new OAuth2 client for Javascript

client_credentials example

authorization_code example

Handling the redirect back

Docs and download

Request bodies in GET requests

Undefined behavior

Why it’s not defined

Why should GET have a body?

Why was it designed not to support bodies?

Our needs have evolved, has HTTP?

Anyway…

JWT should not be your default for sessions

Cookies

Sessions

Where is the session data stored?

Encrypted token

Drawbacks

Why are they popular?

On JWT and scale

Conclusion

Acknowledgements

Ketting 7 released

What is Ketting?

What's new?

Better HAL-Forms support

React bindings: <RequireLogin>

React-bindings: useCollection

React-bindings: refreshOnStale

Deprecation warnings

Removed support for Prefer-Push

Smarter caching of newly created resoures.

State objects now have a .follow() and .followAll() function.

Upgrading

Future plans

A generic middleware pattern in Typescript

The setup

How we want to use it

Running this application

The implementation

HTTP/2 Push is dead

Current alternatives

Apollo-like hooks for REST APIs with React and Ketting

What's Ketting?

React support in Ketting 6

Non-exhaustive list of other changes

Future plans

More links

ECMAScript 4: The missing version

A brief history

What could have been?

Classes

Interfaces

Strict typing

Like 👍

New types

triple-quoted strings.

Packages

Generic functions

E4X

Why should `GET` have a body?

React bindings: `<RequireLogin>`

React-bindings: `useCollection`

React-bindings: `refreshOnStale`

`State` objects now have a `.follow()` and `.followAll()` function.