Migrating JS code from json to x-www-form-urlencoded

This post I'm writing recounts the problems I faced with POSTing application/json content types and why I switched them over to application/x-www-form-urlencoded, and the obstacles I had to overcome to make it work.

To begin with, I'm writing a React app that extracts video frames from Youtube videos. and there is a little state that needs to be kept with each video:

• The currentProject: In my app I call videos "projects", this is just the current video that is being worked on.
• All of the projects: The array of videos created in my database.
• Each project has a few children properties such as the number of frames in the video numFrames and the array of frames I want to extract framesList among others.

And the state is in JSON format, stored in an express.js server which has a public API.

The important part is, each component retrieves some state items with a GET request when it's mounted, and when it's about to unmount it sends POST requests for the states it needs to change. And this is what I will be talking about for the remainder of this post.

Once upon a time, I was happy enough sending POST requests using the application/json content type:

var body = {'currentProject': this.state.project};
// send POST request
let res = await fetch(url, {
    method: 'post',
    body:    JSON.stringify(body),
    headers: { 'Content-Type': 'application/json' }
})

And everything was in harmony server-side and client-side. There was only one problem. It is that I set the content type to application/json. This causes the browser to preflight the requests by sending an OPTIONS request first, and then the POST later. Preflighting can be done for any request type, not just POST.

Normally this OPTIONS request isn't a problem, it only adds a few milliseconds of latency to your app which is negligible. But say you promisify your fetch calls, so that one request is not made until the previous one finishes. preflighting disrupts that sequence and send the OPTIONS request when you were expecting the POST request to be sent, and then POST requests, or whatever request type you were making were sent after some of the next requests have completed. This means that preflighting makes race conditions where the order the requests are made depends on your network speed.

So instead of sending the requests in this order

OPTIONS --> POST --> (some other GET)

The browser sends them in this order:

OPTIONS --> (some other GET) --> POST

If that GET request is supposed to retrieve the state you just POSTed and render it into the next component, then you will be retrieving old and undefined state and this will ruin your app.

So what did I do to attempt to fix this problem? I switched my content types from application/json to application/x-www-form-urlencoded. But it's not as simple as just changing the header. x-www-form-urlencoded has a special format that is a little from JSON, and your body needs to be formatted exactly that way for the request to succeed.

For example:

• JSON's [] is "" in form-urlencodded
• ["abc"] is "abc"
• [1] is "1" (notice how the number was turned into a string)
• Multiple array values are separated with comma, so ["abc", 123] becomes "abc,123".

A consequence of this naming scheme is that it's impossible to type some characters in form-urlencoded format that have a special meaning, like comma. Also some of the JSON structure is lost in the conversion, so now the empty array and the empty string are indistinguishable.

To avoid this, I decided to convert my JSON values to strings using JSON.stringify(). This makes a string out of a JSON object when can then be parsed by JSON.parse() back into an object.

But, I still can't POST this directly because there are illegal characters like [, ] and " which need to be escaped first. Escaping replaces the character with the percentage sign % followed by the hexadecimal number of the character. So = is converted to %3D. The string can be escaped using encodeURIComponent. encodeURIComponent takes a string and escapes all of the special characters in it, so if you pass it ["a", 1], it returns %5B%22a%22%2C%201%5D. The result can then be decoded into the original string using decodeURIComponent. It's necessary to use these two functions to encode data if you are submitting an x-www-form-urlencoded request, or the character replacement I described above will happen.

At this point the POST request bodies look like:

let body = {'someArray': encodeURIComponent(JSON.stringify(["a", 1])), 'someNumber': encodeURIComponent(JSON.stringify(1))}

// I personally dislike using escaped characters in keys.
// So an encodeURIComponent for such key names is redundant
// and returns the original value.

And GET requests are read like this:

let response = await fetch(url);
let body = await response.json();
let someArray = JSON.parse(decodeURIComponent(body.someArray));

We should refactor these into two functions to minimize the chance of writing it incorrectly. I will name them wwwencode and wwwdecode.

const wwwencode = (data) => {
    return encodeURIComponent(JSON.stringify(data))
};

const wwwdecode = (data) => {
    return JSON.parse(decodeURIComponent(data))
};

Implementing server support

Express can handle normal JSON requests and responses thanks to the bodyparser middleware, but bodyparser also has a urlencoded feature that allows it to understand x-www-form-urlencoded requests. This is exactly what we want. This snippet supports not only application/json requests but also application/x-www-form-urlencoded requests. No additional code is required to send and receive urlencoded parameters and responses, specifically there's nothing to put in the app routes.

const express = require('express');
const bodyParser = require('body-parser');
// ..
var app = express();
// support json encoded bodies
app.use(bodyParser.json());
// We don't use extended bodies or make use of node's 
// stringifying modules, we do this ourselves with 
// JSON.stringify. But extended defaults to true.
app.use(bodyParser.urlencoded({ extended: true }));

Now when you want to reference a parameter sent in your route, you use wwwdecode(req.body.someKey), and when you are ready to send the result back you use res.status(200).json('someValue': wwwencode(data)). The wwwencode and wwwdecode functions are the convenience wrappers I made above. The data is not automatically encoded, that is why we must do it manually with these two functions.

And that folks, is all you have to do to send JSON data with x-www-form-urlencoded. It avoids the sometimes problematic OPTIONS request.

And remember, if you see any incorrect information in this post, let me know so I can correct it.