DEV Community 👩‍💻👨‍💻

DEV Community 👩‍💻👨‍💻 is a community of 964,423 amazing developers

We're a place where coders share, stay up-to-date and grow their careers.

Create account Log in
Cover image for Why HTTP streaming upload matters on Web browser
Ryo Ota
Ryo Ota

Posted on

Why HTTP streaming upload matters on Web browser

We want to make the web better. I believe fetch() upload streaming is a key to making the web simpler, safer and faster.

Available, but giving up to shipping ...

Looking back in history, the document of fetch() upload streaming seems to have started 7 years ago. The latest document is under whatwg/fetch.

https://github.com/yutakahirano/fetch-with-streams/commits/master?after=3653e535378143663ade61e13064ce3561fbcf68+34

Main browser vendors have positive reactions as follows.

https://github.com/whatwg/fetch/pull/425#issuecomment-531680634

At last, we have the fetch() upload streaming feature since Chrome 85.

chrome://flags/#enable-experimental-web-platform-features

However, the Chromium team is going to give up to shipping the feature. That is the reason why the post is here.

We worked with a parter but we failed to show benefits of the feature, so we're giving up shipping the feature.

https://bugs.chromium.org/p/chromium/issues/detail?id=688906#c57

The origin trial ends today, Nov 10, 2021.

Why the Web browser needs fetch() upload streaming

HTTP is the core transfer technology on the web. It is an open, widely used and trusted protocol. And we are building more rich and complex applications on the web. Some of them may deal with high-resolution videos and others may provide high-level security with E2E encryption. The fetch() upload streaming allows us to do both upload and data transformations at the same. So, we can upload while encoding rich videos in client-side. We can upload while encrypting files in client-side, which archives E2E encryption. In addition, it is easy to combine multiple transformations and build a complex one. The fetch() upload streaming is more efficient in terms of both time and space than transforming whole data.

The most important thing is that it has high compatibility. Our world is made up of all kinds of software that are integrated together. Streaming upload in HTTP is already available in common software such as Java (JVM languages), Node.js, Go, Swift, curl, etc. These common software has built-in HTTP streaming upload. I believe that built-in availability is the biggest difference from WebSocket, WebRTC and upcoming WebTransport. The streaming upload coming to web browser makes web server simpler, instead of creating another WebSocket API, etc. only for web browser. Eliminating extra entrances or branches can reduce code size, reduce unexpected behaviors and improve maintainability and safety. HTTP is a widely accepted protocol and provide practical API than raw TCP. As far as I know, HTTP is the only one that has this power.

Simple, but powerful

API is very simple as follows. You can pass ReadableStream to fetch(). The streaming upload is the most general transfer in HTTP, which allows us to transfer conventional fixed-length data and infinite data stream.

await fetch(url, {
  method: "POST",
  body: myReadableStream
});
Enter fullscreen mode Exit fullscreen mode

Here is an example of how to create ReadableStream emitting infinite random bytes.

const myReadableStream = new ReadableStream({
  pull(ctrl) {
    // random bytes
    ctrl.enqueue(window.crypto.getRandomValues(new Uint32Array(128)));
  }
});
Enter fullscreen mode Exit fullscreen mode

Here is how to convert MediaStream to ReadableStream. This can apply screen share, web camera, voice or other sources which are from MediaStream.

// Convert MediaStream to ReadableStream
function mediaStreamToReadableStream(mediaStream, timeslice) {
  return new ReadableStream({
    start(ctrl) {
      const recorder = new MediaRecorder(mediaStream);
      recorder.ondataavailable = async (e) => {
        ctrl.enqueue(new Uint8Array(await e.data.arrayBuffer()));
      };
      recorder.start(timeslice);
    }
  });
}

// Usage:
const mediaStream = await navigator.mediaDevices.getDisplayMedia({video: true});
const myReadableStream = mediaStreamToReadableStream(mediaStream, 100);
Enter fullscreen mode Exit fullscreen mode

Everything over HTTP

As another real example, here is a stream encrypting another stream using OpenPGP. The demo shows end-to-end encrypting (E2EE), encrypting, uploading, downloading and decrypting at the same time. In the demo below, the client is a web browser. Yet, curl and gpg are also compatible with sending and receiving since behind-the-system is a pure HTTP streaming upload.

https://github.com/nwtgck/piping-ui-web/issues/433#issuecomment-830734031

Here is another example. This is SSH over HTTP. Everything including SSH client is complete in web browser. SSH client is also implemented in web browser powered by SSHy project. So, this is also E2E encryption under SSH.

https://github.com/nwtgck/piping-ssh-web

Here is VNC over HTTP. VNC is for remote control. VNC client is complete in web browser powered by noVNC project. The application below also provides E2E encryption using openssl aes-256-ctr-command-compatible way which encrypts streamingly.

https://github.com/nwtgck/piping-vnc-web

Those two applications tunnel a TCP port over pure HTTP. So, simple commands using curl and nc or socat can also be a client, not only web browser.

With built-in software

It is worth noting that in those examples, the web browser is just one of clients. This means java.net.HttpURLConnection in the standard library in JVM languages can be a client, net/http in the standard library in Go can be a client, URLSession#uploadTask(withStreamedRequest: request) in Foundation in Swift can be a client. The fetch() upload streaming allows our web browsers to be one. Without it, this highly compatibile feature is not available since there is no alternative and it is impossible to create a polyfill.

Conclusion

The feature allows us to transfer infinitely in real-time and give efficiency in both time and space, which was never allowed before and it is impossible to create polyfill. It also gives us to communicate with other built-in software seemly and keep web servers simpler, safe and more maintainable.

I love the feature. I love the fetch() upload streaming. Infinite stream has infinite possibilities. I hope this post would be a good one for you to know how great this feature is.

More about the feature

The links below would be helpful to know more.

Photo by Robert Zunikoff on Unsplash

Top comments (0)

🤯

"I made 10x faster JSON.stringify() functions, even type safe"

☝️ Must read for JS devs