DEV Community: Ryo Ota

Secure TCP tunnel from anywhere with curl and nc for single connection

Ryo Ota — Wed, 11 May 2022 12:41:38 +0000

This post provides a secure and highly transparent way for port forwarding or tunneling a single TCP connection, using only nc and curl. Moreover, it also provides more secure tunneling with end-to-end encryptions using openssl and socat.

# server host
curl -sSN https://ppng.io/aaa | nc localhost 22 | curl -sSNT - https://ppng.io/bbb

# client host
curl -sSN https://ppng.io/bbb | nc -lp 2222 | curl -sSNT - https://ppng.io/aaa

Why port forwarding from anywhere?

Today's application is everywhere. We often want to access an application on our remote machine. It is useful to make the remote application as if it is in local. To achieve that, port forwarding or tunneling is widely used for accessing a private remote port from a local machine. A common approach for secure port forwarding is using ssh -L. This is a very secure connection over SSH. However, exposing SSH port in public may not be allowed because of firewall or NAT. We sometimes need to port forward seamlessly no matter where we are. Besides, since SSH is very powerful, it allows us to use a shell by default even when we want to use only port forwarding without arbitrary command execution. In addition, some environments may not allow to install and run an SSH server, which usually needs the root permission.

Why nc and curl?

Some tools allow us to port forward from anywhere. However, as far as I know, they require us to install their own dedicated command. Some of them are open source projects, but I wanted a more transparent way because a shorter chain of trust should be better.

I found a higher transparent way that uses existing and widely used commands: nc and curl. The original idea of a port forwarding with socat, curl, and Piping Server was proposed in a Japanese article by @Cryolite. Meanwhile, this post introduces a way using nc (netcat), which is more widely used and installed than socat, and the way allows simple integration with other commands using Unix pipe. It is very helpful when using an untrustable server because we can integrate it with an encryption command such as openssl.

Piping Server

Here is an introduction to Piping Server. I developed Piping Server, which streams any data infinitely between every device over pure HTTP/HTTPS.

nwtgck / piping-server

Infinitely transfer between every device over pure HTTP with pipes or browsers

Piping Server

Infinitely transfer between every device over HTTP/HTTPS

Transfer

Piping Server is simple. You can transfer as follows.

# Send
echo 'hello, world' | curl -T - https://ppng.io/hello

# Get
curl https://ppng.io/hello > hello.txt

Piping Server transfers data to POST /hello or PUT /hello into GET /hello. The path /hello can be anything such as /mypath or /mypath/123/. A sender and receivers who specify the same path can transfer. Both the sender and the recipient can start the transfer first. The first one waits for the other.

You can also use Web UI like https://ppng.io on your browser. A more modern UI is found in https://piping-ui.org, which supports E2E encryption.

Stream

The most important thing is that the data are streamed. This means that you can transfer any data infinitely. The demo below transfers an infinite text stream with seq inf.

Ideas

…

View on GitHub

Piping Server is an HTTP server. A sender and recipient who specify the same path such as /mypath can transfer. The image below is the concept of Piping Server.

The image above shows the sender who does POST /mypath and the recipient who does GET /mypath can transfer. Both the sender and the recipient can start the transfer first. The first one waits for the other. After the sender and recipient connection is established, /mypath is protected not to be connected until the connection is closed. Closing the path frees the path and anyone can start a new transfer at the path.

Both POST and PUT methods are the same effect in Piping Server. Here is a simple example of transferring "hello" to a receiver.

# sender: PUT /mypath
$ echo hello | curl -T - https://ppng.io/mypath

# receiver: GET /mypath
$ curl https://ppng.io/mypath
hello

The server transfers any kind of data including text and binaries. In addition, the data through a Piping Server are infinitely streamed. The data are only on memory just for a moment and never saved. I posted The Power of Pure HTTP, which shows how powerful HTTP is and what Piping Server can do.

Public servers

Here are some available public servers.

https://piping.nwtgck.repl.co may be faster depending on your location. You can host one using docker run -p 8181:8080 nwtgck/piping-server instantly or other ways including portable binaries: https://github.com/nwtgck/piping-server/wiki/How-to-self-host-Piping-Server.

Port forwarding command

Here is the main part of this post.

Suppose you have two machines named "server host" which serves 22 port and "client host", and you want to use the 22 port in the server host as a new 2222 port in the client host. The following commands forward the 22 port in the server host to the 2222 port in the client host.

# server host
curl -sSN https://ppng.io/aaa | nc localhost 22 | curl -sSNT - https://ppng.io/bbb

# client host
curl -sSN https://ppng.io/bbb | nc -lp 2222 | curl -sSNT - https://ppng.io/aaa

-s slient without progress message
-S shows error even when -s used
-N Smooth streaming, disalbing buffering of the output stream

Cheat sheat

Though it's very cool to remember and type the commands, we can go to https://piping-server-command.nwtgck.org to copy the commands.

GNU nc vs OpenBSD nc vs socat

netcat (nc) has some variants. The ways of listening are different as follows.

GNU nc: nc -lp 2222
OpenBSD nc: nc -l 2222
socat: socat TCP-LISTEN:2222,reuseaddr -

# client host (GNU nc)
curl -sSN https://ppng.io/bbb | nc -lp 2222 | curl -sSNT - https://ppng.io/aaa
# client host (OpenBSD nc)
curl -sSN https://ppng.io/bbb | nc -l 2222 | curl -sSNT - https://ppng.io/aaa
# client host (socat)
curl -sSN https://ppng.io/bbb | socat TCP-LISTEN:2222,reuseaddr - | curl -sSNT - https://ppng.io/aaa

SSH demo

Here is a demo video of forwarding 22 port in my "basil" server to my local 2222 port.

SSH is very powerful because you use can create a SOCKS proxy with ssh -D 1080 ... like a simple VPN, do rsync for fast resumable file transfer, use GUI with ssh -X ... and mount a remote file system with SSHFS.

The example is about SSH since it is literally a well-known protocol. This method allows any TCP port to be forwarded for a single connection not only SSH.

Mechanism

The goal of tunneling is as follows, forwarding the 22 port of "server program" to the 2222 port of "client program."

The diagram below illustrates how the two commands above connect each other. All outgoing connections are HTTPS. This means that the tunneling requirements are the same as web browsing. Therefore, you can do the tunneling anywhere.

You can control the timing of port forwarding. Each server host side and client host side can start the command first. Each side can close the tunnel by Ctrl+C at any time and the other side command exits at that time, and this frees the paths. Piping Server guarantees the transferred protocol data is only on memory and never stored. The server also guarantees that after port forwarding is established at paths, no one can establish and get the data at the paths until the pre-established tunnel is closed.

High transparency

I believe "The less you trust in, the more secure you are."

Some tunneling tools require installing additional dedicated CLI. In contrast, the way uses only nc (netcat) and curl commands. They are widely used and accepted, and most people know how they work and have already trusted them. You just use the common commands, not need to install and trust extra tools. We can obviously understand that the commands never send any extra information, such as private information or credentials. This means there is no black box at all. Besides, the commands are short enough to remember.

All communication outside is complete in HTTP/HTTPS. The protocol is also familiar, widely used, accepted, and trusted already. Piping Server is open on GitHub, developed in TypeScript and Node.js. Other implementations in Rust and Go are provided as open-source. The server is designed to keep simple as possible to verify the source and reduce the potential of bugs.

In order to improve transparency more and get independent from the pre-hosted public servers, you can self host Piping Server. When you prefer to limit traffic with basic auth or path, etc., Rich Piping Server, which uses internally the original Piping Server as a library, is useful.

We need to trust a few existing widely accepted command-line tools and Piping Server. Even if the source of the server is public, the server might have vulnerabilities or might be modified maliciously. To avoid those concerns, I will introduce end-to-end encryption (E2E encryption) for higher-level security. E2E encryption allows us not to trust the server and make us secure.

Universal end-to-end encryption

The section describes secure tunneling even if a Piping Server is untrustable. As you know, all HTTPS traffic to the server is encrypted. However, what if the server is compromised or malicious? Tunneling SSH is one of the solutions. This section provides a more universal end-to-end encryption way for any protocol.

The idea is simple, which decrypts incoming streams and encrypts outgoing streams as follows.

# server host (idea)
curl -sSN https://ppng.io/aaa | <decrypt> | nc localhost 22 | <encrypt> |curl -sSNT - https://ppng.io/bbb

# client host (idea)
curl -sSN https://ppng.io/bbb | <decrypt> | nc -lp 2222 | <encrypt> | curl -sSNT - https://ppng.io/aaa

The real commands using openssl are as follows. Although openssl commands are used, any kind of encryption commands can be replaced with existing commands or future-invented commands.

# server host
curl -sSN https://ppng.io/aaa | stdbuf -i0 -o0 openssl aes-256-ctr -d -pass "pass:mypass" -bufsize 1 -pbkdf2 | nc localhost 22 | stdbuf -i0 -o0 openssl aes-256-ctr -pass "pass:mypass" -bufsize 1 -pbkdf2 | curl -sSNT - https://ppng.io/bbb

# client host
curl -sSN https://ppng.io/bbb | stdbuf -i0 -o0 openssl aes-256-ctr -d -pass "pass:mypass" -bufsize 1 -pbkdf2 | nc -lp 2222 | stdbuf -i0 -o0 openssl aes-256-ctr -pass "pass:mypass" -bufsize 1 -pbkdf2 | curl -sSNT - https://ppng.io/aaa

stdbuf -i0 -o0 <command> disables buffers for real-time streaming.
aes-256-ctr is a stream cipher.
-pbkdf2 derives key from the password using PBKDF2. This is optional but makes it more secure.
The password is mypass. You can replaced with better one.

The commands below allow us to input password and hides passwords from commands.

# server host
read -p "password: " -s pass && curl -sSN https://ppng.io/aaa | stdbuf -i0 -o0 openssl aes-256-ctr -d -pass "pass:$pass" -bufsize 1 -pbkdf2 | nc localhost 22 | stdbuf -i0 -o0 openssl aes-256-ctr -pass "pass:$pass" -bufsize 1 -pbkdf2 | curl -sSNT - https://ppng.io/bbb; unset pass

# client host
read -p "password: " -s pass && curl -sSN https://ppng.io/bbb | stdbuf -i0 -o0 openssl aes-256-ctr -d -pass "pass:$pass" -bufsize 1 -pbkdf2 | nc -lp 2222 | stdbuf -i0 -o0 openssl aes-256-ctr -pass "pass:$pass" -bufsize 1 -pbkdf2 | curl -sSNT - https://ppng.io/aaa; unset pass

These commands are also in the cheat sheet.

Web browser, not only terminal and the commands

High interoperability matters because it realizes fewer dependencies and less lock-in. This section shows the interoperability, replacing nc and curl with JavaScript.

Piping SSH and Piping VNC run on Web browser. This means a Web browser is an alternative frontend of curl ... | nc -lp ... | curl ... and a terminal. Those apps have SSH and VNC (RBF protocol) implementations in JavaScript, thanks to authors of based projects. This means that they work on your local device and never process on the server side. The server is only a Piping Server only for data transferring. See my previous post for detail.

Those apps require "fetch() upload streaming", which is only available now in Chromium-based browsers >= 85 with "Experimental Web Platform features" enabled. Every major vendors has positive reaction to the "fetch() upload streamingl". The video below is an example using Piping SSH.

VNC is widely used for remote desktop. The example below has two windows, a Ubuntu on a virtual machine and Piping VNC on Chrome on my local machine. It shows relatively smooth remote desktop.

The port forwarding way has high interoperability because the forwarding command, curl ... | nc ... | curl ... , in the example videos is the same even when a client runs on Web browser. In this example, fetch() in JavaScript replaces nc, curl, and terminal. As another example, java.net.URLConnection and java.net.Socket in Java and net.Dial() and http in Go language can replace them.

E2E encryption VNC

Although the transport to Piping Server is encrypted with TLS because of HTTPS, E2E encryption for VNC is secure if the server is untrustable.

You can put a check in "Encrypt with OpenSSL AES CTR" below in Piping VNC. It has the same effect as openssl aes-256-ctr -d -pass "pass:mypass" -bufsize 1 -pbkdf2 -iter 100000 -md sha256 mentioned in the "Universal end-to-end encryption" section above.

Piping VNC automatically generates a command you should type on the server host side.

How is it possible in Web browser? I made https://github.com/nwtgck/openssl-aes-ctr-stream-npm for using OpenSSL-compatible AES CTR encryption/decryption in Web browser. The concept is very simple as follows.

// Encrypt the ReadableStream `uploadReadableStream`
const encryptedUploadReadableStream = opensslAesCtrStream.aesCtrEncryptWithPbkdf2(
  uploadReadableStream,
  options
);

// Transfer the encrypted stream
fetch(pipingServerUrl, {
  method: "POST",
  body: encryptedUploadReadableStream,
});

(simplified version of https://github.com/nwtgck/piping-vnc-web/blob/8f2df75c7452b515fcc2ba36fe9cbf6218c673fc/core/websock.js#L205-L214)

openssl-aes-ctr-stream implementation is shorter than 200 lines: https://github.com/nwtgck/openssl-aes-ctr-stream-npm/blob/d071752009b0c6262f6c27722b438cb73cdb7cab/src/index.ts. The major browsers provide Web Crypto API. openssl-aes-ctr-stream uses AES-CTR in the Web Crypto API.

Here is a demo video of E2E encryption VNC. Because of the E2E encryption, the screen is laggier than without it.

https://piping-vnc.nwtgck.org

Because the protocol of OpenSSL is open, we can stay highly interoperable with implementation in the other language.

Universal end-to-end encryption over TLS

This section introduces the way to encrypt pre-listened TCP port using TLS with socat. This is also a universal E2E encryption way. The advantages of the way are that you can use well-trusted TLS and encryptions with sharing certificates instead a password.

In the server host, create certificates as follows. It generates server.crt, server.key and server.pem.

# server host
openssl req -x509 -newkey rsa:4096 -keyout server.key -out server.crt -days 3653 -sha256 -nodes -subj '/CN=localhost/' && cat server.key server.crt > server.pem

In a client host, create certificates as follows. It generates client.crt, client.key and client.pem.

# client host
openssl req -x509 -newkey rsa:4096 -keyout client.key -out client.crt -days 3653 -sha256 -nodes -subj '/CN=localhost/' && cat client.key client.crt > client.pem

Share server.crt to the client host and share client.crt to the server host respectively.

server host

Suppose the server host serves 8181 port and you want to forward the 8181 to a client with encryption. Open two terminals in server host and run the following commands. Make sure to run socat in the directory which has server.pem and client.crt.

socat openssl-listen:4433,reuseaddr,cert=server.pem,cafile=client.crt tcp:localhost:8181

curl -sSN https://ppng.io/aaa | nc localhost 4433 | curl -sSNT - https://ppng.io/bbb

client host

Open two terminals in the client host and run the following commands. Make sure to run socat in the directory which has client.pem and server.crt.

curl -sSN https://ppng.io/bbb | nc -lp 5433 | curl -sSNT - https://ppng.io/aaa

socat tcp-listen:8282,reuseaddr openssl-connect:localhost:5433,openssl-commonname=localhost,cert=client.pem,cafile=server.crt

Now, the 8282 port in the client host is the decrypted TCP port forwarded to 8181 in the server host. When the protocol of the 8181 port in the server host is HTTP, you can use curl localhost:8282 in the client host.

Here is a graph to describe how this E2EE works.

[server host 8181 <--> 4433] <-- TLS over HTTPS --> [Piping Server] <-- TLS over HTTPS --> [5433 <--> 8282 client host]

The way is based on Securing Traffic Between two Socat Instances Using SSL found on the socat official page. If you encountered Invalid argument error in macOS with socat, you can upgrade socat 1.7.4.2 or later, which is fixed.

Universal end-to-end encryption over SSH

You can simply use ssh -L as follows when a machine on server host side has SSH server. SSH also multiplexes TCP connections over single SSH connection.

Suppose you want to forward 8080 port in server host with encryption to 8181 port in client host. Suppose the server host listens SSH on 22 port. The command below forwards 22 to 2222 and 8080 to 8181.

# server host
curl -sSN https://ppng.io/aaa | nc localhost 22 | curl -sSNT - https://ppng.io/bbb

# client host 1/2
curl -sSN https://ppng.io/bbb | nc -lp 2222 | curl -sSNT - https://ppng.io/aaa

# client host 2/2
ssh -p 2222 -L 8181:localhost:8080 <server host user>@localhost

Now, you can access localhost:8181 in the client host.

bonus: Multiple TCP connections

The way introduced here allows forwarding a single TCP connection. A single TCP connection is enough for SSH and VNC (RFB protocol). This feature sometimes contributes to greater security since one connection is always guaranteed. However, for instance, HTTP generally requires multiple TCP connections even when HTTP/2, which multiplexes request and response streams, is available.

A simple solution is to multiplex TCP requests over Unix pipe. In order to multiplex TCP requests, Yamux is available created by Hashicorp, who creates Vagrant, Terraform, and so on. The protocol of Yamux is inspired by SPDY, which is the basis of HTTP/2 specification. The protocol spec is found in https://github.com/hashicorp/yamux/blob/master/spec.md. libp2p, which is used in IPFS, also uses Yamux as one choice of multiplexes and maintains Go and Rust versions of Yamux libraries.

I made a yamux CLI and distribute its portable binaries for multi-platforms. You can install it from https://github.com/nwtgck/yamux-cli and replaces nc with yamux as follows.

# server host
curl -sSN https://ppng.io/aaa | yamux localhost 8080 | curl -sSNT - https://ppng.io/bbb

# client host
curl -sSN https://ppng.io/bbb | yamux -l 8181 | curl -sSNT - https://ppng.io/aaa

You can also combine this yamux and E2E commands to make it more secure.

This topic is in a bonus section since its way requires an additional CLI. The way of the section "Universal end-to-end encryption over SSH" also allows multiple TCP connections because ssh -L also multiplexes TCP connections.

bonus: UDP

TCP is a perfect fit for Unix pipe because TCP delivers streams. In contrast, UDP delivers datagrams, so boundaries are needed to deliver multiple datagrams over Unix pipe. I successfully run dig command for forwarded DNS using nc -u. However, multiple datagrams are not processed. To solve this problem, I made an experimental option to yamux cli for UDP.

# server host (experimental)
curl -sSN https://ppng.io/aaa | yamux -u localhost 53 | curl -sSNT - https://ppng.io/bbb

# client host (experimental)
curl -sSN https://ppng.io/bbb | yamux -ul 1053 | curl -sSNT - https://ppng.io/aaa

"Experimental" means that I may change the way of transferring over Yamux. Datagrams from the same address are delivered over the same Yamux stream. The data structure is simple, which has datagram length in 4 bytes in network byte order and raw datagram.

In my experiment, HTTP/3, HTTP over QUIC, can be port forwarded over Piping Server. Actually, I realized HTTP/3 Piping Server over Piping Server.

Links

The cover image by Fabian Jung on Unsplash

Why HTTP streaming upload matters on Web browser

Ryo Ota — Wed, 10 Nov 2021 11:32:42 +0000

We want to make the web better. I believe fetch() upload streaming is a key to making the web simpler, safer and faster.

Available, but giving up to shipping ...

Looking back in history, the document of fetch() upload streaming seems to have started 7 years ago. The latest document is under whatwg/fetch.

https://github.com/yutakahirano/fetch-with-streams/commits/master?after=3653e535378143663ade61e13064ce3561fbcf68+34

Main browser vendors have positive reactions as follows.

https://github.com/whatwg/fetch/pull/425#issuecomment-531680634

At last, we have the fetch() upload streaming feature since Chrome 85.

chrome://flags/#enable-experimental-web-platform-features

However, the Chromium team is going to give up to shipping the feature. That is the reason why the post is here.

We worked with a parter but we failed to show benefits of the feature, so we're giving up shipping the feature.

https://bugs.chromium.org/p/chromium/issues/detail?id=688906#c57

The origin trial ends today, Nov 10, 2021.

Why the Web browser needs fetch() upload streaming

HTTP is the core transfer technology on the web. It is an open, widely used and trusted protocol. And we are building more rich and complex applications on the web. Some of them may deal with high-resolution videos and others may provide high-level security with E2E encryption. The fetch() upload streaming allows us to do both upload and data transformations at the same. So, we can upload while encoding rich videos in client-side. We can upload while encrypting files in client-side, which archives E2E encryption. In addition, it is easy to combine multiple transformations and build a complex one. The fetch() upload streaming is more efficient in terms of both time and space than transforming whole data.

The most important thing is that it has high compatibility. Our world is made up of all kinds of software that are integrated together. Streaming upload in HTTP is already available in common software such as Java (JVM languages), Node.js, Go, Swift, curl, etc. These common software has built-in HTTP streaming upload. I believe that built-in availability is the biggest difference from WebSocket, WebRTC and upcoming WebTransport. The streaming upload coming to web browser makes web server simpler, instead of creating another WebSocket API, etc. only for web browser. Eliminating extra entrances or branches can reduce code size, reduce unexpected behaviors and improve maintainability and safety. HTTP is a widely accepted protocol and provide practical API than raw TCP. As far as I know, HTTP is the only one that has this power.

Simple, but powerful

API is very simple as follows. You can pass ReadableStream to fetch(). The streaming upload is the most general transfer in HTTP, which allows us to transfer conventional fixed-length data and infinite data stream.



await fetch(url, {
  method: "POST",
  body: myReadableStream
});

Here is an example of how to create ReadableStream emitting infinite random bytes.



const myReadableStream = new ReadableStream({
  pull(ctrl) {
    // random bytes
    ctrl.enqueue(window.crypto.getRandomValues(new Uint32Array(128)));
  }
});

Here is how to convert MediaStream to ReadableStream. This can apply screen share, web camera, voice or other sources which are from MediaStream.



// Convert MediaStream to ReadableStream

function mediaStreamToReadableStream(mediaStream, timeslice) {

  return new ReadableStream({

    start(ctrl) {

      const recorder = new MediaRecorder(mediaStream);

      recorder.ondataavailable = async (e) => {

        ctrl.enqueue(new Uint8Array(await e.data.arrayBuffer()));

      };

      recorder.start(timeslice);

    }

  });

}

// Usage:

const mediaStream = await navigator.mediaDevices.getDisplayMedia({video: true});

const myReadableStream = mediaStreamToReadableStream(mediaStream, 100);

Everything over HTTP

As another real example, here is a stream encrypting another stream using OpenPGP. The demo shows end-to-end encrypting (E2EE), encrypting, uploading, downloading and decrypting at the same time. In the demo below, the client is a web browser. Yet, curl and gpg are also compatible with sending and receiving since behind-the-system is a pure HTTP streaming upload.

https://github.com/nwtgck/piping-ui-web/issues/433#issuecomment-830734031

Here is another example. This is SSH over HTTP. Everything including SSH client is complete in web browser. SSH client is also implemented in web browser powered by SSHy project. So, this is also E2E encryption under SSH.

https://github.com/nwtgck/piping-ssh-web

Here is VNC over HTTP. VNC is for remote control. VNC client is complete in web browser powered by noVNC project. The application below also provides E2E encryption using openssl aes-256-ctr-command-compatible way which encrypts streamingly.

https://github.com/nwtgck/piping-vnc-web

Those two applications tunnel a TCP port over pure HTTP. So, simple commands using curl and nc or socat can also be a client, not only web browser.

With built-in software

It is worth noting that in those examples, the web browser is just one of clients. This means java.net.HttpURLConnection in the standard library in JVM languages can be a client, net/http in the standard library in Go can be a client, URLSession#uploadTask(withStreamedRequest: request) in Foundation in Swift can be a client. The fetch() upload streaming allows our web browsers to be one. Without it, this highly compatibile feature is not available since there is no alternative and it is impossible to create a polyfill.

Conclusion

The feature allows us to transfer infinitely in real-time and give efficiency in both time and space, which was never allowed before and it is impossible to create polyfill. It also gives us to communicate with other built-in software seemly and keep web servers simpler, safe and more maintainable.

I love the feature. I love the fetch() upload streaming. Infinite stream has infinite possibilities. I hope this post would be a good one for you to know how great this feature is.

More about the feature

The links below would be helpful to know more.

Photo by Robert Zunikoff on Unsplash

The Power of Pure HTTP – screen share, real-time messaging, SSH and VNC

Ryo Ota — Thu, 17 Dec 2020 15:27:07 +0000

Hi all. Today, I'd like to show you the power of pure HTTP stream, which allows us to do screen sharing, video chatting, real-time text messaging, SSH, VNC, and so on. The demo videos below show SSH and VNC over pure HTTP. Let's dive into this!


SSH on Web browser	VNC on Web browser

Why HTTP?

HTTP is everywhere for everyone and everything, which allows us to communicate in any situation. HTTP is one of the most mature protocols and wildly used. You can find HTTP in web browser in personal computers including Windows, Mac and Linux, smartphones and tablets including Android, iPhone and iPad, curl, wget commands in your terminal, some IoT devices, automation tools such as Shortcuts iOS app and Microsoft Flow, Web Hooks and so on. Additionally, HTTP is an evolving protocol as HTTP/2, HTTP/3 and getting faster.

Usuary, an HTTP request has a short life, used for fetching HTML/CSS/JavaScript, media and API requests. So, this post introduces long-life HTTP request, which does real-time communications over pure HTTP stream without WebSocket and WebRTC.

Piping Server

I made Piping Server, which allows us to transfer data between every device.

nwtgck / piping-server

Infinitely transfer between every device over pure HTTP with pipes or browsers

Piping Server

Infinitely transfer between every device over HTTP/HTTPS

Transfer

Piping Server is simple. You can transfer as follows.

# Send
echo 'hello, world' | curl -T - https://ppng.io/hello

# Get
curl https://ppng.io/hello > hello.txt

You can also use Web UI like https://ppng.io on your browser. A more modern UI is found in https://piping-ui.org, which supports E2E encryption.

Stream

The most important thing is that the data are streamed. This means that you can transfer any data infinitely. The demo below transfers an infinite text stream with seq inf.

Ideas

…

View on GitHub

Piping Server is simple. A sender and recipient who specify the same path such as /hello can transfer. The image below is the concept of transfer.

The image shows the sender who POSTs /mypath and the recipient GETs /mypath can transfer. Both the sender and the recipient can start the transfer first. The first one waits for the other. Both POST and PUT methods are the same in Piping Server.

You can transfer a text by using fetch() in JavaScript like below.

// Send
fetch("https://ppng.io/hello", {
  method: "POST",
  body: "hello, world"
});

// Get
const res = await fetch("https://ppng.io/hello");
console.log(await res.text());
// => "hello, world"

You can also use curl command like below.

You can also transfer binary data such as an image and a video like below. As you can see, the recipient just opened the URL on the browser to get the image.

HTTP is everywhere. So we can transfer data freely without any extra tools.

Infinitely transfer

The most notable feature of Piping Server is allowing you to transfer infinite data. The demo below shows inputting text streams into the web browser.

Send folder

You can transfer a folder (a directory), which has multiple files as follows.

# Send folder
tar c ./mydir | curl -T - https://ppng.io/mypath

# Get folder
curl https://ppng.io/mypath | tar xv

The files are packing while uploading in the sender and unpacking while downloading in the recipient. Stream makes this possible without creating a temporary file.

It is easy to end-to-end encrypt your data and transfer as follows.

send: ... | openssl aes-256-cbc | curl -T ...
get: curl ... | openssl aes-256-cbc -d

It is also easy to reduce the size by compression as follows.

send: ... | gzip | curl -T ...
get: curl ... | zcat

You can transform data as you want such as gpg, zip or tools invented in the future. Combining pipe is efficient in terms of both time and memory. Unix pipe is an amazing way to combine software. The name of Piping Server is derived from Unix pipe.

The most common use case of Piping Server is a file transfer. For transferring files, you can use Piping UI, which allows you to transfer securely with end-to-end encryption over many devices.

Transfer huge data for a long time

Here are simple experiments to transfer data over HTTP using local and remote Piping Server.

The demo video below shows 45TB is transferred for 2,092 hours (87 days) over HTTP via remote Piping Server, using cat /dev/zero | curl -T- ....

The image below shows transferred 1,110TB (≈ 1PB) for 64 days over HTTP via local Piping Server.

These experiments show a huge amount of data can be continuously transferred over a single HTTP request and a single HTTP request lives long enough.

Infinite stream for Web browser

Infinite stream sending over HTTP had come to Web browser at last!

Google Chrome 85 or above has the feature as origin trial. Open chrome://flags and enable "Experimental Web Platform features" as follows

Other main browsers such as Firefox and Safari are also interested in this feature.

Uploading a Request made from a ReadableStream body by yutakahirano · Pull Request #425 · whatwg/fetch

In a nutshell, this feature allows us to send ReadableStream as follows.

fetch("https://example.com", {
  method: "POST",
  body: <ReadableStream here!>
});

Simple text messaging

Here is a simple text messaging on Web browser with fetch() and ReadableStream.

The code below creates ReadableStream from user input and sends the input stream to Piping Server. The recipient just opens the URL on the browser and sees streamed text messages.

const readableStream = new ReadableStream({
  start(ctrl) {
    const encoder = new TextEncoder();
    window.myinput.onkeyup = (ev) => {
      if (ev.key === 'Enter') {
        ctrl.enqueue(encoder.encode(ev.target.value+'\n'));
        ev.target.value = '';
      }
    }
  }
});

fetch("https://ppng.io/mytext", {
  method: 'POST',
  body: readableStream,
  headers: { 'Content-Type': 'text/plain;charset=UTF-8' },
  allowHTTP1ForStreamingUpload: true,
});

(full: https://github.com/nwtgck/piping-server-streaming-upload-htmls/blob/a107dd1fb1bbee9991a9278b10d9eaf88b52c395/text_stream.html)

allowHTTP1ForStreamingUpload in the code is a temporary property in Chrome to allow us to use this feature over HTTP/1.1 (see: 4c75c0c9f730589ad8d6c33af919d6b105be1462 - chromium/src - Git at Google).

Screen sharing

You can share your screen in almost the same way as the text streaming above. Get MediaStream and convert to ReadableStream and send the stream to Piping Server with fetch().

The function mediaStreamToReadableStream() below converts MediaStream to ReadableStream.

(async () => {
  // Get display
  const mediaStream = await navigator.mediaDevices.getDisplayMedia({video: true});
  // Convert MediaStream to ReadableStream
  const readableStream = mediaStreamToReadableStream(mediaStream, 100);

  fetch("https://ppng.io/myvideo", {
    method: 'POST',
    body: readableStream,
    allowHTTP1ForStreamingUpload: true,
  });
})();

// Convert MediaStream to ReadableStream
function mediaStreamToReadableStream(mediaStream, timeslice) {
  return new ReadableStream({
    start(ctrl){
      const recorder = new MediaRecorder(mediaStream);
      recorder.ondataavailable = async (e) => {
        ctrl.enqueue(new Uint8Array(await e.data.arrayBuffer()));
      };
      recorder.start(timeslice);
    }
  });
}

(full: https://github.com/nwtgck/piping-server-streaming-upload-htmls/blob/a107dd1fb1bbee9991a9278b10d9eaf88b52c395/screen_share.html)

The recipient just opens the HTML below with one <video> tag.

<!-- viewer -->
<video src="https://ppng.io/myvideo" autoplay muted></video>

This way is friendy to command-line tools too. You can also view the screen with curl https://ppng.io/myvideo | ffplay -. You can also send your screen with ffmpeg command. See Capture/Desktop – FFmpeg for more info.

Voice and video chatting

For voice or video chatting, all you need to do is to replace the code, const mediaStream = above with:

// Voice
const mediaStream = navigator.mediaDevices.getUserMedia({ audio: { echoCancellation: true } })

// video + voice
const mediaStream = navigator.mediaDevices.getUserMedia({ video: true, audio: { echoCancellation: true } })

(voice: https://github.com/nwtgck/piping-server-streaming-upload-htmls/blob/a107dd1fb1bbee9991a9278b10d9eaf88b52c395/simple_phone.html)

(video + voice: https://github.com/nwtgck/piping-server-streaming-upload-htmls/blob/a107dd1fb1bbee9991a9278b10d9eaf88b52c395/video_chat.html)

Then, you can use the mediaStreamToReadableStream() to converts those MediaStreams to ReadableStreams to specify body: in fetch().

Video filtering

You can get MediaStream from the canvas. The function below creates a video and a canvas in memory and transforms a MediaStream to another one. JSManipulate is used. You may create a filter app like Snap Camera.

// Filter for sepia
async function sepiaMediaStream(mediaStream) {
  const memVideo = document.createElement('video');
  memVideo.srcObject = mediaStream;
  await memVideo.play();

  const width = memVideo.videoWidth;
  const height = memVideo.videoHeight;
  const srcCanvas = document.createElement('canvas');
  const dstCanvas = document.createElement('canvas');
  srcCanvas.width = dstCanvas.width = width;
  srcCanvas.height = dstCanvas.height = height;
  const srcCtx = srcCanvas.getContext('2d');
  const dstCtx = dstCanvas.getContext('2d');

  (function loop(){
    srcCtx.drawImage(memVideo, 0, 0, width, height);
    const frame = srcCtx.getImageData(0, 0, width, height);

    JSManipulate.sepia.filter(frame);
    dstCtx.putImageData(frame, 0, 0);
    setTimeout(loop, 0);
  })();

  return dstCanvas.captureStream();
}

(full: https://github.com/nwtgck/piping-server-streaming-upload-htmls/blob/a107dd1fb1bbee9991a9278b10d9eaf88b52c395/screen_share_with_filter.html)

(demo video: https://youtu.be/VcKJR8D8IFA)

Compression

Compress data with gzip as follows. In Chrome, you can easily compress a stream with readableStream.pipeThrough(new CompressionStream('gzip')).

const readableStream = new ReadableStream({
  pull(ctrl) {
    // random bytes
    ctrl.enqueue(window.crypto.getRandomValues(new Uint32Array(128)));
  }
}).pipeThrough(new CompressionStream('gzip'))

fetch("https://ppng.io/mytext", {
  method: 'POST',
  body: readableStream,
  allowHTTP1ForStreamingUpload: true,
});

(full: https://github.com/nwtgck/piping-server-streaming-upload-htmls/blob/a107dd1fb1bbee9991a9278b10d9eaf88b52c395/gzip_inifinite_stream.html)
The sample code sends infinite random bytes with compression over Piping Server.

End-to-end encryption for infinite stream

You can safely transfer your stream even if a server is untrustable. You can encrypt any ReadableStream with the code below using OpenPGP.js.

// Encrypt ReadableStream with password by OpenPGP
async function encryptStream(readableStream, password) {
  const options = {
    message: openpgp.message.fromBinary(readableStream),
    passwords: [password],
    armor: false
  };
  const ciphertext = await openpgp.encrypt(options);
  return ciphertext.message.packets.write();
}

https://youtu.be/lxpxeB_0UDk is a demo video of end-to-end encrypted screen sharing over Piping Server.

Service Worker is used on the viewer-side. The purpose of using Service Worker is for getting a decrypted video at https://localhost:8080/e2ee_screen_share/swvideo#myvideo. Service Worker is used as a proxy. See the full code for detail: https://github.com/nwtgck/piping-server-streaming-upload-htmls/tree/a107dd1fb1bbee9991a9278b10d9eaf88b52c395/e2ee_screen_share.

Web browsers have Web Crypto, which can generate keys safely and do Diffie-Hellman key exchange an untrustable channel. For example, Piping UI, which is a file transfer tool, exchanges public keys and encrypts a file by using ECDH and OpenPGP.js.

Access the repository below to get other examples using fetch() upload streaming feature with Piping Server.
https://github.com/nwtgck/piping-server-streaming-upload-htmls

SSH over HTTP

As you see, any data can be streamed over HTTP. So, this means a protocol can be over HTTP via Piping Server.

Why Piping Server?

There are some environments that can not release ports public. For such kind of environment, when you have the only outward-connection to HTTP/HTTPS ports, you can use SSH. A possible example is for GitHub Actions, which does not support SSH debug like CircleCI (see: SSH in GitHub Actions over Piping Server).

SSH client in JavaScirpt

I found a wonderful project, SSHy whose JavaScript speaks SSH. The data communication is over WebSocket, so I just need to switch WebSocket to HTTP with fetch(). Unfortunately, although SSHy is not actively maintained now, this is a perfect fit for my proof of concept to speak SSH over HTTP using Piping Server. We could port OpenSSH by using Emscripten, write Rust and compile to Web Assembly, or do something in the future.

By using SSHy, it is possible to SSH with only Web browser and Piping Server. The data streamed to Piping Server is securely encrypted since the communication is SSH.

How to SSH over Piping Server?

Create two sets of connections over Piping Server for duplex communication. One of them is for sending data to your peer. The other one is for receiving data from your peer. On HTTP/2, multiple HTTP requests are bundled into one TCP connection.

The command below is an example to forward 22 port over HTTP via Piping Server. This way was proposed by @Cryolite in a Japanese great post https://qiita.com/Cryolite/items/ed8fa237dd8eab54ef2f. The data to the 22 port is downloading from /path1 and the data from the 22 port is uploading to /path2.

# server-host
socat 'EXEC:curl -NsS https\://ppng.io/path1!!EXEC:curl -NsST - https\://ppng.io/path2' TCP:127.0.0.1:22

The way makes possible NAT traversal without releasing port public over HTTP.

The command below creates the tunnel with the command above. The 22 port is forwarded to 31376 port in your another machine.

# client-host
socat TCP-LISTEN:31376 'EXEC:curl -NsS https\://ppng.io/path2!!EXEC:curl -NsST - https\://ppng.io/path1'

You can do ssh -p 31376 <user>@localhost in the machine in another terminal. This is a versatile way to forward a port to another device, not only SSH.

Transport implementations of SSHy

The implementation below sends bytes over WebSocket.
https://github.com/stuicey/SSHy/blob/82941c8ae15359fd387109dcee3a218808df0bb0/index.html#L259-L264
ws, WebSocket instance has a user-defined method, sendB64() and send Base64-encoded string. A proxy server called stuicey/wsproxy is used, which is for proxying WebSocket to TCP (in this case SSH).

The implementation below receives bytes over WebSocket.
https://github.com/stuicey/SSHy/blob/82941c8ae15359fd387109dcee3a218808df0bb0/index.html#L233-L236

SSH over Piping Server

These sending and receiving parts are replaced fetch() and a way of using Piping Server. The code below is the replaced implementation.
https://github.com/nwtgck/piping-ssh-web/blob/287e89ef05173e69d1302b29acf2abbe858ee78b/index.html#L187-L219

The application is called Piping SSH. Here is a demo video. In it, logging in Ubuntu machine from the web browser and type ls and htop command.

Application: https://piping-ssh.nwtgck.org
GitHub: https://github.com/nwtgck/piping-ssh-web

TIPS: Keep-alive of SSH

In Chrome, an HTTP request is stopped when no bytes arrived for 60 seconds. To resolve the issue, you can set /etc/ssh/sshd_config as follows in your SSH server setting.

# /etc/ssh/sshd_config
# ...
ClientAliveInterval 20
ClientAliveCountMax 3
# ...

VNC over HTTP

VNC (Virtual Network Computing) is widely used for controlling the computer remotely.

Here is the demo video. The front window is a Chrome, Web browser and the back one is a controlled machine on Ubuntu on VirtualBox.

Application: https://piping-vnc.nwtgck.org
GitHub: https://github.com/nwtgck/piping-vnc-web

For Ubuntu 20.04 users, to enable VNC, you can turn on Settings > Sharing and run gsettings set org.gnome.Vino require-encryption false to avoid an error, "Failed when connecting: Unsupported security types (types: 18)".

VNC is also available for Windows. Here is a demo controlling Windows 10 from Chrome. It was more smooth on a real Windows machine since the windows machine in the demo below was running on VirtualBox. UltraVNC is running on the Windows machine.

The fetch-uploading feature is also available on Android Chrome. The demo below controls Windows 10 by an Android smartphone.

For windows users, you can a download tunneling tool over Piping Server here: https://github.com/nwtgck/go-piping-tunnel. It is convenient to create a simple .bat file as follows.

.\piping-tunnel server -p 5900 path1 path2

piping-tunnel has the same feature of the socat + curl command. For mac users, you can install by brew install nwtgck/piping-tunnel/piping-tunnel.

How it works

The application is fully based on noVNC, which is a VNC client written in JavaScript. Only transport implementations are replaced with the way of using fetch and Piping Server instead of WebSocket.

Here is the diff for replacing WebSocket transportation with fetch and Piping Server.
https://github.com/nwtgck/piping-vnc-web/commit/1e1f2863160bfab8c9fbfc4c6970cd2b31135bfd

Network in Web browser

Here is the network in the Chrome DevTools. There are only two pure HTTPS connections. v86 is uploading and 7vk is downloading. As you can see the download size of v86 is increasing. Although 7vk is uploading, the view in the current Chrome says "pending".

fetch() upload streaming

I have been following this feature. Here are useful links to get information about the fetch() upload streaming feature.

whatwg: Fetch Standard
whatwg issue: Uploading a Request made from a ReadableStream body by yutakahirano · Pull Request #425 · whatwg/fetch
Chromium commits: 688906 - Streaming upload support - chromium
fetch() upload streaming - Chrome Platform Status
web.dev: Streaming requests with the fetch API
Firefox: 1387483 - [Meta-Bug] Support ReadableStream as Request.body in fetch API
Firefox: 1469359 - Support ReadableStream as Request.body in fetch API in necko
Safari: 203617 – "ReadableStream uploading is not supported" when fetch()ing a Request that has been logged to console

Public Piping Server

Here are public Piping Servers.

https://ppng.io
- alias: https://piping.ml
- These aliases can be different servers in the future.
https://piping.glitch.me
https://piping-47q675ro2guv.runkit.sh
https://ppng.herokuapp.com (NOTE: Heroku does not support streaming)

Self-hosted Piping Server

Run Piping Server on http://localhost:8080 as follows using Docker.

docker run -p 8080:8080 nwtgck/piping-server

Single binaries are also available on https://github.com/nwtgck/piping-server-pkg.

Here are easier ways to get public your Piping Server is to use Glitch and Runkit.

remix from https://glitch.com/~piping and serve it
clone from https://runkit.com/nwtgck/piping/ and serve it

Piping Server with JWT authentication

To restrict users to use Piping Server, you can use https://github.com/nwtgck/jwt-piping-server with an example using Auth0.

Piping Server in Rust

Piping Server is also written in Rust. This is the fastest Piping Server now.

GitHub: https://github.com/nwtgck/piping-server-rust

Base posts

Here are my posts based on this post.

The link below is the repository of Piping Server.
GitHub: https://github.com/nwtgck/piping-server

Get more information from the link below about Piping Server such as end-to-end encrypted file transfer, with basic auth, real-time drawing and so on.
https://github.com/nwtgck/piping-server/wiki/Ecosystem-around-Piping-Server

Try QUIC in Node.js on Docker

Ryo Ota — Mon, 30 Mar 2020 11:56:31 +0000

Hi! I made a Docker image for QUIC in Node.js for everyone to try QUIC easily.

GitHub repository

Here is the GitHub repository.

nwtgck / docker-node-quic

Docker image for Node.js with QUIC

docker-node-quic

Docker image for Node.js with QUIC

Usage

You can docker-run with docker run -it nwtgck/node-quic as follows.

$ docker run -it nwtgck/node-quic
Welcome to Node.js v14.0.0-pre.
Type ".help" for more information.
> const { createQuicSocket } = require('net');
undefined

You can get more detail examples and descriptions about the usage of QUIC in Node.js in the following.
quic/quic.md at cee2e5d079ca2b55e421d81df1ad131c1bfeecc6 · nodejs/quic

Article

Here is an article to get example.

Try QUIC in Node.js on Docker - DEV Community 👩‍💻👨‍💻

View on GitHub

You can use the docker image by docker run -it nwtgck/node-quic and use QUIC by const { createQuicSocket } = require('net');.

Create echo server

As an example, let's create an echo server.

First, you can create self-signed certificates as follows.

mkdir ssl_certs
cd ssl_certs
openssl genrsa 2024 > server.key
openssl req -new -key server.key -subj "/C=JP" > server.csr
openssl x509 -req -days 3650 -signkey server.key < server.csr > server.crt
cd -

Second, create my_echo_server.js as follows.

// my_echo_server.js

const { createQuicSocket } = require('net');
const fs = require('fs');

const key  = fs.readFileSync('./ssl_certs/server.key');
const cert = fs.readFileSync('./ssl_certs/server.crt');
const ca   = fs.readFileSync('./ssl_certs/server.csr');
const port = 1234;

// Create the QUIC UDP IPv4 socket bound to local IP port 1234
const server = createQuicSocket({ endpoint: { port } });

// Tell the socket to operate as a server using the given
// key and certificate to secure new connections, using
// the fictional 'hello' application protocol.
server.listen({ key, cert, alpn: 'hello' });

server.on('session', (session) => {
  // The peer opened a new stream!
  session.on('stream', (stream) => {
    // Echo server
    stream.pipe(stream);
  });
});

server.on('listening', () => {
  // The socket is listening for sessions!
  console.log(`listening on ${port}...`);
  console.log('input something!');
});

const socket = createQuicSocket({
  client: {
    key,
    cert,
    ca,
    requestCert: true,
    alpn: 'hello',
    servername: 'localhost'
  }
});

const req = socket.connect({
  address: 'localhost',
  port,
});

req.on('secure', () => {
  const stream = req.openStream();
  // stdin -> stream
  process.stdin.pipe(stream);
  stream.on('data', (chunk) => console.log('client(on-secure): ', chunk.toString()));
  stream.on('end', () => console.log('client(on-secure): end'));
  stream.on('close', () => {
    // Graceful shutdown
    socket.close();
  });
  stream.on('error', (err) => console.error(err));
});

Next, enter a docker container as follows.

docker run -it -v $PWD:/playground nwtgck/node-quic bash

And, type the following commands in the container.

# Move the playground
cd /playground/
# Run echo server
node my_echo_server.js

In the container, you can use existing packages because npm command is available. You can expose UDP port by -p 1234:1234/udp with docker run.

Demo

Here is a demo.

Your inputs on stdin are echoed by the server.

Learn more

Official document: quic/quic.md at cee2e5d079ca2b55e421d81df1ad131c1bfeecc6 · nodejs/quic
Echo server ref: A QUIC Update for Node.js
My Japanese post: https://scrapbox.io/nwtgck/Node.js%E3%81%AEQUIC%E3%82%92%E5%85%88%E5%8F%96%E3%82%8A%E3%81%97%E3%81%A6%E4%BD%BF%E3%81%A3%E3%81%A6%E3%81%BF%E3%82%88%E3%81%86

Simple Chat over only HTTP without WebSocket and WebRTC, using Piping Server

Ryo Ota — Thu, 06 Jun 2019 16:01:38 +0000

Hi all! I'd like to show you a simple text chat over HTTP/HTTPS using Piping Server, which was introduced in Data Streaming between Every Device over HTTP/HTTPS.

Demo video

Here is a simple demo video of the chat.

How to use Piping Server in JavaScript?

The purpose of Piping Server is data transfer.
Here is an usage. Send by POST method, Get by GET method.

// Send
fetch("https://ppng.io/mytext", {
  method: "POST",
  body: 'hello, world'
})

// Get
const res = await fetch("https://ppng.io/mytext")
await res.text() // === 'hello, world'

You can change /mytext and "https://ppng.io" freely. An easy way to run Piping Server is using Heroku. You can click [Deploy to Heroku] on Piping Server on GitHub.

By using this, you can build a simple chat without WebSocket or WebRTC.

Try on CodePen

Codepen: https://codepen.io/nwtgck/pen/xNoKgx

Both CodePens are the same.

Full code

Here is the whole code of chat.

<html>
<head>
  <title>Simple Chat via Piping Server</title>
  <style>
    .me {
      text-align: right;
    }
  </style>
</head>
<body>
  <p>
    <input placeholder="Your ID" id='your_id'>
    <input placeholder="Peer ID" id='peer_id'>
    <button onclick='receiveLoop(this)'>Connect</button>
  </p>
  <p style='position: absolute; bottom: 0;'>
    <input placeholder="Message" id='message' size='50'>
    <button onclick="send()">Send</button>
  </p>

  <hr>
  <div id='talks'>
    <!--This will be added by JavaScript -->
  </div>
  <script>
    // Receive-loop
    async function receiveLoop(btn) {
      your_id.disabled = peer_id.disabled = btn.disabled = true;
      while(true) {
        try {
          // Get peer's response
          const res = await fetch(`https://ppng.io/${peer_id.value}-${your_id.value}`);
          // Create talk element
          const talk = document.createElement('div');
          // Set peer's message
          talk.innerText = await res.text();
          // Add peer's message
          talks.appendChild(talk);
        } catch(err) {
          console.error(err);
        }
      }
    }

    // Send your message
    function send() {
      // Send your message
      fetch(`https://ppng.io/${your_id.value}-${peer_id.value}`, {
        'method': 'POST',
        body: message.value
      });
      // Create talk element
      const talk = document.createElement('div');
      talk.innerText = message.value;
      talk.classList.add('me');
      talks.appendChild(talk);
      // Empty your message
      message.value = '';
    }
  </script>
</body>
<html>

Secure chat via Piping Server

You can find more secure chat in the following. The application is written in Vue in TypeScript.
GitHub: https://github.com/nwtgck/piping-chat-web

The features are the following.

End-to-End Encryption by AES GCM
Forward Secrecy by ECDH
Public Key Authentication like SSH
via Piping Server
Static hosting
Progressive Web App (PWA)
Accountless

Thanks

@karanganesan told me that a simpler example will be useful. This is why I wrote this post. Thanks Karan!

can you share any simpler examples of using Piping Server like : only pure vanilla javascript and bare minimum like working chat using piping without any Vue JS or anything.

Data Streaming between Every Device over HTTP/HTTPS

Ryo Ota — Sat, 09 Feb 2019 10:41:17 +0000

Hi all! I'm very new to the DEV community. Today, I'd like to introduce data stream transfer between almost every device over HTTP/HTTPS made for use with Unix Pipe and also for browser!

What do I want to solve?

We sometimes want to transfer data between Mac, Windows, Linux, Unix, iPhone, Android... Although we have AirDrop, Google Drive, Dropbox, Slack, WhatsApp, Skype, netcat, ssh or etc, we need to

Find a common service used by both sender and receiver
Install additional GUI software
Sign up some service
Solve NAT traversal

In addition, almost existing data transfer services are not CUI-friendly or not GUI-friendly.

So, I made a solution to the problems above. The system allows you to

Transfer data with almost every device
Use it without additional installation
Use it with Unix/Linux Pipe and it's engineer-friendly

Actual Usages

I'd like to introduce how to use the system I made.

Here is a demo to transfer seq 100000.

Send: seq 100000 | curl -T - https://ppng.io/myseq
Get: curl https://ppng.io/myseq

Here is a demo to transfer a 100MB file.

Send: cat 100MB.dat | curl -T - https://ppng.io/mydata
Get: curl https://ppng.io/mydata > mydata

You can transfer any data by using pipe. For example,

Compress & Send: cat myfile | gzip | curl -T ...
Get & Decompress: curl ... | zcat > myfile
and
Encrypt & Send: cat myfile | openssl aes-256-cbc | curl -T ...
Get & Decript: curl ... | openssl aes-256-cbc -d > myfile
and
Send directory(zip): zip -q -r - ./mydir | curl -T - https://ppng.io/mydir.zip
Send directory(tar.gz): tar zfcp - ./mydir | curl -T - https://ppng.io/mydir.tar.gz

Here is a demo to transfer data to a browser.

Here is a demo to transfer data from a browser.

This means you can transfer a file from Android/iPhone/iPad... to another device!

Transfer to Multiple Receivers

Terminal Screen Sharing!

Share: script -f >(curl -T - https://ppng.io/myscript >&/dev/null)
View: curl https://ppng.io/myscript
(Use script -F in Mac instead of script -f)

GitHub Repository

The project is Piping Server found in the link below.
GitHub: https://github.com/nwtgck/piping-server

nwtgck / piping-server

Infinitely transfer between every device over pure HTTP with pipes or browsers

Piping Server

Infinitely transfer between every device over HTTP/HTTPS

Transfer

Piping Server is simple. You can transfer as follows.

# Send
echo 'hello, world' | curl -T - https://ppng.io/hello

# Get
curl https://ppng.io/hello > hello.txt

You can also use Web UI like https://ppng.io on your browser. A more modern UI is found in https://piping-ui.org, which supports E2E encryption.

Stream

The most important thing is that the data are streamed. This means that you can transfer any data infinitely. The demo below transfers an infinite text stream with seq inf.

Ideas

…

View on GitHub

This project is written in TypeScript.

Public Servers

Here are public servers. My recommendation is https://ppng.io because it has a short name.

In my policy, you can freely run the servers. I'd like to describe how to run later.

You can get engineer-friendly help from the following.

$ curl https://ppng.io/help
<Help will be displayed here>

How it works

I think Piping Server is like a TURN server in Pear-to-Pear. Piping Server relays your data to another device to solve NAT traversal.

A body of POST/PUT HTTP request is passed to a body of GET HTTP response.

How to Run Piping Server

We have several ways to run Piping Server. I'd like to introduce one by one.

Heroku

Push [Deploy to Heroku] button in https://github.com/nwtgck/piping-server.

Portable Binary Executable

The following commands make a Piping Server run on http://localhost:8888.

wget https://github.com/nwtgck/piping-server-pkg/releases/download/v0.8.7/piping-server-macos
chmod +x piping-server-macos
./piping-server-macos --http-port=8888

You can also find other platform releases in GitHub Releases in https://github.com/nwtgck/piping-server-pkg. The portable executables are produced by zeit/pkg.

Docker

A docker run make a Piping Server run on http://localhost:8888.

docker run -d --restart=always -p 8888:80 nwtgck/piping-server --http-port=80

npm

The following commands make Piping Server run on http://localhost:8888.

# install
npm install -g piping-server
# run
piping-server --http-port=8888

Possibility of Data Transfer over HTTP

I'm conducting an experiment to reveal how many data can be transferred over HTTP.
I started the experiment 55 days and 7 hours ago and over 1000 terabyte data were transferred.

The method of the experiment is as follows.
Infinitely send: cat /dev/urandom | curl -T - localhost:8888/rand
Infinitely discard: curl localhost:8888/rand > /dev/null

Transfer Speed Compared with Go

Here is a simple comparison in transfer speed between TypeScript/Node.js and Go implementations. As a result of the videos, there seem to be almost no differences.

TypeScript/Node.js:

Go:

Repo Go ver.: https://github.com/nwtgck/go-piping-server

Japanese Post

Here is my original Japanese post: https://qiita.com/nwtgck/items/78309fc529da7776cba0.
You can get more information there.

Thank you very much for reading. Have a wonderful day!