DEV Community

Ryo Ota
Ryo Ota

Posted on • Updated on

Data Streaming between Every Device over HTTP/HTTPS

Hi all! I'm very new to the DEV community. Today, I'd like to introduce data stream transfer between almost every device over HTTP/HTTPS made for use with Unix Pipe and also for browser!

What do I want to solve?

We sometimes want to transfer data between Mac, Windows, Linux, Unix, iPhone, Android... Although we have AirDrop, Google Drive, Dropbox, Slack, WhatsApp, Skype, netcat, ssh or etc, we need to

  • Find a common service used by both sender and receiver
  • Install additional GUI software
  • Sign up some service
  • Solve NAT traversal

In addition, almost existing data transfer services are not CUI-friendly or not GUI-friendly.

So, I made a solution to the problems above. The system allows you to

  • Transfer data with almost every device
  • Use it without additional installation
  • Use it with Unix/Linux Pipe and it's engineer-friendly

Actual Usages

I'd like to introduce how to use the system I made.

Here is a demo to transfer seq 100000.

piping seq
Send: seq 100000 | curl -T - https://ppng.io/myseq
Get: curl https://ppng.io/myseq


Here is a demo to transfer a 100MB file.

piping 100 MB file
Send: cat 100MB.dat | curl -T - https://ppng.io/mydata
Get: curl https://ppng.io/mydata > mydata


You can transfer any data by using pipe. For example,

Compress & Send: cat myfile | gzip | curl -T ...
Get & Decompress: curl ... | zcat > myfile
and
Encrypt & Send: cat myfile | openssl aes-256-cbc | curl -T ...
Get & Decript: curl ... | openssl aes-256-cbc -d > myfile
and
Send directory(zip): zip -q -r - ./mydir | curl -T - https://ppng.io/mydir.zip
Send directory(tar.gz): tar zfcp - ./mydir | curl -T - https://ppng.io/mydir.tar.gz


Here is a demo to transfer data to a browser.

piping from CUI to broswer


Here is a demo to transfer data from a browser.

piping from browser to CUI

This means you can transfer a file from Android/iPhone/iPad... to another device!


Transfer to Multiple Receivers

piping multiple

Terminal Screen Sharing!

piping terminal share

Share: script -f >(curl -T - https://ppng.io/myscript >&/dev/null)
View: curl https://ppng.io/myscript
(Use script -F in Mac instead of script -f)

GitHub Repository

The project is Piping Server found in the link below.
GitHub: https://github.com/nwtgck/piping-server

GitHub logo nwtgck / piping-server

Infinitely transfer between every device over pure HTTP, designed for everyone using Unix pipe and even for browser users

Piping Server

npm CodeFactor CircleCI Build status GitHub Actions Docker Automated build Dependabot Status

Infinitely transfer between every device over HTTP/HTTPS
Piping Server hello

Transfer

Piping Server is simple. You can transfer as follows.

# Send
echo 'hello, world' | curl -T - https://ppng.io/hello
Enter fullscreen mode Exit fullscreen mode
# Get
curl https://ppng.io/hello > hello.txt
Enter fullscreen mode Exit fullscreen mode

Piping Server transfers data to POST /hello or PUT /hello into GET /hello. The path /hello can be anything such as /mypath or /mypath/123/. A sender and receivers who specify the same path can transfer. Both the sender and the recipient can start the transfer first. The first one waits for the other.

You can also use Web UI like https://ppng.io on your browser. A more modern UI is found in https://piping-ui.org, which supports E2E encryption.

Stream

The most important thing is that the data are streamed. This means that you can transfer any data infinitely. The demo below transfers an infinite text stream with seq inf.

infnite text stream

Ideas

This project is written in TypeScript.

Public Servers

Here are public servers. My recommendation is https://ppng.io because it has a short name.

In my policy, you can freely run the servers. I'd like to describe how to run later.

You can get engineer-friendly help from the following.

$ curl https://ppng.io/help
<Help will be displayed here>
Enter fullscreen mode Exit fullscreen mode

How it works

I think Piping Server is like a TURN server in Pear-to-Pear. Piping Server relays your data to another device to solve NAT traversal.

A body of POST/PUT HTTP request is passed to a body of GET HTTP response.

How to Run Piping Server

We have several ways to run Piping Server. I'd like to introduce one by one.

Heroku

Push [Deploy to Heroku] button in https://github.com/nwtgck/piping-server.

Portable Binary Executable

The following commands make a Piping Server run on http://localhost:8888.

wget https://github.com/nwtgck/piping-server-pkg/releases/download/v0.8.7/piping-server-macos
chmod +x piping-server-macos
./piping-server-macos --http-port=8888
Enter fullscreen mode Exit fullscreen mode

You can also find other platform releases in GitHub Releases in https://github.com/nwtgck/piping-server-pkg. The portable executables are produced by zeit/pkg.

Docker

A docker run make a Piping Server run on http://localhost:8888.

docker run -d --restart=always -p 8888:80 nwtgck/piping-server --http-port=80
Enter fullscreen mode Exit fullscreen mode

npm

The following commands make Piping Server run on http://localhost:8888.

# install
npm install -g piping-server
# run
piping-server --http-port=8888
Enter fullscreen mode Exit fullscreen mode

Possibility of Data Transfer over HTTP

I'm conducting an experiment to reveal how many data can be transferred over HTTP.
I started the experiment 55 days and 7 hours ago and over 1000 terabyte data were transferred.

The method of the experiment is as follows.
Infinitely send: cat /dev/urandom | curl -T - localhost:8888/rand
Infinitely discard: curl localhost:8888/rand > /dev/null

Transfer Speed Compared with Go

Here is a simple comparison in transfer speed between TypeScript/Node.js and Go implementations. As a result of the videos, there seem to be almost no differences.

TypeScript/Node.js:

Go:

Repo Go ver.: https://github.com/nwtgck/go-piping-server

Japanese Post

Here is my original Japanese post: https://qiita.com/nwtgck/items/78309fc529da7776cba0.
You can get more information there.

Thank you very much for reading. Have a wonderful day!

Discussion (8)

Collapse
normanr profile image
Norman Rasmussen

Your tests may be limited by yes and/or cat and/or /dev/urandom. What about using /dev/zero with dd instead? (dd also allows you to control the block size.) You can test the speed of input data with pv.

Also the CPU usage of the server needs to be considered too, otherwise CPU used by cat/curl and other tools is competing and not taken into account.

Collapse
nwtgck profile image
Ryo Ota Author • Edited

Thank you very much for your smarter suggestion! I thought the comparison was too simple. When I tried another speed comparison, I'll use your suggestion, and server, senders and receivers should be run probably in different machines, considering CPU usages.

Collapse
kylegalbraith profile image
Kyle Galbraith

This is very slick Ryo and I enjoyed the samples you shared in this post. Although it looks like one GIF for the 100MB file is not actually doing that, it would be fun to see that as well.

Have you tried running this inside of a serverless function like AWS Lambda?

Collapse
nwtgck profile image
Ryo Ota Author • Edited

Thanks! I replaced the GIF for 100MB file with the proper one.

Have you tried running this inside of a serverless function like AWS Lambda?

I haven't tried. I don't think Piping Server is designed for serverless or FaaS.
In my comprehension, a serverless function should be stateless. On the other hand, Piping Server has states in each path. Piping Server doesn't store data in storage or database. Data passed from one device to another one are only on memory. That's why I think Piping Server has states.

Collapse
normanr profile image
Norman Rasmussen

Piping is great for one way transfer! For two way similar services are serveo.net or ngrok.com.

Collapse
nwtgck profile image
Ryo Ota Author • Edited

Thanks! I've never known serveo.net, it's so useful, no-install, no-signup!

I think that a benefit of Piping Server, which serveo and ngrok might not have is allowing users to transfer data even from Android/iPhone smartphones with pre-installed browsers.

Collapse
normanr profile image
Norman Rasmussen

Yep, Piping Server is best for http upload/download where data only needs to flow in one direction. serveo is useful if you need two way communications for any reason.

Thread Thread
nwtgck profile image
Ryo Ota Author • Edited

Yes! I agree with you. serveo is so useful, it allows me to use it with only ssh and communicate another device in two way. I knew serveo because you told me it. Thank you very much!

Here is an article which describes two-way communication via Piping Server with only socat and curl. The article was written by another Japanese not me. In it, Piping Server is used like serveo.

Japanese: qiita.com/Cryolite/items/ed8fa237d...
Google Translate: translate.google.com/translate?hl=...