Hi all! I'm very new to the DEV community. Today, I'd like to introduce data stream transfer between almost every device over HTTP/HTTPS made for use with Unix Pipe and also for browser!
We sometimes want to transfer data between Mac, Windows, Linux, Unix, iPhone, Android... Although we have AirDrop, Google Drive, Dropbox, Slack, WhatsApp, Skype,
ssh or etc, we need to
- Find a common service used by both sender and receiver
- Install additional GUI software
- Sign up some service
- Solve NAT traversable
In addition, almost existing data transfer services are not CUI-friendly or not GUI-friendly.
So, I made a solution to the problems above. The system allows you to
- Transfer data with almost every device
- Use it without additional installation
- Use it with Unix/Linux Pipe and it's engineer-friendly
I'd like to introduce how to use the system I made.
Here is a demo to transfer
Here is a demo to transfer a 100MB file.
You can transfer any data by using pipe. For example,
Compress & Send:
cat myfile | gzip | curl -T ...
Get & Decompress:
curl ... | zcat > myfile
Encrypt & Send:
cat myfile | openssl aes-256-cbc | curl -T ...
Get & Decript:
curl ... | openssl aes-256-cbc -d > myfile
zip -q -r - ./mydir | curl -T - https://ppng.ml/mydir.zip
tar zfcp - ./mydir | curl -T - https://ppng.ml/mydir.tar.gz
Here is a demo to transfer data to a browser.
Here is a demo to transfer data from a browser.
This means you can transfer a file from Android/iPhone/iPad... to another device!
script -f >(curl -T - https://ppng.ml/myscript >&/dev/null)
The project is Piping Server found in the link below.
Streaming Data Transfer Server over HTTP/HTTPS: designed for people using pipe in Unix-like OS and even for browser users
Streaming Data Transfer Server over HTTP/HTTPS
Piping server is designed to realize the following three ideas.
- Engineer friendly
Usually, when you want to send data to someone, you need to have an account in the common service among you and his/her In addition, the service may require you to install specific software to share the data.
Piping server is designed to solve the problems above
Piping server allows you to use without any account and frees you from additional software installation because the server is available with Web browsers,
wget commands, which are widely pre-installed in Unix-like OS
Because of HTTP/HTTPS, users can send data across different devices such as Windows, macOS, Unix, Linux, iOS, Android and any device which uses HTTP.
Furthermore, pipe in Unix-like OS allows you to send data in a more efficient and secure way.
This project is written in TypeScript.
Here are public servers. My recommendation is https://ppng.ml because it has a short name.
In my policy, you can freely run the servers. I'd like to describe how to run later.
You can get engineer-friendly help from the following.
$ curl https://ppng.ml/help <Help will be displayed here>
I think Piping Server is like a TURN server in Pear-to-Pear. Piping Server relays your data to another device to solve NAT traversal.
A body of POST/PUT HTTP request is passed to a body of GET HTTP response.
We have several ways to run Piping Server. I'd like to introduce one by one.
Push [Deploy to Heroku] button in https://github.com/nwtgck/piping-server.
The following commands make a Piping Server run on http://localhost:8888.
wget https://github.com/nwtgck/piping-server-pkg/releases/download/v0.8.7/piping-server-macos chmod +x piping-server-macos ./piping-server-macos --http-port=8888
docker run make a Piping Server run on http://localhost:8888.
docker run -d --restart=always -p 8888:80 nwtgck/piping-server --http-port=80
The following commands make Piping Server run on http://localhost:8888.
# install npm install -g piping-server # run piping-server --http-port=8888
I'm conducting an experiment to reveal how many data can be transferred over HTTP.
I started the experiment 55 days and 7 hours ago and over 1000 terabyte data were transferred.
The method of the experiment is as follows.
cat /dev/urandom | curl -T - localhost:8888/rand
curl localhost:8888/rand > /dev/null
Here is a simple comparison in transfer speed between TypeScript/Node.js and Go implementations. As a result of the videos, there seem to be almost no differences.
Repo Go ver.: https://github.com/nwtgck/go-piping-server
Here is my original Japanese post: https://qiita.com/nwtgck/items/78309fc529da7776cba0.
You can get more information there.
Thank you very much for reading. Have a wonderful day!