Building a HTTP Tunnel with WebSocket and Node.JS Stream

Embbnux Ji — Wed, 09 Mar 2022 12:09:56 +0000

When developing apps or bots that integrate with third-party services, it is common to need to expose the local development server to the internet to receive Webhook messages. To achieve this, an HTTP tunnel is required for the local server. This article demonstrates how to use WebSocket and Node.js streams to build an HTTP tunnel tool and transfer big data.

Why Deploy Your Own HTTP Tunnel Service

Many online services provide HTTP tunnels, such as ngrok, which offer paid fixed public domains to connect the local server. It also has a free package, but it only provides a random domain that changes each time the client restarts, making it inconvenient to save the domain in third-party services.

To get a fixed domain, you can deploy your own HTTP tunnel on your server. ngrok also provides an open-source version for server-side deployment, but it is an old 1.x version with some serious reliability issues and not recommended for production.

In addition, with your own server, you can ensure data security.

Introduction about Lite HTTP Tunnel project

Lite HTTP Tunnel is a recently developed HTTP tunnel service that can be self-hosted. You can use the Deploy button in the Github repository to deploy it and obtain a fixed domain for free.

It is built based on Express.js and Socket.io with just a few lines of code. It uses WebSocket to stream HTTP/HTTPS requests from the public server to your local server.

Implementation

Step 1: Build a WebSocket Connection Between Server and Client

To support WebSocket connections at the server-side, we use socket.io:

const http = require('http');
const express = require('express');
const { Server } = require('socket.io');

const app = express();
const httpServer = http.createServer(app);
const io = new Server(httpServer);

let connectedSocket = null;

io.on('connection', (socket) => {
  console.log('client connected');
  connectedSocket = socket;
  const onMessage = (message) => {
    if (message === 'ping') {
      socket.send('pong');
    }
  }
  const onDisconnect = (reason) => {
    console.log('client disconnected: ', reason);
    connectedSocket = null;
    socket.off('message', onMessage);
    socket.off('error', onError);
  };
  const onError = (e) => {
    connectedSocket = null;
    socket.off('message', onMessage);
    socket.off('disconnect', onDisconnect);
  };
  socket.on('message', onMessage);
  socket.once('disconnect', onDisconnect);
  socket.once('error', onError);
});

httpServer.listen(process.env.PORT);

To connect the WebSocket at the client-side:

const { io } = require('socket.io-client');

let socket = null;

function initClient(options) {
  socket = io(options.server, {
    transports: ["websocket"],
    auth: {
      token: options.jwtToken,
    },
  });

  socket.on('connect', () => {
    if (socket.connected) {
      console.log('client connect to server successfully');
    }
  });

  socket.on('connect_error', (e) => {
    console.log('connect error', e && e.message);
  });

  socket.on('disconnect', () => {
    console.log('client disconnected');
  });
}

Step2: Use JWT Token to Protect the WebSocket Connection

At the server-side, we use socket.io middleware to reject invalid connections:

const jwt = require('jsonwebtoken');

io.use((socket, next) => {
  if (connectedSocket) {
    return next(new Error('Connected error'));
  }
  if (!socket.handshake.auth || !socket.handshake.auth.token){
    next(new Error('Authentication error'));
  }
  jwt.verify(socket.handshake.auth.token, process.env.SECRET_KEY, function(err, decoded) {
    if (err) {
      return next(new Error('Authentication error'));
    }
    if (decoded.token !== process.env.VERIFY_TOKEN) {
      return next(new Error('Authentication error'));
    }
    next();
  });  
});

Step 3: Data Stream Transmission

In Node.js, both HTTP Request and Response are streams. On the server side, Request is a Readable stream, while Response is a Writable stream.
The normal data stream transmission in a node.js web server is shown in the following diagram:

And now that our Web Server is inside the local firewall, we use a public server to forward the Request and Response through. Therefore, the user's HTTP data first goes to the Tunnel server, which sends the Request to the Tunnel client, and then the Tunnel client sends the Request to the Local web server to get the Response, which is finally returned to the Tunnel server for transmission to the Client side.

To transmit the Request and Response streams between the Tunnel server and Tunnel client, we implement a TunnelRequest writable stream class and TunnelResponse readable stream class on the Tunnel server side, based on WebSocket, and a TunnelRequest readable stream class and TunnelResponse writable stream class on the Tunnel client side.

Tunnel server side：

const { Writable, Readable } = require('stream');

class TunnelRequest extends Writable {
   // ...
}

class TunnelResponse extends Readable {
  // ...
}

Tunel client side:

const { Writable, Readable } = require('stream');

class TunnelRequest extends Readable {
   // ...
}

class TunnelResponse extends Writable {
   // ...
}

To learn more about Node.js stream, you can refer to the official documentation. For the implementation of TunnelRequest and TunnelResponse, you can visit https://github.com/web-tunnel/lite-http-tunnel/blob/main/lib.js.

After completing all of the above steps, we now support streaming HTTP requests to a local computer and sending responses from the local server back to the original request. This is a lightweight solution, but it is highly stable and easy to deploy in any Node.js environment.

Step 4: Deploy HTTP Tunnel service

We can deploy the HTTP tunnel service to a cloud provider such as Heroku/Render. The project Lite HTTP Tunnel contains a Heroku/Render button in the Github repository, which allows you to deploy the service to Heroku/Render quickly.

So we have introduced about how to transfer HTTP requests based on WebSocket and Node.js Writable and Readable stream. In latest version of Lite HTTP Tunnel, we refactor the project with Duplex stream to support requests from WebSocket. You can check that from source code.

ShareService: a library for building multiple tabs react app

Embbnux Ji — Sat, 21 Aug 2021 17:30:27 +0000

We can use React to create single page apps easily, React helps us to render UI state. But for every new tab, app need to run our business logics and services to have data for react to rendering. So can we centralize our business logics and services?

SharedWorker is the solution. And we have shared-service library to help you centralize your business logics and services and build a multiple tabs react app without pain.

How it work

SharedWorker instance will be reused by cross browser tabs with same domain until all pages are closed.

App creates SharedWorker to load data generate UI state when user open first tab.
App gets UI state from existing SharedWorker instance when user load second tab.
App send action to execute in SharedWorker instance when user has action in web page
SharedWorker instance send UI state to tabs that use it after action executing

Work with ShareService library

I assume you have start a react app with create-react-app.

1 . First step: install it

$ npm install @shared-service/core @shared-service/react

2 . Init shared service in your react app root endpoint:

3 . Connect Shared Service in ShareWorker worker.js file:

4 . In react component:

For simple state logic, we can just use setCount function return useSharedState to update it into SharedService.
For complex actions, we can send it to execute in sharedServiceServer.
Firstly register executor into sharedServiceServer at worker.js:

Send the action in react component:

5 . Data persistence

We can persist data in SharedWorker with IndexedDB. Here we use localforage lib to help us handle storage.

Online demo

We have online demo here. For full demo source code, please refer here.

Problem

As SharedWorker is not supported for Safari and IE now, so we can’t run share-service at the shared worker for those Browser. You can take a look at here to make app work at Safari.

There are some TODO for shared-service library:

Add support for Vue.js library.
Support Browser extension (with background API)
Support Electron main and render process

To make Browser extension and Electron app support the centralization service.

Original link

DEV Community: Embbnux Ji