Pipy proxy: Creating HTTP Tunnel with 10 lines of code

Pipy is an open-source, lightweight, high-performance, modular, programmable, cloud-native network stream processor that is ideal for a variety of use cases ranging from (but not limited to) edge routers, load balancers & proxy solutions, API gateways, static HTTP servers, service mesh sidecars, and other applications. Previously we have covered several use cases, and in this blog post will continue our learning of Pipy by implementing an HTTP Tunnel functionality by only using the PipyJS scripting language.

HTTP tunneling is a technique used to transmit data over the internet through a network that does not support the desired data transfer protocol. It involves encapsulating the data in a way that allows it to be transmitted over the internet using the HTTP protocol, which is widely supported by web servers and clients.

There are several reasons why HTTP tunneling might be used. For example, it can be used to bypass firewalls or other network security measures that block certain types of data transfer. It can also be used to access resources on a private network from a remote location or to connect to a network that uses a proprietary protocol not supported by the client.

To use HTTP tunneling, a client sends a request to a server using the HTTP protocol, with the desired data included in the request body. The server then sends a response back to the client using the HTTP protocol, with the data included in the response body. This allows the data to be transmitted over the internet using the HTTP protocol, bypassing any restrictions or limitations on the network.

Solution

Suppose we have two devices communicating with each other using a private protocol based on TCP, and the firewall only allows HTTP communication. The server is listening on an internal IP and port 8081. Here we use Pipy to simulate a TCP server that responds with "Hi, TCP!" (mimicking the private protocol) when it receives a request.

pipy()

  .listen('127.0.0.1:8081')
  .replaceData(
    () => new Data('Hi, TCP!\n')
  )

Next, we need a proxy on the DMZ side to provide HTTP tunneling. The client establishes an HTTP connection with the proxy (using the HTTP CONNECT method). When the connection is successful, the proxy establishes a connection with the destination server as requested by the client, connecting the client and server. The proxy then forwards the TCP stream sent by the client to the server and sends the server's response back to the client.

PipyJS script

The scripting part is simple. We have a proxy listening on port 80 that checks if the request header is an HTTP CONNECT request. If it is, the proxy retrieves the address and port of the destination server from the path in the request header and establishes a connection with the destination server. If it is not an HTTP CONNECT request, it is treated as a regular HTTP request and we return a 404 Not Found! here without implementing it.

pipy({
  _isTunnel: false,
  _target: null,
})

  //http tunnel
  .listen(80)
  .demuxHTTP().to($ => $
    .handleMessageStart(
      msg => msg.head.method === 'CONNECT' && (_isTunnel = true)
    )
    .branch(
      () => _isTunnel, (
      $ => $.acceptHTTPTunnel(
        msg => (
          _target = msg.head.path,
          new Message({ status: 200 })
        )
      ).to(
        $ => $.connect(() => _target)
      )
    ), (
      //common HTTP request
      $ => $.replaceMessage(new Message({ status: 404 }, 'Not Found!\n'))
    )
    )
  )

Here we have a special filter called acceptHTTPTunnel Filter that implements HTTP tunneling on the server side (there is another filter called connectHTTPTunnel Filter on the client side, which we will talk about later). It redirects the TCP stream after the HTTP CONNECT to a child pipeline (to filter in the above code).

Compared to a regular HTTP proxy, there is only the additional acceptHTTPTunnel filter.

Testing

Our proxy and server are running on host 192.168.1.110, and the client curl is running on host 192.168.1.11.

curl -p -x http://192.168.1.110:80 telnet://127.0.0.1:8081

curl uses the -x (--proxy) option to specify the proxy server and the -p (--proxytunnel) option to use the proxy HTTP tunnel.

What could be done next?

This time, our client curl happens to support using HTTP tunnels. What if the client doesn't support it? That's where connectHTTPTunnel, which we mentioned earlier, comes in. This filter provides HTTP tunneling functionality on the client side: first, it sends an HTTP CONNECT request to establish the tunnel, then it sends the TCP stream through the tunnel, as shown in the following figure.

Interested readers can practice their PipyJS skills by trying to implement this and let us know by posting comments.

Summary

We used about 10 lines of code to add tunneling capabilities to an HTTP proxy. This article briefly introduced the implementation of tunneling, but the function of tunneling is not just to solve connectivity problems. High-level HTTP tunnels can also provide features that lower-level protocols cannot, such as various security authentications to enhance security.

The egress gateway in the service mesh osm-edge provides HTTP tunneling functionality, using an HTTP tunnel to provide governance functions at the high-level protocol between the sidecar and the egress gateway. We will explain this in more detail when we introduce the egress gateway.