DEV Community: Kazuki Higashiguchi

Go 1.18: Cut added to strings/bytes

Kazuki Higashiguchi — Wed, 16 Feb 2022 11:09:26 +0000

Key takeaways

Go 1.18 is expected to be released in February 2022.
New function Cut added to strings and bytes package, which will make our Go code pretty simpler.

Go 1.18

Go 1.18 includes several major new features, generics(Type parameters), Fuzzing and etc.

The official release note is available on the following link.

https://tip.golang.org/doc/go1.18

Go 1.18 is expected to be released in February 2022, but you can download and use the beta version Go 1.18 Beta 2 as the following commands:

$ go install golang.org/dl/go1.18beta2@latest
$ go1.18beta2 download

$ go1.18beta2 version
go version go1.18beta2 darwin/arm64

⚠️ Installing executables with go get is deprecated in Go 1.17. You should use go install instead as shown in the above command example. ref: Deprecation of 'go get' for installing executables

strings.Cut/ bytes.Cut

As always, there are various minor changes and updates to the library. This article brings the Cut function added to strings and bytes package as a topic.

The Cut function is described like this in Go 1.18 release notes.

The new Cut function slices a []byte around a separator. It can replace and simplify many common uses of Index, IndexByte, IndexRune, and SplitN.

The signature of Cut is as follow:

func Cut(s, sep []byte) (before, after []byte, found bool) {

Let's take a look at an example. Here is an example of an implementation that parses username and password from a given header string during basic authentication.

First, if using Go 1.17 and prior...

package main

import (
    "encoding/base64"
    "fmt"
    "strings"
)

const prefix = "Basic "

func main() {
    auth := "Basic R28xMTg6d2VsY29tZSBjdXQ="
    const prefix = "Basic "
    base64Decoded, err := base64.StdEncoding.DecodeString(auth[len(prefix):])
    if err != nil {
        fmt.Printf("error while base64 decoding: %v\n", err)
        return
    }

    decodedString := string(base64Decoded)
    fmt.Printf("decoded: %s\n", decodedString)
    // Output: Go118:welcome cut

    separatorIndex := strings.IndexByte(decodedString, ':')
    if separatorIndex < 0 {
        fmt.Println("not a basic authentication format.")
        return
    }
    username := decodedString[:separatorIndex]
    password := decodedString[separatorIndex+1:]
    fmt.Printf("username: %q, password: %q\n", username, password)
    // username: "Go118", password: "welcome cut"
}

(Available on Go playground)

In this example, find an index of the separator ":" using strings.IndexByte and get username "Go118" and password "welcome cut".

separatorIndex := strings.IndexByte(decodedString, ':')
if separatorIndex < 0 {
    fmt.Println("not a basic authentication format.")
    return
}
username := decodedString[:separatorIndex]
password := decodedString[separatorIndex+1:]

The Cut function makes this code simpler. Let's modify this code using strings.Cut.

username, password, ok := strings.Cut(decodedString, ":")
if !ok {
    fmt.Println("not a basic authentication format.")
    return
}

(Available on Go playground)

Awesome, The code improved pretty simpler! In Go 1.18 and later, you may be lots of opportunities to use the Cut function in your daily Go coding.

The problem Cut function addresses

Then, why does Go add the Cut function? Let's take a look at the proposal.

https://github.com/golang/go/issues/46336

The proposal mentioned the effectiveness of Cut by searching for use of string.Index, strings.IndexBytes, or strings.IndexRune that could use strings.Cut instead in the main Go repository.

That leaves 285 calls. Of those, 221 were better written as Cut, leaving 64 that were not.
That is, 77% of Index calls are more clearly written using Cut. That's incredible!

77% of Index calls, it's incredible number! It found out Cut would replace and simplify the overwhelming majority of the usage of these functions:

eq := strings.IndexByte(rec, '=')
if eq == -1 {
    return "", "", s, ErrHeader
}
k, v = rec[:eq], rec[eq+1:]

with

k, v, ok = strings.Cut(rec, "=")
if !ok {
    return "", "", s, ErrHeader
}

In fact, lots of standard libraries was updated using Cut after implementing it.

https://go-review.googlesource.com/c/go/+/351711

Implementation of Cut

The implementation of Cut looks very short and simple. The Cut function was added to strings and bytes libraries in 351710: bytes, strings: add Cut.

func Cut(s, sep []byte) (before, after []byte, found bool) {
    if i := Index(s, sep); i >= 0 {
        return s[:i], s[i+len(sep):], true
    }
    return s, nil, false
}

The first edition of the current signature was proposed on November 12, 2020.

And then, renamed the third result from ok to found to avoid confusion with the comma-ok form for map access, channel receive, and so on.

If you look around at other languages, you'll see that str.partition of Python and split_once of Rust have similar functionalities.

str.partition: Split the string at the first occurrence of sep, and return a 3-tuple containing the part before the separator, the separator itself, and the part after the separator.

x = text.partition("delimeter")

split_once: Splits the string on the first occurrence of the specified delimiter and returns prefix before delimiter and suffix after delimiter.

pub fn split_once<'a, P>(&'a self, delimiter: P) -> Option<(&'a str, &'a str)>

In particular, the Python case of str.partition helped the Go team to accept introducing Cut.

That is, the fact that str.partition is useful in Python is added evidence for Cut, but we need not adopt the Pythonic signature.

Quoted from issue#46336

History until accepted

This feature was first discussed in issue 40135 and then moved to issue 46336, which was accepted. The following timeline led to the acceptance of the proposal

July 9, 2020: Incoming (opened the issue #40135)
May 24, 2021: Incoming (moved to the issue #46336)
Jun 3, 2021: Incoming → Active
Jun 17, 2021: Active → Likely Accept
July 15, 2021: Likely Accept → Accepted

By the way, a group of Go team members holds proposal review meetings roughly weekly. You can read meeting minutes on the issue. "Incoming," "Active," and etc refer to the state of proposals.

Strictly speaking, there are 11 states of proposals as defined in golang/proposal repository.

State	Description
Incoming	New proposal.
Active	Watch for emerging consensus in the discussions.
Likely Accept	Discussion seems to have reached a consensus to accept the proposal.
Likely Decline	Discussion seems to have reached a consensus to decline the proposal.
Accepted	Accepted and it is moved out of the Proposal milestone into a work milestone.
Declined	Declined, and it is closed.
Declined as Duplicate	Declined because it duplicates a previously decided proposal.
Declined as Infeasible	Directly contradicts the core design of the language or of a package, or impossible to implement efficiently.
Declined as Retracted	Closed or retracted in a comment by the original author
Hold	Waiting for design revisions or additional information, which would take a couple weeks or more.

References

This feature was first discussed in issue 40135 and then moved to issue 46336, which was accepted. The following timeline led to the acceptance of the proposal

Proposals
- 40135: proposal: strings: add Cut | https://github.com/golang/go/issues/40135
- 46336: bytes, strings: add Cut | https://github.com/golang/go/issues/46336
Code changes
- 351710: bytes, strings: add Cut | https://go-review.googlesource.com/c/go/+/351710
- 351711: all: use bytes.Cut, strings.Cut | https://go-review.googlesource.com/c/go/+/351711
- 369981: doc/go1.18: mention bytes.Cut and strings.Cut | https://go-review.googlesource.com/c/go/+/369981
- 382954: bytes, strings: mention Cut in docs for Split and SplitN | https://go-review.googlesource.com/c/go/+/382954

Go: Deep dive into net package learning from TCP server

Kazuki Higashiguchi — Thu, 13 Jan 2022 01:12:48 +0000

FYI: A previous title was “How Go handles network and system calls when TCP server”.

Key takeaways

With the net package, you can create a TCP server with such simple and little code (The Go playground).
If the host in the address parameter is empty or a literal unspecified IP address, listen on all available unicast and anycast IP addresses of the local system.
There are two kinds of DNS resolver: Go's built-in DNS resolver and uses Cgo DNS resolver. By default, Go's built-in DNS resolver is used.
If the port in the address parameter is empty or "0", a port number is automatically chosen by the system call bind.
Go accepts an incoming connection by using the system call accept.

TCP server in Go

By using packages in standard libraries, you make it easier to write and manage your code and ensure that it's reliable. They are well designed and provide more functionality than is normally found in traditional standard libraries. In particular, net package is often used by web application engineers, which provides a portable interface for network I/O, including TCP/IO, UDP, domain name resolution, and Unix domain sockets.

With the net package, you can create a TCP server with such simple and little code (The Go playground).

package main

import (
    "fmt"
    "net"
)

func main() {
    // Listen for incoming connections.
    addr := "localhost:8888"
    l, err := net.Listen("tcp", addr)
    if err != nil {
        panic(err)
    }
    defer l.Close()
    host, port, err := net.SplitHostPort(l.Addr().String())
    if err != nil {
        panic(err)
    }
    fmt.Printf("Listening on host: %s, port: %s\n", host, port)

    for {
        // Listen for an incoming connection
        conn, err := l.Accept()
        if err != nil {
            panic(err)
        }
        // Handle connections in a new goroutine
        go func(conn net.Conn) {
            buf := make([]byte, 1024)
            len, err := conn.Read(buf)
            if err != nil {
                fmt.Printf("Error reading: %#v\n", err)
                return
            }
            fmt.Printf("Message received: %s\n", string(buf[:len]))

            conn.Write([]byte("Message received.\n"))
            conn.Close()
        }(conn)
    }
}

When you run this Go script, it will start a TCP server that listens for incoming connections on port 8888.

$ go run main.go

Listening on host: 127.0.0.1, port: 8888

If you make a TCP connection to this server with nc command, you will see that the server has successfully received the message.

$ echo -n "How's it going?" | nc localhost 8888
Message received.

The implementation looks very simple and easy but knowing what is doing in the net package makes it easier to deal with advanced issues. This article tries to deep dive into the design and implementation of the net package.

Note that the version of Go presented in this article is 1.17.5.

Listen for incoming connections

At the begging of the main function, use net.Listen to listen for incoming connections.

func main() {
    // Listen for incoming connections.
    addr := "localhost:8888"
    l, err := net.Listen("tcp", addr)
    if err != nil {
        panic(err)
    }
    defer l.Close()
    host, port, err := net.SplitHostPort(l.Addr().String())
    if err != nil {
        panic(err)
    }
    fmt.Printf("Listening on host: %s, port: %s\n", host, port)

The signature of net.Listen is as follow.

func Listen(network, address string) (Listener, error)

Network types: "tcp", "tcp4", "tcp6"

The first argument "network" must be "tcp", "tcp4", "tcp6", "unix" or "unixpacket".

"tcp4" means IPv4 only, and "tcp6" means IPv6 only. "tcp" means IPv4 or IPv6 as the comment on net.ListenConfig mentions.

Network and address parameters passed to Control method are not necessarily the ones passed to Listen. For example, passing "tcp” to Listen will cause the Control function to be called with "tcp4" or "tcp6".

Go and Cgo resolver

The second argument "address" means the local network address and there are several patterns for TCP networks.

If the host in the address parameter is empty or a literal unspecified IP address, listen on all available unicast and anycast IP addresses of the local system.

In the above example, I pass the literal unspecified IP address "localhost", and net.Listen casts as the IP "127.0.0.1".

For another example, if you pass an empty address as the following code:

// empty host
addr := ":8888"
l, err := net.Listen("tcp", addr)
if err != nil {
    panic(err)
}
defer l.Close()
host, port, err := net.SplitHostPort(l.Addr().String())
if err != nil {
    panic(err)
}
fmt.Printf("Listening on host: %s, port: %s\n", host, port)

The host and port where the server listens will be "::" and "8888".

$ go run main.go
Listening on host: ::, port: 8888

As described in the comment on net.Dial, if the host is empty or a literal unspecified IP address, for TCP and UDP ":80", "0.0.0.0:80" or "[::]:80.

For TCP, UDP and IP networks, if the host is empty or a literal unspecified IP address, as in ":80", "0.0.0.0:80" or "[::]:80" for TCP and UDP, "", "0.0.0.0" or "::" for IP, the local system is assumed.

Let's take a look at the inner implementation of this behavior. In the net package, several functions are invoked in the order shown in the following figure.

net.Listen invokes ListenConfig.Listen that announces on the local network address.

Then it creates DefaultResolver which type is net.Resolver who actually resolves the network IP address, and then invokes Resolver.resolveAddrList.

type Resolver struct {
    PreferGo bool
    StrictErrors bool
    Dial func(ctx context.Context, network, address string) (Conn, error)
    // contains filtered or unexported fields
}

From there various functions are invoked, most notably Resolver.lookupPort decides whether it uses Go's built-in DNS resolver or uses Cgo DNS resolver.

func (r *Resolver) lookupPort(ctx context.Context, network, service string) (int, error) {
    if !r.preferGo() && systemConf().canUseCgo() {
        if port, err, ok := cgoLookupPort(ctx, network, service); ok {
            if err != nil {
                // Issue 18213: if cgo fails, first check to see whether we
                // have the answer baked-in to the net package.
                if port, err := goLookupPort(network, service); err == nil {
                    return port, nil
                }
            }
            return port, err
        }
    }
    return goLookupPort(network, service)
}

Go's built-in DNS resolver sends DNS requests directly to the server listed in /etc/resolv.conf, and Cgo-based resolver calls C library routines such as getaddrinfo and getnameinfo.

By default, Go's built-in DNS resolver is used because a blocked DNS request consumes only a goroutine, while a blocked C call consumes an operating system thread. See Name Resolution in the GoDoc of the net package for more detail.

Also, You can see a use case of the Cgo DNS resolver from DNS Resolution in Go and Cgo.

A way to dynamically choose a port

In addition, if the port in the address parameter is empty or "0", a port number is automatically chosen.

// a port number will be automatically chosen
addr := "127.0.0.1:" 
l, err := net.Listen("tcp", addr)
if err != nil {
    panic(err)
}
defer l.Close()
host, port, err := net.SplitHostPort(l.Addr().String())
if err != nil {
    panic(err)
}
fmt.Printf("Listening on host: %s, port: %s\n", host, port)

For example, if you pass "127.0.0.1:" to the address parameter, a port number will be automatically chosen every time you run it.

$ go run main.go
Listening on host: 127.0.0.1, port: 51962

$ go run main.go
Listening on host: 127.0.0.1, port: 51978

$ go run main.go
Listening on host: 127.0.0.1, port: 51979

It's an interesting behavior, how is the port automatically selected? Let's take a look at the inner implementation of this behavior. Here is the overview of the net package's implementation.

As you can see, net.Listen invokes net.ListenConfig.Listen.

Then, net.Resolver.resolveAddrList resolves the address as "0" if the port in the address parameter is empty or "0". It is tested at TestLookupPort in the net package.

func TestLookupPort(t *testing.T) {
    // (omit)
    type test struct {
        network string
        name    string
        port    int
        ok      bool
    }
    var tests = []test{
        {"tcp", "0", 0, true},
        {"udp", "0", 0, true},
        {"udp", "domain", 53, true},

        {"--badnet--", "zzz", 0, false},
        {"tcp", "--badport--", 0, false},
        {"tcp", "-1", 0, false},
        {"tcp", "65536", 0, false},
        {"udp", "-1", 0, false},
        {"udp", "65536", 0, false},
        {"tcp", "123456789", 0, false},

        // Issue 13610: LookupPort("tcp", "")
        {"tcp", "", 0, true},
        {"tcp4", "", 0, true},
        {"tcp6", "", 0, true},
        {"udp", "", 0, true},
        {"udp4", "", 0, true},
        {"udp6", "", 0, true},
    }

    // (omit)

    for _, tt := range tests {
        port, err := LookupPort(tt.network, tt.name)
        if port != tt.port || (err == nil) != tt.ok {
            t.Errorf("LookupPort(%q, %q) = %d, %v; want %d, error=%t", tt.network, tt.name, port, err, tt.port, !tt.ok)
        }
        if err != nil {
            if perr := parseLookupPortError(err); perr != nil {
                t.Error(perr)
            }
        }
    }
}

Next, net.ListenConfig.Listen creates a net.sysListener that contains a Listen's parameters and configuration. When the network protocol is "tcp", net.ListenConfig.Listen invokes net.sysListener.listenTCP.

func (lc *ListenConfig) Listen(ctx context.Context, network, address string) (Listener, error) {
    addrs, err := DefaultResolver.resolveAddrList(ctx, "listen", network, address, nil)
    if err != nil {
        return nil, &OpError{Op: "listen", Net: network, Source: nil, Addr: nil, Err: err}
    }
    sl := &sysListener{
        ListenConfig: *lc,
        network:      network,
        address:      address,
    }
    var l Listener
    la := addrs.first(isIPv4)
    switch la := la.(type) {
    case *TCPAddr:
        l, err = sl.listenTCP(ctx, la)
    case *UnixAddr:
        l, err = sl.listenUnix(ctx, la)

    // (omit)
}

net.sysListener.listenTCP invokes net.internetSocket.

func (sl *sysListener) listenTCP(ctx context.Context, laddr *TCPAddr) (*TCPListener, error) {
    fd, err := internetSocket(ctx, sl.network, laddr, nil, syscall.SOCK_STREAM, 0, "listen", sl.ListenConfig.Control)
    if err != nil {
        return nil, err
    }
    return &TCPListener{fd: fd, lc: sl.ListenConfig}, nil
}

Then it invokes net.socket.

func socket(ctx context.Context, net string, family, sotype, proto int, ipv6only bool, laddr, raddr sockaddr, ctrlFn func(string, string, syscall.RawConn) error) (fd *netFD, err error) {
    // (omit)

    if fd, err = newFD(s, family, sotype, net); err != nil {
        poll.CloseFunc(s)
        return nil, err
    }

    // (omit)
}

It returns a network file descriptor that is ready for asynchronous I/O using the network poller. The type representing a network file descriptor is net.netFD.

type netFD struct {
    pfd poll.FD

    family      int
    sotype      int
    isConnected bool 
    net         string
    laddr       Addr
    raddr       Addr
}

pool.FD is the exported type representing a file descriptor which is used by the net and os packages. It is an internal package, so you can't use it directly.

Let's go back to the rest of net.socket.

func socket(ctx context.Context, net string, family, sotype, proto int, ipv6only bool, laddr, raddr sockaddr, ctrlFn func(string, string, syscall.RawConn) error) (fd *netFD, err error) {

    // (omit)

    if laddr != nil && raddr == nil {
        switch sotype {
        case syscall.SOCK_STREAM, syscall.SOCK_SEQPACKET:
            if err := fd.listenStream(laddr, listenerBacklog(), ctrlFn); err != nil {
                fd.Close()
                return nil, err
            }
            return fd, nil
        case syscall.SOCK_DGRAM:
            if err := fd.listenDatagram(laddr, ctrlFn); err != nil {
                fd.Close()
                return nil, err
            }
            return fd, nil
        }
    }
    if err := fd.dial(ctx, laddr, raddr, ctrlFn); err != nil {
        fd.Close()
        return nil, err
    }
    return fd, nil
}

As a prerequisite, TCP (Transmission Control Protocol) is a stream oriented protocol which guarantees that all bytes received will be identical and in the same order as those sent. In contract, UDP (User Datagram Protocol) provides a connectionless datagram service that prioritizes time over reliability.

That's why net.internetSocket chooses the socket type syscall.SOCK_STREAM that provides sequenced, reliable, two-way, connection-byte streams as Linux programmer's manual describes.

func (sl *sysListener) listenTCP(ctx context.Context, laddr *TCPAddr) (*TCPListener, error) {
    fd, err := internetSocket(ctx, sl.network, laddr, nil, syscall.SOCK_STREAM, 0, "listen", sl.ListenConfig.Control)

net.socket invokes net.netFD.listenStream when syscall.SOCK_STREAM is passed.

switch sotype {
case syscall.SOCK_STREAM, syscall.SOCK_SEQPACKET:
    if err := fd.listenStream(laddr, listenerBacklog(), ctrlFn);

Finally, we reached the last place.

func (fd *netFD) listenStream(laddr sockaddr, backlog int, ctrlFn func(string, string, syscall.RawConn) error) error {

    // (omit)

    if err = syscall.Bind(fd.pfd.Sysfd, lsa); err != nil {
        return os.NewSyscallError("bind", err)
    }
    if err = listenFunc(fd.pfd.Sysfd, backlog); err != nil {
        return os.NewSyscallError("listen", err)
    }
    if err = fd.init(); err != nil {
        return err
    }
    lsa, _ = syscall.Getsockname(fd.pfd.Sysfd)
    fd.setAddr(fd.addrFunc()(lsa), nil)
    return nil
}

net.netFD.listenStream executes three system calls: bind, listen, and getsockname.

bind assigns the socket address specified by given address to the socket referred to by the file descriptor.

if err = syscall.Bind(fd.pfd.Sysfd, lsa); err != nil {

listen marks the socket referred to by the file descriptor as a passive socket, that is, as a socket that will be used to accept incoming requests using accept

if err = listenFunc(fd.pfd.Sysfd, backlog); err != nil {

getsockname returns the current address to which the socket is bound. You can know the current address even when the port is randomly chosen.

lsa, _ = syscall.Getsockname(fd.pfd.Sysfd)

Let's go back to the first question: How is the port automatically selected? The answer is the specification of bind.

Ports that can be used by the TCP protocol are 16 bits in the protocol header so they can hold values between 0 - 65535.

The bind uses a zero in the port number to mean "pick random one" and bind its address.

For example, the specification of the bind function in Linux is described in ip(7) — Linux manual page.

An ephemeral port is allocated to a socket in the following circumstances:

the port number in a socket address is specified as 0 when calling bind(2);

Ephemeral port is a port of a transport layer protocol that is used for only a short period of time for the duration of a communication session. It is also called dynamic ports, because it is used on a per request basis. The IANA assigns the the range 49152-65535 for the selection of ephemeral ports as RFC 6056 describes.

The port range varies depending on the server system, but the same is true not only for Linux, but also other systems. For example, in Windows, bind function (winsock.h) assigns a unique port to the application from the dynamic client port range 49152 and 65535 (when on Windows Vista later).

Accept a connection

So far, you have understood the back side of starting to listen for TCP connections. Now, let's go back to the first sample code again.

func main() {
    addr := "localhost:8888"
    l, err := net.Listen("tcp", addr)

    // (omit)

    for {
        // Listen for an incoming connection
        conn, err := l.Accept()
        if err != nil {
            panic(err)
        }
        // Handle connections in a new goroutine
        // (omit)
    }
}

The next part is the mechanism to accept a connection, net.TCPListener.Accept.

net.TCPListener.Accept waits for the next call and returns a generic net.Conn. It implements the Accept method in the net.Listener interface that represents a generic network listener for stream-oriented protocols.

type Listener interface {
    Accept() (Conn, error)
    Close() error
    Addr() Addr
}

It is implemented by various types: FileListener, UnixListener and TCPListener.

In understanding this implementation, I have again prepared a map.

net.TCPListener.Accept invokes net.TCPListener.accept and then it invokes net.netFD.accept.

func (fd *netFD) accept() (netfd *netFD, err error) {
    d, rsa, errcall, err := fd.pfd.Accept()
    if err != nil {
        if errcall != "" {
            err = wrapSyscallError(errcall, err)
        }
        return nil, err
    }

    if netfd, err = newFD(d, fd.family, fd.sotype, fd.net); err != nil {
        poll.CloseFunc(d)
        return nil, err
    }
    if err = netfd.init(); err != nil {
        netfd.Close()
        return nil, err
    }
    lsa, _ := syscall.Getsockname(netfd.pfd.Sysfd)
    netfd.setAddr(netfd.addrFunc()(lsa), netfd.addrFunc()(rsa))
    return netfd, nil
}

Here, it executes two system calls: accept and getsockname.

The accept system call is used with connection-based socket types (SOCK_STREAM, SOCK_SEQPACKET). As you see, SOCK_STREAM is chosen when it's a TCP listener. It extracts the first connection request on the queue of pending connections for listening socket, creates a new connected socket, and returns a new file descriptor referring to that socket.

The getsockname is called to know a new connected socket and file descriptor.

Read and write a connection

net.TCPListener.Accept returns net.Conn that is a generic stream-oriented network connection.

type Conn interface {
    Read(b []byte) (n int, err error)
    Write(b []byte) (n int, err error)
    Close() error
    LocalAddr() Addr
    RemoteAddr() Addr
    SetDeadline(t time.Time) error
    SetReadDeadline(t time.Time) error
    SetWriteDeadline(t time.Time) error
}

By using the net.Conn, you can read and write a connection like this:

func main() {
    addr := "localhost:8888"
    l, err := net.Listen("tcp", addr)

    // (omit)

    for {
        // Listen for an incoming connection
        conn, err := l.Accept()
        if err != nil {
            panic(err)
        }
        // Handle connections in a new goroutine
        go func(conn net.Conn) {
            buf := make([]byte, 1024)
            len, err := conn.Read(buf)
            if err != nil {
                fmt.Printf("Error reading: %#v\n", err)
                return
            }
            fmt.Printf("Message received: %s\n", string(buf[:len]))

            conn.Write([]byte("Message received.\n"))
            conn.Close()
        }(conn)
    }
}

Conclusion

List the key takeaways again.

With the net package, you can create a TCP server with such simple and little code (The Go playground).
If the host in the address parameter is empty or a literal unspecified IP address, listen on all available unicast and anycast IP addresses of the local system.
There are two kinds of DNS resolver: Go's built-in DNS resolver and uses Cgo DNS resolver. By default, Go's built-in DNS resolver is used.
If the port in the address parameter is empty or "0", a port number is automatically chosen by the system call bind.
Go accepts an incoming connection by using the system call accept.

You should now have a more in-depth understanding of the fundamentals of implementing a TCP server.

Reverse HTTP proxy over WebSocket in Go (Part 5)

Kazuki Higashiguchi — Thu, 30 Dec 2021 04:51:58 +0000

Series introduction

In part 4 I talked about how to relay TCP connection from "App" to the peer of WebSocket, especially implementation to take a WebSocket connection available from pools for relaying received requests.

Reverse HTTP proxy over WebSocket in Go (Part 4)

Kazuki Higashiguchi ・ Dec 28 '21

#go #webdev #websocket

In this post, I will be starting to talk about how to send the request to the peer through the WebSocket connection.

Start a WebSocket server (Part 1)
Establish a WebSocket connection (Part 2)
Relay TCP connection from "App" to the peer of WebSocket (Part 3 | Part 4 | Part 5)
Relay TCP connection in WebSocket data to "internal API"
Keep a established connection

Reverse HTTP proxy over WebSocket

A reverse HTTP proxy over WebSocket is a type of proxies, which retrieves resources on behalf on a client from servers and uses the WebSocket protocol as a "tunnel" to pass TCP communication from server to client.

I'll introduce Go sample project forked from root-gg/wsp (I forked it because maintenance has stopped and the Go language and libraries version needed to be updated).

hgsgtk / wsp

HTTP tunnel over Websocket

Send the request to the peer through the WebSocket connection

Let's take a look at the continuation of the previous HTTP handler code, which waits the request from "app" to the endpoint /requests/.

func (s *Server) request(w http.ResponseWriter, r *http.Request) {
    // (omit): [1]: Receive requests to be proxied

    // [2]: Take a WebSocket connection available from pools for relaying received requests.
    request := NewConnectionRequest(s.Config.GetTimeout())
    // "Dispatcher" is running in a separate thread from the server by `go s.dispatchConnections()`.
    // It waits to receive requests to dispatch connection from available pools to clients requests.
    // https://github.com/hgsgtk/wsp/blob/ea4902a8e11f820268e52a6245092728efeffd7f/server/server.go#L93
    //
    // Notify request from handler to dispatcher through Server.dispatcher channel.
    s.dispatcher <- request
    // Dispatcher tries to find an available connection pool,
    // and it returns the connection through Server.connection channel.
    // https://github.com/hgsgtk/wsp/blob/ea4902a8e11f820268e52a6245092728efeffd7f/server/server.go#L189
    //
    // Here waiting for a result from dispatcher.
    connection := <-request.connection
    if connection == nil {
        // It means that dispatcher has set `nil` which is a system error case that is
        // not expected in the normal flow.
        wsp.ProxyErrorf(w, "Unable to get a proxy connection")
        return
    }

    // [3]: Send the request to the peer through the WebSocket connection.
    if err := connection.proxyRequest(w, r); err != nil {
        // An error occurred throw the connection away
        log.Println(err)
        connection.Close()

        // Try to return an error to the client
        // This might fail if response headers have already been sent
        wsp.ProxyError(w, err)
    }
}

In the implementation up to part 4, we were able to identify the WebSocket connection to be relayed. Next, send an HTTP request over the connection.

connection.proxyRequest does it. Let's look at the implementation. There are six steps.

Serialize HTTP request
Send the HTTP request to the peer
Wait the HTTP response is ready
Read the HTTP response from the peer
Wait the HTTP response body is ready
Read the HTTP response body from the peer

Here is a diagram describing the steps of communication over the WebSocket connection.

1. Serialize HTTP request / 2. send an HTTP request on the WebSocket connection

At first, it serializes an HTTP request and send it to the WebSocket connection peer.

func (connection *Connection) proxyRequest(w http.ResponseWriter, r *http.Request) (err error) {
    log.Printf("proxy request to %s", connection.pool.id)

    // [1]: Serialize HTTP request
    jsonReq, err := json.Marshal(wsp.SerializeHTTPRequest(r))
    if err != nil {
        return fmt.Errorf("unable to serialize request : %w", err)
    }

    // [2]: Send the HTTP request to the peer
    // Send the serialized HTTP request to the the peer
    if err := connection.ws.WriteMessage(websocket.TextMessage, jsonReq); err != nil {
        return fmt.Errorf("unable to write request : %w", err)
    }

    // Pipe the HTTP request body to the the peer
    bodyWriter, err := connection.ws.NextWriter(websocket.BinaryMessage)
    if err != nil {
        return fmt.Errorf("unable to get request body writer : %w", err)
    }
    if _, err := io.Copy(bodyWriter, r.Body); err != nil {
        return fmt.Errorf("unable to pipe request body : %w", err)
    }
    if err := bodyWriter.Close(); err != nil {
        return fmt.Errorf("unable to pipe request body (close) : %w", err)
    }

In WebSocket connection, the kind of message is defined in the RFC 6455 called Opcode, and Text Frame and Binary Frame can be used for data exchange.

Opcode	Meaning
0	Continuation Frame
1	Text Frame
2	Binary Frame
8	Connection Close Frame
9	Ping Frame
10	Pong Frame

In this case, it sends a JSON serialized HTTP meta data (URL, Method, Header, and Content-Length) in Text Frame, and sends a request body in Binary Frame.

type HTTPRequest struct {
    Method        string
    URL           string
    Header        map[string][]string
    ContentLength int64
}

func SerializeHTTPRequest(req *http.Request) (r *HTTPRequest) {
    r = new(HTTPRequest)
    r.URL = req.URL.String()
    r.Method = req.Method
    r.Header = req.Header
    r.ContentLength = req.ContentLength
    return
}

SerializeHTTPRequest serializes HTTP request.

req.Header is a value who type is http.Header. A Header represents the key-value pairs in an HTTP header.

type Header map[string][]string

A Header represents the key-value pairs in an HTTP header.

{
    "Accept":["*/*"],
    "User-Agent":["curl/7.77.0"],
    "X-Proxy-Destination":["http://localhost:8081/hello"]
}

json.Marshal returns the JSON encoding of the value. Struct values encode as JSON objects.

jsonReq, err := json.Marshal(wsp.SerializeHTTPRequest(r))

Each exported struct field becomes a member of the object. If you want to customize the encoding struct field, you can use the struct field's tag json.

type HTTPRequest struct {
    Method string `json:"customized"` // {"customized":"GET"}
}

Send JSON data when serialize is complete.

connection.ws.WriteMessage(websocket.TextMessage, jsonReq)

websocket.WriteMessage is the function to write any message in the connection, and websocket.TextMessage is a constant representing a text data message.

Then, pipe the HTTP request body to the WebSocket connection peer.

// Pipe the HTTP request body to the the peer
bodyWriter, err := connection.ws.NextWriter(websocket.BinaryMessage)
if err != nil {
    return fmt.Errorf("unable to get request body writer : %w", err)
}
if _, err := io.Copy(bodyWriter, r.Body); err != nil {
    return fmt.Errorf("unable to pipe request body : %w", err)
}
if err := bodyWriter.Close(); err != nil {
    return fmt.Errorf("unable to pipe request body (close) : %w", err)
}

websocket.NextWriter returns a writer for the next message to send. An application can send and receive messages using the io.WriteCloser and io.Reader interfaces. In this case, The content of r.Body is written to writer.

3. Wait the HTTP response is ready to read

Let's move on the rest code, to receive an HTTP response and parse it. Here is one of difficult points to proxy an HTTP request over an WebSocket connection, because the wsp_server does not know when the wsp_client writes the HTTP response over the WebSocket connection. We need to implement waiting for response to come back.

I'll show you the whole picture first to address the issue.

In advance, the read function waits to receive the HTTP response as an another thread (goroutine) "reader”.

type Connection struct {
    // (omit)
    nextResponse chan chan io.Reader
}

// NewConnection returns a new Connection.
func NewConnection(pool *Pool, ws *websocket.Conn) *Connection {
    // (omit: Initialize a new Connection)

    // Start to listen to incoming messages over the WebSocket connection
    go c.read()

    return c
}

func (connection *Connection) read() {
    defer func() {
        if r := recover(); r != nil {
            log.Printf("WebSocket crash recovered : %s", r)
        }
        connection.Close()
    }()

    for {
        if connection.status == Closed {
            break
        }

        // https://godoc.org/github.com/gorilla/websocket#hdr-Control_Messages
        _, reader, err := connection.ws.NextReader()
        if err != nil {
            break
        }

        if connection.status != Busy {
            // We received a wild unexpected message
            break
        }

        // When it gets here, it is expected to be either a HttpResponse or a HttpResponseBody has been returned.
        //
        // Next, it waits to receive the value from the Connection.proxyRequest function that is invoked in the "server" thread.
        c := <-connection.nextResponse
        if c == nil {
            // We have been unlocked by Close()
            break
        }

        // Send the reader back to Connection.proxyRequest
        c <- reader

        // Wait for proxyRequest to close the channel
        // this notify that it is done with the reader
        <-c
    }
}

When the new WebSocket connection is connected, it starts a goroutine which listens to incoming messages over the WebSocket connection. It continues running until the connection itself is closed or any unexpected error occurs. In this loop, use websocket.NewReader to read incoming messages.

// read loop
for {
    _, reader, err := connection.ws.NextReader()
    if err != nil {
        break
    }
}

NextReader returns the next data message received from the peer. In this case, it means the message from a wsp_client.

func (c *Conn) NextReader() (messageType int, r io.Reader, err error)

The returned messageType, which means opcode in RFC 6455, is either TextMessage or BinaryMessage. In addition to these opcodes, the WebSocket protocol defines three control messages: close (8: Connection Close Frame), ping (9: Ping Frame), and pong (10: Pong Frame). If you implement WebSocket messaging from scratch, you need to process these control messages, but gorilla/websocket is designed so that you don't have to be concern about it.

If the application is not otherwise interested in messages from the peer, then the application should start a goroutine to read and discard messages from the peer.

https://pkg.go.dev/github.com/gorilla/websocket@v1.4.2#hdr-Control_Messages

And, the a must break out of the read loop when websocket.NextReader returns a non-nil error value. gorilla/websocket implements that if you do not break out the read loop and repeat many times, you will get a panic like this:

// gorilla/websocket conn.go
func (c *Conn) NextReader() (messageType int, r io.Reader, err error) {
    // (omit)

    // Applications that do handle the error returned from this method spin in
    // tight loop on connection failure. To help application developers detect
    // this error, panic on repeated reads to the failed connection.
    c.readErrCount++
    if c.readErrCount >= 1000 {
        panic("repeated read on failed websocket connection")
    }

    return noFrame, nil, c.readErr

NextReader will block here until a message is received or the WebSocket connection is closed. The following code in gorolla/websocket keeps reading the next data frame until a TextMessage or BinaryMessage is received. Note that data frame is a message format in the WebSocket protocol defined in RFC 6455.

// gorilla/websocket conn.go
func (c *Conn) NextReader() (messageType int, r io.Reader, err error) {
    // (omit)

    for c.readErr == nil {
        frameType, err := c.advanceFrame()
        if err != nil {
            c.readErr = hideTempErr(err)
            break
        }

        if frameType == TextMessage || frameType == BinaryMessage {
            c.messageReader = &messageReader{c}
            c.reader = c.messageReader
            if c.readDecompress {
                c.reader = c.newDecompressionReader(c.reader)
            }
            return frameType, c.reader, nil
        }
    }

    // (omit)
}

See the following link more detail.

How WebSocket protocol designs bidirectional messaging and implements in Go

Kazuki Higashiguchi ・ Dec 13 '21

#webdev #websocket #go

After a message is received from the peer, it waits to receive the value from the Connection.proxyRequest function that is invoked in the "server" thread.

func (connection *Connection) read() {
    // (omit)
    for {
        // (omit)

        c := <-connection.nextResponse
        if c == nil {
            // We have been unlocked by Close()
            break
        }

        // Send the reader back to Connection.proxyRequest
        c <- reader

        // (omit)
    }
}

When a "server" thread proxies, it sends the HTTP request over the WebSocket, and sends the channel of the io.Reader interface (chan io.Reader) that can read the HTTP response to the field nextResponse, then waits until the value is written in the channel (chan io.Reader) by another thread "reader” (code).

After the thread "reader" detects that the HTTP response from the peer of the WebSocket connection has been written, it sends the value to the channel (chan io.Reader), and the "server" thread can proceed to process the rest procedures.

type Connection struct {
    // (omit)
    nextResponse chan chan io.Reader
}

func (connection *Connection) proxyRequest(w http.ResponseWriter, r *http.Request) (err error) {
    // (omit) [1]: Serialize HTTP request
    // (omit) [2]: Send the HTTP request to the peer

    // [3]: Wait the HTTP response is ready to read
    responseChannel := make(chan (io.Reader))
    connection.nextResponse <- responseChannel
    responseReader, ok := <-responseChannel
    if responseReader == nil {
        if ok {
            // The value of ok is false, the channel is closed and empty.
            // See the Receiver operator in https://go.dev/ref/spec for more information.
            close(responseChannel)
        }
        return fmt.Errorf("unable to get http response reader : %w", err)
    }
}

When we get to this point, we are ready to read the response written over the WebSocket connection.

4. Read the HTTP response from the peer

Let's move on the next step: read the HTTP response from the peer, where gets the serialized HTTP response.

type HTTPResponse struct {
    StatusCode    int
    Header        http.Header
    ContentLength int64
}

func (connection *Connection) proxyRequest(w http.ResponseWriter, r *http.Request) (err error) {
    // (omit)

    // [4]: Read the HTTP response from the peer
    // Get the serialized HTTP Response from the peer
    jsonResponse, err := io.ReadAll(responseReader)
    if err != nil {
        close(responseChannel)
        return fmt.Errorf("unable to read http response : %w", err)
    }

    // Notify the read() goroutine that we are done reading the response
    close(responseChannel)

    // Deserialize the HTTP Response
    httpResponse := new(wsp.HTTPResponse)
    if err := json.Unmarshal(jsonResponse, httpResponse); err != nil {
        return fmt.Errorf("unable to unserialize http response : %w", err)
    }

    // Write response headers back to the client
    for header, values := range httpResponse.Header {
        for _, value := range values {
            w.Header().Add(header, value)
        }
    }
    w.WriteHeader(httpResponse.StatusCode)

    // (omit)
}

The value responseReader implements io.Reader interface, that's why we can read the data in responseReader by io.ReadAll, which is added in Go 1.16.

After that, it closes the channel responseChannel so that the "reader" thread will be notified that the response is finished to read.

func (connection *Connection) read() {
    // (omit)
    for {
        // (omit)

        // Send the reader back to Connection.proxyRequest
        c <- reader

        // Wait for proxyRequest to close the channel
        // this notify that it is done with the reader
        <-c
    }
}

The "reader" thread waits until the "server" thread closes the responseChannel channel because of the requirements of websocket.NextReader. As explained in godoc, we need to consider several specifications:

No more than one goroutine calls the read methods (NextReader, SetReadDeadline, ReadMessage, ReadJSON, SetPongHandler, SetPingHandler) concurrently
Connections support one concurrent reader and one concurrent writer

That's why we need to wait for websocket.NextReader to be consumed before requesting the next one.

Then next, unmarshal JSON into the struct HTTPResponse by json.Unmarshal. json.Unmarshal matches incoming JSON object keys to the keys used by Marshal, either the struct field name or its tag (json:"xxx").

Now we can get the HTTP response headers and status code.

5. Wait the HTTP response body is ready / 6. Read the HTTP response body from the peer

Finally, follow the same steps again to read the HTTP response body again.

func (connection *Connection) proxyRequest(w http.ResponseWriter, r *http.Request) (err error) {
    // (omit)

    // [5]: Wait the HTTP response body is ready
    // Get the HTTP Response body from the the peer
    // To do so send a new channel to the read() goroutine
    // to get the next message reader
    responseBodyChannel := make(chan (io.Reader))
    connection.nextResponse <- responseBodyChannel
    responseBodyReader, ok := <-responseBodyChannel
    if responseBodyReader == nil {
        if ok {
            // If more is false the channel is already closed
            close(responseChannel)
        }
        return fmt.Errorf("unable to get http response body reader : %w", err)
    }

    // [6]: Read the HTTP response body from the peer
    // Pipe the HTTP response body right from the remote Proxy to the client
    if _, err := io.Copy(w, responseBodyReader); err != nil {
        close(responseBodyChannel)
        return fmt.Errorf("unable to pipe response body : %w", err)
    }

    // Notify read() that we are done reading the response body
    close(responseBodyChannel)

    connection.Release()

    return
}

Conclusion

Following part 3 and part 4, I explained how to relay TCP connection from "App" to the peer of WebSocket.

In part 6, I'll explain how to relay TCP connection in WebSocket data to "internal API".

Reverse HTTP proxy over WebSocket in Go (Part 4)

Kazuki Higashiguchi — Tue, 28 Dec 2021 06:01:33 +0000

Series introduction

In part 3 I talked about how to relay TCP connection from "App" to the peer of WebSocket, especially implementation to receive requests to be proxied and to pool the WebSocket connection on the server for relaying in Go.

Reverse HTTP proxy over WebSocket in Go (Part 3)

Kazuki Higashiguchi ・ Dec 16 '21

#go #webdev #websocket

In this post, I will be starting to talk about how to relay TCP connection from "App" to the peer of WebSocket in Go.

Start a WebSocket server (Part 1)
Establish a WebSocket connection (Part 2)
Relay TCP connection from "App" to the peer of WebSocket (Part 3 | Part 4 | Part 5)
Relay TCP connection in WebSocket data to "internal API"
Keep a established connection

Reverse HTTP proxy over WebSocket

I'll introduce Go sample project forked from root-gg/wsp (I forked it because maintenance has stopped and the Go language and libraries version needed to be updated).

hgsgtk / wsp

HTTP tunnel over Websocket

Relay TCP connection to the peer WebSocket

A reverse HTTP proxy over WebSocket relay TCP connection to the peer WebSocket over the WebSocket connection.

HTTP communication is relayed by the following route.

app -[1]-> wsp server -[2](WebSocket)-> wsp client -> internal API

And, these flow are divided into three parts to explain it.

Receive requests to be proxied ([1] in the relay flow)
Pool the WebSocket connection on the server for relaying
Relay TCP connection to the peer WebSocket ([2] in the relay flow)

I explained the 1st and 2nd flow in part 3, so let's move on the 3rd flow.

Let's take a look at the continuation of the previous HTTP handler code, which waits the request from "app" to the endpoint /requests/.

func (s *Server) request(w http.ResponseWriter, r *http.Request) {
    // (omit): [1]: Receive requests to be proxied

    // [2]: Take a WebSocket connection available from pools for relaying received requests.
    request := NewConnectionRequest(s.Config.GetTimeout())
    // "Dispatcher" is running in a separate thread from the server by `go s.dispatchConnections()`.
    // It waits to receive requests to dispatch connection from available pools to clients requests.
    // https://github.com/hgsgtk/wsp/blob/ea4902a8e11f820268e52a6245092728efeffd7f/server/server.go#L93
    //
    // Notify request from handler to dispatcher through Server.dispatcher channel.
    s.dispatcher <- request
    // Dispatcher tries to find an available connection pool,
    // and it returns the connection through Server.connection channel.
    // https://github.com/hgsgtk/wsp/blob/ea4902a8e11f820268e52a6245092728efeffd7f/server/server.go#L189
    //
    // Here waiting for a result from dispatcher.
    connection := <-request.connection
    if connection == nil {
        // It means that dispatcher has set `nil` which is a system error case that is
        // not expected in the normal flow.
        wsp.ProxyErrorf(w, "Unable to get a proxy connection")
        return
    }

    // [3]: Send the request to the peer through the WebSocket connection.
    if err := connection.proxyRequest(w, r); err != nil {
        // An error occurred throw the connection away
        log.Println(err)
        connection.Close()

        // Try to return an error to the client
        // This might fail if response headers have already been sent
        wsp.ProxyError(w, err)
    }
}

In brief, the following process is performed.

Take a WebSocket connection available from pools for relaying received requests
Send the request to the peer through the WebSocket connection

I'll explain the one by one.

Take a WebSocket connection available from pools for relaying received requests

As a pre-requisite, we will start with WebSocket connection already established with the WebSocket client (wsp_client) and held by the server as a pool (Chapter 2).

func (s *Server) Register(w http.ResponseWriter, r *http.Request) {
    // - 1. Upgrade a received HTTP request to a WebSocket connection
    // (omit)
    // - 2. Wait a greeting message from the peer and parse it
    // (omit)

    // 3. Register the connection into server pools.
    // (omit)
    // Add the WebSocket connection to the pool
    pool.Register(ws)
}

There are several possible designs for retrieving the connection from the pools, and I'll explain the pattern of using multiple threads. Specifically, there are two threads running: "Server", which accepts http requests, and "Dispatcher", which dispatches connections from the pools to be used to relay. Here is the Go code to start "Server" and "Dispatcher" that will be called from the main function.

func (s *Server) Start() {
    // (omit)...

    // start the "Dispatcher"
    go s.dispatchConnections()

    // start the "Server"
    s.server = &http.Server{
        Addr:    s.Config.GetAddr(),
        Handler: r,
    }
    go func() { log.Fatal(s.server.ListenAndServe()) }()
}

The go statement starts the execution of a function call as an independent concurrent thread of control, or goroutine, within the same the address space.

Let's move on the implementation of dispatchConnections function that starts a "dispatcher" thread.

type Server struct {
    // (omit)

    // Through dispatcher channel it communicates between "server" thread and "dispatcher" thread.
    // "server" thread sends the value to this channel when accepting requests in the endpoint /requests, 
    // and "dispatcher" thread reads this channel.
    dispatcher chan *ConnectionRequest
}

func (s *Server) dispatchConnections() {
    for {
        //  The operator <- is "receive operator", which expression blocks until a value is available.
        request, ok := <-s.dispatcher
        if !ok {
            // The value of `ok` is false if it is a zero value generated because the channel is closed an empty.
            // In this case, that means server shutdowns.           break
        }

        for {
            // (omit)...

            // Verify that we can use this connection
            if connection.Take() {
                request.connection <- connection
                break
            }
        }

        // (omit)...
    }
}

The type of the field dispatcher in the Server structure is channel. Channel types provide a mechanism for concurrently executing functions to communicate by sending and receiving values of a specified element type.

for {
    request, ok := <-s.dispatcher
    if !ok {
        break
    }
}

The operator x, ok := <-ch is receiver operator. The value of the receive operation <-s.dispatcher is the value received from the channel s.dispatcher. Also, the expression <- blocks until a value is available, so waits until a connection request is sent by "Server" thread.

The value of ok is false if it is a zero value generated because the channel is closed or empty. In this case, that means server shutdowns.

On the other hand, the /request/ handler on the "Server" tread sends a value to this channel.

func (s *Server) request(w http.ResponseWriter, r *http.Request) {
    // (omit): [1]: Receive requests to be proxied

    // Here! Sends a value to dispatcher channel
    request := NewConnectionRequest(s.Config.GetTimeout())
    s.dispatcher <- request

    connection := <-request.connection
    if connection == nil {
        wsp.ProxyErrorf(w, "Unable to get a proxy connection")
        return
    }
}

The operator ch <- x is send statements, which sends a value on a channel. Here it sends the request to the dispatcher channel.

By the way, the type of variable request is ConnectionRequest.

type ConnectionRequest struct {
    connection chan *Connection
}

After sending the value to the s.dispatcher channel, it waits to be available to get the value in the request.connection channel.

func (s *Server) request(w http.ResponseWriter, r *http.Request) {
    // (omit)
    s.dispatcher <- request

    // Here!
    connection := <-request.connection
    if connection == nil {
        wsp.ProxyErrorf(w, "Unable to get a proxy connection")
        return
    }
}

Next, let's look at the "Dispatcher" thread.

To summarize, "Server" thread sends the value to this channel when accepting requests in the endpoint /requests, and "dispatcher" thread reads this channel.

Let's move on the implementation of dispatchConnections function that starts a "dispatcher" thread.

type Server struct {
    // (omit)

    // Through dispatcher channel it communicates between "server" thread and "dispatcher" thread.
    // "server" thread sends the value to this channel when accepting requests in the endpoint /requests, 
    // and "dispatcher" thread reads this channel.
    dispatcher chan *ConnectionRequest
}

func (s *Server) dispatchConnections() {
    for {
        // Runs in an infinite loop and keeps receiving the value from the `server.dispatcher` channel
        // The operator <- is "receive operator", which expression blocks until a value is available.
        request, ok := <-s.dispatcher
        if !ok {
            // The value of `ok` is false if it is a zero value generated because the channel is closed an empty.
            // In this case, that means server shutdowns.
            break
        }

        // A timeout is set for each dispatch request.
        ctx := context.Background()
        ctx, cancel := context.WithTimeout(ctx, s.Config.GetTimeout())
        defer cancel()

    L:
        for {
            select {
            case <-ctx.Done(): // The timeout elapses
                break L
            default: // Go through
            }

            s.lock.RLock()
            if len(s.pools) == 0 {
                // No connection pool available
                s.lock.RUnlock()
                break
            }

            // [1]: Select a pool which has an idle connection
            // Build a select statement dynamically to handle an arbitrary number of pools.
            cases := make([]reflect.SelectCase, len(s.pools)+1)
            for i, ch := range s.pools {
                cases[i] = reflect.SelectCase{
                    Dir:  reflect.SelectRecv,
                    Chan: reflect.ValueOf(ch.idle)}
            }
            cases[len(cases)-1] = reflect.SelectCase{
                Dir: reflect.SelectDefault}
            s.lock.RUnlock()

            _, value, ok := reflect.Select(cases)
            if !ok {
                continue // a pool has been removed, try again
            }
            connection, _ := value.Interface().(*Connection)

            // [2]: Verify that we can use this connection and take it.
            if connection.Take() {
                request.connection <- connection
                break
            }
        }

        close(request.connection)
    }
}

First, dispatchConnections runs in an infinite loop and keeps receiving the value from the server.dispatcher channel.

for {
    request, ok := <-s.dispatcher
    // ...
}

The next step is to set the timeout, if no idle connection is obtained after a predetermined time, the channel will be closed.

// A timeout is set for each dispatch request.
ctx := context.Background()
ctx, cancel := context.WithTimeout(ctx, s.Config.GetTimeout())
defer cancel()

L:
    for {
        select {
        case <-ctx.Done(): // The timeout elapses
            break L
        default: // Go through
        }

        // ...
    }

    close(request.connection)

When the channel is closed, a receive operation on a closed channel can always proceed immediately, yielding the element type's zero value after any previously sent values have been received (See The Go Programming Language Specification#Receive operator more detail). In this case, the "Server" thread waits to receive the value, and will get nil from request.connection channel, so the "Server" will return the error response.

func (s *Server) request(w http.ResponseWriter, r *http.Request) {
    // (omit)
    s.dispatcher <- request

    // Here!
    connection := <-request.connection
    if connection == nil {
        wsp.ProxyErrorf(w, "Unable to get a proxy connection")
        return
    }
}

Also, if you want to know more about timeout using the context package, please refer to the following post.

Timeout using context package in Go

Kazuki Higashiguchi ・ Dec 28 '21

#go

Then, select a pool which has an idle connection by building a select statement dynamically to handle an arbitrary number of pools.

cases := make([]reflect.SelectCase, len(s.pools)+1)
for i, ch := range s.pools {
    cases[i] = reflect.SelectCase{
        Dir:  reflect.SelectRecv,
        Chan: reflect.ValueOf(ch.idle)}
}
cases[len(cases)-1] = reflect.SelectCase{
    Dir: reflect.SelectDefault}
s.lock.RUnlock()

_, value, ok := reflect.Select(cases)
if !ok {
    continue // a pool has been removed, try again
}
connection, _ := value.Interface().(*Connection)

reflect.Select allows us to receive a variable number of channels. See the following post for more information.

Handling with arbitrary channels by reflect.Select

Kazuki Higashiguchi ・ Dec 28 '21

#go

Lastly, Verify that we can use this connection and take it.

if connection.Take() {
    request.connection <- connection
    break
}

connection.Take verify the status of connection whether it's available one or not, then if it's available one, mark it busy.

That's it to take a WebSocket connection available from pools for relaying received requests.

Conclusion

Following part 3, I explained how to relay TCP connection from "App" to the peer of WebSocket. Especially, I focused on the way to take a WebSocket connection available from pools for relaying received requests.

In part 5, I'll explain how to send the request to the peer through the WebSocket connection.

Timeout using context package in Go

Kazuki Higashiguchi — Tue, 28 Dec 2021 04:27:05 +0000

Key takeaways

context.WithTimeout can be used in a timeout implementation.
WithDeadline returns CancelFunc that tells an operation to abandon its work.
timerCtx implements cancel() by stopping its timer then delegating to cancelCtx.cancel, and cancelCtx closes the context.
ctx.Done returns a channel that's closed when work done on behalf of this context should be canceled.

context.WithTimeout

The context package as the standard library was moved from the golang.org/x/net/context package in Go 1.7. This allows the use of contexts for cancellation, timeouts, and passing request-scoped data in other library packages.

context.WithTimeout can be used in a timeout implementation.

func WithTimeout(parent Context, timeout time.Duration) (Context, CancelFunc)

For example, you could implement it as follow (Go playground):

package main

import (
    "context"
    "fmt"
    "log"
    "time"
)

func execute(ctx context.Context) error {
    proc1 := make(chan struct{}, 1)
    proc2 := make(chan struct{}, 1)

    go func() {
        // Would be done before timeout
        time.Sleep(1 * time.Second)
        proc1 <- struct{}{}
    }()

    go func() {
        // Would not be executed because timeout comes first
        time.Sleep(3 * time.Second)
        proc2 <- struct{}{}
    }()

    for i := 0; i < 3; i++ {
        select {
        case <-ctx.Done():
            return ctx.Err()
        case <-proc1:
            fmt.Println("process 1 done")
        case <-proc2:
            fmt.Println("process 2 done")

        }
    }

    return nil
}

func main() {
    ctx := context.Background()
    ctx, cancel := context.WithTimeout(ctx, 2*time.Second)
    defer cancel()

    if err := execute(ctx); err != nil {
        log.Fatalf("error: %#v\n", err)
    }
    log.Println("Success to process in time")
}

Canceling this context releases resources associated with it, so you should call cancel as soon as the operations running in this Context complete.

ctx := context.Background()
ctx, cancel := context.WithTimeout(ctx, 2*time.Second)
defer cancel()

Cancel notification after timeout is received from ctx.Done(). Done returns a channel that's closed when work done on behalf of this context should be canceled. WithTimeout arranges for Done to be closed when the timeout elapses.

select {
case <-ctx.Done():
    return ctx.Err()
}

When you execute this code, you will get the following result. A function call that can be completed in 1s will be finished, but a function call that can be done after 3s will not be executed because a timeout occurs in 2s.

$ go run main.go
process 1 done
2021/12/28 12:32:59 error: context.deadlineExceededError{}
exit status 1

In this way, you can implement timeout easily.

Deep dive into context.WithTimeout

Here's a quick overview.

WithTimeout is a wrapper function for WithDeadline.

func WithTimeout(parent Context, timeout time.Duration) (Context, CancelFunc) {
    return WithDeadline(parent, time.Now().Add(timeout))
}

WithDeadline returns CancelFunc that tells an operation to abandon its work. Internally, a function that calls timerCtx.cancel(), a function that's not exported, will be returned.

func WithDeadline(parent Context, d time.Time) (Context, CancelFunc) {
    // (omit)
    c := &timerCtx{
        cancelCtx: newCancelCtx(parent),
        deadline:  d,
    }
    // (omit)
    return c, func() { c.cancel(true, Canceled) }
}

// (omit)

type timerCtx struct {
    cancelCtx
    timer *time.Timer // Under cancelCtx.mu.

    deadline time.Time
}

A timerCtx carries a timer and a deadline, and embeds a cancelCtx.

type cancelCtx struct {
    Context

    mu       sync.Mutex            // protects following fields
    done     atomic.Value          // of chan struct{}, created lazily, closed by first cancel call
    children map[canceler]struct{} // set to nil by the first cancel call
    err      error                 // set to non-nil by the first cancel call
}

timerCtx implements cancel() by stopping its timer then delegating to cancelCtx.cancel.

func (c *cancelCtx) cancel(removeFromParent bool, err error) {
    if err == nil {
        panic("context: internal error: missing cancel error")
    }
    c.mu.Lock()
    if c.err != nil {
        c.mu.Unlock()
        return // already canceled
    }
    c.err = err
    d, _ := c.done.Load().(chan struct{})
    if d == nil {
        c.done.Store(closedchan)
    } else {
        close(d)
    }
    for child := range c.children {
        // NOTE: acquiring the child's lock while holding parent's lock.
        child.cancel(false, err)
    }
    c.children = nil
    c.mu.Unlock()

    if removeFromParent {
        removeChild(c.Context, c)
    }
}

In the function the context is closed.

Conclusion

I explained how to implement timeout with context package, and dived into internal implementation in it. I hope this helps you understand the Go implementation.

Handling with arbitrary channels by reflect.Select

Kazuki Higashiguchi — Tue, 28 Dec 2021 00:16:14 +0000

Key takeaways

A select statement chooses which of a set of operations will proceed, which lets you wait on multiple channel operations.
reflect.Select allows us to handle an arbitrary number of channels

Select statements

A select statement chooses which of a set of possible send or receive operations will proceed. Go's select lets you wait on multiple channel operations. For example, the following code can perform multiple channel handling.

package main

import (
    "fmt"
    "time"
)

func main() {
    ch1 := make(chan int)
    ch2 := make(chan int)
    done := make(chan struct{})

    go func() {
        time.Sleep(1 * time.Second)
        ch1 <- 1
    }()

    go func() {
        time.Sleep(2 * time.Second)
        ch2 <- 2
    }()

    go func() {
        time.Sleep(3 * time.Second)
        quit <- struct{}{}
    }()

Loop:
    for {
        select {
        case <-done:
            fmt.Println("quit")
            break Loop
        case num := <-ch1:
            fmt.Printf("received: %d\n", num)
        case num := <-ch2:
            fmt.Printf("received: %d\n", num)
        }
    }
}

The implementation pattern of combining for statement and select statement is described in the book "Concurrency in Go" by Katherine Cox-Buday as "The for-select Loop".

for { // Either loop infinitely or range over something
    select {
    // Do some work with channels
    }
}

Aside: labeled break

This is not the main topic, but Loop: means a labeled statements. A labeled statement may be the target of a goto, break or continue statement. In this case, it is trying to do a loop break in the select scope, but for example, the following code will no work as it expected.

for {
    select {
    default:
        break
    }
}

The "Go" way for that kind of situations is to use labels a break on the label.

L:
    for {
        select {
            default:
                break L
        }
    }

Communicate with arbitrary channels

In the above example, the main function listens for transmissions to three channels: ch1, ch2, and done.

select {
case <-done:
    fmt.Println("quit")
    break Loop
case num := <-ch1:
    fmt.Printf("received: %d\n", num)
case num := <-ch2:
    fmt.Printf("received: %d\n", num)
}

However, this does not allow us to receive a variable number of channels. For example, a server holds multiple connections and selects one of them that matches the conditions.

reflect.Select

One option to achieve this is, as the tile says, to use reflect package, especially Select function. It was added in Go 1.1.

func Select(cases []SelectCase) (chosen int, recv Value, recvOK bool)

Select executes a select operation described by the list of cases whose type is reflect.SelectCase. Like the Go select statement, it blocks until at least one of the cases can proceed, makes a uniform pseudo-random choice, and then executes that case.

The first return value chosen is the index of the chosen case, and if that case is a receive operation, the second and third value are returned. The second one is the value received and the third one is a boolean indicating whether the value corresponds to a send on the channel. When the channel is closed, a zero value received.

reflect.SelectCase describes a single case in a select operation.

type SelectCase struct {
    Dir  SelectDir // direction of case
    Chan Value     // channel to use (for send or receive)
    Send Value     // value to send (for send)
}

The kind of case depends on the field Dir whose type is reflect.SelectDir. It indicates the communication direction.

type SelectDir int

const (
    SelectSend    SelectDir // case Chan <- Send
    SelectRecv              // case <-Chan:
    SelectDefault           // default
)

If Dir is SelectSend, the case represents a send operation. If Dir is SelectRecv, the case represents a receive operation. If Dir is SelectDefault, the case represents a default case. For example, the following code will allow you to implement picking up one channel at random from a dynamically generated list of channels.

package main

import (
    "fmt"
    "reflect"
)

func main() {
    // Generate the list of channel
    chs := make([]<-chan int, 0)
    for i := 0; i < 4; i++ {
        ch := make(chan int)
        go func() { ch <- i }()
        chs = append(chs, ch)
    }

    // Convert the list of cases(SelectCase)
    cases := make([]reflect.SelectCase, len(chs))
    for i, ch := range chs {
        cases[i] = reflect.SelectCase{
            Dir:  reflect.SelectRecv,
            Chan: reflect.ValueOf(ch),
            // If Dir is SelectRecv, Send must be a zero Value
            // Send: ...
        }
    }

    // Wait until the channel receives the value (reflect.Select)
    for i := 0; i < 4; i++ {
        chosen, recv, ok := reflect.Select(cases)
        if ok {
            fmt.Printf("chosen: %d, recv: %v\n", chosen, recv)
        }
    }
}

Wait and select an idle connection from connection pools

Let's try to implement reflect.Select to select one of the idle connections from the pooled connections. The design overview is like this:

package main

import (
    "errors"
    "fmt"
    "log"
    "reflect"
)

type Conn struct {
    ID string
    // It may contains some kind of connections like database, websocket and etc.
}

type Pool struct {
    idle chan *Conn
}

type Dispatcher struct {
    // Dispatcher knows the list of connections.
    pools []*Pool
}

func (s *Dispatcher) Select() (*Conn, error) {
    // Convert the list of cases
    cases := make([]reflect.SelectCase, len(s.pools))
    for i, pool := range s.pools {
        cases[i] = reflect.SelectCase{
            Dir:  reflect.SelectRecv,
            Chan: reflect.ValueOf(pool.idle),
        }
    }
    log.Print("dispatcher waiting any idle connection...")
    _, recv, ok := reflect.Select(cases)
    if !ok {
        return nil, errors.New("failed to select a case from connection pool")
    }
    conn, ok := recv.Interface().(*Conn)
    if !ok {
        return nil, errors.New("the type of received connection is invalid")
    }

    return conn, nil
}

func main() {
    // Prepare empty pools
    pools := make([]*Pool, 0)
    for i := 0; i < 4; i++ {
        p := new(Pool)
        p.idle = make(chan *Conn)
        pools = append(pools, p)
    }

    d := Dispatcher{pools: pools}

    // Notify a connection becomes idle
    go func() {
        for i, pool := range pools {
            c := &Conn{ID: fmt.Sprintf("%d", i)}
            pool.idle <- c
        }
    }()

    // Wait and select an idle connection from pools
    for i := 0; i < 4; i++ {
        selected, err := d.Select()
        if err != nil {
            fmt.Printf("err: %#v", err)
        }
        fmt.Printf("selected Connection ID: %s\n", selected.ID)
    }
}

In this example above, the following flow is implemented.

Prepare four pools. Each pool has a channel named idle
The main thread waits for the idle channel by reflect.Select
The channel is notified of an idle connection from another thread

$ go run main.go
2021/12/28 09:24:09 dispatcher waiting any idle connection...
selected Connection ID: 0
2021/12/28 09:24:09 dispatcher waiting any idle connection...
selected Connection ID: 1
2021/12/28 09:24:09 dispatcher waiting any idle connection...
selected Connection ID: 2
2021/12/28 09:24:09 dispatcher waiting any idle connection...
selected Connection ID: 3

Conclusion

reflect.Select allows us to handle an arbitrary number of channels. This article explained the basic knowledge needed to use this coding pattern and introduced some example code.

Reverse HTTP proxy over WebSocket in Go (Part 3)

Kazuki Higashiguchi — Thu, 16 Dec 2021 08:09:13 +0000

Series introduction

In my previous post I talked about how to establish a WebSocket connection in Go.

Reverse HTTP proxy over WebSocket in Go (Part 2)

Kazuki Higashiguchi ・ Dec 15 '21

#go #webdev #websocket

In this post, I will be starting to talk about how to relay TCP connection from "App" to the peer of WebSocket in Go.

Start a WebSocket server (Part 1)
Establish a WebSocket connection (Part 2)
Relay TCP connection from "App" to the peer of WebSocket (Part 3 | Part 4)
Relay TCP connection in WebSocket data to "internal API"
Keep a established connection

Reverse HTTP proxy over WebSocket

I'll use root-gg/wsp as a sample code to explain it. I'll use the forked code to explain because maintenance has stopped and the Go language and libraries version needed to be updated.

hgsgtk / wsp

HTTP tunnel over Websocket

Relay TCP connection to the peer of WebSocket

In the part 1, I presented a demonstration.

HTTP communication is relayed by the following route.

app -[1]-> wsp server -[2](WebSocket)-> wsp client -> internal API

As a pre requirement, a WebSocket connection has already been established between wsp server and wsp client before relaying.

The way to establish WebSocket connection is explained in part 2, but next we need to pool the connection on the server side and use it when relaying.

These flow are divided into three parts to explain it.

Receive requests to be proxied ([1] in the relay flow)
Pool the WebSocket connection on the server for relaying
Relay TCP connection to the peer WebSocket ([2] in the relay flow)

This post describes the 1st and 2nd flow. In part 4, I'll explain the 3rd flow.

1. Receive requests to be proxied

To receive requests to be proxied to "internal API" finally, there are mainly two kinds of API interfaces.

Expose an endpoint (i.e. /requests) that accept HTTP requests
Work as a HTTP proxy used by "app" (i.e. curl -x {wsp server's address} http://localhost:8081)

In the demo example, wsp server (WebSocket server) chose 1st option and exposes the endpoint whose path is /requests.

$ curl -H 'X-PROXY-DESTINATION: http://localhost:8081/hello' http://127.0.0.1:8080/request

Let's read the HTTP handler code in Go (code), which waits the request to /requests/.

func (s *Server) request(w http.ResponseWriter, r *http.Request) {
    // Parse destination URL
    dstURL := r.Header.Get("X-PROXY-DESTINATION")
    if dstURL == "" {
        wsp.ProxyErrorf(w, "Missing X-PROXY-DESTINATION header")
        return
    }
    URL, err := url.Parse(dstURL)
    if err != nil {
        wsp.ProxyErrorf(w, "Unable to parse X-PROXY-DESTINATION header")
        return
    }
    r.URL = URL

    // (omit)
}

wsp defined the custom HTTP header X-PROXY-DESTINATION to specify the final destination URL. For example, when you want to access http://localhost:8081/hello, add a HTTP header X-PROXY-DESTINATION: http://localhost:8081/hello in your request.

2. Pool the WebSocket connection on the server for relaying

We need to make sure that the http handler can recognize and use the connected WebSocket connections.

In this code, an HTTP server that exposes an endpoint /request is set up at a given address. The handler mapped to /request is as follow (code):

func (s *Server) Register(w http.ResponseWriter, r *http.Request) {
    // - 1. Upgrade a received HTTP request to a WebSocket connection
    // (omit)
    // - 2. Wait a greeting message from the peer and parse it
    // (omit)

    // 3. Register the connection into server pools.
    // s.lock is for exclusive control of pools operation.
    s.lock.Lock()
    defer s.lock.Unlock()

    var pool *Pool
    // There is no need to create a new pool,
    // if it is already registered in current pools.
    for _, p := range s.pools {
        if p.id == id {
            pool = p
            break
        }
    }
    if pool == nil {
        pool = NewPool(s, id)
        s.pools = append(s.pools, pool)
    }
    // update pool size
    pool.size = size

    // Add the WebSocket connection to the pool
    pool.Register(ws)
}

First, the server needs to have a WebSocket connection pooled so that it can be used for relay. One design idea to achieve this is to prepare three entities Server, Pool, and Connection as shown below.

In this code, the Server struct have a field pools which is a pointer of the slice of Pool struct.

type Server struct {
    // (omit)

    pools []*Pool

    // (omit)
}

The Pool struct represents the connection from ws client. The definition is as follow:

type Pool struct {
    server *Server
    id     PoolID

    size int

    connections []*Connection
    idle        chan *Connection

    done bool
    lock sync.RWMutex
}

The Connection struct manages a single WebSocket connection because wsp supports multiple connections from a single wsp client at the same time.

type Connection struct {
    pool         *Pool
    ws           *websocket.Conn
    status       ConnectionStatus
    idleSince    time.Time
    lock         sync.Mutex
    nextResponse chan chan io.Reader
}

Then, going back to the handler code, we can see that the server checks whether requesting connection is already registered, and create a new pool if it's not registered in current pools.

func (s *Server) Register(w http.ResponseWriter, r *http.Request) {
    // (omit)

    var pool *Pool
    // There is no need to create a new pool,
    // if it is already registered in current pools.
    for _, p := range s.pools {
        if p.id == id {
            pool = p
            break
        }
    }
    if pool == nil {
        pool = NewPool(s, id)
        s.pools = append(s.pools, pool)
    }

    // (omit)
}

Each ws client issues an unique ID, which is a common design when you want to pool and recognize external devices. By issuing an ID, the server is able to recognize meta information such as ws client source IP and so on.

At the end of the handler function, the server registers a new connection to the created pool.

func (s *Server) Register(w http.ResponseWriter, r *http.Request) {
    // (omit)

    pool.Register(ws)
}

Pool.Register is as follow:

func (pool *Pool) Register(ws *websocket.Conn) {
    // (omit)

    log.Printf("Registering new connection from %s", pool.id)
    connection := NewConnection(pool, ws)
    pool.connections = append(pool.connections, connection)
}

This handler process registered an idle connection that can be used for relay. NewConnection() function mark a new connection as a usable connection for relay in the pool and start a thread (technically, start a new goroutine running) that keeps reading messages over the connected WebSocket connection from the wsp client.

func NewConnection(pool *Pool, ws *websocket.Conn) *Connection {
    // Initialize a new Connection
    c := new(Connection)
    c.pool = pool
    c.ws = ws
    c.nextResponse = make(chan chan io.Reader)
    c.status = Idle

    // Mark that this connection is ready to use for relay
    c.Release()

    // Start to listen to incoming messages over the WebSocket connection
    go c.read()

    return c
}

At initialization, a new Connection is created by state Idle.

Enum in Go

The status Idle means it is opened but not working now. Here is a state diagram describing the behavior of Connection.

In this repository, I made the defined type ConnectionStatus and constants Idle, Busy, and Closed.

// ConnectionStatus is an enumeration type which represents the status of WebSocket connection.
type ConnectionStatus int

const (
    // Idle state means it is opened but not working now.
    // The default value for Connection is Idle, so it is ok to use zero-value(int: 0) for Idle status.
    Idle ConnectionStatus = iota
    Busy
    Closed
)

By the way, if you want not to define ConnectionStatus with Zero-Value, you can skip the zero-value as follows:

// Pattern1: Assign zero-value to blank
const (
    _ State = iota 
    A = iota
    B
)

// Pattern2: Start from 1
const (
    C State = iota + 1
    D
)

Conclusion

This post explained how to relay TCP connection from "App" to the peer of WebSocket, especially implementation to receive requests to be proxied and to pool the WebSocket connection on the server for relaying.

I will explain the rest points in part 3 and beyond.

Relay TCP connection from "App" to the peer of WebSocket (rest)
Relay TCP connection in WebSocket data to "internal API"
Keep a established connection

Reverse HTTP proxy over WebSocket in Go (Part 2)

Kazuki Higashiguchi — Wed, 15 Dec 2021 07:31:26 +0000

Series introduction

In my previous post I talked about how to start WebSocket server in Go.

Reverse HTTP proxy over WebSocket in Go (Part 1)

Kazuki Higashiguchi ・ Dec 14 '21

#go #webdev #websocket

In this post, I will be starting to talk about how to establish a WebSocket connection in Go.

Start a WebSocket server (Part 1)
Establish a WebSocket connection (Part 2)
Keep a established connection (Part 3 | Part 4)
Relay TCP connection from "App" to the peer of WebSocket
Relay TCP connection in WebSocket data to "internal API"

Reverse HTTP proxy over WebSocket

I'll use root-gg/wsp as a sample code to explain it. I'll use the forked code to explain because maintenance has stopped and the Go language and libraries version needed to be updated.

hgsgtk / wsp

HTTP tunnel over Websocket

Establish a WebSocket connection

In my previous post, I explained the implementation for starting a WebSocket server.

To establish a WebSocket connection, the WebSocket client needs to request a handshake to the server.

When you run this client program, it will start to negotiate the handshake, then finished to establish the connection.

$ go run cmd/wsp_client/main.go -config examples/wsp_client.cfg

2021/12/15 15:10:54 Connecting to ws://127.0.0.1:8080/register
2021/12/15 15:10:54 Connected to ws://127.0.0.1:8080/register

Let's read the Go code. First is the main function (cmd/client/main.go), which is the entry point.

package main

import (
    // (omit)
)

func main() {
    // (omit)

    proxy := client.NewClient(config)
    proxy.Start(ctx)

    // (omit: shutdown)
}

Initialize a WebSocket client

The first line is:

proxy := client.NewClient(config)

client.NewClient function returns a pointer of client.Client struct which connects to servers using the WebSocket protocol.

type Client struct {
    Config *Config

    client *http.Client
    dialer *websocket.Dialer
    pools  map[string]*Pool
}

An important fields is dialer whose type is websocket.Dialer to contains options for connection to WebSocket server. The websocket.Dialer struct is like that:

type Dialer struct {
    NetDial func(network, addr string) (net.Conn, error)
    NetDialContext func(ctx context.Context, network, addr string) (net.Conn, error)
    Proxy func(*http.Request) (*url.URL, error)
    TLSClientConfig *tls.Config
    HandshakeTimeout time.Duration
    ReadBufferSize, WriteBufferSize int
    WriteBufferPool BufferPool
    Subprotocols []string
    EnableCompression bool
    Jar http.CookieJar
}

There are some points to understand the WebSocket protocol, see the following post more detail.

Deep dive into WebSocket opening handshake protocol with Go

Kazuki Higashiguchi ・ Dec 11 '21

#webdev #websocket #go

Send a handshake request to a WebSocket server

The next line is:

proxy.Start()

Several functions are called from this code but the important one is the following code.

connection.ws, _, err = connection.pool.client.dialer.DialContext(
    ctx,
    connection.pool.target,
    http.Header{"X-SECRET-KEY": {connection.pool.secretKey}},
)

Dialer.DialContext creates a new client connection to open handshake, then it returns a pointer of websocket.Conn struct which represents a WebSocket connection.

If successful, WebSocket handshake is complete. You can start exchanging messages over the established WebSocket connection.

Greeting messages

Immediately after handshake, we sometimes do greeting to send client meta data (i.e. device ID) or authentication information via WebsSocket messasge.

For this scene, you can receive and send messages by calling the Conn's ReadMessage and WriteMessage.

Data is transmitted using a sequence of frames. This wire format for the data transfer part is described by the ABNF (Augmented BNF for Syntax Specification) in the WebSocket protocol. A overview of the framing is given in the following figure.

In this example, the client sends a message to the server.

greeting := fmt.Sprintf(
    "%s_%d",
    connection.pool.client.Config.ID,
    connection.pool.client.Config.PoolIdleSize,
)
err = connection.ws.WriteMessage(websocket.TextMessage, []byte(greeting))
if err != nil {
    log.Println("greeting error :", err)
    connection.Close()
    return
}

Conn.WriteMessage

func (c *Conn) WriteMessage(messageType int, data []byte) error

And then, websocket.TextMessage is one of opcodes which is defined in RFC 6455 - 11.8. WebSocket Opcode Registry.

Opcode	Meaning
0	Continuation Frame
1	Text Frame
2	Binary Frame
8	Connection Close Frame
9	Ping Frame
10	Pong Frame

websocket.TextMessage means the opcode 1 (Text Frame), which is for exchanging text data over a WebSocket connection.

The server will receive the message sent by the above code (code.

// Handshaking
ws, err := server.upgrader.Upgrade(w, r, nil)
if err != nil {
    wsp.ProxyErrorf(w, "HTTP upgrade error : %v", err)
    return
}

// Receive the message from the client
_, greeting, err := ws.ReadMessage()
if err != nil {
    wsp.ProxyErrorf(w, "Unable to read greeting message : %s", err)
    ws.Close()
    return
}

ReadMessage is a method to read messages from the peer which interprets data in base framing protocol.

func (c *Conn) ReadMessage() (messageType int, p []byte, err error)

When you want to exchange simple text messages over WebSocket connection, you can do it with just two methods I just introduced in this post.

Conclusion

This post explained how to establish a WebSocket connection.

I will explain the rest points in part 3 and beyond.

Keep a established connection

Reverse HTTP proxy over WebSocket in Go (Part 3)

Kazuki Higashiguchi ・ Dec 16 '21

#go #webdev #websocket

Relay TCP connection from "App" to the peer of WebSocket
Relay TCP connection in WebSocket data to "internal API"

Reverse HTTP proxy over WebSocket in Go (Part 1)

Kazuki Higashiguchi — Tue, 14 Dec 2021 10:07:16 +0000

Key takeaways

A reverse HTTP proxy over WebSocket is a type of proxy server which uses the WebSocket protocol as a "tunnel" to pass TCP communication from server to client.
In Go project, gorilla/websocket is widely used to implement WebSocket.
WebSocket is designed to work over HTTP. To achieve compatibility with HTTP, the WebSocket handshake uses the HTTP Upgrade header in order to change from the HTTP protocol to the WebSocket protocol.
Authentication of WebSocket is a difficult design problem, and there are many options.

Reverse HTTP proxy over WebSocket

There are two types of proxies, forward proxy and reverse proxy.

A forward proxy (or gateway, or tunnel, or just "proxy") provides proxy services to a client or a group of clients. It allow users to hide their IP address while accessing a Web service or an API server. In contract, a reverse proxy is the opposite of what a forward proxy does. It is a type of proxy server that retrieves resources on behalf of a client from servers.

Further, this post focuses on a reverse HTTP proxy over WebSocket, in brief, it uses the WebSocket protocol as a "tunnel" to pass TCP communication from server to client.

A case study in Go

There are not that many examples of implementation reverse proxy over WebSocket. In JavaScript, mhzed/wstunnel is well known, in Haskell, erebe/wstunnel is.

In Go, inconshreveable/ngrok and coyove/goflyway is well known, especially ngrok is popular among developers as a SaaS service.

In this post, we will focus on understanding the basic concepts and read prototypical and simpler one root-gg/wsp. wsp is developed by root-gg, which is a reverse HTTP proxy over WebSocket, whose aim is to securely make call to internal APIs from outside. It is difficult to use in production, but it is a good learning material to explain the design of reverse proxy over WebSocket.

However, maintenance has stopped since the days of Go 1.6, so I'll proceed with this post based on code hgsgtk/wsp that I forked and modified for the Go situation in 2021 (Thank you root-gg).

hgsgtk / wsp

HTTP tunnel over Websocket

Demonstration of wsp

It's easier to get a general idea of how it works if you see it in action, so here's a brief demo. In the demonstration, the following four system component are introduced.

Internal API(http://localhost:8081): the server you want to request via the reverse proxy
wsp client: running in a network with the internal API
wsp server(http://localhost:8080): running outside the network
App: starting point for communication to the internal API

Try to send a request to the internal API via WebSocket communication to the endpoint /hello.

$ curl -H 'X-PROXY-DESTINATION: http://localhost:8081/hello' http://127.0.0.1:8080/request

HTTP communication was relayed by the following route.

app -> wsp server -(WebSocket)-> wsp client -> internal API

Here is a terminal image when sending a HTTP request from app.

The internal network restricts incoming requests from outside or does not have any global IPs, so that it is not possible for external server to connect to a client started in an internal network. For example, if the client is launched on your local PC, it is not possible to send a request to the client from an external server.

Therefore, the starting point is wsp client in the internal network. In this design, it starts with a request for a WebSocket connection from wsp client to wsp server.

The following points to implement this design will be explained in this post.

Start a WebSocket server

I will explain the rest points in part 2 and beyond.

Establish a WebSocket connection
Keep a established connection
Relay TCP connection from "App" to the peer of WebSocket
Relay TCP connection in WebSocket data to "internal API"

Start a WebSocket server

The first step is to start up a WebSocket server which waits for requests from WebSocket clients.

As a side note, WebSocket is a mechanism for low-cost, full-duplex communication on Web, which protocol was standardized as RFC 6455. The following diagram, quoted by Wikipedia, describe a communication using WebSocket between client and server.

In the case of wsp, it starts like this.

$ ./wsp_server -config examples/wsp_server.cfg
{
  "Host": "127.0.0.1",
  "Port": 8080,
  "Timeout": 1000,
  "IdleTimeout": 60000,
  "Whitelist": [],
  "Blacklist": [],
  "SecretKey": ""
}

Let's read the Go code. First is the main function (cmd/wsp_server/main.go), which is the entry point.

package main

import (
    // (omit)
)

func main() {
    // (omit)

    server := server.NewServer(config)
    server.Start()

    // (omit: shutdown)
}

server.NewServer function returns a pointer of Server struct which represents a reverse HTTP proxy over WebSocket.

type Server struct {
    Config *Config

    upgrader websocket.Upgrader

    pools []*Pool
    lock  sync.RWMutex
    done  chan struct{}

    dispatcher chan *ConnectionRequest

    server *http.Server
}

An important field is upgrader whose type is websocket.Upgrader to specify parameters for upgrading an HTTP connection to a WebSocket connection. In Go project, gorilla/websocket is widely used to implement WebSocket, and many samples using this library are available on the internet.

WebSocket is designed to work over HTTP. To achieve compatibility with HTTP, the WebSocket handshake uses the HTTP Upgrade header in order to change from the HTTP protocol to the WebSocket protocol.

Fields of Upgrader struct is as follow:

type Upgrader struct {
    HandshakeTimeout time.Duration
    ReadBufferSize, WriteBufferSize int
    Subprotocols []string
    Error func(w http.ResponseWriter, r *http.Request, status int, reason error)
    CheckOrigin func(r *http.Request) bool
    EnableCompression bool
}

If you want to know more about the WebSocket parameters or gorilla/websocket internal implementation, refer to this post Learn WebSocket handshake protocol with gorilla/websocket server.

In NewServer function, it creates a value whose type is websocket.Upgrader with default parameters.

func NewServer(config *Config) (server *Server) {
    // (omit)

    server.upgrader = websocket.Upgrader{}

    // (omit)
    return
}

Next, call server.Start function to start a WebSocket server.

func (server *Server) Start() {
    // (omit)

    r := http.NewServeMux()
    r.HandleFunc("/request", server.request)

    // (omit)

    server.server = &http.Server{
        Addr:    server.Config.GetAddr(),
        Handler: r,
    }
    go func() { log.Fatal(server.server.ListenAndServe()) }()

In this code, an HTTP server that exposes an endpoint /request is set up at a given address. The handler mapped to /request is as follow (code):

func (server *Server) register(w http.ResponseWriter, r *http.Request) {
    secretKey := r.Header.Get("X-SECRET-KEY")
    if secretKey != server.Config.SecretKey {
        wsp.ProxyErrorf(w, "Invalid X-SECRET-KEY")
        return
    }

    ws, err := server.upgrader.Upgrade(w, r, nil)
    if err != nil {
        wsp.ProxyErrorf(w, "HTTP upgrade error : %v", err)
        return
    }

    // (omit)
}

There are several knowledges to implement WebSocket server, so I'll explain it line by line.

Authentication of WebSocket

The first line is:

secretKey := r.Header.Get("X-SECRET-KEY")
if secretKey != server.Config.SecretKey {
    wsp.ProxyErrorf(w, "Invalid X-SECRET-KEY")
    return
}

The WebSocket protocol does not handle authorization or authentication. However, it is a problem that anyone can access the WebSocket server without authentication just because the protocol is not prepared.

Business requirements often require authenticating the connecting WebSocket client and associating it with a registered customer record.

There are several patterns to implement authentication, and Heroku Dev Center - WebSocket Security describes one pattern - "ticket" -based authentication system.

The client-side contacts the server-side to obtain an authorization "ticket".
The server stores the ticket and returns it to the client
The client opens the WebSocket connection, and sends along this "ticket" as part of an initial handshake.
The server can compare this ticket, check an user's condition (i.e. source IPs)

See Authorization and IPs chapter in Armin Ronacher's post Websockets 101 for detail.

There are several options for where and how to tell the "ticket" from the client to the server. A commonly used method for simple authentication is to set the relevant ticket in the HTTP header. For example, Amazon API Gateway provides the option to require API clients to pass the API key as the X-API-Key header.

The sample code implements a similar concept. It validates the value in the header X-SECRET-KEY at initial handshake.

secretKey := r.Header.Get("X-SECRET-KEY")

Also, depending on your business requirements, there are several options for how to issue the "ticket". A idea is to issue "ticket" (or access key, or access token) in advance on their administration UI. Or if the process of asking where the WebSocket server to connect contains before handshake, it is also good idea to issue a one-time "ticket" by "access token" which is issued in advance.

In "ticket" -based authentication system, the server authenticates only at initial handshake, but there is also an idea to authenticate every message. See Klee Thomas's post Authenticating Websockets more detail.

Handshake (HTTP Upgrade)

After authentication, upgrade an HTTP connection to an WebSocket connection.

ws, err := server.upgrader.Upgrade(w, r, nil)
if err != nil {
    wsp.ProxyErrorf(w, "HTTP upgrade error : %v", err)
    return
}

websocket.Upgrader.Upgrade function does it. To put it simply, it checks if the handshake request from the client is valid and returns a successful handshake response if it is.

i.e. a valid handshake request

GET /register HTTP/1.1
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==
Origin: http://example.com
Sec-WebSocket-Version: 13

i.e. a successful handshake response

HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: s3pPLMBiTxaQ9kYGzzhZRbK+xOo=

If you want to know the internal implementation of Upgrader.Upgrade function, see Learn WebSocket handshake protocol with gorilla/websocket server.

Conclusion

This post explained how to start a WebSocket server and WebSocket specification and security considerations to establish a WebSocket connection.

I will explain the rest points in part 2 and beyond.

Establish a WebSocket connection (Part 2)

Reverse HTTP proxy over WebSocket in Go (Part 2)

Kazuki Higashiguchi ・ Dec 15 '21

#go #webdev #websocket

Keep a established connection
Relay TCP connection from "App" to the peer of WebSocket
Relay TCP connection in WebSocket data to "internal API"

How WebSocket protocol designs bidirectional messaging and implements in Go

Kazuki Higashiguchi — Mon, 13 Dec 2021 04:59:20 +0000

WebSocket

WebSocket is a mechanism for low-cost, full-duplex communication on Web, which protocol was standardized as RFC 6455.

The following diagram, quoted by Wikipedia, describe a communication using WebSocket between client and server.

"Handshake" is explained in the following two articles.

This post will focus on "Bidirectional messages"

WebSocket server

Let's learn the WebSocket bidirectional messaging protocol with a specific Go implementation, which is available on a GitHub repository.

package main

import (
    "fmt"
    "log"
    "net/http"

    "github.com/gorilla/websocket"
)

var upgrader = websocket.Upgrader{}

func main() {
    p := 12345
    log.Printf("Starting websocket echo server on port %d", p)

    http.HandleFunc("/", echo)
    if err := http.ListenAndServe(fmt.Sprintf(":%d", p), nil); err != nil {
        log.Panicf("Error while starting to listen: %#v", err)
    }
}

func echo(w http.ResponseWriter, r *http.Request) {
    // Start handshaking
    c, err := upgrader.Upgrade(w, r, nil)
    if err != nil {
        log.Printf("Upgrading error: %#v\n", err)
        return
    }
    defer c.Close()
    // End

    log.Println("Success to handshake with client")

    // Start bidirectional messages
    for {
        mt, message, err := c.ReadMessage()
        if err != nil {
            log.Printf("Reading error: %#v\n", err)
            break
        }
        log.Printf("recv: message %q", message)
        if err := c.WriteMessage(mt, message); err != nil {
            log.Printf("Writing error: %#v\n", err)
            break
        }
    }
    // End bidirectional messages
}

Open handshake

The first half of the code is the server implementation for "Handshake".

// 1. Initializes parameters for upgrading an HTTP connection to a WebSocket connection.
var upgrader = websocket.Upgrader{}

// 2. Starts WebSocket server which waits client HTTP requests on port 12345
func main() {
    p := 12345
    log.Printf("Starting websocket echo server on port %d", p)

    http.HandleFunc("/", echo)
    if err := http.ListenAndServe(fmt.Sprintf(":%d", p), nil); err != nil {
        log.Panicf("Error while starting to listen: %#v", err)
    }
}

func echo(w http.ResponseWriter, r *http.Request) {
    // 3. Start handshaking
    c, err := upgrader.Upgrade(w, r, nil)
    if err != nil {
        log.Printf("Upgrading error: %#v\n", err)
        return
    }
    defer c.Close()
    // End

    log.Println("Success to handshake with client")

    // ...(Bidirectional messages part)

As noted in the comments in the code, to start up a WebSocket server, do the following:

Initializes parameters for upgrading an HTTP connection to a WebSocket connection.
Starts WebSocket server which waits client HTTP requests on port 12345
Start handshaking

When you run this program, it will start an HTTP server on port 12345.

$ go run server/main.go
2021/12/13 11:54:22 Starting websocket echo server on port 12345

You can check if the handshake is actually working by making a curl request (but it's not perfect).

$ curl -X GET http://localhost:12345 \
-H "Connection: upgrade" \
-H "Upgrade: websocket" \
-H "Sec-Websocket-version: 13" \
-H "Sec-Websocket-Key: 08kp54j1E3z4IfuM1m75tQ==" \
-H "Host: localhost:12345" \
-H "Origin: http://localhost:12345"

When you send it, you will see that the request was accepted in stdout.

$ go run main.go
(omit)...
2021/12/13 12:07:19 Success to handshake with client

See Learn WebSocket handshake protocol with gorilla/websocket server for more information.

WebSocket connection

The key to understanding the message exchanging is the return value of this line.

c, err := upgrader.Upgrade(w, r, nil)

The variable c is a pointer of websocket.Conn struct. The Conn type represents a WebSocket connection.

When the handshake is completed, you can start exchanging messages over the established WebSocket connection.

You can receive and send messages by calling the Conn's ReadMessage and WriteMessage.

// Start bidirectional messages
for {
    mt, message, err := c.ReadMessage()
    if err != nil {
        log.Printf("Reading error: %#v\n", err)
        break
    }
    log.Printf("recv: message %q", message)

    // (omit...writing message part)
}
// End bidirectional messages

Use for loop so that the server keeps waiting for messages from the client.

Data Framing

In the WebSocket protocol, data is transmitted using a sequence of frames. This wire format for the data transfer part is described by the ABNF (Augmented BNF for Syntax Specification). A overview of the framing is given in the following figure.

Server and client exchange data in accordance with this method.

Read messages from the peer

ReadMessage is a method to read messages from the peer which interprets data in base framing protocol.

mt, message, err := c.ReadMessage()

The implementation of ReadMessage is as follow:

func (c *Conn) ReadMessage() (messageType int, p []byte, err error) {
    var r io.Reader
    messageType, r, err = c.NextReader()
    if err != nil {
        return messageType, nil, err
    }
    p, err = ioutil.ReadAll(r)
    return messageType, p, err
}

The Conn.NextReader function reads bytes received from the peer.

messageType, r, err = c.NextReader()

Opcode

First return value is messageType which represents WebSocket opcode numbers defined in RFC 6455 - 11.8. WebSocket Opcode Registry.

Opcode	Meaning
0	Continuation Frame
1	Text Frame
2	Binary Frame
8	Connection Close Frame
9	Ping Frame
10	Pong Frame

There are two types of opcode: message for exchanging data and controlling WebSocket connection.

For exchanging data
- Text Frame
- Binary Frame
For controlling WebSocket connection
- Connection Close Frame
- Ping Frame
- Pong Frame
- Contination Frame

The messages for exchanging data distinguishes between text and binary data messages. And, the WebSocket protocol defines four types of control messages: close, ping, pong, and continuation.

Conclusion

This article explains WebSocket bidirectional message using gorilla/websocket server implementation.

Deep dive into WebSocket opening handshake protocol with Go

Kazuki Higashiguchi — Sat, 11 Dec 2021 03:13:05 +0000

In this article, I will explain what exactly WebSocket protocol does when handshaking. What a WebSocket server should do is explained in Learn WebSocket handshake protocol with gorilla/websocket server.

This article focuses what a WebSocket client does, and explains it with Go codes.

WebSocket

WebSocket is a mechanism for low-cost, full-duplex communication on Web, which protocol was standardized as RFC 6455.

The following diagram, quoted by Wikipedia, describe a communication using WebSocket between client and server.

Now, we will focus on the first step "Handshake (HTTP Upgrade)",

WebSocket client

Next, let's learn the WebSocket protocol with client implementation. The sample code is available on a GitHub repository.

package main

import (
    "log"

    "github.com/gorilla/websocket"
)

func main() {
    ctx := context.Background()

    // (omit)... signal handling
    endpointUrl := "ws://localhost:12345"

    dialer := websocket.Dialer{}
    c, _, err := dialer.DialContext(ctx, endpointUrl, nil)
    if err != nil {
        log.Panicf("Dial failed: %#v\n", err)
    }
    defer c.Close()

    // Codes for bidirectional messages (listening to read messages for server)
    done := make(chan struct{})
    go func() {
        defer c.Close()
        defer close(done)

        for {
            _, message, err := c.ReadMessage()
            if err != nil {
                log.Printf("read message: %#v\n", err)
                return
            }
            log.Printf("recv: %s\n", message)
        }
    }()

    // (omit)... periodic message sending and shutdown handling
}

URI schemes "ws" and "wss"

Specify the endpoint of the request to the WebSocket server like this.

endpointUrl := "ws://localhost:12345"

The WebSocket protocol described in RFC 6455 registered two URI schemes.

ws: WebSocket Protocol
- A ws URI identifies a WebSocket server and resource name
wss: WebSocket Protocol over TLS
- A wss URI identifies a WebSocket server and resource name and indicates that traffic over that connection is to be protected via TLS.

Some security considerations about WebSocket are described in RFC 6455 10.6. Connection Confidentiality and Integrity.

Connection confidentiality and integrity is provided by running the WebSocket Protocol over TLS (wss URIs). WebSocket implementations MUST support TLS and SHOULD employ it when communicating with their peers.

In the production use case, we need to use wss to ensure connection confidentiality and integrity, but this sample code is only intended to run locally, so I use ws scheme.

Subprotocols

The next line is an initialization of Dialer struct. A Dialer contains options for connection to WebSocket server.

dialer := websocket.Dialer{}

Here is the definition of Dialer struct.

type Dialer struct {
    NetDial func(network, addr string) (net.Conn, error)
    NetDialContext func(ctx context.Context, network, addr string) (net.Conn, error)
    Proxy func(*http.Request) (*url.URL, error)
    TLSClientConfig *tls.Config
    HandshakeTimeout time.Duration
    ReadBufferSize, WriteBufferSize int
    WriteBufferPool BufferPool
    Subprotocols []string
    EnableCompression bool
    Jar http.CookieJar
}

An important field to understand the WebSocket protocol is Subprotocols.

Subprotocols is like a custom XML schema or doctype declaration. For example, if you're using a subprotocol json, all data is passed as JSON.

    Sec-WebSocket-Protocol: soap, wamp

By specifying subprotocols with a client, you can tell the server which protocol you want to use. For instance:

dialer := websocket.Dialer{
    Subprotocols: []string{"json"},
}
c, _, err := dialer.DialContext(ctx, endpointUrl, nil)
if err != nil {
    log.Panicf("Dial failed: %#v\n", err)
}
fmt.Printf("negotiated protocol: %q\n", c.Subprotocol())
// Output: ""

Conn.Subprotocol function return the negotiated protocol for the connection. In the above example, the WebSocket server does not support json subprotocol, so the negotiated result is "no subprotocol used".

A WebSocket serve specifies its supported protocols, and when the server does not support any subprotocols, the client's demands will not be met.

Opening handshake

Dialer.DialContext creates a new client connection to open handshake.

dialer := websocket.Dialer{}
c, _, err := dialer.DialContext(ctx, endpointUrl, nil)
if err != nil {
    log.Panicf("Dial failed: %#v\n", err)
}
defer c.Close()

You can understand the specification of WebSocket handshake by reading codes inside Dialer.DialContext.

At first, we need to decide the URI scheme. In gorilla/websocket, the URI scheme is determined as follows:

switch u.Scheme {
case "ws":
    u.Scheme = "http"
case "wss":
    u.Scheme = "https"
default:
    return nil, nil, errMalformedURL
}

ws -> http
wss -> https

In the case of wss, specify https as the URI scheme for opening handshake because communication should be over TLS.

Second, we send HTTP GET request by codes in gorilla/websocket.

req := &http.Request{
    Method:     "GET",
    URL:        u,
    Proto:      "HTTP/1.1",
    ProtoMajor: 1,
    ProtoMinor: 1,
    Header:     make(http.Header),
    Host:       u.Host,
}
// (omit)...
req.Header["Upgrade"] = []string{"websocket"}
req.Header["Connection"] = []string{"Upgrade"}
req.Header["Sec-WebSocket-Key"] = []string{challengeKey}
req.Header["Sec-WebSocket-Version"] = []string{"13"}
if len(d.Subprotocols) > 0 {
    req.Header["Sec-WebSocket-Protocol"] = []string{strings.Join(d.Subprotocols, ", ")}
}

The specification of handshake is as follow:

The method of the request MUST be GET as described on RFC 6455 - page 16
The request uses HTTP/1.1.
The request MUST contain a Upgrade header field whose value MUST include the websocket keyword.
The request MUST contain a Connection header field whose value MUST include the upgrade token.
Sec-Websocket-Version is the WebSocket protocol version which is supported by server. Available versions are listed in IANA WebSocket Version Number Registry. RFC 6415 is registered as Version 13, so basically, only 13 should be supported.

Finally, verify that the handshake would be completed correctly by validating the response (here).

if resp.StatusCode != 101 ||
!strings.EqualFold(resp.Header.Get("Upgrade"), "websocket") ||
!strings.EqualFold(resp.Header.Get("Connection"), "upgrade") ||
resp.Header.Get("Sec-Websocket-Accept") != computeAcceptKey(challengeKey) {
    buf := make([]byte, 1024)
    n, _ := io.ReadFull(resp.Body, buf)
    resp.Body = ioutil.NopCloser(bytes.NewReader(buf[:n]))
    return nil, resp, ErrBadHandshake
}

The handshake response of WebSocket server MUST be as follows:

Return HTTP status code 101 (Switching Protocols)
- The HTTP 101 Switching Protocols response code indicates the protocol the server is switching to as requested by a client.
The value in the header Sec-Websocket-Accept should be computed by the value in Sec-Websocket-Key. See more detail on How decided a value set in Sec-WebSocket-Key/Accept header.

Conclusion

This article explains WebSocket handshake protocol using gorilla/websocket client implementation.

Additional articles will be published on the WebSocket protocol from the server perspective and data frame processing when exchanging messages and etc.

Learn WebSocket handshake protocol with gorilla/websocket server

Learn WebSocket handshake protocol with gorilla/websocket server

Kazuki Higashiguchi — Thu, 09 Dec 2021 10:40:09 +0000

Let's learn the WebSocket protocol with a simple implementation of echo WebSocket server and client in Go.

WebSocket

WebSocket is a mechanism for low-cost, full-duplex communication on Web, which protocol was standardized as RFC 6455.

The following diagram, quoted by Wikipedia, describe a communication using WebSocket between client and server.

Let's see how to implement the above design, especially handshaking, using the code which works.

WebSocket in Go

WebSocket server and client can be implemented in various programming languages, but in this article, I use Go language because it is simple and easy to read.

In Go project, gorilla/websocket is widely used to implement WebSocket, and many samples using this library are available on the internet.

I'll explain what gorilla/websocket does in its internal implementation while reading RFC 6455.

WebSocket server

First, let's learn the WebSocket protocol with server implementation. The sample code is available on GitHub repository. I'll explain it line by line.

package main

import (
    "fmt"
    "log"
    "net/http"

    "github.com/gorilla/websocket"
)

var upgrader = websocket.Upgrader{}

func main() {
    p := 12345
    log.Printf("Starting websocket echo server on port %d", p)

    http.HandleFunc("/", echo)
    if err := http.ListenAndServe(fmt.Sprintf(":%d", p), nil); err != nil {
        log.Panicf("Error while starting to listen: %#v", err)
    }
}

func echo(w http.ResponseWriter, r *http.Request) {
    // Start handshaking
    c, err := upgrader.Upgrade(w, r, nil)
    if err != nil {
        log.Printf("Upgrading error: %#v\n", err)
        return
    }
    defer c.Close()
    // End

    for {
        mt, message, err := c.ReadMessage()
        if err != nil {
            log.Printf("Reading error: %#v\n", err)
            break
        }
        log.Printf("recv: message %q", message)
        if err := c.WriteMessage(mt, message); err != nil {
            log.Printf("Writing error: %#v\n", err)
            break
        }
    }
}

This article focuses on the handshake on the WebSocket protocol from WebSocket server perspective.

"Upgrade" an HTTP to a WebSocket

Here is a first line, excluding the import statement.

var upgrader = websocket.Upgrader{}

Upgrader struct specifies parameters for upgrading an HTTP connection to a WebSocket connection. The naming itself, "Upgrader", is also meaningful.

WebSocket is designed to work over HTTP. To achieve compatibility with HTTP, the WebSocket handshake uses the HTTP Upgrade header in order to change from the HTTP protocol to the WebSocket protocol.

The Upgrade header can be used to upgrade an already established client/server connection to a different protocol. To use Upgrade header, Connection: update must be set.

RFC 6455 - 1.2 Protocol Overview illustrates an example of WebSocket handshake.

From client:

    GET /chat HTTP/1.1
    Host: server.example.com
    Upgrade: websocket
    Connection: Upgrade
    Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==
    Origin: http://example.com
    Sec-WebSocket-Protocol: chat, superchat
    Sec-WebSocket-Version: 13

Then server will respond as follow:

    HTTP/1.1 101 Switching Protocols
    Upgrade: websocket
    Connection: Upgrade
    Sec-WebSocket-Accept: s3pPLMBiTxaQ9kYGzzhZRbK+xOo=
    Sec-WebSocket-Protocol: chat

Fields of Upgrader struct is as follow:

type Upgrader struct {
    HandshakeTimeout time.Duration
    ReadBufferSize, WriteBufferSize int
    Subprotocols []string
    Error func(w http.ResponseWriter, r *http.Request, status int, reason error)
    CheckOrigin func(r *http.Request) bool
    EnableCompression bool
}

Important fields to understand WebSocket protocol are HandshakeTimeout and Subprotocols.

Handshake

HandshakeTimeout specifies the duration for the handshake to complete.

The first step to exchange message bidirectionally is handshake where opens a new connection between client and server.

Subprotocols

Subprotocols is like a custom XML schema or doctype declaration.

    Sec-WebSocket-Protocol: soap, wamp

For example, if you're using a subprotocol json, all data is passed as JSON (of course, you needs extra code on the server to parse JSON data).

As a WebSocket server, it specifies the server's supported protocols. Therefore, when the server specifies supported protocol is only soap, server will reject client's handshake with other subprotocols such as json.

Wait handshake from client

Here is a http handler which waits HTTP requests to URI path / on port 12345.

func main() {
    http.HandleFunc("/", echo)
    if err := http.ListenAndServe(fmt.Sprintf(":%d", 12345), nil); err != nil {
        log.Panicf("Error while starting to listen: %#v", err)
    }
}

func echo(w http.ResponseWriter, r *http.Request) {
    c, err := upgrader.Upgrade(w, r, nil)
    if err != nil {
        log.Printf("Upgrading error: %#v\n", err)
        return
    }
    defer c.Close()

    // ...(omit)
}

echo function first performs WebSocket handshake procedure in response to a request from client. The detailed implementation is in Upgrader.Upgrade function.

c, err := upgrader.Upgrade(w, r, nil)

Upgrade upgrades the HTTP server connection to the WebSocket protocol.

Handshake request validation

Upgrade function begins with gorilla/websocket codes which validate client HTTP request.

    if !tokenListContainsValue(r.Header, "Connection", "upgrade") {
        return u.returnError(w, r, http.StatusBadRequest, badHandshake+"'upgrade' token not found in 'Connection' header")
    }

    if !tokenListContainsValue(r.Header, "Upgrade", "websocket") {
        return u.returnError(w, r, http.StatusBadRequest, badHandshake+"'websocket' token not found in 'Upgrade' header")
    }

    if r.Method != "GET" {
        return u.returnError(w, r, http.StatusMethodNotAllowed, badHandshake+"request method is not GET")
    }

    if !tokenListContainsValue(r.Header, "Sec-Websocket-Version", "13") {
        return u.returnError(w, r, http.StatusBadRequest, "websocket: unsupported version: 13 not found in 'Sec-Websocket-Version' header")
    }

The specification of handshake is as follow:

The request MUST contain a Connection header field whose value MUST include the upgrade token.
- A Connection header is often set to keep-alive or close in normal HTTP communication.
The request MUST contain a Upgrade header field whose value MUST include the websocket keyword.
- A Upgrade header is used to upgrade an already established client/server connection to a different protocol.
The method of the request MUST be GET as described on RFC 6455 - page 16
Sec-Websocket-Version is the WebSocket protocol version which is supported by server. Available versions are listed in IANA WebSocket Version Number Registry. RFC 6415 is registered as Version 13, so basically, only 13 should be supported.

Sec-Websocket-Key

There is also gorilla/websocket codes on the way in Upgrade function.

challengeKey := r.Header.Get("Sec-Websocket-Key")
if challengeKey == "" {
    return u.returnError(w, r, http.StatusBadRequest, "websocket: not a websocket handshake: 'Sec-WebSocket-Key' header is missing or blank")
}

Sec-Websocket-Key is used mainly for two purposes according to RFC 6455 and the discussion on stack overflow.

Ensure that server understands WebSocket protocol
Prevent clients accidentally requesting WebSocket upgrade not expecting it

See How decided a value set in Sec-WebSocket-Key/Accept header for more information.

Return HTTP status code 101 (Switching Protocols)

Upgrade function ends with gorilla/websocket codes which write HTTP headers.

p = append(p, "HTTP/1.1 101 Switching Protocols\r\nUpgrade: websocket\r\nConnection: Upgrade\r\nSec-WebSocket-Accept: "...)
p = append(p, computeAcceptKey(challengeKey)...)
p = append(p, "\r\n"...)

The HTTP 101 Switching Protocols response code indicates the protocol the server is switching to as requested by a client. This code includes in this response an Upgrade response header field whose value is websocket.

When the client receives this response, it understands that the communication protocol has been upgraded to WebSocket, and the first step, handshake, is completed.

Conclusion

This article explains WebSocket handshake protocol using gorilla/websocket server implementation.

Additional articles will be published on the WebSocket protocol from the client perspective and data frame processing when exchanging messages and etc.

Deep dive into WebSocket opening handshake protocol with Go
- It explains the WebSocket handshaking from the client perspective.
How WebSocket protocol designs bidirectional messaging and implements in Go