DEV Community: Andy Pan

【gnet】: Support Windows platform and add several codecs for encoding/decoding TCP frames

Andy Pan — Thu, 12 Dec 2019 14:28:01 +0000

Github

https://github.com/panjf2000/gnet

Introduction

gnet is a high-performance(much better performance than the net package in Go), lightweight, non-blocking, event-driven networking framework written in pure Go, it supports Linux, freeBSD and Windows platforms now.
All features:

[x] High-performance event-loop under networking model of multiple threads/goroutines
[x] Built-in load balancing algorithm: Round-Robin
[x] Built-in goroutine pool powered by the library ants
[x] Built-in memory pool with bytes powered by the library pool
[x] Concise APIs
[x] Efficient memory usage: Ring-Buffer
[x] Supporting multiple protocols: TCP, UDP, and Unix Sockets
[x] Supporting two event-driven mechanisms: epoll on Linux and kqueue on FreeBSD
[x] Supporting asynchronous write operation
[x] Flexible ticker event
[x] SO_REUSEPORT socket option
[x] Built-in multiple codecs to encode/decode network frames into/from TCP stream: LineBasedFrameCodec, DelimiterBasedFrameCodec, FixedLengthFrameCodec and LengthFieldBasedFrameCodec, referencing netty codec, also supporting customized codecs
[x] Supporting Windows platform with ~~event-driven mechanism of IOCP~~ Go stdlib: net

Releasing a high-performance goroutine pool in Go

Andy Pan — Wed, 02 Oct 2019 16:18:57 +0000

ants

A goroutine pool for Go

Github Page:
https://github.com/panjf2000/ants

Library ants implements a goroutine pool with fixed capacity, managing and recycling a massive number of goroutines, allowing developers to limit the number of goroutines in your concurrent programs.

Features:

Automatically managing and recycling a massive number of goroutines.
Periodically purging overdue goroutines.
Friendly interfaces: submitting tasks, getting the number of running goroutines, tuning capacity of pool dynamically, closing pool.
Handle panic gracefully to prevent programs from crash.
Efficient in memory usage and it even achieves higher performance than unlimited goroutines in golang.
Nonblocking mechanism.

Tested in the following Golang versions:

1.8.x
1.9.x
1.10.x
1.11.x
1.12.x
1.13.x

How to install

go get -u github.com/panjf2000/ants

How to use

Just take a imagination that your program starts a massive number of goroutines, resulting in a huge consumption of memory. To mitigate that kind of situation, all you need to do is to import ants package and submit all your tasks to a default pool with fixed capacity, activated when package ants is imported:

package main

import (
    "fmt"
    "sync"
    "sync/atomic"
    "time"

    "github.com/panjf2000/ants/v2"
)

var sum int32

func myFunc(i interface{}) {
    n := i.(int32)
    atomic.AddInt32(&sum, n)
    fmt.Printf("run with %d\n", n)
}

func demoFunc() {
    time.Sleep(10 * time.Millisecond)
    fmt.Println("Hello World!")
}

func main() {
    defer ants.Release()

    runTimes := 1000

    // Use the common pool.
    var wg sync.WaitGroup
    syncCalculateSum := func() {
        demoFunc()
        wg.Done()
    }
    for i := 0; i < runTimes; i++ {
        wg.Add(1)
        _ = ants.Submit(syncCalculateSum)
    }
    wg.Wait()
    fmt.Printf("running goroutines: %d\n", ants.Running())
    fmt.Printf("finish all tasks.\n")

    // Use the pool with a function,
    // set 10 to the capacity of goroutine pool and 1 second for expired duration.
    p, _ := ants.NewPoolWithFunc(10, func(i interface{}) {
        myFunc(i)
        wg.Done()
    })
    defer p.Release()
    // Submit tasks one by one.
    for i := 0; i < runTimes; i++ {
        wg.Add(1)
        _ = p.Invoke(int32(i))
    }
    wg.Wait()
    fmt.Printf("running goroutines: %d\n", p.Running())
    fmt.Printf("finish all tasks, result is %d\n", sum)
}

Integrate with http server

package main

import (
    "io/ioutil"
    "net/http"

    "github.com/panjf2000/ants/v2"
)

type Request struct {
    Param  []byte
    Result chan []byte
}

func main() {
  pool, _ := ants.NewPoolWithFunc(100000, func(payload interface{}) {
        request, ok := payload.(*Request)
        if !ok {
            return
        }
        reverseParam := func(s []byte) []byte {
            for i, j := 0, len(s)-1; i < j; i, j = i+1, j-1 {
                s[i], s[j] = s[j], s[i]
            }
            return s
        }(request.Param)

        request.Result <- reverseParam
  })
    defer pool.Release()

    http.HandleFunc("/reverse", func(w http.ResponseWriter, r *http.Request) {
        param, err := ioutil.ReadAll(r.Body)
        if err != nil {
            http.Error(w, "request error", http.StatusInternalServerError)
        }
        defer r.Body.Close()

        request := &Request{Param: param, Result: make(chan []byte)}

        // Throttle the requests traffic with ants pool. This process is asynchronous and
        // you can receive a result from the channel defined outside.
        if err := pool.Invoke(request); err != nil {
            http.Error(w, "throttle limit error", http.StatusInternalServerError)
        }

        w.Write(<-request.Result)
    })

    http.ListenAndServe(":8080", nil)
}

Functional options for ants pool

type Options struct {
    // ExpiryDuration set the expired time (second) of every worker.
    ExpiryDuration time.Duration

    // PreAlloc indicate whether to make memory pre-allocation when initializing Pool.
    PreAlloc bool

    // Max number of goroutine blocking on pool.Submit.
    // 0 (default value) means no such limit.
    MaxBlockingTasks int

    // When Nonblocking is true, Pool.Submit will never be blocked.
    // ErrPoolOverload will be returned when Pool.Submit cannot be done at once.
    // When Nonblocking is true, MaxBlockingTasks is inoperative.
    Nonblocking bool

    // PanicHandler is used to handle panics from each worker goroutine.
    // if nil, panics will be thrown out again from worker goroutines.
    PanicHandler func(interface{})
}

func WithOptions(options Options) Option {
    return func(opts *Options) {
        *opts = options
    }
}

func WithExpiryDuration(expiryDuration time.Duration) Option {
    return func(opts *Options) {
        opts.ExpiryDuration = expiryDuration
    }
}

func WithPreAlloc(preAlloc bool) Option {
    return func(opts *Options) {
        opts.PreAlloc = preAlloc
    }
}

func WithMaxBlockingTasks(maxBlockingTasks int) Option {
    return func(opts *Options) {
        opts.MaxBlockingTasks = maxBlockingTasks
    }
}

func WithNonblocking(nonblocking bool) Option {
    return func(opts *Options) {
        opts.Nonblocking = nonblocking
    }
}

func WithPanicHandler(panicHandler func(interface{})) Option {
    return func(opts *Options) {
        opts.PanicHandler = panicHandler
    }
}

ants.Optionscontains all optional configurations of ants pool, which allows you to customize the goroutine pool by invoking option functions to set up each configuration in NewPool/NewPoolWithFuncmethod.

Customize limited pool

ants also supports customizing the capacity of pool. You can invoke the NewPool method to instantiate a pool with a given capacity, as following:

// Set 10000 the size of goroutine pool
p, _ := ants.NewPool(10000)

Submit tasks

Tasks can be submitted by calling ants.Submit(func())

ants.Submit(func(){})

Tune pool capacity in runtime

You can tune the capacity of ants pool in runtime with Tune(int):

pool.Tune(1000) // Tune its capacity to 1000
pool.Tune(100000) // Tune its capacity to 100000

Don't worry about the synchronous problems in this case, the method here is thread-safe (or should be called goroutine-safe).

Pre-malloc goroutine queue in pool

ants allows you to pre-allocate memory of goroutine queue in pool, which may get a performance enhancement under some special certain circumstances such as the scenario that requires a pool with ultra-large capacity, meanwhile each task in goroutine lasts for a long time, in this case, pre-mallocing will reduce a lot of costs when re-slicing goroutine queue.

// ants will pre-malloc the whole capacity of pool when you invoke this method
p, _ := ants.NewPool(100000, ants.WithPreAlloc(true))

Release Pool

pool.Release()

About sequence

All tasks submitted to ants pool will not be guaranteed to be addressed in order, because those tasks scatter among a series of concurrent workers, thus those tasks would be executed concurrently.

Benchmarks

In this benchmark-picture, the first and second benchmarks performed test cases with 1M tasks and the rest of benchmarks performed test cases with 10M tasks, both in unlimited goroutines and ants pool, and the capacity of this ants goroutine-pool was limited to 50K.

BenchmarkGoroutine-4 represents the benchmarks with unlimited goroutines in golang.
BenchmarkPoolGroutine-4 represents the benchmarks with a ants pool.

Benchmarks with Pool

In above benchmark picture, the first and second benchmarks performed test cases with 1M tasks and the rest of benchmarks performed test cases with 10M tasks, both in unlimited goroutines and ants pool, and the capacity of this ants goroutine-pool was limited to 50K.

As you can see, ants performs 2 times faster than goroutines without pool (10M tasks) and it only consumes half the memory comparing with goroutines without pool. (both in 1M and 10M tasks)

Benchmarks with PoolWithFunc

Throughput (it is suitable for scenarios where tasks are submitted asynchronously without waiting for the final results)

100K tasks

1M tasks

10M tasks

Performance Summary

In conclusion, ants performs 2~6 times faster than goroutines without a pool and the memory consumption is reduced by 10 to 20 times.

License

Source code in gnet is available under the MIT License.

Relevant Articles

Goroutine 并发调度模型深度解析之手撸一个高性能协程池

Users of ants (please feel free to add your projects here ~~)

Releasing a high-performance, lightweight, non-blocking and event-loop networking library written in pure Go

Andy Pan — Mon, 30 Sep 2019 16:40:02 +0000

Github Page:

https://github.com/panjf2000/gnet

gnet is an Event-Loop networking framework that is fast and small. It makes direct epoll and kqueue syscalls rather than using the standard Go net package, and works in a similar manner as libuv and libevent.

The goal of this project is to create a server framework for Go that performs on par with Redis and Haproxy for packet handling.

gnet sells itself as a high-performance, lightweight, non-blocking network library written in pure Go which works on transport layer with TCP/UDP/Unix-Socket protocols, so it allows developers to implement their own protocols of application layer upon gnet for building diversified network applications, for instance, you get a HTTP Server or Web Framework if you implement HTTP protocol upon gnet while you have a Redis Server done with the implementation of Redis protocol upon gnet and so on.

gnet derives from project evio while having higher performance.

Features

High-performance Event-Loop under multi-threads/goroutines model
Built-in load balancing algorithm: Round-Robin
Concise APIs
Efficient memory usage: Ring-Buffer
Supporting multiple protocols: TCP, UDP, and Unix Sockets
Supporting two event-notification mechanisms: epoll in Linux and kqueue in FreeBSD
Supporting asynchronous write operation
Flexible ticker event
SO_REUSEPORT socket option

Key Designs

Multiple-Threads/Goroutines Model

Multiple Reactors Model

gnet redesigns and implements a new built-in multiple-threads/goroutines model: 『Multiple Reactors』 which is also the default multiple-threads model of netty, Here's the schematic diagram:

and it works as the following sequence diagram:

Multiple Reactors + Goroutine-Pool Model

You may ask me a question: what if my business logic in EventHandler.React contains some blocking code which leads to a blocking in event-loop of gnet, what is the solution for this kind of situation？

As you know, there is a most important tenet when writing code under gnet: you should never block the event-loop in the EventHandler.React, otherwise it will lead to a low throughput in your gnet server, which is also the most important tenet in netty.

And the solution for that would be found in the subsequent multiple-threads/goroutines model of gnet: 『Multiple Reactors with thread/goroutine pool』which pulls you out from the blocking mire, it will construct a worker-pool with fixed capacity and put those blocking jobs in EventHandler.React into the worker-pool to unblock the event-loop goroutines.

This new networking model is under development and about to be delivered soon and its architecture diagram of new model is in here:

and it works as the following sequence diagram:

Before you can benefit from this new networking model in handling blocking business logic, there is still a way for you to handle your business logic in networking: you can utilize the open-source goroutine-pool to unblock your blocking code, and I now present you ants: a high-performance goroutine pool in Go that allows you to manage and recycle a massive number of goroutines in your concurrency programs.

You can import ants to your gnet server and put your blocking code to the ants pool in EventHandler.React, which makes your business code non-blocking.

Auto-scaling Ring Buffer

gnet utilizes Ring-Buffer to cache TCP streams and manage memory cache in networking.

Getting Started

Installation



$ go get -u github.com/panjf2000/gnet

Usage

It is easy to create a network server with gnet. All you have to do is just make your implementaion of gnet.EventHandler interface and register your event-handler functions to it, then pass it to the gnet.Serve function along with the binding address(es). Each connection is represented as a gnet.Conn interface that is passed to various events to differentiate the clients. At any point you can close a client or shutdown the server by return a Close or Shutdown action from an event.

The simplest example to get you started playing with gnet would be the echo server. So here you are, a simplest echo server upon gnet that is listening on port 9000:

Echo server without blocking logic



package main

import (
    "log"

    "github.com/panjf2000/gnet"
)

type echoServer struct {
    *gnet.EventServer
}

func (es *echoServer) React(c gnet.Conn) (out []byte, action gnet.Action) {
    top, tail := c.ReadPair()
    out = top
    if tail != nil {
        out = append(top, tail...)
    }
    c.ResetBuffer()
    return
}

func main() {
    echo := new(echoServer)
    log.Fatal(gnet.Serve(echo, "tcp://:9000", gnet.WithMulticore(true)))
}

As you can see, this example of echo server only sets up the EventHandler.React function where you commonly write your main business code and it will be invoked once the server receives input data from a client. The output data will be then sent back to that client by assigning the out variable and return it after your business code finish processing data(in this case, it just echo the data back).

Echo server with blocking logic



package main

import (
    "log"
    "time"

    "github.com/panjf2000/ants/v2"
    "github.com/panjf2000/gnet"
)

type echoServer struct {
    *gnet.EventServer
    pool *ants.Pool
}

func (es *echoServer) React(c gnet.Conn) (out []byte, action gnet.Action) {
    data := c.ReadBytes()
    c.ResetBuffer()

    // Use ants pool to unblock the event-loop.
    _ = es.pool.Submit(func() {
        time.Sleep(1 * time.Second)
        c.AsyncWrite(data)
    })

    action = gnet.DataRead
    return
}

func main() {
    // Create a goroutine pool.
    poolSize := 64 * 1024
    pool, _ := ants.NewPool(poolSize, ants.WithNonblocking(true))
    defer pool.Release()
    echo := &echoServer{pool: pool}
    log.Fatal(gnet.Serve(echo, "tcp://:9000", gnet.WithMulticore(true)))
}

Like I said in the 『Multiple Reactors + Goroutine-Pool Model』section, if there are blocking code in your business logic, then you ought to turn them into non-blocking code in any way, for instance you can wrap them into a goroutine, but it will result in a massive amount of goroutines if massive traffic is passing through your server so I would suggest you utilize a goroutine pool like ants to manage those goroutines and reduce the cost of system resource.

I/O Events

Current supported I/O events in gnet:

EventHandler.OnInitComplete is activated when the server is ready to accept new connections.
EventHandler.OnOpened is activated when a connection has opened.
EventHandler.OnClosed is activated when a connection has closed.
EventHandler.React is activated when the server receives new data from a connection.
EventHandler.Tick is activated immediately after the server starts and will fire again after a specified interval.
EventHandler.PreWrite is activated just before any data is written to any client socket.

Ticker

The EventHandler.Tick event fires ticks at a specified interval.
The first tick fires immediately after the Serving events and if you intend to set up a ticker event, remember to pass a option: gnet.WithTicker(true) to gnet.Serve.



events.Tick = func() (delay time.Duration, action Action){

    log.Printf("tick")

    delay = time.Second

    return

}

UDP

The gnet.Serve function can bind to UDP addresses.

All incoming and outgoing packets will not be buffered but sent individually.
The EventHandler.OnOpened and EventHandler.OnClosed events are not available for UDP sockets, only the React event.

Multi-threads

The gnet.WithMulticore(true) indicates whether the server will be effectively created with multi-cores, if so, then you must take care with synchronizing memory between all event callbacks, otherwise, it will run the server with single thread. The number of threads in the server will be automatically assigned to the value of runtime.NumCPU().

Load balancing

The current built-in load balancing algorithm in gnet is Round-Robin.

SO_REUSEPORT

Servers can utilize the SO_REUSEPORT option which allows multiple sockets on the same host to bind to the same port and the OS kernel takes care of the load balancing for you, it wakes one socket per accpet event coming to resolved the thundering herd.

Just use functional options to set up SO_REUSEPORT and you can enjoy this feature:



gnet.Serve(events, "tcp://:9000", gnet.WithMulticore(true)))

Performance

Contrasts to the similar networking libraries

On Linux (epoll)

Test Environment



# Machine information

        OS : Ubuntu 18.04/x86_64

       CPU : 8 Virtual CPUs

    Memory : 16.0 GiB

# Go version and configurations

Go Version : go1.12.9 linux/amd64

GOMAXPROCS=8

Echo Server

HTTP Server

On FreeBSD (kqueue)

Test Environment



# Machine information

        OS : macOS Mojave 10.14.6/x86_64

       CPU : 4 CPUs

    Memory : 8.0 GiB

# Go version and configurations

Go Version : go version go1.12.9 darwin/amd64

GOMAXPROCS=4

Echo Server

HTTP Server

License

Source code in gnet is available under the MIT License.

DEV Community: Andy Pan

【gnet】: Support Windows platform and add several codecs for encoding/decoding TCP frames

Github

Introduction

Releasing a high-performance goroutine pool in Go

ants

Features:

Tested in the following Golang versions:

How to install

How to use

Integrate with http server

Functional options for ants pool

Customize limited pool

Submit tasks

Tune pool capacity in runtime

Pre-malloc goroutine queue in pool

Release Pool

About sequence

Benchmarks

Benchmarks with Pool

Benchmarks with PoolWithFunc

Throughput (it is suitable for scenarios where tasks are submitted asynchronously without waiting for the final results)

100K tasks

1M tasks

10M tasks

Performance Summary

License

Relevant Articles

Users of ants (please feel free to add your projects here ~~)

Releasing a high-performance, lightweight, non-blocking and event-loop networking library written in pure Go

Github Page:

Features

Key Designs

Multiple-Threads/Goroutines Model

Multiple Reactors Model

Multiple Reactors + Goroutine-Pool Model

Auto-scaling Ring Buffer

Getting Started

Installation

Usage

Echo server without blocking logic

Echo server with blocking logic

I/O Events

Ticker

UDP

Multi-threads

Load balancing

SO_REUSEPORT

Performance

Contrasts to the similar networking libraries

On Linux (epoll)

Test Environment

Echo Server

HTTP Server

On FreeBSD (kqueue)

Test Environment

Echo Server

HTTP Server

License

Thanks

Relevant Articles