DEV Community: Javad Rajabzadeh

🚀 otp package (high performance TOTP and HOTP generator)

Javad Rajabzadeh — Thu, 03 Apr 2025 18:10:13 +0000

If you're looking to implement two-factor authentication (2FA) in your Go applications, the otp offers a high-performance, zero-dependency solution for generating and validating TOTP and HOTP one-time passwords, fully compliant with RFC 4226 and RFC 6238.

Binding nodejs is available here.

Key Features

Zero Dependencies – Fully self-contained, no third-party libraries.
High Performance – Optimized for speed and minimal memory usage.
Support for TOTP & HOTP – RFC-compliant implementation.
Custom Digits & Algorithms – Use 6, 8, or 10 digits with SHA1, SHA256, or SHA512.
Secure – Constant-time validation to prevent timing attacks.
Clock Skew Tolerance – Allows minor time drift in TOTP validation.
Authenticator App Compatible – Supports otpauth:// URL generation for Google Authenticator and others.
Secure Secret Generator – Cryptographically secure random base32 secrets.
Well Tested – Passes all RFC test vectors, includes fuzzing and benchmarks.

Installation

go get -u github.com/Ja7ad/otp

Benchmarks

Compared to popular packages like pquerna/otp, this package:

Is ~2x faster

Uses 30–50% less memory

Has zero allocations in some core paths

Usage Example

package main

import (
    "fmt"
    "github.com/Ja7ad/otp"
    "log"
    "time"
)

func main() {
    secret, err := otp.RandomSecret(otp.SHA1)
    if err != nil {
        log.Fatal(err)
    }

    t := time.Now()

    code, err := otp.GenerateTOTP(secret, t, otp.DefaultTOTPParam)
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println("Generated TOTP code:", code)

    valid, err := otp.ValidateTOTP(secret, code, t, otp.DefaultTOTPParam)
    if err != nil {
        log.Fatal(err)
    }

    if valid {
        fmt.Println("The TOTP code is valid.")
    } else {
        fmt.Println("The TOTP code is invalid.")
    }

    url, err := otp.GenerateTOTPURL(otp.URLParam{
        Issuer:      "https://example.com",
        Secret:      secret,
        AccountName: "foobar",
        Period:      otp.DefaultTOTPParam.Period,
        Digits:      otp.DefaultTOTPParam.Digits,
        Algorithm:   otp.DefaultTOTPParam.Algorithm,
    })
    if err != nil {
        log.Fatal(err)
    }

    fmt.Println("TOTP URL:", url.String())
}

Conclusion

The otp package is a robust, fast, and easy-to-integrate solution for adding 2FA to your Go applications. Whether you're building a secure login flow or a developer tool, this package delivers production-grade OTP generation and validation without the bloat.

GRPC captcha validator middleware

Javad Rajabzadeh — Tue, 13 Jun 2023 04:27:03 +0000

The GRPC Captcha Validation Middleware is a Go module that provides a reusable middleware component for validating captcha challenges on GRPC servers. The middleware intercepts incoming requests, verifies the captcha challenge for the request using a configurable provider, and rejects the request if the challenge is not valid.

The module is designed to be configurable, with options to specify the captcha provider, challenge validation endpoint, and custom headers for the validation request. It uses the Google Protobuf library to parse the GRPC method descriptor and extract custom options defined using protobuf extensions.

Feature

support Google, Cloudflare, hcaptcha provider to verify challenge
support unary and stream server middleware
captcha validation for specific rpc methods

Install (Go >= 1.18)

$ go get -u github.com/GoFarsi/grpc-captcha-validation

Document: https://pkg.go.dev/github.com/GoFarsi/grpc-captcha-validation

Example

example captcha for GRPC server:

syntax = "proto3";
package example;
option go_package = "github.com/GoFarsi/grpc-captcha-validation/_example/proto";

import "captcha.proto";

service GreetingService {
  rpc Greeting(GreetingRequest) returns(GreetingResponse) {
    option(captcha.captcha) = {
      check_challenge: true,
      provider: GOOGLE,
    };
  }

  rpc GreetingStream(stream GreetingRequest) returns(stream GreetingResponse) {
    option(captcha.captcha) = {
      check_challenge: true,
      provider: GOOGLE,
    };
  }
}

message GreetingRequest {
  string name = 1;
}

message GreetingResponse {
  string message = 1;
}

package main

import (
    "context"
    "fmt"
    gcv "github.com/GoFarsi/grpc-captcha-validation"
    "github.com/GoFarsi/grpc-captcha-validation/_example/proto"
    "google.golang.org/grpc"
    "google.golang.org/grpc/reflection"
    "io"
    "log"
    "net"
)

type GreetingService struct {
    proto.UnimplementedGreetingServiceServer
}

func (g *GreetingService) Greeting(ctx context.Context, in *proto.GreetingRequest) (*proto.GreetingResponse, error) {
    return &proto.GreetingResponse{
        Message: fmt.Sprintf("greeting %s", in.Name),
    }, nil
}

func (g *GreetingService) GreetingStream(srv proto.GreetingService_GreetingStreamServer) error {
    ctx := srv.Context()
    for {
        select {
        case <-ctx.Done():
            return ctx.Err()
        default:
        }

        req, err := srv.Recv()
        if err == io.EOF {
            log.Println("exit")
            return nil
        }
        if err != nil {
            log.Printf("receive error %v", err)
            continue
        }

        resp := proto.GreetingResponse{Message: fmt.Sprintf("Greeting %s", req.Name)}
        if err := srv.Send(&resp); err != nil {
            log.Printf("send error %v", err)
        }
    }
}

func main() {
    listener, err := net.Listen("tcp", ":8080")
    if err != nil {
        log.Fatalln(err)
    }

    c := gcv.NewCaptcha("foo", "", "", "")

    srv := grpc.NewServer(
        grpc.ChainUnaryInterceptor(c.UnaryServerInterceptor()),
        grpc.ChainStreamInterceptor(c.StreamServerInterceptor()),
    )

    reflection.Register(srv)

    proto.RegisterGreetingServiceServer(srv, &GreetingService{})

    log.Fatalln(srv.Serve(listener))
}

Use new heap memory percentage strategy to control automatic GC frequency

Javad Rajabzadeh — Thu, 30 Mar 2023 07:21:13 +0000

The value parts which contain pointers and are known clearly alive by Go runtime are called (scannable) root value parts (call roots below). Value parts allocated on stack and direct parts of global (package-level) variables are the main sources of roots. To simplify the implementation, the official Go runtime views all value parts allocated on the stack as roots, including value parts without pointers. Root memory blocks host roots. For the official standard Go runtime, before version 1.18, roots had no impact on the new heap memory percentage strategy; since version 1.18, they have. The new heap memory percentage strategy is configured through a GOGC value, which may be set via the GOGC environment variable or modified by calling the runtime/debug.SetGCPercent function. The default value of the GOGC value is 100. We may set the GOGC environment variable as off or call the runtime/debug.SetGCPercent function with a negative argument to disable the new heap memory percentage strategy. Note: doing these will also disable the two-minute auto GC strategy. To disable the new heap memory percentage strategy solely, we may set the GOGC value as math.MaxInt64. If the new heap memory percentage strategy is enabled (the GOGC value is non-negative), when the scan+mark phase of a GC cycle is just done, the official standard Go runtime (version 1.18+) will calculate the target heap size for the next garbage collection cycle, from the non-garbage heap memory total size (call live heap here) and the root memory block total size (called GC roots here).

according to the following formula:

Target heap size = Live heap + (Live heap + GC roots) * GOGC / 100

For versions before 1.18, the formula is:

Target heap size = Live heap + (Live heap) * GOGC / 100

If the heap memory total size (approximately) exceeds the calculated target heap size, the next GC cycle will start automatically.

Note: the minimum target heap size is (GOGC * 4 / 100)MB, which is also the target heap size for the first GC cycle.

We could use the GODEBUG=gctrace=1 environment variable option to output a summary log line for each GC cycle. Each GC cycle summary log line is formatted like the following text (in which each # presents a number and ... means ignored messages by the current article).

gc # @#s #%: ..., #->#-># MB, # MB goal # MB stacks # MB globals ...

Note: in the following outputs of examples, to keep the each output line short, not all of these fields
will be shown.

Let’s use an unreal program as an example to show how to use this option:

package main

import (
    "math/rand"
    "time"
)

var x [512][]*int

func garbageProducer() {
    rand.Seed(time.Now().UnixNano())
    for i := 0; ; i++ {
        n := 6 + rand.Intn(6)
        for j := range x {
            x[j] = make([]*int, 1<<n)
            for k := range x[j] {
                x[j][k] = new(int)
            }
        }
        time.Sleep(time.Second / 1000)
    }
}
func main() {
    garbageProducer()
}

$ GODEBUG=gctrace=1 go run main.go
gc 1 @0.018s 0%: 0.015+0.37+0.023 ms clock, 0.060+0.12/0.25/0.22+0.093 ms cpu, 3->3->0 MB, 4 MB goal, 0 MB stacks, 0 MB globals, 4 P
gc 2 @0.039s 1%: 0.23+0.74+0.003 ms clock, 0.92+0.57/0.26/0.025+0.015 ms cpu, 3->3->0 MB, 4 MB goal, 0 MB stacks, 0 MB globals, 4 P
gc 3 @0.044s 1%: 0.022+0.42+0.025 ms clock, 0.091+0.30/0.30/0.37+0.10 ms cpu, 3->3->0 MB, 4 MB goal, 0 MB stacks, 0 MB globals, 4 P
gc 4 @0.052s 1%: 0.048+0.67+0.024 ms clock, 0.19+0.13/0.42/0.24+0.099 ms cpu, 3->4->0 MB, 4 MB goal, 0 MB stacks, 0 MB globals, 4 P
# command-line-arguments
gc 1 @0.000s 15%: 0.062+0.94+0.002 ms clock, 0.25+0.074/0.86/0.10+0.011 ms cpu, 4->6->5 MB, 4 MB goal, 0 MB stacks, 0 MB globals, 4 P
gc 2 @0.023s 2%: 0.015+2.0+0.017 ms clock, 0.063+0.057/1.6/0.53+0.069 ms cpu, 10->11->7 MB, 12 MB goal, 0 MB stacks, 0 MB globals, 4 P

Here, the #->#-># MB and # MB goal fields are what we have most interests in. In a #->#-># MB field:

the last number is non-garbage heap memory total size (a.k.a. live heap).
the first number is the heap size at a GC cycle start, which should be approximately equal to the target heap size (the number in the # MB goal field of the same line). From the outputs, we could find that
the live heap sizes stagger much (yes, the above program is deliberately designed as such), so the GC cycle intervals also stagger much.
the GC cycle frequency is so high that the percentage of time spent on GC is too high. The above outputs show the percentage varies from 8% to 12%.

The reason of the findings is that the live heap size is small (staying roughly under 5MiB) and
staggers much, but the root memory block total size is almost zero.

One way to reduce the time spent on GC is to increase the GOGC value:

$ GOGC=1000 GODEBUG=gctrace=1 go run main.go
gc 1 @0.074s 2%: ..., 38->43->15 MB, 40 MB goal, ...
gc 2 @0.810s 1%: ..., 160->163->9 MB, 167 MB goal, ...
gc 3 @1.285s 1%: ..., 105->107->11 MB, 109 MB goal, ...
gc 4 @1.835s 1%: ..., 125->128->10 MB, 129 MB goal, ...
gc 5 @2.331s 1%: ..., 114->117->18 MB, 118 MB goal, ...
gc 6 @3.250s 1%: ..., 199->201->8 MB, 204 MB goal, ...
gc 7 @3.703s 1%: ..., 96->98->18 MB, 100 MB goal, ...
gc 8 @4.580s 1%: ..., 201->204->10 MB, 207 MB goal, ...
gc 9 @5.111s 1%: ..., 118->119->3 MB, 122 MB goal, ...
gc 10 @5.306s 1%: ..., 43->43->4 MB, 44 MB goal, ....

From the above outputs, we could find that, after increase the GOGC value to 1000, GC cycle intervals
and the heap sizes at GC cycle beginning both become much larger now. But GC cycle intervals
and live heap sizes still stagger much, which might be a problem for some programs. The following
sections will introduce some ways to solve the problems.

🚀 Package Zapper (write logs but easier)

Javad Rajabzadeh — Wed, 22 Mar 2023 17:35:13 +0000

Wrapping the zap log uber in the zapper package makes it easy to use zap to write logs to varianty core.

feature

available cores (console, file, sentry, json)
show stacktrace in log
log file rotation using lumberjack
create multiple core in same time
add service details to zapper
compatibility with ELK using json core
compatibility with some software architecture scenario
100% compatibility with sentry using sentry core

package main

import (
    "github.com/GoFarsi/zapper"
    "log"
)

func main() {
    z := zapper.New(false, zapper.WithDebugLevel())
    if err := z.NewCore(
        zapper.ConsoleWriterCore(true),
        zapper.FileWriterCore("./test_data", nil),
    ); err != nil {
        log.Fatal(err)
    }

    z.Debug("debug log")
    z.Info("info log")
    z.Warn("warn log")
    z.Error("error log")
}

Repo: https://github.com/GoFarsi/zapper
Document: https://pkg.go.dev/github.com/GoFarsi/zapper

GRPC over unix socket Protocol

Javad Rajabzadeh — Wed, 07 Dec 2022 08:36:10 +0000

ja7ad / grpc-unix-socket

GRPC over unix socket Protocol

Unix sockets, or Unix Domain Sockets, allow bidirectional data exchange between processes running on the same machine In order to transmit data between processes, the file system provides a reliable and efficient mechanism. The kernel is the only component involved in communication between processes. The processes communicate by reading and writing to the same socket file, which is managed by the kernel. Kernels handle communication details, such as synchronization, buffering, and error handling, and ensure that data is delivered reliably and correctly.

Advantage :

Fast communication
Efficient
Low overhead
Stability

Disadvantages :

Limited on same machine

sequenceDiagram

Client->>+Server: Health Request (1.02187 µs per response)

Server->>+Client: Serve message

$ netstat -a -p --unix | grep grpc
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
unix  2      [ ACC ]     STREAM

…

View on GitHub

Unix sockets, or Unix Domain Sockets, allow bidirectional data exchange between processes running on the same machine.
In order to transmit data between processes, the file system provides a reliable and efficient mechanism. The kernel is the only component involved in communication between processes. The processes communicate by reading and writing to the same socket file, which is managed by the kernel. Kernels handle communication details, such as synchronization, buffering, and error handling, and ensure that data is delivered reliably and correctly.

Advantage :

Fast communication
Efficient
Low overhead
Stability

Disadvantages :

Limited on same machine

$ netstat -a -p --unix | grep grpc

(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
unix  2      [ ACC ]     STREAM     LISTENING     119977   98036/main           /tmp/grpc.sock

A TCP/IP socket is a mechanism for communicating between processes over a network.

package main

import (
    "google.golang.org/grpc"
    "google.golang.org/grpc/health"
    "google.golang.org/grpc/health/grpc_health_v1"
    "google.golang.org/grpc/reflection"
    "log"
    "net"
    "os"
    "os/signal"
    "syscall"
)

const (
    // unix socket
    PROTOCOL = "unix"
    SOCKET   = "/tmp/grpc.sock"

    // tcp protocol
    PROTOCOL_TCP = "tcp"
    ADDR         = "localhost:3000"
)

func main() {
    ln, err := net.Listen(PROTOCOL, SOCKET)
    if err != nil {
        log.Fatal(err)
    }

    tcpLn, err := net.Listen(PROTOCOL_TCP, ADDR)
    if err != nil {
        log.Fatal(err)
    }

    c := make(chan os.Signal, 1)
    signal.Notify(c, os.Interrupt, syscall.SIGTERM)
    go func() {
        <-c
        os.Remove(SOCKET)
        os.Exit(1)
    }()

    srv := grpc.NewServer()
    grpc_health_v1.RegisterHealthServer(srv, health.NewServer())
    reflection.Register(srv)

    go func() {
        log.Printf("grpc ran on tcp protocol %s", ADDR)
        log.Fatal(srv.Serve(tcpLn))
    }()

    log.Printf("grpc ran on unix socket protocol %s", SOCKET)
    log.Fatal(srv.Serve(ln))
}

Test Benchmark and profiling

package main

import (
    "context"
    "google.golang.org/grpc"
    "google.golang.org/grpc/health/grpc_health_v1"
    "net"
    "os"
    "runtime/pprof"
    "testing"
    "time"
)

func Benchmark_UNIX(b *testing.B) {
    f, err := os.Create("unix.prof")
    if err != nil {
        b.Fatal(err)
    }

    if err := pprof.StartCPUProfile(f); err != nil {
        b.Fatal(err)
    }

    defer pprof.StopCPUProfile()

    dialer := func(addr string, t time.Duration) (net.Conn, error) {
        return net.Dial(PROTOCOL, addr)
    }

    conn, err := grpc.Dial(SOCKET, grpc.WithInsecure(), grpc.WithDialer(dialer))
    if err != nil {
        b.Fatal(err)
    }

    health := grpc_health_v1.NewHealthClient(conn)
    ctx := context.Background()

    for i := 0; i < b.N; i++ {
        _, err := health.Check(ctx, &grpc_health_v1.HealthCheckRequest{})
        if err != nil {
            b.Fatal(err)
        }
    }
}

func Benchmark_TCP(b *testing.B) {
    f, err := os.Create("tcp.prof")
    if err != nil {
        b.Fatal(err)
    }

    if err := pprof.StartCPUProfile(f); err != nil {
        b.Fatal(err)
    }

    defer pprof.StopCPUProfile()

    conn, err := grpc.Dial(ADDR, grpc.WithInsecure())
    if err != nil {
        b.Fatal(err)
    }

    health := grpc_health_v1.NewHealthClient(conn)
    ctx := context.Background()

    for i := 0; i < b.N; i++ {
        _, err := health.Check(ctx, &grpc_health_v1.HealthCheckRequest{})
        if err != nil {
            b.Fatal(err)
        }
    }
}

Profile result over unix socket 100k request (Profiling Visualization Image):

goos: linux
goarch: amd64
pkg: github.com/Ja7ad/grpc-unix-socket/server
cpu: Intel(R) Core(TM) i5-3570 CPU @ 3.40GHz
Benchmark_UNIX
Benchmark_UNIX-4          100000            102187 ns/op            4960 B/op         96 allocs/op

Profile result over TCP 100k request (Profiling Visualization Image):

goos: linux
goarch: amd64
pkg: github.com/Ja7ad/grpc-unix-socket/server
cpu: Intel(R) Core(TM) i5-3570 CPU @ 3.40GHz
Benchmark_TCP
Benchmark_TCP-4           100000            127188 ns/op            4961 B/op         96 allocs/op

Result of 100k benchmark with 20 time test as new/old (benchstat)

name     old time/op    new time/op    delta
_UNIX-4     100µs ± 2%     100µs ± 2%    ~     (p=0.668 n=17+20)
_TCP-4      125µs ± 1%     124µs ± 1%  -0.15%  (p=0.031 n=20+20)

name     old alloc/op   new alloc/op   delta
_UNIX-4    4.96kB ± 0%    4.96kB ± 0%    ~     (p=0.454 n=20+20)
_TCP-4     4.96kB ± 0%    4.96kB ± 0%    ~     (all equal)

name     old allocs/op  new allocs/op  delta
_UNIX-4      96.0 ± 0%      96.0 ± 0%    ~     (all equal)
_TCP-4       96.0 ± 0%      96.0 ± 0%    ~     (all equal)

UseCase

When you run many services on the same machine and run them in containers or hosts, you can use unix sockets for efficient and fast communication.

Containers need to share sock files via volumes, since unix sockets are over socket files.

docker volume create --name=socket
docker run socket:/var/socket serviceA
docker run socket:/var/socket serviceB

How to test server/client?

Server :

run server (go1.19) :

$ go run -mod vendor grpc.go

benchmark command :

$ go test -bench=. -benchtime=100000x -benchmem

Go high performance http server with Forker

Javad Rajabzadeh — Tue, 11 Oct 2022 19:03:47 +0000

GoFarsi / forker

Get high performance http requests with fork process

forker

Get high performance http requests with fork process (forker inspired prefork feature in web servers)

prefork is a solution which is also used by other webservers

A server instance is opened for each processor core and incoming requests are shared between these instances

In order to distribute the load more evenly and handle more requests per second, there is a master that starts by the user, which then starts child processes on the other processor cores based on configuration

memory between the different processes is not shared, beacuse goroutines are independent processes

how to install

go get -u github.com/Ja7ad/forker

forker benchmark

forker tested 500 concurrent with 200k http requests

oha -c 500 -n 200000 --latency-correction --disable-keepalive http://localhost:8080

Summary
  Success rate: 1.0000
  Total:    24.4907 secs
  Slowest:  0.5000 secs
  Fastest:  0.0004 secs
  Average:  0.0610 secs
  Requests/sec: 8166.3762

  Total data:   2.10 MiB
  Size/request: 11 B
  Size/sec: 87.72 KiB

Response time histogram:
  0.000

…

View on GitHub

Get high performance http requests with fork process (forker inspired prefork feature in web servers)

prefork is a solution which is also used by other webservers

A server instance is opened for each processor core and incoming requests are shared between these instances

memory between the different processes is not shared, beacuse goroutines are independent processes

Benchmark With oha 200k request with 500 concurrent

Summary:
  Success rate: 1.0000
  Total:    24.4907 secs
  Slowest:  0.5000 secs
  Fastest:  0.0004 secs
  Average:  0.0610 secs
  Requests/sec: 8166.3762

  Total data:   2.10 MiB
  Size/request: 11 B
  Size/sec: 87.72 KiB

Response time histogram:
  0.000 [1]     |
  0.050 [82408] |■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.100 [98705] |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
  0.150 [16180] |■■■■■
  0.200 [2245]  |
  0.250 [361]   |
  0.300 [59]    |
  0.350 [33]    |
  0.400 [3]     |
  0.450 [1]     |
  0.500 [4]     |

Latency distribution:
  10% in 0.0291 secs
  25% in 0.0404 secs
  50% in 0.0557 secs
  75% in 0.0754 secs
  90% in 0.0989 secs
  95% in 0.1170 secs
  99% in 0.1581 secs

Details (average, fastest, slowest):
  DNS+dialup:   0.0259 secs, 0.0001 secs, 0.3010 secs
  DNS-lookup:   0.0000 secs, 0.0000 secs, 0.0239 secs

How to Install?

go get -u github.com/Ja7ad/forker

Example

package main

import (
    "github.com/Ja7ad/forker"
    "log"
    "net/http"
)

func main() {
    srv := &http.Server{
        Handler: GreetingHandler(),
    }

    f := forker.New(srv)

    log.Fatalln(f.ListenAndServe(":8080"))

}

func GreetingHandler() http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        w.Write([]byte("greeting!!!"))
    }
}

Merge files in a path with goMerge

Javad Rajabzadeh — Tue, 02 Nov 2021 10:03:09 +0000

goMerge

Merge some files in a path with custom extension in golang

ja7ad / goMerge

Merge some files in a path with custom extension in golang

goMerge

Merge some files in a path with custom extension in golang

Feature

Walk in a path with a goroutine using parallelism
Merge files with custom extension
Remove unmerged files with remove flag

Install goMerge

go get -u github.com/Ja7ad/goMerge

Example

package main

import (
    fileMerger "github.com/Ja7ad/goMerge"
    "log"
)

func main() {
    err := fileMerger.Merge("./test", ".log", "./test/merged.txt", true)
    if err != nil {
        log.Fatal(err)
    }
}

View on GitHub

Feature

Walk in a path with a goroutine using parallelism
Merge files with custom extension
Remove unmerged files with remove flag

Install goMerge

go get -u github.com/Ja7ad/goMerge

Example

package main

import (
    fileMerger "github.com/Ja7ad/goMerge"
    "log"
)

func main() {
    err := fileMerger.Merge("./test", ".log", "./test/merged.txt", true)
    if err != nil {
        log.Fatal(err)
    }
}

Go Power for vscode

Javad Rajabzadeh — Wed, 30 Jun 2021 02:53:12 +0000

Go Power is an extension for GoLang, supporting more Go snippets and popular Go packages.

Version 1.0.0

Main snippets for go
Fiber package snippets

Download Extension for vscode

Project for contributing

Algorithms for Go

Javad Rajabzadeh — Tue, 22 Jun 2021 12:35:28 +0000

Binary search

package main

import (
    "fmt"
    "time"
)

//works when the array is sorted
func binarySearch(arr []int, x int) int {
    i := 0
    j := len(arr)
    for i != j {
        var m = (i + j) / 2
        if x == arr[m] {
            return m
        }
        if x < arr[m] {
            j = m
        } else {
            i = m + 1
        }
    }
    return -1
}

func main() {
    items := []int{2, 3, 5, 7, 11, 13, 17}

    fmt.Println(binarySearch(items, 1))
    //print -1
    fmt.Println(binarySearch(items, 7))
    //print 3
    fmt.Println(binarySearch(items, 19))
    //print -1

    // *** simplified speed test ***

    items = make([]int, 10000000)
    for i := 0; i < len(items); i++ {
        items[i] = i
    }
    count := 100

    start := time.Now()

    for i := 0; i < count; i++ {
        binarySearch(items, 7777777)
    }

    delta := time.Now().Sub(start)
    nanoseconds := delta.Nanoseconds() / int64(count)

    fmt.Println(nanoseconds)
    // about 88 nanoseconds
}

Fast linear search

package main

import (
    "fmt"
    "time"
)

func fastLinearSearch(arr []int, x int) int {
    i := 0
    count := len(arr)
    arr = append(arr, x)
    for true {
        if arr[i] == x {
            if i < count {
                return i
            }
            return -1
        }
        i++
    }
    return -1
}

func main() {
    items := []int{2, 3, 5, 7, 11, 13, 17}

    fmt.Println(fastLinearSearch(items, 1))
    //print -1
    fmt.Println(fastLinearSearch(items, 7))
    //print 3
    fmt.Println(fastLinearSearch(items, 19))
    //print -1

    // *** simplified speed test ***

    items = make([]int, 10000000)
    for i := 0; i < len(items); i++ {
        items[i] = i
    }
    count := 100

    start := time.Now()

    for i := 0; i < count; i++ {
        fastLinearSearch(items, 7777777)
    }

    delta := time.Now().Sub(start)
    nanoseconds := delta.Nanoseconds() / int64(count)

    fmt.Println(nanoseconds)
    // about 34 148 000 nanoseconds
}

Interpolation search

package main

import (
    "fmt"
    "time"
)

//works when the array is sorted
func interpolationSearch(arr []int, x int) int {
    low := 0
    high := len(arr) - 1

    for (arr[low] < x) && (x < arr[high]) {
        var mid = low + ((x-arr[low])*(high-low))/(arr[high]-arr[low])

        if arr[mid] < x {
            low = mid + 1
        } else if arr[mid] > x {
            high = mid - 1
        } else {
            return mid
        }
    }

    if arr[low] == x {
        return low
    }
    if arr[high] == x {
        return high
    }
    return -1
}

func main() {
    items := []int{2, 3, 5, 7, 11, 13, 17}

    fmt.Println(interpolationSearch(items, 1))
    //print -1
    fmt.Println(interpolationSearch(items, 7))
    //print 3
    fmt.Println(interpolationSearch(items, 19))
    //print -1

    // *** simplified speed test ***

    items = make([]int, 10000000)
    for i := 0; i < len(items); i++ {
        items[i] = i
    }
    count := 100

    start := time.Now()

    for i := 0; i < count; i++ {
        interpolationSearch(items, 7777777)
    }

    delta := time.Now().Sub(start)
    nanoseconds := delta.Nanoseconds() / int64(count)

    fmt.Println(nanoseconds)
    // about 41 nanoseconds
}

Linear search

package main

import (
    "fmt"
    "time"
)

func linearSearch(arr []int, x int) int {
    i := 0
    count := len(arr)
    for i < count {
        if arr[i] == x {
            return i
        }
        i++
    }
    return -1
}

func main() {
    items := []int{2, 3, 5, 7, 11, 13, 17}

    fmt.Println(linearSearch(items, 1))
    //print -1
    fmt.Println(linearSearch(items, 7))
    //print 3
    fmt.Println(linearSearch(items, 19))
    //print -1

    // *** simplified speed test ***

    items = make([]int, 10000000)
    for i := 0; i < len(items); i++ {
        items[i] = i
    }
    count := 100

    start := time.Now()

    for i := 0; i < count; i++ {
        linearSearch(items, 7777777)
    }

    delta := time.Now().Sub(start)
    nanoseconds := delta.Nanoseconds() / int64(count)

    fmt.Println(nanoseconds)
    // about 20 910 000 nanoseconds
}

Bubble sort

package main

import (
    "fmt"
    "time"
)

// Time Complexity O(n^2)
// Space Complexity O(1)

func bubbleSort(arr []int) []int {
    length := len(arr)
    items := make([]int, length)
    copy(items, arr)
    for i := 0; i < length; i++ {
        for j := i + 1; j < length; j++ {
            if items[j] < items[i] {
                var tmp = items[j]
                items[j] = items[i]
                items[i] = tmp
            }
        }
    }
    return items
}

func main() {
    items := []int{4, 1, 5, 3, 2}

    sortItems := bubbleSort(items)
    // sortItems is {1, 2, 3, 4, 5}

    // *** simplified speed test ***

    items = make([]int, 200)
    for i := 0; i < len(items); i++ {
        items[i] = i
    }
    var tmp = items[5]
    items[5] = items[6]
    items[6] = tmp
    count := 10000
    start := time.Now()

    for i := 0; i < count; i++ {
        bubbleSort(items)
    }

    delta := time.Now().Sub(start)
    nanoseconds := delta.Nanoseconds()

    fmt.Println(sortItems)
    fmt.Println(nanoseconds)
    // about 683 000 000 nanoseconds
}

Counting sort

package main

import (
    "fmt"
    "math"
    "time"
)

// Time Complexity O(n+k)
// Space Complexity O(k)

func countingSort(arr []int) []int {
    length := len(arr)
    items := make([]int, length)
    copy(items, arr)

    var min = math.MaxInt32
    var max = math.MinInt32
    for _, x := range arr {
        if x > max {
            max = x
        }
        if x < min {
            min = x
        }
    }

    var counts = make([]int, max-min+1)

    for _, x := range arr {
        counts[x-min]++
    }

    var total = 0
    for i := min; i <= max; i++ {
        var oldCount = counts[i-min]
        counts[i-min] = total
        total += oldCount
    }

    for _, x := range arr {
        items[counts[x-min]] = x
        counts[x-min]++
    }
    return items
}

func main() {
    items := []int{4, 1, 5, 3, 2}

    sortItems := countingSort(items)
    // sortItems is {1, 2, 3, 4, 5}

    // *** simplified speed test ***

    items = make([]int, 200)
    for i := 0; i < len(items); i++ {
        items[i] = i
    }
    var tmp = items[5]
    items[5] = items[6]
    items[6] = tmp
    count := 10000
    start := time.Now()

    for i := 0; i < count; i++ {
        countingSort(items)
    }

    delta := time.Now().Sub(start)
    nanoseconds := delta.Nanoseconds()

    fmt.Println(sortItems)
    fmt.Println(nanoseconds)
    // about 58 000 000 nanoseconds
}

Merge sort

package main

import (
    "fmt"
    "time"
)

// Time Complexity O(n log(n)))
// Space Complexity O(n)

func mergeSort(arr []int) []int {
    items := make([]int, len(arr))
    copy(items, arr)
    doMergeSort(items)
    return items
}

func doMergeSort(items []int) {
    length := len(items)
    if length == 1 {
        return
    }

    var lLeft = length / 2
    var left = make([]int, lLeft)
    copy(left, items[:lLeft])
    var lRight = length - lLeft
    var right = make([]int, lRight)
    copy(right, items[lLeft:])

    doMergeSort(left)
    doMergeSort(right)

    merge(left, right, items)
}

func merge(left []int, right []int, result []int) {
    l := 0
    r := 0
    i := 0

    for l < len(left) && r < len(right) {
        if left[l] < right[r] {
            result[i] = left[l]
            l++
        } else {
            result[i] = right[r]
            r++
        }
        i++
    }
    var length = len(left) - l
    copy(result[i:i+length], left[l:])
    i = i + length
    length = len(right) - r
    copy(result[i:i+length], right[r:])
}

func main() {
    items := []int{4, 1, 5, 3, 2}

    sortItems := mergeSort(items)
    // sortItems is {1, 2, 3, 4, 5}

    // *** simplified speed test ***

    items = make([]int, 200)
    for i := 0; i < len(items); i++ {
        items[i] = i
    }
    var tmp = items[5]
    items[5] = items[6]
    items[6] = tmp
    count := 10000
    start := time.Now()

    for i := 0; i < count; i++ {
        mergeSort(items)
    }

    delta := time.Now().Sub(start)
    nanoseconds := delta.Nanoseconds()

    fmt.Println(sortItems)
    fmt.Println(nanoseconds)
    // about 439 000 000 nanoseconds
}

Quick sort

package main

import (
    "fmt"
    "time"
)

// Time Complexity from O(n log(n)) to O(n^2)
// Space Complexity O(log(n))

func doSort(items []int, fst int, lst int) {
    if fst >= lst {
        return
    }
    i := fst
    j := lst
    x := items[(fst+lst)/2]

    for i < j {
        for items[i] < x {
            i++
        }
        for items[j] > x {
            j--
        }
        if i <= j {
            var tmp = items[i]
            items[i] = items[j]
            items[j] = tmp
            i++
            j--
        }
    }
    doSort(items, fst, j)
    doSort(items, i, lst)
}

func quicksort(arr []int) []int {
    length := len(arr)
    items := make([]int, length)
    copy(items, arr)
    doSort(items, 0, length-1)
    return items
}

func main() {
    items := []int{4, 1, 5, 3, 2}

    sortItems := quicksort(items)
    // sortItems is {1, 2, 3, 4, 5}

    // *** simplified speed test ***

    items = make([]int, 200)
    for i := 0; i < len(items); i++ {
        items[i] = i
    }
    var tmp = items[5]
    items[5] = items[6]
    items[6] = tmp
    count := 10000
    start := time.Now()

    for i := 0; i < count; i++ {
        quicksort(items)
    }

    delta := time.Now().Sub(start)
    nanoseconds := delta.Nanoseconds()

    fmt.Println(sortItems)
    fmt.Println(nanoseconds)
    // about 83 000 000 nanoseconds
}

Redix sort

package main

import (
    "fmt"
    "math"
    "time"
)

// Time Complexity O(nk)
// Space Complexity O(n+k)

func listToBuckets(items []int, cBase int, i int) [][]int {
    var buckets = make([][]int, cBase)

    var pBase = int(math.Pow(
        float64(cBase), float64(i)))
    for _, x := range items {
        //Isolate the base-digit from the number
        var digit = (x / pBase) % cBase
        //Drop the number into the correct bucket
        buckets[digit] = append(buckets[digit], x)
    }

    return buckets
}

func bucketsToList(buckets [][]int) []int {
    result := []int{}

    for _, bucket := range buckets {
        result = append(result, bucket...)
    }

    return result
}

func radixSort(arr []int, cBase int) []int {
    maxVal := 0
    for i, value := range arr {
        if i == 0 || value > maxVal {
            maxVal = value
        }
    }

    length := len(arr)
    items := make([]int, length)
    copy(items, arr)

    i := 0
    for math.Pow(float64(cBase), float64(i)) <= float64(maxVal) {
        items = bucketsToList(listToBuckets(items, cBase, i))
        i++
    }

    return items
}

func main() {
    items := []int{4, 1, 5, 3, 2}

    sortItems := radixSort(items, 10)
    // sortItems is {1, 2, 3, 4, 5}

    // *** simplified speed test ***

    items = make([]int, 200)
    for i := 0; i < len(items); i++ {
        items[i] = i
    }
    var tmp = items[5]
    items[5] = items[6]
    items[6] = tmp
    count := 10000
    start := time.Now()

    for i := 0; i < count; i++ {
        radixSort(items, 10)
    }

    delta := time.Now().Sub(start)
    nanoseconds := delta.Nanoseconds()

    fmt.Println(sortItems)
    fmt.Println(nanoseconds)
    // about 532 200 000 nanoseconds
}

Getting Started with Go Context

Javad Rajabzadeh — Tue, 22 Jun 2021 12:32:53 +0000

Applications in golang use Contexts for controlling and managing very important aspects of reliable applications, such as cancellation and data sharing in concurrent programming. This may sound trivial but in reality, it’s not that so. The entry point for the contexts in golang is the context package. It is extremely useful and probably one of the most versatile packages of the entire language. If you haven’t come across anything dealing with contexts yet, you probably will very soon (or maybe you just didn’t pay much attention to it). The usage of context is so widespread that multiple other packages rely on it and assume you will do the same. It is definitely a key component on golang’s ecosystem.

Here’s the official documentation for the context package https://golang.org/pkg/context/. It’s really great and filled with practical examples. In an attempt to extend those, let’s have a look into things I have came across in real applications.

Context with value

One of the most common uses for contexts is to share data, or use request scoped values. When you have multiple functions and you want to share data between them, you can do so using contexts. The easiest way to do that is to use the function context.WithValue. This function creates a new context based on a parent context and adds a value to a given key. You can think about the internal implementation as if the context contained a map inside of it, so you can add and retrieve values by key. This is very powerful as it allows you to store any type of data inside the context. Here’s an example of adding and retrieving data with a context.

package main

import (
    "context"
    "fmt"
)

func main() {
    ctx := context.Background()
    ctx = addValue(ctx)
    readValue(ctx)
}

func addValue(ctx context.Context) context.Context {
    return context.WithValue(ctx, "key", "test-value")
}

func readValue(ctx context.Context) {
    val := ctx.Value("key")
    fmt.Println(val)
}

One important aspect of the design behind context package is that everything returns a new context.Context struct. This means that you have to remember to work with the returned value from operations and possibly override old contexts with new ones. This is a key design in immutability.

Using this technique you can pass along the context.Context to concurrent functions and as long as you properly manage the context you are passing on, it’s good way to share scoped values between those concurrent functions (meaning that each context will keep their own values on its scope). That’s exactly what net/http package does when handling HTTP requests. To elaborate on that let’s have a look into the next example.

Middlewares

A great example and use case for request scoped values is working with middlewares in web request handlers. The type http.Request contains a context which can carry scoped values throughout the HTTP pipeline. It is very common to see code where middlewares are added to the HTTP pipeline and the results of the middlewares are added to the http.Request context. This is a very useful technique as you can rely on something you know happened to in your pipeline already on later stages. This also enables you to use generic code to handle HTTP request, while respecting the scope where you want to share the data (instead of sharing data on global variables for example). Here’s an example of a middleware that leverages the request context.

package main

import (
    "context"
    "log"
    "net/http"

    "github.com/google/uuid"
    "github.com/gorilla/mux"
)

func main() {
    router := mux.NewRouter()
    router.Use(guidMiddleware)
    router.HandleFunc("/ishealthy", handleIsHealthy).Methods(http.MethodGet)
    http.ListenAndServe(":8080", router)
}

func handleIsHealthy(w http.ResponseWriter, r *http.Request) {
    w.WriteHeader(http.StatusOK)
    uuid := r.Context().Value("uuid")
    log.Printf("[%v] Returning 200 - Healthy", uuid)
    w.Write([]byte("Healthy"))
}

func guidMiddleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        uuid := uuid.New()
        r = r.WithContext(context.WithValue(r.Context(), "uuid", uuid))
        next.ServeHTTP(w, r)
    })
}

Context Cancellation

Another very useful feature of context in golang is cancelling things that are related. This is very important when you want to propagate your cancellation. It’s a good practice to propagate the cancellation signal when you receive one. Let’s say you have a function where you start tens of goroutines. That main function waits for all goroutines to finish or a cancellation signal before proceeding. If you receive the cancellation signal you may want to propagate it to all your goroutines, so you don’t waste compute resources. If you share the same context among all goroutines you can easily do that.
To create a context with cancellation you only have to call the function context.WithCancel(ctx) passing your context as parameter. This will return a new context and a cancel function. To cancel that context you only need to call the cancel function.

package main

import (
    "context"
    "fmt"
    "io/ioutil"
    "net/http"
    neturl "net/url"
    "time"
)

func queryWithHedgedRequestsWithContext(urls []string) string {
    ch := make(chan string, len(urls))
    ctx, cancel := context.WithCancel(context.Background())
    defer cancel()
    for _, url := range urls {
        go func(u string, c chan string) {
            c <- executeQueryWithContext(u, ctx)
        }(url, ch)

        select {
        case r := <-ch:
            cancel()
            return r
        case <-time.After(21 * time.Millisecond):
        }
    }

    return <-ch
}

func executeQueryWithContext(url string, ctx context.Context) string {
    start := time.Now()
    parsedURL, _ := neturl.Parse(url)
    req := &http.Request{URL: parsedURL}
    req = req.WithContext(ctx)

    response, err := http.DefaultClient.Do(req)

    if err != nil {
        fmt.Println(err.Error())
        return err.Error()
    }

    defer response.Body.Close()
    body, _ := ioutil.ReadAll(response.Body)
    fmt.Printf("Request time: %d ms from url%s\n", time.Since(start).Nanoseconds()/time.Millisecond.Nanoseconds(), url)
    return fmt.Sprintf("%s from %s", body, url)
}

Each request is fired in a separate goroutine. The context is passed to all requests that are fired. The only thing that is being done with the context is that it gets propagated to the HTTP client. This allows a graceful cancellation of the request and underlying connection when the cancel function is called. This is a very common patter for functions that accept a context.Context as argument, they either actively act on the context (like checking if it was cancelled) or they pass it to an underlying function that deals with it (in this case the Do function that receives the context through the http.Request).

Context Timeout

Timeouts are a really common pattern for making external requests, like querying a database or fetching data from another service either through HTTP or gRPC (both support contexts). Handling those scenarios is quite easy using the context package. All you have to do is call the function context.WithTimeout(ctx, time) passing your context and the actual timeout. Like this:

ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)

You still receive the cancel function in case you want to manually trigger that. It works the same way as a normal context cancellation.
The behaviour for this case is very straightforward. In case the timeout is reached, the context is cancelled. Is case of a HTTP call, it works essentially the same as the example above.

gRPC

Context is a fundamental piece of gRPC implementation in golang. It is used both to share data (what is called metadata) and to control flow, like cancelling a stream or request.

// Server implementation receiving metadata
func (*server) Sum(ctx context.Context, req *calculatorpb.SumRequest) (*calculatorpb.SumResponse, error) {
    log.Printf("Sum rpc invoked with req: %v\n", req)
    md, _ := metadata.FromIncomingContext(ctx)
    log.Printf("Metadata received: %v", md)
    return &calculatorpb.SumResponse{
        Result: req.NumA + req.NumB,
    }, nil
}

// Client implementation sending metadata
func sum(c calculatorpb.CalculatorServiceClient) {
    req := &calculatorpb.SumRequest{
        NumA: 3,
        NumB: 10,
    }
    ctx := metadata.AppendToOutgoingContext(context.Background(), "user", "test")
    res, err := c.Sum(ctx, req)
    if err != nil {
        log.Fatalf("Error calling Sum RPC: %v", err)
    }
    log.Printf("Response: %d\n", res.Result)
}

// Server implementation handling context cancellation
func (*server) Greet(ctx context.Context, req *greetpb.GreetRequest) (*greetpb.GreetResponse, error) {
    log.Println("Greet rpc invoked!")

    time.Sleep(500 * time.Millisecond)

    if ctx.Err() == context.Canceled {
        return nil, status.Error(codes.Canceled, "Client cancelled the request")
    }

    first := req.Greeting.FirstName
    return &greetpb.GreetResponse{
        Result: fmt.Sprintf("Hello %s", first),
    }, nil
}

// Client implementation using timeout context cancellation
func greetWithTimeout(c greetpb.GreetServiceClient) {
    req := &greetpb.GreetRequest{
        Greeting: &greetpb.Greeting{
            FirstName: "Ricardo",
            LastName:  "Linck",
        },
    }
    ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
    defer cancel()
    res, err := c.Greet(ctx, req)
    if err != nil {
        grpcErr, ok := status.FromError(err)

        if ok {
            if grpcErr.Code() == codes.DeadlineExceeded {
                log.Fatal("Deadline Exceeded")
            }
        }

        log.Fatalf("Error calling Greet RPC: %v", err)
    }
    log.Printf("Response: %s\n", res.Result)
}

OpenTelemetry

OpenTelemetry also relies heavily on context for what is called Context Propagation. That is a way to tied up requests happening in different systems. The way to implement that is to Inject span information into the context you are going to send as part of the protocol you are using (HTTP or gRPC, for instance). On the other service you need to Extract the span information out of the context.

Build CI/CD pipelines in Go with github actions and Docker

Javad Rajabzadeh — Tue, 22 Jun 2021 09:46:58 +0000

This tutorial will show you how to setup a CI/CD pipeline using GitHub Actions. The pipeline will test, build and publish a Go app to Docker Hub when changes are pushed to a GitHub repository.

Overview

Here is an overview of what’s in this guide:

We start by creating a simple console app in Go that prints Hello World along with the version number of the app.
We will create a GitHub repository and assign Docker Hub credentials using the Secrets feature of GitHub.
We will create a GitHub workflow that runs Go tests each time a change is pushed to the main branch. The workflow will be configured to build a Docker image and push it to Docker Hub when a tag is pushed to the repository.
The workflow will extract a version number from a Git tag and assign it to a Go variable at build time so the latest version is printed when the app is run.
We will see the workflows in action. First by seeing what happens when we push code that fails tests. We will fix the code and see it pass, then we will see what happens when we push a tag.

1. Create Go Project

First let’s create a simple Go program that prints out Hello World along with the version number of our app.
Create a folder to store our files:

javad@fedora:~$ mkdir ~/golang-pipeline
javad@fedora:~$ cd ~/golang-pipeline

Create main.go and main_test.go

javad@fedora:~$ touch main.go
javad@fedora:~$ touch main_test.go

Add the following code to main.go

package main

import "fmt"

var version = "dev"

func main() {

 fmt.Printf("Version: %s\n", version)

 fmt.Println(hello())
}

func hello() string {
 return "Hello Glang"
}

Note: the version variable is assigned the string dev and the string returned by the hello function is intentionally wrong so that our tests fail. Later we will see how the GitHub action can be configured to automatically replace the version string at build time.

Add the following code to main_test.go

package main

import "testing"

func TestHello(t *testing.T) {

 want := "Hello Golang"

 got := hello()

 if want != got {
  t.Fatalf("want %s, got %s\n", want, got)
 }
}

Make sure our tests run and fail by running:

javad@fedora:~$ go test

--- FAIL: TestHello (0.00s)
    main_test.go:9: want Hello Golang, got Hello Glang
FAIL
exit status 1
FAIL    hello   0.003s

It fails because we made the return value of the hello function wrong on purpose so we can test the CI/CD pipeline catches errors.

We can also test the app runs by running:

javad@fedora:~$ go run main.go

Version: dev
Hello glang

Notice it prints out dev for the version. When we run the release version from the published image on Docker Hub, we will see the version number that was tagged when pushing a release to the GitHub repo.

2. Create GitHub Repository

Login to your GitHub account and create a new repository. For the following example I have created a new public repo (exmple golang-pipeline)

3. Assign Docker Hub Credentials as Secrets

For this step you will need to login to your Docker Hub account and generate an access token. Once you’ve done that, navigate to the Settings screen of the GitHub repository then click on Secrets.

Create Secret DOCKER_USERNAME

Create Secret DOCKER_ACCESS_TOKEN

Secrets for pipeline

4. Create GitHub Workflow

We are now ready to create the GitHub workflow. Create a folder inside your repository called .github/workflows and add a file called push.yml.

javad@fedora:~$ mkdir -p .github/workflows
javad@fedora:~$ touch .github/workflows/push.yml

Add the following config to the push.yml file:

name: golang-pipeline
on: push
jobs:
  test:
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags')
    steps:
      - uses: actions/checkout@v2
      - name: Run Unit Tests
        run: go test

  deploy:
    runs-on: ubuntu-latest
    needs: test
    if: startsWith(github.ref, 'refs/tags')
    steps:
      - name: Extract Version
        id: version_step
        run: |
          echo "##[set-output name=version;]VERSION=${GITHUB_REF#$"refs/tags/v"}"
          echo "##[set-output name=version_tag;]$GITHUB_REPOSITORY:${GITHUB_REF#$"refs/tags/v"}"
          echo "##[set-output name=latest_tag;]$GITHUB_REPOSITORY:latest"
      - name: Print Version
        run: |
          echo ${{steps.version_step.outputs.version_tag}}
          echo ${{steps.version_step.outputs.latest_tag}}
      - name: Set up QEMU
        uses: docker/setup-qemu-action@v1

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v1

      - name: Login to DockerHub
        uses: docker/login-action@v1
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}

      - name: PrepareReg Names
        id: read-docker-image-identifiers
        run: |
          echo VERSION_TAG=$(echo ${{ steps.version_step.outputs.version_tag }} | tr '[:upper:]' '[:lower:]') >> $GITHUB_ENV
          echo LASTEST_TAG=$(echo ${{ steps.version_step.outputs.latest_tag  }} | tr '[:upper:]' '[:lower:]') >> $GITHUB_ENV
      - name: Build and push
        id: docker_build
        uses: docker/build-push-action@v2
        with:
          push: true
          tags: |
            ${{env.VERSION_TAG}}
            ${{env.LASTEST_TAG}}
          build-args: |
            ${{steps.version_step.outputs.version}}

Replace Variables in the environment with $ ( example push.yml )

The workflow config is quite simple. You will notice there are two jobs, one called test and one called deploy. The test job has one step that uses an if to make sure it only runs when changes are pushed to the main branch or a tag is pushed.

The deploy job requires the test job to run first so that if the tests fail, the Docker image will not be built. If the test job does pass, the remaining steps will run. The first step extracts the version number from a git tag that has the following format v1.0.0.

The next two steps setup the environment so that Docker images can be built. Finally, a step is run that signs into your Docker Hub account using the credentials stored in GitHub Secrets, then the Docker image is built and pushed to Docker Hub.

You Can see you run action pipeline in https://github.com/USERNAME/golang-pipeline/actions your repo.

5. Create Dockerfile

Now we need to create a Dockerfile for the GitHub action to use when building the image. Create a file called Dockerfile at the root of the repo.

javad@fedora:~$ touch Dockerfile

Add the following to the Dockerfile:

FROM golang:1.16.4-buster AS builder

ARG VERSION=dev

WORKDIR /go/src/app
COPY main.go .
RUN go build -o main -ldflags=-X=main.version=${VERSION} main.go 

FROM debian:buster-slim
COPY --from=builder /go/src/app/main /go/bin/main
ENV PATH="/go/bin:${PATH}"
CMD ["main"]

The config above uses a multi-stage build process to build the Go app then copy it to a slim Debian image. If you want to make the image smaller, you can change it to alpine or scratch.

Let’s test the Dockerfile by building it with the following command:

javad@fedora:~$ docker build -t golang-pipeline:dev .

Sending build context to Docker daemon  114.2kB
Step 1/9 : FROM golang:1.16.4-buster AS builder
1.16.4-buster: Pulling from library/golang
d960726af2be: Pull complete 
e8d62473a22d: Pull complete 
8962bc0fad55: Pull complete 
65d943ee54c1: Pull complete 
f2253e6fbefa: Pull complete 
6d7fa7c7d5d3: Pull complete 
e2e442f7f89f: Pull complete 
Digest: sha256:ab32429d40c1b734ed4f036838ac516352182f9414563478fa88a1553c4a4414
Status: Downloaded newer image for golang:1.16.4-buster
 ---> 96129f3766cf
Step 2/9 : ARG VERSION=dev
 ---> Running in 819610ae34de
Removing intermediate container 819610ae34de
 ---> 9eec806043be
Step 3/9 : WORKDIR /go/src/app
 ---> Running in 46c0cf6a36b4
Removing intermediate container 46c0cf6a36b4
 ---> 1f1f66dfcc4f
Step 4/9 : COPY main.go .
 ---> 963a3bcf262b
Step 5/9 : RUN go build -o main -ldflags=-X=main.version=${VERSION} main.go
 ---> Running in c18447aa254a
Removing intermediate container c18447aa254a
 ---> 3d5364678624
Step 6/9 : FROM debian:buster-slim
buster-slim: Pulling from library/debian
69692152171a: Pull complete 
Digest: sha256:f077cd32bfea6c4fa8ddeea05c53b27e90c7fad097e2011c9f5f11a8668f8db4
Status: Downloaded newer image for debian:buster-slim
 ---> 80b9e7aadac5
Step 7/9 : COPY --from=builder /go/src/app/main /go/bin/main
 ---> f96694436dca
Step 8/9 : ENV PATH="/go/bin:${PATH}"
 ---> Running in 0423a0a1b60a
Removing intermediate container 0423a0a1b60a
 ---> 8ac59bcbe805
Step 9/9 : CMD ["main"]
 ---> Running in 1491bcf193c6
Removing intermediate container 1491bcf193c6
 ---> 190ba9407211
Successfully built 190ba9407211
Successfully tagged golang-pipeline:dev

Now run the image and see if it prints the incorrect string Hello Glang:

javad@fedora:~$ docker run --rm golang-pipeline:dev

Version: dev
Hello glang

We can assign a version to the image by using --build-arg VERSION=1.0.0 while building the image. For example:

javad@fedora:~$ docker build -t golang-pipeline:1.0.0 . --build-arg VERSION=1.0.0

Sending build context to Docker daemon  114.2kB
Step 1/9 : FROM golang:1.16.4-buster AS builder
 ---> 96129f3766cf
Step 2/9 : ARG VERSION=dev
 ---> Using cache
 ---> 9eec806043be
Step 3/9 : WORKDIR /go/src/app
 ---> Using cache
 ---> 1f1f66dfcc4f
Step 4/9 : COPY main.go .
 ---> Using cache
 ---> 963a3bcf262b
Step 5/9 : RUN go build -o main -ldflags=-X=main.version=${VERSION} main.go
 ---> Running in 352fb8de7969
Removing intermediate container 352fb8de7969
 ---> 0950776088fa
Step 6/9 : FROM debian:buster-slim
 ---> 80b9e7aadac5
Step 7/9 : COPY --from=builder /go/src/app/main /go/bin/main
 ---> 91938ce4f326
Step 8/9 : ENV PATH="/go/bin:${PATH}"
 ---> Running in 44541fe9de77
Removing intermediate container 44541fe9de77
 ---> c8924b6b45b3
Step 9/9 : CMD ["main"]
 ---> Running in 424b93b94419
Removing intermediate container 424b93b94419
 ---> 9da417e7399b
Successfully built 9da417e7399b
Successfully tagged golang-pipeline:1.0.0

The docker image built can be found here:

javad@fedora:~$ docker images

REPOSITORY                TAG             IMAGE ID       CREATED          SIZE
golang-pipeline           1.0.0           9da417e7399b   22 seconds ago   71.2MB
golang-pipeline           dev             190ba9407211   3 minutes ago    71.2MB
golang                    1.16.4-buster   96129f3766cf   2 weeks ago      862MB
debian                    buster-slim     80b9e7aadac5   2 weeks ago      69.2MB

The reason this works, is because we use -ldflags to modify the version variable at compile time. The --build-arg assigns the version to VERSION and is used in the following line:

RUN go build -o main -ldflags=-X=main.version=${VERSION} main.go

Okay, we are now ready to push our code to the GitHub repository.

6. Push Code to Main

Now that we know our project builds and we have created a repository, we are ready to push our code to the main branch. Let’s initialise Git, commit our changes, add the origin, and push the changes by running the following commands (replace the repo URL with your own):

javad@fedora:~$ git init
javad@fedora:~$ git add .
javad@fedora:~$ git commit -m "first commit"
javad@fedora:~$ git remote add origin git@github.com:USERNAME/golang-pipeline
javad@fedora:~$ git branch -M main
javad@fedora:~$ git push -u origin main

7. Check Build Failed

Go to your GitHub repository and click Actions. We should see the GitHub action running. Since we are pushing to the main branch, it will run the test job and it should fail.

Github Action test fail

8. Fix Code so Test Passes

Let’s fix the broken test.
Open main.go and modify the return value of the hello function so that it returns Hello Golang.

javad@fedora:~$ go test

PASS
ok      hello   0.003s

Commit and push the changes.

javad@fedora:~$ git add .
javad@fedora:~$ git commit - "Fix hello function return value"
javad@fedora:~$ git push

The GitHub action should run again and now the test will pass.

Github Action test successfully

Github Action status

9. Push Tag

Now that we know the test passes, let’s push a release tag. We will tag this version as v1.0.0

javad@fedora:~$ git tag v1.0.0
javad@fedora:~$ git push --tags

After pushing the tags, the GitHub action will run again and this time it will build the Docker image and push it to your Docker Hub repository.

How to use Go Modules with Private Git repository

Javad Rajabzadeh — Fri, 18 Jun 2021 08:32:12 +0000

The Go module system has greatly improved the way dependencies are managed. If you are new to Go Modules.

Public Repositories

After configuring everything correctly, it is relatively straightforward to include Go packages from public repositories. Typical starting points for me are as follows:

module github.com/Ja7ad/project

go 1.16

require (
    github.com/pkg/errors
    github.com/spf13/cobra
    github.com/spf13/viper
)

Private Repositories

Go uses Git to pull the versions of the dependencies you specify. To access any private repositories, the git configuration for wherever Go is running (e.g. a Docker container or your laptop) needs to be configured with the appropriate credentials.

GitHub

git config --global url."https://${user}:${personal_access_token}@github.com".insteadOf "https://github.com"

GitLab

git config --global url."https://oauth2:${personal_access_token}@privategitlab.com".insteadOf "https://privategitlab.com"

# or 

git config --global url."https://${user}:${personal_access_token}@privategitlab.com".insteadOf "https://privategitlab.com"

BitBucket

git config --global url."https://${bitbucket_id}:${bitbucket_token}@privatebitbucket.com".insteadOf "https://privatebitbucket.com"

CI/CD Pipeline

Dockerfile can be added via this method and add private repository to the image:


# ---------------------------------------------------------------------
#  The first stage container, for building the application
# ---------------------------------------------------------------------
FROM golang:1.16-stretch as builder

COPY . /app

# Add the keys
ARG github_user
ENV github_user=$github_user
ARG github_personal_token
ENV github_personal_token=$github_personal_token

WORKDIR /app/cmd/webapp

RUN git config \
    --global \
    url."https://${github_user}:${github_personal_token}@@github.com".insteadOf \
    "https://github.com"

RUN GIT_TERMINAL_PROMPT=1 \
    GOARCH=amd64 \
    GOOS=linux \
    CGO_ENABLED=0 \
    go build -v --installsuffix cgo --ldflags="-s" -o myapp

# ---------------------------------------------------------------------
#  The second stage container, for running the application
# ---------------------------------------------------------------------
FROM alpine:3.8
COPY --from=builder /app/cmd/webapp/myapp /app/myapp

WORKDIR /app

ENTRYPOINT ["/myapp"]

You can use the following method for docker-compose.yml:


version: '3.6'
services:
  app:
    container_name: my_go_app_container
    build:
      # context can/may/will be different per-project setup
      context: ../
      dockerfile: GitDockerfile
      args:
        - github_user=private_user
        - github_personal_token=private_token
    image: my_go_app_image
    # other configs...

SSH

Another way to do this would be to use your SSH key every time you connect, and to set .gitconfig like the following to ensure SSH is used each time:

git config --global url."git@github.com".insteadOf "https://github.com"