DEV Community: Didik Tri Susanto

Implement Your Own Request Middleware for Go HTTP Server

Didik Tri Susanto — Fri, 02 May 2025 14:54:15 +0000

I believed this topic is a lot in the internet but I want to write it anyway, because documenting your own learning journey and share it to the world is good right?

Request middleware performs some specific function on the HTTP request or response at a specific stage in the HTTP pipeline before the business logic / app handler.

Currently middleware is a common term but in different programming language or framework sometimes it called filter. For illustration could be like this

As you can see, middleware could be layered for more than one before hitting app handler or business logic then later it will return the response. HTTP Request comes in to first layer or middleware, check the necessary and if all conditions are passed then it allowed to continue the journey. If failed, it could also returned the failed response anyway.

Basic Middleware

Let say we have simple HTTP API server

func rootHandler() http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        resp := defaultResponse{
            Status:      "OK",
            Description: "Success",
        }

        j, _ := json.Marshal(resp)
        w.Header().Add("Content-Type", "application/json")
        w.Write(j)
    })
}

func main() {
    mux := http.NewServeMux()

    mux.Handle("GET /", rootHandler())

    s := &http.Server{
        Addr:    ":8080",
        Handler: mux,
    }

    // ...
}

if we run the program go run main.go and visit http://localhost:8080 it will return 200 success with JSON response:

{
  "status": "OK",
  "description": "Success"
}

Now lets create our first middleware. Our middleware is to write a simple log that indicating the request is passing through. The middleware should be a function like this

func logMiddleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        log.Print("Executing logMiddleware")
        next.ServeHTTP(w, r)
    })
}

Then we will modify the mux handler to use the middleware

func main() {
    mux := http.NewServeMux()
    mux.Handle("/", logMiddleware(rootHandler()))
    // ...
}

The result is when we’re visiting the API, it also wrote the log that already specified in the middleware logic.

Adding More Middleware

Interesting. Now try to add more middleware. This time we will check if the API request header contains x-signature, if not exists then we will return error.

Lets create the middleware function

func requestSignatureMiddleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        log.Print("Executing requestSignatureMiddleware")
        h := r.Header.Get("x-signature")
        if h == "" {
            resp := errorResponse{
                Status:  http.StatusUnauthorized,
                Error:   "INVALID_MISSING_SIGNATURE",
                Message: "missing request signature",
            }

            j, _ := json.Marshal(resp)
            w.Header().Add("Content-Type", "application/json")
            w.WriteHeader(http.StatusUnauthorized)
            w.Write(j)
            return
        }
        next.ServeHTTP(w, r)
    })
}

We want to call logMiddleware first then continue to requestSignatureMiddleware. Our mux handler would be look like this now

func main() {
    mux := http.NewServeMux()
    mux.Handle("/", logMiddleware(requestSignatureMiddleware(rootHandler())))
    // ...
}

Here’s the result of the combined middlewares

First request without x-signature header will returned 401 HTTP status
Second request with additional x-signature header passed middleware logic and returned 200 HTTP status.

Cleaning Up

You may notice that our mux handler with multiple middlewares is little bit messy because it is calling function into function. What if we have many middlewares?

logMiddleware(requestSignatureMiddleware(authMiddleware(rateLimitMiddleware(rootHandler)))))

Hard to see.

Here’s what we can do, create a chain function to loop all middleware functions. You can named the function whatever you want.

// Middleware type for cleaner middleware chaining
type Middleware func(http.Handler) http.Handler

// Chain creates a single middleware from multiple middlewares
func Chain(middlewares ...Middleware) Middleware {
    return func(next http.Handler) http.Handler {
        for i := len(middlewares) - 1; i >= 0; i-- {
            next = middlewares[i](next)
        }
        return next
    }
}

Initiate the Chain with our middlewares and finally call it in our mux handler.

func main() {
    mux := http.NewServeMux()

    // Create a chain of middlewares
    middlewareChain := Chain(
        func(next http.Handler) http.Handler { return logMiddleware(next) },
        func(next http.Handler) http.Handler { return requestSignatureMiddleware(next) },
    // Add more middlewares here as needed
    )

    mux.Handle("GET /", middlewareChain(rootHandler()))
    // ...
}

Now it’s more manageable and it will still got same result as before.

So here’s the full code of this experiment https://gist.github.com/didikz/98de868f20887285f5aabc80f4dddcfa

Happy coding!

Large CSV Processing Using Go

Didik Tri Susanto — Sat, 23 Nov 2024 03:00:29 +0000

The idea is:

Given a large dummy CSV (1 million rows) contains sample of customer data and do processing with goals below:

Extract the data from the CSV

Calculate how many data / rows

Grouping how many customers for each city

Sort cities by customers count from highest to lowest

Calculate processing time

Sample CSV of the customers can be downloaded here https://github.com/datablist/sample-csv-files

Load And Extract Data

Apparently Go has standard lib for CSV processing. We don't need 3rd party dependency to solve our problem anymore which is nice. So the solution is pretty straightforward:

  // open the file to a reader interface
  c, err := os.Open("../data/customers-1000000.csv")
  if err != nil {
    log.Fatal(err)
  }
  defer c.Close()

  // load file reader into csv reader
  // Need to set FieldsPerRecord to -1 to skip fields checking
  r := csv.NewReader(c)
  r.FieldsPerRecord = -1
  r.ReuseRecord = true
  records, err := r.ReadAll()
  if err != nil {
    log.Fatal(err)
  }

Open the file from the given path
Load opened file to csv reader
Holds all extracted csv records / rows value into records slice for later processing

FieldsPerRecord is set to -1 because I want to skip fields checking on the row since fields or column count could be different in each format

At this state we already able to load and extract all the data from csv and ready to next processing state. We also will able to know how many rows in CSV by using function len(records).

Grouping Total Customer to Each City

Now we are able to iterate the records and create the map contains city name and total customer looks like this:

["Jakarta": 10, "Bandung": 200, ...]

City data in csv row is located in 7th index and the code will look like this

  // create hashmap to populate city with total customers based on the csv data rows
  // hashmap will looks like be ["city name": 100, ...]
  m := map[string]int{}
  for i, record := range records {
    // skip header row
    if i == 0 {
    continue
    }
    if _, found := m[record[6]]; found {
      m[record[6]]++
    } else {
      m[record[6]] = 1
    }
  }

If the city map is not exists, create new map and set the customer total as 1. Otherwise just increment the total number of given city.

Now we have map m contains collection of city and how many customer inside it. At this point we already solved problem of grouping how many customer for each city.

Sorting Highest Total Customer

I tried to find is there any function in standard lib to sort the map but unfortunately I couldn't find it. Sorting only possible for slice because we can rearrange the data order based on the index position. So yeah, let's make a slice from our current map.

// convert to slice first for sorting purposes
dc := []CityDistribution{}
for k, v := range m {
  dc = append(dc, CityDistribution{City: k, CustomerCount: v})
}

Now how we sorted it by the CustomerCount from highest to lowest? The most common algorithm for this is using bubble short. Although it's not the fastest but it could do the job.

Bubble Sort is the simplest sorting algorithm that works by repeatedly swapping the adjacent elements if they are in the wrong order. This algorithm is not suitable for large data sets as its average and worst-case time complexity is quite high.

Reference: https://www.geeksforgeeks.org/bubble-sort-algorithm/

Using our slice, it will loop over the data and check the next value of the index and swap it if current data is less than next index. You can check the detail algorithm on the reference website.

Now our sorting process could be like this

// use bubble sort
dcCount := len(dc)
for i := 0; i < dcCount; i++ {
  swapped := false
  for j := 0; j < dcCount-i-1; j++ {
    if dc[j].CustomerCount < dc[j+1].CustomerCount {
      temp := dc[j]
      dc[j] = dc[j+1]
      dc[j+1] = temp
      swapped = true
    }
  }

  if !swapped {
    break
  }
}

By the end of the loop, the final slice will give us a sorted data.

Calculate Processing Time

Calculate processing time is quite simple, we get timestamp before & after executing the main process of the program and calculate the difference. In Go the approach should be simple enough:

func main() {
    start := time.Now() // start timing for processing time
    // the main process
    // ...
    duration := time.Since(start)
    fmt.Println("processing time (ms): ", duration.Milliseconds())
}

The Result

Run the program with command

go run main.go

The printed out would be rows count, sorted data, and processing time. Something like this below:

As expected of Go performance, it handled 1 million rows csv under 1 second!

All the completed codes already publish on my Github Repository:

https://github.com/didikz/csv-processing/tree/main/golang

Lesson Learned

CSV processing in Go is already available in standard lib, no need to use 3rd party lib
Processing the data is quite easy. The challenge was to find out how to sort the data because need to do manually

What's Come in Mind?

I was thinking my current solution might can be optimized further because I looped all the records extracted csv to map and if we checked at ReadAll() source, it also have loop to create the slice based on the given file reader. By this, 1 Mil rows could produce 2 x loops for 1 Mil data which is not nice.

I thought if I could read data directly from File reader it only needs 1 loop because I could create map directly from it. Except the records slice will be used elsewhere but not in this case.

I still have no time to figure it out yet, but I also thought some downside if I will do it manually:

Probably need handle more errors of the parsing process
I am not sure how significant it will reduce the processing time to consider the workaround will be worth it or not

Happy Coding!

Building Simple Real-Time System Monitor using Go, HTMX, and Web Socket

Didik Tri Susanto — Sun, 17 Nov 2024 03:41:11 +0000

I was finding a fun project to work with Go, HTMX, and Tailwwindcss and ended up built a simple real-time web based system monitor with the power of web socket. Here’s the result.

It shows system information, memories, disk, CPU, and running processes, updated automatically every 5 seconds.

I’ll break down the code little bit in this post.

Stacks

HTTP Server

type HttpServer struct {
    subscriberMessageBuffer int
    Mux                     http.ServeMux
    subscribersMutex        sync.Mutex
    subscribers             map[*subscriber]struct{}
}

type subscriber struct {
    msgs chan []byte
}

It’s quite straightforward. HttpServer contains a http.ServeMux as http handler and subscribers for web socket broadcasting later. subscriber is simply has msgs channel for data update.

Since it only needs to serve a single HTML file, then it needs URL to show the page, and one URL for web socket connection.

func NewHttpServer() *HttpServer {
    s := &HttpServer{
        subscriberMessageBuffer: 10,
        subscribers:             make(map[*subscriber]struct{}),
    }

    s.Mux.Handle("/", http.FileServer(http.Dir("./views")))
    s.Mux.HandleFunc("/ws", s.subscribeHandler)
    return s
}

Web Socket Connection & Subscriber

Endpoint /ws will handling web socket connection and managing a subscriber. First it will initiate a new subscriber and added it to a map in the http server structure. Lock will be used to prevent race condition since we will use go routine later.

func (s *HttpServer) subscribeHandler(w http.ResponseWriter, r *http.Request) {
    err := s.subscribe(r.Context(), w, r)
    if err != nil {
        fmt.Println(err)
        return
    }
}

func (s *HttpServer) addSubscriber(subscriber *subscriber) {
    s.subscribersMutex.Lock()
    s.subscribers[subscriber] = struct{}{}
    s.subscribersMutex.Unlock()
    fmt.Println("subscriber added", subscriber)
}

Web socket is starting accept a connection and via loop, we will detect an incoming channel msgs from subscriber and write it to web socket.

func (s *HttpServer) subscribe(ctx context.Context, w http.ResponseWriter, r *http.Request) error {
    var c *websocket.Conn
    subscriber := &subscriber{
        msgs: make(chan []byte, s.subscriberMessageBuffer),
    }

    s.addSubscriber(subscriber)

    c, err := websocket.Accept(w, r, nil)
    if err != nil {
        return err
    }

    defer c.CloseNow()

    ctx = c.CloseRead(ctx)
    for {
        select {
        case msg := <-subscriber.msgs:
            ctx, cancel := context.WithTimeout(ctx, time.Second)
            defer cancel()
            err := c.Write(ctx, websocket.MessageText, msg)
            if err != nil {
                return err
            }
        case <-ctx.Done():
            return ctx.Err()
        }
    }
}

Auto Update

Auto update the system info data is handled by go routine. We will build a html response that will be sent via web socket and htmx will handle the update on the html side.

func main() {
    fmt.Println("Starting system monitor")
    s := server.NewHttpServer()

    go func(s *server.HttpServer) {
        for {
            hostStat, _ := host.Info()
            timestamp := time.Now().Format("2006-01-02 15:04:05")
            html := `
            <span hx-swap-oob="innerHTML:#data-timestamp">` + timestamp + `</span>
            <span hx-swap-oob="innerHTML:#system-hostname">` + hostStat.Hostname + `</span>
            <span hx-swap-oob="innerHTML:#system-os">` + hostStat.OS + `</span>
            `
            s.Broadcast([]byte(html))
            time.Sleep(time.Second * 5)
        }
    }(s)
    // ...
}

Syntax hx-swap-oob="innerHTML:#data-timestamp" in htmx is tell us that swap a component inside data-timestamp id in our HTML. All swapping mechanism will be the same for other system information components.

All swappable html components will be sent via Broadcast(msg) method and later will be sent via channel every 5 seconds.

func (s *HttpServer) Broadcast(msg []byte) {
    s.subscribersMutex.Lock()
    for subscriber := range s.subscribers {
        subscriber.msgs <- msg
    }
    s.subscribersMutex.Unlock()
}

The HTMX View

It’s plain HTML file and for Tailwindcss I simple used CDN for that

<script src="https://cdn.tailwindcss.com"></script>

Same idea for HTMX and web socket extension for using CDN.

<script src="https://unpkg.com/htmx.org@2.0.3" integrity="sha384-0895/pl2MU10Hqc6jd4RvrthNlDiE9U1tWmX7WRESftEDRosgxNsQG/Ze9YMRzHq" crossorigin="anonymous"></script>
<script src="https://unpkg.com/htmx-ext-ws@2.0.1/ws.js"></script>

How to connect to the web socket?

The system monitor page is expected to receives all the data by web socket so I can set it from the main div container. Specify hx-ext=”ws”to tell HTMX for using web socket extension and ws-connect=”/ws” to tell web socket to connect via /ws URL.

<body class="bg-gray-700 text-white">
    <div class="container mx-auto p-8" hx-ext="ws" ws-connect="/ws">
   <!-- all the data -->
    </div>
</body>

Full Code

Here is the full version of the code https://github.com/didikz/gosysmon-web and you may want to play around with your own version.

Happy coding!

How To Create Package in Go

Didik Tri Susanto — Sat, 05 Oct 2024 03:47:43 +0000

For reusable purpose, package is a good start to manage your Go codes because we can import and use it to our program.

Lets create a simple main file to begin with

package main

import "fmt"

func main() {
   fmt.Println("hello world!")
}

It's a simple hello world program that printed out hello world string when we run go run main.go .

Now lets initiate a package with creating the module first. Commonly it's using a git repository even we have no plan to push it to any repository out there. In this example I will use my own Github repository and use mygopackageas package name.

git mod init github.com/didikz/mygopackage

Then create a subdirectory, for example I use models and create a user.go inside of it. The structure should be looks like this

Inside user.go I would simple create a struct and a receiver that could be imported later in the main.go . I also set package name following the current directory name as models

package models

type User struct {
    Id        int
    FirstName string
    LastName  string
    Address   string
}

func (user *User) GetName() string {
    return user.FirstName + " " + user.LastName
}

Get back to main.go and now we can try to import the package and use the defined struct. Use the module name initiated before along with the package name.

import "github.com/didikz/mygopackage/models"

Now to use the user model from the package we can write like this

var user models.User
user.Id = 1
user.FirstName = "Didik"
user.LastName = "Tri Susanto"
user.Address = "Malang"

// or alternatively
user := models.User{Id: 1, FirstName: "Didik", LastName: "Tri Susanto", Address: "Malang"}

fmt.Println(user.GetName())

All set. Next, If we run go run main.go then it should printed out Didik Tri Susanto

Easy right?

The final main.go file now should be like this

package main

import (
    "fmt"

    "github.com/didikz/mygopackage/models"
)

func main() {
    user := models.User{Id: 1, FirstName: "Didik", LastName: "Tri Susanto", Address: "Malang"}
    fmt.Println(user.GetName())
}

That's it and happy coding!