DEV Community: sean

Seamless Integration Testing for Your Go Application on GitHub Actions with PostgreSQL

sean — Fri, 22 Nov 2024 12:21:07 +0000

Introduction

Integration testing is crucial for ensuring your Go application works flawlessly with external dependencies like databases. In this blog, we’ll explore how to set up and run integration tests for a Go application using GitHub Actions. We’ll configure a PostgreSQL database within the CI pipeline, streamline the test process, and ensure your codebase is reliable and production-ready with every push. Let’s dive in!.

We created unit test and integrations in a previous article here!. In this article we want to run these tests on all commits to our github repository.

Github Actions

They are a continuous integration and continuous delivery (CI/CD) platform that allows you to automate your build, test and deployment pipeline.
Github Actions lets you run workflows when other events happen in your repository

Github Workflows

A workflow is a configurable automated process that will run one or more jobs. Workflows are defined by a YAML file checked in to your repository and will run when triggered by an event in your repository. Workflows are defined in the .github/workfows.

Event is a specific activity in a repository that triggers a workflow run. In our case this will be a push to our branch.
Jobs is a set of steps in a workflow that is executed on the same runner.
Runners is a server that runs your workflows when they are triggered. Each runner can runner a single job at a time.

Workflow Yaml

The first step would be to create .github/workflows folder where our yaml file will be located.
Next is to create the yaml file in this case we will name it ci-test.yml.

name: ci-test

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

env:
  POSTGRES_USER: postgres
  POSTGRES_PASSWORD: Password123
  POSTGRES_DB: crud_db

jobs:
  build:
    name: tests
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres
        env:
          POSTGRES_USER: ${{ env.POSTGRES_USER }}
          POSTGRES_PASSWORD: ${{ env.POSTGRES_PASSWORD }}
          POSTGRES_DB: ${{ env.POSTGRES_DB }}
        ports:
          - 5432:5432
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5

    steps:
      - uses: actions/checkout@v4

      - name: Set up Go
        uses: actions/setup-go@v4
        with:
          go-version: "1.22"

      - name: Install dbmate for golang migrations
        run: |
          sudo curl -fsSL -o /usr/local/bin/dbmate https://github.com/amacneil/dbmate/releases/latest/download/dbmate-linux-amd64
          sudo chmod +x /usr/local/bin/dbmate
          which dbmate

      - name: Construct DB URL
        id: construct_url
        run: echo "DB_URL=postgres://${{ env.POSTGRES_USER }}:${{ env.POSTGRES_PASSWORD }}@localhost:5432/${{ env.POSTGRES_DB }}?sslmode=disable" >> $GITHUB_ENV

      - run: env

      - name: Make Migrations
        run: make migrations URL=${{ env.DB_URL }}

      - name: Seed test DB
        run: go run db/seed.go

      - name: Test
        run: make test

Yaml Description

The first part is to name the action in this case it is ci-test.

Workflow Triggers

The second section describes triggers. Events that trigger the action. In this file we have two events that will trigger the running of this jobs, pushes and pull requests targeting the main branches. This ensures that every code change intended for production is tested before merged, maintaining the integrity of the project.

Environment Variables

Github workflows support global and job-specific environment variables. This variables describe postgres credentials that we will user later in our yaml file.

Job

jobs:
  build:
    name: tests
    runs-on: ubuntu-latest

Here we have assigned a name to the job that will perform the core tasks, which are building and testing our code.
Runner - describes where the workflow will run which will is a Ubuntu viritual machine.

Services

Github Actions workflows allow you to define services. In this case we need a postgres database to run our tests against.

A PostgreSQL container is created using the official PostgreSQL Docker image.
The container is configured with environment variables we declared earlier

Workflow Steps

First step is to checkout the repository code

- uses: actions/checkout@v4

This line fetches the latest version of the repository, providing access to all source files.

Second step is to setup golang in the runner.

- name: Set up Go
  uses: actions/setup-go@v4
  with:
    go-version: "1.22"

The third step is installing dbmate on our runner. Dbmate is a migration tools that will manage application migrations.

- name: Install dbmate for golang migrations
  run: |
    sudo curl -fsSL -o /usr/local/bin/dbmate https://github.com/amacneil/dbmate/releases/latest/download/dbmate-linux-amd64
    sudo chmod +x /usr/local/bin/dbmate
    which dbmate

Fourth is to construct the db url

- name: Construct DB URL
  id: construct_url
  run: echo "DB_URL=postgres://${{ env.POSTGRES_USER }}:${{ env.POSTGRES_PASSWORD }}@localhost:5432/${{ env.POSTGRES_DB }}?sslmode=disable" >> $GITHUB_ENV

Fifth is runnig db migrations to setup our relations that will seed with date

- name: Make Migrations
  run: make migrations URL=${{ env.DB_URL }}

The second last action is to seed the database with test data.

- name: Seed test DB
  run: go run db/seed.go

The seed.go file seeds the data ase with test data. Setting up a realistic test environment. To inspect this file further, visit here

The final stage is to execute our go test using the make file

- name: Test
  run: make test

This workflow will now run each time we make a pull request or push code to our main branch

Some Advantages of Adopting Github Action.

As we have seen github action allow you to do

Automated Testing - run tests consistently on every code change.
Have Database Integretions - provide a real postgres environment for testing, simulating production conditions
Reproducible Environment - Github action use containerized services and predefined steps to ensure consistent results across all runs.
Fast Feedback loop - They enable developers to receive quick feedback if something breaks, allowing for faster issue resolution.
Simplified Collaboration - They ensure all contributors' changes are verified before emerging, maintaining code quality and project stability

Conclusions

By leveraging GitHub Actions, this workflow streamlines testing and database setup, ensuring robust and reliable software development.
Visit the github repository to view the code that is being tested with the action described above.

Testing REST APIs in Go: A Guide to Unit and Integration Testing with Go's Standard Testing Library

sean — Fri, 15 Nov 2024 07:52:06 +0000

Introduction

This article is going to take you through how to use unit test and integration test to improve you development experience as you create rest apis in golang.

Unit tests are designed to verify the functionality of the smallest, individual parts of an application, often focusing on a single function or method. These tests are conducted in isolation from other parts of the code to ensure each component works as expected on its own.
Integration tests, on the other hand, assess how different modules or components of the application work together. In this article, we’ll focus on integration testing for our Go application, specifically checking that it interacts correctly with a PostgreSQL database by successfully making and executing SQL queries.

This article assumes that you are familiar with golang and how to create rest api in golang the main focus will be on creating test for your routes (unit tests) and testing your sql query functions (integration tests) for reference visit the github to have a look at the project.

Setting Up

Assuming you have setup your project similar to the one linked above you will have a folder structure similar to this

test_project
|__cmd
   |__api
      |__api.go
   |__main.go
|__db
   |___seed.go
|__internal
   |___db
       |___db.go
   |___services
       |___records
           |___routes_test.go
           |___routes.go
           |___store_test.go
           |___store.go
       |___user
           |___routes_test.go
           |___routes.go
           |___store_test.go
           |___store.go
|__test_data
|__docker-compose.yml
|__Dockerfile
|__Makefile

Testing in golang is easy compored to other language you may have encountered becase of the inbuilt testing package that provides tools needed to write tests.
Test files are named with _test.go this suffix allows for go to target this files for execution when running the command go test.

The entrypoint for our project is the main.go file located in the cmd folder

// main.go

package main

import (
    "log"

    "finance-crud-app/cmd/api"
    "finance-crud-app/internal/db"

    "github.com/gorilla/mux"
    "github.com/jmoiron/sqlx"
    _ "github.com/lib/pq"
)

type Server struct {
    db  *sqlx.DB
    mux *mux.Router
}

func NewServer(db *sqlx.DB, mux *mux.Router) *Server {
    return &Server{
        db:  db,
        mux: mux,
    }
}

func main() {

    connStr := "postgres://postgres:Password123@localhost:5432/crud_db?sslmode=disable"

    dbconn, err := db.NewPGStorage(connStr)
    if err != nil {
        log.Fatal(err)
    }
    defer dbconn.Close()

    server := api.NewAPIServer(":8085", dbconn)
    if err := server.Run(); err != nil {
        log.Fatal(err)
    }
}

From the code you can see we are creating a new api server by passing a database connection and port number. After creating the server we run it on the stated port.

The NewAPIServer command comes from the api.go file which

// api.go
package api

import (
    "finance-crud-app/internal/services/records"
    "finance-crud-app/internal/services/user"
    "log"
    "net/http"

    "github.com/gorilla/mux"
    "github.com/jmoiron/sqlx"
)

type APIServer struct {
    addr string
    db   *sqlx.DB
}

func NewAPIServer(addr string, db *sqlx.DB) *APIServer {
    return &APIServer{
        addr: addr,
        db:   db,
    }
}

func (s *APIServer) Run() error {
    router := mux.NewRouter()
    subrouter := router.PathPrefix("/api/v1").Subrouter()

    userStore := user.NewStore(s.db)
    userHandler := user.NewHandler(userStore)
    userHandler.RegisterRoutes(subrouter)

    recordsStore := records.NewStore(s.db)
    recordsHandler := records.NewHandler(recordsStore, userStore)
    recordsHandler.RegisterRoutes(subrouter)

    log.Println("Listening on", s.addr)

    return http.ListenAndServe(s.addr, router)
}

For this api we are using mux as our http router.

Integration Test

We have a user Store struct that handles sql queries related to the user entity.

// store.go
package user

import (
    "errors"
    "finance-crud-app/internal/types"
    "fmt"
    "log"

    "github.com/jmoiron/sqlx"
)

var (
    CreateUserError   = errors.New("cannot create user")
    RetrieveUserError = errors.New("cannot retrieve user")
    DeleteUserError   = errors.New("cannot delete user")
)

type Store struct {
    db *sqlx.DB
}

func NewStore(db *sqlx.DB) *Store {
    return &Store{db: db}
}

func (s *Store) CreateUser(user types.User) (user_id int, err error) {
    query := `
    INSERT INTO users
    (firstName, lastName, email, password)
    VALUES ($1, $2, $3, $4)
    RETURNING id`

    var userId int
    err = s.db.QueryRow(query, user.FirstName, user.LastName, user.Email, user.Password).Scan(&userId)
    if err != nil {
        return -1, CreateUserError
    }

    return userId, nil
}

func (s *Store) GetUserByEmail(email string) (types.User, error) {
    var user types.User

    err := s.db.Get(&user, "SELECT * FROM users WHERE email = $1", email)
    if err != nil {
        return types.User{}, RetrieveUserError
    }

    if user.ID == 0 {
        log.Fatalf("user not found")
        return types.User{}, RetrieveUserError
    }

    return user, nil
}

func (s *Store) GetUserByID(id int) (*types.User, error) {
    var user types.User
    err := s.db.Get(&user, "SELECT * FROM users WHERE id = $1", id)
    if err != nil {
        return nil, RetrieveUserError
    }

    if user.ID == 0 {
        return nil, fmt.Errorf("user not found")
    }

    return &user, nil
}

func (s *Store) DeleteUser(email string) error {

    user, err := s.GetUserByEmail(email)
    if err != nil {
        return DeleteUserError
    }
    // delete user records first
    _, err = s.db.Exec("DELETE FROM records WHERE userid = $1", user.ID)
    if err != nil {
        return DeleteUserError
    }

    _, err = s.db.Exec("DELETE FROM users WHERE email = $1", email)
    if err != nil {
        return DeleteUserError
    }
    return nil
}

In the file above we have 3 pointer receiver methods:

CreateUser
GetUserByEmail
GetUserById

For these methods to perform their function they have to interact with an external systems which ,in this case that is Postgres DB .

To test this methods we will first create a store_test.go file. In go we usually name our test files after the file we are targetting to test and add the suffix _test.go .

// store_test.go

package user_test

import (
    "finance-crud-app/internal/db"
    "finance-crud-app/internal/services/user"
    "finance-crud-app/internal/types"
    "log"
    "os"
    "testing"

    "github.com/jmoiron/sqlx"
    _ "github.com/lib/pq"
)

var (
    userTestStore *user.Store
    testDB        *sqlx.DB
)

func TestMain(m *testing.M) {
    // database
    ConnStr := "postgres://postgres:Password123@localhost:5432/crud_db?sslmode=disable"
    testDB, err := db.NewPGStorage(ConnStr)
    if err != nil {
        log.Fatalf("could not connect %v", err)
    }
    defer testDB.Close()
    userTestStore = user.NewStore(testDB)

    code := m.Run()
    os.Exit(code)
}

func TestCreateUser(t *testing.T) {
    test_data := map[string]struct {
        user   types.User
        result any
    }{
        "should PASS valid user email used": {
            user: types.User{
                FirstName: "testfirsjjlkjt-1",
                LastName:  "testlastkjh-1",
                Email:     "validuser@email.com",
                Password:  "00000000",
            },
            result: nil,
        },
        "should FAIL invalid user email used": {
            user: types.User{
                FirstName: "testFirstName1",
                LastName:  "testLastName1",
                Email:     "test1@email.com",
                Password:  "800890",
            },
            result: user.CreateUserError,
        },
    }

    for name, tc := range test_data {
        t.Run(name, func(t *testing.T) {
            value, got := userTestStore.CreateUser(tc.user)
            if got != tc.result {
                t.Errorf("test fail expected %v got %v instead and value %v", tc.result, got, value)
            }
        })
    }

    t.Cleanup(func() {
        err := userTestStore.DeleteUser("validuser@email.com")
        if err != nil {
            t.Errorf("could not delete user %v got error %v", "validuser@email.com", err)
        }
    })
}

func TestGetUserByEmail(t *testing.T) {
    test_data := map[string]struct {
        email  string
        result any
    }{
        "should pass valid user email address used": {
            email:  "test1@email.com",
            result: nil,
        },
        "should fail invalid user email address used": {
            email:  "validuser@email.com",
            result: user.RetrieveUserError,
        },
    }

    for name, tc := range test_data {
        got, err := userTestStore.GetUserByEmail(tc.email)
        if err != tc.result {
            t.Errorf("test fail expected %v instead got %v", name, got)
        }
    }
}

func TestGetUserById(t *testing.T) {
    testUserId, err := userTestStore.CreateUser(types.User{
        FirstName: "userbyid",
        LastName:  "userbylast",
        Email:     "unique_email",
        Password:  "unique_password",
    })
    if err != nil {
        log.Panicf("got %v when creating testuser", testUserId)
    }

    test_data := map[string]struct {
        user_id int
        result  any
    }{
        "should pass valid user id used": {
            user_id: testUserId,
            result:  nil,
        },
        "should fail invalid user id used": {
            user_id: 0,
            result:  user.RetrieveUserError,
        },
    }

    for name, tc := range test_data {
        t.Run(name, func(t *testing.T) {
            _, got := userTestStore.GetUserByID(tc.user_id)
            if got != tc.result {
                t.Errorf("error retrieving user by id got %v want %v", got, tc.result)
            }
        })
    }

    t.Cleanup(func() {
        err := userTestStore.DeleteUser("unique_email")
        if err != nil {
            t.Errorf("could not delete user %v got error %v", "unique_email", err)
        }
    })
}

func TestDeleteUser(t *testing.T) {
    testUserId, err := userTestStore.CreateUser(types.User{
        FirstName: "userbyid",
        LastName:  "userbylast",
        Email:     "delete_user@email.com",
        Password:  "unique_password",
    })
    if err != nil {
        log.Panicf("got %v when creating testuser", testUserId)
    }

    test_data := map[string]struct {
        user_email string
        result     error
    }{
        "should pass user email address used": {
            user_email: "delete_user@email.com",
            result:     nil,
        },
    }

    for name, tc := range test_data {
        t.Run(name, func(t *testing.T) {
            err = userTestStore.DeleteUser(tc.user_email)
            if err != tc.result {
                t.Errorf("error deletig user got %v instead of %v", err, tc.result)
            }
        })
    }

    t.Cleanup(func() {
        err := userTestStore.DeleteUser("delete_user@email.com")
        if err != nil {
            log.Printf("could not delete user %v got error %v", "delete_user@email.com", err)
        }
    })
}

Lets go through the file looking at what each section does.

The first action is to declare the variables userTestStore and testDB. These variables will be used to store pointers to the user store and db respectively. The reason we have declared them in the global file scope is because we want all functions in the test file to have access to the pointers.

The TestMain function allows us to do some setting up actions before the main test are run. We are initially connecting to the postgres store and saving the pointer into our global variable.
We have used that pointer to create a userTestStore that we will use to execute the sql queries we are trying to connect.

defer testDB.Close() closes the database connection after the test has completed

code := m.Run() runs the rest of the test function before returning and exiting.

TestCreateUser function will handle the testing of the create_user function. Our goal is to test if the function will create the user if a unique email is passed and the function should not be able to create a user if a non-unique email has already been used to create another user.

First we create the test data that we will use to test both case scenarios.

test_data := map[string]struct {
        user   types.User
        result any
    }{
        "should PASS valid user email used": {
            user: types.User{
                FirstName: "testfirsjjlkjt-1",
                LastName:  "testlastkjh-1",
                Email:     "validuser@email.com",
                Password:  "00000000",
            },
            result: nil,
        },
        "should FAIL invalid user email used": {
            user: types.User{
                FirstName: "testFirstName1",
                LastName:  "testLastName1",
                Email:     "test1@email.com",
                Password:  "800890",
            },
            result: user.CreateUserError,
        },
    }

I will loop through the map executing create_user function with the test date as parameters and compare if the returned value is the same the result we expect

for name, tc := range test_data {
        t.Run(name, func(t *testing.T) {
            value, got := userTestStore.CreateUser(tc.user)
            if got != tc.result {
                t.Errorf("test fail expected %v got %v instead and value %v", tc.result, got, value)
            }
        })
    }

For cases where the returned result is not the same as the expected result then our test will fail

The last part of this function is using the inbuilt testing package function Cleanup. This function registered a function that will be called when all the function in the test have already been executed. In our example case here we are using the function to clear up user data that was used during this test function execution.

Unit Tests

For our unit tests we are going to test the route handlers for our api. In this case the routes related to the user entity. Observe below.

package user

import (
    "finance-crud-app/internal/services/auth"
    "finance-crud-app/internal/types"
    "finance-crud-app/internal/utils"
    "fmt"
    "net/http"
    "strconv"

    "github.com/go-playground/validator/v10"
    "github.com/gorilla/mux"
)

type Handler struct {
    store types.UserStore
}

func NewHandler(store types.UserStore) *Handler {
    return &Handler{store: store}
}

func (h *Handler) RegisterRoutes(router *mux.Router) {
    router.HandleFunc("/login", h.handleLogin).Methods("POST")
    router.HandleFunc("/register", h.handleRegister).Methods("POST")

    // secured routes
    router.HandleFunc("/users/{userID}", auth.JWTAuthMiddleWare(h.handleGetUser, h.store)).Methods(http.MethodGet)
}

func (h *Handler) handleLogin(w http.ResponseWriter, r *http.Request) {
    var user types.LoginUserPayload
    if err := utils.ParseJSON(r, &user); err != nil {
        utils.WriteError(w, http.StatusBadRequest, fmt.Errorf("invalid payload: %v", err))
        return
    }

    if err := utils.Validate.Struct(user); err != nil {
        errors := err.(validator.ValidationErrors)
        utils.WriteError(w, http.StatusBadRequest, fmt.Errorf("invalid payload: %v", errors))
        return
    }

    u, err := h.store.GetUserByEmail(user.Email)
    if err != nil {
        utils.WriteError(w, http.StatusBadRequest, fmt.Errorf("invalid credentials"))
        return
    }

    if !auth.ComparePasswords(u.Password, []byte(user.Password)) {
        utils.WriteError(w, http.StatusBadRequest, fmt.Errorf("invalid credentials"))
        return
    }

    token, err := auth.CreateJWT(u.ID)
    if err != nil {
        utils.WriteError(w, http.StatusBadRequest, fmt.Errorf("error %v", err))
        return
    }

    utils.WriteJSON(w, http.StatusAccepted, map[string]string{"token": token})
}

func (h *Handler) handleRegister(w http.ResponseWriter, r *http.Request) {
    var user types.RegisterUserPayload
    if err := utils.ParseJSON(r, &user); err != nil {
        utils.WriteError(w, http.StatusBadRequest, err)
        return
    }

    if err := utils.Validate.Struct(user); err != nil {
        errors := err.(validator.ValidationErrors)
        utils.WriteError(w, http.StatusBadRequest, fmt.Errorf("invalid payload: %v", errors))
        return
    }

    hashedPassword, err := auth.HashPassword(user.Password)
    if err != nil {
        utils.WriteError(w, http.StatusInternalServerError, err)
        return
    }

    userId, err := h.store.CreateUser(types.User{
        FirstName: user.FirstName,
        LastName:  user.LastName,
        Email:     user.Email,
        Password:  hashedPassword,
    })

    if err != nil {
        utils.WriteError(w, http.StatusInternalServerError, err)
        return
    }
    utils.WriteJSON(w, http.StatusCreated, map[string]int{"user_id": userId})
}

func (h *Handler) handleGetUser(w http.ResponseWriter, r *http.Request) {
    vars := mux.Vars(r)
    str, ok := vars["userID"]
    if !ok {
        utils.WriteError(w, http.StatusBadRequest, fmt.Errorf("missing user ID"))
        return
    }

    userID, err := strconv.Atoi(str)
    if err != nil {
        utils.WriteError(w, http.StatusBadRequest, fmt.Errorf("invalid user ID"))
        return
    }

    user, err := h.store.GetUserByID(userID)
    if err != nil {
        utils.WriteError(w, http.StatusInternalServerError, err)
        return
    }

    utils.WriteJSON(w, http.StatusOK, user)
}

We have 3 function here that we would want to test

HandleLogin
HandleRegister
HandleGetUser

HandleGetUser

The handleGetUser function in this handler retrieves user details based on a user ID provided in the HTTP request URL. It starts by extracting the userID from the request path variables using the mux router. If the userID is missing or invalid (non-integer), it responds with a 400 Bad Request error. Once validated, the function calls the GetUserByID method on the data store to retrieve user information. If an error occurs during retrieval, it returns a 500 Internal Server Error. On success, it responds with a 200 OK status, sending the user details as JSON in the response body.

As stated before for you to for as to test the handler functions we need to create a routes_test.go. See mine below

package user

import (
    "finance-crud-app/internal/types"
    "net/http"
    "net/http/httptest"
    "testing"

    "github.com/gorilla/mux"
)

type mockUserStore struct{}

func (m *mockUserStore) DeleteUser(email string) error {
    return nil
}

func (m *mockUserStore) GetUserByEmail(email string) (types.User, error) {
    return types.User{}, nil
}

func (m *mockUserStore) CreateUser(u types.User) (userId int, err error) {
    return 1, nil
}

func (m *mockUserStore) GetUserByID(id int) (*types.User, error) {
    return &types.User{}, nil
}

func TestGetUserHandler(t *testing.T) {
    userStore := &mockUserStore{}
    handler := NewHandler(userStore)

    t.Run("should fail to get user with user_id that is not a number", func(t *testing.T) {
        req, err := http.NewRequest(http.MethodGet, "/user/abc", nil)
        if err != nil {
            t.Fatal(err)
        }

        rr := httptest.NewRecorder()
        router := mux.NewRouter()

        router.HandleFunc("/user/{userID}", handler.handleGetUser).Methods(http.MethodGet)

        router.ServeHTTP(rr, req)

        if rr.Code != http.StatusBadRequest {
            t.Errorf("expected status code %d, got %d", http.StatusBadRequest, rr.Code)
        }
    })

    t.Run("should pass to get user with numeric id", func(t *testing.T) {
        req, err := http.NewRequest(http.MethodGet, "/user/23", nil)
        if err != nil {
            t.Fatal(err)
        }

        rr := httptest.NewRecorder()
        router := mux.NewRouter()

        router.HandleFunc("/user/{userID}", handler.handleGetUser).Methods(http.MethodGet)
        router.ServeHTTP(rr, req)

        if rr.Code != http.StatusOK {
            t.Errorf("expected status code %d, got %d", http.StatusBadGateway, rr.Code)
        }
    })
}

Our New Handler function requires a user store as a parameter for it to create a handler struct.
Since we do not need actual store we create a mock struct and create receiver functions that mock the function of the actual struct. We do this because we are handling the store function tests separetly therefore we dont need to test that part of the code in the handler tests.

The test function TestGetUserHandler test two case scenarios, the first is attempting to retrieve a user without providing the user id

t.Run("should fail to get user with user_id that is not a number", func(t *testing.T) {
        req, err := http.NewRequest(http.MethodGet, "/user/abc", nil)
        if err != nil {
            t.Fatal(err)
        }

        rr := httptest.NewRecorder()
        router := mux.NewRouter()

        router.HandleFunc("/user/{userID}", handler.handleGetUser).Methods(http.MethodGet)

        router.ServeHTTP(rr, req)

        if rr.Code != http.StatusBadRequest {
            t.Errorf("expected status code %d, got %d", http.StatusBadRequest, rr.Code)
        }
    })

The test is expected to pass if the http request responds with a 400 status code.

The second test case scenario is cases where we are retrieving, user information by using the correct url containing a valid user id. In this test case we expected a response with 200 status code. If not that test will have failed.

t.Run("should pass to get user with numeric id", func(t *testing.T) {
        req, err := http.NewRequest(http.MethodGet, "/user/23", nil)
        if err != nil {
            t.Fatal(err)
        }

        rr := httptest.NewRecorder()
        router := mux.NewRouter()

        router.HandleFunc("/user/{userID}", handler.handleGetUser).Methods(http.MethodGet)
        router.ServeHTTP(rr, req)

        if rr.Code != http.StatusOK {
            t.Errorf("expected status code %d, got %d", http.StatusBadGateway, rr.Code)
        }
    })

Conclusion

We have managed to implement unit test in our project by creating test for our route handlers. We have seen how to use mocks to only test a small unit of code. We have been able to developed integration test for our function that interact with Postgresql DB.
If you would want some hands on time with the project code clone the repo from github here

Instrumenting Django Applications using OpenTelemetry

sean — Thu, 18 Jul 2024 09:00:11 +0000

In this articles we are going to go through instrumentating your django application using OTel. The project will demonstrate how to add logging for Prometheus and how to visualised spans using Jaeger.

Introduction

OpenTelemetry is a vendor-neutral open source Observability framework for instrumenting, generating, collecting, and exporting telemetry data such as traces, metrics, and logs.

It provides end-to-end tracing, allowing you to follow requests as they travel through various services and components. This helps in understanding the flow and identifying bottlenecks or failures.
Enables the easy collection of metrics such as latency, request count, error rates, and resource utilization helps in monitoring the health and performance of your application.
Enables correlating logs with traces and metrics provides a detailed view of what happens at each step of the request lifecycle.

Prequisites

To follow along with this article you require:

Docker installed
python and/or poetry

Step 1: Install Dependencies

Setup your python environment then install the following packages



poetry add django opentelemetry-sdk opentelemetry-instrumentation-django gunicorn django-prometheus opentelemetry-exporter-otlp celery

After installing the project dependencies setup up you django application



poetry run django-admin startproject config .

Step 2: Setup Django App

We will use an todo application to demonstrate some of the features that we can use from OpenTelemetry.

Create a todo app in our django application and added to the setting.py inside the config folder



poetry run python3 manage.py startapp todo

After this your settings should be similart to this



INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles', 
    'django_prometheus',

    'todo'
]

update your todo/urls.py file to be



from django.urls import path 

from . import views

urlpatterns = [
  path("", views.home, name="home"),
  path('create-todo', views.create_todo, name="create-todo")
]

we will also add a templates folder that contains the html that will be rendered by the view below. You will find the template files in this link.

update your templates settings in the settings.py file to be



TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [BASE_DIR / 'templates'],
        'APP_DIRS': True,
        'OPTIONS': {
            'context_processors': [
                'django.template.context_processors.debug',
                'django.template.context_processors.request',
                'django.contrib.auth.context_processors.auth',
                'django.contrib.messages.context_processors.messages',
            ],
        },
    },
]

Before create the view for the application we will need a model to store our todos

Create the following model within the models.py file in the todo folder



from django.db import models
from datetime import datetime

# Create your models here.

class Todo(models.Model):
  text: str = models.TextField(null=False)
  created_at: datetime = models.DateTimeField(auto_now_add=True, null=True)
  title: str = models.CharField(max_length=100, null=False)
  is_completed: bool = models.BooleanField(default=False)

  def __str__(self) -> str:
    return self.title

Next create the views for our todo application



from django.shortcuts import get_object_or_404, redirect
from django.template.response import TemplateResponse
from django.http import HttpRequest, HttpResponse


from todo.models import Todo
from todo.task import task_created_alert

def home(request: HttpRequest) -> HttpResponse:

  # with trace.
  todos = Todo.objects.all()

  return TemplateResponse(
    request,
    "todos/home.html",
    {
      "todos": todos
    },
  )

def create_todo(request: HttpRequest) -> HttpResponse: 
  if request.method == 'POST':
    title = request.POST.get('title')
    description = request.POST.get('description')

    created_todo = Todo.objects.create(title=title, text=description)

    return redirect('home')

  return TemplateResponse(
    request, 
    "todos/create_todo.html"
  )

Add our new urls to the application urls in config/urls.py file



from django.contrib import admin
from django.urls import path, include

urlpatterns = [
    path('admin/', admin.site.urls),
    path('', include("todo.urls"))
]

For this application we are also using Celery for taks queues in django. To setup celery in your app create celery.py file inside the config folder.



# celery.py

import os

from celery import Celery

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'config.settings')

app = Celery('config')

app.config_from_object('django.conf:settings', namespace='CELERY')

app.autodiscover_tasks()

to complete the integration process of celery to django navigate to the init.py file in the config folder and adjust it to be as follows.



# config/__init__.py

from .celery import app as celery_app

__all__ = ('celery_app',)

Step 3: Instrumenting the application

In this step we are going to instrument our todo application. First we are going to tackle logging of our application by integrating the django-prometheus package we installed earlier update settings.py file to be



MIDDLEWARE = [
    'django_prometheus.middleware.PrometheusBeforeMiddleware',
    'django.middleware.security.SecurityMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
    'django_prometheus.middleware.PrometheusAfterMiddleware',
]

By updating our middleware we allow django_prometheus to log all request made to our application we can also add logging for django models by modifying our todo model.



from django.db import models
from datetime import datetime
from django_prometheus.models import ExportModelOperationsMixin

# Create your models here.

class Todo(ExportModelOperationsMixin('todo'), models.Model):
  text: str = models.TextField(null=False)
  created_at: datetime = models.DateTimeField(auto_now_add=True, null=True)
  title: str = models.CharField(max_length=100, null=False)
  is_completed: bool = models.BooleanField(default=False)

  def __str__(self) -> str:
    return self.title

Django prometheus makes it easy to automatically add logging to our applicaiton.

Next is to add otel spans into a our application to monitor the lifecyle of request.

For production environments we will create a gunicorn.config.py file in the root directory and modify it as follows:



import os

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import (
    OTLPSpanExporter,
)

from opentelemetry.sdk.resources import Resource, SERVICE_NAME
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor


TRACING_EXPORTER_ENDPOINT = os.environ.get('JAEGER_ENDPOINT', 'http://127.0.0.1:4317')

def post_fork(server, worker):
    server.log.info("Worker spawned (pid: %s)", worker.pid)

    resource = Resource(attributes={
        SERVICE_NAME: "todo-application"
    })

    traceProvider = TracerProvider(resource=resource)


    processor = BatchSpanProcessor(OTLPSpanExporter(endpoint=TRACING_EXPORTER_ENDPOINT))
    traceProvider.add_span_processor(processor)
    trace.set_tracer_provider(traceProvider)

In the file above we are exporting application spans to Jaeger application at a port 4317 where jaeger is expecting to receive trace information.

Modify celery.py to enable it to export spans from celery to Jaeger



# config/celery.py
import os
from opentelemetry.sdk.resources import SERVICE_NAME, Resource
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import (
    BatchSpanProcessor,
)

from celery import Celery

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'config.settings')

TRACING_EXPORTER_ENDPOINT = os.environ.get('JAEGER_ENDPOINT', 'http://127.0.0.1:4317')

resource = Resource(attributes={
  SERVICE_NAME: "celery-worker"
})

traceProvider = TracerProvider(resource=resource)

TRACING_EXPORTER_ENDPOINT = os.environ.get('JAEGER_ENDPOINT', TRACING_EXPORTER_ENDPOINT)
processor = BatchSpanProcessor(OTLPSpanExporter(endpoint=TRACING_EXPORTER_ENDPOINT))
traceProvider.add_span_processor(processor)
trace.set_tracer_provider(traceProvider)

tracer = trace.get_tracer(__name__)

app = Celery('config')

app.config_from_object('django.conf:settings', namespace='CELERY')


app.autodiscover_tasks()

Next is to modify the wsgi.py file found in config.py by adding the following line to automatically instrument the whole application



"""
WSGI config for config project.

It exposes the WSGI callable as a module-level variable named ``application``.

For more information on this file, see
https://docs.djangoproject.com/en/5.0/howto/deployment/wsgi/
"""
import os

from opentelemetry.instrumentation.django import DjangoInstrumentor


from django.core.wsgi import get_wsgi_application

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'config.settings')
DjangoInstrumentor().instrument()


application = get_wsgi_application()

We have been able to instrument our django + celery application at this point but there is still some customisation we have to do. Inorder for us to view the span of a view that uses celery tasks ques we have to use context propagation.

The create_do view function creates a todo record in the model and then passes the created todo title to task_created_alert function that runs for 10 seconds.

To propagate the context from the view to the task_queue function modify the view function to be as follows:



from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator

def create_todo(request: HttpRequest) -> HttpResponse: 
  if request.method == 'POST':
    title = request.POST.get('title')
    description = request.POST.get('description')

    created_todo = Todo.objects.create(title=title, text=description)

    # add this
    carrier = {}
    TraceContextTextMapPropagator().inject(carrier) 
    task_created_alert.delay(title=created_todo.title, headers=carrier)
   # end here
    sleep(5)
    return redirect('home')

  return TemplateResponse(
    request, 
    "todos/create_todo.html"
  )

The updated function retrieves the current trace contexts and todo title then passes it to our task function.

Let create the task that should run after creation of a todo.
Create a task.py file in the todo folder



from time import sleep
from celery import shared_task
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator
from opentelemetry import trace


tracer = trace.get_tracer(__name__)
@shared_task
def task_created_alert(title, headers):
  ctx = TraceContextTextMapPropagator().extract(carrier=headers)


  with tracer.start_as_current_span('tack_create_alert', context=ctx):
    span = trace.get_current_span()
    span.set_attribute("title", title)
    sleep(10)

  return True

tasks.py file has task_created_alert function which we called in create_todo view function. We create another context from the context provided when calling the function.

We are simulating a task that takes 10 seconds to view the generated span from this tasks.

Step 4: Setup Observabilty tools
As previously stated we are using docker compose to run this application with all its dependencies.
First we need a dockerfile for our python application



FROM python:3.12-slim 

# Set environment variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1

# Install Poetry
RUN pip install poetry

# Set work directory
WORKDIR /code

# Copy only the necessary files to install dependencies
COPY pyproject.toml poetry.lock /code/

# Install dependencies
RUN poetry config virtualenvs.create false \
    && poetry install --no-dev --no-interaction --no-ansi

# Copy project
COPY . /code/

# Expose port 8000 for the app
EXPOSE 8000

Then the docker compose file which will be located at the root folder of our application



services: 
  web:
    build: .
    command: gunicorn --config gunicorn.config.py --workers=4 --bind 0.0.0.0:8000 config.wsgi:application
    volumes:
      - .:/code
    environment:
      - OTEL_SERVICE_NAME=todo-app
      - OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true
      - CELERY_BROKER_URL=amqp://guest:guest@rabbitmq:5672/
      - JAEGER_ENDPOINT=http://jaeger:4317/
      - OTEL_LOGS_EXPORTER=otlp
    ports:
      - "8000:8000"

  celery:
    build: .
    command: celery --app=config worker --loglevel=info --logfile=logs/celery.log
    volumes: 
      - .:/code 
    environment:
      - OTEL_SERVICE_NAME=todo-app
      - OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true
      - CELERY_BROKER_URL=amqp://guest:guest@rabbitmq:5672/
      - JAEGER_ENDPOINT=http://jaeger:4317/

  rabbitmq:
    image: rabbitmq:3-management
    container_name: 'rabbitmq'
    ports:
        - 5672:5672
        - 15672:15672

  jaeger:
    image: jaegertracing/all-in-one:1.58
    ports:
      - "16686:16686"
      - "4318:4318"
      - "6831:6831"
      - "4317:4317"
    environment:
      - LOG_LEVEL=debug

  prometheus:
    image: prom/prometheus
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    command: 
      - '--config.file=/etc/prometheus/prometheus.yml'
    ports:
      - 9090:9090

prometheus configurations



global:
 scrape_interval: 10s
 evaluation_interval: 10s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
    - targets: ['127.0.0.1:9090']

  - job_name: 'web-app-stuff'
    metrics_path: '/metrics'
    scrape_interval: 5s
    static_configs:
    - targets: ['web:8000']
      labels:
        alias: "web-app"

by adding the configurations we are able to run our application and view spans that are generated when creating a todo

Step 5: Usage
Visit web app at web-app and create a task in your todo
the visit the jaeger ui dashboard search for POST create-todo operation to view the span for the request

We are also able to view our logs using the Prometheus dashboard

Summary

We have gone through how to implement a django app using celery and also to propagate request in the app
Useful links

Upload and Delete file from Amazon S3 Bucket in Go using Presigned URLs

sean — Mon, 22 Apr 2024 06:25:50 +0000

Introduction

This article will demonstrate how to securely upload and delete files from Amazon S3 storage using signed urls using the golang sdk.
Amazon Simple Storage Service (Amazon S3) is an object storage service.

Prerequsites

An aws s3 bucket where we will be storing our files.
An access key and secret access key that have been granted to a user with the necessary permission to upload and delete objects from the s3 bucket.

Benefits of Using Presinged URLS

Controlled Access - set an expiration time for the URL, ensuring temporary access
Security - accesss is limited to specific s3 objects, reducing the risk of unauthorized access

Demonstration

We will create a REST api that has two routes, upload route that receives a file that will be resized and uploaded to the bucket and delete which will take in the filename to be deleted and delete it.

Setup the API

Create a go project using go mod init <name_of_project>. Once done create a main.go file and create the following functions.

func main() {
    fmt.Println("hello world")
    setupRoutes()
}

func setupRoutes() {
    http.HandleFunc("/upload", UploadHandler())
    http.HandleFunc("/delete", DeleteHandler())
    http.ListenAndServe(":8080", nil)
}

func UploadHandler() http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {

    }
}

func DeleteHandler() http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {


    }
}

We have setup an http routes to handle the delete and upload action for our project

Setup Environment Variables

We are going to load our environment variables from a .env file. Create .env file in the root of the project and fill it with bucket details in this format

AWS_BUCKET_NAME="<bucket-name-here>"
AWS_REGION="<aws-region-here>"
AWS_S3_BUCKET_ACCESS_KEY="<aws-bucket-access-key-here>"
AWS_S3_BUCKET_SECRET_ACCESS_KEY="<aws-bucket-secret-key-here>"

Once environment variables are setup we need load them into our project. For this this i will use viper

Create a folder named utils. Inside the folder create a file named config.go

Update the file to be

package utils

import (
    "log"
    "path/filepath"

    "github.com/spf13/viper"
)

type Config struct {
    AWS_BUCKET_NAME                 string `mapstructure:"AWS_BUCKET_NAME"`
    AWS_REGION                      string `mapstructure:"AWS_REGION"`
    AWS_S3_BUCKET_ACCESS_KEY        string `mapstructure:"AWS_S3_BUCKET_ACCESS_KEY"`
    AWS_S3_BUCKET_SECRET_ACCESS_KEY string `mapstructure:"AWS_S3_BUCKET_SECRET_ACCESS_KEY"`
}

func LoadViperEnvironment(path string) (config Config, err error) {
    viper.SetConfigFile(filepath.Join(path, ".env"))
    viper.SetConfigType("env")

    // Read the config file
    if err := viper.ReadInConfig(); err != nil {
        log.Fatalf("Error reading config file, %s", err)
    }

    err = viper.Unmarshal(&config)
    if err != nil {
        log.Fatalf("Unable to decode into struct, %v", err)
    }

    return config, nil
}

We will use the code above to load the environment variables into our routes by updating setupRoutes() in main.go

func setupRoutes() {
    // Load configurations from environment variables from .env
    // at root of project
    cwd, err := os.Getwd()
    if err != nil {
        log.Fatalf("Error getting current working directory: %v", err)
    }
    config, err := utils.LoadViperEnvironment(cwd)
    if err != nil {
        log.Fatalf("Error loading config: %s", err)
    }
    http.HandleFunc("/upload", UploadHandler(config))
    http.HandleFunc("/delete", DeleteHandler(config))
    http.ListenAndServe(":8080", nil)
}

We updated setupRoutes to import the app configs from the .env file and then pass those configs to the separate routes where they will be used when creating s3 bucket presigned urls.

Setup S3 bucket Client

Create another folder in the root directory named repo inside repo we will create a file named repo.go.

For this stage of the project I will need to download the necessary modules from the golang sdk

Write the following commands in the terminal to get the sdk modules.

go get github.com/aws/aws-sdk-go-v2
go get github.com/aws/aws-sdk-go-v2/credentials
go get github.com/aws/aws-sdk-go-v2/service/s3

First lets add the following code into the repo.go file

type Repo struct {
    s3Client          *s3.Client
    s3PresignedClient *s3.PresignClient
}

func NewS3Client(accessKey string, secretKey string, s3BucketRegion string) *Repo {
    options := s3.Options{
        Region:      s3BucketRegion,
        Credentials: aws.NewCredentialsCache(credentials.NewStaticCredentialsProvider(accessKey, secretKey, "")),
    }

    client := s3.New(options, func(o *s3.Options) {
        o.Region = s3BucketRegion
        o.UseAccelerate = false
    })

    presignClient := s3.NewPresignClient(client)
    return &Repo{
        s3Client:          client,
        s3PresignedClient: presignClient,
    }
}

func (repo Repo) PutObject(bucketName string, objectKey string, lifetimeSecs int64) (*v4.PresignedHTTPRequest, error) {
    request, err := repo.s3PresignedClient.PresignPutObject(context.TODO(), &s3.PutObjectInput{
        Bucket: aws.String(bucketName),
        Key:    aws.String(objectKey),
    }, func(opts *s3.PresignOptions) {
        opts.Expires = time.Duration(lifetimeSecs * int64(time.Second))
    })
    if err != nil {
        log.Printf("Couldn't get a presigned request to put %v:%v. Here's why: %v\n",
            bucketName, objectKey, err)
    }
    return request, err
}

func (repo Repo) DeleteObject(bucketName string, objectKey string, lifetimeSecs int64) (*v4.PresignedHTTPRequest, error) {
    request, err := repo.s3PresignedClient.PresignDeleteObject(context.TODO(), &s3.DeleteObjectInput{
        Bucket: aws.String(bucketName),
        Key:    aws.String(objectKey),
    }, func(opts *s3.PresignOptions) {
        opts.Expires = time.Duration(lifetimeSecs * int64(time.Second))
    })
    if err != nil {
        log.Printf("Couldn't get a presigned request to delete %v:%v. Here's why: %v\n",
            bucketName, objectKey, err)
    }

    return request, err
}

func (repo Repo) UploadFile(file image.Image, url string) error {
    var buf bytes.Buffer
    err := jpeg.Encode(&buf, file, nil)
    if err != nil {
        return nil
    }
    body := io.Reader(&buf)
    request, err := http.NewRequest(http.MethodPut, url, body)
    if err != nil {
        return err
    }

    request.Header.Set("Content-Type", "image/jpeg")

    client := &http.Client{}
    resp, err := client.Do(request)
    if err != nil {
        log.Println("Error sending request:", err)
        return err
    }
    defer resp.Body.Close() 
    return err
}

func (repo Repo) DeleteFile(url string) error {
    request, err := http.NewRequest(http.MethodDelete, url, nil)
    if err != nil {
        return err
    }
    response, err := http.DefaultClient.Do(request)
    if err != nil {
        return err
    }

    log.Printf("Delete Request Response status code: %v", response.StatusCode)
    return err
}

Create a struct Repo that holds our s3client and s3PresingedClient.
NewS3Client takes in accesskey, secretkey, s3bucket region and returns the defined repo struct with all containing the s3 clients
The PutObject function that creates a presigned url for uploading files into the bucket. The function required the bucketname, objectkey(filename), lifetime(how long will the presinged url will be valid)
The DeleteObject function creates a presigned url for deleting a file from the bucket. The function required bucketname, objectkey(filename) and lifetime to create return the url.
UploadFile function takes in the url generate from the Putobject function and perform a put request populated by a file.
DeleteFile function takes in the url generated from DeleteObject function and makes a delete request to the url.

Hooking Up everything

We will now update our routes to use the configuration from .env and upload and delete files using the repo functions

Upload Action

func UploadHandler(config utils.Config) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {

        file, _, err := r.FormFile("image")
        if err != nil {
            w.WriteHeader(http.StatusBadRequest)
            fmt.Fprintf(w, "Error reading file: %v", err)
            return
        }
        defer file.Close()

        imageDoc, _, err := image.Decode(file)
        if err != nil {
            w.WriteHeader(http.StatusInternalServerError)
            fmt.Fprintf(w, "Error encoding image: %v", err)
            return
        }

        repo := repo.NewS3Client(
            config.AWS_S3_BUCKET_ACCESS_KEY,
            config.AWS_S3_BUCKET_SECRET_ACCESS_KEY,
            config.AWS_REGION,
        )

        // Create singed url used for uploading file
        presignedurl, err := repo.PutObject(config.AWS_BUCKET_NAME, 'filename', 60)
        if err != nil {
            log.Fatalf("Error generating presigned url for put object: %s", err)
        }
        // Uploading document to S3
        err = repo.UploadFile(file, presignedurl.URL)
        if err != nil {
            w.WriteHeader(http.StatusInternalServerError)
            fmt.Fprintf(w, "Error uploading image to S3: %v", err)
            return
        }

        w.WriteHeader(http.StatusOK)
        fmt.Fprintf(w, "Successfully uploaded image: %s", filename)
    }
}

We take the file from the request and decode it before uploading it to the bucket

Delete Action

func DeleteHandler(config utils.Config) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        // Read request body
        type RequestBody struct {
            Filename string `json:"filename"`
        }
        var data RequestBody
        if r.Body != nil {
            json.NewDecoder(r.Body).Decode(&data)
        }

        if data.Filename == "" {
            w.WriteHeader(http.StatusInternalServerError)
            fmt.Fprintf(w, "Filename must be provided")
            return
        }

        repo := repo.NewS3Client(
            config.AWS_S3_BUCKET_ACCESS_KEY,
            config.AWS_S3_BUCKET_SECRET_ACCESS_KEY,
            config.AWS_REGION,
        )

        presignedurl, err := repo.DeleteObject(config.AWS_BUCKET_NAME, data.Filename, 60)
        if err != nil {
            w.WriteHeader(http.StatusInternalServerError)
            log.Fatalf("Error generating presigned url for put object: %s", err)
            return
        }

        err = repo.DeleteFile(presignedurl.URL)
        if err != nil {
            w.WriteHeader(http.StatusInternalServerError)
            log.Fatalf("Error deleting file from s3 bucket: %v", err)
            return
        }

        w.WriteHeader(200)
        fmt.Fprintf(w, "Successfully deleted file: %s", data.Filename)
    }
}

The delete action works similar to the upload handler difference is we expect a json of the format

{
 "filename": "<filename-here>",
}

The file the specified name will be deleted.

Conclusion

The article has goes through the process of deleting and uploading files by using presigned url. Further Reading Material on this

Proxy Server in Go

sean — Sat, 20 Jan 2024 05:51:53 +0000

A proxy server is a system or router that provides a gateway between users and the internet. It helps prevent cyber attackers from entering a private network. There are two types of proxy servers Forward and Reverse .

Reverse Proxy

Reverse proxy servers are installed before the internal servers. Before connection to your internal server, the reverse proxy intercepts all requests from the client before they reach the internal server.

They are generally used to improve security and performance.

Forward Proxy

Forward proxy server are installed between the internal client and external servers. They are meant to intercept request made by internal clients and make requests on behalf of the clients to the external server. It acts on behalf of the client. They are generally used for security reasons.

In this article we are going to implement a reverse proxy server in go

To access the code in this articles go to github.

Setting Up Our Project

First we need to create all the required folder and files.

mkdir -p cmd internal/configs internal/server settings

cmd : It will contain entrypoint of our application.
interna/configs: Will hold code to load configurations values from the yaml file
internal/server : Will hold code to run the HTTP proxy server.
settings : Will contain configuration yaml file for our project.

Let's create all the files required for this project

touch Makefile settings/config.yaml cmd/main.go internal/configs/config.go internal/server/proxy_handler.go internal/server/healthcheck.go internal/server/server.go

Makefile: Allow us to easy run multiple bash commands at once
config.yaml: Hold reverse proxy configurations
main.go: Entrypoint of our program
config.go : Hold code that loads configurations
proxy_handlers.go: Contain functionthat create new proxy and proxy handler
healtcheck.go: Simple http handler to check if our server is running or not.
server.go: It will be used to create and run HTTP server.

Run HTTP Server

We will run the demo http servers in docker containers. Open the Makefile and pasted the following code that runs and stops the the three servers on port 9001, 9002, 9002. And also runs the proxy server.

## run: starts demo http services
.PHONY: run-containers
run-containers:
    docker run --rm -d -p 9001:80 --name server1 kennethreitz/httpbin
    docker run --rm -d -p 9002:80 --name server2 kennethreitz/httpbin
    docker run --rm -d -p 9003:80 --name server3 kennethreitz/httpbin

## stop: stops all demo services
.PHONY: stop
stop:
 docker stop server1
 docker stop server2
 docker stop server3

## run: starts demo http services
.PHONY: run-proxy-server
run-proxy-server:
 go run cmd/main.go

Now when you run make run and make stop respectively it will run and stop the docker containers.

Configure reverse proxy server

We will first configure our server using the config.yaml file.
The file needs to describe
name : Name of the resource we are targeting
endpoint : Endpoint the resource will be serving at
destination_url : Where the incoming request at endpoint will be forwarded to.

server:
  host: "localhost"
  listen_port: "8080"

resources:
  - name: Server1
    endpoint: /server1
    destination_url: "http://localhost:9001"
  - name: Server2
    endpoint: /server2
    destination_url: "http://localhost:9002"
  - name: Server3
    endpoint: /server3
    destination_url: "http://localhost:9003"

Loading Settings from config file

The code to load configurations will be in internal/config/config.go

The code uses Viper to load configuration files in the application.

package configs

import (
 "fmt"
 "strings"
 "github.com/spf13/viper"
)
type resource struct {
 Name            string
 Endpoint        string
 Destination_URL string
}
type configuration struct {
 Server struct {
  Host        string
  Listen_port string
 }
 Resources []resource
}
var Config *configuration
func NewConfiguration() (*configuration, error) {
 viper.AddConfigPath("settings")
 viper.SetConfigName("config")
 viper.SetConfigType("yaml")
 viper.AutomaticEnv()
 viper.SetEnvKeyReplacer(strings.NewReplacer(`.`, `_`))
 err := viper.ReadInConfig()
 if err != nil {
  return nil, fmt.Errorf("error loading config file: %s", err)
 }
 err = viper.Unmarshal(&Config)
 if err != nil {
  return nil, fmt.Errorf("error reading config file: %s", err)
 }
 return Config, nil
}

Creating a proxy and a proxy handler

In file internal/server/proxy_handler.go paste the following code

package server

import (
    "fmt"
    "net/http"
    "net/http/httputil"
    "net/url"
    "strings"
    "time"
)

func NewProxy(target *url.URL) *httputil.ReverseProxy {
    proxy := httputil.NewSingleHostReverseProxy(target)
    return proxy
}

func ProxyRequestHandler(proxy *httputil.ReverseProxy, url *url.URL, endpoint string) func(http.ResponseWriter, *http.Request) {
    return func(w http.ResponseWriter, r *http.Request) {
        fmt.Printf("[ PROXY SERVER ] Request received at %s at %s\n", r.URL, time.Now().UTC())
        // Update the headers to allow for SSL redirection
        r.URL.Host = url.Host
        r.URL.Scheme = url.Scheme
        r.Header.Set("X-Forwarded-Host", r.Header.Get("Host"))
        r.Host = url.Host

        // trim reverseProxyRouterPrefix
        path := r.URL.Path
        r.URL.Path = strings.TrimLeft(path, endpoint)

        // Note that ServeHttp is non blocking and uses a go routine under the hood
        fmt.Printf("[ PROXY SERVER ]Proxying request to %s at %s\n", r.URL, time.Now().UTC())
        proxy.ServeHTTP(w, r)
    }
}

The file has two functions NewProxy and ProxyRequestHandler

NewProxy: Function takes an urland returns a new http reverse proxy

ProxyRequestHandler : takes the proxy create by NewProxy, destination URL and endpoint and returns a http handler.

Creating HTTP server for our proxy

We will create the Run function that we called in main.go.
In the code snippet, we create the Run function which

Loads the configurations by calling NewConfiguration function from our config package.
Create a new server using net/http package.
Register route called /ping to call http handler ping
Iterate through the configuration resources describing the different endpoints and register into our router
We finally start the server

package server

import (
    "fmt"
    "net/http"
    "net/url"
    "reverse-proxy-learn/internal/configs"
)

// Run start the server on defined port
func Run() error {
    // load configurations from config file
    config, err := configs.NewConfiguration()
    if err != nil {
        return fmt.Errorf("could not load configuration: %v", err)
    }

    // Creates a new router
    mux := http.NewServeMux()

    // register health check endpoint
    mux.HandleFunc("/ping", ping)

    // Iterating through the configuration resource and registering them
    // into the router.
    for _, resource := range config.Resources {
        url, _ := url.Parse(resource.Destination_URL)
        proxy := NewProxy(url)
        mux.HandleFunc(resource.Endpoint, ProxyRequestHandler(proxy, url, resource.Endpoint))
    }

    // Running proxy server
    if err := http.ListenAndServe(config.Server.Host+":"+config.Server.Listen_port, mux); err != nil {
        return fmt.Errorf("could not start the server: %v", err)
    }

    return nil
}

We will also need to implement the /ping route handler. Paste the snippet below in the file internal/server/healthcheck.go

package server

import "net/http"

// ping returns a "pong" message
func ping(w http.ResponseWriter, r *http.Request) {
    w.Write([]byte("pong"))
}

Entrypoint for the Application

Inside cmd/main.go paste the following

package main

import (
    "log"

    "reverse-proxy-learn/internal/server"
)

func main() {
    if err := server.Run(); err != nil {
        log.Fatalf("could not start the server: %v", err)
    }
}

We can finally run our program buy calling server.Run() and handling any error if returned.

Running and testing Proxy

To run the program we need to first run our destination servers.

make run-containers

Once destnation servers are running run the proxy server

make run-proxy-server

which run the code.

To test that the proxy works.
Run the snippet below to attempt to connect to server 1

curl -I http://localhost:8080/server1

The response should be in this format

HTTP/1.1 200 OK
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Content-Length: 9593
Content-Type: text/html; charset=utf-8
Date: Sat, 20 Jan 2024 05:24:51 GMT
Server: gunicorn/19.9.0

We can verify that proxy as intended by looking at the servers logs

[ PROXY SERVER ] Request received at /server1 at 2024-01-20 05:24:30.616616225 +0000 UTC
[ PROXY SERVER ]Proxying request to http://localhost:9001 at 2024-01-20 05:24:30.616692726 +0000 UTC
[ PROXY SERVER ] Request received at /server1 at 2024-01-20 05:24:51.927329445 +0000 UTC
[ PROXY SERVER ]Proxying request to http://localhost:9001 at 2024-01-20 05:24:51.927399846 +0000 UTC

To take a run the code locally fork the code from github.com