DEV Community: Zaffere

Golang Functional Options Pattern

Zaffere — Thu, 18 May 2023 09:10:23 +0000

The Functional Options Pattern is Creational Design Pattern which allows you to build complex structs using a constructor which takes in zero or more functions as arguments.

Sometimes we find ourselves having to deal with a domain object that takes in optional fields:

type Student struct {
  Name string
  Age int
  Address *string // optional
  PhoneNumber *int // optional
}

func NewStudent(name string, address string, age int, phoneNumber int)

If all fields are required, instantiating a new student each time would be neat and easy to read:

student := NewStudent(name string, address string, age int, phoneNumber int)

Even if we needed 10x students, using the constructor method as above would be we all well too:

student1 := NewStudent("Jon", "Kings Road", 12, 12345)
student2 := NewStudent("Tim", "Sixth Ave", 12, 11223)
student3 := NewStudent("Sara", "Orchard Road", 12, 22112)
student4 := NewStudent("Vanessa", "River Valley", 13, 43321)
student5 := NewStudent("Steve", "Killiney Road", 12, 55423)

However, if we had 100 students new students and some provided their address and phone numbers while some don't, creating new students would get real messy real fast:
(With only 4 fields in the struct, this doesn't seem like a real problem however, in production, your domain model typically consists of more than just 4 fields which makes it easier to make mistakes and harder to read)

student1 := NewStudent("Jon", "River Valley", 12, 0)
student2 := NewStudent("Alex", "", 10, 22222)
student3 := NewStudent("Jones", "Kings Road", 12, 0)
student4 := NewStudent("Rachel", "", 12, 0)
student5 := NewStudent("Esther", "Killiney Road", 13, 55555)
...

To help with this, we can use whats called an Optional method/function pattern

Create a custom Type of type:function that takes in a pointer to the struct you want to build
Expose a method that takes in those optional values and returns a function with the same function signature of the custom function type, this returned function of type:Option function would be responsible for adding those optional values to the struct

// Custom Type which is a function that takes in a pointer of the Student struct
type OptionalStudentValues func(s *Student)

func NewStudent(name string, age int, opts ...OptionalStudentValues) *Student {
    student := &Student{
        Name: name,
        Age:  age,
    }

    // Checks id there are optional values needed to be passed to the new Student struct
    if len(opts) > 0 {
        for _, opt := range opts {
            // execute the function retruned by the exposed methods, passing in the student struct
            opt(student)
        }
    }
    return student
}

// Implementation of the Optional function
func WithAddress(addr string) OptionalStudentValues {
    return func(s *Student) {
        s.Address = &addr
    }
}

func WithPhoneNumber(num int) OptionalStudentValues {
    return func(s *Student) {
        s.PhoneNumber = &num
    }
}

func main() {
    studentAddr := "Address string"
    studentPhoneNumber := 12345

    //  With optional fields
    s := NewStudent("Tim", 20, WithAddress(studentAddr), WithPhoneNumber(studentPhoneNumber))
    // No optional fields ...
    s2 := NewStudent("Mary", 12)

    log.Printf("This is the newly created student with optional fields : %+v", s)
    log.Printf("This is the newly created student with no optional fields : %+v", s2)
}

Scaling PostgreSQL read performance

Zaffere — Tue, 11 Apr 2023 15:32:10 +0000

Scaling your PostgreSQL read performance

We've all been at the situation where we find our database queries takes way too long to execute. This typically happens when our tables or columns grows too large and which causes the table operational cost to increase as well. A common approach here would be to simply add a read replica and redirect all reads to this replica. This approach works by redirecting all read queries to a replicated database while our writes goes to the main/master database.

However, there are some cost effective solutions we might want to consider before spinning up a read replica.

We will take a look at two methods that, when used together, has the potential to increase our read capacities significantly.

Indexing

First we will take a look at indexing.

What is indexing?
Indexing is a technique that sorts data in a table into a data structure that is ordered and allows fast retrieval speed. The data structure used is typically a B-tree where it creates a multi-level tree structure that breaks a database down into fixed-size blocks or pages.

For example, remember our good old friend, Yellow Pages? (Yellow Pages is a telephone directories of businesses used in the 90s) Lets say you want to find the address of a specific company among thousands of different companies. Now if the listings in Yellow Pages weren't ordered in any form, you can expect to flip through each page and check every company until you find the address you were looking for. This is called a sequential scan in PostgreSQL and as you can imagine, depending on the size of your database, this would take up a significant amount of time.

To solve this problem, we can use Indexes. With Indexes, we can sort our database table in an orderly manner. In the Yellow Pages example, we can order all companies in alphabetical order and to find the address of Tesla, we know we can start searching on the pages where company names starts with a "T". In doing so we are way more efficient in finding what we need as we know we can skip the first 20 alphabets (which may in itself involve multiple pages).

How to create an index in PostgreSQL
Let's see the theory above in action.

We will create a table that stores details of multiple companies and run queries with and without creating an index on them to compare the execution time:

First lets create a new database in PostgreSQL.

Note: I used NodeJS, Sequelize and FakerJS to quickly seed some random data into my database.

Creating the table

module.exports = {
  up: async (queryInterface, Sequelize) => {
    await queryInterface.createTable('companies', {
      id: {
        type: Sequelize.INTEGER,
        allowNull: false,
        autoIncrement: true,
        primaryKey: true,
      },
      name: Sequelize.TEXT,
      category: Sequelize.TEXT,

    });

  },

  down: async (queryInterface, Sequelize) => {
    await queryInterface.dropTable('companies');
  }
};

Seeding the table

const {faker} = Faker

const generateRandomCategory = () => {
  const categories = ['Tech', 'Health Care', 'Food & Beverage', 'Tourism'];
  const randomIdx = Math.round(Math.random() * 3)
  return categories[randomIdx]
}

module.exports = {
  async up (queryInterface, Sequelize) {

    const companyList = [];

    for (i = 0; i < 10000; i ++) {
      const companyName = faker.company.name
      const newCompany = {
        name: companyName(),
        category: generateRandomCategory(),
      }
      companyList.push(newCompany)
    }

    await queryInterface.bulkInsert('companies', companyList)

Looking at our current database, if we ran a SELECT * FROM companies WHERE name='Rau Group' we'll see that we have an entry where the company name == 'Rau Group'

Now lets run an EXPLAIN ANALYZE command to see the planning and execution time PostgresSQL takes to select all from companies where name is Rau Group:

Based off the results, we can see that PostgreSQL had to make a Sequential scan (go through each entry in the table) and took 3.908ms to plan and 70.679ms to execute the query. Not too bad..

Let's see if we can do better?

Lets create an index on the name column and run the query again:

Creating an index in PostgreSQL is quite straight forward.
We can simply use PostgreSQL's CREATE INDEX command.
CREATE INDEX <index_name> ON companies (table_column):

And now we run the same query:

Looking at the results now, we see that PostgreSQL uses an Index Scan and took 0.602ms and 2.418ms to plan and execute respectively.

As mentioned, database indexes are stored in a B-Tree data structure and the Index Scan performs a B-tree traversal, walks through the leaf nodes to find all matching entries, and fetches the corresponding table data.
This improves the query by more than half and hence leads to better database performance.Indexing works well if we have identified frequently queried columns in our table.

Cons of using indexes

One of the downsides of creating an index in PostgreSQL is that indexes slow down data entry or modification. Whenever a new row is added that contains a column with an index, that index is modified as well
Increased disk space

Partitioning

Now we'll take a look at partitioning in PostgreSQL.
There are two methods of partitioning (Horizontal/Vertical partitioning) but we will focus our attention on Horizontal Partitioning here.

What is a partition?
To partition means to slice our single large table horizontally (by rows) or vertically (by columns) and splitting each half(partition) into multiple smaller tables identified by a specified key. As you can already imagine, each table has halved in size improving our read queries quite a tiny bit already.

When our tables in our database gets too large, performance and scaling are affected. As table size increase, more data scanning, swapping pages to memory and other table operation cost also increases. Partitioning helps by dividing our large table into smaller tables (categorised by your desired field/key) thus reducing table scans and memory swap problems which ultimately increases performance.

In our example above with the Yellow pages, seeing that each company has a category associated with it (and assuming that we have identified this as part of our data access pattern), we can go a step further in improving our read queries by partitioning the companies table into each category.

Methods of PostgreSQL partition
PostgreSQL allows two types of partitions: Declarative & Inheritance Partitioning. Both has their limitations but here we will use Declarative Partitioning.

There are multiple methods of partitions in PostgreSQL(of which, we will use List):

Range Partitioning
The table is partitioned into “ranges” defined by a key column or set of columns, with no overlap between the ranges of values assigned to different partitions.
List Partitioning
The table is partitioned by explicitly listing which key value(s) appear in each partition.
Hash Partitioning
The table is partitioned by specifying a modulus and a remainder for each partition. Each partition will hold the rows for which the hash value of the partition key divided by the specified modulus will produce the specified remainder

Creating a partition
In most real-life scenarios, we often would be implementing partitioning on an already existing database and table. Here arises the problem of database availability. While migrating our tables to partitions with live data, because this process might take a long period of time and the fact that database locking will be invoked on the transaction, this could cost significantly on database availability. With this method, I see the potential of minimising database locks hence keeping availability

NOTE: There are other strategies to tackle this problem, I am following this approach based off my own research and knowledge. Be sure to do your own research before implementing anything at this scale in production

This is a 5 step process we performed in a transaction. If any of the steps fails, then our database would rollback to its original state, keeping everything safe and sound.

Step 1
Rename the original (going to be legacy/old table) table. We do this so that we can create our partitioned table with the same name as before, allowing our application to still make queries from this table.

ALTER TABLE companies RENAME TO companies_old;

Step 2
Create the new partitioned table with the old name. Using the old name for the same reasons as mentioned above in Step 1. Here we partition by List with value 'category'

CREATE TABLE companies (
        id SERIAL,
        name TEXT,
        category TEXT
) PARTITION BY LIST (category);

Step 3
Create the partition tables to hold data specific to each category

CREATE TABLE tech_companies PARTITION OF companies FOR VALUES IN ('Tech');
CREATE TABLE health_care_companies PARTITION OF companies FOR VALUES IN ('Health care');
CREATE TABLE fnb_companies PARTITION OF companies FOR VALUES IN ('Food & Beverage');
CREATE TABLE tourism_companies PARTITION OF companies FOR VALUES IN ('Tourism');

Step 4
Store each partition's data in a temporary table which will be inserted into their respective partition table afterwards

WITH tech_companies AS (
        SELECT * FROM companies_old WHERE category = 'Tech'
), fnb_companies AS (
        SELECT * FROM companies_old WHERE category = 'Food & Beverage'
), health_care_companies AS (
        SELECT * FROM companies_old WHERE category = 'Health Care'
), tourism_companies AS (
        SELECT * FROM companies_old WHERE category = 'Tourism'
)

Step 5
Finally, we insert respective data into the partition table

INSERT INTO companies SELECT * FROM tech_companies;
INSERT INTO companies SELECT * FROM fnb_companies;
INSERT INTO companies SELECT * FROM health_care_companies;
INSERT INTO companies SELECT * FROM tourism_companies;

Here is the full DDL:

BEGIN;
-- rename the legacy/old table
ALTER TABLE companies RENAME TO companies_old;

-- Create a new partitioned table with the legacy/old table's name to ensure application compatibility moving forward from here
CREATE TABLE companies (
        id SERIAL,
        name TEXT,
        category TEXT
) PARTITION BY LIST (category);

-- Create a new partition table for each category values
CREATE TABLE tech_companies PARTITION OF companies FOR VALUES IN ('Tech');
CREATE TABLE health_care_companies PARTITION OF companies FOR VALUES IN ('Health care');
CREATE TABLE fnb_companies PARTITION OF companies FOR VALUES IN ('Food & Beverage');
CREATE TABLE tourism_companies PARTITION OF companies FOR VALUES IN ('Tourism');

-- Select data from legacy/old table to be moved into partition tables
WITH tech_companies AS (
        SELECT * FROM companies_old WHERE category = 'Tech'
), fnb_companies AS (
        SELECT * FROM companies_old WHERE category = 'Food & Beverage'
), health_care_companies AS (
        SELECT * FROM companies_old WHERE category = 'Health Care'
), tourism_companies AS (
        SELECT * FROM companies_old WHERE category = 'Tourism'
)

-- Insert data to partition tables
INSERT INTO companies SELECT * FROM tech_companies;
INSERT INTO companies SELECT * FROM fnb_companies;
INSERT INTO companies SELECT * FROM health_care_companies;
INSERT INTO companies SELECT * FROM tourism_companies;
COMMIT;

Inspecting our new partitioned tables
Now if executing the above commands succeeds, we can go ahead and inspect our table. In your PSQL terminal run:

\d+ companies

Looking at the details, we see that our "companies" table is in fact a partitioned table, with a partition key of "LIST" grouped by the "category" column and that it has 4 total partitions.

We have successfully partitioned our one large table.

Now lets run some tests to see how this improves our read queries!

Analysing the query
Now running the same SELECT * WHERE name=Rau Group command, let's see what the results gives us.

Keep in mind that we now have only partitioned our tables and have yet to add indexes to our partitioned tables. With that, PostgreSQL took 27.435ms & 11.117ms to plan and execute respectively. Still way quicker than querying from a regular table.

Lets see how much more improvement we can make with indexes in our partitioned tables.
Creating an index on a partitioned table is the same as with a regular table. We will create an index on company's name based off our knowledge of the data access pattern:

CREATE INDEX tech_companies_name_idx ON tech_companies (name);
CREATE INDEX fnb_companies_name_idx ON fnb_companies (name);
CREATE INDEX health_care_companies_name_idx ON health_care_companies (name);
CREATE INDEX tourism_companies_name_idx ON tourism_companies (name);

Now let's make the same query and take a look at the results:

We can see that we have gained a slight improvement here with 1.76ms execution time.

Partitioning can help greatly improve the speed of read queries on really large tables if done correctly by splitting groups of data into smaller tables which in turn allows PostgreSQL's table operation cost to significantly reduce. Adding indexes to partitioned tables can further improve this. Whats important is that we have to identify our data access patterns, properly normalise our tables and learn to find grouping patterns in our data.

Cons of partitioning

Lack of fault tolerance. Unlike distributed replication, if this instance fails, then your database is unavailable
Foreign keys referencing partitioned tables, as well as foreign key references from a partitioned table to another table, are not supported because primary keys are not supported on partitioned tables
If the number of partitions is out-grown, you have the same issue with the partitions. A full re-partitioning would be needed to increase the partition count

Please feel free to leave any comments or corrections on my implementations ✌️

Deployment Strategies

Zaffere — Sun, 29 Jan 2023 14:07:25 +0000

Choosing the right deployment strategy can be tricky especially when considering zero downtime for our end users.

Traditional deployments were typically done in the middle of the night (when we assume most of our users are fast asleep). It requires us to take down our servers, add our new feature/code and restart our servers again. Although this strategy could work for some of us and its the simplest, you can already see 2 obvious trade-offs.

Deployments must happen at a specific time (typically dead in the night!)
Deployments may take a long time, so depending on its duration, NO users are able to use our software as we are deploying (downtime!)

Deployment Strategies

Here I will share 3 of the most common deployment strategies together with its trade-offs.

Blue/Green Deployment

With the Blue/Green deployment strategy, we have two identical environments deployed on our servers sitting behind a load balancer. Each environment is independent of each other but share the same exact configurations.

During our pre-release phase, either one of these environments would hold the current version with the previously up to date features. All our services are served from this server and all our users are routed to this server from the load balancer.
(Green is the most updated in this case)

Now we have a new feature ready to go to production. Instead of deploying our new feature to the green server (currently live), we deploy our new feature to the blue server, the one that is currently idle. This means we now have two servers that are "live", with our green server serving our users in production with the old features while the blue server is still sitting idle with our latest features.

We can complete our final testing on the blue server and once done, we can reconfigure our load balancers to route production traffic to the blue server instead.

Live users can now see the new feature and good news was that we never had to bring our servers down when adding new features.

And if there were any bugs discovered in our newly live feature, we can simply reconfigure our load balancers to route users to the the previous server

Pros:

Zero/Minimal downtime
Fairly straight forward to implement
Easy rollbacks if new feature fails
Allows for testing in a live environment

Cons:

May be expensive having two live servers
Can get tricky at scale

Canary Deployment

Canary deployment are very similar to blue-green deployment, but uses a slightly different approach. Here we only have one environment but multiple nodes. This deployment strategy is common when using Kubernetes but can also be done without.

Let's take for example a 3 node replica of a service. We have 3 instances running live with its load distributed evenly among them.

They each have the same exact version of the service.

This time, when we have a new feature available to go live, we deploy it to a single node. We then route only a small percentage of our users to the newly deployed feature.

This gives us the ability to let a very small percentage of our users do a live "test" of the feature before rolling it out to 100% of our users.

Now when we have determined that the new feature is safe to use with no bugs detected, we deploy our new feature to all other nodes and simply split the load of the service evenly again.

Pros

Zero/minimal downtime
Flexibility to experiment with new features
Less costly than blue-green method
Quick rollback

Cons

Can get complex to set up
Needs observability metrics

Feature Flagging

Feature flagging is another common concept when it comes to deploy/release of new features.

With feature flagging (also commonly known as feature toggles) you can toggle on/off any feature available to users during runtime. This can be as simple as conditionals determining the code path that will be executed.

const features = {
    'newFeature': true,
}

if (features.newFeature) {
    RenderNewFeature()
} else {
    RenderOldFeature()
}

Feature flagging allows you to bundle multiple releases into a single deploy but based on flags, only made available to a subset of users (or none at all depending on your requirements) hence in a way, reduces the total number of deployments.

Control of when and who can use your latest feature is entirely up to you now with a simple toggle.

Pros

Adding/removing a feature is extremely simple
Reduced deployments as you can now deploy multiple features at once but only release those required at the moment
No need to maintain long-lived feature branches. We can directly push our new features to the main branch and toggle wether or not to release it
If a feature is causing problems, flipping the flag off will disable it, solving the problem immediately without the need to roll back a complex release

Cons

You may need to know in advanced what features are to be included per deployment
If not implemented properly, code can get chunky
Adding more feature flags makes testing and management exponentially more difficult over time. It becomes very hard to tell which flags are necessary or obsolete

Summary

There is no one-fits-all solution and as with everything, there are trade-offs that we have to consider before choosing the right deployment strategy. From how we are serving our applications to the problems that we are trying to solve. These are just some of the solution I have worked with and thought it'd be nice to share with the community.

Golang GRPC Implementation

Zaffere — Wed, 18 Jan 2023 05:01:47 +0000

💡 This page is a rough guide on how to setup a GRPC and REST endpoint using a multiplexer

1. Creating the entry point

📌 Initialising main.go

Main.go

Main.go is where we define our application server.

This is the entry point to our API

We can go ahead and set up:

Our Application Server
GRPC server
Gin framework

We use CMUX to handle multiplexing in our endpoint. This basically listens for requests and based off it’s headers (HTTP1, HTTP2..), it serves the request to either our HTTP or GRPC servers

Application Server

$ mkdir cmd/main.go

package main

import (
    "fmt"
    "log"
    "net"
    "net/http"
    "strings"

    "github.com/ZAF07/tigerlily-e-bakery-inventories/api/router"
    "github.com/ZAF07/tigerlily-e-bakery-inventories/api/rpc"
    "github.com/ZAF07/tigerlily-e-bakery-inventories/internal/db"
    "github.com/ZAF07/tigerlily-e-bakery-inventories/internal/pkg/env"
    "github.com/ZAF07/tigerlily-e-bakery-inventories/internal/pkg/logger"
    "github.com/ZAF07/tigerlily-e-bakery-inventories/internal/service/inventory"
    "github.com/gin-gonic/gin"
    "github.com/soheilhy/cmux"
    "google.golang.org/grpc"
)

func main() {
    logs := logger.NewLogger()
    logs.InfoLogger.Println("Starting up server ...")

    // Set ENV vars
    env.SetEnv()

    // Spin up the main server instance
    lis, err := net.Listen("tcp", ":8000")
    if err != nil {
        logs.ErrorLogger.Println("Something went wrong in the server startup")
        log.Fatalf("Error connecting tcp port 8000")
    }
    logs.InfoLogger.Println("Successfull server init")

    // Start a new multiplexer passing in the main server
    m := cmux.New(lis)

    // Listen for HTTP requests first
    // If request headers don't specify HTTP, next mux would handle the request
    httpListener := m.Match(cmux.HTTP1Fast())
    grpclistener := m.Match(cmux.Any())

    // Run GO routine to run both servers at diff processes at the same time
    go serveGRPC(grpclistener)
    go serveHTTP(httpListener)

    fmt.Printf("Inventory Service Running@%v\n", lis.Addr())

    if err := m.Serve(); !strings.Contains(err.Error(), "use of closed network connection") {
        log.Fatalf("MUX ERR : %+v", err)
    }

}

GRPC Server

Here we define a GRPC server to serve request

// cmd/main.go
package main

import (
    "log"
    "net"

    "github.com/ZAF07/tigerlily-e-bakery-inventories/api/rpc"
    "github.com/ZAF07/tigerlily-e-bakery-inventories/internal/db"
    "github.com/ZAF07/tigerlily-e-bakery-inventories/internal/pkg/logger"
    "github.com/ZAF07/tigerlily-e-bakery-inventories/internal/service/inventory"
    "google.golang.org/grpc"
)

// GRPC Server initialisation
func serveGRPC(l net.Listener) {
    grpcServer := grpc.NewServer()

    // Register GRPC stubs (pass the GRPCServer and the initialisation of the service layer)
    rpc.RegisterInventoryServiceServer(grpcServer, inventory.NewInventoryService(db.NewDB()))

    if err := grpcServer.Serve(l); err != nil {
        log.Fatalf("error running GRPC server %+v", err)
    }
}

Here we define our Gin Framework server

Gin Server

// cmd/main.go

import (
    "log"
    "net"

    "github.com/gin-gonic/gin"
)

// HTTP Server initialisation (using gin)
func serveHTTP(l net.Listener) {
    h := gin.Default()
    router.Router(h)
    s := &http.Server{
        Handler: h,
    }
    if err := s.Serve(l); err != cmux.ErrListenerClosed {
        log.Fatalf("error serving HTTP : %+v", err)
    }

Complete example of main.go

// cmd/main.go
package main

import (
    "fmt"
    "log"
    "net"
    "net/http"
    "strings"

    "github.com/ZAF07/tigerlily-e-bakery-inventories/api/router"
    "github.com/ZAF07/tigerlily-e-bakery-inventories/api/rpc"
    "github.com/ZAF07/tigerlily-e-bakery-inventories/internal/db"
    "github.com/ZAF07/tigerlily-e-bakery-inventories/internal/pkg/env"
    "github.com/ZAF07/tigerlily-e-bakery-inventories/internal/pkg/logger"
    "github.com/ZAF07/tigerlily-e-bakery-inventories/internal/service/inventory"
    "github.com/gin-gonic/gin"
    "github.com/soheilhy/cmux"
    "google.golang.org/grpc"
)

func main() {
    logs := logger.NewLogger()
    logs.InfoLogger.Println("Starting up server ...")

    // Set ENV vars
    env.SetEnv()

    // Spin up the main server instance
    lis, err := net.Listen("tcp", ":8000")
    if err != nil {
        logs.ErrorLogger.Println("Something went wrong in the server startup")
        log.Fatalf("Error connecting tcp port 8000")
    }
    logs.InfoLogger.Println("Successfull server init")

    // Start a new multiplexer passing in the main server
    m := cmux.New(lis)

    // Listen for HTTP requests first
    // If request headers don't specify HTTP, next mux would handle the request
    httpListener := m.Match(cmux.HTTP1Fast())
    grpclistener := m.Match(cmux.Any())

    // Run GO routine to run both servers at diff processes at the same time
    go serveGRPC(grpclistener)
    go serveHTTP(httpListener)

    fmt.Printf("Inventory Service Running@%v\n", lis.Addr())

    if err := m.Serve(); !strings.Contains(err.Error(), "use of closed network connection") {
        log.Fatalf("MUX ERR : %+v", err)
    }

}

// GRPC Server initialisation
func serveGRPC(l net.Listener) {
    grpcServer := grpc.NewServer()

    // Register GRPC stubs (pass the GRPCServer and the initialisation of the service layer)
    rpc.RegisterInventoryServiceServer(grpcServer, inventory.NewInventoryService(db.NewDB()))

    if err := grpcServer.Serve(l); err != nil {
        log.Fatalf("error running GRPC server %+v", err)
    }
}

// HTTP Server initialisation (using gin)
func serveHTTP(l net.Listener) {
    h := gin.Default()
    router.Router(h)
    s := &http.Server{
        Handler: h,
    }
    if err := s.Serve(l); err != cmux.ErrListenerClosed {
        log.Fatalf("error serving HTTP : %+v", err)
    }
}

2. Defining our routes

This is where we map incoming traffic to the controller layer depending on the path.

The routers would send the incoming request to the appropriate controllers which would map them to its respective services

$ mkdir api/router/router.go

// api/router/router.go

package router

import (
    "github.com/ZAF07/tigerlily-e-bakery-inventories/api/controller"
    "github.com/ZAF07/tigerlily-e-bakery-inventories/internal/pkg/middleware"
    "github.com/gin-contrib/cors"
    "github.com/gin-gonic/gin"
)

// Router method gets imported in main.go
func Router(r *gin.Engine) *gin.Engine {

    // Set CORS config
    r.Use(cors.New(cors.Config{
        AllowCredentials: false,
        // Allowing only localhost:8080 access
        AllowOrigins: []string{"http://localhost:8080"},
        AllowMethods: []string{"GET", "POST", "PUT", "DELETE", "OPTION", "HEAD", "PATCH", "COMMON"},
        AllowHeaders: []string{"Content-Type", "Content-Length", "Authorization", "accept", "origin", "Referer", "User-Agent"},
    }))

// Use ustom middleware for all routes
    r.Use(middleware.CORSMiddleware())

    // Get all inventory
    // Get all invetory by type
    // inventoryAPI := new(controller.InventoryAPI)
    inventoryAPI := controller.NewInventoryAPI()
    inventory := r.Group("inventory")
    {
        inventory.GET("", inventoryAPI.GetAllInventories)
        inventory.GET("/:type",inventoryAPI.GetInventoryByType)

        // Get user details and past pre-checkout cart item
        cache := r.Group("cache")
        cacheAPI := new(controller.CacheAPI)
        {
            cache.GET("/:user_uuid", cacheAPI.GetUserDetails)
        }
    }

    return r
}

3. Connecting to Database with GORM:

Next we would want somewhere we can persist data to.

We will use an ORM library for GO to simplify things.

Create a package to write code to connect to our database

$ mkdir internal/db

Create a file to store the code for the package

$ touch internal/db/db.go

Create connection, store connection object in a struct or variable and export to packages that needs to use the Database

// db.go

import (
    // Go standard libraries
    "fmt"
    "log"

    // Importing required packages
    "github.com/ZAF07/tigerlily-e-bakery-inventories/internal/pkg/env"
    "github.com/ZAF07/tigerlily-e-bakery-inventories/internal/pkg/logger"
    "github.com/jinzhu/gorm"
    _ "github.com/jinzhu/gorm/dialects/postgres"
)

// Init a variable to hold the connection
var ORM *gorm.DB

// ...or a struct
type Db struct {
    db *gorm.DB
}

// Init method to be called whenever we want to create a database instance
func NewDB() (*gorm.DB) {
    connectDB()
    // If using variable
    return ORM

    // If using struct
    return &Db{
            db: ORM
        }
}

// Function to make the connection and return the connection object
func connectDB() () {

    logs := logger.NewLogger()
    // Make the connection
    db, err := gorm.Open("postgres",  env.GetDBEnv())
    if err != nil {
        logs.ErrorLogger.Printf("Couldn't connect to Database %+v", err)
        log.Fatalf("Error connectiong to Database : %+v", err)
    }
    logs.InfoLogger.Println("Successfully connected to Database")
    fmt.Println("SUCCEEDED CONNECTING TO DB")

    // Store connection object in variable
    ORM = db
}

4. Defining our Controller layer:

The controller is the initial layer in our endpoint to interact with the request made from the client.

Here we will serialise and structure the request data and hand it over to the service layer to handle our business logic

Create the controller package:

// api/controller/inventory.go
package controller

import (
    "context"
    "fmt"
    "log"
    "net/http"
    "strconv"

    "github.com/ZAF07/tigerlily-e-bakery-inventories/api/rpc"
    "github.com/ZAF07/tigerlily-e-bakery-inventories/internal/db"
    "github.com/ZAF07/tigerlily-e-bakery-inventories/internal/pkg/logger"
    "github.com/ZAF07/tigerlily-e-bakery-inventories/internal/service/inventory"
    "github.com/gin-gonic/gin"
    "github.com/jinzhu/gorm"
)

// A struct (object) representing the controller with all its methods
type InventoryAPI struct {
    db *gorm.DB
    logs logger.Logger
}

// Init the DB here (open a connection to the DB) and pass it along to service and repo layer later on
func NewInventoryAPI() *InventoryAPI {
    return &InventoryAPI{
        db: db.NewDB(),
        logs: *logger.NewLogger(),
    }
}

// One of many Controller method
// Takes in a pointer to gin.Context, where the client request lives
func (a InventoryAPI) GetAllInventories(c *gin.Context) {

    // Custom logger
    a.logs.InfoLogger.Println(" [CONTROLLER] GetAllInventories Request recieved")

    // Serialize and structure incoming data before handing them over to the service layer
    limit , lErr := strconv.Atoi(c.Query("limit"))
    if lErr != nil {
        a.logs.ErrorLogger.Printf("Error converting limit query string to int %+v", lErr)
    }
    offset , oErr := strconv.Atoi(c.Query("offset"))
    if oErr != nil {
        a.logs.ErrorLogger.Printf("Error converting offset query string to int %+v", oErr)
    }

    // Construct the request object 
    req := rpc.GetAllInventoriesReq{
        Limit: int32(limit),
        Offset: int32(offset),
    }

    // Init a new service instance passing the DB instance (service will pass this DB inatance to the repo layer later on)
    service := inventory.NewInventoryService(a.db)
    fmt.Printf("service %+v", service)
    ctx := context.Background()
    resp, err := service.GetAllInventories(ctx, &req)
    if err != nil {
        log.Fatalf("Error getting response from service layer")
    }

    // Wait for service layer to respond with data before sending them over to the client
    c.JSON(http.StatusOK,
        gin.H{
            "message": "Success",
            "status": http.StatusOK,
            "data": resp,
        },
    )
}

// Another controller method
func (a InventoryAPI) GetInventoryByType(c *gin.Context) {
    c.JSON(http.StatusOK,
        gin.H{
            "message": "Success",
            "status": http.StatusOK,
            "data": "This is all the pastries by type",
        },
    )
}

5. Defining our Service Layer

The service layer is the heart of our API.

This is where we run our main business logic.

Like how the controller calls the service layer, the service layer calls the repo layer and waits for data returned

Define our service layer:

$ mkdir internal/service/inventory.go

// internal/service/inventory.go
package inventory

import (
    "context"
    "log"

    "github.com/ZAF07/tigerlily-e-bakery-inventories/api/rpc"
    "github.com/ZAF07/tigerlily-e-bakery-inventories/internal/pkg/logger"
    "github.com/ZAF07/tigerlily-e-bakery-inventories/internal/repository/inventory"
    "github.com/jinzhu/gorm"
    _ "github.com/jinzhu/gorm/dialects/postgres"
)

// Data structure representing the service layer. Down below we create methods for this struct to receive
// This struct typically holds the repo layer instance, only made available to use after initialising the repo, passing in the DB instance first
type Service struct {
    db *gorm.DB
    inventory inventory.InventoryRepo
    logs logger.Logger
    rpc.UnimplementedInventoryServiceServer
}

// This gurantees that Service struct implements the interface
var _ rpc.InventoryServiceServer = (*Service)(nil)

// We initialise a new repo instance at the same time we initialise the service layer
// THE CONTROLLER SHOULD START THE DB INIT AND PASS THE INSTANCE TO SERVICE AND SERVICE TO REPO !!
func NewInventoryService(DB *gorm.DB) *Service {
    return&Service{
        db: DB,
        // Init the repo layer
        inventory: *inventory.NewInventoryRepo(DB),
        logs: *logger.NewLogger(),
    }
}

func (srv Service) GetAllInventories(ctx context.Context, req *rpc.GetAllInventoriesReq) (resp *rpc.GetAllInventoriesResp, err error) {
    srv.logs.InfoLogger.Println(" [SERVICE] GetAllInventories Running service layer")

    // Use the repo instance (the repo should be tied to this service struct field)
    in, err := srv.inventory.GetAllInventories(req.Limit, req.Offset)
    if err != nil {
        srv.logs.ErrorLogger.Printf("Database Error : %+v", err)
        log.Fatalf("Database err %+v", err)
    }

    // Run logic with the data returned by GORM
    i := []*rpc.Sku{}
    for _, sku := range in {
        i = append(i, &rpc.Sku{
            Name: sku.Name,
            Price: sku.Price,
            SkuId: sku.SkuID,
            ImageUrl: sku.ImageURL,
            Type: sku.Type,
            Description: sku.Description,
        })
    }

    // Return results back to controller layer
    resp = &rpc.GetAllInventoriesResp{
        Inventories: i,
    }

    return
}

6. Define our Repository layer

The repository layer talks DIRECTLY to the DB, returning all data returned by DB to the service layer. Service layer runs logic and returns data back to the controller layer

Define our repository layer:

$ mkdir internal/repository/inventory.go

// internal/repository/inventory.go
package inventory

import (
    "fmt"

    "github.com/ZAF07/tigerlily-e-bakery-inventories/internal/models"
    "github.com/jinzhu/gorm"
    _ "github.com/jinzhu/gorm/dialects/postgres"
)

// This gurantees that Repo struct implements the interface
var _ inventoryRepo = (*InventoryRepo)(nil)

// Create an interface to prevent unwanted use of these methods
type inventoryRepo interface {
    GetAllInventories(limit, offset int32) (items []*models.Skus, err error)
}

type InventoryRepo struct {
    db *gorm.DB
}

// Receives the db instance as argument and sets it in the struct before returning the struct itself
func NewInventoryRepo(db *gorm.DB) *InventoryRepo {
    return &InventoryRepo{
        db: db,
    }
}

// Makes the query to DB via GORM methods, returns data to service layer
func (m InventoryRepo) GetAllInventories(limit, offset int32) (items []*models.Skus, err error) {
    fmt.Println("HELLO?")
    m.db.Debug().Find(&items)
    return
}

What is a browser?

Zaffere — Wed, 18 Jan 2023 04:55:09 +0000

What is a web Browser

In most devices you have, be it a phone (android or mac), a tablet or a laptop, you will find a web browser installed. A web browser is a software we use to browse the internet. There are 5 major browsers: Chrome, Internet Explorer, Firefox, Safari and Opera. (maybe 4, excluding IE).

The browser’s main functionality is to fetch and display resources we request from the internet like Facebook or youtube. Resources can also be a simple image, pdf file or even a JSON file.

Each browser has a few main components common to each other:

1) User Interface

This are GUI (Graphical User Interface) users interact with. Stuff like the address bar, back/forward buttons, bookmarking menu etc..

2) Browser Engine

Think of this as the controller. It comprises of a few other engines (rendering engine & javascript engine) to render and style the webpage you see upon receiving the response of your request

3) Rendering engine

Responsible for the layout of the webpage. If the requested content was a HTML & CSS file, the rendering engine would know to parse the response with HTML & CSS and displays the result to the user. This is where the DOM (document object model) is constructed & styling is made.

4) Javascript Engine

Most response comes with some javascript code to create dynamic components or to fetch additional content from the web. The javasript engine is responsible for interpreting and executing the javascript code. The primary difference between a Rendering engine and a JavaScript engine lies around the dependency with browser. The rendering engine is tightly coupled with browser engine, on the other hand, a JavaScript engine can be worked upon even without a browser.

5) Networking

The networking component is responsible for making HTTP requests. Read more here https://hpbn.co/primer-on-browser-networking/

6) UI backend

used for drawing basic widgets like combo boxes and windows. This backend exposes a generic interface that is not platform specific. Underneath it uses operating system user interface methods.

7) Data Storage

This is a persistence layer. The browser may need to save all sorts of data locally, such as cookies. Browsers also support storage mechanisms such as localStorage, IndexedDB, WebSQL and FileSystem

The Rendering Engine

The rendering engine’s main purpose is to render (parse & display) the requested & received content to the user on the browser.

Different browsers use different rendering engines: Internet Explorer uses Trident, Firefox uses Gecko, Safari uses WebKit. Chrome and Opera (from version 15) use Blink, a fork of WebKit.

WebKit is an open source rendering engine which started as an engine for the Linux platform and was modified by Apple to support Mac and Windows. See webkit.org for more details.

The rendering engine will start getting the contents of the requested document from the networking layer. This will usually be done in 8kB chunks.

After that, this is the basic flow of the rendering engine:

1) Parsing HTML to construct DOM tree

2) Render tree construction

3) Layout of the render tree

4) Painting the render tree

What are Websockets?

Zaffere — Wed, 18 Jan 2023 02:23:25 +0000

What are `Websockets`

Websocket is a communication protocol between client and server.

It establishes a two-way communication protocol (full duplex communication) and keeps a persistent connection between client & server.

Websockets allows near real-time communication between client and browser

We can send data in text or binary with websockets.

HTTP VS Websockets

With HTTP 1.1, we have the “keep alive” header. So only initial request from a client requires a TCP handshake. Subsequently all client requests just gets a response back from the server.

But it still requires the client to send a request to the server over the HTTP protocol and wait for a response from the server. Server just waits to be asked for resources.

Websockets also uses HTTP and TCP/IP. HTTP for initial request where it sends a connection upgrade request to the server (to upgrade the protocol from HTTP to web socket)
TCP is also involved but same thing only on initial request from client.

So the difference between HTTP and websockets is that websockets persists its connection and allows bidirectional communication between servers and clients where typical HTTP does not.

HTTP is request/response meaning that regular HTTP protocols requires the client to send a request in order for a response.
Websocket allows duplex communication so that means the server doesn’t need to wait for a request before fires a response.

Alternatives to websockets

Event Source / Server Sent Events

This is quite similar to websockets but is mono-directional meaning the server can only push data to the client. something like a webhook in the client side
Polling

There are generally two different types of polling, Short & Long polling.

Short polling sends a single request to the server at an interval regardless of whether the server has ‘new data’ whereas Long polling sends a single request to the server and waits for ‘new data’ to be available.

In short polling, we may expect to receive an empty response from the server when there are no new updates. This could have an impact server load depending on the interval time the client sends a request to the server. We are generally wasting resources this way as the server has to parse request headers, query for new updates and handle the response object.

Long polling on the other hand sends a request and waits for updates on the server side. Meaning we only send one request each time. But this still has its overhead as we still need to establish the TCP Handshake each time we send a request. This can get quite expensive if done in regular intervals in terms of server resource usage.

All these solutions comes with an overhead and needs proper planning and determining its trade-offs before implementing.

Why use websocket

Websocket allows near real time updates, it is fast by design (keeps a single persistent connection) and doesn’t require constant request/respond cycles.

This is especially useful when developing a real-time chat application or when you need your client to react as quickly as possible to events that happen within the architecture

Go Memory Allocation

Zaffere — Wed, 18 Jan 2023 01:43:19 +0000

What is memory allocation

Memory allocation is the process of how programming languages store or keep track of in-memory data like variables, arguments or any other data in your program’s runtime.

There are two places where the runtime keeps track of memory; Heap & Stack

Heap

A Heap (same name as the data structure but not related at all) is a portion of physical memory on your machine where global variables, arguments or any data used by the runtime is stored. Heaps are generally shared by the entire program meaning that all stacks have access to data stored in the heap.

Data in the heap stays in the heap until it is freed or your program terminates.

Go has aGarbage Collector and it is responsible for freeing up unused data in the heap automatically for us (we can manipulate this behaviour with the runtime package)

Stack

Stack is a short lived memory space to store data that is local to a function or a thread (goroutine).

Memory in the stack are removed once the function or goroutine returns/terminates.

Any new memory that is created inside of a stack that is passed and used outside of the stack (*****************passed up the stack*****************) is said to have escaped to the heap. This is when the memory is “transferred” over to the heap from the stack.

Passing down the stack

Passing down the stack refers to passing an address of a variable instead of the value of the variable itself down to a function.

Example of Passing down the stack

package main

func main() {
    num := 2
    add(&num)
}

func add(i int) {
    result := *i + 10
    return result
}

Passing up the stack

Passing up the stack refers to passing an address of a variable instead of the value of the variable itself up/outside the function.

Example of passing up the stack:

package main

func main() {
    num := getNum()
}

func getNum() int {
    number := 7
    // This causes the variable number to be moved to the heap
    return &number 
}

Handling Application Configurations

Zaffere — Wed, 18 Jan 2023 01:05:19 +0000

Problem Statement 😬:

When developing an application, we require external dependencies, for example, a database, cache store, an object storage (AWS S3) and so on. Often times, these services contain private data and should never be exposed to the public, making handling credentials to these external services securely a priority. You don't want your credentials to your database sitting in plain sight in your codebase which is checked in to your code repository, available for anyone who has access to your codebase (or just about anyone if your repository is set to public) to view. So setting your application's configurations as simple constants in your codebase is never a good idea.

There are a couple of ways to solve this problem.

Storing configurations in a separate file
This approach has you creating a file, separate from your main codebase (config.yml for example) and storing it in a private server or a private repository, accessible only by those who are responsible for setting up and maintaining these external dependencies.

But there are also some problems with this approach!

It is easy to mistakenly check in the config file to your repository!
In a microservice architecture, these configuration files could be scattered everywhere and doesn't follow a standard format making it difficult to manage

Storing configurations in environment variables
With this approach, we simply set our credentials in the environment. All we have to do then, is to reference these variables in our codebase

package main

conn := os.getEnv("POSTGRES")
db, err := sql.Open("postgres", conn)

This approach can potentially solve some of the problems mentioned in the previous step (Storing configs in separate files)

There is little chance that these credentials will be checked into our repository and definitely safer than storing them as constants.

We may have a single point of managing these credentials, but still has the potential for human error with no standard naming format.

Other solutions...

AWS Secrets could be a good fit if you are already hosting on AWS. They provide APIs to manage, retrieve, and rotate database credentials, API keys, and other secrets throughout their lifecycles

A more "advanced" way of handling application configuration is to use ETCD as our main storage for credentials. ETCD is a key/value store and is highly available by design. We could expose a user interface for developers to configure these credentials, which would store them in our ETCD clusters where our application would read from.
Although this step requires some development time, it should reduce the problems mentioned above.

But as always, we have to identify the trade-offs present in all scenarios and based on our requirements, choose the solution that fits us best.

These are a few options I have used to handle application configuration and i hope you find it useful!

Please add a comment if you see anything that can be improved!👍