DEV Community: Robin

To Comment or Not To Comment

Robin — Mon, 20 Sep 2021 02:27:59 +0000

Sometimes I feel like writing comment is a controversial topic in programming. Some will take the stance of:

Code is like humor. When you have to explain it, it’s bad.

Of which, I don't 100% agree with.

Mis-interpretation

Let's take a bit of a bad example:

function applyDiscount(to: number, rate: number): number {
   ...
}

At a glance, the code is quite obvious: Apply discount to a number, pass on the rate and the function will return the final result.

Two months goes by and somebody else wanted to use this function and at a glance they use:

const finalPrice = applyDiscount(100, 10);

expecting the finalPrice = 90 but instead got finalPrice = -900

In this case, there's an illusion where the code is already self-documenting and self explanatory, but interpretation of a word between each developer might be different! What rate means for the original coder is the rate of discount after it is divided by 100. But the next coder who tried to use it interpreted it as a percentage number.

Naming things is always regarded as one of the hardest thing in programming. Mis-interpretation will always happen when glossary of words between developers are different.

Too Much Focus on What but not Why

Consider this bad and exaggerated example:

function applyDiscount(to: number, percentage: number): number;

...

function submitOrder(order: Oder) {
  order.setFinalPrice(applyDiscount(order.subTotal, 10);
  ...
}

Sure now the code is obvious, every order will be given 10% discount. But it's now missing an important context on:

Why should every order be applied 10% discount?

Most of the time we can easily understand the code on what it does by tracing it line by line. But when things gets complicated, only focusing on what the code does loses the context where we need to know

Why the code is written as it is?
Or, why there's a branching condition?

Comment your intent

Most of the times, comment should never explain what the code does. But rather what the intention is.

Let's again take a look of this simple example:

function doPayment(cardNumber: string) {
  if (isVisa(cardNumber)) {
    processPaymentOnlyForVisa(cardNumber);
    return;
  }

  ...
}

Sure we know how to read that there's a special treatment for payment using Visa. But after maybe 1-2 months or when the original coder leave, the context of why Visa payment should be different is lost.

It might be business decision, it might be caused by some technical hurdles.
But as long as the intention was not documented, the cause is lost for the next person who touched the code.

Cover image credits to: Artur Shamsutdinov

Playing with GraphQL

Robin — Sat, 20 Feb 2021 05:01:23 +0000

For the past few years, GraphQL have risen in popularity it becomes hard for me to ignore now.

Now I've put some effort into learning GraphQL by making a simple app and here's what I found.

Problem Statement

What is it that GraphQL tries to solve?

In a typical REST API. Every endpoint is structured around ONE resource. This causes a common issue in Frontend (FE) development where upon encountering relationship between resource, the FE client have to fetch to multiple REST endpoints.

Also a REST endpoint usually expose all fields of a resource in the endpoint. Causing FE to sometimes, or most of the times, fetch unneeded data.

These are called overfetching. Sometimes we overfetch by calling multiple endpoints just to get the full picture of a resource. Sometimes we overfetch by getting unnecessary data presented by an endpoint.

A Look inside GraphQL Query

GraphQL introduce us to another layer of language to learn to. Let's take a look at this simple sample GQL

query  {
  products (price: "100,200") {
    id
    name
    price
  }
}

Very simple, and can be interpreted as:

SELECT id, name, price
FROM products 
WHERE price between 100 and 200

Now GraphQL doesn't have a concept of separate endpoints per resource. Every request comes to a single root endpoint and all resources can be accessed by the query.

Let's have a case where we want to fetch every available discount to the product

query {
  products (first: 10, price: "100,200") {
    id
    name
    price
    discounts {
      rate
    }
  }
}

Again, very simple SQL equivalent would be

SELECT p.id, p.name, p.price, d.rate
FROM products p
LEFT JOIN discounts d
WHERE p.price between 100 and 200
LIMIT 10

Easy right? And orginally I thought that GraphQL can do this kind of optimization in the background. Except it turns out to be NOT TRUE at all.

Shifting Overfetching Around

In the previous scenario, what GraphQL does are

Query the products and limit it to 10 results
Foreach entry in products result, get the product available discounts.

This is called N+1 problem where basically we have to query N+1 times to database. Of which N equals to how many results in the previous step. In this case, we fetch 10 product once, and we query discounts at least once per product, equaling 10+1 separate query to database.

This makes me thinking. So is what GraphQL does, is just shifting around the overfetching problem away from frontend to backend and introduce another myriad of problems that does not exist in a typical REST API?

Is problem of Overfetching overstated by GraphQL enthusiast?

Dataloader

Now GraphQL have batching technique that might make nested query to become at least 1+1 instead of N+1.

The simplest implementation is to pool each id of the parent context into something that is called data loader. And after all the parent ids are collected, dataloader can do 1 batch query to database to get all the child.

But this again, adds another layer of problem. How do we attach the list of child results into its correct parent?

With dataloader, we have to return it into the same order as the parent. Giving us another layer of post processing.

Personal Take

Maybe it's just me but back before the GraphQL land and time, joining resource via code was (or still is? IDK) considered as hacky and might introduce unwanted bugs.

Seeing now that it becomes the norm with GraphQL kinda rubs me in the wrong way.

I haven't even talked about caching and mutation problem in GraphQL that doesn't exist in a typical REST API, yet. And the N+1 problem already gives me a bad impression about GraphQL.

For now at least I think GraphQL is more of a solution to a problem that I personally don't face.

Disclaimer

To other people who loves GraphQL and want to bash me in the head for this.

I am not trying to say GraphQL is bad for you. Just that it is not the right fit for me. At least for now.

The Industry Hates Me For Being Honest

Robin — Fri, 05 Jun 2020 03:59:53 +0000

Over the past 2 months I have interviewed with several companies and 2 of them stood out at rejecting me. Let's call them Company A, and Company B!

I know right? Developers are bad at naming!

I guess part of it is my fault too! I am old and slow. Doesn't like competitive programming and already forgot most of how to answer algorithm question. I can't even answer some of the algorithm question I've flawlessly answered back when I was fresh out of University.

Company A

When I got chance to interview at Company A, they require me to answer HackerRank style interview. It's my first time in nearly a decade of working in the industry to feel like I'm in a classroom exam again. I hate it, and I deliberately voiced my distaste to the answers comment:

// Paraphrasing
// I'm sorry, I'm dumb!
// I never faced anything like this in real world work...
// ......

But guess what? My answer still pass the score, have a call with their VP, which proceed to have another call with their Lead Engineer.

Talked about my experience with Event Driven System and CQRS+ES and they decided that I am:

Arrogant
Too RND in my tech stack
And overkill in CQRS+ES

And decided they don't need me.
They hate me for having a headstrong personality which translates as Arrogance to the perceiving end.

Company B

Another HackerRank style interview. Guess I passed their score this time without me typing some strong comment and proceed to have another test with their Lead Engineer.
This time they want 5 question answered in google docs within 60 minutes.
Two of them stood out to me for being impossible to work on 12 minutes (60 / 5 if you're wondering). Or maybe I'm just old and dumb?!
The others are just questions copied word for word from Geeks For Geeks.

One of the question requires me to write a password brute force attack to an imaginary API.

The other requires me to find a combination of math + or - operation from a strings of numbers that results in a number.

My Arrogance kicks in and I start typing a comment

// Paraphrasing
// I am sorry but I feel this is impossible for me to think of in 12 minutes 
// (60 / 5 if you're wondering)
// But I know you guys got this question from Rosseta Code!
// Here's the link, but I don't know the logic behind it

See? I've worked on this question back when I was still a University student and remember where to look at.

Unsurprisingly, I've heard the feedback that I was rejected although I've answered one of their question FLAWLESSLY. I know they are being sarcastic at this point. haha.

I was trying to be honest about what I can and can't do in the N minutes timeframe and the Industry hates me.

I guess The Industry love people who can grind GFG or other algorithm websites, remember the solutions out of their head, and quietly answer their genuinely original question without pointing the flaws back at them.

Lack of default constructor is hurting Go

Robin — Mon, 16 Mar 2020 06:55:01 +0000

How many times have you written a factory function just to initialize a struct with proper default value?

For example:

// PagingOption ...
type PagingOption struct {
    Page int
    Size int
}

We usually want BOTH to have default values. Let's say page=1&size=25, but anywhere we want to use PagingOption we are either

Write and use a factory function which properly build a PagingOption
Write and use an Init function which properly build a PagingOption
Instantiate using object literal and fill it with default value
Validate PagingOption in every consumer and fallback to default value if necessary

Option 1 forces all developer to remember not to use new(PagingOption) or PagingOption{}.
Option 2 forces all developer to remember calling .Init after initializing PagingOption
Options 3 forces all developer to know what is the agreed default values.
Options 4 forces all developer to remember checking default value before using the PagingOption

All options contribute to human error which can be mitigated just by having a proper constructor.

Marrying Actor, DDD, and Worker Pattern using Go

Robin — Wed, 11 Mar 2020 12:18:25 +0000

Imagine a Domain event called Order.Placed.
Each Order.Placed contains:

{
    "customer": "CUST-001",
    "merchant": "MRCN-001",
    "payment": "CARD-001",
    "items": [
        {"id": "IT-001", "qty": 1},
        {"id": "IT-002", "qty": 2}
    ],
    "promo": "PROM-001"
}

Everytime an order is placed, there must be something in the Backend service that:

Getting customer's detail
Getting merchant's detail
Getting promotion's detail
Getting each product / item detail
Calculate sum of the order and deduct it with promotion
Make an order based on the gathered information, and notify it to merchant
Make an invoice based on the gathered information, and notify it to customer
Make a payment through payment service based on the invoice
Flag the order state as paid
Notify merchant that payment is done to said order
Notify customer that payment success to said invoice

That's Tremendously Ridiculous amount of work needed to be done in the Backend side.
It sounds wrong to just make an API endpoint to handle these work, especially if your Platform can receive a ludicrous amount of order per minute across multiple region.

Aggregate Root disguised as Actor

First of all, let's take on the rate of order issue.
Order can be placed at any time by millions of customer at the same time and the system must NEVER lose any of it.
System can defer processing an order by placing it in a queue which can be persisted and recovered in case of server restart, crash, or whatever disaster that comes.

Let's design an object called Actor

An Actor is an object which process messages being sent to it
So each Actor have an inbox which acts as a queue
The inbox can contain any type of message, but usually 1 actor handle 1 kind of message
The actor processor is a function delegated by its maker
Each actor can only have 1 type of processor but can have multiple instance of it. Let's call it worker.
A processor might, or might not produce a result
A processor might, or might not produce an error

package actor

// Processor is the delegate which process a message
// @worker is its assigned worker number (starts from 1) in case we make more than 1 worker
// @actor is the reference to which actor that receives the message
// @message is the current individual message from actor's inbox
type Processor func(worker int, actor *Actor, message interface{}) (interface{}, error)

// Exception handler in case processor produce an error
// @worker is its assigned worker number (starts from 1) in case we make more than 1 worker
// @actor is the reference to which actor that receives the message
// @err is the error that happened after trying to process a message
type Exception func(worker int, actor *Actor, err error)

// Options when initializeing an Actor
type Options struct {
    Worker      int              // number of worker / processor go routine, defaults = 1
    Output      chan interface{} // output channel, on which Actor will send in after process is done
}

// configure fallbacks to default
func (opt *Options) configure() {
    if opt.Worker <= 0 {
        opt.Worker = 1
    }
}

// Actor ...
type Actor struct {
    inbox     chan interface{}
    outbox    chan interface{}
    process   Processor
    exception Exception
}


// New instance of an Actor with w as number of worker
// @p is the processor function
// @e is the exception handler
func New(p Processor, e Exception, opt *Options) *Actor {
    opt.configure()

    actor := &Actor{
        inbox:     make(chan interface{}, opt.Worker),
        outbox:    opt.Output,
        process:   p,
        exception: e,
    }

    actor.start(0, opt.Worker)
    return actor
}

// start the actor with n number of worker
func (actor *Actor) start(idx, n int) {
    if idx == n {
        return
    }

    // worker number starts from 1
    go actor.work(idx + 1)
    actor.start(idx+1, n)
}
func (actor *Actor) work(w int) {
    for message := range actor.inbox {
        result, err := actor.process(w, actor, message)

        if err != nil && actor.exception != nil {
            actor.exception(w, actor, err)
            continue
        }

        if actor.outbox != nil {
            actor.outbox <- result
            continue
        }
    }
}

// Queue a message to inbox
func (actor *Actor) Queue(messages ...interface{}) {
    go func() {
        for _, message := range messages {
            actor.inbox <- message
        }
    }()
}

We'll test the actor with simple test case:

func Test_Actor(t *testing.T) {
    word= [...]string{"One", "Two", "Three"}
    actor := New(func(w int, actor *Actor, message interface{}) (interface{}, error) {
        result := words[w-1]

        fmt.Println("worker", w,
            "receive", message,
            "processed as", result,
            "send to?", actor.outbox)

        return result, nil
    }, func(w int, actor *Actor, err error) {
        fmt.Println(err)
    }, &Options{Worker: 3})

    wg := sync.WaitGroup{}
    for i := 0; i <= 10; i++ {
        wg.Add(1)
        go func(i int) {
            idx := i % 3
            word := words[idx]
            actor.Queue(word)
            wg.Done()
        }(i)
    }

    wg.Wait()
}

The test code creates an actor with 3 worker
Each worker do the same type of job, which only returns word based on its worker number
We loop 10 times and queue a word into the actor
Lastly, we wait for the wait group to be done

robin.bastian$ go test -timeout=10s -run "^(Test_Actor)$"
worker 2 receive One processed as Two send to? <nil>
worker 2 receive One processed as Two send to? <nil>
worker 3 receive Two processed as Three send to? <nil>
worker 3 receive One processed as Three send to? <nil>
worker 1 receive Two processed as One send to? <nil>
worker 1 receive Three processed as One send to? <nil>
worker 1 receive Three processed as One send to? <nil>
worker 1 receive Two processed as One send to? <nil>
worker 1 receive One processed as One send to? <nil>
worker 2 receive Three processed as Two send to? <nil>
worker 3 receive Two processed as Three send to? <nil>
PASS
ok      github.com/bastianrob/go-experiences/generator/actor    0.006s

From the test results, we can see:

Exactly 3 workers are spawned with one type of processor
Order of message processed is NOT guaranteed
Each process returns a result but is not being sent to actor's outbox

Stopping the Actor

Next, we'll need a mechanism to Stop an actor. When an actor is stopped, we collect all pending messages and return it to the caller.

First of all, we add an exit mechanism to the Actor by adding:

type Actor struct {
    ...
    // exit mechanism
    exit       chan struct{}
    workgroup  *sync.WaitGroup // worker waitgroup
    inboxgroup *sync.WaitGroup // inbox waitgroup
}

func New(p Processor, e Exception, opt *Options) *Actor {
    opt.configure()

    actor := &Actor{
        ...
        exit:       make(chan struct{}),
        workgroup:  &sync.WaitGroup{},
        inboxgroup: &sync.WaitGroup{},
    }

    actor.start(0, opt.Worker)
    return actor
}

exit is an exit channel which will be used to signal each worker to stop processing a message
workgroup is a wait group that counts how many worker is still active
inboxgroup is a wait group that counts how many messages is still in the inbox

Next, we implement all of those exit mechanism:

inboxgroup is added everytime a message is queued into inbox
workgroup is added everytime a worker is spawned
Each of it tries to listen to both inbox and exit channel by using select case statement
inboxgroup is decreased everytime a message is done processed by a worker
workgroup is decreased everytime a worker quits when it receives signal from exit channel

func (actor *Actor) Queue(messages ...interface{}) {
    // add length of message to inbox wait group
    actor.inboxgroup.Add(len(messages))
    go func() {
        for _, message := range messages {
            actor.inbox <- message
        }
    }()
}

func (actor *Actor) work(w int) {
    actor.workgroup.Add(1)       // worker group is added
    defer actor.workgroup.Done() // defer flagging worker group as done

    for {
        select {
        case message := <-actor.inbox: // waits for message to come from inbox
            result, err := actor.process(w, actor, message)

            if err != nil && actor.exception != nil {
                actor.exception(w, actor, err)
                actor.inboxgroup.Done() // flag 1 message as done
                continue
            }

            if actor.outbox != nil {
                actor.outbox <- result
                actor.inboxgroup.Done() // flag 1 message as done
                continue
            }

            // flag 1 message as done
            actor.inboxgroup.Done()
        case <-actor.exit: // listen on exit signal
            return
        }
    }
}

Lastly, we implements the Stop method to:

Close the exit channel so it is broadcasted to all workers
Wait for all workers to finish exiting
Gather the pending messages that's still lingering in inbox
Decrease the inboxgroup counter everytime we finished collecting a message
Wait for all pending messages to be collected
Close the inbox channel
And report all the pending messages to caller

// Stop actor from processing any message
func (actor *Actor) Stop() (pendings []interface{}) {
    // stop all worker from processing any inbox
    close(actor.exit)
    actor.workgroup.Wait()

    // gather pending messages inside inbox and flag it as done
    go func() {
        for message := range actor.inbox {
            pendings = append(pendings, message)
            actor.inboxgroup.Done()
        }
    }()

    // wait for pending messages gathering to be completed and close the inbox channel
    actor.inboxgroup.Wait()
    close(actor.inbox)

    // return gathered pending messages
    return pendings
}

And now it's time to test the stop mechanism:

func Test_ActorStop(t *testing.T) {
    mux := sync.Mutex{}
    var processed []interface{}
    actor := New(func(w int, actor *Actor, message interface{}) (interface{}, error) {
        mux.Lock()
        processed = append(processed, message)
        mux.Unlock()

        return nil, nil
    }, func(w int, actor *Actor, err error) {
        fmt.Println(err)
    }, &Options{Worker: 5})

    expected := 0
    for i := 1; i <= 100; i++ {
        go actor.Queue(i)
        expected = expected + i
    }

    pendings := actor.Stop()
    combined := append(processed, pendings...)

    sum := 0
    for _, e := range combined {
        sum = sum + e.(int)
    }
    if sum != expected {
        t.Error("Sum of 1-100 must be", expected, "but received", sum)
    }

    fmt.Println("PEND", pendings)
    fmt.Println("PROC", processed)
}

The test code creates an Actor with 5 workers
We feed the actor with number from 1 to 100
So the expected sum of 1 + 2 + ... + 100 should be 5050
In the processor function, we collect all processed number into a variable called var processed []interface{}
We use mutex to guard it because it will be accessed by 5 workers working from different go routine
We then collect all pending numbers by calling actor.Stop and store it to a variable called pendings
We combine both processed and pendings into slice called combined and sum all numbers inside it
We compare the sum of all combined numbers with the expected number

robin.bastian$ go test -timeout=10s -run "^(Test_ActorStop)$"
PEND [7 21 8 22 10 23 63 24 57 25 26 27 28 29 30 31 73 64 65 66 67 68 58 59 60 61 62 71 70 79 74 75 76 77 69 78 72 82 80 81 84 83 85 86 93 87 88 89 90 91 92 96 94 95 98 97 99 100]
PROC [6 11 3 4 5 43 13 33 14 34 15 35 16 36 17 18 37 38 39 40 41 42 56 44 45 46 47 48 49 50 51 52 53 54 55 19 9 12 1 32 20 2]
PASS
ok      github.com/bastianrob/go-experiences/generator/actor    0.006s

Directing Actors

Individually, an actor can do its job fine enough. But it will be better if we can chain actors together to build a pipeline processing!
So let's think of what we already have:

Each actor have an inbox and outbox
While inbox is required, an outbox is optional
Now we want to chain output from one actor as input to another actor
In other words: we can assign one actor's outbox with another actor's inbox

First, we'll have to refactor some of our Actor's code:

type Options struct {
    ...
    Name        string // actor's name
    Output      *Actor // output actor, on which source actor will send a message after process is done
}

type Actor struct {
    ...
    // metadata
    name string
}

func (actor *Actor) work(w int) {
    ...

    for {
        select {
        case message := <-actor.inbox: // waits for message to come from inbox
            ...
            if actor.outbox != nil {
                actor.outbox.Queue(result)
                actor.inboxgroup.Done() // flag 1 message as done
                continue
            }
            ...
        }
    }
}

We added name to an actor so we can identify an actor easily.
And in the worker, if current actor's outbox is not nil, we queue the process output to next actor.

And then we write a function called Direct:

package actor

// Direct inbox of a target actor, as source actor's outbox
func Direct(actors ...*Actor) {
    var source *Actor
    for _, target := range actors {
        if source == nil {
            source = target
            continue
        }

        source.outbox = target.inbox
        source = target
    }
}

Again, we'll write a test to ensure the behaviour is within expectation

func Test_ActorDirected(t *testing.T) {
    errPrinter := func(w int, actor *Actor, err error) {
        fmt.Println("worker:", w, "err:", err)
    }

    bale := New(func(w int, actor *Actor, in interface{}) (interface{}, error) {
        return in, nil
    }, errPrinter, &Options{
        Worker: 3,
        Name:   "Bale",
    })
    bane := New(func(w int, actor *Actor, in interface{}) (interface{}, error) {
        switch {
        case in == "I AM VENGEANCE":
            return "I AM INEVITABLE", nil
        case in == "I AM THE NIGHT":
            return "I AM BANE", nil
        case in == "I'M BATMAN":
            return "I WILL BREAK YOU", nil
        default:
            return nil, errors.New("WHATEVER YOU SAY")
        }
    }, errPrinter, &Options{
        Worker: 3,
        Name:   "Bane",
    })
    subtitle := New(func(w int, actor *Actor, in interface{}) (interface{}, error) {
        fmt.Println("worker:", w, "actor:", actor.name, "receive:", in)
        if in != "I AM INEVITABLE" && in != "I AM BANE" && in != "I WILL BREAK YOU" {
            t.Error("Bane's subtitle must be one of:", "I AM INEVITABLE", "I AM BANE", "I WILL BREAK YOU")
        }
        return nil, nil
    }, errPrinter, &Options{
        Worker: 3,
        Name:   "Subtitle",
    })

    Direct(bale, bane, subtitle)

    bale.Queue("I AM VENGEANCE", "I AM THE NIGHT", "I'M BATMAN", "HEY HO!")
    bale.Stop()
    bane.Stop()
    subtitle.Stop()
}

In this test, we:

Have 3 actors named Bale, Bane, and Subtitle
Bale will say I AM VENGEANCE, I AM THE NIGHT, I'M BATMAN, and HEY HO! to Bane
Bane will say I AM INEVITABLE, I AM BANE, and I WILL BREAK YOU to Subtitle
Bane will be confused by HEY HO! and responds WHATEVER YOU SAY to errPrinter
Subtitle will print whatever Bane says

Test result shows:

robin.bastian$ go test -timeout 10s -run "^(Test_ActorDirected)$"
worker: 3 actor: Subtitle receive: I AM BANE
worker: 2 actor: Subtitle receive: I WILL BREAK YOU
worker: 1 actor: Subtitle receive: I AM INEVITABLE
worker: 2 actor: Bane err: WHATEVER YOU SAY
PASS
ok      github.com/bastianrob/go-experiences/generator/actor    0.006s

Mocking Fake Microservices

Back to the example domain event in the beginning of this article, we have 7 microservices with CRUD implementation with Order microservice in the center of it all.
We'll try to mock the CRUD implementation to all of the microservices:

package mock

// CRUD contract
type CRUD interface {
    Get(id string) (interface{}, error)
    Create(dao interface{}) error
    Update(dao interface{}) error
}

// APIClient generic mock implementation of CRUD interface
type APIClient struct {
    GetFunc    func(id string) (interface{}, error)
    CreateFunc func(dao interface{}) error
    UpdateFunc func(dao interface{}) error
}

// Get mock, please implement GetFunc
func (ac *APIClient) Get(id string) (interface{}, error) {
    return ac.GetFunc(id)
}

// Create mock, please implement CreateFunc
func (ac *APIClient) Create(dao interface{}) error {
    return ac.CreateFunc(dao)
}

// Update mock, please implement UpdateFunc
func (ac *APIClient) Update(dao interface{}) error {
    return ac.UpdateFunc(dao)
}

CRUD is a contract to imaginary API which
APIClient is the mocked / fake implementation of a CRUD

We'll use these later to mock CRUD call all microservices.

Each of those microservice have their own data transfer object.
Let's call them: dto

package dto

type Customer struct {
    ID    string
    Email string
    Name  string
}

type Invoice struct {
    ID       string
    Order    string
    Customer string
    Promo    string
    Subtotal int
    Discount int
    Total    int
}

type Merchant struct {
    ID    string
    Email string
    Name  string
}

type Payment struct {
    ID        string
    InvoiceID string
    MethodID  string // card, cash, balance, whatever
    Amount    int
}

type Product struct {
    ID    string
    Name  string
    Price int
}

type Promotion struct {
    ID       string
    Name     string
    Discount int // percentage
}

And since we're in the context of order we can access its data access object (dao):

package dao

import "time"

// OrderItem DAO
type OrderItem struct {
    ID    string
    Name  string
    Qty   int
    Price int
}

// Order DAO
type Order struct {
    ID           string
    Date         time.Time
    State        OrderState
    CustomerID   string
    CustomerName string
    MerchantID   string
    MerchantName string
    Items        []*OrderItem
    Total        int
}

Lastly we create a command package which, as its name implies, command our order service to do something

package command

// LineItem ordered item & qty
type LineItem struct {
    ID  string
    Qty int
}

// PlaceOrder command
type PlaceOrder struct {
    Customer string
    Merchant string
    Payment  string
    Promo    string
    Items    []LineItem
}

Order as Aggregate Root

We need a service which orchestrate all required data fom multiple services.
We call it Aggregate Root and in this case, it's in the order context.

So let's start thinking about the flow:

Order is working as the Aggregate Root so let's just call it Root
Root can receive a command called PlaceOrder which contains all necessary information for Root to work
Each time Root receives a message, it will orchestrate calls to multiple services needed to place an order

package order

import (
    "errors"
    "fmt"
    "time"

    "github.com/bastianrob/go-experiences/generator/order/pkg/dao"
    "github.com/bastianrob/go-experiences/generator/order/pkg/dto"

    "github.com/bastianrob/go-experiences/generator/actor"
    "github.com/bastianrob/go-experiences/generator/mock"
    "github.com/bastianrob/go-experiences/generator/order/pkg/command"
)

// Services collection
type Services struct {
    Customer mock.CRUD
    Invoice  mock.CRUD
    Merchant mock.CRUD
    Order    mock.CRUD
    Payment  mock.CRUD
    Product  mock.CRUD
    Promo    mock.CRUD
}

// Config for order service
type Config struct {
    Worker   int
    Services Services
}

// Root aggregate root of order
type Root struct {
    *actor.Actor
    services Services
}

// NewAggregateRoot for order
func NewAggregateRoot(cfg *Config) *Root {
    root := &Root{
        services: cfg.Services,
    }

    n := cfg.Worker
    if n <= 0 {
        n = 10
    }

    worker := &actor.Options{Worker: n}
    root.Actor = actor.New(root.processor, root.exception, worker)

    return root
}

func (root *Root) processor(w int, a *actor.Actor, msg interface{}) (interface{}, error) {
    if msg == nil {
        return nil, errors.New("Order message is empty")
    }

    var customer *dto.Customer
    var merchant *dto.Merchant
    var promo *dto.Promotion

    // 1. Converts message to command
    cmd := msg.(*command.PlaceOrder)

    // 2. Fetch required information
    // Uses goroutine because we all have verbose if err
    errc := make(chan error)
    go func(errc chan<- error) {
        cust, err := root.services.Customer.Get(cmd.Customer)
        if err != nil {
            errc <- err
            return
        }
        customer = cust.(*dto.Customer)

        mcr, err := root.services.Merchant.Get(cmd.Merchant)
        if err != nil {
            errc <- err
            return
        }
        merchant = mcr.(*dto.Merchant)

        prm, err := root.services.Promo.Get(cmd.Promo)
        if err != nil {
            errc <- err
            return
        }
        promo = prm.(*dto.Promotion)

        errc <- nil
    }(errc)

    // 3. Wait for fetch to complete and listen to any error occurred
    if err := <-errc; err != nil {
        return nil, err
    }

    // 4. Get product details and calculate the total
    order := &dao.Order{
        Date:         time.Now(),
        State:        dao.New,
        CustomerID:   customer.ID,
        CustomerName: customer.Name,
        MerchantID:   merchant.ID,
        MerchantName: merchant.Name,
        Items:        make([]*dao.OrderItem, len(cmd.Items)),
    }
    for i, entry := range cmd.Items {
        it, err := root.services.Product.Get(entry.ID)
        if err != nil {
            return nil, errors.New("Failed to get item with ID: " + entry.ID)
        }

        item := it.(*dto.Product)
        order.Items[i] = &dao.OrderItem{
            ID:    item.ID,
            Name:  item.Name,
            Qty:   entry.Qty,
            Price: item.Price,
        }
        order.Total += (entry.Qty * item.Price)
    }

    // 5. Persist the order data to database
    err := root.services.Order.Create(order)
    if err != nil {
        return nil, errors.New("Failed to create a new order: " + err.Error())
    }

    // 6. Create the invoice through API
    discount := order.Total * promo.Discount / 100
    invoice := &dto.Invoice{
        Order:    order.ID,
        Customer: order.CustomerID,
        Promo:    promo.ID,
        Subtotal: order.Total,
        Discount: discount,
        Total:    (order.Total - discount),
    }
    err = root.services.Invoice.Create(invoice)
    if err != nil {
        // TODO: Recovery strategy, delete the order? or flag it if you wish
        return nil, errors.New("Failed to create a payment: " + err.Error())
    }

    // 7. Make a payment through API call
    payment := &dto.Payment{
        InvoiceID: invoice.ID,
        MethodID:  cmd.Payment,
        Amount:    invoice.Total,
    }
    err = root.services.Payment.Create(payment)
    if err != nil {
        // TODO: Recovery strategy to both order and invoice
        return nil, errors.New("Failed to create a payment: " + err.Error())
    }

    return order, nil
}

func (root *Root) exception(w int, a *actor.Actor, err error) {
    fmt.Println("Exception occurred at worker:", w, "with err:", err)
}

As we can see in the code above:

Root owns an Actor
The processor is an actor's processor which orchestrate all call to multiple services

Now let's test the code by writing some unit tests:

package order

import (
    "errors"
    "fmt"
    "sync"
    "testing"
    "time"

    "github.com/bastianrob/go-experiences/generator/actor"
    "github.com/bastianrob/go-experiences/generator/mock"
    "github.com/bastianrob/go-experiences/generator/order/pkg/command"
    "github.com/bastianrob/go-experiences/generator/order/pkg/dao"
    "github.com/bastianrob/go-experiences/generator/order/pkg/dto"
)

func Test_OrderAsAggregateRoot(t *testing.T) {
    customerAPIMock := &mock.APIClient{
        GetFunc: func(id string) (interface{}, error) {
            time.Sleep(20 * time.Millisecond) // simulate 20ms latency
            return &dto.Customer{
                ID:   id,
                Name: "I am your customer",
            }, nil
        },
    }
    merchantAPIMock := &mock.APIClient{
        GetFunc: func(id string) (interface{}, error) {
            time.Sleep(20 * time.Millisecond) // simulate 20ms latency
            return &dto.Merchant{
                ID:   id,
                Name: "I am your merchant",
            }, nil
        },
    }
    promotionAPIMock := &mock.APIClient{
        GetFunc: func(id string) (interface{}, error) {
            time.Sleep(20 * time.Millisecond) // simulate 20ms latency
            return &dto.Promotion{
                ID:       id,
                Name:     "10% discount",
                Discount: 10,
            }, nil
        },
    }
    invoiceAPIMock := &mock.APIClient{
        CreateFunc: func(obj interface{}) error {
            time.Sleep(20 * time.Millisecond) // simulate 20ms latency
            inv := obj.(*dto.Invoice)
            inv.ID = "INV-001"
            return nil
        },
    }
    orderAPIMock := &mock.APIClient{
        CreateFunc: func(obj interface{}) error {
            time.Sleep(20 * time.Millisecond) // simulate 20ms latency
            inv := obj.(*dao.Order)
            inv.ID = "INV-001"
            return nil
        },
    }
    paymentAPIMock := &mock.APIClient{
        CreateFunc: func(obj interface{}) error {
            time.Sleep(20 * time.Millisecond) // simulate 20ms latency
            pay := obj.(*dto.Payment)
            pay.ID = "PMT-001"
            return nil
        },
    }
    productAPIMock := &mock.APIClient{
        GetFunc: func(id string) (interface{}, error) {
            time.Sleep(20 * time.Millisecond) // simulate 20ms latency
            switch id {
            case "ITEM-001":
                return &dto.Product{
                    ID:   id,
                    Name: "I am item 001",
                }, nil
            case "ITEM-002":
                return &dto.Product{
                    ID:   id,
                    Name: "I am item 002",
                }, nil
            }
            return nil, errors.New("404")
        },
    }

    // root is order actor which acts as an aggregate root
    // have by default 10 workers
    root := NewAggregateRoot(&Config{
        Worker: 20,
        Services: Services{
            Customer: customerAPIMock,
            Merchant: merchantAPIMock,
            Invoice:  invoiceAPIMock,
            Order:    orderAPIMock,
            Payment:  paymentAPIMock,
            Product:  productAPIMock,
            Promo:    promotionAPIMock,
        },
    })

    // waiter is an actor which hears from root and keep tracks of how many order have we done processing
    wg := &sync.WaitGroup{}
    wg.Add(100)
    waiter := actor.New(
        func(w int, a *actor.Actor, message interface{}) (interface{}, error) {
            wg.Done()
            return nil, nil
        },
        func(w int, a *actor.Actor, err error) {
            wg.Done()
        },
        &actor.Options{Worker: 10},
    )

    // clock in to check how long we're porcessing 100 command
    start := time.Now()

    // place order 100 times
    var orders []interface{}
    for i := 0; i < 100; i++ {
        orders = append(orders, &command.PlaceOrder{
            Customer: "CUST-001",
            Merchant: "MRCN-001",
            Payment:  "CARD-001",
            Promo:    "DISC-10",
            Items: []command.LineItem{{
                ID:  "ITEM-001",
                Qty: 1,
            }, {
                ID:  "ITEM-002",
                Qty: 1,
            }},
        })
    }
    root.Queue(orders...)

    actor.Direct(root.Actor, waiter)
    fmt.Println("We are waiting")
    wg.Wait()

    // we have at least 7 fake services and each takes simulated 20ms to complete
    // so total time it takes to complete 100 command * 140ms = 14sec
    // 14 sec if we only have 1 worker.
    // Ideally we can cut it down to minimal of 0.7sec because we have 20 workers
    dur := time.Since(start)
    if dur.Seconds() >= 1. {
        t.Error("Total processing time should not exceed 1sec")
    }

    fmt.Println("Duration:", dur)
}

In the test:

We mock all API call by using the mock.APIClient
Each API call is given 20ms sleep to simulate latency
We instantiate an Aggregate Root and give it 20 worker
We have total of 7 services and each call cost 20ms so 1 message should cost around 140ms
We give 100 command to the Aggregate Root
If each command takes 140ms processing time, and we're giving 100. Total time it takes is 14000ms or 14s if done serially
But we give 20 workers to the Aggregate Root so the fastest we can achieve is t = 14s / 20 = 0.7s
We direct Aggregate Root's output to another actor called waiter
The waiter simply waits for Aggregate Root to finish processing 100 commands.
And then we track the duration it takes for Aggregate Root to complete 100 commands.

robin.bastian$ go test -timeout 3s -run "^(Test_OrderAsAggregateRoot)$"
We are waiting
Duration: 881.459161ms
PASS
ok      github.com/bastianrob/go-experiences/generator/order/internal   0.888s

Distributed Processing

Now imagine 100 order placed / sec in the test case above is not enough.
We can make a cluster of Aggregate Root, have them listen to a message broker like NSQ or RabbitMQ, and let the message broker handle the load balancing.

Shenanigan's github https://github.com/bastianrob/go-experiences/tree/master/generator

Making Simple Scheduler using Go Routine & Channel

Robin — Sat, 07 Mar 2020 13:45:59 +0000

Learn how to schedule an event to be running at specific time

Recently Product team in my job requires me to have a scheduled event that might or might not run at a specific set of interval.

And the story goes like this:

As a [role] I want [resource] document to be tracked after it was Assigned to [role], for every [N] minutes until it was Handled

So let's do a case study on Go Routine, and Go Channel!

Designing the Scheduler

It have to be re-usable and shared across all projects
It have to survive server restart
It have to be able to re-schedule event

So now, we have these barebone requirements. We can start to write some code.

Event Object

First we have to write Event object. It is the data that a Scheduler receive

package scheduler

import (
    "time"
)

// Attachment data associated with an event
// Can be anything stored in bytes
type Attachment struct {
    Name        string
    ContentType string
    Body        []byte
}

// Event which will run on scheduler
type Event struct {
    datetime    string // RFC3339 please
    attachments []Attachment
}

// NewEvent create a new instance of immutable Event
func NewEvent(d string, att []Attachment) *Event {
    // we copy the attachment slice to another memory to avoid mutability
    cpy := make([]Attachment, len(att))
    copy(cpy, att)

    return &Event{
        datetime:    d,
        attachments: cpy,
    }
}

// Date get event datetime, parsed into RFC3339 format
func (e *Event) Date() (time.Time, error) {
    return time.Parse(time.RFC3339, e.datetime)
}

// Attachments returns a copy of attachments slice
// This is done to ensure immutability of event
func (e *Event) Attachments() []Attachment {
    cpy := make([]Attachment, len(e.attachments))
    copy(cpy, e.attachments)

    return cpy
}

Event is designed to be immutable so none of these event can be changed once scheduled.
Each event contains 2 fields

datetime = RFC3339 string representation of a date & time
attachments = List of any object associated to an Event

Both of those fields are private and can only be set from the NewEvent factory.
But both can get fetched from 2 getter methods called Date dan Attachments.

The Scheduler

Next in the scheduler, we know that we have to create a method called Schedule which takes an Event as its parameter:

package scheduler

// Scheduler ...
type Scheduler struct {}

// Schedule an event
func (s *Scheduler) Schedule(e *Event) error {
    // TODO:
}

TODO-1 Validate Event

// Scheduler error collection
var (
    ErrEventInPast = errors.New("Event datetime is in the past")
    ErrTimeInvalid = errors.New("Datetime format is not in RFC3339")
)

// Schedule an event
func (s *Scheduler) Schedule(e *Event) error {
    date, err := e.Date()
    if err != nil {
        return ErrTimeInvalid
    }

    now := time.Now()
    if date.Unix() <= now.Unix() {
        return ErrEventInPast
    }

    ...
}

TODO-2 Use Go routine to wait for event

// Schedule an event
func (s *Scheduler) Schedule(e *Event) error {
    ...
    // fire a go routine
    go func(e *Event) {
        now := time.Now()
        target, _ := e.Date()
        waitDuration := target.Sub(now) // compare

        select {
            case <-time.After(waitDuration):
        }
    }(e)
}

TODO-3 Add Event Handler

This line below:

select {
    case <-time.After(waitDuration):
}

Will block the goroutine for waitDuration long, and then executes whatever code under the case

But we don't have the event handler handler for now, so it's time to design it.

The event handler must be supplied by the caller
Each Scheduler can only have 1 handler
Everytime an Event is triggered, we'll call the registered handler

The concept really is close with Java delegates so that's what we'll call it.

// EventHandler delegates
type EventHandler func(*Scheduler, *Event)

// Scheduler ...
type Scheduler struct {
    delegate EventHandler
}

// New instance of scheduler
func New(d EventHandler) *Scheduler {
    return &Scheduler{
        delegate: d,
    }
}

Again, we make the Scheduler immutable so the delegate can only be instantiated via New factory function, and can't be changed while it's running.

The EventHandler delegate accepts 2 parameter, Scheduler and Event. This means any delegate will always receive the correct reference to its Scheduler and Event.

And then, we complete our Schedule method:

// Schedule an event
func (s *Scheduler) Schedule(e *Event) error {
    date, err := e.Date()
    if err != nil {
        return ErrTimeInvalid
    }

    now := time.Now()
    if date.Unix() <= now.Unix() {
        return ErrEventInPast
    }

    // fire a go routine
    go func(e *Event) {
        now := time.Now()
        target, _ := e.Date()
        waitDuration := target.Sub(now) // compare

        select {
            case <-time.After(waitDuration):
                s.delegate(s, e)
        }
    }(e)
    return  nil
}

Reviewing the Requirements

It looks like we're done with the Scheduler but let's go back and look at the requirements:

It have to survive server restart

So, we'll at least have to consider Stop method which stops all scheduled events, collects it, and report it back to caller:

// Stop all running scheduler and report all pending events
func (s *Scheduler) Stop() (events []*Event) {
    // TODO:
}

TODO-1 Stop all scheduled events!

So in the select case statement in Schedule method. Not only we have to wait for Event datetime, we also have to listen to another channel which signals as out Stop event. So lets call it just that:

// Scheduler ...
type Scheduler struct {
    delegate EventHandler
    stop     chan struct{}
}

// New instance of scheduler
func New(d EventHandler) *Scheduler {
    return &Scheduler{
        delegate: d,
        // initialize stop channel
        stop:     make(chan struct{}),
    }
}

In the Schedule method we'll listen to the stop channel:

// Schedule an event
func (s *Scheduler) Schedule(e *Event) error {
    ...

    // fire a go routine
    go func(e *Event) {
        now := time.Now()
        target, _ := e.Date()
        waitDuration := target.Sub(now) // compare

        select {
            case <-time.After(waitDuration):
                s.delegate(s, e)
            case <-s.stop:
                // TODO:
        }
    }(e)
    return  nil
}

And then in the Stop method:

// Stop all running scheduler and report all pending events
func (s *Scheduler) Stop() (events []*Event) {
    // close stop channel & it will be broadcasted to all consumer
    close(s.stop)

    // TODO:
}

TODO-2 Collecting pending events

So far we have orchestrated to stop all running go routine but we haven't collected all the pending events yet.

To do this, we'll once again utilize channel. Let's call it pendings

// Scheduler ...
type Scheduler struct {
    delegate EventHandler
    stop     chan struct{}
    pendings chan *Event
}

// New instance of scheduler
func New(d EventHandler) *Scheduler {
    return &Scheduler{
        delegate: d,
        // initialize stop channel
        stop:     make(chan struct{}),
        // initialize buffered event channel
        pendings: make(chan *Event, 3),
    }
}

pendings is a buffered channel with length = 3 which means it can hold up to 3 value until producer have to wait for any consumer to fetch a value, freeing up a buffer.

// Schedule an event
func (s *Scheduler) Schedule(e *Event) error {
    ...
    // fire a go routine
    go func(e *Event) {
        ...
        select {
            ...
            case <-s.stop:
                s.pendings <- e
        }
    }(e)
    return  nil
}

And then in the Stop function we'll have to collect all pending Events being put in pendings channel

// Stop all running scheduler and report all pending events
func (s *Scheduler) Stop() (events []*Event) {
    close(s.stop)

    for e := range s.pendings {
        events = append(events, e)
    }

    return events
}

Looks good? but wait!

for e := range s.pendings

Is iterating over a channel.
BUT NOBODY is CLOSING THE pendings CHANNEL SO IT WON'T EVER QUIT!

TODO-3 Closing the pendings channel

So how do we know when to close pendings channel? We'll need to:

Manually count how many go routine was fired.
Decreement it everytime the go routine exit. Either it is done handling event, or forcefully stopped.
Watch the counter to go down to zero, to then close the pendings channel

!@#E!T#T!V@$Y

Luckily, we have sync.WaitGroup to our rescue! It does everything we listed above so let's code right away:

// Scheduler ...
type Scheduler struct {
    delegate EventHandler
    stop     chan struct{}
    pendings chan *Event
    wg       *sync.WaitGroup
}

// New instance of scheduler
func New(d EventHandler) *Scheduler {
    return &Scheduler{
        delegate: d,
        // initialize stop channel
        stop:     make(chan struct{}),
        // initialize buffered event channel
        pendings: make(chan *Event, 3),
        wg:       &sync.WaitGroup{},
    }
}

Next, we want to call wg.Add(1) every time we fire a go routine.
And We want to call wg.Done() every time a go routine exits.

// Schedule an event
func (s *Scheduler) Schedule(e *Event) error {
    ...
    s.wg.Add(1)
    // fire a go routine
    go func(e *Event) {
        ...
        defer s.wg.Done()
        select {
            ...
            case <-s.stop:
                s.pendings <- e
        }
    }(e)
    return  nil
}

Lastly! we want to watch and wait for the counter to go down to zero.

// Stop all running scheduler and report all pending events
func (s *Scheduler) Stop() (events []*Event) {
    ...
    go func() {
        s.wg.Wait()
        close(s.pendings)
    }()
    ...
}

In the Stop method, we are waiting for wg counter to drop to zero by calling wg.Wait().
And we do it in another go routine so it doesn't block the Stop execution which collects pending events.

Reviewing the Requirements again

It have to be re-usable and shared across all projects

This is a standalone go package called scheduler and can be shared to whoever needed a scheduler

It have to survive server restart

Actually because scheduler is a standalone go package we'll invert the responsibilities of persisting pending events to the user / caller.

All scheduler cant provide is just a Stop method that can be called which reports any pending events to the caller.

It have to be able to re-schedule event

By designing the EventHandler delegate to accept Scheduler and Event as its parameter. We leave the re-schedule implementation to the delegate.

The delegate can just call Scheduler.Schedule method to re-schedule any event.

Full Code of the Scheduler

package scheduler

import (
    "errors"
    "sync"
    "time"
)

// Scheduler error collection
var (
    ErrEventInPast = errors.New("Event datetime is in the past")
    ErrTimeInvalid = errors.New("Datetime format is not in RFC3339")
)

// EventHandler delegates
type EventHandler func(*Scheduler, *Event)

// Scheduler ...
type Scheduler struct {
    delegate EventHandler
    stop     chan struct{}
    pendings chan *Event
    wg       *sync.WaitGroup
}

// New instance of scheduler
func New(d EventHandler) *Scheduler {
    return &Scheduler{
        delegate: d,
        // initialize stop channel
        stop: make(chan struct{}),
        // initialize buffered event channel
        pendings: make(chan *Event, 3),
        wg:       &sync.WaitGroup{},
    }
}

// Schedule an event
func (s *Scheduler) Schedule(e *Event) error {
    date, err := e.Date()
    if err != nil {
        return ErrTimeInvalid
    }

    now := time.Now()
    if date.Unix() <= now.Unix() {
        return ErrEventInPast
    }

    s.wg.Add(1)
    // fire a go routine
    go func(e *Event) {
        now := time.Now()
        target, _ := e.Date()
        waitDuration := target.Sub(now)

        defer s.wg.Done()
        select {
        case <-time.After(waitDuration):
            s.delegate(s, e)
        case <-s.stop:
            s.pendings <- e
        }
    }(e)
    return nil
}

// Stop all running scheduler and report all pending events
func (s *Scheduler) Stop() (events []*Event) {
    close(s.stop)
    go func() {
        s.wg.Wait()
        close(s.pendings)
    }()

    for e := range s.pendings {
        events = append(events, e)
    }

    return events
}

Testing our Scheduler

func Test_Scheduler(t *testing.T) {
    one := time.Now().Add(1 * time.Second) // next 1 sec
    two := one.Add(1 * time.Second)        // next 2 sec
    tri := two.Add(1 * time.Second)        // next 3 sec
    att := []Attachment{
        {Name: "Here!"}, // we need this to test immutability
    }

    sch := New(func(s *Scheduler, e *Event) {
        if e.attachments[0].Name == "THERE!" {
            t.Error("Name should be Here! not THERE!")
        }
    })
    ev1 := NewEvent(one.Format(time.RFC3339), att)
    sch.Schedule(ev1)

    ev2 := NewEvent(two.Format(time.RFC3339), att)
    sch.Schedule(ev2)

    ev3 := NewEvent(tri.Format(time.RFC3339), att)
    sch.Schedule(ev3)

    // Name changed after all events have been scheduled
    att[0].Name = "THERE!"

    // sleep for 2 secs to leave only 3rd event as pending
    time.Sleep(2 * time.Second)
    // stop scheduler and collect the pending events
    pendings := sch.Stop()
    if len(pendings) != 1 {
        t.Error("Pendings should only left the last event")
    } else {
        last := pendings[0]
        if last != ev3 {
            t.Error("Pendings[0] should equal to ev3")
        }
    }
}

And test result shows:

Running tool: go test -timeout 30s -run ^(Test_Scheduler)$

PASS
ok      /Go/pkg/scheduler   2.512s
Success: Tests passed.

Hope you guys enjoy this case study ✌️

Gracefully Shutdown Your Go App

Robin — Thu, 05 Mar 2020 07:26:25 +0000

Shutdown Horror Story

Imagine you have a backend service which process a queue of operations.
Around 20 message comes to this queue every second.
Each operation takes around 3 seconds to complete.
And have 20 parallel workers to process each message in the queue.

And then there is light

Or no light!

Your engineering team want to deploy a hotfix to a critical bug in the worker processing code and they need it now.
Otherwise, the data processed by currenly live worker might produce wrong result.

You begin to sweat, usually deployment starts at dawn when traffic is low / no traffic at all.
You haven't got time, resource, and possibly proficiency, to properly implements blue-green nor canary delpoyment.
The currently running instance, is all you have and it needs to be restarted, introducing downtime, and possibly corrupting/losing data that is being processed right now in the worker.

Your CD pipeline, Mr. Travis (or jenkins, actions, circle, whatever) starts his workflow.

He build the backend service for you, push it to your VM instance, and then, the time comes...
He sends SIGTERM signal to your backend service.
He waits, and waits for you backend service to stop whatever it is doing, so he can close shops.
But the signal never came...

Mr. Travis have to forcefully kill your backend service by sending SIGKILL signal so OS' terminator will hunt, and kill your service without dignity.

Your backend service died while still handling N amount of messages
Your engineering team must then do an autopsy to the killed service by examining log.
Hopeful that they can find whatever messagse haven't done processing, so they can re-queue them manually.

Gracefully Shutdown your Service

Welcome to the light, young padawan.
Here, you will learn how to gracefully execute order 66...

The Concept

So, when OS (unix system), process, or people wants to shut your application down, they can send various types of signal, which then can be interrupted by our app to prepare shutthing down process.

There are lots of signal types, but we'll only focus on 2 types of signal

SIGINT or Signal Interrupt. Typically sent when a user press CTRL+C to exit app
SIGTERM or Signal Terminate. Typically sent by an app to kill another app. Most likely from administrative tools.

Both signal can be used to politely ask app to terminate their process, cleaning up any hanging operations.
OS will wait for 30s for app to shutdown, otherwise, it will send SIGKILL. Which is another type of signal that can't be intercepted and will forcefully shutdown app.

Basic Implementation in `Go`

Using go channel we can make our program wait for a signal

wait := make(chan bool)
before := time.Now()
//this will be executed asynchronously
go func() {
    time.Sleep(3 * time.Second)
    wait <- false
}()

<-wait
fmt.Println("I am done")
fmt.Println(int(time.Since(before).Seconds()), "sec")

bastianrob$ go run main.go
I am done
3 sec

And then in go, we can listen to OS' signal by using

sig := make(chan os.Signal)
signal.Notify(sig, syscall.SIGINT, syscall.SIGTERM)

Notice that sig is a channel that waits from SIGINT and SIGTERM signal.
We'll use this to setup our teardown process

package gracefully

// Serve HTTP gracefuly
func Serve(listenAndServe func() error, teardown func(context.Context) error) error {
    term := make(chan os.Signal) // OS termination signal
    fail := make(chan error)     // Teardown failure signal

    go func() {
        signal.Notify(term, syscall.SIGINT, syscall.SIGTERM)
        <-term // waits for termination signal

        // context with 30s timeout
        ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
        defer cancel()

        // all teardown process must complete within 30 seconds
        fail <- teardown(ctx)
    }()

    // listenAndServe blocks our code from exit, but will produce ErrServerClosed when stopped
    if err := listenAndServe(); err != nil && err != http.ErrServerClosed {
        return err
    }

    // after server gracefully stopped, code proceeds here and waits for any error produced by teardown() process @ line 26
    return <-fail
}

And in the main.go:

func main() {
    server := &http.Server{
        Addr: ":8080",
        // Handler: your API handler
    }

    if err := gracefully.Serve(server.ListenAndServe, func(ctx context.Context) error {
        if err := server.Shutdown(ctx); err != nil {
            return err
        }

        // unplug from message broker
        // unplug from service mesh
        // remove temporary files
        // wait for all pending queue/topic processor to finish
        // etc, yada-yada

        return nil
    }); err != nil {
        log.Fatalln("ERR:", err)
    }
}

RBAC in REST API using Go

Robin — Tue, 03 Mar 2020 06:01:16 +0000

Role Based Access Control, or RBAC.
Might be one of the more interesting challenge I've faced. It is:

Needed by business
Complicated
But quite rewarding to solve
While felt over-engineered at the same time

Dahe** is RBAC

Imagine a scenario where you have 4 roles inside your company, which have access to 1 type of resource.
Let's say:

Manager
Ops
CS
Client

These roles have different kind of interaction with one type of resource. Let's say it's an Inquiry, where:

Client can submit New Inquiry
CS views Inquiry from Client an then assign it to an Ops
Client can only view his own Inquiry
CS can only view Inquiry with status = New
Ops can only view Inquiry assigned to them
Manager can view all Inquiry which have been assigned to Ops

This is what RBAC is, where we try to Control Access to a resource/action Based on Role

Hence the name: Role Based Access Control

Minimum Effort Implementation

The easiest way to implement this is to filter resource access through Frontend:

As a Client I must ONLY be able to view my Inquiry
As a CS I must ONLY be able to view New Inquiry
As an Ops I must ONLY be able to view Inquiry assigned to me
As a Manager I must ONLY be able to view Inquiry assigned to Ops

Neat right? Looks like some JIRA stories for Frontend.
BUT NOT SO FAST...

If there's no restriction enforced by Backend, then:

Let's say Ops A knows how to construct a query, therefore
All Inquiry data can be leaked just by querying GET /inquiries
There could be sensitive / confidential data which must never be able to seen by other than specified role
e.g: Client's profile, address, phone, etc...

This is why RBAC must be implemented by Backend

REST API

In typical REST API we usually creates endpoint based on a Resource (hence the R from REST... J.K. It's not!)

Example:

'GET /inquiries?param1={v1}&param2={v2}...' # Get list of inquiries based on some query parameter
'POST /inquiries'                           # Create a new inquiry
'POST /inquiries/{id}/assign'               # Assign an inquiry to a someone

With Inquiry data structure:

{
    "id": "INQ-0001",
    "created_by": "client@email.com",
    "status": "Assigned",
    "assignee": "ops.one@company.email"
}

And we want to enforce RBAC on these endpoints

Mapping the rules

Now we have:

Four roles
Lots of rules
Three endpoints, and
Inquiry data structure

Next we'll have to map permission to a role based on guideline below:

what's my role? what resource am I trying to access? at what endpoint? is it allowed? if it is, what's the rule?

So, let's start from `Client's` rule

We'll use YAML to structure our rule

client:
  inquiry:
    get:
      allow: true
      ensure:
        query:
          - key: created_by
            operator: "="
            value: "ctx.email"

    create:
      allow: true

    assign:
      allow: false

Now, we have a rule for Client, how do we read it?

* If I am a `client`, want to access `inquiry`:
  * For endpoint `get`, I am allowed, only when `url query`
    * Contains a key named `create_by`
    * And the value must be `=` to value of `ctx.email`
  * For endpoint `create`, I am allowed, without restriction
  * For endpoint `assign`, I am not allowed to access

Next, `CS'` rule

cs:
  inquiry:
    get:
      allow: true
      enforce:
        query:
          - key: status
            value: "New"

    create:
      allow: false

    assign:
      allow: true

We read it as:

* If I am a `cs`, want to access `inquiry`:
  * For endpoint `get`, I am allowed, but `url query`
    * Will be enforced with a key named `status`
    * And the value is `status=New`
  * For endpoint `create`, I am not allowed to access
  * For endpoint `assign`, I am allowed, without restriction

Next, `OPS'` rule

ops:
  inquiry:
    get:
      allow: true
      ensure:
        query:
          - key: assignee
            operator: "="
            value: "ctx.email"

    create:
      allow: false

    assign:
      allow: false

We read it as:

* If I am an `ops`, want to access `inquiry`:
  * For endpoint `get`, I am allowed, only when `url query`
    * Contains a key name `assignee`
    * And the value must be `=` to value of `ctx.email`
  * For endpoint `create`, I am not allowed to access
  * For endpoint `assign`, I am not allowed to access

Lastly, `Manager's` rule

manager:
  inquiry:
    get:
      allow: true
      enforce:
        query:
          - key: status
            value: "Assigned"

    create:
      allow: false

    assign:
      allow: true

We read it as:

* If I am a `manager`, want to access `inquiry`:
  * For endpoint `get`, I am allowed, but `url query`
    * Will be enforced with a key named `status`
    * And the value is `status=Assigned`
  * For endpoint `create`, I am not allowed to access
  * For endpoint `assign`, I am allowed, without restriction

ctx is context.Context object in golang which we usually pass around from end to end

RBAC engine

Now comes the part where we should code the RBAC engine to check those rules we have listed.

Let's start from Rule

Rule is the smallest unit in our YAML
It is used to compare actual incoming request, VS expectation we have set in Rule

YAML example:

key: status
operator: "="
value: "New"
--------
key: created_by
operator: "="
value: "ctx.email"

Go code equivalent of a Rule:

  // Rule of a permission
  type Rule struct {
      Key      string `yaml:"key"`
      Operator string `yaml:"operator"`
      Value    string `yaml:"value"`
  }

From the YAML we have 2 types of Rule

Rule.Value = {a string} e.g: status=New
Rule.Value = ctx.{field} e.g: created_by=ctx.email

For the first type, we take the rule value as is
But for the 2nd type, we have to take the expected value from context object, splitted by .

So we have to prepare a method, owned by Rule, to get expected value, from context.
Let's name it FromContext

  // FromContext get actual rule.Value from ctx if rule.Value starts with ctx
  // otherwise, return rule.Value as is
  func (rule Rule) FromContext(ctx context.Context) interface{} {
      ...
  }

Next, we write unit tests scenario to ensure FromContext behave as we wanted it to be:

  // Semi BDD style unit testing
  // I think the code is self explanatory, we just wanted to:
  // call rule.FromContext(ctx), and
  //   want it to either panics, or
  //   produce a correct result
  func TestRule_FromContext(t *testing.T) {
      tests := []struct {
          given  string
          then   string
          rule   rbac.Rule
          ctx    func() context.Context
          want   interface{}
          panics bool
      }{{
          given: "Non ctx rule.Value", then: "return value should be rule.Value as is",
          rule: rbac.Rule{Value: "something"},
          ctx:  func() context.Context { return context.Background() },
          want: "something",
      }, {
          given: "rule.Value with ctx", then: "return value should be taken from ctx",
          rule: rbac.Rule{Value: "ctx.email"},
          ctx: func() context.Context {
              return context.WithValue(context.Background(), rbac.ContextKey("email"), "someone@email.com")
          },
          want: "someone@email.com",
      }, {
          given: "rule.Value with deep nested ctx", then: "return value should be taken from ctx",
          rule: rbac.Rule{Value: "ctx.access.id"},
          ctx: func() context.Context {
              return context.WithValue(context.Background(), rbac.ContextKey("access"), map[string]interface{}{
                  "id": "IDX-0001",
              })
          },
          want: "IDX-0001",
      }, {
          given: "rule.Value with deep nested ctx, but at 4th level its not a map", then: "code should panic",
          rule: rbac.Rule{Value: "ctx.access.id.name"},
          ctx: func() context.Context {
              return context.WithValue(context.Background(), rbac.ContextKey("access"), map[string]interface{}{
                  "id": "IDX-0001",
              })
          },
          panics: true,
      }, {
          given: "rule.Value with deep nested ctx, but does not exists", then: "code should panic",
          rule: rbac.Rule{Value: "ctx.something.not.exists"},
          ctx: func() context.Context {
              return context.Background()
          },
          panics: true,
      }}
      for _, tt := range tests {
          t.Run(tt.given, func(t *testing.T) {
              if !tt.panics {
                  got := tt.rule.FromContext(tt.ctx())
                  assert.Equal(t, tt.want, got, tt.then)
              } else {
                  assert.Panics(t, func() {
                      tt.rule.FromContext(tt.ctx())
                  }, tt.given)
              }
          })
      }
  }

Next, we actually write the code to satisfy these test scenario

  // FromContext get actual rule.Value from ctx if rule.Value starts with ctx
  // otherwise, return rule.Value as is
  func (rule Rule) FromContext(ctx context.Context) interface{} {
      if !strings.HasPrefix(rule.Value, "ctx") {
          return rule.Value
      }

      paths := strings.Split(rule.Value, ".")
      var ctxval interface{}

      // starts from 1, as we exclude the ctx part
      for i := 1; i < len(paths); i++ {
          ctxkey := paths[i]

          //Get current context index
          if i == 1 {
              ctxval = ctx.Value(ContextKey(ctxkey))
          } else {
              // if rule.Value is nested more than 1 level, we assume the context value is of type map[string]interface{}
              // otherwise, panic
              var ok bool
              kvp := ctxval.(map[string]interface{})
              ctxval, ok = kvp[ctxkey]
              if !ok || ctxval == nil {
                  ctxval = nil
              }
          }
      }

      return ctxval
  }

Running the test (from VS Code)

  Running tool: /usr/local/opt/go/libexec/bin/go test -timeout 30s github.com/bastianrob/go-experiences/rbac -run ^(TestRule_FromContext)$

  ok      github.com/bastianrob/go-experiences/rbac
  Success: Tests passed.

Now imagine we have to actually check whether a Request complies with given Rule or not
Let's make another method for Rule and name it Comply

  // Comply checks does request value complies with our rule
  func (rule Rule) Comply(expected, actual interface{}) bool {
      ...
  }

We write another unit tests scenario to ensure Comply behave as we wanted it to be:

  func TestRule_Comply(t *testing.T) {
      type args struct {
          expected interface{}
          actual   interface{}
      }
      tests := []struct {
          given string
          then  string
          rule  rbac.Rule
          args  args
          want  bool
      }{{
          given: "With rule: actual must be = expected", then: "query complies with our rule",
          rule: rbac.Rule{
              Operator: "=",
          },
          args: args{
              expected: "something",
              actual:   "something",
          },
          want: true,
      }, {
          given: "With rule: actual must be != expected", then: "query complies with our rule",
          rule: rbac.Rule{
              Operator: "!=",
          },
          args: args{
              expected: "something",
              actual:   "another",
          },
          want: true,
      }, {
          given: "With rule operator not known", then: "query does not complies",
          rule: rbac.Rule{
              Operator: "unknwon",
          },
          want: false,
      }}
      for _, tt := range tests {
          t.Run(tt.given, func(t *testing.T) {
              got := tt.rule.Comply(tt.args.expected, tt.args.actual)
              assert.Equal(t, tt.want, got, tt.then)
          })
      }
  }

Again, we actually write the code to satisfy these test scenario

  // Comply checks does request value complies with our rule
  func (rule Rule) Comply(expected, actual interface{}) bool {
      switch rule.Operator {
      case "!=":
          return !reflect.DeepEqual(expected, actual)
      case "=":
          return reflect.DeepEqual(expected, actual)
      }

      // doesn't comply if we don't recognize the rule operator
      return false
  }

Running the test (from VS Code)

  Running tool: /usr/local/opt/go/libexec/bin/go test -timeout 30s github.com/bastianrob/go-experiences/rbac -run ^(TestRule_Comply)$

  ok      github.com/bastianrob/go-experiences/rbac
  Success: Tests passed.

Ensurer and Enforcer

Next, we have Ensurer and Enforcer. As the name implies:

Ensurer ensure that request must complies with all the rules attached in Ensurer
Enforcer enforce that no matter what you request, we'll always enforce all the rules attached in Enforcer
Both can be targeted to either query, header, or path
And both contains list of Rules we have defined in the previous point

// Ensurer data model
// Can either ensure query, header, or path
type Ensurer struct {
    Query  []Rule `yaml:"query"`
    Header []Rule `yaml:"header"`
    Path   []Rule `yaml:"path"`
}


// Enforcer structure is just like an Ensurer
type Enforcer Ensurer

For this example, we'll only implement Query Ensurer and Enforcer

// QueryComplies check whether query request complies with rules
func (ens Ensurer) QueryComplies(r *http.Request) error {
    ...
}

// QueryComplies enforce query request from rule
func (enf Enforcer) QueryComplies(r *http.Request) error {
    ...
}

Then, we write unit tests to ensure QueryComplies behave as we wanted it to be:

func TestEnsurer_QueryComplies(t *testing.T) {
    type args struct {
        method string
        url    string
    }
    tests := []struct {
        given   string
        then    string
        ensurer rbac.Ensurer
        context func() context.Context
        args    args
        wantErr bool
    }{{
        given: "Query: id=0001&name=John and Rule: id=0001&name=ctx.name and ctx.name=John",
        then:  "QueryComplies must not return error",
        args: args{
            // query: {id: "0001", name: "John"}
            url: "http://api.example.com/resources?id=0001&name=John",
        },
        ensurer: rbac.Ensurer{
            Query: []rbac.Rule{
                // id IS 0001, and name EQUALS to value stored in context.name
                {Key: "id", Operator: "=", Value: "0001"},
                {Key: "name", Operator: "=", Value: "ctx.name"},
            },
        },
        context: func() context.Context {
            // we give the context.name = "John"
            return context.WithValue(context.Background(), rbac.ContextKey("name"), "John")
        },
    }}
    for _, tt := range tests {
        t.Run(tt.given, func(t *testing.T) {
            r, _ := http.NewRequest(tt.args.method, tt.args.url, nil)
            r = r.WithContext(tt.context())

            err := tt.ensurer.QueryComplies(r)
            if tt.wantErr {
                assert.Error(t, err, tt.given)
            } else {
                assert.NoError(t, err, tt.given)
            }
        })
    }
}

func TestEnforcer_QueryComplies(t *testing.T) {
    type args struct {
        method string
        url    string
    }
    tests := []struct {
        given    string
        then     string
        enforcer rbac.Enforcer
        context  func() context.Context
        args     args
        want     map[string]string
        wantErr  bool
    }{{
        given: "Query: id=nil&name=nil and Rule: id=0001&name=ctx.name and ctx.name=John",
        then:  "QueryComplies must not return error, and query must be re-written by enforcer",
        args: args{
            url: "http://api.example.com/resources?id=nil&name=nil",
        },
        enforcer: rbac.Enforcer{
            Query: []rbac.Rule{
                // query: {id: "0001", name: "John"}
                {Key: "id", Value: "0001"},
                {Key: "name", Value: "ctx.name"},
            },
        },
        context: func() context.Context {
            // we give the context.name = "John"
            return context.WithValue(context.Background(), rbac.ContextKey("name"), "John")
        },
        want: map[string]string{
            "id":   "0001",
            "name": "John",
        },
    }}
    for _, tt := range tests {
        t.Run(tt.given, func(t *testing.T) {
            r, _ := http.NewRequest(tt.args.method, tt.args.url, nil)
            r = r.WithContext(tt.context())

            err := tt.enforcer.QueryComplies(r)
            if tt.wantErr {
                assert.Error(t, err, tt.given)
            } else {
                assert.NoError(t, err, tt.given)
                assert.Equal(t, len(tt.want), len(r.URL.Query()))
                for key, val := range tt.want {
                    assert.Equal(t, val, r.URL.Query().Get(key), tt.then)
                }
            }
        })
    }
}

Then, we write actual QueryComplies code which satisfy our unit tests:

// QueryComplies check whether query request complies with rules
func (ens Ensurer) QueryComplies(r *http.Request) error {
    if ens.Query == nil || len(ens.Query) <= 0 {
        return nil
    }

    ctx := r.Context()
    for _, rule := range ens.Query {
        actual := r.URL.Query().Get(rule.Key)
        expected := rule.FromContext(ctx)

        if !rule.Comply(expected, actual) {
            return fmt.Errorf("Query rule violation: ensure '%s' %s '%v', instead got: '%s'",
                rule.Key, rule.Operator, expected, actual)
        }
    }

    // all query complies with rules
    return nil
}

// QueryComplies enforce query request from rule
func (enf Enforcer) QueryComplies(r *http.Request) error {
    q := r.URL.Query()
    ctx := r.Context()
    for _, rule := range enf.Query {
        expected := rule.FromContext(ctx)
        valueStr, isString := expected.(string)
        if !isString {
            return ErrNotString
        }

        q.Set(rule.Key, valueStr)
    }

    r.URL.RawQuery = q.Encode()
    // all query enforced with rules
    return nil
}

And the test result is:

Running tool: /usr/local/opt/go/libexec/bin/go test -timeout 30s github.com/bastianrob/go-experiences/rbac -run ^(TestEnsurer_QueryComplies)$

ok      github.com/bastianrob/go-experiences/rbac   0.014s
Success: Tests passed.

--------

Running tool: /usr/local/opt/go/libexec/bin/go test -timeout 30s github.com/bastianrob/go-experiences/rbac -run ^(TestEnforcer_QueryComplies)$

ok      github.com/bastianrob/go-experiences/rbac    0.014s
Success: Tests passed.

Stitching it all together

Now we have:

Ensurer and Enforcer
Each have their own set of Rules
Lastly we need to stitch them all together in an RBAC object

// Error collection
var (
    ErrNotString   = errors.New("Expected value is not a string")
    ErrNoRole      = errors.New("You have no role assigned to you")
    ErrRoleUnknown = errors.New("You have an unknown role assigned to you")
    ErrForbidden   = errors.New("You are not allowed to access specified resource")
)


// Permission of an endpoint
type Permission struct {
    Allow   bool     `yaml:"allow"`
    Ensure  Ensurer  `yaml:"ensure,omitempty"`
    Enforce Enforcer `yaml:"enforce,omitempty"`
}

// Endpoint is a map of {endpoint: permission}
type Endpoint map[string]Permission

// Resource is a map of {resource: endpoint}
type Resource map[string]Endpoint

// RBAC is a map of {role: resource}
type RBAC map[string]Resource

// FromFile creates a new RBAC object from .yaml file
func FromFile(path string) *RBAC {
    f, err := ioutil.ReadFile(path)
    if err != nil {
        return nil
    }

    rbac := &RBAC{}
    err = yaml.Unmarshal(f, rbac)
    if err != nil {
        return nil
    }

    return rbac
}

Now we have an actual object called RBAC, we can parse YAML rule into RBAC using FromFile factory function.

Next, we have to write Authorize method for RBAC

// Authorize a request based on its role, resource, and endpoint
func (rbac RBAC) Authorize(r *http.Request, role, resource, endpoint string) error {
    ...
}

And we write a unit tests for Authorize:

func TestRBAC_Authorize(t *testing.T) {
    rbo := rbac.FromFile("./test.yaml")
    fmt.Printf("%+v", rbo)

    type args struct {
        req      func() *http.Request
        role     string
        resource string
        endpoint string
    }
    tests := []struct {
        given, when, then string
        args              args
        wantErr           bool
        queryResult       map[string]string
    }{
        // As a client
        {
            given: "Role is Client & email = client.one@email.com",
            when:  "?created_by=client.one@email.com", then: "is allowed",
            args: args{
                req: func() *http.Request {
                    req, _ := http.NewRequest("", "http://api.example.com/inquiries?created_by=client.one@email.com", nil)
                    ctx := context.WithValue(context.Background(), rbac.ContextKey("email"), "client.one@email.com")

                    return req.WithContext(ctx)
                },
                role:     "client",
                resource: "inquiry",
                endpoint: "get",
            },
        }, {
            given: "Role is Client & email = client.one@email.com",
            when:  "?created_by=client.other@email.com", then: "is not allowed",
            args: args{
                req: func() *http.Request {
                    req, _ := http.NewRequest("", "http://api.example.com/inquiries?created_by=client.other@email.com", nil)
                    ctx := context.WithValue(context.Background(), rbac.ContextKey("email"), "client.one@email.com")

                    return req.WithContext(ctx)
                },
                role:     "client",
                resource: "inquiry",
                endpoint: "get",
            },
            wantErr: true,
        }, {
            given: "Role is Client & email = client.one@email.com",
            when:  "query is not given", then: "is not allowed",
            args: args{
                req: func() *http.Request {
                    req, _ := http.NewRequest("", "http://api.example.com/inquiries", nil)
                    ctx := context.WithValue(context.Background(), rbac.ContextKey("email"), "client.one@email.com")

                    return req.WithContext(ctx)
                },
                role:     "client",
                resource: "inquiry",
                endpoint: "get",
            },
            wantErr: true,
        }, {
            given: "Role is Client & email = client.one@email.com",
            when:  "trying to create", then: "is allowed",
            args: args{
                req: func() *http.Request {
                    req, _ := http.NewRequest("POST", "http://api.example.com/inquiries", nil)
                    ctx := context.WithValue(context.Background(), rbac.ContextKey("email"), "client.one@email.com")

                    return req.WithContext(ctx)
                },
                role:     "client",
                resource: "inquiry",
                endpoint: "create",
            },
        }, {
            given: "Role is Client & email = client.one@email.com",
            when:  "trying to assign", then: "is not allowed",
            args: args{
                req: func() *http.Request {
                    req, _ := http.NewRequest("POST", "http://api.example.com/inquiries/INQ-0001/assign", nil)
                    ctx := context.WithValue(context.Background(), rbac.ContextKey("email"), "client.one@email.com")

                    return req.WithContext(ctx)
                },
                role:     "client",
                resource: "inquiry",
                endpoint: "assign",
            },
            wantErr: true,
        },

        // As CS
        {
            given: "Role is CS",
            when:  "query is not given", then: "status=New is enforced",
            args: args{
                req: func() *http.Request {
                    req, _ := http.NewRequest("", "http://api.example.com/inquiries", nil)
                    ctx := context.WithValue(context.Background(), rbac.ContextKey("email"), "cs.one@company.com")

                    return req.WithContext(ctx)
                },
                role:     "cs",
                resource: "inquiry",
                endpoint: "get",
            },
            queryResult: map[string]string{
                "status": "New",
            },
        }, {
            given: "Role is CS",
            when:  "query ?status is given", then: "status=New is still enforced",
            args: args{
                req: func() *http.Request {
                    req, _ := http.NewRequest("", "http://api.example.com/inquiries?status=Assigned", nil)
                    ctx := context.WithValue(context.Background(), rbac.ContextKey("email"), "cs.one@company.com")

                    return req.WithContext(ctx)
                },
                role:     "cs",
                resource: "inquiry",
                endpoint: "get",
            },
            queryResult: map[string]string{
                "status": "New",
            },
        }, {
            given: "Role is CS",
            when:  "trying to create", then: "is not allowed",
            args: args{
                req: func() *http.Request {
                    req, _ := http.NewRequest("POST", "http://api.example.com/inquiries", nil)
                    ctx := context.WithValue(context.Background(), rbac.ContextKey("email"), "cs.one@company.com")

                    return req.WithContext(ctx)
                },
                role:     "cs",
                resource: "inquiry",
                endpoint: "create",
            },
            wantErr: true,
        }, {
            given: "Role is CS",
            when:  "trying to assign", then: "is allowed",
            args: args{
                req: func() *http.Request {
                    req, _ := http.NewRequest("POST", "http://api.example.com/inquiries/INQ-0001/assign", nil)
                    ctx := context.WithValue(context.Background(), rbac.ContextKey("email"), "cs.one@company.com")

                    return req.WithContext(ctx)
                },
                role:     "cs",
                resource: "inquiry",
                endpoint: "assign",
            },
        },

        // As an Ops
        {
            given: "Role is Ops & email = ops.one@company.com",
            when:  "?assignee=ops.one@company.com", then: "is allowed",
            args: args{
                req: func() *http.Request {
                    req, _ := http.NewRequest("", "http://api.example.com/inquiries?assignee=ops.one@company.com", nil)
                    ctx := context.WithValue(context.Background(), rbac.ContextKey("email"), "ops.one@company.com")

                    return req.WithContext(ctx)
                },
                role:     "ops",
                resource: "inquiry",
                endpoint: "get",
            },
        }, {
            given: "Role is Ops & email = ops.one@company.com",
            when:  "?assignee=ops.other@company.com", then: "is not allowed",
            args: args{
                req: func() *http.Request {
                    req, _ := http.NewRequest("", "http://api.example.com/inquiries?assignee=ops.other@company.com", nil)
                    ctx := context.WithValue(context.Background(), rbac.ContextKey("email"), "ops.one@company.com")

                    return req.WithContext(ctx)
                },
                role:     "ops",
                resource: "inquiry",
                endpoint: "get",
            },
            wantErr: true,
        }, {
            given: "Role is Ops & email = ops.one@company.com",
            when:  "query is not supplied", then: "is not allowed",
            args: args{
                req: func() *http.Request {
                    req, _ := http.NewRequest("", "http://api.example.com/inquiries", nil)
                    ctx := context.WithValue(context.Background(), rbac.ContextKey("email"), "ops.one@company.com")

                    return req.WithContext(ctx)
                },
                role:     "ops",
                resource: "inquiry",
                endpoint: "get",
            },
            wantErr: true,
        }, {
            given: "Role is Ops & email = ops.one@company.com",
            when:  "trying to created", then: "is not allowed",
            args: args{
                req: func() *http.Request {
                    req, _ := http.NewRequest("POST", "http://api.example.com/inquiries", nil)
                    ctx := context.WithValue(context.Background(), rbac.ContextKey("email"), "ops.one@company.com")

                    return req.WithContext(ctx)
                },
                role:     "ops",
                resource: "inquiry",
                endpoint: "create",
            },
            wantErr: true,
        }, {
            given: "Role is Ops & email = ops.one@company.com",
            when:  "trying to assign", then: "is not allowed",
            args: args{
                req: func() *http.Request {
                    req, _ := http.NewRequest("POST", "http://api.example.com/inquiries/INQ-0001/assign", nil)
                    ctx := context.WithValue(context.Background(), rbac.ContextKey("email"), "ops.one@company.com")

                    return req.WithContext(ctx)
                },
                role:     "ops",
                resource: "inquiry",
                endpoint: "assign",
            },
            wantErr: true,
        },

        // As a manager
        {
            given: "Role is Manager",
            when:  "query is not given", then: "status=Assigned is enforced",
            args: args{
                req: func() *http.Request {
                    req, _ := http.NewRequest("", "http://api.example.com/inquiries", nil)
                    ctx := context.WithValue(context.Background(), rbac.ContextKey("email"), "manager@company.com")

                    return req.WithContext(ctx)
                },
                role:     "manager",
                resource: "inquiry",
                endpoint: "get",
            },
            queryResult: map[string]string{
                "status": "Assigned",
            },
        }, {
            given: "Role is Manager",
            when:  "query ?status is given", then: "status=Assigned is still enforced",
            args: args{
                req: func() *http.Request {
                    req, _ := http.NewRequest("", "http://api.example.com/inquiries?status=New", nil)
                    ctx := context.WithValue(context.Background(), rbac.ContextKey("email"), "manager@company.com")

                    return req.WithContext(ctx)
                },
                role:     "manager",
                resource: "inquiry",
                endpoint: "get",
            },
            queryResult: map[string]string{
                "status": "Assigned",
            },
        }, {
            given: "Role is Manager",
            when:  "trying to create", then: "is not allowed",
            args: args{
                req: func() *http.Request {
                    req, _ := http.NewRequest("POST", "http://api.example.com/inquiries", nil)
                    ctx := context.WithValue(context.Background(), rbac.ContextKey("email"), "manager@company.com")

                    return req.WithContext(ctx)
                },
                role:     "manager",
                resource: "inquiry",
                endpoint: "create",
            },
            wantErr: true,
        }, {
            given: "Role is Manager",
            when:  "trying to assign", then: "is allowed",
            args: args{
                req: func() *http.Request {
                    req, _ := http.NewRequest("POST", "http://api.example.com/inquiries/INQ-0001/assign", nil)
                    ctx := context.WithValue(context.Background(), rbac.ContextKey("email"), "manager@company.com")

                    return req.WithContext(ctx)
                },
                role:     "manager",
                resource: "inquiry",
                endpoint: "assign",
            },
        },
    }
    for _, tt := range tests {
        t.Run(tt.given, func(t *testing.T) {
            req := tt.args.req()
            got := rbo.Authorize(req, tt.args.role, tt.args.resource, tt.args.endpoint)
            if tt.wantErr {
                assert.Error(t, got, "when: %s, then: %s", tt.when, tt.then)
            } else {
                assert.NoError(t, got, "when: %s, then: %s", tt.when, tt.then)
                for key, val := range tt.queryResult {
                    assert.Equal(t, val, req.URL.Query().Get(key), "when: %s, then: %s", tt.when, tt.then)
                }
            }
        })
    }
}

Last, we write Authorize code to satisf our unit tests

// Authorize a request based on its role, resource, and endpoint
func (rbac RBAC) Authorize(r *http.Request, role, resource, endpoint string) error {
    permission, exists := rbac[role][resource][endpoint]
    if !exists {
        return ErrRoleUnknown
    }

    if !permission.Allow {
        return ErrForbidden
    }

    // Ensure query compliance
    err := permission.Ensure.QueryComplies(r)
    if err != nil {
        return err
    }

    // Enforce query compliance
    err = permission.Enforce.QueryComplies(r)
    if err != nil {
        return err
    }

    return nil
}

And the test results shows:

Running tool: /usr/local/opt/go/libexec/bin/go test -timeout 30s github.com/bastianrob/go-experiences/rbac -run ^(TestRBAC_Authorize)$

ok      github.com/bastianrob/go-experiences/rbac    (cached)
Success: Tests passed.

So, Again What the Hell is `ctx`

We see a lot of ctx being thrown around so, let me explain it in chronologically, from end-to-end

* Think of `ctx` as browser `cookies`
* As a user logged in our application, we respond to them by setting `cookie` into the browser
* This `cookie` is typically a string of encrypted data about logged in user's profile (think `JWT`)
* This `cookie` will always get sent to server for each and every request to our domain
* At the start of the request, our code have to:
  * Check existence of `cookie`
    * Returns 401 if not exists (unauthorized)
    * Proceed if exists
* Unwrap the `cookie` value, which in our case typpically contains `role` and `email` of the logged in user
* And then, we set all of the unwrapped `cookie` value into HTTP request `context`
* In this case we set `ctx.name`
* Then, we call the `RBAC` engine to `Authorize` the request
  * `RBAC` ensures the request satisfy all rules inside the `Ensurer`
  * `RBAC` enforce all rules inside the `Enforcer` by rewriting the HTTP request object
* If HTTP request pass all the rules in `RBAC` engine, we then proceed to pass the HTTP request to `Business Service Layer`
* Because `RBAC` engine relies heavily on the HTTP query, `Business Service Layer` must be written to always converts the HTTP query into actual `Database Query`. e.g:
  * `/inquiries?created_by=client@email.com` must be translated to `SELECT * FROM inquiries WHERE created_by = @url.query.created_by`
  * `/inquiries?status=New` must be translated to `SELECT * FROM inquiries WHERE status = @url.query.status`

What does this all means? This is nuts

Let us take a journey and assume position as each of the roles

As a `Client`

client:
  inquiry:
    get:
      allow: true
      ensure:
        query:
          - key: created_by
            operator: "="
            value: "ctx.email"

    create:
      allow: true

    assign:
      allow: false

* I am currently logged in as `client.one@email.com`
* I am trying to access `GET /inquiries`
  * `RBAC` states: `ensure query: created_by = ctx.email`
  * `GET /inquiries` doesn't have any query, therefore rule is not satisfied
  * Returned as `Unauthorized`
* I am trying to access `GET /inquiries?created_by=client.other@email.com`
  * `RBAC` states: `ensure query: created_by = ctx.email`
  * `GET /inquiries?created_by=client.other@email.com` have the `created_by` query and will be compared against `ctx.email`
  * `ctx.email` is `client.one@gmail.com`, but supplied query is `client.other@email.com`, therefore rule is not satisfied
  * Returned as `Unauthorized`

So now as a Client even though I can read and construct a query, I still can't get any data from other people

As a `CS`

cs:
  inquiry:
    get:
      allow: true
      enforce:
        query:
          - key: status
            value: "New"

    create:
      allow: false

    assign:
      allow: true

* I am currently logged in as `cs.one@company.com`
* I am trying to access `GET /inquiries`
  * `RBAC` states: `enforce query: status = New`
  * `GET /inquiries` doesn't have any query, but `Enforcer` will forcefully write it as `?status=New`
* I am trying to access `GET /inquiries?status=Assigned`
  * `RBAC` states: `enforce query: status = New`
  * `GET /inquiries?status=Assigned` have the `status` query and the value is `Assigned`
  * `Enforcer` doesn't care with the value requested, and will forcefully re-write it as `?status=New`

So now as a CS even though I can read and construct a query, I can only get inquiries with status=New

As an `Ops`

ops:
  inquiry:
    get:
      allow: true
      ensure:
        query:
          - key: assignee
            operator: "="
            value: "ctx.email"

    create:
      allow: false

    assign:
      allow: false

* I am currently logged in as `ops.one@company.com`
* I am trying to access `GET /inquiries`
  * `RBAC` states: `ensure query: assignee = ctx.email`
  * `GET /inquiries` doesn't have any query, therefore rule is not satisfied
  * Returned as `Unauthorized`
* I am trying to access `GET /inquiries?assignee=ops.other@company.com`
  * `RBAC` states: `ensure query: status = ctx.email`
  * `GET /inquiries?status=ops.other@company.com` have the `created_by` query and will be compared against `ctx.email`
  * `ctx.email` is `ops.one@company.com`, but supplied query is `ops.other@company.com`, therefore rule is not satisfied
  * Returned as `Unauthorized`

So now as an Ops even though I can read and construct a query, I still can't get any data from any other Ops

As a `Manager`

manager:
  inquiry:
    get:
      allow: true
      enforce:
        query:
          - key: status
            value: "Assigned"

    create:
      allow: false

    assign:
      allow: true

* I am currently logged in as `manager@company.com`
* I am trying to access `GET /inquiries`
  * `RBAC` states: `enforce query: status = Assigned`
  * `GET /inquiries` doesn't have any query, but `Enforcer` will forcefully write it as `?status=Assigned`
* I am trying to access `GET /inquiries?status=New`
  * `RBAC` states: `enforce query: status = Assigned`
  * `GET /inquiries?status=New` have the `status` query and the value is `New`
  * `Enforcer` doesn't care with the value requested, and will will forcefully re-write it as `?status=Assigned`

So now as a Manager even though I can read and construct a query, I can only get inquiries with status=Assigned

MONGOREPO Abstraction Using Generic Constructor in Go

Robin — Mon, 02 Mar 2020 16:02:23 +0000

CRUD to resource

Typically we have this kind of CRUD endpoints to a resource:

'GET    /{resources}'       # Get a list of resource
'GET    /{resources}/{id}'  # Get one resource based on its ID
'POST   /{resource}'        # Create a new resource
'PUT    /{resource}/{id}'   # Update an existing resource based on its ID
'DELETE /{resource/{id}'    # Flag an existing resource as virtually deleted based on its ID

Which then brings us to the next point

Repository Pattern

Imagine a scenario where we have database object named Person

type Person struct {
    ID   primitive.ObjecID `bson:"_id,omitempty"`
    Name string            `bson:"name"`
}

With Repository Pattern typically we'll define a repository struct which mimics our CRUD operations:

type PersonRepo struct {}

func(r *PersonRepo) Get(ctx context.Context) ([]*Person, error) {}
func(r *PersonRepo) GetOne(ctx context.Context, id string) (*Person, error) {}
func(r *PersonRepo) Create(ctx context.Context, p *Person) error {}
func(r *PersonRepo) Update(ctx context.Context, p *Person) error {}
func(r *PersonRepo) Delete(ctx context.Context, id string) error {}

Abstraction, or the lack thereof

Imagine a scenario where we have more (let's say N amount) database object.
Each of them have similar CRUD signature and need their own repository.

In C# or Java who have generic, we can usually code something like this:

public class Repo<T> {
    public T Get() {}
    public T GetOne(string id) {}
    public void Create(T obj) {}
    public void Update(T obj) {}
    public void Delete(string id) {}
}

Where Repo<T> can be instantiated at runtime (or compile time) for diferrent type of dbo:

var personRepo = new Repo<Person>();
var enemyRepo = new Repo<Enemy>();
// etc, yada-yada, whatever

But here in Golang:

We don't have generic
Not OOP

We (read: software engineer/developer/programmer/coder drones/highly trained monkeys/whatever) often need to type type {Resource}Repo struct{...} N amount of times.

One solution is to learn meta programming but that's not the topic I'll touch today...

Now how do we abstract Repo<T> in Go?

First, we define the MongoRepo struct along with its factory

package mongorepo

// MongoRepo is repository that connects to MongoDB
// one instance of MongoRepo is responsible for one type of collection & data type
type MongoRepo struct {
    collection  *mongo.Collection
    constructor func() interface{}
}

// New creates a new instance of MongoRepo
func New(coll *mongo.Collection, cons func() interface{}) *MongoRepo {
    return &MongoRepo{
        collection: coll,
        constructor: cons,
    }
}

Here MongoRepo have 2 fields:

collection: is the mongodb collection which the MongoRepo have access to
constructor: is the constructor/factory of object which we want to abstract

So, think of the constructor as generic type T.
But instead of storing the type information, we are storing the function on how to create a new object T.

To see how constructor works, we move on to the CRUD implementation

var (
    virtualDelete = bson.M{"$set": bson.M{"deleted": true}}
)

// Get a list of resource
// The function is simply getting all entries in r.collection for the sake of example simplicity
func (r *MongoRepo) Get(ctx context.Context) ([]interface{}, error) {
    cur, err := r.collection.Find(ctx, bson.M{})
    if err != nil {
        return nil, err
    }

    var result []interface{}
    defer cur.Close(ctx)
    for cur.Next(ctx) {
        entry := r.constructor() // call to constructor
        if err = cur.Decode(entry); err != nil {
            return nil, err
        }

        result = append(result, entry)
    }

    return result, nil
}


// GetOne resource based on its ID
func (r *MongoRepo) GetOne(ctx context.Context, id string) (interface{}, error) {
    _id, _ := primitive.ObjectIDFromHex(id)
    res := r.collection.FindOne(ctx, bson.M{"_id": _id})
    dbo := r.constructor()
    err := res.Decode(dbo)
    return dbo, err
}

// Create a new resource
func (r *MongoRepo) Create(ctx context.Context, obj interface{}) error {
    _, err := r.collection.InsertOne(ctx, obj)
    if err != nil {
        return err
    }

    return nil
}

// Update a resource
func (r *MongoRepo) Update(ctx context.Context, id string, obj interface{}) error {
    _id, _ := primitive.ObjectIDFromHex(id)
    _, err := r.collection.UpdateOne(ctx, bson.M{"_id": _id}, obj)
    if err != nil {
        return err
    }

    return nil
}

// Delete a resource, virtually by marking it as {"deleted": true}
func (r *MongoRepo) Delete(ctx context.Context, id string) error {
    _id, _ := primitive.ObjectIDFromHex(id)
    _, err := r.collection.UpdateOne(ctx, bson.M{"_id": _id}, virtualDelete)
    if err != nil {
        return err
    }

    return nil
}

Notice @line#9 of Get method, we have:

entry := r.constructor()

Or @line#3 of GetOne method:

dbo := r.constructor()

This is where we trick the abstraction of T in Go, which I termed as Generic Constructor (CMIIW), just because Go doesn't have generic.

So what's the point of all of this, bro? Why all the shenanigan?

So:

We can freely pass any generic constructor function
To not having to type / make {Resource}Repo N number of times

e.g:

type Person struct{}

type Enemy struct{}

// Initialize mongo connection
ctx := context.Background()
conn := os.Getenv("MONGO_CONN")
mongoopt := options.Client().ApplyURI(conn)
mongocl, _ := mongo.Connect(ctx, mongoopt)
mongodb := mongocl.Database("dbname")

// personRepo, points to 'person' collection
personRepo := mongorepo.New(
    mongodb.Collection("person"),
    func() interface{} {
        return &Person{}
    }))

// enemyRepo, points to 'enemy' collection
enemyRepo := mongorepo.New(
    mongodb.Collection("enemy"),
    func() interface{} {
        return &Enemy{}
    }))

Compare it to language with generic:

var personRepo = new Repo<Person>();
var enemyRepo = new Repo<Enemy>();

I think it's already quite similar. ✌️

Conclusion

+ If we have lots of database object with similar CRUD operation, I think this can save us lots of time.

- We now rely on interface{} which defeats the purpose of strongly typed language.

Ironic, how lack of generic makes code even more unsafe when we try to do abstraction around it...

Golang, Microservices, and Monorepo

Robin — Sun, 01 Mar 2020 13:52:48 +0000

I've been coding in golang for the last 2 years (2018 - 2020), building microservices and committing it to a monorepo.
These years have got me thinking on how to structure Go project based on those 2 criteria: microservice and monorepo

Why Microservice

Business comes and go quickly, especially at the early phase of a startup.
The benefits of using microservice are:

Each microservice is treated as 1 business domain
Easier to spawn a new microservice without having any dependency to other services
It's also easier to take down a microservice in case the business initiative failed, without having to refactor any code

But there are also some downsides on using microservice architecture.
In my experience, code sharing is one of the issues. Which can be tackled by using monorepo

Why Monorepo

One benefit of monorepo is code sharing.
But what the f is code sharing???

Imagine a scenario without monorepo:

There's an e-commerce named TOKOPA'EDI
They have 2 microservice named payment and invoice
Everytime a payment is made, it needs to copy the payment information into invoice

When invoice service accept payment information, invoice service do not have type Payment struct in its repository.
Therefore invoice service have to do some duplication of payment struct.

With monorepo, payment struct can be shared across multiple microservices

Project Structure

project
├── cmd                     # contains microservices 
│   ├── '{microservice a}'  # "microservice a" project which contains "main" function to start app
│   │   └── main.go
│   └── '{microservice b}'  # "microservice b" project, also only contains "main" function to start app
│       └── main.go
├── docker                  # dockefile for each microservices
│   ├── '{microservice a}'
│   └── '{microservice b}'
├── internal                # internal package of each microservice. codes inside internal package only visible to its own microservice and cannot be shared
│   ├── '{microservice a}'
│   │   ├── rest            # HTTP entry point for REST API
│   │   ├── grpc            # gRPC entry
│   │   ├── repo            # repository / connector to database
│   │   └── service         # business logic layer which is handled by the microservice
│   └── '{microservice b}'
│       ├── rest
│       ├── grpc
│       ├── repo
│       └── service
├── pkg                     # public package directory which can be shared across project / microservices
│   └── models              # data model which usually 1:1 relation with database
│       ├── something.go
│       └── another.go
└── readme.md

With above project structure, every object / code inside pkg is fair game and can be shared across microservices without having to copy-pasting object from another repository.

CMIIW ✌️

DEV Community: Robin

To Comment or Not To Comment

Mis-interpretation

Too Much Focus on What but not Why

Comment your intent

Playing with GraphQL

Problem Statement

A Look inside GraphQL Query

Shifting Overfetching Around

Dataloader

Personal Take

Disclaimer

The Industry Hates Me For Being Honest

Company A

Company B

Lack of default constructor is hurting Go

Marrying Actor, DDD, and Worker Pattern using Go

Aggregate Root disguised as Actor

Stopping the Actor

Directing Actors

Mocking Fake Microservices

Order as Aggregate Root

Distributed Processing

Making Simple Scheduler using Go Routine & Channel

Designing the Scheduler

Event Object

The Scheduler

TODO-1 Validate Event

TODO-2 Use Go routine to wait for event

TODO-3 Add Event Handler

Reviewing the Requirements

TODO-1 Stop all scheduled events!

TODO-2 Collecting pending events

TODO-3 Closing the pendings channel

Reviewing the Requirements again

Full Code of the Scheduler

Testing our Scheduler

Gracefully Shutdown Your Go App

Shutdown Horror Story

Gracefully Shutdown your Service

The Concept

Basic Implementation in Go

RBAC in REST API using Go

Dahe** is RBAC

Minimum Effort Implementation

REST API

Mapping the rules

So, let's start from Client's rule

Next, CS' rule

Next, OPS' rule

Lastly, Manager's rule

RBAC engine

Let's start from Rule

Ensurer and Enforcer

Stitching it all together

So, Again What the Hell is ctx

What does this all means? This is nuts

As a Client

As a CS

As an Ops

As a Manager

MONGOREPO Abstraction Using Generic Constructor in Go

CRUD to resource

Repository Pattern

Abstraction, or the lack thereof

Conclusion

Golang, Microservices, and Monorepo

Why Microservice

Why Monorepo

Project Structure

Basic Implementation in `Go`

So, let's start from `Client's` rule

Next, `CS'` rule

Next, `OPS'` rule

Lastly, `Manager's` rule

So, Again What the Hell is `ctx`

As a `Client`

As a `CS`

As an `Ops`

As a `Manager`