DEV Community: Girish Talekar

HTTP Authorization in Go using Casbin, Redis and PostgreSQL

Girish Talekar — Sat, 24 Jun 2023 04:10:12 +0000

First scenario :

To understand how casbin works, i started with very simple authorization requirement

Requirements :

1) I have 2 api’s ‘/project’ and ‘/channel’
2) I want only authorized user should be able to access this api’s

Approach :

Rather then implementing our own authorization model we decided of using an existing/opensource package, we came across Casbin which is flexible and has many feature’s inbuilt.

Basically in casbin there are rules which you need to first configure in .conf file and specify each rule in .csv file

The conf file is based on PERM metamodel (Policy, Effect, Request, Matchers)

Request:

For instance, a request definition may look like this: r={sub,obj,act}

It actually defines the parameter name and order which we should provide for access control matching function.

Policy:

Define the model of the access strategy. It defines the name and order of the fields in the Policy rule document.

Matchers:

Matching rules of Request and Policy.

For example: m = r.sub == p.sub && r.act == p.act && r.obj == p.obj

Effect:

It can be understood as a model in which a logical combination judgment is performed again on the matching results of Matchers.

Example: e = some(where(p.eft == allow))

Create a policy.csv as below

p, alice, /project, *
p, alice, /channel, *
p, bob, /channel, GET
p, bob, /project, GET

1) I want alice should have full access to all api’s
2) I want bob should only be able to read

The conf file for above requirement looks like this

model.conf

[request_definition]
r = sub, obj, act

[policy_definition]
p = sub, obj, act

[policy_effect]
e = some(where (p.eft == allow))

[matchers]
m = r.sub == p.sub && keyMatch(r.obj, p.obj) && (r.act == p.act || p.act == "*")

Integrating together with golang code

package main

import (
    "net/http"

    "github.com/casbin/casbin"
    "github.com/labstack/echo"
)

type Enforcer struct {
    enforcer *casbin.Enforcer
}

func (e *Enforcer) Enforce(next echo.HandlerFunc) echo.HandlerFunc {
    return func(c echo.Context) error {
        user, _, _ := c.Request().BasicAuth()
        method := c.Request().Method
        path := c.Request().URL.Path

        result, _ := e.enforcer.EnforceSafe(user, path, method)

        if result {
            return next(c)
        }
        return echo.ErrForbidden
    }
}

func main() {
    e := echo.New()
    enforcer := Enforcer{enforcer: casbin.NewEnforcer("model.conf", "policy.csv")}
    e.Use(enforcer.Enforce)
    e.GET("/project", func(c echo.Context) error {
        return c.JSON(http.StatusOK, "project get allowed")
    })
    e.POST("/project", func(c echo.Context) error {
        return c.JSON(http.StatusOK, "project post allowed")
    })

    e.GET("/channel", func(c echo.Context) error {
        return c.JSON(http.StatusOK, "channel get allowed")
    })

    e.POST("/channel", func(c echo.Context) error {
        return c.JSON(http.StatusOK, "channel post allowed")
    })
    e.Logger.Fatal(e.Start("0.0.0.0:3000"))
}

Github link

Tests:

curl -X POST http://bob:@0.0.0.0:3000/channel                                                                 
{"message":"Forbidden"}
curl http://bob:@0.0.0.0:3000/channel                                                                         
"ok channel get"
curl -X POST http://alice:@0.0.0.0:3000/channel                                                               
"ok channel post"
curl -X GET http://alice:@0.0.0.0:3000/channel                                                                
"ok channel get"
curl -X POST http://bob:@0.0.0.0:3000/project                                                                 
{"message":"Forbidden"}

More practical scenario :

After getting some understanding, i want to implement it in my project

Requirements:

1) I want the policy/rules to be added dynamically though api’s and we want it get applied as soon as it get added
2) I don’t want to fetch all the rules from db for every request, so we want to use some caching layer
3) I want to group the rules as per roles, so that i don’t have to add for every verbs(GET, POST etc), i just want to give a role to a user and whatever rule it is associated with should get applied, some thing like this

p, admin, /project, *
p, admin, /channel, *
p, user, /channel, GET
p, user, /project, GET
g, alice, admin
g, bob, user

The conf file for above requirement will look like this

[request_definition]
r = sub, obj, act

[policy_definition]
p = sub, obj, act

[role_definition]
g = _, _

[policy_effect]
e = some(where (p.eft == allow))

[matchers]
m = (r.sub == p.sub || g(r.sub, p.sub)) && keyMatch(r.obj, p.obj) && (r.act == p.act || p.act == "*")

Approach:

1) Add middleware which will take care of authorization
2) Check the rule in redis if user has the access (to reduce load on db)
3) If cache miss happens then load the policy’s from db and store the rule for that user in redis for subsequent requests.

Note: on every update of the rule though api, we need to delete the cache for that user, this is not covered in this example

Storing the Policy’s in the DB :

CREATE TABLE casbin_rule (
    id SERIAL PRIMARY KEY,
    ptype VARCHAR(100),
    v0 VARCHAR(100),
    v1 VARCHAR(100),
    v2 VARCHAR(100),
    v3 VARCHAR(100),
    v4 VARCHAR(100),
    v5 VARCHAR(100)
);

INSERT INTO casbin_rule(ptype, v0, v1, v2) VALUES('p', 'admin', '/project', '*');
INSERT INTO casbin_rule(ptype, v0, v1, v2) VALUES('p', 'admin', '/channel', '*');
INSERT INTO casbin_rule(ptype, v0, v1, v2) VALUES('p', 'user', '/project', 'GET');
INSERT INTO casbin_rule(ptype, v0, v1, v2) VALUES('p', 'user', '/channel', 'GET');

INSERT INTO casbin_rule(ptype, v0, v1) VALUES('g', 'alice', 'admin');
INSERT INTO casbin_rule(ptype, v0, v1) VALUES('g', 'bob', 'user');

Middleware and code :

package main

import (
    "context"
    "fmt"
    "log"
    "net/http"
    "strconv"
    "time"

    "github.com/casbin/casbin/v2"
    gormadapter "github.com/casbin/gorm-adapter/v3"
    "github.com/labstack/echo/v4"
)

func Authenticate(adapter *gormadapter.Adapter) echo.MiddlewareFunc {
    return func(next echo.HandlerFunc) echo.HandlerFunc {
        return func(e echo.Context) (err error) {

            ctx := e.Request().Context()

            user, _, _ := e.Request().BasicAuth()
            method := e.Request().Method
            path := e.Request().URL.Path

            key := fmt.Sprintf("%s-%s-%s", user, path, method)

            result := RedisCache.Get(ctx, key)
            val, err := result.Result()
            if err == nil {
                boolValue, err := strconv.ParseBool(val)
                if err != nil {
                    log.Fatal(err)
                }

                if !boolValue {
                    return &echo.HTTPError{
                        Code:    http.StatusForbidden,
                        Message: "not allowed",
                    }
                }
                return next(e)
            }

            // Casbin enforces policy
            ok, err := enforce(ctx, user, path, method, adapter)
            if err != nil || !ok {

                return &echo.HTTPError{
                    Code:    http.StatusForbidden,
                    Message: "not allowed",
                }
            }
            if !ok {
                return err
            }
            return next(e)
        }
    }
}

func enforce(ctx context.Context, sub string, obj string, act string, adapter *gormadapter.Adapter) (bool, error) {
    // Load model configuration file and policy store adapter
    enforcer, err := casbin.NewEnforcer("./examples/group_model.conf", adapter)
    if err != nil {
        return false, fmt.Errorf("failed to load policy from DB: %w", err)
    }
    // Load policies from DB dynamically
    err = enforcer.LoadPolicy()
    if err != nil {
        return false, fmt.Errorf("error in policy: %w", err)
    }
    // Verify
    ok, err := enforcer.Enforce(sub, obj, act)
    if err != nil {
        return false, fmt.Errorf("error in policy: %w", err)
    }
    key := fmt.Sprintf("%s-%s-%s", sub, obj, act)
    RedisCache.Set(ctx, key, strconv.FormatBool(ok), time.Hour)
    return ok, nil
}

Done, that’s all you need to do, below are the tests with result

curl -X POST http://hari:@0.0.0.0:3000/project                                                                  ✔  10849  12:37:10
{"message":"Forbidden"}
curl -X POST http://alice:@0.0.0.0:3000/project                                                                 ✔  10851  12:39:21
"project post allowed"
curl -X POST http://bob:@0.0.0.0:3000/project                                                                   ✔  10852  12:39:29
{"message":"Forbidden"}
curl -X GET http://bob:@0.0.0.0:3000/project                                                                    ✔  10853  12:39:42
"project get allowed"
curl -X GET http://alice:@0.0.0.0:3000/project                                                                  ✔  10854  12:39:50
"project get allowed"
curl -X POST http://bob:@0.0.0.0:3000/channel                                                                   ✔  10855  12:40:11
{"message":"Forbidden"}
curl -X POST http://gt:@0.0.0.0:3000/channel                                                                    ✔  10856  12:40:28
{"message":"Forbidden"}

Now if you add another user say

INSERT INTO casbin_rule(ptype, v0, v1) VALUES('g', 'ray', 'user');

and do

curl -X GET http://ray:@0.0.0.0:3000/channel                                                                  ✔  10854  12:39:50
"channel get allowed"

it will work as expected, no need to restart the server

Boom! all things are working as expected happy coding, let me know you suggestion/feedback so that i can improve “learning should never stop”.

Source code

How to download large amount of datadog logs in parallel

Girish Talekar — Sat, 17 Jun 2023 13:12:12 +0000

Liquid syntax error: Unknown tag 'endraw'

Internals of goroutines and Channels

Girish Talekar — Tue, 29 Mar 2022 05:52:16 +0000

Why go is popular? because of concurrency and in golang how we achieve concurrency? through goroutines.

Before you get into details of goroutines we need to understand Threads and Processes

What is the difference between process and threads ?

How process works

The program has been loaded into the computer’s memory in binary form. Now what?

An executing program needs more than just the binary code that tells
the computer what to do. The program needs memory and various operating
system resources in order to run. A “process” is what we call a program
that has been loaded into memory along with all the resources it needs
to operate. The “operating system” is the brains behind allocating all
these resources, and comes in different flavors such as macOS, iOS,
Microsoft Windows, Linux, and Android. The OS handles the task of
managing the resources needed to turn your program into a running
process.

A process run independently and in isolation of other process, it can not directly access shared data in other processes.Switching from one process to another requires some time (relatively) for saving and loading registers, memory maps, and other resources. This helps in isolation of the process and that's why you’ve been able to quit that program without affecting others.

There are three main costs of a process switch

First is the kernel needs to store the contents of all the CPU registers for that process, then restore the values for another process.
The kernel also needs to flush the CPU’s mappings from virtual memory to physical memory as these are only valid for the current process.
Finally there is the cost of the operating system context switch, and the overhead of the scheduler function to choose the next process to occupy the CPU.

There are a surprising number of registers in a modern processor. Because a process switch can occur at any point in a process’ execution, the operating system needs to store the contents of all of these registers because it does not know which are currently in use

How Threads works

A process can have multiple threads and

Because threads share the same address space as the process and other threads within the process, the operational cost of communication between the threads is low, which is an advantage. The disadvantage is that a problem with one thread in a process will certainly affect other threads and the viability of the process itself

How Goroutines works

Goroutines take the idea of threads a step further.

Goroutines are cooperatively scheduled, rather than relying on the kernel to manage their time sharing.

The switch between goroutines only happens at well defined points, when an explicit call is made to the Go runtime scheduler.

The compiler knows the registers which are in /use and saves them automatically.

Because the heap and stack overwriting each other would be
catastrophic, the operating system usually arranges to place an area of
unwritable memory between the stack and the heap to ensure that if they
did collide, the program will abort.

This is called a guard page, and effectively limits the stack size of a process, usually in the order of several megabytes.

Because it is hard to predict the stack requirements of a particular thread, a large amount of memory is reserved for each thread’s stack along with a guard page. The downside is that as the number of threads in your program increases, the amount of available address space is reduced.

Internals of Go Scheduling

Arguably, one of the more important aspects of the Go run-time is the goroutine scheduler. The runtime keeps track of each goroutine, and will schedule them to run in turn on a pool of thr eads belonging to the process. Goroutines are separate from threads but rely upon them to run, and scheduling goroutines onto threads effectively is crucial for the efficient performance of Go programs. The idea behind goroutines is that they are capable of running concurrently,like threads, but are also extremely lightweight in comparison. So, while there might be multiple threads created for a process running a Go program, the ratio of goroutines to threads should be much higher than 1-to-1. Multiple threads are often necessary to ensure that goroutines are not unnecessarily blocked. When one goroutine makes a blocking call,the thread running it must block. Therefore, at least one more thread should be created by the runtime to continue the execution of other goroutines that are not in blocking calls. Multiple threads are allowed to run in parallel upto a programmer defined maximum, which is stored in the variable GOMAXPROCS[6].

It is important to keep in mind that all the OS sees is a single user level process requesting and running multiple threads.The concept of scheduling goroutines onto these threads is merely a construct in the virtual environment of the runtime.When we refer to the Go runtime and scheduler in this pa-per we are referring to these higher level entities, which are completely separate from the operating system.

In the Go runtime, there are three main C-structs that help keep track of everything and support the runtime and scheduler:

THE G STRUCT

A G struct represents a single goroutine[9]. It contains the fields necessary to keep track of its stack and current status. It also contains references to the code that it is responsible for running. See figure 2.

THE M STRUCT

The M struct is the Go runtime’s representation of an OS thread[9]. It has pointers to fields such as the global queue of G’s, the G that it is currently running, its own cache, and a handle to the scheduler. See figure 3.

THE SCHED STRUCT

The Sched struct is a single, global struct[9] that keeps track of the different queues of G’s and M’s and some other information the scheduler needs in order to run, such as the global Sched lock. There are two queues containing G structs, one is the runnable queue where M’s can find work, and the other is a free list of G’s. There is only one queue pertaining to M’s that the scheduler maintains; the M’s in this queue are idle and waiting for work. In order to modify these queues,the global Sched lock must be held. See figure 4.

The runtime starts out with several G’s. One is in charge of garbage collection, another is in charge of scheduling, and one represents the user’s Go code. Initially, one M is created to kick off the runtime. As the program progresses,more G’s may be created by the user’s Go program, and more M’s may become necessary to run all the G’s. As this happens, the runtime may provision additional threads upto GOMAXPROCS. Hence at any given time, there are at most GOMAXPROCS active M’s.

Since M’s represent threads, an M is required to run a goroutine. An M without a currently associated G will pick up a G from the global runnable queue and run the Go code belonging to that G. If the Go code requires the M to block,for instance by invoking a system call, then another M will be woken up from the global queue of idle M’s. This is done to ensure that goroutines, still capable of running, are not blocked from running by the lack of an available M.

System calls force the calling thread to trap to the kernel,causing it to block for the duration of the system call execution. If the code associated with a G makes a blocking system call, the M running it will be unable to run it or any other G until the system call returns. M’s do not exhibit the same blocking behavior for channel communication, even though goroutines block on channel communication. The operating system does not know about channel communication, and the intricacies of channels are handled purely by the runtime. If a goroutine makes a channel call, it may need to block, but there is no reason that the M running that G should be forced to block as well. In a case such as this, the G’s status is set to waiting and the M that was previously running it continues running other G’s until the channel communication is complete. At that point the G’s status is set back to runnable and will be run as soon as there is an M capable of running it.

Get free NSE & BSE data(bhavcopy) based on customizable index and date

Girish Talekar — Tue, 16 Mar 2021 11:49:28 +0000

Why there is need for this:

For Technical analysis of stocks, for generating graphs and for many more such usecases most of the people requires End of Day stocks data called as Bhavcopy. I didn't find a place where i can get both Indian stock exchanges bhavcopy freely, hence the idea of creating a simple, customizable and public app, so that any one can download the NSE and BSE bhavcopy free.

Simple solution to the problem:

I have created a public website, below is the working of the same. BhavCopy-Downloader export data in csv format that is easily imported in leading technical analysis software. It connects to NSE and BSE servers to download this data, so it can be considered 100% authentic. I have made this highly customizable so that any one can update the index's as per there need and download the bhavcopy. Please check the contribution section for the same.

Public Github repository of the project:

I have create a public github repo called as BhavCopy-Downloader where all the source code of the current project is present. There are 2 folders called as nse and bse where all the stocks bhavcopy's are stored for a particular date. The bhavcopy stored so far is based on the request made by someone from the app, Other bhavcopy will be downloaded and stored as per request.

What feathers it provides:

Download EOD data for both the index NSE and BSE with delivered quantity and delivered quantity percentage
Can select different funds like Equities, f&o etc.
Can select different index from NSE and BSE like NIFTY50, BSE100 etc.
Get back date data
Api is public any one can use directly, https://bhavcopy-backend.herokuapp.com

Architecture of the project:

Created a flow diagram to get batter understand of the project

Details on how to use the application:

Select the Stock Exchange from which the data is required eg. NSE/BSE
Select Fund for the particular exchange, currently it is configured for Equity only, in future more options will be added
Select the Index for which the data is required default is All.
One's index is selected all the stock in that index get appear in textarea which is editable. User can make changes to the list.
Select date, to download the specific day data, default is previous date.
Click on download to download the bhavcopy in csv format.

Contribution is always welcome:

Since the github repo is public any one can contribute. For keeping all the index up to date i require help, that's why i have made that highly customizable.

All the index and stocks in that are stored in json files at "./src/NSEIndexConfigs" and "./src/BSEIndexConfigs". You can create new index, update the existing index based on NSE and BSE index change or add new one which i have missed. Create a pull request and submit the changes.

Also there is "./src/config.json" in which UI related config are present.

Inputs are always welcome! I want to make contributing to this project as easy and transparent as possible, whether it's:

Reporting a bug
Discussing the current state of the code
Submitting a fix
Proposing new features

Feedback or comments

Tell me your openioun or thoughts on this, Wheather it is negative or postive doesn't matter. Together we can make this batter.