DEV Community

Arseny Zinchenko
Arseny Zinchenko

Posted on • Originally published at rtfm.co.ua on

Go: checking public repositories list in Github. Go slices comparison. The first Golang experience.

#go #programming #beginners

The task is to write a tool which will be started from a Jenkin’s job by a cron and will check an organization’s public repositories list in the Github.

A Docker-image build and a Jenkins job are described in the Jenkins: a job to check a Github organization’s public repositories list post.

Then it has to compare received repositories list with a predefined list with allowed repositories and if they will not be equal – send an alert to a Slack channel.

The idea is to have such a check in case if developers accidentally will create a public repository instead of private or will change a private repository to the public and to get a notification about such issue.

It could be written with bash and curl, or Python with urllib and use Github API directly, but I want to get some Go experience so will use it.

In fact – this is my the very first self-written Golang code which I didn’t use a lot and even don’t know its syntax, but will try to use some existing knowledge with C/Python, add some logic and of course with the Google’s help and write something workable.

To work with API Github the go-github package will be used.

Let’s begin.

Add imports, the function was created automatically with the vim-go plugin:

import (
    "fmt"
    "github.com/google/go-github/github"
)

func main() {
    fmt.Println("vim-go")
}
Enter fullscreen mode Exit fullscreen mode

Add $GOPATH:

$ sudo mkdir /usr/local/go
$ sudo chown setevoy:setevoy /usr/local/go/
$ export GOPATH=/usr/local/go && export GOBIN=/usr/local/go/bin
Enter fullscreen mode Exit fullscreen mode

Install package:

$ go get
/home/setevoy/Scripts/Go/GTHB
./gitrepos.go:5:2: imported and not used: "github.com/google/go-github/github"
Enter fullscreen mode Exit fullscreen mode

Getting repositories list from Github

Copy and paste the first example from the package’s README, an organization name will be taken from the $GITHUB_ORG_NAMEand let’s try get something.

Set the variable:

$ export GITHUB_ORG_NAME="rtfmorg"
Enter fullscreen mode Exit fullscreen mode

And code:

package main
import (
    "context"
    "fmt"
    "github.com/google/go-github/github"
)

func main() {
    client := github.NewClient(nil)
    opt := &github.RepositoryListByOrgOptions{Type: "public"}
    repos, _, _ := client.Repositories.ListByOrg(context.Background(), os.Getenv("GITHUB_ORG_NAME"), opt)
    fmt.Printf(repos)
}
Enter fullscreen mode Exit fullscreen mode

In the original example, organization name was hardcoded but here we will take it from a variable which will be set in a Jenkin’s job using os.Getenv.

Add context and os imports.

Run the code:

$ go run go-github-public-repos-checker.go
command-line-arguments
./go-github-public-repos-checker.go:17:15: cannot use repos (type []*github.Repository) as type string in argument to fmt.Printf
Enter fullscreen mode Exit fullscreen mode

Errr…

Okay.

I thought client.Repositories.ListByOrg will return actually list – just because of the List in its name :-)

Check what data type we have in the repos object. Use reflect:

package main
import (
    "os"
    "context"
    "fmt"
    "reflect"
    "github.com/google/go-github/github"
)
func main() {
    client := github.NewClient(nil)
    opt := &github.RepositoryListByOrgOptions{Type: "public"}
    repos, _, _ := client.Repositories.ListByOrg(context.Background(), os.Getenv("GITHUB_ORG_NAME"), opt)
    fmt.Println(reflect.TypeOf(repos).String())
}
Enter fullscreen mode Exit fullscreen mode

Run:

$ go run go-github-public-repos-checker.go
[]*github.Repository
Enter fullscreen mode Exit fullscreen mode

Good… It’s really list ([]), apparently pointed to a structure – github.Repository.

Let’s check with the go doc:

$ go doc github.Repository
type Repository struct {
ID               *int64           `json:"id,omitempty"`
NodeID           *string          `json:"node_id,omitempty"`
Owner            *User            `json:"owner,omitempty"`
Name             *string          `json:"name,omitempty"`
FullName         *string          `json:"full_name,omitempty"`
Description      *string          `json:"description,omitempty"`
...
Enter fullscreen mode Exit fullscreen mode

Yup – it’s structure and also we see all its fields.

To disable Go's “imported and not used: “reflect” message – set it as _reflect:

package main
import (
    "os"
    "context"
    "fmt"
    _"reflect"
    "github.com/google/go-github/github"
)
func main() {
    client := github.NewClient(nil)
    opt := &github.RepositoryListByOrgOptions{Type: "public"}
    repos, _, _ := client.Repositories.ListByOrg(context.Background(), os.Getenv("GITHUB_ORG_NAME"), opt)
    for _, repo := range repos {
        fmt.Printf("Repo: %s\n", *repo.Name)
    }
}
Enter fullscreen mode Exit fullscreen mode

Here from the repos list we are getting an element’s ID and its value.

Can add ID’s as well:

...
    for id, repo := range repos {
        fmt.Printf("%d Repo: %s\n", id, *repo.Name)
    }
...
Enter fullscreen mode Exit fullscreen mode 
$ go run go-github-public-repos-checker.go
0 Repo: org-repo-1-pub
1 Repo: org-repo-2-pub
Enter fullscreen mode Exit fullscreen mode

Comparing lists in Go

Now need to add another one list which will keep repositories list from Github.

Allowed to be public repositories will be passed from Jenkins in a list view separated by spaces:

$ export ALLOWED_REPOS="1 2"
Enter fullscreen mode Exit fullscreen mode

Attempt number one

Create allowedRepos of the string type and save allowed public repositories list in it:

...
    allowedRepos := []string{os.Getenv("ALLOWED_REPOS")}
    for id, repo := range repos {
        fmt.Printf("%d Repo: %s\n", id, *repo.Name)
    }
    fmt.Printf("Allowed repos: %s\n", allowedRepos) 
...
Enter fullscreen mode Exit fullscreen mode

But here is an issue which I faced with a bit later when started comparing lists. Will see it shortly.

Run the code:

$ go run go-github-public-repos-checker.go
0 Repo: org-repo-1-pub
1 Repo: org-repo-2-pub
Allowed repos: [1 2]
Enter fullscreen mode Exit fullscreen mode

Okay – it works.

Now need to make checks between two repositories lists – repos with repositories from Github and allowedRepos.

The first solution was next:

...
    allowedRepos := []string{os.Getenv("ALLOWED_REPOS")}
    fmt.Printf("Allowed repos: %s\n", allowedRepos)
    for id, repo := range repos {
        for _, i := range allowedRepos {
            if i == *repo.Name {
                fmt.Printf("Index: %d, repo %s found in Allowed as %s\n", id, *repo.Name, i)
            } else {
                fmt.Printf("ALARM: repo %s was NOT found in Allowed!\n", *repo.Name)
            }
        }
    }
...
Enter fullscreen mode Exit fullscreen mode

Run:

$ go run go-github-public-repos-checker.go
Allowed repos: [1 2]
ALARM: repo org-repo-1-pub was NOT found in Allowed!
ALARM: repo org-repo-2-pub was NOT found in Allowed!
Enter fullscreen mode Exit fullscreen mode

Looks like it works?

Repositories with the 1 and 2 names != org-repo-1-pub and org-repo-2-pub.

But the problem which might be already obvious for some readers appeared when I did a “back-check”, i.e. when I set $ALLOWED_REPOS with real names to get the OK result:

$ export ALLOWED_REPOS="org-repo-1-pub org-repo-2-pub"
Enter fullscreen mode Exit fullscreen mode

Check:

$ go run go-github-public-repos-checker.go
Allowed repos: [org-repo-1-pub org-repo-2-pub]
ALARM: repo org-repo-1-pub was NOT found in Allowed!
ALARM: repo org-repo-2-pub was NOT found in Allowed!
Enter fullscreen mode Exit fullscreen mode

And why?

Because of in the allowedRepos we have not a list aka slice and Go – but just a string.

Let’s add the i variable’s output and indexes numbers:

...
    for r_id, repo := range repos {
        for a_id, i := range allowedRepos {
            fmt.Printf("ID: %d Type: %T Value: %s\n", a_id, allowedRepos, i)
            if i == *repo.Name {
                fmt.Printf("Index: %d, repo %s found in Allowed as %s\n", r_id, *repo.Name, i)
            } else {
                fmt.Printf("ALARM: repo %s was NOT found in Allowed!\n", *repo.Name)
            }
        }
    }
...
Enter fullscreen mode Exit fullscreen mode

Check:

$ go run go-github-public-repos-checker.go
Allowed repos: [org-repo-1-pub org-repo-2-pub]
ID: 0 Type: []string Value: org-repo-1-pub org-repo-2-pub
ALARM: repo org-repo-1-pub was NOT found in Allowed!
ID: 0 Type: []string Value: org-repo-1-pub org-repo-2-pub
ALARM: repo org-repo-2-pub was NOT found in Allowed!
Enter fullscreen mode Exit fullscreen mode

allowedRepos is really a list but with the only 0 element which keeps the “org-repo-1-pub org-repo-2-pub” value.

Attempt number two

To make this working – need to convert the allowedRepos to a real list.

Let’s use the strings package:

...
    // allowedRepos := []string{os.Getenv("ALLOWED_REPOS")}                                                                                                                                                     
    allowedRepos := strings.Fields(os.Getenv("ALLOWED_REPOS"))
    fmt.Printf("Allowed repos: %s\n", allowedRepos)

    for r_id, repo := range repos {
        for a_id, i := range allowedRepos {
            fmt.Printf("ID: %d Type: %T Value: %s\n", a_id, allowedRepos, i)
            if i == *repo.Name {
                fmt.Printf("Index: %d, repo %s found in Allowed as %s\n", r_id, *repo.Name, i)
            } else {
                fmt.Printf("ALARM: repo %s was NOT found in Allowed!\n", *repo.Name)
            }
        }
    }
...
Enter fullscreen mode Exit fullscreen mode

allowedRepos now will be filled with the strings.Fields().

Check result:

$ go run go-github-public-repos-checker.go
Allowed repos: [org-repo-1-pub org-repo-2-pub]
ID: 0 Type: []string Value: org-repo-1-pub
Index: 0, repo org-repo-1-pub found in Allowed as org-repo-1-pub
ID: 1 Type: []string Value: org-repo-2-pub
ALARM: repo org-repo-1-pub was NOT found in Allowed!
ID: 0 Type: []string Value: org-repo-1-pub
ALARM: repo org-repo-2-pub was NOT found in Allowed!
ID: 1 Type: []string Value: org-repo-2-pub
Index: 1, repo org-repo-2-pub found in Allowed as org-repo-2-pub
Enter fullscreen mode Exit fullscreen mode

Okay – much better now but this also will not work as there some “false-positive” results because of each repos‘s element is compared with each allowedRepos‘s element.

Attempt number three

Let’s rewrite it and now let’s try to use a repo‘s indexes to chose an element from the allowedRepos:

...
    for r_id, repo := range repos {
        fmt.Printf("%d %s\n", r_id, *repo.Name)
        if *repo.Name != allowedRepos[r_id] {
            fmt.Printf("ALARM: repo %s was NOT found in Allowed!\n", *repo.Name)
        } else {
            fmt.Printf("Repo %s found in Allowed\n", *repo.Name)
        }
    }
...
Enter fullscreen mode Exit fullscreen mode

I.e. from the repos we getting an element’s ID and then checking the allowedRepos‘s element with the same ID.

Run:

$ go run go-github-public-repos-checker.go
Allowed repos: [org-repo-1-pub org-repo-2-pub]
0 org-repo-1-pub
Repo org-repo-1-pub found in Allowed
1 org-repo-2-pub
Repo org-repo-2-pub found in Allowed
Enter fullscreen mode Exit fullscreen mode

Nice!

But another issue can happen now…

What if the order in both lists will differ?

Set instead of the:

$ export ALLOWED_REPOS="org-repo-1-pub org-repo-2-pub"
Enter fullscreen mode Exit fullscreen mode

Repositories names in the reversed order:

$ export ALLOWED_REPOS="org-repo-2-pub org-repo-1-pub"
Enter fullscreen mode Exit fullscreen mode

Check:

$ go run go-github-public-repos-checker.go
Allowed repos: [org-repo-2-pub org-repo-1-pub]
0 org-repo-1-pub
ALARM: repo org-repo-1-pub was NOT found in Allowed!
1 org-repo-2-pub
ALARM: repo org-repo-2-pub was NOT found in Allowed!
Enter fullscreen mode Exit fullscreen mode

D’oh!

Attempt number four

The next idea was to use the reflect module and its DeepEqual() function.

Add one more list called actualRepos, add repositories taken from Github with the append and then compare them:

...
    allowedRepos := strings.Fields(os.Getenv("ALLOWED_REPOS"))
    var actualRepos []string
    for _,  repo := range repos {
        actualRepos = append(actualRepos, *repo.Name)
    }
    fmt.Printf("Allowed: %s\n", allowedRepos)
    fmt.Printf("Actual: %s\n", actualRepos)
    fmt.Println("Slice equal: ", reflect.DeepEqual(allowedRepos, actualRepos))
...
Enter fullscreen mode Exit fullscreen mode

Run:

$ go run go-github-public-repos-checker.go
Allowed: [org-repo-2-pub org-repo-1-pub]
Actual: [org-repo-1-pub org-repo-2-pub]
Slice equal:  false
Enter fullscreen mode Exit fullscreen mode

And again no… Although if revert order back – it will work:

$ export ALLOWED_REPOS="org-repo-1-pub org-repo-2-pub"
$ go run go-github-public-repos-checker.go
Allowed: [org-repo-1-pub org-repo-2-pub]
Actual: [org-repo-1-pub org-repo-2-pub]
Slice equal:  true
Enter fullscreen mode Exit fullscreen mode

So, need to go back to the loop’s variant but to check somehowrepo‘s value in the whole allowedRepos list and only if it’s absent – create an alert.

Attempt number five

The solution was next: create a dedicated function which will check all allowedRepos‘s elements in a loop and will return true if the value will be found and false otherwise. Then – we can use this in themain()‘s loop.

Let’s try.

Create a isAllowedRepo()function which will accept two arguments – a repository name to be checked and the allowed repositories list and will return a boolean value:

...
func isAllowedRepo(repoName string, allowedRepos []string) bool {
    for _, i := range allowedRepos {
        if i == repoName {
            return true
        }
    }
    return false
}
...
Enter fullscreen mode Exit fullscreen mode

Then in the main() – run a loop over all repos‘s elements, pass them one by one to the isAllowedRepo() and then print a result:

...
    for _, repo := range repos {
        fmt.Printf("Checking %s\n", *repo.Name)
        fmt.Println(isAllowedRepo(*repo.Name, allowedRepos))
    }
...
Enter fullscreen mode Exit fullscreen mode

Let’s test.

First, restore allowed repositories list in the initial order:

$ export ALLOWED_REPOS="org-repo-1-pub org-repo-2-pub"
go run go-github-public-repos-checker.go
Checking org-repo-1-pub
true
Checking org-repo-2-pub
true
Enter fullscreen mode Exit fullscreen mode

Good!

And reversed order:

$ export ALLOWED_REPOS="org-repo-2-pub org-repo-1-pub"
$ go run go-github-public-repos-checker.go
Checking org-repo-1-pub
true
Checking org-repo-2-pub
true
Enter fullscreen mode Exit fullscreen mode

Still works…

Now remove one of the allowed repositories:

$ export ALLOWED_REPOS="org-repo-2-pub"
go run go-github-public-repos-checker.go
Checking org-repo-1-pub
false
Checking org-repo-2-pub
true
Enter fullscreen mode Exit fullscreen mode

Great!

Now when we will get false – we can raise an alert.

The whole code now. Leaving “as is” just for history:

package main
import (
    "os"
    "context"
    "fmt"
    _"reflect"
    "strings"
    "github.com/google/go-github/github"
)
func isAllowedRepo(repoName string, allowedRepos []string) bool {
    for _, i := range allowedRepos {
        if i == repoName {
            return true
        }
    }
    return false
}
func main() {
    client := github.NewClient(nil)
    opt := &github.RepositoryListByOrgOptions{Type: "public"}
    repos, _, _ := client.Repositories.ListByOrg(context.Background(), os.Getenv("GITHUB_ORG_NAME"), opt)
    // allowedRepos := []string{os.Getenv("ALLOWED_REPOS")}
    allowedRepos := strings.Fields(os.Getenv("ALLOWED_REPOS"))
//  var actualRepos []string
/*
    for _,  repo := range repos {
        actualRepos = append(actualRepos, *repo.Name)
    }
    fmt.Printf("Allowed: %s\n", allowedRepos)
    fmt.Printf("Actual: %s\n", actualRepos)
    fmt.Println("Slice equal: ", reflect.DeepEqual(allowedRepos, actualRepos))
*/
    for _, repo := range repos {
        fmt.Printf("Checking %s\n", *repo.Name)
        fmt.Println(isAllowedRepo(*repo.Name, allowedRepos))
    }
/*
    for r_id, repo := range repos {
        for _, i := range allowedRepos {
            fmt.Printf("Checking %s and %s\n", *repo.Name, i)
            if i == *repo.Name {
                fmt.Printf("Index: %d, repo %s found in Allowed as %s\n", r_id, *repo.Name, i)
                break
            } else {
                fmt.Printf("ALARM: repo %s was NOT found in Allowed!\n", *repo.Name)
            }
        }
    }
*/
/*
    for r_id, repo := range repos {
        fmt.Printf("%d %s\n", r_id, *repo.Name)
        if *repo.Name != allowedRepos[r_id] {
            fmt.Printf("ALARM: repo %s was NOT found in Allowed!\n", *repo.Name)
        } else {
            fmt.Printf("Repo %s found in Allowed\n", *repo.Name)
        }
    }
*/
/*
    for r_id, repo := range repos {
        for a_id, i := range allowedRepos {
            fmt.Printf("ID: %d Type: %T Value: %s\n", a_id, allowedRepos, i)
            if i == *repo.Name {
                fmt.Printf("Index: %d, repo %s found in Allowed as %s\n", r_id, *repo.Name, i)
            } else {
                fmt.Printf("ALARM: repo %s was NOT found in Allowed!\n", *repo.Name)
            }
        }
    }
*/
}
Enter fullscreen mode Exit fullscreen mode

Golang Slack

And the final thing is to add a Slack notification.

Use ashwanthkumar/slack-go-webhook here.

Configure Slack’s WebHook, get its URL.

Add a new sendSlackAlarm() function which will accept two arguments – repository name and its URL:

...
func sendSlackAlarm(repoName string, repoUrl string) {
    webhookUrl := os.Getenv("SLACK_URL")
    text := fmt.Sprintf(":scream: ALARM: repository *%s* was NOT found in Allowed!", repoName)
    attachment := slack.Attachment{}
    attachment.AddAction(slack.Action{Type: "button", Text: "RepoURL", Url: repoUrl, Style: "danger"}) 
    payload := slack.Payload{
        Username:    "Github checker",
        Text:        text,
        Channel:     os.Getenv("SLACK_CHANNEL"),
        IconEmoji:   ":scream:",
        Attachments: []slack.Attachment{attachment},
    }
    err := slack.Send(webhookUrl, "", payload)
    if len(err) > 0 {
        fmt.Printf("error: %s\n", err)
    }
}
...
Enter fullscreen mode Exit fullscreen mode

Add the sendSlackAlarm() execution to the main() if the isAllowedRepo() returned false:

...
    for _, repo := range repos {
        fmt.Printf("\nChecking %s\n", *repo.Name)
        if isAllowedRepo(*repo.Name, allowedRepos) {
            fmt.Printf("OK: repo %s found in Allowed\n", *repo.Name)
        } else {
            fmt.Printf("ALARM: repo %s was NOT found in Allowed!\n", *repo.Name)
            sendSlackAlarm(*repo.Name, *repo.HTMLURL)
        }
    }
...
Enter fullscreen mode Exit fullscreen mode

repo.HTMLURL we found from the go doc github.Repository.

Add $SLACK_URL and $SLACK_CHANNEL environment variables:

$ export SLACK_URL="https://hooks.slack.com/services/T1641GRB9/BA***WRE"
$ export SLACK_CHANNEL="#general"
Enter fullscreen mode Exit fullscreen mode

Restore the full repositories list on reversed order:

$ export ALLOWED_REPOS="org-repo-2-pub org-repo-1-pub"
Enter fullscreen mode Exit fullscreen mode

Check:

$ go run go-github-public-repos-checker.go
Checking org-repo-1-pub
OK: repo org-repo-1-pub found in Allowed
Checking org-repo-2-pub
OK: repo org-repo-2-pub found in Allowed
Enter fullscreen mode Exit fullscreen mode

Okay…

Remove one allowed:

$ export ALLOWED_REPOS="org-repo-2-pub"
Enter fullscreen mode Exit fullscreen mode

Check again:

$ go run go-github-public-repos-checker.go
Checking org-repo-1-pub
ALARM: repo org-repo-1-pub was NOT found in Allowed!
Checking org-repo-2-pub
OK: repo org-repo-2-pub found in Allowed
Enter fullscreen mode Exit fullscreen mode

And Slack notification:

Done.

The script is available in the setevoy-tools Github repository.

Similar posts

Top comments (2)

Collapse
 
theodesp profile image
Theofanis Despoudis •

Do you think it was worth it?

Collapse
 
setevoy profile image
Arseny Zinchenko •

Yes, absolutely.
Can't say I'm in love with Go now - but at least I better understand it.
Not planning to use it in future for coding but have to know it to read some source code.
And anyway - in the last couple of years not so often have a chance to write anything even in Python/C/bash - so any experience/practice is excellent for me.

Read next

mikeyoung44 profile image

100x Faster Diffusion Planning with DiffuserLite's Refinement Approach

Mike Young -

kanav_gathe profile image

Day 14/90: Essential Linux & Git-GitHub Cheat Sheet 📝 #90DaysOfDevOps

Kanav Gathe -

yassinesallami profile image

How to create an image authentication system with python streamlit and canva!

Yassine Sallami -

mdarifulhaque profile image

2641. Cousins in Binary Tree II

MD ARIFUL HAQUE -