DEV Community: Panagiotis Georgiadis

GitOps in a nutshell

Panagiotis Georgiadis — Tue, 04 Jan 2022 14:33:29 +0000

What GitOps is

GitOps is a set of best practices to update or rollback your application, nowadays mostly in a cloud (e.g. k8s) environment. As the name implies, the whole process is controlled by Git, and the code includes both the application source and the infrastructure configuration. The goal, as usual, is to deploy fast and reliably.

In 2021, the first OpenGitOps Standard v1 was created, to make sure we all GitOps enthusiasts speak the same language. For more information go to opengitops.dev.

Hm, what about DevOps?

DevOps is a mindset. GitOps is a set of best practices for deployments. They can both co-exist together.

So what those "best" practices say?

The entire system (infra & apps or services) is described in a declarative way. Imagine like taking a cab, you tell the taxi driver where you would like to go (declarative way), instead of giving him instructions in GoogleMaps fashion (e.g. turn here - turn there).

pseudocode example

if (SERVER is online) {
    if (SERVER == Ubuntu)    {use apt-get}
    if (SERVER == Fedora)    {use dnf}
    if (SERVER == SUSE)      {use zypper}
}

The desired state is versioned in Git, giving you the confidence that it cannot easily just change, and if that happens, then there will be a log entry about it, with a complete history.
Merged PRs are applied to the system automatically or semi-automatically (meaning to inform the admin a new desirable state has been detected, so they can decide when to apply it).
If there is a drift, reconciliation should detect this and try to fix it. If it cannot be fixed after a couple of minutes later, then an alert should be fired.

Given Kubernetes is the new platform to deploy apps in scale, let's see two deployment scenarios: one with GitOps and one without.

How to deploy in k8s without GitOps

Someone commits the source code for the app in Github, the CI system runs the tests and builds the container and pushes it into a container registry.
The CI (or another system) that has access to the k8s cluster, creates a deployment of the app using a bash script full of kubectl, helm, or any other command to interact with the cluster.
The application should now be deployed on the k8s cluster. Hopefully there were no problems.

Problems with this workflow

The cluster status is changing using kubectl commands provided via a script or other programs talking directly with the API.
The CI system has some kind of authorized access to the k8s cluster, and usually, this CI system is hosted somewhere externally, unless you have the tech & budget to build everything internally.

How to deploy in k8s with GitOps

First step is the same: someone commits the source code for the app in Github, the CI system runs the tests and builds the container and pushes it into a container registry.
Someone (or another system) goes into another repository, the one that holds the the k8s manifests for the app and makes the necessary changes (e.g. update the container image tag the newly built one).
A k8s GitOps controller running into the cluster monitors this repo and detects this change. So it applies the new changes to the cluster according to the git repo.

Solutions used:

State is changed based on manifests, not commands or interacting on the fly with the API.
No external system needs access to the k8s cluster. The change is happening from within the cluster itself.

GitOps use-case examples

Most common scenario for GitOps is for continuous deployment. If you use k8s and you handle your apps and services in a declarative manner, this is the most straightforward way of deploying them.
Easy roll-back of anything you deploy in your cluster no matter if you develop it or not. e.g. some databases or some monitoring tools. You can also deploy them in a similar fashion.
Detects configuration drift, meaning someone is doing stuff while they shouldn't.
Multi cluster scenario. When you have more than one cluster the usual question is what is running where and who changed what, when, why and so on. Having a git history and each cluster config in git, gives you all the answer you need without cluster debugging -- just git diff

However, some might still don't adopt GitOps because ...

Git is not your cup of tea (duh!)
If you want to debug something on the cluster, it might be that the GitOps controller restores your changes back to the desired state. So you might want to disable it while doing this.
GitOps is all about the deployment. For everything else (testing, building, monitoring, etc), you need to figure it out on your own.

If you are looking for a full blown solution to set everything up for you with one click, have a look at https://docs.openshift.com/container-platform/4.7/cicd/gitops/understanding-openshift-gitops.html

Mocking the filesystem in Go using Afero

Panagiotis Georgiadis — Wed, 16 Dec 2020 17:21:06 +0000

Let's start the project

cd $GOPATH/src/github.com/drpaneas
mkdir checkfile; cd checkfile
git init -q
go mod init
touch main.go
code . # for vscode

Coding Task

Write a function that checks if a specific directory exists on disk. For example:

if folderExists(directory) {
  // do something
}

Bonus: Feel free to add fileExists(filename) as homework.

Acceptance Criteria

Use https://github.com/spf13/afero for filesystem abstraction
Make sure it works cross-platform
For logging use https://github.com/Sirupsen/logrus
Put the function into pkg/utils package
Write a unit test
Make sure there are testable examples

Hints:

Writing a unit test for filesystem checking is not trivial. You should NOT create a real file on the system, because then your test will depend on the I/O of the file system itself. The last resort is mocking the filesystem. There are quite a few powerful libraries like spf13/afero for this purpose (mocking of a filesystem). These packages will create temporary files in the background and clean up afterward.
As an exercise, try to write the test first. Let it fail. Then try to fix it, by writing the actual implementation of the function.

Extra Help

https://golangcode.com/check-if-a-file-exists/
https://www.youtube.com/watch?v=ifBUfIb7kdo
https://github.com/spf13/afero#using-afero-for-testing
An example using Afero for testing: http://krsacme.com/golang-fs-tests/
For logging https://linuxhint.com/golang-logrus-package/

NOTICE:

If you are unable to mock this test, in worst case scenario, create a folder pkg/utils/testdata and put actual files you can use for your test.

Let's code

// main.go
package main

import (
    "os"

    log "github.com/sirupsen/logrus"
)

// exists reports whether the named file or directory exists
func exists(path string, isDir bool) bool {
    if path == "" {
        log.Debug("Path is empty")
        return false
    }

    info, err := os.Stat(path)
    if err != nil {
        if os.IsNotExist(err) { // look for the specific error type
            return false
        }
    }

    return isDir == info.IsDir()
}

// FolderExists reports whether the provided directory exists.
func FolderExists(path string) bool {
    return exists(path, true)
}

// FileExists reports whether the provided directory exists.
func FileExists(path string) bool {
    return exists(path, false)
}


func main() {
    // Just for demo purposes
    if FolderExists("logs") {
        log.Info("Folder exists")
    } else {
        log.Error("Folder does not exist")
    }

    if FileExists("helloworld.txt") {
        log.Info("File exists")
    } else {
        log.Error("File does not exist")
    }
}

One way to test this is by actually using the program itself.
Running it as it is, it's expected to fail:

ERRO[0000] Folder does not exist                        
ERRO[0000] Folder does not exist

If we create the two necessary things we look for, then it's expected to work:

# Create the file and directory
touch helloworld.txt
mkdir logs

# Run it again
go run main.go
INFO[0000] Folder exists                                
INFO[0000] File exists

Normally you would create separate package and not use main.
So let's do it!

mkdir -p pkg/utils/
touch pkg/utils/filesystem.go

The code will look like this:

// main.go
// -------

package main

import (
    "github.com/drpaneas/checkfile/pkg/utils"
    log "github.com/sirupsen/logrus"
)

func main() {
    if utils.FolderExists("logs") {
        log.Info("Folder exists")
    } else {
        log.Error("Folder does not exist")
    }

    if utils.FileExists("helloworld.txt") {
        log.Info("File exists")
    } else {
        log.Error("File does not exist")
    }
}

// pkg/utils/filesystem.go
// -----------------------

package utils

import (
    "os"

    log "github.com/sirupsen/logrus"
)

// exists reports whether the named file or directory exists
func exists(path string, isDir bool) bool {
    if path == "" {
        log.Debug("Path is empty")
        return false
    }

    info, err := os.Stat(path)
    if err != nil {
        if os.IsNotExist(err) { // look for the specific error type
            return false
        }
    }

    return isDir == info.IsDir()
}

// FolderExists reports whether the provided directory exists.
func FolderExists(path string) bool {
    return exists(path, true)
}

// FileExists reports whether the provided directory exists.
func FileExists(path string) bool {
    return exists(path, false)
}

Just to be on the safe side we didn't forget anything major, let's run it again to make sure everything is alright.

# Run it again
go run main.go
INFO[0000] Folder exists                                
INFO[0000] File exists

Test with testdata

Ok,we need to write tests for pkg/utils/filesystem.go.
One way of doing this is by accessing a real file system to do testing, then we're talking about what you'd call system or integration testing. That's fine in itself: for instance, you could run such tests in a throw-away container as part of a CI pipeline.
But with this approach you can't sensibly do what is called unit-testing.

To proceed with that, we are going to use the generated boilerplate code that vscode happily offers (give you have the Go extensions installed).

Place the mouse cursor over the FolderExists() function and press the keys: CTRL+SHIFT+P.
From the drop-down menu, type: Go: test and select the one for the function.
This will create the filesystem_test.go file and populate it with a table test.
Add your test logic there:

// pkg/utils/filesystem_test.go
// ----------------------------

package utils

import "testing"

// Add those variables as well
var (
    existingFolder    = "./testdata/a-folder-that-exists"
    nonExistingFolder = "./testdata/a-folder-that-does-not-exists"
)

func TestFolderExists(t *testing.T) {
    type args struct {
        path string
    }
    tests := []struct {
        name string
        args args
        want bool
    }{
        // ----- test logic start ----- //
        {
            "Returns true when given folder exists",
            args{path: existingFolder},
            true,
        },
        {
            "Returns false when given folder does not exist",
            args{path: nonExistingFolder},
            false,
        },
        // ----- test logic end ----- //
    }
    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            if got := FolderExists(tt.args.path); got != tt.want {
                t.Errorf("FolderExists() = %v, want %v", got, tt.want)
            }
        })
    }
}

The tests is relying upon real data on the actual hard drive.
Run tests and you will see they will fail:

go test ./... -v
?       github.com/drpaneas/checkfile   [no test files]
=== RUN   TestFolderExists
=== RUN   TestFolderExists/Returns_true_when_given_folder_exists
    filesystem_test.go:33: FolderExists() = false, want true
=== RUN   TestFolderExists/Returns_false_when_given_folder_does_not_exist
--- FAIL: TestFolderExists (0.00s)
    --- FAIL: TestFolderExists/Returns_true_when_given_folder_exists (0.00s)
    --- PASS: TestFolderExists/Returns_false_when_given_folder_does_not_exist (0.00s)
FAIL
FAIL    github.com/drpaneas/checkfile/pkg/utils 0.098s
FAIL

Create the necessary testdata and try again:

mkdir -p pkg/utils/testdata/a-folder-that-exists

$ go test ./... -v?       github.com/drpaneas/checkfile   [no test files]
=== RUN   TestFolderExists
=== RUN   TestFolderExists/Returns_true_when_given_folder_exists
=== RUN   TestFolderExists/Returns_false_when_given_folder_does_not_exist
--- PASS: TestFolderExists (0.00s)
    --- PASS: TestFolderExists/Returns_true_when_given_folder_exists (0.00s)
    --- PASS: TestFolderExists/Returns_false_when_given_folder_does_not_exist (0.00s)
PASS
ok      github.com/drpaneas/checkfile/pkg/utils (cached)

Now, see the coverage. Go into the package and run cover.

 cover
PASS
coverage: 77.8% of statements
ok      github.com/drpaneas/checkfile/pkg/utils 0.293s

PS: You need to have this snippet into your ~/.bashrc:

# Go coverage trick
cover () {
    t="/tmp/go-cover.$$.tmp"
    go test -coverprofile=$t $@ && go tool cover -html=$t && unlink $t
}

Let's enhance the testing coverage, by adding:

        {
            "Returns false when provided path is empty",
            args{path: ""},
            false,
        },

Run the tests again and the coverage:

go test ./...

$ cover ./pkg/utils
ok      github.com/drpaneas/checkfile/pkg/utils 0.232s  coverage: 88.9% of statements

Add the tests for the other function as well:

The variables first:

var (
    existingFile      = "./testdata/a-file-that-exists"
    nonExistingFile   = "./testdata/a-file-that-doesn-not-exist"
)

The test code:

        {
            "Returns true when given file exists",
            args{path: existingFile},
            true,
        },
        {
            "Returns false when given file does not exist",
            args{path: nonExistingFile},
            false,
        },
        {
            "Returns false when provided path is empty",
            args{path: ""},
            false,
        },

And create the appropriate file for testing:

touch pkg/utils/testdata/a-file-that-exists

Run the tests (they should all pass):

$ go test -v ./...?       github.com/drpaneas/checkfile   [no test files]
=== RUN   TestFolderExists
=== RUN   TestFolderExists/Returns_true_when_given_folder_exists
=== RUN   TestFolderExists/Returns_false_when_given_folder_does_not_exist
=== RUN   TestFolderExists/Returns_false_when_provided_path_is_empty
--- PASS: TestFolderExists (0.00s)
    --- PASS: TestFolderExists/Returns_true_when_given_folder_exists (0.00s)
    --- PASS: TestFolderExists/Returns_false_when_given_folder_does_not_exist (0.00s)
    --- PASS: TestFolderExists/Returns_false_when_provided_path_is_empty (0.00s)
=== RUN   TestFileExists
=== RUN   TestFileExists/Returns_true_when_given_file_exists
=== RUN   TestFileExists/Returns_false_when_given_file_does_not_exist
=== RUN   TestFileExists/Returns_false_when_provided_path_is_empty
--- PASS: TestFileExists (0.00s)
    --- PASS: TestFileExists/Returns_true_when_given_file_exists (0.00s)
    --- PASS: TestFileExists/Returns_false_when_given_file_does_not_exist (0.00s)
    --- PASS: TestFileExists/Returns_false_when_provided_path_is_empty (0.00s)
PASS
ok      github.com/drpaneas/checkfile/pkg/utils 0.301s

Check the testing coverage:

cover ./pkg/utils
ok      github.com/drpaneas/checkfile/pkg/utils 0.268s  coverage: 100.0% of statements

Although we already hit the 100% testing coverage, I would like to add another test-case.

Add these two code block respectively:

// TestFolderExists()
// ...
        {
            "Returns false when provided path is not a directory",
            args{path: existingFile},
            false,
        },

// TestFileExists
// ...
        {
            "Returns false when provided path is not a file",
            args{path: existingFolder},
            false,
        },

... and that's it!
Run again the tests to verify everything works and we are good to go.

go test ./...
?       github.com/drpaneas/checkfile   [no test files]
ok      github.com/drpaneas/checkfile/pkg/utils 0.332s

Test without testdata

Ok now let's do the same without the actual testdata.
To unit-test your function you need to replace something it uses with something "virtualised".
Exactly how to do that is an open question.
Right now, a work is being done on bringing filesystem virtualisation right into the Go standard library, but while it's not there yet, there's a 3rd-party package which has been doing that for ages, — for instance, github.com/spf13/afero.

First let's remove the actual real data, and see the tests fail:

rm -r ./pkg/utils/testdata

So now the tests are failing obviously:

go test ./...
?       github.com/drpaneas/checkfile   [no test files]
--- FAIL: TestFolderExists (0.00s)
    --- FAIL: TestFolderExists/Returns_true_when_given_folder_exists (0.00s)
        filesystem_test.go:47: FolderExists() = false, want true
--- FAIL: TestFileExists (0.00s)
    --- FAIL: TestFileExists/Returns_true_when_given_file_exists (0.00s)
        filesystem_test.go:86: FileExists() = false, want true
FAIL
FAIL    github.com/drpaneas/checkfile/pkg/utils 0.404s
FAIL

To do that we will use Afero.
Writing a unit test for filesystem checking is not trivial. You should NOT create a real file on the system, because then your test will depend on the I/O of the file system itself.

Let's modify our code to use Afero for both implementation and test.
So for the filesystem.go just add:

// Making the global, sharing the memory filesystem
var (
    FS  afero.Fs     = afero.NewMemMapFs()
    AFS *afero.Afero = &afero.Afero{Fs: FS}
)

and replace the os.Stat to AFS.Stat. That's all.
For the filesystem_test.go just add:

func init() {
    // create test files and directories
    AFS.MkdirAll(existingFolder, 0755)
    afero.WriteFile(AFS, existingFile, []byte("test file"), 0755)
}

Run the tests and they should pass:

go test ./... -v
?       github.com/drpaneas/checkfile   [no test files]
=== RUN   TestFolderExists
=== RUN   TestFolderExists/Returns_true_when_given_folder_exists
=== RUN   TestFolderExists/Returns_false_when_given_folder_does_not_exist
=== RUN   TestFolderExists/Returns_false_when_provided_path_is_empty
=== RUN   TestFolderExists/Returns_false_when_provided_path_is_not_a_directory
--- PASS: TestFolderExists (0.00s)
    --- PASS: TestFolderExists/Returns_true_when_given_folder_exists (0.00s)
    --- PASS: TestFolderExists/Returns_false_when_given_folder_does_not_exist (0.00s)
    --- PASS: TestFolderExists/Returns_false_when_provided_path_is_empty (0.00s)
    --- PASS: TestFolderExists/Returns_false_when_provided_path_is_not_a_directory (0.00s)
=== RUN   TestFileExists
=== RUN   TestFileExists/Returns_true_when_given_file_exists
=== RUN   TestFileExists/Returns_false_when_given_file_does_not_exist
=== RUN   TestFileExists/Returns_false_when_provided_path_is_empty
=== RUN   TestFileExists/Returns_false_when_provided_path_is_not_a_file
--- PASS: TestFileExists (0.00s)
    --- PASS: TestFileExists/Returns_true_when_given_file_exists (0.00s)
    --- PASS: TestFileExists/Returns_false_when_given_file_does_not_exist (0.00s)
    --- PASS: TestFileExists/Returns_false_when_provided_path_is_empty (0.00s)
    --- PASS: TestFileExists/Returns_false_when_provided_path_is_not_a_file (0.00s)
PASS
ok      github.com/drpaneas/checkfile/pkg/utils 0.349s

Notice we have no testdata any more.
Basically, Afero allows you to not use os directly but write all your code in a way so that instead of the os package it calls methods on an instance of a type implementing particular interface: in the production code that object is a thin shim for the os package, and in the testing code it's replaced by whatever you wish; afero has a readily-available in-memory FS backend to do this.

To sum up, although it's a good idea to do this, there might be cases where Afero fails to offer you this isolation.
For example if you try to do anything on top of the basic filesystem usage, then Afero might not be enough.
For example, if you try to extract, zip or compress a fail, you will see that those functions are not compatible with Afero, thus you will have either to send a PR to afero or suffer by creating lot's of extra code for achieving that.