DEV Community: Paweł Zaremba

Embedding Types in Go Reduces Noise

Paweł Zaremba — Thu, 30 Apr 2020 11:08:05 +0000

My previous post was about one of the concurrency patterns that can be achieved with go.
I immediately got some feedback on it, and this is where I will address one of them.

You have seen the code reimplementing sync.WaitGroup to have a semaphore in it.

type WaitGroup interface {
    Add(delta int)
    Done()
    Wait()
}

type SemaphoredWaitGroup struct {
    sem chan bool
    wg  sync.WaitGroup
}

func (s *SemaphoredWaitGroup) Add(delta int) {
    s.wg.Add(delta)
    s.sem <- true
}

func (s *SemaphoredWaitGroup) Done() {
    <-s.sem
    s.wg.Done()
}

func (s *SemaphoredWaitGroup) Wait() {
    s.wg.Wait()
}

It has been brought to my attention (Thanks, Wojtek!), that there is a "cleaner" way to do it in Go.

Enter: Type Embedding

Those coming from other languages (e.g.: PHP), you might notice that there is no such thing as an extends keyword.
There is no subclassing in the popular way. What we do have is type embedding.

To achieve that we change SemaphoredWaitGroup:

type SemaphoredWaitGroup struct {
    sem chan bool
-   wg  sync.WaitGroup
+   sync.WaitGroup
}

All methods from the "inner" (embedded) struct are now a part of the "outer" (embedding) type.
It is even possible to access them directly by calling them as if they were actually defined locally. E.g:

func (s *SemaphoredWaitGroup) newMethod() {
    s.Wait()
}

There is a catch though. when we re-define the methods from the inner part (here we have: Add(delta int) and Done()), we need to change the internal calls

func (s *SemaphoredWaitGroup) Add(delta int) {
-   s.Add(delta)
+   s.WaitGroup.Add(delta)
    s.sem <- true
}

func (s *SemaphoredWaitGroup) Done() {
    <-s.sem
-   s.Done()
+   s.WaitGroup.Done()
}

because the s.Add(delta) and s.Done() would be recursive calls, and would result in a "stack overflow" error in this particular instance.

Also, we don't need to have the Wait() method pass calls through to the underlying sync.WaitGroup struct.

A complete example is available on Go Playground.

This was originally posted at: https://www.tegh.net/9

Semaphored Wait Group

Paweł Zaremba — Wed, 29 Apr 2020 13:19:59 +0000

One of the first things you might like to try when starting your journey with Go are the concurrency patterns.
You will probably start using goroutines to run things "in the background",
and you will also get to know channels that allow for safe communication between the sub-processes.

When you'll want to spawn many goroutines, you will surely get to know WaitGroups to wait for them to finish.
There will also be the defer statement which helps you to not forget to clean up after a function has finished running (and remember, deferred calls are executed in reverse order).

Let's start with a simple example:

package main

import (
    "log"
    "sync"
    "time"
)

func worker(id int, wg *sync.WaitGroup) {
    defer wg.Done()
    defer log.Printf("#%d done", id)
    log.Printf("#%d starting", id)
    time.Sleep(time.Second)
}

func main() {
    var wg sync.WaitGroup
    for i := 1; i <= 100; i++ {
        wg.Add(1)
        go worker(i, &wg)
    }
    wg.Wait()
    log.Printf("all done")
}

This works well, but what if we had a million items to process, and the actual work would be a memory-heavy operation?
We don't want to start a million goroutines.
We need something to limit the number of goroutines being run at the same time.

This is where a semaphore will come in handy.
We'll define how many concurrent workers we want, and the semaphore will not allow starting new goroutines until a slot is free.

Using a Channel to Create a Simple Semaphore

Let's say we want a maximum of 5 workers running at any time.
We'll need a buffered channel, and we do not really care what type of values it holds.

sem := make(chan bool, 5)

Whenever we want to start a goroutine, we'll push a value to the channel:

        wg.Add(1)
        sem <- true
        go worker(i, &wg, sem) // we need to pass the channel to the worker

When we're done - we'll take one value out of the channel:

func worker(id int, wg WaitGroup, sem <-chan bool) {
    defer wg.Done()
    defer log.Printf("#%d done", id)
    log.Printf("#%d starting", id)
    time.Sleep(time.Second)
    <-sem
}

Please notice, that wen we added the sem argument to the worker, we also converted the semaphore to a read-only channel using <-chan syntax.

But wait..., we do not want to change the code too much.
Can we just create our own implementation of a wait-group that will have the desired feature?

Yes we can, and Go's interfaces will help us do it.

Go's Implicit Interfaces - The Ultimate Decoupling

In most other languages the interface has to be defined before it can be implemented.
In Go - it is the other way around.
We can create interfaces that satisfy our needs and if anything (internal or external) has the methods with matching signatures,
Go will treat it as a type that implements the interface.

A Tour of Go: Implicit interfaces decouple the definition of an interface from its implementation, which could then appear in any package without prearrangement.

So, even though we cannot change the sync.WaitGroup type, we can extract an interface that matches our current requirements. We'll have:

type WaitGroup interface {
    Add(delta int)
    Done()
    Wait()
}

and change worker function's signature to:

func worker(id int, wg WaitGroup)

Now, we can create our own implementation of our WaitGroup interface with a built-in semaphore.

type SemaphoredWaitGroup struct {
    sem chan bool
    wg   sync.WaitGroup
}

func (s *SemaphoredWaitGroup) Add(delta int) {
    s.wg.Add(delta)
    s.sem <- true
}

func (s *SemaphoredWaitGroup) Done() {
    <-s.sem
    s.wg.Done()
}

func (s *SemaphoredWaitGroup) Wait() {
    s.wg.Wait()
}

Finally, instead of using the default sync.WaitGroup we can use our own SemaphoredWaitGroup:

    wg := SemaphoredWaitGroup{sem: make(chan bool, 5)}

and we have ourselves a limited pool of workers.

The complete example is available as a GitHub Gist
and on Go Playground.

Original post at: https://www.tegh.net/knowledge/semaphored-wait-group/

Money Is Not Everything

Paweł Zaremba — Mon, 27 Apr 2020 14:46:44 +0000

I am currently searching for new work.
I tend to phrase it as "looking for new opportunities" or "challenges".
But let's call it as it is. I am on a job hunt.

The money question

Quite often I find myself in an interview baffled by a question about my financial terms.
Some interviewers require a range, some a single number.
It is easier if the job ad has the offered range specified from the beginning, but only a little.

Having been a professional software developer for almost 15 years (i.e. since I started my self-employment in 2006) I am not shy to ask for the top offered value.
Unfortunately there is a catch with this tactic.

I have found that MANY companies LIE about the actual salary range they are capable or willing to pay.
I have been turned down a few times because my salary expectations were said to be too high.
The recruiters wrote: "we value your expertise, and we would like to have you on board, but you a) have too much of experience and we would not be able to challenge you enough or b) we cannot meet financial expectations". WTF, I ask you?

I have since came to realise, that I do not want to work for a company that misleads its prospective employees.
Just do not waste time.

I guess I needed to get this out of my chest and in the open.

In both of my previous jobs I got to enjoy the feeling of not having to work at all, because the people there and the things we got to do, made it seem like so much fun.
The most important thing is that we bonded and supported each other. This allowed everyone to grow. No money can buy that!

Tell me what you think is fair

That is my default answer now.

Money is important. It is the means with which we can support our goals and dreams, but it is not the deciding factor.
As it turns out - it never has been!

I am sure we can come to a mutually beneficial arrangement if we just keep honest and open about our expectations and capabilities.

Now, Let's talk how we can work together!

original post: https://www.tegh.net/misc/money-is-not-everything/

Automated testing in a build pipeline

Paweł Zaremba — Tue, 14 Apr 2020 18:08:03 +0000

A testing mindset with its' scripts would be incomplete if we did not put them to good use.

Ideally, tests should be run every time we intend to share our code with anyone - be it our team or the whole world.
It is a tech lead's wet dream to work in an environment where every dev thoroughly tests their solutions.
As this might be impossible, we can have the next best thing.

We can make tests run automatically every time new or modified code is introduced into the codebase.

Let's assume we do everything in Docker and any new code is built by a server that prevents merging changes until a new image is successfully created.

We'll use the tests.sh from before and add a test_scenario.sh which checks if all the use cases we need are covered.

# When we run the script:
bash test_scenario.sh

# We get:
Test #1: PASS
Test #2: PASS
Test #3: PASS
Test #4: PASS
Test #5: PASS
Test #6: PASS
Test #7: PASS

Now, we can create a Dockerfile that will use the above script to check if our app is behaving properly.

FROM bash
COPY test*.sh ./

RUN bash test_scenario.sh

The docker image described above will not be built if the tests fail. Like this:

# Try to build the image
docker build .

# We get:

Sending build context to Docker daemon  4.608kB
Step 1/3 : FROM bash
 ---> 78664daf24f4
Step 2/3 : COPY test*.sh ./
 ---> fe75cddc7f0f
Step 3/3 : RUN bash test_scenario.sh
 ---> Running in 010702b02aea
Test #1: PASS
Test #2: PASS
Test #3: PASS
Test #4: PASS
Test #5: PASS
Test #6: PASS
Test #7: FAIL
The command '/bin/sh -c bash test_scenario.sh' returned a non-zero code: 1

With this approach, there will never exist a runnable version of our app that did not pass all the required tests.
When we fix the app, the test will work again.

# [...]
Step 3/3 : RUN bash test_scenario.sh
 ---> Running in c2d555481c28
Test #1: PASS
Test #2: PASS
Test #3: PASS
Test #4: PASS
Test #5: PASS
Test #6: PASS
Test #7: PASS
Removing intermediate container c2d555481c28
 ---> d41d9ecb5ed9
Successfully built d41d9ecb5ed9

... and we will have a new image built and ready to use.

Of course, there are different types of tests. Some of them will not be able to run before the application is actually built and deployed to a testing environment.
Real-life requires more sophisticated configurations, but the general idea is here.

Both *.sh files and the Dockerfile are available on GitHub.

Multiple tests run in bulk using Bash

Paweł Zaremba — Wed, 25 Mar 2020 15:24:24 +0000

Having briefly explained what a test is we can come to an inevitable conclusion that having only a single test is never enough.
We usually write a lot of them and it is useful to group them.

Assuming that the tests we are using conform to the contract of returning proper exit statuses, the whole group can technically be treated like a single test.

Let's put them inside a tests.sh script:

test -f file-a.txt
test -f file-b.txt
test -f file-c.txt

Now, (I assume that the script is an executable file) let's run it:

./tests.sh
echo $?
# returns: 1

All is well. The required files do not exist, so the test is failing.

Create the files one by one and run the test each time:

touch file-a.txt
./tests.sh
echo $?
# returns: 1
touch file-b.txt
./tests.sh
echo $?
# returns: 1
touch file-c.txt
./tests.sh
echo $?
# returns: 0

All is well again. The test passes.

Now, let's make sure the test is solid.
Delete the first file and run the test again.

rm file-a.txt
./tests.sh
echo $?
# returns: 0

Wait, what? The test returns 0? It should fail!

The thing is that when we put some commands into a script, only the last one's return status is returned by the script itself.
To ensure that the script will exit when a command within that script exits with a non-zero status, we use the set bash builtin with the -e option.

Put #!/bin/bash (just to make sure we will be running Bash) on the first line and set -e on the second line of the test script and run it yet again.

Now the test fails as it should.

Run a series of tests to ensure all relevant situations are being checked.

rm file-?.txt

./tests.sh
echo $? # should return 1

touch file-a.txt
touch file-b.txt
touch file-c.txt
./tests.sh
echo $? # should return 0

rm file-a.txt
./tests.sh
echo $? # should return 1

touch file-a.txt
rm file-b.txt
./tests.sh
echo $? # should return 1

touch file-b.txt
rm file-c.txt
./tests.sh
echo $? # should return 1

This shows the general idea behind testing.

I have been purposefully avoiding using the terms test cases and test suites as they are usually more abstract concepts I do not wish to get into here.

What is a test in software development?

Paweł Zaremba — Wed, 25 Mar 2020 15:22:13 +0000

When joining a new project (and maybe even in a previously unknown programming language) we can get overwhelmed by the complexity of the test suites. Sometimes it can even be a nightmare to run them, not to mention understand what is going on. There are lots of levels the tests can be run on.

Let's start with the basics.

Anything that checks if a requirement is met is a test.

How do we apply this mindset in real life?

In software engineering, we want something that we can run and immediately see if what the project requires is what the solution provides.

Fortunately, we have the concept of an exit status. Well behaved commands return such a status (sometimes called exit code). A 0 means that the command finished successfully. A non-zero value is usually considered to be an error code.

If a command utilizes this concept it is a perfect tool to use for testing.

We can print the exit status on unix-like operating systems with echo $?.

The simplest example is the test command:

test -e a-non-existent-file.txt
echo $?
# returns: 1

test -e .
echo $?
# returns: 0

This is already a ready-to-use test. We can put it into a CI/CD pipeline which will fail if the README file is missing.