DEV Community: Pascal Dennerly

Quitting early for the win!

Pascal Dennerly — Sat, 14 Dec 2024 01:01:16 +0000

Super short summary: Exiting your program when you get an error can be a good idea. Using gobail will make your life easier.

When you get an error in your Go code, you'll normally see something like this:

err := myFunc()
if err != nil {
    return fmt.Errorf("doing my thing: %w", err)
}

You'll notice a few things in this example:

You have to check whether there is an error
There is some text that will help diagnose the error
And the error is wrapped, passed back with the text

So what happens next? Well it all happens again. You check the error value, describe it, pass it back. And then it all starts again.

Why do we do this? Why all this effort?

Well it all depends on the software that you're writing. When you get an error, you have to make a decision. What happens to that error?

If you're writing an HTTP API responding to a request then eventually you're going to get to some kind of HTTP handler and you'll turn that error into some response - perhaps a little 400 with a polite reminder format their requests properly or possibly a 500 and a worrying message about the health of your application. Alternatively, if you're writing some kind of CLI tool then you might decide that the error will eventually get passed all the way back to your main function. For any kind of program, you might decide that enough is enough - the program should just end because you can't do anything else.

Let's look at this last option. When is it appropriate to just exit your program? Here's a few reasons that I can think of:

a. There's nothing else to be done, the error is so bad that everything has to stop right now
b. There are no consequences to terminating the program (no cleanup is required, no status to respond with)
c. Stopping early is desirable, perhaps you have a monitor that restarts the process cleanly

Whatever the reason, you need to consider how you're going to exit cleanly. Now the first thing you're likely to try is this:

err := myFunc()
if err != nil {
    fmt.Printf("doing my thing: %v", err)
    os.Exit(1)
}

It looks pretty similar to our original error handling code but with a couple of important differences. The first is kind of obvious - there's a honking great stop-right-g*****n-now statement there. Your program is not going to continue. The second point is perhaps more important. The code that is calling this sample doesn't have worry about handling any errors. There are no additional code paths that have to be tested - we can trust that that the calling code doesn't have an if block to test because there's nothing returned that we have to check.

Testing your exits

So I was being, perhaps, a little enthusiastic when I suggested you should just trust that your exit code is just going to work. You should probably check that the reason that you new program has stopped is for the right reason.

Attempt 1 - just run your program

This feels like it should be easy. Here's some things to consider - in the most easy case, you're just going to run your program and trigger an error condition. For example, get your CLI tool to open a file that doesn't exist. You can do this manually for a few simple cases. When the number of tests goes up, you'll probably need some kind of automation around it to help you.

Quick side note - this is probably the subject of a whole other blog post but my current favourite way of testing CLI tools uses godog to write tests. It can be a little complex but I've found it supremely powerful. Here are some good examples of how I've approached it with layli and wait-for.

This approach will get you very far but sometimes it can be difficult to create the conditions that will properly exercise all of the code paths that you want to be confident about.

Attempt 2 - mock out the exit

OK so now we're going to be using some of the features of the Go language. We don't actually have to call os.Exit - we can call something that looks like it. So look at this:

type ExitFunc func(code int)

var customExit ExitFunc = os.Exit

func myFunc() {
    err := someOtherFunc()
    if err != nil {
        fmt.Printf("doing my thing: %v", err)
        customExit(1)
    }
}

So how are we going to take advantage of this for our testing? As the function has now been turned into a variable (customExit) then we can replace the value with something else that we want to do. Like so...

package sameAsTheAboveCode

func TestOMGItsAllGoneWrong(t *testing.T) {
    oldExit := customExit
    defer func() {
        // Make sure we reset the exit so it can be used elsewhere
        customExit = oldExit
    }

    exitCalled := false
    exitCode := 0
    customExit = func(code int) {
        exitCalled = true
        exitCode = code
    }

    // Set up the mocks so that someOtherFunc returns an error

    myFunc()

    // Assume that we're using the fantastic stretchr/testify library here
    assert.True(t, exitCalled)
    assert.Equal(t, 1, exitCode)
}

This is a much more unit-test friendly approach. You can check that the exit code used is correct - and that you actually called the exit function.

On the surface, this looks great but there is one big problem - if your test passes, then you program will continue and execute the rest of the function when you expected it to exit. It will continue even though the test setup means that the rest of the execution is not valid and cause problems for your tests, such as causing panics.

Removing code to remove tests

Well, this sounds a bit extreme!

I feel like I should explain... Typically in "well managed" companies you need to make sure that every line of code has been proved to be working before if can be put in front of your customers. Using the above techniques you may not be able to generate the correct coverage metrics to attest that you're good. Even if it's trivial to reason about.
All of the examples above have assumed that when we get an error then we have to check it to decide what to do (Exit with vengeance). Wouldn't it be great if we were able to exit without having to check that there was an error?

Let's see what we can do.

func checkExit(err error) {
   fmt.Printf("doing my thing: %v", err)
   customExit(1)
}

func myFunc() {
    checkExit(someOtherFunc())
}

Take a look at the above example. The functionality is the same but the implementation of myFunc is much simpler now - there are no conditionals. We can check the implementation of the checkExit function in it's own tests, meaning that whatever is new in myFunc() can be much more easily verified.

Introducing gobail

A new libraray, gobail has been created that allows you to e confident that if there is an error then it will be handled without having to add complexity to your own code. It looks like this:

func myFunc() {
    gobail.Run(someOtherFunc()).OrExit("with message")
    gobail.Run(yetAnotherFunc()).OrExit("with message")
}

This library has been fully tested, with coverage metrics, to prove it. You can safely use it without having to worry that an error will be skipped. It will also handle functions with 2 return values, like so:

gobail.Return2(return2ValsAndError()).OrExitMsg("something went wrong: %v")

Notice as well that you an include the error that is causing all of your problems.

It's also possible to panic instead of exiting, printing our a stack trace and other contextual information from the program when the panic is invoked. Take a look at the docs for more details.

Isolating dependencies

As you write software with gobail, you will notice that you will mostly have to use it when talking to external libraries. This has the additional code that you typically need to write to handle all of the error cases can be wrapped in a call to Return or Return2 and assume that we will exit when necessary.

Conclusion

Sometimes it's desirable to exit your program instead of handling your errors in detail. The gobail library has been created and validated so that you don't have to worry about the detail of proving this.

If you find an improvement that could be made or just have a suggestion then raise a PR or an issue on the repo and the developers will get on it when they can!

Introducing Layli

Pascal Dennerly — Fri, 24 Nov 2023 22:20:15 +0000

This is a story about how my simple little alternative to PlantUML came about. Please check out layli if you just want to see the end result.

I've been an avid user of PlantUML for years. I love the ability to define my diagrams as code. Sequence diagrams, component diagrams - just define what connects to what and you have a diagram that you can put in git and render in to any image format you choose.

But.

BUT....

...very quickly, you find that your component diagrams end up looking a bit messy. So you use linetype ortho to get the straight lines with corners, rather than curves. Then you have to arrange the boxes around the page and you add hidden connections (-[hidden]-> and play with the line lengths (using --> and --->, perhaps even ---->). All of a sudden you discover that your simple diagram becomes a mess of crossing lines and kludges to separate the nodes so that it looks pretty.

So what to do? I love the ability to specify my diagrams as code. This works really nicely for sequence diagrams but my experience with component diagrams is often...disappointing.

A manifesto

I was frustrated with what I was working on and had a couple of insights. Surely there should be some simple guidelines that guarantee pretty diagrams? So I made some notes and eventually came up with this list or rules:

Edge paths must travel across a "path grid"
Node borders must be on the "path grid"
Edges must meet nodes at "ports"
Ports must sit on the "path grid"
Where a port is not specified, layli may select a "default" port on any side of the node
Nodes must be layed out so that the total area marked by the outside bounds of every node is as small as possible
Nodes must be seperated by at least 1 space on the "path grid"
Edges must must not cross!
Edges must follow a grid path (ie. not curved or diagonal)
Edges must be as short as possible
Edges must have as few corners as possible
Edge paths may sit on top of each other at the beginning or at the end

I think this is a reasonable start. It almost looks like a manifesto!

Getting started on Layli

After a little time prevaricating, I set out to clobber the keyboard a little and Layli slowly came in to being. The first thing I had to do was figure out how to arrange the boxes on my diagram (the nodes). I decided that the simplest way to do this was to just implement a primitive flow layout - just arrange them in to a square arrangement, row by column.

Next up, I had to draw connections between the nodes (edges). To do this, I looked back to an blog that I read a couple years ago that described using Dijkstra's shortest path algorithm to plot paths between nodes. I forget the details but this seemed to be an intriguing approach. After a little work, I had something that looked pretty decent:

After a little more work, I was able to implement some more features. A handful of different node arrangements alongside trying to shuffle the order that we draw paths so we don't get crossing lines.

There're more examples here.

What next

I've got a few items on my TODO list. Allowing you to specify nodes absolutely - giving them coordinates on your diagram, adding some styling, making the path-finding algorithm prettier. Who know's?

Maybe this is something that you'd like to contribute to? Just raise an issue or a pull request and I'd be happy to take a look!

Download layli and start using it now!

What will make me happy, as a #hacktoberfest maintainer

Pascal Dennerly — Mon, 05 Oct 2020 20:54:21 +0000

The original version of this appeared as a tweet thread a few days ago:
https://twitter.com/pascaldoesgo/status/1311799941791133697?s=19

Hey #hacktoberfest beginners!

I'm a very small scale maintainer and here are some things that you could do that I, personally, would value greatly! I suspect most maintainers would too. I personally look after 3 open source projects, essentially written to solve problems.

Shameless plug, here they are:

abbreviate

httpref

s3backup

As you can see, it's not much but it's mine.

Deciding on a contribution

First of all, I'd probably take a look at open issues on a project. A lot of projects will tag them with labels like help wanted or beginners. You can see quite quickly whether a project might have something they think is a good starting point for you. If you have an idea of what you could do to implement the change then just ask on the issue. I would happily spend a little time on feedback.

If you think you have an idea for an entirely new change that doesn't already exist then raise your own issue to discuss it first. This has been called "talk first" or talk then code development. It'll help me understand what's coming in. And help me to give you some guidance.

What changes can you make?

4 1-line puntuation changes in separate PRs is no help to anyone. I've had at 3 pull requests of this type already, including changes to the code of conduct and project license. The changes accidentally changed the legal meaning of the license text, moving it away from being the kind of project I want it to be. This is not cool!

Honest grammar and punctuation fixes are valued! If you've spent some time reviewing a README and you think you can make it better, then THANK YOU! I am a human and I old enough and had my ego punctured enough to realise that I make mistakes. All the time! If you spot something that you think isn't quite right the raise that PR.

Adding and improving documentation is the best contribution! It's the bit most developers struggle with. Usage guides, installation instructions, FAQs. These are great candidates for documentation changes that always seem to need more work. Certainly for me, if I'm intimately familiar with a tool or some code then I find it harder to understand what it takes to start using it. An outside perspective is actually a great asset for this kind of contribution.

Making that change

Now this section is more about my personal style of contributing. It's an approach when extending my own projects as well as at work.

First off open a draft PR early, fill out the PR template, make me aware of what you're doing. This achieves a few things all in one go.

Let's you use the CI/CD tooling where it's available. This depends very much on what the project is using to do their testing and deploying. It probably won't work for Github actions but might for Travis or CircleCI etc.
Makes me aware of an incoming change.
Gives me an opportunity to give you some early feedback for larger changes

I'm happy to review your code and help steer you if you don't feel confident or feel you just need some advice.

You don't have to be a confident developer to start coding. I remember my first attempts at code. I didn't know what I was doing for a long time. To be fair, I still make it up a little now. 😅

What you can expect from me

Be mindful that I have a full time job, small kids, a wife, people that call me on the phone, DIY, a laptop that we share amongst us.

I'll try to help you as much as I can but I can't promise to be totally on it all the time.

I will take a look at all changes that come in a few days! If you've spent the time to raise a pull request then I owe it to you to give it the proper amount of attention.

What I want you to get something from #hacktoberfest

Most importantly, have fun!

I mean a free T-Shirt is all very nice but I want you to enjoy what you're doing. You'll be more engaged and find it easier to contribute.

Ultimately #hacktoberfest is about building the Open Source community and joining in makes you a part of that.

Introducing Abbreviationist

Pascal Dennerly — Tue, 29 Sep 2020 22:19:38 +0000

Just a bit of blatant self promotion here. I've been playing with my little CLI tool abbreviate and wondering what I could do next. I also need to practice my very new front-end skills so I put together this site to meld the 2 things:

https://abbreviationist.herokuapp.com/

It's not much but after so many years hiding in the backend, putting up a simple site with my own HTML and CSS is a big deal for me.

If you have any comments or feedback then feel free to let me know. :)

Fun Coding

Pascal Dennerly — Sat, 28 Mar 2020 21:15:09 +0000

Fun Coding
/verb/

To develop code for ones own pleasure. This may or may not be related to your day job. Perhaps a side project, perhaps not. The principle is that it's for fun.

Rewarding Github contributions

Pascal Dennerly — Tue, 10 Mar 2020 21:00:43 +0000

So I'll probably be cross posting on Twitter or something but here goes...

How do you reward people that take the time to contribute?

I recently had someone fix a bug in httpref. No prompting, just a PR out of the blue that nailed the problem. I was shocked, and surprised, and so so glad that someone had taken the time.

Apart from a cursory 'thanks' when I hit merge and an impromptu shout out on Twitter I couldn't really do anything.

Is this something that anyone has been able to do? Just show someone how much you appreciate what they've done?

Discuss...

Learning Frontend as a Backend dev

Pascal Dennerly — Thu, 27 Feb 2020 22:10:47 +0000

Hey all, I thought I'd pull the community for tips on learning Frontend development.

I've been writing backend software in various forms for many, many years now. Even so, Frontwnd feels magical and scary.

It's definitely a skill I'm missing, but I wouldn't know where to start now.

Has anyone done this particular journey? Any tips?

THIS_IS_A_CONSTANT

Pascal Dennerly — Thu, 27 Feb 2020 20:18:13 +0000

A junior engineer asked a great question while we were pairing

"Why does this variable have to be in upper case?"

Well, because it's static final, of course.

But why?

So I had to think about this one and wondered why myself. What's so different? And why did people decide this was a good idea?

So the obvious difference is that because it's final, it can't be modified after it has been initialised. And it's static, meaning that it's a property of the class and not of this instance.

It's as close as you really get, in Java, to a const in other languages.

But, he asked, why does that mean we have to make it upper case with underscores? Won't the compiler tell us when you try to change it?

Well it's a convention that we used to tell us that we need to use this field differently, like we used to back in the old C days, and it's stuck from there.

But what was so different about C?

So there wasn't really the concept of a immutables for a while in C either. BUT we did have the #define directive. This told the compiler that the next thing it saw on the line would be a symbol that would be replaced exactly with the thing after that.

So you would see something like:
#define STR "my string"

When the compiler saw STR, it would literally replace that symbol with "my string" when it was processing that file.

It was important to know that you were dealing with something special, something that could break in REALLY weird ways! Anyone remember the old "end of file when ';' expected - but it's RIGHT THERE!"?

So here we are in 2020, writing Java variables with different case because it looks like a constant, because we have to know to treat it differently, almost with reverence, because of lessons learned in another language.

Same post at the top of my feed

Pascal Dennerly — Thu, 20 Feb 2020 23:51:55 +0000

Hey,

I signed up tonight and clicked on a post to see what it's about. But now that article, as important and insightful as it is, appears at the top of my feed and I can't figure out how to remove it.

Any suggestions?

Testing Go CLI Tools

Pascal Dennerly — Wed, 12 Feb 2020 21:08:19 +0000

Testing Go CLI Tools (part 1)

This has all started from a very simple tweet from Mat Ryer

I don’t use func main In #golang 😂

But seriously, I always make a new little “run” function and then immediately call that, passing in args and io.Writer for stdout etc.

It’s the easiest way I’ve found to make func main testable. https://t.co/8BQ0EEmU5x
— mat_ryer.go (@matryer) February 12, 2020

What is going on?

Interlude in to my testing journey

So I have a confession to make - I like testing my code. I read about it first in Joel Spolsky’s book “Joel on Software”. Unit testing was a thing that I did a little 20 years ago when I was writing C for mobile phones and then C++ for Windows applications (before there were Apps). There was no common unit testing framework that people always looked to, we just cobbled it together with what we had.

Then I moved to a company that did C# and discovered that there WERE frameworks for you to use. In this case, NUnit. Well, that was a revelation. Everything was tested, every code path and every combination of if statement. It was great to feel like your code worked!

Lets move forward a few years. Unit testing again in C++ with CPPUnit, meeting actual people whose full time job it was to test your software. I even learned some of the theory behind testing, passing a proper QA certification - ISTQB. Only the basic one mind, but it was a start. TDD became a thing that I tried and failed to do. Testing still involved all the code paths.

So now we arrive at the present day. I’ve become less bad. I do TDD before I’ve even created the file that the Production code will go in. I test behaviours and not implementation. Occasionally I even make it so I can refactor my code without needing to change the tests. I’m not so worried about getting all of the code paths checked multiple times but I still care when there are gaps.

Testing CLI tools

It’s actually quite hard to test everything in a command line tool. When you think about it, how is a Unit Testing framework going to help you to parse command line arguments? What happens when some of the output goes to STDOUT? What about STDERR? And how do we get any kind of code coverage statistics?

BATS

My first attempt at solving this uses a great framework called BATS. This is essentially a Bash based Unit Testing framework. You can use this to to make calls to you CLI tool and validate the output. You can see how I’ve done this in a couple of places:

goclitem: a simple, opinionated, Go CLI application template
abbreviate: a tool for abbreviating long strings (it’s a pre-cursor to goclitem, which is why the structure is a little different)

Here we can see that I’m building the tool and then actually invoking it, testing my expectations against the real fully built tool. It’s great. BUT…

We can’t check the code coverage
We can’t use the nice testing tools that Go and 3rd party packages provide

Calling `main()`

This was the next obvious approach. This didn’t work that well. There was none of the control offered by the regular unit tests but there was difficulty intercepting the output.

Using `run`

Now we get on to Matt Ryer’s comment. What is it trying to do? Essentially this allows us to have an entry point to the CLI without having be locked in to main. In part 2, I’ll dig in to how this is used.

Go Interface Generation

Pascal Dennerly — Fri, 26 Oct 2018 21:58:35 +0000

Go Interface Generation

I’d like to rave about a tool that I discovered recently. But first…a little history.

I’ve been working with Buffalo and enjoying it very much. It’s all there in front of you, in Go (my very favourite language) and it’s easy to take your service from the basic out-of-the-box template to adding your own domain models and business logic.

But I also have a background in Java of the Big Business kind. We’re talking full DDD and 3-tier applications in web containers, JCA components and fancy messaging solutions.

Now the thing I like about 3-tier applications is the strong seperation of concerns. Buffalo, as easy as it is, doesn’t have this same seperation. Buffalo deals with this by making strong use of integration tests using a full database implementation and utilities to manage database schemas and populating the database with data according to what is defined in your text fixtures.

In short, I want to be able to take the business logic out of the handler and put it in a repository that I can abstract away so that when I do my testing (first), I can make sure that I’m only testing 1 thing at a time.

How do we achieve this in Buffalo?

Well the obvious way is to move the logic for making queries against the database in to its own type, passing in the dependencies as members of this class. Stub out the dependencies - a route for validating outputs against the inputs that we insert.

Buffalo makes extensive use of the incredibly useful package Pop. Unfortunately, access to the database in this package is via the Connection type. It’s unfortunate because it is a struct, which means that we can’t just insert a mock version of it in to the units that we’d like to test.

To perform this injection of a mock, we need to create an interface that exposes each of the exported functions on the Connection type.

After a little searching, I found ifacemaker. This tools reads the struct and creates an interface that allows access to each of the exported functions on that type.

To run the tool, I did this:

ifacemaker --file ~/go/src/github.com/gobuffalo/pop/connection.go --struct Connection --iface DBConnection --pkg repo

This presented the following type:

package repo

// DBConnection ...
type DBConnection interface {
        String() string
        // URL returns the datasource connection string
        URL() string
        // MigrationURL returns the datasource connection string used for running the migrations
        MigrationURL() string
        // MigrationTableName returns the name of the table to track migrations
        MigrationTableName() string
        // Open creates a new datasource connection
        Open() error
        // Close destroys an active datasource connection
        Close() error
        // Transaction will start a new transaction on the connection. If the inner function
        // returns an error then the transaction will be rolled back, otherwise the transaction
        // will automatically commit at the end.
        Transaction(fn func(tx *Connection) error) error
        // NewTransaction starts a new transaction on the connection
        NewTransaction() (*Connection, error)
        // Rollback will open a new transaction and automatically rollback that transaction
        // when the inner function returns, regardless. This can be useful for tests, etc...
        Rollback(fn func(tx *Connection)) error
        // Q creates a new "empty" query for the current connection.
        Q() *Query
        // TruncateAll truncates all data from the datasource
        TruncateAll() error
}

We can now use this in our Go mocking tools - for example github.com/stretchr/testify/mock. But if we’re going to use a mock, we have to generate one. Here’s where we turn to mockery. The first time you use it - it’ll fail. This is because it doesn’t populate the package imports and package names (for example, Connection rather than pop.Connection). Fortunately this is easy to fix, giving this:

package repo

import (
    "github.com/gobuffalo/pop"
)

// DBConnection represents a Buffalo pop connection to a database
type DBConnection interface {
    String() string
    // URL returns the datasource connection string
    URL() string
    // MigrationURL returns the datasource connection string used for running the migrations
    MigrationURL() string
    // MigrationTableName returns the name of the table to track migrations
    MigrationTableName() string
    // Open creates a new datasource connection
    Open() error
    // Close destroys an active datasource connection
    Close() error
    // Transaction will start a new transaction on the connection. If the inner function
    // returns an error then the transaction will be rolled back, otherwise the transaction
    // will automatically commit at the end.
    Transaction(fn func(tx *pop.Connection) error) error
    // NewTransaction starts a new transaction on the connection
    NewTransaction() (*pop.Connection, error)
    // Rollback will open a new transaction and automatically rollback that transaction
    // when the inner function returns, regardless. This can be useful for tests, etc...
    Rollback(fn func(tx *pop.Connection)) error
    // Q creates a new "empty" query for the current connection.
    Q() *pop.Query
    // TruncateAll truncates all data from the datasource
    TruncateAll() error
}

The next time we run mockery, it magically works. Take a look at this snippet, if it compiles then you have yourself an interface over a struct to unlock your unit tests.

The following code should work, although you may need to fill in the blanks:


func AcceptConnection(tx DBConnection) {}

func TestMocks(t *testing.T) {
    tx := &pop.Connection{}
    mx := &mocks.DBConnection{}

    AcceptConnection(tx)
    AcceptConnection(mx)
}

If you get this far then you have all of the components needed to add a testable repository tier in your Buffalo app.

Semaphores and Threadpools in Go

Pascal Dennerly — Wed, 03 Oct 2018 20:43:56 +0000

Semaphores and Threadpools in Go

Here’s a thing I was thinking about today. I had to make a load of REST calls to an API. The first one was a query to return matching objects and then for each result, make another to get more details. Version one made each request one after the other and it was slow.

Let’s look at some code to show what’s going on.

package main

import (
    "log"
    "time"
)

func main() {
    start := time.Now()

    // Imagine this is the result of a search on a REST API
    results := []int{10, 15, 8, 3, 17, 20, 1, 6, 10, 9, 13, 19}

    for _, d := range results {
        // Imagine this is a long running operation, perhaps another
        // REST API call
        log.Printf("Waiting for %d seconds\n", d)
        time.Sleep(time.Second * time.Duration(d))
    }

    log.Printf("Total time taken: %s\n", time.Now().Sub(start))
}

Example 1

You can see this eaxmple code, along with all the others in this page here.

This waits quite a while then gives the output:

2018/10/03 23:31:19 Waiting for 10 seconds
2018/10/03 23:31:29 Waiting for 15 seconds
2018/10/03 23:31:44 Waiting for 8 seconds
2018/10/03 23:31:52 Waiting for 3 seconds
2018/10/03 23:31:55 Waiting for 17 seconds
2018/10/03 23:32:12 Waiting for 20 seconds
2018/10/03 23:32:32 Waiting for 1 seconds
2018/10/03 23:32:33 Waiting for 6 seconds
2018/10/03 23:32:39 Waiting for 10 seconds
2018/10/03 23:32:49 Waiting for 9 seconds
2018/10/03 23:32:58 Waiting for 13 seconds
2018/10/03 23:33:11 Waiting for 19 seconds
2018/10/03 23:33:30 Total time taken: 2m11.012685s

Over 2 minutes? Surely we could make this faster?

Adding goroutines

What’s the simplest way to make this faster?

It’s to make all the API calls at the same time. Let’s pretend we’re doing that in our example and use go routines to parralelise that work.

for _, d := range results {
    go func() {
        // Imagine this is a long running operation, perhaps another
        // REST API call
        log.Printf("Waiting for %d seconds\n", d)
        time.Sleep(time.Second * time.Duration(d))
    } ()
}

Example 2

Let’s look at what this gives us:

2018/10/05 20:53:11 Waiting for 1 seconds
2018/10/05 20:53:11 Waiting for 17 seconds
2018/10/05 20:53:11 Waiting for 19 seconds
2018/10/05 20:53:11 Waiting for 19 seconds
2018/10/05 20:53:11 Waiting for 19 seconds
2018/10/05 20:53:11 Waiting for 19 seconds
2018/10/05 20:53:11 Total time taken: 571µs

Errrrrm, that’s not right! It’s not obvious but what I think is happening is that the value of d is being modified while the different goroutines are running, so several of them accidentally pick up the same value, that’s different from the value of d when each on was launched. And then all of the goroutines end and the program exits before the log package has written everything to the console.

Don’t end early

We need a way to wait until all of the threads are done. For this, let’s use a sync.WaitGroup. This will allow us to safely track the number of goroutines that have been created and then wait until they done.

var wg sync.WaitGroup

for _, d := range results {
    wg.Add(1)

    go func() {
        defer wg.Done()

        // Imagine this is a long running operation, perhaps another
        // REST API call
        log.Printf("Waiting for %d seconds\n", d)
        time.Sleep(time.Second * time.Duration(d))
    } ()
}

wg.Wait()

Example 3

This time we get this:

2018/10/05 21:43:57 Waiting for 6 seconds
2018/10/05 21:43:57 Waiting for 19 seconds
2018/10/05 21:43:57 Waiting for 19 seconds
2018/10/05 21:43:57 Waiting for 19 seconds
2018/10/05 21:43:57 Waiting for 19 seconds
2018/10/05 21:43:57 Waiting for 19 seconds
2018/10/05 21:43:57 Waiting for 19 seconds
2018/10/05 21:43:57 Waiting for 19 seconds
2018/10/05 21:43:57 Waiting for 19 seconds
2018/10/05 21:43:57 Waiting for 19 seconds
2018/10/05 21:43:57 Waiting for 19 seconds
2018/10/05 21:43:57 Waiting for 19 seconds
2018/10/05 21:44:16 Total time taken: 19.003385s

Well at least we’re getting the correct number of log entries and the time looks about right.

Using correct values across goroutines

This one should be a little easier, we just need to make sure that we pass in the data to the goroutine function that we’re invoking. Right?

for _, d := range results {
    wg.Add(1)

    go func(secs int) {
        defer wg.Done()

        // Imagine this is a long running operation, perhaps another
        // REST API call
        log.Printf("Start waiting for %d seconds\n", secs)
        time.Sleep(time.Second * time.Duration(secs))
        log.Printf("Finished waiting for %d seconds\n", secs)
    } (d)
}

Example 4

Look! We ran them all in order, and they finished when they were done! :)

2018/10/05 22:21:20 Start waiting for 19 seconds
2018/10/05 22:21:20 Start waiting for 20 seconds
2018/10/05 22:21:20 Start waiting for 3 seconds
2018/10/05 22:21:20 Start waiting for 17 seconds
2018/10/05 22:21:20 Start waiting for 10 seconds
2018/10/05 22:21:20 Start waiting for 1 seconds
2018/10/05 22:21:20 Start waiting for 8 seconds
2018/10/05 22:21:20 Start waiting for 6 seconds
2018/10/05 22:21:20 Start waiting for 15 seconds
2018/10/05 22:21:20 Start waiting for 9 seconds
2018/10/05 22:21:20 Start waiting for 13 seconds
2018/10/05 22:21:20 Start waiting for 10 seconds
2018/10/05 22:21:21 Finished waiting for 1 seconds
2018/10/05 22:21:23 Finished waiting for 3 seconds
2018/10/05 22:21:26 Finished waiting for 6 seconds
2018/10/05 22:21:28 Finished waiting for 8 seconds
2018/10/05 22:21:29 Finished waiting for 9 seconds
2018/10/05 22:21:30 Finished waiting for 10 seconds
2018/10/05 22:21:30 Finished waiting for 10 seconds
2018/10/05 22:21:33 Finished waiting for 13 seconds
2018/10/05 22:21:35 Finished waiting for 15 seconds
2018/10/05 22:21:37 Finished waiting for 17 seconds
2018/10/05 22:21:39 Finished waiting for 19 seconds
2018/10/05 22:21:40 Finished waiting for 20 seconds
2018/10/05 22:21:40 Total time taken: 20.003293s

Great…but…they all run at once. Let’s pretend that you’re trying to call a microservice that has some serious perforance issues so making all of those calls at the same time could cause problems. I know, I know, 12 calls at once isn’t really much but let’s pretend that it is. Well we say pretend, I’ve seen real life services in Production that would struggle for many different reasons. Something that I’ve seen more than once is exponential performance drop off - increasing the number of connections or requests increases response times exponentially. And watch out, it’s surprisingly easy to do if you’re not careful!

Limiting the number of concurrent requests

So here we get in to some proper computer-y stuff. Ideally we would use a thread pool so that only a certain number of operations can be carried out at the same time. But this is Go and we’re not really using threads. So what else can we use?

What we need here is to maintain a count of how many goroutines are active right now and stop more from running so we don’t exceed that magic number. We need to block goroutines when we don’t have enough capactity. We need to use a semaphore!

There are a few approaches to using semaphores but fortunately we’re using Go so we can take advantage of channels to give us the correct behaviour without having to explicitly use a thread pool.

For this we need to create a buffered channel, so a channel with certain number of slots that can be filled at the same time. But what do we fill it with? Does it matter? Well it doesn’t, really. Did you know you can pass an empty struct in to channel? Lot’s of details here.

Let’s see how this works:

var wg sync.WaitGroup
sem := make(chan struct{}, 3)

for _, d := range results {
        wg.Add(1)

        go func(wait int) {
                defer func() {
                        wg.Done()

                        // Release the semaphore resource
                        <-sem
                }()

                // Aquire a single semaophore resource
                sem <- struct{}{}

                // Imagine this is a long running operation, perhaps another
                // REST API call
                log.Printf("Start waiting for %d seconds\n", wait)
                time.Sleep(time.Second * time.Duration(wait))
                log.Printf("Finished waiting for %d seconds\n", wait)
        }(d)
}

wg.Wait()

So what does this look like?

2018/10/06 00:12:09 Start waiting for 19 seconds
2018/10/06 00:12:09 Start waiting for 15 seconds
2018/10/06 00:12:09 Start waiting for 10 seconds
2018/10/06 00:12:19 Finished waiting for 10 seconds
2018/10/06 00:12:19 Start waiting for 3 seconds
2018/10/06 00:12:22 Finished waiting for 3 seconds
2018/10/06 00:12:22 Start waiting for 8 seconds
2018/10/06 00:12:24 Finished waiting for 15 seconds
2018/10/06 00:12:24 Start waiting for 17 seconds
2018/10/06 00:12:28 Finished waiting for 19 seconds
2018/10/06 00:12:28 Start waiting for 10 seconds
2018/10/06 00:12:30 Finished waiting for 8 seconds
2018/10/06 00:12:30 Start waiting for 6 seconds
2018/10/06 00:12:36 Finished waiting for 6 seconds
2018/10/06 00:12:36 Start waiting for 20 seconds
2018/10/06 00:12:38 Finished waiting for 10 seconds
2018/10/06 00:12:38 Start waiting for 9 seconds
2018/10/06 00:12:41 Finished waiting for 17 seconds
2018/10/06 00:12:41 Start waiting for 13 seconds
2018/10/06 00:12:47 Finished waiting for 9 seconds
2018/10/06 00:12:47 Start waiting for 1 seconds
2018/10/06 00:12:48 Finished waiting for 1 seconds
2018/10/06 00:12:54 Finished waiting for 13 seconds
2018/10/06 00:12:56 Finished waiting for 20 seconds
2018/10/06 00:12:56 Total time taken: 47.005534s

Example 5

We did it! Admittedly not as quick as earlier, but we only do 3 things at once and therefore not totally overloading our pretend fragile API.

Passing data on to the next step

Now it wouldn’t it be nice if after we make our calls to this API, we processed the results all together? Maybe we’re trying to aggregate something or use the individual bits of data to draw a pretty picture.

Let’s use another channel to pass these response on.

var wg sync.WaitGroup
sem := make(chan struct{}, 3)
responses := make(chan int)

for _, d := range results {
        wg.Add(1)

        go func(wait int) {
                defer func() {
                        // Release the semaphore resource
                        <-sem
                        wg.Done()
                }()

                // Aquire a single semaophore resource
                sem <- struct{}{}

                // Imagine this is a long running operation, perhaps another
                // REST API call
                log.Printf("Start waiting for %d seconds\n", wait)
                time.Sleep(time.Second * time.Duration(wait))
                log.Printf("Finished waiting for %d seconds\n", wait)

                responses <- wait / 2
        }(d)
}

wg.Wait()

for r := range responses {
        log.Printf("Got result %d", r)
}

Example 6

Which gives…

2018/10/06 00:37:04 Start waiting for 19 seconds
2018/10/06 00:37:04 Start waiting for 17 seconds
2018/10/06 00:37:04 Start waiting for 15 seconds
2018/10/06 00:37:19 Finished waiting for 15 seconds
2018/10/06 00:37:21 Finished waiting for 17 seconds
2018/10/06 00:37:23 Finished waiting for 19 seconds
fatal error: all goroutines are asleep - deadlock!

goroutine 1 [semacquire]:
sync.runtime_Semacquire(0xc0000120a8)
        /usr/local/go/src/runtime/sema.go:56 +0x39
sync.(*WaitGroup).Wait(0xc0000120a0)
        /usr/local/go/src/sync/waitgroup.go:130 +0x64
main.main()
        /home/dnnrly/projects-host/sempool/example6/main.go:42 +0x184

goroutine 4 [chan send]:
main.main.func1(0xc00009e000, 0xc0000120a0, 0xc00009e060, 0xa)
        /home/dnnrly/projects-host/sempool/example6/main.go:30 +0x6d
created by main.main
        /home/dnnrly/projects-host/sempool/example6/main.go:22 +0x158

I’ve abreviated this log output, there is a lot of it.

Well, that’s clearly not good! We have a deadlock and looking at the lines of code indicated, it’s a problem with waiting for the sync.WaitGroup to finish and trying to acquire the semaphore. If you run this example yourself you can see it tells us which lines we are blocking on.

Hmmm, the first thing I tried didn’t work. You, reader - go do something else for a couple of hours while I figure this out.

Moving to mutexes

So I am writing this literally 8 hours afte that last paragraph. Admittedly I was doing other things during those 8 hours but still, it took some figuring out. To help me out, I created an alterntive to example 6 with logging. Take a look at the repo if you’re interested.

It appears that the problem is that if we use channels to store the ‘result’ of our calculation then we have a problem. Essentially by using make(chan int), we are creating another buffered channel. But this time, with zero slots - it needs to be read from immediately or else it will block. Which is exactly what happened. Our worker goroutine blocked immediately, meaning that we never get to the end of the goroutines and we deadlock.

It looks like I made the classic mistake of trying to use channels to synchronize rather than comminicate. What we really want to do in this particular situation is append to a slice so that it can be processed later on. To do this we have to make sure that we don’t suffer from other race conditions that might mess up the append operation - the append operation is definitely not thread safe.

To get around this, we use a sync.Mutex to force all of the goroutines to perform the append one at a time.

        var wg sync.WaitGroup
        sem := make(chan struct{}, 3)
        responses := []int{}
        mutex := &sync.Mutex{}

        for _, d := range results {
                wg.Add(1)

                go func(wait int) {
                        defer func() {
                                // Release the semaphore resource
                                <-sem
                                wg.Done()
                        }()

                        // Aquire a single semaophore resource
                        sem <- struct{}{}

                        // Imagine this is a long running operation, perhaps another
                        // REST API call
                        log.Printf("Start waiting for %d seconds\n", wait)
                        time.Sleep(time.Second * time.Duration(wait))
                        log.Printf("Finished waiting for %d seconds\n", wait)

                        mutex.Lock()
                        responses = append(responses, wait/2)
                        mutex.Unlock()
                }(d)
        }

        wg.Wait()

        for _, r := range responses {
                log.Printf("Got result %d", r)
        }

And look upon the glorious success:

2018/10/06 22:16:48 Start waiting for 8 seconds
2018/10/06 22:16:48 Start waiting for 19 seconds
2018/10/06 22:16:48 Start waiting for 10 seconds
2018/10/06 22:16:56 Finished waiting for 8 seconds
2018/10/06 22:16:56 Start waiting for 17 seconds
2018/10/06 22:16:58 Finished waiting for 10 seconds
2018/10/06 22:16:58 Start waiting for 3 seconds
2018/10/06 22:17:01 Finished waiting for 3 seconds
2018/10/06 22:17:01 Start waiting for 15 seconds
2018/10/06 22:17:07 Finished waiting for 19 seconds
2018/10/06 22:17:07 Start waiting for 10 seconds
2018/10/06 22:17:13 Finished waiting for 17 seconds
2018/10/06 22:17:13 Start waiting for 1 seconds
2018/10/06 22:17:14 Finished waiting for 1 seconds
2018/10/06 22:17:14 Start waiting for 6 seconds
2018/10/06 22:17:16 Finished waiting for 15 seconds
2018/10/06 22:17:16 Start waiting for 9 seconds
2018/10/06 22:17:17 Finished waiting for 10 seconds
2018/10/06 22:17:17 Start waiting for 20 seconds
2018/10/06 22:17:20 Finished waiting for 6 seconds
2018/10/06 22:17:20 Start waiting for 13 seconds
2018/10/06 22:17:25 Finished waiting for 9 seconds
2018/10/06 22:17:33 Finished waiting for 13 seconds
2018/10/06 22:17:37 Finished waiting for 20 seconds
2018/10/06 22:17:37 Got result 4
2018/10/06 22:17:37 Got result 5
2018/10/06 22:17:37 Got result 1
2018/10/06 22:17:37 Got result 9
2018/10/06 22:17:37 Got result 8
2018/10/06 22:17:37 Got result 0
2018/10/06 22:17:37 Got result 7
2018/10/06 22:17:37 Got result 5
2018/10/06 22:17:37 Got result 3
2018/10/06 22:17:37 Got result 4
2018/10/06 22:17:37 Got result 6
2018/10/06 22:17:37 Got result 10
2018/10/06 22:17:37 Total time taken: 49.006433s

Ay you can tell, I’m rather happy with this result.

Conclusion

The 2 hardest things in computing are naming things, concurrency and off-by-one errors.
Someone famous in the Go community, on Twitter (I forget who)

So we’ve looked at how we can do work (make API requests?) concurrently and what we can do to limit just how concurrent we want to be. We then looked at how we can pass results from this concurrent work on to the next step in the process, without accidentally reading data in corruptly or causing a deadlocks.

DEV Community: Pascal Dennerly

Quitting early for the win!

Testing your exits

Attempt 1 - just run your program

Attempt 2 - mock out the exit

Removing code to remove tests

Introducing gobail

Isolating dependencies

Conclusion

Introducing Layli

A manifesto

Getting started on Layli

What next

What will make me happy, as a #hacktoberfest maintainer

Hey #hacktoberfest beginners!

Deciding on a contribution

What changes can you make?

Making that change

What you can expect from me

What I want you to get something from #hacktoberfest

Introducing Abbreviationist

Fun Coding

Rewarding Github contributions

Learning Frontend as a Backend dev

THIS_IS_A_CONSTANT

Same post at the top of my feed

Testing Go CLI Tools

Testing Go CLI Tools (part 1)

Interlude in to my testing journey

Testing CLI tools

BATS

Calling main()

Using run

Go Interface Generation

Go Interface Generation

How do we achieve this in Buffalo?

Semaphores and Threadpools in Go

Semaphores and Threadpools in Go

Adding goroutines

Don’t end early

Using correct values across goroutines

Limiting the number of concurrent requests

Passing data on to the next step

Moving to mutexes

Conclusion

Calling `main()`

Using `run`