DEV Community: Preslav Mihaylov

Lessons learnt from publishing my programming course

Preslav Mihaylov — Sat, 05 Feb 2022 16:08:04 +0000

I’ve been working on my course — Learn Programming with Go, One Game at a Time, for nearly a year & made many mistakes along the way.

Nevertheless, I finally published it & earned $2,158.25 in a month:

Here’s what I’ve learnt along the way 👇

Getting started is scariest

One of the scariest things of making my course was getting started.

I’ve been teaching programming to 2000+ students since 2015 in bootcamps & universities, but I never felt ready to make my own course.

I realised that I never will be & took the leap. Best decision ever!

It’s harder than it looks

Initially, I thought designing the curriculum & recording the videos is the bulk of the effort.

There was so much more — choosing the right platform, video editing, marketing, designing the logo, etc…

I thought that making the course would take me ~3 months in total. I ended up spending more than a year working on it.

But most of that time was lost because I had no clue what I had to do & missed many opportunities to move efforts in parallel.

For example, I first spent 3 months recording all the videos & just then started to look for a video editor to edit all of them. If I do this again, I’d rather do both of those in parallel — record one week’s worth of content & send it over for editing.

What’s more, I started thinking about the course poster only after everything else was ready. Little did I know that would also take more than a month because the illustrator I hired had too many orders in their queue.

If I had known all this & optimised my workflows, I expect I would have been able to finish everything in 5–6 months.

Choosing the best platform

In terms of platform, I thought about building my own at first, but quickly realised that’s infeasible, unless I have a year or so to spare.

Hence, I decided to use one of the existing teaching platforms. The ones I was considering were Teachable, Slip and Udemy.

The first two looked good but I thought they were more suited for creators with established audiences.

I chose Udemy because I could keep most of the profits from my channels but they also offered free marketing on my behalf for a bigger commission, which was fair.

Pitfalls to watch out for when recording

Recording the videos seemed trivial but there were some pitfalls to watch out for:

Maximise your IDE/Browser/etc font size.
Record at max possible quality.
Don’t record \w low energy. Hard to fix later.
Have the video editor in mind. Don’t make them guess.

To video edit or not

At first I thought about doing it myself.

And I did… for a single lecture, which took me more than 20 hours.

Totally not worth it. Hire a good video editor who’ll do it much faster at a reasonable price.

Fiverr is a great place to find one.

The beautiful course poster

Fiverr is also the place I found an amazing illustrator for the course poster.

Some might consider this micro optimisation, but for me, it was totally worth it. I don’t care if it brings more engagement or not, I just love it.

The dreadful marketing…

In terms of marketing, I shared the course \w my inner circle first — friends & family who gave me good initial reviews.

Next, I shared it via Linkedin to my professional circle.

But what worked best was sharing it to the Golang community on reddit.

What didn’t work was using twitter promotions. My tweet got good engagement, but not many actually bought the course.

I spent around $115 for a single day, which lead to 3 people buying the course.

I’m probably just a marketing newbie ¯\_(ツ)_/¯

Conclusion

Finally, no matter how optimised my setup is, the most important thing for me is delivering very high-quality content.

I’d rather grow to be a niche instructor with courses students adore vs. pushing my way through with growth hacks & delivering sugarcoated money-grabbers.

If you wanna learn more about my journey, follow @PreslavMihaylov on Twitter.

How to set up Vim for Go Development

Preslav Mihaylov — Sun, 11 Apr 2021 08:35:41 +0000

One of the major reasons I love programming in Go is that I can do it in any IDE or editor I want, including my favourite Vim editor.

Unlike Java, for example, which is highly coupled to IntelliJ or Eclipse, Go is much more lightweight and the tools you use with it can be integrated with any editor.

In this guide, I will help you setup your Vim as a fully-fledged Go IDE, including the plugins you need to install but also setting up the important options & mappings which you'll use daily.

Prerequisites

You'll need to have Go installed & your Go environment setup, namely your $GOPATH and related subdirectories - /bin /pkg /src.

You'll also need to use nvim instead of vim or have your vim updated to version 8+.

The Main Vim plugin for Go

One of the only plugins you need for Go is called vim-go. It aggregates all the tools you need to get started with Go development.

To install it via pathogen:

git clone https://github.com/tpope/vim-pathogen ~/.vim/bundle

Other installation options.

After you've installed it, you can customise it using these options which are highly opinionated, but work best for me:

" disable all linters as that is taken care of by coc.nvim
let g:go_diagnostics_enabled = 0
let g:go_metalinter_enabled = []

" don't jump to errors after metalinter is invoked
let g:go_jump_to_error = 0

" run go imports on file save
let g:go_fmt_command = "goimports"

" automatically highlight variable your cursor is on
let g:go_auto_sameids = 0

For syntax highlighting, use these options:

let g:go_highlight_types = 1
let g:go_highlight_fields = 1
let g:go_highlight_functions = 1
let g:go_highlight_function_calls = 1
let g:go_highlight_operators = 1
let g:go_highlight_extra_types = 1
let g:go_highlight_build_constraints = 1
let g:go_highlight_generate_tags = 1

Finally, install all the required go tools by opening vim and running:

:GoInstallBinaries

coc.nvim - Autocompletion & linting

In the past, I've relied on YouCompleteMe for auto-completion or vim-go's native integration with vim for auto-completion & ALE for linting/static code analysis.

Nowadays, I prefer using coc.nvim and its plugins to do all that as I've found it to be more reliable than all of those tools.

For installation instructions, follow the Quick Start guide.

After you've installed it, run this command to install the necessary coc extension for Go:

vim -c 'CocInstall -sync coc-go coc-html coc-css coc-json|q'

The html/css/json extensions are optional but who doesn't deal with those nowadays. You'd probably need them at some point.

Afterwards, open vim & type :CocConfig to open coc's configuration file. These are my opinionated settings, stripped from the non-go related settings:

{
  "suggest.noselect": false,
  "diagnostic.errorSign": "✘",
  "diagnostic.warningSign": "!",
  "diagnostic.infoSign": "?",
  "diagnostic.checkCurrentLine": true,
  "coc.preferences.formatOnSaveFiletypes": [
    "javascript",
    "html",
    "json",
    "css",
    "scss",
    "go"
  ],
  "coc.preferences.hoverTarget": "float",
  "languageserver": {
    "golang": {
      "command": "gopls",
      "rootPatterns": ["go.mod"],
      "filetypes": ["go"]
    }
  },
  "go.goplsOptions": {
    "staticcheck": true
  }
}

For all auxiliary coc settings, add these settings to your .vimrc.

Mappings For The Most Useful Commands

Having great tools is not enough. You need to know how to use them. Hence, it is best to customize some of the mappings which vim-go and coc provide you so that you're more effective when dealing with Go code.

Here are the mappings I use most often and the way I've configured them:

Manage unit tests in the current file

I've mapped running all the tests in the current file to <leader>-t which translates to \ + t on my Mac & Linux.

autocmd BufEnter *.go nmap <leader>t  <Plug>(go-test)

I've also mapped \ + tt to run the current test function only, instead of running all of them:

autocmd BufEnter *.go nmap <leader>tt <Plug>(go-test-func)

Finally, I use \ + c to toggle the coverage profile for the current file I'm in:

autocmd BufEnter *.go nmap <leader>c  <Plug>(go-coverage-toggle)

Inspect a Go Codebase

Show the function signature for a given routine with \ + i:

autocmd BufEnter *.go nmap <leader>i  <Plug>(go-info)

Show the interfaces a type implements with \ + ii:

autocmd BufEnter *.go nmap <leader>ii  <Plug>(go-implements)

Describe the definition of a given type with \ + ci:

autocmd BufEnter *.go nmap <leader>ci  <Plug>(go-describe)

See the callers of a given function with \ + cc:

autocmd BufEnter *.go nmap <leader>cc  <Plug>(go-callers)

Find all references of a given type/function in the codebase with \ + cr:

nmap <leader>cr <Plug>(coc-references)

Go to definition/Go back with Ctrl+d and Ctrl+a:

nmap <C-a> <C-o>
nmap <C-d> <Plug>(coc-definition)

Refactor Go Code

Not many options here, but there's renaming the symbol your cursor is on with \ + r:

nmap <leader>r <Plug>(coc-rename)

Occasional Maintenance of Tooling

To update all Go tools, run this from vim:

:GoUpdateBinaries

And updating all coc plugins:

:CocUpdate

To upgrade vim-go or coc.nvim, delete the folders from ~/.vim/bundle and git clone the repos again.

Setup The Cool gruvbox Theme (Optional)

Install gruvbox using pathogen:

git clone https://github.com/morhetz/gruvbox ~/.vim/bundle

Enable it in your .vimrc with some opinionated extra options:

colorscheme gruvbox
autocmd ColorScheme * highlight CocErrorFloat guifg=#ffffff
autocmd ColorScheme * highlight CocInfoFloat guifg=#ffffff
autocmd ColorScheme * highlight CocWarningFloat guifg=#ffffff
autocmd ColorScheme * highlight SignColumn guibg=#adadad

Want to see how it looks?

Setup tmux For Terminal Multiplexing (Optional)

If you want to manage multiple terminal tabs in the same window, use tmux (See my how to guide):

This is one of the most useful tools I have setup because it allows me to do my code editing in vim and manage all sorts of other terminal-related tasks in separate tabs all inside the same window.

Conclusion

And that's everything I use for my daily Go programming.

This setup has worked extremely well for me and have in mind that I'm also dealing with Uber's huge Go Monorepo, where I've been able to integrate my vim environment seamlessly (with some extra settings which you probably won't need.

If you want to see my entire vim development environment, check out all my dotfiles and default-setups repos, which include my full setup, including external programs, the terminal, vim, tmux, colorschemes, etc.

Happy Gophing!

Check out some of my other Go series:
Integrating your Go Service with ELK
gRPC With Go Crash Course

Thread-Safety in Go - an Overlooked Concern

Preslav Mihaylov — Tue, 29 Dec 2020 18:04:02 +0000

When you first start tinkering with concurrency in any language, what's the most sought after goal is extracting the maximum output from the hardware you have available.

That's why we use it and study it in the first place - to achieve optimal parallelism & get some significant speed-up for our apps out of the box.

However, a not so glamorous part of studying the subject is understanding how to write thread-safe code. The techniques and principles which will enable you to keep your application well-behaved, even after scaling it to dozens of threads.

Even though this is an important thing to consider while writing concurrent code, it is often overlooked in most learning resources about concurrency. This problem is even more relevant in the Go community due to two common misunderstandings I will cover in the article.

What does thread-safety mean anyways?

Let's first explore what thread-safety is exactly. One reasonable way to define the term is to think of it in terms of correctness.

If a component conforms to its specification, it is correct. Say you have an Interval struct which defines two limits - lower and higher. The important invariant instances of the struct have to maintain is lower < higher. If at any point, this invariant is not true, then this class is incorrect.

A thread-safe class, on the other hand is able to sustain its correctness even after being bombarded by a dozen of threads, accessing it concurrently.

Now, when designing thread-safe components, there are also other concerns one has to consider. One such concern is achieving reasonable performance - if a component performs worse in a multi-threaded environment, instead of a single-threaded one, it's obviously not a great solution.

However, for the scope of this article, we will just worry about making a component thread-safe - i.e. correct in a multi-threaded environment.

Why should you care about thread-safety?

Most people understand that thread-safety is an issue. When exploring the topic of concurrency in any language, they've probably seen at least a single thread-safety issue.

Consider this example:

When I run this code on my laptop, I get as result 9303. The second time, it was a different number. When you run it, it might even be correct.

This is an example of non-thread-safe code. The expected behaviour for this routine is to print 10000 when I increment the counter that many times, albeit concurrently.

Instead, when I ran it the first time I got 9303. The second time was a different number which was still incorrect. Try running it on your end and see what's your lucky number.

Given this example, most people acknowledge that writing thread-safe code is important as otherwise, you are subject to sporadic bugs which would be very hard to trace, debug and reproduce.

Even so, the concern of writing thread-safe code is still quite overlooked by many due to two main reasons:

They think they don't need to worry about it because they don't directly spawn threads in their applications
They think they don't need to worry because they use channels and there is no memory sharing when using them

Unfortunately both of these assumptions are incorrect for most people. Let's see why.

You are already writing multi-threaded applications

Let's first cover the subtle fact that you are already writing multi-threaded applications even if you haven't spawned a single goroutine in it.

Have you ever created an http handler using Go's standard http package?

Congratulations, you are already writing multi-threaded software!

Although you aren't spawning goroutines in your codebase directly, you are using a framework which spawns one for every incoming http request.

This means that there are already multiple threads, executing your code. Hence, it needs to be thread-safe to avoid subtle bugs, which would be quite hard to find.

By the way, this is applicable to any multi-threaded framework or library you might use. Even if you're simply spawning a timer to execute some background task, you're already dealing with a multi-threaded application.

Go channels are great, but not always applicable

There's this notorious technique, applied in Go, of not communicating by sharing memory but sharing memory by communicating.

You can achieve this via Go's channel primitive. It's a data structure which enables one to achieve thread-safe code in a multi-threaded environment. It's nice because you don't need any synchronisation whatsoever as long as you use it to share data across threads.

It, undoubtedly, is a great solution for avoiding thread-safety issues when writing concurrent code. However, it is not applicable in all situation.

Let's explore an example

Take this simple web chat implementation:

This looks like a perfectly reasonable application, where no goroutines are spawned. When you test it out via a rest client, it appears to work great!

However, if you deploy this code into production and enough users start using it, you will start receiving a flood of bug reports which you can hardly reproduce.

What's more, changing this code to work with channels in quite impractical as the resulting code will be hard to understand and reason about. A much more practical solution is sticking to the good old synchronisation primitives, implemented in the sync package.

And this is where the classic, well-known practices for achieving thread-safety are applicable in Go, despite the advent of channels.

Are you curious if you have any non-thread-safe classes in your own web application?

One common source of thread-safety bugs in web applications is having a struct field of type slice or map whose state you change in any function. If that struct instance is shared across multiple http requests and you don't have any synchronisation, then that's probably a non-thread-safe component.

Some good resources on thread-safety & concurrency

Most existing resources cover the subject of thread-safety in a very shallow way. I know this as after consuming many of them, I've still felt that I don't understand it at all.

For example, one of the only books on concurrency in go - Concurrency in Go, dedicates a single chapter on covering the problems of thread-safety. And it doesn't even go into enough detail exploring what options we have for solving them or even demonstrating all possible sources of thread-safety violations.

Once you read that chapter you would, at best, have a shallow understanding of the fact that you need to be careful when writing concurrent code. Oh, and that there is something called a lock which appears to be helpful for resolving that issue.

So far, I've found a single resource on concurrency which goes into enough depth on the subject, exploring why thread-safety issues happen and how to avoid them - Java Concurrency in Practice.

If you want to explore concurrency and thread-safety beyond this article, then I highly suggest you check out that book. My book notes can also greatly aid you in digesting it.

Oh, and I know it's a Java book, not a Go one but trust me. It's still better than all the Go books/courses on the subject.

Additionally, I'm planning on releasing several additional articles on the subject, covering the subject in more detail. If you want to get a notification once I release the next article in the series, make sure to subscribe.

Conclusion

Thread-safety is an important concern for stable & reliable applications, nowadays as most of us are already writing multi-threaded applications.

If you want to avoid the subtle bugs arising from thread-safety issues, you should make sure you sufficiently understand the topic.

In order to bridge the gaps in your knowledge, check out Java Concurrency in Practice. Also subscribe for the blog for the next articles in these series on thread-safety in Go.

Good and Bad Practices - a Limiting Perspective

Preslav Mihaylov — Sun, 15 Nov 2020 19:32:31 +0000

Growing up as software engineers, we typically spend the first few years of our careers studying various technologies and concepts from the programming area - what a list is, what a hash table is, what's JavaScript, what's HTTP, what is a client-server architecture...

The goal of this venture is for one to accumulate the necessary skills & knowledge to land their first jobs.

But apart from understanding "what" something is, over the years, the preliminary knowledge required to land your first job has expanded to also cover topics which help one not just write software that works, but also write "good code".

Most aspiring software engineers nowadays are advised to read up about the SOLID principles, about what "clean code" is, they're also taught to split up their applications using the MVC "architecture" and have separate models, views and controllers.

Concepts vs. Principles

The difference between learning how a linked list works and what the S in SOLID stands for is that the former is a concept which is easier to understand as it solves a problem much more familiar to a beginner - it helps one efficiently store data in particular use-cases.

The Single Responsibility Principle, on the other hand, is a principle which has been carved out in our software engineering books thanks to the deliberate efforts of generations of programmers who've accumulated years of both good and bad experience and they've distilled what they've learnt into a single-lined principle.

But the simplicity of it is deceptive.

Memorising a single-lined principle and a few examples of applying it it simple and anyone can do it. Any junior developer learns to cite the S in SOLID by heart before their first job interview.

However, it is much more challenging to understand it.

To understand and appreciate the solution, one must first understand the problem it solves.

Then, you can look at the principle as a clever approach to dealing with a common problem.

Trying to use a principle without understanding the problem it solves is like trying to use a hammer when you don't know what a nail is.

But we as a society, have always tried to circumvent the long and hard road to true understanding. We're always looking for the shortcut.

Hence, we are teaching what SOLID is quite well, but we're not teaching why it exists or what problem it solves nearly as well.

Suddenly, you receive a hammer and everything looks like a nail you can hit.

Good and Bad Practices

Thanks to all this, we've developed the concepts of good and bad practices.

We've come so far as to be disgusted from certain patterns in our codebases. Using a global variable or directly initialising a dependency in your class instead of using dependency injection is considered a sin.

Why?

Because it's a "bad practice". It is not "clean code".

However, in programming there are no perfect choices. There are only trade-offs.

Every little idiom or witty technique you apply in your codebase has a cost associated with it. Most often, that cost is in increasing the complexity of the codebase.

After you do this too many times, it suddenly takes an hour to initialise a simple utility class you need for a small part of the larger task you are solving.

You want to directly construct a new class? No, you can't do that. You should use a factory instead. But wait, to make things even more sophisticated, we'll make it a "builder factory" so that the code is "extensible".

This vicious cycle continues to a point where you have a small project, for which you need people with PhDs in software design to understand. And I've actually seen that.

I've worked on a giant project with half a million lines of code, which is easier to understand than a 10k lines project.

That is the price we pay for over-engineering by blindly following the "good practices".

Oh, and by the way, this problem extends beyond code. It has become prevalent in our system designs as well.

For example, most projects built with a "micro services" architecture I've seen don't really need micro services. They could have just sticked to the not-so-glossy monolith and saved themselves dozens of problems, which the shiny buzzword brings with itself.

Dealing With Imperfect Principles

Although I've been painting a not-so-rosy picture of all the "good practices" in this article so far, note that there is nothing wrong with the principles themselves. They have become popular and widely adopted because they are quite effective in solving some of the problems we often face while writing code.

However, by not understanding the problems they solve and the problems they introduce, it is not easy to recognise when to use them and when to avoid them.

Therefore, before applying any sort of design patterns or good practices, read up about them. Don't just pick up the first vague article which either simply explains how to use them or, worse, try to sell them by outlining all the benefits while avoiding the costs.

What's even better, try to figure that out yourself. Oftentimes, techniques for writing good code address the problem of making your project more maintainable.

What that means is based on context.

Sometimes, making your project more maintainable means writing code which is easy to test. Other times, it is about writing code which is easier to read.

Some techniques contribute to more specialised use-cases. Such are, for example, making the frameworks and peripherals of your codebase more detached from the rest of the project.

And note that certain techniques might aid you in some of those goals, while impeding you in others.

For example, applying dependency injection might make writing tests easier, but it makes initialising and using your components harder. Is it worth the cost?

It depends. Sometimes it is, other times it isn't. You are the one to decide as only you understand your project well enough.

A Case Study

Oftentimes, sticking to simpler idioms, condemned as bad practices, might actually be more beneficial in your use case.

Check out Go's standard library http package for example. In it, there is a global http client variable.

Perhaps some bells start ringing right now - that makes the package harder to test, it raises the risk for thread-safety issues, it breaks encapsulation, etc etc.

And although most of those claims might be true, there is another thing which also holds true - it makes using the package much easier for simple use-cases.

Thanks to that decision, in order to make a simple GET request, all you need to do is:

http.Get("http://example.com/")

That's it. There are no initialisations, no configurations, no need to look into how to configure your timeouts & headers. You just stick to the reasonable defaults which the global variable gives you and in most cases, that works just fine.

This is an example of how, for the Go team, the widely adopted wisdom of avoiding global variables didn't apply to this specific use case they were aiming for - making the http package easy to use.

Conclusion

In software, there are no silver bullets. Not even the globally-recognised principles for writing good code.

Every pattern, principle and idiom is created with a specific problem in mind. And all of them have a certain cost you'll have to pay for applying them.

In order to use them effectively, you should examine both sides of the coin.

Otherwise, you will have to continue your endeavour in software engineering by blindly following certain patterns while condemning others as bad practices.

At that point, decision making will no longer be about making rational choices, but more about maintaining religious beliefs.

My thoughts on Java Concurrency in Practice

Preslav Mihaylov — Tue, 06 Oct 2020 14:15:53 +0000

One of my key goals this year was to gain a good understanding of concurrency and multithreading. I've always had a tangent understanding of it at best. The book Java Concurrency in Practice was one of my first picks.

Sure, I've used multithreading concepts like promises & ajax in JavaScript before. I've also spawned go routines & used mutexes in Go. But I've never felt I'm proficient enough to state that I have a good understanding of this subject.

Hence, to bridge this gap in knowledge & skills, I decided to invest in several concurrency related books & courses. My intention was to start from JCIP and then move on to some additional courses. Initially, I felt that just reading this book won't be sufficient to understand the subject thoroughly.

However, after going through this book I didn't bother looking into any other concurrency course at all. It is one of the most succinct and yet thorough books on a given topic that I've ever read.

Additionally, I would recommend you to go through this book even if you're not using Java at all. It will give you a very profound understanding of whatever framework/mechanism your language of choice uses to tackle concurrency.

Here's why...

Completeness

Java Concurrency in Practice is very thorough as it takes you from the basics you need to understand about concurrent programs all the way to the advanced topics, where the "dessert" is.

However, it is also quite succinct as it doesn't drift into unnecessary details or complexities which aren't foundational to the subject.

Many books and courses fail to achieve this.

Some drift too much towards the theoretical and impractical. This makes the reader impatient and eventually, repels him from the book/course altogether. Most classical computer science books are like this, making them quite unattractive to novices and professionals alike.

Others skip the foundations and simply focus on showing you "how it's done in X", where X is your favourite language. This kind of content might, at best, leave you hungrier than you were at the beginning. Worst case scenario, it might fool you into believing that "you know it all" and not study the subject further.

There are very few courses & books which achieve the golden balance between theory and practice.

Accessibility

Many books tend to be written in a very high-level way, skipping appropriate examples and using inaccessible language/terminology.

These kinds of books require unnecessary additional attention in order to understand the context alone. This is quite deterring, especially when the reader realises that there are good alternative books on the subject half-way through.

I avoid investing too much time in such books as reading those often contributes solely to "ego-reading" - giving you the prestige of a person who has read a book, which is considered a hard read. Understanding it is, though, is not a given.

Now, I would consider Java Concurrency in Practice to be quite accessible and written in a very pedagogical way. It is full of examples to supplement the book & aid you in "getting it".

However, I wouldn't mark it as an easy read. It took me substantial effort to go through the book. I read through each chapter at least twice, while also taking notes along the way.

Critique

Lack of exercises

What the book unfortunately lacks is a set of good exercises.

That would have been tremendously helpful as it supplements the read-through with "getting your hands dirty".

This is why I had to tinker with the concepts in the IDE myself. This was the only way to back my theoretical understanding with enough practice.

Google doesn't have any good suggestions either. The top result which pops up after you search for "concurrency projects" suggests some quite impractical projects. Projects such as creating your own concurrent hash map or implementing an atomic variable.

I find this to be bad exercise advice as all these are concurrency primitives. Knowing how to implement them yourself is useless.

They are already available out of the box and the book goes into enough depth to explain how they work.

What's more, these exercises won't prepare you for an actual concurrency-related problem. Just because you know the concurrency primitives doesn't mean you can use them effectively.

It would have been great if there were a couple of exercises in addition to each chapter. This would aid one in putting the concepts covered to practice.

Although the book is lagging on this important point, fortunately it is still very well-made in all other aspects. Additionally, the provided examples help one bridge the gap even though you'll have to do some homework afterwards.

The book doesn't cover modern tools & frameworks

The other critique I've often encountered is that the book doesn't cover the most modern practices in tackling concurrency. One could expect this given the book's age.

However, I don't consider this to be that important. The book focuses on providing you with understanding of the fundamental concurrency concepts and problems.

After you go through it, learning about the latest concurrency framework is a breeze.

It is far more beneficial to understand the concepts covered in JCIP than it is to know the API of the latest framework.

How to approach this book

JCIP quite thoroughly covers the subject of concurrency and the author has also written it in an accessible way. However, this is not a "feel-good" book. It's not meant to be read from cover to cover. It is also not meant to be among the collection of 100 books you've read this year. It will definitely impede you in achieving that "paramount" goal.

I'd advise readers to go through most chapters multiple times, especially the first five ones. In addition to that, I'd suggest one to take notes and even spend some time in the IDE tinkering with the concepts learnt.

Have in mind that the author doesn't spend too much time explaining the APIs of the library routines they use. Don't be taken aback by this, simply keep a browser tab open, ready to look up the API documentation of the methods being used.

Now, I wouldn't recommend these measures with any book as it is quite the time investment. But it's definitely worth it for this one.

There are easier books & courses on the subject out there.

But almost all of them leave you with a "there's something missing" feeling. This makes you hop from one piece of content to another.

This contributes to that never ending feeling that you don't have a good enough understanding of concurrency.

Java Concurrency in Practice is not like that. It will leave you fulfilled & confident that you understand the subject. After going through it, you'd rarely want to look to another book/course on the subject.

But it does demand its share of attention and care. Don't pass on this opportunity, it is definitely worth it.

Prerequisites

Although JCIP is accessible, it still demands some backbone knowledge before starting it.

Make sure that you are comfortable with an object-oriented programming language. You should also be comfortable opening the official documentation of an API over a stack overflow question.

In addition to that, the book doesn't cover the very basics of multithreading.

Concepts such as what a thread is, how it works or how to start one up. Make sure you spend a couple of days/weeks studying that beforehand.

Conclusion

Java Concurrency in Practice is one of the best books I've read on concurrency. I would recommend it to anyone, even if Java is not their language of choice.

Its lack of exercises does force one to do some homework. But even so, the course makes most other content on concurrency obsolete.

Due to this, JCIP does demand considerable attention, despite it being accessible, but it's definitely worth it.

Using Kibana to Debug Production Issues

Preslav Mihaylov — Sun, 16 Aug 2020 16:52:01 +0000

In the last chapter of these series, we covered how to create your first Kibana dashboard for a sample web application.

Although that will help you setup Kibana for your production services, what you'll be doing 99% of the time would be to actually use the dashboard to monitor & debug issues you observe on production.

This is the final post in the series for monitoring Go applications \w Kibana. It will give you some practice using your newly created dashboard to identify problems with a sample web application.

So if you've setup Kibana for the first time or you've just joined your new team & seen your Kibana dashboards without a clue what's going on, this post is for you!

The Starting Point

First, make sure you have docker and docker-compose installed.

For docker, follow the installation instructions for your platform here.
For docker-compose, follow the installation instructions here.

Afterwards, download this exercise’s repository on the kibana-debugging-tutorial branch:

git clone --branch kibana-debugging-tutorial https://github.com/preslavmihaylov/tutorials

Boot up all docker containers:

docker-compose up

Run this script to import the dashboard to Kibana from the project’s root:

./import-kibana-dashboard.sh

This is the exact same dashboard as the one we create in this post.

Now, you should be good to go.

Verify everything is working properly by going to http://localhost:5601/app/kibana#, opening the Dashboard tab and making sure you see some incoming traffic with occasional errors:

Your Goal

In this tutorial, you have a black box application, which has the following HTTP API:

/payments/execute
/payments/list
/payments/authhold

In addition to these endpoints, what you'll see in the dashboard are some additional properties:

method - the HTTP method invoked
countryISO2 - the ISO2 of a user's country (e.g. US, FR, UK)
userID - the ID of the user making the request
paymentMethod - the payment method used for making a transaction
userType - the type of user (e.g. trial, individual, business)

Inspect the panels in the dashboard to see some sample data for these properties.

This API is the same as the one seen in the previous tutorials from the series.

The difference is that this time, the application fails under certain hardcoded conditions.

Example:

The application fails when the country is “US” and
the endpoint is “/payments/list” and the method is “PUT”

This is just an example, of course, it doesn’t necessarily mean that the application returns an error under these exact conditions (or does it?).

However, the failure conditions will be of that kind.

Your task is to use the Kibana dashboard you are given, apply some filters to narrow down the errors and figure out what are the exact conditions which cause an error.

There are 5 error conditions which you have to find out.

Here are the hints:

The application fails when ________ and ________
The application fails when ________ and ________
The application fails when ________
The application fails when ________ and ________ and ________ and ________
The application fails when ________ and ________ and ________ and ________

The hints are ordered by the difficulty of finding the given error condition - The first condition is the easiest to find.

One caveat to have in mind is that you won’t be 100% sure that a given error condition you think you’ve found is correct.

Instead, focus on finding the first two error conditions first, write them down and then consult the walkthrough for the first two issues to verify if your assumptions are correct.

Then, continue with the third and final two error conditions and check the walkthrough for each of those steps again.

The Walkthrough

Before you go through this, I urge you to attempt the exercise on your own.

Use this only if you get stuck or if you want to check out some alternative ways to debug the issues.

Well, without further ado, let’s get started.

Finding Issue #1

Issue #1 is:

The application fails when ________ and ________

First, go to your Errors table and filter the dashboard to only show the errors:

This will only filter the errors in the view, which will hide the irrelevant details we aren’t interested in.

After we do this, this is an example of what your dashboard should look like:

Notice the applied filter at the top-left. That’s the bar which shows all filters applied to the current dashboard. This is one of the main mechanisms you’ll use when debugging production issues.

Next, notice how the greatest amount of errors appears to happen on the /payments/authhold endpoint.

Apply that to the current filter as well:

After we apply this filter, take a look at your tables:

It seems that the errors per value are evenly distributed across all tables except for the HTTP Method. There is an abnormal amount of errors which occur when the HTTP method is POST.

Apply a filter on the POST method and let’s see the results:

Notice that the rest of the errors per value remain evenly distributed.

This means that our first error condition is:

The application fails when the endpoint is /payments/authhold and the HTTP method is POST

That’s a good start.

One important thing to do before you move on, however, is to exclude the current error conditions from the rest.

This will allow you to focus on the errors that are left, rather than rediscover the errors which you’ve already found.

It is enough to exclude one of the error conditions (as both of them have to match).

To do this, clear all current filters except for the error filter:

Next, exclude the endpoint /payments/authhold:

This is how your filters bar should look like:

Of course, it is possible (and is the case) that there are other error conditions that aren’t the one we found for the endpoint we excluded.

However, we will have to do some heuristics like this to narrow down the rest of the errors until there is no more room to “zoom in on”.

At that point, we will have to relax our filters a bit until something we are yet to discover pops up.

Typically, when working in a production environment, you would approach this kind of issue by finding a few problems at a time.

Only after you’ve fixed the most outstanding issues, will you narrow down on the rest of the issues in your application which aren’t so pervasive.

However, in this context, it would be quite shallow to only get a single error condition to discover. This is why you’ll have to discover all 5 at once.

But don’t worry, I’ve made sure the error conditions are quite distinct to one another in order to enable you to more easily discover them.

Let’s move on to the next issue.

Finding Issue #2

Issue #2 is:

The application fails when ________ and ________

To find this one, we’ll take a very similar approach to the previous one. Take a good look at the errors distribution at a high level.

Notice the outstanding amount of errors for the /payments/list endpoint.

Filter by that value and take a look at the dashboard again:

As previously, there is one outstanding value which catches the eye. it is the ES country.

Filter by that value now:

As you see, it looks like all other values are evenly distributed when we filter by endpoint /payments/list and country ES.

This means, that the second error condition our application has is:

The application fails when the endpoint is /payments/list and the country is ES

Nicely done. Now remove the filters we added in this step and exclude one of the error conditions we just discovered.

In this case, I would choose to exclude the ES country as excluding another endpoint will make the resulting view way too narrow.

This is what your filters bar should look like:

Let’s move on.

Finding Issue #3

Issue #3 is:

The application fails when ________

Taking a look at the view we have now, there is nothing that immediately pops out:

So it's hard to narrow down in any direction from here.

However, there is one view on our dashboard which we’ve neglected thus far.

It is the detailed logs view at the bottom of the dashboard.

It will be the one which will guide us in the right direction:

Do you notice something abnormal?

The amount of errors for user ID 68 tends to be quite high in comparison with other user IDs.

Perhaps there is something wrong with that user. Apply a filter on his user ID and see what the result is:

Notice that after we filter user ID 68, there is nothing extraordinary which pops out. Perhaps the application unconditionally fails for this user every time?

To test this hypothesis remove all filters except for the user ID. This is what the filters bar should look like:

After we do this, notice that there isn’t a single successful request for this user:

This is our third error condition:

The application fails when the user ID is 68

Now that we’ve discovered this issue, there are still two more to go.

Finding them now will be significantly easier than at the beginning because we’re going to filter all other error conditions so that only those remaining will pop up.

To do this, remove all filters and exclude the erroneous values from the previous issues along with the user ID we just found:

Finding Issues #4 and #5

Issues #4 and #5 are:

The application fails when ________ and ________ and ________ and ________
The application fails when ________ and ________ and ________ and ________

After applying all filters for the rest of the errors, you should have a very small number of errors remaining.

This is normal as these two errors are the hardest to hit due to the many input values which cause them.

Finding them first at the beginning is extremely difficult because of that.

However, now that we’ve narrowed down our errors so much, seeing the root cause for these issues is a lot easier.

Take a look at the dashboard:

It seems that all errors are caused by the /payments/execute endpoint. This first part of the error condition is obvious and shared by both issues.

The rest of the values are not so obvious but they are all present in the current view.

We simply have to find out what combination of each value will match issue #4 and, respectively, issue #5.

At this point, we can simply guess until the solution pops out.

Let’s start by filtering the GET method.

The solution to any of the issues isn’t obvious yet:

Let’s continue by filtering the AMEX payment method:

It appears we’ve found one of our issues.

However, there is one problem - it currently appears that the error condition consists of five different inputs, while both issues consist of four inputs.

Now, we have to find the odd one out.

We can do this by removing the filter on any of the parameters on-screen.

After several iterations of trial-and-error, you will figure out that when we remove the filter on the GET method, it can be seen that the error happens on other methods as well:

This leads us to the conclusion that issue #4 is:

The application fails when the endpoint is /payments/execute and the payment method is AMEX and the country is AU and the user type is corporate

Now, exclude one of those parameters, e.g. the AMEX payment method and the final issue is evident:

Issue #5 is:

The application fails when the endpoint is /payments/execute and the payment method is Cash and the country is BG and the user type is trial

Conclusion

Congratulations. 👏👏👏

You’ve successfully completed the tutorial.

You should now have a good practice using Kibana to analyze real issues with your services.

This concludes the series on Integrating your Go service with the ELK Stack. You should now have a decent understanding of:

How to install & setup ELK
How to use structured logging in your services
How to create Kibana dashboards
How to analyse production issues in the Kibana dashboard

Well Done & happy logging. 👌

How to create a Logging Dashboard with Kibana

Preslav Mihaylov — Sun, 09 Aug 2020 07:30:56 +0000

In this tutorial, I'll show you how to create a dashboard for your application's structured logs in Kibana.

This is a follow-up to this article, which covers how to instrument your Go application \w structured logging for use by Kibana (in this tutorial).

We'll use Kibana v7.6 but any version you're using should work. It's just that your UI might look a bit different & you'll have to adjust.

Let's jump straight in! We'll stick to simple panels, which suite most of the use-cases you'd need.

The Starting Point

First, make sure you have docker and docker-compose installed.

For docker, follow the installation instructions for your platform here.
For docker-compose, follow the installation instructions here.

Afterwards, download this exercise’s repository on the kibana-dashboard-tutorial branch:

git clone --branch kibana-dashboard-tutorial https://github.com/preslavmihaylov/tutorials

What you already have is a Go application, which randomly produces logs for endpoints /payments/execute, /payments/list, /payments/authhold.

Starting from here, you will build a dashboard for effectively monitoring & analysing incoming traffic.

In the following section, I will walk you through how to achieve this step by step.

The Walkthrough

Boot up all containers

Invoke this command to bring up all containers:

docker-compose up

For all following steps, make sure you never execute docker-compose down.

That command will not only stop all containers but delete them as well.

If you're seeing some issues with booting up elasticsearch, here is a reference to some common issues & how to handle them:

elasticsearch exit code 78
elasticsearch exit code 137

Setup your application’s baseline saved search

Go to http://localhost:5601/app/kibana#/home and open the Discover tab.

First, expand the tab names in the bottom-left as they are unfamiliar at first:

Then, find the Discover tab in the top-left:

Set the index pattern if you haven’t already:

On the next step, choose @timestamp as a time filter:

Open the Discover tab again. This is what the initial view should look like:

Currently, kibana shows all log lines from our application without any filtered fields. We can exclude all fields we don’t need:

Next, select only the fields which make sense for our application.

Those are endpoint, method, countryISO2, userID, paymentMethod, userType, error, msg.

For each of them, click Add to add them to the selected fields in the visualised log:

In the end, this is what your Selected fields should be:

If you did these steps properly, this is an example of what your logs should look like:

We will use this saved search as a baseline view for our dashboard.

Click Save at the top-left to later reuse this view for our dashboard:

This is one of the most useful views as it allows one to inspect the details of the events which happen in your application.

Setting up your initial Kibana dashboard

Open the Dashboard tab to create your first dashboard:

Follow the instructions on-screen:

After this, you should see an empty dashboard which doesn’t show anything:

Click the Add button at the top-left to add a new visualisation:

You should see the name of the saved search you created in the previous step. Choose that:

Next, resize it as you see fit and you should see the first panel in your dashboard:

This view is great for inspecting the details of all requests & using it for additional filtering on a given property.

However, we will need to add some more visualisations to be able to see aggregations of the data in our dashboard.

Save your new dashboard with a descriptive name and proceed to the next section:

Add a data table to monitor requests per endpoint

In this step, we will create a data table which will show how many requests we get per endpoint.

Click the Edit button at the top-left to edit the contents of the dashboard and select Add to create a new view.

In the panel which pops up, create a new visualisation from the bottom-left:

Next, create a new data table:

On the next screen, select your saved search’s name as the data source to use.

Afterwards, you should see a screen which simply shows the total count of requests in a table. This is because the table currently aggregates all requests & shows the total count for them.

We will aggregate the data in the table by the endpoint keys. This will show the count of each request per endpoint.

To do this, add a new Bucket which splits rows:

As an aggregation, use Terms. This is a simple aggregation which simply groups the data by the unique values for the key we’ll choose:

Aggregate by the endpoint key and use the following configuration:

The Group other values in separate bucket will display all data which is too little and doesn’t fit in the table (the limit is 5 rows, which we specify) in a general “Other” bucket.

For example, if we had 7 different endpoint and five of them get 80% of the traffic, the 20% other values will be aggregated in a row called “Other”.

Now, click the Play button at the top of the window to see how your table looks. You should see the count of requests per endpoint:

This view now looks quite useful. With it, we can track how much traffic we get per endpoint. Click the Save button at the top-left, give the new view a title (e.g. Calls by Endpoints) and rearrange your dashboard to look aesthetically pleasing:

Let’s now do the same for the rest of our significant fields.

Add a data table for all significant fields

In this step, add a new table, following the same steps as the previous one, for method, countryISO2, userType and paymentMethod.

There is no need to include a view for userID as the granularity of that field is typically way too big to make a sensible dashboard.

However, you can usually take that by making custom filters on some of the log lines in the saved search view.

After you go through this, here is what the dashboard should look like:

Now, you can see aggregations for all significant data points you are interested in.

If you want to further inspect e.g. what payment methods one uses when calling the /payments/execute endpoint, you can “zoom in” on the data by filtering for that data.

Try it out and explore for yourself:

Here is a challenge for you, zoom in on the errors and identify which is the endpoint which has the most errors.

The scope of this exercise is creating the dashboard, but don’t worry. You will get to play with it in the following exercise.

Our dashboard already looks quite informative.

But there is one final touch it needs.

Add a histogram for tracking success/error rate

In this step, we will diverge from the good old data table and add a histogram to track the success-to-error ratio over time.

The rest of the views in the dashboard are typically used to find the root cause of an issue.

The view we’ll create now will enable us to quickly identify if there is an issue.

Create a new visualisation, but this time, select a Vertical Bar:

In the beginning, you will simply see a single bar with the total count of requests in the given time period.

For starters, add an X-Axis which will create time intervals for each of the lines in the histogram. This will allow us to see the tendency of events over time.

This time, the selected aggregation is Date histogram on the @timestamp field:

Click the Play button and see how the view changes to reflect the count of events over time:

This is cool, but it would be useful if we could actually see how many of those requests are success and how many are errors.

To do this, let’s customize the existing Y-Axis to only aggregate successful requests.

First, change the Y-Axis aggregation from Count to Sum bucket.

Then, use Filters as the Bucket aggregation. This will allow us to use KQL (Kibana Query Language) to make any custom filter we like.

Add a filter which says select all messages which are not empty strings:

Next, add a custom label “Success” for this Y-Axis at the bottom of the panel.

Click the Play button to see the results. You should now see the number of successful requests every 30s. The default color, however, is a bit off, so change it at the top-right to green:

Now, it looks a lot better:

We aren’t done yet.

It’s now time to stack the errors on this Y-Axis as well.

Collapse the current Y-Axis settings and add a new one:

The setup for this Y-Axis is simpler.

Choose Sum bucket as an aggregation again and Terms as an aggregation for the bucket. Choose errors.keyword as the field to use.

Keep the rest of the settings as-is.

Add a custom label “Errors”, change the color to red and here’s what you get:

This is the success-to-error ratio, aggregated per 30s.

This view will now enable you to see the % of errors for all endpoints with a single glance.

You could also zoom in on the data again and see the % of errors per endpoint/payment method/userType/etc.

Save this view, add it to your dashboard, arrange it nicely at the top and gaze at your beautiful application dashboard:

Finale

Congratulations. 👏👏👏

You’ve successfully completed the tutorial.

You should now have a decent practice using Kibana to create basic dashboards for your application.

The final step of this exercise is to add such a dashboard for your real production application, buy a 146-inch monitor, attach it to the wall in your office, open your app’s dashboard, enter full-screen and enable Auto-refresh.

After you do this, all your colleagues from the other teams will stare at it enviously.

But unless you know how to effectively use Kibana for debugging real production issues, that’s all the value you’ll get from it - a shiny dashboard to show-off with in the office.

In the following tutorial, you’ll learn how to now capitalize on this dashboard and use it to instantly discover the root cause of production issues.

todocheck v0.2.0 is live!

Preslav Mihaylov — Sat, 01 Aug 2020 17:43:56 +0000

The next release of todocheck - the tool that helps you track & keep TODOs actionable is live!

First time you hear about it? - check this out first.

In version 0.2.0, the main focus was extending support to new programming languages & issue trackers.

Hence, there are now five new languages supported - R, PHP, Rust, Swift & Groovy.

Support has been also provided for two new issue trackers - Pivotal Tracker & Redmine.

Additionally, one useful new feature is that todocheck now supports passing in your issue tracker authentication token via an environment variable - this will make it a lot easier to integrate the tool in your CI environment!

Finally, you can now specify todocheck's output to be in JSON format. This provides the opportunity to create IDE plugins or include support for todocheck into linter aggregators.

See the full changelog here & don't forget to update your binary to the latest release!

Eliminate the undocumented TODOs with todocheck

Preslav Mihaylov — Sun, 19 Jul 2020 17:47:16 +0000

Yesterday, I released todocheck - a new kind of static code analyser for annotated TODOs.

Way too often, we let leftover TODOs slip into our main branch, which leaves your coworkers puzzles, looking at it a year from now.

They're thinking - what did I mean by "TODO: Move this to the users package"? What is the users package? It doesn't seem to exist anymore.

todocheck helps you fix this by forcing you to mark all your TODOs against an existing, open issue in your issue tracker.

That way, if you, at some point, close the issue, thinking you're done, the CI pipeline will sparkle in red as there is an open, unaddressed TODO in your main branch.

No longer can developers close a half-baked issue, rushing for the weekly sprint review to say "I'm done!".

How todocheck works

You run todocheck as a standalone binary from the root of your project and it looks for issues with the following format:

// TODO J123: Fix this typo
func fuu() {
    ...
}

In case the linked issue J123 is open, todocheck will not report any error. In case you close the issue or it doesn't exist, todocheck will show an error:

ERROR: Issue is closed.
myproject/main.go:12: // TODO J123: Fix this typo

ERROR: Issue doesn't exist.
myproject/main.go:14: // TODO J321: A non-existent issue

If there is an unannotated TODO in your code base, todocheck will also report it as a malformed TODO:

ERROR: Malformed todo.
myproject/main.go:16: // TODO - This is not a valid annotated todo

Only TODOs with valid, open issues are allowed to exist in the codebase.

By integrating todocheck in your development workflow & CI pipeline, you can ensure that there will be no half-baked issue closed with pending TODOs in the codebase.

Does todocheck work with my language?

You can see the list of supported programming languages here.

There is also support for various public & private issue trackers. See the list here.

But your favorite language/issue tracker is not in the list?

Then feel free to open an issue & describe your needs. Oftentimes, adding support for new languages/issue trackers is a matter of configuration, so you might see your tool in the above lists pretty soon. Here is an example of such a request.

Conclusion

If you want to learn more about included features & configurations, consult the project's README.

Why don't you give todocheck a try right now? Download the latest release & check out the Quickstart section to get started in no-time.

Like the project? Show some love by ⭐-ing it on GitHub.

A Concise Guide to the Latest Go Generics Draft Design

Preslav Mihaylov — Thu, 18 Jun 2020 13:35:23 +0000

Recently, the Go team announced an updated draft design for their Generics in Go proposal. It goes into a lot of details about why certain decisions were made, implementation details, etc.

In this article, my goal is to summarise the major upcoming changes, as the whole draft design can be a mouthful for many.

I will provide some code snippets to demonstrate the major features as well as give you the chance to experiment yourself with them, thanks to the new Go playground with support for generics.

Type Constraints in Generic Functions

Here's how a generic function looks like without any constraints on the type parameter:

Try it out!

To add constraints on the generic type, you can demand that it implements a given interface:

Try it out!

To specify multiple type parameters, separate them by commas:

Try it out!

Constraints on multiple type parameters are written the same way you write function argument types:

Try it out!

And here's how to specify different types for both parameters:

Try it out!

Type Lists & comparable

Instead of constraining types based on a set of methods, you can constraint them based on a set of supported types. For example, you could specify that you accept a generic type which can only be an int or a long.

This allows you to e.g. leverage operators like "less than", "greater than", which are only available for basic types in Go:

Try it out!

You also have an out-of-the-box constraint called comparable, which constraints types to those which support the == and != operators.

Try it out!

Interfaces using these constraints - type lists and/or comparable, cannot be implemented by a struct.

They can only be used for type constraint definitions.

Generic Types

Structs can be defined using a generic type. Once specified, in the type declaration, there is no need to specify the type for all the functions of the type:

Try it out!

You can also do that in interfaces. This is especially useful when a type constraint depends on itself.

E.g. you have a type constraint for T, which requires an Equal method, which accepts a T parameter:

Try it out!

In case you need to specify a type parameter, who has a state-modifying function (e.g. a setter), then you can specify a pointer type constraint:

Try it out!

Notice how this example above requires that you explicitly specify the type you'll use in the function - FromStrings(Settable)...

This is because whenever the type is not present as a function argument, the compiler cannot infer the actual type once the code is compiled. Hence, you need to explicitly specify it.

Conclusion

This article was purposefully succinct and straightforward. Hopefully, it will help you quickly get up-to-date on the latest generics draft design, while you're drinking your morning coffee.

However, there is a lot of rationale behind any single design choice around generics in Go. If you are interested to dive deeper into this subject, check out the official draft design doc.

So, do you like the latest Go Generics draft design?

Let me know in the comments section below.

How to use Structured Logs in your Go Application

Preslav Mihaylov — Sat, 13 Jun 2020 05:23:12 +0000

The Elastic stack (also referred to as ELK) can bring a lot of value to your production services. But it is not that much of value if you don't use structured logs in your services.

In one of my latest posts, I wrote about what ELK is and why you should care. I also wrote a tutorial about how to integrate ELK with your Go app.

In this article, I will walk you through how to integrate structured logging in your Go services. We will use a sample HTTP service with a few basic endpoints and we'll use the zap library to emit logs on error/success, which would also include some domain-specific info.

The Starting Point

Download this tutorial’s repository:

Currently, the repo has a main.go file which has http handlers for three different endpoints:

/payments/execute
/payments/list
/payments/authhold

The handlers themselves don’t do anything special, they simply return a status OK when invoked.

Additionally, you have a logging middleware which you’ll be working in.

Currently, what it does is it wraps the standard http.ResponseWriter into a statusWriter, which records the returned http status code.

We will use it to determine if the http response was successful or not.

Setting up the REST Client

In addition to the Go application, you have a file go-elk-exercise-reqs.json in the branch’s root. It is an exported set of requests you can import in a REST client of your choice.

The one I use, and recommend, is Insomnia. If you want to use that one, simply import the file in your insomnia workspace like this:

Next, choose import data from file & import the file you’re provided:

If everything was successful, you’ll get all necessary HTTP requests imported in you workspace with all necessary data already setup (e.g. headers).

Alternatively, if the import doesn’t work for your rest client, here is what the requests are, so that you can add them manually:

Execute Payment:

List Payments:

Auth Hold:

What We'll be Doing

In this tutorial, we'll complete the loggingMiddleware function by emitting structured logs on success (status code is less than 400) or error (status code is greater than or equal to 400).

The structured logs should contain the following keys:

endpoint - the invoked endpoint
method - the HTTP method used
countryISO2, userID, paymentMethod, userType - domain-specific keys for this application

In case of a success, we'll write “Inbound request succeeded” via logger.Info.

In case of a bad status code, we'll write “Inbound request failed with status [returned status code]” via logger.Error in the default error key.

Here is an example log line we'll emit by the end of the tutorial:

In the following section, I will walk you through how to achieve this step by step.

The Walkthrough

In this section, we will go through the application & implement structured logging step by step.

A noteworthy thing to notice is how we are plugging in logging as a middleware.

This makes the developers writing the domain-specific logic unaware of the implementation details of how their http invocations are logged.

Additionally, it allows us to reuse the logging middleware on each endpoint we want instrumented, rather than copy-pasting the code in each handler.

And finally, this application is kept simple on purpose. In a real application, you might want to include this middleware as a Fx module which you include as part of your baseline Fx modules for each of your microservices.

This gives all your microservices some baseline logging capabilities out of the box.

Let’s get started.

Log on success/failure

We’ll start by simply emitting log messages on each successful or failed http call.

First, create an empty zap.Field array:

We’ll use this array in the later steps to populate it with the specific keys we want to emit in our logs.

Next, use logger.Info and logger.Error in the appropriate spots with their respective success/error messages. Also include all logging fields which are currently empty, but will be filled in in the next steps.

We also encapsulate the error message in a zap.Error field. This will propagate the error message in a separate error field and will also include the stack trace.

Test that everything works correctly by running your application using
go run main.go and sending an http request from your rest client.

If everything went well, here is a sample log line you should see on your screen:

And here’s how an error log should look like:

Test this by returning an http.StatusBadRequest in any of the endpoints and invoking it.

These logs will also be available in logs/go.log.

Let’s move on with adding the rest of the keys we’ll need.

Include the endpoint and method in your logs

To include the invoked endpoint and the HTTP method in the structured logs, you will have to take them from the *http.Request object.

Add the respective fields in the fields array we introduced previously:

Make a similar test as in the previous section and see that your logs now include the endpoint and http method:

Your error logs should also include the new keys.

Include the domain-specific fields in the logs

This step is pretty similar to the previous one and is hence, straightforward.

Here, we extract the headers from the http request and add them to our logs:

After you add this, we should be able to see the log as expected from the beginning of this exercise:

Conclusion

If you got this far, well done!

You should now have a good grasp of what structured logs are and how to use them in your Go services.

If you want to learn more about how to leverage it, check out some of the other articles I've linked at the beginning of the post.

Stay tuned for more upcoming Go/Microservice tutorials!

How to integrate your Go service with ELK

Preslav Mihaylov — Sat, 16 May 2020 07:03:24 +0000

In my last post, I shared how much value the ELK stack could bring for your application in terms of the monitoring capabilities it gives you.

In this post, I will walk you through how to integrate your Go application with ELK, what are the different parts of ELK, how they work and how to create a basic configuration for them.

Let's jump straight in, shall we?

The Starting Point

First, make sure you have docker and docker-compose installed.

For docker, follow the installation instructions for your platform here.
For docker-compose, follow the installation instructions here.

Currently, the repo simply has a main.go file which logs random log lines on certain intervals. Occasionally, it propagates error log lines.

We will use this as a backbone to integrate the go app with ELK.

Setting Up the ELK Stack

ELK stands for Elasticsearch, Logstash and Kibana.

These are three different tools, created by the elastic company, which are typically used together to enable developers & sys admins to monitor production systems based on application or system logs.

There is one additional component the ELK stack, which is Beats.

Beats is a collection of small programs which are fine-tuned to gather different kinds of logging data - e.g. from files, from OS system logs, from Windows event logs, etc.

Since ELBK doesn’t sound as fancy as ELK, we still refer to the stack as the ELK stack.

In the following sections, we will explore each of these components separately & install them accordingly.

What is Elasticsearch?

Elasticsearch is a NoSQL database which is optimized for storing structured documents.

This database also has a REST API in order to easily interact with it.

A structured document is one which structures its metadata into JSON format.

This can be any JSON-serialized object. This can be used, for example, to create a user-facing search bar in your application.

Instead of doing manual & lame string processing of search terms, you can outsource this to elasticsearch which will do a much better job than both of us.

However, in our use-case, we will use Elasticsearch for storing structured logs. A structured log is one, which organizes its contents into a JSON object.

This is an example of an unstructured log:

This is an example of a structured log:

Using a structured log, you can later index into these keys and analyse bulks of your logs fast & seamlessly. This is exactly what the ELK stack provides.

Elasticsearch, in particular, will store this log line in a way optimized for search & analysis later.

Setting up Elasticsearch

To install elasticsearch, first create a docker-compose.yml file in the root of the repo and add this initial yaml:

Let’s break this config down.

In docker-compose, we enumerate the services we want to bring up in this manner:

In this case, we only bring up a single service which is called elasticsearch. You can choose any name you like, but this one is the most descriptive, given what it does.

Let’s now interpret the elasticsearch service config line by line.

This means that we are pulling an already setup docker image with the given version.

This enables us to not bother with setting up an image from scratch. Someone has already provided that out of the box, so we are reusing it.

Otherwise, you will have to bother with setting up the environment, installing dependencies, configuring them, installing elasticsearch manually, etc.

With this line, we are getting all this out of the box.

With this line, we are indicating that the local directory ./config/elasticsearch.yml will be mounted on /usr/share/elasticsearch/… in the container and it will be read-only (indicated by the ro flag).

The designated folder is where the elasticsearch default config directory resides.

The file ./config/elasticsearch.yml is not created yet, but we’ll create it soon.

This part designates that we will make health checks on the container by running the command:

The command will run periodically, every 3s with a 3s timeout.

We will do 10 retries in total and if all fail, docker marks this container as unhealthy and notifies dependent containers.

The final part of the configuration maps the container’s internal 9200 port to our real 9200 port:

This is all we need to add to docker-compose.yml for elasticsearch. Finally, add this configuration in ./config/elasticsearch.yml:

What this configuration means is that we will use single-node discovery mode.

Typically, elasticsearch runs in a cluster of nodes with masters & slaves in them which have to be configured.

To keep things simple & digestible for this exercise, we will use a single elasticsearch node.

The network.host option indicates that elasticsearch can bind to any network interface.

Starting Elasticsearch

To boot up our new container, run docker-compose up.

If all went well, you should start seeing the elasticsearch logs after a while and the container should not show any error lines or anything pointing to an issue:

At this point, have in mind that there might be some issues which might pop-up.

Here is a reference to common issues on elasticsearch startup:

elasticsearch exit code 78
elasticsearch exit code 137

If all went went, you can continue with the next component in our stack - logstash.

What is Logstash?

Logstash is a component which stands between elasticsearch & applications and does several things:

Filtering input documents based on some criteria (e.g. ignore all DEBUG log lines)
Adding extra metadata to the input log lines (e.g. ip, location info, timestamp, etc.)
Sanitizing the log lines if needed (e.g. turning unstructured logs into structured ones)

In reality, you can interact with elasticsearch directly, but logstash is typically used as a helpful intermediary to save yourself some work on the application side.

Setting up Logstash

Add another service to your docker-compose.yml file:

This config section is simpler than the previous one.

First, we use the standard logstash docker image as a baseline.

As before, we are mounting the ./config/logstash.conf file into logstash’s default config location.

Finally, the depends_on section indicates that this service should only boot up when the elasticsearch service is up and healthy.

Otherwise, the container won’t start at all.

Next up, add this logstash configuration in ./config/logstash.conf:

The logstash configuration typically has three sections - input, filter & output.

The input section specifies where logstash accepts data from. It has a built-in plugin called beats, which automatically integrates with beats applications, such as filebeat, which we’ll use later.

Here, we indicate that we’ll accept input from beats applications on port 5044.

In the filter section, you can add any custom filters or mappings (e.g. transforming one input field to another in a custom format).

Here, we only specify that the source structured log will come from the message field, which is the default key where filebeat stores the actual application log.

This will enable us to index the custom application keys in our log, not only the ones plugged in by filebeat and/or logstash.

The final output section specifies where the filtered logs should be forwarded to.

It can by default connect to elasticsearch (as we’ve done here) and integrate with it via its default port 9200.

If you change elasticsearch’s default port to something else, you will have to modify this section as well.

Now run both elasticsearch and logstash and make sure they both boot up without any issues:

If everything goes well, your output should look like this:

What is filebeat?

Filebeat is a beats application (there are more), which is responsible for monitoring changes in log files and forwarding them to logstash.

Without filebeat, you would have to manage your application’s integration with logstash yourself.

The way it works is - you specify a path to your logs which it should monitor & it will handle propagating those logs on regular intervals to logstash.

Setting up filebeat & your Go application

In this section, we will add filebeat & the Go application as docker containers.

Add this to your docker-compose.yml:

The bulk of the new sections are pretty similar to what we already did for the rest of our services.

Both new services depend on elasticsearch being healthy, both depend on their respective baseline docker images.

The differences are in the volumes sections.

For our go app, we’ve mounted the current directory in a subdirectory of the container’s GOPATH.

The logs, on the other hand, have been mounted in both containers.

Our Go application will use that directory to output its logging info.

The filebeat service will use it to monitor changes in the logging info.

In addition to that, we’ve mounted the filebeat configuration in the standard config directory on the container.

For our Go app, we’ve also specified that once the container is setup, we’ll run our application via the command:

Now, add this to config/filebeat.yml:

This configuration specifies that the service will listen for changes to all log files in /logs. Additionally, It will output any new logging info to logstash via port 5044.

We also whitelisted that port in our logstash service config for inputs from beats applications.

What is Kibana?

The final piece of the setup is Kibana. It is a front-end for analysing the data stored in Elasticsearch.

With it, you can visualize a huge array of data via graphs, pie charts, data tables, etc, in order to monitor what’s happening in your application via its logs.

You will see this in action in the final section of this exercise.

Setting up Kibana

Add this final piece of config to your docker-compose.yml file:

There is nothing new in this configuration section.

It installs the baseline docker image, depends on elasticsearch, has a health check of its own and maps its internal port 5601 to the machine’s real port 5601.

In addition to this, you can, optionally, add a dependency of your golang application on kibana to make sure your entire container stack boots up only after Kibana’s server is ready:

Now, test that everything works properly by running docker-compose up.

Now, you should be able to go to http://localhost:5601/app/kibana and see Kibana’s home page load.

If you didn’t do that last optional step, you might have to wait awhile before Kibana boots up:

If you instead get asked for defining an Index Pattern, use logstash-*:

On the next step, select @timestamp as a time filter:

This finishes the Kibana setup.

In the final section, we will test that everything is integrated correctly by searching through our application logs via Kibana.

Search through your logs via Kibana

Open Kibana and expand your tabs first in the bottom-left corner of the site:

Next up, open the Discover tab:

After you open it, you should see all your logs here:

Filter only the application log fields on the left:

And voila, you should be able to see all your application logs:

Conclusion

This wraps up the tutorial to integrate a Go service with the ELK stack.

You should now have a good understanding of how the ELK stack works & each of its different components. Additionally, you should now have a good grasp of how to configure your ELK setup correctly & wire it to your Go application.

From here, you can start playing with Kibana by setting up useful dashboards based on your data.

Watch out for a follow-up tutorial for doing just that!