DEV Community: Peter Strøiman

Getting Started with Neovim

Peter Strøiman — Mon, 05 May 2025 11:33:36 +0000

You are interested in learning vim? Good. What you learn will be useful knowledge forever.

I will guarantee you that once you learn vim, it will affect how you use any editor. Even if you don't use vim; whichever editor you use, you will want to install plugins to emulate vim behaviour.

Vim emulation plugins exist for virtually every text editor, including general writing applications such as Obsidian.

Learn modal editing

Vim's unique approach to editing originates from operating in different modes, separating the activity of writing text from editing text. Once you master this approach, there is no going back; as the editor ceases to get in your way. You don't reach for the mouse to navigate, as your fingers are already close to the keys you need to press.

Vim has a dedicated program vimtutor to teach the fundamentals. Run with the -h argument to which options it has.

❯ vimtutor -h

Or just run vimtutor without arguments to start from the beginning in English.

Configuring vim/neovim into a usable environment can be a mouthful. You can install a "distro" that come provides a curated set of plugins. I will suggest you try one of these:

But now, you are started.

Experimenting with configuration

Feeling adventurous, try both, and compare for yourself.

Disclaimer: I do not know if or how the following tip can be translated to Windows.

By default, neovim loads the configuration from $HOME/.config/nvim, but the last part of the path can be configured with the environment variable NVIM_APPNAME.

So you could install the two configurations in two separate folders and switch between them:

> NVIM_APPNAME=lazy nvim # loads from $HOME/.config/lazy
> NVIM_APPNAME=kick nvim # loads from $HOME/.config/kick

I've recreated my configuration from scratch a few times, and I always start in a new folder, using this approach; keeping my existing configuration that I know is working as a fallback; until I'm happy with the new setup.

Learn From the Old Masters.

I also recommend watching vim-casts by Drew Neil. They are rather old, targeting older versions, so do take the information with a grain of salt. But the principles of vim; buffers, registers, navigation, tab, macros, etc.; all that is still valuable information today; and will help grok fundamentals of vim and neovim.

Use, Then Biuld

When you've used a distro for a year; I suggest you "create" a configuration from scratch, possibly following a video tutorial.

And after you've used that for a year, I suggest you create your own from scratch.

Why You Should Boycott VS Code

Peter Strøiman — Mon, 05 May 2025 09:57:46 +0000

Microsoft's Visual Studio Code, commonly just called VS Code, is a great piece of software. This is in no small part due to all extensions on the extensive marketplace providing extreme flexibility. It has proven it to be adaptable to most, if not all, software development projects.

My editor of choice, neovim, provides no such benefits out of the box, requiring me to configure everything, install language plugins, code completion, myself.

Vim and neovim are by far more powerful editors, but lacking the ease-of-use of VSCode, and easy discoverability of new features; I don't recommend beginners to use these. Configuration is done through code, not a nice UI allowing you to browse a marketplace and take popularity into consideration when choosing between different plugins for the same task.

And while both pieces of software are distributed as open-source software, Microsoft is actively preventing the modification and distribution of the source code under the ethos of open-source.

What is Open-source Software?

For software to be considered "open-source", you must be able to make modifications, distribute the modifications, and run the software or your modified software for any purpose without any restrictions.

The term "open-source" is predated by "free software"

The FSF (free software foundation)'s goal was to promote the development and use of free software, which they defined as software that grants users the freedom to run, study, share, and modify the code.¹

"Free software" had a unclear meaning however, as "free" is unclear. It was free as in "free speech", not "free beer". "Open-source" was coined instead to avoid confusion.

While Richard Stallman and FSF didn't adopt this, as the term might indicate you only had the ability to inspect software, the Open Source Initiative still governs the core principles; that open-source software can be modified and shared, regardless of its purpose.

OSI have reviewed many licenses. They have have published a list of approved licenses that meet the criteria of open-source.

Distributing Source Code is not Open-source.

MongoDB is a database engine by MongoDB Inc.² who provides hosted MongoDB databases as a service. The source code for MongoDB is available under the conditions of their own proprietary SSPL license. The license provides a lot of flexibility in how you can use the software; you can even use it with no licensing fees to power your own web applications. But if you offer MongoDB as a service; the license impose some extreme requirements.

MongoDB's approach is IMHO reasonable and fair. They are basically making the software freely available to anyone who is not a direct competitor to their own hosted service.

But it's not open-source, as you do not have the freedom to modify and run the code without restrictions. The license was rejected as an open-source license by OSI, Red Hat, and Debian.

How Visual Studio Code Violates the Ethos

Visual Studio Code is distributed under the MIT license, a permissive license that allows you to do anything with the source code. This license is recognised by the OSI as an open-source license. And quite predictably, a few forks of VSCode exists. VSCodium seems to be the most popular.

This was all perfectly well, if it wasn't because of the barriers Microsoft impose on derived works ...

VSCode has little value without any extensions. The VSCode marketplace has an almost infinite list of extensions, so almost no matter which tool or language you use, you will find an extension to provide support. If you want a new programming language to succeed, it's almost vital that you have proper VSCode integration.

Microsoft also provides VSCode extensions as well. And this is where the problem lies, as you are not allowed to use these extensions in any a fork of VSCode. Two popular extensions provided by Microsoft are pylance and C/C++ for Visual Studio Code. Both of these contain the following condition in their license:

You may install and use any number of copies of the software only with Microsoft Visual Studio, Visual Studio for Mac, Visual Studio Code, Azure DevOps, Team Foundation Server, and successor Microsoft products and services to develop and test your applications.

The "Open-source Theatre"

I call this the "Open-source Theatre", somewhat inspired by the "Security Theater"

Security theater is the practice of implementing security measures that are considered to provide the feeling of improved security while doing little or nothing to achieve it.³

So while Microsoft distributes VSCode itself as an open-source product, providing a feeling of commitment to open-source, they are not just "doing little" to achieve it, they are actively fighting against this. The ethos dictates that you should be able to make modifications and run without restrictions, but they are actively placing barriers on those modifications.

I would call this a legal loophole, as they managed to place open-source software in a context where they can violate the principles of open-source in the surrounding context.

The Value of VSCode is it's Community.

As I mentioned, I think VSCode is a good product, particularly as your first editor.

The value of VSCode comes from the ecosystem of extensions. While some are developed by Microsoft, the majority are developed by individuals or small teams made who provided their solutions as free and open-source software. In effect, a large number of individual developers are increasing the value of VSCode, often in their spare time; but under the assumption that they are contributing to an open-source community.

This isn't just a theoretical case, of some outdated licenses they forgot to update. Microsoft are actively enforcing the restriction, as this issue from the vscodium project shows. The plugin used to work from VSCodium, but Microsoft has actively taken measures to prevent this.

To be fair

While researching for this article, I notice that the restrictions is in the license for the compiled .vsix file distributed to the VSCode marketplace. Their source code have more permissive licenses. Maybe you could compile the extensions your self and use them? But the situation isn't clear from what I learned.

pylance has a license I think should be fine, but it's not an OSI approved license.⁴ At first, things looked better for the C/C++ extension, as it uses the OSI approved MIT license. But it also includes a notice that the code depends on runtime files "built and distributed by Microsoft; these are governed by the more restrictive proprietary license".

I did not research whether you could legally use these plugins if you were to compile them yourself. So these two plugin could be possible, but requiring extra work.

It doesn't change the fact that Microsoft are taking steps to actively prevent using the compiled version distributed to the VSCode marketplace, thus actively imposing barriers for VSCode forks.

Conclusion

The revolution of open-source software is by far the most important change in the software industry I've witnessed during my career. When I started, software development required the purchase of tools, compilers and component libraries. Bugs in libraries meant bugs in your own code; with no ability to fix them yourself.

Open source has democratised software development. Large companies like Netflix, Meta, Google, have distributed tools as free and open-source software.

And the linux kernel is a massive open-source project. It's safe to say that the internet runs on Linux; and Google is also making a fair bit of profit from 2.5 billion active Android devices.

Even VSCode is build on the open-source Electron framework, created by Github (later acquired by Microsoft) as a foundation for their Atom editor. And Electron is based on Chromium, the open-source core powering Google Chrome.

I don't argue that companies are obligated to distributing their products as open-source. Not at all. One of my favourite pieces of software, Obsidian is closed source, and I have no objection to that. They as a company need to make a profit, and they are free to chose their own strategy.⁵

But I do expect when a company release a product as open-source, they support the ability to modify and distribute, rather than actively imposing barriers for derived works, while using this legal loophole to justify an "open-source" label.

Open-source is the foundation for a free internet.⁶ Open-source is important. Open-source is worth fighting for.

But in the land of open-source software, VSCode is a wolf in sheep's clothing. It should be exposed for what it is.

If you are using VSCode, I encourage you to try something new. See if VSCodium works for you. Or if you are ready to move on to vim or neovim (see my getting started guide for some batteries-included options). Or try emacs.⁷

Hell, even check out JetBrains' products and see if they work for you. JetBrains have build development tools for decades, and have an insane amount of experience in that field. Unfortunately, their tools are closed-source⁸ paid products.⁹

But JetBrains don't pretend to be what they are not.

https://en.wikipedia.org/wiki/Open_source#Open_source_as_a_term ↩
https://en.wikipedia.org/wiki/MongoDB ↩
https://en.wikipedia.org/wiki/Security_theater ↩
Pylance source code is distributed under the Creative Commons Attribute 4.0 International license. Creative Commons is normally used for other creative arts, such as images, graphics, audio, etc. I've never seen it applied to software before. AFAIK it does follow the ethos of open-source software. According to my research is not an OSI approved license. But that could simply be because it was never evaluated for that purpose. ↩
Obsidian is a safe investment because data is stored in an open format, markdown files. If at any point in the future I should want to abandon Obsidian, while I ca't rely on a fork suited for my purposes, I can still access my notes; using a plain text editor; like vim. ↩
"Free" as in "free speech". An internet free as in "free beer" is very unlikely to support free speech. But if it does, it would be due to open-source projects. ↩
Emacs seem to have decreased significantly in popularity. Many of the advantages emacs previously had over vim are easily achievable in neovim today. ↩
JetBrains have a staggering amount of open-source projects, including the Kotlin programming language. But to the best of my knowledge, their IDEs are closed source. ↩
There appears to be free plans for non-commercial use. ↩

Using Go maps as a simple Fake repository

Peter Strøiman — Mon, 21 Apr 2025 09:28:51 +0000

This article explores how you can take advantage of Go's interface mechanics to create very simple stubs for testing.

The example here is a piece of code that sends a "password confirmation email" to a user. The implementation needs to look up an account, to generate the correct email.

Sending an account validation email

The actual process doesn't run in the same transaction as the original user request; it runs as a reaction to a domain event, that a user has registered with the site. The event itself doesn't contain any user information, only an AccountID; as well as a generic EventID. To generate and send the email, the implementation needs to get the account with that ID.

You could imagine a simple type depending on two interface types, one for a repository to load the account, and one for sending an email:

type EmailValidator struct {
    AccountRepository
    Mailer
}

func (v EmailValidator) ProcessEvent(DomainEvent) error {
  // ...
}

The AccountRepository might accommodate many different purposes, and have a range of methods for those purposes.

type AccountRepository interface {
    GetAccount(id AccountID) (Account, error)
    Insert(Account) error
    Update(Account) error
    FindByEmail(string) (account Account, found bool, err error)
}

But for the purpose of the EmailValidator, this interface is bloated. The EmailValidator only needs the first method, GetAccount. This design violates the Interface Segregation Principle; the EmailValidator is forced to depend on methods it doesn't need.

Narrowing down the interface

The code base will have an implementation of an AccountRepository somewhere, providing the capabilities of finding, and updating Account information, bringing cohesion to serialising/deserialising account entities. But the interface we depend on need not specify all the capabilities of the implementation. Multiple interfaces can exist, and the same implementation can satisfy them all.

When I first started writing Go, it was after 15 years of C# experience. For the components that are wired up through an IoC container, it was a common approach to have an 1-to-1 relationship between classes and interfaces, making the interface effective describe the capabilities of a component in the system.

But we can invert this principle, and let the interface describe the dependencies of a component. So instead I create the AccountLoader interface; describing what the EmailValidator depends on.

package registration

type AccountLoader interface {
    GetAccount(AccountID) (Account, error)
}

type EmailValidator struct {
    AccountLoader
    Mailer
}

By having the interface next to the EmailValidator this further enhances the relationship that this interface describes the dependencies of the validator component.

Go's type system makes this approach viable, as types do not need to know about the interfaces they implement. C#, Java, or C++ require that class definitions explicitly state which interfaces they support. While the same granularity is possible, language designs of these languages impose constraints to code structure, and it's not an approach I've witnessed during my 15 years of experience with C#.

In reality, Go is a compiled language that supports Duck Typing on interface values.

So this change didn't break any implementation. Any previous account repository implementation is still a valid slot to fill in as a dependency to the validator type.

Creating a fake repository for testing

A common practice for testing in languages like C# is to use a mocking library to automatically generate programmable mocks to inject as dependencies.

With single-method interfaces, a hand-written stub or fake becomes a much more viable solution. For the AccountLoader, a simple map type is enough when it just has to find an entity by ID.

type fakeRepo map[AccountID]Account

func (r fakeRepo) GetAccount(id AccountID) (Account, error) {
    var err error
    res, found := r[id]
    if !found {
        err = ErrNotFound
    }
    return res, err
}

This implementation demonstrates another property, that any type can have methods, meaning any type can be a valid implementation of an interface.

Creating a helper to convert a found bool value to an error can reduce the code further:

type fakeRepo map[AccountID]Account

func (r fakeRepo) GetAccount(id AccountID) (Account, error) {
    res, found := r[id]
    return res, foundToError(found)
}

func foundToError(found bool) error {
    if found {
        return nil
    } else {
        return ErrNotFound
    }
}

The fake repository is now just 5 lines of code. While the abstraction through the foundToError helper may at first make the code less readable; if this is a helper function used ubiquitously in test code, it becomes part of the mental model, making the reduced version just as readable as the first implementation.

Using the fake repository

Writing a new test using the fake repository is quite easy. Using map as the underlying type allows it to be constructed using a map-literal.

func TestSendEmailValidationChallenge(t *testing.T) {
    // InitAccount is assumed create a valid initialized account, i.e. a non-zero ID
    acc := InitAccount() 
    event := acc.GenerateEmailValidationRequestEvent()

    mailer := FakeMailer{}
    validator := EmailValidator{
        Repository: repo{acc.ID: acc},
        Mailer:     mailer,
    }
    assert.NoError(t, validator.ProcessEvent(event))
    mailer.AssertOutgoingEmailTo(t, acc.Email)
}

Creating an initialising the fake repository is just a one-liner embedded in the EmailValidator struct literal.

This makes the test code simpler than any implementation using a programmable mock which would that you first create the mock, then program expectations, before it can be used in a dependency.

Conclusion

Go's interfaces provides great flexibility because of two features, separating it from most compiled languages:

Types do not need to know about the interfaces they implement
Any type can have methods and implement an interface.

This makes it a viable approach to create small interfaces that describe the dependencies of a component, rather than the capabilities.

You can easily create simple implementations for testing, often making the tests simpler than using a programmable mocking framework (they exist for Go as well, but not as used).

Furthermore, as the hand written implementations implement an interface describing a dependency of the component under test; so they are coupled to the components they are used to tests, strengthening cohesion in test code.

The Beauty of Go, Introduction

Peter Strøiman — Mon, 21 Apr 2025 09:28:17 +0000

This series is about Go, a simple, yet powerful, language that has some unique features in its design.

Go has some unique features, and is together with OCaml the only compiled language I've used that supports Duck Typing, i.e., the ability for a type to conform to an "interface" without the type definition explicitly stating this.

In this series, I share some of the interesting aspects of Go, and share some solutions that take advantage of the capabilities of the Go language.

How TDD reduces cost by reducing waste

Peter Strøiman — Wed, 16 Apr 2025 12:23:59 +0000

For more than half my career, I've been a practitioner of TDD, Test-driven development. The practice of writing test code before writing production code. I strongly believe that it is by far the most important skill any developer can learn.

Unfortunately, TDD is often poorly understood, even by many who practice it; yet, poorly practiced TDD will still often be beneficial. But a challenge developers can face is that management may perceive TDD as increased cost, when the exact opposite is true; TDD reduces cost because it reduces waste.

Waste

Agile software development is sometimes compared to Lean Manufacturing, which is highly influenced by the Toyota Production System. In the 1988 booklet published by Toyota they describe it as:

The TPS is a framework for conserving resources by eliminating waste.

Examples of waste in a production facility could be excessive inventory, overproduction, moving or transporting parts or equipment, underutilised workers, idle time, relearning, and obviously, defective products.

While civil engineering, and industrial production, are both poor analogies of software development, the concept of "waste" is comparable, although the sources of waste are different.

Some examples of waste in software development are are: long running abandoned branches (unproductive paths), merge conflict resolution, knowledge loss through handoffs, unnecessary complexity, and obviously, product defects (bugs).

Waste means added costs, and reduction of waste translates directly to reduced cost.

How TDD reduces waste

The type of waste that is commonly understood to be reduced by a good test strategy is is product defects/bugs. Thorough testing may detect defects, but TDD can prevent defects; i.e., many defects would never be committed to source control in the first place. A good TDD process would be able to cover the majority of the required behaviour. TDD itself is not a complete solution to this problem, and there are tests worth writing after code was written.¹

But defects is not the only source of waste reduced by TDD.

Truncating an unproductive path

As developers, we are continuously making assumptions, often not realising it. We assume that we can use component or technique X to implement functionality Y. We assume that the path we take will solve the problem. We may write many lines of code assuming it has the intended effect.

Occasionally, the assumption is flawed, and we must adjust the current work-in-progress accordingly. Occasionally, the assumption is flat-out invalid. Pursuing X was so flawed to begin with that all work based on the assumption is useless for the purpose of Y. The more work we have carried out, the more waste produced by following an unproductive path.

The way to reduce unproductive paths is to validate the assumption as early as possible. If the assumption is technical in nature, executing snippets of code continuously is often the most efficient way to validate them.

When the developer uses TDD they will quickly work towards a simple working solution using X, initially with a lot of aspects uncovered, and even an incomplete tests. TDD effectively validates the design.

Reduce time debugging

At the time of writing this, I have not launched a debugger for 4 years! A debugger is a tool you may use when your code when the code does not have the intended effect, and you can't immediately identify why.

When working in small increments, running tests early, maybe even after one line of change at a time, you get the immediate feedback if a change had the intended effect.

In poorly understood part about TDD is that TDD is a feedback tool. When practicing TDD, the first iteration of a "test" may be nothing like a test at all; the test itself will change while developing a feature and will eventually mature into a "test" describing describe desired behaviour. Just as the production code evolves slowly, so may can test code.

The granularity of the process can vary greatly. When taking the first step into a new area of the system, a very granular approach is likely. The more confident you are with the way forward, the greater the step you might take.

But the role of the test is to describes the next step you intend take towards having the desired behaviour, and have automated feedback that your work had the intended effect. And as issues are revealed early, debugging is rarely necessary, as the issue is easy to identify.

Decrease unnecessary complexity

TDD can help avoid unnecessary complexity. How effectively depends on how well the test suite facilitate refactoring. Developers or teams new to TDD have a tendency to write test suites more closely coupled to implementation details, making the test suite resist refactoring. As developers gain more TDD experience, their test suites tend to describe the system at a higher level of abstraction, reflecting system behaviour rather than design decisions.

The test suite now serves as a safety harness facilitating refactoring. Developers who fully embrace this will happily commit a simple naïve solution to the problem, defering complex design decisions and refactoring to patterns until new behaviour dictate.

Often, the elegant solution doesn't present itself immediately, and by being able to defer the design decision; we have bought ourselves time, increasing the likelyhood of finding a simple solution to before the features demand it.

Note: Behaviour does not necessarily mean business rules. Behaviour can be technical in nature. Access token renewal is a behaviour of the system. It's not a behaviour domain experts may care about, but they may care about how fault-tolerant the system is overall; making developers add new behaviour such as retrying failed attempts using an exponential backoff mechanism. Writing tests for this in a way that can describe interaction with an external system as well as the passing or time is almost essential.² Manual testing access token expiration against a real identity provide is almost guaranteed to be flaky if you need to wait 2 hours for access tokens to expire.³

Conclusion

TDD is often misunderstood. The word "test" is somewhat misleading, as TDD is much more than testing.

TDD is a process that incorporates instant feedback into the daily work of the developer. This helps reduce unproductive paths as a source of waste while coding.

An outcome of an effective TDD process is a test suite that helps detect regression of behaviour, reducing defects as a source of waste; while facilitating refactoring.

This permits developers to choose the simplest possible solution initially, having confidence they can add complexity when changing requirements dictate it, reducing excessive complexity as a source of waste.

Effective TDD is a skill that needs to be honed like any other skill. Once mastered, teams practicing TDD are able to reduce cost due to the reduction of waste.

Appendix: Examples

This are two real examples from code bases I've worked on where a very granular process helped build the system, and solve one problem at a time. And in both cases, fast feedback was essential to moving forward quickly.

Discovering the wrong assumption with GORM.

I was new to GORM, a micro-ORM for Go. I started by creating a very simple database schema by hand, a single table with a UUID primary key column. I created an entity type in code where the ID was initially represented by a string, partly because Go doesn't natively have a UUID datatype.

To see if I could insert and read, I created a roundtrip test, a test that inserts the entity, then reads it again and verifies that the new copy is identical to the original. I wrote the following function to read an Entity:

func Get(id string) (entity Entity, err error) {
    err = db.Take(&entity, id).Error
    return
}

But the function didn't work! The function returned an error, not an entity. After reading the documentation, I learned that when using a string identifier, a different syntax is necessary:

func Get(id string) (entity Entity, err error) {
    err = db.Take(&entity, "id=?", id).Error
    return
}

And this worked.

Because I created a test as a means of fast feedback, and started with just basic static functions and a hardcoded connection, I was able to identify my incorrect assumption immediately; significantly reducing the debugging process, compared to detecting the issue in the context of a more complex business rule implementation; and having to figure out where the problem lies.

Over time I would rearrange code, grouping functionality into a dedicated repository with configurable connection options. But code structure was a different problem, and as I was venturing into unknown territory, I used a very granular approach and a tight feedback cycle.

I first solved the problem of writing the actual lines of code that can read and write data; before addressing the problem of how to organise code in a larger codebase.

First time working with RabbitMQ

The first time I had to work with RabbitMQ I had a similar problem. There were plenty of unknowns. How do I even connect? What is the valid connection string? How do I segregate data?

The first couple of iterations involved visual inspection in the admin UI. For example, I tried to send a message to a queue. Messages should not be sent directly to queues, but exchanges, but that was a different problems to solve. The first queue was created manually in the rabbit admin UI. I wrote the code to send a message to the queue, and saved, letting the test framework run the code automatically. Then I inspected manually in the admin UI to see that the message would appear.

After dealing with message publishing issues, I could then start to write code that would consume messages from a queue. The test would now send a message, and wait for one to appear. Next, I could add a verification step that the received message had an identical message body. I now had something that started to look like a test.

Now I would manually delete the queue I had created, and the test would of course fail. I wrote the code to create missing queues, and the test would verify that the queue was created. Running the test again would verify that creating a missing queue would fail when it wasn't missing.

At that point in time, I no longer needed the admin UI. It served as the best way to get feedback in the very early iterations, when I didn't have the code to read messages, and write both the publishing and consumption code in one sitting before working on the feedback would have required significantly more debugging time.

Up to this point in time, all code existed only in a test file. I had dealt with the problem of how to write the lines of code that can actually interact with the RabbitMQ server. But the test provided feedback measured in milliseconds, and now I could rearrange the code into a Publisher and a Subscriber component in the system. The tests would at the same time be reorganised to describe publishing messages through the Publisher and consuming them through the Consumer, and every time I saved, a message would pass through RabbitMQ, confirming that the refactored code worked.

Finally, I could address the problem at a higher level of abstraction. How domain events in one area of the system will eventually trigger another process to happen in another part of the system. For the most part, domain logic would be tested using a mocked Publisher, verifying domain logic alone.

Messages were still passed through RabbitMQ as part of the test suite. The actual message routing was controlled by RabbitMQ configuration, so if this wasn't tested, an essential part of the system behaviour would go untested.

Domain event publishing and handling worked reliably, and I never had to debug the code.

To this day, whenever I add RabbitMQ behaviour to a project, I follow the same path.

I will typically write TDD tests covering application and business logic as a whole, HTTP endpoints as a whole (described using HTTP headers, body, status codes, and mocking application logic), and data storage as a whole (using a real database). This leaves at least one type of test to add after code was written. Do all layers connect properly? E.g., do IOC containers wire up dependencies correctly? Can a user complete a basic flow from login to checkout in a complete system? ↩
Libraries like lolex for JavaScript, and the experimental synctest package in Go 1.24 allow tests to simulate the passing of time; verifying behaviour without actually having to wait. ↩
This is the standard access token expiration for one major identity provider. ↩

How Gost-DOM avoids making HTTP calls

Peter Strøiman — Tue, 18 Feb 2025 12:53:10 +0000

This article is about the implementation of Gost-DOM, the headless browser written in Go.

Web applications written in Go are very easy to test. An web application serves HTTP requests from a single function, ServeHTTP. Test code can just call this function to test the web application behaviour, but still express the test in terms of http requests, responses, headers, body, and status codes; rather than controller method calls.

I wanted Gost-DOM to take advantage of this for two reasons:

Performance - The test is just calling Go code, avoiding the overhead of a TCP stack
Isolation - By eliminating the need to manage TCP ports, it becomes much easier to run tests in isolation.

Part 2 is really the most important part here. Although there is nothing preventing tests from running in isolation; the management of TCP ports increases complexity of the test setup.

The HTTP client

I wanted to build this in a way that makes the core of the browser unaware of how HTTP requests are handled. Fortunately, Go's excellent standard library has the right tools out of the box.

Outgoing HTTP requests are handled by an http.Client instance; which abstracts the transport layer through the RoundTripper interface.

type Client struct {
    // Transport specifies the mechanism by which individual
    // HTTP requests are made.
    // If nil, DefaultTransport is used.
    Transport RoundTripper

  // ...
}

A great feature of Go is that many essential operations are abstracted by single-method interfaces. The RoundTripper has just one method:

type RoundTripper interface {
    RoundTrip(*Request) (*Response, error)
}

This means, I just need to create a type implementing the RoundTripper interface, that calls ServeHTTP.

Implementing the interface

The http handler function receives an http.ResponseWriter. This is not a concrete type, but an interface.

In addition to promoting a testable design of your production code, Go also supports great test tools in the net/http/httptest package. The ResponseRecorder is a valid ResponseWriter that test code can pass to the implementation, and simplifies dealing with response body streams.

But the ResponseRecorder is also so kind to actually generate a proper *http.Response, that can be retrieved by calling, ResponseRecorder.Result().

The type representing the request is the same in the RoundTripper and the Handler, so I made a copy, to avoid mutation in the tested server code affects the request object generated in the browser. The first version of the TestRoundtripper looked like this:

// A TestRoundTripper is an implementation of the [http.RoundTripper] interface
// that communicates directly with an [http.Handler] instance.
type TestRoundTripper struct{ http.Handler }

func (h TestRoundTripper) RoundTrip(req *http.Request) (*http.Response, error) {
    rec := httptest.NewRecorder()
    // Make a copy, so the http handler doesn't mutate the outgoing request
    reqCopy := new(http.Request)
    *reqCopy = *req 
    h.ServeHTTP(rec, reqCopy)
    return rec.Result(), nil
}

This worked fine in the beginning, but had some issues.

A slightly odd design decision

In the midst of the very well designed API is what I find to be a somewhat odd design decision.

The *Request passed to the RoundTripper, and the *Request passed to your http Handler is the same type, but they are really two different things. The one is an outgoing request, and the other is an incoming request.

While they share are similarities, they are not the entirely the same, and the type has properties that are only relevant for processing incoming requests, such as decoding form data. And there have different rules determining if a request is valid.

First, the outgoing request is allowed to have a nil request body, where the incoming request always have a body.

Then, the incoming request is a source of a context.Context, which I didn't initialise either.

Fixing the request was easy:

func (h TestRoundTripper) RoundTrip(req *http.Request) (*http.Response, error) {
    rec := httptest.NewRecorder()
    body := req.Body
    if body == nil {
        body = nullReader{}
    }
    clientReq, err := http.NewRequest(req.Method, req.URL.String(), req.Body)
    if err != nil {
        return nil, err
    }
    clientReq.Header = req.Header
    clientReq.Trailer = req.Trailer
    h.ServeHTTP(rec, clientReq)
    return rec.Result(), nil
}

And now, the browser itself will create HTTP requests using an instance of the http.Client provided. Test code can control the client, replacing default HTTP request behaviour with one that calls your HTTP handler, and an outgoing request from the browser is now just a simple function call.

Adding cookie support

Go also provides a cookie jar. So adding cookie support was so simple that giving it a headling was completely overkill.

The function that constructs an http.Client communicating with a simple handler is as simple as:

import (
    "net/http"
    "net/http/cookiejar"
)

func NewHttpClientFromHandler(handler http.Handler) http.Client {
    cookiejar, err := cookiejar.New(nil)
    if err != nil {
        panic(err)
    }
    return http.Client{
        Transport: TestRoundTripper{Handler: handler},
        Jar:       cookiejar,
    }
}

Testing identity provider integration

While this is not a feature yet, it is intended to extend this functionality to support multiple HTTP handlers, simulating different host names.

This could be valuable in testing an OAuth authentication flow, or sign in using external identity providers. You could create a test HTTP handler simulating the behaviour of an identity provider, and test the flow independently of the external provider.

In my previous experience, this is often accomplished with a real identity provider, configured with some test users. But this approach has some drawbacks:

The tests may fail due to a temporary outage of an external service (in the worst case, preventing deployeing a critical)
Tests may fail due to account lockout.
Using "real test users" makes test code depend on an external context, i.e., there are factors affecting the outcome of the test that is not specified in the test.
Developers may not have privileges to administer test users, preventing them from writing the right test, waiting for an administrator to add new users.

By mocking an IDP, you gain full control of test environment flow.¹ Then you'd need to setup the round tripper to support two hostnames:

https://app.example.com/ Calls your application http handler
https://idp.example.com/ Calls your mock identity provider web app.

And you'd still be able to run everything in parallel.

Intrigued? Check out Gost-DOM. And stay tuned for more nitty-gritty details about how I build a headless browser in Go.

And please, spread the word. 🙏

I want to emphasise that this tool is meant to support the majority of tests written to support a TDD flow. That doesn't mean there shouldn't be added more tests after the fact to find critical integration issues, and I certainly recommend having ONE automated tests exercise the login flow if your application integrates with an external identity provider. But I don't want that to be part of the normal development flow. ↩

How Gost-DOM is implemented

Peter Strøiman — Tue, 18 Feb 2025 12:52:31 +0000

Gost-DOM is headless browser written in Go. It is written with the purpose of supporting a TDD feedback loop while building web applications in Go.

This article is a short introduction to the the series about the technical solutions. Future articles will go more into details of specific solutions.

It started as an experiment, but as I progressed I strongly felt that this could become an extremely valuable tool for web development in Go.

On February 6th, I released version 0.1, signifying that the project had reached a level that it could fulfil minimal use cases. But as the versioning scheme indicates:

This is a very early prototype
The API may not be stable. I believe I have found a reasonably good structure, but I am still learning.

In general, the browser now supports minimal HTMX apps, e.g., clicking elements with HTMX handlers, including boosted links, that update history, form submission using HTMX with content swapping.

Gost was specifically written with HTMX in mind, and it features a V8 engine for JavaScript support. Work is in progress to also support Goja, a pure Go JavaScript engine. Gost has a cookie support, and Gost can bypass the TCP stack for extremely fast test execution. It supports full isolation of test cases with no fuzz, and parallel tests as a result (depending only on your code). The goal is "ludicrous speed" for web application TDD feedback loop.

In future articles, I will dive in to specific aspects and solution I have applied during the implementation of Gost-DOM.

But I invite you to try out Gost-DOM, and spread the word.

I created a headless browser in Go. Here's what I learned

Peter Strøiman — Sat, 08 Feb 2025 08:18:33 +0000

I have more then 20 years of web development experience, both using server-side rendering, and React. React is a great framework for advanced UI; but it does bring a lot of complexity compared to SSR.

When I watched The Primeagen's introduction to HTMX, I had a eureka moment. A tool that enables the same smooth UX as React for the most parts; but with the simplicity of SSR. And when HTMX doesn't cut it, and a hybrid is a very plausible solution. PWAs are not possible, but who really need that? (who are willing to pay for the complexity)

I started learning HTMX, but as a long time TDD practitioner, I wanted to write tests to provide fast feedback helping drive implementation of behaviour.

Testing HTTP servers in Go is extremely easy, as an HTTP server is just a function; easily callable from test code. This makes testing the HTTP layer like testing any other part of the Go codebase, supporting mocking of dependencies as necessary.

But request/response verification of an HTMX test is tightly coupled to implementation details instead of behaviour and responsibility. The application behaviour is a result of a choreography of attributes in the HTML, backend route handlers, and the responses headers and response bodies intended trigger a specific behaviour in the browser.

I searched for what people used to test HTMX applications, and the answer was: browser automation, mostly playwright.

Such a tool has so much overhead that it discourages a TDD process. In my mind, there is only one way, a headless browser with an interface in the same language as your backend code, allowing tests to mock dependencies. And there wasn't one for Go.

So I decided to build my one: Gost-DOM.

Eliminating the unknowns

To get something like this working, I knew that there were two major uncertainties I had to address.

Parse HTML into a DOM
Execute JavaScript

I started exploring both options, and after just two days, I had an early prototype that was able to parse an HTML page with a <script> tag, and execute it.

It gave me confidence to believe that I could pull it off. But I hadn't fully grasped the scope of the project.

Parsing HTML

I'm not foreign to language parsers, generating an AST from source code. But HTML parsing has a unique twist. You cannot just exist with a non-zero exit code if the HTML is malformed. It must generate a DOM; and it's clearly specified how.

I decided to defer the problem of malformed content for the first prototype, and I quickly had a prototype that could parse a simple document into a DOM. I started implementing specific rules, such as executing scripts when they are mounted. Well, that's about it at the time.

While I was getting feedback on the actual DOM implementation in the golang subreddit, a friendly mentioned I might want to consider using the x/net/html package. This is a an HTML parser that already handles all the weird rules for malformed HTML.

While the types themselves couldn't work as the DOM in my scenario, and the parser didn't handle the DOM tree construction rules, it did handle the HTML parsing rule, including dealing with malformed HTML.

I decided to discard my own parser, and parse HTML into the x/net/html structure, then iterate that tree to build my own.

This solution has limitations, such as document.write doesn't work; the input stream has already been consumed when it's executed. But the solution is perfect for the first version of the tool. The priority is to support testing modern web applications, and the solution allowed me to quickly turn address other problems.

Embedding and extending V8

As fun as it would be to write a JavaScript engine, I would never have gotten to the point I am now, had I written my own JavaScript engine. But Go support linking to C code, so embedding V8 was a definite possibility, and I'm confident it has the features needed in a browser context.

I searching for anything about embedding V8 into Go, and I found more than I was looking for. A project already existed, exposing V8 to Go, a project called v8go. This hadn't been maintained for some time, but tommie's fork had not just been kept up-to-date with latest v8 versions. It automatically pulls the latest version from chromium sources!

I added tommie's version to my project, but soon learned, that many essential features of V8 wasn't exposed in to Go code. When the DOM itself is implemented in Go, I need objects in V8 to wrap Go objects, and build the prototype hierarchy in Go. All essential V8 features for this use case was not available in v8go.

I have added these in my own fork. Most of my changes have been merged to tommie's fork now, a few still only exist in my fork.

Learning CGo

To add the functionality, I needed to learn the V8 API, as well as CGo, the part of Go that allows calling C code, and calling back from C code.

As Rogchap, the original author of v8go, had already creates the framework, I could start with little knowledge, and copy/paste his code, adapting to the new V8 API functions that needed to be exposed. But the header files had a weird structure, which I realised was due to another problem.

Go and C++ is like oil and water

Go cannot call C++ code, just C code. Rogchap had implemented a solution using preprocessor directives. When header files are compiled as part of the Go compilation, it sees C compatible code, and when they are part of the C++ compilation, it appears as C++ classes.

The Go types only store pointers, so the types the the Go compiler see doesn't need to be complete. Go just need to know that they are pointer types. The following snippet of header file shows the general approach:

#ifdef __cplusplus
namespace v8 {
// Forward declaration, the header file only uses pointers to this, so 
// the header file doesn't actually _need_ to include v8's "isolate.h"
class Isolate;
}
// Define a v8Isolate type, which C++ code sees as the v8 Isolate class
typedef v8::Isolate v8Isolate;

extern "C" { // The following functions should have C-calling conventions.
#else
// For C code, v8Isolate is defined as a struct
typedef struct v8Isolate v8Isolate;
#endif

// There is a C-compatible function, NewIsolate() which returns a pointer to
// a V8 Isolate. C++ compilation will know what it is, the v8::Isolate class.
// Go code will just know that it's a pointer, and that's all that Go code
// needs to know. It's still a nominal type, so Go will still check that you 
// use the right types.
extern v8Isolate* NewIsolate();
#ifdef __cplusplus
}  // extern "C"
#endif

I can safely say, if Rogchap hadn't created this, my project wouldn't have had a V8 engine. I have written C++ and assembler code, so I understand the fundamentals of C compilation and linking, as well as C++ scoping rules, and the use of smart pointers for resource management; which V8 relies heavily on. But it's 20 years ago since I last wrote C++. I don't think I would have discovered this pattern on my own.

Passing Go objects to C

In order for an object in JavaScript to wrap an object in the host the V8 object needs to keep some reference to the host object.

For this purpose, V8 has the concept of an external value which can hold a generic pointer. But Go pointers aren't safe to pass to C for this usage. Go's garbage collector can move objects in memory, updating pointer variables accordingly. So any pointer held on to by C code would become invalid.

Go provides two solutions for this.

runtime.Pinner can prevent an object from being moved by the garbage collector, making pointers "safe" to pass to C code.
cgo.Handle creates an integer (uintptr) handle for the object. Internally, handles are generated by an atomically incrementing number, and a synchronised map convert a handle to a pointer.

Of course, both mechanisms prevent the object from being garbage collected, so you need to explicitly cleanup the object after use.

V8go didn't use any of these tho mechanisms, but had its own system which works exactly like the cgo handle.

A mistake was made

It took some time before I realised I had made a mistake. Before I started, I had already made a technical decision based on my existing knowledge: Integrate V8. It never occurred to me that there could be other options. There were two alternate options I should have considered, and one I should have picked:

SpiderMonkey, the JavaScript engine for Firefox.
Goja, a JavaScript engine implemented in Go.

Could spider monkey have been a better choice than V8? V8 isn't exactly easy to use outside C++ due to it's heavy reliance on C++ style resource management.

But Goja! A pure JavaScript engine! If I had known about this up front, I have used this instead.

I started adding support for Goja as an alternate engine. It's not ready yet, but when it catches up with the V8 integration, I will switch to Goja as the default engine, eliminating the need for CGo.

Integrating with Goja is significantly easier than integrating with V8

V8 support will not go away, as I should be able to safely rely on this being updated with latest JavaScript features.

Implementing an Object-Oriented API

To build a browser, you have to implement the DOM specification. And that specification is very 90s style object-oriented in its design. It is full of of circular references; and subclasses overriding behaviour of superclasses.

Everything in the DOM is a node, and you can append children to nodes. Children all know about their parent. Specialised nodes could be Text, Comment, and Element, which again has an HTMLElement subclass, further broken down into HTMLBodyElement, HTMLFormElement, HTMLAnchorElement, etc.

A first solution could be something like:

type Node interface {
    AppendChild(Node) Node
    // Unexported; we need to call it to maintain the tree, but client
    // code shouldnt.
    setParent(Node)
}

type Element interface {
    Node
    // methods that exist on element
}

type node struct {
    parent Node
}

type element struct {
    node
    tagName string
}

func (n *node) AppendChild(child Node) {
    n.childNodes = append(n.childNodes, child)
    child.setParent(n)
}

func (n *node) setParent(p Node) {
    n.parent = p
}

But this code doesn't work.

The type *node is a valid Node implementation, which makes *element one too as it embeds node. But when you call AppendChild on an element, it is a method on the embedded node that is executed. This method updates the parent of the new node to itself, the receiver value. Which is the embedded node, not the element.

So the DOM API, which is beyond my control, isn't a good fit for Go.

A solution. Was it the right one?

The solution I chose is to add a new method, SetSelf to Node, and single type that embeds a node must call it with a reference to itself as a Node.

type node struct {
    self   Node
    parent Node
}

func (n *node) SetSelf(self Node) {
    n.self = self
}

func (n *node) AppendChild(child Node) {
    n.childNodes = append(n.childNodes, child)
    child.setParent(n.self)
}

func NewElement(tagName string) Element {
    result := &element{newNode(), tagName}
    result.SetSelf(result)
    return result
}

The element tells the embedded node what it's own Node implementation is, pointing to the element, not the embedded node. So for an element, the Node passed to SetSelf can still be cast back to Element, and any method on Node, where *element has provided its own implementation, it is now the method on *element which is called, not the one on *node.

I'm not really happy with this. The necessity of the call is not very obvious from the code; and if a specialised node type forget to call it, the system will fail in ways that does little to reveal why.

The And as I don't want to have to implement every HTML element in the same package, the method also needs to be exported (what other languages call public).

I am wondering if a Strategy Pattern would have been a better solution, but that presents other challenges, like methods and attributes on specialised elements are exposed. E.g., a <form> element is represented by an HTMLFormElement and it has a specific properties and methods, e.g., a method property, and a requestSubmit method. And the HTMLTemplateElement hides all its children in a document fragment, accessed through a content attribute.

But so far, the code works as intended, and perhaps a different solution will present later?

p.s. If you have actual experience building a rendering engine, I'd like to hear from you, about your experiences.

"You Know Nothing"

I have been in the software development industry for 25 years, and most of that has been web development. I am surprised at how much I didn't know.

There are special elements with special behaviour. For example, the <template> which doesn't have any children. Instead, it's children in the HTML are added to a document fragment accessible in the content property. Yet they are rendered when reading outerHTML. This means that the HTMLTemplateElement needs to override outerHTML (object-oriented API).

Attributes aren't just attributes

The first good half of my career was server-side rendering of HTML (really the only option back then). Later, I have written a lot of React using the JSX syntax, which is almost indistinguishable from HTML.

So I was rather surprised, when I learned that attributes on elements aren't just attributes. After all, for both SSR and React, I write attributes in an HTML-like style, not revealing any difference, except for a few with different names. Yet there are two types of attributes: data attributes and idl attributes. The attributes in the HTML are data attributes; an accessible in the DOM using getAttribute and setAttribute.

The IDL Attributes are exposed as properties on the JavaScript element objects. They often reflect an identically named data attribute, but the can implement specific behaviour.

For example, the HTMLFormElement has a method IDL attribute. This will always return either "get" or "post", no matter the value of the data attribute. The default value is "get", and any invalid value will result in "get".

form = document.createElement("form")
form.getAttribute("method")
// null
form.method
// "get"
form.method = "invalid"
form.getAttribute("method")
// "invalid"
form.method
// "get"
form.setAttribute("post")
form.getAttribute("method")
// "post"
form.method
// "post"
form.setAttribute("pOsT")
form.getAttribute("method")
// "pOsT"
form.method
// "post"

Implementing IDL attributes isn't too problematic; they often expose behaviour I need anyway; e.g., both the method and action IDL attributes on a form implement logic necessary for form posting anyway.

But I was rather surprised I didn't know of this distinction.

Some IDL attributes also have different names then their data attribute counterpart, for example all aria- data attributes have camel-cased IDL attribute names, e.g., arialLabel. And the data attributes for and class are reserved JavaScript words, so the corresponding IDL attributes are named htmlFor and className.

React developers will undoubtedly recognise the latter two.

Where's the event loop?

I was browsing through the V8 API documentation, searching for the place to register a callback handling errors caused by setTimeout or setInterval callbacks; but nothing like that existed.

That's because those functions aren't in V8 at all because they are actually not part of the ECMAScript specification.

Browsers and node.js just happen to implement these functions. I did know that the two actually don't have the same interface; they differ in return type of the returned handle. The handle is a number in a browser, while it's an object type in node.

In hindsight, that alone should have been the dead giveaway that it's not implemented by V8. If chrome and node.js both use V8, and a global function returns different types, then it's probably not supplied by V8 itself.

So to write a headless browser, you need to write an event loop.

Autogenerating Code

Much of the behaviour of the DOM is specified by Web IDL specifications. After the first couple of JavaScript wrappers were written by hand, I started to investigate the possibility of autogenerating this code. I found webref, repository that contains a collection of all the IDL files, and is automatically updated. I added this as a submodule to start consuming those files.

Over time that process led to two new separate Go packages that have made available separately, as they have general purpose use:

gost-dom/webref - A package that exposes a subset of the IDL data as native Go types.
gost-dom/generators - Helper types for code generation. This is a layer on top of jennifer, providing an interface that lends itself better to composition.

Cutting corners

It's a long tradition in JavaScript to use polyfills to backport new browser features to older browsers. The same polyfills can also be used to quickly add functionality to Gost, and many already exists. Currently, XPathEvaluator is a pure JavaScript using slightly modified code from jsdom¹.

Many smaller functions, and constants, are only implemented in JavaScript code. For example, node type constants are in JavaScript space:

Node.ELEMENT_NODE = 1;
Node.ATTRIBUTE_NODE = 2;
Node.TEXT_NODE = 3;
Node.CDATA_SECTION_NODE = 4;
// etc

For CSS selector queries, I use a Go package, CSS. This works with x/net/html, the types that were used during HTML parsing. So when performing a CSS query, I recreate an "x" HTML tree from the Gost DOM, run the query on that tree, and translate the result back to the original elements.

Unfortunately, the library doesn't support checking if an element itself matches a CSS selector. So eventually I had to create my own matcher for Element.matches. But that is a minimal implementation, only used here, and only supporting the selectors that HTMX uses.

The really cool part!

As mentioned in the beginning, testing an HTTP server in Go, is just a matter of calling an HTTP handler function.

A headless browser in Go can do the same. From day one, I added the ability to bypass the TCP layer completely, connecting directly to the http handler implementation.

b := browser.NewFromHandler(myserver.RootHttpHandler)
// The host is ignored, but cookies don't work if it's not there.
window := b.Open("http://localhost:1234/")
window.Document().GetElementById("login-link").Click()

There's no overhead of the TCP stack, no hassle of managing startup and shutdown. You can run all tests in parallel with full isolation; of your own code supports isolation. Each browser has it's own isolated V8 instance.

The state of Gost

I recently released version 0.1. This release signifies that:

This has very limited functionality, but enough to support some real usage patterns.
The API is not guaranteed to be stable.

But it is now in the state where you it covers basic scenarios like a "login flow"

Navigate to the "index".
Click an HTMX boosted link that should navigate to a page requiring authentication, responding with a hx-push-url header, and the login page
Verify that the current location is the login page, and the history has a new entry.
Fill out the form (setting the "value" data attribute on the input fields).
Click the form's submit button (<button type="submit"> or <input type="submit">, both works).
Verify that the user is now redirected to the page they were trying to access, the location is updated, and the history has a new entry.

Any JavaScript/DOM feature that is not directly affected by this flow isn't supported yet. E.g., there is a setTimeout function, but it disregards the timeout value; the callback is scheduled for immediate execution. There isn't yet a setInterval.

What's up.

The currently planned next feature is to have a proper event loop supporting setInterval, as well as the clear function. And to support time travel in order to test behaviour that requires time to pass, without actually having to wait in the test. Very much like ~~lolex~~ sinon fake timers.

But that shouldn't take too long to implement. I'll just inject fake timers on the JavaScript side.

But user feedback can to a large degree change priorities. If real users struggle with specific problems due to lack of support, that will be a natural focus of attention.

Please go use it

I believe that this would be an extremely useful tool for anyone using Go for web development with SSR and JavaScript, in particular HTMX. And as the popularity of this stack increases, this tool becomes even more relevant.

In theory, this should work with React apps as well; but the current focus is HTMX support.

I invite everyone to try it out. Use it for the simple cases, as explained above.

And please, spread the word.

You can find Gost here.

jsdom doesn't have an XPathEvaluator class, but it does implement the global xpath evaluation functions. My adaptation was to write a class using those functions. ↩

Create an auto-merging workflow on Github

Peter Strøiman — Mon, 06 Jan 2025 12:16:09 +0000

How do you prevent bad commits from entering the main branch?

You can add a workflow that builds and runs your test suite. This will notify you immediately when this has happened. But the bad commits are already there, possibly blocking other developers. Or worse, preventing a crucial hotfix from reaching production.

You can submit a pull request, a popular popular way for teams to work. But pull requests are intended to handle the process for feedback of proposed changes. For single-developer repos, or teams that don't need this feedback process, using pull requests just to prevent broken builds from entering the default branch easily overcomplicates things.

A much simpler process is to just push to a different branch, have a let github automatically merge the changes to your main branch if verification succeeds.

This article shows how to setup such a workflow that runs almost seamlessly, and how to deal with a few challenges along the way.

Set the permissions

This setup requires a github workflow that pushes to your repository. In order for this to work, you need to set the proper permissions for the project.

Go to the settings page for your project, and open the "Actions > General" settings, where you must configure "Workflow permissions" to "Read and write permissions".

Once this is done, workflow actions can push to your repository.

Create/update a verification workflow

You must have a verification workflow. I assume that you already have an existing workflow, and that it has a push trigger.

Normally, the trigger is not set to run on all branches, so you need to add the branch name to use. Here it's called auto-merge. If you don't already have a push trigger, be sure to add it.

Remember the name of the workflow, here it's Build. It is needed in the next step.

# .github/workflows/build.yaml
name: Build

on:
  push:
    branches: [ "main", "auto-merge" ]

For a team setting, each developer could have their own branch, and you can use a wildcard to react to them all.

About github workflows

If you are new to workflows, here are some fundamentals.

A github workflow is started from a trigger. A common configuration is to use both the push trigger, to run the workflow when there has been pushed to a branch, and the pull_request trigger, which obviously triggers when a pull request is created, or new code pushed to the pull request.

There are many kinds of triggers, including triggers that are completely unrelated to code. E.g., I used a scheduled job trigger to renew server certificates from a github workflow.

A verification workflow would normally use actions/checkout to fetch the code from git. For a push trigger, the branch will by default be checked out. For a pull_request trigger, a merge commit will be checked out, i.e. it is the result of a merge with the default branch that is verified by the workflow.

A common default default is to have both push and pull_request triggers on the default branch, here main.

# .github/workflows/build.yaml
name: Build

on:
  push:
    branches: [ "main", "auto-merge" ]
  pull_request:
    branches: [ "main" ]


jobs:
  build:
    name: Build and test the code
    runs-on: ubuntu_latest
    steps:
    # Uses can use pre-made actions. 
    # actions/checkout will fetch your code.
    - uses: actions/checkout@v4
    # Frameworks can often be configured using other actions.
    # Github have good starting points for most project types.
    - name: Build and test
      # Make sure you run the steps necessary. 
      # Github have good starting points for most project types.
      run: ./build-and-test.sh

Create a new workflow in the default branch.

The new workflow uses the workflow_run trigger, a trigger that can react to events of another workflow.

This workflow is not associated with a branch, it points to another workflow. Because of that, it is global to the github project and must exist in the default branch; typically named main or master.

The auto-merge workflow should be run when the verification workflow is completed, and the workflow was executed on the auto-merge branch.

# .github/workflows/auto-merge.yaml
name: Auto-merge
on:
  workflow_run:
    workflows: [Build]
    branches: [auto-merge]
    types: [completed]

So while the workflow with a workflow_run trigger workflows on a project level, it can still filter on the branches that were the trigger a verification workflow run.

Note: You can use wildcards in your branch names if you have multiple auto-merge branches.

Create a job

A job does the actual work. While this is triggered on a completed verification workflow, a completed workflow can still have failed. To handle that, add a condition to check the outcome of the completed workflow.

jobs:
  on-success:
    runs-on: ubuntu-latest
    if: ${{ github.event.workflow_run.conclusion == 'success' }}

Fetch the code

To fetch the code, we use the actions/checkout action.

The first unexpected issue is that the action checks out the default branch, not the auto-merge branch; which was the original source of a trigger. While a push trigger will by default check out the branch that was updated, the workflow_run trigger does not. We must check out the right branch in the workflow ourselves.

The branch name exists as data on the associated event of the trigger. This contains information about the completed workflow, including the name of the branch that triggered the first workflow. This value is found in the variable github.event.workflow_run.head_branch.

Note: For a single auto-merging branch, we could just have duplicated the branch name, but for multiple branches, it's necessary to read this value from the event.

Shallow Clones

The next issue is that by default, the checkout action creates a shallow clone, i.e., there is no history. You cannot push to a branch, if you don't locally have all commits from, and including, the head of the target branch to your branch.

The easiest solution is to add fetch-depth: 0 to the action. This pulls the full history, problem solved.

    steps:
      - uses: actions/checkout@v4
        with:
          ref: ${{ github.event.workflow_run.head_branch }}
          fetch-depth: 0

For large repositories with a lot of history, or large binary files in the history, this can increase the runtime significantly. But there are ways to deal with this problem.

Smarter handling of shallow clones

Instead of fetching full history, we can fetch just enough history to be able to push.

Note: This is much more tricky, so for smaller repositories, the previous method would be advisable. Particularly, if you/the team don't feel comfortable with shell scripting.

The changes to make compared to the previous simple version are:

Remove the ref and fetch-depth options.
Use git remote set-branches ... to tell git there is a different remote branch we want to use.
Use git fetch ... to fetch the relevant commits, i.e. just enough shallow history, to be able to push the changes.
Checkout the source branch locally.

    steps:
      - uses: actions/checkout@v4
      - name: Tell git we want the `auto-merge` branch
        run: git remote set-branches --add origin ${{ github.event.workflow_run.head_branch }}
      - name: Fetch the target branch
        run: git fetch --shallow-since="`git show --no-patch --format=%ci HEAD`"
      - name: Checkout target branch
        run: git checkout ${{ github.event.workflow_run.head_branch }}

Here is a more detailed explanation of the changes.

1. Remove `ref` and `fetch-depth`

Because we need to have all commits in the source branch not reachable from the target branch (default branch), we need the target branch in our working copy. The simplest solution is that start the process with the target branch checked out.

2. Add new remote branch

In a shallow clone, git doesn't fetch all remote branches, only the target branch. The following command tells git explicitly that this is a branch we want to work with.

git remote set-branches --add origin ${{ ... .head_branch }}

Again, for a single branch, you could duplicate the branch name, but it's so simple to avoid it.

3. Fetch the remote branch.

With the remote branch added, we can fetch the branch using git fetch.

In a shallow clone, fetch will fetch commits that are after the current branch, i.e., it will fetch all the commits that are new in the auto-merge branch. But if the target branch is ahead of the source branch, i.e., the default branch contains commits not yet in the auto-merge branch, git fetch fetches the entire history, defeating the purpose of the shallow clone to begin with.

This scenario is less likely for a single-developer setup, but very likely to happen occasionally in a team setup where team members use individual auto-merge branches.

To keep the clone shallow, the command option --shallow-since="" is used to not fetch commits older than the current HEAD (which is the default branch). The commit timestamp for HEAD is found using git show --no-patch --format=%ci HEAD.

git fetch --shallow-since="`git show --no-patch --format=%ci HEAD`"

Backticks here is a special shell feature that executes the git show ... command, and uses the textual output as input for the command, here for the arguments for the git fetch command

fetch will now fetch exactly all the commits that are necessary for the branch to be pushed if possible.

4. Check out the source branch

How we have just enough commits in the local database to push. We checkout the branch

git checkout ${{ github.event.workflow_run.head_branch }}

Push

Both the simple, and the shallow clone workflow variations, have left us with the source branch
checked out. All that is left is to push it to the remote target branch, i.e. push auto-merge to main in this example. Here HEAD is used, so the script doesn't need to know what the branch actually is.

jobs:
  on-success:
    # ...
    steps:
      # ...
      - name: Push to main
        run: git push origin HEAD:main

By default, git will only perform a fast-forward merge on push. So the workflow will fail if there are new commits on the target branch. In this case, you need to deal with the conflict, by either merging or rebasing your changes off the new master; just as you normally would.

The error could be one of two:

The push failed because it wasn't a fast-forward.
The push failed because you didn't have the remote target HEAD in your local git database. This was because they were not fetched as they were committed earlier than the target HEAD.

No matter which, the cause is still the same, there are new commits in the default branch not reachable from the HEAD of your local branch.

There is another potential error, you have rewritten the commit time in the history. That shouldn't be a normal workflow, so this workflow is not designed to handle that.

The full workflow file

This is the full workflow file. Adapt the following to your own workflow.

The verification workflow is named Build.
The branch to run on is named auto-merge.
The default branch is named main.

# .github/workflows/auto-merge.yaml
name: Auto-merge
on:
  workflow_run:
    workflows: [Build]
    branches: [auto-merge]
    types: [completed]

jobs:
  on-success:
    runs-on: ubuntu-latest
    if: ${{ github.event.workflow_run.conclusion == 'success' }}
    steps:
      - uses: actions/checkout@v4
      - name: Tell git we want the `auto-merge` branch
        run: git remote set-branches --add origin ${{ github.event.workflow_run.head_branch }}
      - name: Fetch the target branch
        run: git fetch --shallow-since="`git show --no-patch --format=%ci HEAD`"
      - name: Checkout target branch
        run: git checkout ${{ github.event.workflow_run.head_branch }}
      - name: Push current head to main
        run: git push -v origin HEAD:main

Pushing to the right branch locally

You can push directly to the branch from the command line:

> git push origin HEAD:auto-merge

Here, the remote is assumed to be named origin, the default for most workflows. But this can be simplified, so you just need to perform a normal git push.

To set that up, you need to make some changes in your local git repository configuration, found in the .git/config file of your working directory.

First sub-optimal solution

By adding a push refspec for the remote, you can configure git to push to a different remote branch.

# .git/config
[remote "origin"]
    url = git@github.com:username/repository.git
    fetch = +refs/heads/*:refs/remotes/origin/*
    push = refs/heads/main:refs/heads/auto-merge

With this setting, when you pull the main branch from your local working directory, you get the main branch from the remote repository, but when you push the main branch, it will be pushed to the auto-merge branch on the remote. If your changes are good, your team mates will quickly get them when they pull.

Using this setup, you can work on the main branch locally exactly the same way you would normally.

But this also changes the default behaviour for other branches.

A less intrusive configuration

A solution to not break default behaviour for other branches is to create a new remote with the same url. Then you can configure the default to use this remote when pushing. Now other branches are not affected by the change to the push configuration, only the default branch has different behaviour.

# .git/config
[remote "origin"]
    url = git@github.com:username/repository.git
    fetch = +refs/heads/*:refs/remotes/origin/*
[remote "origin-main"]
    url = git@github.com:username/repository.git
    fetch = +refs/heads/*:refs/remotes/origin/*
    push = refs/heads/main:refs/remotes/origin/auto-merge
[branch "main"]
    remote = origin
    pushRemote = origin-main
    merge = refs/heads/main

The local workflow

With everything setup, your workflow should looks remarkably familiar:

> git pull # or git pull --rebase
> git commit -am "Change 1"
> git commit -am "Change 2"
> git commit -am "Change 3"
> git push
# In case of a conflict
> git pull # or git pull --rebase
# Fix merge conflicts
> git push

The only differences are:

There is the delay of the build before you know what the outcome is.
You local branch will appear to be ahead of the default branch after a push, until you fetch again after a successful build.

But commits failing the verification will not appear in the main branch.

The git configuration is local

Remember, the repository configuration is local, i.e., it exists only on the local machine. Every developer need to set the configuration on every computer they use.

It's a simple solution to a simple problem

This setup will not prohibit bad commits from reaching the default branch. Developers can still push to directly to the default branch.

For a team project, every developer needs configure their git configuration to push to an auto-merge branch, requiring uncommon initial custom configuration.

But for those projects that don't need complicated processes, this small change can help prevent bad commits from reaching the default branch in a completely non-intrusive way.

Go-DOM - 1st major milestone

Peter Strøiman — Mon, 18 Nov 2024 07:38:32 +0000

After just under 2 weeks of work; I finally reached the first major milestone for Go-DOM.

Now, the browser will download and execute remote JavaScript when building the DOM tree.

A Brief History

The project started as a crazy idea; seeing that Go and HTMX is a stack that is gaining some popularity;

Go already has all the tools you need to test pure server-side rendering. But when adding a library like HTMX, the behaviour of the app is the result of a choreography between the initial DOM; the attributes on interactive elements; the HTTP endpoints reached, and the content delivered by those endpoints; both response headers and body. To verify the behaviour from the user's point of view, you need a browser; or at least a test harness that behaves ... not entirely unlike a browser.¹

Searching for "headless browser in go" only led to results suggesting to use a real browser in headless mode. This combination that has a huge overhead; discouraging the fast and efficient TDD loop. Relying on a real browser will typically slow you down instead of speed you up.²

So the idea was sparked; write a headless browser in pure Go as a testing tool for web applications;

The first uncertainties to address was the parsing of HTML; as well as script execution. I managed to quite quickly; within 2 days to address both of these. I had a very rudimentary HTML parser; as well as I had integrated v8 into the code base³ and make Go objects accessible to JavaScript code.

The HTML parser was later removed, as go x/net/html already implements an HTML parser; dealing with all the quirks of HTML parsing. Parsing a well-formed document isn't a terribly difficult problem to solve. It's gracefully dealing with malformed HTML where it becomes tricky.

After some time, I also managed to get inline script execution to run at the right moment; i.e. the script is executes when the element is actually connected to the DOM, not after the full HTML has been parsed.

Working with HTTP requests

After being able to process an HTML document with inline script; the next step was to actually download scripts from source. This needed to integrate an HTTP layer; such that the browser fetches content itself; rather than being fed content.

The http.Client also allows you to control the actual transport layer using the http.RoundTripper interface. Normally you start a server; which will listen for requests on a TCP port. In this context TCP serves as the transport layer; but is itself irrelevant for the processing of HTTP requests. Due to simplicity of the standard HTTP stack in Go; an entire HTTP server is represented by a single function, func Handle(http.ResponseWriter, *http.Request).

The headless browser can completely bypass the overhead of the TCP stack and call this function directly using a custom RoundTripper.

Now the browser can perform HTTP requests, but the browser code itself is ignorant of the fact that the HTTP layer is bypassed. And with this came the ability to download the script during DOM parsing.

Example Code

Let's explore a simple test, as it looks now in the code base (the code uses Ginkgo and Gomega, a somewhat overlooked combination IMHO)

First, the test creates a simple HTTP handler which serves two endpoints, /index.html, and /js/script.js.

It("Should download and execute script from script tags", func() {
  // Setup a server with test content
  server := http.NewServeMux()
  server.HandleFunc(
    "GET /index.html",
    func(res http.ResponseWriter, req *http.Request) {
      res.Write(
        []byte(
          `<html><head><script src="/js/script.js"></script></head><body>Hello, World!</body>`,
        ),
      )
    },
  )
  server.HandleFunc(
    "GET /js/script.js",
    func(res http.ResponseWriter, req *http.Request) {
      res.Header().Add("Content-Type", "text/javascript")
      res.Write([]byte(`var scriptLoaded = true`))
    },
  )

  // ...

The intention here is merely to verify that the script is executed. To do that, the script produces an observable side effect: It sets a value in global scope.

To verify that the script was executed is just a matter of examining the global scope, which is done by executing ad-hoc JavaScript from the test itself; verifying the outcome of the expression.

The code to create the browser, load the index file, and verify the observed side effect

browser := ctx.NewBrowserFromHandler(server)
Expect(browser.OpenWindow("/index.html")).Error().ToNot(HaveOccurred())
Expect(ctx.RunTestScript("window.scriptLoaded")).To(BeTrue())

Test execution is also pretty fast. The part of the test suite involving JavaScript execution currently consists of 32 tests running in 23 milliseconds.

Next milestone, integrate HTMX.

As the project was initially conceived while trying to verify an HTMX application, a reasonable next goal is to support just that case. A simple HTMX application with a button and a counter, which increases when the button is pressed. This will drive the need for:

AnXMLHttpRequest implementation needs to be in place. Work is underway for that.
An XPathEvaluator. I believe this can be poly-filled to begin with.
Event propagation. Only DOMContentLoaded and load events are emitted right now. Elements need to support more events; such as click; as well as methods to trigger them.
- This might also require proper event capture and bubbling.

And then ...

Following that, more advanced user interaction; proper form handling, e.g., input handline (e.g., pressing enter in an <input> field submits the form. This typically also involves some kind of URL redirection; which drives the need for a history object, etc.

Integration external sites

With the ability to control the transport layer; we can provide tests with unique abilities; we can mock out external sites that the system would depend on at run-time. E.g., for a given host name, the test could provide another Go HTTP Handler simulating the behaviour.

The most obvious example is the use of external identity providers. The test could simulate the behaviour of a login flow; not having to force you to create dummy accounts in an external system, have test failures because of outages in an external system, or simply being unable to automate the process at all because of 2FA or a Captcha introduced by the identity provider.

Another use case is the use of API-heavy libraries, like map libraries, that incur a usage cost. Mock out the external site to not receive an extra bill for running your test suite.

But given the fact that you can pass a custom http.RoundTripper; it should be easy enough to create an implementation that forwards the call depending on the host.

Usability over Compatibility

Creating a 100% whatwg standards compliant implementation is quite an endeavour; I didn't fully comprehend the scope until I actually started reading parts of the whatwg specifications. The goal is to create a tool helping write tests for web applications. Full compatibility is a long-term goal; but until the project has reached some level of usability, I will not start filling out holes of missing compliance.

For that reason; features that are more likely to be used in actual applications are more likely to be prioritised. A feature request pointing to an actual test giving the wrong result is likely to be prioritised. A feature request for the implementation of a specific standard is likely to be rejected.

Spread the word

I believe this can be a very useful tool for many developers, so if you read this, let your coworkers know it exists. This is so far a spare time project, of which I have plenty of at the moment; but that will not be the case forever.

If you want to see this live, spread the word ...

Perhaps you would even sponsor this? Do you have a large company building web applications in Go? Feel free to contact me.

Find the project here: https://github.com/stroiman/go-dom

Kudos if you managed to catch the homage to a popular BBC radio play. ↩
This is based on personal experience. Doing TDD right will speed you up due to the fast feedback cycle. The overhead of a real browser tends to make you write tests after the production code; loosing their benefit of the feedback loop a fast test suite gives you. ↩
Groundwork had already been laid by the v8go project. However; not all features of v8 are exposed to Go code; including necessary features for embedding native objects. I was able to add those in a separate fork; which is still WIP. ↩

Go-DOM - A headless browser written in Go.

Peter Strøiman — Wed, 06 Nov 2024 13:16:26 +0000

Having too little to do sometimes results in a crazy idea, and this time; it was to write a headless browser in Go, with full DOM implementation and JavaScript support, by embedding the v8 engine.

It all started working with writing an HTMX application, and the need to test it that made me curious if there was a pure Go implementation of a headless browsser.

Searching for "go headless browser" only resulted in search results talking about automating a headless browser, i.e. using a real browser like Chrome of Firefox in headless mode.

But nothing in pure Go.

So I started building one.

Why A Headless Browser in Go?

It may seem silly because writing a headless browser will never work like a real browser; and as such wouldn't really verify that your application works correctly in all the browsers you have decided to support. Neither does this allow you to get nice features such as screenshots of the application when things stop working.

So why then?

The Need for Speed!

To work in an effective TDD loop, tests must be fast. Slow test execution discourages TDD, and you loose the efficiency benefits a fast feedback loop provides.

Using browser automation for this type of verification has severe overheads, and such tests are typically written after the code was written; and as such, they no longer serve as a help writing the correct implementation; but are reduced to a maintenance burden after the fact; that only occasionally detect a bug before your paying customers do.

The goal is to create a tool that supports a TDD process. To be usable, it needs to run in-process.

It needs to be written in Go.

Less Flaky Tests

Having the DOM in-process enables writing better wrappers on top of the DOM; which can help providing a less erratic interface for your tests, like testing-library does for JavaScript.

Rather than depending on CSS classnames, element IDs, or DOM structure, you write your tests in a user-centric language, like this.

Type "me@example.com" in the textbox that has the label, "Email"

Or in hypothetical code.

testing.GetElement(Query{
  role: "textbox",
  // The accessibility "name" of a textbox _is_ the label
  name: "Email",
}).type("me@example.com")

This test doesn't care if the label is implemented as <label for="...">, <input aria-label="Email">, or <input aria-labelledby="email-label">.

This decouples verification of behaviour from UI changes; but it does enforce that the text, "Email" is associated with the input field in accessible way. This couples the test to how the user interacts with the page; including those relying on screen readers for using your page.

This achieves the most important aspect of TDD; to write tests coupled to concrete behaviour.¹

Although it's probably technically possible to write the same tests for an out-of-process browser; the benefit of native code is essential for the type of random access of the DOM you most likely need for these types of helpers.

An example: JavaScript

To exemplify the type of test, I will use a similar example from JavaScript; also an application using HTMX. The test verifies a general login flow from requesting a page requiring authentication.

It's a bit long, as I've combined all setup and helper code in one test function here.

it("Redirects to /local after a successful login", async () => {
  // Setup - stub the authentication, and create a stubbed user
  // using a test helper
  sinon
    .stub(auth, "authenticate")
    .withArgs({
      email: "jd@example.com",
      // matchPassword helper is used, as passwords are wrapped in a class
      // preventing accidental disclosure in logs, console out, etc.
      password: matchPassword("s3cret"),
    })
    .resolves(
      auth.AuthenticateResult.success(createUser({ firstName: "John" })),
    );
  const url = `http://127.0.0.1:${port}/auth/login?redirectUrl=%2Flocal`;
  // Request private page. This _should_ generate a redirect
  const wrapper = await DOMWrapper.open(url); // Just a helper around jsdom
  const browser = wrapper.browser;
  // Once HTMX is ready, it emits an `htmx:load` event. Then verify that it was 
  // correctly redirected.
  await wrapper.waitFor("htmx:load");
  expect(wrapper.url.pathname).to.equal("/auth/login");
  // Use testing-library to fill out and submit the form
  let screen = wrapper.screen;
  const username = screen.getByRole("textbox", { name: "Email" });
  const password = screen.getByLabelText("Password");
  await userEvent.type(username, "jd@example.com");
  await userEvent.type(password, "s3cret"); // password has no role
  // Wait for a new `htmx:load` event, while clicking the submit button
  // at the same time.
  await wrapper.runAndWaitFor(
    ["htmx:load"],
    userEvent.click(screen.getByRole("button", { name: "Sign in" })),
  );
  // After the new new page has been loaded, verify that the username
  // is displayed (i.e. the stubbed user is used), and the correct
  // URL is used.
  screen = testingLibrary.within(browser.window.document.body);
  const heading = screen.getByRole("heading", { level: 1 });
  expect(heading.innerHTML).to.equal("Hi, John");
  expect(wrapper.url.pathname).to.equal("/local");
});

In simple terms the test does the following:

Stub out the authentication function, simulating a successful response.
Request a page that requires authentication
Verify that the browser redirects to the login page, and the browser URL is updated. ²
Fill out the form with the expected values, and submit.
Verify that the browser redirects to the originally requested page, and it shows information for the stubbed user.

Internally the test starts an HTTP server. Because the this runs in the test process, mocking and stubbing of business logic is possible. The test use jsdom to communicate with the HTTP server; which both parse the HTML response into a DOM, but also executes client-side script in a sandbox which has been initialised, e.g. with window as the global scope.³

This enables writing tests of the HTTP layer, where validating the contents of the response is not enough. In this case; that the response is processed by HTMX as intended.

But apart from waiting for some HTMX events, so as to not proceed to early (or too late) the test doesn't actually care about HTMX. In fact, if I remove HTMX from the form, resorting to classical redirects, the test still pass.

(If I remove the HTMX <script> tag completely, the test will timeout waiting for HTMX events)

Speed? Check!

While the previous test was a little slower than desired; it's reasonably fast, completing in typically 150-180ms. This is far too slow for the majority of the test suite, but it's fast enough to serve as a feedback loop while working on that particular feature.

This test is not part of a normal TDD run. They are run when I work on that feature; or just before committing; ensuring nothing broke. Which is a completely normal way to deal with "slow tests".

Potential Speed Improvements

The JavaScript example uses a real HTTP server started on a random port. The server runs in the process of the test runner, which is why we can stub and mock business logic.

In Go, HTTP requests are handled by an http.Handler, making it very easy to consume the HTTP handling logic without actually launching an HTTP server.

And this is something that the go-dom code does handle right now, and currently the test suite runs in zero milliseconds, rounded to the nearest millisecond.⁴

Mocking and Parallel Tests?

The ability to run parallel tests only depends on your code's ability to run in parallel. As this can consume an http.Handler, each test can create its own handler; each with different dependencies replaced with test doubles as fits the individual test.

This allows you to test the HTTP layer as a whole; using stubbed business logic.

Current State of the Project?

Close to nothing is implemented; the current state is the result of about one and a half day's work. I have a basic streaming tokeniser that can consume an http response stream, which is passed to a parser that returns a Node.

The code can currently process the string <html></html> (no spaces allowed yet) into an HTMLHtmlElement.

Next steps are

Improve the parser just slightly, and get a few more element types implemented
~~Embed the v8 engine, addressing the primary uncertainty~~; how are Go objects made accessible to JavaScript, and how Go code can inspect the result of mutations from JavaScript code.
- I made en extremely simple POC using v8go, and what appears to be best maintained fork. It lacked necessary features; which I have added in my own fork

Future of the Project?

This will most likely die :(

I am not even working on a Go project where this would be valuable (I was working on a node.js project). It was the joy of seeing how jsdom helped serve as a feedback loop for the authentication flow that sparkled a stupid idea that was fun to pursue. And as a person with ADHD, this is a typical pattern for me. I start something that is fun, work on it; until something else hits the radar and intrigues me.

Unless ...

Other developers think this is a good idea and want to help building this.

I believe that such a tool would be extremely helpful for any Go project combining server-side rendering with client-side scripts, including HTMX-based applications.

The project is found here: https://github.com/stroiman/go-dom

The goal of TDD is not to write unit tests. That is an extremely common; but utterly incorrect misconception. ↩
The important part is not that we are redirected; but that the browser history has the correct entries, providing sensible behaviour of browser back/forward functionality. The test should really have verified the contents of history, or perhaps even used actively used the navigation API to go back and forth. As such; the test could be improved in regards to describing expected behaviour from the user's point of view. ↩
Wring a headless browser in JavaScript has an unfair advantage; as your simulated DOM are already valid JavaScript objects. In Go, extra work is necessary to allow client-side scripts to mutate the DOM, and let the outcome be accessible in test code. ↩
As reported by Ginkgo. This doesn't include the overhead of building and launching, which has a noticeable, but very short delay. ↩

My three epiphanies of TDD

Peter Strøiman — Sun, 14 Jul 2024 14:37:00 +0000

I once had a talented developer join my team. He was quick to understand the sometimes odd idioms used in the code base. But he wasn't that comfortable writing tests. When talking about his past experiences he said, "we often didn't have time to write tests".

What strikes me as odd is, that writing the right tests first makes you faster. Of course, if you first write code, and only proceed to write tests after you have already fixed all issues and established that the code works as intended; then yes, writing tests would be an investment in time with a questionable return on investments. But if you write the tests before you write the code, the tests help you implement the code faster, with fewer bugs, and more maintainable.

In my career, I've had three epiphanies while practicing TDD. The first epiphany was:

Writing tests first makes you faster; not slower

Let me write about the background story of all three events, and how they helped me become a better programmer.

The first epiphany

This was the first project, where I consistently wrote tests before I wrote code. Prior to that, I had been from time to time, adding tests to existing code. But on this particular project project, there was a shared understanding that the code should be covered by tests. However, apart from me, only one other developer consistently wrote code test first. A few would sometimes write the test first, but most wrote the tests after writing production code (often resulting in complex tests that themselves had bugs, so they didn't actually test anything)

It was of course a learning process for me, I didn't get good at writing tests immediately, and today I would question if I was even practicing TDD.

I wrote tests under the assumption that it was an investment. I would spend a little bit more time developing a feature. The payoff would be reduced maintenance costs. It turns out I was wrong in a way I didn't expect.

One day, something marvellous happened

One day I was working on a feature that required modifications to ALL the layers of the application. The application was a Windows Application build using WPF following an MVVM pattern, communicating with a .NET server using WCF. This is a list of the layers I needed to modify:

WPF Views
View Models
Internal model in the frontend
Communication layer in the frontend
WCF contracts
Communication layer in the backend
Domain model in the backend
Entity framework mappings
Database migrations.

With the exception of the WPF views, that were basically untestable, I had implemented every thing I needed in every layer by writing tests first, starting from the UI (that's what I though outside-in meant back then). I had been coding for quite a while, probably a few hours, when I was finally ready to launch the application and test everything manually before handing it over to testers for further scrutiny when something totally unexpected happened:

It worked the first time!

Did I not make any mistakes during several hours of coding? Of course I had. I had made many mistakes, but they had all been caught early by the tests.

Now I started thinking, how long time would I have spent debugging, if those errors had not been caught early? At that time we didn't have a reliable mechanism for hot code reloading, so every code change would require a relaunch of the application. For a proper stateless web application that's not a big a deal; your state is in the already open browser window. But for a Windows application, I need to shut down the application, navigate through the menus to get the application into the desired state, just to get to the point, where I can start debugging. Combine that with the build times, it would mean about 1-2 minutes from making a code change just to get to the point where I could reproduce the unexpected behaviour. Add to this that debugging was not trivial, as I would have no idea of knowing initially if it was the server or the client that had the bug.

The time saved, by not debugging was enormous!

I had no doubt at all, the time necessary to find the bugs in debugging sessions would far exceed the relatively minor amount of time I had spent writing the tests.

The pattern continued; from time to time, I could work on a feature that worked the first time. It wasn't always - far from it. But often when the feature didn't work initially, the problem was often trivial to find, typically a bad bindings in the WPF views, as they were virtually impossible to test.

Writing tests first, made me faster; not slower

These days, more than 10 years later, I almost never use a debugger. I don't even have one configured in neovim, my editor of choice.

The second epiphany

A few years later, I was working on a web application. This was at a time when SPAs were still not that common, and our application was delivering server-rendered HTML, with a wee bit of JavaScript. All backend code was developed using TDD as this time all team members would write the tests first.

The tooling for testing JavaScript was far from what we have today, neither was the editor support. To test code that manipulates the DOM, you basically needed to run them in a browser. For that purpose we had an HTML page that would load QUnit, our test code, and our production code.

Previously, I had been using various Visual Studio plugins that allowed me to run tests without leaving Visual Studio. Now, when a file was saved, I needed to switch focus to the browser, and refresh the page. Fortunately, there was tool called LiveReload that could monitor the file system for changes, and trigger the browser to reload. Now, every time I saved the file, the tests would run. And they were FAST.

I didn't even know what fast meant before this. I though the C# test suite was fast. Maybe we could run the test suite with thousands of tests in 10 seconds, or maybe even just a second. That is excellent for a CI server. But add compilation and startup time to that, and the fastest the feedback loop could be would still be measured in a magnitude of seconds.

But the browser updated instantaneously. My guess is that it was less than 200 milliseconds from saving a file to seeing the result on screen.

Sometimes the HTML page would not show any test results at all. In that case, the JavaScript would have syntax errors, e.g. mismatching parenthesis or braces, an easy mistake to make as our JavaScript code relied heavily on callback functions, and there wasn't a lot of help from the editor.

This experience profoundly changed the way I wrote code. I would no longer write the code to make a test pass and then save. I would save my changes every time the code was syntactically valid; typically every 5-20 seconds. Did I need to add a new UI event handler? First, add an empty event handler, save, BAM! Immediate feedback on a misplaced closing brace.

In case of an error, I wouldn't bother trying to find out what I had done wrong; I would just undo my change, and try again. Quite simply, spending 30 seconds trying to identify which parenthesis or bracket that was misplaced would be silly when it takes 10 seconds to rewrite the intended code.

Having instantaneous feedback while writing code profoundly changed the way I worked.

Today, I will go to great lengths to setup my tooling to ensure the fastest possible feedback. Why do I still use Mocha for testing JavaScript code when more modern alternatives exist? Because none of them can match the speed of mocha (despite at least one of them makes the claim to speed)

The third epiphany

A few years later, I needed to connect to a RabbitMQ from a node.js project. I already had a little experience with RabbitMQ, but in Go, not node.js.

But what I realised here was that there was a series of problems that needed to be solved individually. The first problem was just establishing a connection. So I wrote something like this:

describe.only("Rabbitmq communication", () => {
  it("can connect", async () => {
    await amqplib.connect("amqp://guest:guest@localhost");
    await sleep(10000)
  })
})

This would give me the feedback I needed. First, no errors are thrown. Second, has a connection been established? I would manually verify that in the RabbitMQ management UI that lists active connections.

This was the first time I wrote a "test" that wasn't a test, i.e. I didn't even try to think about what kind of verification would make sense here. But I didn't write it for verification, I needed a way to quickly execute bits of code to get feedback, and that is exactly the functionality that mocha provided making it the perfect tool for the job.

In essence, the typical "unit test tools" basically just allow you to write a lot of small tasks, or pieces of code, that can be executed individually, and for which errors are reported. In addition to that, they also often provide an easy way to be selective about which tasks to run, allowing you to pick what is relevant for the problem you are working on, improving the feedback loop, and possibly reducing noise in the output.

After I was able to successful connect, I wrote the code to create a channel over the connection, and then the code to send a message to a queue. Once again, I would manually inspect in the RabbitMQ management console that it had indeed arrived at the queue, which I had also created manually in the management UI. Then I would proceed to writing code to receive messages, and now I had the first meaningful assertion to write: verify that the content of the message received is identical to the content sent.

After that, I no longer depended on manual inspection. I was now in a state with feedback measured in milliseconds and the process started to speed up dramatically; I wrote code to create queues programatically, create exchanges and bindings¹, send the message through the exchange. After each code change, within milliseconds, I knew if the message was still coming through.

Eventually, I had uncovered the unknowns about how to use the library, and the "test file" contained all pieces of code necessary to setup the infrastructure. At that point in time, I would start extracting the code into a class, gradually moving the different parts of the code from the test file to production file. Still, with millisecond feedback, I would know if a message would still pass through RabbitMQ after every small change to the code.

The code that started its life inline in a single test function eventually evolved into a production-mature module for communicating with RabbitMQ.

The Third Epiphany: TDD is NOT about writing unit tests, it is about feedback.

I had long known about the idea that TDD is the "red, green, refactor" cycle. But this was the first time that refactoring was an essential part of the process, and it was the feedback that allowed refactoring.

Eventually, the test changed characteristics. Rather than describing RabbitMQ interaction, they described how certain domain events should trigger other business processes, such as, "When a new user has registered, a welcome email should be sent". The test did not actually call the business logic, neither was the "register user", not the "send welcome email" code called from these tests. This had the responsibility of ensuring that a well defined event would eventually trigger a call to a use case, which had the responsibility of sending a mail. Both of these functions had been developed using different tests of tests.

Eventually RabbitMQ was reduced to an implementation detail from the perspective of the test that verified the behaviour of a Publisher and Subscriber. In fact, RabbitMQ could potentially be replaced with another messaging technology without modifying the tests. The test suite would then be able to verify that business processes that required a temporal decoupling would still run as intended after replacing the messaging infrastructure. Other tests verified behaviour of a technical concern rather than from the business rule side. E.g. how message retry would work in the case of a failure during message processing, e.g. a broken SMTP connection.

Authoritative sources to back my conclusions

Before publishing this article with claims to what TDD is really about, I decided to see if there are any sources where Kent Beck himself opine on the matter. I learned that Kent Beck has a YouTube channel. Most of the content is really old, but I was delighted to learn that he express exactly the same point of view as I do: He repeatedly states that the goal is to get fast feedback.

Building a Tokyo Tyrant client library

This video from 2010 is a small fraction of what was a longer video course on TDD by Kent Beck. Unfortunately, the comments track suggests, the the rest is lost. Only the first 10 minutes are now available, but during those, Kent Beck specifically mentions that the primary reason for TDD is to get feedback.

The actual use case is creating a Java library for connecting to Tokyo Tyrant. Early on, he expects to end up with a class TyrantClient to which the test will eventually interact. That's where he imagines that the test will end up, but he doesn't even try to write that test. Rather, he writes a "test" that tries to open a TCP socket just to explore how to connect with the server. Next, he writes code, still in "the test" that transmits binary data over the socket directly; at which point in time the video ends.

But the very short video is enough to show that his way of approaching the problem is almost identical to how I started using RabbitMQ from node.js; Let's just get feedback on whether or not we can establish a connection without worrying about how the "test" will look when completed.

Use TCR to implement a rope data structure

More recently, Kent Beck has been experimenting with a process called TCR, Test && Commit || Revert. The idea is that after every file save, tests are automatically executed. If they pass, all changes are automatically committed to source control; otherwise, all changes are reverted, i.e. your working directory is reverted to the previous green state.

This process encourages very small changes, and you basically end up writing code in way similar to what I did during my second epiphany.

The only important difference is that in TCR, the undo is automatic, rather than manual, I would work in just as small batches as Kent Beck shows. One minor difference is that in my case, a failing test could be the expected outcome, assuming it failed with the expected error. But apart from that the process was quite similar. If I didn't see the expected outcome, I'd more often undo than diagnose, potentially work in even smaller steps during the next attempt.

Interview with Dan North

The third source is an interview with Dan North, the "inventor" of BDD. Originally, BDD was exactly the same as TDD was supposed to be. But in one particular project, the word "test" gave the wrong impression. Testers objected, that programmers would take their jobs (and would be bad at it), and programmers objected that they weren't testers. As the focus of the programmers was different from that of the testers, which would not be replaced, changing the term, "test" to "behaviour" made all the difference.

Since then, a lot have happened in the BDD community, and today is more considered an acceptance-test-driven outside-in TDD approach, with more tools surrounding and supporting the process, such as Cucumber. But just as TDD has been misinterpreted, so has the intention of cucumber

In the interview, Dan states that BDD was originally intended to exactly the same as TDD was about: A process to iteratively work on the design, and he states directly "TDD ... has nothing to do with testing".

Why and when is TDD the most efficient development method?

I believe that TDD is the most efficient method for developing the vast majority of your code base, as the fast feedback encourages you to solve the problems one at a time; potentially trying to identify the smallest possible solvable problem. You can iteratively work on the design of your code, and extract abstractions when you discover them; or remove abstractions when you realise they are no longer helpful. You can make these changes in very small steps, immediately validating your assumptions, truncating long unproductive paths.

When is TDD not the right choice?

TDD isn't helpful when executing small pieces of code doesn't provide meaningful feedback. The most obvious example is UI code when the focus not behaviour, but style, colours and layout. In this case, the only meaningful feedback is visually inspecting the result. Does everything look as intended? Does it look good?

Fast feedback is still important for efficiency, so be sure to get the fasted possible feedback. If you are writing web applications, you are probably used to hot code reload in the browser, providing near-instantaneous feedback. What about desktop applications? Automated e-mail? PDF Generation?².

But for the vast majority of the code, a good TDD process provides the fastest possible meaningful feedback in the development process.

Do I have a good test suite?

You can find a lot of literature about which properties a test suite should exhibit. Much of it is BS, and much describes properties that are often observed in a good test suite, not the properties that make the test suite good.

In this article I have tried to describe, both through my own experiences, and by referencing videos with some of the pioneers of modern TDD³, that TDD is really about the feedback, allowing you to solve one small problem at a time, and facilitating the ability to refactor to patterns as you discover them.

So in essence, to identify if you have a good test suite, ask yourself the following two questions:

Do my tests provide fast feedback when implementing new behaviour?
Do my tests allow me to refactor safely?

If the answer to both questions is "yes", you have a good test suite.

Exchanges and bindings are part of the RabbitMQ infrastructure and controls how messages published are distributed to different queues owned by different consumes, respecting specific routing rules. ↩
In one project I needed to write code for PDF generation. I invested time into finding the tools that would automatically create and reload a PDF file in a viewer after each code change. It took some time, but the time saved by the fast feedback was worth it, allowing a fast iterative approach to writing the code. Again, the tool I used to write the code was my test framework, and once the PDF was correct, the output file was copied to an "expected" folder, used in snapshot testing to prevent a regression. I generally would advice against shapshot testing, but for this purpose it was perfect. ↩
Kent Beck only claims to have rediscovered TDD, not invented it. ↩

DEV Community: Peter Strøiman

Getting Started with Neovim

Learn modal editing

Experimenting with configuration

Learn From the Old Masters.

Use, Then Biuld

Why You Should Boycott VS Code

What is Open-source Software?

Distributing Source Code is not Open-source.

How Visual Studio Code Violates the Ethos

The "Open-source Theatre"

The Value of VSCode is it's Community.

To be fair

Conclusion

Using Go maps as a simple Fake repository

Sending an account validation email

Narrowing down the interface

Creating a fake repository for testing

Using the fake repository

Conclusion

The Beauty of Go, Introduction

How TDD reduces cost by reducing waste

Waste

How TDD reduces waste

Truncating an unproductive path

Reduce time debugging

Decrease unnecessary complexity

Conclusion

Appendix: Examples

Discovering the wrong assumption with GORM.

First time working with RabbitMQ

How Gost-DOM avoids making HTTP calls

The HTTP client

Implementing the interface

A slightly odd design decision

Adding cookie support

Testing identity provider integration

How Gost-DOM is implemented

I created a headless browser in Go. Here's what I learned

Eliminating the unknowns

Parsing HTML

Embedding and extending V8

Learning CGo

Go and C++ is like oil and water

Passing Go objects to C

A mistake was made

Implementing an Object-Oriented API

A solution. Was it the right one?

"You Know Nothing"

Attributes aren't just attributes

Where's the event loop?

Autogenerating Code

Cutting corners

The really cool part!

The state of Gost

What's up.

Please go use it

Create an auto-merging workflow on Github

Set the permissions

Create/update a verification workflow

About github workflows

Create a new workflow in the default branch.

Create a job

Fetch the code

Shallow Clones

Smarter handling of shallow clones

1. Remove ref and fetch-depth

2. Add new remote branch

3. Fetch the remote branch.

4. Check out the source branch

Push

The full workflow file

Pushing to the right branch locally

First sub-optimal solution

A less intrusive configuration

The local workflow

The git configuration is local

It's a simple solution to a simple problem

Go-DOM - 1st major milestone

A Brief History

1. Remove `ref` and `fetch-depth`