DEV Community: Joost van Wollingen

Alert on Test Results Easily With Alerticorn 0.1

Joost van Wollingen — Wed, 19 Feb 2025 21:35:05 +0000

Sometimes, you want to push test results to your team chat automatically.

For example, in my current job, we associate sections of E2E tests with specific teams. When a test related to their functionality fails, they need to be notified so they can take a look.

Sometimes I come across a test that fails under unknown circumstances, and I just can’t reproduce it right now. But I want to be alerted when it happens again, so I can investigate.

With these usecases in mind (and mostly as a fun hobby project), I built Alerticorn — a lightweight framework that makes sending notifications to team chats as easy as possible.

How Alerticorn Works

Currently, Alerticorn extends JUnit 5, listening to results from test classes and test methods. If necessary, sends an alert to Slack, Teams or Discord.

With the right configuration in place, sending a chat message is as simple as this:

@ExtendWith(MessageExtension::class)
class MyTest {

  @Message(title = "The flux capacitator failed")
  @Test
  fun `flux capacitator test`() {
    // your test
  }
}

Messages can be:

Routed to different platforms and channels
Customised with more information
Generated with details from thrown exceptions
Filtered based on the outcome of the test

What’s Next? TestNG and Kotest Support

Currently, Alerticorn supports JUnit 5, but I plan to add support for TestNG and Kotest , making it useful regardless of your test framework.

Directly calling RunWith.message is already supported, offering more flexibility for different use cases.

The Philosophy Behind Alerticorn

Alerticorn is built with a few core principles in mind:

Convention over configuration — The defaults cover most use cases, but you can configure it fully if needed.
No unnecessary dependencies — It’s designed to be lightweight.
Extensibility — A plug-in system allows easy integration with other chat platforms or test runners.
Learning and fun — This project has been an excellent way to explore Kotlin, ServiceLoader, AI-assisted coding, Javadoc/KDoc, technical documentation (Docusaurus), Maven Central publishing, GitLab Actions, and more.

Curious?

Do you want to try it out? See a use case for it? Or maybe you think it’s a silly idea? Either way, I’d love to hear your thoughts in the comments.

In the meantime, you can find the documentation and source code on GitHub.

Building a Test Automation Library to Create and Manipulate Test Data

Joost van Wollingen — Thu, 30 May 2024 20:42:57 +0000

The past 4 years I’ve worked on a test automation library that is primarily used for end-to-end test automation, with some additional uses. In that time it has proven to be a very flexible, easily maintained and extensible piece of software. In this article I’ll share some of the underlying concepts and considerations we’ve used to build the library.

Goals — or “How Should This Thing Work?”

Before setting up this library, there had been some earlier work done that was mostly pieced together JSON snippets, with text concatenation all over the place. Writing a new test meant copy/pasting tons of JSON every time. To top it off, it was written in an tool and language our developers had no knowledge of. Obviously, they wanted no part of it. It was not something I felt was fit to establish a strong foundation of test automation on top of.

Goals we set for the library included:

It should hide complexity when unnecessary
It should give control when required
It should be easy to use, and easy to contribute to

What Are We Automating Again?

The intended use for this library was twofold:

Use it to automate tests against the public facing API of our platform
Make it easy to reach certain states with the data it creates

Our public API is a highly configurable one. Each consumer will call the same endpoints, but based on the configuration the required inputs can be very different. Endpoints must be called in a certain order and data most be provided consistently across these endpoints. For example: if you provide an address in call #1 it must be in the same country you provide in call #2.

Completing the series of calls results in an entity being created, which has a complicated lifecycle, with different states, transitions and transition guards. The library should make it easy to create that initial entity, but also modify the state of the entity as desired.

Hiding Complexity

As described above, providing valid inputs to the API across multiple calls requires quite some specific domain and product knowledge. That’s fine if you care about testing the particulars of one set of configuration, but if you simply need an entity of type X to test something else elsewhere on the platform you might not be too enthusiastic about figuring out all those details.

This is what I mean with hiding complexity, the library should provide its users an easy way to create complex data, without them having to know what data results in a valid entity, nor how to orchestrate the calls that are required.

This is where we leverage Kotlin’s concrete interfaces, providing a blueprint for the process and for the pieces of data that must be provided.

The Product interface provides a default implementation for createShoppingCart, while the classes implementing provide specific, valid data through the interface methods.

Sensible Defaults

Creating tests for a new product simply means implementing the methods that provide the data for a valid policy. All that remains to create that policy is calling a single line:

motor.createShoppingCart()

Using Datafaker, we provide localized and randomized data. Akin to James Bach’s galumphing, this helped us to detect issues with inputs “that shouldn’t make a difference” more than once.

Need something specific?

The setup with the Product interface is flexible enough to allow users to create very specific policies where needed, falling back to defaults for values that aren’t specified.

val verySpecificRiskAnswers = setOf(Answer("ALLRISK", "YES"))

motor.createShoppingCart(riskAnswers = verySpecificRiskAnswers)

Mind Your Consumer

With this design for the test data library, we allow users to easily create randomized, valid test data with a one-line statement. No knowledge is required of the underlying API’s nor what data is required for a valid policy. At least…as long as they want to keep away from that complexity. The library is flexible enough to grant that control, and specify every input in detail.

Usage

The Test Data Library powers different types of test and a service

The test data library is used to power both E2E tests and system tests on integrated environments.

Next to test engineers and developers, also business stakeholders often need test data to work with. You’ll probably be familiar with requests on Slack for data like X, Y or Z, and it might be cumbersome to create it manually every time.

For this specific purpose we’ve build a web application with a thin UI layer that allows business stakeholders to easily create test data right from their browser. If they don’t have any specific requirements the sane defaults in the library make this a one click operation. If they have more specific demands, the UI allows for the same level of control as at the library level.

Results

The library has helped us to stay on top of an ever growing number of products, maintaining tests and rolling out changes with ease. A common change only has to be implemented in the Product interface, and all tests automatically follow the updated process.

Adding a new product can be done in a matter of minutes and it’s very clear what information is needed to be able to start testing.

The main thing to keep in mind when working on the library is that there are several consumers, and that the API of the library should remain stable. Breaking changes may complicate the upgrade path of consumers.

As the platform evolves and newer versions of API’s are adopted by production services it’s also important to keep the clients in the library in sync with new endpoints.

Meet the Captain

Joost van Wollingen — Sat, 27 Jan 2024 22:52:43 +0000

Since I’m no artist , we’re starting from AI generated images, and from there try to modify them to fit the game. Today, I started on a first draft for a character: The Captain.

First I generated some portraits of “seasoned navy captains, with a scar, smoking a cigar, cartoony”. I’ll have to up my prompt engineering in the coming time.

This is what the Captain could look like according to AI

I threw the portrait I liked most into aseprite, a sprite editor & pixel art tool. It came up as a good tool to do the graphics in for our kind of game, and I got myself a license through Steam.

My first goal was to take the AI generated still, and make a speech animation. I grabbed the Captain’s mustache and lowerlip, added a frame where these part and painted in some basic teeth. For a draft, in a tool I never used before, it’s good enough for now.

More mustache than man

Next, I wanted to simulate some sort of scanlines, for the videocall sequences where we’d use these portraits. I’ve probably overcomplicated things, used too many layers and animated it clumsily, but the end result I was pretty happy with. Reminds me of those old, green, monochrome monitors.

The Captain uses some old naval technology for video conferencing.

Finally, I tried my hand at an overlay for the portrait + scanlines, to have somewhat of a border around it, and tried some more animations, such as the blinking light and the scrolling letters. That’s far too clumsy to do neatly in aseprite as far as I can tell for now.

Here’s the Captain on the phone with you, giving orders.

Doing these in aseprite is a lot of fun, although I should probably take some time to study the tool and read the documentation. For now, I’ve been going by experience from other graphical editors I’ve worked with throughout the years. We’ll get there!

Surfacing

Joost van Wollingen — Sat, 27 Jan 2024 08:22:42 +0000

Lightning cracks, thick rain drops fall down relentlessly, making it almost impossible to see outside. The wind howls around the bridge, rolling the ship in the giant waves. We’re being hunted.

Building in Public

It’s a game we’ve dreamed about for years now. We never started development, but that changes today. At this stage, it’s not much more than a few loosely related thoughts, ideas and experiments we want to run. There is a million things to learn, such as Golang, Ebitenengine, game design, sound design, music, graphics, aseprite, entity component system, artificial intelligence.

This blog will be used to document our journey, the things we learn along the way, and the progress the game is making. We’ll be building in public, to keep us on our toes.

The Game

In the game the player will pilot a destroy-class ship, with a unique, challenging control-scheme. Fights will take place in the air and with submerged enemies. You can expect updates as we refine our thinking around the game, its design and gameplay loop.

The Latest

We publish every build at the following url, fully playable in your browser thanks to WASM !

Play the game at https://ubootgame.github.io/ubootgame/

jq for testers

Joost van Wollingen — Wed, 24 Jan 2024 08:46:56 +0000

As test automation engineers we often touch many different tools, protocols, data, and systems. Whatever we need to get the job done. Wrangling data can be quite burdensome, for example when preparing data for a next test or to verify expectations against data retrieved. Data can be in the wrong format, have additional unwanted fields, lack fields, or the receiving system expects a different structure.

Nowadays a lot of the data that we deal with is JSON. Simple enough to deal with and widely supported by tons of tools. It gets annoying when some of the issues above pop-up and you want to change the data for a lot of tests, or it’s just a lot of data.

Here is where jq can help. Jq is an open source command line tool, that makes it easy to deal with json in all kinds of scenarios. Below I’m sharing a few of the most useful applications I’ve come across for jq.

For the examples in this article I’m using data and endpoints from https://fakestoreapi.com/.

Pretty print & color json

Often systems return minified json, without white space and line breaks. While a bit more efficient to transmit, it makes it a lot harder to read for humans. Jq can help. Given a piece of json on your clipboard simply run:

pbpaste | jq

If you pass in JSON to the jq command it will automatically pretty-print and color the input. Nice!

Select only one field from an array of objects

https://fakestoreapi.com/users returns a list of users and their details. To grab only the address fields of each entry:

curl https://fakestoreapi.com/users | jq '.[].address'

Slice the data

In order to grab the first 3 elements from an array you’d run:

curl https://fakestoreapi.com/users | jq '.[0:3]'

Construct a new object

Sometimes you want to shuffle the data you have into another structure. jq again makes this easy.

 curl https://fakestoreapi.com/users | jq '.[] | {user: "\(.name.firstname) \(.name.lastname)", phone: .phone}'

Count

Calling the length function gets you the number of items in an array or the number of key-value pairs in an object. Very useful if you want to run some quick verifications on the data.

curl https://fakestoreapi.com/users | jq length

Get a list of all users whose e-mail address contains ‘w’

curl https://fakestoreapi.com/users|jq 'map(select(.email | contains("w")))'

Count all users with a ‘w’ in their e-mail address

curl https://fakestoreapi.com/users|jq 'map(select(.email | contains("w")))' | jq length

There’s a lot more you can do with jq, check out the documentation for much, much more you can do with it.

Learn the fundamentals of Test Automation, in my course on Pluralsight- Test Automation: The Big Picture(affiliate link).

Catching java.io.IOException: Broken pipe with the help of toxiproxy

Joost van Wollingen — Sun, 21 Jan 2024 15:33:31 +0000

Recently I’ve moved into a new team, which is responsible for the front- and backend of a backoffice application. The alerting for our applications is configured to trigger whenever log lines come in at the error level.

As the backend was throwing a lot of java.io.IOException: Broken pipe exceptions I started to investigate. My team members shared that this error was thrown whenever users navigate away from a page, while the backend is still transmitting data to them.

Observability had not received a lot of attention for this application and the signal to noise ratio in our logs and alerts was very low. I decided that we needed to stop logging this error, especially because this alert was not actionable: it is not recoverable, and there is nothing actually functionally wrong.

I started setting up a testbed to reproduce the hypothesis that this exception occurred when the client hangs up unexpectedly.

I decided to go with the locally running service and toxiproxy. Toxiproxy is an open source project that allows you to sit between client and server and introduce network related issues such as latency, and limited bandwidth.

Installing toxiproxy

Toxiproxy works with a server component and a cli to configure said server. Installing it is easy enough, simply run:

brew install toxiproxy

Setting up the toxic

In our case first I set up the toxiproxy server to listen on port 4444 and forward any requests to my application at port 8080. This configuration was named “connectionreset”.

Next we added a toxic to this proxy, to limit the data after receiving 1 byte.

toxiproxy-server &
toxiproxy-cli create -listen localhost:4444 -upstream localhost:8080 connectionreset
toxiproxy-cli toxic add --type limit_data -a bytes=1 connectionreset

Reproducing the broken pipe exception

The plan was to call the service with curl through toxiproxy. On the first attempt the service logged a broken pipe exception, perfect! I started implementing a method to catch this exception and lower the log level. For this we have a RestExceptionHandler class, annotated with @_ControllerAdvice_. Any exceptions at the API level are handled centrally by that class.

Because we still want to know if this happens a lot, I also created a metric, which we’d increment every time we caught a broken pipe. That way, if we’d still be able to see if an abnormal amount of exceptions started to occur.

Then I rebooted the server and tried to call again with another endpoint. Nothing! Not entirely what we expected, as at least there should’ve been an INFO log and an increased metric. Hmm, what’s going on here? The original endpoint I tested with still responded as I expected.

My first hunch was that somehow the two endpoint controllers were different, and/or the controller advice wasn’t being picked up by one of them. But because I had no trouble with other exceptions handled by the rest exception handler that couldn’t be it. After investigating with Mykola, we figured out the size of the response mattered. Although toxiproxy would kill the connection after receiving the first byte, for small responses the server would’ve already finished sending all the data and not log any errors.

With this we were satisfied and the test setup with toxiproxy proved to be very useful:

It allowed me to induce the fault condition
It allowed me to explore the situations in which the exception was thrown — all from my local machine
It helped me in verifying the changes to our application to handle broken pipe exceptions more gracefully

Reading material

Here are some of the materials that helped me figure this out.

My Pluralsight course — Test Automation: The Big Picture(affiliate link),covers why fast feedback is so important and what types of test automation you can expect to find out there.

Evolution of Test Automation for a Monolith

Joost van Wollingen — Tue, 02 Jan 2024 22:06:16 +0000

The monolith. Dreaded for its sluggishness, tangledness and license costs. Mongering fear in the hearts of developers, who tame it through incantations long lost to the void of attrition. The call for new features and profits overshadowing the long lasting desire to replace it. Its expensive maintenance delegated to consulting shamans, whose native development rituals make little sense to modern man.

A menacing monolith. Generated via https://hotpot.ai/art-generator

Deciphering its glyphs, marks and forgotten functionalities we have discovered ways to tolerate its dull presence in our microservices landscape, where heralds sound the progress of technocracy.

As we continuously push forward, the monolith slowly withers in its corner, remembering the attention it would revel in during the Platform’s early days. Slowly smothered, deconstructed and derelict, it becomes an empty husk, which no longer commands but is instructed. As its decommissioning lurks on the horizon, we review a few test automation patterns applied throughout the years.

Out with the Old, in with the New

Initially the testing team of external consultants was tasked with training us in the Ways of the Monolith. SoapUI was the main tool used, with the code written partially in a foreign language, and long, long, long if-else trees and string concatenation of all over the place. The tests focused on the Monolith, mostly ignoring the New Platform. Clearly, this would not do.

With Kotlin being the main language at our company, and the ambition to have more involvement of developers in test automation, it was an easy choice. We felt a low entry barrier for contributions by developers was important to create a context where ownership of test automation is shared across test and development teams. With Kotlins interop with Java, we’d have access to the full array of test libraries from the Java ecosystem and still get to enjoy the benefits Kotlin brings in terms of conciseness and syntax.

Manual Operation Required

Some of the old tests depended on batch jobs to be ran…manually. After the job completed, you could start the next part of the ‘automated’ test. Working with the team managing the Monolith we figured out how to run the batch jobs from code, speeding up the tests and removing the manual element completely. A nice reminder to not settle for mediocre testability in your applications and test automation.

Atomic Tests Blew up the System

Atomic tests (of the non-exploding kind) can help you pin-point faster and with more accuracy what the problem is with your SUT. In our case we set out enthusiastically to increase coverage and write more tests. This worked great for the New Platform side of things, whizzing along quickly, creating all the entities we needed for our tests. But the Monolith choked. Queues would fill up, database transactions would get into deadlock, test verifications would timeout and application server instances would disappear as soon as we started increasing the amount of data created by tests.

Anti Pattern: Non Atomic Tests

We needed a way to get around the unreliability of the Monolith. Knowing full well that we were simply trading our current problems for a new set of problems, we decided to drop atomic tests in favour of non atomic tests. Instead of creating fresh test data for every operation we’d want to test, we’d create test data once and re-use it for a chain of operations. This way we could alleviate the bottleneck, still get test (some) test results and keep moving forward. There’s a few issues with this approach:

A failed test means that you won’t have results for all the tests that would’ve run after that test.
Failing tests mean that you have to investigate not one but many operations to figure out what is wrong. This increased the load on the team, or Test Engineer on Duty. It takes more time to determine the issue.
Separate parts of the test really need to be separate. If you need data in a specific state for a specific operation you may have to order the tests or create a new test all together after all. Which then increases the pressure on the bottleneck again.

Having identified all of this upfront we still decided to move on and to transpose the test set to atomic tests again as soon as it was possible.

Comparing Notes

During the strangling of the Monolith there will be a period of time where both the Monolith and the Platform operate together. While extracting functionalities to new applications (part of) the functionality lives on in the Monolith, and production will still depend on the monolithic part functioning. In our case copies of entities were kept in both the New Platform and the Monolith. One strategy we applied here, was to ‘compare notes’ for each test. An entity created in the New Platform must look the same in the Monolith. Any difference should be investigated, as our migration strategy dictated consistency across both.

Unlocking Faster Feedback

As we edge closer to decommissioning the Monolith, less and less features depend on it. However, by keeping it in scope for our automated tests we suffer a penalty: slow feedback. Test runs including the Monolith take about 30 mins, and still suffer from unreliable results. Again, there is a trade off to be made.

As the New Platform is now the main driver of most major processes, feedback on the New Platform has become more important than feedback on the Monolith. Our test set did not make that distinction, and any test run would verify both the New Platform and the Monolith.

In order to have faster feedback, we made our tests configurable. An environment variable now determines what parts of our non-atomic tests concerning the Monolith can be skipped. This reduced the test lead time to ~5 minutes. Tests with the Monolith included can still be run on demand. A next step is to return to atomic tests and have even more manageable tests.

It ain’t pretty but it works

Of course, some of the things described above come with a certain risk. Risk of spending too much time on inspecting flaky tests. Risk of the Monolith failing and us consciously not running the tests for it. Reality is that we don’t always get to work with perfect systems. And that’s ok. Please just don’t apply any of this blindly. Always weigh your options, in relation to your goals and risk appetite. The only reason this worked in my context is because of:

A great team of test engineers, with the culture to meticulously check failing tests.
A shift-left test approach, where a lot of the tests are executed early on.

If you want to get started with test automation, check out my Pluralsight course — Test Automation: The Big Picture, it’s a perfect starting point! (affiliate link)

Journalbeat - Error while reading event: failed to get realtime timestamp: 99

Joost van Wollingen — Fri, 03 Apr 2020 09:29:05 +0000

Recently I was setting up an instance of Journalbeat to publish journald logging to our ELK stack. I wanted to publish only the logs for a select set of applications, so I was using the following configuration.

- paths: []
seek: cursor
cursor_seek_fallback: tail
include_matches:
- "systemd.unit=name_of_application" #this will not work!
- "systemd.unit=another_application" #this will not work!

In this case, upon restarting Journalbeat, it kept repeating the following error in its logs:

Error while reading event: failed to get realtime timestamp: 99

If I removed the “include_matches”-section everything worked just fine. In the end, I found the answer in this Github issue: use the full name of the systemd unit, including .service.

- paths: []
seek: cursor
cursor_seek_fallback: tail
include_matches:
- "systemd.unit=name_of_application.service"
- "systemd.unit=another_application.service"

Copy text from your VSCode terminal by selecting it

Joost van Wollingen — Wed, 05 Feb 2020 14:08:05 +0000

If you are used to working on a terminal a lot you’ve probably come to love the feature to copy the text you select to your clipboard. Recently I’ve started using VSCode more and more and often ran into the issue that my muscle memory tricked me into selecting some text followed by quickly pasting and executing something else entirely because VSCode’s terminal wasn't configured to copy on select. Turns out this feature has been in VSCode since 2018, you just have to enable it.

Go to settings (⌘,)
Search for terminal.integrated.copyOnSelection
Check the checkmark

Happy copy/pasting!

How my mistake made the build 20 minutes slower

Joost van Wollingen — Wed, 05 Feb 2020 11:40:32 +0000

In order to deploy the same version of your code on all environments, it is important to create immutable artifacts of your application and only do deployments using those artifacts. The artifacts should be created by your CI server, to ensure a consistent environment is used to create the artifacts every time. That’s exactly what I was doing the other day for an application consisting out of more than 15 components. Each component needed to be built in Azure DevOps and subsequently published as a Build Artifact.

The initial version of the pipeline ran over 25 minutes to publish the 15 or so artifacts: NodeJS applications including their node_modules. The task taking the most time was the PublishBuildArtifacts task, some applications took 20 minutes for that step alone — really slow!.

24 minutes is definitely too long for a pipeline!

Of course, this operation should be much faster, so I deducted that I wrongly assumed that the PublishBuildArtifacts task would create a ZIP file and than upload that as the artifact. Instead, I think, the task takes every individual file in the folder mentioned and uploads them one by one.

By adding the ArchiveFiles task the running time for the build was reduced to a total of 1 minute 30 seconds. The longest publish task is now taking ~20 seconds, down from 20+ minutes! All by adding a single task that creates a ZIP file first. Find a snippet below to get started.

https://medium.com/media/8d030f5dd409ada653170c6f56a2625c/href