Learning TDD by doing: Acceptance testing with Umbraco's management API

Remember: 🔴 Red, 🟢 Green, ♻️ Refactor

In my previous post I speculated about end-to-end testing with Umbraco's management API. I have tried it out and I can confirm that it works! In this post I will dive into what I wanted to achieve, how I did it and what challenges I faced along the way.

The goal

My goal was to automate content scenarios in order to provide a consistent initial state for acceptance tests. The result should be a suite of tests that interact with the website like an end user would.

If you want to skip straight to the end result: check out my github repository with testing examples. I shared all my code there.

D-Inventor / automated-testing-in-umbraco

A working example of integration- and unittests with Umbraco. A demonstration of various concepts for testing your Umbraco website

Umbraco 16 automated testing setup

This project is a fully functioning setup for automated testing with Umbraco 16. You can use this project as a reference or starting point to get started with testing on your Umbraco website. The tests are set up with Test Driven Development (TDD) in mind.

Tools

The most important tools that are used in the automated tests are as follows:

Name	Description
xUnit v3	The testing framework. You can use any testing framework that you like though
NSubstitute	Library for mocking. Any mocking library will work. This project doesn't do extensive mocking, but for example IPublishedValueFallback is a mandatory parameter for any published content item, even if you don't actually use it. It's just convenient to insert a mock.
Test Containers	Automatically creates docker containers while running tests. It is used to create an empty SQL Server database that is automatically cleaned up after testing.

…

View on GitHub

The journey

There were two steps to my plan: First, I created a client that could communicate with Umbraco's management API. Then, I built an interface on top of the client, so that I could describe the different kinds of content in easy-to-use models. I also created a small website to create a realistic test-case to use these tests on.

Step 1: Management API Client

Umbraco's management API is the interface that the Umbraco backoffice communicates with when you edit content. It is built with external integrations in mind.

To create the API client, I used NSwagStudio. With this tool, I can consume the OpenAPI spec that Umbraco exposes and automatically generate a C# client. Although it didn't go perfectly, the generated client was very usable and Umbraco appears to have designed very nice and clear models and endpoints. The generated C# code couldn't compile right away, because for some reason it depended on a type FileResponse, which doesn't exist. Fortunately, creating an empty class was enough to make it work.

NSwagStudio does not generate the authentication code, so I had to extend the generated client to include authentication. This was easy enough with Umbraco's documentation on external access. After this, I was able to write a crude script that could access the backoffice and produce some basic content items.

Step 2: A layer of convenience

The client works, but preparing a scenario using the Management API Client alone takes a lot of code. My goal was to be able to describe a scenario in code the same way as I would describe it to a coworker. So before I got to work, I wrote down some example sentences that I would use to describe a test scenario:

• Given a content page with a tag, when I go to the page, I see related content with the same tag.
• Given a content page with the content from our figma design, when I go to the page, I see the page just like in our design
• Given a news overview with many news articles, when I load more articles, I see the next N articles

I designed a test that would describe a simple scenario:

[Fact]
public void ShouldDisplayTitleFromContent()
{
    // given
    var contentPage = Scenario
        .WithContentPage()
        .WithHeader(title: "My content page");

    // when
    User.Visits(contentPage);

    // then
    Assert.Equal("My content page", CurrentPage.Title);
}

The focus here was to get the 'given' part working. I used test-driven development to design and build this scenario builder. The tests that I wrote are included in the github repository, so you can have a look at those if it interests you. The fun thing about test-driven development is that a test makes me think about how I'm going to use the code that I'm about to write.

In the end, I didn't get exactly the design that I wanted, simply because "I want content with a header" is too vague, but I did get close. The test above ended up looking like this:

[Fact]
public async Task ShouldDisplayTitleFromContent()
{
    // given
    Scenario.Homepage()
        .HasContent(Variation.Invariant, content => content.WithHeader(title: "Welcome to the website"));
    await Scenario.BuildAsync(TestContext.Current.CancellationToken);
    var homepage = new HomePageObject(await Browser.NewPageAsync(), Scenario.Website());

    // when
    await homepage.GoToAsync();

    // then
    await Expect(homepage.Title).ToHaveTextAsync("Welcome to the website");
}

Instead of saying "The homepage has a header", it now says "The invariant content of the homepage has a header". It's a bit longer, but also more explicit. It also allows you to specify cultures and variants. Very useful when you want to test multilingual sites or if you use segments to personalize content. Additionally, I couldn't get around an explicit call to BuildAsync, because I need a trigger to actually make the changes.

Challenges / difficulties

Here are some of the things that surprised me, some things that I found difficult and some roadblocks that I came across while developing this.

Visual regression testing? Yes, but not in dotnet

The biggest challenge and also my largest surprise, was the fact that there don't seem to be any visual regression testing tools available for dotnet. Playwright for example supports comparing screenshots in the version for javascript / typescript, but does not support this for its dotnet version. I had no plans to create my own visual regression testing tool, so I had to give up. It might have been better to create my scenario builder in javascript / typescript, in retrospect.

This was somewhat disappointing for me. In my team at work, it is very important that our websites look like the designs that we make upfront. One of our success criteria is that we can recreate the design using content in Umbraco. Being able to match a browser screenshot with an export from our design software would have been very valuable.

Domains in Umbraco didn't quite work

Just before writing this blog, I submitted a bug report to Umbraco, because it appears that Umbraco has a stale caching layer on top of their domain service. When I ran the tests more than once or on multiple scenario's, I would run into bad status codes because the domain was reserved for the content of the previous scenario, even though that content was deleted.

I worked around this issue by creating my own custom endpoint in the management api. A sort of "delete everything" endpoint that deletes all the content in the content tree and makes sure that domains are deleted explicitly first. Although it's not ideal, it's a workable solution and shows how flexible the system as a whole is.

Content is inherently complex

My example on GitHub and the example code above is simple on purpose. String values are easy to model. But how do you model related media? Images with crops? What about a blocklist? These types of content are inherently complex and there is no single best answer. Blocklists have quite complex rules, take this article in the Umbraco documentation about block-level-variance for example. Media has its own section in the backoffice and introduces an additional external dependency: the filesystem. Do I need to recreate all media items on every test as well? Or maybe I can simply reserve a set of media items in the backoffice and reuse them? It really depends on your use-case and I haven't quite figured this out for myself yet.

Layers and abstractions

I found it difficult to assign clear responsibilities between the ScenarioBuilder and the DocumentClient. I had a gut-feeling that somewhere here is supposed to be a "repository". Intuitively, the ScenarioBuilder seemed to play the role of repository, because it translated domain models into DTOs for the management api, but after building everything, the IDocumentClient ended up looking more like a repository than the ScenarioBuilder. In the end, I couldn't really make a proper case for either, so neither of these types turned into a repository. This once again shows how test-driven development helps you make better design choices. I didn't need a repository, so I didn't make one.

final thoughts

Most of all, I'm very excited that I managed to make it work. I have to admit that without visual regression testing, I'm not exactly sure how I'm going to use this technology. That doesn't mean that it's useless though. I feel like there is a lot of potential and I am very happy with how easy everything turned out to be.

Please check out my repository in GitHub with testing examples. It includes everything that I talked about in this post. Let me know your thoughts: do you have extensive test suites for your Umbraco sites? What kind of tests do you find most valuable?

Thank you for reading and I'll see you in my next blog! 😊