DEV Community: Oleksii Holub

Prefer Fakes Over Mocks

Oleksii Holub — Tue, 13 Oct 2020 11:16:01 +0000

The primary purpose of software testing is to detect any potential defects in a program before it reaches its intended consumers. This is typically achieved by establishing functional requirements which define supported user interactions as well as expected outcomes, and then validating them using (automated) tests.

Consequentially, the value provided by such tests is directly dependent on how well the scenarios they simulate resemble the way the software is actually used. Any deviation therein diminishes that value, as it becomes harder to reason about the state of the would-be production system based on the result of a test run.

In an ideal world, our test scenarios, including the environment they execute in, should perfectly match real-life conditions. This is always desirable, but might not always be practical, as the system can rely on components that are difficult to test with, either because they are not available or because their behavior is inconsistent or slow.

A common practice in cases like this is to replace such dependencies with lightweight substitutes that act as test doubles. Although doing so does lead to lower confidence, it's often an unavoidable trade-off when it comes to designing a robust and consistent test suite.

That said, while test doubles can be implemented in different ways, most of the time developers tend to resort to mocking as the default choice. This is unfortunate, as it leads to overuse of mocks where other forms of substitutes are typically more suitable, making tests implementation-aware and fragile as a result.

When writing tests, I prefer to avoid mocks as much as possible and rely on fake implementations instead. They require a bit of additional upfront investment, but provide many practical advantages which are important to consider.

In this article we will look at the differences between these two variants of test doubles, identify how using one over the other impacts test design, and why using fakes often results in more manageable test suites.

Drawbacks of mocking

As we enter the realm of software terminology, words slowly start to lose their meaning. Testing jargon is exceptionally notorious in this regard, as it always seems to create a lot of uncertainty among developers.

Unsurprisingly, the concept of "mock" or how it's fundamentally different from other types of substitutes is also one of those cases. Despite its highly ubiquitous usage, this term doesn't have a single universally accepted interpretation.

According to the original definitions introduced by Gerard Meszaros, a mock is a very specific type of substitutes which is used to verify interactions between the system under test and its dependencies. Nowadays, however, the distinction has become a bit blurry, as this term is commonly used to refer to a broader class of objects created with frameworks such as Moq, Mockito, Jest, and others.

Such substitutes may not necessarily be mocks under the original definition, but there's very little benefit in acknowledging these technicalities. So to make matters simpler, we will stick to this more colloquial understanding of the term throughout the article.

Generally speaking, a mock is a substitute, that pretends to function like its real counterpart, but returns predefined responses instead. From a structural standpoint, it does implement the same external interface as the actual component, however that implementation is entirely superficial.

In fact, a mock is not intended to have valid functionality at all. Its purpose is rather to mimic the outcomes of various operations, so that the system under test exercises the behavior required by a given scenario.

Besides that, mocks can also be used to verify side-effects that take place within the system. This is achieved by recording method calls and checking if the number of times they appear and their parameters match the expectations.

Let's take a look at how all of this works in practice. As an example, imagine that we're building a system that relies on some binary file storage represented by the following interface:

public interface IBlobStorage
{
    Task<Stream> ReadFileAsync(string fileName);

    Task DownloadFileAsync(string fileName, string outputFilePath);

    Task UploadFileAsync(string fileName, Stream stream);

    Task UploadManyFilesAsync(IReadOnlyDictionary<string, Stream> fileNameStreamMap);
}

As we can see, it provides basic operations to read and upload files, as well as a few more specialized methods. The actual implementation of the above abstraction does not concern us, but for the sake of complexity we can pretend that it relies on some expensive cloud vendor and doesn't lend itself well for testing.

Built on top of it, we also have another component which is responsible for loading and saving text documents:

public class DocumentManager
{
    private readonly IBlobStorage _storage;

    public DocumentManager(IBlobStorage storage) =>
        _storage = storage;

    private static string GetFileName(string documentName) =>
        $"docs/{documentName}";

    public async Task<string> GetDocumentAsync(string documentName)
    {
        var fileName = GetFileName(documentName);

        await using var stream = await _storage.ReadFileAsync(fileName);
        using var streamReader = new StreamReader(stream);

        return await streamReader.ReadToEndAsync();
    }

    public async Task SaveDocumentAsync(string documentName, string content)
    {
        var fileName = GetFileName(documentName);

        var data = Encoding.UTF8.GetBytes(content);
        await using var stream = new MemoryStream(data);

        await _storage.UploadFileAsync(fileName, stream);
    }
}

This class gives us an abstraction over raw file access and exposes methods that work with text content directly. Its implementation is not particularly complex, but let's imagine we want to test it anyway.

As we've identified earlier, using the real implementation of IBlobStorage in our tests would be troublesome, so we have to resort to test doubles. One of the simpler ways to approach this is, of course, by creating mock implementations:

[Fact]
public async Task I_can_get_the_content_of_an_existing_document()
{
    // Arrange
    await using var documentStream = new MemoryStream(
        new byte[] {0x68, 0x65, 0x6c, 0x6c, 0x6f}
    );

    var blobStorage = Mock.Of<IBlobStorage>();

    Mock.Get(blobStorage)
        .Setup(bs => bs.ReadFileAsync("docs/test.txt"))
        .ReturnsAsync(documentStream);

    var documentManager = new DocumentManager(blobStorage);

    // Act
    var content = await documentManager.GetDocumentAsync("test.txt");

    // Assert
    content.Should().Be("hello");
}

[Fact]
public async Task I_can_update_the_content_of_a_document()
{
    // Arrange
    var blobStorage = Mock.Of<IBlobStorage>();
    var documentManager = new DocumentManager(blobStorage);

    // Act
    await documentManager.SaveDocumentAsync("test.txt", "hello");

    // Assert
    Mock.Get(blobStorage).Verify(bs => bs.UploadFileAsync(
        "docs/test.txt",
        It.Is<Stream>(s => /* stream verification */)
    ));
}

In the above code snippet, the first test attempts to verify that the consumer can retrieve a document, given it already exists in the storage. To facilitate this precondition, we configure the mock in such way that it returns a hardcoded byte stream when ReadFileAsync() is called with the expected file name.

However, in doing so, we are inadvertently making a few very strong assumptions about how DocumentManager works under the hood. Namely, we assume that:

Calling GetDocumentAsync() in turn calls ReadFileAsync()
File name is formed by prepending docs/ to the name of the document

These specifics may be true now, but they can easily change in the future. For example, it's not a stretch to imagine that we may decide to store files under a different path or replace the call to ReadFileAsync() with DownloadFileAsync(), as a means to preemptively cache files locally.

In both cases, the changes in the implementation won't be observable from the user perspective as the surface-level behavior will remain the same. However, because the test we wrote relies on internal details of the system, it will start failing, indicating that there's an error in our code, when in reality there isn't.

The second scenario works a bit differently, but also suffers from the same issue. To verify that a document is correctly persisted in the storage when it gets saved, it checks that a call to UploadFileAsync() takes place in the process.

Again, it's not hard to imagine a situation where the underlying implementation can change in way that breaks this test. For example, we may decide to optimize the behavior slightly by not uploading the documents straight away, but instead keeping them in memory and sending in batches using UploadManyFilesAsync().

An experienced mocking practitioner might argue that some of these shortcomings can be mitigated if we configure our mocks to be less strict. In this instance, we can modify the test so it expects a call to any of the upload methods rather than a specific one, while also not checking the parameters at all:

[Fact]
public async Task I_can_update_the_content_of_a_document()
{
    // Arrange
    var eitherUploadMethodCalled = false;

    var blobStorage = Mock.Of<IBlobStorage>();

    Mock.Get(blobStorage).Setup(bs => bs.UploadFileAsync(
        It.IsAny<string>(), // any parameter -> OK
        It.IsAny<Stream>()  // any parameter -> OK
    )).Callback(() => eitherUploadMethodCalled = true);

    Mock.Get(blobStorage).Setup(bs => bs.UploadManyFilesAsync(
        // any parameter -> OK
        It.IsAny<IReadOnlyDictionary<string, Stream>>()
    )).Callback(() => eitherUploadMethodCalled = true);

    var documentManager = new DocumentManager(blobStorage);

    // Act
    await documentManager.SaveDocumentAsync("test.txt", "hello");

    // Assert
    eitherUploadMethodCalled.Should().BeTrue();
}

The mocking framework we're using (Moq) doesn't allow us to directly verify that either one of the given methods was called, so we need a workaround. To do this, we inject a callback that sets the value of a variable to true and then use it to check the outcome accordingly.

As you can probably see, this change increased the complexity of the test rather significantly, as suddenly we find ourselves dealing with some additional state and a much more involved mocking setup. It also became less clear what exactly is it that we're trying to verify or whether we're doing it correctly, making the whole scenario harder to reason about.

Even despite all that effort, this test is still not as resilient as we would've wanted. For example, adding another method to IBlobStorage and calling it from DocumentManager will cause the test to break as the mock wasn't previously taught how to deal with it. You can see how all of these issues and complexity can only extrapolate poorly in real projects with large test suites.

One way or another, tests that rely on mocks are inherently coupled to the implementation of the system and are fragile as the result. This does not only impose an additional maintenance cost, as such tests need to be constantly updated, but makes them considerably less valuable as well.

Instead of providing us with a safety net in the face of potential regressions, they lock us into an existing implementation and discourage evolution. Because of that, introducing substantial changes and refactoring code becomes a much more difficult and ultimately discouraging experience.

Behavioral testing with fakes

Logically, in order to avoid strong coupling between tests and the underlying implementations, our test doubles need to be completely independent from the scenarios they're used in. As you can probably guess, that's exactly where fakes come in.

In essence, a fake is a substitute that represents a lightweight but otherwise completely functional alternative to its real counterpart. Instead of merely trying to fulfil the contract with preconfigured responses, it provides an actually valid end-to-end implementation.

Although its functionality resembles that of the real component, a fake implementation is intentionally made simpler by taking certain shortcuts. For example, rather than relying on a remote database server, the fake can be programmed to use an in-memory provider instead. This makes it more accessible in testing, while retaining most of its core behavior.

In contrast to mocks, fakes are usually not created in runtime via dynamic proxies, but defined statically like other regular types. While it is technically possible to generate a fake implementation using mocking frameworks as well, there are rarely any benefits in doing so.

Now let's come back to our file storage interface and make a fake implementation that we can use in tests. Here's one of the ways that we can do it:

public class FakeBlobStorage : IBlobStorage
{
    private readonly Dictionary<string, byte[]> _files =
        new Dictionary<string, byte[]>(StringComparer.Ordinal);

    public Task<Stream> ReadFileAsync(string fileName)
    {
        var data = _files[fileName];
        var stream = new MemoryStream(data);

        return Task.FromResult<Stream>(stream);
    }

    public async Task DownloadFileAsync(string fileName, string outputFilePath)
    {
        await using var input = await ReadFileAsync(fileName);
        await using var output = File.Create(outputFilePath);

        await input.CopyToAsync(output);
    }

    public async Task UploadFileAsync(string fileName, Stream stream)
    {
        await using var buffer = new MemoryStream();
        await stream.CopyToAsync(buffer);

        var data = buffer.ToArray();
        _files[fileName] = data;
    }

    public async Task UploadManyFilesAsync(IReadOnlyDictionary<string, Stream> fileNameStreamMap)
    {
        foreach (var (fileName, stream) in fileNameStreamMap)
        {
            await UploadFileAsync(fileName, stream);
        }
    }
}

As seen above, our fake blob storage uses a hash map to keep track of uploaded files and their content. For the more high-level operations such as DownloadFileAsync() and UploadManyFilesAsync(), the real implementation may be using some optimized routines, but here we are just composing existing functionality.

Note that the above implementation doesn't make any assumptions about how it's going to be used in tests. Instead it provides what can effectively be a drop-in replacement for the actual blob storage component in our system.

Because of that, it's also important that the fake replicates the behavior of the real dependency as closely as possible. This means that we might have to consider various nuances, some of which are:

Whether the file names are treated as case-sensitive
Whether ReadFileAsync() throws on a non-existing file or returns an empty stream
Whether UploadFileAsync() throws on an existing file or just overwrites it

Not getting these aspects right doesn't invalidate the implementation altogether, but can make it less valuable in specific edge-case scenarios. At the end of the day, even when using fakes, we won't be able to gain the same level of confidence as we would by testing in a real environment, which is why proper end-to-end is still necessary.

It may also appear that the details we have to take into account here are not that different from the implementation-aware assumptions we were making when using mocks, as neither are actually governed by the interface of the component. However, the major distinction is that the coupling we institute here is between the test double and the real implementation of the component, rather than between the test double and the internal specifics of its consumer.

Since our fake is defined separately from the tests themselves, its design doesn't rely on any particular interactions with the rest of the system. As a result, changing the implementation of DocumentManager should not cause the tests to fail.

With that out of the way, let's consider how incorporating fakes actually affects the design of our scenarios. The initial instinct would probably be to convert it over to something like this:

[Fact]
public async Task I_can_get_the_content_of_an_existing_document()
{
    // Arrange
    var blobStorage = new FakeBlobStorage();
    var documentManager = new DocumentManager(blobStorage);

    await using var documentStream = new MemoryStream(
        new byte[] {0x68, 0x65, 0x6c, 0x6c, 0x6f}
    );

    await blobStorage.UploadFileAsync("docs/test.txt", documentStream);
    //          still implementation-aware --^

    // Act
    var content = await documentManager.GetDocumentAsync("test.txt");

    // Assert
    content.Should().Be("hello");
}

Here we take an existing test and rather than configure a mock to return a preconfigured response, we create a fake blob storage and fill it with data directly. This way we don't need to assume that retrieving a document should call a certain method, but instead just rely on the completeness of the behavior provided by our fake.

However, despite being able to eliminate most of the assumptions, we didn't get rid of all of them. Namely, our test still expects that calling GetDocumentAsync() should look for the file inside the docs/ namespace, as that's where we're uploading it to in the arrange phase.

This problem stems from the fact that we are yet again relying on how DocumentManager interacts with IBlobStorage, but this time it's not caused by the test double but by the design of the test itself. To avoid it, we need to adapt the scenario so that it revolves around the external behavior of the system and not its relationship with the dependencies.

Here is how we can achieve that:

[Fact]
public async Task I_can_get_the_content_of_a_previously_saved_document()
{
    // Arrange
    var blobStorage = new FakeBlobStorage();
    var documentManager = new DocumentManager(blobStorage);

    await documentManager.SaveDocumentAsync("test.txt", "hello");

    // Act
    var content = await documentManager.GetDocumentAsync("test.txt");

    // Assert
    content.Should().Be("hello");
}

Now, rather than create a file directly through FakeBlobStorage, we do it using DocumentManager instead. This brings the scenario closer to how an actual consumer would interact with the class that we are testing.

From the perspective of the behavior of the system, the only thing we care about is whether a document that was saved can be retrieved afterwards. Any unrelated details do not concern us, so there is no reason to be testing them.

Because of this, we don't have to worry about where the file is persisted inside the storage, how exactly it gets uploaded, which format or encoding is used, and other similar aspects. Since the test above only validates the external behavior, it doesn't have any overreaching assumptions about internal specifics.

It is also worth noting how little code we had to write, comparing to our previous attempts when we relied on mocking. This additional benefit comes from the fact that well-designed fakes are inherently reusable, which helps a lot with maintainability.

If you are used to purist unit testing, this approach may seem a little bit weird at first, since we're not verifying the outcomes of individual operations, but rather how they fit together to create cohesive functionality. In the grand scheme of things, the latter is far more important, as the confidence we derive directly depends on how accurately our tests match the way the software is actually used.

Testing the test doubles

Since fakes are used to provide a realistic and potentially non-trivial implementation, it makes sense that their behavior should be tested as well. The idea of testing test doubles may appear bizarre, as we never do it with mocks, but in this case it is actually perfectly reasonable.

In fact, it's even common to define fakes as part of the main project, where the rest of the code resides, rather than with the tests. Many libraries and frameworks often also provide fake implementations as part of their core package, in order to make it easier for other developers to write their own tests as well.

The process of testing fakes is not in any way different from testing normal production code. For example, in case with our FakeBlobStorage, we can verify important aspects of its behavior like so:

[Fact]
public async Task Previously_uploaded_file_can_be_retrieved()
{
    // Arrange
    var blobStorage = new FakeBlobStorage();

    var fileData = new byte[] {0x68, 0x65, 0x6c, 0x6c, 0x6f};
    await using var fileStream = new MemoryStream(fileData);
    await blobStorage.UploadFileAsync("test.txt", fileStream);

    // Act
    await using var actualFileStream = await blobStorage.ReadFileAsync("test.txt");
    var actualFileData = actualFileStream.ToArray();

    // Assert
    actualFileData.Should().Equal(fileData);
}

[Fact]
public async Task Trying_to_retrieve_non_existing_file_throws()
{
    // Arrange
    var blobStorage = new FakeBlobStorage();

    // Act & assert
    await Assert.ThrowsAnyAsync<Exception>(() => blobStorage.ReadFileAsync("test.txt"));
}

[Fact]
public async Task File_names_are_case_sensitive()
{
    // Arrange
    var blobStorage = new FakeBlobStorage();

    await using var fileStream1 = new MemoryStream(new byte[] {1, 2, 3});
    await using var fileStream2 = new MemoryStream(new byte[] {4, 5, 6});

    await blobStorage.UploadFileAsync("test.txt", fileStream1);
    await blobStorage.UploadFileAsync("TEST.txt", fileStream2);

    // Act
    await using var actualFileStream1 = await blobStorage.ReadFileAsync("test.txt");
    var actualFileData1 = actualFileStream1.ToArray();

    await using var actualFileStream2 = await blobStorage.ReadFileAsync("TEST.txt");
    var actualFileData2 = actualFileStream2.ToArray();

    // Assert
    actualFileData1.Should().NotEqual(actualFileData2);
}

These tests make sure that the fake implementation we've built actually works like it's supposed to. As long as they pass, we can be confident that using FakeBlobStorage doesn't produce incorrect results in other scenarios.

Summary

Due to the popularity of mocking frameworks and the convenience they provide, many developers find that there is very little incentive to write test doubles by hand. However, relying on dynamically-generated mocks can be dangerous, as it typically leads to implementation-aware testing.

In many cases, it may be a better idea to use fakes instead. These are test doubles that represent complete but simplified implementations of their real-life counterparts, rather than just a set of prearranged responses.

Because fakes are naturally separate from the scenarios in which they are used, it's easier to design tests that are not coupled to internal specifics of the system. Besides that, their self-contained nature makes them reusable as well, which lends itself to better long-term maintainability.

Follow me on Twitter to get notified when I post a new article ✨

Pure-Impure Segregation Principle

Oleksii Holub — Mon, 24 Aug 2020 17:13:15 +0000

About two months ago I published an article titled "Unit Testing is Overrated" where I shared my thoughts on how developers place way too much faith in that testing approach and why it often isn't the best tool for the job. While I didn't expect that post to do particularly well, in three weeks it managed to get over 100K views and 1000 comments, even despite its controversial nature (or, perhaps, owing to it?).

It was really interesting to follow the discussions that unfolded, given the vast contrast of opinions people seemed to have on the subject. And while most commenters mainly shared their personal experiences, a few have also voiced criticism of the way some arguments were presented.

In particular, one person mentioned that the drawbacks I've described, especially those pertaining to abstractions and mocking, are really just a byproduct of object-oriented programming and its inherent flaws. Had my examples been designed with functional principles in mind, many of the outlined problems would never have surfaced.

More specifically, the suggested solution was to refactor the presented class hierarchy by extracting the pure business logic away from the rest of the code. Getting rid of the impure dependency eliminates the need for mocking, which in turn simplifies unit testing.

This exact approach was actually mentioned in later parts of the post as well, albeit in a slightly different context. Although it does make isolated testing easier for that particular snippet of code, it doesn't actually invalidate the main issues raised by the article.

That said, I also think that the underlying principle of code separation based on purity is very important and often overlooked. When used correctly, it can positively influence software design, providing benefits in terms of readability, portability and, as mentioned, unit testing.

Depending on who you ask, this principle may have different names, such as functional core, imperative shell, impure-pure-impure sandwich, and some others. And while most developers seem to agree on its value, there's still some misunderstanding remaining as to how it's applied beyond simple academic examples.

At the end of the day, just like with any other software development pattern, its usefulness is entirely situational. However, it offers a good mental model for reasoning about non-determinism in code, which is relevant regardless of context.

In this article we will look at what actually makes something pure or impure, why is that important to us, and how we can leverage that knowledge to write better software. I will show examples of where applying this principle lends to better design, as well scenarios where it might not be as helpful.

Note: as usual, the code samples in this article are written in C#, but the main ideas apply to any language.

Pure vs impure

As I'm writing this in 2020, there is no doubt that most readers are already familiar with the concept of purity in programming. Nevertheless, let's go over it one more time to make sure we are on the same page.

In essence, pure code is code encapsulated within a function, whose evaluation is influenced only by its parameters and whose evaluation influences only its returned value. In other words, a pure function doesn't have any implicit arguments, doesn't depend on or interact with external state, and doesn't generate any observable side-effects.

Conversely, a function which breaks at least one of those two rules is called impure. To illustrate this, let's look at a very simple example:

public static bool IsFoodEdible(DateTimeOffset expiration) =>
    DateTimeOffset.Now < expiration;

public static bool IsFoodEdible(DateTimeOffset expiration, DateTimeOffset instant) =>
    instant < expiration;

While both versions of the IsFoodEdible function are similar, only one of them is pure. The first overload gets the current time from the system clock, creating an implicit dependency on some external state. In practice, this means that evaluating the function multiple times may very well produce different results even for the same input parameters, which violates the first rule of purity.

The other version takes the current time as an explicit parameter instead and thus does not exhibit that problem. Regardless of whether we call that function now or ten years into the future, the result is guaranteed to always be the same for the same input. In other words, the behavior of the function depends only on the parameters that were passed to it and nothing else.

Because of that, the second function shown in the above example is pure, while the first one isn't. Additionally, the following variant would be impure as well:

public static void IsFoodEdible(DateTimeOffset expiration, DateTimeOffset instant)
{
    if (instant < expiration)
        Console.WriteLine("It's edible.");
    else
        Console.WriteLine("It's not edible.");
}

In this case, the impurity comes from the fact that this function generates side-effects by interacting with the standard output stream. Since its evaluation influences something other than its returned value, it breaks the second rule we outlined earlier.

As a general rule, any function that doesn't return anything (or whose return value may be ignored) is guaranteed to be impure, because a pure function without a return value is inherently useless. Furthermore, if a function executes asynchronously, it's also a reliable giveaway that a function is impure, since asynchrony naturally comes from I/O operations.

Finally, the function in the following example may seem impure at a first glance too, but actually isn't:

public static bool AllFoodEdible(IReadOnlyList<DateTimeOffset> expirations, DateTimeOffset instant)
{
    for (var i = 0; i < products.Count; i++)
    {
        if (instant >= expirations[i])
            return false;
    }

    return true;
}

Seeing as AllFoodEdible mutates the value of i during the course of its execution, one could think that such a function is not pure either. However, because the variable i is encapsulated within a local scope and cannot be accessed from outside, this mutation is not externally observable and, as such, does not make the function impure.

Now, of course it wouldn't make much sense to classify code based on these seemingly arbitrary traits if purity didn't provide us with some useful benefits. Indeed, since pure functions are deterministic and have no side-effects, they possess the following intrinsic qualities:

Easy to reason about
Can be safely cached
Can be safely parallelized
Testable in isolation
Don't execute asynchronously
Don't influence other functions

Judging by this list alone, it's rather clear that pure code is extremely flexible and convenient to work with. In fact, the initial instinct may be that we should optimize our design in such way that we focus exclusively on writing pure code.

Unfortunately, that's not possible because purity, in itself, is not an indication of quality, but rather of purpose. Any program will invariably have impure code, as it's required to handle infrastructural concerns, such as reading user input, persisting data, making changes in the environment, and all the other things that make our software actually useful.

These aspects are dictated by the functional requirements of the software and not so much by its design. That means that we can't simply eliminate impurities from our code, at least not without also changing how it works.

Having said that, one very important characteristic of impurity is that it's inherently contagious. Any function that depends on the execution of an impure function becomes impure as well:

// Impure function
public static string GetId() => Guid.NewGuid().ToString();

// Impure function (calls an impure function)
public static string GetFilePath(string dirPath, string name) =>
    dirPath + name + GetId();

// Pure function (takes the result of impure function as a parameter)
public static string GetFilePath(string dirPath, string name, string id) =>
    dirPath + name + id;

Depending on how the code is structured and how it interacts with non-deterministic and effectful operations, impurities may make up a larger or smaller portion of the whole. That, in turn, is something we can actually control.

In order to reap the most benefit out of pure functions, we need to design software in a way that limits impure interactions and delays them as much as possible. Ideally, we should strive to push them as far out as we can, towards the boundaries of the system.

Flattening the dependency tree

Although the concept of purity forms the foundation of functional programming, it isn't given as much thought in the object-oriented world. In fact, the main purpose of object-oriented design is to aggregate related behavior in a single contextual entity, which usually involves state and mutations.

Software written with OOP in mind follows a hierarchical design, where objects are composed together to represent different layers of abstraction in a connected fashion. Any impurities that may exist in those objects are free to spread from child to parent, potentially contaminating the entire dependency tree.

To better understand what that means in practice, let's revisit an example from my previous article. The idea was to build a simple web API application that calculates user's sunrise and sunset times based on their IP. This functionality can be modeled using three classes:

LocationProvider to get a location from an IP address, using a public GeoIP database
SolarCalculator to calculate solar times from that location
SolarTimesController to expose the result through an HTTP endpoint

public class LocationProvider
{
    private readonly HttpClient _httpClient;

    /* ... */

    public async Task<Location> GetLocationAsync(IPAddress ip)
    {
        // Pure
        var ipFormatted = !ip.IsLocal()
            ? ip.MapToIPv4().ToString()
            : "";

        // Impure
        var json = await _httpClient.GetJsonAsync($"http://ip-api.com/json/{ipFormatted}");

        // Pure
        var latitude = json.GetProperty("lat").GetDouble();
        var longitude = json.GetProperty("lon").GetDouble();

        return new Location(latitude, longitude);
    }
}

public class SolarCalculator
{
    private readonly LocationProvider _locationProvider;

    /* ... */

    private DateTimeOffset CalculateSunrise(Location location, DateTimeOffset date)
    {
        /* Pure (implementation omitted) */
    }

    private DateTimeOffset CalculateSunset(Location location, DateTimeOffset date)
    {
        /* Pure (implementation omitted) */
    }

    public async Task<SolarTimes> GetSolarTimesAsync(IPAddress ip, DateTimeOffset date)
    {
        // Impure
        var location = await _locationProvider.GetLocationAsync(ip);

        // Pure
        var sunrise = CalculateSunrise(location, date);
        var sunset = CalculateSunset(location, date);

        return new SolarTimes(sunrise, sunset);
    }
}

[ApiController, Route("solartimes")]
public class SolarTimeController : ControllerBase
{
    private readonly SolarCalculator _solarCalculator;

    /* ... */

    [HttpGet("by_ip")]
    public async Task<IActionResult> GetSolarTimesByIp(DateTimeOffset? date)
    {
        // Impure
        var result = await _solarCalculator.GetSolarTimesAsync(
            HttpContext.Connection.RemoteIpAddress,
            date ?? DateTimeOffset.Now
        );

        return Ok(result);
    }
}

Note how these three classes represent a vertical slice from a potentially much more involved object hierarchy. Schematically, the flow of data in this relationship can be depicted like so:

 [ LocationProvider ]
           |
           ↓
  [ SolarCalculator ]
           |
           ↓
[ SolarTimesController ]

This is a very typical scenario for "classically" designed object-oriented software. You'll probably find it extremely familiar if you have experience working on code that follows the n-tier architecture or any other similar pattern.

If we consider this relationship from a standpoint of purity, we'll notice that the entire call chain shown previously is impure. And while for LocationProvider it makes sense because it performs non-deterministic I/O, the SolarCalculator is impure only due to its dependency on the former.

That design is not ideal, because we lose out on the benefits of pure functions without really getting anything in return. Now if we wanted to, for example, test SolarCalculator.GetSolarTimesAsync in isolation, we would only be able do that with the help of an autotelic abstraction and a mock object, which is not desirable.

This issue could've been avoided if we architected our code with the pure-impure segregation principle in mind. Let's see how we can refactor our classes to push the impurities out of SolarCalculator:

public class LocationProvider
{
    /* ... */
}

public static class SolarCalculator
{
    public static SolarTimes GetSolarTimes(Location location, DateTimeOffset date)
    {
        // Pure
        var sunrise = CalculateSunrise(location, date);
        var sunset = CalculateSunset(location, date);

        return new SolarTimes(sunrise, sunset);
    }
}

[ApiController, Route("solartimes")]
public class SolarTimesController
{
    private readonly LocationProvider _locationProvider;

    /* ... */

    [HttpGet("by_ip")]
    public async Task<IActionResult> GetSolarTimesByIp(DateTimeOffset? date)
    {
        // Impure
        var location = await _locationProvider.GetLocationAsync(
            HttpContext.Connection.RemoteIpAddress
        );

        // Pure
        var result = SolarCalculator.GetSolarTimes(
            location,
            date ?? DateTimeOffset.Now
        );

        return Ok(result);
    }
}

Previously, the method in SolarCalculator took an IP address as a parameter and relied on LocationProvider to get the coordinates it maps to. After refactoring, the method is now static and instead takes the location directly, skipping the previously required impure operation.

Of course, that impurity didn't just disappear into thin air, our software still needs to get the location anyhow. The difference is that now this concern is pushed out towards the boundary of the system, which, in this case, is represented by the controller.

In doing that, we also flattened the hierarchy so that all of the dependencies are aggregated at the boundary. The data flow now looks a bit more like a pipeline instead:

[ LocationProvider ]  [ SolarCalculator ]
          |                   |
          |_____         _____|
                ↓       ↓
        [ SolarTimesController ]

The benefit of this design is that our pure business logic is no longer contaminated by effectful code, which means we can take advantage of the useful properties listed in the previous section. For example, if we wanted to parallelize or test SolarCalculator, it's much easier to do so now than it was before.

Interleaved impurities

Although very useful, the type of "lossless" refactoring shown earlier only works if the data required by the function can be easily encapsulated within input parameters. Unfortunately, this is not always the case.

Often a function may need to dynamically resolve data from an external API or a database, with no way of knowing about it beforehand. This typically results in an implementation where pure and impure concerns are interleaved with each other, creating a tightly coupled cohesive structure.

To illustrate a scenario like that, let's take a look at a slightly more involved example. The following snippet contains a class called RecommendationsProvider which is responsible for generating song suggestions for a user of some music streaming service:

public class RecommendationsProvider
{
    private readonly SongService _songService;

    /* ... */

    public async Task<IReadOnlyList<Song>> GetRecommendationsAsync(string userName)
    {
        // 1. Get user's own top scrobbles
        // 2. Get other users who listened to the same songs
        // 3. Get top scrobbles of those users
        // 4. Aggregate the songs into recommendations

        // Impure
        var scrobbles = await _songService.GetTopScrobblesAsync(userName);

        // Pure
        var scrobblesSnapshot = scrobbles
            .OrderByDescending(s => s.ScrobbleCount)
            .Take(100)
            .ToArray();

        var recommendationCandidates = new List<Song>();
        foreach (var scrobble in scrobblesSnapshot)
        {
            // Impure
            var otherListeners = await _songService
                .GetTopListenersAsync(scrobble.Song.Id);

            // Pure
            var otherListenersSnapsot = otherListeners
                .Where(u => u.TotalScrobbleCount >= 10_000)
                .OrderByDescending(u => u.TotalScrobbleCount)
                .Take(20)
                .ToArray();

            foreach (var otherListener in otherListenersSnapsot)
            {
                // Impure
                var otherScrobbles = await _songService
                    .GetTopScrobblesAsync(otherListener.UserName);

                // Pure
                var otherScrobblesSnapshot = otherScrobbles
                    .Where(s => s.Song.IsVerifiedArtist)
                    .OrderByDescending(s => s.Song.Rating)
                    .Take(10)
                    .ToArray();

                recommendationCandidates.AddRange(
                    otherScrobblesSnapshot.Select(s => s.Song)
                );
            }
        }

        // Pure
        var recommendations = recommendationCandidates
            .OrderByDescending(s => s.Rating)
            .Take(200)
            .ToArray();

        return recommendations;
    }
}

The above algorithm works by retrieving the user's most listened songs, finding other people who have also listened to the same titles, and then extracting their top songs as well. Those songs are then aggregated into a list of recommendations and returned back to the caller.

It's quite clear that this function would benefit greatly from being pure, seeing how much business logic is encapsulated within it. Unfortunately, the technique we relied upon earlier won't work here.

In order to fully isolate GetRecommendationsAsync from its impure dependencies, we would have to somehow supply the function with an entire list of songs, users, and their scrobbles upfront. If we assume that we're dealing with data on millions of users, it's obvious that this would be completely impractical and likely even impossible.

A seemingly simple way we could try to work around this problem is to split the function into smaller pieces, each handling one of the four stages of the algorithm separately:

public class RecommendationsProvider
{
    /* ... */

    // Pure
    public static IReadOnlyList<string> HandleOwnScrobbles(IReadOnlyList<Scrobble> scrobbles) =>
        scrobbles
            .OrderByDescending(s => s.ScrobbleCount)
            .Take(100)
            .Select(s => s.Song.Id)
            .ToArray();

    // Pure
    public static IReadOnlyList<string> HandleOtherListeners(IReadOnlyList<User> users) =>
        users
            .Where(u => u.TotalScrobbleCount >= 10_000)
            .OrderByDescending(u => u.TotalScrobbleCount)
            .Take(20)
            .Select(u => u.UserName)
            .ToArray();

    // Pure
    public static IReadOnlyList<Song> HandleOtherScrobbles(IReadOnlyList<Scrobble> scrobbles) =>
        scrobbles
            .Where(s => s.Song.IsVerifiedArtist)
            .OrderByDescending(s => s.Song.Rating)
            .Take(10)
            .Select(s => s.Song)
            .ToArray();

    // Pure
    public static IReadOnlyList<Song> FinalizeRecommendations(IReadOnlyList<Song> songs) =>
        songs
            .OrderByDescending(s => s.Rating)
            .Take(200)
            .ToArray();

    public async Task<IReadOnlyList<Song>> GetRecommendationsAsync(string userName)
    {
        // Impure
        var scrobbles = await _songService.GetTopScrobblesAsync(userName);

        // Pure
        var songIds = HandleOwnScrobbles(scrobbles);

        var recommendationCandidates = new List<Song>();
        foreach (var songId in songIds)
        {
            // Impure
            var otherListeners = await _songService
                .GetTopListenersAsync(songId);

            // Pure
            var otherUserNames = HandleOtherListeners(otherListeners);

            foreach (var otherUserName in otherUserNames)
            {
                // Impure
                var otherScrobbles = await _songService
                    .GetTopScrobblesAsync(otherListener.UserName);

                // Pure
                var songsToRecommend = HandleOtherScrobbles(otherScrobbles);

                recommendationCandidates.AddRange(songsToRecommend);
            }
        }

        // Pure
        return FinalizeRecommendations(recommendationCandidates);
    }
}

By extracting all of the pure code out of GetRecommendationsAsync, we can now write unit tests that verify that the intermediate stages of the algorithm work as intended. On the surface, it looks as though we managed to achieve exactly what we wanted.

However, instead of having one cohesive element to reason about, we ended up with multiple fragments, each having no meaning or value of its own. While unit testing of individual parts may have become easier, the benefit is very questionable, as it provides no confidence in the correctness of the algorithm as a whole.

Ultimately, we weren't able to push impurities out towards the system boundaries -- what we did was simply push the pure code further in instead. In other words, the flow of data in the program remains completely unchanged.

The main issue is that each stage of the recommendation algorithm depends on additional data derived from the previous stages. Since this behavior is inherently non-deterministic, it's impossible to express it using pure functions.

Pure "enough" code

Although it's convenient to treat purity as an objectively provable characteristic, it's actually a bit more nuanced than that. As a matter of fact, one could say that purity is a relative concept, not an absolute one.

To understand what exactly I mean by that, let's take a look at an example:

public static int FindIndexOf(IEnumerable<Item> items, Item item)
{
    var i = 0;

    foreach (var o in items)
    {
        if (o == item)
            return i;

        i++;
    }

    throw new Exception("Item not found.");
}

This is a very simple function that attempts to find an index that corresponds to the position of an item in a sequence or throw an exception in case of failure. The negative outcome is assumed to be very improbable in this scenario, hence why an exception is used as opposed to a fallback value.

According to the criteria of purity, this function is not pure because the result of its evaluation is not entirely encapsulated within the returned value. Throwing an exception is an effectful operation, since it can change the behavior of the function above in the call stack, or lead to the termination of the program altogether.

However, despite all that, the function is still deterministic, cacheable, parallelizable, and testable, as long as we remember to handle the exception that may be raised in certain circumstances. Even though it's not technically pure, it still retains most of the important properties we care about.

Let's consider an even simpler example:

public static int Wrap(int value, int period) => value % period;

Seeing as the above code literally just represents a mathematical expression, it seems logical that it must be pure. However, this function shares the exact same problem as the one in the previous snippet.

The modulus operator has an exceptional outcome, which occurs when the supplied divisor is equal to zero. If we were to try and invoke Wrap(123, 0), it would throw an exception, indicating that the function is actually impure as well.

Notably, this problem could be avoided if we used something like Option<int> as return type instead. This approach eliminates the need for an exception (and this is how Darklang does it), but comes at an expense of making basic arithmetic operations appear more cumbersome.

In any case, even though the code we wrote originally doesn't satisfy the theoretical definition of purity, it might be pure enough for our usage scenario.

Let's also take a look at an opposite situation:

public static string GetOutputPath(Report report, string outputDir)
{
    var fileExtension = report.Format == ReportFormat.Html
        ? "html"
        : "txt";

    var fileName = $"{report.Name}.{fileExtension}";

    return Path.Combine(outputDir, fileName);
}

The code above assembles a file path for the provided report by combining the output directory with the generated file name. It calls the Path.Combine method, whose behavior relies on the value of the Path.DirectorySeparatorChar constant, as it indicates which directory separator is used by the operating system.

Since it is a constant and its value is guaranteed to always be the same for the duration of the program's lifetime, our function is pure (as long as we also disregard exceptions). However, it's pure only within the current session.

If we imagine that we're building a cross-platform solution, it's logical that we treat specifics of each platform as environmental parameters. In other words, for code that is expected to run seamlessly on Windows and Linux, the path separator constant essentially acts as a global variable.

Assuming our goal is to test GetOutputPath in isolation, simply relying on the parameters of the function is not enough. We would also need to execute tests on each of the supported operating systems, to make sure it actually works with all possible path separators.

In this case, the fact that the function is pure does not provide us with sufficient guarantees. While it's pure by definition, it's not pure enough for what we need.

As you can see, the concept of purity gets a bit hazy once you start digging into specifics. In reality, everything around us is inherently impure, so what we accept as pure really depends on what level of abstraction we choose to operate at.

If you decide to follow the rules pedantically, you'll find the idea of modelling any problem domain with pure functions quickly becomes impractical. However, it's important to remember that the goal is not purity in itself, but rather the benefits it provides.

At the end of the day, the entire notion of purity is just a mathematical model, which may not necessarily translate very well to applied programming. As a developer of the project, it is up to you to draw the line and decide what makes sense and what doesn't.

Summary

Overall, purity is a pretty useful concept, as it helps us understand how some operations may make our code non-deterministic, difficult to reason about, and cumbersome to test in isolation. Impure interactions are not bad on their own, but the constraints they impose are contagious in nature and may spread to other parts of the application.

The pure-impure segregation principle aims to limit impurities to an essential minimum, by decoupling them from the rest of the code. Ultimately, the goal is to push all non-pure operations towards the outermost layers of the system, while keeping the domain layer comprised entirely of pure functions.

Designing software in such way leads to an architecture that resembles a pipeline rather than a hierarchy, which favors functional style of programming. Depending on the project, this may aid in expressing the flow of data more clearly, among other useful benefits.

However, this is not always practical and there are scenarios where extracting pure code comes at a cost of severely reduced cohesiveness. In any case, if your goal is to facilitate testing without mocking, architecting your solution for high-level testing is likely going to be a much better time investment.

Follow me on Twitter to get notified when I post a new article ✨

Unit Testing is Overrated

Oleksii Holub — Tue, 07 Jul 2020 13:54:57 +0000

The importance of testing in modern software development is really hard to overstate. Delivering a successful product is not something you do once and forget about but is rather a continuous recurring process. With every line of code that changes, software must remain in a functional state, which implies the need for rigorous testing.

Over time, as the software industry evolved, testing practices have matured as well. Gradually moving towards automation, testing approaches have also influenced software design itself, spawning mantras like test-driven development, emphasizing patterns such as dependency inversion, and popularizing high-level architectures that are built around it.

Nowadays, automated testing is embedded so deeply within our perception of software development, it's hard to imagine one without the other. And since that ultimately enables us to produce software quickly without sacrificing quality, it's hard to argue that it's not a good thing.

However, despite there being many different approaches, modern "best practices" primarily push developers specifically towards unit testing. Tests, whose scope lies higher on Mike Cohn's pyramid are either written as part of a wider suite (often by completely different people) or even disregarded entirely.

The benefit of this approach is often supported by the argument that unit tests provide the most value during development because they're able to catch errors quickly and help enforce design patterns that facilitate modularity. This idea has become so widely accepted that the term "unit testing" is now somewhat conflated with automated testing in general, losing part of its meaning and contributing to confusion.

When I was a less experienced developer, I believed in following these "best practices" to the letter, as I thought that would make my code better. I didn't particularly enjoy writing unit tests because of all the ceremony involved with abstractions and mocking, but it was the recommended approach after all, so who am I to know better.

It was only later, as I've experimented more and built more projects, that I started to realize that there are much better ways to approach testing and that focusing on unit tests is, in most cases, a complete waste of time.

Aggressively popularized "best practices" often have a tendency of manifesting cargo cults around them, enticing developers to apply design patterns or use specific approaches without giving them a much needed second thought. In the context of automated testing, I find this prevalent when it comes to our industry's unhealthy obsession with unit testing.

In this article I will share my observations about this testing technique and go over why I believe it to be inefficient. I'll also explain which approaches I'm currently using instead to test my code, both in open source projects and day-to-day work.

Note: this article contains code examples which are written in C#, but the language itself is not (too) important to the points I'm making.

Note 2: I've come to realize that programming terms are completely useless at conveying meanings because everyone seems to understand them differently. In this article I will be relying on the "standard" definitions, where unit testing targets smallest separable parts of code, end-to-end testing targets software's outermost entry points, while integration testing is for everything in-between.

Fallacies of unit testing

Unit tests, as evident by the name, revolve around the concept of a "unit", which denotes a very small isolated part of a larger system. There is no formal definition of what a unit is or how small it should be, but it's mostly accepted that it corresponds to an individual function of a module (or method of an object).

Normally, when the code isn't written with unit tests in mind, it may be impossible to test some functions in complete isolation because they can have external dependencies. In order to work around this issue, we can apply the dependency inversion principle and replace concrete dependencies with abstractions. These abstractions can then be substituted with real or fake implementations, depending on whether the code is executing normally or as part of a test.

Besides that, unit tests are expected to be pure. For example, if a function contains code that writes data to the file system, that part needs to be abstracted away as well, otherwise the test that verifies such behavior will be considered an integration test instead, since its coverage extends to the unit's integration with the file system.

Considering the factors mentioned above, we can reason that unit tests are only useful to verify pure business logic inside of a given function. Their scope does not extend to testing side-effects or other integrations because that belongs to the domain of integration testing.

To illustrate how these nuances affect design, let's take a look at an example of a simple system that we want to test. Imagine we're working on an application that calculates local sunrise and sunset times, which it does through the help of the following two classes:

public class LocationProvider : IDisposable
{
    private readonly HttpClient _httpClient = new HttpClient();

    // Gets location by query
    public async Task<Location> GetLocationAsync(string locationQuery) { /* ... */ }

    // Gets current location by IP
    public async Task<Location> GetLocationAsync() { /* ... */ }

    public void Dispose() => _httpClient.Dispose();
}

public class SolarCalculator : IDiposable
{
    private readonly LocationProvider _locationProvider = new LocationProvider();

    // Gets solar times for current location and specified date
    public async Task<SolarTimes> GetSolarTimesAsync(DateTimeOffset date) { /* ... */ }

    public void Dispose() => _locationProvider.Dispose();
}

Although the design above is perfectly valid in terms of OOP, neither of these classes are actually unit-testable. Because LocationProvider depends on its own instance of HttpClient and SolarCalculator in turn depends on LocationProvider, it's impossible to isolate the business logic that may be contained within methods of these classes.

Let's iterate on that code and replace concrete implementations with abstractions:

public interface ILocationProvider
{
    Task<Location> GetLocationAsync(string locationQuery);

    Task<Location> GetLocationAsync();
}

public class LocationProvider : ILocationProvider
{
    private readonly HttpClient _httpClient;

    public LocationProvider(HttpClient httpClient) =>
        _httpClient = httpClient;

    public async Task<Location> GetLocationAsync(string locationQuery) { /* ... */ }

    public async Task<Location> GetLocationAsync() { /* ... */ }
}

public interface ISolarCalculator
{
    Task<SolarTimes> GetSolarTimesAsync(DateTimeOffset date);
}

public class SolarCalculator : ISolarCalculator
{
    private readonly ILocationProvider _locationProvider;

    public SolarCalculator(ILocationProvider locationProvider) =>
        _locationProvider = locationProvider;

    public async Task<SolarTimes> GetSolarTimesAsync(DateTimeOffset date) { /* ... */ }
}

By doing so we were able to decouple LocationProvider from SolarCalculator, but in exchange our code nearly doubled in size. Also note that we had to drop IDisposable from both classes because they no longer own their dependencies and thus have no business taking responsibility for their lifecycle.

While these changes may seem as an improvement to some, it's important to point out that the interfaces we've defined serve no practical purpose other than making unit testing possible. There's no need for actual polymorphism in our design, so, as far as our code is concerned, these abstractions are autotelic.

Let's try to reap the benefits of all that work and write a unit test for SolarCalculator.GetSolarTimesAsync:

public class SolarCalculatorTests
{
    [Fact]
    public async Task GetSolarTimesAsync_ForKyiv_ReturnsCorrectSolarTimes()
    {
        // Arrange
        var location = new Location(50.45, 30.52);
        var date = new DateTimeOffset(2019, 11, 04, 00, 00, 00, TimeSpan.FromHours(+2));

        var expectedSolarTimes = new SolarTimes(
            new TimeSpan(06, 55, 00),
            new TimeSpan(16, 29, 00)
        );

        var locationProvider = Mock.Of<ILocationProvider>(lp =>
            lp.GetLocationAsync() == Task.FromResult(location)
        );

        var solarCalculator = new SolarCalculator(locationProvider);

        // Act
        var solarTimes = await solarCalculator.GetSolarTimesAsync(date);

        // Assert
        solarTimes.Should().BeEquivalentTo(expectedSolarTimes);
    }
}

Here we have a basic test that verifies that SolarCalculator works correctly for a known location. Since unit tests and their units are tightly coupled, we're following the recommended naming convention, where the test class is named after the class under test, and the name of the test method follows the Method_Precondition_Result pattern.

In order to simulate the desired precondition in the arrange phase, we have to inject corresponding behavior into the unit's dependency, ILocationProvider. In this case we do that by substituting the return value of GetLocationAsync() with a location for which the correct solar times are already known ahead of time.

Note that although ILocationProvider exposes two different methods, from the contract perspective we have no way of knowing which one actually gets called. This means that by choosing to mock a specific one of these methods, we are making an assumption about the underlying implementation of the method we're testing (which was deliberately hidden in the previous snippets).

All in all, the test does correctly verify that the business logic inside GetSolarTimesAsync works as expected. However, let's expand on some of the observations we've made in the process.

Unit tests have a limited purpose

It's important to understand that the purpose of any unit test is very simple: verify business logic in an isolated scope. Depending on which interactions you intend to test, unit testing may or may not be the right tool for the job.

For example, does it make sense to unit test a method that calculates solar times using a long and complicated mathematical algorithm? Most likely, yes.

Does it make sense to unit test a method that sends a request to a REST API to get geographical coordinates? Most likely, not.

If you treat unit testing as a goal in itself, you will quickly find that, despite putting a lot of effort, most tests will not be able to provide you with the confidence you need, simply because they're testing the wrong thing. In many cases it's much more beneficial to test wider interactions with integration tests, rather than focusing specifically on unit tests.

Interestingly, some developers do end up writing integration tests in such scenarios, but still refer to them as unit tests, mostly due to confusion surrounding the concept. Although it could be argued that a unit size can be chosen arbitrarily and can span multiple components, this makes the definition very fuzzy, ultimately just turning overall usage of the term completely useless.

Unit tests lead to more complicated design

One of the most popular arguments in favor of unit testing is that it enforces you to design software in a highly modular way. This builds on an assumption that it's easier to reason about code when it's split into many smaller components rather than a few larger ones.

However, it often leads to the opposite problem, where the functionality may end up becoming unnecessarily fragmented. This makes it much harder to assess the code because a developer needs to scan through multiple components that make up what should have been a single cohesive element.

Additionally, the abundant usage of abstraction, which is required to achieve component isolation, creates a lot of unneeded indirection. Although an incredibly powerful and useful technique in itself, abstraction inevitably increases cognitive complexity, making it further more difficult to reason about the code.

Through that indirection we also end up losing some degree of encapsulation that we were able to maintain otherwise. For example, the responsibility of managing lifetimes of individual dependencies shifts from components that contain them to some other unrelated service (usually the dependency container).

Some of that infrastructural complexity can be also delegated to a dependency injection framework, making it easier to configure, manage, and activate dependencies. However, that reduces portability, which may be undesirable in some cases, for example when writing a library.

At the end of the day, while it's clear that unit testing does influence software design, it's highly debatable whether that's really a good thing.

Unit tests are expensive

Logically, it would make sense to assume that, since they are small and isolated, unit tests should be really easy and quick to write. Unfortunately, this is just another fallacy that seems to be rather popular, especially among managers.

Even though the previously mentioned modular architecture lures us into thinking that individual components can be considered separately from each other, unit tests don't actually benefit from that. In fact, the complexity of a unit test only grows proportionally to the number of external interactions the unit has, due to all the work that you must do to achieve isolation while still exercising required behavior.

The example illustrated previously in this article is very simple, but in a real project it's not unusual to see the arrange phase spanning many long lines, just to set preconditions for a single test. In some cases, the mocked behavior can be so complex, it's almost impossible to unravel it back to figure out what it was supposed to do.

Besides that, unit tests are by design very tightly coupled to the code they're testing, which means that any effort to make a change is effectively doubled as the test suite needs to be updated as well. What makes this worse is that very few developers seem to find doing that an enticing task, often just pawning it off to more junior members on the team.

Unit tests rely on implementation details

The unfortunate implication of mock-based unit testing is that any test written with this approach is inherently implementation-aware. By mocking a specific dependency, your test becomes reliant on how the code under test consumes that dependency, which is not regulated by the public interface.

This additional coupling often leads to unexpected issues, where seemingly non-breaking changes can cause tests to start failing as mocks become out of date. It can be very frustrating and ultimately discourages developers from trying to refactor code, because it's never clear whether the error in test comes from an actual regression or due to being reliant on some implementation detail.

Unit testing stateful code can be even more tricky because it may not be possible to observe mutations through the publicly exposed interface. To work around this, you would normally inject spies, which is a type of mocked behavior that records when a function is called, helping you ensure that the unit uses its dependencies correctly.

Of course, when you not only rely on a specific function being called, but also on how many times it happened or which arguments were passed, the test becomes even more coupled to the implementation. Tests written in such way are only useful if the internal specifics are not ever expected to change, which is a highly unreasonable expectation to have.

Relying too much on implementation details also makes the tests themselves very complex, considering how much setup is required to configure mocks in order to simulate a specific behavior, especially when the interactions are not that trivial or when there are a lot of dependencies. When the tests get so complicated that their own behavior is hard to reason about, who is going to write tests to test the tests?

Unit tests don't exercise user behavior

No matter what type of software you're developing, its goal is to provide value for the end user. In fact, the primary reason why we're writing automated tests in the first place is to ensure that there are no unintended defects that would diminish that value.

In most cases, the user works with the software through some top-level interface such as a UI, CLI, or API. While the code itself might involve numerous layers of abstractions, the only one that matters to the user is the one they get to actually see and interact with.

It doesn't even matter if a few layers deep there's a bug in some part of the system, as long as it never surfaces to the user and doesn't affect the provided functionality. Conversely, it makes no difference that we may have full coverage on all the lower-level pieces, if there's a defect in the user interface that renders our system effectively useless.

Of course, if you want to ensure that something works correctly, you have to check that exact thing and see if it does. In our case, the best way to gain confidence in the system is to simulate how a real user would interact with the top-level interface and see if it works properly according to expectations.

The problem with unit tests is that they're the exact opposite of that. Since we're always dealing with small isolated pieces of our code that the user doesn't directly interact with, we never test the actual user behavior.

Doing mock-based testing puts the value of such tests under an even bigger question, because the parts of our system that would've been used otherwise are replaced with mocks, further distancing the simulated environment from reality. It's impossible to gain confidence that the user will have a smooth experience by testing something that doesn't resemble that experience.

Pyramid-driven testing

So why would we, as an industry, decide that unit testing should be the primary method of testing software, given all of its existing flaws? For the most part, it's because testing at higher levels has always been considered too hard, slow, and unreliable.

If you refer to the traditional test pyramid, you will find that it suggests that the most significant part of testing should be performed at the unit level. The idea is that, since coarse-grained tests are assumed to be slower and more complicated, you will want to concentrate efforts towards the bottom of the integration spectrum to end up with an efficient and maintainable test suite:

The metaphorical model offered by the pyramid is meant to convey that a good testing approach should involve many different layers, because focusing on the extremes can lead to issues where the tests are either too slow and unwieldy, or are useless at providing any confidence. That said, the lower levels are emphasized as that's where the return on investment for development testing is believed to be the highest.

Top-level tests, despite providing the most confidence, often end up being slow, hard to maintain, or too broad to be included as part of typically fast-paced development flow. That's why, in most cases, such tests are instead maintained separately by dedicated QA specialists, as it's usually not considered to be the developer's job to write them.

Integration testing, which is an abstract part of the spectrum that lies somewhere between unit testing and complete end-to-end testing, is quite often just disregarded entirely. Because it's not really clear what exact level of integration is preferable, how to structure and organize such tests, or for the fear that they might get out of hand, many developers prefer to avoid them in favor of a more clear-cut extreme which is unit testing.

For these reasons, all testing done during development typically resides at the very bottom of the pyramid. In fact, over time this has become so commonplace that development testing and unit testing are now practically synonymous with each other, leading to confusion that is only further perpetuated by conference talks, blog posts, books, and even some IDEs (all tests are unit tests, as far as JetBrains Rider is concerned).

In the eyes of most developers, the test pyramid looks somewhat like this instead:

While the pyramid is a noble attempt to turn software testing into a solved problem, there are obviously many issues with this model. In particular, the assumptions it relies on might not be true for every context, especially the premise of highly integrated test suites being slow or hard.

As humans, we are naturally inclined to rely on information passed on to us from those who are more experienced, so that we can benefit from the knowledge of past generations and apply our second thinking system on something more useful instead. This is an important evolutionary trait that makes our species extremely fit for survival.

However, whenever we extrapolate experiences into guidelines, we tend to think of them as being good on their own, forgetting about the circumstances that are integral to their relevancy. The reality is that circumstances change, and once perfectly reasonable conclusions (or best practices) might not apply so well anymore.

If we look back, it's clear that high-level testing was tough in 2000, it probably still was in 2009, but it's 2020 outside and we are, in fact, living in the future. Advancements in technology and software design have made it a much less significant issue than it once was.

Most modern application frameworks nowadays provide some sort of separate API layer used for testing, where you can run your application in a simulated in-memory environment that is very close to the real one. Virtualization tools like Docker also make it possible to execute tests that rely on actual infrastructural dependencies, while still remaining deterministic and fast.

We have solutions like Mountebank, WireMock, GreenMail, Appium, Selenium, Cypress, and countless others that simplify different aspects of high-level testing that were once considered unapproachable. Unless you're developing desktop applications for Windows and are stuck with UIAutomation framework, you will likely have many options available.

On one of my previous projects, we had a web service which was tested at the system boundary using close to a hundred behavioral tests that took just under 10 seconds to run in parallel. Sure, it's possible to get much faster execution time than that with unit tests but given the confidence they provide this was a no-brainer.

The slow test fallacy is, however, not the only false assumption that the pyramid is based on. The idea of having the majority of testing concentrated at the unit level only works out if those tests actually provide value, which of course depends on how much business logic is contained within the code under test.

Some applications may have a lot of business logic (e.g. payroll systems), some may have close to none (e.g. CRUD apps), most are somewhere in between. Majority of the projects I've personally worked on didn't have nearly enough of it to warrant extensive coverage with unit tests but had plenty of infrastructural complexity on the other hand, which would benefit from integration testing.

Of course, in an ideal world one would evaluate the context of the project and come up with a testing approach that is most suitable for the problem at hand. In reality, however, most developers don't even begin to think about it at all, instead just blindly stacking mountains of unit tests following what the best practices seemingly advise you to do.

Finally, I think it's fair to say, the model provided by the test pyramid is just too simplistic in general. The vertical axes present the testing spectrum as a linear scale, where any gain in confidence you get by going up is apparently offset by an equivalent amount of loss in maintainability and speed. This may be true if you compare the extremes, but not necessarily so for the rest of the points in between.

It also doesn't account for the fact that isolation has a cost in itself and isn't something that comes for free simply by "avoiding" external interactions. Given how much effort it takes to write and maintain mocks, it's entirely possible that a less-isolated test can be cheaper and end up providing more confidence, albeit running slightly slower.

If you consider these aspects, it seems likely that the scale is not linear after all and that the point of highest return on investment resides somewhere closer to the middle rather than at the unit level:

All in all, when you're trying to establish an efficient test suite for your project, the test pyramid isn't the best guideline you can follow. It makes a lot more sense to focus on what's relevant specifically to your context, instead of relying on "best practices".

Reality-driven testing

At the most basic level, a test provides value if it grants certainty that the software is working correctly. The more confident we feel, the less we have to rely on ourselves to spot potential bugs and regressions while introducing changes in code, because we trust our tests to do that for us.

That trust in turn depends on how accurately the test resembles the actual user behavior. A test scenario operating at the system boundary without knowledge of any internal specifics is bound to provide us with greater confidence (and thus, value) than a test working at a lower level.

In essence, the degree of confidence we gain from tests is the primary metric by which their value should be measured. Pushing it as high as possible is also the primary goal.

Of course, as we know, there are other factors in play as well, such as cost, speed, ability to parallelize, and whatnot, which are all important. The test pyramid makes strong assumptions about how these things scale in relation to each other, but these assumptions are not universal.

Moreover, these factors are also secondary to the primary goal of obtaining confidence. An expensive test that takes a really long time to run but provides a lot of confidence is still infinitely more useful than an extremely fast and simple test that does nothing.

For that reason, I find it best to write tests that are as highly integrated as possible, while keeping their speed and complexity reasonable.

Does this mean that every test we write should be an end-to-end test? No, but we should be trying to get as far as we can in that direction, while keeping the downsides at an acceptable level.

What's acceptable or not is subjective and depends on the context. At the end of the day, it's important that those tests are written by developers and are used during development, which means they shouldn't feel like a burden to maintain and it should be possible to run them for local builds and on CI.

Doing this also means that you will likely end up with tests that are scattered across different levels of the integration scale, with seemingly no clear sense of structure. This isn't an issue we would have had with unit testing, because there each test is coupled to a specific method or a function, so the structure usually ends up mirroring that of the code itself.

Fortunately, this doesn't matter because organizing tests by individual classes or modules is not important in itself but is rather a side-effect of unit testing. Instead, the tests should be partitioned by the actual user-facing functionality that they are meant to verify.

Such tests are often called functional because they are based on the software's functional requirements that describe what features it has and how they work. Functional testing is not another layer on the pyramid, but instead a completely orthogonal concept.

Contrary to the popular belief, writing functional tests does not require you to use Gherkin or a BDD framework, but can be done with the very same tools that are typically used for unit testing. For example, consider how we can rewrite the example from the beginning of the article so that the tests are structured around supported user behavior rather than units of code:

public class SolarTimesSpecs
{
    [Fact]
    public async Task User_can_get_solar_times_automatically_for_their_location() { /* ... */ }

    [Fact]
    public async Task User_can_get_solar_times_during_periods_of_midnight_sun() { /* ... */ }

    [Fact]
    public async Task User_can_get_solar_times_if_their_location_cannot_be_determined() { /* ... */ }
}

Note that the actual implementation of the tests is hidden because it's not relevant to the fact that they're functional. What matters is that the tests and their structure are driven by the software requirements, while their scope can theoretically range anywhere from end-to-end to even unit level.

Naming tests in accordance to specifications rather than classes has an additional advantage of removing that unnecessary coupling. Now, if we decide to rename SolarCalculator to something else or move it to a different directory, the test names won't need to be updated to reflect that.

By adhering to this structure, our test suite will effectively take form of a living documentation. For example, this is how the test suite is organized in CliWrap (the underscores are replaced with spaces by xUnit):

As long as a piece of software does something at least remotely useful, it will always have functional requirements. Those can be either formal (specification documents, user stories, etc.) or informal (verbally agreed upon, assumed, JIRA tickets, written on toilet paper, etc.)

Turning informal specifications into functional tests can often be difficult because it requires us to take a step away from code and challenge ourselves to think from a user's perspective. What helps me with my open source projects is that I start by creating a readme file where I list a bunch of relevant usage examples, and then encode those into tests.

To summarize, we can conclude that it's best to partition tests based on threads of behavior, rather than the code's internal structure.

If we combine all these ideas, we get a mental framework that provides us with a clear goal for writing tests and a good sense of organization, while not relying on any assumptions. We can use it to establish a test suite for our project that focuses on value, and then scale it according to priorities and limitations relevant to the current context.

Functional testing for web services (via ASP.NET Core)

There might still be some confusion as to what constitutes functional testing or how exactly it's supposed to look especially if you've never done it before, so it makes sense to show a simple but complete example. For this, we will turn the solar calculator from earlier into a web service and cover it with tests according to the rules we've outlined in the previous part of the article. This app will be based on ASP.NET Core, which is a web framework I'm most familiar with, but the same idea should also equally apply to any other platform.

Our web service is going to expose endpoints to calculate sunrise and sunset times based on the user's IP or provided location. To make things a bit more interesting, we'll also add a Redis caching layer to store previous calculations for faster responses.

The tests will work by launching the app in a simulated environment where it can receive HTTP requests, handle routing, perform validation, and exhibit almost identical behavior to an app running in production. At the same time, we will also use Docker to make sure our tests rely on the same infrastructural dependencies as the real app does.

Let us first go over the implementation of the web app to understand what we're dealing with. Note, some parts in the code snippets below are omitted for brevity, but you can also check out the full project on GitHub.

First off, we will need a way to get the user's location by IP, which is done by the LocationProvider class we've seen in earlier examples. It works simply by wrapping an external GeoIP lookup service called IP-API:

public class LocationProvider
{
    private readonly HttpClient _httpClient;

    public LocationProvider(HttpClient httpClient) =>
        _httpClient = httpClient;

    public async Task<Location> GetLocationAsync(IPAddress ip)
    {
        // If IP is local, just don't pass anything (useful when running on localhost)
        var ipFormatted = !ip.IsLocal() ? ip.MapToIPv4().ToString() : "";

        var json = await _httpClient.GetJsonAsync($"http://ip-api.com/json/{ipFormatted}");

        var latitude = json.GetProperty("lat").GetDouble();
        var longitude = json.GetProperty("lon").GetDouble();

        return new Location
        {
            Latitude = latitude,
            Longitude = longitude
        };
    }
}

In order to turn location into solar times, we're going to rely on the sunrise/sunset algorithm published by US Naval Observatory. The algorithm itself is too long to include here, but the rest of the implementation for SolarCalculator is as follows:

public class SolarCalculator
{
    private readonly LocationProvider _locationProvider;

    public SolarCalculator(LocationProvider locationProvider) =>
        _locationProvider = locationProvider;

    private static TimeSpan CalculateSolarTimeOffset(Location location, DateTimeOffset instant,
        double zenith, bool isSunrise)
    {
        /* ... */

        // Algorithm omitted for brevity

        /* ... */
    }

    public async Task<SolarTimes> GetSolarTimesAsync(Location location, DateTimeOffset date)
    {
        /* ... */
    }

    public async Task<SolarTimes> GetSolarTimesAsync(IPAddress ip, DateTimeOffset date)
    {
        var location = await _locationProvider.GetLocationAsync(ip);

        var sunriseOffset = CalculateSolarTimeOffset(location, date, 90.83, true);
        var sunsetOffset = CalculateSolarTimeOffset(location, date, 90.83, false);

        var sunrise = date.ResetTimeOfDay().Add(sunriseOffset);
        var sunset = date.ResetTimeOfDay().Add(sunsetOffset);

        return new SolarTimes
        {
            Sunrise = sunrise,
            Sunset = sunset
        };
    }
}

Since it's an MVC web app, we will also have a controller that provides endpoints to expose the app's functionality:

[ApiController]
[Route("solartimes")]
public class SolarTimeController : ControllerBase
{
    private readonly SolarCalculator _solarCalculator;
    private readonly CachingLayer _cachingLayer;

    public SolarTimeController(SolarCalculator solarCalculator, CachingLayer cachingLayer)
    {
        _solarCalculator = solarCalculator;
        _cachingLayer = cachingLayer;
    }

    [HttpGet("by_ip")]
    public async Task<IActionResult> GetByIp(DateTimeOffset? date)
    {
        var ip = HttpContext.Connection.RemoteIpAddress;
        var cacheKey = $"{ip},{date}";

        var cachedSolarTimes = await _cachingLayer.TryGetAsync<SolarTimes>(cacheKey);
        if (cachedSolarTimes != null)
            return Ok(cachedSolarTimes);

        var solarTimes = await _solarCalculator.GetSolarTimesAsync(ip, date ?? DateTimeOffset.Now);
        await _cachingLayer.SetAsync(cacheKey, solarTimes);

        return Ok(solarTimes);
    }

    [HttpGet("by_location")]
    public async Task<IActionResult> GetByLocation(double lat, double lon, DateTimeOffset? date)
    {
        /* ... */
    }
}

As seen above, the /solartimes/by_ip endpoint mostly just delegates execution to SolarCalculator, but also has very simple caching logic to avoid redundant requests to 3rd party services. The caching is done by the CachingLayer class which encapsulates a Redis client used to store and retrieve JSON content:

public class CachingLayer
{
    private readonly IConnectionMultiplexer _redis;

    public CachingLayer(IConnectionMultiplexer connectionMultiplexer) =>
        _redis = connectionMultiplexer;

    public async Task<T> TryGetAsync<T>(string key) where T : class
    {
        var result = await _redis.GetDatabase().StringGetAsync(key);

        if (result.HasValue)
            return JsonSerializer.Deserialize<T>(result.ToString());

        return null;
    }

    public async Task SetAsync<T>(string key, T obj) where T : class =>
        await _redis.GetDatabase().StringSetAsync(key, JsonSerializer.Serialize(obj));
}

Finally, all of the above parts are wired together in the Startup class by configuring request pipeline and registering required services:

public class Startup
{
    private readonly IConfiguration _configuration;

    public Startup(IConfiguration configuration) =>
        _configuration = configuration;

    private string GetRedisConnectionString() =>
        _configuration.GetConnectionString("Redis");

    public void ConfigureServices(IServiceCollection services)
    {
        services.AddMvc(o => o.EnableEndpointRouting = false);

        services.AddSingleton<IConnectionMultiplexer>(
            ConnectionMultiplexer.Connect(GetRedisConnectionString()));

        services.AddSingleton<CachingLayer>();

        services.AddHttpClient<LocationProvider>();
        services.AddTransient<SolarCalculator>();
    }

    public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
    {
        if (env.IsDevelopment())
            app.UseDeveloperExceptionPage();

        app.UseMvcWithDefaultRoute();
    }
}

Note that we didn't have our classes implement any autotelic interfaces because we're not planning to use mocks. It may happen that we will need to substitute one of the services in tests but it's not yet clear now, so we avoid unnecessary work (and design damage) until we're sure we need it.

Although it's a rather simple project, this app already incorporates a decent amount of infrastructural complexity by relying on a 3rd party web service (GeoIP provider) as well as a persistence layer (Redis). This is a rather common setup which a lot of real-life projects can relate to.

With a classical approach focused on unit testing, we would find ourselves targeting the service layer and possibly the controller layer of our app, writing isolated tests that ensure that every branch of code executes correctly. Doing that would be useful to an extent but could never give us confidence that the actual endpoints, with all of the middleware and peripheral components, work as intended.

Instead, we will write tests that target the endpoints directly. To do that, we will need to create a separate testing project and add a few infrastructural components that will support our tests. One of them is FakeApp which is going to be used to encapsulate a virtual instance of our app:

public class FakeApp : IDisposable
{
    private readonly WebApplicationFactory<Startup> _appFactory;

    public HttpClient Client { get; }

    public FakeApp()
    {
        _appFactory = new WebApplicationFactory<Startup>();
        Client = _appFactory.CreateClient();
    }

    public void Dispose()
    {
        Client.Dispose();
        _appFactory.Dispose();
    }
}

The majority of the work here is already done by WebApplicationFactory, which is a utility provided by the framework that allows us to bootstrap the app in-memory for testing purposes. It also provides us with API to override configuration, service registrations, and the request pipeline if needed.

We can use an instance of this object in tests to run the app, send requests with the provided HttpClient, and then check that the response matches our expectations. This instance can be either shared among multiple tests or instead created separately for each one.

Since we also rely on Redis, we want to have a way to spin up a fresh server to be used by our app. There are many ways to do it, but for a simple example I decided to use xUnit's fixture API for this purpose:

public class RedisFixture : IAsyncLifetime
{
    private string _containerId;

    public async Task InitializeAsync()
    {
        // Simplified, but ideally should bind to a random port
        var result = await Cli.Wrap("docker")
            .WithArguments("run -d -p 6379:6379 redis")
            .ExecuteBufferedAsync();

        _containerId = result.StandardOutput.Trim();
    }

    public async Task ResetAsync() =>
        await Cli.Wrap("docker")
            .WithArguments($"exec {_containerId} redis-cli FLUSHALL")
            .ExecuteAsync();

    public async Task DisposeAsync() =>
        await Cli.Wrap("docker")
            .WithArguments($"container kill {_containerId}")
            .ExecuteAsync();
}

The above code works by implementing the IAsyncLifetime interface that lets us define methods which are going to be executed before and after the tests run. We are using these methods to start a Redis container in Docker and then kill it once the testing has finished.

Besides that, the RedisFixture class also exposes ResetAsync method which can be used to execute the FLUSHALL command to delete all keys from the database. We will be calling this method to reset Redis to a clean slate before each test. As an alternative, we could also just restart the container instead, which takes a bit longer but is probably more reliable.

Now that the infrastructure is set up, we can move on to writing our first test:

public class SolarTimeSpecs : IClassFixture<RedisFixture>, IAsyncLifetime
{
    private readonly RedisFixture _redisFixture;

    public SolarTimeSpecs(RedisFixture redisFixture)
    {
        _redisFixture = redisFixture;
    }

    // Reset Redis before each test
    public async Task InitializeAsync() => await _redisFixture.ResetAsync();

    [Fact]
    public async Task User_can_get_solar_times_for_their_location_by_ip()
    {
        // Arrange
        using var app = new FakeApp();

        // Act
        var response = await app.Client.GetStringAsync("/solartimes/by_ip");
        var solarTimes = JsonSerializer.Deserialize<SolarTimes>(response);

        // Assert
        solarTimes.Sunset.Should().BeWithin(TimeSpan.FromDays(1)).After(solarTimes.Sunrise);
        solarTimes.Sunrise.Should().BeCloseTo(DateTimeOffset.Now, TimeSpan.FromDays(1));
        solarTimes.Sunset.Should().BeCloseTo(DateTimeOffset.Now, TimeSpan.FromDays(1));
    }
}

As you can see, the setup is really simple. All we need to do is create an instance of FakeApp and use the provided HttpClient to send requests to one of the endpoints, just like you would if it was a real web app.

This specific test works by querying the /solartimes/by_ip route, which determines user's sunrise and sunset times for the current date based on their IP. Since we're relying on an actual GeoIP provider and don't know what the result is going to be, we're performing property-based assertions to ensure that the solar times are valid.

Although those assertions will be able to catch a multitude of potential bugs, it doesn't give us full confidence that the result is fully correct. There are a couple of different ways we can improve on this, however.

An obvious option would be to replace the real GeoIP provider with a fake instance that will always return the same location, allowing us to hard-code the expected solar times. The downside of doing that is that we will be effectively reducing the integration scope, which means we won't be able to verify that our app talks to the 3rd party service correctly.

As an alternative approach, we can instead substitute the IP address that the test server receives from the client. This way we can make the test more strict, while maintaining the same integration scope.

To accomplish this, we will need to create a startup filter that lets us inject a custom IP address into request context using middleware:

public class FakeIpStartupFilter : IStartupFilter
{
    public IPAddress Ip { get; set; } = IPAddress.Parse("::1");

    public Action<IApplicationBuilder> Configure(Action<IApplicationBuilder> nextFilter)
    {
        return app =>
        {
            app.Use(async (ctx, next) =>
            {
                ctx.Connection.RemoteIpAddress = Ip;
                await next();
            });

            nextFilter(app);
        };
    }
}

We can then wire it into FakeApp by registering it as a service:

public class FakeApp : IDisposable
{
    private readonly WebApplicationFactory<Startup> _appFactory;
    private readonly FakeIpStartupFilter _fakeIpStartupFilter = new FakeIpStartupFilter();

    public HttpClient Client { get; }

    public IPAddress ClientIp
    {
        get => _fakeIpStartupFilter.Ip;
        set => _fakeIpStartupFilter.Ip = value;
    }

    public FakeApp()
    {
        _appFactory = new WebApplicationFactory<Startup>().WithWebHostBuilder(o =>
        {
            o.ConfigureServices(s =>
            {
                s.AddSingleton<IStartupFilter>(_fakeIpStartupFilter);
            });
        });

        Client = _appFactory.CreateClient();
    }

    /* ... */
}

Now we can update the test to rely on concrete data:

[Fact]
public async Task User_can_get_solar_times_for_their_location_by_ip()
{
    // Arrange
    using var app = new FakeApp
    {
        ClientIp = IPAddress.Parse("20.112.101.1")
    };

    var date = new DateTimeOffset(2020, 07, 03, 0, 0, 0, TimeSpan.FromHours(-5));
    var expectedSunrise = new DateTimeOffset(2020, 07, 03, 05, 20, 37, TimeSpan.FromHours(-5));
    var expectedSunset = new DateTimeOffset(2020, 07, 03, 20, 28, 54, TimeSpan.FromHours(-5));

    // Act
    var query = new QueryBuilder
    {
        {"date", date.ToString("O", CultureInfo.InvariantCulture)}
    };

    var response = await app.Client.GetStringAsync($"/solartimes/by_ip{query}");
    var solarTimes = JsonSerializer.Deserialize<SolarTimes>(response);

    // Assert
    solarTimes.Sunrise.Should().BeCloseTo(expectedSunrise, TimeSpan.FromSeconds(1));
    solarTimes.Sunset.Should().BeCloseTo(expectedSunset, TimeSpan.FromSeconds(1));
}

Some developers might still feel uneasy about relying on a real 3rd party web service in tests, because it may lead to non-deterministic results. Conversely, one can argue that we do actually want our tests to incorporate that dependency, because we want to be aware if it breaks or changes in unexpected ways, as it can lead to bugs in our own software.

Of course, using real dependencies is not always possible, for example if the service has usage quotas, costs money, or is simply slow or unreliable. In such cases we would want to replace it with a fake (preferably not mocked) implementation to be used in tests instead. This, however, is not one of those cases.

Similarly to how we did with the first one, we can also write a test that covers the second endpoint. This one is simpler because all input parameters are passed directly as part of URL query:

[Fact]
public async Task User_can_get_solar_times_for_a_specific_location_and_date()
{
    // Arrange
    using var app = new FakeApp();

    var date = new DateTimeOffset(2020, 07, 03, 0, 0, 0, TimeSpan.FromHours(+3));
    var expectedSunrise = new DateTimeOffset(2020, 07, 03, 04, 52, 23, TimeSpan.FromHours(+3));
    var expectedSunset = new DateTimeOffset(2020, 07, 03, 21, 11, 45, TimeSpan.FromHours(+3));

    // Act
    var query = new QueryBuilder
    {
        {"lat", "50.45"},
        {"lon", "30.52"},
        {"date", date.ToString("O", CultureInfo.InvariantCulture)}
    };

    var response = await app.Client.GetStringAsync($"/solartimes/by_location{query}");
    var solarTimes = JsonSerializer.Deserialize<SolarTimes>(response);

    // Assert
    solarTimes.Sunrise.Should().BeCloseTo(expectedSunrise, TimeSpan.FromSeconds(1));
    solarTimes.Sunset.Should().BeCloseTo(expectedSunset, TimeSpan.FromSeconds(1));
}

We can keep adding tests like this one to ensure that the app supports all possible locations, dates, and handles potential edge cases such as the midnight sun phenomenon. However, you may not want to re-test the entire pipeline each time but focus just on the business logic that calculates the solar times.

Doing that would imply that we need to isolate SolarCalculator from LocationProvider somehow and that in turn implies mocking which we want to avoid. Fortunately, there is a more clever way to achieve that.

We can alter the implementation of SolarCalculator by separating the pure and impure parts of the code away from each other:

public class SolarCalculator
{
    private static TimeSpan CalculateSolarTimeOffset(Location location, DateTimeOffset instant,
        double zenith, bool isSunrise)
    {
        /* ... */
    }

    public SolarTimes GetSolarTimes(Location location, DateTimeOffset date)
    {
        var sunriseOffset = CalculateSolarTimeOffset(location, date, 90.83, true);
        var sunsetOffset = CalculateSolarTimeOffset(location, date, 90.83, false);

        var sunrise = date.ResetTimeOfDay().Add(sunriseOffset);
        var sunset = date.ResetTimeOfDay().Add(sunsetOffset);

        return new SolarTimes
        {
            Sunrise = sunrise,
            Sunset = sunset
        };
    }
}

What changed is now, instead of relying on LocationProvider to provide it, the GetSolarTimes method takes location as an explicit parameter. Doing that means that we also no longer require dependency inversion, as there are no dependencies to invert.

To wire everything back together, all we need to do is update the controller:

[ApiController]
[Route("solartimes")]
public class SolarTimeController : ControllerBase
{
    private readonly SolarCalculator _solarCalculator;
    private readonly LocationProvider _locationProvider;
    private readonly CachingLayer _cachingLayer;

    public SolarTimeController(
        SolarCalculator solarCalculator,
        LocationProvider locationProvider,
        CachingLayer cachingLayer)
    {
        _solarCalculator = solarCalculator;
        _locationProvider = locationProvider;
        _cachingLayer = cachingLayer;
    }

    [HttpGet("by_ip")]
    public async Task<IActionResult> GetByIp(DateTimeOffset? date)
    {
        var ip = HttpContext.Connection.RemoteIpAddress;
        var cacheKey = ip.ToString();

        var cachedSolarTimes = await _cachingLayer.TryGetAsync<SolarTimes>(cacheKey);
        if (cachedSolarTimes != null)
            return Ok(cachedSolarTimes);

        // Composition instead of dependency injection
        var location = await _locationProvider.GetLocationAsync(ip);
        var solarTimes = _solarCalculator.GetSolarTimes(location, date ?? DateTimeOffset.Now);

        await _cachingLayer.SetAsync(cacheKey, solarTimes);

        return Ok(solarTimes);
    }

    /* ... */
}

Since our existing tests are not aware of implementation details, this simple refactoring didn't break them in any way (which would not be the case with unit tests). With that done, we can write some additional light-weight tests to cover the business logic more extensively, while still not mocking anything:

[Fact]
public void User_can_get_solar_times_for_New_York_in_November()
{
    // Arrange
    var location = new Location
    {
        Latitude = 40.71,
        Longitude = -74.00
    };

    var date = new DateTimeOffset(2019, 11, 04, 00, 00, 00, TimeSpan.FromHours(-5));
    var expectedSunrise = new DateTimeOffset(2019, 11, 04, 06, 29, 34, TimeSpan.FromHours(-5));
    var expectedSunset = new DateTimeOffset(2019, 11, 04, 16, 49, 04, TimeSpan.FromHours(-5));

    // Act
    var solarTimes = new SolarCalculator().GetSolarTimes(location, date);

    // Assert
    solarTimes.Sunrise.Should().BeCloseTo(expectedSunrise, TimeSpan.FromSeconds(1));
    solarTimes.Sunset.Should().BeCloseTo(expectedSunset, TimeSpan.FromSeconds(1));
}

[Fact]
public void User_can_get_solar_times_for_Tromso_in_January()
{
    // Arrange
    var location = new Location
    {
        Latitude = 69.65,
        Longitude = 18.96
    };

    var date = new DateTimeOffset(2020, 01, 03, 00, 00, 00, TimeSpan.FromHours(+1));
    var expectedSunrise = new DateTimeOffset(2020, 01, 03, 11, 48, 31, TimeSpan.FromHours(+1));
    var expectedSunset = new DateTimeOffset(2020, 01, 03, 11, 48, 45, TimeSpan.FromHours(+1));

    // Act
    var solarTimes = new SolarCalculator().GetSolarTimes(location, date);

    // Assert
    solarTimes.Sunrise.Should().BeCloseTo(expectedSunrise, TimeSpan.FromSeconds(1));
    solarTimes.Sunset.Should().BeCloseTo(expectedSunset, TimeSpan.FromSeconds(1));
}

Although these tests no longer exercise the full integration scope, they are still driven by functional requirements of the app. Because we already have another high-level test that covers the entire endpoint, we can keep these ones more narrow without sacrificing overall confidence. This trade-off makes sense if we're trying to improve execution speed, but I would recommend to stick to high-level tests as much as possible, at least until it becomes a problem.

Finally, we may also want to do something to ensure that our Redis caching layer works correctly as well. Even though we're using it in our tests, it never actually returns a cached response because the database gets reset between tests.

The problem with testing things like caching is that they can't be defined by functional requirements. A user, with no awareness of the app's internal affairs, has no way of knowing whether the responses are returned from cache or not.

However, if our goal is only to test the integration between the app and Redis, we don't need to write implementation-aware tests and can do something like this instead:

[Fact]
public async Task User_can_get_solar_times_for_their_location_by_ip_multiple_times()
{
    // Arrange
    using var app = new FakeApp();

    // Act
    var collectedSolarTimes = new List<SolarTimes>();

    for (var i = 0; i < 3; i++)
    {
        var response = await app.Client.GetStringAsync("/solartimes/by_ip");
        var solarTimes = JsonSerializer.Deserialize<SolarTimes>(response);

        collectedSolarTimes.Add(solarTimes);
    }

    // Assert
    collectedSolarTimes.Select(t => t.Sunrise).Distinct().Should().ContainSingle();
    collectedSolarTimes.Select(t => t.Sunset).Distinct().Should().ContainSingle();
}

The test will query the same endpoint multiple times and assert that the result always remains the same. This is enough to ensure that the responses are cached properly and then returned in the same way as normal responses.

At the end of the day we have a simple test suite that looks like this:

Note that the duration of the tests is pretty good, with the fastest integration test completing at 55ms and the slowest being under a second (due to suffering from cold start). Considering that these tests involve the entire lifecycle, include all dependencies and infrastructure, while relying on a grand total of zero mocks, I would say that this is more than acceptable.

If you want to tinker with the example project yourself, you can find it on GitHub.

Drawbacks and considerations

Unfortunately, there is no silver bullet and the approaches described in this article also suffer from some potential drawbacks. In the interest of fairness, it makes sense to mention them as well.

One of the biggest challenges I've found when doing high-level functional testing is figuring out a good balance between usefulness and usability. Compared to approaches that focus specifically on unit testing, it does take more effort to ensure that such tests are sufficiently deterministic, don't take too long, can run independently of each other, and are generally usable during development.

The wide scope of tests also implies the need for a deeper understanding of the project's dependencies and technologies it relies upon. It's important to know how they're used, whether they can be easily containerized, which options are available and what are the trade-offs.

In the context of integration testing, the "testability" aspect is not defined by how well the code can be isolated, but instead by how well the actual infrastructure accommodates and facilitates testing. This puts a certain prerequisite on the responsible person and the team in general in terms of technical expertise.

It may also take some time to set up and configure the testing environment, as it includes creating fixtures, wiring fake implementations, adding custom initialization and cleanup behavior, and so on. All these things need to be maintained as the project scales and becomes more complicated.

Writing functional tests in itself involves a bit more planning as well, because it's no longer just about covering every method of every class, but rather about outlining software requirements and turning them into code. Understanding what those requirements are and which of them are functional can also be tricky sometimes, as it requires an ability to think from a user's perspective.

Another common concern is that high-level tests often suffer from a lack of locality. If a test fails, either due to unmet expectations or because of an unhandled exception, it's usually unclear what exactly caused the error.

Although there are ways to mitigate this issue, ultimately it's always going to be a trade-off: isolated tests are better at indicating the cause of an error, while integrated tests are better at highlighting the impact. Both are equally useful, so it comes down to what you consider to be more important.

At the end of the day, I still think functional testing is worth it even despite these shortcomings, as I find that it leads to a better developer experience overall. It's been a while since I've done classic unit testing and I'm not looking forward to starting again.

Summary

Unit testing is a popular approach for testing software, but mostly for the wrong reasons. It's often touted as an effective way for developers to test their code while also enforcing best design practices, however many find it encumbering and superficial.

It's important to understand that development testing does not equate to unit testing. The primary goal is not to write tests which are as isolated as possible, but rather to gain confidence that the code works according to its functional requirements. And there are better ways to achieve that.

Writing high-level tests that are driven by user behavior will provide you with much higher return on investment in the long run, and it isn't as hard as it seems. Find an approach that makes the most sense for your project and stick to it.

Here are the main takeaways:

Think critically and challenge best practices
Don't rely on the the test pyramid
Separate tests by functionality, rather than by classes, modules, or scope
Aim for the highest level of integration while maintaining reasonable speed and cost
Avoid sacrificing software design for testability
Consider mocking only as a last resort

There are also other great articles about alternative testing approaches in modern software development. These are the ones I've personally found really interesting:

Follow me on Twitter to get notified when I post a new article ✨

Simulating Return Type Inference in C#

Oleksii Holub — Tue, 10 Mar 2020 09:32:34 +0000

Above everything else in software development, I really enjoy building frameworks that enable other developers to create something cool. Sometimes, when chasing that perfect design I have in mind, I find myself coming up with weird hacks that really push the C# language to the limit.

One such case happened not so long ago, when my coworker and I were looking at how to avoid specifying generic arguments, in places where the compiler should seemingly be able to guess it based on the return type. He said it was impossible seeing as C# can only infer generic arguments from method parameters -- however I was able to come up with a way to prove him wrong.

In this article I will show a little trick I came up with to simulate return type inference, as well as some examples where that can be useful.

Type inference

Type inference, in general, is the ability of a compiler to automatically detect the type of a particular expression, without having the programmer explicitly specify it. This feature works by analyzing the context in which the expression is evaluated, as well as the constraints imposed by the flow of data in the program.

By being able to detect the type automatically, languages that support type inference allow writing more succinct code, while still maintaining the full benefits of a static type system. This is why most mainstream statically-typed languages have type inference, in one form or another.

C#, being one of those languages, has type inference as well. The simplest possible example of it is the var keyword:

var x = 5;              // int
var y = "foo";          // string
var z = 2 + 1.0;        // double
var g = Guid.NewGuid(); // Guid

When doing a combined declaration and assignment operation with the var keyword, you don't need to specify the type of the variable. The compiler is able to detect it on its own based on the expression on the right side.

In a similar vein, C# also allows initializing an array without having to manually specify its type:

var array = new[] {"Hello", "world"}; // string[]

In this case, the compiler can see that we're initializing the array with two string elements so it can safely conclude that the resulting array is of type string[]. In some (very rare) cases, it can even infer the type of the array based on the most specific common type among the individual elements:

var array = new[] {1, 2, 3.0}; // double[]

However, the most interesting aspect of type inference in C# is, of course, generic methods. When calling a method with a generic signature, we can omit type arguments as long as they can be deduced from the values passed to the method parameters.

For example, we can define a generic method List.Create<T> that creates a list from a sequence of elements:

public static class List
{
    public static List<T> Create<T>(params T[] items) => new List<T>(items);
}

Which in turn can be used like this:

var list = List.Create(1, 3, 5); // List<int>

In the above scenario we could've specified the type explicitly by writing List.Create<int>(...) but we didn't have to. The compiler is able to detect it automatically based on the parameters we passed into the method, which are constrained by the same type as the returned list itself.

Interestingly enough, all of the examples shown above are in fact based on the same form of type inference, which works by analyzing the constraints imposed by other expressions, whose type is already known. In other words, it examines the flow of data that goes in and draws conclusions about the data that goes out.

There are scenarios, however, where we may want the type inference to work in the opposite direction. Let's see where that could be useful.

Option type

If you have been writing code in a functional style before, it's very likely that you're intimately familiar with the Option<T> type. It's a container that encapsulates a single value (or absence thereof) and allows us to perform various operations on the content without actually observing its state.

In C#, an option type is usually defined by encapsulating two fields -- a generic value and a flag that indicates whether that value is actually set. It could look something like this:

public readonly struct Option<T>
{
    private readonly T _value;
    private readonly bool _hasValue;

    private Option(T value, bool hasValue)
    {
        _value = value;
        _hasValue = hasValue;
    }

    public Option(T value)
        : this(value, true)
    {
    }

    public TOut Match<TOut>(Func<T, TOut> some, Func<TOut> none) =>
        _hasValue ? some(_value) : none();

    public void Match(Action<T> some, Action none)
    {
        if (_hasValue)
            some(_value);
        else
            none();
    }

    public Option<TOut> Select<TOut>(Func<T, TOut> map) =>
        _hasValue ? new Option<TOut>(map(_value)) : new Option<TOut>();

    public Option<TOut> Bind<TOut>(Func<T, Option<TOut>> bind) =>
        _hasValue ? bind(_value) : new Option<TOut>();
}

This API design is fairly basic. The implementation above hides the actual value away from consumers, surfacing it only through the Match(...) method, which unwraps the container by handling both of its potential states. Additionally, there are Select(...) and Bind(...) methods that can be used to safely transform the value, regardless of whether it's actually been set or not.

Also, in this example, Option<T> is defined as a readonly struct. Seeing as it's mainly returned from methods and used in local scopes, this decision makes sense from a performance point of view.

Just to make things convenient, we may also want to provide factory methods that help construct new instances of Option<T> in a more fluent manner:

public static class Option
{
    public static Option<T> Some<T>(T value) => new Option<T>(value);

    public static Option<T> None<T>() => new Option<T>();
}

Which can be used like this:

public static Option<int> Parse(string number)
{
    return int.TryParse(number, out var value)
        ? Option.Some(value)
        : Option.None<int>();
}

As you can see, in case with Option.Some<T>(...), we can drop the generic argument because the compiler can infer it based on the type of value, which is int. On the other hand, the same doesn't work with Option.None<T>(...) because it doesn't have any parameters, hence why we need to specify the type manually.

Even though the type argument for Option.None<T>(...) seems to be inherently obvious from the context, the compiler is not able to deduce it. This is because, as mentioned earlier, type inference in C# only works by analyzing the data that flows in and not the other way around.

Of course, ideally, we would want the compiler to figure out the type of T in Option.None<T>(...) based on the return type this expression is expected to have, as dictated by the signature of the containing method. If not, we would want it to at least get the T from the first branch of the conditional expression, where it was already inferred from value.

Unfortunately, neither of these is possible with C#'s type system because it would need to work out the type in reverse, which it cannot do. That said, we can help it a little bit.

We can simulate return type inference by having Option.None return a special non-generic type representing an option with deferred initialization, which can be coerced into Option<T>. Here's how that would look:

public readonly struct Option<T>
{
    private readonly T _value;
    private readonly bool _hasValue;

    private Option(T value, bool hasValue)
    {
        _value = value;
        _hasValue = hasValue;
    }

    public Option(T value)
        : this(value, true)
    {
    }

    // ...

    public static implicit operator Option<T>(NoneOption none) => new Option<T>();
}

public readonly struct NoneOption
{
}

public static class Option
{
    public static Option<T> Some<T>(T value) => new Option<T>(value);

    public static NoneOption None { get; } = new NoneOption();
}

As you can see, Option.None now returns a dummy NoneOption object, which essentially represents an empty option whose type hasn't been decided yet. Because NoneOption is not generic, we were able to drop the generic arguments and turn Option.None into a property.

Additionally, we made it so Option<T> implements an implicit conversion from NoneOption. Although operators themselves can't be generic in C#, they still retain type arguments of the declaring type, which allows us to define this conversion for every possible variant of Option<T>.

All of this allows us to write Option.None and have the compiler coerce it automatically to the destination type. From the consumer's point of view, it looks as though we've implemented return type inference:

public static Option<int> Parse(string number)
{
    return int.TryParse(number, out var value)
        ? Option.Some(value)
        : Option.None;
}

Result type

Just like we did with Option<T>, we may want to apply the same treatment to Result<TOk, TError>. This type fulfills a similar purpose, except that it also has a fully fledged value representing the negative case, instead of just being empty.

Here's how we could implement it:

public readonly struct Result<TOk, TError>
{
    private readonly TOk _ok;
    private readonly TError _error;
    private readonly bool _isError;

    private Result(TOk ok, TError error, bool isError)
    {
        _ok = ok;
        _error = error;
        _isError = isError;
    }

    public Result(TOk ok)
        : this(ok, default, false)
    {
    }

    public Result(TError error)
        : this(default, error, true)
    {
    }

    // ...
}

public static class Result
{
    public static Result<TOk, TError> Ok<TOk, TError>(TOk ok) =>
        new Result<TOk, TError>(ok);

    public static Result<TOk, TError> Error<TOk, TError>(TError error) =>
        new Result<TOk, TError>(error);
}

And here's how it would then be used:

public static Result<int, string> Parse(string input)
{
    return int.TryParse(input, out var value)
        ? Result.Ok<int, string>(value)
        : Result.Error<int, string>("Invalid value");
}

As you can see, the situation regarding type inference is even more dire here. Neither Result.Ok<TOk, TError>(...) nor Result.Error<TOk, TError>(...) have enough parameters to infer both generic arguments, so we are forced to specify them manually in both cases.

Having to write out these types every time leads to visual noise, code duplication, and bad developer experience in general. Let's try to rectify this using the same technique from earlier:

public readonly struct Result<TOk, TError>
{
    private readonly TOk _ok;
    private readonly TError _error;
    private readonly bool _isError;

    private Result(TOk ok, TError error, bool isError)
    {
        _ok = ok;
        _error = error;
        _isError = isError;
    }

    public Result(TOk ok)
        : this(ok, default, false)
    {
    }

    public Result(TError error)
        : this(default, error, true)
    {
    }

    public static implicit operator Result<TOk, TError>(DelayedResult<TOk> ok) =>
        new Result<TOk, TError>(ok.Value);

    public static implicit operator Result<TOk, TError>(DelayedResult<TError> error) =>
        new Result<TOk, TError>(error.Value);
}

public readonly struct DelayedResult<T>
{
    public T Value { get; }

    public DelayedResult(T value)
    {
        Value = value;
    }
}

public static class Result
{
    public static DelayedResult<TOk> Ok<TOk>(TOk ok) =>
        new DelayedResult<TOk>(ok);

    public static DelayedResult<TError> Error<TError>(TError error) =>
        new DelayedResult<TError>(error);
}

Here we similarly defined DelayedResult<T> that represents the initialized part of Result<TOk, TError>. Again, we're using implicit conversion operators to coerce the delayed instance into the destination type.

Doing all that enables us to rewrite our code like this:

public static Result<int, string> Parse(string input)
{
    return int.TryParse(input, out var value)
        ? (Result<int, string>) Result.Ok(value)
        : Result.Error("Invalid value");
}

This is a bit better but not ideal. The problem here is that the conditional expression in C# doesn't coerce its branches directly to the expected type, but instead tries to convert the type of the negative branch into the type of the positive branch. Because of that, we need to explicitly cast the positive branch into Result<int, string> to specify the common denominator.

However, this issue can be completely avoided if we just use a conditional statement instead:

public static Result<int, string> Parse(string input)
{
    if (int.TryParse(input, out var value))
        return Result.Ok(value);

    return Result.Error("Invalid value");
}

I'm much more satisfied with this setup. We were able to drop the generic arguments entirely, while maintaining the same signature and type safety as before. Again, from a high-level perspective this may look as if the generic arguments were somehow inferred from the expected return type.

However, you may have noticed that there's a bug in the implementation. If the types of TOk and TError are the same, there's an ambiguity as to which state DelayedResult<T> actually represents.

For example, imagine we were using our result type in the following scenario:

public interface ITranslationService
{
    Task<bool> IsLanguageSupportedAsync(string language);

    Task<string> TranslateAsync(string text, string targetLanguage);
}

public class Translator
{
    private readonly ITranslationService _translationService;

    public Translator(ITranslationService translationService)
    {
        _translationService = translationService;
    }

    public async Task<Result<string, string>> TranslateAsync(string text, string language)
    {
        if (!await _translationService.IsLanguageSupportedAsync(language))
            return Result.Error($"Language {language} is not supported");

        var translated = await _translationService.TranslateAsync(text, language);
        return Result.Ok(translated);
    }
}

Here Result.Error<TError>(...) and Result.Ok<TOk>(...) both return DelayedResult<string>, so the compiler struggles to figure out what to do with it:

Cannot convert expression type 'DelayedResult<string>' to return type 'Result<string,string>'

Luckily, the fix is simple -- we just need to represent each of the individual states separately:

public readonly struct Result<TOk, TError>
{
    private readonly TOk _ok;
    private readonly TError _error;
    private readonly bool _isError;

    private Result(TOk ok, TError error, bool isError)
    {
        _ok = ok;
        _error = error;
        _isError = isError;
    }

    public Result(TOk ok)
        : this(ok, default, false)
    {
    }

    public Result(TError error)
        : this(default, error, true)
    {
    }

    public static implicit operator Result<TOk, TError>(DelayedOk<TOk> ok) =>
        new Result<TOk, TError>(ok.Value);

    public static implicit operator Result<TOk, TError>(DelayedError<TError> error) =>
        new Result<TOk, TError>(error.Value);
}

public readonly struct DelayedOk<T>
{
    public T Value { get; }

    public DelayedOk(T value)
    {
        Value = value;
    }
}

public readonly struct DelayedError<T>
{
    public T Value { get; }

    public DelayedError(T value)
    {
        Value = value;
    }
}

public static class Result
{
    public static DelayedOk<TOk> Ok<TOk>(TOk ok) =>
        new DelayedOk<TOk>(ok);

    public static DelayedError<TError> Error<TError>(TError error) =>
        new DelayedError<TError>(error);
}

Going back to the code from earlier, it will now work exactly as expected:

public class Translator
{
    private readonly ITranslationService _translationService;

    public Translator(ITranslationService translationService)
    {
        _translationService = translationService;
    }

    public async Task<Result<string, string>> TranslateAsync(string text, string language)
    {
        if (!await _translationService.IsLanguageSupportedAsync(language))
            return Result.Error($"Language {language} is not supported");

        var translated = await _translationService.TranslateAsync(text, language);
        return Result.Ok(translated);
    }
}

Summary

Although type inference in C# has its limits, we can push them a bit further with the help of implicit conversion operators. Using a simple trick shown in this article, we can simulate return type inference, enabling some potentially interesting design opportunities.

Follow me on Twitter to get notified when I post a new article ✨

💻 CliWrap — Execute Shell Commands with Extensive Support for Piping

Oleksii Holub — Thu, 27 Feb 2020 11:17:27 +0000

Hey there!

Recently I've been working on an update for my library, CliWrap, which can be used to automate execution of shell commands from C#. Today I released version 3.0 which is a major change with a lot of improvements, mainly the new support for piping. ⭐

You can download the library from NuGet. In this post I will give a short overview of what you can do with this library. 🚀

Executing a command

The following is a basic example that shows how to asynchronously execute a command by specifying command line arguments:

using CliWrap;

var result = await Cli.Wrap("path/to/exe")
    .WithArguments("--foo bar")
    .ExecuteAsync();

// Result contains:
// -- result.ExitCode        (int)
// -- result.StartTime       (DateTimeOffset)
// -- result.ExitTime        (DateTimeOffset)
// -- result.RunTime         (TimeSpan)

In this scenario, all of the streams are redirected into CliWrap's equivalent of /dev/null, avoiding unnecessary memory allocations. This approach is useful if you want to execute a command, but don't care about what it writes to the console. You still get the returned exit code, which is usually enough to determine whether the command ran successfully or not.

By default, ExecuteAsync() will throw a CommandExecutionException if the underlying process returned a non-zero exit code. You can choose to disable this check.

Command configuration and execution are not coupled in any way so you can separate them:

var cmd = Cli.Wrap("path/to/exe").WithArguments("--foo bar");
var result = await cmd.ExecuteAsync();

Every call to ExecuteAsync() is stateless and will spawn a new process for each execution.

Executing a command with buffering

You can also execute a command while buffering its outputs in memory:

using CliWrap.Buffered;

var result = await Cli.Wrap("path/to/exe")
    .WithArguments("--foo bar")
    .ExecuteBufferedAsync();

// Result contains:
// -- result.StandardOutput  (string)
// -- result.StandardError   (string)
// -- result.ExitCode        (int)
// -- result.StartTime       (DateTimeOffset)
// -- result.ExitTime        (DateTimeOffset)
// -- result.RunTime         (TimeSpan)

Calling ExecuteBufferedAsync() is similar to ExecuteAsync(), but the returned result contains two extra fields: StandardOutput and StandardError. These contain the aggregated text data produced by the underlying command. This approach is useful if you want execute a command and then inspect what it wrote to the console. Note, however, that some commands may produce really large outputs or even write binary content instead of text, in which case it's better to use direct piping methods (explained later).

By default, this method will assume that the underlying command uses Console.OutputEncoding for writing text to the console. If it doesn't, you can override it using one of the overloads:

// Treat both stdout and stderr as UTF8-encoded text streams
var result = await Cli.Wrap("path/to/exe")
    .WithArguments("--foo bar")
    .ExecuteBufferedAsync(Encoding.UTF8);

// Treat stdout as ASCII-encoded and stderr as UTF8-encoded
var result = await Cli.Wrap("path/to/exe")
    .WithArguments("--foo bar")
    .ExecuteBufferedAsync(Encoding.ASCII, Encoding.UTF8);

Getting process ID

The promise returned by ExecuteAsync() and ExecuteBufferedAsync() is in fact not Task<T> but CommandTask<T>. It's a special object that similarly can be awaited, but on top of that it contains information about the ongoing execution.

You can inspect the task object and get the ID of the underlying process that is represented by this command execution:

var task = await Cli.Wrap("path/to/exe")
    .WithArguments("--foo bar")
    .ExecuteAsync();

var processId = task.ProcessId;

await task;

Lazily mapping the result of an execution

Additionally, you can transform the result of CommandTask<T> lazily with the help of Select() method:

// We're only interested in the exit code
var exitCode = await Cli.Wrap("path/to/exe")
    .WithArguments("--foo bar")
    .ExecuteAsync()
    .Select(r => r.ExitCode);

// We're only interested in stdout
var stdOut = await Cli.Wrap("path/to/exe")
    .WithArguments("--foo bar")
    .ExecuteBufferedAsync()
    .Select(r => r.StandardOutput);

Configuring arguments and other options

CliWrap has a fluent interface with various overloads to help configure different options related to command execution. For example, here are three alternative ways you can configure command line arguments:

// Set arguments directly (no formatting, no escaping)
var command = Cli.Wrap("git")
    .WithArguments("clone https://github.com/Tyrrrz/CliWrap --depth 10");

// Set arguments from a list (joined to a string, with escaping)
var command = Cli.Wrap("git")
    .WithArguments(new[] {"clone", "https://github.com/Tyrrrz/CliWrap", "--depth", "10"});

// Build arguments from parts (joined to a string, with formatting and escaping)
var command = Cli.Wrap("git")
    .WithArguments(a => a
        .Add("clone")
        .Add("https://github.com/Tyrrrz/CliWrap")
        .Add("--depth")
        .Add(10));

While all of these approaches can be used interchangeably, the last two take care of escaping automatically for you. Moreover, the builder approach has some overloads to help with formatting, which makes it the preferred approach in most situations.

Besides command line arguments, you can also configure other aspects, such as environment variables and working directory:

var command = Cli.Wrap("git")
    .WithWorkingDirectory("path/to/repo/")
    .WithArguments(a => a
        .Add("commit")
        .Add("-m")
        .Add("my commit"))
    .WithEnvironmentVariables(e => e
        .Set("GIT_AUTHOR_NAME", "John")
        .Set("GIT_AUTHOR_EMAIL", "john@email.com"));

Additionally, you can use WithValidation() to configure whether the command will throw an exception in case the execution finishes with a non-zero exit code:

var commandNoCheck = Cli.Wrap("git").WithValidation(CommandResultValidation.None);
var commandWithCheck = Cli.Wrap("git").WithValidation(CommandResultValidation.ZeroExitCode); // (default)

Note that each call to any of these mentioned WithXyz() methods returns a completely new immutable object, with the corresponding property set to the specified value. That means you can safely re-use parts of your commands as you see fit:

var command1 = Cli.Wrap("git")
    .WithWorkingDirectory("path/to/repo/")
    .WithArguments("--version")
    .WithEnvironmentVariables(e => e
        .Set("GIT_AUTHOR_NAME", "John")
        .Set("GIT_AUTHOR_EMAIL", "john@email.com"));

var command2 = command1.WithArguments("pull");

In the above example, command1 and command2 are separate objects, sharing all configuration options except command line arguments.

Timeout and cancellation

Command execution is asynchronous by nature because it involves a completely separate process. Often you may want to implement an abortion mechanism to stop the execution before it finishes, either by a manual trigger or a timeout.

To do that with CliWrap, you simply need to pass a CancellationToken that represents the cancellation signal:

using var cts = new CancellationTokenSource();

// Cancel automatically after a timeout of 10 seconds
cts.CancelAfter(TimeSpan.FromSeconds(10));

// If this is canceled, a TaskCanceledException is thrown
var result = await Cli.Wrap("path/to/exe").ExecuteAsync(cts.Token);

When an execution is canceled, the underlying process is killed and the ExecuteAsync() method throws an exception of type OperationCanceledException (or its derivative, TaskCanceledException). You will have to catch this exception to recover from cancellation.

All other execution models like ExecuteBufferedAsync() similarly accept cancellation tokens as well.

Executing a command as an event stream

Besides executing a command as a task, CliWrap also supports an alternative model, in which an execution is represented as an event stream. In this scenario, the underlying command may trigger the following events:

StartedCommandEvent -- received just once, when the command starts executing. Contains the process ID.
StandardOutputCommandEvent -- received every time the underlying process writes a new line to the output stream. Contains the text as string.
StandardErrorCommandEvent -- received every time the underlying process writes a new line to the error stream. Contains the text as string.
ExitedCommandEvent -- received just once, when the command successfully finishes executing. Contains the exit code.

There are two ways you can start a command and listen to its events. One of them is through an asynchronous pull-based stream:

using CliWrap.EventStream;

var cmd = Cli.Wrap("foo").WithArguments("bar");

await foreach (var cmdEvent in cmd.ListenAsync())
{
    switch (cmdEvent)
    {
        case StartedCommandEvent started:
            _output.WriteLine($"Process started; ID: {started.ProcessId}");
            break;
        case StandardOutputCommandEvent stdOut:
            _output.WriteLine($"Out> {stdOut.Text}");
            break;
        case StandardErrorCommandEvent stdErr:
            _output.WriteLine($"Err> {stdErr.Text}");
            break;
        case ExitedCommandEvent exited:
            _output.WriteLine($"Process exited; Code: {exited.ExitCode}");
            break;
    }
}

The ListenAsync() method starts the command and returns an object of type IAsyncEnumreable<CommandEvent>, which you can iterate over using the await foreach construct introduced with C# 8. In this scenario, back-pressure is performed by locking the pipes until an event is processed, which means there's no buffering of data in memory.

Alternatively, you can also start a command as an observable push-based stream instead:

using CliWrap.EventStream;
using System.Reactive;

await cmd.Observe().ForEachAsync(cmdEvent =>
{
    switch (cmdEvent)
    {
        case StartedCommandEvent started:
            _output.WriteLine($"Process started; ID: {started.ProcessId}");
            break;
        case StandardOutputCommandEvent stdOut:
            _output.WriteLine($"Out> {stdOut.Text}");
            break;
        case StandardErrorCommandEvent stdErr:
            _output.WriteLine($"Err> {stdErr.Text}");
            break;
        case ExitedCommandEvent exited:
            _output.WriteLine($"Process exited; Code: {exited.ExitCode}");
            break;
    }
});

In this case, Observe() starts the command and returns an IObservable<CommandEvent>. You can use the set of extensions provided by Rx.NET to transform, filter, throttle, and otherwise manipulate the stream. There is no locking in this scenario so the data is pushed as soon as it's available.

Both ListenAsync() and Observe() also have overloads that accept custom encoding and/or a cancellation token.

Piping

Most of the features you've seen so far are based on CliWrap's core model of piping. It lets you redirect input and output streams of the underlying process to form a complex execution pipeline.

To facilitate piping, Command object has three methods:

WithStandardInputPipe(PipeSource source)
WithStandardOutputPipe(PipeTarget target)
WithStandardErrorPipe(PipeTarget target)

By default, every command is piped from PipeSource.Null and is itself piped to PipeTarget.Null, which are CliWrap's equivalents of /dev/null. You can change that and, for example, have the command pipe its standard input from one file and redirect its standard output to another:

await using var input = File.OpenRead("input.txt");
await using var output = File.Create("output.txt");

await Cli.Wrap("foo")
    .WithStandardInputPipe(PipeSource.FromStream(input))
    .WithStandardOutputPipe(PipeTarget.ToStream(output))
    .ExecuteAsync();

The exact same thing can be expressed in a terser way using pipe operators:

await using var input = File.OpenRead("input.txt");
await using var output = File.Create("output.txt");

await (input | Cli.Wrap("foo") | output).ExecuteAsync();

Besides raw streams, PipeSource and PipeTarget both have factory methods that help express different piping directions:

PipeSource.Null -- represents an empty pipe source.
PipeSource.FromStream() -- pipes data from any readable stream.
PipeSource.FromBytes() -- pipes data from a byte array.
PipeSource.FromString() -- pipes from a text string (supports custom encoding)
PipeSource.FromCommand() -- pipes data from standard output of another command.
PipeTarget.Null -- represents a pipe target that discards all data.
PipeTarget.ToStream() -- pipes data into any writeable stream.
PipeTarget.ToStringBuilder() -- pipes data as text into StringBuilder (supports custom encoding).
PipeTarget.ToDelegate() -- pipes data as text, line-by-line, into Action<string> or Func<string, Task> (supports custom encoding).
PipeTarget.Merge() -- merges multiple pipes into one.

The pipe operator also has overloads for most of these. Below you can see some examples of what you can do.

Pipe a string into stdin:

await ("Hello world" | Cli.Wrap("foo")).ExecuteAsync();

Pipe stdout as text into a StringBuilder:

var stdOutBuffer = new StringBuilder();
await (Cli.Wrap("foo") | stdOutBuffer).ExecuteAsync();

Pipe a binary HTTP stream into stdin:

using var httpClient = new HttpClient();
await using var input = await httpClient.GetStreamAsync("https://example.com/image.png");

await (input | Cli.Wrap("foo")).ExecuteAsync();

Pipe stdout of one command into stdin of another:

await (Cli.Wrap("foo") | Cli.Wrap("bar") | Cli.Wrap("baz")).ExecuteAsync();

Pipe stdout and stderr into those of parent process:

await using var stdOut = Console.OpenStandardOutput();
await using var stdErr = Console.OpenStandardError();

var cmd = Cli.Wrap("foo") | (stdOut, stdErr);
await cmd.ExecuteAsync();

var cmd = Cli.Wrap("foo") |
    (Console.WriteLine, Console.Error.WriteLine);

await cmd.ExecuteAsync();

Pipe stdout into a file and stderr into a StringBuilder:

await using var file = File.Create("output.txt");
var buffer = new StringBuilder();

var cmd = Cli.Wrap("foo") |
    (PipeTarget.ToStream(file), PipeTarget.ToStringBuilder(buffer));

await cmd.ExecuteAsync();

Pipe stdout into multiple files simultaneously:

await using var file1 = File.Create("file1.txt");
await using var file2 = File.Create("file2.txt");
await using var file3 = File.Create("file3.txt");

var target = PipeTarget.Merge(
    PipeTarget.ToStream(file1)
    PipeTarget.ToStream(file2)
    PipeTarget.ToStream(file3));

await (Cli.Wrap("foo") | target).ExecuteAsync();

Pipe a string into a command, that command into another command, and then into parent's stdout and stderr:

var cmd = "Hello world" | Cli.Wrap("foo")
    .WithArguments("print random") | Cli.Wrap("bar")
    .WithArguments("reverse") | (Console.WriteLine, Console.Error.WriteLine);

await cmd.ExecuteAsync();

As you can see, piping enables a wide range of different use cases. It's not only used for convenience, but also to improve memory efficiency when dealing with large and/or binary inputs and outputs. With the help of CliWrap's pipe operators, configuring pipelines is really easy -- just imagine doing the same with System.Diagnostics.Process manually.

The different execution models which we saw earlier, ExecuteBufferedAsync(), ListenAsync() and Observe() are all based on the concept of piping, but these approaches are not exclusive. For example, you can create a piped command and start it as an event stream:

await using var input = File.OpenRead("input.txt");
await using var output = File.Create("output.txt");

var cmd = input | Cli.Wrap("foo") | output;

await foreach (var cmdEvent in cmd.ListenAsync())
{
    // ...
}

This works because internally these methods call PipeTarget.Merge to add additional pipe targets while preserving those configured earlier.

Working with Expression Trees in C#

Oleksii Holub — Mon, 17 Feb 2020 08:42:03 +0000

Expression trees is an obscure, although very interesting feature in .NET. Most people probably think of it as something synonymous with object-relational mapping frameworks, but despite being its most common use case, it's not the only one. There are a lot of creative things you can do with expression trees, including code generation, transpilation, metaprogramming, and more.

In this article I will give an overview of what expression trees are and how to work with them, as well as show some interesting scenarios where I've seen them used to great effect.

What is an expression tree?

When it comes to programming languages, an expression describes some operation on data that produces a certain result. It's one of the foundational constructs of any language.

As an example of a very simple expression, consider 2 + 3. It consists of a constant, a plus operator, and another constant. We can evaluate this expression and get the result, which is 5.

Of course, expressions vary in complexity and can contain different combinations of constants, variables, operators and function calls. For example, the following piece of code is also an expression:

!string.IsNullOrWhiteSpace(personName)
    ? "Greetings, " + personName
    : null;

Looking at the above expression, we can also consider two of its aspects: what it does and how it does it.

When it comes to the former, the answer is pretty simple -- it generates a greeting based on the person's name, or produces a null. If this expression was returned by a function, that would be the extent of information we could derive from its signature:

string? GetGreeting(string personName) { /* ... */ }

As for how it does it, however, the answer is a bit more detailed. This expression consists of a ternary conditional operator, whose condition is evaluated by negating the result of a call to method string.IsNullOrWhiteSpace with parameter personName, whose positive clause is made up of a "plus" binary operator that works with a constant string expression "Greetings, " and the parameter expression, and whose negative clause consist of a sole null expression.

The description above may seem like a mouthful, but it outlines the exact syntactic structure of the expression. It is by this higher-order representation that we're able to tell how exactly it's evaluated.

To make things more clear, we can also illustrate this representation with the following diagram:

 { Ternary conditional }
      |      |     |
    +-+      |     +-----+
    |        |           |
 (true)   (false)   (condition)
    |        |           |
    |        |           +---- { (!) }
    |        +------+             |
 { (+) }            |             |
   | |           { null }   { Method call }
   | |                         |      |
   | +---------+               |      |
   |           |               |      +------- { string.IsNullOrWhiteSpace }
   |           |               |
   |     { personName }        +-------- { personName }
   |
   +---- { "Greetings, " }

As you can see, at the highest level we have the ternary conditional operator which is itself made up of other expressions, which are made up of other expressions, and so on. The relationship between individual components is hierarchical, resembling an upside-down tree.

Although it's inherently obvious to us as humans, in order to interpret this representation programmatically, we need a special data structure. This data structure is what we call an expression tree.

Constructing expression trees manually

In C#, expression trees can be used in either of two directions: we can create them directly via an API and then compile them into runtime instructions, or we can disassemble them from supplied lambda expressions. In this part of the article we will focus on the first one.

The framework offers us with an API to construct expression trees through the Expression class located in the System.Linq.Expressions namespace. It exposes various factory methods that can be used to produce expressions of different types.

Some of these methods are:

Expression.Constant(...) -- creates an expression that represents a value.
Expression.Variable(...) -- creates an expression that represents a variable.
Expression.New(...) -- creates an expression that represents an initialization of a new instance.
Expression.Assign(...) -- creates an expression that represents an assignment operation.
Expression.Equal(...) -- creates an expression that represents an equality comparison.
Expression.Call(...) -- creates an expression that represents a specific method call.
Expression.Condition(...) -- creates an expression that represents branching logic.
Expression.Loop(...) -- creates an expression that represents repeating logic.

As a simple exercise, let's recreate the expression we've looked into in the previous part of the article:

public Expression ConstructGreetingExpression()
{
    var personNameParameter = Expression.Parameter(typeof(string), "personName");

    // Condition
    var isNullOrWhiteSpaceMethod = typeof(string)
        .GetMethod(nameof(string.IsNullOrWhiteSpace));

    var condition = Expression.Not(
        Expression.Call(isNullOrWhiteSpaceMethod, personNameParameter));

    // True clause
    var trueClause = Expression.Add(
        Expression.Constant("Greetings, "),
        personNameParameter);

    // False clause
    var falseClause = Expression.Constant(null, typeof(string));

    // Ternary conditional
    return Expression.Condition(condition, trueClause, falseClause);
}

Let's digest what just happened here.

First of all, we're calling Expression.Parameter in order to construct a parameter expression. We will be able to use it to resolve the value of a particular parameter.

Following that, we are relying on reflection to resolve a reference to the string.IsNullOrWhiteSpace method. We use Expression.Call to create a method invocation expression that represents a call to string.IsNullOrWhiteSpace with the parameter resolved by the expression we created earlier. To perform a logical "not" operation on the result, we're calling Expression.Not to wrap the method call. Incidentally, this expression constitutes the condition part of the ternary expression we're building.

To compose the positive clause, we're constructing an "add" operation with the help of Expression.Add. As the operands, we're providing a constant expression for string "Greetings, " and the parameter expression from earlier.

Then, for the negative clause, we're using Expression.Constant to create a null constant expression. To ensure that the null value is typed correctly, we explicitly specify the type as the second parameter.

Finally, we're combining all of the above parts together to create our ternary conditional operator. If you take a moment to trace what goes into Expression.Condition, you will realize that we have essentially replicated the tree diagram we've seen earlier.

However, this expression isn't particularly useful on its own. Since we've created it ourselves, we're not really interested in its structure -- we want to be able to evaluate it instead.

In order to do that, we have to create an entry point by wrapping everything in a lambda expression. To turn it into an actual lambda, we can call Compile which will produce a delegate that we can invoke.

Let's update the method accordingly:

public Func<string, string?> ConstructGreetingFunction()
{
    var personNameParameter = Expression.Parameter(typeof(string), "personName");

    // Condition
    var isNullOrWhiteSpaceMethod = typeof(string)
        .GetMethod(nameof(string.IsNullOrWhiteSpace));

    var condition = Expression.Not(
        Expression.Call(isNullOrWhiteSpaceMethod, personNameParameter));

    // True clause
    var trueClause = Expression.Add(
        Expression.Constant("Greetings, "),
        personNameParameter);

    // False clause
    var falseClause = Expression.Constant(null, typeof(string));

    var conditional = Expression.Condition(condition, trueClause, falseClause);

    var lambda = Expression.Lambda<Func<string, string?>>(conditional, personNameParameter);

    return lambda.Compile();
}

As you can see, we were able to construct a lambda expression by specifying its body (which is our conditional expression) and the parameter that we defined earlier. We also indicated the exact type of the function this expression represents by supplying a generic argument.

By compiling the expression tree, we can convert the code it represents into runtime instructions. The delegate returned by this method can be used to evaluate the expression:

var getGreeting = ConstructGreetingFunction();

var greetingForJohn = getGreeting("John");

However, if we try to run this, we will get an error:

The binary operator Add is not defined for the types 'System.String' and 'System.String'.

Hmm, that's weird. I'm pretty sure the + operator is defined for strings, otherwise how else would I be able to write "foo" + "bar"?

Well, actually the error message is correct, this operator is indeed not defined for System.String. Instead what happens is that the C# compiler automatically converts expressions like "foo" + "bar" into string.Concat("foo", "bar"). In cases with more than two strings this provides better performance because it avoids unnecessary allocations.

When dealing with expression trees, we're essentially writing the "final" version of the code. So instead of Expression.Add we need to call string.Concat directly.

Let's change our code to accommodate for that:

public Func<string, string?> ConstructGreetingFunction()
{
    var personNameParameter = Expression.Parameter(typeof(string), "personName");

    // Condition
    var isNullOrWhiteSpaceMethod = typeof(string)
        .GetMethod(nameof(string.IsNullOrWhiteSpace));

    var condition = Expression.Not(
        Expression.Call(isNullOrWhiteSpaceMethod, personNameParameter));

    // True clause
    var concatMethod = typeof(string)
        .GetMethod(nameof(string.Concat), new[] {typeof(string), typeof(string)});

    var trueClause = Expression.Call(
        concatMethod,
        Expression.Constant("Greetings, "),
        personNameParameter);

    // False clause
    var falseClause = Expression.Constant(null, typeof(string));

    var conditional = Expression.Condition(condition, trueClause, falseClause);

    var lambda = Expression.Lambda<Func<string, string?>>(conditional, personNameParameter);

    return lambda.Compile();
}

Now, if we try to compile and run our function, it behaves as expected:

var getGreetings = ConstructGreetingFunction();

var greetingsForJohn = getGreetings("John"); // "Greetings, John"
var greetingsForNobody = getGreetings(" ");  // <null>

I think this is pretty awesome. We built an expression tree, compiled it in-memory, and now we can evaluate it using a delegate.

Constructing statements

So far we've only talked about expressions, but what about statements? Can we dynamically compile code that contains multiple statements or are we limited to expressions?

The main difference between expressions and statements is that statements don't produce results. That means we can't really string them into a single expression.

For example, consider the following two statements:

// Two statements:
Console.Write("Hello ");
Console.WriteLine("world!");

There's no way for us to compose these into one expression, like we could have with StringBuilder, for instance:

// Single expression:
new StringBuilder().Append("Hello ").AppendLine("world!");

Fortunately, the expression tree model allows us to represent statements as well. To do that, we need to put them inside a Block expression.

Here is how it works:

public Expression CreateStatementBlock()
{
    var consoleWriteMethod = typeof(Console)
        .GetMethod(nameof(Console.Write), new[] {typeof(string)});

    var consoleWriteLineMethod = typeof(Console)
        .GetMethod(nameof(Console.WriteLine), new[] {typeof(string)});

    return Expression.Block(
        Expression.Call(consoleWriteMethod, Expression.Constant("Hello ")),
        Expression.Call(consoleWriteLineMethod, Expression.Constant("world!")));
}

We can then similarly compile a delegate and invoke it:

var block = CreateStatementBlock();
var lambda = Expression.Lambda<Action>(block).Compile();

lambda();

// Hello world!

Now, if we inspect the block.Type property, which denotes the result type of the expression, we will see that it's System.Void. Essentially the lambda expression we've built is just this:

var lambda = () =>
{
    Console.Write("Hello ");
    Console.WriteLine("world!");
};

For more complex use cases, we may declare and reference variables from inside the block expression:

public Expression CreateStatementBlock()
{
    var consoleWriteMethod = typeof(Console)
        .GetMethod(nameof(Console.Write), new[] {typeof(string)});

    var consoleWriteLineMethod = typeof(Console)
        .GetMethod(nameof(Console.WriteLine), new[] {typeof(string)});

    var variableA = Expression.Variable(typeof(string), "a");
    var variableB = Expression.Variable(typeof(string), "b");

    return Expression.Block(
        // Declare variables in scope
        new[] {variableA, variableB},

        // Assign values to variables
        Expression.Assign(variableA, Expression.Constant("Foo ")),
        Expression.Assign(variableB, Expression.Constant("bar")),

        // Call methods
        Expression.Call(consoleWriteMethod, variableA),
        Expression.Call(consoleWriteLineMethod, variableB));
}

If we compile and evaluate this expression, we will see the following output in the console:

var block = CreateStatementBlock();
var lambda = Expression.Lambda<Action>(block).Compile();

lambda();

// Foo bar

So despite the fact that we are building expression trees, we are not actually limited only to expressions. We can just as easily model blocks of statements too.

Converting expressions to readable code

We know how to compile our expressions into runtime instructions, but what about readable C# code? It could be useful if we wanted to display it or just to have some visual aid while testing.

The good news is that all types that derive from Expression override the ToString method with a more specific implementation. That means we can do the following:

var s1 = Expression.Constant(42).ToString(); // 42

var s2 = Expression.Multiply(
    Expression.Constant(5),
    Expression.Constant(11)).ToString();     // (5 * 11)

The bad news, however, is that it only works nicely with simple expressions like the ones above. For example, if we try to call ToString on the ternary expression we compiled earlier, we will get:

var s = lambda.ToString();

// personName => IIF(Not(IsNullOrWhiteSpace(personName)), Concat("Greetings, ", personName), null)

While fairly descriptive, this is probably not the text representation one would hope to see.

Luckily, we can use the ReadableExpressions NuGet package to get us what we want. By installing it, we should be able to call ToReadableString to get the actual C# code that represents the expression:

var code = lambda.ToReadableString();

// personName => !string.IsNullOrWhiteSpace(personName) ? "Greetings, " + personName : null

As you can see, it even replaced the string.Concat call with the plus operator to make it closer to code that a developer would typically write.

Additionally, if you are using Visual Studio and want to inspect expressions by visualizing them as code, you can install this extension. It's very helpful when debugging large or really complex expressions.

Optimizing reflection calls

When it comes to compiled expressions, one of the most common usage scenarios is reflection-heavy code. As we all know, reflection can be quite slow because of late binding, however by compiling the code at runtime we can achieve better performance.

Let's imagine we have a class which has a private method that we want to invoke from the outside:

public class Command
{
    private int Execute() => 42;
}

With the help of reflection, this is quite simple:

public static int CallExecute(Command command) =>
    (int) typeof(Command)
        .GetMethod("Execute", BindingFlags.NonPublic | BindingFlags.Instance)
        .Invoke(command, null);

Of course, invoking the method like that can cause significant performance issues if we put it in a tight loop. Let's see if we can optimize it a bit.

Before we jump into expressions, we can first optimize the above code by separating the part that resolves MethodInfo from the part that invokes it. If we're going to call this method more than once, we don't have to use GetMethod every time:

public static class ReflectionCached
{
    private static MethodInfo ExecuteMethod { get; } = typeof(Command)
        .GetMethod("Execute", BindingFlags.NonPublic | BindingFlags.Instance);

    public static int CallExecute(Command command) => (int) ExecuteMethod.Invoke(command, null);
}

That should make things better, but we can push it even further by using Delegate.CreateDelegate. This way we can create a re-usable delegate and avoid the overhead that comes with MethodInfo.Invoke. Let's do that as well:

public static class ReflectionDelegate
{
    private static MethodInfo ExecuteMethod { get; } = typeof(Command)
        .GetMethod("Execute", BindingFlags.NonPublic | BindingFlags.Instance);

    private static Func<Command, int> Impl { get; } =
        (Func<Command, int>) Delegate.CreateDelegate(typeof(Func<Command, int>), ExecuteMethod);

    public static int CallExecute(Command command) => Impl(command);
}

Alright, that's probably as good as it can get with reflection. Now let's try to do the same using compiled expressions:

public static class ExpressionTrees
{
    private static MethodInfo ExecuteMethod { get; } = typeof(Command)
        .GetMethod("Execute", BindingFlags.NonPublic | BindingFlags.Instance);

    private static Func<Command, int> Impl { get; }

    static ExpressionTrees()
    {
        var instance = Expression.Parameter(typeof(Command));
        var call = Expression.Call(instance, ExecuteMethod);
        Impl = Expression.Lambda<Func<Command, int>>(call, instance).Compile();
    }

    public static int CallExecute(Command command) => Impl(command);
}

In all of these approaches we're relying on static constructors to initialize the properties in a lazy and thread-safe manner. This ensures that all of the heavy-lifting happens only once, the first time the members of these classes are accessed.

Now let's pit all of these techniques against each other and compare their performance using Benchmark.NET:

public class Benchmarks
{
    [Benchmark(Description = "Reflection", Baseline = true)]
    public int Reflection() => (int) typeof(Command)
        .GetMethod("Execute", BindingFlags.NonPublic | BindingFlags.Instance)
        .Invoke(new Command(), null);

    [Benchmark(Description = "Reflection (cached)")]
    public int Cached() => ReflectionCached.CallExecute(new Command());

    [Benchmark(Description = "Reflection (delegate)")]
    public int Delegate() => ReflectionDelegate.CallExecute(new Command());

    [Benchmark(Description = "Expressions")]
    public int Expressions() => ExpressionTrees.CallExecute(new Command());

    public static void Main() => BenchmarkRunner.Run<Benchmarks>();
}

|                Method |       Mean |     Error |    StdDev | Ratio |
|---------------------- |-----------:|----------:|----------:|------:|
|            Reflection | 192.975 ns | 1.6802 ns | 1.4895 ns |  1.00 |
|   Reflection (cached) | 123.762 ns | 1.1063 ns | 1.0349 ns |  0.64 |
| Reflection (delegate) |   6.419 ns | 0.0646 ns | 0.0605 ns |  0.03 |
|           Expressions |   5.383 ns | 0.0433 ns | 0.0383 ns |  0.03 |

As you can see, compiled expressions outperform reflection across the board, even though the approach with CreateDelegate comes really close. Note however that while the execution times are similar, CreateDelegate is more limited than compiled expressions -- for example, it cannot be used to call constructor methods.

This approach of using expression trees for dynamic method invocation is commonplace in various frameworks and libraries. For example:

AutoMapper uses them to speed up object conversion
NServiceBus uses them to speed up its behavior pipeline
Marten uses them to speed up entity mapping

Implementing generic operators

Something else we can do with compiled expressions is implement generic operators. These can be pretty useful if you're writing a lot of mathematical code and want to avoid duplication.

As you know, operators in C# are not generic. This means that every numeric type defines its own version of the multiply and divide operators, among other things. As a result, code that uses these operators also can't be generic either.

Imagine that you had a function that calculates three-fourths of a number:

public int ThreeFourths(int x) => 3 * x / 4;

// ThreeFourths(18) -> 13

Defined as it is, it only works when used with numbers of type int. If we wanted to extend it to support other types, we'd have to add some overloads:

public int ThreeFourths(int x) => 3 * x / 4;

public long ThreeFourths(long x) => 3 * x / 4;

public float ThreeFourths(float x) => 3 * x / 4;

public double ThreeFourths(double x) => 3 * x / 4;

public decimal ThreeFourths(decimal x) => 3 * x / 4;

This is suboptimal. We are introducing a lot of code duplication which only gets worse as this method is referenced from other places.

It would've been better if we could just do something like this instead:

public T ThreeFourths<T>(T x) => 3 * x / 4;

But unfortunately that doesn't compile, seeing as the * and / operators are not available on every type that can be specified in place of T. Sadly, there's also no constraint we could use to limit the generic argument to numeric types.

However, by generating code dynamically with expression trees we can work around this problem:

public T ThreeFourths<T>(T x)
{
    var param = Expression.Parameter(typeof(T));

    // Cast the numbers '3' and '4' to our type
    var three = Expression.Convert(Expression.Constant(3), typeof(T));
    var four = Expression.Convert(Expression.Constant(4), typeof(T));

    // Perform the calculation
    var operation = Expression.Divide(Expression.Multiply(param, three), four);

    var lambda = Expression.Lambda<Func<T, T>>(operation, param);

    var func = lambda.Compile();

    return func(x);
}

// ThreeFourths(18) -> 13
// ThreeFourths(6.66) -> 4.995
// ThreeFourths(100M) -> 75

That works well and we can reuse this method for numbers of any type. Although, seeing as our generic operation doesn't have type safety, you may be wondering how is this approach any different from just using dynamic?

Surely, we could just write our code like this and avoid all the trouble:

public dynamic ThreeFourths(dynamic x) => 3 * x / 4;

Indeed, functionally these two approaches are the same. However, the main difference and the advantage of expression trees is the fact they are compiled, while dynamic isn't. Compiled code has the potential to perform much faster.

That said, in the example above we're not benefitting from this advantage at all because we're recompiling our function every time anyway. Let's try to change our code so that it happens only once.

In order to achieve that, we can apply the same pattern as the last time. Let's put the delegate inside a generic static class and have it initialized from the static constructor. Here's how that would look:

public static class ThreeFourths
{
    private static class Impl<T>
    {
        public static Func<T, T> Of { get; }

        static Impl()
        {
            var param = Expression.Parameter(typeof(T));

            var three = Expression.Convert(Expression.Constant(3), typeof(T));
            var four = Expression.Convert(Expression.Constant(4), typeof(T));

            var operation = Expression.Divide(Expression.Multiply(param, three), four);

            var lambda = Expression.Lambda<Func<T, T>>(operation, param);

            Of = lambda.Compile();
        }
    }

    public static T Of<T>(T x) => Impl<T>.Of(x);
}

// ThreeFourths.Of(18) -> 13

Due to the fact that the compiler generates a version of the Impl class for each argument of T, we end up with an implementation of three-fourths for each type encapsulated in a separate class. This approach gives us a thread-safe lazy-evaluated generic dynamic function.

Now, with the optimizations out of the way, let's again use Benchmark.NET to compare the different ways we can calculate three-fourths of a value:

public class Benchmarks
{
    [Benchmark(Description = "Static", Baseline = true)]
    [Arguments(13.37)]
    public double Static(double x) => 3 * x / 4;

    [Benchmark(Description = "Expressions")]
    [Arguments(13.37)]
    public double Expressions(double x) => ThreeFourths.Of(x);

    [Benchmark(Description = "Dynamic")]
    [Arguments(13.37)]
    public dynamic Dynamic(dynamic x) => 3 * x / 4;

    public static void Main() => BenchmarkRunner.Run<Benchmarks>();
}

|      Method |     x |       Mean |     Error |    StdDev | Ratio | RatioSD |
|------------ |------ |-----------:|----------:|----------:|------:|--------:|
|      Static | 13.37 |  0.6077 ns | 0.0176 ns | 0.0147 ns |  1.00 |    0.00 |
|     Dynamic | 13.37 | 19.3267 ns | 0.1512 ns | 0.1340 ns | 31.82 |    0.78 |
| Expressions | 13.37 |  1.9510 ns | 0.0163 ns | 0.0145 ns |  3.21 |    0.08 |

As you can see, the expression-based approach performs about nine times faster than when using dynamic. Considering that these are the only two options we can use to implement generic operators, this is a pretty good case for compiled expression trees.

Compiling dictionary into a switch expression

Another fun way we can use expression trees is to create a dictionary with a compiled lookup. Even though the standard .NET System.Collections.Generic.Dictionary is insanely fast on its own, it's possible to make its read operations even faster.

While a typical dictionary implementation may be pretty complicated, a lookup can be represented in a form of a simple switch expression:

// Pseudo-code
public TValue Lookup(TKey key) => key.GetHashCode() switch
{
    // No collisions
    9254 => value1,
    -101 => value2,

    // Collision
    777 => key switch
    {
        key3 => value3,
        key4 => value4
    },

    // ...

    // Not found
    _ => throw new KeyNotFoundException(key.ToString())
};

The function above attempts to match the hash code of the specified key with the hash code of one of the keys contained within the dictionary. If it's successful, then the corresponding value is returned.

Even though hash codes are designed to be unique, inevitably there will be collisions. In such cases, when the same hash code matches with multiple different values, there is an inner switch expression that compares the actual key and determines which value to return.

Finally, if none of the cases matched, it throws an exception signifying that the dictionary doesn't contain the specified key.

The idea is that, since a switch is faster than a hash table, dynamically compiling all key-value pairs into a switch expression like the one above should result in a faster dictionary lookup.

Let's try it out. Here's how the code for that would look:

public class CompiledDictionary<TKey, TValue> : IDictionary<TKey, TValue>
{
    private readonly IDictionary<TKey, TValue> _inner = new Dictionary<TKey, TValue>();

    private Func<TKey, TValue> _lookup;

    public CompiledDictionary() => UpdateLookup();

    public void UpdateLookup()
    {
        // Parameter for lookup key
        var keyParameter = Expression.Parameter(typeof(TKey));

        // Expression that gets the key's hash code
        var keyGetHashCodeCall = Expression.Call(
            keyParameter,
            typeof(object).GetMethod(nameof(GetHashCode)));

        // Expression that converts the key to string
        var keyToStringCall = Expression.Call(
            keyParameter,
            typeof(object).GetMethod(nameof(ToString)));

        // Expression that throws 'not found' exception in case of failure
        var exceptionCtor = typeof(KeyNotFoundException)
            .GetConstructor(new[] {typeof(string)});

        var throwException = Expression.Throw(
            Expression.New(exceptionCtor, keyToStringCall),
            typeof(TValue));

        // Switch expression with cases for every hash code
        var body = Expression.Switch(
            typeof(TValue), // expression type
            keyGetHashCodeCall, // switch condition
            throwException, // default case
            null, // use default comparer
            _inner // switch cases
                .GroupBy(p => p.Key.GetHashCode())
                .Select(g =>
                {
                    // No collision, construct constant expression
                    if (g.Count() == 1)
                        return Expression.SwitchCase(
                            Expression.Constant(g.Single().Value), // body
                            Expression.Constant(g.Key)); // test values

                    // Collision, construct inner switch for the key's value
                    return Expression.SwitchCase(
                        Expression.Switch(
                            typeof(TValue),
                            keyParameter, // switch on actual key
                            throwException,
                            null,
                            g.Select(p => Expression.SwitchCase(
                                Expression.Constant(p.Value),
                                Expression.Constant(p.Key)
                            ))),
                        Expression.Constant(g.Key));
                }));

        var lambda = Expression.Lambda<Func<TKey, TValue>>(body, keyParameter);

        _lookup = lambda.Compile();
    }

    public TValue this[TKey key]
    {
        get => _lookup(key);
        set => _inner[key] = value;
    }

    // The rest of the interface implementation is omitted for brevity
}

The method UpdateLookup takes all of the key-value pairs contained within the inner dictionary and groups them by the hash codes of their keys, which are then transformed into switch cases. If there is no collision for a particular hash code, then the switch case is made up of a single constant expression that produces the corresponding value. Otherwise, it contains an inner switch expression that further evaluates the key to determine which value to return.

Let's see how well this dictionary performs when benchmarked against the standard implementation:

public class Benchmarks
{
    private readonly Dictionary<string, int> _normalDictionary =
        new Dictionary<string, int>();

    private readonly CompiledDictionary<string, int> _compiledDictionary =
        new CompiledDictionary<string, int>();

    [Params(10, 1000, 10000)]
    public int Count { get; set; }

    public string TargetKey { get; set; }

    [GlobalSetup]
    public void Setup()
    {
        // Seed the dictionaries with values
        foreach (var i in Enumerable.Range(0, Count))
        {
            var key = $"key_{i}";

            _normalDictionary[key] = i;
            _compiledDictionary[key] = i;
        }

        // Recompile lookup
        _compiledDictionary.UpdateLookup();

        // Try to get the middle element
        TargetKey = $"key_{Count / 2}";
    }

    [Benchmark(Description = "Standard dictionary", Baseline = true)]
    public int Normal() => _normalDictionary[TargetKey];

    [Benchmark(Description = "Compiled dictionary")]
    public int Compiled() => _compiledDictionary[TargetKey];

    public static void Main() => BenchmarkRunner.Run<Benchmarks>();
}

|              Method | Count |      Mean |     Error |    StdDev | Ratio |
|-------------------- |------ |----------:|----------:|----------:|------:|
| Standard dictionary |    10 | 24.995 ns | 0.1821 ns | 0.1704 ns |  1.00 |
| Compiled dictionary |    10 |  9.366 ns | 0.0511 ns | 0.0478 ns |  0.37 |
|                     |       |           |           |           |       |
| Standard dictionary |  1000 | 25.105 ns | 0.0665 ns | 0.0622 ns |  1.00 |
| Compiled dictionary |  1000 | 14.819 ns | 0.1138 ns | 0.1065 ns |  0.59 |
|                     |       |           |           |           |       |
| Standard dictionary | 10000 | 29.047 ns | 0.1201 ns | 0.1123 ns |  1.00 |
| Compiled dictionary | 10000 | 17.903 ns | 0.0635 ns | 0.0530 ns |  0.62 |

We can see that the compiled dictionary performs lookups about 1.6-2.8 times faster. While the performance of the hash table is consistent regardless of how many elements are in the dictionary, the expression tree implementation becomes slower as the dictionary gets bigger. This can potentially be remedied by adding another switch layer for indexing.

Parsing DSLs into expressions

One other interesting usage scenario, that I'm personally really fond of, is parsing. The main challenge of writing an interpreter for a custom domain-specific language is turning the syntax tree into runtime instructions. By parsing the grammar constructs directly into expression trees, this becomes a solved problem.

As an example, let's write a simple program that takes a string representation of a mathematical expression and evaluates its result. To implement the parser, we will use the Sprache library.

public static class SimpleCalculator
{
    private static readonly Parser<Expression> Constant =
        Parse.DecimalInvariant
            .Select(n => double.Parse(n, CultureInfo.InvariantCulture))
            .Select(n => Expression.Constant(n, typeof(double)))
            .Token();

    private static readonly Parser<ExpressionType> Operator =
        Parse.Char('+').Return(ExpressionType.Add)
            .Or(Parse.Char('-').Return(ExpressionType.Subtract))
            .Or(Parse.Char('*').Return(ExpressionType.Multiply))
            .Or(Parse.Char('/').Return(ExpressionType.Divide));

    private static readonly Parser<Expression> Operation =
        Parse.ChainOperator(Operator, Constant, Expression.MakeBinary);

    private static readonly Parser<Expression> FullExpression =
        Operation.Or(Constant).End();

    public static double Run(string expression)
    {
        var operation = FullExpression.Parse(expression);
        var func = Expression.Lambda<Func<double>>(operation).Compile();

        return func();
    }
}

As you can see, the parsers defined above (Constant, Operator, Operation, FullExpression) all yield objects of type Expression and ExpressionType, which are both defined in System.Linq.Expressions. The expression tree is essentially our syntax tree, so once we parse the input we have all the required information to compile the runtime instructions represented by it.

You can try it out by calling Run:

var a = SimpleCalculator.Run("2 + 2");        // 4
var b = SimpleCalculator.Run("3.15 * 5 + 2"); // 17.75
var c = SimpleCalculator.Run("1 / 2 * 3");    // 1.5

Note that this simple calculator is just an example of what you can do, it doesn't respect operator precedence and doesn't understand nested expressions. Implementing a parser for that would be out of scope of covering expression trees, but if you want to see how a proper calculator like that would look, check out Sprache.Calc. Also, if you want to learn more about parsing, check out my blog posts about parsing in C# and parsing in F#.

Making things even faster

While compiled expressions execute really fast, compiling them can be relatively expensive.

In most cases that's completely fine, but you may want to take the performance even further by using FastExpressionCompiler. This library provides a drop-in replacement for the Compile method called CompileFast, which executes much faster.

For example, here's a simple benchmark that shows the difference:

public class Benchmarks
{
    private static Expression Body { get; } =
        Expression.Add(Expression.Constant(3), Expression.Constant(5));

    [Benchmark(Description = "Compile", Baseline = true)]
    public Func<int> Normal() => Expression.Lambda<Func<int>>(Body).Compile();

    [Benchmark(Description = "Compile (fast)")]
    public Func<int> Fast() => Expression.Lambda<Func<int>>(Body).CompileFast();

    public static void Main() => BenchmarkRunner.Run<Benchmarks>();
}

|         Method |      Mean |     Error |    StdDev | Ratio | Allocated |
|--------------- |----------:|----------:|----------:|------:|----------:|
|        Compile | 38.435 us | 0.2131 us | 0.1889 us |  1.00 |   3.53 KB |
| Compile (fast) |  4.497 us | 0.0662 us | 0.0619 us |  0.12 |   1.21 KB |

As you can see, the performance improvement is pretty noticeable. The reason it runs so fast is because the CompileFast version skips all the verifications that normal Compile does to ensure that the expression tree is valid.

This library (as part of FastExpressionCompiler.LightExpression) also offers a drop-in replacement for Expression and all of its static factory methods. These alternative implementations construct expressions which may in some cases perform much faster than their default counterparts. However, I still recommend to benchmark it on your particular use cases to ensure that it actually provides an improvement.

Inferring expression trees from code

So far we've explored how to construct expression trees manually. The cool thing about expression trees in .NET though is that they can also be created automatically as well.

The way this works is that you can infer an expression tree by simply specifying a lambda expression like you would if you were to define a delegate. C# compiler will take care of the rest.

Consider this snippet of code:

Func<int, int, int> div =
    (a, b) => a / b;

Expression<Func<int, int, int>> divExpr =
    (a, b) => a / b;

Both of these assignments look exactly the same, but the actual assigned value is different. While in the first case we will get a delegate which can be executed directly, the second will provide us with an expression tree that represents the structure of the supplied lambda expression. This is essentially the same LambdaExpression that we were creating when compiling code ourselves, only now it represents code written statically as opposed to dynamically.

For example, we can inspect the expression tree produced by the compiler:

Expression<Func<int, int, int>> divExpr =
    (a, b) => a / b;

foreach (var param in divExpr.Parameters)
    Console.WriteLine($"Param: {param.Name} ({param.Type.Name})");

// Param: a (Int32)
// Param: b (Int32)

And, just like with expression trees created manually, we can compile it into a delegate:

Expression<Func<int, int, int>> divExpr =
    (a, b) => a / b;

var div = divExpr.Compile();

var c = div(10, 2); // 5

Essentially, in this context, you can think of divExpr as a recipe that contains the ingredients needed to create div, the final product.

Note, however, that while direct assignment shown previously works, you can't do something like this:

Func<int, int, int> div = (a, b) => a / b;

// Compilation error
Expression<Func<int, int, int>> divExpr = div;

The expression must be defined in-place in order to work. Because the disassembly happens during compile time, not runtime, the compiler needs to know exactly what it's dealing with.

Although this approach is incredibly useful, it has certain limitations. Specifically, the supplied lambda expression must not contain any of the following:

Null-coalescing operator (obj?.Prop)
Dynamic variables (dynamic)
Asynchronous code (async/await)
Default or named parameters (func(a, b: 5), func(a))
Parameters passed by reference (int.TryParse("123", out var i))
Multi-dimensional array initializers (new int[2, 2] { { 1, 2 }, { 3, 4 } })
Assignment operations (a = 5)
Increment and decrement (a++, a--, --a, ++a)
Base type access (base.Prop)
Dictionary initialization (new Dictionary<string, int> { ["foo"] = 100 })
Unsafe code (via unsafe)
Throw expressions (throw new Exception())
Tuple literals ((5, x))

On top of all that, you cannot use this method to construct expression trees from multi-line lambdas. That means this won't compile:

// Compilation error
Expression<Func<int, int, int>> divExpr = (a, b) =>
{
    var result = a / b;
    return result;
};

And, more importantly, this won't work either:

// Compilation error
Expression<Action> writeToConsole = () =>
{
    Console.Write("Hello ");
    Console.WriteLine("world!");
};

Most of these limitations come from the fact that this feature was designed with IQueryable in mind and many of the language constructs listed above don't really make sense when it comes to querying data. That said, there are a lot of other scenarios where they can be useful.

There is a suggestion to extend compile-time expression trees and it's tracked by this issue on GitHub. We'll see where it goes.

For now, let's move these limitations aside and explore some of the ways we can use expression trees constructed with this approach.

Identifying type members

The most common use case for expression trees obtained in such manner is to identify type members. This approach allows us to extract information on fields, properties, or methods from a supplied lambda expression.

For example, assume we have the following class:

public class Dto
{
    public Guid Id { get; set; }

    public string Name { get; set; }
}

If we wanted to get the PropertyInfo that represents its Id property, we could use reflection to do it like this:

var idProperty = typeof(Dto).GetProperty(nameof(Dto.Id));

Console.WriteLine($"Type: {idProperty.DeclaringType.Name}");
Console.WriteLine($"Property: {idProperty.Name} ({idProperty.PropertyType.Name})");

// Type: Dto
// Property: Id (Guid)

That works completely fine. For example, if we were designing an API for a validation library, it could look like this:

public class Validator<T>
{
    // Add validation predicate to the list
    public void AddValidation<TProp>(string propertyName, Func<TProp, bool> predicate)
    {
        var propertyInfo = typeof(T).GetProperty(propertyName);

        if (propertyInfo is null)
            throw new InvalidOperationException("Please provide a valid property name.");

        // ...
    }

    // Evalute all predicates
    public bool Validate(T obj) { /* ... */ }

    /* ... */
}

Which we would be able to use like this:

var validator = new Validator<Dto>();
validator.AddValidation<Guid>(nameof(Dto.Id), id => id != Guid.Empty);
validator.AddValidation<string>(nameof(Dto.Name), name => !string.IsNullOrWhiteSpace(name));

var isValid = validator.Validate(new Dto { Id = Guid.NewGuid() }); // false

However, the problem here is that all of our validators are effectively untyped. We have to specify the generic argument in AddValidation so that our predicates are aware of what they're working with, but this setup is very volatile.

If we were to, for example, change the type of Dto.Id from Guid to int, everything will still compile but the code will no longer work correctly because our predicate expects the type to be Guid. Also, we'd be lucky if our users were to provide the property names using nameof, in reality there will probably be magic strings instead. All in all, this code is not refactor-safe.

With expressions we can completely remedy this:

public class Validator<T>
{
    public void AddValidation<TProp>(
        Expression<Func<T, TProp>> propertyExpression,
        Func<TProp, bool> predicate)
    {
        var propertyInfo = (propertyExpression.Body as MemberExpression)?.Member as PropertyInfo;

        if (propertyInfo is null)
            throw new InvalidOperationException("Please provide a valid property expression.");

        // ...
    }

    public bool Validate(T obj) { /* ... */ }

    /* ... */
}

With the new interface we can write our code like this instead:

var validator = new Validator<Dto>();
validator.AddValidation(dto => dto.Id, id => id != Guid.Empty);
validator.AddValidation(dto => dto.Name, name => !string.IsNullOrWhiteSpace(name));

var isValid = validator.Validate(new Dto { Id = Guid.NewGuid() }); // false

This works exactly the same, except that now we don't need to specify generic arguments manually, there are no magic strings, and the code is completely safe to refactor. If we change the type of Dto.Id from Guid to int, our code will rightfully no longer compile.

Many existing libraries are using expression trees for this purpose, including:

FluentValidation uses it to setup validation rules
EntityFramework uses it for entity configuration
Moq uses it to build mocks

Providing context to assertions

Often when I'm writing test suites for my projects, I find myself spending time decorating assertions with informational error messages. For example:

[Test]
public void IntTryParse_Test()
{
    // Arrange
    const string s = "123";

    // Act
    var result = int.TryParse(s, out var value);

    // Assert
    Assert.That(result, Is.True, "Parsing was unsuccessful");
    Assert.That(value, Is.EqualTo(124), "Parsed value is incorrect");
}

By doing that, the errors produced by failed assertions become more descriptive. This makes it easier to understand what went wrong without having to look inside the test implementation:

X IntTryParse_Test [60ms]
  Error Message:
    Parsed value is incorrect
  Expected: 124
  But was:  123

In a perfect world, however, it would be nice if the error message simply contained the code of the assertion. That way I would know which exact check failed and why.

Luckily, this is something we can do with the help of expressions. To facilitate that, we can create a helper method that will wrap the assertion in an expression:

public static class AssertEx
{
    public static void Express(Expression<Action> expression)
    {
        var act = expression.Compile();

        try
        {
            act();
        }
        catch (AssertionException ex)
        {
            throw new AssertionException(
                expression.Body.ToReadableString() +
                Environment.NewLine +
                ex.Message);
        }
    }
}

This method is really simple. All it does is try to run the delegate represented by the expression and, if the underlying assertion fails, it prints the expression along with the error.

Let's update our test code to make use of this method:

[Test]
public void IntTryParse_Test()
{
    // Arrange
    const string s = "123";

    // Act
    var result = int.TryParse(s, out var value);

    // Assert
    AssertEx.Express(() => Assert.That(result, Is.True));
    AssertEx.Express(() => Assert.That(value, Is.EqualTo(124)));
}

Now, when this test fails we will instead get the following error message:

X IntTryParse_Test [99ms]
  Error Message:
    Assert.That(value, Is.EqualTo(124))
  Expected: 124
  But was:  123

As you can see, the error message now specifies the exact assertion that failed. This gives us more context which helps determine what actually went wrong.

With the advent of .NET Core 3.0, the .NET team has also added a new attribute, CallerArgumentExpression. This attribute was meant to be supported by a language feature that was planned for C# 8 but unfortunately it didn't make it. Currently, the attribute doesn't do anything, but we should see this change in one of the future versions of the language.

The goal of this attribute is to provide the ability to "sniff" the expression passed to the specified parameter. For example, we should be able to define a method like this:

public static void Assert(
    bool condition,
    [CallerArgumentExpression("condition")] string expression = "")
{
    if (!condition)
        throw new AssertionFailedException($"Condition `{expression}` is not true");
}

Which will then produce a detailed exception message if the assertion fails:

Assert(2 + 2 == 5);

// Exception:
// Condition `2 + 2 == 5` is not true

Note that with this approach we will only be able to obtain the expression as a string, which will be the same expression specified in the source code. This can be used to provide a somewhat similar experience as shown with AssertEx.Express above.

Traversing and rewriting expression trees

In order to analyze expression trees, we need to be able to traverse them in a recursive descent manner, starting from the body of the lambda expression and going down to every expression it's made out of. This could be done manually with a large switch expression that calls into itself.

Fortunately, we don't have to reinvent the wheel because the framework already provides a special class for this purpose called ExpressionVisitor. It's an abstract class that has a visitor method for every expression type so you can simply inherit from it and override the methods you're interested in.

For example, we can implement a visitor that prints out all the binary and method call expressions it encounters:

public class Visitor : ExpressionVisitor
{
    protected override Expression VisitMethodCall(MethodCallExpression node)
    {
        Console.WriteLine($"Visited method call: {node}");

        return base.VisitMethodCall(node);
    }

    protected override Expression VisitBinary(BinaryExpression node)
    {
        Console.WriteLine($"Visited binary expression: {node}");

        return base.VisitBinary(node);
    }
}

Expression<Func<double>> expr = () => Math.Sin(Guid.NewGuid().GetHashCode()) / 10;

new Visitor().Visit(expr);

// Visited binary expression: (Sin(Convert(NewGuid().GetHashCode(), Double)) / 10)
// Visited method call: Sin(Convert(NewGuid().GetHashCode(), Double))
// Visited method call: NewGuid().GetHashCode()
// Visited method call: NewGuid()

As you can see by the order of the logs, the visitor first encounters the binary expression that makes up the lambda body, then digs inside, revealing a call to Math.Sin whose parameter is also expressed as a call to GetHashCode on the result of NewGuid.

You may have noticed that the visitor methods on ExpressionVisitor all return Expressions. That means that besides merely inspecting them, the visitor can choose to rewrite or completely replace expressions with different ones.

Let's change our visitor so that it catches all calls to method Math.Sin and rewrites them into Math.Cos:

public class Visitor : ExpressionVisitor
{
    protected override Expression VisitMethodCall(MethodCallExpression node)
    {
        var newMethodCall = node.Method == typeof(Math).GetMethod(nameof(Math.Sin))
            ? typeof(Math).GetMethod(nameof(Math.Cos))
            : node.Method;

        return Expression.Call(newMethodCall, node.Arguments);
    }
}

Expression<Func<double>> expr = () => Math.Sin(Guid.NewGuid().GetHashCode()) / 10;
var result = expr.Compile()();

Console.WriteLine($"Old expression: {expr.ToReadableString()}");
Console.WriteLine($"Old result: {result}");

var newExpr = (Expression<Func<double>>) new Visitor().Visit(expr);
var newResult = newExpr.Compile()();

Console.WriteLine($"New expression: {newExpr.ToReadableString()}");
Console.WriteLine($"New result value: {newResult}");

// Old expression: () => Math.Sin((double)Guid.NewGuid().GetHashCode()) / 10d
// Old result: 0.09489518488876232
// New expression: () => Math.Cos((double)Guid.NewGuid().GetHashCode()) / 10d
// New result value: 0.07306426748550407

As you can see, the new expression is structurally identical but with Math.Sin replaced by Math.Cos. Both expressions are completely independent and can be compiled to produce their respective delegates.

Using this approach we can arbitrarily rewrite supplied expressions, generating derivatives that behave differently. It can be very helpful when creating dynamic proxies. For example, a popular mocking library Moq uses this technique to build stubs at runtime.

Transpiling code into a different language

Now that we know that we can use ExpressionVisitor to analyze and rewrite expression trees, it's not too hard to guess that we can also use it to transpile expressions into another language. The goal of such a tool would be to convert code from one language to another, while retaining its functional behavior.

Let's imagine we're building a library that allows users to convert C# expressions to their equivalent F# representations. For example, we want to be able to do this:

Expression<Action<int, int>> expr = (a, b) => Console.WriteLine("a + b = {0}", a + b));

var fsharpCode = FSharpTranspiler.Convert(expr);

To facilitate that, we can create a class called FSharpTranspiler which will internally use a special ExpressionVisitor to traverse the expression tree and write valid F# code. It could look something like this:

public static class FSharpTranspiler
{
    private class Visitor : ExpressionVisitor
    {
        private readonly StringBuilder _buffer;

        public Visitor(StringBuilder buffer)
        {
            _buffer = buffer;
        }

        // ...
    }

    public static string Convert<T>(Expression<T> expression)
    {
        var buffer = new StringBuilder();
        new Visitor(buffer).Visit(expression);

        return buffer.ToString();
    }
}

With this setup, we can inject a StringBuilder into our visitor and use that as the output buffer. While the visitor takes care of navigating the tree, we need to make sure we're emitting valid code on each expression type.

Writing a full C# to F# transpiler would be too complicated and way outside of the scope of this article. For the sake of simplicity let's limit our job to support expressions similar to the one we've seen in the initial example. To handle these, we will need to translate Console.WriteLine into correct usage of printfn.

Here's how we can do it:

public static class FSharpTranspiler
{
    private class Visitor : ExpressionVisitor
    {
        private readonly StringBuilder _buffer;

        public Visitor(StringBuilder buffer)
        {
            _buffer = buffer;
        }

        protected override Expression VisitLambda<T>(Expression<T> node)
        {
            _buffer.Append("fun (");
            _buffer.AppendJoin(", ", node.Parameters.Select(p => p.Name));
            _buffer.Append(") ->");

            return base.VisitLambda(node);
        }

        protected override Expression VisitMethodCall(MethodCallExpression node)
        {
            if (node.Method.DeclaringType == typeof(Console) &&
                node.Method.Name == nameof(Console.WriteLine))
            {
                _buffer.Append("printfn ");

                if (node.Arguments.Count > 1)
                {
                    // For simplicity, assume the first argument is a string (don't do this)
                    var format = (string) ((ConstantExpression) node.Arguments[0]).Value;
                    var formatValues = node.Arguments.Skip(1).ToArray();

                    _buffer.Append("\"");
                    _buffer.Append(Regex.Replace(format, @"\{\d+\}", "%O"));
                    _buffer.Append("\" ");

                    _buffer.AppendJoin(" ", formatValues.Select(v => $"({v.ToReadableString()})"));
                }
            }

            return base.VisitMethodCall(node);
        }
    }

    public static string Convert<T>(Expression<T> expression)
    {
        var buffer = new StringBuilder();
        new Visitor(buffer).Visit(expression);

        return buffer.ToString();
    }
}

So now we can try to convert our expression from earlier and see what it returns:

var fsharpCode = FSharpTranspiler.Convert<Action<int, int>>(
    (a, b) => Console.WriteLine("a + b = {0}", a + b));

// fun (a, b) -> printfn "a + b = %O" (a + b)

This produces a string that contains valid F# code which should compile into an equivalent anonymous function. Let's run it in F# interactive to make sure it works correctly:

> let foo = fun (a, b) -> printfn "a + b = %O" (a + b)
val foo : a:int * b:int -> unit

> foo (3, 5)
a + b = 8
val it : unit = ()

Translating code from one language to another is definitely not a simple task, but it can be incredibly useful in certain scenarios. One example could be sharing validation rules between backend and frontend by converting C# predicate expressions into JavaScript code.

Summary

Expression trees provide us with a formal structure of code that lets us analyze existing expressions or compile entirely new ones directly at runtime. This feature makes it possible to do a bunch of cool things, including writing transpilers, interpreters, code generators, optimize reflection calls, provide contextual assertions, and more. I think it's a really powerful tool that deserves a lot more attention.

Some other interesting articles on the topic:

I also recommend reading about code quotations in F# which is a similar feature to expression trees but with more powerful language support.

Follow me on Twitter to get notified when I post a new article ✨

Refactoring C# code using partial classes

Oleksii Holub — Mon, 03 Feb 2020 09:24:36 +0000

As our code grows, we regularly find ourselves seeking new ways to keep it well structured and organized. Systematic refactoring is a necessity but often doesn't come very easily.

One of the challenges we often face is deciding how to group different parts of a bigger class together. Even with a good degree of separation, sometimes we end up with classes that might be a bit too much to reason about.

From the earliest versions of the language, C# provided a construct called regions. Although it can be helpful when trying to organize code, most seem to agree that using regions is generally an anti-pattern. Even if their usage can be justified, their benefits often come at a rather steep cost in terms of readability.

I do believe that being able to group code to form logical blocks is useful, however I agree that regions cause more problems than they solve. For that reason, I've been actively using partial classes instead, which in many ways can be used for a similar purpose without suffering from the same drawbacks.

Partial classes is a C# feature that lets you split the definition of a type into multiple parts, each potentially in its own file. During the build, compiler collects all of the parts and combines them together to produce a single class, as if it was defined in one place. It's enabled by adding the partial keyword in the definition.

In this article I will show you how I typically utilize partial classes when refactoring my own code. Hopefully, the examples here will entice you to try this approach as well.

Extracting static members

One thing that I like to do nearly all the time is separate static properties and methods from the rest of the class. That might seem like an arbitrary criteria, but I find it makes sense because we do reason about static and non-static members in different ways.

Let's have a look at an example. Imagine we're working on an abstraction called PartitionedTextWriter that implements the rolling file concept -- it acts as a streaming text writer that automatically switches to a new file after reaching a certain character threshold in the previous one.

The class is initialized with a base path and it needs to use that to generate file names for each partition. Because that's pure business logic without side effects, it makes perfect sense to put it into a static helper method.

Usually, mixing static and non-static members can be quite confusing. Let's see how that looks when we use partial classes instead:

public partial class PartitionedTextWriter : TextWriter
{
    private readonly string _baseFilePath;
    private readonly long _partitionLimit;

    private int _partitionIndex;
    private TextWriter _innerWriter;
    private long _partitionCharCount;

    public override Encoding Encoding { get; } = Encoding.UTF8;

    public PartitionedTextWriter(string baseFilePath, long partitionLimit)
    {
        _baseFilePath = baseFilePath;
        _partitionLimit = partitionLimit;
    }

    private void InitializeInnerWriter()
    {
        // Get current file path by injecting partition identifier in the file name
        // E.g. MyFile.txt, MyFile [part 2].txt, etc
        var filePath = GetPartitionFilePath(_baseFilePath, _partitionIndex);

        _innerWriter = File.CreateText(filePath);
    }

    public override void Write(char value)
    {
        // Make sure the underlying writer is initialized
        if (_innerWriter == null)
            InitializeInnerWriter();

        // Write content
        _innerWriter.Write(value);
        _partitionCharCount++;

        // When the char count exceeds the limit,
        // start writing to a new file
        if (_partitionCharCount >= _partitionLimit)
        {
            _partitionIndex++;
            _partitionCharCount = 0;

            _innerWriter?.Dispose();
            _innerWriter = null;
        }
    }

    protected override void Dispose(bool disposing)
    {
        if (disposing)
            _innerWriter?.Dispose();

        base.Dispose(disposing);
    }
}

public partial class PartitionedTextWriter
{
    // Pure helper function
    private static string GetPartitionFilePath(string baseFilePath, int partitionIndex)
    {
        if (partitionIndex <= 0)
            return baseFilePath;

        // Inject "[part x]" in the file name
        var fileNameWithoutExt = Path.GetFileNameWithoutExtension(baseFilePath);
        var fileExt = Path.GetExtension(baseFilePath);
        var fileName = $"{fileNameWithoutExt} [part {partitionIndex + 1}]{fileExt}";

        var dirPath = Path.GetDirectoryName(baseFilePath);
        if (!string.IsNullOrWhiteSpace(dirPath))
            return Path.Combine(dirPath, fileName);

        return fileName;
    }
}

As a developer reading this code for the first time, you will most likely appreciate this separation. When we're dealing with the notions of creating new files, we don't really care as much about how GetPartitionFilePath is implemented. Similarly, if we wanted to know how GetPartitionFilePath works, the rest of the code would likely act as unrelated noise.

One could argue that we could've instead moved our helper method to a different static class. That could work in some cases, especially if that method is going to be reused in other places as well. However, that would also make the method less discoverable and I generally prefer to keep dependencies as close to the source as possible, in order to reduce cognitive overhead.

Note that in this example both partial definitions of the class are placed in the same file. Since our primary goal is to group code rather than shred it to pieces, keeping things close makes more sense. I would consider moving the partitions to separate files only if they get too big to keep in one place.

This idea works especially well when combining with the "Resource acquisition is initialization" pattern. Using partial classes we can group methods responsible for initialization and separate them from the rest of the class.

In the following example we have a class called NativeDeviceContext which is a wrapper for a device context resource in the Windows operating system. The class can be constructed by providing a handle to the native resource, but the consumers will not be doing this manually. Instead they will be calling one of the available static methods such as FromDeviceName(...) that will take care of the initialization for them.

Again, let's see how it looks when we split out the static methods:

// Resource management concerns
public sealed partial class NativeDeviceContext : IDisposable
{
    public IntPtr Handle { get; }

    public NativeDeviceContext(IntPtr handle)
    {
        Handle = handle;
    }

    ~NativeDeviceContext()
    {
        Dispose();
    }

    public void SetGammaRamp(GammaRamp ramp)
    {
        // Call a WinAPI method via p/invoke
        NativeMethods.SetDeviceGammaRamp(Handle, ref ramp);
    }

    public void Dispose()
    {
        NativeMethods.DeleteDC(Handle);
        GC.SuppressFinalize(this);
    }
}

// Resource acquisition concerns
public partial class NativeDeviceContext
{
    public static NativeDeviceContext? FromDeviceName(string deviceName)
    {
        var handle = NativeMethods.CreateDC(deviceName, null, null, IntPtr.Zero);

        return handle != IntPtr.Zero
            ? new NativeDeviceContext(handle)
            : null;
    }

    public static NativeDeviceContext? FromPrimaryMonitor() { /* ... */ }

    public static IReadOnlyList<NativeDeviceContext> FromAllMonitors() { /* ... */ }
}

Similarly to the previous example, this makes the code a lot more readable by visually separating two unrelated (albeit coupled) concerns -- resource initialization and resource management.

Separating interface implementations

Another interesting thing we can do with partial classes is separate interface implementations. More often than not, members responsible for implementing interfaces don't really contribute to the core behavior of the class, so it makes sense to push them out.

For example, let's take a look at HtmlElement, a class that represents an element in the HTML DOM. It implements IEnumerable<T> for iterating over its children and ICloneable to facilitate deep copying.

Using partial classes we can arrange our code like this:

// Core concerns
public partial class HtmlElement : HtmlNode
{
    public string TagName { get; }
    public IReadOnlyList<HtmlAttribute> Attributes { get; }
    public IReadOnlyList<HtmlNode> Children { get; }

    public HtmlElement(string tagName,
        IReadOnlyList<HtmlAttribute> attributes,
        IReadOnlyList<HtmlNode> children)
    {
        /* ... */
    }

    public HtmlElement(HtmlElement other)
    {
        /* ... */
    }

    public string? GetAttributeValue(string attributeName) { /* ... */ }

    public IEnumerable<HtmlNode> GetDescendants() { /* ... */ }
}

// Implementation of IEnumerable<T>
public partial class HtmlElement : IEnumerable<HtmlNode>
{
    public IEnumerator<HtmlNode> GetEnumerator() => Children.GetEnumerator();

    IEnumerator IEnumerable.GetEnumerator() => GetEnumerator();
}

// Implementation of ICloneable
public partial class HtmlElement : ICloneable
{
    public object Clone() => new HtmlElement(this);
}

Putting interface implementations in partial classes can help us reduce the "routing noise" caused by methods that forward calls upstream. Additionally, since C# allows us to specify the class signature on each partition separately, we can conveniently group members that belong to the same interface.

This approach is also very useful when combined with conditional compilation. Occasionally, we may want to introduce API that depends on features available in a specific version of the framework. To do that, we have to use the #if directive which acts similarly to regions, making our code less readable.

Partial classes can help us make things tidier. Let's take a look at an example where we override DisposeAsync but only if we're building the assembly against .NET Standard 2.1:

public partial class SegmentedHttpStream : Stream
{
    private readonly HttpClient _httpClient;
    private readonly string _url;
    private readonly long _segmentSize;

    private Stream? _currentStream;

    public SegmentedHttpStream(HttpClient httpClient,
        string url, long length, long segmentSize)
    {
        /* ... */
    }

    /* Skipped overrides for Stream methods */

    protected override void Dispose(bool disposing)
    {
        if (disposing)
            _currentStream?.Dispose();

        base.Dispose(disposing);
    }
}

#if NETSTANDARD2_1
public partial class SegmentedHttpStream
{
    // This method is not available in earlier versions of the standard
    public override async ValueTask DisposeAsync()
    {
        if (_currentStream != null)
            await _currentStream.DisposeAsync();

        await base.DisposeAsync();
    }
}
#endif

The clear benefit of using partial classes in such cases is that we are able to completely mitigate the noise caused by the conditional blocks. It looks much better when they are pushed outwards instead of being in between code.

Organizing private classes

It's not all that uncommon to have private classes. These are convenient when we want to avoid namespace pollution while defining a type that's only used within one place. Typical case for this is when we need to implement a custom interface to override certain behavior in a third party library or a framework.

As an example, imagine we're exporting a sales report as an HTML document and we're using the Scriban engine to do it. In this particular scenario, we need to configure it so that templates can be resolved from the resources embedded in the assembly rather than from the file system. In order to do that, the framework expects us to provide a custom implementation of ITemplateLoader.

Seeing as our custom loader is going to be used only within this class, it makes perfect sense to define it as private class. However, with C# being as verbose as it is, private classes may introduce unwanted noise into our code.

Using partial classes, though, we can clean it up like this:

public partial class HtmlReportRenderer : IReportRenderer
{
    public async ValueTask<string> RenderReportAsync(SalesReport report, string templateCode)
    {
        var template = Template.Parse(templateCode);

        var templateContext = new TemplateContext
        {
            TemplateLoader = new CustomTemplateLoader(), // reference the private class
            StrictVariables = true
        };

        var model = new ScriptObject();
        model.SetValue("report", report, true);

        templateContext.PushGlobal(model);

        return await template.RenderAsync(templateContext);
    }
}

public partial class HtmlReportRenderer
{
    // This type is only used within HtmlReportRenderer
    private class CustomTemplateLoader : ITemplateLoader
    {
        private static readonly string ResourceRootNamespace =
            $"{typeof(HtmlReportRenderer).Namespace}.Templates";

        private static StreamReader GetTemplateReader(string templatePath)
        {
            var resourceName = $"{ResourceRootNamespace}.{templatePath}";

            var assembly = Assembly.GetExecutingAssembly();

            using var stream = assembly.GetManifestResourceStream(resourceName);
            if (stream == null)
                throw new MissingManifestResourceException("Template not found.");

            return new StreamReader(stream);
        }

        public string GetPath(
            TemplateContext context,
            SourceSpan callerSpan,
            string templateName) => templateName;

        public string Load(
            TemplateContext context,
            SourceSpan callerSpan,
            string templatePath) => GetTemplateReader(templatePath).ReadToEnd();

        public async ValueTask<string> LoadAsync(
            TemplateContext context,
            SourceSpan callerSpan,
            string templatePath) => await GetTemplateReader(templatePath).ReadToEndAsync();
    }
}

Grouping arbitrary code

We don't always need a special case to decide to use partial classes. In fact, sometimes it just feels right to split parts of our code into some logical blocks.

In this example we have a command line application that formats files. Both the options and the command behavior are defined as part of a single class, which may be a little confusing.

By using partial classes, we can split and group different parts of the class like so:

// Core options
public partial class FormatCommand
{
    [CommandOption("files", 'f', IsRequired = true, Description = "List of files to process.")]
    public IReadOnlyList<FileInfo> Files { get; set; }

    [CommandOption("config", 'c', Description = "Configuration file.")]
    public FileInfo? ConfigFile { get; set; }
}

// Options related to formatting
public partial class FormatCommand
{
    [CommandOption("indent-size", Description = "Override: indent size.")]
    public int? IndentSize { get; set; } = 4;

    [CommandOption("line-length", Description = "Override: line length.")]
    public int? LineLength { get; set; } = 80;

    [CommandOption("insert-eof-newline", Description = "Override: insert new line at EOF.")]
    public bool? InsertEofNewLine { get; set; } = false;
}

// Command implementation
[Command("format", Description = "Format files.")]
public partial class FormatCommand : ICommand
{
    private readonly IFormattingService _formattingService;

    public FormatCommand(IFormattingService formattingService)
    {
        _formattingService = formattingService;
    }

    private Config LoadConfig() { /* ... */ }

    public async ValueTask ExecuteAsync(IConsole console)
    {
        var config = LoadConfig();

        foreach (var file in Files)
        {
            await _formattingService.FormatAsync(config, file.FullName);
            console.Output.WriteLine($"Formatted: {file.FullName}");
        }
    }
}

Summary

Partial classes can be used for more than just auto-generated code. It's a powerful language feature that enables creative ways to arrange code into smaller logically-independent units. This can be very helpful when we want to reduce cognitive load or to simply keep things a bit more organized.

Since we're on the topic of refactoring, consider also checking out a few interesting ways we can use extension methods to write cleaner code. Similarly to partial classes, they might have more uses than you thought.

Follow me on Twitter to get notified when I post a new article ✨

CliFx - opinionated framework for building CLI applications

Oleksii Holub — Fri, 31 Jan 2020 10:29:33 +0000

Hi!

Recently I've been working on an application framework for building command line interfaces and today I'm happy to say that it's out of pre-release! Its goal is to provide a solid abstraction layer over command line arguments while offering a great experience for your users, so that you can focus all your efforts on the actual business logic of your application.

Here's the link: https://github.com/Tyrrrz/CliFx/

CliFx is a simple to use, yet powerful framework for building command line applications. Its primary goal is to completely take over the user input layer, letting you forget about the infrastructure and instead focus on writing your application. This framework uses a declarative class-first approach for defining commands, avoiding excessive boilerplate code and complex configurations.

An important property of CliFx, when compared to some other libraries, is that it's not just a parser -- it's a complete application framework. The main goal of the library is to provide a consistent and enjoyable development experience when building command line applications. At its core, CliFx is highly opinionated, giving preference to convention over configuration, strictness over extensibility, consistency over ambiguity, and so on.

Features

Complete application framework, not just an argument parser
Requires minimal amount of code to get started
Configuration via attributes
Handles conversions to various types, including custom types
Supports multi-level command hierarchies
Allows graceful command cancellation
Prints errors and routes exit codes on exceptions
Provides comprehensive and colorful auto-generated help text
Highly testable and easy to debug
Targets .NET Framework 4.5+ and .NET Standard 2.0+
No external dependencies

Screenshots

Usage

Quick start
Binding arguments
Argument syntax
Value conversion
Multiple commands
Reporting errors
Graceful cancellation
Dependency injection
Testing
Debug and preview mode
Reporting progress
Environment variables

Quick start

To turn your application into a command line interface you need to change your program's Main method so that it delegates execution to CliApplication.

The following code will create and run a default CliApplication that will resolve commands defined in the calling assembly. Using fluent interface provided by CliApplicationBuilder you can easily configure different aspects of your application.

public static class Program
{
    public static async Task<int> Main() =>
        await new CliApplicationBuilder()
            .AddCommandsFromThisAssembly()
            .Build()
            .RunAsync();
}

In order to add functionality to your application, you need to define at least one command. Commands are essentially entry points through which the user can interact with your application. You can think of them as something similar to controllers in ASP.NET Core.

To define a command, just create a new class that implements the ICommand interface and annotate it with [Command] attribute:

[Command]
public class HelloWorldCommand : ICommand
{
    public ValueTask ExecuteAsync(IConsole console)
    {
        console.Output.WriteLine("Hello world!");

        // Return empty task because our command executes synchronously
        return default;
    }
}

To implement ICommand, the class needs to define the ExecuteAsync method. This is the method that gets called when the user executes the command.

To facilitate both asynchronous and synchronous execution, this method returns a ValueTask. Since the simple command above executes synchronously, we can just put return default at the end. In an asynchronous command, however, we would use the async/await keywords instead.

As a parameter, this method takes an instance of IConsole, an abstraction around the system console. You should use this abstraction in places where you would normally use System.Console, in order to make your command testable.

With this basic setup, the user can execute your application and get a greeting in return:

> myapp.exe

Hello world!

Out of the box, your application now also supports the built-in help and version options:

> myapp.exe --help

MyApp v1.0

Usage
  myapp.exe

Options
  -h|--help         Shows help text.
  --version         Shows version information.

> myapp.exe --version

v1.0

Binding arguments

Commands can be configured to take input from command line arguments. To do that, we need to add properties to the command and annotate them with special attributes.

In CliFx, there are two types of argument bindings: parameters and options. Parameters are positional arguments that are identified by the order they appear in, while options are identified by their names.

Here's an example command that calculates a logarithm of a value, which uses both a parameter and an option:

[Command]
public class LogCommand : ICommand
{
    [CommandParameter(0, Description = "Value whose logarithm is to be found.")]
    public double Value { get; set; }

    [CommandOption("base", 'b', Description = "Logarithm base.")]
    public double Base { get; set; } = 10;

    public ValueTask ExecuteAsync(IConsole console)
    {
        var result = Math.Log(Value, Base);
        console.Output.WriteLine(result);

        return default;
    }
}

The above command has two inputs:

Value which is a parameter with order 0.
Base which is an option with name base and short name b.

Let's try running --help to see how this command is supposed to be used:

> myapp.exe --help

MyApp v1.0

Usage
  myapp.exe <value> [options]

Parameters
* value             Value whose logarithm is to be found.

Options
  -b|--base         Logarithm base.
  -h|--help         Shows help text.
  --version         Shows version information.

As we can see, in order to execute this command, at a minimum, the user has to supply a value:

> myapp.exe 10000

4

They can also set the base option to override the default logarithm base of 10:

> myapp.exe 729 -b 3

6

> myapp.exe 123 --base 4.5

3.199426017362198

On the other hand, if the user fails to provide the required parameter, they will get an error:

> myapp.exe -b 10

Missing value for parameter <value>.

Differences between parameters and options:

Parameters are identified by their relative order. Options are identified by two dashes followed by their name, or a single dash followed by their short name (single character).
Parameters can't be optional. Options are usually optional (as evident by the name), but can be configured to be required as well.
Parameters technically have a name, but it's only used in the help text.
Options can be configured to use the value of an environment variable as a fallback.
Both parameters and options can take multiple values, but such a parameter must be last in order to avoid ambiguity. Options are not limited in this aspect.

As a general guideline, prefer to use parameters for required inputs that the command can't work without. Use options for non-required inputs, or when the command has too many required inputs, or when specifying the option name explicitly provides a better user experience.

Argument syntax

This library supports an argument syntax which is based on the POSIX standard. To be fair, nobody really knows what the standard is about and very few tools actually follow it as they're supposed to, so for the purpose of having dashes and spaces, CliFx is using the "standard command line syntax".

In more detail, the following examples are all valid:

myapp --foo bar sets option "foo" to value "bar"
myapp -f bar sets option 'f' to value "bar"
myapp --switch sets option "switch" without value
myapp -s sets option 's' without value
myapp -abc sets options 'a', 'b' and 'c' without value
myapp -xqf bar sets options 'x' and 'q' without value, and option 'f' to value "bar"
myapp -i file1.txt file2.txt sets option 'i' to a sequence of values "file1.txt" and "file2.txt"
myapp -i file1.txt -i file2.txt sets option 'i' to a sequence of values "file1.txt" and "file2.txt"
myapp jar new -o cookie sets option 'o' to value "cookie" and retains two unbound arguments "jar" and "new"

Note that CliFx purposely employs a context-free parser when consuming command line arguments. That means that every input is parsed the same way.

This also means that myapp -i file1.txt file2.txt will always be parsed as an option with multiple values, even if the underlying bound property is not enumerable. For the same reason, unseparated arguments such as myapp -ofile will be treated as five distinct options 'o', 'f', 'i', 'l', 'e', instead of 'o' being set to "file".

When it comes to command name and parameters, they must appear in a strict order, before any options. The parser can't distinguish between arguments that make up a part of the command name and arguments that belong to command parameters, which is why the non-option arguments are bound at a later stage. It is done by trying to find a command that matches the longest sequence of arguments starting from the first, binding any remaining arguments to positional parameters.

The above design may seem like a deficiency, but it actually provides value in the fact that it's deterministic -- given a set of command line arguments, the semantics behind them always remain the same. This leads to a more consistent experience for both you as a developer, as well as for the users of your application.

Value conversion

Parameters and options can have different underlying types:

Standard types
- Primitive types (int, bool, double, ulong, char, etc)
- Date and time types (DateTime, DateTimeOffset, TimeSpan)
- Enum types
String-initializable types
- Types with a constructor that accepts a single string parameter (FileInfo, DirectoryInfo, etc)
- Types with a static method Parse that accepts a single string parameter (and optionally IFormatProvider)
Nullable versions of all above types (decimal?, TimeSpan?, etc)
Collections of all above types
- Array types (T[])
- Types that are assignable from arrays (IReadOnlyList<T>, ICollection<T>, etc)
- Types with a constructor that accepts a single T[] parameter (HashSet<T>, List<T>, etc)

When defining a parameter of an enumerable type, keep in mind that it has to be the only such parameter and it must be the last in order. Options, on the other hand, don't have this limitation.

Example command with an array-backed parameter:

[Command]
public class FileSizeCalculatorCommand : ICommand
{
    // FileInfo is string-initializable and IReadOnlyList<T> can be assgined from an array,
    // so the value of this property can be mapped from a sequence of arguments.
    [CommandParameter(0)]
    public IReadOnlyList<FileInfo> Files { get; set; }

    public ValueTask ExecuteAsync(IConsole console)
    {
        var totalSize = Files.Sum(f => f.Length);

        console.Output.WriteLine($"Total file size: {totalSize} bytes");

        return default;
    }
}

> myapp.exe file1.bin file2.exe

Total file size: 186368 bytes

Same command, but using an option for the list of files instead:

[Command]
public class FileSizeCalculatorCommand : ICommand
{
    [CommandOption("files")]
    public IReadOnlyList<FileInfo> Files { get; set; }

    public ValueTask ExecuteAsync(IConsole console)
    {
        var totalSize = Files.Sum(f => f.Length);

        console.Output.WriteLine($"Total file size: {totalSize} bytes");

        return default;
    }
}

> myapp.exe --files file1.bin file2.exe

Total file size: 186368 bytes

Multiple commands

Complex command line applications may have more than a single command in order to facilitate different workflows. In even more complex applications there may be multiple levels of commands, forming a hierarchy.

Whichever case it is, CliFx takes care of everything for you. All you need to do is specify appropriate command names in the attributes:

// Default command, i.e. command without a name
[Command]
public class DefaultCommand : ICommand
{
    // ...
}

// Child of default command
[Command("cmd1")]
public class FirstCommand : ICommand
{
    // ...
}

// Child of default command
[Command("cmd2")]
public class SecondCommand : ICommand
{
    // ...
}

// Child of FirstCommand
[Command("cmd1 sub")]
public class SubCommand : ICommand
{
    // ...
}

The user can access other commands by specifying the name before any other arguments, e.g. myapp.exe cmd1 arg1 -p 42.

In a multi-command application you may also choose to not have a default command, in which case executing your application without any arguments will just show the help text.

Requesting help on the application above will show:

> myapp.exe --help

MyApp v1.0

Usage
  myapp.exe [command]

Options
  -h|--help         Shows help text.
  --version         Shows version information.

Commands
  cmd1
  cmd2

You can run `myapp.exe [command] --help` to show help on a specific command.

As you can see, only two commands are listed here because cmd1 sub is not an immediate child of the default command. We can further refine our help query to get information on cmd1:

> myapp.exe cmd1 --help

Usage
  myapp.exe cmd1 [command]

Options
  -h|--help         Shows help text.

Commands
  sub

You can run `myapp.exe cmd1 [command] --help` to show help on a specific command.

Reporting errors

You may have noticed that commands in CliFx don't return exit codes. This is by design as exit codes are considered a higher-level concern and thus handled by CliApplication, not by individual commands.

Commands can report execution failure simply by throwing exceptions just like any other C# code. When an exception is thrown, CliApplication will catch it, print the error, and return an appropriate exit code to the calling process.

If you want to communicate a specific error through exit code, you can instead throw an instance of CommandException which takes an exit code as a parameter. When a command throws an exception of type CommandException, it is assumed that this was a result of a handled error and, as such, only the exception message will be printed to the error stream. If a command throws an exception of any other type, the full stack trace will be printed as well.

[Command]
public class DivideCommand : ICommand
{
    [CommandOption("dividend", IsRequired = true)]
    public double Dividend { get; set; }

    [CommandOption("divisor", IsRequired = true)]
    public double Divisor { get; set; }

    public ValueTask ExecuteAsync(IConsole console)
    {
        if (Math.Abs(Divisor) < double.Epsilon)
        {
            // This will print the error and set exit code to 1337
            throw new CommandException("Division by zero is not supported.", 1337);
        }

        var result = Dividend / Divisor;
        console.Output.WriteLine(result);

        return default;
    }
}

> myapp.exe --dividend 10 --divisor 0

Division by zero is not supported.


> echo Exit code was %errorlevel%

Exit code was 1337

Graceful cancellation

It is possible to gracefully cancel execution of a command and preform any necessary cleanup. By default an app gets forcefully killed when it receives an interrupt signal (Ctrl+C or Ctrl+Break), but you can easily override this behavior.

In order to make a command cancellation-aware, you need to call console.GetCancellationToken(). This method returns a CancellationToken that will trigger when the user issues an interrupt signal. Note that any code that comes before the first call to GetCancellationToken() will not be cancellation-aware and as such will terminate instantly. Any subsequent calls to this method will return the same token.

[Command("cancel")]
public class CancellableCommand : ICommand
{
    public async ValueTask ExecuteAsync(IConsole console)
    {
        console.Output.WriteLine("Printed");

        // Long-running cancellable operation that throws when canceled
        await Task.Delay(Timeout.InfiniteTimeSpan, console.GetCancellationToken());

        console.Output.WriteLine("Never printed");
    }
}

Keep in mind that a command may delay cancellation only once. If the user decides to press Ctrl+C again after the first time, the execution will be forcefully terminated.

Dependency injection

CliFx uses an implementation of ITypeActivator to initialize commands and by default it only works with types that have parameterless constructors.

In real-life scenarios, however, your commands will most likely have dependencies that need to be injected. CliFx doesn't come with its own dependency container but it makes it really easy to integrate any container of your choice.

For example, here is how you would configure your application to use Microsoft.Extensions.DependencyInjection (aka the built-in dependency container in ASP.NET Core).

public static class Program
{
    public static async Task<int> Main()
    {
        var services = new ServiceCollection();

        // Register services
        services.AddSingleton<MyService>();

        // Register commands
        services.AddTransient<MyCommand>();

        var serviceProvider = services.BuildServiceProvider();

        return await new CliApplicationBuilder()
            .AddCommandsFromThisAssembly()
            .UseTypeActivator(serviceProvider.GetService)
            .Build()
            .RunAsync();
    }
}

Testing

CliFx provides an easy way to write functional tests for your commands thanks to the IConsole interface.

You can use VirtualConsole to replace the application's stdin, stdout and stderr with your own streams. It has multiple constructor overloads allowing you to specify the exact set of streams that you want. Streams that are not provided are replaced with stubs, i.e. VirtualConsole doesn't leak to System.Console in any way.

Let's assume you want to test a simple command such as this one.

[Command]
public class ConcatCommand : ICommand
{
    [CommandOption("left")]
    public string Left { get; set; } = "Hello";

    [CommandOption("right")]
    public string Right { get; set; } = "world";

    public ValueTask ExecuteAsync(IConsole console)
    {
        console.Output.Write(Left);
        console.Output.Write(' ');
        console.Output.Write(Right);

        return default;
    }
}

By substituting IConsole you can write your test cases like this:

[Test]
public async Task ConcatCommand_Test()
{
    // Arrange
    using var stdout = new StringWriter();
    var console = new VirtualConsole(stdout);

    var command = new ConcatCommand
    {
        Left = "foo",
        Right = "bar"
    };

    // Act
    await command.ExecuteAsync(console);

    // Assert
    Assert.That(stdout.ToString(), Is.EqualTo("foo bar"));
}

And if you want, you can also test the whole application in a similar fashion:

[Test]
public async Task ConcatCommand_Test()
{
    // Arrange
    using var stdout = new StringWriter();
    var console = new VirtualConsole(stdout);

    var app = new CliApplicationBuilder()
        .AddCommand(typeof(ConcatCommand))
        .UseConsole(console)
        .Build();

    var args = new[] {"--left", "foo", "--right", "bar"};
    var envVars = new Dictionary<string, string>();

    // Act
    await app.RunAsync(args, envVars);

    // Assert
    Assert.That(stdout.ToString(), Is.EqualTo("foo bar"));
}

Generally, the first approach is more preferable as it's less verbose and lets you test a specific command in complete isolation.

Debug and preview mode

When troubleshooting issues, you may find it useful to run your app in debug or preview mode. To do it, simply pass the corresponding directive to your app along with other command line arguments.

If your application is ran in debug mode (using the [debug] directive), it will wait for debugger to be attached before proceeding. This is useful for debugging apps that were ran outside of the IDE.

> myapp.exe [debug] cmd -o

Attach debugger to PID 3148 to continue.

If preview mode is specified (using the [preview] directive), the app will short-circuit by printing consumed command line arguments as they were parsed. This is useful when troubleshooting issues related to command routing and argument binding.

> myapp.exe [preview] cmd arg1 arg2 -o foo --option bar1 bar2

Parser preview:
cmd <arg1> <arg2> [-o foo] [--option bar1 bar2]

You can also disallow these directives, e.g. when running in production, by calling AllowDebugMode and AllowPreviewMode methods on CliApplicationBuilder.

var app = new CliApplicationBuilder()
    .AddCommandsFromThisAssembly()
    .AllowDebugMode(true) // allow debug mode
    .AllowPreviewMode(false) // disallow preview mode
    .Build();

Reporting progress

CliFx comes with a simple utility for reporting progress to the console, ProgressTicker, which renders progress in-place on every tick.

It implements a well-known IProgress<double> interface so you can pass it to methods that are aware of this abstraction.

To avoid polluting output when it's not bound to a console, ProgressTicker will simply no-op if stdout is redirected.

var progressTicker = console.CreateProgressTicker();

for (var i = 0.0; i <= 1; i += 0.01)
    progressTicker.Report(i);

Environment variables

An option can be configured to use the value of a specific environment variable as a fallback.

[Command]
public class AuthCommand : ICommand
{
    [CommandOption("token", IsRequired = true, EnvironmentVariableName = "AUTH_TOKEN")]
    public string AuthToken { get; set; }

    public ValueTask ExecuteAsync(IConsole console)
    {
        console.Output.WriteLine(AuthToken);

        return default;
    }
}

> $env:AUTH_TOKEN="test"

> myapp.exe

test

Environment variables can be used as fallback for options of enumerable types too. In this case, the value of the variable will be split by Path.PathSeparator (which is ; on Windows, : on Linux).

Link to GitHub repo: https://github.com/Tyrrrz/CliFx/

Interview question: async & await (C#)

Oleksii Holub — Wed, 22 Jan 2020 11:59:26 +0000

Q: What is the purpose of async/await keywords?

These keywords allow writing asynchronous non-blocking code in a synchronous fashion.

This feature is facilitated by the Task/Task<T> classes or ValueTask/ValueTask<T> structs. These types represent an abstraction around an operation that may execute asynchronously.

We use await keyword to materialize the task into resulting value. Methods that contain await must be marked with the async keyword.

Q: What's the difference between asynchronous programming and multithreaded programming?

An asynchronous task does not necessarily represent execution on a separate thread. You can think of an asynchronous operation as just an aggregation of two events -- start and finish.

A good example of an asynchronous operation is reading a file from a hard drive. To read individual bytes of a file, operating system issues requests to the driver software, which in turn tells the drive to seek to a specific position by moving its mechanical head. The process of moving the head around is asynchronous, it's not an operation that runs on CPU, it's just a physical task you have to wait for completing. This is the type of "pure" async operation that can be represented with the async/await pattern.

That said, an asynchronous task may also represent some CPU-bound calculation happening on a separate thread, but this is an implementation detail. This is useful when you want to delegate execution to a different thread in order to not block the calling thread, while disguising it as an asynchronous operation. You can do that by calling Task.Run().

Overall, it's fair to say that every multithreaded execution can be represented as an asynchronous operation, but not every asynchronous operation necessarily employs additional threads.

Q: How does it work?

Let's take the following method as an example:

public async Task DoAsync()
{
    Console.WriteLine("Before await");

    await Task.Delay(TimeSpan.FromSeconds(1));

    Console.WriteLine("Between awaits");

    await Task.Delay(TimeSpan.FromSeconds(1));

    Console.WriteLine("After await");
}

Functionally, this code works by printing "Before await" to the console, waiting 1 second, printing "Between awaits", waiting 1 second again, and then printing "After await".

Everything until the first await is executed synchronously just like it would in a normal method. That means "Before await" will be printed on the same context that called this method.

The next thing that happens is that we create and run a new task using a static helper method Task.Delay(). This is a simple task that does nothing and automatically turns into completed after a specified delay.

Upon reaching the await keyword, the runtime will return control back to the calling method, which may or may not await it. If it does await, the same happens again, returning control to the caller of that method in turn, until it reaches a method in the call stack that either does not await (usually event handlers on a message loop thread) or materializes the task synchronously (e.g. entry point in a console application).

Once the task completes (after 1 second), the execution is returned back to our method, which continues by printing "Between awaits". By default, the execution will continue on the same context as the one that started executing this method.

Then the story repeats again by running and waiting on a new task, finally printing "After await". After the last message, the execution is returned to the caller method (if it awaited on DoAsync) so that it can continue executing further.

Q: What happens if we execute an asynchronous method but don't await it?

Nothing special, the operation represented by the task will continue its lifetime normally, but the result won't be observed. The task object itself will eventually be reclaimed by garbage collector.

Note that we can also use .ContinueWith() to handle the result in a call-back fashion.

Q: What happens if an exception is thrown within an asynchronous method?

If the method is awaited, the exception will instantly propagate to the calling method, then to the caller of that method, and so on, as long as the whole chain is awaited.

Otherwise, the exception will be considered unobserved, which can lead (in some versions of the framework) to the application crashing as soon as the task is disposed by the finalizer.

Q: Is it possible to make a lambda that executes asynchronously?

Yes.

var result = await new Func<Task>(async () => await Task.Delay(100));

Q: What happens when a method returns a Task without awaiting it?

Example:

public Task WaitAsync() => SomeOtherMethodAsync();

In this case, if an exception is thrown within asynchronous part of SomeOtherMethodAsync(), the WaitAsync() method will not be listed in the stack trace.

Q: Task type implements IDisposable, when are we supposed to dispose tasks?

The Dispose() method is not supposed to be invoked manually, a task will be disposed automatically when you await it or when it's reclaimed by the GC.

Q: What is the purpose of ConfigureAwait()?

By default, after the awaited task is completed, the execution continues on the originally captured context, i.e. the same thread that invoked the method. You can override that behavior by specifying ConfigureAwait(false), indicating that the execution may continue on a different context.

It's generally recommended using ConfigureAwait(false) wherever possible as it can offer minor performance gain and help prevent deadlocks.

Q: Why is the default behavior to continue on the captured context?

For compatibility reasons, due to constraints when working with Windows UI and classic ASP.NET applications. In one you can only interact with controls from the main thread, in the other you can only serve the response on the same thread that handled the request, so it was important that the execution continued on the captured context.

Q: What is the difference between Task and ValueTask?

The main difference is that Task is a class while ValueTask is a struct. Latter was added to the BCL to alleviate unnecessary pressure on the garbage collector caused by asynchronous methods that often return synchronously (e.g. cached result). You can think of ValueTask as a discriminated union of Task or a synchronous result. Since recently, ValueTask can also represent a result signaled by ValueTaskSource.

Q: What are the main downsides of using asynchronous methods compared to synchronous methods?

Asynchronous methods can be harder to debug, especially because exceptions thrown from asynchronous methods have difficult to read stack traces. Also, running many asynchronous tasks in a tight loop can put pressure on the garbage collector.

Formatting XAML files on build with XAMLStyler

Oleksii Holub — Thu, 09 Jan 2020 19:37:59 +0000

Recently, I've decided to switch from Visual Studio to Rider as my default .NET development environment. The main problem with Visual Studio was that was too slow when paired with ReSharper and, unfortunately, too useless when not. Rider offered me the best of both worlds.

However, one of the things that made the jump really difficult was the absence of XAMLStyler extension in Rider. It's an incredibly useful plugin that automatically formats your XAML files on save, letting you completely forget about sorting attributes, maintaining indentation, and other mundane things like that. After using it for three years, the idea of writing XAML without it was unbearable.

The problem is that .NET, historically, has been a rather closed ecosystem. If you wanted to extend the developer experience, that typically meant installing a custom Visual Studio extension or writing your own. Everything revolved around the same IDE, same workflow, same stack, same set of tools, and there wasn't much choice.

With the advent of .NET Core the situation started changing. We are now seeing a mentality shift where everything is evolving towards more modular and portable components, with even .NET SDK itself being shipped as a command line tool. Overall, .NET development experience is starting to resemble that of Node.js, which I personally think is a great thing.

Among other things, .NET Core also introduced the concept of .NET Core Global Tools. This feature enables any .NET developer to quickly download, install and run custom command line tools without leaving the terminal.

Since recently, XAMLStyler is also available as a .NET custom tool, which means you can run it as a CLI instead of relying on the Visual Studio extension. In this article I will show you how I integrated it into my build process, ensuring all XAML files are always properly formatted, regardless of which IDE I'm using.

Although I will be talking about XAMLStyler in particular, the approach described here can easily be extrapolated to any other tool.

What is XAMLStyler?

Given that I've praised it so much, it makes sense to say a few words about what is it that XAMLStyler actually does.

If you've worked with either WPF, UWP or Xamarin, chances are you already know how annoying it is to maintain good formatting in your XAML files. As developers, we like when everything is symmetrical, tidy and consistent. As developers, we also like when everything is automated.

Let's say we have the following piece of XAML that renders some UI:

<Grid>
  <Grid.ColumnDefinitions>
    <ColumnDefinition Width="*" /><ColumnDefinition Width="Auto" />
  </Grid.ColumnDefinitions>

  <!-- Update notice -->
  <TextBlock Grid.Column="0" VerticalAlignment="Center" Foreground="{DynamicResource SecondaryTextBrush}">
    <Run Text="Update is available to version:" />
    <Run Text="v" /><Run Text="{Binding NewVersion}" FontWeight="SemiBold" />
  </TextBlock>

  <!-- Confirmation button -->
  <Button Margin="6" Grid.Column="1" Command="{Binding UpdateCommand}" Content="UPDATE NOW" ></Button>
</Grid>

At first glance it might look sufficiently clean as it is, but it's not. The attributes are not sorted, element declarations are far too long, the closing tag on the button is redundant. What's worse -- there is no consistency whatsoever.

Luckily, I can just run XAMLStyler on the code above and get something that looks like this:

<Grid>
  <Grid.ColumnDefinitions>
    <ColumnDefinition Width="*" />
    <ColumnDefinition Width="Auto" />
  </Grid.ColumnDefinitions>

  <!--  Update notice  -->
  <TextBlock
    Grid.Column="0"
    VerticalAlignment="Center"
    Foreground="{DynamicResource SecondaryTextBrush}">
    <Run Text="Update is available to version:" />
    <Run Text="v" /><Run FontWeight="SemiBold" Text="{Binding NewVersion}" />
  </TextBlock>

  <!--  Confirmation button  -->
  <Button
    Grid.Column="1"
    Margin="6"
    Command="{Binding UpdateCommand}"
    Content="UPDATE NOW" />
</Grid>

This is way better. Individual elements are now arranged vertically, long attribute declarations broken into separate lines, redundant code gone, even the comments are formatted as well.

Note how it didn't touch the two consecutive Run elements in my code. That's because splitting them into multiple lines would result in a different layout being rendered. XAMLStyler is aware of these nuances and doesn't make changes that could introduce unwanted side-effects.

Another thing I really like about it, is how it sorts attributes. They are not merely sorted alphabetically, but also in accordance with the categories to which they belong. For example, properties like Margin and VerticalAlignment are placed above most other attributes, which makes them easier to find.

This and all other aspects of the formatting behavior are fully configurable in settings or via an external configuration file.

Using it as a .NET Core global tool

The release of .NET Core 2.1 introduced us with a feature that lets us use the so-called global tools. These are essentially console applications published as NuGet packages, that you can easily download and run.

To use XAMLStyler as a global tool, we just need to install it with the following command:

> dotnet tool install XamlStyler.Console --global

This downloads the corresponding NuGet package, extracts its contents to a shared directory, then puts the executable on the system PATH.

Following that, we can now run xstyler from the command line. To process all XAML files in a directory, we can use:

> xstyler --directory f:\Projects\Softdev\LightBulb\ --recursive

Processing: f:\Projects\Softdev\LightBulb\LightBulb\App.xaml
Processing: f:\Projects\Softdev\LightBulb\LightBulb\Views\RootView.xaml
Processing: f:\Projects\Softdev\LightBulb\LightBulb\Views\Components\AdvancedSettingsTabView.xaml
Processing: f:\Projects\Softdev\LightBulb\LightBulb\Views\Components\WhitelistSettingsTabView.xaml
Processing: f:\Projects\Softdev\LightBulb\LightBulb\Views\Components\GeneralSettingsTabView.xaml
Processing: f:\Projects\Softdev\LightBulb\LightBulb\Views\Components\HotKeySettingsTabView.xaml
Processing: f:\Projects\Softdev\LightBulb\LightBulb\Views\Components\LocationSettingsTabView.xaml
Processing: f:\Projects\Softdev\LightBulb\LightBulb\Views\Dialogs\MessageBoxView.xaml
Processing: f:\Projects\Softdev\LightBulb\LightBulb\Views\Dialogs\SettingsView.xaml
Processed 9 of 9 files.

This is nice, but not ideal. As evident by the name, global tools are installed system-wide, which is convenient for some one-off utilities but doesn't work so well with tools that your project relies on. That's because the project repository is no longer self-contained -- other developers (or future you) will now have to also manually install this tool on their machines, which adds an an extra unnecessary step.

Having to take any additional steps after git clone makes the developer experience worse and introduces indeterminism, so we want to avoid that. After all, it's always nice to keep the repository as a single source of truth.

Installing as a local tool

.NET Core local tools weren't really a thing until .NET Core 3.0 came out. Previously, there was a way to install a global tool inside a directory to make it "local" but that was a hack more than anything. In earlier versions of .NET Core we also had DotNetCliToolReference, but it's deprecated now and can't be used in new projects.

With the latest SDK, however, we can now install local tools simply by dropping the --global option:

> dotnet tool install XamlStyler.Console

Note that if we try to run this in our project's repository we will get the following error:

Cannot find a manifest file.
For a list of locations searched, specify the "-d" option before the tool name.
If you intended to install a global tool, add `--global` to the command.
If you would like to create a manifest, use `dotnet new tool-manifest`,
  usually in the repo root directory.

As the message states, in order to install local tools we will first need to create a manifest file in the root of the project repository. We can do this using the suggested command:

> dotnet new tool-manifest

That creates an empty manifest at /.config/dotnet-tools.json. We can now run the original command again, which will add a tool entry to this manifest file:

> dotnet tool install XamlStyler.Console

You can invoke the tool from this directory using the following commands:
  'dotnet tool run xstyler' or 'dotnet xstyler'.

Tool 'xamlstyler.console' (version '3.2001.0') was successfully installed.
Entry is added to the manifest file f:\Projects\Softdev\LightBulb\.config\dotnet-tools.json.

As long as the manifest file is tracked by git, any developer who clones the repository can run a simple command to download and install all tools listed in the manifest file:

> dotnet tool restore

Note that by installing XAMLStyler as a local tool, it is no longer added on the PATH, so we have to run it as dotnet xstyler instead of just xstyler. The name will be resolved anywhere in the repository or in any of its descendant directories.

In most cases, when working with .NET custom CLI tools, you will want to have them installed locally for your project. This provides a higher degree of portability as opposed to global tools.

Integrating in the build process

Things are much better now that XAMLStyler is installed locally in our project, but we still have to perform manual actions. This is unacceptable.

We can go a step further and execute the tool during build by extending the default workflow. With MSBuild-based projects this is possible by adding a custom target.

A target is a set of tasks that can be either invoked manually or configured to run automatically after some other, usually built-in, target. As a trigger we can use the default BeforeBuild target which is executed right before the build starts.

Let's update our project file accordingly:

<Project Sdk="Microsoft.NET.Sdk.WindowsDesktop">

  <PropertyGroup>
    <OutputType>WinExe</OutputType>
    <UseWPF>true</UseWPF>
  </PropertyGroup>

  <!-- ... -->

  <Target Name="Format XAML" AfterTargets="BeforeBuild">
    <Exec Command="dotnet tool restore" />
    <Exec Command="dotnet xstyler -r -d &quot;$(MSBuildProjectDirectory)&quot;" />
  </Target>

</Project>

As you can see, this target will trigger automatically on each build, execute dotnet tool restore and then run XAMLStyler recursively on the project's directory.

By executing the restore first, we ensure that the tool is always available. Once it's downloaded and installed, running the restore again will just complete instantly so we don't have to worry about any performance issues. I've found this to be the most reliable approach after experimenting with a few different ones.

Now, when we run dotnet build on the project, it will also execute XAMLStyler to format all XAML files inside of it. The same will happen if I build my project in Visual Studio, Rider, VS Code, or anywhere else. Also, since the tool is installed locally and restored as part of the build, any developer who clones the repository won't have to take any additional steps to get up and running.

Summary

XAMLStyler is an awesome tool that will make you forget about formatting in your XAML files once and for all. By integrating it into the build process, we ensure that all XAML files will adhere to a consistent and clean format, no matter who's working on it and where.

With .NET Core 3.0 we can now install locally-scoped command line tools. This is better than the previously available global tool concept because it's more portable. By using local tools we can also easily integrate custom workflows into our project.

If you're interested to learn more about .NET Core global and local tools, check out this article by Andrew Lock and another one by Stuart Lang. There's also a curated list of .NET custom tools maintained by Nate McMaster.

Follow me on Twitter to get notified when I post a new article ✨

Parsing in F# with FParsec

Oleksii Holub — Sun, 05 Jan 2020 18:45:55 +0000

Two months ago I wrote an article about monadic parser combinators in C# and how you can use this approach to express complex recursive grammar. To extend this topic a bit, I also wanted to show how the same approach can be applied in a functional-first language such as F#.

FParsec may not be the only, but is definitely the most popular F# framework for building parsers. Designed as a clone to Haskell's Parsec, with which it shares a very similar API, this library is focused on high throughput and human-readable error messages.

As a personal learning exercise, I've decided to port LtGt from Sprache to FParsec. Just by simply rewriting the same rules in a new framework, I saw a 7.35x performance improvement, which is pretty impressive!

In this article I'll guide you through the basics of this library and show how you can write parsers with it.

Parser primitives

Every parser in FParsec is an instance of Parser<'Result, 'State> which is a type of function that takes a CharStream<'State> and returns Reply<'Result>. Yes, parsers in FParsec can have state which means you can express context-sensitive grammar, but I'm not going to talk about it here. All we need to know is that a parser is a function that takes an input and produces a result or a failure.

Similarly to Sprache and other parser combinator libraries, FParsec builds upon a set of powerful primitives. These are accessible as top-level functions and provide starting points for even the most complicated parsers.

Among these are:

anyChar -- parses any single character.
pchar -- parses a specific character.
anyOf -- parses any of the specified characters.
satisfy -- parses any character that satisfies a predicate.
letter, digit, upper, lower -- parses a character that belongs to a specific category.
pstring -- parses a specified string.

These can be used to parse basic inputs but, being primitives, they are obviously not very useful on their own. In order to compose these simple parsers into more sophisticated ones, we need to use combinators.

In essence, combinators are just functions that take existing parsers as parameters and construct new, enhanced versions of them. I personally like to divide them into three logical groups:

Chaining combinators
Grouping combinators
Mapping combinators

Let's take a look at some of them.

Chaining multiple parsers

At the base of all chaining combinators in FParsec stands the bind operator (>>=). It constructs a new parser based on the result of the previous one.

We can use the bind operator to chain multiple sequential parsers and combine them into a higher-order parser. As a simple example, here's how we can express two consecutive characters that appear in opposite case:

open FParsec

let isOppositeCase a b = isUpper a <> isUpper b

let sawtooth = anyChar >>= fun a -> satisfy <| isOppositeCase a >>= fun b -> preturn (a, b)
//             ~~~~~~~              ~~~~~~~    ~~~~~~~~~~~~~~~~              ~~~~~~~~~~~~~~
// parse any char --^                  ^         ^                              ^
//                                     |         |                              |
// then any char satisfying predicate --         -- partially-applied function  |
//                                                                              |
//             then combine the results in a tuple ------------------------------

The parser sawtooth will succeed on inputs such as "aB", "Aa", "dP" and similar, but will fail on "ab", "AA", "dp".

Although it is quite flexible, the bind operator isn't very convenient to use. It's too verbose and, unlike this particular case, we rarely need the result of the previous parser to construct the next one.

This is why FParsec also offers a few high level chaining operators:

.>> -- chains two sequential parsers and retains the result of the one on the left.
>>. -- chains two sequential parsers and retains the result of the one on the right.
.>>. -- chains two sequential parsers and combines both of their results in a tuple.

For example, here's how we can compose a parser that will consume "5,9" and turn it into an F# tuple consisting of characters '5' and '9', discarding the comma in the middle:

let commaSeparatedDigits = digit .>> pchar ',' .>>. digit

Individual parsers in the expression are chained pairwise left to right. Here's the same code but with parentheses highlighting how it works:

let commaSeparatedDigits = ((digit .>> pchar ',') .>>. digit)
//                           ~~~~~ ^              ^  ^ ~~~~~
//                                 |              |  |
//   take result of the left side --              ---- take both

We can further improve this by using skipChar instead of pchar to avoid unnecessary allocations for the result we're not interested in:

let commaSeparatedDigits = digit .>> skipChar ',' .>>. digit

This kind of parser chaining can be useful to express grammar rules with a fixed structure. Sometimes, however, we also need to express repetition, which is when a certain symbol may appear more than once. To do that, we can use one of the sequence combinators that FParsec offers:

many -- chains the same parser until it fails.
sepBy -- chains the same parser separated by another parser.
manyTill -- chains the same parser until another parser succeeds.
manyChars, manyCharsTill -- same as many and manyTill but optimized for strings.

For example, we can use manyChars to enhance the original commaSeparatedDigits parser so that it can handle multiple consecutive digits around the comma:

let commaSeparatedDigits = manyChars digit .>> skipChar ',' .>>. manyChars digit
//                         ~~~~~~~~~~~~~~~                       ~~~~~~~~~~~~~~~
//                              ^--  take many digits instead of one  --^
//                                    (manyChars produces a string)

This will now work on inputs such as "1337,69", producing a tuple of strings instead of characters. We can also change it a bit and use sepBy so that it can handle an arbitrary number of digits separated by commas:

let manyCommaSeparatedDigits = manyChars digit |> sepBy <| skipChar ','
//                                                ~~~~~~~~~~~~~~~~~~~~~
//        apply the parser many times separated by commas --^

If we run the last parser on "5,96,10" we will get a matching list of strings ["5"; "96"; "10"], which we can operate on.

I like to apply F#'s forward and backward pipes in compositions like this because it makes the parsers more readable -- the expression above is equivalent to sepBy (manyChars digit) (skipChar ',').

Note that the parsers created with these sequence combinators will always succeed -- if the parser x fails, many x will simply produce an empty list. If you need the underlying parser to succeed at least once, you can use the non-empty variants of these combinators instead: many1, sepBy1, many1Chars, etc.

Grouping alternatives

Another way we can combine parsers is by grouping them. This lets us defer complex grammar rules to multiple separate parsers.

We can do this with the choice operator (<|>). For example, to combine letter and digit into one parser, we may write:

let letterOrDigit = letter <|> digit

The above parser will succeed on "a", "B" as well as "0", "4". Similarly, we can write a parser for a string that contains only letters or digits:

let alphanumericString = manyChars (letter <|> digit)

Of course, we can also chain the choice operator multiple times to provide more than two alternatives:

let letterOrDigitOrSpecial = letter <|> digit <|> pchar '*' <|> pchar '#' <|> pchar '%'

When we're dealing with many alternatives, however, it's better to use the choice function instead. It's an optimized version of the <|> operator that takes multiple parsers as a sequence:

let letterOrDigitOrSpecial =
    choice [
        letter
        digit
        pchar '*'
        pchar '#'
        pchar '%'
    ]

Not only is the parser above going to work on a bunch of different inputs, it will also produce informative error messages if the parser fails. For example, if we try to use it to parse "!", it will say:

Failure:
Error in Ln: 1 Col: 1
!
^
Expecting: decimal digit, letter, '#', '%' or '*'

An important thing to note about choice combinators in FParsec is that all of them are non-backtracking by default. This ensures the path of least resistance for writing high performance code but may lead to slightly unexpected results.

For example, in the following snippet one of the alternative parsers (fooXyz) will never be evaluated:

let fooBar = pstring "foo" .>>. pstring "bar"
let fooXyz = pstring "foo" .>>. pstring "xyz"

let fooBarOrFooXyz =
    choice [
        fooBar
        fooXyz
    ]

// Input:         fooxyz
//                ~~~===
//                  ^  ^
// fooBar:          |  |
// 1. take "foo" ----  |   success, change state
// 2. take "bar" -------   error: expected "bar", found "xyz"
//
// fooXyz will not be tried because fooBar changed state

Trying to run fooBarOrFooXyz on "fooxyz" will fail, but will work well with "foobar". This is because the underlying parser fooBar partially succeeds on "fooxyz" and changes the parser state.

To instruct FParsec to backtrack in such cases, we can wrap the individual parsers in attempt which will reset the state after the underlying parser fails:

let fooBar = pstring "foo" .>>. pstring "bar"
let fooXyz = pstring "foo" .>>. pstring "xyz"

let fooBarOrFooXyz =
    choice [
        attempt fooBar
        attempt fooXyz
    ]

// Input:         fooxyz
//                ~~~===
//                  ^  ^
// fooBar:          |  |
// 1. take "foo" ----  |   success, change state
// 2. take "bar" -------   error: expected "bar", found "xyz"
// 3. reset state
//
//                fooxyz
//                ~~~~~~
//                  ^  ^
// fooXyz:          |  |
// 1. take "foo" ----  |   success, change state
// 2. take "xyz" -------   success, change state
// 3. produce result: ("foo", "xyz")

This will work as expected. Of course, you should use attempt sparingly to avoid unnecessary backtracking. Alternatively, it's also possible to selectively avoid changing parser state by using a variant of chaining combinators:

let fooBar = pstring "foo" .>>.? pstring "bar"
let fooXyz = pstring "foo" .>>.? pstring "xyz"
//   state won't change here --^

// ...so backtracking is not necessary
let fooBarOrFooXyz =
    choice [
        fooBar
        fooXyz
    ]

These variants of chaining combinators are just like the regular ones, except that the constructed parser is treated as a single whole instead of two separate parsers:

>>=? -- same as >>= but doesn't change state.
>>? -- same as >>. but doesn't change state.
.>>? -- same as .>> but doesn't change state.
.>>.? -- same as .>>. but doesn't change state.

Mapping results

Let's not forget that the main purpose of a parser is to extract semantics from text. In programming terms, that means converting raw character strings to some domain types.

There are two main operators associated with that:

Return operator (>>%) -- sets the result of a parser to the specified value.
Map operator (|>>) -- applies a function to the result of a parser.

We can use the return operator to construct a parser that will simply produce a constant if it succeeds:

type ShapeType = Circle | Rectangle

// This parser produces ShapeType
let shapeType =
    choice [
        skipString "circle" >>% Circle
        skipString "rectangle" >>% Rectangle
    ]

Combination of skipString and >>% is so common that it also has its own optimized shorthand stringReturn:

let shapeType =
    choice [
        stringReturn "circle" Circle
        stringReturn "rectangle" Rectangle
    ]

The example parser above will turn "circle" into ShapeType.Circle and "rectangle" into ShapeType.Rectangle.

To perform more complex conversions, we can use the map operator which lets us transform the result using a function. For example, here we map a tuple into our own domain type:

// Single-case union
type NameValuePair = NameValuePair of string * string

// Parses: name=value;
let nameValuePair =
    (anyChar |> manyCharsTill <| skipChar '=')  // take a string of any chars until equals sign
    .>>.
    (anyChar |> manyCharsTill <| skipChar ';')  // take a string of any chars until semicolon
    |>> NameValuePair                           // map the tuple to a domain type

The reason we were able to simply write |>> NameValuePair is because the type's constructor expects a tuple of two strings, which is exactly what we're supplying it with. In a general form we can use any function:

let nameValuePair =
    (anyChar |> manyCharsTill <| skipChar '=')
    .>>.
    (anyChar |> manyCharsTill <| skipChar ';')
    |>> fun (name, value) -> NameValuePair (name, value)
//           ~~~~  ~~~~~
//            ^      ^
//            |      |
//      deconstructed tuple

JSON processor using FParsec

There are still a lot of other primitives and combinators in FParsec. In fact, with around 200 of them in total, covering them all here would be both ambitious and unnecessary.

Instead, let's take a look at how it all fits together by writing our own JSON parser.

I know that the official documentation has a tutorial on exactly this topic and on top of that I've already shown how to write a JSON processor in C# using Sprache in my previous article. That being said, I think JSON grammar has a perfect mixture of small and non-trivial rules that makes it a pretty good "hello world" of parser frameworks.

Before we can begin, however, we have to establish what is it that we want our parsers to produce. Since we're dealing with JSON, being a typical context-free language, its structure can be expressed using a syntax tree.

Thanks to F#'s recursive discriminated unions, defining ASTs is really trivial:

namespace MyJsonProcessor

type JsonNode =
    // Literals
    | JsonNull                            // null
    | JsonBool of bool                    // true | false
    | JsonNumber of float                 // 10 | 5.63
    | JsonString of string                // "foobar"
    // Complex types
    | JsonArray of List<JsonNode>         // [ "hello", 10, true ]
    | JsonObject of Map<string, JsonNode> // { "count": 4, "items": [5, 1, -8, 3.14] }

These will represent the individual nodes in the tree and later we're going to add some logic to navigate through the hierarchy.

Let's create a new module called JsonGrammar in the same file. The functions in this module will represent individual grammar rules so their names will match those of the actual data types but with camel case instead.

The first parser that we'll write is going to be jsonNull, which is really easy:

namespace MyJsonProcessor

// ...

module JsonGrammar =

    open FParsec

    let jsonNull = stringReturn "null" JsonNull .>> spaces
    //                           ~~~~  ~~~~~~~~     ~~~~~~
    //                            ^        ^           ^-- skip trailing whitespace
    //                            |        |
    //                match this --        -- produce this

We are using stringReturn to consume a string "null" and return the corresponding value -- union case JsonNull.

Since whitespace is ignored in JSON, we have to also account for it in our parsers or else they will fail when they encounter any whitespace character. We can do that by chaining our parser with spaces which will consume and discard any trailing spaces. As long as we do that at the end of every parser, we will be fine.

The traditional way of dealing with insignificant whitespace involves writing a separate lexer component, which parses raw characters into so-called tokens. It can be done with FParsec as well and it provides many benefits, but for the sake of simplicity we'll be writing a scanner-less parser this time.

If you're following along and your IDE is complaining that the type of a parser can't be inferred -- help it by explicitly specifying it as let jsonNull : Parser<_, unit> = .... We're not going to be using state so we can set it to unit. By the end of this exercise we will have an entry point function that will help the F#'s compiler correctly determine the generic types, but for now we can write them out manually.

With JsonNull out of the way, let's proceed on to our next data type, JsonBool:

namespace MyJsonProcessor

// ...

module JsonGrammar =

    // ...

    let jsonBoolTrue = stringReturn "true" <| JsonBool true .>> spaces
    let jsonBoolFalse = stringReturn "false" <| JsonBool false .>> spaces

    let jsonBool = jsonBoolTrue <|> jsonBoolFalse

Since a boolean can be in either of two states, we can handle them separately and combine them using the choice operator. This is an advantage of combinatory parsing -- we can split a complex grammar rule into many simpler ones.

When it comes to jsonNumber, FParsec already does most of the work by providing us with pfloat, a parser that matches text that represents a floating point number and converts it to float (which is an alias for System.Double in F#). That means we can just write our parser like this:

namespace MyJsonProcessor

// ...

module JsonGrammar =

    // ...

    let jsonNumber = pfloat .>> spaces |>> JsonNumber

Note how we used the map operator (|>>) in the parser above. This takes the result of pfloat and pipes it into JsonNumber's constructor, which conveniently expects a float.

Moving on to JsonString, where we need to handle text of any length between two double quotes. To make our life easier we can introduce a few helper functions:

namespace MyJsonProcessor

// ...

module JsonGrammar =

    // ...

    // Applies popen, then pchar repeatedly until pclose succeeds,
    // returns the string in the middle
    let manyCharsBetween popen pclose pchar = popen >>? manyCharsTill pchar pclose

    // Parses any string between popen and pclose
    let anyStringBetween popen pclose = manyCharsBetween popen pclose anyChar

    // Parses any string between double quotes
    let quotedString = skipChar '"' |> anyStringBetween <| skipChar '"'
    // is equivalent to: anyStringBetween (skipChar '"') (skipChar '"')

The combinator manyCharsBetween applies popen, then repeatedly applies pchar until it encounters pclose. We build upon it and define a higher-level combinator anyStringBetween which will parse a string between two parsers consisting of any characters. It's effectively the same as the .*? regular expression.

Finally, we also define quotedString which is anyStringBetween with double quotes already applied. Note how the use of forward and backward pipes makes the code fluent -- the order of tokens in this expression actually matches the order in which the parser consumes them!

Now, defining jsonString becomes really trivial:

namespace MyJsonProcessor

// ...

module JsonGrammar =

    // ...

    let jsonString = quotedString .>> spaces |>> JsonString

Ok, to be fair this doesn't handle escape sequences like \t, \n, etc. which are allowed in JSON strings. Handling those is not very interesting so I will cheat here by claiming that this is left as an exercise for the reader. 🙂

With all of the literals out of the way, we can group them into a single parser. This is not necessary but it makes it slightly easier to reason about the grammar.

namespace MyJsonProcessor

// ...

module JsonGrammar =

    // ...

    let jsonLiteral =
        choice [
            jsonNull
            jsonBool
            jsonNumber
            jsonString
        ]

Our two remaining cases, JsonArray and JsonObject, are slightly more complicated. That's because the corresponding grammar rules are recursive -- arrays and objects can both be nested within each other.

What makes it harder is that the order of declarations in a file matters in F# as opposed to C#. This means we can't reference a parser if it hasn't been defined above, so it's not possible to introduce a cyclic recursion that easily.

FParsec provides its own way to work around this problem with the createParserForwardedToRef function. We can use it to construct a dummy jsonNode parser, which we can later update using its reference:

namespace MyJsonProcessor

// ...

module JsonGrammar =

    // ...

    let jsonNode, jsonNodeRef = createParserForwardedToRef()

At this point we can temporarily pretend that jsonNode is already defined and will evaluate any valid JSON node, including arrays and objects.

Now let's parse an array which is an enumeration of comma-separated items contained within square brackets. Each item in an array can be any literal, an object, or another array as well -- or in other words, any node.

We can define a helper function called manyContained which will take care of the list inside the brackets. With that, the implementation of jsonArray looks like this:

namespace MyJsonProcessor

// ...

module JsonGrammar =

    // ...

    // Parses: popen p psep p psep p psep ... p pclose
    let manyContained popen pclose psep p = between popen pclose <| sepBy p psep

    let jsonArray =
        jsonNode                       // parse JSON nodes...
        |> manyContained               // contained within...
            (skipChar '[' .>> spaces)  // openning square bracket...
            (skipChar ']' .>> spaces)  // and closing square bracket...
            (skipChar ',' .>> spaces)  // separated by commas
        |>> JsonArray

The combinator between popen pclose p is an optimized version of popen >>. p .>> pclose. We pipe sepBy into it, which will apply a parser repeatedly using a separator. By combining these two combinators, we get a yet another combinator which will apply an opening parser, then the main parser delimited by the separator parser, and finally the closing parser.

Ultimately, the implementation of jsonArray comes down to just passing the correct arguments to our helper function.

Dealing with JsonObject is pretty similar except that we have to parse an enumeration of properties instead of just nodes. A property in JSON is a quoted string followed by a colon and a value. Conveniently, we've already defined a useful combinator called quotedString when we were writing jsonString earlier, which we can now use like so:

namespace MyJsonProcessor

// ...

module JsonGrammar =

    // ...

    // Produces a tuple: (string, JsonNode)
    let jsonProperty =
        quotedString .>> spaces .>> skipChar ':' .>> spaces .>>. jsonNode .>> spaces

    let jsonObject =
        jsonProperty
        |> manyContained
            (skipChar '{' .>> spaces)
            (skipChar '}' .>> spaces)
            (skipChar ',' .>> spaces)
        |>> Map.ofList
        |>> JsonObject

Finally, now that we have parsers for all types of nodes defined, we can actually implement jsonNode using its reference we've obtained earlier:

namespace MyJsonProcessor

// ...

module JsonGrammar =

    // ...

    do jsonNodeRef :=
        choice [
            jsonObject
            jsonArray
            jsonLiteral
        ]

Tada! Our parser is technically complete. To test it out, we can use the run function to evaluate a parser on a string:

[<EntryPoint>]
let main _ =
    run jsonNode "{ \"arr\": [1, 3.14, false, null] }" |> printfn "%O"
    0

// Prints:
Success: JsonObject
  (map
     [("arr",
       JsonArray [JsonNumber 1.0; JsonNumber 3.14; JsonBool false; JsonNull])])

Of course, a JSON processor isn't very useful if it doesn't have any means for querying data. To that end, let's create another module and add some business logic:

namespace MyJsonProcessor

// ...

module Json =

    open FParsec

    /// Tries to parse a string as a JSON node.
    let tryParse source =
        // Discard leading whitespace and ensure the parser reaches end of stream
        let jsonNodeFull = spaces >>. JsonGrammar.jsonNode .>> eof

        // Run parser and convert FParsec's result to F#'s standard result
        match run jsonNodeFull source with
        | Success (res, _, _) -> Result.Ok res
        | Failure (err, _, _) -> Result.Error err

    /// Tries to extract a boolean value from a node.
    let tryBool (node : JsonNode) =
        match node with
        | JsonBool b -> Some b
        | _ -> None

    /// Tries to extract a string value from a node.
    let tryString (node : JsonNode) =
        match node with
        | JsonString s -> Some s
        | _ -> None

    /// Tries to extract a float value from a node.
    let tryFloat (node : JsonNode) =
        match node with
        | JsonNumber n -> Some n
        | _ -> None

    /// Tries to extract an int value from a node.
    let tryInt (node : JsonNode) =
        node |> tryFloat |> Option.map int

    /// Tries to get an item by its index.
    let tryItem (i : int) (node : JsonNode) =
        match node with
        | JsonArray a -> a |> List.tryItem i
        | _ -> None

    /// Tries to get a child node by its name.
    let tryChild (name : string) (node : JsonNode) =
        match node with
        | JsonObject o -> o |> Map.tryFind name
        | _ -> None

By defining our own tryParse function, we hide FParsec's API behind a layer of abstraction so that our library is slightly easier to use. We also wrap the original jsonNode in a parser that will discard leading whitespace and ensure that the end of the stream has been reached -- the latter is important because we don't want to only partially parse the input.

Functions tryBool, tryString, tryFloat, tryInt make it easier to extract values without having to do pattern matching yourself. To make it simpler to work with objects and arrays, there's also tryChild which will get an object child by its name and tryItem which will get an array item by its index.

Let's put it all together and test it on some real input:

[<EntryPoint>]
let main _ =
    let str = """
    {
        "quiz": {
            "sport": {
                "q1": {
                    "question": "Which one is correct team name in NBA?",
                    "options": [
                        "New York Bulls",
                        "Los Angeles Kings",
                        "Golden State Warriors",
                        "Huston Rocket"
                    ],
                    "answer": "Huston Rocket"
                }
            },
            "math": {
                "q1": {
                    "question": "5 + 7 = ?",
                    "options": [
                        "10",
                        "11",
                        "12",
                        "13"
                    ],
                    "answer": "12"
                },
                "q2": {
                    "question": "12 - 8 = ?",
                    "options": [
                        1,
                        2,
                        3,
                        4
                    ],
                    "answer": 4
                }
            }
        }
    }
    """

    // Get the value of quiz.sport.q1.options[2]
    match Json.tryParse str with
        | Ok result ->
            result
            |> Json.tryChild "quiz"
            |> Option.bind (Json.tryChild "sport")
            |> Option.bind (Json.tryChild "q1")
            |> Option.bind (Json.tryChild "options")
            |> Option.bind (Json.tryItem 2)
            |> Option.bind (Json.tryString)
            |> Option.iter (printfn "Value: %s")
            0
        | Error err ->
            printfn "Parsing failed: %s" err
            1

Once we run this piece of code, we should see Value: Golden State Warriors. Awesome, we've verified that our parser works and made a pretty simple but useful API around it.

The finished project is also available as a gist here.

Wait, but where are the monads?

If you've come here after my previous article or were otherwise expecting to see monads here, you might be a bit surprised that there were none so far. This is because the combinators FParsec provides out of the box are so powerful you very rarely need to resort to it.

We could have, for example, defined our JSON array parser from earlier like this instead:

let jsonArray = parse {
    // [
    do! skipChar '['
    do! spaces

    // ...
    let! items = sepBy jsonNode <| (skipChar ',' .>> spaces)

    // ]
    do! skipChar ']'
    do! spaces

    return JsonArray items
}

The parse computation expression allows us to chain sequential parsers in a more imperative way. Under the hood it also uses the bind operator (>>=) but provides a slightly cleaner syntax when dealing with a lot of parsers. If you've used monads in C#, this is similar to LINQ comprehension syntax but way more powerful.

Stephan Tolksdorf, author of FParsec, discourages the use of this syntax in the documentation because it's rarely called for and may negatively impact performance.

That said, some grammar rules are easier to express using this syntax. For example, let's say we were parsing an HTML element and needed to match a closing tag that has the same name as the opening tag:

let htmlElement = parse {
    // <div ...>
    do! skipChar '<'
    let! tagName = manyChars (letter <|> digit)
    do! spaces

    let! attributes = many attribute
    do! spaces

    do! skipChar '>'
    do! spaces

    // ...
    let! children = many elementChild

    // </div>
    do! skipString "</"
    do! skipString tagName // use the result of the earlier parser
    do! spaces

    do! skipChar '>'
    do! spaces

    return HtmlElement (tagName, attributes, children)
}

Writing the same thing using binds may not be as pleasant.

Summary

FParsec is an incredibly robust framework for building parsers with the combinatorial approach. With some careful fine-tuning, parsers written with this library may even outperform traditional hand-rolled parsers. If you're using .NET and want to build a text processor, compiler, or a DSL interpreter -- FParsec is likely a no-brainer.

Should you ever get stuck, I recommend reading the API reference and the user guide.

In case you're curious to see some other examples of FParsec in action, check out the official samples. There are also some open source projects that depend on this library, like GraphQL.NET, QSharp Compiler, and LtGt.

Other articles about FParsec: by Jake Witcher, by Phillip Trelford, by Tamizh Vendan.

Follow me on Twitter to get notified when I post a new article ✨

💡 LightBulb v2 released!

Oleksii Holub — Sat, 23 Nov 2019 13:40:46 +0000

After over a year since last release and long months full of work, I'm happy to release LightBulb v2 to the world!

For those who don't know, LightBulb is a Windows background application that controls the gamma of your display(s), gradually reducing blue light as it gets late. It's similar to the Windows built-in NightLight feature and apps like f.lux, but offers much wider customization options, more features, better performance, an elegant UI, and it's completely open-source!

⭐ You can download it here but please note that you also need to have .NET Core 3 runtime for desktop apps installed.

The main new feature of v2 is the completely reworked UI and the addition of the dashboard screen which shows up-to-date relevant information. Here you can see the sundial which indicates where you are in the 24-hour cycle separated by phases of day-time, night-time, and transitions.

Also, I've added the brightness configuration to go with color temperature. This will let you personalize LightBulb even further and your eyes will be thankful. 👀

Settings have also seen a few enhancements, namely the ability to manually fine-tune every configuration value, including your location. Previous versions of LightBulb required you to have active internet connection in order to accurately calculate sunrise & sunset times, but not anymore -- you can set it manually or use the text box to search for any valid address.

For the full list of changes (there are over 30 of them), check out the changelog! ❤️