DEV Community: Carlos Schults

Finding .NET Memory Leaks through Soak Testing

Carlos Schults — Tue, 02 Nov 2021 15:25:57 +0000

As you’re probably aware, C# is a modern, garbage collected language, which means you don’t have to take care of object disposal as you would in languages like C++. However, that doesn’t mean .NET engineers don’t face memory problems, such as memory leaks.

In this post, we’ll walk you through the basics of memory leaks in .NET: what they are, how they occur, and why they matter. We’ll also show you how it’s possible to use techniques and tools already at your disposal — in particular, soak testing — to diagnose and fix memory issues.

Let’s get started.

.NET Garbage Collection 101

To start, we’ll briefly explain how the .NET garbage collection process works. To learn about GC in more detail, refer to Microsoft’s Fundamentals of garbage collection.

An Intro To GC

At a high level, GC works like this.

The analysis starts with an object that’s guaranteed to be alive, which are called “root objects.” For instance, local variables.
From the root object, the GC starts to follow all of its references until it reaches the end of the graph. Every visited object is marked as alive.
The process starts again for the other root objects.
By the end of the process, unvisited objects are deemed unreachable and marked for deletion.
Dead objects are cleaned from memory, but that leaves a lot of “holes” in memory. It’s better if the free areas in memory are kept together, so memory is compacted after the cleaning process.

What we’ve just described is an oversimplified view of GC; we left a lot of details out on purpose. However, you can already see that the process is a lot of work. It takes a while to complete, and thread execution is halted during the process.

Thus, garbage collection, despite being valuable, is costly for your app’s performance. How to get out of this conundrum?

Talking ‘Bout My Generation: Making GC More Efficient

To solve this dilemma, we need to run the collection process as infrequently as we can get away with it. Since every time counts, we’ve got to make sure GC is as efficient as possible.
.NET achieves this by using an approach that understands that not all objects are created equal: it organizes them in different spaces in memory called “generations.”

Generation 0 (G0) is the first generation, where newly created objects go by default. This generation is the one where collection happens more often, based on the assumption that young objects are less likely to have relationships with other objects and are more likely to be eligible for collection as a result.

Generation 1 (G1) is the next generation. Objects from G0 who survive a collection are promoted to G1. Here, collection happens less frequently.

Finally, you have Generation 2 (G2), which is the generation for long lived objects. Here, collection happens very rarely. Objects that make it to this generation are likely to be alive for as long as the application is in use.

GC operates on some assumptions. One of them is that large objects live longer.

If a large object were to be allocated in G0, that would create performance problems. It’d have to progress until G2, and the process of compacting it would be costly.

But that’s not what happens. Large objects (> 85 KB in size) are automatically promoted to an area called the Large Object Heap (LOH.) Unlike the Small Object Heap (SOH), the LOH doesn’t have generations, it’s collected along with G2.

.NET Memory Leaks 101

Having covered the fundamentals of the garbage collection process, let’s move on to memory leaks.

What Are Memory Leaks In .NET and Why Do They Happen?

Wikipedia defines memory leaks as follows:

[...] a memory leak is a type of resource leak that occurs when a computer program incorrectly manages memory allocations in a way that memory which is no longer needed is not released. A memory leak may also happen when an object is stored in memory but cannot be accessed by the running code.

It might seem counterintuitive that .NET can suffer from memory leaks from the definition above. After all, aren’t we talking about a managed environment?

Michael Shpilt argues that are two types of memory leaks in .NET:

managed memory leaks, which happen when you have obsolete objects that don’t get collected.
unmanaged memory leaks, which happen if you allocate unmanaged memory and don’t free it.

Why Should You Care About Memory Leaks?

Memory leaks can give you a severe headache if you don’t have them on your radar.

Suppose you have a number of short-lived objects which are no longer needed, but they can’t be collected, because they’re being referenced by other objects. The fact that they’re unnecessarily kept alive will impact performance: besides the unnecessary memory they use, they will cause collections, which are costly.

Besides degrading performance, severe memory leaks can cause an OutOfMemory exception which will crash your application.

Seeing a Memory Leak In Action

Now, I’ll use soak testing to allow us to see a memory leak in action. This will be a simulated example; I’ll use an application I deliberately modified to introduce a memory problem (described in this blog post).

You can download the modified version from this GitHub repo. Clone it using Git or download everything as a .zip file.

[HttpGet]
        [Route("{id:int}", Name = nameof(GetSingleFood))]
        public ActionResult GetSingleFood(ApiVersion version, int id)
        {
            strings.Add(new String('x', 10 * 1024)); // this is the offending line
            FoodEntity foodItem = _foodRepository.GetSingle(id);

strings in the code above refer to a static property of type ConcurrentBag. To sum it up:

with each request to this particular endpoint—which is the endpoint we’ll be testing during our endurance test—I create a useless big string.
the created string is added to a static collection, which ensures its reference will live on and not be collected.

So, with just two lines of code we ensure the allocation of unnecessary memory and its artificially induced longevity.

Now, I’ll go to my terminal, access the project’s folder and run the following commands:

cd SampleWebApiAspNetCore
dotnet run --configuration Release

After that, I can open my browser at https://localhost:5001/API/v1/Foods to see the API returning an array of foods:

// 
//

{
    "value": [
        {
            "id": 2,
            "name": "Hamburger",
            "type": "Main",
            "calories": 1100,
            "created": "2021-08-24T08:28:57.6607216-03:00",
            "links": [
                {
                    "href": "https://localhost:5001/api/v1/foods/2",
                    "rel":"self",
                    "method": "GET"
                },
                {
                    "href": "https://localhost:5001/api/v1/foods/2",
                    "rel":"delete_food",
                    "method": "DELETE"
                },
                {
                    "href": "https://localhost:5001/api/v1/foods/2",
                    "rel":"create_food",
                    "method": "POST"
                }
            ],
        }
    ]
}

Running Soak Testing With k6

Earlier, I’ve mentioned there are techniques and tools you can use to fight memory problems. It’s now time to meet the technique (soak testing) and the tool (k6) I’ll use in this tutorial.

What is Soak Testing?

There are many kinds of performance testing, and soak testing is one of them.

Unlike load testing, which is targeted at evaluating the app’s performance at a given point in time, soak testing determines how reliable the application is over an extended period of time. That’s why soak testing is also called endurance testing.

How does soak testing work in practice? You basically put a lot of pressure on your system for a few hours. That way, you simulate what would’ve been days of traffic in way less time, potentially uncovering memory leaks, poor configuration, bugs or edge cases that would only surface after an extended period of time.

Getting Started With k6

For our tests, we’ll use k6, which is an open-source tool for scriptable performance testing.

If you want to follow along, please refer to the installation guide. To test your installation, try to follow the minimum test example from k6’s documentation.

Creating The Test Script

k6 is a scripted tool. That means you write a script that details the steps that will be taken during the test. In the case of k6, you write scripts in JavaScript, which is great: JavaScript is the language of the web, so it’s very likely you and your coworkers have at least some familiarity with it.

I created a file called soak.js, with the following content:

import http from 'k6/http';
import { sleep } from 'k6';

export let options = {
  stages: [
    { duration: '2m', target: 200 },
    { duration: '56m', target: 200 },
    { duration: '2m', target: 0 },
  ],
};

const API_BASE_URL = 'https://localhost:5001/API/v1/';

export default function () {
  http.batch([
    ['GET', `${API_BASE_URL}Foods/1/`],
    ['GET', `${API_BASE_URL}Foods/2/`],
    ['GET', `${API_BASE_URL}Foods/3/`],
    ['GET', `${API_BASE_URL}Foods/4/`],
  ]);

  sleep(1);
}

Now, the explanations:

The script starts with the necessary imports;
Then, it declares an object named options, that represents the configuration for the test, detailing three different stages:
- the first will last for 2 minutes, ramping from 0 to 200 VUs (virtual users)
- the second will last for 56 minutes, staying at 200 users
- the last one will go down to zero users again, lasting for 2 minutes
After that, we declare the URL for the calls, and the endpoints of the API that will be hit during the test.

As you can see, with k6 you only need a few lines of code to create a test that will put the application under a heavy pressure for an extended period of time. More specifically, it’s this line:

{ duration: '56m', target: 200 },

that’s responsible for making this test a “soak”, or endurance test. It ensures k6 will hit the API simulating 200 users for almost an hour.

Of course, the configuration we’re using is just an example. In real usage scenarios, you implement a k6 script simulating a realistic user flow with the expected frequent traffic.

Running The Test Script

Now, I’ll run the test. I go to my terminal and run this:

k6 run soak.js --out cloud

Before this, I logged in my k6 Cloud account using my account token. The --out cloud option means that the test itself is run locally, but the results get sent to k6 Cloud, where I can analyze them afterward.

Analyzing The Results

The image above displays the results of the first test. Here’s the overview for the test run:

514676 requests were made.
From those, 37511 resulted in failure (or 7.29%)—these are tracked by the red line.
The average request rate was 687 requests per second, peaking at 797.71—this value is tracked in the chart with the purple line.
The average response time was 34 milliseconds, tracked with the sky blue line.

From the graph above, we can see that, after some 13 minutes of testing, the application simply couldn’t handle the pressure and crashed, and after that point, all requests resulted in failure until the end of the test run.

Now, take a look at the following image:

This is a screenshot from Visual Studio’s profiler. As you can see, even before reaching the 2 minute mark, the program’s memory consumption skyrocketed and two garbage collections had already taken place!

Changing The Application’s Code

“Fixing” the memory issue of this application would be the easiest task ever, since I was the one who introduced it in the first place! Here’s what the correct version of the method should look like:

[HttpGet]
        [Route("{id:int}", Name = nameof(GetSingleFood))]
        public ActionResult GetSingleFood(ApiVersion version, int id)
        {
            FoodEntity foodItem = _foodRepository.GetSingle(id);

            if (foodItem == null)
            {
                return NotFound();
            }

            return Ok(ExpandSingleFoodItem(foodItem, version));
        }

Were this a real situation, though, how could the results from this test run help us in detecting the underlying issue?

Since we’re testing the Foods endpoint on this API, a good place to start the investigation would be the controller action responsible for that specific GET endpoint.

In our example, we would quickly find the code responsible for creating a lot of memory leakage. In a real scenario, that would be the starting point of a debugging session. With the help of a profiler, you’d be more likely to find the culprit.

Running Soak Testing With k6, Again

After “fixing” the code, I’ll now run the tests again, so we can compare the results of both test runs.

Here are the results:

This time 2473732 requests were made.
From those, only 764 resulted in failure, which corresponds to 0.03%, as you can see in the red line in the chart.
This time, the purple line, tracking the successful requests, remains stable for virtually the entire session, just taking a deep dive about midway through the tests, but then making a full recovery.
At the same spot, you can see the request time—tracked by the sky blue line—skyrocketed.
The average request rate was 143 requests per second, peaking at 800.
The average response time was 21 milliseconds.

Conclusion

.NET isn’t immune to memory issues, among which memory leaks are probably the most insidious ones. Memory leaks are often silent; if you don’t watch out for them, they can sneak in your application and corrode its performance until it’s too late.

The good news is that it’s possible to fight back. By understanding the fundamentals of GC and memory management, you can write code in such a way that minimizes the likelihood of problems.

Education can only take you so far, though. At some point, you have to leverage the tools and techniques available to you.That includes profilers, such as CodeTrack.

Another great tool for your arsenal is k6. Although more widely known as a load testing tool, k6 also enables you to perform soak/endurance testing to your applications, which can help you uncover memory issues.

Intro to Testing ASP.NET APIs with k6 - When Unit Tests Meet Load Testing

Carlos Schults — Thu, 06 May 2021 08:41:29 +0000

Software organizations are always on the hunt for ways to satisfy and delight their customers. Application performance is certainly a part of that, and ensuring good performance requires testing. Load testing is one of the types of performance testing and, in this post, you'll see how to use k6 to perform load testing on a .NET RESTful API.

By the end of the post, you’ll have a more solid knowledge of load testing, why it’s important and how it’s performed. You’ll also have practical experience performing load testing with k6, and you’ll understand some of the unique ways in which this tool sets itself apart from competitors. With all of that being said, let’s get started!

Load Testing: The What, Why, and How

Before getting our hands dirty, let’s start with some fundamentals. What is load testing?

Defining Load Testing

Load testing is one of the many types of web performance testing. It consists of verifying how an application behaves under a heavy load—hence the name. With the help of load testing, you can assess the response of your app to increasing demand and then improve it accordingly.

Why Should You Care About Load Testing?

Why is load testing important? Well, as we’ve explained, when it comes to online services, it’s essential to provide a great user experience to your customers, since the competition is only a few clicks away. And performance is a crucial part of the overall user experience of a website.

But why use load testing specifically? In short, you should adopt load testing when you're concerned about the availability and scalability of your website. Even though the frontend is responsible for relevant performance issues, you should focus on the backend to ensure your performance and reliability goals when scaling your site. And load testing is exactly the tool you need to verify how your website reacts when put under load.

How Do You Perform Load Testing

How is load testing actually performed? As you’ll see, that can vary, even because load testing tools can come in different categories. However, the gist of it is that load testing simulates multiple users sending multiple requests to the app, concurrently. You choose the number of users and requests based on the estimated load you expect your app to be able to handle.

Rolling Up Your Sleeves: Let's Load Test An ASP.NET API

After our brief introduction to load testing, it’s time to learn how to do it in practice.

Obtaining The Requirements

The application we’ll be testing throughout this tutorial is a RESTful API written in ASP.Net Core. To be able to run the application, you’ll need the .NET SDK version 5.0 or later.

Presenting the Sample Application

For this tutorial, our sample application will be the sample API developed using ASP.Net Core, demonstrated in this tutorial.

Download or clone the complete application from this GitHub repository.

As soon as you get the code, cd into the project’s directory and run the following command:



dotnet run

Now, open your browser and go to localhost:<port>/api/TodoItems to see the resources that are returned.

Let's Start Testing

Here’s where the fun begins: we’ve gotten ourselves an API, so let’s test it.

One thing to keep in mind when doing performance testing is that, as we’ve mentioned, it comes in different types. More specifically, we can split load testing tools into two main camps: scriptable and non-scriptable ones.

Scriptable tools are those that allow you to use a full-fledged scripting language to write your tests. Such tools might offer a steeper learning curve and, in some situations, consume slightly more resources. However, the upside is that they enable users to write flexible and more complex tests.

On the other end of the spectrum, you have non-scriptable tools. They tend to be simpler and consume fewer resources. But on the other hand, they're way less powerful when it comes to the capabilities they offer.

This post is about k6, which is a powerful scriptable tool. But we won’t jump into k6 right away. Instead, we’ll start by testing our API with a non-scriptable tool. That way, you’ll be able to appreciate the differences between those two categories of tools.

Testing a GET Endpoint With a Non-Scriptable Tool

As I’ve mentioned earlier, it’s possible to divide load testing tools into two broad groups: scriptable and non-scriptable ones. So, to appreciate the qualities of the former group, you need at least a little bit of experience with the latter. That’s why before we get to k6 we’ll take a brief detour to show you how testing an endpoint with a non-scriptable tool looks like.

The non-scriptable tool we’ll use is wrk. Feel free to follow along with this part of the tutorial but, since it’s a detour, I won’t provide detailed instructions for it. With that out of the way, let’s begin.

After installing wrk, I’ll run my sample project again by executing:



dotnet run

Then, I’ll use wrk to test the GET endpoint for the TodoItem resource:



./wrk -t12 -c400 -d30s https://localhost:5001/api/TodoItems/

This is exactly the same "basic usage" example taken from the project's README. It runs a 30 seconds test that uses 12 threads keeping 400 connections open. What do the results look like? See in the following image:

Let's unpack that. After a short text summarizing the executed test (30 seconds, 12 threads, 400 connections) we see statistics regarding the latency and requests per second. For each of those we see, among other info, the average value (1), the standard deviation (2), the maximum value (3).

We also see the number of requests that were performed, in how much time, and the total volume of data that was read (5). Finally, we see the total amount of requests per second and the volume of data transferred.

Testing a Get Endpoint with k6

Now it’s time to repeat the previous test, this time using k6.

Installing

Let’s start by installing the tool. If you use macOS, install k6 is a breeze using homebrew:



brew install k6
```

Users of Debian, Ubuntu or other Debian-based distros should run the following commands:
```bash
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 379CE192D401AB61
echo "deb https://dl.bintray.com/loadimpact/deb stable main" | sudo tee -a /etc/apt/sources.list
sudo apt-get update
sudo apt-get install k6
```

If you’re on Windows, you can [download the.msi installer](https://k6.io/docs/getting-started/installation/#windows-msi-installer). Users of other platforms, take a look at the [k6 installation guide](https://k6.io/docs/getting-started/installation/).

After the installation is complete, regardless of OS, you should be able to run {% raw %}`k6 version` to see the current version number and verify the installation was a success.

#### Writing and Running Your First Test Script

To perform k6 tests, you’ll need to create a test script. Test scripts in k6 are written in JavaScript, which is a perfect choice, considering most web developers will already be familiar with the language—as opposed to something like Python, Ruby, or a custom DSL.

Using your favorite editor, create a file called **script.js**. I’ll use Visual Studio Code:

```
code script.js
```

Then, add the following code to the new file:

```javascript
import http from 'k6/http';

export default function () {
  http.get('http://localhost:5000/Api/TodoItems');
}
```

Of course, make sure that you’ve started the API. Also, change the port number if necessary.

After saving the test script, you can test it by running:

```
k6 run script.js
```

The results should look something like this:
```
          /\      |‾‾| /‾‾/   /‾‾/   
     /\  /  \     |  |/  /   /  /    
    /  \/    \    |     (   /   ‾‾\  
   /          \   |  |\  \ |  (‾)  | 
  / __________ \  |__| \__\ \_____/ .io

  execution: local
     script: script.js
     output: -

  scenarios: (100.00%) 1 scenario, 1 max VUs, 10m30s max duration (incl. graceful stop):
           * default: 1 iterations for each of 1 VUs (maxDuration: 10m0s, gracefulStop: 30s)


running (00m00.1s), 0/1 VUs, 1 complete and 0 interrupted iterations
default ✓ [ 100% ] 1 VUs  00m00.1s/10m0s  1/1 iters, 1 per VU

     data_received..................: 1.8 kB 33 kB/s
     data_sent......................: 816 B  15 kB/s
     http_req_blocked...............: avg=22ms    min=5ms     med=22ms    max=39ms    p(90)=35.6ms   p(95)=37.3ms 
     http_req_connecting............: avg=499.9µs min=0s      med=499.9µs max=999.8µs p(90)=899.82µs p(95)=949.8µs
     http_req_duration..............: avg=3.99ms  min=999.7µs med=3.99ms  max=6.99ms  p(90)=6.39ms   p(95)=6.69ms 
       { expected_response:true }...: avg=3.99ms  min=999.7µs med=3.99ms  max=6.99ms  p(90)=6.39ms   p(95)=6.69ms 
     http_req_failed................: 0.00%  ✓ 0 ✗ 2
     http_req_receiving.............: avg=0s      min=0s      med=0s      max=0s      p(90)=0s       p(95)=0s     
     http_req_sending...............: avg=0s      min=0s      med=0s      max=0s      p(90)=0s       p(95)=0s     
     http_req_tls_handshaking.......: avg=19ms    min=0s      med=19ms    max=38ms    p(90)=34.2ms   p(95)=36.1ms 
     http_req_waiting...............: avg=3.99ms  min=999.7µs med=3.99ms  max=6.99ms  p(90)=6.39ms   p(95)=6.69ms 
     http_reqs......................: 2      37.030248/s
     iteration_duration.............: avg=54ms    min=54ms    med=54ms    max=54ms    p(90)=54ms     p(95)=54ms   
     iterations.....................: 1      18.515124/s
```
#### Improving The Test

You’ve just written and run your first k6 test script. Congrats! However, we need to beef it up a little bit so it compares with the test we’ve done using wrk. Our k6 test is, at the moment, a simple request. It sends a get request to the specified URL and then waits for 1 second.

When running a test with k6, you can pass parameters to the test. Run the following command:

```
k6 --vus 400 --duration 30s run script.js
```

You can see we’ve used two parameters here.  The **--duration** option defines that the test should run for 30 seconds.

The **--vus** option stands for [virtual users](https://k6.io/docs/misc/glossary/#virtual-users), which are used to send the requests to the system under test.To understand the concept of VUs, think of them like parallel infinite loops. If you go back to our script, you'll see there is a `default` function. Scripts in k6 need to have that `default` function as a requirement. That's the entry point for the test script. The code inside such a function is what the VUs execute; it gets executed over and over again, for the whole duration of the tests.

Besides the `default` function, you can have additional code. However, code outside `default` is run only once per VU. You use that code for initialization duties, such as importing different modules or loading something from your filesystem.

That being said, let’s see what the result of the command’s execution looks like:

```
          /\      |‾‾| /‾‾/   /‾‾/   
     /\  /  \     |  |/  /   /  /    
    /  \/    \    |     (   /   ‾‾\  
   /          \   |  |\  \ |  (‾)  | 
  / __________ \  |__| \__\ \_____/ .io

  execution: local
     script: script.js
     output: -

  scenarios: (100.00%) 1 scenario, 400 max VUs, 1m0s max duration (incl. graceful stop):
           * default: 400 looping VUs for 30s (gracefulStop: 30s)


running (0m00.9s), 400/400 VUs, 8 complete and 0 interrupted iterations
default   [   3% ] 400 VUs  00.9s/30s

...

running (0m30.0s), 000/400 VUs, 202254 complete and 0 interrupted iterations
default ✓ [ 100% ] 400 VUs  30s

     data_received..................: 60 MB  2.0 MB/s
     data_sent......................: 36 MB  1.2 MB/s
     http_req_blocked...............: avg=1.66ms   min=0s    med=0s      max=2.68s p(90)=0s      p(95)=0s     
     http_req_connecting............: avg=213.12µs min=0s    med=0s      max=1.32s p(90)=0s      p(95)=0s     
     http_req_duration..............: avg=27.65ms  min=0s    med=24ms    max=1.96s p(90)=40ms    p(95)=54.99ms
       { expected_response:true }...: avg=27.65ms  min=0s    med=24ms    max=1.96s p(90)=40ms    p(95)=54.99ms
     http_req_failed................: 0.00%  ✓ 0     ✗ 404508
     http_req_receiving.............: avg=703.91µs min=0s    med=0s      max=1.35s p(90)=0s      p(95)=999.1µs
     http_req_sending...............: avg=167.24µs min=0s    med=0s      max=1.39s p(90)=0s      p(95)=0s     
     http_req_tls_handshaking.......: avg=1.3ms    min=0s    med=0s      max=2.65s p(90)=0s      p(95)=0s     
     http_req_waiting...............: avg=26.77ms  min=0s    med=24ms    max=1.46s p(90)=39.99ms p(95)=51ms   
     http_reqs......................: 404508 13474.167948/s
     iteration_duration.............: avg=59.23ms  min=998µs med=49.99ms max=2.84s p(90)=78.99ms p(95)=98.99ms
     iterations.....................: 202254 6737.083974/s
     vus............................: 380    min=380 max=400 
     vus_max........................: 400    min=400 max=400 
```
What you see above is what the results look like for me after running the latest version of the command. As you can see, there’s a lot going on here. Let’s walk through some of the main pieces of information:

*   `data_received`: that’s the total amount of data received (109 MB) at a rate of 3.4 MB per second;
*   `http_req_duration`: information about the duration of the HTTP requests performed, including the average, median, minimum and maximum values;
*   `http_req_failed`: the rate of requests that failed;
*   `http_req_waiting`: time spent waiting for the server’s response.

To learn more about the metrics displayed in k6’s results, you can [refer to the documentation](https://k6.io/docs/using-k6/metrics/).

### Testing With a Higher Load

We’ll now perform a different test, sending a PUT request instead of a GET one. We’ll also use an additional option of k6 that allows us to simulate an increase in the load on our application.

For that, I’ll create a new test script and name it script2.js. Here’s what its code looks like:

```javascript
import http from 'k6/http';

const url = 'https://localhost:5001/api/TodoItems/';

export let options = {
  stages: [
    { duration: '15s', target: 100 },
    { duration: '30s', target: 500 },
    { duration: '15s', target: 0 },
  ],
};  

export default function () {

  const headers = { 'Accept': '*/*', 'Content-Type': 'application/json', 'Host': 'localhost:5001' };
  const data = {
      name: 'Send e-mail',
      isComplete: false      
  };

  http.put(url, JSON.stringify(data), { headers: headers });
}
```

As you can see, in the script above we’re sending a PUT request to our API, trying to add a new TodoItem resource. Before that, though, we create an `options` variable in which we define three stages for our test. In the first one, which lasts for 15 seconds, k6 will increase the number of VUs from 0 to 100. Then, it will ramp up to 500 for another 30seconds, before ramping down to 0 in the last 15 seconds.

Here’s what the results look like:

```

          /\      |‾‾| /‾‾/   /‾‾/   
     /\  /  \     |  |/  /   /  /    
    /  \/    \    |     (   /   ‾‾\  
   /          \   |  |\  \ |  (‾)  | 
  / __________ \  |__| \__\ \_____/ .io

  execution: local
     script: script2.js
     output: -

  scenarios: (100.00%) 1 scenario, 500 max VUs, 1m30s max duration (incl. graceful stop):
           * default: Up to 500 looping VUs for 1m0s over 3 stages (gracefulRampDown: 30s, gracefulStop: 30s)


running (0m00.8s), 006/500 VUs, 6743 complete and 0 interrupted iterations
default   [   1% ] 006/500 VUs  0m00.8s/1m00.0s

...

running (1m00.0s), 000/500 VUs, 671850 complete and 0 interrupted iterations
default ✓ [ 100% ] 000/500 VUs  1m0s

     data_received..............: 32 MB   537 kB/s
     data_sent..................: 84 MB   1.4 MB/s
     http_req_blocked...........: avg=73.54µs min=0s med=0s      max=530.01ms p(90)=0s      p(95)=0s     
     http_req_connecting........: avg=16.56µs min=0s med=0s      max=212ms    p(90)=0s      p(95)=0s     
     http_req_duration..........: avg=19.56ms min=0s med=14ms    max=307.99ms p(90)=41.99ms p(95)=55ms   
     http_req_failed............: 100.00% ✓ 671850 ✗ 0    
     http_req_receiving.........: avg=34.6µs  min=0s med=0s      max=229.99ms p(90)=0s      p(95)=0s     
     http_req_sending...........: avg=85.33µs min=0s med=0s      max=157ms    p(90)=0s      p(95)=999µs  
     http_req_tls_handshaking...: avg=56.37µs min=0s med=0s      max=364ms    p(90)=0s      p(95)=0s     
     http_req_waiting...........: avg=19.44ms min=0s med=14ms    max=272.99ms p(90)=41ms    p(95)=54.99ms
     http_reqs..................: 671850  11196.657731/s
     iteration_duration.........: avg=20.07ms min=0s med=14.99ms max=632.99ms p(90)=42.99ms p(95)=56.99ms
     iterations.................: 671850  11196.657731/s
     vus........................: 1       min=1    max=500
     vus_max....................: 500     min=500  max=500
```

## Here's Where a Scriptable Tool Shines: Testing a User Scenario

Up until now, we’ve been testing using fairly simple scripts. You might be wondering what’s all the fuss about scriptable tools after all. Well, here’s the thing: a non-scriptable tool might be enough if you only need to perform basic requests.

However, there are scenarios in which a non-scriptable tool can really make a difference. One of those is where you need to test a user scenario. In other words, you might need to verify a real usage workflow in your app. You simply can’t do that with a non-scriptable tool.

Up until now, we've been testing an endpoint in isolation. Often, when monitoring your services, you might have found a REST API underperforming at a certain load level and want to simulate this behavior again. A non-scriptable tool is often enough for this type of testing.

Such a type of verification, however, isn’t the most realistic. Why? Well, users don’t behave like that. They don’t do things in a completely isolated way. In the real world, your application continuously responds to a flow of real-user interactions, e.g., visit a page, log in, list items, purchase some items, and so on.

Testing real-world scenarios allow you to validate critical business logic or the most frequent user flows. If you want to mimic this type of interaction, a scriptable tool makes this job possible and more manageable.

The following script is a simple example of how testing a user workflow could look like. Using k6, we hit the GET endpoint, retrieving the existing resources. We then get the id of the first object, and use that to send another get  request, obtaining that resource. After that, we set the value of the `completed` attribute and send a PUT request to update the resource on the server. Finally, we use a check to verify whether the response has the expected status code.

```javascript
import http from 'k6/http';
import { check,  sleep } from 'k6';

export let options = {
    vus: 30,
    duration: '40s'
}

export default function () {
  let url = 'https://localhost:5001/api/TodoItems';

  // getting all todo items
  let response = http.get(url);

  // parsing the response body
  let obj = JSON.parse(response.body);

  // retrieving the id from the first resource
  let idFirstItem = obj[0].id;

  // retrieving the resource with the selected id
  response = http.get(`${url}/${idFirstItem}`);
  let item = JSON.parse(response.body);

  // setting the item as complete
  item.complete = true;

  // updating the item
  const headers = { 'Content-Type': 'application/json' };
  let res = http.put(`${url}/${idFirstItem}`, JSON.stringify(item), { headers: headers });

  // checking the response has the 204 (no content) status code
  check(res, {
    'is status 204': (r) => r.status === 204,
  });

  // random think time between 0 and 5 seconds
  sleep(Math.random() * 5);
}
```

In the previous example, the test runs the same user flow continuously until the test duration ends. But with k6, and you can also simulate different user flows with varying load patterns to run simultaneously. This flexibility allows you to test your application more precisely.

For example, you could run 50 virtual users doing some actions and 100 virtual users doing something different for a different period while generating a constant request rate to one endpoint. 

```javascript
export const options = {

  scenarios: {
    scenario1: {
      executor: 'constant-vus',
      duration: "1m",
      vus: 50,
      exec: "userFlowA"
    },

    scenario2: {
      executor: 'ramping-vus',
      stages: [
        { duration: '1m', target: 100 },
        { duration: '30s', target: 0 },
      ],
      exec: "userFlowB"
    },

    scenario3: {
      executor: 'constant-arrival-rate',
      rate: 200,
      timeUnit: '1s',
      duration: '1m',
      preAllocatedVUs: 50,
      exec: "hammerEndpointA"
    }
  }
}
```

To learn more about configuring advanced traffic patterns, check out the [Scenarios API](https://k6.io/docs/using-k6/scenarios/).


## Here's Where k6 Shines: Setting Performance Goals

The ability to set performance goals for your application is an aspect that differentiates k6 from other load testing tools. With k6, you can verify the performance of your app against expected baselines in a way that’s [not that different from assertions in unit testing](https://k6.io/unit-testing-for-performance/).

How does that work in practice? There are two main options that k6 provides: checks and thresholds. 

[Checks ](https://k6.io/docs/using-k6/checks/) allow you to set expectations in your script and verify those expectations automatically. We’ve used one check in our previous example. Even if the checks fail, the execution of the script doesn’t stop. 

[Thresholds](https://k6.io/docs/using-k6/thresholds/), on the other hand, do interrupt the script’s execution. They’re criteria you can use to fail your load test script when the system under test doesn’t meet the expectations.

Here are some examples of thresholds you could use:

- The system generates at the most 1% errors.
- Response time for 99% of requests should be below 400ms.
- Response time for a specific endpoint must always be below 300ms.

Since k6 allows you to define custom metrics, you can always define thresholds on them as well. Here’s an example of what thresholds look like in an actual script:

```javascript
export let options = {
  thresholds: {
    http_req_failed: ['rate<0.01'],   // http errors should be less than 1% 
    http_req_duration: ['p(99)<400'], // 99% of requests should be below 400ms
  },
};
```

## k6: When Unit Tests Meet Load Testing

In this post, you’ve learned how to get started with load testing using k6. We’ve covered some fundamentals on load testing: you’ve learned the definition of this technique, and the differences between scriptable and non-scriptable load testing tools.

k6 belongs to the latter group. It allows developers to author test scenarios in JavaScript—a language they’re likely to already know. Because of that, k6 is a powerful tool, enabling the performing not only of HTTP benchmarking, but also the verification of realistic usage scenarios.

Perhaps the greatest differentiator of k6 is that it bridges the gap between unit testing and load testing. Developers can use the same workflow they’ve been using for years: creating tests, adding verifications with pass/fail criteria, adding those to the CI/CD pipeline, and then be notified when the tests do fail.

Mutation Testing: What It Is and How It Makes Code Coverage Matter

Carlos Schults — Thu, 23 Jul 2020 21:45:43 +0000

Photo by Christina @ wocintechchat.com on Unsplash

Editorial note: This post was originally written for the NCrunch blog. You can check out the original here, at their site.

I've been fascinated by mutation testing since I found out about it. I thought I'd finally found the answer to so many problems I had when writing tests. With mutation testing, I now had a way to really trust my tests. At last, code coverage had meaning again.

Then, I was dumbstruck as I realized that very few developers shared my interest in mutation testing. In fact, I dare say that most developers haven't even heard about it. And that's a shame because they—and we, as an industry—are missing out on so many benefits.

So, this post is my humble attempt to remedy the situation. I'll start by explaining the current dilemmas developers face regarding the reliability of the tests they write. Then, I'll proceed to show you how mutation testing is the answer to these dilemmas. I'll explain what it is, how it can make your tests more trustworthy, and how it can turn code coverage into the valuable metric it should be.

The Problem: Trustworthiness of Tests

When learning about unit tests—or automated tests in general—most people will ask the same or a similar question: How do I know my tests are right? That's a legitimate concern. If your tests aren't trustworthy, then you might be better off with no tests at all.

So what's the answer? How do people deal with the problem of test trustworthiness without relying on mutation testing?

Making Tests Reliable Without Mutation Testing

There are techniques developers employ to improve the reliability of their tests, and we'll briefly cover some of them in this section. If you're experienced with unit testing, you're probably familiar with them. Let's dive in.

Keep Your Tests Simple

The first technique we'll cover here to improve the reliability of your tests is just to keep them simple. And by "simple" I mean with less cyclomatic complexity. The lower the cyclomatic complexity of a given piece of code, the likelier it is that it actually does what you think it does. Simple code is easier to reason about, which is a property you definitely want your unit tests to have.

Keep test code simple to the point of being obvious. That means, for instance, avoiding loops or decision structures. Also, avoid doing anything fancy to compute the expected result (more on that in the next section). Hard-code it instead.

Don't Duplicate Implementation Code

Let's say you're doing the Roman numerals kata. Resist the temptation to automatically generate the expected values ("I" for 1, "II" for 2, and so on). Instead, hard-code the values. If the repetition really bothers you and your test framework allows it, use parametrized tests.

Why would that be a problem? Simple: The fancier your test code gets, the more likely it's duplicating production code. If that's the case, you might be unlucky enough to find yourself in the situation where your production code is wrong (it doesn't solve the problem as it's supposed to do) but the tests pass. That's one of the worst possible scenarios. It's even worse than having no tests at all.

Ensure You See the Test Failing

Ensure each test fails at least once before it passes. If you see the test failing when you think it should be failing and vice versa, that's a sign you're moving in the right direction. It doesn't guarantee anything, but it decreases the likelihood the test is passing due to a coincidence.

Here's how you'd do it. As soon as you get to the green phase, damage the implementation code in such a way that one or more tests should fail. You could invert conditionals, replace strings or numeric literals with random values, or even delete an if-statement. If you manage to sabotage production code and get away with it, that's not a good sign. Your test suite is either wrong or incomplete. In a sense, you're testing the tests.

Developers who employ TDD (test-driven development) kind of already do that by definition. Since you write a failing test and then proceed to make it pass, you're seeing the test fail. Of course, the test should fail in the expected manner. Meaning that if you're performing an assertion, the test should fail due to an assertion failure and not, say, because the method under test throws an exception. Yes, this is better than nothing, but it still might not be enough. Since a unit test represents a single-use case, it's totally possible to introduce a defect to production code in such a way that this particular test still passes.

We Must Do Better: Enter Mutation Testing

So you've just applied the technique described in the last section. Good! Not perfect, though. Here comes a problem. You can't just insert a lot of defects and run the tests, because you wouldn't be able to identify which defect was responsible for the tests failing. The correct way to do it is to insert a single deliberate defect, run all the tests, verify their result, and then roll back the change. After that, you can introduce another mistake, run all the tests again, verify the result, roll back the change...rinse and repeat. Needless to say, such an approach is extremely slow, tedious, and error-prone.

That's where mutation testing comes in.

What's Mutation Testing, Anyway?

Mutation testing is nothing more, nothing less, than automating the whole "sabotaging production code and running tests to see if they fail" process you just saw. To use mutation testing, you need a mutation testing framework. The framework will alter production code, introducing defects that are called "mutations." For each mutation introduced, the framework will again run the suite of unit tests. If all tests pass, we say the mutation survived. That's a bad thing. It means that either your suite is lacking tests or the existing ones are wrong.

If, on the other hand, one or more tests fail, that means the mutation was killed, which is a good thing. The framework will repeat that process until it's tested the relevant portion of the codebase. When it's all done you can check the results, which will contain the number of mutations introduced, as well as the ratio of surviving vs. killed mutants.

Mutation Testing Makes Code Coverage Better

One of the most controversial topics in the unit testing world is the argument about code coverage. Some developers say that getting to full coverage is essential; others will argue that's a useless metric. Who's right?

First of all, you have to understand that this issue isn't black and white. As is the case with pretty much everything in software, there's some nuance. Of course code coverage isn't useless. Knowing that your codebase has, say, 10% of test coverage is definitely a useful piece of data. Such a coverage is way too low: Seeing the green bar won't offer them any confidence. That's not to say that having 100% coverage is necessarily a good thing in itself. You could have tests that don't have assertions, for instance. Yes, this is a contrived example, but something like that could (and sometimes does) happen.

A more common occurrence would be to just have tests that don't exercise enough paths in the software. In short: Low code coverage is definitely a bad thing, but high (or full) code coverage is not necessarily a good thing since it says nothing about the quality of the tests in the suite.

Since mutation testing does verify the quality of the test suite, it's the missing piece of the puzzle. If your codebase has a high code coverage and the results of mutation tests show that most or all mutations introduced are being killed, then smile! You probably have a great test suite in place!

Embrace Mutation Testing Today

In today's post, we talked about the problem of test trustworthiness, then proceeded to review some techniques and guidelines you can use to overcome that challenge. Finally, we saw how mutation testing is the superior approach to solving that problem.

Here's the thing: The techniques we covered are good guidelines to follow when writing unit tests. Your tests will benefit from abiding by them, whether you employ mutation testing or not. But guidelines can only take you so far. They depend too much on human willpower and discipline, and we all have limited amounts of those. In order to take the quality of your tests to the next level, you need to embrace automation.

It takes just a few minutes of googling to find a mutation testing tool for your preferred tech stack. Do that today, and stop missing out on the benefits that mutation testing can provide you and your team!

C# Unit Testing: Getting Started With TDD

Carlos Schults — Mon, 13 Jul 2020 22:19:17 +0000

This post is part of a series. See all the articles in the series.

Today I bring you another post to help you get started with C# unit testing. I've already covered the basics of unit tests, explaining what they are and why they're so important. After that, I showed you how to get started with unit testing with a practical example. Today, we go one step further, exploring the TDD methodology.

You've probably have heard of TDD, but you might be confused as to what it is. This isn't your fault, by the way. There's quite a lot of misconception surrounding this acronym. Some people even use it interchangeably with unit testing. In this post, you're going to learn why they're wrong, and more.

We start the post with a brief definition of TDD. You'll learn not only that TDD stands for Test-Driven Development, but also that it's not a testing technique, despite the name. After that, I explain what TDD is and what its benefits are.

After the "what" and "why" are both out of our way, we'll be ready to have some fun. I'm going to show you, in practice, how to get started with TDD, developing a solution for a famous programming exercise. Sounds good? Then, let's dig in.

C# Unit Testing & TDD Basics

I've mentioned earlier that TDD is not a testing technique. What is it, though? And how it's related to C# unit testing (or unit testing in general, for that matter?)

Defining TDD

As you've seen, TDD stands for Test-Driven Development. It is a technique or methodology of software development that uses unit tests to drive the development of the application.

Instead of doing the more intuitive thing, which would be writing unit tests after the production code, the TDD methodology states that you should start by writing a failing unit test. Then you write the production code, but only what's necessary to make the test pass.

I guess you're now wondering at least two things:

How does that work in practice?
Why write code in such a weird way?

That's what we're going to see next.

The TDD Phases

Test-driven development relies on the repetition of an incredibly short cycle. This cycle is composed of three phases:

First, you write a test that represents a specific requirement of the functionality you're trying to implement.
You then make the test pass, writing the minimum amount of production code you can get away with.
If necessary, you refactor the code to eliminate duplication or other problems.

Since the functionality doesn't exist yet, the test you write in step #1 will fail. That is, in languages such as Python or Ruby. In the case of statically typed languages such as Java or C#, the code won't even compile. For our purposes, code not compiling counts as a test failure.

In step #2, you have to make the test pass, but nothing beyond that. What I mean is that your goal here isn't to solve the problem, at least not yet. Instead, your only job is to make the test pass, writing the least possible amount of code. Cheating—for instance, returning a hard-coded value—is not only OK but encouraged, as you'll soon see.

Finally, the third phase is the only one that allows you to write production code without having to create a failing test first. But you can't create new classes or functions; you can only refactor the code you wrote in the previous step, to make it more readable, to eliminate duplication, or to solve another problem. And, of course, the test should still pass.

People often use refer to the TDD as "red-green-refactor" because most unit testing tools use red to denote failing tests and green for passing tests.

Why Use TDD?

The hard thing to understand when getting started with TDD isn't the how. The "how" is trivial: write a test, make it pass, maybe refactor, rinse, repeat. The troubling part is the "why." Why develop software in such a non-intuitive way?

I'll talk more of the TDD philosophy in future articles. In a nutshell, applying TDD ensures you'll have testable code from the beginning. It will encourage you to design your code in a simple and modular way.

But perhaps, the main advantage of TDD is increasing the developer's confidence on their code. By developing one tiny step at a time, you'll never be able to get a lot wrong, since you're doing too little. Knowing that you're only one failing test away from having working code is reassuring.

C# Unit Testing & TDD: The Hands-On Guide To Get Started

I've covered how to get started with C# unit testing in the past. I've also covered the required tools and how to get started. However, I won't assume you've read those articles. Instead, I'll cover everything from scratch. So, you'll be able to follow the tutorial even if you have zero experience with unit testing.

Our Problem: The String Calculator Kata

For our tutorial, we'll write a solution for Roy Osherov's String Calculator kata. A coding kata is a programming exercise, meant to allow developers to practice fundamental agile software-engineering practices, such as refactoring, and—you've guessed it—TDD.

For simplicity's sake, I'll ignore some of the kata's requirements. What follows are the requirements we're going to use:

We're going to create a class called StringCalculator, with a single static method with the signature static int Add(string numbers);
The method takes a string representing numbers separated by a comma, and return their sum.
If we pass an empty string, the method should return zero.
Passing a single number should result in the number itself.
If we pass negative numbers, the method should throw an ArgumentException, with the message "Negative numbers not allowed:" followed by the negatives that were specified.
The method should ignore numbers greater than 1000 should. So, "1,2,1000" should result in 1003, but "1,2,1001" should result in 3.

Creating The Production Project

For this tutorial, I'll be using the community edition of Visual Studio 2019. If you don't already have it, you can download it and install it from free.

Open VS and click on "Create a new project," like in the following image:

In the opened window, choose Class Library (.NET Core) as the template for the new project. Then, click on "Next":

The next screen simply asks you for a name for the project and the solution. I chose" StringCalculatorKata" for both the project and the solution. You'll also have to provide a location for saving the project files. When you're done, just click "Create."

If everything went well, you should see the default class open for you in Visual Studio. Go to Solution Explorer and delete that class; we're not going to need it.

Creating The Test Project

Now, it's time to create the test project. We could this in two ways: creating a regular "Class Library" project and then adding the necessary dependencies to it, or creating a unit test project right away. We'll go with the latter since it makes the whole thing easier.

You know the drill: right-click the solution, go to "Add," then "New Project…". Then, choose the template "NUnit Test Project (.NET Core)."

Then, you'll be required to provide a name and a location for the project. I like to follow the naming convention of naming the test project after the production project, with a ".Test" added. So, I pick "StringCalculatorKata.Test." Finish the creation of the project.

If everything went right, you should now see a new class that looks like this:

public class Tests
{
    [SetUp]
    public void Setup()
    {
    }

    [Test]
    public void Test1()
    {
        Assert.Pass();
    }
}

Let's do a few things. First, get rid of the Setup() method. We won't need it. Then, add a new method with the code below:

[Test]
public void Test2()
{
    Assert.Fail();
}

So, we now have two tests, one that should pass and another that should fail. Let's run them to see if they're working correctly. Go to the "Run" menu and click on "Run All Tests."

Now, open the Test Explorer window (View -> Test Explorer). It should look like this:

It looks like everything is working fine! But before we start doing our coding kata exercise, there are two final steps we need to take. First, let's rename the test class. Go to the solution explorer, expand the unit test project, and delete its default test class. Then, right-click the test project, go to "Add," then "New class…" and add a new class called "StringCalculatorKata." Alternatively, you can rename the existing class.

The second thing we have to do is to ensure our test project can see our production project. To solve that, we're adding a reference.

Go to the solution explorer again, right-click the test project, then go to "Add" and click on "Reference…".

In the new window, select "Projects" on the left panel, and then select the StringCalculatorKata project, which should be the only one available:

Then, you just have to click on "OK," and now you're ready to go.

Starting Our Coding Kata

Now, we're ready to write our first failing test. So, open the StringCalculatorTest class and add the following method to it:

[Test]
public void Add_EmptyStringAsParam_ReturnsZero()
{
    Assert.AreEqual(0, StringCalculator.Add(string.Empty));
}

In our first test case, we test the simplest possible scenario. That is, we call the Add method passing an empty string, which, according to the requirements you saw before, should result in 0. Of course, neither the Add method nor the StringCalculator class exists, so our code doesn't even compile. Well, congratulations! You've successfully performed the first step in the red-green-refactor cycle by writing a failing test! Remember: in statically-typed languages such as C#, failure to compile counts as a failed test.

So, our first step is to get rid of the compilation error. If you hover over "StringCalculator," you should see a little pop-up explaining the error and offering possible fixes:

Click on "Show potential fixes" and then on "Generate new type…". You should then see a window prompting you for the details and location of the new type. Change the "access" to "public" and the location to the production project, which is "StringCalculatorKata." The window should look like this:

Click on "OK." Now, if you open solution explorer and expand the StringCalculatorKata project, you should see the StringCalculator.cs class lurking around there. Cool.

However, our code still doesn't compile. And that's because, despite creating the production class, we didn't add the Add method to it. So, let's do it in the same way we did with the class.

Hover over the "Add" word until the help pop-up shows up with the message "' StringCalculator' does not contain a definition for 'Add.'" Click on Show potential fixes, and then click on "Generate method 'StringCalculator.Add'."

You'll see that the production class now contains a method called Add, with double as a return type. We want the method to return int, so let's change that. Let's also change the parameter name to "numbers" to match the coding kata's requirements. At this point, your complete StringCalculator class should look like this:

public class StringCalculator
{
    public static int Add(string numbers)
    {
        throw new NotImplementedException();
    }
}

Now your code should compile. Run the test again, and you'll see that it fails, with a message like this:

Add_EmptyStringAsParam_ReturnsZero
   Source: StringCalculatorTest.cs line 8
   Duration: 43 ms

  Message: 
    System.NotImplementedException : The method or operation is not implemented.
  Stack Trace: 
    StringCalculator.Add(String numbers) line 9
    StringCalculatorTest.Add_EmptyStringAsParam_ReturnsZero() line 10

We have a truly failing test. Are we ready to write production code? Not so fast. Sure, our test fails, but it fails in the wrong way. Since our test contains an assertion, we expected a failed assertion. Instead, what we've got is a failure due to the method under test throwing an exception.

The fix here is simple. Let's just change the Add method, so it returns any number different from zero:

public static int Add(string numbers)
{
    return -1;
}

Now, run the test again, and you'll see the error message is now this:

Add_EmptyStringAsParam_ReturnsZero
   Source: StringCalculatorTest.cs line 8
   Duration: 76 ms

  Message: 
      Expected: 0
      But was:  -1

Making The Test Pass

We're now finally ready to make the test pass. As I've said earlier, to make a test pass, you're not only allowed but encouraged to cheat. In our case, we can simply make the Add method return zero:

public static int Add(string numbers)
{
    return 0;
}

Writing The Second Test: A Single Number

The requirements say that passing a single number should return the number itself. That's sound like a useful thing to test:

[Test]
public void Add_StringContainingSingleNumber_ReturnsTheNumberItself()
{
    Assert.AreEqual(5, StringCalculator.Add("5"));
}

The test fails with the following message:

Add_StringContainingSingleNumber_ReturnsTheNumberItself
   Source: StringCalculatorTest.cs line 14
   Duration: 56 ms

  Message: 
      Expected: 5
      But was:  0

How can we make the test above pass in the laziest possible way? How about this:

public static int Add(string numbers)
{
    if (numbers == string.Empty)
        return 0;

    return 5;
}

Testing Two Numbers

Since we've already tested the Add method by passing zero numbers (an empty string) and a single number, it feels like the next natural step for us now would be to write a test for the scenario of adding two numbers. So, let's do just that.

[Test]
public void Add_TwoNumbersSeparatedByComma_ReturnsTheirSum()
{
    var numbers = "7,8";
    var expectedResult = 15;
    Assert.AreEqual(expectedResult, StringCalculator.Add(numbers));
}

The test above naturally fails since our method currently returns 0 when it gets an empty string and five otherwise. How can we change it, so this new test passes, the older tests continue to pass, in a way that doesn't solve the problem generally?

This is an idea:

public static int Add(string numbers)
{
    if (numbers == string.Empty)
        return 0;

    if (numbers.Contains(','))
        return 15;

    return 5;
}

Testing Three Numbers

Have you noticed that, up until now, we haven't done any refactoring? Well, we're getting closer to the point when our tests drive us to include some nasty duplication to our code. Then, we'll use refactoring to change the code in a way that gets closer to a general solution.

Let's see if we can do that by testing the scenario with three numbers:

[Test]
public void Add_ThreeNumbersSeparatedByComma_ReturnsTheirSum()
{
    var numbers = "1, 2, 3";
    var expected = 6;
    Assert.AreEqual(expected, StringCalculator.Add(numbers));
}

The test will naturally fail. Since the provided string contains commas, we fall into the conditional branch that returns 15. Our challenge now is to change the production method in a way that makes this test pass. Can we do it without going to the general solution to the problem?Let's see.

public static int Add(string numbers)
{
    if (numbers == string.Empty)
        return 0;

    if (numbers == "1, 2, 3")
        return 6;

    if (numbers.Contains(','))
        return 15;

    return 5;
}

By comparing the specified param with the exact input used in the test, we can make the test pass while avoiding going for the general solution. However, now we have managed to create code duplication. Can you see it? We're making two comparisons against the value of numbers, one right after the other. Let's see if we can get rid of that duplication.

public static int Add(string numbers)
{
    if (numbers == "1, 2, 3")
        return 6;

    if (numbers.Contains(','))
        return 15;

    int.TryParse(numbers, outint result);
    return result;
}

By leveraging the TryParse method from the System.Int32 type, I've managed to get rid of the first if instruction. We've also used a feature introduced in C# 7 called "out variables." This feature allows us to use out parameters without having to declare them beforehand.

All tests still pass, so I can't write more production code. What should the next test be?

Testing More Than Three Numbers

The requirements don't say we should only be able to handle three numbers. So, let's create another test case to cover the scenarios with 4, 5, or more numbers. While we're at it, we can also include the requirement of ignoring numbers greater than 1000.

To do this without having to create a lot of test methods, we're going to leverage NUnit's parametrized tests feature, adding a single method with several test cases:

[TestCase("1,2,3,4", 10)]
[TestCase("8,7,20", 35)]
[TestCase("5,0,4,1001", 9)]
[TestCase("5,0,4,1000", 1009)]
[TestCase("26,6,90", 122)]
public void Add_MoreThanThreeNumbersSeparatedByComma_ReturnsTheirSum(
    string input, int result)
{
    Assert.AreEqual(result, StringCalculator.Add(input));
}

Notice that the third test case exemplifies the requirement that says we should ignore numbers greater than 1000. The next test case, however, shows that 1000 should not be ignored. If you run the tests, you'll see that test explorer shows each test case as a distinct test.

How can we make this test pass? Honestly, by this post, it's way easier to go for the correct implementation than it is to cheat. So, let's do just that:

public static int Add(string numbers)
{
    var parts = numbers.Split(',');
    var result = 0;

    foreach (var part in parts)
    {
        int.TryParse(part, outint number);

        if (number <= 1000)
            result += number;
    }

    return result;
}

The code above should be easy to understand. We just split the string into parts using the comma as the delimiter. Then, for each part, we parse it to an integer, verify whether it's equal or less than a thousand, and, if so, we add it to the result variable. Finally, we return the result.

We're Not Done Yet

The requirements say that negative numbers shouldn't be allowed. Let's add a test for that! For brevity's sake, we'll add a single test method with several test cases, so we're forced to go for the correct implementation right away:

[TestCase("1,2,3,4,5,-5")]
[TestCase("-1,1,2,9")]
[TestCase("5,6,8,-5")]
public void Add_StringContainingNegativeNumbers_Throws(string numbers)
{
    Assert.Throws<ArgumentException>(() => StringCalculator.Add(numbers));
}

For this test, we're asserting not against a return value. Rather, we're checking whether the method under test throws an exception.

Remember that the requirements say we should throw an exception with a message saying that negatives are not allowed. We should also include a list of the negatives that were passed. This will require some changes in our method:

public static int Add(string numbers)
{
    var parts = numbers.Split(',');
    var result = 0; 
    var negatives = new List<int>();

    foreach (var part in parts)
    {
        int.TryParse(part, outint number);

        if (number < 0)
            negatives.Add(number);
        elseif (number <= 1000)
            result += number;
    }

    if (negatives.Count > 0)
    {
        var negativesList = string.Join(',', negatives);
        var exceptionMessage = $"Negative numbers not allowed: {negativesList}.";
        throw new ArgumentException(exceptionMessage);
    }

    return result;
}

As you can see, right at the beginning, we define a List<int> to store the negatives we find while iterating over all the numbers. Inside the loop, we verify whether the current number is negative. If it is, we add it to the list. If it isn't, we verify whether it's less than or equals to 1000, in which we case we add it to the result variable.

After the loop, we verify whether the negatives list has any elements. If it has, we create an exception message that includes the specified negatives and then throw a new ArgumentException. Otherwise, we return the result.

Conclusion

This post was a practical guide on how to get started with TDD in C#. So, where do you go from here?

Well, most things in life you learn by doing. Programming is certainly one of those things. So, if you want the concepts you've seen today to really sink in, you've got to practice.

The code I've written during this post is available as a public repository on GitHub. Go there, clone it using Git,, and start playing with it.

You'll see that I created one commit for each step in the TDD cycle. That way, it becomes easier for future readers to visualize all the steps in the process by going through the project's history, one commit at a time.

There are improvements that can be made to the code I shared today. For instance, the final Add method could be written in a shorter, clearer, more efficient way, using LINQ. You could add more test cases. Also, the kata requirements ask for a specific exception message in the case of negative numbers. Even though we've implemented the message as specified, we didn't write a test for it. We could do that as part of your practice.

Finally, stay tuned to this blog. This post is part of a series, to which I intend to add more parts.

Thanks for reading, and until the next time!

Get Started Quickly With Java Logging

Carlos Schults — Tue, 05 May 2020 15:26:53 +0000

You've already seen how to get started with C# logging as quickly as possible. But what if you're more of a Java guy or gal? Well, then we've got your back, too: today's post will get you up to speed with logging using C#'s older cousin, Java.

As in the previous post in this series, we'll not only provide a quick guide but also go into more detail about logging, diving particularly into the what and why of logging.

So, let's get you started with Java logging as quickly as possible. When you're finished reading, you'll know how to approach logging in your Java codebase.

The Simplest Possible Java Logging

For this simple demo, I'm going to use the free community version of IntelliJ IDEA. I'm also assuming that you have the Java JDK installed on your machine.

First, open IntelliJ IDEA and click on "Create New Project":

On the next screen, select "Java" as the type of project, on the left panel. You'll also need to point to the JDK's path, in case you haven't yet.

Third step---mark the "Create project from template" option and then select "Command Line App":

I've called my project "JavaLoggingDemo," as you can see from the image below.

After doing the steps above, you should see the Main class that was automatically created for you. It should look like this:

package com.company;

public class Main {

    public static void main(String[] args) {
    // write your code here
    }
}

Let's get to work. Replace the "// write your code here" comment with the following line:

System.out.println("Hello there!");

Now you have an application that displays a friendly greeting.

Next, let's run our app. Go to Run > Run 'Main' or use the shortcut Shift + F10 (Windows and Linux) or Control + R on Mac.

If everything went well, you should see something like this:

Nice. But let's say the requirements for our little app changed, as they always seem to do. Now we need our app not only to display a message but also to log it somewhere. How to go about that?

There are a lot of sophisticated options we could use, but let's try and do the simplest thing we can. Edit the code in your main function so it looks like this:

public static void main(String[] args) throws IOException {
    String message = "Hello there!";
    System.out.println(message);
    Files.write(Paths.get("log.txt"), message.getBytes());
}

Notice the changes. We've added a new variable (message) to store the greeting's text. Then there's the printing. And finally, there's a new line, which is meant to save the text to a file.

We also had to add a throw declaration to our method since the last line can throw an IOException. Additionally, we've added some new import declarations. The whole code now looks like this:

package com.company;

import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Paths;

public class Main {

    public static void main(String[] args) throws IOException {
        String message = "Hello there!";
        System.out.println(message);
        Files.write(Paths.get("log.txt"), message.getBytes());
    }
}

When you run your application again, it should behave the same as before. But the last line added will create a log file for you. It'll be located in your project folder:

That's pretty much it. You've just written your first Java logger! And with that first, small step, you're ready to try larger ones.

What Is Application Logging?

First things first. Let's briefly define what application logging is.

According to Wikipedia,

In computing, a log file is a file that records either events that occur in an operating system or other software runs, or messages between different users of a communication software. Logging is the act of keeping a log.

A little bit vague, right? I tend to prefer the definition we gave in our C# logging post:

Application logging involves recording information about your application’s runtime behavior to a more persistent medium.

With the what out of the way, let's get to the why of logging.

What’s the Motivation for Logging?

I think the key part of the logging definition we've presented in the previous section was the word persistent. Why would we need to record information about the behavior of our app in a persistent medium?

Well, for the same reason we record anything to a persist medium: so we can get back to it later. The question then becomes "Why would you want to do that with events that happened in an application execution?"

It all boils down to the nature of software itself.

A piece of software in production is a very complicated thing, more often than not running in a totally non-controlled environment.

How do you know it's going to work? And when things go wrong, how can you know what exactly went off the rails?

You can always cross your fingers and hope, but as they say, hope is not a strategy. Logging is, though!

In a nutshell, you use logging so that you can after-the-fact debug your application. By reading the log entries, you can understand the events that happened with your application, or even retrace the steps taken by a user, in order to diagnose and fix issues.

Evolving Our Approach: What Should We Capture? And To Where?

Being able to do an after-the-fact investigation on your app sounds like a great benefit --- and, indeed, it is. Our humble logger isn't able to provide us this benefit, though, since all it can do is write some text to a file.

Our job now should be to evolve our logger, turning it into a richer source of information for its future readers.

So how do we do that?

The key point to understand is the nature of a log entry.

Some of this is familiar ground, but I'm going to recap here for completeness' sake. Think of a log entry as an event. Something that is of interest to your application happened in a given instant in time. So the value of your log entry derives from the data you capture about the event.

The following list contains some examples of things that you'll probably want to capture.

A timestamp. Exactly when did the event take place? (You'll want to record times in UTC and using a standard format such as ISO-8601, in case you're logging to a file.)
Context that makes it clear to a reader what the entry's about. Just recording “Hello, user!” might prove confusing weeks or months later. A better message might say, “Recorded user greeting of ‘Hello, user!'”
Tags and/or categories for each entry, to enable later search and classification.
Log levels, such as “error,” “warning,” or “information,” that allow even more filtering and context.

After deciding what things you want to capture, the next step is defining where you want to log to. Even though I've used a file in our first example, there's nothing stopping you from logging to other media, such as a database (relational or not), the windows event viewer, or even the console.

Enter the Java Logging Framework

Time to get back to our demo. Here's the thing: it was designed to give you a quick start on the Java logging world. But now you're equipped to look at it in a more critical way, so let's do just that.

As it turns out, our humble logger is inadequate in several important ways.

For one thing, the code always overwrites the log file. A log strategy that only allows you to see the last event isn't terribly useful, isn't it?
And since we're talking about overwriting the file, it probably wouldn't hurt to think about file permissions, how to deal with concurrent access, and that sort of thing.
Finally, it sounds like an awful lot of work having to write boilerplate code in each logging call to get log levels, timestamps, and all that jazz I mentioned in the previous section.

I could go on and on, but you've got the picture: coming up with a sound Java logging approach requires a lot of thought. I have good news, though. As it turns out, people have already solved these problems. Not only that, they provide their solutions for you, ready to use and often for free, in the form of a logging framework.

A logging framework is a package you install in your application that provides easy and configurable logging for you. With a little bit of configuration, you can make one-line calls as simple as the one in our toy example. But this time, they'll have consistent and nicely formatted log entries, and they'll have answers to all those questions from the previous section.

Installing a Logging Framework

The first thing we're going to do is go back to our codebase and delete that line that writes to the file. Then we can proceed to install a logging framework called log4j. This is only one of the options that are out there, and I do encourage you to try alternatives later. But since this is a quick guide, it makes sense to go with log4j since it's a widely used framework.

On the Project tool windows, right-click on the "JavaLoggingDemo" module and then on "Add Framework Support...":

Now, select "Maven" and click OK:

After you've done this, the IDE will create a pom.xml file for you and open it for editing:

Now, copy and paste the following XML text into your file:

<dependencies>
      <dependency>
        <groupId>org.apache.logging.log4j</groupId>
        <artifactId>log4j-api</artifactId>
        <version>2.10.0</version>
      </dependency>
      <dependency>
        <groupId>org.apache.logging.log4j</groupId>
        <artifactId>log4j-core</artifactId>
        <version>2.10.0</version>
      </dependency>
    </dependencies>

The whole file should look like this by now:

<?xml version="1.0" encoding="UTF-8"?>
<project 
         
         xsi_schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>groupId</groupId>
    <artifactId>JavaLoggingDemo</artifactId>
    <version>1.0-SNAPSHOT</version>
    <dependencies>
        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-api</artifactId>
            <version>2.10.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-core</artifactId>
            <version>2.10.0</version>
        </dependency>
    </dependencies>     
</project>

After adding the dependencies to the file, you should see a small pop-up in the bottom-right corner of your screen. Click on "Import Changes" to finish the installation.

Configuring the Logger

Now it's time to configure log4j.

What I'm going show you is one way of doing this; it's not the only one, and it's probably not even the best one, whatever "best" may mean. But it's definitely shorter than a lot of the tutorials you see out there.

Pretty much all you have to do is to create a configuration file and then write some lines of code. Don't believe me? Well, let me just show you, then.

First, go to the "Project" tool window on IntelliJ. Expand the folders and locate the "resources" folder, under JavaLoggingDemo > src > main, just like the image below:

Now, right-click on the resources folder, and then select New > File:

When you're prompted for a name, enter "log4j2.xml." After the file is created, paste the following text into it:

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="INFO">
    <Appenders>
        <File name="FileAppender" fileName="proper.log" immediateFlush="false" append="true">
            <PatternLayout pattern="%d{yyy-MM-dd HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n"/>
        </File>
    </Appenders>
    <Loggers>
        <Root level="ALL">
            <AppenderRef ref="FileAppender"/>
        </Root>
    </Loggers>
</Configuration>

Editing the Code

First, at the top of the file, add the following two import declarations:

import org.apache.logging.log4j.Logger;
import org.apache.logging.log4j.LogManager;

Then, add the following line to the top of the Main class, which declares a private field to hold the instance of the logger:

private static final Logger logger = LogManager.getLogger(Main.class);

Finally, add the following line of code to where you earlier had the call to Files.write(Paths.get("log.txt"), message.getBytes()):

logger.info(message);

The whole class should now look like this:

package com.company;

import org.apache.logging.log4j.Logger;
import org.apache.logging.log4j.LogManager;

public class Main {

    private static final Logger logger = LogManager.getLogger(Main.class);

    public static void main(String[] args) {
        String message = "Hello there!";
        System.out.println(message);
        logger.info(message);
    }
}

And we're done! Now all you have to do is run the application.

Checking the Results

Now navigate back to the application folder and notice a new file there, called "proper.log." If you open it using a text editor, you should see the following:

2019-09-25 15:39:29.739 [main] INFO  com.company.Main - Hello there!

Let's break this line into its components.

2017-12-31 15:39:29.739

First, we have the timestamp, in the ISO-8601-compliant format.

[main]

This refers to the name of the thread from which the event originated.

INFO

Here we have the logging level.

com.company.Main

Then, the name of the class.

Hello there!

Last, but not least, the message itself.

The Power of a Java Logging Framework

I think you'll agree that our logger just got a lot more useful with the update we've made.

Just by creating a config file and writing a few lines of code, we were able to configure a realistic example of a logger. This isn't all there is to it, of course, but it's already enough to be useful in a real application. What's more, it gives you a pretty good idea of the power a logging framework can put into your hands.

The XML configuration file is one the places this power really manifests itself. It offers you a lot of options to configure each entry that you log, such as the name of the log file or if the logger should append to it or overwrite it, just to name a few.

A lot of the flexibility log4j provides is due to something called an appender. The appender is the component that effectively writes the message to some medium, and there are many of them available. This means it's possible to direct your logs somewhere else entirely, just by adding a new appender to the XML file, without even touching your application code.

Separation of concerns at its finest, if you ask me.

Happy Learning!

What we've seen today is just the tip of the iceberg. Make no mistake: you now have a lot of learning ahead of you. But you have the advantage of already having a live, functional, and realistic setup to work with. Start by playing with and tweaking it.

Here are some tips on what you can do:

Learn how to configure different loggers, one for each log level.
Research and try out the different appenders available.
Play with the options available for the "layout" entry.
Learn about the other ways you can configure log4j.
You have lots of options for logging frameworks; try some of them, once you're confident enough with log4j.

And to learn more about Java logging and logging strategies in general, you probably won't find a better place then Scalyr's blog. Scalyr offers a log aggregation tool, which means that once you have lots of log files and data, they’ll help you organize, search, and make sense of all these data. So stay tuned for more!

Learning something from zero can be an overwhelming task. Fortunately for you, you don't have to do that. We've given you the fundamentals upon which you can build a solid and long-lasting body of knowledge.

Now it's up to you.

PS: If you're interested in further reading on the topic of logging in other languages and platforms, take a look at some of our available guides: