Soak Testing with K6

What is Soak Testing

A soak test, it's like a load test that runs for an extended period. For the following examples I decided to use K6, an extensible testing tool.

Why is useful

Typically with a normal load test, we send a bunch of requests for a short period and monitor how the platform performs. While this may be useful for most scenarios, the typical load test might miss some issues that would come up only when the system is stressed for a long period. For example, during a major event, we might have a lot of clients using our platform for hours and this might cause problems related to memory leaks, logs filling up the disk space and so on...

Example of Web API with a memory leak

The following example has been taken from Microsoft Diagnostic Samples - This Web API will cause the target to leak memory (amount specified by {kb}).

Observe that by hitting the API Endpoint, the memory usage keeps growing (the GC Heap Size (MB) increases)

By running a conventional load test for a short period, it returns a 200 so we think that everything it's working as expected:

However, when we run a Soak Test for a long period, we notice that some of the requests are failing:

Investigating the exceptions we can see what the issue is:

System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.

And this is the full stack trace:

fail: Microsoft.AspNetCore.Server.Kestrel[13]

      Connection id "0HMHUTUB1JJR6", Request id "0HMHUTUB1JJR6:00000003": An unhandled exception was thrown by the application.

      System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
         at System.Collections.Generic.List`1.set_Capacity(Int32 value)
         at System.Collections.Generic.List`1.AddWithResize(T item)
         at MemoryLeakApi.Controllers.MemoryLeakController.memleak(Int32 kb) in /src/Controllers/MemoryLeakController.cs:line 18
         at lambda_method1(Closure , Object , Object[] )
         at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.SyncObjectResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeActionMethodAsync()
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeNextActionFilterAsync()

      --- End of stack trace from previous location ---
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()

      --- End of stack trace from previous location ---
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
         at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
         at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)

Conclusion

As we can see, a conventional load test would have missed the issue (in this case a memory leak) - keep in mind that this was just an example.

In a real-world scenario, you would want to run a soak test for several hours on a lower environment or even in production if your application can handle it.

Then set some alerts on whatever monitoring software you use and check out the results to narrow down and fix any issues.