DEV Community

Agile Actors Hellas profile image Georgios Pligoropoulos
Georgios Pligoropoulos for Agile Actors Hellas

Posted on

Which endpoints are tested? Answered, instantly

#api #automation #productivity #testing

They told us it was impossible. They were wrong.

And they kept asking the same anxious question...
Which endpoints are tested?
A question that usually shows up right when you are trying to enjoy that lunch break where you promised yourself you would not open a laptop.

You want this answered now. Instantly. For hundreds of scenarios.
So you open Swagger UI.
You stare at the endpoints.
You map an endpoint to whatever name the autogenerated client felt like giving it.
You search.
Multiple versions.
Same method names.
Different clients.
...Of course!
You filter results.
Wrong client.
Ignore that.
Ignore this.
Not a scenario.
Still not a scenario.
You finally find the right class.
You count invocations.
One. Two. Maybe three.
Was that all of them?
Now do it again.
Every endpoint.
Every version.
Every Swagger file.

Somewhere around here you realize you’re not testing anymore.
And this could end here, as a sad story of a low budget.
But every story has a moment where everything changes. The year is 2025 and LLMs are any developer's best pals, cheaply available.

Frankly speaking, you can buy an electric drill. You can take the conscious decision to not care how the electric drill is built, and only care that it does its job well enough. An LLM could code the whole thing, teach us how to use it and even write this blog post for the tool as well, if we really wanted to.

Step 0: Symmetry Doesn't Happen

Before we began, we confirmed that without any exceptions NSwag was already used consistently across the entire project. It is a library that parses the Swagger Json and generates C# classes and methods that correspond to endpoints. Because without a generated client, the same endpoint might be called in ten different ways across the codebase. Then your coverage question turns into archaeology. Who said obsession does not pay off?

Symmetry in the code, a purely technical project, no business specific context .. sounds like the perfect recipe for automation Step by Step.

Step 1: Get Requests from Swagger

The swagger json file looks like that

"/HealthCheck": {
  "get": {
    "tags": [
      "HealthCheck"
    ],
    "summary": "Method for health checking api version 1",
    "parameters": [
      {
        "name": "Accept-Language",
        "in": "header",
        "schema": {
          "type": "string"
        }
      }
    ],
    "responses": {
      "200": {
        "description": "OK",
        "content": {
Enter fullscreen mode Exit fullscreen mode

Read the NSwag configuration JSON. Inside it you will find the link to the Swagger JSON.

Fetch the Swagger JSON, parse it, and collect all paths along with their HTTP methods (GET, POST, PUT, etc.). In the current implementation, the key is simply Method + Path.

Yes, you could go further and track different scenario variants by parameter combinations. But if your first milestone is "every request is covered at least once", that extra complexity is just glitter on a fire alarm.

Step 2: Forget about Regex and bring a magician on Board

Today's magician is Microsoft's Code Analysis aka Roslyn:

<PackageReference Include="Microsoft.CodeAnalysis" Version="4.11.0" />
<PackageReference Include="Microsoft.CodeAnalysis.CSharp" Version="4.11.0" />
<PackageReference Include="Microsoft.CodeAnalysis.CSharp.Workspaces" Version="4.11.0" />
<PackageReference Include="Microsoft.CodeAnalysis.Workspaces.MSBuild" Version="4.11.0" />
Enter fullscreen mode Exit fullscreen mode

We are building something beautiful, and beauty needs structure. Roslyn allows you to navigate the codebase properly and search with semantics.

A quick Cmd+F search through the client reveals there is a comment called Operation Path that matches the url and the Method name is just above.

public virtual async System.Threading.Tasks.Task<GetProductDetailsResponse> ProductDetailsAsync(string productId, string accept_Language)
{
    if (productId == null)
        throw new System.ArgumentNullException("productId");

    var client_ = _httpClient;
    var disposeClient_ = false;
    try
    {
        using (var request_ = new System.Net.Http.HttpRequestMessage())
        {
            if (accept_Language != null)
                request_.Headers.TryAddWithoutValidation("Accept-Language", ConvertToString(accept_Language, System.Globalization.CultureInfo.InvariantCulture));
            request_.Method = new System.Net.Http.HttpMethod("GET");
            request_.Headers.Accept.Add(System.Net.Http.Headers.MediaTypeWithQualityHeaderValue.Parse("text/plain"));

            var urlBuilder_ = new System.Text.StringBuilder();
            if (!string.IsNullOrEmpty(_baseUrl)) urlBuilder_.Append(_baseUrl);
            // Operation Path: "Product/productDetails/{productId}"
            urlBuilder_.Append("Product/productDetails/");
Enter fullscreen mode Exit fullscreen mode

So Regex and pray or Roslyn and play? Either way, now you have an automated mapping from Swagger request to NSwag-generated method name.

Step 3: The Brute Force Part

Don't get me wrong, humans are great, but humans are not meant to do the same search 600 times.

Finding where those client methods are called throughout the repository seems like a Cmd+Shift+F of the ProductDetailsAsync ?
Well.. do you remember that time that you picked the username definitely-not-taken that you were sure to be unique but it wasn't after all ? It is one of those times!
You soon realize that the method is named exactly the same among versions, which you didn't think of, plus the method happens to be invoked inside the autogenerated code itself, and your luck is so great that some library happens to use the same method name for a completely different reason.

Let code analysis scan the entire solution. Iterate every project, every C# document (.cs file), and collect invocations of any kind.
If the string representation of an invocation matches one of the method names you collected, keep it.

What you want out of each invocation, and can get thanks to Roslyn, is a couple of things:

  • Filepath: in which file we find this invocation
  • Containing Class: Looking at the ancestors in the syntax tree which is the first Class that we encounter
  • Line & Column Number: To be able to pinpoint it exactly in the file

And, the most useful of all, the Definition of the Method:

  • Filepath: Where the file is found or dummy string if outside of the project
  • Definition Class: The class that defines the method that was invoked

Voila! With the dictionary method name -> all the invocation info you can now start filtering, filtering, filtering to ensure that only the ones involved in the scenarios of the suite are included.

In other words:

  • Keep only invocations that belong to the current NSwag client (and the correct API version)
  • Exclude invocations inside NSwag-generated code.
  • Exclude calls from places unrelated to scenarios, so the numbers reflect real test coverage.

Step 4: Show it to the World

As the fan of the CPU slows down, count, export to CSV and if you feel like showing off, plot the statistics into a bar chart.

Request Count
GET /MeinePost 0
GET /order/parcelStamp/size 1
GET /order/parcelStamp/config 3
POST /AddressValidation 7

Final Step: Remove the blindfold

The call to action becomes obvious. If a request has a zero count, it is not involved in any scenario at all.

From experience, covering each request at least once is the first meaningful milestone. Once you hit that milestone, the conversation can become creative and interesting: deeper scenario variants, data combinations, edge cases, and all the fun stuff.

Eagle's Eye View

You can, but should you ?

The coding of the project was faster than the writing of this blog post. Think about it. Efficiency was at its peak. But when speed increases, something else must give, and it’s neither computing power nor electricity.

You used the drill but you did not learn how to build a drill, didn't you ?
For sure you learned how prompt engineering can construct the entire project but merely understanding what you see does not mean that you actually learned how to do it.
Learning requires what the education industry now calls productive struggle and there is a great TED talk explaining it, if you want to know more: https://www.youtube.com/watch?v=YBH8rQv4aTQ

You will not believe how much I am hesitating of writing a suggestion here, as the temptation to not follow it myself is real, but here it is: Give the LLM work that you already know how to do yourself and it is just boring and slow to do on your own. Just don't let the LLM think on your behalf.

There's no going back. Choose wisely.

"Which endpoints are tested?" Answered instantly.

"Why do they matter?" That meeting is still on Monday.

At Agile Actors, we thrive on challenges with a bold and adventurous spirit. We confront problems directly, using cutting-edge technologies in the most innovative and daring ways. If you’re excited to join a dynamic learning organization where knowledge flows freely and skills are refined to excellence, consider joining our exceptional team. Let’s conquer new frontiers together. Check out our openings and choose the Agile Actors Experience!

Top comments (0)