How (and why) I created my API testing tool, Yapitest

My career started out working for a company that tested C++/C code for industrial applications with strict standards, so I learned the importance of testing. Then I switched towards web-based/devops roles and never saw anyone being that thorough again.

In my current role, I had to create a platform for developers to run their tests and we needed to support a certain set of technologies, namely typescript and C#. The decision was Playwright for websites, xUnit for C# testing, but the teams had to agree on something for API testing, and we settled on playwright again. In the process of my proof-of-concept for teams to begin their API testing. I had to sadly write a few tests myself.

I absolutely hated it.

I cannot stand the terrible syntax of expect(var).toBe("salad") and verbose constructions like that. The non-technical QA team members had an even harder time figuring it out. For one, you have to deal with setting up a development environment for people who have no idea where to begin. Additionally, the playwright startup time takes a while, even if you're just testing APIs which could be done with just a few curl requests. For actual websites, playwright is great, but for APIs it's overkill. I thought about Postman, but as a seasoned Neovim user the thought of a GUI makes me sick. Also, I've heard they may or may not steal your data.

So I came up with something on my own. The prerequisites were as follows:

• Easy to write and more importantly understand tests
• Usable on the command line
• Fast

And Yapitest was born (it stands for "Yaml API Testing" if that wasn't clear)

I initially wrote it in Python before switching to Rust, because that's what I use for a lot of scripting in my DevOps role and it's easy to prototype in. Another thing that any DevOps engineer is familiar with is Yaml, whether it be helm charts, Pulumi configs, or GitHub Workflows, Yaml is everywhere. So that's what I settled on. Yaml for the test specs and python for the v0.1 implementation.

Implementation

The first thing I decided on was what I want my tests to look like. I wanted them to be easily readable even for someone who has never looked at a yapitest file before, and I think I nailed it:

test-create-and-get-post:
  setup: create-user
  steps:
    - path: /api/post/create
      id: create-post
      method: POST
      headers:
        API-Token: $setup.token
      data:
        title: "Some Title"
        body: "Some message"
      assert:
        status-code: 201
    - path: /api/post/$create-post.response.post_id
      assert:
        body:
          title: "Some Title"
          body: "Some message"

You have a test name, and a 'steps' section which lays out which endpoint you're hitting with the method, headers, and data all plainly laid out. Then there's an 'assert' block that can validate status-codes, and response bodies. Embracing DRY principles, fields in any of these can refer to values that were in earlier steps, to reduce the amount of duplicates.

Then, I created a crappy flask web service to test against. It has primitive forms of users, and posts. Nothing too complicated, I just needed something to ping against.

Finally, I worked on the implementation. The meat of the code lies in the Configs and the Steps. A test is just a collection of steps, and the config is a place where you can store values or other groups of steps that may be needed in multiple places.

Configs

The Configs were the first issue that I ran into. I wanted configs to be usable across many tests, but I knew that having one massive config file for a plethora of tests may not be suitable for every project. So I decided to have multiple config files. If you have tests/users/config.yaml, then that config only applies to all tests inside of tests/users/. If I have tests/posts/config.yaml, then those tests only apply to that tests/posts directory. They do not know about eachother. However, if you have tests/config.yaml, that would then apply to all tests inside of tests/. So I decided to create a parent/child relationship with the configs. If a config is in a subdirectory of another config, that other config is it's "parent". If the parent and child both have a value defined, then the child's value wins out.
This was accomplished by first finding all of the configs, then simply iterating up until we found either another config, a .git directory, or the root of the filesystem. Then on test discovery, we find the closest related Config and set it as the test's config, and then we have access to all of the configs that are applicable to that test. The configs are also fairly simple to understand:

// Variables that can be used anywhere
vars:
  default-url:
    env: BASE_URL
    default: http://127.0.0.1:8181
  sample-user:
    env: SAMPLE_USERNAME
    default: test-user-123
  sample-password:
    env: SAMPLE_PASSWORD
    default: p4ssw0rd123!

// Basically another variables dedicated to urls "base" is a keyword that means it's the default URL for everything unless overridden.
urls:
  base: $vars.default-url

// Reusable steps sets that can be run in tests
step-sets:
  create-user:
    once: true
    steps:
      - id: create-user
        path: /api/user/create
        method: POST
        data:
          username: setup-test-user
          password: setup-pass-123!
    output:
      username: $create-user.data.username
      password: $create-user.data.password
      token: $create-user.response.token

The reasoning behind test steps was you may need a fresh user for a dozen tests, and sticking those requests in a all those tests quickly blows up your test files and adds a ton of duplicate code. I wanted to reduce that in the easiest way possible. Additionally, I want you to be able to read outputs from those steps, so the step-sets have outputs that can be assigned to values from those steps.

Tests

The next step was to write the test steps. This was fairly simple in that most of the complexity was in the yaml files. We read the data in from the data block, replace any $ values with the values from the config, and then use the specified request method on the URL + Path and get the response, then run through the key/values of the assertions and if anything is wrong, fail out.

Then it came down to discovering the tests. It simply reads in the test data, turns them all into TestSteps, and then calls each step in succession, failing the test if any assertions fail.

Migrating to Rust

After writing a fully-featured python implementation, I decided to switch to Rust. As anyone knows, Python is not strongly typed and bugs could be lurking anywhere. Everything worked, but something in the back of my mind said "this code may not be safe to rely on", which is fairly important for a piece of testing software. I decided to go fully in the opposite direction and use the language with the strongest type system in the world, Rust. (I also hadn't written rust and figured it would be a good project). A part of me wanted to write it in C++, but I don't like make or cmake and didn't want that smoke. Also crates.io could provide me with packages to do the yaml/json parsing and C++'s build system is a nightmare.

One of the first issues with Rust is that I had no idea how to use it. I understand C/C++ pointers, but rust references threw me off, particularly the rules around them. Being unable to modify mut vars in two places in the same function (for some reason, I could be wrong) was confusing. I knew that the things were happening in sequence and the code would be safe, but the compiler wasn't having it. I decided to embrace it and believe that the compiler knew something that I didn't, so I continued working on it. The "Ownership " confused me as well. I wasn't sure why I couldn't return references to values that I had created inside the same function, because you can do that in C++, but I eventually came to understand why that is the case. Dangling pointers and what-have-you.

My initial plan was to have a Config, Test, and TestStep structs, and then when the tests run they add the results to those structs like I had in the Python classes, but it didn't feel Rust-y enough to make everything mut everywhere. So I pivoted to having the Test run to return a TestResult, and the TestStep return a TestStepResult so there would be no/minimal mut. This was a big pivot in my understanding of the "best" way to use Rust, and I'm glad I did it this way. The code eventually became easier to write and understand, and I didn't have to worry about values being shared all over the place.

Conclusion

So now I have my package working exactly as I desired. I think it fixed a lot of the issues I see with API testing at least at my company and believe that it could help others too. There still are some features I need to add (timing assertions, regex support), but that's just more of a reason to keep using rust, which I have grown to like.

Thank you for reading this far, I know it was not that in-depth but it gives a top-level summary of my experience writing this software and a brief insight into how a newcomer looks at Rust in particular.

Check out my Yapitest Repository and give it a whirl (and maybe a star). Thanks again!