What is TestFleet?
Think of it like a Postman collection you can run on a schedule, with actual assertions to validate the response from your APIs. If you've ever used Datadog's synthetics, you'll be familiar with how TestFleet works. Just set up an instance of the control server and test runner, and you're ready to create multi-step API tests.
Why I built it
Before I even knew what this project was going to be, I had already decided that I wanted to build something with distributed worker nodes. A fleet of distributed processes doing coordinated work and reporting back to a central server. I just needed the use case to build it, and API monitoring turned out to be a great fit. It requires a comprehensive control plane to create and schedule jobs, and benefits from a swarm of worker nodes executing that work.
The Moving Parts
Control Server
This is TestFleet's brain, and the only component you'll interact with directly. It serves both the TestFleet API & the control server UI. The API exposes endpoints for CRUD operations on scenes, frames, & runners, and the React frontend interfaces with the API directly to deliver a seamless, single deployment experience. It's also responsible for scheduling tests, monitoring runner health, and aggregating test results from multiple test runners.
Test Runner
It would be cool if we could continue the analogy and call this the heart, but in reality, it's a headless execution engine, and relies on an intermediary layer to get tests from the control server. The test runner itself is not a server, and cannot receive HTTP requests. It does however use HTTP to send its test results, and, ironically, its own heartbeat back to the control server.
I mentioned earlier that the control server aggregates test results from multiple runners, which brings us to the core functionality that makes this a 'fleet' of test engines. You can deploy many runners, across various deployment regions, providers, etc., as long as they A: Can send HTTP requests to the control server from where they are deployed, and B: Can connect to Redis, more on that later.
Redis & MongoDB
So if the test runner isn't a server, how does it get jobs from the control server? The control server pushes jobs to a Redis pub/sub queue, which is then broadcast to every runner connected to Redis & subscribed to the channel the control server is publishing to.
MongoDB is where all the data for the control server is persisted. This includes scenes, frames, runners, user profiles, test results, & some platform configuration data.
I go into a lot more detail on the design choices I made later in this post, but kept this section intentionally short for those just interested in the high level description.
How it works
Inviting Users
This application uses OAuth to authenticate users, so you are required to set up an OAuth app for one of the supported providers (Google, Microsoft, GitHub, Okta). Once that's set up and ready to go, you need to deploy the control server with the OAuth configuration variables, and your bootstrap account email. This is the root user of the application, and will be invited with admin privileges when the control server is spun up for the first time. You can then proceed to log in with this email via SSO, and from there you can proceed to invite additional users through the UI.
Creating Test Runners
You need at least one runner to execute tests, but you can have as many as you need for your use case. To create a runner, first make sure you have resources provisioned for its deployment, and that the network is able to connect to your Redis instance, as well as send HTTP requests to your control server. Now you can go to the control server UI, and create a new runner. After choosing a runner name, you will see a window with the runner API key & secret. At this point, you can deploy your runner with the credentials, and within 30 seconds, the runner should send its first heartbeat to the control server, signaling its ready status. You can check the status of the runner in the Runners page.
Building Multi-Step API Tests
My primary goal was to make a tool that didn't get in its own way. Since this is a platform tool, it's inherently technical, but that doesn't mean it has to be difficult to understand. One of the ways I tried to achieve that was with the naming convention. In TestFleet, an API test is called a scene, and every step in that scene is referred to as a frame.
To generate a test, you need to navigate to the Scenes page, where you will find a list of your current scenes, and a prompt to create a new scene. You can click this and enter the initial configuration for the scene. After you have created the scene, you will see the scene editor window. At the top of the editor, you have your variables. You can generate scene scope variables here. You will also be able to see the frame extracted variables once you have generated frames that extract variables from their response.
It's useful to have an API test planned out before generating frames, but there are tools to help make building frames easier for you. While building out a frame, you can test the frame as configured. This will send an HTTP request with the destination & payload defined in the frame. Keep in mind that any extracted variables referenced in a frame will resolve to null values since the variable's value is not yet defined. Create your assertions to test the validity of the response, and extract any JSON or response headers you need for future frames. Once you're ready, save the frame. At this point, you can either add more frames, or you can commit the changes to the scene by clicking 'save scenes' on the top right of the scene editor.
Now your scene is ready to run! You can either wait for the scene to run on its cron schedule, or you can force a run in the scenes UI. You can navigate to the 'Runs' page to view all runs in list view, or you can view runs for a specific scene by opening the scene, and clicking on its 'Runs' tab.
Running it yourself
To make deploying as simple as possible, I have created a Helm chart repo for this project. The README has instructions for deploying via Helm, but if you want to test it out locally, you can clone the control server repo and use Docker Compose configuration to build the project on your machine. Check the control server README for detailed instructions on how to run locally.
Design decisions
Redis Pub/Sub:
One of the first design decisions I made when planning this project was to deliver jobs to the test runner via broadcast. Since having multiple runners deployed is a core feature, I wanted a way to deliver job payloads to N runners, in the most robust way possible. After some consideration, I landed on using Redis Pub/Sub as the intermediary layer between the runner and the control server.
I considered WebSockets for like 10 seconds before deciding they were unnecessary. There's no reason for a runner to have a constant connection to the control server, and at scale, could introduce overhead. It also complicates reconnect logic for potential runner outages, and makes resetting connections more difficult.
I also considered using HTTP for all communication, but that required turning each runner into a server, which would require the control server to know the address of each runner, and simply did not align with the goals of the project.
Since I was already familiar with message-brokers & event streaming from prior work with Apache Kafka & RabbitMQ, those were the first tools I considered using. The problem with both of those, was the messages would be consumed by a single consumer, and I needed the job to be broadcast to N runners. After a bit of digging, I found that Redis pub/sub is designed to broadcast by default, and on top of that, it is a much lighter and more manageable dependency.
Golang for Test Runner:
Goroutines. This one was a no brainer. Go is my favorite language, so realizing the runner was basically built to take advantage of Go's concurrency patterns was really exciting. I ended up using a worker pool to process each job sent from the control server. The MAX_WORKERS environment variable allows users to configure how many workers are created when the runner is spun up, and defaults to 3. Jobs are stored in an unbuffered channel, where workers will pull from once it is ready to process a job. If all workers are busy, the goroutine responsible for reading from Redis will be blocked on handing that job off. If this happens & Redis delivers a new job, those scenes will be lost. I don't envision a scenario where a tool like this, that runs a scene at most once per minute, could ever be overwhelmed in this way, but because it is technically possible, I made the number of workers configurable.
MongoDB:
This is another decision where adoptability was weighed more heavily than other factors. What I found most interesting about designing this project, was how different the architecture was from if I were building a managed platform. MongoDB is a readily available dependency, and doesn't require SQL expertise to manage. On top of that, I can skip a lot of DB configuration on control server startup, such as DB migrations. Would an SQL database have been more robust? Probably, but I think for this project, balancing function with adoptability was more important than having the most performant platform possible.
Control Server:
Once again, in the name of adoptability, the control server bundles both the UI and the API into a single package. The API is a standard Express REST API, and the frontend is a React/Vite application. I'm not a full-stack developer, and I'm not as familiar with JavaScript as my co-collaborator Suleman. The frontend was designed & integrated into the control server by him, and any changes that I made to the frontend are done with the assistance of Claude Code.
Future Plans
Alerts:
As I'm writing this blog post, I'm also actively working on alerting through Slack notifications. When I was considering what would make the cut for the initial release, I spent the most time contemplating whether or not to include alerting. I decided that it wasn't critical to the purpose of the tool, but I know how useful it can be for teams, so it's currently at the top of my list of new features.
Postman Import Support:
I haven't spent as much time thinking about how I want to implement this, but I think it would be cool to export a collection from Postman & import it as a new scene in TestFleet. I've also thought about having a JSON template for creating scenes, but I think supporting Postman collections makes more sense.
Secrets Manager Integration:
It would be nice to import secrets from a trusted secrets manager and use those in scenes instead of having to hold real credentials in scene configs.
CI/CD Triggers:
I think it would be really cool if I created a way to kick off a scene from outside the platform, via web hook or API trigger. If you had a really comprehensive scene, you could trigger it via deploy pipelines, and potentially use the scene for CI/CD gating based on the results of the run.
Try it out
If you've made it this far, thanks for reading my post! The purpose of me writing this was to gather as much feedback as possible, so if you have any comments, recommendations, critiques, I would appreciate you leaving a comment. Thanks!
If you're interested in trying TestFleet out, you can download the control server repo to run the project locally using the Docker Compose configuration. Just make sure you have Docker running, copy the .env template and fill in the required fields, and use the make build command to build the project.
Top comments (0)