How deterministic context batching reduced approximate token usage by ~81.7% while building an agentic API testing prototype
See It in Action
I also recorded a short demo showing the complete workflow, from loading the OpenAPI specification to generating and executing tests, and auto healing:
I originally built this as part of my GSoC 2026 proposal for foss42.
I wasn't selected for GSoC, but while working on the proposal I had already built a working prototype of the system I wanted to contribute: an agentic API testing workflow that could read an OpenAPI specification, generate tests, execute them, and reason about failures.
And while building it, I ran into a problem that had little to do with HTTP itself.
The OpenAPI specification was becoming too large to send to an LLM efficiently.
That led me to build OpenAPI Context Batching -- a deterministic approach for giving an LLM only the relevant part of an OpenAPI specification instead of repeatedly sending the entire API.
In my prototype experiment, this reduced approximate token usage from ~50,000 tokens to ~9,200 tokens across the same five-interaction workflow; an 81.7% reduction🤯.
The Problem: OpenAPI Specs are Massive
The original goal of my project was to let an AI agent read an OpenAPI specification, generate test plans, and analyze API failures. But I quickly ran into a huge problem.
Real OpenAPI specifications are gigantic. They contain hundreds of endpoints, nested schemas, and complex validation rules. If you dump that entire JSON file into an LLM prompt, you run into two major problems:
- It burns through thousands of tokens per single session.
- The model gets completely overwhelmed and starts hallucinating fake endpoints.
If I just want the AI agent to test a /categories endpoint, it absolutely does not need to see the schemas for /payments or /cart or the endpoints not related to it. I realized I needed a way to give the model a smarter context instead of just a bigger one.
The Golden Rule: AI Plans, Dart Executes
Before I could optimize the tokens, I had to lock down the core architecture. I established one strict rule for the whole system: the AI figures out what to test, and Dart does the actual testing.
Letting an LLM directly execute HTTP calls is a terrible idea because it can easily invent fake responses. To prevent this, the model is constrained to generate a structured JSON test plan.
I built a standalone, headless Dart execution class called ApiTestRunner. This runner ingests the JSON test plan and performs the real HTTP requests using the networking layer of API Dash. The AI never touches the network directly.
The Solution: Deterministic Context Batching
So, how do we actually shrink a massive OpenAPI spec? We do not ask another AI to summarize it. We use deterministic code because OpenAPI is highly structured.
I wrote a custom Dart algorithm called OpenAPI Context Batching. Instead of sending the full specification, the script parses it and breaks it apart deterministically. Here is exactly how it works under the hood:
- Domain Splitting: The script reads the raw OpenAPI JSON and splits the endpoints by their root domain, such as
/author/categories. - Recursive Reference Resolution: OpenAPI heavily uses
$refto reuse shared schemas. If you only extract the endpoint path, the AI will not know what the data payload actually looks like. I built a recursive resolver that digs through the endpoint and finds every$ref. - Deep Schema Extraction: The resolver pulls in only the specific, deeply nested schemas required for that exact endpoint. This completely avoids sending the bloated
components/schemasobject. - Focused Context Map: The output is a clean
Map<String, String>where each entry is the fully resolved, batched schema for just one domain.
When the agent needs to reason about a request, it only receives this highly focused batched schema. Because the algorithm is deterministic, the exact same OpenAPI specification always produces the exact same endpoint partitions every time.
The Guardrailed Retry Loop
To make the system even smarter, I built a conversational agentic mode with a self healing loop.
If the Dart execution engine runs a test and gets a failing non 2xx status code, it intercepts the error. It then injects the exact status code and response body back into the LLM prompt, asking the agent to fix the parameters automatically.
To make sure this does not result in an AI driven infinite loop that drains credits, I hardcoded a strict limit of three maximum retries.
The Results: 81.7% Token Reduction
I benchmarked this approach on a custom shop.json OpenAPI specification during my prototype development. The result was significant: approximately 81.7% fewer input tokens for the tested workflow.
| Approach | Approximate Token Usage |
|---|---|
| Entire OpenAPI specification | ~50,000 tokens |
| Context Batching | ~9,200 tokens |
By only sending the dependency aware context, I achieved an 81.7% reduction in token usage for a five step conversation workflow. Not only did it save a massive amount of tokens, but the focused context meant the AI responded faster and stopped hallucinating.
| Approach | Visualization | Approx tokens |
|---|---|---|
| Entire OpenAPI passed at once |  |
~50,000 |
| OpenAPI Context Batching |  |
~9,200 |
The Biggest Takeaway
The coolest thing I learned from this project is that we should not use LLMs for everything.
Parsing files, finding dependencies, and executing HTTP requests are tasks best handled by deterministic software. Generating test plans and reasoning about edge cases are where the LLM actually shines.
Building this engine independently of the Flutter UI also allowed me to expose it as an MCP server, meaning external tools like Claude Desktop can trigger real test runs. It was an incredible learning experience, and I am super excited to keep exploring how we can build smarter, more efficient developer tools!
Top comments (2)
Really nice approach. The part I find most interesting isn’t actually the 81.7% token reduction — it’s that you’ve essentially built deterministic retrieval over the OpenAPI dependency graph.
With traditional RAG, context selection is probabilistic: embeddings → similarity search → top-k chunks. Here, OpenAPI already gives you an explicit dependency graph through paths, operations, and $ref, so you can calculate the exact dependency closure and give the LLM only what it needs. No embeddings, no similarity threshold, and much less risk of retrieving irrelevant context.
I’d only make one distinction around “LLMs directly executing HTTP calls.” I don’t think an LLM invoking HTTP through a strictly validated tool is inherently problematic. The important boundary is:
LLM decides → deterministic/validated tool executes → real response returns → LLM reasons about the result.
Your ApiTestRunner is essentially a very clean implementation of that boundary anyway.
More broadly, this demonstrates something we’re going to need much more in agent systems: instead of continuously increasing context windows, reduce the context deterministically whenever the domain structure allows it.
Great engineering work — especially for a prototype.
Thank you ❤️