This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady
🦥 SlothSwitch: We're making AI agents smarter. I'm trying to make them cheaper.
What if the smartest thing an AI agent could do was skip an unnecessary LLM call?
As AI agents become more capable, they also introduce a growing inference cost. Many workflows repeatedly invoke generative models for tasks that could be handled through structured decisions and direct tool execution.
Consider a simple request: List the open pull requests in a GitHub repository.
Does an agent really need a generative LLM to decide which tool to call?
I built SlothSwitch, an open-source decision and execution layer for agentic workflows, to explore a different approach.
Its primary objective is simple: minimize LLM inference costs as much as possible by avoiding unnecessary generative calls while preserving task reliability, policy enforcement, and execution visibility.
Instead of sending every request through a generative LLM, SlothSwitch uses a decision model to select appropriate structured actions and reserves generative inference for tasks that actually need it.
The goal isn't to eliminate LLMs. It's to stop paying for inference that a task doesn't need.
What I Built
SlothSwitch sits between incoming user requests and the tools an AI agent can use. It determines how a request should be handled, checks the proposed action, and routes execution through an appropriate path.
The system combines four key components:
- Laya — Decision Model: Selects an action from a defined workflow schema, helping route structured requests without automatically invoking a generative LLM.
- Policy Engine: Evaluates whether a proposed action is permitted before execution.
- GitHub Tool Integration: Executes supported repository operations.
- Gemma — Generative LLM: Handles tasks requiring language generation, such as producing natural-language summaries, using local inference.
- Execution Traces: Provide visibility into decisions, model usage, and execution outcomes.
How SlothSwitch Works
The intended workflow is:
- A user submits a natural-language request.
- Laya selects an appropriate structured action.
- The policy engine evaluates the proposed action.
- If authorized, the corresponding tool executes the operation.
- If the task requires generative language capabilities, Gemma handles the generation step.
- Execution details are recorded for inspection and evaluation.
For example, retrieving a list of open pull requests can follow a structured tool-execution path, whereas summarizing changes across several pull requests can require Gemma's generative capabilities.
This distinction is central to SlothSwitch: use decision models for decisions, tools for actions, and generative LLMs when generation adds value.
The system is also designed to support clarification, abstention, and human approval where appropriate.
Demo
🎬 Watch the 90-second SlothSwitch demo:
[https://youtu.be/eM3XmrOOK5M]
The demo shows the actual application and its supported execution workflow, illustrating how SlothSwitch processes requests and exposes execution behavior.
The objective is to make AI workflows more cost-conscious by avoiding unnecessary generative inference instead of automatically invoking an LLM for every request.
Code
💻 GitHub Repository: [https://github.com/dushyantzz/Slothswitch.git]
SlothSwitch is an open-source experiment in reducing unnecessary LLM inference costs through decision-based routing and structured tool execution.
The project explores how decision models, policy checks, local generative inference, and execution traces can work together in agentic workflows.
Contributions and feedback are welcome, particularly around decision accuracy, inference efficiency, evaluation methodology, and AI agent security.
How I Built It
I built SlothSwitch around a simple principle:
Decision-making and language generation do not always need to be performed by the same model.
1. Laya for Decision-Making
Laya is used as the decision model. It selects an action from a defined workflow schema, allowing supported structured requests to be routed toward the appropriate tool action.
The objective is to avoid invoking a generative LLM when a structured decision is sufficient.
2. Gemma for Generative Tasks
Gemma is the generative LLM used for tasks requiring language generation. It can run locally, giving the system a local inference option rather than requiring every generation task to depend on a hosted model API.
SlothSwitch does not treat Gemma and Laya as interchangeable components: Laya handles action selection, while Gemma handles generative language tasks.
3. Policy-Governed Tool Execution
After an action is selected, the policy engine evaluates whether it is permitted. Authorized actions can then be executed through the relevant tool integration, such as GitHub.
This provides an execution boundary between selecting an action and carrying it out.
4. Evaluation and Measurement
I evaluated SlothSwitch on a 48-example test set to assess action selection, outcome accuracy, task completion, and generative-call reduction.
The currently reported results are:
| Evaluation Metric | SlothSwitch Result |
|---|---|
| Evaluation examples | 48 |
| Strict action accuracy | 44/48 — 91.7% |
| Outcome accuracy | 45/48 — 93.8% |
| End-to-end task success | 41/48 — 85.4% |
| Generative calls made | 6/48 — 12.5% |
| Generative calls avoided | 42/48 — 87.5% |
| Successful direct tool calls | 29/29 — 100% |
The direct-tool result refers to the 29 direct tool calls reported in the evaluation and should not be interpreted as a universal success rate.
Baseline Comparison
The reported evaluation also compared SlothSwitch against an always-generative baseline and a deterministic-rules baseline.
| Metric | Always-Generative Baseline | Deterministic Rules | SlothSwitch |
|---|---|---|---|
| Strict action accuracy | 0/48 (0%) | 30/48 (62.5%) | 44/48 (91.7%) |
| Outcome accuracy | 6/48 (12.5%) | 39/48 (81.2%) | 45/48 (93.8%) |
| End-to-end task success | 6/48 (12.5%) | 27/48 (56.2%) | 41/48 (85.4%) |
| Generative calls | 48 | 6 | 6 |
| Generative calls avoided vs. always-generative | 0% | 87.5% | 87.5% |
These are the currently reported benchmark results. The baseline model identities, evaluation configuration, and underlying artifacts should be verified before publication.
What Do These Results Mean for LLM Costs?
SlothSwitch reportedly avoided 42 out of 48 generative calls, an 87.5% reduction in generative-call count compared with the always-generative baseline.
This is important because generative inference can be a significant operating expense in applications that process large numbers of requests.
However, reducing call count does not automatically translate into an identical percentage reduction in monetary cost. Actual savings depend on model pricing, input and output tokens, local compute costs, decision-model overhead, and the workload being evaluated.
Laya also introduces inference overhead, so SlothSwitch does not claim to be faster or cheaper on every individual request.
The objective is to minimize unnecessary generative inference and improve overall cost efficiency while preserving successful task execution.
Why Does Open Innovation Matter?
Reducing LLM costs should not require every developer to build a proprietary routing system from scratch.
Open-source tools and open-weight models make it possible to experiment with alternative architectures, inspect model behavior, and adapt AI infrastructure to specific workloads.
Gemma provides an opportunity to run generative inference locally, while Laya explores the use of a decision model for structured action selection.
Open implementations also make it easier to investigate failures, reproduce evaluations, improve policies, and collaborate on more efficient AI systems.
For SlothSwitch, transparency matters because developers should be able to understand which action was selected, whether it was authorized, which execution path was used, and whether generative inference was necessary.
Local Gemma inference does not make the entire workflow offline: GitHub operations still require network access and appropriate authorization.
My goal is to make cost-conscious AI execution easier to experiment with, evaluate, and improve through open collaboration.
My Agent Session
The Lazy Payoff
The main motivation behind SlothSwitch is to push unnecessary LLM inference costs as low as practically possible.
Every generative call should serve a purpose. If a task can be completed through a structured decision and an authorized tool, invoking a generative LLM may add unnecessary expense and latency.
On the current 48-example evaluation set, SlothSwitch reported:
- 87.5% fewer generative calls than the always-generative baseline.
- 91.7% strict action accuracy.
- 93.8% outcome accuracy.
- 85.4% end-to-end task success.
These results provide an initial basis for exploring decision-based routing as a way to reduce unnecessary inference in structured agentic workflows.
The next steps are to improve decision accuracy, expand the evaluation dataset, and measure actual monetary savings across different models, workloads, and token volumes.
The goal is not to make AI agents less capable. It's to stop spending generative inference on work that doesn't need it, while retaining generative models for tasks where they provide genuine value.
Don't generate everything. Don't pay for unnecessary inference. Let the task determine the execution path.
That's SlothSwitch. 🦥
Prize Categories
Best Use of Gemma — provided the submitted implementation meaningfully uses Gemma in accordance with the challenge requirements.
SlothSwitch is submitted for the Slow and Steady challenge because it explores how open-source decision models, local Gemma inference, and structured tool execution can help reduce unnecessary generative inference costs.
Top comments (0)