I Tested a Local Gemma Model on 12 Support Tickets. It Wasn't Ready - So I Built ModelExit.

Alen Sebastian — Sun, 18 Oct 2026 20:16:06 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle.

A small SaaS company doesn't want to pay for a bigger AI model than it needs. If thousands of support requests are being processed through a paid API, running an open-weight model locally sounds like a straightforward way to reduce that bill.

Until something goes wrong.

A billing ticket is misclassified. A response comes back as broken JSON. Worse, an account-security issue fails to reach a person. The API savings are easy to calculate; the cost of those mistakes isn't.

That is the problem I wanted to work on with ModelExit.

The project doesn't promise to replace an expensive API automatically. It answers a smaller, more useful question: is this particular local model sufficiently reliable for this particular workflow to justify a limited pilot, and what evidence says otherwise?

What I Built

ModelExit is a local-first evaluation tool for structured support-ticket triage. It uses a real open-weight model, runs each case against independently labelled expected answers, and produces a report that someone else can inspect.

A developer can load their own CSV or JSONL dataset, select a model served by LM Studio, and see exactly where it succeeds, fails, or returns unusable output. They can optionally upload saved responses from an existing cloud workflow for a side-by-side comparison. ModelExit never calls a cloud provider itself.

The output is an Exit Passport: a JSON and Markdown record of the model, dataset fingerprint, metrics, failures, and migration-readiness decision.

Where the commercial value could be

The buyer I have in mind is not somebody looking for another LLM leaderboard. It's the independent developer or small SaaS team asking whether an existing AI feature can be made cheaper without introducing an unacceptable operational risk.

Consider three realistic use cases:

A SaaS founder wants to evaluate local inference before routing a portion of customer-support traffic away from a paid provider.
A small development agency wants to offer clients a fixed-scope AI migration assessment, backed by a repeatable test and an evidence report rather than a confident recommendation.
An operations lead needs something concrete to show colleagues: which ticket types failed, which require human review, and whether the suggested change merits a pilot at all.

The potential side hustle is an AI migration preflight service: collect a representative labelled workload, test candidate local models, document failure cases, and deliver an evidence-based go/no-go recommendation. ModelExit is the tool that makes that service repeatable.

That is a business hypothesis, not a proven market. I haven't sold an assessment or onboarded a paying user. But it is a specific service with a specific deliverable, rather than an abstract promise to “save money with AI.”

There is an important distinction here: lower inference spend is not necessarily lower operating cost. Local hardware, power, maintenance, latency, and the human effort needed to correct wrong decisions still count. ModelExit currently measures quality and latency; it does not claim to calculate a company's actual savings.

How I Built It

The application is Python + Streamlit, with Pydantic for validated data contracts and httpx for local model calls. Inference goes through LM Studio's OpenAI-compatible endpoint on localhost.

I tested it with [google/gemma-4-12b-qat] running locally. No paid AI API or API key is needed.

For each support ticket, Gemma is asked to return four fields:

{
  "category": "account",
  "priority": "high",
  "needs_human": true,
  "order_id": null
}

The correct labels are not shown to the model. A deterministic evaluator compares the prediction against the pre-labelled answer. A failed inference or malformed response counts as failure, rather than being quietly retried until the score improves.

The scoring reports full-field exact matches, per-field accuracy, high-priority and human-review recall, invalid responses, latency, and critical-case misses. A curated Failure Frontier groups original and altered ticket pairs so that a change in wording can be inspected directly. It is a curated set of cases, not autonomous discovery of every possible failure.

The decision policy is deliberately strict: READY FOR LIMITED PILOT requires at least 10 evaluated cases, at least 85% full-field exact match, no invalid outputs, and no critical misses. A critical miss forces NOT READY. Other outcomes can lead to NEEDS REVIEW or INSUFFICIENT EVIDENCE.

Those thresholds are transparent MVP rules, not calibrated production-safety standards. A serious deployment would require far more representative data and human oversight.

The Experiment: A Good-Looking Score Can Hide the Wrong Failure

I ran a final end-to-end check with 12 synthetic support tickets, with gold labels fixed before inference. The model was real; the workload was synthetic. Most cases in this final fixture were not complete original/changed pairs, so I treat this run as a comparison and integration check, not a complete Failure Frontier study. The recorded outputs and full report are in the public validation folder.

Gemma completed 11 of 12 calls and matched all four fields on 7 of 12 tickets (58.3%). It correctly handled 4 of 4 predefined critical cases, but returned one unusable output: the model exhausted its generation budget on internal reasoning and produced no final JSON.

The result was NEEDS REVIEW, not a green light for migration.

I also tested the new Cloud vs Local Comparison with an intentionally mixed, synthetic baseline. Both sides scored 7/12 exact matches, but the simulated baseline missed two critical cases while Gemma missed none.

That result demonstrates how the comparison feature behaves, not that Gemma is safer than GPT, Claude, or any actual cloud provider. The baseline outputs and their latency values were invented for a controlled functional check. No cloud API was called, and the chart is not a real-world model benchmark.

The point is about the decision, not declaring a winner. Two systems can post the same overall score while failing in different places. For a support workflow, those places matter. A ticket that requires human escalation has a different operational consequence from a harmless category mismatch.

ModelExit does not conceal those differences behind one average. It shows the errors case by case and keeps its local readiness decision independent of the optional baseline.

Demo

The demo follows the actual workflow: load labelled tickets → select Gemma in LM Studio → run local evaluation → inspect the optional baseline comparison → export the Exit Passport.

If you'd rather inspect the recorded data first, the final experiment includes the labelled cases, preserved Gemma outputs, a clearly marked synthetic baseline, a comparison summary, and both exported Passport formats.

Code

Repository: github.com/Bolshoi777/ModelExit · License: MIT

It includes a Windows quick start, the Streamlit interface, a CLI runner, example datasets, deterministic tests, and recorded experiment evidence. It doesn't download model weights automatically; you choose and run an open-weight model through LM Studio.

A Report Instead of a Claim

The part I'm most interested in isn't the dashboard. It's what gets handed to somebody making a deployment decision.

An Exit Passport records the actual model ID, run settings, dataset SHA-256, denominators for every metric, individual failures, and the reasons for the final status. It can be inspected later, shared with a teammate, or used as a baseline for another candidate model.

That has commercial value because an assessment needs to be reviewable, not just plausible. A customer evaluating an AI migration should be able to ask “Which exact cases failed, and what did you measure?” and get an answer.

It also creates room for a service business around the tool: test-data preparation, running a fixed-scope assessment, interpreting failures, and planning a limited pilot with a human fallback. None of that requires selling another monthly AI subscription.

Why Does Open Innovation Matter?

A locally served open-weight model changes who controls the experiment.

I can choose which model to load, run repeated tests without a per-request cloud API bill, keep the evaluation workload on my own machine, and inspect every response. I can swap the inference model without rewriting the entire evaluator. And because the evaluation code is published, the pass/fail rules and readiness thresholds are not a black box.

This matters for small teams that can't justify building a proprietary evaluation platform just to investigate a migration. It also matters for teams working with internal tickets that shouldn't be sent to a third-party AI service solely for testing.

“Local” doesn't mean free: compute, hardware, maintenance, and data-handling responsibilities remain. But openness makes the trade-offs measurable and the setup easier to adapt.

What This Prototype Cannot Tell You Yet

Twelve synthetic examples are nowhere near enough to approve a real production migration. The current evaluator covers one narrowly defined JSON classification workflow, not arbitrary AI applications. The optional cloud baseline in the published experiment is simulated, and I haven't measured an actual commercial provider's costs or responses. Nor have I validated customer willingness to pay.

Those are not details to hide in fine print; they're the next work to do.

The logical next step is to test a larger, consented real-world workload against genuinely saved cloud responses, then add configurable business-risk weights and operating-cost assumptions. Only then would a real migration assessment begin to justify a commercial recommendation.

Prize Categories

Best Use of Gemma - Gemma is the real open-weight inference model in the recorded local experiment. ModelExit evaluates its structured outputs, exposes its failure cases, and produces an auditable readiness report.

I built ModelExit because “the smaller model seems good enough” is not a migration plan. A better plan starts with a dataset, visible failures, and a decision you can defend.

If you maintain an AI feature in a small product: what would you need to see before moving even 10% of its traffic to a local model?

Hey DEV! I'm Alen 👋

Alen Sebastian — Mon, 12 Oct 2026 21:51:10 +0000

I'm Alen. I mostly work with Python, automation, AI agents, and Web3.

I like building my own tools, experimenting with new ideas, and figuring out how things work.

Currently working on a few open-source projects. Always open to interesting collaborations and freelance work.

DEV Community: Alen Sebastian