Two weeks ago I had zero dollars, one gaming PC with an RTX 3060 Ti, and a stubborn idea: what if the cost of running a small, useful LLM-backed API could be close to zero, if you just... hosted the model yourself?
Here's what I shipped, and the specific things that broke along the way — because the failures were more instructive than the plan.
The idea
A small API that turns plain English into working code artifacts:
-
/v1/regex— "validate a Brazilian CEP" → a working regex, an explanation, and match/no-match examples -
/v1/sql— "list the 10 customers who bought the most last month" → a SQL query -
/v1/commit-message— a diff description → a conventional commit message -
/v1/json-schema— "an e-commerce product with name, price, category" → a JSON Schema
Nothing revolutionary. The point wasn't the idea — it was proving the economics work when the model is yours.
The stack
-
Ollama running
qwen2.5-coder:7blocally on a Windows box with an RTX 3060 Ti (8GB VRAM — plenty for a 7B coder model at Q4 quantization) -
FastAPI wrapping it, with a JSON-only system prompt and a tolerant parser (models don't always respect "respond with JSON only," so I regex out the first valid
{...}block ifjson.loadsfails on the raw text) - Cloudflare Tunnel (the free "quick tunnel," not a named/paid one) to expose it publicly without port forwarding or a static IP
- RapidAPI as the billing/marketplace layer — free tier + paid tier, they handle subscriptions so I don't have to build that
Total infrastructure cost: $0. The GPU was already sitting there.
What actually broke (the useful part)
Windows process persistence is not what you think. Anything you launch directly in an SSH session — even with Start-Process, even backgrounded — dies the moment the SSH session closes, because Windows ties it to a Job Object scoped to that session. The fix that actually works: wrap the command in a .bat, register it as a Scheduled Task (schtasks /create ... /sc onlogon), and trigger it once immediately with schtasks /run. That survives disconnects.
"Free static domain" isn't always what it says. I tried ngrok's free static domain (a real, permanent feature — not a myth) to get a stable URL instead of Cloudflare's rotating one. It works great in a browser. It's useless for an API: ngrok's free tier shows a mandatory interstitial warning page to any request that doesn't send a specific ngrok-skip-browser-warning header — and a marketplace proxy calling your API on a customer's behalf will never send that header. Every single API call returns an HTML warning page instead of JSON. I found this by testing the raw endpoint externally with no special headers before trusting the "solved" checkbox — which is the actual lesson: test the exact path a real client will take, not the happy path you control.
LLMs don't reliably return valid JSON, even when told to. The fix wasn't a smarter prompt — it was a tolerant extractor: try json.loads on the raw text, then try pulling a fenced
block, then fall back to a brace-matching regex. Three tries, cheap, and it turned "the model sometimes wraps JSON in markdown" from a 502 error into a non-issue.
**A brand-new marketplace listing needs its own free tier.** I gated the public demo (`/demo/*`) with a simple in-memory per-IP daily counter (5 free tries/day) instead of requiring signup at all for the landing page. Letting people try before they subscribe converts better than a wall.
## The economics, honestly
RapidAPI takes 25% of marketplace revenue, plus a small payout processing fee. That's real, and it means the math on a $9.99/month plan is closer to $7 net than $10. Worth knowing before you price.
## What's next
The backend is done. The actual bottleneck now is distribution — a working API with zero users doesn't pay rent. If you're building something adjacent (dev tools, LLM-backed APIs, or you're just curious about the self-hosted-model economics), I'd genuinely like to hear what you're seeing.
**API listing:** [https://rapidapi.com/JoaoPauloNA/api/plain-english-to-code-api](https://rapidapi.com/JoaoPauloNA/api/plain-english-to-code-api) (free tier, no credit card to try the demo)
---
*Building this in public as I go — happy to answer questions about the self-hosted LLM economics, the FastAPI/Ollama integration, or the Windows automation quirks above.*
Top comments (0)