DEV Community: Simangaliso Vilakazi

How I Replaced Gemini with a Self-Hosted LLM for Two Production Apps

Simangaliso Vilakazi — Sat, 27 Jun 2026 13:56:38 +0000

A while back I wrote about my terminal-inspired portfolio and the products it indexes. Two of those products lean on a language model: the portfolio terminal at smngvlkz.com that you can ask questions, and PayChasers, which generates OPTIONAL payment follow-up emails. Both started on Google's Gemini 3 Flash. Both now run on a model I host myself, with a fallback chain that keeps them alive when my hardware is not.

This is the story of that move. The experiment that started it, why I committed to it, what the architecture looks like, the night it broke, and the parts I still have not solved.

It started as an experiment

When Qwen 3.5 was announced, it made me curious about how far open models have actually come. Instead of reading benchmarks, I tested it the way I like to learn things, by running it.

It began as a small experiment on my base Mac mini. I pulled Qwen through Ollama just to see how capable the model would be running directly on a local machine. The results were far better than I expected. Good enough that I stopped thinking of it as a toy and started thinking about production.

Why move off Gemini at all

Gemini 3 Flash worked. The integration was a few lines and the quality was good. So this was not a "the API is bad" story. It was three smaller pulls that added up.

The first was cost shape. PayChasers generates optional email drafts on demand, and every preview is a few thousand tokens of system prompt plus output. That is fine at zero users and a slow leak at volume. The marginal cost of an inference I run on a machine I already own is electricity.

The second was control and privacy. I wanted to choose the model, pin it, and change the prompt contract without a provider deprecating something underneath me. I also did not love sending client names and payment context to a third party when I did not have to.

The third was the economics of treating AI as infrastructure rather than a metered API. Once the model runs on hardware I control, it stops being a per-call expense and becomes shared infrastructure that multiple applications can use. The same inference server now powers two different products. That reframing is the whole point.

Getting it to production was the hard part

The original plan was to host the model on Oracle Cloud using one of their free Ampere ARM instances in the Johannesburg region. If you have ever tried to get one, you know the struggle. Free tier ARM capacity is brutally limited, and after more than 200 automated retry attempts across two days, I still could not get one.

So I pivoted. I wrote a lightweight reverse proxy, set up a Cloudflare Tunnel on one of my domains, and routed production traffic to the model running on my Mac at home. No ports opened on my home network, no static IP, just a tunnel from Cloudflare's edge to the machine on my desk.

There is an honest tension here worth naming. Privacy was one of my reasons for leaving Gemini, and routing client data to a box on my desk is its own tradeoff. The tunnel keeps inbound ports closed and lets Cloudflare terminate TLS at the edge, and the reverse proxy sits in front of Ollama rather than exposing it directly. But this is a single machine I own, not a vendor's hardened, multi-tenant platform, and tightening access control on that endpoint, a service token rather than relying on an obscure hostname, is firmly on the list. Self-hosting moves the privacy boundary onto you, it does not remove it.

It was meant to be temporary. The Oracle instance eventually did come through, but by then the home setup was working well, so I did not throw it away. Instead I kept the Mac mini as the primary and gave Oracle a different job, the always-on backup. More on that in a moment.

This was a small full-circle moment. The Linux and infrastructure fundamentals I picked up during my bootcamp days and years of self-teaching showed up in a real production context. Provisioning tunnels, configuring DNS, writing a proxy service, setting up persistent services. All of it coming together for something real.

One deliberate decision was to keep the infrastructure simple. There are a lot of frameworks and agent systems appearing in the space right now. I focused on straightforward tooling that solved the problems I actually had.

The shape of the system

The Mac mini, exposed through Cloudflare tunnel, is the primary. It is fast but it is not always on, because it is a machine in my home. The Oracle Cloud VM is the fallback. It is slower and smaller, but it stays up around the clock.

Every app talks to a thin client that knows about both, tries the fast one first, and silently falls back to the reliable one.

Vercel app
   |
   v
[ primary: Mac mini via Cloudflare tunnel ]  --fail/timeout-->  [ fallback: Oracle Cloud VM ]
        fast, not always on                                          slow, always on

The failover client

This is the whole idea in one function. Hit the primary with a timeout. If anything goes wrong, the status, the timeout, a dropped tunnel, fall through to the fallback.

const PRIMARY_URL = process.env.OLLAMA_PRIMARY_URL || "http://localhost:11434";
const FALLBACK_URL = process.env.OLLAMA_FALLBACK_URL || PRIMARY_URL;

async function fetchWithFallback(path: string, body: object): Promise<Response> {
    try {
        const res = await fetch(`${PRIMARY_URL}${path}`, {
            method: "POST",
            headers: { "Content-Type": "application/json" },
            body: JSON.stringify(body),
            signal: AbortSignal.timeout(15000),
        });
        if (!res.ok) throw new Error(`Primary failed (${res.status})`);
        return res;
    } catch {
        const res = await fetch(`${FALLBACK_URL}${path}`, {
            method: "POST",
            headers: { "Content-Type": "application/json" },
            body: JSON.stringify(body),
        });
        if (!res.ok) {
            const text = await res.text().catch(() => "Unknown error");
            throw new Error(`Ollama request failed (${res.status}): ${text}`);
        }
        return res;
    }
}

A few small choices that matter more than they look:

The primary gets a 15 second timeout, the fallback does not. The thinking was that the fallback's job is to answer at all, so I let it take its time. In practice that means an unbounded fetch, which can hang if Oracle is reachable but wedged. A long timeout would be the more defensible version of the same idea, and I have not added one yet
The catch swallows why the primary failed, no log, no signal. Fine for failing over, bad for diagnosing, and something I would tighten before I called this production hardened.
The fallback URL defaults to the primary, so the same code runs locally with one Ollama instance and no special config.
Failover is transparent. The caller never knows which machine answered.

Load-aware model selection

Running your own models means you also get to decide which model serves which request. I do a very simple version of routing based on how many requests are in flight.

let activeRequests = 0;

function selectModel(): string {
    // 1 request: best quality. 2+: lighter model that handles concurrency.
    return activeRequests > 1 ? FALLBACK_MODEL : PRIMARY_MODEL;
}

The intent is that a single visitor gets qwen3.5:latest, the better model, and the moment requests overlap, new ones drop to qwen2.5-coder:7b, which is lighter under concurrency. It is one counter and a ternary, the cost and quality trade off in miniature.

I will be honest about how well it actually works, because it is more idea than guarantee. activeRequests is a module-level counter, so on serverless it only sees concurrency inside a single warm instance, not across the fleet. Worse, in the streaming path it is decremented in a finally that runs when the function returns the Response, which is before the stream has finished generating. So for the streaming features, which is most of what these apps do, the counter is near zero almost all the time and the downgrade rarely fires. It works on the non-streaming path, where the count wraps the full call. Right now it is more a hook I reached for early than a load balancer that earns the name.

I also pass two Ollama options that earn their keep:

keep_alive: -1 keeps the model resident in memory so the next request does not pay the cold load.
think: false turns off the reasoning tokens, because for a portfolio terminal and an email draft I want the answer, not the monologue.

Not everything should hit the model

The cheapest inference is the one you never run. Previously my portfolio terminal used Gemini 3 Flash for natural language queries while common commands were handled locally without AI. I kept that split when I moved the natural language layer onto my own infrastructure.

const lowerQuery = query.toLowerCase().trim();

if (lowerQuery === "help") { /* return static command list */ }
if (lowerQuery === "list all") { /* return products + systems from data */ }
if (lowerQuery === "show activity") { /* return GitHub/GitLab stats */ }

const showMatch = lowerQuery.match(/^show\s+([\w-]+)\s+(\w+)$/);
if (showMatch) { /* answer straight from structured data */ }

// only open-ended natural language falls through to the model

help, list, show, and explain are answered straight from the typed data. Only genuinely open-ended questions stream from the model. It is faster, it is free, and it is more reliable than asking a 7B model to format a list it could get wrong.

Streaming the answer

For the open-ended path, the portfolio streams tokens over server-sent events. Ollama returns newline-delimited JSON, so the route reads the body, split on newlines, and re-emits each token as an SSE frame.

const reader = response.body!.getReader();
const decoder = new TextDecoder();
let buffer = "";

while (true) {
    const { done, value } = await reader.read();
    if (done) break;
    buffer += decoder.decode(value, { stream: true });
    const lines = buffer.split("\n");
    buffer = lines.pop() || "";
    for (const line of lines) {
        if (!line.trim()) continue;
        const chunk = JSON.parse(line);
        const token = chunk.message?.content || "";
        if (token) {
            controller.enqueue(encoder.encode(
                `data: ${JSON.stringify({ type: "token", content: token })}\n\n`
            ));
        }
    }
}

Both products stream responses token by token and run entirely on infrastructure I control.

Constraining the model so the output is usable

PayChasers is where the prompt work actually lives, because the output is not a chat bubble, it is an email that gets sent to someone's client. Two things make a self-hosted 7B model reliable enough for that.

First, the model never writes real values. It writes placeholders, and the app fills them in. This keeps the model from hallucinating an amount or a name.

CRITICAL: You MUST use these exact placeholder variables instead of real values:
- {clientName} for the recipient's name
- {dueDate} for the due date
- {amount} for the amount owed
- {daysOverdue} for the number of days overdue

For example: "Hey {clientName}," NOT "Hey John,".
Return ONLY valid JSON.

Second, the tone escalates with how late the payment is, decided in code, not left to the model's mood.

function determineTone(daysOverdue: number) {
    if (daysOverdue >= 14) return "urgent";
    if (daysOverdue >= 7) return "firm";
    return "friendly";
}

And because a local model will occasionally wrap its JSON in a code fence or stray <think> block no matter how firmly you ask, the parser is defensive rather than trusting.

function extractJson(text: string) {
    const cleaned = text.replace(/<think>[\s\S]*?<\/think>/g, "").trim();
    try { return JSON.parse(cleaned); } catch {}

    const fence = cleaned.match(/```(?:json)?\s*([\s\S]*?)```/);
    if (fence) { try { return JSON.parse(fence[1].trim()); } catch {} }

    const first = cleaned.indexOf("{");
    const last = cleaned.lastIndexOf("}");
    if (first !== -1 && last > first) {
        try { return JSON.parse(cleaned.slice(first, last + 1)); } catch {}
    }
}

Self-hosting a smaller model means you trade some of the provider's polish for parsing your own. That is a fair trade when the upside is control and cost.

The night the power went out

Then I learned the lesson that every self-hoster learns eventually.

There was a small power outage one night around 20:00. The Mac mini, my primary inference node, switched off, and it never came back on. I only realised the next morning.

PayChasers failed over to the Oracle backup automatically, exactly as it should have. But the floating terminal in my portfolio had no failover, so it just sat there dead all night. Anyone who was bored enough to try and poke at my portfolio that night got nothing.

Two lessons came out of that morning:

Every service that needs inference needs a failover. Not just the ones I remembered to set it up for. The portfolio terminal got the same fetchWithFallback client that PayChasers already had.
A 12-hour outage I did not even notice is a monitoring problem, not just me being forgetful. Mostly. Partly forgetful.

Self-hosting your own AI is great until you are the one on call at 8am on a Saturday, and there is no one else to escalate to, because it is your own thing

Knowing when the homelab is down

So I built the monitoring I should have had first. PayChasers runs a small cron that health-checks both Ollama endpoints and emails me, but only on state transition, up to down or down to up. It keeps the last known state in Upstash Redis so it does not spam me every 5 minutes while the mini is asleep.

const ENDPOINTS = [
    { name: "primary-mac", url: process.env.OLLAMA_PRIMARY_URL },
    { name: "fallback-oracle", url: process.env.OLLAMA_FALLBACK_URL },
];

// hit /api/tags on each, compare to stored state in Redis,
// send a Resend email only when ok flips. Auth via CRON_SECRET.

Now when the mini goes offline, traffic quietly shifts to Oracle and I get exactly one email telling me so. That is the entire operations story, and that is the amount of operations story I want for a side project.

What I have not solved

I want to be honest about the edges, because the architecture above is the easy part.

My evaluation is still vibes. I read the generated emails, decide they look good, and ship. I do not have an eval harness scoring tone, placeholder correctness, or JSON validity across a fixed set of cases. I should. When I claim qwen3.5 is "better" than qwen2.5-coder for a request, that is intuition, not a benchmark.

The irony is that the plumbing is already there. PayChasers runs PostHog for the product funnel, signups, chases created, upgrades. Capturing AI events would be trivial. A draft_generated, draft_accepted, draft_edited, draft_regenerated funnel would tell me, with real users, how often a generated email ships untouched versus gets rewritten. That acceptance rate is a real quality signal, and it is the cheapest first step from vibes towards measurement. I just have not wired it yet.

My model selection is instinct, not measurement. I picked these Qwen models because they ran well on my hardware and read well in practice. A systematic version would measure latency, quality, and cost per model and route on data.

And I have not touched retrieval. Both apps stuff their full context into the system prompt, which is fine at this size and would fall apart the moment the data outgrew the window. There is no RAG here, and I have not yet had to reach for it.

I am pointing at these on purpose. The move off Gemini taught me serving, the cost and reliability tradeoff, basic routing, and prompt constraining by doing them. The next layer, real evaluation and measured model choice, is the part I am learning now.

What it adds up to

Open models have come a long way. It is becoming genuinely practical to run useful AI systems on relatively small infrastructure. No GPU cluster required. What started as a small experiment on a base mini is now live for real users across two products, on infrastructure I own.

This is not a finished system. It is a snapshot of how I run a model I control today, and a map of what I am building next

I built a Zapier integration for my solo SaaS. Here's every decision I made

Simangaliso Vilakazi — Mon, 06 Apr 2026 11:05:33 +0000

I am a full-stack software engineer in Durban, South Africa. I work full-time and build SaaS products after hours. PayChasers is one of them. It's a payment chasing tool for anyone who's owed money and tired of asking twice.

Today I submitted it to Zapier's partner program. Here's exactly how it went and what I would do differently.

What PayChasers does

PayChasers handles chasing payment reminders for invoices, rent, deposits, failed payments, and personal debts — anything where someone owes you money and you'd rather not send awkward reminders yourself.

You add a chase. You choose how to handle it. Send a follow-up manually when you're ready, or turn on Auto Chase and let PayChasers handle the escalation for you: friendly nudge, due today, overdue, firm. Email or WhatsApp. The tone sharpens as the payment ages. You stay in control of every message. Nothing goes out without your templates, your tone, your timing.

It works. But the product lives in isolation. If someone tracks invoices in Google Sheets, manages deals in HubSpot, or processes payments through Stripe, they have to manually create chases in PayChasers. That's friction. Friction kills adoption.

So I built a Zapier integration.

Why Zapier and not direct integrations

I considered building native integrations. A Stripe webhook listener, a Google Sheets sync, a HubSpot connector. Each one would take days. Each one adds a maintenance surface. Each one serves one app.

Zapier serves 7000+ apps with one integration.

I built one API surface and let Zapier be the bridge. A user who wants "Hubspot deal closes -> create a chase in PayChasers" can build that in 2 minutes without me writing a single line of HubSpot code.

The trade-off is control. I can't fine-tune each experience. But for a solo engineer, the leverage is hard to beat.

What the integration exposes

Triggers (things that happen in PayChasers)

New Chase - fires when a user creates a chase
Chase Overdue - fires when a chase passes its due date
Chase Paid - fires when a chase is marked as paid
Follow-up Sent - fires when an automated follow-up email goes out

Actions (things Zapier can do in PayChasers)

Create Chase - create a new chase with client details, amount, due date, currency, and chase type
Send Follow-up - trigger a follow-up on an existing chase
Mark as Paid - mark a chase as paid

Auth

API key authentication. Users generate a key in Settings, paste it into Zapier. Simple, no OAuth dance. The key is hashed with SHA-256 before storage. I never see the raw key after it's generated.

The API layer

I built a /api/v1/ REST API specifically for this. It's separate from the internal app routes. Endpoints:

GET    /api/v1/me                   — verify auth
GET    /api/v1/chases               — list chases
POST   /api/v1/chases               — create a chase
POST   /api/v1/chases/:id/paid      — mark paid
POST   /api/v1/chases/:id/follow-up — send follow-up
GET    /api/v1/clients              — list clients
GET    /api/v1/triggers/*           — polling triggers for Zapier

The triggers use Zapier's polling model. Zapier hits the trigger endpoint every few minutes, and I return items sorted by most recent. Zapier deduplicates by ID.

I considered webhooks (Zapier's REST Hook model) but polling was simpler to implement and doesn't require me to manage subscription state. For a product at my scale, polling every few minutes is fine and I dont see any issues.

The embed strategy

Zapier offers a "Full Zapier Experience" embed. A JavaScript widget that lets users discover and build "Zaps" directly inside your app. I built the page for it but designed it with two states:

Before approval: An informational page explaining whats coming. no dead UI, no broken embed. Just a clear explanation of what Zapier automation enables and what it will look like.

After approval: Set one environment variable (NEXT_PUBLIC_ZAPIER_FZE_CLIENT_ID), ready to redeploy and the embed loads. Users can browse Zap templates and build automations without leaving PayChasers.

Zero code changes needed when the switch flips. One env var.

Ref-based signup attribution

This is the part I almost skipped and I am glad I didn't.

When someone clicks through from a Zap template to sign up, I wanted to know which template drove them. So the signup page accepts a ?ref= parameter:

/signup?ref=zapier-stripe → "Chase failed Stripe payments automatically."
/signup?ref=zapier-sheets → "Turn your spreadsheet into a payment chasing machine."
/signup?ref=zapier-hubspot → "Close the deal. Chase the payment. Automatically."
/signup?ref=zapier-forms → "From form submission to follow-up in seconds."
/signup?ref=zapier → "Automate your payment follow-ups."

Each ref swaps the hero headline and subtitle on the signup page. The person sees copy that matches their intent.

The ref is also stored on the user record in the database as signupRef. After 30 days of beinfg live I can run:

SELECT signupRef, COUNT(*) FROM "User"
WHERE signupRef IS NOT NULL
GROUP BY signupRef
ORDER BY count DESC;

That tells me which integration is driving signups, which use case resonates, and where to double down in listing copy. Without it, I would know signups are happening but not why.

One column. Massive signal.

The partner review process

Submitting to Zapier's partner program is straightforward but not instant. Here's how it went:

Build the integration on Zapier's developer platform
Submit for review. Their team assigns a reviewer.
Compliance check. The reviewer checked my homepage, terms of service, and privacy policy to confirm the country of operation. He couldn't find it. Fair, it wasn't there.
Fix and respond. I added "South Africa" to my Terms of Service, Privacy Policy, and Contact page within the hour. Updated the Governing Law clause to reference South African law.
Wait for Beta. After passing review, the integration moves to Beta in the Zapier app directory.

The compliance ask caught me off guard but makes sense. They're publishing your app in their directory, they need to know where you operate.

If you're building a Zapier integration, put your country and legal jurisdiction on your site before you submit. Save yourself a round trip.

What I would do differently

Start with the API earlier. The /api/v1/ layer is clean and useful beyond Zapier. It could serve a mobile app, a CLI, or any other client. I should have built it as a first-class citizen from day one instead of bolting it on for Zapier.

Don't overthink trigger architecture. I spent time debating polling vs webhooks. Polling is fine. Ship it. You can add webhooks later if scale demands it.

Add signup attribution from the start. The signupRef field took 10 minutes to add. It should have been there from the first signup form.

Put your legal jurisdiction on your site from day one. Not just for Zapier, for any partnership, integration, or compliance review. It took me an hour to fix. It should have been there from the start.

Current status

PayChasers has paying users across multiple countries. We are early, but the signal is real.

The integration has been submitted and is in review. The embed page is built and waiting. The attribution tracking is live. When the green light comes, it's one env var and a deploy.

If you're an indie hacker thinking about integrations, Zapier is high leverage. One integration surface, thousands of connections. Build the API, submit the integration, track your refs, and let the platform do the distribution.

I am Simangaliso, building PayChasers from South Africa. If you've ever sent a "just following up" email manually, PayChasers was built for you. paychasers.com

A Terminal-Inspired Portfolio of Shipped and Researched Products (2026)

Simangaliso Vilakazi — Fri, 02 Jan 2026 11:47:38 +0000

This is a submission for the New Year, New You Portfolio Challenge Presented by Google AI

About Me

I am a full-stack software engineer based in South Africa, focused on building durable software that ships and lasts.
I care about clear system boundaries, low-maintenance architectures, and products that solve real problems without unnecessary complexity. Most of my work lives in production: SaaS tools, automation systems, payment flows, and community platforms.
This portfolio is meant to reflect how I actually work. Pragmatic, intentional, and biased toward execution.

Portfolio

You can visit it at: https://smngvlkz.com

This portfolio is structured like internal system documentation rather than a traditional marketing site. It highlights:

Live and archived products I have shipped
Clear product intent, status, and scope
Ongoing research and exploratory work
SaaS, automation, payments, and platform work
Open-source and community contributions

AI Integration

Design principle:
AI is used only where it improves inspectability or reduces cognitive load. All core functionality remains deterministic and debuggable without AI.

This portfolio includes an interactive terminal (SYSTEM.QUERY) powered by Google Gemini 3 Flash.

Users can query the entire portfolio data directly:

Structured commands (handled locally for speed):

help - list available commands
list all - show all products and contributions
show activity - show GitHub & GitLab stats, streak, sessions
show [id] [field] - show specific data for any item in the portfolio
explain [id] - show full breakdown for any product or contribution
list fields - show queryable fields

Natural language queries (powered by Gemini) - examples:

"what tech does paychasers use?"
"compare the infrastructure of all products."
"which product uses blockchain?"
"what's the Cape Community Blog built with?"

The system prompt constrains Gemini to portfolio data only, with terminal-style output formatting. Common queries are handled locally without an API call. Complex or natural language queries fall back to Gemini.

This demonstrates practical AI integration: fast where possible, intelligent when needed, without over-engineering.

Live Activity Data (GitHub & GitLab)

SYSTEM.ACTIVITY pulls live data from GitHub and GitLab using authenticated API requests.

A server-side route aggregates commits, repositories, contribution streaks, and session metadata into a unified activity model. This model is used in two places:

Rendered as live system activity stats in the interface, including a 111-day contribution heatmap that visualizes real contributions with interactive tooltips (showing source breakdown: GitHub/GitLab commits, date, and day of week). Only days with actual contributions are inspectable on hover.
Exposed to SYSTEM.QUERY, allowing users to inspect and query the same data via the terminal.

This keeps activity up to date without manual updates and demonstrates real external API integration alongside AI-powered querying, with a single source of truth shared across UI and terminal

How I Built It

Framework: Next.js
Language: Typescript, JavaScript
Styling: Tailwind CSS
AI: Google Gemini 3 Flash
Deployment: Google Cloud Run (fully managed, container-based)

I used AI-assisted development to iterate on structure, copy clarity, and information hierarchy, while keeping the final implementation intentionally simple and deterministic.

The site favors static rendering, fast load times, and a small surface area for long-term maintainability. The terminal feature adds interactivity without compromising the minimal aesthetic.

What I'm Most Proud Of

The restraint in the design. Nothing exists without a reason
Clear labeling of product status (live, inactive, archived)
Showing real shipped work instead of demo projects
AI integration that fits the product's identity (terminal queries, not chatbot fluff)
A portfolio that reflects engineering maturity, not trend-chasing

This is not a highlight reel. It is a snapshot of how I build software today.