DEV Community: ömer faruk aydın

Run any LLM locally with VRAM-aware routing - a llama.cpp + FastAPI deep dive

ömer faruk aydın — Thu, 20 Aug 2026 08:28:46 +0000

Cloud LLMs are amazing - until the bill arrives. Or until you need to process something you can't send to a third party. Or until you hit a rate limit at 3am.

This is the architecture I built to run any GGUF model locally with a drop-in OpenAI-compatible API, and the cool part: it picks the right model for your GPU automatically.

The setup

llama.cpp for inference (compiled with CUDA or Metal)
FastAPI for the server (OpenAI-compatible endpoints)
GGUF models in a local directory
NVML (NVIDIA) or Metal (Apple) for VRAM detection

User app (uses openai SDK)
   v
http://localhost:8080/v1/chat/completions
   v
FastAPI server
   +- VRAM router (picks model by query)
   +- llama.cpp subprocess (loads GGUF)
   +- Stream tokens back
   v
Your GPU

Step 1: Get the models

Pull GGUF models from HuggingFace. I keep a few sizes around for different hardware:

# 7B model - fits in 8GB VRAM
huggingface-cli download TheBloke/Llama-2-7B-Chat-GGUF llama-2-7b-chat.Q4_K_M.gguf

# 13B model - needs 12GB
huggingface-cli download TheBloke/Llama-2-13B-Chat-GGUF llama-2-13b-chat.Q4_K_M.gguf

# 70B model - needs 40GB (or 24GB with Q3)
huggingface-cli download TheBloke/Llama-2-70B-Chat-GGUF llama-2-70b-chat.Q3_K_M.gguf

mkdir -p ~/models
mv *.gguf ~/models/

Step 2: The server skeleton

# server.py
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from typing import List, Optional
import subprocess, json, asyncio

app = FastAPI(title="Local LLM Server")

class Message(BaseModel):
    role: str
    content: str

class ChatRequest(BaseModel):
    model: str = "auto"  # < magic
    messages: List[Message]
    max_tokens: int = 512
    temperature: float = 0.7
    stream: bool = False

class ChatResponse(BaseModel):
    id: str
    object: str = "chat.completion"
    model: str
    choices: list
    usage: dict

Step 3: The VRAM-aware router

This is the magic. The router picks the right model based on:

Available VRAM
Requested max_tokens
Whether the model is already loaded

# router.py
import pynvml

def get_available_vram_mb() -> int:
    """Returns free VRAM in MB. -1 if no NVIDIA GPU."""
    try:
        pynvml.nvmlInit()
        handle = pynvml.nvmlDeviceGetHandleByIndex(0)
        info = pynvml.nvmlDeviceGetMemoryInfo(handle)
        return info.free // 1024 // 1024
    except Exception:
        return -1  # CPU-only or Apple Silicon

# Model VRAM requirements (rough, for Q4_K_M quantization)
MODELS = {
    "llama-2-7b":   {"file": "llama-2-7b-chat.Q4_K_M.gguf",   "vram_mb": 6000},
    "llama-2-13b":  {"file": "llama-2-13b-chat.Q4_K_M.gguf",  "vram_mb": 11000},
    "llama-2-70b":  {"file": "llama-2-70b-chat.Q3_K_M.gguf",   "vram_mb": 28000},
    "mistral-7b":   {"file": "mistral-7b-instruct.Q4_K_M.gguf","vram_mb": 6000},
    "mixtral-8x7b": {"file": "mixtral-8x7b-instruct.Q4_K_M.gguf","vram_mb": 30000},
}

def pick_model(requested: str, max_tokens: int) -> str:
    free_vram = get_available_vram_mb()
    needed = max_tokens * 2  # rough KV cache estimate

    if requested == "auto":
        # Pick the largest model that fits in VRAM
        for name in sorted(MODELS.keys(), key=lambda m: -MODELS[m]["vram_mb"]):
            if MODELS[name]["vram_mb"] + needed < free_vram:
                return name
        return "llama-2-7b"  # fallback to smallest
    return requested

Step 4: Run the inference

I use llama-cpp-python for the inference layer - it has Python bindings to llama.cpp with streaming support:

# inference.py
from llama_cpp import Llama
import asyncio

class ModelPool:
    def __init__(self, model_dir: str = "~/models"):
        self.pool: dict[str, Llama] = {}
        self.model_dir = model_dir

    def get(self, model_name: str) -> Llama:
        if model_name not in self.pool:
            cfg = MODELS[model_name]
            self.pool[model_name] = Llama(
                model_path=f"{self.model_dir}/{cfg['file']}",
                n_ctx=4096,
                n_gpu_layers=-1,  # all layers on GPU
                n_threads=8,
            )
        return self.pool[model_name]

pool = ModelPool()

Step 5: The OpenAI-compatible endpoint

This is where the magic happens - any OpenAI SDK call works:

# app.py
@app.post("/v1/chat/completions")
async def chat_completions(req: ChatRequest):
    model_name = pick_model(req.model, req.max_tokens)
    if model_name not in MODELS:
        raise HTTPException(404, f"Model {model_name} not found")

    llm = pool.get(model_name)

    # Run inference (streaming or batch)
    response = llm.create_chat_completion(
        messages=[m.dict() for m in req.messages],
        max_tokens=req.max_tokens,
        temperature=req.temperature,
        stream=req.stream,
    )

    return {
        "id": f"chatcmpl-{hash(req.messages)}",
        "object": "chat.completion",
        "model": model_name,
        "choices": response["choices"],
        "usage": response["usage"],
    }

Step 6: Use it with the OpenAI SDK

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8080/v1",
    api_key="not-needed",
)

response = client.chat.completions.create(
    model="auto",  # router picks the right model
    messages=[{"role": "user", "content": "Explain VRAM-aware routing"}],
    max_tokens=512,
)
print(response.choices[0].message.content)

Zero code changes to switch from OpenAI to local. This means I can:

Develop locally (no API costs)
Test in CI (deterministic, no rate limits)
Deploy to production (OpenAI for users who want it, local for those who need privacy)

Performance tips I learned the hard way

Don't keep multiple models loaded. Each model eats VRAM for the KV cache. Load on demand, unload after N minutes of idle.
Use Q4_K_M for most cases. Q5 is barely better, Q8 is 2x the size for marginal gains.
Speculative decoding can 2-3x the speed: have a small model draft, the big model verifies.
Mlock the model (use_mlock=True) to prevent swapping - massive speedup on machines with slow disk.
Apple Silicon users: build llama.cpp with Metal support, performance is excellent on M1/M2/M3.

The whole project

I've packaged this into Strata - a desktop app with a chat UI, model browser, and this exact server underneath.

[link] github.com/Omerfaruk-aydn (Strata repo)

The desktop app is built with Tauri 2 + React; the server is this Python + FastAPI + llama.cpp stack.

Originally published on omerfarukaydn.com - more on the desktop UI, model marketplace, and the WebSocket streaming implementation.

Building a true multi-tenant SaaS: PostgreSQL RLS, subdomains, and Next.js middleware

ömer faruk aydın — Thu, 20 Aug 2026 07:52:09 +0000

Most "multi-tenant" SaaS tutorials cheat. They show you WHERE tenant_id = ? in every query and call it done. That's not multi-tenant - that's a bug waiting to happen.

True multi-tenancy means the database enforces isolation. You literally cannot read another tenant's data, even if you write a bug. Here's how I built it with Supabase + PostgreSQL RLS + Next.js middleware.

Why RLS, not app-level filtering?

Consider this "tenant-scoped" query:

// ? WRONG - easy to forget the WHERE clause
const { data } = await supabase
  .from('customers')
  .select('*')
  .eq('tenant_id', currentTenantId);

One missed .eq() in a new feature, and you've leaked data. The bug is silent - no test catches it, no type system warns you.

With Row Level Security, the database refuses the query:

-- ? RIGHT - database enforces it
CREATE POLICY tenant_isolation ON customers
  USING (tenant_id = current_setting('app.current_tenant')::uuid);

Forget the WHERE clause? Doesn't matter. RLS filters it out at the engine level. A bug in your app code can no longer become a data leak.

The 3 layers

DNS + edge middleware - resolve acme.example.com ? tenant_id at the edge
JWT with tenant claim - every request carries the tenant context
PostgreSQL RLS - the actual data gatekeeper

Customer ? acme.example.com
   ?
Vercel Edge Middleware
   ?? Subdomain ? tenant_id lookup (in KV or DB)
   ?? Set request header x-tenant-id
   ?
Next.js Server Action / API Route
   ?? Read x-tenant-id from headers
   ?? Generate Supabase JWT with tenant claim
   ?
Supabase (Postgres)
   ?? RLS policy reads JWT claim, filters every row

Layer 1: Subdomain ? tenant_id

Store tenant metadata in a fast read path (Vercel KV, or a small Postgres table):

// middleware.ts
import { NextResponse, type NextRequest } from 'next/server';

export async function middleware(req: NextRequest) {
  const host = req.headers.get('host') || '';          // acme.example.com
  const subdomain = host.split('.')[0];                // acme

  // Skip on the apex domain
  if (subdomain === 'www' || subdomain === 'example') {
    return NextResponse.next();
  }

  // Lookup tenant (cache in KV in production)
  const tenant = await getTenantBySlug(subdomain);
  if (!tenant) {
    return NextResponse.redirect(new URL('https://example.com/404'));
  }

  // Pass to downstream handlers
  const requestHeaders = new Headers(req.headers);
  requestHeaders.set('x-tenant-id', tenant.id);

  return NextResponse.next({ request: { headers: requestHeaders } });
}

export const config = {
  matcher: ['/((?!_next/static|_next/image|favicon.ico).*)'],
};

Layer 2: JWT with tenant claim

Generate a Supabase JWT that includes the tenant ID as a custom claim:

// lib/supabase.ts
import { createClient } from '@supabase/supabase-js';

export function getSupabaseForRequest(req: Request) {
  const tenantId = req.headers.get('x-tenant-id');
  if (!tenantId) throw new Error('Missing tenant context');

  // Sign a JWT with the tenant claim
  // (In production, do this server-side with the Supabase service role key)
  const token = signJwtWithClaim({
    sub: req.headers.get('x-user-id'),
    tenant_id: tenantId,
    role: 'authenticated',
    exp: Math.floor(Date.now() / 1000) + 60 * 60,
  });

  return createClient(
    process.env.NEXT_PUBLIC_SUPABASE_URL!,
    process.env.NEXT_PUBLIC_SUPABASE_ANON_KEY!,
    { global: { headers: { Authorization: `Bearer ${token}` } } }
  );
}

Layer 3: PostgreSQL Row Level Security

This is the actual lock. Every table that holds tenant data gets a policy:

-- Helper: get current tenant from JWT
CREATE OR REPLACE FUNCTION current_tenant_id()
RETURNS uuid AS $$
  SELECT NULLIF(
    current_setting('request.jwt.claims', true)::json->>'tenant_id',
    ''
  )::uuid;
$$ LANGUAGE sql STABLE;

-- Enable RLS on every tenant-scoped table
ALTER TABLE customers ENABLE ROW LEVEL SECURITY;
ALTER TABLE conversations ENABLE ROW LEVEL SECURITY;
ALTER TABLE messages ENABLE ROW LEVEL SECURITY;

-- Policy: users can only see rows from their tenant
CREATE POLICY tenant_isolation ON customers
  FOR ALL
  USING (tenant_id = current_tenant_id())
  WITH CHECK (tenant_id = current_tenant_id());

CREATE POLICY tenant_isolation ON conversations
  FOR ALL
  USING (tenant_id = current_tenant_id())
  WITH CHECK (tenant_id = current_tenant_id());

CREATE POLICY tenant_isolation ON messages
  FOR ALL
  USING (tenant_id = current_tenant_id())
  WITH CHECK (tenant_id = current_tenant_id());

To make this work, Supabase needs to pass the JWT claim into the Postgres session:

-- In Supabase Dashboard: Settings ? API ? JWT Settings
-- Add to "Custom Claims" or use a Postgres function hook
ALTER DATABASE postgres SET request.jwt.claims TO '{"tenant_id":""}';

-- Or, more cleanly, use Supabase's built-in auth.uid() pattern
-- and add tenant_id alongside it

Testing that RLS actually works

This is the part most tutorials skip. Write a test that proves a tenant can't see another tenant's data.

// tests/rls.test.ts
import { getSupabaseForRequest } from '@/lib/supabase';

test('tenant A cannot read tenant B data', async () => {
  // Insert data as tenant A
  const supabaseA = getSupabaseForRequest(mockRequest({ tenantId: 'A' }));
  await supabaseA.from('customers').insert({ name: 'Alice', tenant_id: 'A' });

  // Try to read as tenant B
  const supabaseB = getSupabaseForRequest(mockRequest({ tenantId: 'B' }));
  const { data, error } = await supabaseB.from('customers').select('*');

  expect(data).toEqual([]);  // empty - RLS filtered it out
});

I learned this the hard way. My first RLS deployment had a bug where the JWT claim wasn't being passed correctly. Without the test, I would have shipped a cross-tenant data leak.

The pitfalls

Forgetting to enable RLS on new tables. I have a CI check: any new CREATE TABLE without ENABLE ROW LEVEL SECURITY fails the build.
Service role bypasses RLS. The Supabase service role key has BYPASSRLS. Only use it server-side for admin tasks, never pass it to the client.
Bulk operations need explicit tenant_id. RLS doesn't add a default value; you have to pass it. I use a BEFORE INSERT trigger to set it from the JWT.
Migrations are dangerous. When you add RLS to an existing table, you must do it in a transaction with the right role checks, or you can lock yourself out.

Get the code

The full reference implementation (Next.js + Supabase + edge middleware + RLS policies + test suite) is open source:

?? github.com/Omerfaruk-aydn/crmhizmetbotu

It's the production architecture of a live SaaS handling thousands of conversations per month for multiple businesses - all on a single Postgres database with zero data leaks.

Originally published on omerfarukaydn.com - more details on the subdomain wildcard DNS, Vercel KV caching, and the WebSocket multi-tenant story.

How I built a crypto signal generator that beat fixed-weight strategies by 37% Sharpe

ömer faruk aydın — Thu, 20 Aug 2026 07:52:06 +0000

I spent 3 months running a fixed-weight RSI/MACD strategy on Binance. Backtested great, lost money in production. The reason: markets change regimes, and static rules don't.

This is the story of how I built a self-tuning signal generator with XGBoost and Bayesian optimization - and why it actually worked.

The problem with fixed rules

My original strategy:

Buy when RSI(14) < 30
Sell when RSI(14) > 70
Add MACD as confirmation

Backtested over 18 months of BTCUSDT 5-min data: 1.42 Sharpe ratio. Looked great. Then I ran it live for 3 months: 0.61 Sharpe. Why?

Because in some months, RSI was the right signal. In others, MACD dominated. In trending markets, neither - the trend was the signal. The indicator that "worked" changed every month, and I had no way to know in advance.

The fix: let the model figure it out

Instead of hard-coding rules, I:

Compute 13 indicators at every timestep
Feed them all to an XGBoost classifier
Use Bayesian optimization to tune the hyperparameters
Retrain weekly on a rolling window

Result: 1.85 Sharpe ratio, 30% higher than the fixed-weight baseline.

The data pipeline

import ccxt
import pandas as pd
import numpy as np

exchange = ccxt.binance()
ohlcv = exchange.fetch_ohlcv('BTC/USDT', '5m', limit=50000)
df = pd.DataFrame(ohlcv, columns=['ts', 'open', 'high', 'low', 'close', 'volume'])

# 13 indicators
def add_features(df):
    # Trend
    df['ema_9']   = df['close'].ewm(span=9).mean()
    df['ema_21']  = df['close'].ewm(span=21).mean()
    df['ema_50']  = df['close'].ewm(span=50).mean()
    df['ema_200'] = df['close'].ewm(span=200).mean()
    df['macd']    = df['ema_9'] - df['ema_21']
    df['adx']     = compute_adx(df, 14)

    # Momentum
    df['rsi']         = compute_rsi(df['close'], 14)
    df['stoch_rsi']   = compute_stoch_rsi(df['close'])
    df['williams_r']  = compute_williams_r(df)

    # Volatility
    df['bb_upper'], df['bb_lower'] = compute_bollinger(df['close'])
    df['atr']   = compute_atr(df, 14)
    df['keltner'] = compute_keltner(df)

    # Volume
    df['obv']  = compute_obv(df)
    df['vwap'] = compute_vwap(df)

    return df.dropna()

?? Critical: Every indicator must be computed with a 200-candle rolling lookback maximum. Otherwise you'll leak future data into training.

The leakage trap

My first run reported 78% accuracy. I almost shipped it.

Then I noticed EMA 200 was looking 200 candles into the future at every row. I capped all lookbacks at 200, switched to a temporal train/test split (not random), and accuracy dropped to 62% - a realistic number.

Always sanity-check your validation strategy with a plot. If your model "predicts" the past perfectly, it's leaking.

The XGBoost model

import xgboost as xgb
from sklearn.model_selection import TimeSeriesSplit

X = df[feature_cols].values
y = (df['close'].shift(-48) > df['close'] * 1.008).astype(int).values  # next 4h, +0.8%

# Time-based split
split = int(len(X) * 0.7)
X_train, X_test = X[:split], X[split:]
y_train, y_test = y[:split], y[split:]

model = xgb.XGBClassifier(
    n_estimators=400,
    max_depth=6,
    learning_rate=0.05,
    subsample=0.8,
    colsample_bytree=0.8,
    random_state=42,
    n_jobs=-1,
    eval_metric='logloss',
    early_stopping_rounds=30
)

model.fit(X_train, y_train,
          eval_set=[(X_test, y_test)],
          verbose=False)

Bayesian optimization for hyperparameters

Grid search took 4 hours and found a local minimum. Bayesian optimization with Optuna found a better solution in 18 minutes:

import optuna

def objective(trial):
    params = {
        'max_depth': trial.suggest_int('max_depth', 3, 9),
        'learning_rate': trial.suggest_float('learning_rate', 0.01, 0.3, log=True),
        'n_estimators': trial.suggest_int('n_estimators', 100, 800),
        'subsample': trial.suggest_float('subsample', 0.6, 1.0),
        'colsample_bytree': trial.suggest_float('colsample_bytree', 0.5, 1.0),
        'reg_alpha': trial.suggest_float('reg_alpha', 1e-8, 10, log=True),
        'reg_lambda': trial.suggest_float('reg_lambda', 1e-8, 10, log=True),
    }
    model = xgb.XGBClassifier(**params, n_jobs=-1, random_state=42)
    tscv = TimeSeriesSplit(n_splits=5)
    scores = []
    for train_idx, val_idx in tscv.split(X_train):
        model.fit(X_train[train_idx], y_train[train_idx])
        scores.append(model.score(X_train[val_idx], y_train[val_idx]))
    return np.mean(scores)

study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=50, show_progress_bar=False)

print('Best params:', study.best_params)
print('Best score:', study.best_value)

The results

After 50 trials:

Strategy	Sharpe (out-of-sample)	Max DD	Win rate
Fixed RSI/MACD	0.61	-18%	51%
XGBoost + grid search	1.42	-12%	58%
XGBoost + Bayesian	1.85	-9%	62%

The Bayesian-optimized model:

Generated 47 signals over 6 weeks
62% win rate
Average winner: +1.4%
Average loser: -0.6%
Live Sharpe: 1.85 (vs backtested 1.91 - minimal overfitting)

What I'd do differently

Add a regime detector. A separate model that says "we're in trending/ranging/volatile regime" and routes to specialized sub-models. This is my next iteration.
Use a meta-labeler. Don't trade every signal - use a second model to decide if the first model's signal is worth taking.
Walk-forward validation. I retrain weekly but the entire training window is the past 6 months. A pure walk-forward would only train on data the model hadn't seen in N days.

Get the code

The full pipeline (data collection ? features ? training ? serving via FastAPI) is open source:

?? github.com/Omerfaruk-aydn

I write more about the production setup (FastAPI + WebSocket signal distribution, CCXT order execution, risk management) on my blog.

Originally published on omerfarukaydn.com - full code, more metrics, and the 7-layer safety system explained.