DEV Community: Ahmet Ozan GAZİ

I Built a Self-Hosted ChatGPT on My Laptop. Here's What Broke.

Ahmet Ozan GAZİ — Mon, 02 Nov 2026 07:57:12 +0000

From DevOps to AI Platform Engineering — Part 1

I'm a DevOps engineer. I've spent years running Kubernetes clusters, writing Terraform and keeping production alive. Lately, every job interview has a new question in it: "What have you done with AI?"

Calling the OpenAI API from a script didn't feel like a good answer. So I set myself a different goal: run an open-weight LLM myself and serve it to other people like a real platform, with API keys, rate limits, usage tracking and numbers I measured myself.

This is part 1 of a 3-part series. Everything here runs on one laptop, a MacBook Pro M1 Pro with 16 GB of RAM. Part 2 covers observability and load testing. Part 3 moves the same platform to GPUs on Amazon EKS.

I call the platform OzAI. To be clear from the start: OzAI is not a model I trained. It's a self-hosted platform built on open-weight models (Qwen3 8B and Qwen2.5-Coder 3B). The engineering is in serving them.

The architecture

Ollama runs the models. It runs natively on macOS, not in Docker (more on that below).
LiteLLM is the gateway: an OpenAI-compatible API with per-user virtual keys, model allow-lists, rate limits and a request log in Postgres.
Open WebUI is a ChatGPT-style UI that talks only to the gateway.
SearXNG is a self-hosted search engine, so the model can answer questions about recent events.
Grafana shows dashboards built on the gateway's request log (part 2).

Everything except Ollama runs in Docker Compose and binds to 127.0.0.1 only. Nothing is exposed to the internet yet.

Decision #1: Ollama runs on the host, not in a container. Docker on macOS runs containers inside a Linux VM with no access to the Apple GPU. Ollama in a container would run on the CPU only. On the host it uses Metal, and the containers reach it at host.docker.internal:11434. It looks less tidy in a compose file, but it's the difference between a usable system and an unusable one.

Step 1: How fast is a laptop, really?

Before building anything, I measured. Here is one request per model, 200 max tokens, no concurrency:

Model                    Disk     Cold load   Prompt eval   Generation
qwen2.5-coder:3b (Q4)    1.9 GB   2.1 s       120 tok/s     53.3 tok/s
qwen3:8b (Q4_K_M)        5.2 GB   3.9 s        87 tok/s     22.4 tok/s

Two things surprised me:

The 8B model is less than half as fast as the 3B one. Generating tokens is limited by memory bandwidth, not compute. For every token, the model's weights have to be read from memory again. A model 2.7× bigger is roughly 2.4× slower.
22 tokens per second is fine for one person. That's faster than most people read. The real question is what happens with more than one user, which comes later.

Quantization (Q4_K_M = roughly 4 bits per weight) is what makes an 8B model fit in 5.2 GB at all. Without it, the same model needs around 16 GB just for its weights, which is all the RAM in this laptop.

Step 2: A gateway, not just a model

Running a model is easy. Letting other people use it safely is the platform engineering part, and that's where LiteLLM comes in.

The model list maps friendly names to real models:

model_list:
  - model_name: ozai            # fast chat
    litellm_params:
      model: ollama_chat/qwen3:8b
      api_base: http://host.docker.internal:11434
      reasoning_effort: none
      num_ctx: 8192
  - model_name: ozai-think      # same model, thinking on
    litellm_params:
      model: ollama_chat/qwen3:8b
      api_base: http://host.docker.internal:11434
      num_ctx: 8192
  - model_name: ozai-coder
    litellm_params:
      model: ollama_chat/qwen2.5-coder:3b
      api_base: http://host.docker.internal:11434

Then every client gets its own virtual key with its own limits:

curl localhost:4000/key/generate \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -d '{"key_alias":"demo-friend","models":["ozai"],"rpm_limit":10}'

I tested it the way I'd test any API gateway:

No key                              -> 401
Friend key asking for ozai-coder    -> 403 (not on its allow-list)
12 requests in a minute, limit 10   -> 9 x 200, then 429
                                       (one earlier request counted in the same window)

Any tool that speaks the OpenAI API can now use OzAI by changing two settings: the base URL and the key. That turned out to matter a lot for the IDE integration below.

What broke (the useful part)

Nothing below threw an error. Every one of these problems showed up as "it works, but badly."

1. A "thinking" model made every answer take two minutes

Qwen3 is a reasoning model. Before it answers, it writes a hidden chain of thought. One simple question produced over 600 streamed reasoning chunks before the first word of the actual answer. The full response took about two minutes, partly because it was queued behind another request on the same GPU.

There was a second, quieter problem. With a small max_tokens, the model spent the whole budget on thinking and returned an empty answer.

Fix: two aliases for the same weights. ozai has thinking turned off (reasoning_effort: none) and is the default. ozai-think is there for hard questions. Users choose speed or depth by picking a model name. Nobody has to know about prompt flags.

2. A $0 budget blocked everything

I gave a test key max_budget: 0, assuming that meant "no spending allowed." Local models cost $0, so I expected every request to pass. Instead, every request was rejected:

Budget has been exceeded! Current cost: 0.0, Max budget: 0.0

The check is spend >= budget, and 0 ≥ 0. For self-hosted models, either leave the budget unset or set a custom per-token price so the budget means something.

3. The UI was quietly competing with its own users

The chat UI felt slow even with a single user. The gateway log explained why. After every answer, Open WebUI sends extra requests to the same model: one to generate a chat title, one for tags, one for follow-up suggestions.

On one GPU, those requests queue in front of the user's next question.

Fix: In Open WebUI's admin settings, point background tasks at the small 3B model (ozai-coder) and turn off tags and follow-ups. After that, answers started almost immediately again.

4. More users don't get more throughput

I wrote a small load generator (Python threads streaming from the gateway) and measured time to first token (TTFT) and throughput at different concurrency levels. Each request asked for 128 tokens:

Users   TTFT p50   TTFT max   Per-user tok/s   Total tok/s
1         0.3 s      0.3 s        19.5            19.5
2         3.4 s      6.7 s        14.5            19.7
4         9.3 s     18.6 s         8.7            20.8
8        21.3 s     43.3 s         4.7            20.7

Total throughput stays flat at about 20 tokens per second. More users just split the same output into smaller slices. The worst TTFT is roughly your place in the queue × one answer. In practice, requests are processed one at a time.

For a DevOps engineer this was the key lesson: an LLM server does not scale like a web server. Adding users doesn't use idle capacity, because the hardware is already working at full speed for one request. Production inference servers like vLLM fix this with continuous batching, which runs many requests through the model in the same pass. That's part 3.

5. The model is confidently wrong, and has no idea what day it is

An 8B model has to compress a lot of world knowledge into 5 GB, and it shows. Ask it a precise technical question and it answers with total confidence, even when it's wrong. The KEDA Kafka scaler's default lagThreshold is 10. OzAI says 1000:

The second answer shows the other limit. The model only knows what was in its training data, and it reports a knowledge cutoff of October 2023. The current F1 season is out of reach.

Both problems have the same cure: stop relying on what the model remembers, and give it the facts at question time. Later in this series I add document RAG and measure it. On 10 KEDA questions like this one, accuracy goes from 1/10 to 8/10.

Fix (part one of RAG): web search through a self-hosted SearXNG instance. Open WebUI searches, pulls in the relevant parts of the top results and gives them to the model as context. No search API keys, and no queries sent to a third-party AI provider.

It still didn't work. That led to the best bug of the project.

6. The bug with no error: silent prompt truncation

Web search was on and SearXNG was returning results. The model still said "I don't have access to live data."

The gateway's request log had the clue. Every Open WebUI request had exactly 2,050 prompt tokens. It didn't matter whether the question was short or long, or whether search was on or off.

So I ran a test. I sent about 5,000 tokens of filler text with a secret word at the very end:

Sent:   ~5,000 tokens, secret word at the END
Got:    prompt_tokens: 2050, answer: BANANA

The model found the word, but the gateway reported only 2,050 prompt tokens. About 3,000 tokens had disappeared, and since the end survived, they came from the beginning. Ollama was silently cutting long prompts to fit its context window. It kept the end and dropped the beginning, with no error and no warning. Open WebUI puts the search results at the start of the prompt, so they were cut every single time.

Fix: one line, num_ctx: 8192, on the gateway model. Then I moved the secret word to the start of the prompt and repeated the test:

Sent:   ~5,000 tokens, secret word at the START
Got:    prompt_tokens: 5430, answer: BANANA

The whole prompt now reached the model. Memory use went from 5.6 GB to 6.2 GB.

This is the lesson I'd put on a poster for anyone moving from DevOps to AI infrastructure: in LLM serving, "no error" doesn't mean "correct." A web server that drops half a request fails loudly. A model server that drops half a prompt just gives a worse answer. Watch your token counts.

7. IDE integration: the gateway was the wrong path for autocomplete

Because the gateway speaks the OpenAI API, connecting VS Code (with the Continue extension) took one config block. Chat and code edits go through the gateway with their own key, so that usage shows up in the logs.

Autocomplete was different. Code completion uses fill-in-the-middle (FIM): the model gets the code before and after the cursor and fills the gap. Through the gateway, the coder model answered like a chatbot instead:

"Certainly! The Fibonacci sequence is a series of numbers where..."

The gateway's completions endpoint was dropping the FIM parameters (suffix, raw) on the way to Ollama. Called directly, Ollama completed the function correctly:

def fibonacci(n):
    if n <= 1:
        return n
    else:
        return fibonacci(n-1) + fibonacci(n-2)

Fix: chat goes through the gateway, and autocomplete goes straight to Ollama. That's the dashed line in the architecture diagram. It's a trade-off: autocomplete requests don't appear in the usage logs. For a local setup, I'll take working completions over complete logs.

Cursor is another story. It calls custom models from its own servers, so it can't reach localhost. That needs a public, authenticated endpoint, which is coming later with Cloudflare Tunnel and Access.

What I'd tell another DevOps engineer

The gateway is where your skills transfer. Keys, quotas, rate limits, routing, request logs: this is API platform work you already know. The model is just another upstream.
Measure before you optimize. One table of tokens per second at 1, 2, 4 and 8 users explained more than any blog post I read.
LLMs fail quietly. Truncated prompts, empty answers caused by thinking, confident hallucinations. None of them return a 500. Your monitoring has to look at token counts and answer quality, not just status codes.
Small local models are fast and cheap, but not reliable as a source of facts. Give them context (search, documents) instead of trusting what they remember.

What's next

Part 2: observability and load testing. A Grafana dashboard for TTFT, latency percentiles and tokens per second, built from the gateway's request log. Plus why LiteLLM's Prometheus metrics weren't an option.
Part 3: from laptop to production. vLLM on Amazon EKS with GPU nodes, autoscaling on queue depth with KEDA, and the cost of self-hosting compared with paying per token.

All configs, benchmarks and notes are in the repo: github.com/ozy31/self-hosted-llm-platform

If you're making the same move from DevOps into AI infrastructure, I'd like to hear what surprised you.

AWS Site-to-Site VPN Connection and Distrubution over AWS Transit Gateway.

Ahmet Ozan GAZİ — Tue, 24 Jun 2025 11:40:03 +0000

AWS Site-to-Site VPN and Transit Gateway Cross-Account Routing

Introduction

In enterprise cloud infrastructures, establishing secure and scalable network connections between different AWS accounts and VPCs (Virtual Private Cloud) is of critical importance. AWS provides powerful services such as Site-to-Site VPN and Transit Gateway to meet this need. In this article, we will explain step by step how to route network traffic between multiple AWS accounts using Site-to-Site VPN and Transit Gateway.

1. Basic Concepts

Site-to-Site VPN

AWS Site-to-Site VPN creates an encrypted tunnel between your on-premises data center and your AWS VPC. This enables secure data transmission between your on-premises network and cloud environment.

Transit Gateway

AWS Transit Gateway allows you to connect multiple VPCs and on-premises networks through a central hub. This structure simplifies network management and increases scalability.

2. Scenario and Architecture Definition

Account A: Main AWS account connected to on-premises data center.
Account B and C: VPCs located in different AWS accounts.
Objective: Provide secure and centralized access from on-premises data center to VPCs in Account B and C.

3. Step-by-Step Setup

3.1. Creating Transit Gateway (Account A)

Sign in to AWS Management Console with Account A.
Follow the steps: VPC Dashboard > Transit Gateways > Create Transit Gateway.
Give the Transit Gateway a meaningful name and configure necessary settings.
Note the Transit Gateway ID.

3.2. Creating Transit Gateway Attachments

a) Connecting Account A's VPC to Transit Gateway

Create a new attachment from the Transit Gateway Attachments section in VPC Dashboard.
Select the relevant VPC in Account A and connect it to the Transit Gateway.

b) Creating VPC Attachments in Other Accounts (B and C)

In Account B and C, enable Transit Gateway sharing through Resource Access Manager (RAM).
Share the Transit Gateway from Account A.
In Account B and C, connect their own VPCs to the Transit Gateway using the shared Transit Gateway.

3.3. Site-to-Site VPN Setup

Create Customer Gateway and Virtual Private Gateway in Account A.
Create Site-to-Site VPN Connection and enter your on-premises network information.
Connect the VPN connection to the Transit Gateway (create Attachment).

3.4. Route Table and Routing

Add routes to each VPC's route table to route traffic to other VPCs and on-premises network through the Transit Gateway.
Define rules in the Transit Gateway Route Table to route incoming traffic to the correct VPC or VPN.

3.5. Security Groups and NACLs

Configure security groups and network access control lists (NACL) to allow relevant ports and protocols.

4. Testing and Validation

Test the connection by pinging or SSH connecting to resources (e.g., EC2) in Account B and C from the on-premises network.
Monitor traffic using AWS VPC Flow Logs and CloudWatch.

5. Best Practices

Segment Transit Gateway Route Tables to ensure network isolation.
Securely manage resource sharing with IAM and RAM.
Use two tunnels for high availability in VPN connections.

Conclusion

With AWS Site-to-Site VPN and Transit Gateway, you can establish a centralized, secure, and scalable network architecture between different AWS accounts and on-premises networks. This structure greatly simplifies network management in large-scale and multi-account architectures.

MFA AWS VIA TERMINAL

Ahmet Ozan GAZİ — Mon, 20 May 2024 10:48:12 +0000

In today's digital landscape, securing access to cloud services is paramount, particularly as cyber threats become increasingly sophisticated. Amazon Web Services (AWS), a leading cloud service provider, offers robust security features to protect sensitive data and infrastructure. One essential security measure is Multi-Factor Authentication (MFA), which enhances the traditional username and password login process by requiring an additional verification step. This essay will explore the process of configuring and using MFA to access AWS from a local terminal, highlighting the steps involved, the benefits of this security approach, and practical tips for seamless integration. By implementing MFA, users can significantly bolster their security posture, ensuring that access to their AWS environments is both secure and efficient.
In this article I want draw your attention to some point which we can make it happen with ease. First of all we need to do some configuration like installing necessary packages such as os, subprocess, pyotp, json, base64, binascli, configparser, dotenv. These packages are prerequisites for making the scripts work .
As a next step we need to define the environment variables which we'll us it in our bash scripts. To do seo we create a .env file to keep all of our environmental variables to call them in the cli commands.

When configuring Multi-Factor Authentication (MFA) to securely access Amazon Web Services (AWS) from a local terminal, one of the crucial steps involves creating a .env file. This file, short for "environment variables," is essential for securely storing and managing sensitive information, such as AWS access keys, secret keys, and MFA device serial numbers. By using a .env file, users can avoid hard-coding these credentials directly into their scripts or applications, which enhances security and simplifies configuration management. The .env file should be placed in the root directory of the project and structured in a way that each variable is declared on a new line.
When it comes to creating a script for all of these configuration, naturally we need to acquire a snappy script which includes as follows,
1- Imports and Environment Setup:

Imports the necessary modules.
Loads environment variables from the .env file.

2- Configuration Reading:

Reads AWS credentials from the specified source profile in the AWS credentials file.

3-Validation:

Verifies that the TOTP secret is valid and base32 encoded.

4- MFA Token Generation:

Generates a current MFA token using the TOTP secret.

5- Session Token Retrieval:

Uses AWS STS to obtain a session token with the MFA token.

6- Credentials Update:

Updates the AWS credentials file with the temporary session credentials.

Additionally, ensure the ~/.aws/profile under the $HOME directory is correctly configured for your AWS CLI.

Here is the one example for script that we may use;
aws-login.py

import os
import subprocess
import pyotp
import json
import base64
import binascii
import configparser
from dotenv import load_dotenv

# Load environment variables from .env file
load_dotenv()

# Retrieve variables from the environment
mfa_serial_number = os.getenv('MFA_SERIAL_NUMBER')
totp_secret = os.getenv('TOTP_SECRET')
source_profile = os.getenv('SOURCE_PROFILE', 'dev-mfa')
aws_profile = os.getenv('AWS_PROFILE', 'default')

# Read credentials from the source profile
config = configparser.ConfigParser()
credentials_file_path = os.path.expanduser('~/.aws/credentials')
config.read(credentials_file_path)

if source_profile not in config:
    print(f"Profile '{source_profile}' not found in {credentials_file_path}.")
    exit(1)

aws_access_key_id = config[source_profile].get('aws_access_key_id')
aws_secret_access_key = config[source_profile].get('aws_secret_access_key')

if not aws_access_key_id or not aws_secret_access_key:
    print(f"Missing credentials in profile '{source_profile}'.")
    exit(1)

# Ensure TOTP secret is base32 encoded
try:
    base64.b32decode(totp_secret, casefold=True)
except binascii.Error:
    print("The provided TOTP secret is not valid base32 encoded. Please check the secret.")
    exit(1)

# Generate MFA token
totp = pyotp.TOTP(totp_secret)
mfa_token = totp.now()

# Get temporary session token using MFA
try:
    response = subprocess.check_output([
        'aws', 'sts', 'get-session-token',
        '--serial-number', mfa_serial_number,
        '--token-code', mfa_token,
        '--output', 'json',
        '--profile', source_profile
    ])
except subprocess.CalledProcessError as e:
    print(f"Error getting session token: {e}")
    exit(1)

# Parse the response
response_json = json.loads(response)
credentials = response_json['Credentials']

# Update the AWS credentials file with the temporary credentials
if aws_profile not in config.sections():
    config.add_section(aws_profile)

config[aws_profile]['aws_access_key_id'] = credentials['AccessKeyId']
config[aws_profile]['aws_secret_access_key'] = credentials['SecretAccessKey']
config[aws_profile]['aws_session_token'] = credentials['SessionToken']

with open(credentials_file_path, 'w') as configfile:
    config.write(configfile)

print(f"🙌🏻 Welcome to Starlet Technologies AWS Cloud Services. All logins and acitivities are monitoring and recording.")
print("⏰ Your AWS CLi session will expire at:", credentials['Expiration'])

## Verify the identity of the temporary credentials
#try:
#    identity = subprocess.check_output(['aws', 'sts', 'get-caller-identity', '--profile', aws_profile])
#    print("Current Identity:", identity.decode('utf-8'))
#except subprocess.CalledProcessError as e:
#    print(f"Error verifying identity: {e}")
#    exit(1)
#
## Attempt to list S3 buckets
#try:
#    s3_buckets = subprocess.check_output(['aws', 's3', 'ls', '--profile', aws_profile])
#    print("S3 Buckets:", s3_buckets.decode('utf-8'))
#except subprocess.CalledProcessError as e:
#    print(f"Error listing S3 buckets: {e}")
#    exit(1)

In order to make above script run, we use

python aws-login.py

As a expected output we can see ;

For the troubleshooting purposes we recommend you to check if you have registered profile under ~/.aws/credentials file.