DEV Community

Janak Shrestha
Janak Shrestha

Posted on

Self-Host a Qwen3.8-27B-Uncensored LLM on Oracle Cloud with Your Own Domain and integrate with AI coding agents

#selfhosted #oracle #llm #devops

Oracle Cloud gave me the 1 month trial server, llama.cpp gave me the model server, and Caddy handled HTTPS. It sounds simple, but I lost hours to a Cloudflare proxy setting, two layers of firewalls and a Let's Encrypt rate limit. This post is the clean path, with the commands I used and the mistakes to skip.

What you'll build

By the end you'll have:

  • A 27B parameter model running on your own Oracle Cloud VM
  • An OpenAI-compatible API at https://localai.janakkumarshrestha0.com.np/v1
  • A web chat interface on the same domain
  • Automatic HTTPS with a free certificate that renews itself
  • A setup you can plug into tools like Hermes Agent, or anything that accepts a custom base URL

Before you start: what to expect

  • This is CPU-only inference. On my VM, the 27B Q4_K_M model generated about 4.5 tokens/s and processed prompts at about 40 tokens/s. That's comfortable for chat and testing, and slow for coding agents.
  • Oracle's Always Free tier can't run a 27B model. The free Arm VM is limited to 2 OCPUs and 12 GB of RAM. I used my free-trial credit on a VM.Standard.E5.Flex with 24 OCPUs and 42 GB of RAM, which stops working when the trial ends. On Always Free, a 7–9B model is the realistic choice, and every step below still applies.
  • You need a Linux VM with a public IP, SSH access, and a domain whose DNS you control (I used Cloudflare).

Prepare your server

Log in over SSH and update the system:

sudo apt update && sudo apt upgrade -y
# check 
lscpu | grep -E "Model name|^CPU\(s\)"
free -h
df -h
Enter fullscreen mode Exit fullscreen mode

server specs

My VM showed 48 vCPUs (24 cores with 2 threads each) and 42 GB of RAM. As a rule, a Q4 model needs roughly its file size in RAM, plus extra for the context.

Next, find the VM's public IP:

curl -s ifconfig.me
# 92.4.88.45
Enter fullscreen mode Exit fullscreen mode

Install llama.cpp and run the model in tmux

Install it and add it to your PATH:

curl -LsSf https://llama.app/install.sh | sh
echo 'export PATH="$HOME/.llama-app:$PATH"' >> ~/.bashrc
source ~/.bashrc
Enter fullscreen mode Exit fullscreen mode

Get your favourite AI models from huggingface.co

Models take a some time to download and load, and the server has to keep running after you close SSH. So I run it inside tmux

tmux new -s llama
Enter fullscreen mode Exit fullscreen mode

Everything you type from here on runs inside that session.

llama serve -hf JonathanColetti/Qwen3.8-27B-Uncensored-GGUF:Q4_K_M
Enter fullscreen mode Exit fullscreen mode

You'll see warnings about unused tensors and image tokens. They're harmless. When it says listening on ..., stop it with Ctrl+C and start the version you'll keep:

llama serve -hf JonathanColetti/Qwen3.8-27B-Uncensored-GGUF:Q4_K_M \
  --host 127.0.0.1 --port 8080 \
  --api-key iamkingjanak \
  --parallel 1 -c 65536 -t 24
Enter fullscreen mode Exit fullscreen mode

What the flags do:

  • --host 127.0.0.1: only Caddy on the same machine can reach it. Don't bind to the public IP. It isn't on the network interface, and llama fails with couldn't bind HTTP server socket.
  • --api-key: clients must send Authorization: Bearer <key>. Generate a long random one with openssl rand -hex 24.
  • --parallel 1: one request at a time. The default of 4 slots shared the CPU and made every request slower.
  • -c 32768: a sane context size. Without it my server asked for 262,144 tokens.
  • -t 24: one thread per physical core. Generation is limited by memory bandwidth, not thread count.

Detach with Ctrl+B, then D. The server keeps running. To get back to it later:

tmux attach -t llama
Enter fullscreen mode Exit fullscreen mode

Test locally 

curl http://localhost:8080/health

curl -s http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer iamkingjanak" \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen","messages":[{"role":"user","content":"what is the capital city of nepal?"}]}'
Enter fullscreen mode Exit fullscreen mode

You should get a JSON reply containing Kathmandu. If that works, the model is fine and everything after this is networking.

Resposne

Point your domain at the oracle VM

In Cloudflare DNS, add an A record:

  • Name: localai (or any subdomain)
  • IPv4: your VM's public IP
  • Proxy status: DNS only (grey cloud)

DNS record

This matters. With the orange proxy on, browsers see Cloudflare's addresses and you get Error 522: Connection timed out, and Let's Encrypt validates against Cloudflare instead of your server. Cloudflare's free plan also cuts requests off at about 100 seconds, which a slow CPU model easily exceeds.

Verify it:

dig +short localai.janakkumarshrestha0.com.np
curl -s ifconfig.me
Enter fullscreen mode Exit fullscreen mode

ip

Both must print your VM's IP. If dig shows a different IP, the proxy is still on.

Open ports 80 and 443

Caddy needs port 80 (certificate validation) and 443 (HTTPS). Traffic passes through two firewalls, and both must allow it.

a. The OCI security list

Go to Networking → VCN → Subnets → your subnet → Security List → Add Ingress Rules. Add TCP rules for destination ports 80 and 443 from 0.0.0.0/0.

OCI ingress rules for ports 80 and 443

Leave "Source Port Range" empty (All). I once typed the service port there. It never matches, because a client's source port is random, and the rule silently does nothing.

b. The Ubuntu firewall (iptables)

Ubuntu images on OCI ship with iptables rules that end with a REJECT. Look at them:

sudo iptables -L INPUT -n --line-numbers
Enter fullscreen mode Exit fullscreen mode

New rules must sit above the REJECT, because rules are read top to bottom.

sudo iptables -I INPUT 5 -m state --state NEW -p tcp --dport 80 -j ACCEPT
sudo iptables -I INPUT 5 -m state --state NEW -p tcp --dport 443 -j ACCEPT
sudo netfilter-persistent save
Enter fullscreen mode Exit fullscreen mode

Or delete the REJECT rule so the OCI security list is your only gate:

sudo iptables -D INPUT -j REJECT --reject-with icmp-host-prohibited
# sudo iptables -D INPUT 5 (rule line no)
sudo netfilter-persistent save
Enter fullscreen mode Exit fullscreen mode

Testing reachability publicly: running curl -v http://92.4.88.45 from the other vm often fails with No route to host, because a VM usually can't reach its own public IP on OCI. Test from your laptop or phone instead.

Install Caddy and get HTTPS 

sudo apt install -y caddy
sudo nano /etc/caddy/Caddyfile
Enter fullscreen mode Exit fullscreen mode

Replace the contents with:

{
    email you@example.com
}

localai.janakkumarshrestha0.com.np {
    encode zstd gzip

    reverse_proxy 127.0.0.1:8080 {
        flush_interval -1
        transport http {
            dial_timeout 5s
            response_header_timeout 30m
            read_timeout 30m
        }
    }
}
Enter fullscreen mode Exit fullscreen mode
  • email is used for certificate expiry notices.
  • flush_interval -1 streams tokens as they're generated instead of buffering them.
  • The 30-minute timeouts keep Caddy from cutting off a slow model that is still processing a long prompt.

Format, validate and apply it:

sudo caddy fmt --overwrite /etc/caddy/Caddyfile
sudo caddy validate --config /etc/caddy/Caddyfile
sudo systemctl reload caddy
sudo journalctl -u caddy -f
Enter fullscreen mode Exit fullscreen mode

You're waiting for this line:

"msg":"certificate obtained successfully"
Enter fullscreen mode Exit fullscreen mode

Test from the outside 

curl https://localai.janakkumarshrestha0.com.np/health

curl https://localai.janakkumarshrestha0.com.np/v1/models \
  -H "Authorization: Bearer iamkingjanak"
Enter fullscreen mode Exit fullscreen mode

Testing

Open the domain in a browser to get the llama web UI, which asks for the API key:

Entering the API key in the web UI

And here it is answering:

Chat examples

Connect your AI Agents: Pi, OMP, Hermes, Openclaw

Any OpenAI-compatible client needs these values:

  • Endpoint URL: https://localai.janakkumarshrestha0.com.np/v1
  • API key: iamkingjanak

Examples

  • Hermes agent

Hermes cli

  • Hermes Desktop

Hermes Desktop

Agents send very long prompts, often 10–20K tokens. At about 40 tokens/s prompt processing, the first reply can take minutes. Hermes also wants a context of about 64k, so you may need -c 65536. Use one agent at a time, because two at once will queue and retry each other.

  • htop during token generation

htop during inference

Top comments (0)