DEV Community

Dakota Huang
Dakota Huang

Posted on

From Localhost to a Real Host: Deploy a Model Gateway in 6 Verified Steps

#tutorial #docker #llm #devops

Localhost lies to you. It hides cold starts, public exposure, and quota drift. A working local endpoint is a prototype. A working remote endpoint is a service. This tutorial deploys a minimal model gateway to a free server. Every step ends with a verification command. Nothing is trusted until it passes.

What you are building

A tiny HTTP gateway. It forwards chat requests to a free model endpoint. It enforces a timeout and maps upstream errors to 502. It exposes a /healthz route for checks. Total code: about 50 lines. Deployment target: a free server with Docker.

Why a real host changes the game

Free model endpoints behave on localhost. They misbehave from a remote host. DNS resolution, TLS handshakes, and cold starts appear. Your laptop also sleeps. A server does not. The local-to-remote gap hides most failures. Close that gap early.

Prerequisites

You need two things. A free model endpoint you can call. A free server you can SSH into. This tutorial uses MonkeyCode's free model access and its free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Exact quotas and hardware vary. Read the current dashboard before you build.

Step 1: Pin the contract

Write down what the endpoint must do. A contract is three lines, not a document. POST /v1/chat with a JSON body. A 200 response with JSON. Any failure returns a structured error.

{
  "endpoint": "/v1/chat",
  "request": { "model": "string", "messages": ["role", "content"] },
  "success": { "status": 200, "body": "json" },
  "failure": { "status": "4xx or 5xx", "body": "json" }
}
Enter fullscreen mode Exit fullscreen mode

Verify the baseline. Call the endpoint from your laptop. Save the exact response. This is your reference. If the baseline fails, stop. No deployment fixes a broken upstream.

Step 2: Write the gateway

Keep the gateway small. Fifty lines is enough. It forwards the request, applies a timeout, and maps errors. No caching. No queues. No auth yet.

const http = require('http');
const UPSTREAM = process.env.UPSTREAM_URL;
const TIMEOUT_MS = Number(process.env.TIMEOUT_MS || 20000);

http.createServer(async (req, res) => {
  if (req.url === '/healthz') {
    res.writeHead(200, { 'content-type': 'application/json' });
    return res.end(JSON.stringify({ ok: true }));
  }
  if (req.method !== 'POST' || req.url !== '/v1/chat') {
    res.writeHead(404, { 'content-type': 'application/json' });
    return res.end(JSON.stringify({ error: 'not_found' }));
  }
  let body = '';
  for await (const chunk of req) body += chunk;
  const controller = new AbortController();
  const timer = setTimeout(() => controller.abort(), TIMEOUT_MS);
  try {
    const upstream = await fetch(UPSTREAM, {
      method: 'POST',
      headers: { 'content-type': 'application/json' },
      body,
      signal: controller.signal,
    });
    const text = await upstream.text();
    res.writeHead(upstream.status, { 'content-type': 'application/json' });
    res.end(text);
  } catch (err) {
    res.writeHead(502, { 'content-type': 'application/json' });
    res.end(JSON.stringify({ error: 'upstream_failed', detail: err.name }));
  } finally {
    clearTimeout(timer);
  }
}).listen(8080);
Enter fullscreen mode Exit fullscreen mode

The gateway buffers the full upstream response before replying. That keeps the client contract simple. It also prevents partial JSON from leaking to callers.

Verify locally. Run UPSTREAM_URL=<your endpoint> node gateway.js. Then check the health route.

curl -s http://localhost:8080/healthz
curl -s -X POST http://localhost:8080/v1/chat \
  -H "content-type: application/json" \
  -d '{"model":"your-model","messages":[{"role":"user","content":"ping"}]}'
Enter fullscreen mode Exit fullscreen mode

Expect {"ok":true} first. Expect a real model reply second.

Step 3: Containerize

Containers make the server reproducible. Use a non-root user. Do not bake the upstream URL into the image. Pass it as an environment variable at runtime.

FROM node:20-alpine
WORKDIR /app
COPY gateway.js .
RUN addgroup -S app && adduser -S app -G app
USER app
EXPOSE 8080
CMD ["node", "gateway.js"]
Enter fullscreen mode Exit fullscreen mode

Verify the image locally. Run the same curl from Step 2.

docker build -t gateway .
docker run --rm -p 8080:8080 \
  -e UPSTREAM_URL=<your endpoint> gateway
curl -s http://localhost:8080/healthz
Enter fullscreen mode Exit fullscreen mode

Same result as Step 2. That is the point. The container must not change behavior.

Step 4: Deploy to the free server

Copy the two files. Build the image on the server. Run the container with a restart policy. A free server can reboot without warning. The policy handles that.

scp gateway.js Dockerfile user@server:/opt/gateway/
ssh user@server 'cd /opt/gateway && docker build -t gateway .'
ssh user@server 'docker run -d --name gateway \
  --restart unless-stopped \
  -p 8080:8080 \
  -e UPSTREAM_URL=<your endpoint> \
  gateway'
Enter fullscreen mode Exit fullscreen mode

Verify from your laptop, not from the server. Remote verification proves the port is public.

curl -s http://SERVER_IP:8080/healthz
Enter fullscreen mode Exit fullscreen mode

If this fails, check the firewall. Free servers often block ports by default. Open port 8080 in the provider panel.

Step 5: Verify the failure path

The happy path is not enough. Force the upstream to fail. Point the container at a closed port. Expect a clean 502. Then restore the real URL.

ssh user@server 'docker stop gateway'
ssh user@server 'docker run -d --name gateway-bad \
  -p 8081:8080 \
  -e UPSTREAM_URL=http://127.0.0.1:9 \
  gateway'
curl -s -X POST http://SERVER_IP:8081/v1/chat \
  -H "content-type: application/json" \
  -d '{"messages":[]}'
ssh user@server 'docker rm -f gateway-bad'
ssh user@server 'docker start gateway'
Enter fullscreen mode Exit fullscreen mode

Expect a 502 with {"error":"upstream_failed"}. This proves the gateway degrades honestly. A silent hang is worse than a fast error.

Step 6: Add a real healthcheck

Docker HEALTHCHECK runs inside the container. It must test the gateway, not the upstream. Use /healthz. A cheap check beats a clever one.

HEALTHCHECK --interval=30s --timeout=3s --retries=3 \
  CMD wget -qO- http://127.0.0.1:8080/healthz || exit 1
Enter fullscreen mode Exit fullscreen mode

Rebuild and redeploy. Then inspect the status.

ssh user@server 'cd /opt/gateway && docker build -t gateway .'
ssh user@server 'docker rm -f gateway && docker run -d --name gateway \
  --restart unless-stopped -p 8080:8080 \
  -e UPSTREAM_URL=<your endpoint> gateway'
ssh user@server 'docker inspect --format="{{.State.Health.Status}}" gateway'
Enter fullscreen mode Exit fullscreen mode

Expect healthy after about 30 seconds. If it stays unhealthy, check the container logs.

Limitations

Free servers have small CPUs and less memory. Do not run a heavy model there. This gateway forwards; it does not cache, rate-limit, or authenticate. Add auth before exposing it to the public internet. Free model quotas change. Re-check the provider dashboard weekly. This setup suits prototypes and internal tools. It is not an SLA.

Who should not use this

Teams with production SLAs should avoid free tiers. Multi-region failover needs a real platform. Compliance requirements need audit trails. High traffic needs a load balancer and metrics. Use this tutorial for learning, demos, and low-traffic tools.

Closing

If you need both free model access and a free server, MonkeyCode's free options are a reasonable starting point. The steps above work with any provider. Verify the current terms before you commit. Then verify your deployment. Every step here has a command. Run them all.

Top comments (0)