DEV Community

SmartGate profile image lizer yang
lizer yang for SmartGate

Posted on Originally published at smartgate.network

How to Deploy an AI Gateway in a Private Cloud: MCP and Audit

#self #what #mcp #model

Short answer: you need three things inside your own VPC - an MCP endpoint that speaks
Streamable HTTP, a runtime whose configuration refuses to start instead of falling back to a
development origin, and an audit path (retention window, export quota, per-key limits) that you
now operate. A shared counter store holds rate-limit and budget counters, a relational store
holds audit and usage rows, and the console that lists your MCP servers is part of the same
deployment, reading the same environment variables.

Key takeaways

  • Configuration is the deployment. The shipped absoluteUrl helper builds every absolute link from one environment variable; leave it unset and the app answers happily while pointing at a development origin.
  • The protocol contract is one endpoint. MCP is JSON-RPC 2.0 over a transport, and the current transport is Streamable HTTP: one URL, POST with a JSON or SSE response, session state in a header.
  • Your MCP server list is configuration you version. Each upstream is one row - transport, URL, credential reference, timeout - and the list view is a convenience, not the record.
  • Timestamps are formatted for humans, not for evidence. The app renders dates through one helper that pins an en-US long format, and activity is shown as relative age; the audit export is where an absolute UTC value has to come from.
  • Report windows are boundary-aligned, not rolling. A "month" figure resets on the first of the calendar month while the other windows count days back from now, so a quota period and a dashboard month will not reconcile unless you align them on purpose.
  • AI agent security is the part a hosted plan used to do for you. Per-key rate limits, budget caps, log retention and the export quota all move inside your perimeter.
  • Do this next: put your resolved base URL, your MCP endpoint path, your retention window and your export quota on one page of deployment notes, then check each against what the running environment actually resolves.

Self-hosted AI gateway: what the deployment owns

A hosted plan hides three surfaces behind a login. When the gateway moves into your cloud, all
three become yours: the runtime and its configuration, the web layer your operators use, and the
audit trail.

Sizing the runtime is the easy half: one container per replica, no persistent local state, one
inbound endpoint, two outbound dependencies. The half that breaks private-cloud rollouts is
configuration, and it rarely breaks loudly. The typical failure is a silent fallback - the
process starts healthy, serves traffic, and emits absolute URLs pointing at an origin nobody
outside your network can reach. The shipped implementation makes that concrete:

# lib/utils.ts — source lines 105–105 (absoluteUrl)
function absoluteUrl(path: string) {
Enter fullscreen mode Exit fullscreen mode 
# lib/utils.ts — source lines 107–107 (absoluteUrl)
}
Enter fullscreen mode Exit fullscreen mode

That is the whole function with its return line elided; the provenance table at the end of this
page records the exact source range, and the omitted line names a development origin that must
never reach a published page. One variable, NEXT_PUBLIC_APP_URL, decides the origin of every
absolute link the console emits: canonical tags, share cards, links inside audit exports, and the
endpoint URL you hand to client teams. The helper concatenates the path straight onto that value,
so it trims no trailing slash and validates no scheme. Set it to the origin your users type, then
prove it after the first deploy by fetching one page and grepping the HTML for the wrong origin.

A deployment order that avoids the trap:

  1. Choose the public origin - the hostname on your TLS certificate, not the load balancer's internal name - and keep it in infrastructure config, not in the image.
  2. Make a missing value a boot failure rather than a warning. The check is one line, and it is the difference between a five-minute fix and a week of wrong links.
  3. Log the resolved origin at boot, so every rolling deploy says which value it picked up.

What is Model Context Protocol, in deployment terms

The Model Context Protocol (MCP) is a JSON-RPC 2.0 message contract between an AI client and a
server that exposes capabilities: tools it can call, resources it can read, prompts it can fetch.
Three properties matter once you host it. It is transport-bound but not transport-specific, so
your gateway publishes one endpoint and the messages on it are JSON-RPC request/response pairs.
It starts with an initialize handshake in which the server declares its protocol version and
capabilities, so nothing about a server is knowable until that call succeeds. And a server may
hold session state, which turns a rolling deploy into a session-lifecycle question.

Keep build-time metadata and a runtime handshake apart, because both appear in the deployment and
they answer different questions. Here is the build-time half from the shipped console:

# lib/brand-metadata.ts — source lines 64–77 (buildBrandViewport)
function buildBrandViewport(): Viewport {
  return {
    themeColor: [
      {
        media: "(prefers-color-scheme: light)",
        color: brandTheme.light.themeColor,
      },
      {
        media: "(prefers-color-scheme: dark)",
        color: brandTheme.dark.themeColor,
      },
    ],
  };
}
Enter fullscreen mode Exit fullscreen mode

The function returns a constant - the light and dark theme colours for the viewport - and it runs
during a build, so it is baked into the served HTML and can never report a runtime problem. The MCP
handshake is the opposite kind of object:
a live call whose answer changes with every deploy. Deployers who conflate the two ship a readiness
probe that passes while clients cannot initialize at all. The rule that follows: after a rollout,
point one real client at the endpoint, call initialize, and require a capability list in the
response before calling the deploy finished. A process-level check is a floor, not an acceptance
test. The walkthrough on this site maps the same handshake onto the primitives the gateway exposes
(Model Context Protocol explained).

MCP server list: the configuration you version

An aggregating gateway rarely talks to one server; it fronts a list. That list is configuration,
and in a private cloud it is also your allow-list and your blast radius, so treat it like firewall
rules: in version control, reviewed, diffable. One row per upstream is enough - a stable id your
logs can carry, the transport and endpoint, the credential reference, a request timeout, and the
subset of tools you allow through. Each row is also a decision about hosting a protocol server:
the transport, the endpoint and the timeouts are the parts a private deployment cannot
inherit from somebody else's hosted version. The credential reference is a pointer into your secret store;
the token itself never belongs in the list.

Three habits keep the list honest. Give every entry its own timeout so one slow upstream cannot hold
a client request open. Start with the entries you can reach and mark the rest unhealthy rather than
refusing to boot, unless a missing upstream is fatal for your traffic - a decision to write down, not
to leave to a default. And keep a field that records when each entry last changed. The console's list view
renders those timestamps through a single helper:

# lib/utils.ts — source lines 96–103 (formatDate)
function formatDate(input: string | number): string {
  const date = new Date(input);
  return date.toLocaleDateString("en-US", {
    month: "long",
    day: "numeric",
    year: "numeric",
  });
}
Enter fullscreen mode Exit fullscreen mode

One formatter, one format: en-US, long month, numeric day, year. Two consequences follow. The
rendered date is a display artefact - pinned to a fixed locale rather than your operator's - so it
is not the value you quote in a compliance answer or compare against a stored row; read the stored
UTC value and format it at the edge of the answer. And because one helper feeds every date in the
console, a formatting change moves every column at once. A published
server list reference is worth comparing your own list against.

Model Context Protocol news: name the revision you deployed

The protocol moves in dated revisions rather than semantic versions: the transport page you deploy
against carries a revision identifier, and the next revision may change the transport, the
authorization flow or the shape of a result. Both revisions still answer, which is why "we support MCP" is not something a deployment can claim - "we
implement the transport as specified in this revision" is.

Write the revision into your deployment notes next to the endpoint path, keep a compatibility
matrix of the clients you tested against it, and re-run the handshake when you upgrade either
side. Two traps make this cheaper to do than to skip. A client that ignores a field the server
added is a client bug, and the revision written down saves an afternoon of blame. And a revision that changes a default can pass every smoke test while quietly
changing behaviour, so re-test the handshake and one real tool call rather than trusting a green
build.

Freshness is the other half of reading a changelog, and the console renders it as relative age:

# lib/utils.ts — source lines 110–115 (timeAgo)
export const timeAgo = (timestamp: Date, timeOnly?: boolean): string => {
  if (!timestamp) return "never";
  return `${ms(Date.now() - new Date(timestamp).getTime())}${
    timeOnly ? "" : " ago"
  }`;
};
Enter fullscreen mode Exit fullscreen mode

The helper turns a timestamp into a short relative string and returns the literal never when the
value is falsy. For an activity column that is the right trade - staleness is what you want to see
at a glance. In an evidence trail it is the wrong one: "3 days ago" cannot be compared with a log
line, and never collapses two different failures into one word, a field that was never written
and a record that predates your retention window. Keep the absolute value in the store, and alert on never as its own condition: an unwritten
timestamp is a pipeline bug, not a quiet period. Revision history is tracked in
protocol versions and transports.

Streamable HTTP: one endpoint, two response shapes

Streamable HTTP is the transport your gateway has to expose. The server publishes a single endpoint. A client sends a JSON-RPC
message with an HTTP POST whose Accept header lists both application/json and
text/event-stream; the server may answer with one JSON object or with a server-sent event
stream, and it may open a GET stream for messages it initiates. Session state travels in a
header the server issues during initialize, and resumption after a dropped connection uses the
last event id the client saw.

Translate that into load-balancer settings before you go live. Response buffering has to be off, or
your event stream arrives in one lump at the end and clients time out waiting. Idle timeouts have
to exceed your longest stream, or the stream needs heartbeats to keep the connection warm. If you
run more than one replica, decide where session state lives - affinity that pins a session to the
replica that created it, or a shared store both replicas can read. A per-IP request limit at the
edge is a reasonable defence, as long as it counts requests rather than connections, or
long-lived streams get cut mid-conversation.

The client half of the contract is worth reading once, because it tells you what your error
responses have to look like:

# lib/utils.ts — source lines 117–137 (fetcher)
async function fetcher<JSON = any>(
  input: RequestInfo,
  init?: RequestInit,
): Promise<JSON> {
  const res = await fetch(input, init);

  if (!res.ok) {
    const json = (await res.json()) as { error?: string };
    if (json.error) {
      const error = new Error(json.error) as Error & {
        status: number;
      };
      error.status = res.status;
      throw error;
    } else {
      throw new Error("An unexpected error occurred");
    }
  }

  return res.json();
}
Enter fullscreen mode Exit fullscreen mode

This helper treats any non-2xx response as an exception, lifts the server's own error string
into the thrown error, and attaches the HTTP status; a body without that field degrades to a
generic message. That is the contract to satisfy: when your gateway rejects a request, return JSON
carrying an error string and the status code, not the ingress's HTML error page. Get it wrong
and every client integration reports the same useless sentence while your operators have nothing
to grep. The gateway that puts this transport in front of upstream endpoints is
MCP gateway, and the contract to hand clients is documented at
the MCP endpoint docs.

An MCP server example that survives your own network

Start with the smallest example that exercises every layer you will later depend on: an internal
service exposing one MCP endpoint over Streamable HTTP, one read-only tool, and token auth on the
internal hop even though nothing outside the VPC can reach it. The protocol logic is the part that
will not surprise you; URL and configuration handling is the part that does, which is why the
interesting code here is a two-line helper:

# lib/brand-metadata.ts — source lines 79–82 (brandEmailLogoUrl)
function brandEmailLogoUrl(baseUrl: string): string {
  const base = baseUrl.replace(/\/$/, "");
  return `${base}${brandAssets.emailLogo}`;
}
Enter fullscreen mode Exit fullscreen mode

Strip a trailing separator from the base before joining, so a base URL written with and without a
trailing slash produces the same endpoint. Copy that discipline for the path you publish:
normalise the base once at startup - one separator decision, one path prefix - and log the fully
resolved endpoint in the boot output. From then on, anything that builds an endpoint URL must go
through the normaliser rather than concatenating configuration strings at the call site, which is
how a double slash appears that your client tolerates and your logs do not.

Then let the example earn its keep. Put it behind your internal load balancer before it has users,
so transport, TLS and timeout settings are exercised while the blast radius is one reader. Require
auth from the first request rather than adding it before the first real user, since retrofitting
auth onto a live endpoint means a maintenance window. Assert that the URL your client builds is
byte-for-byte the one you logged at boot. And add the second server only once the first has
survived a rolling deploy, because a list of one that restarts cleanly teaches you more than a
list of five that has never been restarted. A worked example is
the MCP server example, and the Python version is
the Python MCP server tutorial.

MCP resources: cache windows and what "this month" means

Resources are the read path of the protocol: URIs a client can list and read, plus the hints that
let a client cache them and know when they changed. The moment your gateway caches an upstream
resource, the cache window becomes a deployment decision you have to write down, because it
decides how stale an answer can be when two views disagree. Treat it like any other TTL: pick it
from the upstream's change rate, state it in the deployment notes, and make the cache observable.

The reporting windows inside the console make the same point about boundaries:

# lib/analytics/reports.ts — source lines 23–33 (periodCutoff)
function periodCutoff(
  period: AnalyticsPeriod | "90d",
): Date {
  const now = new Date();
  if (period === "month") {
    return new Date(now.getFullYear(), now.getMonth(), 1);
  }
  const days =
    period === "7d" ? 7 : period === "90d" ? 90 : 30;
  return new Date(now.getTime() - days * 86400000);
}
Enter fullscreen mode Exit fullscreen mode

A month window starts at the first day of the calendar month; the others count a fixed number of
days back from now, with a default that catches anything unrecognised. So "this month" is
boundary-aligned rather than a rolling thirty days, and both flavours depend on the server's
clock. Two consequences. If your quota or billing period is a contract month, align the dashboard
on that definition deliberately instead of discovering the difference during a reconciliation. And
put a real clock under the deployment - UTC in the container, NTP on the host - because a drifting
clock silently moves every boundary, and the symptom is a report that looks approximately right.
The resource surface itself is covered in
MCP resources, prompts and sampling, and the log side
of the same story is the logs documentation.

AI agent security: keys, caps, retention, exports

Everything a hosted plan did for you on the security side moves inside your perimeter at once, so
the honest way to plan a private-cloud deployment is to name four controls instead of one feature.
Keys. One credential per consuming service or team, so revocation is a single action.
Caps. A per-key request rate and a spend ceiling enforced before the request is served rather
than reported after it. Retention. A window per entitlement, a job that deletes older rows,
and a record of what it removed, so "how long do you keep this" has an answer that is a
configuration value plus a job log. Exports. A quota on the heaviest query your gateway
exposes, measured per team and per window, because an unbounded audit export is a self-inflicted
denial of service.

The console's economics add one subtlety worth designing around:

# lib/analytics/build-savings-payload.ts — source lines 19–24 (shouldShowRoiLine)
function shouldShowRoiLine(
  roiPct: number | null,
  estUsdAvoided: number,
): boolean {
  return (roiPct ?? 0) >= 30 || estUsdAvoided >= 5;
}
Enter fullscreen mode Exit fullscreen mode

That is the condition under which the savings line renders: an ROI of at least thirty percent, or
at least five dollars of estimated avoided spend. It is a sensible display rule, and it produces a trap:
a metric that appears only above a floor makes its own absence ambiguous. "No line" can mean nothing to report or nothing measured, and a reader
will assume the first. If anybody downstream reconciles cost from your deployment, expose the
underlying numbers through your own export path and alert on the counters behind the panel rather
than on the panel - the same rule applies to the denials, the throttled keys and the retention job.
Prompt-side hardening for the same agents is covered in
secure prompt handling in AI applications,
and the operator-facing controls are listed under
the audit and compliance features.

How SmartGate compares

For a single developer machine none of this applies: point a client at the hosted endpoint and
move on. The decision only becomes interesting when several teams, several keys and an audit
question are involved.

Where the data lives Who operates the audit path What you pay
Hosted multi-tenant gateway Vendor cloud Vendor: retention tiers are a plan feature Free tier: 2M tokens a month, the full primitive set, no card; Pro from $18/mo
Self-hosted open-source proxy Your cloud You write the retention, export and quota code yourself Engineering time; the control plane is yours to build
Private-cloud SmartGate deployment (Teams / Enterprise) Your cloud You operate it, on the same entitlement retention job and export API quoted above Teams from $55/mo; Enterprise by contract, 1200 req/min/key and HMAC-signed budgets
Bring your own gateway Your cloud You, for every stage Your time plus the platform fees you already pay

The pricing shape is the differentiator: the platform fee is fixed, and a share of measured savings
starts only once they pass a threshold, which makes capacity planning predictable - usually the real
requirement behind wanting the gateway in your own cloud.

How to get started

  1. Choose the two stores before you choose the runtime. A shared counter store for rate-limit and budget counters, a relational store for audit, usage and configuration. Both inside the VPC, neither with a public endpoint.
  2. Make configuration strict. A missing base URL, counter store or database configuration must stop the process, and the resolved origin must be logged at boot.
  3. Publish one endpoint. Streamable HTTP on a single path behind your internal load balancer, with buffering off and idle timeouts sized for event streams.
  4. Write the server list down. Transport, endpoint, credential reference, timeout and allowed tools per upstream, in version control, reviewed like a firewall rule.
  5. Pick your windows on purpose. Retention per entitlement, cache TTL per resource class, and one definition of "this month" that your reports and your quota both use.
  6. Own the controls. Keys per team, caps enforced before the request, an export quota per team and window, and a monitored retention job.
  7. Prove the deployment with a client. After every rollout, call the handshake and one real tool call from outside the cluster, not just a process health check.

Prototype against the hosted endpoint before you build the private-cloud path - the free tier is
2M tokens a month with the full primitive set and no card: start free.
The endpoint contract is in the docs, the plan ceilings the
policy validator enforces are on the pricing page, and a
private-cloud or enterprise deployment (HMAC budgets, 1200 req/min/key, contract token pools)
starts from the contact form.

FAQ

Do I need the console to run the gateway, or only the API?
The gateway runtime needs the two stores and its configuration; the console is the operational
half - server list, usage views, exports. Deploy them together the first time so the configuration
contract is exercised end to end, then decide how much of the console stays reachable.

What is the smallest viable private-cloud deployment?
One runtime replica, one shared counter store, one relational store, one internal load balancer in
front of the endpoint. That is enough to exercise transport, auth, counters and audit. Add a second
replica when you need availability, and settle session state at the same moment.

Why does the health check pass while clients cannot connect?
Because a process check answers "is the process up", and MCP is a runtime handshake. Call
initialize from a real client as part of the acceptance test, and treat a successful process
probe as a floor rather than proof.

How do I know which services are talking to the gateway?
One key per service or team. The gateway then attributes requests, rate limits, budget and audit
rows to a named credential, and revoking a leaked key is a single action instead of a fleet-wide
rotation.

What happens to sessions during a rolling deploy?
They end, unless session state lives somewhere both the old and the new replica can read. Decide between
session affinity and a shared session store before the first rollout, and record the choice in your
runbook.

How long should audit rows live?
As long as the entitlement says and no longer. Retention is a job: it reads the window per team,
deletes by cutoff, and reports what it removed - so the answer in a review is a configuration
value plus a job record, not an estimate.

Limitations and what this does not do

  • This is the application's deployment contract, not a Terraform module. Subnets, TLS termination, secret distribution, backups and clock discipline are yours; the article covers the configuration, endpoint, list, window and control surfaces the application expects.
  • Five of the eight excerpts come from the web layer, not the gateway runtime. They are quoted because that layer is part of what you operate once the gateway is inside your VPC; if you run the API only and drive it from your own console, read those sections as the contract to reproduce rather than as code you will deploy.
  • One excerpt is a window, not a whole body. The URL helper is shown with its return statement elided, because that line names a development origin the publish gate rejects. The prose describes what the omitted line does and the provenance table records the exact range.
  • Boundary-aligned reports are not rolling reports. A calendar-month window is the right default for a quota period and the wrong one for a trailing-thirty-day review; the page does not pick for you, it tells you which one you have.
  • The display rules are display rules. The savings line, the relative timestamps and the formatted dates are presentation decisions, and none of them is an audit source.

Sources

Method note

The code in this article is not transcribed. Each block was cut directly out of the slice body
returned by the SmartGate slice API and re-asserted byte-for-byte as a substring of that body
before publication; the first line inside every fence records the file and the exact source lines.
Symbols were pinned by whole-name containment (rule A level 2) and confirmed by the service's
slot-proof endpoint before being written into the prose - 8 of 8 planned sections pinned, no
abstentions and no misses. Every section keyword is a measured search phrase with non-zero Google Ads
monthly volume, which is why the headings read like questions. One section quotes a window rather than a whole body: the lines above and below it are
described in prose and the provenance table records the range.

Slice provenance

# SERP keyword Symbol File Source lines How it was pinned sha256(12)
1 self hosted ai gateway absoluteUrl lib/utils.ts 105–105, 107–107 rule A L2 → slot-proof 8e7969e18129
2 what is model context protocol buildBrandViewport lib/brand-metadata.ts 64–77 rule A L2 → slot-proof 842c02c9af77
3 mcp server list formatDate lib/utils.ts 96–103 rule A L2 → slot-proof 47fd6f571d88
4 model context protocol news timeAgo lib/utils.ts 110–115 rule A L2 → slot-proof cff1836d7c7b
5 streamable http fetcher lib/utils.ts 117–137 rule A L2 → slot-proof cf0c5de4e764
6 mcp server example brandEmailLogoUrl lib/brand-metadata.ts 79–82 rule A L2 → slot-proof 560ba51f2d45
7 mcp resources periodCutoff lib/analytics/reports.ts 23–33 rule A L2 → slot-proof b3b9db65dd2d
8 ai agent security shouldShowRoiLine lib/analytics/build-savings-payload.ts 19–24 rule A L2 → slot-proof 7eea86a41890

Every fenced block above was cut from the slice body and re-asserted against it byte-for-byte before
publication. 8 of 8 sections pinned, 0 abstentions, 0 misses.

Top comments (0)