Deploying a Custom LLM in Production: Four Architectures, Only One Works

Four architectures, one 8B model, and the data that drove every decision.

Deploying a custom large language model in production is rarely a single decision. It is a sequence of tradeoffs, and each one surfaces a new constraint you did not see coming.

This post documents how we deployed an 8B parameter model (FP16) on Google Cloud Platform for Polyphemus, Rhesis's adversarial test generation service. Polyphemus is an uncensored testing model built to attack LLM applications through prompt injection, goal hijacking, and other adversarial scenarios that safety-aligned models refuse to generate.

We went through four distinct architectures before landing on something that actually works at scale. All four started on Cloud Run (Google's serverless container platform) before we ultimately moved inference to a persistent GPU endpoint.

The short version: we started by baking the model into a Docker image, hit disk space limits in CI, moved the model to Cloud Storage, waited 40 minutes for cold starts, found a way to stream the model directly instead of downloading it in full, and finally moved inference off a serverless platform entirely onto a persistent GPU endpoint with a proper serving engine.

Each change was driven by data. The performance numbers we collected at each stage are what guided the decisions, not gut feeling.

The broader lesson, though, applies well beyond GCP: managed cloud services for AI workloads are genuinely hard to beat. They handle the difficult infrastructure problems: hardware availability, scaling, persistent endpoints at the cost of price and some flexibility. If you can afford the price, you probably should not be building the alternative yourself.

Challenge 1: Image Size and Startup Time
First attempt: baking the model into the Docker image
The first approach was simple in theory. If the model needs to be on the container, put it there at build time. The Dockerfile would copy the weights into the image, and when the container started, everything would already be in place.

This worked locally. It did not work in CI. Figure 1 shows why.

Figure 1: Baking model weights into the Docker image. The 30+ GB image size caused CI failures and slow pushes to GCR.
An 8B FP16 model (8 billion parameters stored in 16-bit floating point precision, the standard full-precision format before any quantization) occupies roughly 16 GB on disk. Add the Python runtime, CUDA libraries, and application code, and the image exceeded 30 GB. GitHub Actions runners ran out of disk space mid-build. Even when a build completed, pushing a 30+ GB image to Google Container Registry took long enough to make the pipeline unusable.

There is also a subtler problem: every code change forces a full rebuild that includes the model weights. Docker layer caching helps when the cache is warm, but a fresh CI environment or any structural Dockerfile change triggers a complete re-upload. Docker images are for code and dependencies. Model weights are data. Conflating the two creates compounding problems as the model grows.

Second attempt: model weights in Google Cloud Storage
The fix was to separate the model from the container. We uploaded the weights once to a Google Cloud Storage bucket and changed the startup sequence to download the model at runtime before loading it onto the GPU.

Upload model weights to GCS (one-time operation)

huggingface-cli download --local-dir ./model-cache
gsutil -m cp -r ./model-cache gs:///cache/

The container image shrank to a few gigabytes. Builds became fast and predictable. The tradeoff was startup time: container starts, downloads ~16 GB over the network, loads weights into GPU memory, then begins serving. On a Cloud Run instance with an NVIDIA L4 GPU, the GCS download alone took 20–30 minutes. Total cold start: around 40 minutes. For a service that scales to zero, that is not acceptable.

Third attempt: mounting the bucket with GCS Fuse
GCS Fuse lets you mount a Cloud Storage bucket as a filesystem inside a container. Instead of downloading the model in full before loading it, the model loading code reads files directly from the mount point (see Figure 2).

Figure 2: GCS Fuse mount. The bucket is exposed as /mnt/model inside the container; model files stream on demand as they are read.

Cloud Run supports GCS Fuse natively and the mount is configured in the service definition, and the container sees the bucket as a regular directory.

gcloud run deploy rhesis-polyphemus \
--image=gcr.io//rhesis-polyphemus:latest \
--region=us-central1 \
--add-volume=name=model-vol,type=cloud-storage,bucket= \
--add-volume-mount=volume=model-vol,mount-path=/mnt/model \
--gpu=1 \
--gpu-type=nvidia-l4 \
--memory=32Gi \
--cpu=8

This reduced startup time from ~40 minutes to roughly 15–20 minutes. A real improvement, but it did not solve the underlying latency problem. The model was still loading from a network-backed filesystem, and under concurrent load, there was a bigger issue we had not yet hit.

Figure 3 summarizes how deployment time changed across the first three approaches.

Figure 3: Deployment time evolution across the first three approaches.

Key insight: separating model weights from container images is not optional at scale. The right mental model is: images ship code, object storage ships data. GCS + Fuse is a reasonable middle ground, but it still carries network-backed startup costs.

Challenge 2: Cloud Run Becomes a Bottleneck
Cloud Run with GPU support is a managed serverless environment. That design works well for stateless web services. It does not map well onto LLM inference.

The core problem is that Cloud Run is built around request-response isolation. Each request goes to a worker process, and the platform scales by adding or removing instances. For an LLM, this prevents the GPU batching that makes inference efficient. The GPU works best when it processes multiple sequences simultaneously using batched attention. Isolated workers prevent that from happening naturally.

To understand how bad the problem actually was, we needed a baseline. We turned to Vertex AI Model Garden (Google's managed endpoint service for open models) and deployed the same model there. Not as our final solution, just as a reference point: what does good look like on this infrastructure?

We measured this directly: 20 requests per test type (latency, throughput, and concurrent), recording mean response time across each. Figure 4 shows what the numbers looked like.

Figure 4: Cloud Run GPU vs. Vertex AI Model Garden baseline. Mean response times under three test types.

Cloud Run failed to handle concurrent load reliably: success rates dropped to 37.5–50%. Vertex AI maintained 100% success rate across all test types. Throughput on Vertex AI was 0.19 requests/second versus 0.03 on Cloud Run under the same load. The Model Garden deployment confirmed that the managed endpoint infrastructure handled concurrent requests significantly better than Cloud Run.

Vertex AI endpoints maintain a persistent GPU instance. The model stays loaded in GPU memory between requests. No cold start per request. The platform handles queuing and batching at the infrastructure level.

Cloud Run GPU is fine for development or low-traffic scenarios where cold starts are acceptable. Under concurrent load, the serverless model actively works against LLM efficiency. If you need production-grade LLM serving, a persistent endpoint is the right starting point.

Challenge 3: Getting the Right Inference Engine
Custom container on Vertex AI: still not fast enough
With the baseline confirmed, we deployed our own container to a Vertex AI endpoint. Same HuggingFace-based inference server we had been running on Cloud Run, now on a g2-standard-8 machine with a single NVIDIA L4 GPU.

Better than Cloud Run under load but the latency numbers were still disappointing. Under the throughput test, mean response time was 173.6 seconds. Under concurrent load, 140.4 seconds. We tried upgrading to an A100 GPU. The improvement was modest.

The data in Table 1 made it clear: the bottleneck was never the hardware. It was the inference engine.

Switching to vLLM
A Google engineer who works with production LLM deployments pointed us toward vLLM. The suggestion was to stop treating LLM serving like a standard single-request inference workload and optimize for concurrency from the start.

vLLM is an inference engine built specifically for LLM serving. The key difference from a standard HuggingFace loop is how it handles memory and batching. vLLM uses PagedAttention, which manages the KV cache in fixed-size pages rather than allocating contiguous memory per sequence. This lets it serve many concurrent requests without the memory fragmentation that forces standard servers to serialize.

The practical effect: vLLM batches multiple in-flight requests together on the GPU, processing them simultaneously rather than one at a time. For an 8B model on a single L4, this changes the throughput profile substantially. Here is the vLLM serving command we use:

python -m vllm.entrypoints.openai.api_server \
--host=0.0.0.0 \
--port=8080 \
--model=gs:///cache/ \
--tensor-parallel-size=1 \
--swap-space=16 \
--gpu-memory-utilization=0.9 \
--max-model-len=4096 \
--dtype=auto \
--max-num-seqs=256 \
--disable-log-stats \
--guided-decoding-backend=auto

A few parameters worth understanding:

--gpu-memory-utilization=0.9 tells vLLM to use 90% of GPU memory for the KV cache. On a 24 GB L4, that leaves room for model weights (~16 GB in FP16) and allocates ~5 GB to the cache.
--max-num-seqs=256 allows up to 256 sequences in flight simultaneously. PagedAttention makes this feasible without running out of memory.
--swap-space=16 provides 16 GB of CPU memory as overflow when GPU memory is under pressure.
--dtype=auto lets vLLM detect the model's native precision from the checkpoint. FP16 checkpoint loads in FP16.

We use the official Vertex AI vLLM container image maintained by Google: pytorch-vllm-serve. It is pre-configured for Vertex AI, handles the /ping health check, and exposes an OpenAI-compatible API at /v1/chat/completions. The results speak for themselves: Figure 5 shows how the three serving options compare across all test types.

Figure 5: Mean response time across three test types for three serving configurations on the same L4 GPU: HF Service (the same model deployed via Vertex AI Model Garden's managed container), Custom Container (our own Docker image with a standard HuggingFace inference server), and vLLM (an open-source inference engine optimized for concurrent LLM serving).
The custom container performed worst across all tests. vLLM had the best single-request latency and the most consistent concurrent performance. The HuggingFace-based server showed lower mean latency under concurrent load, but that is misleading: it was handling fewer requests per second, so each individual request waited less in a shorter queue. Under real concurrent traffic, vLLM wins.

edit_note
Choosing the right inference engine matters more than choosing the right GPU tier. Switching from HuggingFace to vLLM on the same L4 hardware reduced latency under concurrent load by roughly 70%. Upgrading from L4 to A100 with the same custom container improved single-request latency by about 25% but did not fix concurrency.

Final Architecture
The production system runs on two separate deployment pipelines, each managed by its own GitHub Actions workflow (see Figure 6).

Model deployment is managed by the polyphemus-vertex-ai.yml file. It triggers on the changes to apps/polyphemus/model_deployment/ or can be run manually with environment and configuration options. The deployment script uploads the model to the Vertex AI model registry and deploys it to the endpoint. Updates are handled as rolling replacements: the new model takes 100% of traffic, the old model drops to 0%, and is then undeployed. No downtime, no manual traffic splitting.

Figure 6: The production architecture: a lightweight Cloud Run proxy routes requests to a persistent Vertex AI endpoint running vLLM.
The second polyphemus.yml file handles the API service. Polyphemus itself is now a lightweight FastAPI proxy running on Cloud Run with no GPU and no local model weights. It receives requests, authenticates them, applies rate limiting, and forwards them to the Vertex AI endpoint via rawPredict. The model lives in GCS and is loaded by vLLM at endpoint startup. The Cloud Run service scales to zero between periods of activity; the Vertex AI endpoint maintains a persistent GPU instance.

To keep the two in sync, the deploy step queries the Vertex AI endpoint ID by name before deploying the Cloud Run service, ensuring the proxy always points to a live endpoint.

The Cost Picture
Moving to Vertex AI endpoints is not free, and it is worth being direct about that.

A Vertex AI endpoint running a g2-standard-8 machine with a single NVIDIA L4 GPU costs roughly $1.30–$1.50 per hour in us-central1 (check current pricing in the Google Cloud console, as rates change). That is around $950–$1,100/month for a 24/7 persistent endpoint. Cloud Run with GPU scales to zero, so you only pay when requests arrive, but as we showed, it cannot handle concurrent load reliably.

For low-traffic or bursty workloads, Cloud Run GPU may still be cheaper despite the latency tradeoffs — an L4 GPU on Cloud Run costs roughly $0.80–$1.00 per hour, but since it scales to zero, you only pay when requests are actively being served. For anything requiring consistent low-latency responses under concurrent load, the Vertex AI endpoint cost is justified. The 70% latency improvement and 100% success rate under load are the value you are paying for.

One additional cost factor: Vertex AI endpoint deployments take 15–30 minutes to complete. Frequent model updates add up. Batching model changes and using the rolling deployment pattern (new model at 100% traffic, old model undeployed after) keeps this manageable.

edit_note
The managed services path is genuinely hard to beat if you can pay the price. Running your own persistent GPU inference infrastructure including handling hardware provisioning, health checks, auto-recovery, traffic routing costs more in engineering time than most teams realize.

Lessons Learned
Looking back, the path we took made sense given what we knew at each step. But there are a few things we would do differently from the start.

Model weights do not belong in Docker images.
The moment you put a 16 GB model into a container, you have created a build artifact that is too large for standard CI runners, too slow to push, and too expensive to store multiple versions of. Separate model from code from day one.

Storage strategy determines startup speed.
Downloading at startup is slow. GCS Fuse is faster but still adds latency. For Vertex AI endpoints, loading the model from a GCS path directly via vLLM is the cleanest approach; vLLM handles the download internally, and the endpoint only starts serving after the model is fully loaded.

Cloud Run GPU is not the right tool for concurrent LLM inference.
It works for development or single-request flows. Under concurrent load, the serverless model prevents GPU batching. Success rates dropped to 37.5–50% in our tests. For production serving, start with a persistent endpoint.

The inference engine matters more than the GPU.
Switching from HuggingFace to vLLM on the same hardware reduced concurrent latency by ~70% and eliminated throughput degradation. Upgrading the GPU tier improved single-request latency by ~25% but did not fix concurrency. Pick the right engine before you pick the right GPU.

Check GPU quota before starting a deployment.
Discovering a quota limit after a 30-minute deployment attempt is unpleasant. We added a quota verification step to the pipeline to catch this early.

When this stack is overkill.
Not every use case needs a persistent Vertex AI endpoint. If your model is called infrequently (a few requests per hour), Cloud Run GPU with scale-to-zero will cost significantly less and the cold start penalty is acceptable. If you are doing batch inference rather than real-time serving, a Vertex AI batch prediction job is a better fit than a persistent endpoint. And if your team is still in early experimentation, the 15–30 minute deployment cycle of Vertex AI endpoints will slow you down. Start with Cloud Run, validate your use case, then migrate when concurrent load actually becomes a problem.

Wrapping Up
Four iterations. One 8B FP16 model. In retrospect, the final stack is straightforward, comprising GCS for model storage, Vertex AI for the persistent GPU endpoint, and vLLM as the inference engine. But each component choice was validated by measurement rather than assumption, and we would not have known the right answer without going through the wrong ones first.

The Polyphemus service now runs as a lightweight FastAPI proxy on Cloud Run, forwarding requests to a Vertex AI endpoint that runs vLLM on a g2-standard-8 machine with an NVIDIA L4 GPU. Deployments are automated through GitHub Actions, with separate workflows for the API service and the model endpoint.

If you are starting a similar project: skip the Cloud Run GPU phase for anything that needs to handle concurrent requests. Start with Vertex AI endpoints and vLLM. The managed infrastructure handles the hard parts such as hardware provisioning, health checks, traffic routing and vLLM's continuous batching makes the GPU work efficiently from the beginning. Yes, it costs more. But it works reliably, and that reliability has a value that is easy to underestimate until you are debugging a 50% success rate in production.

Polyphemus is part of the Rhesis platform for AI safety testing.

FAQ
Can I use vLLM on Cloud Run instead of Vertex AI?
Technically yes, but Cloud Run's serverless model still limits GPU batching. vLLM's efficiency gains come from continuous batching across concurrent requests, which requires persistent GPU state between requests. Cloud Run's scale-to-zero behavior conflicts with that. For low-traffic scenarios it may work, but you will not get the full benefit of vLLM without a persistent endpoint.

What is GCS Fuse and when should I use it?
GCS Fuse is a FUSE adapter that mounts a Google Cloud Storage bucket as a local filesystem. It is useful when you want to avoid downloading large files in full before using them and the data transfers as it is read. For LLM serving, it reduces startup latency compared to sequential download, but still adds overhead versus loading from a local disk. It is a reasonable middle step, not a final solution.

How does Vertex AI handle model updates without downtime?
Vertex AI endpoints support traffic splitting between multiple deployed models. The rolling deployment pattern we use: deploy the new model version at 100% traffic (which automatically sets the old version to 0%), then undeploy the old version. This avoids downtime and lets you roll back by redeploying the previous version if something goes wrong.

Is this approach specific to GCP?
This architecture pattern applies across cloud providers: object storage for model weights, a persistent managed endpoint for inference, and an optimized serving engine. AWS SageMaker endpoints and Azure ML managed endpoints follow similar logic. The specific tools differ (S3 instead of GCS, SageMaker Model Registry instead of Vertex AI Model Registry), but the tradeoffs are the same. Managed inference endpoints cost more than rolling your own, and they handle the operational complexity that is easy to underestimate.

What are the cost implications compared to self-hosted alternatives?
A Vertex AI endpoint with a single L4 GPU runs ~$1.30–$1.50/hour, or roughly $950–$1,100/month for 24/7 availability. Self-hosting on a raw GCE instance with the same GPU would be cheaper on paper. The difference is engineering time: managed endpoints handle health checks, hardware failure recovery, traffic routing, and scaling. For most teams, the operational overhead of self-hosted GPU infrastructure outweighs the cost savings.

References
Vertex AI Prediction overview — Google Cloud documentation on Vertex AI endpoints.
Vertex AI Model Garden — Browse and deploy open models directly on Vertex AI.
vLLM documentation — Official vLLM docs including PagedAttention and serving configuration.
Cloud Run GPU support — Google Cloud documentation on attaching GPUs to Cloud Run services.
GCS Fuse on Cloud Run — How to mount Cloud Storage buckets as volumes in Cloud Run.
Serving open models on Vertex AI — Guide to deploying open-source models using the official Vertex AI containers.

Md Asaduzzaman Miah

Deploying a Custom LLM in Production: Four Architectures, Only One Works | Rhesis AI Blog

How we deployed an 8B parameter model on GCP from oversized Docker images to a low-latency Vertex AI endpoint with vLLM. Real data, real tradeoffs.

rhesis.ai