This article provides a step by step deployment guide for Gemma 4 to a Google Compute Engine hosted GPU enabled system. A suite of Python MCP tools is built to simplify management of the vLLM hosted Gemma 4 deployment with Antigravity CLI.
What is this project trying to Do?
This project is a DevOps/SRE assistant that uses a Gemma 4 model hosted on GCE with GPU. It provides tools to provision the Docker container and deploy the model, as well as for observability and performance testing.
This project is similar to a previous project that targeted GPU hosted Gemma4 instances on GCP:
Gemma-SRE: Self-Hosted vLLM Infrastructure Agent
Antigravity CLI
Antigravity CLI is the follow-on successor to Gemini CLI- the terminal driven, agent assisted coding tool.
Full details on installing Antigravity CLI are here:
Getting Started with Antigravity CLI
Testing the Antigravity CLI Environment
Once you have all the tools in place- you can test the startup of Antigravity CLI.
You will need to authenticate with a Google Cloud Project or your Google Account:
agy
This will start the interface:
Full Installation Instructions
The detailed installation instructions for Antigravity CLI are here:
Getting Started with Antigravity CLI
Python MCP Documentation
The official GitHub Repo provides samples and documentation for getting started:
Where do I start?
The strategy for starting MCP development for model management is a incremental step by step approach.
First, the basic development environment is setup with the required system variables, and a working Antigravity CLI configuration.
Then, a minimal Python MCP Server is built with stdio transport. This server is validated with Antigravity CLI in the local environment.
This setup validates the connection from Antigravity CLI to the local server via MCP. The MCP client (Antigravity CLI) and the Python MCP server both run in the same local environment.
Setup the Basic Environment
At this point you should have a working Python environment and a working Antigravity CLI installation. The next step is to clone the GitHub samples repository with support scripts:
cd ~
git clone https://github.com/xbill9/gemma4-tips
Then run init.sh from the cloned directory.
The script will attempt to determine your shell environment and set the correct variables:
g2-4-2B-qat-L4-devops-agent
source init.sh
If your session times out or you need to re-authenticate- you can run the set_env.sh script to reset your environment variables:
g2-4-2B-qat-L4-devops-agent
source set_env.sh
Variables like PROJECT_ID need to be setup for use in the various build scripts- so the set_env script can be used to reset the environment if you time-out.
Model Management Tool with MCP Stdio Transport
One of the key features that the standard MCP libraries provide is abstracting various transport methods.
The high level MCP tool implementation is the same no matter what low level transport channel/method that the MCP Client uses to connect to a MCP Server.
The simplest transport that the SDK supports is the stdio (stdio/stdout) transport — which connects a locally running process. Both the MCP client and MCP Server must be running in the same environment.
The connection over stdio will look similar to this:
# Initialize FastMCP server
mcp = FastMCP("Self-Hosted vLLM DevOps Agent")
Running the Python Code
First- switch the directory with the Python version of the MCP sample code:
~/gemma4-tips/g2-4-2B-qat-L4-devops-agent
Run the release version on the local system:
make install
Processing ./.
The project can also be linted:
xbill@penguin:~/gemma4-tips/g2-4-2B-qat-L4-devops-agent$ make lint
ruff check .
All checks passed!
ruff format --check .
7 files already formatted
mypy .
Success: no issues found in 7 source files
Success: no issues found in 6 source files
Getting Started with Gemma 4 on GPU
The Official vLLM repo also has Gemma4 specific information:
Gemma 2B
Gemma 2B is the latest model in the Gemma4 series. More details are available here:
google/gemma-2b · Hugging Face
and here:
Get started with Gemma models | Google AI for Developers
NVIDIA L4 on GCE
The NVIDIA L4 support on Google Cloud Compute Engine is documented here:
Introducing G2 VMs with NVIDIA L4 GPUs | Google Cloud Blog
Quantization-Aware Training (QAT)
One of the key issues with deploying models is available GPU memory based in the hardware footprint. Along with the new 2B model release- new versions of the model that use QAT to reduce the memory requirements have been released:
Gemma 4 QAT models: Optimizing model compression for mobile and laptop efficiency
QAT Model Versions
What are the differences between the versions on Hugging Face?
> what is the difference between google/gemma-4-2B-it-qat-q4_0-unquantized and https://huggingface.co/google/gemma-
4-2B-it-qat-q4_0-gguf and google/gemma-4-2B-it-qat-w4a16-ct
The differences between these three Gemma 4 2B QAT model variants lie primarily in their serialization format,
quantization state, and intended deployment environment.
All three are part of the Quantization-Aware Training (QAT) collection, meaning they were trained to compensate
for the precision loss of 4-bit quantization.
### 📊 Quick Comparison Table
Feature | qat-q4_0-unquantized | qat-q4_0-gguf | qat-w4a16-ct
-----------------|---------------------------------|--------------------------|--------------------------------
Format | Standard Transformers | GGUF | Compressed Tensors ( -ct )
| (Safetensors) | |
Quantization | Unquantized (Half-precision) | 4-bit (Integer) | 4-bit Weights / 16-bit Act.
Primary Tool | Hugging Face Transformers / | llama.cpp / LM Studio | vLLM / SGLang
| Research | |
Target Hardware | GPUs (A100/H100/L4) for fine- | CPU / Apple Silicon / | NVIDIA L4 / H100 (Cloud
| tuning | Consumer GPU | Serving)
Best Use Case | Speculative decoding, research | Zero-setup local | High-throughput cloud serving
| | deployment |
──────
### 🔍 Detailed Breakdown
#### 1. google/gemma-4-2B-it-qat-q4_0-unquantized
• What it is: These are the weights extracted directly from the QAT pipeline before the final compression step.
• Why use it: Since the weights are still in high precision (typically bfloat16 ), this model is ideal for:
• Research and Fine-tuning: If you want to continue training on top of a QAT-optimized base.
• Speculative Decoding: It can serve as a high-quality "target" model for speculative decoding setups.
• Custom Formats: Used as a source for converting to formats like MLX or CoreML.
#### 2. google/gemma-4-2B-it-qat-q4_0-gguf
• What it is: A quantized version using the GGUF format, optimized for the llama.cpp ecosystem.
• Why use it: This is the "plug-and-play" version for local users.
• Portability: It runs on almost anything—MacBooks (Metal), Linux (CUDA/ROCm), and even Windows CPUs.
• Efficiency: Extremely low RAM usage compared to the unquantized version.
#### 3. google/gemma-4-2B-it-qat-w4a16-ct
• What it is: Serialized using the Compressed Tensors standard. It uses 4-bit weights with 16-bit activations (
w4a16 ).
• Why use it: This is the "production-grade" model for cloud inference.
• VRAM Savings: On an NVIDIA L4 GPU, this model frees up ~18 GB of VRAM compared to the standard bfloat16
version, allowing it to handle up to 512 concurrent users on a single card (as noted in the GEMINI.md for
this workspace).
• Tool Calling: Specifically optimized for vLLM with gemma4 tool-calling and reasoning parsers.
More documentation is available here:
Gemma 4 model overview | Google AI for Developers
MCP stdio Transport
One of the key features that the MCP protocol provides is abstracting various transport methods.
The high level tool MCP implementation is the same no matter what low level transport channel/method that the MCP Client uses to connect to a MCP Server.
The simplest transport that the SDK supports is the stdio (stdio/stdout) transport — which connects a locally running process. Both the MCP client and MCP Server must be running in the same environment.
In this project Antigravity CLI is used as the MCP client to interact with the Python MCP server code.
Antigravity CLI mcp_config.json
A sample MCP server file is provided in the .agents directory:
{
"mcpServers": {
"gpu-devops-agent": {
"command": "python3",
"args": [
"/home/xbill/gemma4-tips/gpu-2B-qat-L4-devops-agent/server.py"
],
"env": {
"GOOGLE_CLOUD_PROJECT": "aisprint-491218",
"GOOGLE_CLOUD_LOCATION": "us-east4",
"VLLM_BASE_URL": "https://gpu-2b-qat-l4-devops-agent-289270257791.us-east4.run.app",
"MODEL_NAME": "/mnt/models/gemma-4-2B-it-qat-w4a16-ct"
}
}
}
}
Validation with Antigravity CLI
The final connection test uses Antigravity CLI as a MCP client with the Python code providing the MCP server:
MCP Servers
Plugins (~/.gemini/antigravity-cli/plugins)
> ✓ google-dev-knowledge Tools: search_documents, answer_query, get_documents
✓ gpu-devops-agent Tools: save_hf_token, get_vllm_endpoint, list_vertex_models, list_bucket_models,
analyze_cloud_logging, +26 more
Model Lifecycle Management via MCP
The MCP tools provide a complete suite of agent-oriented operations for managing vLLM deployment on GCE, Cloud Run or a TPU.
Overview of MCP tools :
Here is the output of the get_help tool:
### 🛠️ GCP Gemma 4 SRE Agent Help & Configuration
You can configure this MCP server using the following environment variables:
GCP Configuration:
• GOOGLE_CLOUD_PROJECT : Your GCP Project ID.
• Current Value: comglitn
• GOOGLE_CLOUD_LOCATION : The GCP Region/Location.
• Current Value: us-east4
• GOOGLE_CLOUD_ZONE : The GCP Zone for GCE VM deployment.
• Current Value: us-east4-a
• BUCKET_NAME : GCS Bucket used to store model weights.
• Current Value: comglitn-bucket
General serving:
• MODEL_NAME : Default Hugging Face repository or GCS path.
• Current Value: google/gemma-4-2B-it-qat-w4a16-ct
• VLLM_BASE_URL : The explicit URL of your vLLM GCE service. (If not set, it is auto-discovered via GCE VM
external IP)
• Current Value: Not set (auto-discovering)
### ℹ️ Active Mode Summary
The server is running in GCP GCE VM mode targeting a g2-standard-4 host VM with NVIDIA L4 GPU.
### 🧰 Available MCP Tools
Below is a summary of the tools exposed by this SRE/DevOps agent:
#### 🐳 Infrastructure & Deployment
• start_gce : Starts an existing GCE instance, or provisions a new one if none exists.
• status_gce : Checks GCE instance status.
• stop_gce : Stops GCE instance.
• check_vllm : Checks the status of the vLLM container and engine running on the GCE instance.
• deploy_vllm : Deploys vLLM to GCP GCE g2-standard-4 (NVIDIA L4) VM instance.
• destroy_vllm : Deletes the GCP GCE vLLM VM instance.
• status_vllm : Checks GCE instance status.
• update_vllm_scaling : Scales GCE instance type vertically.
• get_vllm_deployment_config : Generates the gcloud compute command and startup script.
• get_vllm_gpu_deployment_config : Generates a GKE manifest for GPU (NVIDIA L4).
• check_gpu_quotas : Checks GPU/Accelerator quotas for a region.
• get_vllm_endpoint : Returns the current active vLLM endpoint URL.
#### 📊 Model Management
• list_vertex_models : Lists models in the Vertex AI Registry.
• list_bucket_models : Lists model weights in GCS bucket.
• save_hf_token : Securely saves a Hugging Face API token to Secret Manager.
• get_vertex_ai_model_copy_instructions : Instructions to copy model from Vertex AI Model Garden to GCS.
• get_huggingface_model_copy_instructions : Instructions to download model from Hugging Face and upload to GCS.
• get_huggingfacehub_download_path : Resolves local cache path using huggingface_hub.
#### 📊 Monitoring & Status
• get_metrics : Fetches raw Prometheus metrics from the running vLLM service's /metrics endpoint.
• get_system_status : Provides a high-level status dashboard of the service and health.
• get_endpoint : Verifies connectivity and returns the active service URL.
• get_model_details : Retrieves detailed model metadata and engine state from /v1/models .
• verify_model_health : Deep health check by querying the model with a simple prompt and measuring latency.
#### 📈 Performance & Benchmarking
• run_benchmark : Runs performance/concurrency benchmark sweeps against the vLLM GPU endpoint.
#### 💬 Interaction & Diagnostics
• query_gemma4 : Primary tool to query the self-hosted model with standard chat message format.
• query_gemma4_with_stats : Queries the model and returns streaming performance statistics (TTFT, throughput).
• query_vllm : Direct text completions querying tool.
• analyze_cloud_logging : Fetches logs from GCP Logging and analyzes them using the model.
• analyze_gpu_logs : Fetches service logs and uses Gemma 4 to analyze them for SRE/DevOps errors.
• suggest_sre_remediation : Suggests remediation plans for SRE errors using the model.
Checking System status
The status can be checked with an MCP tool:
> get_system_status
Here is the live dashboard system status:
### 🌀 GPU vLLM System Status
• vLLM Health: 🟢 Online (http://35.199.2.239:8080)
• Hosting Status: 🟢 Running (gpu-2b-qat-l4-devops-agent)
👉 Next Step: The self-hosted DevOps agent is fully operational. You can now use query_gemma4 to
interact with the model.
Cross Check The Deployed Model
Once the model starts locally- the MCP tool allows for model verification:
> verify_model_health
I will run the model verification health check to query the endpoint and measure current response latency.
● Bash(GOOGLE_CLOUD_PROJECT="aisprint-491218" VLLM_BASE_URL="http://35.199.2.239:8080" GOO...) (ctrl+o to
expand)
The model health check has successfully PASSED with excellent response times:
### 🩺 Model Verification Results:
• Model: google/gemma-4-E2B-it-qat-w4a16-ct
• Response: "Hello! Yes, I am working. I am Gemma 4, a large la..."
• Response Latency: 0.39 seconds
• Status: Healthy and fully operational
and model details:
> get_active_models
I will query the active models served by the vLLM engine endpoint using the model details tool.
● Bash(GOOGLE_CLOUD_PROJECT="aisprint-491218" VLLM_BASE_URL="http://35.199.2.239:8080" GOO...) (ctrl+o to
expand)
The active model currently served by the vLLM engine at http://35.199.2.239:8080 is:
### 🧩 Model Details
• Model ID: google/gemma-4-E2B-it-qat-w4a16-ct
• Object Type: model
• Owned By: vllm
• Endpoint Status: Healthy ✅
and status:
> get status
I will fetch the current live status dashboard for you.
● Bash(GOOGLE_CLOUD_PROJECT="aisprint-491218" VLLM_BASE_URL="http://35.199.2.239:8080" GOO...) (ctrl+o to
expand)
Here is the current system status:
### 🌀 GPU vLLM System Status
• vLLM Health: 🟢 Online (http://35.199.2.239:8080)
• Hosting Status: 🟢 Running (gpu-2b-qat-l4-devops-agent)
👉 Next Step: The self-hosted model is active and running cleanly. You can query the agent or interact
with the model using the query_gemma4 tool.
Check the Cloud GCE VM
Once Deployed — the model is visible as a standard VM:
Review the Model
Check the model details:
> what are the model details
The active model details served by your vLLM engine at http://35.199.2.239:8080 are:
### 🧩 Active Model Details
• Model Identifier: google/gemma-4-E2B-it-qat-w4a16-ct
• Object Type: model
• Serving Engine: vllm
• Quantization: INT4 Weights & FP16 Activations (QAT Compressed Tensors format)
• VRAM Weight Footprint: ~1.1 GB on a single NVIDIA L4 GPU (leaving ~21.7 GB free for KV Cache).
Once everything is online — you can send queries to the Model:
> query_gemma4_with_stats what is gemma?
### 📊 Performance Stats
• Model: google/gemma-4-E2B-it-qat-w4a16-ct
• Time to First Token (TTFT): 0.131 seconds
• Total Generation Time: 1.412 seconds
• Throughput: 104.64 tokens/second
• Total Tokens generated (approx.): 134
### 💬 Model Response
│ I am Gemma 4, a Large Language Model developed by Google DeepMind.
│
│ I am part of the Gemma family of open weights models. Gemma models are designed to be powerful,
│ efficient, and accessible for a wide range of tasks in areas like reasoning, generation, and
│ understanding natural language.
│
│ If you are asking about the general concept of "Gemma" models, they are a series of AI models released
by
│ Google that are built upon the research and technology used to create Gemini models. Being "open
weights"
│ means that their underlying parameters and structures are made available to the public so that
│ researchers and developers can study, build upon, and deploy them for their own applications.
Benchmark the Local Model
The local model was benchmarked:
The benchmark sweep has completed successfully. Here are the compiled results across all combinations of context
window sizes and concurrency levels:
The analysis is reviewed:
> summarize benchmark results
Here is a summary of the benchmark results from benchmark_report.md for the Gemma 4 2B QAT model on an
NVIDIA L4 GPU:
### 1. Latency Profile
• Low Load (1 User): Average latency scales from 0.08s (4 tokens) to 1.43s (16K tokens).
• Medium Load (128 Users): Stays under 1.00s for context sizes up to 512 tokens.
• High Load (2048 Users): Ranges from 15.60s (4 tokens) up to 37.68s (16K tokens) due to request queuing.
### 2. Throughput Capabilities
• Peak Throughput: Achieved 166.2 requests/sec at 256 concurrent users (4-token context).
• High Concurrency Throughput: Sustains ~64–91 req/s at 2048 users for context sizes up to 2048 tokens.
• Large Context Throughput: Drops to 14.8 req/s at 16K tokens under 2048 users.
### 3. Stability & Success Rates
• 100% Success Rate is maintained across all context windows up to 1024 concurrent users.
• At 2048 Concurrency:
• 100% success up to 4096 tokens.
• 99.7% success at 8192 tokens.
• 53.4% success at 16,384 tokens (indicating the performance boundary/VRAM limitation of the single L4
GPU).
This is visualized:
GPU Memory Layout
What is the memory layout?
> what is the memory layout on the GPU memory for the model
Here is the physical memory layout of the 24 GB GPU VRAM on the NVIDIA L4 for the Gemma 4 2B QAT model:
+--------------------------------------------------------------------------------+
| NVIDIA L4 GPU VRAM (Total: 24.0 GB) |
+------------------------------------+-------------------------+-----------------+
| vLLM Engine Allocation (95%: 22.8 GB) | Unallocated |
| | (5%: 1.2 GB) |
+------------------+-----------------+-------------------------+-----------------+
| Model Weights | CUDA & OS | KV Cache | Safe Margin |
| (INT4 Quantized) | Run Context | (FP8 Quantized) | (Prevents OOM) |
| | | | |
| ~1.1 GB | ~1.2 GB | ~20.5 GB | 1.2 GB |
| | | | |
| 2 Billion | Kernel launches | 2,993,064 max tokens | |
| params @ 4-bit | & memory buffers| concurrent capacity | |
+------------------+-----------------+-------------------------+-----------------+
### Layout Breakdown:
1. Model Weights (~1.1 GB):
• By using 4-bit Quantization-Aware Training (QAT), the 2 billion parameters consume only 0.5 bytes
per parameter (INT4).
• Weight Footprint: 2.0 billion × 0.5 bytes ≈ 1.0 GB (plus minor model architecture overhead).
2. CUDA Context & Runtime Overhead (~1.2 GB):
• Memory consumed by loaded CUDA kernels, vLLM runtime buffers, and basic GPU management libraries.
3. KV Cache (~20.5 GB):
• Allocated dynamically for active queries. At FP8 quantization, this space fits 2,993,064 concurrent
tokens across running user contexts.
4. System Overhead Margin (1.2 GB):
• Kept strictly unallocated via --gpu-memory-utilization 0.95 to safeguard against memory spikes
during execution.
Compare to Other Deployments
The GCE 2B GPU deployment was reviewed:
I have analyzed and compared the Gemma 4 2B benchmarks across the requested platforms: TPU v6e-1, TPU v6e-
4, GPU L4 VM (GCE), and Cloud Run (L4).
The complete comparison report has been compiled and saved as an artifact:
• 53.4% success at 16,384 tokens (indicating the performance boundary/VRAM limitation of the single L4
GPU).
• 2B Platform Comparison Report: 2b_platform_comparison.md
### 🚀 Key Comparison Highlights
1. Latency Stability (TPU v6e-4 Winner):
• At high concurrency (1024 concurrent users with a 1024-token context), the TPU v6e-4 maintains an
average latency of 0.869 seconds, compared to 9.720 seconds on the GPU L4 GCE VM and 23.222 seconds on
Cloud Run.
• This makes TPU v6e-4 the optimal platform for sub-second, real-time SRE diagnostics under mass
parallel loads.
2. Throughput Efficiency (GPU L4 VM Winner under Low-Mid Loads):
• The dedicated GPU L4 VM (running the QAT w4a16 quantized model) achieves the highest throughput under
low-to-medium concurrency. For a 1024-token context size under 16 concurrent users, the GPU VM handles
50.98 req/s vs. 30.60 req/s on the TPU v6e-4.
3. Scale Limitations (TPU v6e-1):
• The single-chip TPU v6e-1 has exceptional peak throughput at 256 users for short contexts (137.30
req/s), but runs out of memory or times out at high concurrency combined with larger contexts (e.g.
8192 tokens).
4. Serverless Penalty (Cloud Run L4):
• Serving over Cloud Run shows heavy cold-start wait times (up to 22.7s for single requests) and
container queuing overheads, restricting maximum throughput to ~24.5 req/s.
What about other Gemma models?
summarize
Here is a summary of the diagnostic and benchmarking work completed during this session:
### 1. 🛠 Execution of 2D Grid Concurrency Sweep
• We successfully launched and executed a 2D performance sweep (13 context windows up to 16K × 12
concurrency levels up to 2048 users) for the Gemma 4 2B QAT model ( google/gemma-4-E2B-it-qat-w4a16-ct )
served via vLLM on a single NVIDIA L4 GPU GCE VM.
• Results Summary Artifact: benchmark_results_summary.md
### 2. 📊 Platform Performance Comparison (TPU vs. GPU)
• We parsed and compared 2B performance across TPU v6e-1, TPU v6e-4, GPU L4 GCE VM, and Cloud Run.
• Findings: The TPU v6e-4 is the latency champion under massive concurrency (0.869s latency under 1024
users vs. 9.72s for GPU GCE and 23.22s for Cloud Run). Under low-to-medium loads, the GPU L4 VM yields the
highest throughput.
• Platform Comparison Artifact: 2b_platform_comparison.md
### 3. 🧠 Model Parameter Size Analysis (2B vs. 4B, 12B, 26B, 31B)
• We evaluated parameter scaling. The 2B QAT is highly concurrent (fits inside ~4 GB VRAM, leaving ~19 GB
free for the KV cache), sustaining 1024+ concurrent users. The 12B QAT model (occupying ~6 GB VRAM) serves
as the optimal sweet spot, providing advanced SRE reasoning while sustaining up to 64 concurrent users on a
single L4 GPU.
• Model Size Comparison Artifact: model_size_comparison.md
### 💰 4. Financial Cost Analysis
• Under high loads (256 concurrency), GCE GPU VMs offer the lowest cost-per-million tokens ($0.0016 vs. $0.
023 on TPU v6e-4).
• For intermittent, spiky workloads (e.g. SRE alert diagnostics), serverless Cloud Run (L4) ($3.76/day
active time) or Spot GCE GPU VMs (~$3.60/day) are the most cost-effective because they scale down or run at
heavily discounted rates.
• Cost Analysis Artifact: cost_analysis.md
And Price/Performance?
The winner is…
I have compiled a detailed Price-to-Performance analysis for the Gemma 4 2B model across all platforms.
You can read the full report here:
• Price/Performance Analysis Report: price_performance_analysis.md
### 📊 Key Insights:
1. GPU L4 VM (The Value Leader):
• The dedicated GPU L4 VM running the QAT model is the absolute price-to-performance winner. On a Spot
instance, it delivers 513.27 Req/s per dollar at high concurrency (256 users, 1K context). On an On-
Demand instance, it delivers 171.09 Req/s per dollar.
2. TPU v6e-1 High Concurrency Efficiency:
• The TPU v6e-1 scales extremely well as concurrency climbs, delivering 51.26 Req/s per dollar at 256
concurrent users.
3. TPU v6e-4 Over-Provisioning:
• Deploying the small 2B model on a TPU v6e-4 (4 chips, $5.50/hr) is highly inefficient. Because 2B is
so small, the extra chips do not yield a corresponding throughput increase (63.89 req/s on v6e-4 vs.
70.22 req/s on v6e-1), dropping the price/performance metric to 11.62 Req/s per dollar. TPU v6e-4 is
over-provisioned for 2B and should be reserved for 12B/31B models.
Summary
The strategy for using MCP for Gemma 4 GPU deployment with Antigravity CLI was validated with a incremental step by step approach.
A minimal stdio transport MCP Server was started from Python source code and validated with Antigravity CLI running as a MCP client in the same local environment. This Python server provided all of the management tools to deploy and troubleshoot GCE Model deployments.
Top comments (0)