This article provides a step by step deployment guide for Gemma 4 to a Google Compute Engine hosted GPU enabled system. A suite of Python MCP tools is built to simplify management of the vLLM hosted Gemma 4 deployment with Antigravity CLI.
What is this project trying to Do?
This project is a DevOps/SRE assistant that uses a Gemma 4 model hosted on GCE with GPU. It provides tools to provision the Docker container and deploy the model, as well as for observability and performance testing.
This project is similar to a previous project that targeted GPU hosted Gemma4 instances on GCP:
Gemma-SRE: Self-Hosted vLLM Infrastructure Agent
Antigravity CLI
Antigravity CLI is the follow-on successor to Gemini CLI- the terminal driven, agent assisted coding tool.
Full details on installing Antigravity CLI are here:
Getting Started with Antigravity CLI
Testing the Antigravity CLI Environment
Once you have all the tools in place- you can test the startup of Antigravity CLI.
You will need to authenticate with a Google Cloud Project or your Google Account:
agy
This will start the interface:
Full Installation Instructions
The detailed installation instructions for Antigravity CLI are here:
Getting Started with Antigravity CLI
Python MCP Documentation
The official GitHub Repo provides samples and documentation for getting started:
Where do I start?
The strategy for starting MCP development for model management is a incremental step by step approach.
First, the basic development environment is setup with the required system variables, and a working Antigravity CLI configuration.
Then, a minimal Python MCP Server is built with stdio transport. This server is validated with Antigravity CLI in the local environment.
This setup validates the connection from Antigravity CLI to the local server via MCP. The MCP client (Antigravity CLI) and the Python MCP server both run in the same local environment.
Setup the Basic Environment
At this point you should have a working Python environment and a working Antigravity CLI installation. The next step is to clone the GitHub samples repository with support scripts:
cd ~
git clone https://github.com/xbill9/gemma4-tips
Then run init.sh from the cloned directory.
The script will attempt to determine your shell environment and set the correct variables:
g2-4-26B-qat-L4-devops-agent
source init.sh
If your session times out or you need to re-authenticate- you can run the set_env.sh script to reset your environment variables:
g2-4-26B-qat-L4-devops-agent
source set_env.sh
Variables like PROJECT_ID need to be setup for use in the various build scripts- so the set_env script can be used to reset the environment if you time-out.
Model Management Tool with MCP Stdio Transport
One of the key features that the standard MCP libraries provide is abstracting various transport methods.
The high level MCP tool implementation is the same no matter what low level transport channel/method that the MCP Client uses to connect to a MCP Server.
The simplest transport that the SDK supports is the stdio (stdio/stdout) transport — which connects a locally running process. Both the MCP client and MCP Server must be running in the same environment.
The connection over stdio will look similar to this:
# Initialize FastMCP server
mcp = FastMCP("Self-Hosted vLLM DevOps Agent")
Running the Python Code
First- switch the directory with the Python version of the MCP sample code:
~/gemma4-tips/g2-4-26B-qat-L4-devops-agent
Run the release version on the local system:
make install
Processing ./.
The project can also be linted:
xbill@penguin:~/gemma4-tips/g2-4-26B-qat-L4-devops-agent$ make lint
ruff check .
All checks passed!
ruff format --check .
7 files already formatted
mypy .
Success: no issues found in 7 source files
Success: no issues found in 6 source files
Getting Started with Gemma 4 on GPU
The Official vLLM repo also has Gemma4 specific information:
Gemma 26B
Gemma 2B is the latest model in the Gemma4 series. More details are available here:
google/gemma-4-26B-A4B-it · Hugging Face
and here:
Get started with Gemma models | Google AI for Developers
NVIDIA L4 on GCE
The NVIDIA L4 support on Google Cloud Compute Engine is documented here:
Introducing G2 VMs with NVIDIA L4 GPUs | Google Cloud Blog
Quantization-Aware Training (QAT)
One of the key issues with deploying models is available GPU memory based in the hardware footprint. Along with the new 2B model release- new versions of the model that use QAT to reduce the memory requirements have been released:
Gemma 4 QAT models: Optimizing model compression for mobile and laptop efficiency
QAT Model Versions
What are the differences between the versions on Hugging Face?
> what is the difference between
gemma-4-26B-A4B-it-qat-q4_0-unquantized and
gemma-4-26B-A4B-it-qat-q4_0-gguf and
google/gemma-4-26B-A4B-it-qat-w4a16-ct
While we wait for your 26B model to finish loading its weights into the GPU, here is the breakdown of the
differences between those three specific Gemma 4 26B MoE (Mixture of Experts) QAT variants:
### 1. gemma-4-26B-A4B-it-qat-q4_0-unquantized
• Format: High-precision weights (usually bfloat16).
• What it is: These are the "master" weights that came directly out of the Quantization-Aware Training (QAT)
process. Even though they are stored in high precision, the model has already "learned" how to behave like a 4-
bit model.
• Primary Use: You would use this if you want to convert the model to a new, custom format or for research
purposes where you need the full precision gradient/activation data.
### 2. gemma-4-26B-A4B-it-qat-q4_0-gguf
• Format: GGUF (GPT-Generated Unified Format).
• What it is: This version is quantized using the q4_0 method (4-bit quantization) specifically for the
llama.cpp ecosystem.
• Primary Use: Local deployment. This is what you would use if you wanted to run this 26B model on your laptop,
an Apple Silicon Mac (using LM Studio/Ollama), or a consumer NVIDIA GPU with limited VRAM. It is optimized for
"one user" scenarios with low latency.
### 3. google/gemma-4-26B-A4B-it-qat-w4a16-ct
• Format: Compressed-Tensors ( -ct ) with w4a16 (4-bit Weights, 16-bit Activations).
• What it is: This is the version we are currently deploying on your GCE instance. It uses the compressed-
tensors standard natively supported by vLLM.
• Primary Use: Cloud/Enterprise Serving. It is designed for high-concurrency environments (multiple users at
once). Because it keeps the activations in 16-bit while quantizing weights to 4-bit, it maintains higher
accuracy for complex tasks like tool-calling and reasoning compared to standard 4-bit formats.
──────
### Summary Comparison Table
Feature | Unquantized | GGUF ( q4_0 ) | Compressed-Tensors ( w4a16-ct )
-------------------------|-------------------------|------------------------|----------------------------------
Precision | bfloat16 (High) | 4-bit (Compressed) | 4-bit Weight / 16-bit Activation
Target Engine | PyTorch / Transformers | llama.cpp / Ollama | vLLM / SGLang
Best Hardware | Research Servers | Mac Studio / Local PCs | NVIDIA L4 / A100 / H100
Main Advantage | Maximum Fidelity | Smallest File Size | Highest Throughput
│ [!NOTE]
│ All three models share the 26B-A4B architecture, which means they have 26 Billion total parameters, but only
│ ~4 Billion are active for any single token generation (Mixture of Experts), allowing for the speed of a 4B
│ model with the knowledge of a 26B model.
More documentation is available here:
Gemma 4 model overview | Google AI for Developers
MCP stdio Transport
One of the key features that the MCP protocol provides is abstracting various transport methods.
The high level tool MCP implementation is the same no matter what low level transport channel/method that the MCP Client uses to connect to a MCP Server.
The simplest transport that the SDK supports is the stdio (stdio/stdout) transport — which connects a locally running process. Both the MCP client and MCP Server must be running in the same environment.
In this project Antigravity CLI is used as the MCP client to interact with the Python MCP server code.
Antigravity CLI mcp_config.json
A sample MCP server file is provided in the .agents directory:
{
"mcpServers": {
"gpu-devops-agent": {
"command": "python3",
"args": [
"/home/xbill/gemma4-tips/gpu-26B-qat-L4-devops-agent/server.py"
],
"env": {
"GOOGLE_CLOUD_PROJECT": "aisprint-491218",
"GOOGLE_CLOUD_LOCATION": "us-east4",
"VLLM_BASE_URL": "https://gpu-26b-qat-l4-devops-agent-289270257791.us-east4.run.app",
"MODEL_NAME": "/mnt/models/gemma-4-26B-A4B-it-qat-w4a16-ct"
}
}
}
}
Validation with Antigravity CLI
The final connection test uses Antigravity CLI as a MCP client with the Python code providing the MCP server:
MCP Servers
Plugins (~/.gemini/antigravity-cli/plugins)
> ✓ google-dev-knowledge Tools: search_documents, answer_query, get_documents
✓ gpu-devops-agent Tools: save_hf_token, get_vllm_endpoint, list_vertex_models, list_bucket_models,
analyze_cloud_logging, +26 more
Model Lifecycle Management via MCP
The MCP tools provide a complete suite of agent-oriented operations for managing vLLM deployment on GCE, Cloud Run or a TPU.
Overview of MCP tools :
Here is the output of the get_help tool:
### 🛠️ GCP Gemma 4 SRE Agent Help & Configuration
You can configure this MCP server using the following environment variables:
GCP Configuration:
• GOOGLE_CLOUD_PROJECT : Your GCP Project ID.
• Current Value: comglitn
• GOOGLE_CLOUD_LOCATION : The GCP Region/Location.
• Current Value: us-east4
• GOOGLE_CLOUD_ZONE : The GCP Zone for GCE VM deployment.
• Current Value: us-east4-a
• BUCKET_NAME : GCS Bucket used to store model weights.
• Current Value: comglitn-bucket
• MODEL_NAME : Default Hugging Face repository or GCS path.
• Current Value: google/gemma-4-2B-it-qat-w4a16-ct
• VLLM_BASE_URL : The explicit URL of your vLLM GCE service. (If not set, it is auto-discovered via GCE VM
external IP)
• Current Value: Not set (auto-discovering)
#### 🐳 Infrastructure & Deployment
• start_gce : Starts an existing GCE instance, or provisions a new one if none exists.
• status_gce : Checks GCE instance status.
• stop_gce : Stops GCE instance.
• check_vllm : Checks the status of the vLLM container and engine running on the GCE instance.
• deploy_vllm : Deploys vLLM to GCP GCE g2-standard-4 (NVIDIA L4) VM instance.
• destroy_vllm : Deletes the GCP GCE vLLM VM instance.
• status_vllm : Checks GCE instance status.
• update_vllm_scaling : Scales GCE instance type vertically.
• get_vllm_deployment_config : Generates the gcloud compute command and startup script.
• get_vllm_gpu_deployment_config : Generates a GKE manifest for GPU (NVIDIA L4).
• check_gpu_quotas : Checks GPU/Accelerator quotas for a region.
• get_vllm_endpoint : Returns the current active vLLM endpoint URL.
#### 📊 Model Management
• list_vertex_models : Lists models in the Vertex AI Registry.
• list_bucket_models : Lists model weights in GCS bucket.
• save_hf_token : Securely saves a Hugging Face API token to Secret Manager.
• get_vertex_ai_model_copy_instructions : Instructions to copy model from Vertex AI Model Garden to GCS.
• get_huggingface_model_copy_instructions : Instructions to download model from Hugging Face and upload to GCS.
• get_huggingfacehub_download_path : Resolves local cache path using huggingface_hub.
#### 📊 Monitoring & Status
• get_metrics : Fetches raw Prometheus metrics from the running vLLM service's /metrics endpoint.
• get_system_status : Provides a high-level status dashboard of the service and health.
• get_endpoint : Verifies connectivity and returns the active service URL.
• get_model_details : Retrieves detailed model metadata and engine state from /v1/models .
• verify_model_health : Deep health check by querying the model with a simple prompt and measuring latency.
#### 📈 Performance & Benchmarking
• run_benchmark : Runs performance/concurrency benchmark sweeps against the vLLM GPU endpoint.
#### 💬 Interaction & Diagnostics
• query_gemma4 : Primary tool to query the self-hosted model with standard chat message format.
• query_gemma4_with_stats : Queries the model and returns streaming performance statistics (TTFT, throughput).
• query_vllm : Direct text completions querying tool.
• analyze_cloud_logging : Fetches logs from GCP Logging and analyzes them using the model.
• analyze_gpu_logs : Fetches service logs and uses Gemma 4 to analyze them for SRE/DevOps errors.
• suggest_sre_remediation : Suggests remediation plans for SRE errors using the model.
Find and Exterminate
First find a zone with available GPU:
The find_quota scan across multiple Google Cloud regions is complete. You have significant available quota
for NVIDIA L4 GPUs in several regions.
### ✅ Available NVIDIA L4 Quota Summary
All listed regions currently show 8 units of available quota (both standard and Spot/Preemptible):
• us-central1 (Iowa)
• us-east1 (South Carolina)
• us-east4 (Northern Virginia) - Note: While quota is available, us-east4-a previously reported a resource
stockout.
• us-east5 (Columbus, Ohio)
• us-west1 (Oregon)
• us-west4 (Las Vegas)
• northamerica-northeast1 (Montreal)
• europe-west1 (Belgium)
• europe-west2 (London)
• europe-west4 (Netherlands)
• asia-southeast1 (Singapore)
### 🚀 Recommendation
Since we encountered a stockout in us-east4-a , I recommend attempting your deployment in one of the following
high-availability zones:
1. us-central1-a (Iowa)
2. us-east1-b (South Carolina)
3. us-west1-a (Oregon)
deploy_with_search()
Checking System status
The status can be checked with an MCP tool:
> get_system_status
414.2k in | 3.8k out | 3.0k thinking
● Bash(curl -v http://35.243.250.174:8080/health) (ctrl+o to expand)
The Gemma 4 26B MoE QAT system is currently healthy and fully operational. While the automated
get_system_status tool is reporting "Offline" due to a configuration mismatch in the background MCP server
(which is still defaulting to the 4B instance settings), I have verified the 26B deployment directly.
### 🌀 Gemma 4 26B MoE System Status
• Instance Name: gpu-26b-qat-l4-devops-agent
• Zone: us-east1-b
• External IP: 35.243.250.174
• vLLM Health: 🟢 Online (Verified via direct API check)
• Model ID: /mnt/models/gemma-4-26B-A4B-it-qat-w4a16-ct
• Serving Engine: vLLM (Compressed-Tensors w4a16)
### 📊 Performance Summary
• TTFT (Time to First Token): 0.536s
• Generation Speed: 62.99 tokens/s
• KV Cache: FP8 Enabled
• VRAM Utilization: ~95% (Optimized for MoE concurrency)
│ [!TIP]
│ Your previous query ("What is Gemma?") was successful with the new configuration. If you would like to run a
│ full benchmark sweep on this 26B model, you can now use the run_benchmark tool or call
│ query_gemma4_with_stats for individual prompts.
Cross Check The Deployed Model
Once the model starts locally- the MCP tool allows for model verification:
> verify_model_health
5.7k in | 438 out | 367 thinking
● gpu-devops-agent/verify_model_health(Verify model health) (ctrl+o to expand)
588.1k in | 6.4k out | 3.0k thinking
● Bash(rm /home/xbill/.gemini/antigravity-cli/brain/341f5c29-5818-4321-a71f-23056ab20e18/scratc...) (ctrl+o to
expand)
The model health check is successful. I have verified the deep operational health of the Gemma 4 26B MoE QAT
model.
### ✅ Model Health Check: PASSED
• Model: /mnt/models/gemma-4-26B-A4B-it-qat-w4a16-ct
• Status: Healthy 🟢
• Verification Response: "Yes, I am working correctly! How can I help you today?"
• Inference Latency: 0.47 seconds (Highly optimized for MoE architecture)
### ⚙️ System Metrics
• Active IP: 35.243.250.174 (GCE Instance)
• Engine: vLLM with CompressedTensors support.
• KV Cache: FP8 enabled (Optimized for context window stability).
and model details:
> get_active_models
The active model currently served by the vLLM engine is:
### 🧩 Active Models
• Model ID: /mnt/models/gemma-4-26B-A4B-it-qat-w4a16-ct
• Object Type: model
• Owned By: vllm
• Max Model Length: 32,768 tokens
• Created: 2026-07-12 (Unix: 1783891987 )
This model is the Gemma 4 26B MoE QAT variant, utilizing 4-bit weights and 16-bit activations, optimized for
high-throughput serving on your NVIDIA L4 GPU.
and status:
> get status
### 🚀 Gemma 4 26B MoE QAT Status
• vLLM Engine: 🟢 Active & Healthy (Verified via direct API check)
• Model: google/gemma-4-26B-A4B-it-qat-w4a16-ct
• Instance IP: 35.243.250.174 (Port 8080)
• Current Latency: 0.47s (Inference)
• Current Throughput: ~63 tokens/s
### 🛠 Infrastructure
• Compute Instance: gpu-26b-qat-l4-devops-agent (Running)
• Zone: us-east1-b
• Accelerator: NVIDIA L4 GPU (16GB VRAM)
• Quantization: Weights 4-bit / Activations 16-bit (QAT)
Check the Cloud GCE VM
Once Deployed — the model is visible as a standard VM:
Review the Model
Check the model details:
> what are the model details
The active model is a Gemma 4 26B Mixture of Experts (MoE) variant, specifically optimized for cloud serving on
NVIDIA L4 hardware using Quantization-Aware Training (QAT).
### 🧩 Core Model Metadata
• Model ID: /mnt/models/gemma-4-26B-A4B-it-qat-w4a16-ct
• Architecture: Gemma4ForConditionalGeneration (26B total parameters / 4B active per token).
• Quantization: w4a16-ct (4-bit Weights / 16-bit Activations).
• Context Window: 32,768 tokens (Max Length).
• Storage Path: Local GCS FUSE mount at /mnt/models .
### 🚀 Serving Configuration
• Engine: vLLM (Optimized for Compressed-Tensors).
• KV Cache: FP8 precision (reduces memory usage to allow for larger batch sizes).
• Hardware: 1x NVIDIA L4 GPU (16GB VRAM) on GCP GCE.
• Performance: ~63 tokens/second with sub-600ms Time-to-First-Token (TTFT).
### ⚖️ Permissions & Capabilities
• Sampling: Enabled (Supports temperature , top_p , etc.)
• Logprobs: Enabled (Supports detailed token probability analysis)
• Reasoning: Supported (Gemma 4 native tool-calling and reasoning chains)
and metrics:
> get_metrics
12.5k in | 900 out | 360 thinking
● Bash(curl -s http://35.243.250.174:8080/metrics | grep "cache") (ctrl+o to expand)
The real-time metrics for the Gemma 4 26B MoE QAT deployment (scraped directly from the vLLM Prometheus
endpoint at 35.243.250.174 ) are as follows:
### 📈 Current Performance Metrics
• Requests Running: 0.0 (Idle)
• Requests Waiting: 0.0 (No queue)
• Generation Tokens Processed: 790 (Total since last restart)
• KV-Cache Usage: 0.0% (Memory is currently clear)
• Prompt Tokens Processed: 75 (Total since last restart)
### ⚙️ Engine Configuration Metrics
• GPU Memory Utilization: 0.95 (95% VRAM reserved for KV cache)
• KV Cache Dtype: fp8 (High-efficiency 8-bit cache enabled)
• Total GPU Blocks: 10,656
• Total KV Cache Capacity: 89,326 tokens
• Prefix Caching: Enabled (Optimized for repeated prompts)
### 📊 Token Accounting
Metric | Value
-------------------------------------------------------|-------------------------------------------------------
Prefill Tokens (Total) | 75
Generation Tokens (Total) | 790
Cache Hits (Total) | 0
Tokens per GPU Block | 16
│ [!NOTE]
│ The model is currently idle but fully initialized. The fp8 KV cache configuration allows for a significant
│ context window (~89k tokens) even on a single NVIDIA L4 GPU.
Once everything is online — you can send queries to the Model:
> query_gemma4_with_stats what is gemma?
15.5k in | 418 out | 0 thinking
● Bash(rm /home/xbill/.gemini/antigravity-cli/brain/341f5c29-5818-4321-a71f-23056ab20e18/scratc...) (ctrl+o to
expand)
### 📊 Performance Stats (Gemma 4 26B MoE QAT)
• Model: /mnt/models/gemma-4-26B-A4B-it-qat-w4a16-ct
• Time to First Token (TTFT): 0.563s
• Total Generation Time: 10.828s
• Tokens per Second: 63.03 tokens/s
• Total Tokens (approx.): 647
### 💬 Model Response: "What is Gemma?"
│ Gemma is a family of lightweight, state-of-the-art open models developed by Google DeepMind.
│
│ If Gemini is Google's massive "closed" engine, Gemma is the smaller "open" sibling designed for developers to
│ download, customize, and run locally. It inherits the same research and technology used in Gemini, offering
│ high-level reasoning and instruction-following in a compact, efficient format.
### ⚙️ Why use Gemma 4 26B MoE?
• Efficiency: Uses Mixture of Experts (MoE), meaning it has the knowledge of a 26B model but only activates ~4B
parameters per token for speed.
• Privacy: Since it runs on your private GCE instance, data never leaves your infrastructure.
• Cost: Once deployed, you avoid per-token API fees while maintaining performance comparable to much larger
models.
│ [!TIP]
│ The model is performing at a highly efficient 63 tokens/second. You can continue using
│ query_gemma4_with_stats for further prompts.
Benchmark the Local Model
The local model was benchmarked:
The benchmark sweep has completed successfully. Here are the compiled results across all combinations of context
window sizes and concurrency levels
The analysis is reviewed:
> summarize benchmark results compare and contrast
The 2D Grid Benchmark Sweep for the Gemma 4 26B MoE QAT model is complete.
The results demonstrate the high efficiency of the Mixture of Experts (MoE) architecture combined with 4-bit
QAT and FP8 KV-caching. The model maintained a 100% success rate up to 1024 concurrent users even at the
largest context windows.
### 📊 Performance Summary
• Peak Throughput: ~81.8 Requests/sec (at 256 concurrent users).
• Context Stability: 100% success rate up to 8192 tokens across all concurrency levels.
• MoE Advantage: Typical TTFT remains under 600ms, providing the responsiveness of a 4B model with the
reasoning power of 26B.
• The "Memory Wall": At the extreme edge (2048 users @ 16k context), the success rate dropped to 48% as the
system reached the physical VRAM limits of the NVIDIA L4 GPU.
### 📄 Full Report & Artifacts
I have generated a detailed performance analysis which you can view here:
• Benchmark Report: benchmark_report_26b.md
• Performance Chart: benchmark_chart.png
• Raw Data: benchmark_sweep_results.csv
This is visualized:
Compare to Other Deployments
The GCE deployment was reviewed:
> summarize gemma4_family_comparison.md
The gemma4_family_comparison.md report evaluates the Gemma 4 QAT family (2B, 4B, 12B, 26B MoE) on a single
NVIDIA L4 GPU.
### 🚀 Key Takeaways:
• The "MoE Miracle": The 26B MoE model is the clear winner for SRE tasks. It provides 26B-level intelligence
but operates at 4B-level speeds (activating only ~4B parameters per token).
• Surprising Large-Context Performance: At the 16k context window, the 26B MoE is the fastest model in the
entire family, outperforming even the 2B and 4B variants in both latency (4.19s) and throughput.
• Throughput Scaling:
• 2B QAT: The speed leader for simple tasks (~166 Req/s).
• 26B MoE QAT: The efficiency leader (~82 Req/s), consistently beating the 12B dense model.
• SRE Recommendation:
• Use 4B for high-volume, simple automation.
• Use 26B MoE for complex reasoning, multi-tool orchestration, and large-log analysis. It offers the best
balance of reasoning depth and serving performance.
│ [!NOTE]
│ The 26B MoE's ability to outperform smaller models at 16k context suggests its architecture is better
│ optimized for large KV-cache blocks when paired with fp8 quantization.
what about across other deployments?
### 📊 Cross-Deployment Summary
Your current configuration on GCE is highly optimized and actually outperforms the AWS L4 deployment in
throughput for most standard workloads.
Metric | GCE (Current L4) | AWS (Legacy L4) | RTX 6000 (Local)
---------------------------|---------------------------|---------------------------|---------------------------
Peak Throughput | 81.8 Req/s | 72.2 Req/s | ~90 Req/s
Avg Throughput (1k) | 50.8 Req/s | 26.1 Req/s | ~85 Req/s
Latency (1k context) | 1.52s | 2.70s | < 1.0s
Throughput (16k) | 17.1 Req/s | 19.9 Req/s | ~27 Req/s
### 💡 Key Findings
• GCE Optimization: Your current GCE setup reaches nearly 2x the throughput of the AWS L4 instance for medium-
sized context windows (1k tokens). This is largely due to the aggressive 0.95 memory utilization and fp8
KV-cache settings we applied.
• Infrastructure Parity: While the RTX 6000 is naturally faster due to its higher memory bandwidth, your GCE L4
deployment is very competitive at large context windows (16k), holding 63% of the RTX 6000's performance on
much cheaper hardware.
• Model Advantage: All reports confirm that the 26B MoE variant is the only way to get 26B reasoning on a
single 16GB L4. The dense 26B models consistently struggle or fail to load on single-L4 instances.
### 🚀 Key Performance Summary
• GCE is the Cloud Leader: Your current GCE L4 deployment is significantly more optimized than the AWS L4
equivalent, achieving nearly 2x the throughput for standard 1k context windows (50.8 vs 26.1 Req/s).
• Competitive with Local Hardware: While the local RTX 6000 is the overall speed champion, your GCE L4
deployment holds 63% of its performance at large (16k) context windows, which is highly efficient for a 16GB
data-center card.
• Latency Advantage: GCE maintains a much tighter latency profile (1.52s) compared to AWS (2.70s) for 1k token
requests.
### 💡 Infrastructure Insights
• Optimization Success: The superior performance on GCE is attributed to our aggressive 0.95 memory utilization
and fp8 KV-cache configuration.
• Architecture Benefit: The 26B MoE architecture is confirmed as the only viable way to run 26B-level
intelligence on a single 16GB L4 card; dense 26B models typically fail to load or lack sufficient KV-cache
space on the same hardware.
Cost Analysis
The current Google Cloud pricing is used to evaluate the cost per token:
> how does the GCE with GPU pricing compare to cloud run and other deployment options
> how does the GCE with GPU pricing compare to cloud run and other deployment options
● Create(/home/xbill/.gemini/antigravity-cli/brain...1-a71f-23056ab20e18/pricing_comparison.md) (ctrl+o to expand)
20.9k in | 1.8k out | 689 thinking
The pricing comparison reveals that your current GCP GCE Spot deployment is by far the most cost-effective way
to run the Gemma 4 26B MoE model.
### 💵 Cost Summary (NVIDIA L4)
Deployment Option | Hourly Rate (Spot) | Hourly Rate (On-Demand)
-------------------------------------|------------------------------------|------------------------------------
GCP GCE (Current) | ~$0.16 / hr | ~$0.62 / hr
GCP Cloud Run | N/A | ~$0.67 / hr (active)
AWS EC2 | ~$0.30 / hr | ~$0.98 / hr
### 💡 Strategic Breakdown
• The Spot Advantage: By using Spot VMs on GCE, you are paying ~75% less than you would on AWS and
significantly less than a 24/7 Cloud Run instance.
• Cloud Run (Serverless): This only becomes cheaper if your model is idle for more than 75% of the day, as it
can "scale to zero." However, with the 26B model, "cold starts" (loading the 13GB weights) can take several
minutes, which might degrade user experience.
• The MoE Value Proposition: Because the 26B MoE model runs on a single L4 GPU (where a dense 26B model would
likely require two), you are getting double the intelligence per dollar.
### 📄 Detailed Report
For a full breakdown including "Intelligence per Dollar" analysis, see:
• Pricing Comparison: pricing_comparison.md
At the current Spot rate, running this 26B-class agent 24/7 costs approximately $115/month, which is vastly
cheaper than equivalent per-token costs for large closed models.
Summary
The strategy for using MCP for Gemma 4 GPU deployment with Antigravity CLI was validated with a incremental step by step approach.
A minimal stdio transport MCP Server was started from Python source code and validated with Antigravity CLI running as a MCP client in the same local environment. This Python server provided all of the management tools to deploy and troubleshoot GCE Model deployments.
Top comments (0)