DEV Community: Google Developer Experts

12B Gemma 4 Deployment with NVIDIA Blackwell 6000, MCP, Cloud Run, and Antigravity CLI

xbill — Mon, 22 Jun 2026 02:44:39 +0000

This article provides a step by step deployment guide for Gemma 4 to a Google Cloud Run hosted GPU enabled system,. A suite of Python MCP tools is built to simplify management of the vLLM hosted Gemma 4 deployment with Antigravity CLI.

What is this project trying to Do?

This project is a DevOps/SRE assistant that uses a Gemma 4 model hosted on Cloud Run with GPU. It provides tools to provision the Docker container and deploy the model, as well as for observability and performance testing.

This project is similar to a previous project that targeted GPU hosted Gemma4 instances on GCP:

Gemma-SRE: Self-Hosted vLLM Infrastructure Agent

Antigravity CLI

Antigravity CLI is the follow-on successor to Gemini CLI- the terminal driven, agent assisted coding tool.

Full details on installing Antigravity CLI are here:

Getting Started with Antigravity CLI

Testing the Antigravity CLI Environment

Once you have all the tools in place- you can test the startup of Antigravity CLI.

You will need to authenticate with a Google Cloud Project or your Google Account:

agy

This will start the interface:

Full Installation Instructions

The detailed installation instructions for Antigravity CLI are here:

Getting Started with Antigravity CLI

Python MCP Documentation

The official GitHub Repo provides samples and documentation for getting started:

GitHub - modelcontextprotocol/python-sdk: The official Python SDK for Model Context Protocol servers and clients

Where do I start?

The strategy for starting MCP development for model management is a incremental step by step approach.

First, the basic development environment is setup with the required system variables, and a working Antigravity CLI configuration.

Then, a minimal Python MCP Server is built with stdio transport. This server is validated with Antigravity CLI in the local environment.

This setup validates the connection from Antigravity CLI to the local server via MCP. The MCP client (Antigravity CLI) and the Python MCP server both run in the same local environment.

Setup the Basic Environment

At this point you should have a working Python environment and a working Antigravity CLI installation. The next step is to clone the GitHub samples repository with support scripts:

cd ~
git clone https://github.com/xbill9/gemma4-tips

Then run init.sh from the cloned directory.

The script will attempt to determine your shell environment and set the correct variables:

gpu-12B-6000-devops-agent
source init.sh

If your session times out or you need to re-authenticate- you can run the set_env.sh script to reset your environment variables:

gpu-12B-6000-devops-agent
source set_env.sh

Variables like PROJECT_ID need to be setup for use in the various build scripts- so the set_env script can be used to reset the environment if you time-out.

Model Management Tool with MCP Stdio Transport

One of the key features that the standard MCP libraries provide is abstracting various transport methods.

The high level MCP tool implementation is the same no matter what low level transport channel/method that the MCP Client uses to connect to a MCP Server.

The simplest transport that the SDK supports is the stdio (stdio/stdout) transport — which connects a locally running process. Both the MCP client and MCP Server must be running in the same environment.

The connection over stdio will look similar to this:

# Initialize FastMCP server
mcp = FastMCP("Self-Hosted vLLM DevOps Agent")

Running the Python Code

First- switch the directory with the Python version of the MCP sample code:

~/gemma4-tips/gpu-12B-6000-devops-agent

Run the release version on the local system:

make install
Processing ./.

The project can also be linted:

xbill@penguin:~/gemma4-tips/gpu-12B-6000-devops-agent$ make lint
ruff check .
All checks passed!
ruff format --check .
6 files already formatted
mypy .
Success: no issues found in 6 source files

MCP stdio Transport

One of the key features that the MCP protocol provides is abstracting various transport methods.

The high level tool MCP implementation is the same no matter what low level transport channel/method that the MCP Client uses to connect to a MCP Server.

In this project Antigravity CLI is used as the MCP client to interact with the Python MCP server code.

Antigravity CLI mcp_config.json

A sample MCP server file is provided in the .agents directory:

{
  "mcpServers": {
    "gpu-devops-agent": {
      "command": "python3",
      "args": [
        "/home/xbill/gemma4-tips/gpu-12B-6000-devops-agent/server.py"
      ],
      "env": {
        "GOOGLE_CLOUD_PROJECT": "aisprint-491218",
        "GOOGLE_CLOUD_LOCATION": "us-central1",
        "VLLM_BASE_URL": "https://gpu-12b-6000-devops-agent-289270257791.us-central1.run.app",
        "MODEL_NAME": "/mnt/models/gemma-4-12B-it"
      }
    }
  }
}

Validation with Antigravity CLI

The final connection test uses Antigravity CLI as a MCP client with the Python code providing the MCP server:

MCP Servers

Plugins (~/.gemini/antigravity-cli/plugins)
> ✓ google-dev-knowledge Tools: search_documents, answer_query, get_documents
   ✓ gpu-26b-6000-devops-agent Tools: save_hf_token, get_vllm_endpoint, list_vertex_models, list_bucket_models,
                                analyze_cloud_logging, +19 more

Getting Started with Gemma 4 on GPU

The Official vLLM repo also has Gemma4 specific information:

Releases · vllm-project/vllm

Model Lifecycle Management via MCP

The MCP tools provide a complete suite of agent-oriented operations for managing vLLM deployment on Cloud Run or a TPU.

The MCP server tools are used to deploy the model to Cloud Run with TPU:

deploy the model

Checking System status

The status can be checked with an MCP tool:

>● gpu-devops-agent/get_model_details(Model details check) (ctrl+o to expand)

  The vLLM service gpu-12b-6000-devops-agent is Healthy and fully operational.

  ### 🌀 Current vLLM Status

  • Service Status: 🟢 Ready
  • vLLM Engine Health: ✅ Healthy
  • Active Model: gpu-12b-6000-devops-agent 
  • Revision: gpu-12b-6000-devops-agent-00006-kbs 
  • Endpoint: https://gpu-12b-6000-devops-agent-wgcq55zbfq-uc.a.run.app

Cross Check The Deployed Model

Once the model starts locally- the MCP tool allows for model verification:

● gpu-devops-agent/verify_model_health(Model health verification) (ctrl+o to expand)

  The gpu-12b-6000-devops-agent service is Online and functioning correctly.
  ### Service Status Summary
  • Cloud Run Status: 🟢 Ready
  • vLLM Health: 🟢 Online
  • Model Health Check: ✅ Passed
      • Latency: 0.77 seconds
      • Endpoint: https://gpu-12b-6000-devops-agent-wgcq55zbfq-uc.a.run.app
  • Latest Revision: gpu-12b-6000-devops-agent-00006-kbs

and model details:

> get_model_details
● gpu-devops-agent/get_model_details(Model details check) (ctrl+o to expand)

  ### 🧩 Model Details Summary

  The vLLM engine is currently serving the following model:

  • Model ID: gpu-12b-6000-devops-agent 
  • Owned By: vllm 
  • Health Status: ✅ Healthy 
  • Active Endpoint: https://gpu-12b-6000-devops-agent-289270257791.us-central1.run.app

and status:

 > get_system_details

8.1k in | 829 out | 424 thinking
● Read(/home/xbill/.gemini/antigravity-cli/brain/...e05/.system_generated/steps/160/output.txt) (ctrl+o to expand)
  ### 🖥️ System Details: gpu-12b-6000-devops-agent 
  The service is currently running on Google Cloud Run (Gen2) with the following hardware and software
  configuration:

  #### 🏗️ Infrastructure & Resources
  • GPU Type: nvidia-rtx-pro-6000 (1 unit)
  • CPU: 20 vCPUs (with CPU Boost enabled)
  • Memory: 80 GiB RAM
  • Execution Environment: gen2 
  • Scaling: Min: 1 instance | Max: 3 instances
  • Region: us-central1 

  #### 🧠 vLLM Engine Configuration

  • Model Path: /mnt/models/gemma-4-12B-it-text-fp8 
  • Context Window: 131,072 tokens
  • Quantization: fp8 
  • KV Cache Dtype: fp8 
  • GPU Memory Utilization: 95% ( 0.95 )
  • Features Enabled:
      • enable-chunked-prefill 
      • enable-prefix-caching 
      • enable-auto-tool-choice (with gemma4 parser)

  #### 📈 Live Resource Metrics

  • vLLM Health: 🟢 Online 
  • Resident Memory: ~3.0 GB
  • Virtual Memory: ~24.8 GB
  • Open File Descriptors: 65 (Max: 25,000)
  • KV Cache Usage: 0% (Idle between benchmark steps)
  • Prefix Cache Hits: 308M tokens (High hit rate due to repetitive benchmark prompts)

Check the Cloud Run deployment status:

show the cloud run docker status

12.1k in | 841 out | 525 thinking
● Read(/home/xbill/.gemini/antigravity-cli/brain/...3fe05/.system_generated/tasks/task-166.log) (ctrl+o to expand)

  ### 🐳 Cloud Run Container ("Docker") Status

  The service gpu-12b-6000-devops-agent is running with the following container configuration:

  #### 🖼️ Image & Entrypoint
  • Image: vllm/vllm-openai:latest 
  • Command: bash 
  • Arguments:
    -c "vllm serve /mnt/models/gemma-4-12B-it-text-fp8 --served-model-name gpu-12b-6000-devops-agent --enable-
  log-requests --enable-chunked-prefill --enable-prefix-caching --generation-config auto --enable-auto-tool-
  choice --tool-call-parser gemma4 --reasoning-parser gemma4 --dtype bfloat16 --quantization fp8 --kv-cache-dtype
  fp8 --max-num-seqs 8 --gpu-memory-utilization 0.95 --tensor-parallel-size 1 --load-format runai_streamer --port
  8080 --host 0.0.0.0 --max-model-len 131072"

  #### ⚙️ Runtime Environment

  • Container Port: 8080 
  • Concurrency: 16 (max simultaneous requests per container)
  • Timeout: 3600s (1 hour)
  • CPU Throttling: false (Always allocated)
  • Startup CPU Boost: true 

  #### 🩺 Health & Readiness

  • Startup Probe: tcpSocket on port 8080 
      • Initial Delay: 240s 
      • Failure Threshold: 120 
      • Period: 15s 
  • Service Readiness: ✅ Ready: True (Last transition: 2026-06-21T23:10:27Z)

  #### 📦 Storage & Volumes

  • Volume: model-volume (Cloud Storage GCSFuse)
  • Bucket: aisprint-491218-bucket 
  • Mount Path: /mnt/models

Check the Cloud Run Service

Once Deployed — the model is visible as a standard Cloud Run Service:

Review the Model

Check the model details:

> what are the model details?

  The gpu-12b-6000-devops-agent is a specialized SRE/DevOps assistant running a self-hosted Gemma 4 12B model.
  ### 🧩 Model Configuration Details

   Detail | Configuration
  ---------------------------------------------|-----------------------------------------------------------------
   Model ID | google/gemma-4-12B-it (served as gpu-12b-6000-devops-agent )
   Quantization | fp8 (Weights) / fp8 (KV Cache)
   Context Window | 131,072 tokens
   GPU Hardware | NVIDIA RTX PRO 6000 (Cloud Run Gen2)
   Serving Engine | vLLM (with gemma4 reasoning and tool-call parsers)
   Key Features | Chunked prefill, Prefix caching, Auto-tool choice
   VRAM Usage | 95% ( 0.95 utilization)

  ### 🛠️ Active Capabilities

  The model is optimized for Zero-shot SRE diagnostics and infrastructure management. It has native support for:

  • Structured Reasoning: It uses a dedicated reasoning parser to output step-by-step SRE remediation plans.
  • Function Calling: Automatically selects tools for log analysis ( analyze_gpu_logs ), scaling (
  update_vllm_scaling ), and system monitoring.
  • Performance: Currently undergoing a high-concurrency benchmark sweep to validate latency across its full
  context window.

Once everything is online — you can send queries to the Model:

● gpu-devops-agent/query_gemma4_with_stats(Gemma 4 query with stats) (ctrl+o to expand)

  Gemma is a family of lightweight, state-of-the-art open models built by Google DeepMind, sharing the same
  technology and research used to create the Gemini models.

  ### 📊 Performance Statistics

  The query was processed by the self-hosted gpu-12b-6000-devops-agent with the following metrics:

  • Tokens per Second: 81.20 tokens/s 
  • Time to First Token (TTFT): 0.539s 
  • Total Generation Time: 8.186s 
  • Total Tokens: ~621 

  ### 💬 Model Response Summary

  • "Open" Architecture: Unlike closed models (like GPT-4), Gemma's weights are public, allowing developers to
  download, customize, and run it locally.
  • Efficient & Capable: Designed to run on consumer hardware (laptops, mobile) while maintaining high
  performance.
  • Safety-Tuned: Built with Google's rigorous safety standards to ensure helpful and safe interactions.
  • Flexible Sizes: Available in various parameter counts (e.g., 2B, 7B, 27B) to balance speed and power.

Benchmark the Local Model

The local model was benchmarked:

I have completed the benchmark sweep and generated a full performance report.

The analysis is reviewed:


> summarize the 12B on 6000 results

  The benchmark results for the Gemma 4 (12B) model on the NVIDIA RTX PRO 6000 (Blackwell) hardware demonstrate a
  massive performance leap, particularly in high-concurrency and high-context stability.
  ### 🚀 12B Blackwell Performance Snapshot

   Metric | Low Load (1-8 Users) | Medium Load (64 Users) | Peak Load (256 Users)
  ---------------------------|---------------------------|---------------------------|---------------------------
   Throughput (Req/s) | ~3 - 24 Req/s | ~72.6 Req/s | ~88.3 Req/s 🚀
   Avg Latency (s) | 0.2s - 0.4s | 0.6s - 1.0s | 1.6s - 4.5s
   Success Rate | 100% | 100% | 100%
  ──────
  ### 🔍 Key Result Highlights

  #### 1. Throughput: 3.8x Faster than L4
  The RTX 6000 Blackwell reached a peak throughput of 88.3 Requests/second (at 256 users, 128 context). For
  comparison, the NVIDIA L4 GPU saturates at 23.5 RPS for the same model. This 3.8x improvement allows a single
  Blackwell instance to handle nearly 4x the request density of an L4.
  #### 2. Context Mastery (Up to 16,384 Tokens)

  Thanks to the 96GB GDDR7 VRAM, the 12B model maintains high throughput even as the context window scales:
  • 4,096 tokens: Maintained 75.1 Req/s at peak concurrency.
  • 8,192 tokens: Maintained 57.4 Req/s at peak concurrency.
  • 16,384 tokens: Delivered 31.5 Req/s at peak concurrency with 100% success.
  • Note: On the 24GB L4 GPU, 16k context windows typically cause OOM (Out of Memory) errors or 0 RPS.
  #### 3. Sub-Second Latency Floor
  The Blackwell architecture provides an extremely low latency floor for real-time SRE applications:

  • Small Context ( < 512 tokens): Average latency stays under 1 second for up to 128 concurrent users.
  • Medium Context (4,096 tokens): Average latency stays under 2 seconds for up to 256 concurrent users.
  ──────
  ### 📊 Comparative Scaling Table (RTX 6000 Blackwell)

   Context Size | 1 User (Latency) | 32 Users (Req/s) | 256 Users (Req/s)
  ---------------------------|---------------------------|---------------------------|---------------------------
   128 tokens | 0.23s | 52.8 Req/s | 88.3 Req/s
   1,024 tokens | 0.22s | 58.9 Req/s | 78.4 Req/s
   4,096 tokens | 0.41s | 46.4 Req/s | 75.1 Req/s
   16,384 tokens | 0.78s | 21.3 Req/s | 31.5 Req/s

  │ [!IMPORTANT]
  │ Summary Verdict: Moving the 12B model to the Blackwell RTX 6000 transforms it from a "fast chat" model into a
  │ "high-density automation" engine. It can process massive streams of log data (16k context) at 30+ RPS,
  │ something previously only possible on enterprise TPU clusters.

This is visualized:

How does this compare to other Hardware Options?

> compare to other hardware options at 12B
6.9k in | 728 out | 472 thinking
● Read(/home/xbill/gemma4-tips/tpu-12B-v6e1-devops-agent/grid_benchmark_results.csv) (ctrl+o to expand)

  The Gemma 4 (12B) model shows a clear performance hierarchy across hardware, with the RTX 6000 Blackwell (96GB)
  serving as a highly balanced, high-density workhorse compared to the specialized TPU v6e and the standard
  NVIDIA L4.

  ### 📊 12B Hardware Comparison Matrix

   Metric | NVIDIA L4 (24GB) | RTX 6000 Blackwell (96GB) | TPU v6e-1 (Trillium)
  ---------------------------|---------------------------|---------------------------|---------------------------
   Peak Throughput | ~23.5 RPS | ~88.3 RPS 🚀 | ~72.8 RPS
   Success Rate (16k Ctx) | 0% (OOM) | 100% (up to 512 users) | 100% (up to 2048 users)
   Avg Latency (Short Ctx) | 0.2s - 0.5s | 0.2s - 0.4s | 0.2s - 0.4s
   Max Context Stability | ~8k tokens | 131k tokens | 4k - 8k tokens*

  *Note: In the TPU v6e-1 sweep, performance dropped significantly after 4k context, with 8k and 16k contexts
  failing (0% success) due to the specific memory allocation/sharding on a single chip.
  ──────
  ### 🔍 Comparative Insights
  #### 1. RTX 6000 Blackwell vs. TPU v6e-1 (Trillium)
  • Throughput Leadership: Surprisingly, the Blackwell RTX 6000 delivered higher peak throughput (~88 RPS) than
  the single-chip TPU v6e-1 (~72 RPS) for the 12B model. This highlights the massive bandwidth of GDDR7 (1.79
  TB/s) on Blackwell.
  • Context Ceiling: The RTX 6000 (96GB) is far more resilient for long-context tasks. While the TPU v6e-1 began
  failing at 8k tokens, the Blackwell GPU handled up to 16k context with 100% success and is capable of scaling
  up to 131k context with reduced concurrency.

  #### 2. Blackwell vs. NVIDIA L4

  • 4x Density Jump: The Blackwell GPU provides a 3.8x throughput increase over the L4. This means a single RTX
  6000 instance can replace four L4 instances while maintaining better latency.
  • Reliability: The L4 is strictly limited to low-context, low-concurrency tasks. The Blackwell card removes
  these constraints, allowing enterprise-grade automation to run on a single card.
  ──────
  ### ⚖️ Strategic Recommendation for 12B

  • 🏆 The Gold Standard: RTX 6000 Blackwell
      • Why: Best-in-class throughput (88 RPS) and the highest stability for long-context (16k-128k) workloads.
      It is the most versatile choice for any 12B DevOps deployment.
  • 🥈 The Budget Choice: NVIDIA L4
      • Why: Ideal for individual developer use or low-traffic SRE sidecars where context length stays below 8k.
  • 🥉 Specialized Scaling: TPU v6e-4/8
      • Why: While a single v6e-1 chip is comparable to a Blackwell card, moving to a v6e-4 or v6e-8 cluster
      would drastically exceed GPU performance for massive batch analysis (as seen in the 31B results).

  Summary Result: For the Gemma 4 (12B) model, the RTX 6000 Blackwell is the most performant and stable single-
  accelerator option currently available in the fleet.

What about other Gemma Model sizes?

> how does this compare to the 2B,4B,26B,31B across hardware deployments

31.4k in | 553 out | 306 thinking
● Read(/home/xbill/gemma4-tips/gpu-4B-L4-devops-agent/benchmark_report.md) (ctrl+o to expand)

  This final cross-matrix compares the Gemma 4 family across all model sizes and hardware platforms. The results
  highlight the NVIDIA RTX PRO 6000 (Blackwell) as the strongest single-card performer, while the TPU v6e
  provides unmatched scale for the largest models.
  ### 🏆 Master Cross-Model Hardware Comparison

   Model Size | NVIDIA L4 (24GB) | RTX 6000 Blackwell (96GB) | TPU v6e (Trillium) | Best Fit
  ------------------|------------------|---------------------------|-------------------------|-------------------
   2B | ~140 tokens/s | ~210 tokens/s | - | Edge / Mobile SRE
   4B | ~90 tokens/s | ~120 tokens/s | - | Real-time Chat
   12B | ~23 RPS | ~88 RPS 🚀 | ~73 RPS (v6e-1) | SRE Standard
   26B | ~131 tokens/s | ~1,232 tokens/s ⚡ | ~445 tokens/s (v6e-4) | High-Density SRE
   31B | ~140 tokens/s | ~345 tokens/s | ~8,500 tokens/s (v6e-8) | Enterprise Audit
  ──────
  ### 🔍 Architectural Comparisons by Model Tier
  #### 1. Small Models (2B & 4B)
  • Performance: These models are extremely efficient on all hardware.
  • Hardware Preference: RTX 6000 Blackwell is the winner due to its superior GDDR7 bandwidth, delivering roughly
  30% better throughput than the L4 for these lightweight tasks.
  #### 2. Medium Models (12B - The "SRE Standard")
  • The Blackwell Revolution: The jump from L4 to Blackwell for the 12B model is the most dramatic in the fleet
  (3.8x throughput increase).
  • Context Stability: Blackwell handles 16k context with 100% success at 512 concurrent users, while the L4
  saturates early and the TPU v6e-1 begins to degrade past 4k tokens.

  #### 3. Large Models (26B & 31B)

  • 26B Performance: The RTX 6000 Blackwell is surprisingly optimized for the 26B model, reaching 1,232 tokens/s.
  This outperforms the TPU v6e-4 for this specific model size.
  • 31B Throughput King: The TPU v6e-8 is the undisputed leader for 31B, delivering 8.5k tokens/s. This is 25x
  faster than the Blackwell GPU, proving that TPU sharding scales exponentially better for ultra-large parameter
  counts.
  ──────
  ### ⚖️ Strategic Deployment Recommendation

   Deployment Goal | Hardware | Model
  -------------------------------------|------------------------------------|------------------------------------
   Fastest Interactive SRE Chat | RTX 6000 Blackwell | Gemma 4 26B
   Most Stable 16k+ Log Analysis | RTX 6000 Blackwell | Gemma 4 12B
   Global Enterprise Batch Audit | TPU v6e-8 (Trillium) | Gemma 4 31B
   Cost-Effective Sidecar Monitor | NVIDIA L4 | Gemma 4 4B

  │ [!IMPORTANT]
  │ Summary: For any task requiring long context (16k-131k tokens) or high concurrency (100+ users) on a single
  │ instance, the RTX 6000 Blackwell (96GB) is the most versatile and performant accelerator in the current fleet.

Cost Breakdowns

Model serving costs were compared:

> do a cost analysis

6.5k in | 373 out | 302 thinking
● Read(/home/xbill/gemma4-tips/gpu-31B-6000-devops-agent/cost_per_token_analysis.md) (ctrl+o to expand)

  This cost analysis evaluates the financial efficiency of Gemma 4 across the fleet, factoring in both the
  machine-hour rates and the token throughput measured in our benchmarks.

  ### 💵 Hardware Cost Foundations (GCP Tier 1)
   Resource | Hourly Rate (Est.) | Billing Model | Best For
  --------------------------------|--------------------|----------------------------|----------------------------
   NVIDIA L4 (Cloud Run) | ~$0.88 / hr | Per-second (Scale-to-zero) | Spiky, low-traffic SRE task
   RTX 6000 Blackwell (Cloud Run) | ~$3.76 / hr | Per-second (Scale-to-zero) | Interactive, high-context S
   TPU v6e-1 (Single Chip) | ~$1.38 / hr | On-demand (Always-on) | Steady production 12B servi
   TPU v6e-8 (Full Pod) | ~$11.00 / hr | On-demand (Always-on) | Enterprise-scale 31B batchi
  ──────
  ### 📈 Cost Per Million (CPM) Tokens (at Peak Concurrency)
  Below is the cost efficiency for generating 1 million tokens across different tiers.

  #### 1. The 12B Tier (The SRE Sweet Spot)

   Hardware | Peak Tokens/s | Cost Per Million (CPM) | Verdict
  ---------------------|---------------------|------------------------|------------------------------------------
   TPU v6e-1 | ~9,300 t/s | $0.04 | Winner for 24/7 throughput
   NVIDIA L4 | ~3,000 t/s | $0.08 | Winner for scale-to-zero
   RTX 6000 Blackwell | ~11,300 t/s | $0.09 | Best performance/dollar for high-context
  #### 2. The 26B / 31B Tier (Enterprise Intelligence)

   Hardware | Peak Tokens/s | Cost Per Million (CPM) | Verdict
  --------------------------|------------------------|------------------------|----------------------------------
   TPU v6e-8 (31B) | ~8,500 t/s | $0.36 | Global leader in efficiency
   RTX 6000 Blackwell (26B) | ~1,232 t/s | $0.85 | High-performance interactive pro
   RTX 6000 Blackwell (31B) | ~345 t/s | $3.03 | Premium context, niche use
  ──────
  ### 🔍 Financial Insights
  #### 1. The "Scale-to-Zero" Multiplier

  While the TPU v6e-1 has the lowest theoretical CPM ($0.04), it costs $33.12/day even if idle. In contrast, an
  NVIDIA L4 on Cloud Run costs $0.00 if idle.
  • Threshold: If your SRE agent is active for less than ~2 hours per day, the L4 or Blackwell GPU on Cloud Run
  is significantly cheaper than a TPU VM.

  #### 2. The High-Context Premium

  The RTX 6000 Blackwell is the only card that maintains its throughput efficiency as context grows.

  • 16k Context Analysis: While other cards OOM or drop throughput by 90%, Blackwell remains stable. Its CPM for
  a 16k context audit is effectively 8x lower than an L4 (which would require multiple retries and smaller
  shards).

  #### 3. Enterprise Batching Winner

  For massive 31B workloads, the TPU v6e-8 is essentially unbeatable. Its CPM of $0.36 is nearly 10x cheaper than
  running the same model on a single Blackwell GPU for batch tasks.

  ### ⚖️ Final Summary

  • 🏆 Best Budget SRE (Spiky Load): NVIDIA L4 ($0.08 CPM).
  • 🚀 Best High-Performance SRE: RTX 6000 Blackwell ($0.09 CPM + Ultra-low Latency).
  • 👑 Best Enterprise Throughput: TPU v6e-8 ($0.36 CPM for 31B).

Summary

The strategy for using MCP for Gemma 4 GPU deployment with Antigravity CLI was validated with a incremental step by step approach.

A minimal stdio transport MCP Server was started from Python source code and validated with Antigravity CLI running as a MCP client in the same local environment. This Python server provided all of the management tools to deploy and troubleshoot Cloud Run Model deployments.

[Gemini API in Action] Building MemeFinder: A Native Mac Menu Bar Widget for Finding Memes via Text Using Gemini Vision & Semantic Embeddings

Evan Lin — Mon, 22 Jun 2026 00:41:17 +0000

The Origin: Mid-Conversation, Where on Earth Is That Meme?

Anyone who chats a lot has a folder full of memes on their phone and computer, but the moment you actually need one — the conversation is rolling, you want to drop a "thanks but no thanks" or an "I'm trash" reaction — you can't find it. The filename is IMG_4821.jpg, the photo library has no categories, and search is a non-starter.

I first came across a wonderful open-source project, ShiQu1218/MemeTalk. It builds a local meme semantic-search system with Python + Streamlit + SQLite: it scans your local meme folder, indexes images with OCR and vector embeddings, then does multi-route retrieval. Feature-complete, but research-oriented and requires opening a browser to run Streamlit.

What I wanted was something closer to an "everyday handy tool":

A native Mac app, one search box. I type what I'm looking for and the relevant meme pops up. Click it and it's copied straight to the clipboard.

So MemeFinder was born. This post records its journey from zero to "menu-bar resident + global hotkey," and several representative pitfalls along the way.

System Design and Architecture

The core concept is simple: point at a local meme folder → have Gemini build an index for each image → type to do a semantic search → click to copy.

I made three key technical decisions:

Native SwiftUI app, not Electron. Copying images to the clipboard, global hotkeys, menu-bar residency — with AppKit these are all first-class citizens.
Gemini does two things: the vision model gemini-3-flash-preview reads the text in each image and generates a Traditional Chinese description plus emotion tags; gemini-embedding-2 turns that semantics into a 768-dimensional vector.
Hybrid semantic-vector + keyword search. Pure keyword recall for Chinese is too poor; only semantic vectors achieve "type a related description and find the image."

System Architecture Flow

The project is deliberately split into two Swift Package targets:

Target	Type	Contents
`MemeFinder`	library	Logic, models, services, ViewModels (all unit-tested)
`MemeFinderApp`	executable	SwiftUI views + menu-bar shell (thin layer, depends on the library)

This split isn't decorative — it directly determines whether the tests can run smoothly, as "Pitfall #2" will explain.

Core Implementation

1. Auto-tagging memes with the Gemini vision model

During indexing, each image is sent to the vision model with a request to output only JSON: the text in the image, a Traditional Chinese description, tags, and emotion. responseMimeType is set to application/json to keep the output format stable:

public static func annotateRequest(apiKey: String, imageData: Data, mimeType: String) -> URLRequest {
    let prompt = """
    你是迷因圖標註助手。請閱讀這張圖，輸出 JSON，欄位：
    ocr_text(圖中所有文字), description(用繁體中文描述畫面與梗),
    tags(3-8 個繁體中文關鍵字陣列), emotion(單一情緒詞)。只輸出 JSON。
    """
    let body: [String: Any] = [
        "contents": [[
            "parts": [
                ["text": prompt],
                ["inline_data": ["mime_type": mimeType, "data": imageData.base64EncodedString()]]
            ]
        ]],
        "generationConfig": ["responseMimeType": "application/json"]
    ]
    // ... set URL, x-goog-api-key header, POST body
}

2. Hybrid semantic + keyword ranking

After the query string is embedded into a vector, we compute cosine similarity for every image, then add weight for keywords that hit the OCR text and tags, and merge-sort:

public func search(queryEmbedding: [Float], queryText: String,
                   in images: [IndexedImage], limit: Int) -> [SearchResult] {
    let tokens = queryText.lowercased().split(whereSeparator: { $0.isWhitespace }).map(String.init)
    let results: [SearchResult] = images.compactMap { image in
        let cos = cosineSimilarity(queryEmbedding, image.embedding)
        let haystack = (image.ocrText + " " + image.tags.joined(separator: " ")).lowercased()
        let matches = tokens.filter { !$0.isEmpty && haystack.contains($0) }.count
        let boost = 0.1 * Float(min(matches, 3))   // keyword boost capped at 0.3
        let score = cos + boost
        return score > 0 ? SearchResult(image: image, score: score) : nil
    }
    return Array(results.sorted { $0.score > $1.score }.prefix(limit))
}

The whole search engine is a pure function, with Gemini hidden behind a protocol, so this logic can be fully unit-tested offline without hitting the real API.

Major Pitfalls and Solutions

The real time sink in this project was never the happy path — it was the pitfalls below.

Pitfall #1: The mysterious `GeminiError error 0` — indexing and search both fail

App packaged, key set, folder chosen, hit search — and nothing shows below, just GeminiError error 0.

Rather than guessing, I hit the embedding endpoint once with a real key and printed the response:

curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-embedding-2:embedContent" \
  -H "x-goog-api-key: $KEY" \
  -d '{"content":{"parts":[{"text":"貓"}]},"output_dimensionality":768}'

The evidence was unmistakable:

{ "embedding": { "values": [ -0.0063, -0.0200, ... ] } }

The problem: my parser was reading the plural embeddings[0].values (that's the batchEmbedContents batch-endpoint format), but the single embedContent call returns the singular embedding.values. So every embed call failed — indexing each image failed, embedding the query string failed — all throwing badResponse (shown in the UI as GeminiError error 0).

[Solution]
Fix the parser to read the singular embedding.values, keeping the plural format as a fallback; I also hardened the annotation parser (a thinking model sometimes returns a textless "thought" part first, so skip to the first part that actually has text):

public static func embedding(fromEmbedContent data: Data) throws -> [Float] {
    guard let root = try? JSONSerialization.jsonObject(with: data) as? [String: Any] else {
        throw GeminiError.badResponse("cannot parse embedContent payload")
    }
    // A single embedContent returns {"embedding":{"values":[...]}}
    if let embedding = root["embedding"] as? [String: Any],
       let values = embedding["values"] as? [Double] {
        return values.map(Float.init)
    }
    // batchEmbedContents is {"embeddings":[{"values":[...]}]} — tolerate it too
    if let embeddings = root["embeddings"] as? [[String: Any]],
       let values = embeddings.first?["values"] as? [Double] {
        return values.map(Float.init)
    }
    throw GeminiError.badResponse("cannot parse embedContent payload")
}

Lesson: trust the actual API response over your memory or secondhand docs. A single line of curl saved countless guesses.

Pitfall #2: SwiftPM's `main` entry-point conflict and the SwiftUICore linking error

I initially made the whole project a single executableTarget with the tests depending on it directly. The result: tests failed to link no matter what. An executable target needs a main entry point, but that entry point only exists at the UI step's @main App; and casually adding a placeholder main.swift then conflicts with @main (Swift doesn't allow two entry points in one target). Worse, SwiftUI in an executable target spews SwiftUICore.tbd ... not an allowed client linker warnings.

[Root cause analysis and solution]
This is actually an architecture problem, not a compilation problem. The right approach is to split the project into two layers:

MemeFinder (library target): all logic, models, services, ViewModels — the tests depend only on this layer, it has no entry point, and it links cleanly as a library. ViewModels import Combine (not SwiftUI) to get ObservableObject.
MemeFinderApp (executable target): only SwiftUI views and @main, with import MemeFinder to use the public types above.

After the split, the library and tests don't touch SwiftUI at all, the linker warnings disappear, and the @main conflict no longer exists. "What the tests need to depend on" often forces out clean module boundaries.

Pitfall #3: Parallel indexing's rate limit and "I want to stop indexing halfway"

The first version indexed one image at a time, serially calling Gemini (annotate then embed). For hundreds of images this was painfully slow. So I switched to bounded parallelism with withTaskGroup (at most 4 at once), which brought three new problems:

The Gemini free tier has a rate limit — too much concurrency triggers 429.
The user wants to cancel halfway through a large folder.
Parallel completion order is chaotic, but the results need stable sorting.

[Solution]
Handle the three problems separately, all converging in the same buildIndex:

429 backoff retry: retry only GeminiError.rateLimited with exponential backoff (max 3 attempts); other errors are recorded without retry.
Cooperative cancellation: honor Task.isCancelled; on cancel, stop scheduling new work and keep the completed portion. Even the backoff Task.sleep lets CancellationError propagate normally instead of swallowing it and firing one more API call.
Stable sorting: collect results into a [path: image] dictionary, then reassemble the output in the order of the pre-sorted file list, decoupled from completion order.

// Seed maxConcurrent tasks first, then refill one per completion — strictly cap concurrency
for _ in 0..<maxConcurrent { if !scheduleNext() { break } }
while let res = await group.next() {
    if let img = res.image { resultsByPath[res.path] = img }
    if let err = res.error { errors.append(err) }
    done += 1
    progress(done, total)
    _ = scheduleNext()
}

Incidentally, the HTTP status code was also extracted into a pure function mapResponse(data:statusCode:): 429 → rateLimited, other non-2xx → httpError(code), 2xx → return the data. The retry logic then has a basis, and this part is easy to test too.

Pitfall #4: Evolving from a "windowed app" into "menu-bar resident + global hotkey"

Whether a tool is pleasant to use comes down to "how many steps to summon it." I wanted to hit ⌃⌘M mid-conversation to bring up the search popover, with the app tucked into the menu bar, not occupying the Dock. This step hit two classic macOS pitfalls:

(a) Does a global hotkey need accessibility permission? No. Use Carbon's RegisterEventHotKey to register a fixed hotkey, which doesn't need Accessibility permission (unlike monitoring the whole keyboard). But under Swift 6 strict concurrency, the C event callback has to dispatch through a static id → instance registry, requiring nonisolated(unsafe) and relying on the invariant that "Carbon events are delivered on the main thread" for safety. If ⌃⌘M is already taken, RegisterEventHotKey returns failure — in which case we silently degrade, log a line, and the menu-bar icon still works.

(b) The timing race in the menu-bar right-click menu. The initial approach was "set statusItem.menu → performClick → immediately clear menu," but clearing synchronously fights AppKit's menu-tracking loop, and the menu flashes and disappears.

[Solution]
Pop the menu up directly, fully bypassing the assign-and-clear of statusItem.menu:

@objc private func statusButtonClicked() {
    guard let event = NSApp.currentEvent else { togglePopover(); return }
    if event.type == .rightMouseUp {
        // Pop up directly; don't assign then synchronously clear statusItem.menu
        // (it races AppKit's menu-tracking loop)
        if let button = statusItem?.button {
            NSMenu.popUpContextMenu(makeMenu(), with: event, for: button)
        }
    } else {
        togglePopover()
    }
}

Finally, adding LSUIElement = true to the Info.plist produced by build-app.sh makes the Dock icon disappear, and MemeFinder officially becomes a pure menu-bar tool.

Pitfall #5: The settings form is blank — one symptom, three layers of cause

After moving to the menu-bar version, a user reported "the settings window is completely blank." This seemingly simple bug, peeled apart, actually had three layers, each highly representative.

Layer 1: a Form collapses to zero height inside a hand-rolled NSWindow.
Originally the settings screen lived in SwiftUI's native Settings { } scene, which sizes it sensibly. After the refactor it was hosted in a hand-rolled NSWindow(contentViewController: NSHostingController(rootView: SettingsView())), and SettingsView ended with only .frame(width: 460) — width only, no height. NSWindow(contentViewController:) sizes the window from the content's natural size, but a SwiftUI Form is vertically greedy; with no constraint, its natural height resolves to nearly 0, so the window opens as a 460-wide, near-zero-height blank strip. The fix is just to add a height:

.padding(20)
// When hosted in a hand-rolled NSWindow (not a SwiftUI Settings scene), a Form
// with no height constraint collapses to ~0, turning the window into a blank strip.
.frame(width: 460, height: 320)

Layer 2: ⌘, and the menu-bar "Settings…" go down two different paths.
After adding the height, the user said "still blank." On follow-up I found out he was summoning settings with ⌘,, while the menu-bar right-click "Settings…" went down a different path. The reason: ⌘, in a SwiftUI app triggers the Settings { } scene, and to dodge a state-sharing problem during the refactor, I had set that to Settings { EmptyView() }:

// During the refactor, the Settings scene was left empty to avoid state-sharing
// — so ⌘, opens a blank window
var body: some Scene {
    Settings { EmptyView() }
}

In other words, settings had two entry points pointing at different things: ⌘, pointed at the empty scene, the menu-bar "Settings…" pointed at the real window. The fix unifies the two paths — let the Settings scene host the real SettingsView (so ⌘, works directly), and make the menu-bar "Settings…" open the same native settings window too:

Settings {
    SettingsView(vm: appDelegate.settings, indexing: appDelegate.indexing,
                 onReindex: { appDelegate.reindexNow() },
                 onCancel: { appDelegate.cancelReindex() })
}

// The menu-bar "Settings…" now opens the same Settings scene
@objc private func openSettings() {
    NSApp.activate(ignoringOtherApps: true)
    NSApp.sendAction(Selector(("showSettingsWindow:")), to: nil, from: nil)
}

This also leverages the fact that a SwiftUI App body is @MainActor-isolated — so reading the @MainActor appDelegate.settings directly from the body is legal, with no extra bridging needed.

Layer 3 (the most insidious): open doesn't reload a menu-bar app at all.
The biggest time-waster in the process was that after recompiling, I'd ask the user to open MemeFinder.app, yet he kept seeing the old behavior. Because MemeFinder is an LSUIElement menu-bar-resident app — when an instance is already running, open only wakes the existing old process instead of relaunching with the new binary. So we were actually testing the same old build the whole time. The correct dev loop is to truly kill it first, then run from source:

killall MemeFinderApp 2>/dev/null; swift run MemeFinderApp

This layer reminds me: when debugging, first confirm "what you're testing really is the version you changed" — otherwise all your reasoning is built on faulty observations.

On the "Development Process" Itself

This project was driven almost entirely by an AI agent workflow of spec → plan → subagent task-by-task implementation → two-stage review: each feature started with a design spec, was broken into independently testable small tasks, every task wrote a failing test first (TDD) before implementing, and after completion an independent review agent checked spec compliance and code quality, followed by one final whole-branch review.

Several of the pitfalls — GeminiError error 0, the library/executable split, swallowing CancellationError during backoff, the menu timing race — were in fact caught half the time during the review stage, not written correctly on the first pass. This echoes that old principle: having tests as armor, and someone (or an agent) seriously reading the diff, matters far more than writing fast. The final project maintains 47 unit tests and a zero-warning release build.

Results and Benefits

Type to find, click to paste: type a Chinese description in the menu-bar popover, semantic search instantly lists relevant memes, click one to copy it to the clipboard and paste straight into LINE / Slack / Messages.
Privacy-friendly, searchable offline: images and the index live locally (~/Library/Application Support/MemeFinder/index.json); only the "build the index" step calls Gemini.
A truly handy tool: ⌃⌘M is available anytime, menu-bar resident, no Dock footprint; incremental indexing only processes new/changed images, and indexing can show progress and be canceled.
A clean, maintainable architecture: a two-layer library/executable design, Gemini hidden behind a protocol, pure logic fully covered by tests.

All the development code for this project is open-sourced on GitHub: kkdai/meme-finder-app. Feel free to clone it, point it at your own meme-collection folder, and experience the joy of "type to find your meme"!

MTP Speculative Decoding with the 12B Gemma 4 QAT Model on NVIDIA L4, Cloud Run, MCP, and…

xbill — Sun, 21 Jun 2026 14:34:08 +0000

MTP Speculative Decoding with the 12B Gemma 4 QAT Model on NVIDIA L4, Cloud Run, MCP, and Antigravity CLI

This article provides a step by step deployment guide for Gemma 4 to a Google Cloud Run hosted GPU enabled system. The latest 12B model is used with QAT formats and deployed with MTP Speculative Decoding. A suite of Python MCP tools is built to simplify management of the vLLM hosted Gemma 4 deployment with Antigravity CLI.

What is this project trying to Do?

This project is similar to a previous project that targeted GPU hosted Gemma4 instances on GCP:

Gemma-SRE: Self-Hosted vLLM Infrastructure Agent

Antigravity CLI

Antigravity CLI is the follow-on successor to Gemini CLI- the terminal driven, agent assisted coding tool.

Full details on installing Antigravity CLI are here:

Getting Started with Antigravity CLI

Testing the Antigravity CLI Environment

Once you have all the tools in place- you can test the startup of Antigravity CLI.

You will need to authenticate with a Google Cloud Project or your Google Account:

agy

This will start the interface:

Full Installation Instructions

The detailed installation instructions for Antigravity CLI are here:

Getting Started with Antigravity CLI

Python MCP Documentation

The official GitHub Repo provides samples and documentation for getting started:

GitHub - modelcontextprotocol/python-sdk: The official Python SDK for Model Context Protocol servers and clients

Where do I start?

The strategy for starting MCP development for model management is a incremental step by step approach.

First, the basic development environment is setup with the required system variables, and a working Antigravity CLI configuration.

Then, a minimal Python MCP Server is built with stdio transport. This server is validated with Antigravity CLI in the local environment.

This setup validates the connection from Antigravity CLI to the local server via MCP. The MCP client (Antigravity CLI) and the Python MCP server both run in the same local environment.

Setup the Basic Environment

At this point you should have a working Python environment and a working Antigravity CLI installation. The next step is to clone the GitHub samples repository with support scripts:

cd ~
git clone https://github.com/xbill9/gemma4-tips

Then run init.sh from the cloned directory.

The script will attempt to determine your shell environment and set the correct variables:

gpu-12B-qat-mtp-L4-devops-agent
source init.sh

If your session times out or you need to re-authenticate- you can run the set_env.sh script to reset your environment variables:

gpu-12B-qat-mtp-L4-devops-agent
source set_env.sh

Variables like PROJECT_ID need to be setup for use in the various build scripts- so the set_env script can be used to reset the environment if you time-out.

Model Management Tool with MCP Stdio Transport

One of the key features that the standard MCP libraries provide is abstracting various transport methods.

The high level MCP tool implementation is the same no matter what low level transport channel/method that the MCP Client uses to connect to a MCP Server.

The connection over stdio will look similar to this:

# Initialize FastMCP server
mcp = FastMCP("Self-Hosted vLLM DevOps Agent")

Running the Python Code

First- switch the directory with the Python version of the MCP sample code:

~/gemma4-tips/gpu-12B-qat-mtp-L4-devops-agent

Run the release version on the local system:

make install
Processing ./.

The project can also be linted:

xbill@penguin:~/gemma4-tips/gpu-12B-qat-mtp-L4-devops-agent$ make lint
ruff check .
All checks passed!
ruff format --check .
6 files already formatted
mypy .
Success: no issues found in 6 source files

Getting Started with Gemma 4 on GPU

The Official vLLM repo also has Gemma4 specific information:

Releases · vllm-project/vllm

Gemma 12B

Gemma 12B is the latest model in the Gemma4 series. More details are available here:

Introducing Gemma 4 12B: a unified, encoder-free multimodal model

and here:

Gemma 4 12B: The Developer Guide

NVIDIA L4 on Cloud Run

The L4 on Google Cloud Run provides a cost-effective “scale to zero” option for providing inference:

Google Cloud Run Adds Support for NVIDIA L4 GPUs, NVIDIA NIM, and Serverless AI Inference Deployments at Scale | NVIDIA Technical Blog

Quantization-Aware Training (QAT)

One of the key issues with deploying models is available GPU memory based in the hardware footprint. Along with the new 12B model release- new versions of the model that use QAT to reduce the memory requirements have been released:

Gemma 4 QAT models: Optimizing model compression for mobile and laptop efficiency

QAT Model Versions

What are the differences between the versions on Hugging Face?

> what is the difference between google/gemma-4-12B-it-qat-q4_0-unquantized and https://huggingface.co/google/gemma-
  4-12B-it-qat-q4_0-gguf and google/gemma-4-12B-it-qat-w4a16-ct

▸ Thought for 1s, 602 tokens
  Analyzing Gemma QAT Variations

● Read(/home/xbill/gemma4-tips/gpu-12B-qat-L4-devops-agent/GEMINI.md) (ctrl+o to expand)

  Here is a comparison of the three Gemma 4 QAT (Quantization-Aware Training) model configurations:

  ### 1. google/gemma-4-12B-it-qat-q4_0-unquantized 

  • Format: Unquantized, high-precision weights (usually float16 or bfloat16) extracted directly from the QAT
  training pipeline.
  • Target Engine: Standard deep learning/fine-tuning framework runtimes (e.g., PyTorch, Transformers, Jax).
  • Primary Use Case:
      • Serving as a high-precision reference baseline.
      • Custom downstream quantization, compilation, or MLX conversion.
      • Research, fine-tuning, or running Speculative Decoding (where this serves as the primary model alongside
      its matching MTP assistant/draft model).


  ### 2. google/gemma-4-12B-it-qat-q4_0-gguf 

  • Format: GGUF (specifically quantized using the standard q4_0 4-bit method).
  • Target Engine: Local CPU, Apple Silicon, or consumer GPU engines (e.g., llama.cpp , Ollama, LM Studio).
  • Primary Use Case:
      • Zero-setup local execution with minimal memory requirements.
      • Running LLMs efficiently on consumer-grade hardware.


  ### 3. google/gemma-4-12B-it-qat-w4a16-ct 

  • Format: Compressed Tensors format ( -w4a16-ct ), which stores 4-bit weights while performing computation using
  16-bit activations.
  • Target Engine: High-performance, high-throughput cloud inference engines (specifically vLLM or SGLang ).
  • Primary Use Case:
      • High-concurrency cloud deployments utilizing GPU accelerators (e.g., NVIDIA L4).
      • Optimizing GPU memory usage and improving inference speed while maintaining near-lossless model accuracy
      via QAT.

  ──────
  ### 📊 Summary Reference Table

   Checkpoint Configuration | Serialization Format | Target Platform | Primary Engine
  ---------------------------|---------------------------|----------------------------|----------------------------
    ...-qat-q4_0-unquantized | Unquantized ( BF16 / FP16 | Custom compilation / | Transformers / Speculative
                             | ) | research | Drafts
    ...-qat-q4_0-gguf | GGUF (4-bit weight) | Local Desktop/Server | llama.cpp / LM Studio /
                             | | (CPU/GPU) | Ollama
    ...-qat-w4a16-ct | compressed-tensors (4- | Cloud GPU Deployments | vLLM / SGLang 
                             | bit/16-bit) | |

  For more details on QAT architecture and configuration, see GEMINI.md.
  ──────
  ### Summary of Work

  • Reviewed Context: Analyzed the workspace documentation GEMINI.md to compare the unquantized, GGUF, and
  compressed-tensors configurations.

MCP stdio Transport

One of the key features that the MCP protocol provides is abstracting various transport methods.

The high level tool MCP implementation is the same no matter what low level transport channel/method that the MCP Client uses to connect to a MCP Server.

In this project Antigravity CLI is used as the MCP client to interact with the Python MCP server code.

Antigravity CLI mcp_config.json

A sample MCP server file is provided in the .agents directory:

{
  "mcpServers": {
    "gpu-devops-agent": {
      "command": "python3",
      "args": [
        "/home/xbill/gemma4-tips/gpu-12B-qat-mtp-L4-devops-agent/server.py"
      ],
      "env": {
        "GOOGLE_CLOUD_PROJECT": "aisprint-491218",
        "GOOGLE_CLOUD_LOCATION": "us-east4",
        "VLLM_BASE_URL": "https://gpu-12b-qat-mtp-289270257791.us-east4.run.app",
        "MODEL_NAME": "/mnt/models/gemma-4-12B-it-qat-w4a16-ct"
      }
    }
  }
}

Validation with Antigravity CLI

The final connection test uses Antigravity CLI as a MCP client with the Python code providing the MCP server:

MCP Servers

Plugins (~/.gemini/antigravity-cli/plugins)
> ✓ google-dev-knowledge Tools: search_documents, answer_query, get_documents
   ✓ gpu-devops-agent Tools: save_hf_token, get_vllm_endpoint, list_vertex_models, list_bucket_models,
                       analyze_cloud_logging, +21 more

Checking System status

The status can be checked with an MCP tool:

> vllm_status

● gpu-devops-agent/status_vllm(Checking status of the vLLM deployment via MCP) (ctrl+o to expand)

  Here is the current status of the gpu-12b-qat-mtp Cloud Run deployment:

  • Service Status: 🟢 Ready (all configurations and routing conditions are met)
  • Latest Revision: gpu-12b-qat-mtp-00032-nwb (deployed on 2026-06-21)
  • Service URL: https://gpu-12b-qat-mtp-wgcq55zbfq-uk.a.run.app

Cross Check The Deployed Model

Once the model starts locally- the MCP tool allows for model verification:

> verify_model_health

● gpu-devops-agent/verify_model_health(Verifying model health for gpu-12b-qat-mtp via MCP) (ctrl+o to expand)

  The model health check has passed successfully:

  • Model: /mnt/models/gemma-4-12B-it-qat-w4a16-ct 
  • Response: "Yes, I am working! I am ready to help you with any..." 
  • Latency: 0.88 seconds

and model details:

> get_active_models

131.0k in | 677 out | 415 thinking
● Bash(ENDPOINT="https://gpu-12b-qat-mtp-wgcq55zbfq-uk.a.run.app"; TOKEN=$(gcloud auth pri...) (ctrl+o to
expand)

  The active model on the deployment is:

  • Model ID: /mnt/models/gemma-4-12B-it-qat-w4a16-ct 
  • Owned by: vllm 

  (Note: The /health check returned a 503 in get_model_details because the request was
  unauthenticated, and Cloud Run requires an IAM identity token for routing. When queried with the proper
  authorization headers, the endpoint returns a clean 200 OK health status).

and status:

Here is the current system status:

  ### 🌀 GPU Cloud Run System Status ( get_system_status )

  • vLLM Health: 🟢 Online ( [https://gpu-12b-qat-l4-devops-agent-289270257791.us-east4.run.app](https://gpu-12b-qat-l4-devops-agent-289270257791.us-east4.run.app) )
  • Cloud Run Service Status: 🟢 Ready
  ──────
  ### 🧩 Active Model Details ( get_model_details )

  • Active Model: /mnt/models/gemma-4-12B-it-qat-w4a16-ct 
  • Health Endpoint: Healthy ✅
  ──────

Review the Model

Check the model details:

> what are the model details including vllm optimization

  Here are the model and vLLM optimization details for the active gpu-12b-qat-mtp deployment:

  ### 🧩 Core Model Details

  • Primary Model: google/gemma-4-12B-it-qat-w4a16-ct (Quantization-Aware Training, INT4 weights with 16-
  bit activations)
  • Speculative/Assistant Model (MTP): google/gemma-4-12B-it-assistant (Multi-Token Prediction)
  • Speculative Tokens: 3 ( --spec-tokens=3 , --spec-method=mtp )
  ──────
  ### 🚀 vLLM Performance Optimizations

   Category | Optimization Parameter | Value/Setting | Purpose
  ---------------------|---------------------------------|---------------|----------------------------------
   Inference Kernel | VLLM_DISABLE_FLASHINFER | 0 (Enabled) | Uses high-performance FlashInfer
                       | | | kernels for attention
                       | | | operations.
   Sampler | VLLM_USE_FLASHINFER_SAMPLER | 1 (Enabled) | Enables FlashInfer's optimized
                       | | | sampling kernels, decreasing
                       | | | time-to-first-token.
   Memory Allocation | PYTORCH_CUDA_ALLOC_CONF | expandable_se| Minimizes VRAM memory
                       | | gments:True | fragmentation and helps prevent
                       | | | Out-Of-Memory (OOM) errors.
   KV Cache | --kv-cache-dtype | fp8 | Quantizes key-value cache to 8-
                       | | | bit precision, doubling the
                       | | | available context capacity.
   KV Cache Allocation | --gpu-memory-utilization | 0.95 | Allocates 95% of GPU VRAM to
                       | | | vLLM's storage caches.
   Batching & Context | --max-num-batched-tokens | 8192 | Sets maximum batched prefill
                       | | | tokens to allow higher
                       | | | throughput under concurrency.
   Chunking | --enable-chunked-prefill | Enabled | Chunks massive requests to avoid
                       | | (Implicit) | stalling shorter requests in
                       | | | queue.
   Load Strategy | --safetensors-load-strategy | prefetch | Prefetches model weights during
                       | | | container boot-up to reduce
                       | | | startup time.
   Scheduling | --async-scheduling | Enabled | Enables asynchronous scheduling
                       | | | loop in vLLM to pipeline
                       | | | execution.
   Context Limit | --max-model-len | 32768 | Limits maximum context length to
                       | | | keep memory reservation stable.
   Concurrency Limit | --max-num-seqs | 8 | Caps maximum concurrent request
                       | | | sequences active in a forward
                       | | | pass.
   Threading | MKL_NUM_THREADS , | 1 | Forces single-threading on CPU
                       | OMP_NUM_THREADS | | to avoid CPU core contention on
                       | | | the host.

───────────────────────────────────────────────────────────────────────────────────────────────────────────────
>

and check the metrics:

  Here is a summary of the key metrics retrieved from the active gpu-12b-qat-mtp vLLM engine:

  ### 🚀 Performance & Throughput

  • Prefill Tokens Processed: 29,389,524 tokens (prefilled)
  • Generation Tokens Processed: 9,494 tokens (generated)
  • Active Running Requests: 0.0 (Idle)
  • Active Waiting Requests: 0.0 
  • KV-Cache Usage: 0.0% (Idle)

  ### 🔮 Speculative Decoding / MTP Statistics

  • Total Draft Iterations: 139 
  • Total Speculative/Draft Tokens Proposed: 417 tokens
  • Total Speculative Tokens Accepted: 23 tokens
  • Speculative Acceptance by Position:
      • Position 0: 11 tokens accepted
      • Position 1: 6 tokens accepted
      • Position 2: 6 tokens accepted
      • Overall Acceptance Rate: ~5.5% (demonstrating why MTP is heavily bound by scheduling and sequential
      evaluation overhead on a single GPU for typical workloads)

  ### 🕒 Latency & Queuing (Cumulative)

  • Average Time to First Token (TTFT): ~0.28 seconds
  • Average Request Queue Time: ~0.016 milliseconds (minimal queue delay)
  • Average End-to-End Latency: ~0.28 seconds (highly responsive)

Once everything is online — you can send queries to the Model:

● gpu-devops-agent/query_gemma4_with_stats(Querying Gemma 4 via MCP with stats returned) (ctrl+o to expand)

  Here are the performance stats and the model response to your query:

  ### 📊 Performance Statistics

  • Active Model: /mnt/models/gemma-4-12B-it-qat-w4a16-ct 
  • Time to First Token (TTFT): 0.215 seconds
  • Total Generation Time: 12.60 seconds
  • Generation Speed: 18.16 tokens/s
  • Total Generated Tokens: ~225 tokens
  ──────
  ### 💬 Model Response

────────────────────────────────────────────────────────────
  Depending on the context, Gemma usually refers to one of two things: Google's family of open AI models, or
  a gemstone.

  #### 1. In Artificial Intelligence (Google’s AI Models)

  Gemma is a family of lightweight, state-of-the-art open models developed by Google DeepMind.

  It is built using the same research and technology as Google's larger Gemini models, but is designed to be
  small and efficient enough for developers to run on their own hardware (such as laptops or single-GPU
  servers).

  • Open Weights: The model weights are publicly available, allowing developers to fine-tune, download, and
  host them privately.
  • Sizes: Available in various sizes (e.g., 2B, 7B, 9B, 27B parameters, and the latest 12B/31B iterations
  in Gemma 4) to balance capability and hardware constraints.
  • Versatility: Capable of text generation, summarization, coding, and logical reasoning.

  #### 2. In Gemology & Names

  • Gemstone: "Gemma" is the Latin word for a precious gemstone or jewel.
  • Name: A popular feminine given name in several European languages (especially Italian), translating to
  "precious stone" or "gem."

Benchmark the Local Model

The local model was benchmarked:

The benchmark sweep has completed successfully. Here are the compiled results across all combinations of context
  window sizes and concurrency levels:

The analysis is reviewed:

> summarize benchmark results

> summarize the benchmark results
  Here is a summary of the benchmark results comparing the Standard QAT (No MTP) configuration against
  various Multi-Token Prediction (MTP) settings on a single NVIDIA L4 GPU:
  ──────
  ### 1. Standard Workloads (Short Contexts, 128 Tokens)

  For typical developer and agent workloads, MTP/Speculative Decoding degrades performance.

  • Why? The sequential overhead of running the draft model ( google/gemma-4-12B-it-assistant ) on a single
  GPU offsets the memory bandwidth savings when the primary model is already highly optimized.
  • Optimized MTP-3 Impact: By enabling FlashInfer and increasing max-num-batched-tokens to 8192 , we
  recovered significant throughput under high concurrency. At 512 concurrent users, throughput for MTP-3
  improved from 8.4 req/s to 12.84 req/s (a +52.8% increase).

  Throughput Comparison (Requests/sec at 128 context):

   Concurrency (Users) | Standard QAT (No MTP) | MTP-4 | MTP-3 | MTP-2 | Optimized MTP-3 (Tuned)
  ---------------------|-----------------------|-----------|-----------|----------|-------------------------
            8 | 18.9 | 11.3 | 8.8 | 8.0 | 8.53
           64 | 23.8 | 12.9 | 11.9 | 11.9 | 12.12
           512 | 23.7 | 13.0 | 8.4 | 8.0 | 12.84
  ──────
  ### 2. Large Context Workloads (16,384 Tokens)

  At very large context lengths, inference becomes heavily memory-bandwidth bound due to the massive size of
  the KV cache. Here, MTP improves performance.

  • Why? Speculative decoding allows the GPU to evaluate multiple tokens per target model forward pass,
  reducing the frequency of loading the giant KV cache from High Bandwidth Memory (HBM) to GPU SRAM.
  • Best Configuration: MTP-3 (3 tokens) and MTP-4 yield the highest overall throughputs. Under high
  concurrency (512 users), MTP-3 outperforms Standard QAT by +11.4%. Under extreme load (2048 users),
  Optimized MTP-3 maintains a stable 6.66 req/s throughput.

  Throughput Comparison (Requests/sec at 16K context):

   Concurrency (Users) | Standard QAT (No MTP) | MTP-4 | MTP-3 | MTP-2 | Optimized MTP-3 (Tuned)
  ---------------------|-----------------------|-----------|-----------|----------|-------------------------
            8 | 5.16 | 6.55 | 0.15 | 0.17 | 0.14
           64 | 7.39 | 8.11 | 7.65 | 8.00 | 7.61
           512 | 7.54 | 8.24 | 8.40 | 7.70 | 8.13
          2048 | 6.30 | 8.00 | 7.55 | 8.01 | 6.66
  ──────
  ### 💡 Core Recommendations

  1. Production DevOps Agent (Default): Keep MTP Disabled. The standard QAT model ( google/gemma-4-12B-it-
  qat-w4a16-ct ) alone provides the highest throughput (up to 23.8 req/s) and lowest latencies for standard
  agent interactions.
  2. Long-Context Tasks (16K+): Enable MTP-3 or MTP-4 with FlashInfer and batched token tuning enabled to
  accelerate processing times and improve throughput by up to 27%.

Wait, What was that? I thought speculative decoding improved performance!

 why didn't MTP improve performance

  Multi-Token Prediction (MTP) / Speculative Decoding failed to improve performance for standard workloads
  (short context sizes) due to three key technical bottlenecks on the single NVIDIA L4 GPU configuration:

  ### 1. Compute-Bound vs. Memory-Bound Regimes

  • When MTP Helps (Memory-Bound): LLM decoding is typically memory-bandwidth bound because the GPU has to
  fetch the entire model's weights from High Bandwidth Memory (HBM) to SRAM for every single token generated.
  Speculative decoding speeds this up by using a tiny draft model to propose tokens, allowing the main model
  to verify them in one joint forward pass.
  • When MTP Fails (Compute-Bound): When you have multiple concurrent users, vLLM groups these requests into
  batches. As the batch size increases, the workload shifts from memory-bound to compute-bound (fully
  utilizing the Tensor Cores). In this regime, weight loading is no longer the bottleneck; instead, raw
  compute capability is. Running the draft model sequentially to generate speculative tokens adds extra
  compute steps to the queue, increasing latency and reducing throughput.

  ### 2. Draft Model Execution & Sequential Verification Overhead

  • Speculative decoding requires the GPU to run the assistant model sequentially (e.g., 3 draft steps)
  before running the target model once to verify them.
  • The Gemma 4 12B QAT model is already extremely fast natively on the L4 GPU. Because of this, the
  overhead of running the draft model sequentially is disproportionately high compared to the target model.
  If even a few proposed tokens are rejected, the wasted draft passes result in a net-negative speedup.

  ### 3. VRAM Contention and Scheduler Limits

  • Assistant Model Overhead: Loading the speculative draft model ( google/gemma-4-12B-it-assistant )
  consumes about 0.8 GB of VRAM. While small, this reduces the space available for vLLM's KV cache.
  • Token Budget Contention: Speculative decoding requires reservation of extra token slots in the scheduler.
  As vLLM warns, this limits the maximum concurrent sequences the engine can schedule per step, causing
  requests to queue up and increasing time-in-queue under high concurrency.

So what are the bottlenecks?

> what are the key constraints for the deployment on this hardware profile

  Deploying serving stacks on this specific hardware profile—a single NVIDIA L4 GPU (24GB VRAM) combined
  with Cloud Run Gen2 (4 vCPUs, 16GB System RAM)—introduces several critical constraints:

  ### 1. GPU VRAM Limitations (24 GB)

  • Model Footprint vs. KV Cache: The L4's 24GB VRAM is the primary ceiling. The baseline Gemma 4 12B model
  in half-precision (bfloat16) consumes ~24GB of memory, leaving 0 GB for the KV cache and causing immediate
  Out-of-Memory (OOM) crashes under concurrent load.
  • The QAT Requirement: To serve the model, we must use the Quantization-Aware Training (QAT) 4-bit
  compressed format ( w4a16-ct ). This compresses the model weights to ~6 GB, freeing up ~18 GB for the KV
  cache.
  • Speculative Model Tax: Enabling MTP adds the assistant model weights to VRAM (~0.8 GB), further
  squeezing the KV cache space.

  ### 2. GPU Memory Bandwidth (300 GB/s)

  • Compared to H100 (3.35 TB/s) or A100 (2.0 TB/s), the L4 has relatively low memory bandwidth (300 GB/s).
  This makes single-user, memory-bound autoregressive decoding natively slower, which is why optimizing the
  kernels (FlashInfer) and memory alignment is critical.

  ### 3. Cloud Run CPU Throttling ( --no-cpu-throttling )

  • Cloud Run Gen2 defaults to throttling CPU down to near-zero when no requests are actively being
  processed. For GPU serving, this must be disabled ( --no-cpu-throttling ). If CPU throttling is active,
  the container's CPU cannot coordinate with the GPU quickly enough, resulting in severe timeouts, failed
  startup health probes, and model-loading hangs.

  ### 4. GCS FUSE UID/GID Permissions

  • Model weights are mounted from Google Cloud Storage (GCS) using Cloud Run's FUSE volume mount.
  • The official vllm/vllm-openai container runs as the non-root user vllm (UID/GID 1001 ). If you do
  not explicitly mount the GCS volume with the options mount-options=uid=1001;gid=1001 , the container will
  fail to boot with a Permission Denied error because the non-root user cannot read the mounted model
  files.

  ### 5. System RAM (16 GB) and vCPUs (4)

  • The host system only has 16 GB of System RAM and 4 vCPUs. Loading model weights from the GCS mount into
  the GPU requires buffering through system memory. During boot, vLLM's initialization and model loading
  must be carefully managed (e.g. using safetensors-load-strategy=prefetch ) to prevent exceeding the
  host's 16GB memory ceiling, which triggers immediate container evictions (OOMKilled).

Cost Analysis

OK so there are some limitations with this hardware profile. What about the cost benefits?

> how cost effective is this deployment?

  This deployment profile (Cloud Run Gen2 GPU + NVIDIA L4) is exceptionally cost-effective for SRE/DevOps
  automation workloads, primarily due to its serverless architecture, hardware efficiency, and the optimized
  model format.
  Here is a breakdown of the cost dynamics:
  ### 1. Serverless Scale-to-Zero Advantage

  Unlike traditional setups on VMs (Compute Engine) or Kubernetes (GKE) where you pay for the GPU 24/7 even
  when idle, Cloud Run allows you to configure min-instances=0 .

  • Idle Cost: $0 when no SRE alerts or queries are active.
  • Cold Start Consideration: While scaling to zero saves maximum budget, it incurs a cold start penalty
  (~1–2 minutes to mount GCS weights and boot vLLM). For critical SRE workloads, keeping min-instances=1 
  is recommended, which costs a fixed baseline but is still cheaper than full VM hosting due to Cloud Run's
  granular allocation.

  ### 2. High Performance-per-Dollar of the L4 GPU

  The NVIDIA L4 (based on the Ada Lovelace architecture) is designed specifically for low-cost, energy-
  efficient inference:

  • Cloud Run GPU Pricing: Estimated at roughly $0.40 to $0.50 per hour for the GPU resource alone, plus
  minimal CPU/Memory allocation.
  • Total Hourly Cost (Active): Approximately $0.60 to $0.70/hour when running.
  • Comparison to A100/H100: An A100 instance generally costs $2.00 to $3.50/hour. Since the optimized Gemma
  4 12B QAT model achieves a highly respectable ~24 req/s on a single L4, you get a massive percentage of
  A100-level throughput at a fraction of the hardware cost.

  ### 3. Efficiency Gains from Quantization (QAT)

  Using the QAT INT4 model ( w4a16-ct ) directly improves cost-effectiveness:

  • Memory Reduction: Compressing the weights from 24GB to 6GB allows the model to fit comfortably on a
  single 24GB L4 GPU.
  • Avoiding Multi-GPU Costs: Without quantization, serving a 12B model in bfloat16 would require either a
  much more expensive GPU (like an A100 40GB/80GB) or multiple L4 GPUs in a tensor-parallel setup, which
  would double or triple the hourly cost.

  ### 4. Summary Financial Comparison

   Metric | Dedicated VM (e.g., g2-… | Cloud Run (Scale-to-Zero… | Cloud Run (Always-On, mi…
  ----------------------|--------------------------|---------------------------|----------------------------
   Hourly Rate (Active) | ~$0.70 | ~$0.70 | ~$0.70
   Monthly Cost | ~$504.00 (fixed) | ~$42.00 (usage-based) | ~$504.00 (max baseline)
   Best Fit | Continuous 24/7 high- | Intermittent SRE alert | Real-time SRE response,
                        | throughput workloads | analysis, dev/testing | zero cold-start tolerance

───────────────────────────────────────────────────────────────────────────────────────────────────────────────

One last cross-check:

> one final review- MTP or not with this deployment? and what about QAT?

  • The Only Exception: If your agent is routinely processing massive datasets or long-context documents
  (16K+ tokens in a single request), MTP will provide a 10% to 27% speedup. Otherwise, the overhead is not
  worth the trade-off.
  ──────
  ### 2. Quantization-Aware Training (QAT): YES (Mandatory)

  Using the QAT compressed weights ( w4a16-ct ) is absolutely critical and mandatory for this hardware
  profile.

  • Why? A single NVIDIA L4 GPU has only 24GB of VRAM.
      • If you deploy the standard bfloat16 model, the weights consume almost all 24GB, leaving 0 GB for the
      KV cache. The service will crash with Out-of-Memory (OOM) errors as soon as multiple requests arrive.
      • The QAT INT4 model compresses the weights to ~6 GB, leaving ~18 GB of VRAM entirely for the KV cache.
      This allows the system to easily handle up to 512 concurrent users with a 100% success rate.
  • Zero Quality Loss: Because this is QAT (quantization simulation during training) rather than standard
  post-training quantization, you get the memory savings of a 4-bit model with the logical accuracy and
  reasoning capabilities of the full 16-bit baseline model.

  ### Final Production Stack Recommendation:

  │ Serve google/gemma-4-12B-it-qat-w4a16-ct using vLLM on a single L4 GPU with MTP disabled, and
  │ FlashInfer enabled for optimal throughput (~24 req/s) and maximum concurrent capacity.

> so MTP is bad?

  MTP is not bad in general—it is just wrong for this specific deployment use case.

  Think of MTP as a highly specialized optimization tool. It works exceptionally well in certain situations,
  but acts as a bottleneck in others.

  ### When MTP is Good:

  • Single-User / Local Environments: If you are running a model locally on your laptop (e.g., Apple Silicon
  │ Serve google/gemma-4-12B-it-qat-w4a16-ct using vLLM on a single L4 GPU with MTP disabled, and
  or a desktop GPU) for a single user, the system is strictly memory-bound. Here, MTP significantly
  accelerates token generation (often by 1.5x to 2x) because it reduces the time spent loading model weights
  from memory.
  • Extremely Long Contexts (16K+ tokens): When processing huge logs or long documents, loading the massive
  KV cache dominates the GPU's time. MTP reduces the frequency of these expensive memory transfers,
  resulting in a 10% to 27% throughput improvement even under high concurrency.

  ### When MTP is Bad (This DevOps Agent Profile):

  • High Concurrency (Batched Requests): When multiple users query the DevOps agent at the same time, vLLM
  batches the requests. This shifts the GPU from a memory-bound state to a compute-bound state (where Tensor
  Cores are running at 100%). In this state, the sequential execution of the draft model adds extra compute
  overhead, causing requests to queue up and slowing everything down.
  • Hardware Resource Constraints: On a single 24GB GPU, loading the draft model takes up precious VRAM and
  restricts the scheduling capacity of the main engine, causing throughput to degrade under standard log
  analysis sizes.

  ### Summary

  MTP is a latency optimizer for single-user, memory-bound tasks. It is not a throughput optimizer for
  concurrent, compute-bound backend APIs. Since DevOps/SRE agents are backend APIs that handle concurrent
  workloads, turning MTP off yields better overall performance.

Summary

The strategy for using MCP for Gemma 4 GPU deployment with Antigravity CLI was validated with a incremental step by step approach.

Chatting with your Data: Conversational Analytics in BigQuery

Aryan Irani — Sat, 20 Jun 2026 07:45:01 +0000

For the last decade, the workflow for Business Intelligence hasn't changed much: A business stakeholder asks a question, a Data Engineer writes the ad-hoc SQL, and a dashboard is built. But as data scales to the petabyte level, this reactive cycle creates massive bottlenecks.

What if business users could just chat directly with the database?

Enter BigQuery Conversational Analytics. Google Cloud has effectively turned the traditional data warehouse into an active participant. By leveraging Gemini, Conversational Analytics allows users to query massive datasets using natural language. It understands the intent, generates the complex SQL, and returns the data (or geographic visualizations) instantly.

The Problem with "Text-to-SQL" Toys

We've all seen the basic "Text-to-SQL" AI wrappers on Twitter. They look great in a controlled demo, but they fall apart in production. Why? Because raw LLMs don't understand your company's unique business logic. If an AI doesn't know that your definition of "Net Profit" excludes returned items, the data it returns is not just wrong - it's dangerous.

To deploy AI over enterprise data, you need strict governance.

You need to be able to explicitly teach the AI your database schema. You need Dataplex Glossaries to lock down business terminology so the AI never guesses a formula. You need Parameterized Verified Queries to ensure highly sensitive financial reports use pre-approved SQL. And crucially, you need Financial Controls (like Maximum Bytes Billed) so a business user asking a vague question doesn't trigger a $5,000 table scan.

BigQuery Conversational Analytics isn't just an AI wrapper; it's a governed enterprise agent.

The Playlist: Build It Yourself

I spent the last few weeks using this product. To show you exactly how you can get started, I recorded a 4-part vide playlist on YouTube.

If you are a Data Engineer, Cloud Architect, or BI Analyst, this series will show you exactly how to build and govern your own AI data agents from scratch:

📺 Part 1: The AI Reasoning Pipeline We dive into the BigQuery Studio UI and test Gemini's ability to perform comparative analysis on the Google Trends public dataset without writing a single line of SQL.

📺 Part 2: Building Custom Data Agents An AI can't magically understand your database schema. I show you how to connect your tables and write System Instructions to explicitly control the SQL that Gemini generates.

📺 Part 3: Enterprise Data Governance We tackle the hardest part of Enterprise AI. I show you how to lock down your agent using Column Metadata, Dataplex Glossaries, and strict financial controls to prevent petabyte-scale billing surprises.

📺 Part 4: Automating Multi-Table Relational Joins In the grand finale, we use everything we've built to force the agent to write a flawless, massive 3-table relational JOIN from a single natural language prompt.

The role of the Data Engineer is shifting from writing ad-hoc SQL to governing autonomous data pipelines. If you're building in Google Cloud, I highly recommend getting hands-on with this.

You can watch the full series here: https://youtube.com/playlist?list=PL_MCVBMm-9sogdOzjqIcbX-dAizXux91c&si=890s67FrqxYfZTeu

Feel free to reach out if you have any issues/feedback at aryanirani123@gmail.com.

Skills over System Prompts: Building an Anki Tutor with the Antigravity SDK

Ertuğrul Demir — Fri, 19 Jun 2026 10:09:03 +0000

AI has made me a little lazier.

Not dramatically lazy. Not "the robots will do everything" lazy. More like: once you get used to asking an agent to do boring work, every small manual workflow starts looking suspicious.

Anki is a perfect example.

Anki is great. I use it to remember things I study, subjects I work on, and the weird little decisions hidden inside codebases. Spaced repetition works. The problem is not Anki.

The problem is me.

I can already see the rot setting in. On complex cards, my brain starts negotiating with itself. "Yeah, I basically knew that." "Close enough." "I would have remembered it in context." Then I press Good and move on.

That is not studying. That is self-certified vibes.

What I actually wanted was a study buddy sitting on top of my real Anki collection. Someone to ask the card, wait for my answer, reveal the real answer, compare it honestly, explain the gap, and only then help decide whether it was Again, Hard, Good, or Easy.

AI is annoyingly good for that.

It is also useful when taking over a new project. When I enter a repo, I do not only want a summary. I want to be quizzed later on the key decisions, the architecture, the gotchas, and the "why is it like this?" parts. Anki is great for that too.

But I am still lazy.

I am not going to manually write every card. I am not going to keep every deck updated by hand. And if I am studying from my phone, I am definitely not going to type long answers into a chat just so the agent can grade me. Voice needs to work too.

So the project quickly stopped being "connect Gemini to Anki."

It became a small agent system:

a terminal tutor for focused review sessions
a Telegram tutor for studying from my phone, including voice answers
a deck builder that creates cards from web research or a local codebase
a watch mode that can notice code changes and create cards while I work

That is a lot of behavior.

My first instinct was the usual one: write a bigger system prompt. Tell the agent how to run a study session. Tell it how to write good flashcards. Tell it how to inspect a codebase and turn architecture into cards. Tell it how to behave differently in Telegram. Tell it not to touch scheduling unless I approve.

That works for about ten minutes.

Then the system prompt becomes a junk drawer.

The hard part was not giving the agent tools.

The hard part was giving it habits.

That is where the Google Antigravity SDK fit really well. It gives you the agent runtime as a Python library: custom tools, reusable skills, lifecycle hooks, safety policies, streaming, triggers, and multiple ways to run the same agent logic from different surfaces.

What the Antigravity SDK Gives You

The Antigravity SDK is not just a wrapper around a chat model.

It gives you programmatic access to the same agent runtime behind Google Antigravity 2.0 and the Antigravity CLI, but from Python.

That matters because a real agent is not only a model call. A real agent needs:

tools
memory across turns
permissions
hooks
skills
streaming
triggers
safety around side effects

The SDK puts those behind one main abstraction: Agent.

The smallest useful version really is tiny:

import asyncio
from google.antigravity import Agent, LocalAgentConfig

async def main():
    config = LocalAgentConfig()
    async with Agent(config) as agent:
        response = await agent.chat("What files are in the current directory?")
        print(await response.text())

if __name__ == "__main__":
    asyncio.run(main())

Install it with:

pip install google-antigravity

Then set a Gemini API key from Google AI Studio:

export GEMINI_API_KEY="your-key-here"

That is the hello world.

The useful version starts when you compose the runtime features around a real workflow.

In this project, the Antigravity SDK pieces mapped like this:

Antigravity SDK capability	Where I used it
`Agent` / `LocalAgentConfig`	the terminal tutor, Telegram tutor, and deck builder all run on the same agent runtime
Custom Python tools	AnkiConnect actions like `get_due_cards`, `show_answer`, `rate_card`, and `add_notes`
`skills_paths`	shared `review-buddy`, `plain-cards`, and `codebase-cards` behavior packages
Lifecycle hooks	sync on session start/end, deck backup before writes, audit log after scheduling changes, tool-error recovery
Safety policies	practice mode blocks `rate_card` so cram sessions cannot change real scheduling
Streaming	the deck builder prints progress while the agent researches and creates cards
Triggers	watch mode reacts to `.py` file changes and asks the agent to card important changes
Built-in read-only tools	codebase mode lets the agent inspect a repo without editing it

That list is the reason this worked better as an SDK project than as one giant prompt around a model call.

Now, the first useful step: give the agent hands.

Giving the Agent Hands: Anki as Python Tools

Anki already has an HTTP API through the AnkiConnect add-on. The entire bridge is basically one POST to localhost:

def invoke(action: str, **params):
    response = requests.post(
        "http://localhost:8765",
        json={"action": action, "version": 6, "params": params},
        timeout=30,
    )
    response.raise_for_status()
    payload = response.json()
    if payload["error"]:
        raise RuntimeError(payload["error"])
    return payload["result"]

From there, the agent tools are just normal Python functions.

A simplified version:

def list_decks() -> str:
    """List all Anki decks with their due counts."""
    decks = invoke("deckNames")
    stats = invoke("getDeckStats", decks=decks)
    return json.dumps(stats)


def get_due_cards(deck: str = "", limit: int = 5) -> str:
    """Return due cards without revealing the answer side."""
    query = f'deck:"{deck}" is:due' if deck else "is:due"
    card_ids = invoke("findCards", query=query)[:limit]
    cards = invoke("cardsInfo", cards=card_ids)
    return json.dumps(cards)


def rate_card(card_id: int, rating: int) -> str:
    """Submit a user-confirmed Anki rating: 1 Again, 2 Hard, 3 Good, 4 Easy."""
    invoke("answerCards", answers=[{"cardId": card_id, "ease": rating}])
    return json.dumps({"rated": card_id, "rating": rating})

Then register them with the SDK:

from google.antigravity import LocalAgentConfig

config = LocalAgentConfig(
    tools=[list_decks, get_due_cards, rate_card],
)

That is one of the nicest parts of the SDK: custom tools do not require a separate server. For this version, I did not need MCP, a framework, a schema generator, or a second process.

The agent can call plain Python.

In the real project I ended up with more tools:

list_decks
get_due_cards
show_answer
rate_card
find_notes
add_note
add_notes
update_note
suspend_card
unsuspend_card
undo
get_stats
sync

That was enough to make the tutor useful.

This is the first pattern:

Put capabilities in tools.

Tools are the agent's hands. But hands are not behavior.

For behavior, I used skills.

The Problem with Giant System Prompts

At first, I tried to describe everything in the agent's system instructions.

The tutor needs to know how to run a review session:

show the question
wait for my answer
reveal the answer
compare my answer
suggest a rating
wait for confirmation
only then update Anki scheduling

It also needs to know how to write good cards:

one fact per card
answer-first backs
no trivia padding
no vague questions
no giant essay cards

Then the deck builder needs another workflow:

research a topic
extract the important facts
create cards
verify they exist in Anki

Then the codebase deck builder needs a different workflow:

inspect the repo breadth-first
find key abstractions
explain responsibilities and data flow
avoid making cards for random syntax

Then Telegram needs shorter replies because nobody wants a wall of Markdown on a phone.

You can put all of that into one system prompt.

But you should not.

A giant system prompt has three problems:

It pollutes every task. The agent is thinking about codebase exploration while you are reviewing Spanish verbs.
It is hard to reuse. The same card-writing rules need to appear in the terminal tutor, Telegram tutor, and deck builder.
It rots. Every new behavior gets pasted into the same blob until nobody knows which rule controls what.

This is exactly the problem skills solve.

The shape changed from this:

system prompt = tutor rules
              + card-writing rules
              + codebase-exploration rules
              + Telegram style rules
              + safety reminders
              + whatever I forgot last week

Into this:

system prompt  = identity + hard safety floor
review-buddy   = study-session behavior
plain-cards    = card-writing behavior
codebase-cards = repo-exploration behavior
hooks/policies = enforcement and receipts

That is the real pattern behind the title.

Not "make the prompt better."

Make the prompt smaller.

Skills over System Prompts

A skill is a folder with a SKILL.md file inside it.

My project has three:

.agents/skills/
  plain-cards/
    SKILL.md
  review-buddy/
    SKILL.md
  codebase-cards/
    SKILL.md

Each skill starts with a tiny bit of frontmatter.

For example, the review skill begins like this:

---
name: review-buddy
description: Playbook for running an interactive Anki review session — quiz one card at a time, grade recall together, submit ratings, repair noisy or broken cards.
---

That description is not just documentation for humans. It is the lightweight discovery layer. The agent can see what skills exist, then load the full instructions only when the task calls for them.

A skill is not a service. It is not an MCP server. It is not a deployment. It is a behavior package sitting on disk, ready to be pulled into the agent when needed.

Then the SDK loads the skill directory:

config = LocalAgentConfig(
    system_instructions=SYSTEM_INSTRUCTIONS,
    tools=ALL_TOOLS,
    skills_paths=[".agents/skills"],
)

The key idea is simple:

The system prompt says who the agent is. Skills say what job it is currently doing.

For this project, the system prompt stays small. It says the agent is a friendly flashcard tutor working with a real Anki collection.

The details live in skills.

`review-buddy`: the study session playbook

This skill describes how to run a review session.

It covers the rhythm:

ask one card at a time
hide the answer until the user attempts it
reveal and teach briefly
suggest a rating
wait for confirmation
handle noisy or broken cards
close with a recap

This is not code. It is behavioral protocol.

That distinction matters. The review flow is not tied to terminal I/O, Telegram messages, or AnkiConnect. It is just the way a good tutor should behave.

`plain-cards`: the card-writing style guide

This skill handles card quality.

It tells the agent to write cards that are:

atomic
answer-first
lean
verified
free of filler
easy to review months later

A bad flashcard is worse than no flashcard. It creates fake progress. The model can generate ten cards in seconds, but without a style guide it will happily generate ten vague cards that future me will hate.

So card writing became a skill.

`codebase-cards`: the repo exploration protocol

This one is for turning source code into Anki cards.

The agent is told to inspect the repo breadth-first, identify architecture, data flow, responsibilities, and gotchas, then turn only the useful findings into cards.

That skill powers code mode in the deck builder:

python deck_builder.py "overall architecture" --path ~/my/project --count 6

The focus hint changes, but the exploration protocol stays the same.

This is the second pattern:

Put reusable behavior in skills.

Not in the system prompt. Not duplicated across entrypoints. Not buried in Python conditionals.

A skill is just a file, but it changes the shape of the whole project.

One Behavior Layer, Three Surfaces

Once the behavior lived in skills, adding new surfaces became much easier.

The architecture looked like this:

                         .agents/skills/
                  ┌──────────┼──────────┐
                  │          │          │
           review-buddy  plain-cards  codebase-cards
                  │          │          │
                  └──────────┼──────────┘
                             │
                    LocalAgentConfig
                             │
       ┌─────────────────────┼─────────────────────┐
       │                     │                     │
  terminal tutor        Telegram tutor        deck builder
    tutor.py          telegram_tutor.py      deck_builder.py

The terminal tutor is the simplest surface:

async with Agent(config) as agent:
    await run_interactive_loop(agent)

The Telegram tutor uses the same agent differently:

async def chat_response(agent: Agent, prompt: str) -> str:
    response = await agent.chat(prompt)
    return "".join([token async for token in response])

The deck builder streams output as it works:

response = await agent.chat(message)
async for token in response:
    print(token, end="", flush=True)

Different surfaces. Same runtime. Same skills.

That is the part I liked most. Telegram did not need a copied review prompt. The deck builder did not need its own card-writing manifesto. The codebase mode did not need a separate app-specific doctrine.

They all loaded the same skill directory.

The Terminal Tutor

The terminal version is the baseline.

Start Anki, run the tutor, and ask naturally:

python tutor.py

Then:

quiz me on XYZ

The tutor lists due cards, asks one question, waits for my answer, reveals the real Anki answer, compares, teaches, and suggests a rating.

The important part: it does not update scheduling just because the model thinks I got the answer right.

The review loop is human-in-the-loop by design:

Agent: I would rate this Good (3). You had the main idea but missed the date.
User: yes
Agent: rated 3. Next card...

Or I can override it:

Agent: I would rate this Hard (2).
User: actually 1
Agent: rated Again (1). Let's reinforce it.

Spaced repetition is stateful. A bad rating affects the future schedule. So the model can suggest, but I decide.

That is not just a prompt preference. It is the product boundary.

The Telegram Tutor

The second surface was Telegram.

Not because Telegram is fancy. Because the best study app is the one I actually open.

The Telegram bot long-polls the Bot API, sends messages into the same Antigravity agent, and returns the response. It also supports voice notes: speak the answer, transcribe it, and feed the transcript back into the tutor as text.

The agent gets a small extra instruction:

TELEGRAM_INSTRUCTIONS = """
You are chatting through Telegram on a phone. Keep replies short and plain
text only — no markdown headers, tables, or code fences. One card per message.
"""

Everything else stays shared.

Same Anki tools. Same hooks. Same skills.

I also added due-card nudges without spending model tokens. Every 30 minutes, plain Python checks Anki deck counts. If cards are waiting, the bot sends a short reminder:

25 cards waiting (X 5, Y 8). Say 'quiz me' to start.

No LLM needed. No reasoning needed. Just deterministic code.

This became a useful design rule:

Do not use the model for work a for loop can do.

The agent is for tutoring. The nudge is just a counter.

The Deck Builder

The third surface is a deck builder.

It has two modes.

Web mode:

python deck_builder.py "Ottoman Empire" --deck "History" --count 8

Codebase mode:

python deck_builder.py "error handling and edge cases" --path ~/my/project --count 6

Web mode gives the agent a small research toolset: Wikipedia search, Wikipedia read, and URL fetch. Then it asks the agent to create cards using the plain-cards skill.

Codebase mode is more interesting. The SDK can give the agent built-in file tools scoped to a workspace. I enabled read-only access:

from google.antigravity.types import BuiltinTools, CapabilitiesConfig

config = LocalAgentConfig(
    tools=[add_notes, list_decks],
    workspaces=[code_path],
    capabilities=CapabilitiesConfig(
        enabled_tools=BuiltinTools.read_only()
    ),
    skills_paths=[".agents/skills"],
)

That means the agent can inspect the target repo, but not edit it.

For a deck builder, that is the right permission boundary. It needs to read code and create Anki notes. It does not need to modify the project.

This is where codebase-cards activates. The agent explores the repo, identifies the concepts worth remembering, then writes cards through add_notes.

At the end, I do not trust the model's narration. The script queries Anki to verify the cards exist.

def cards_in_anki(deck: str) -> int:
    result = json.loads(find_notes(f'deck:"{deck}" tag:auto-researched', 100))
    return len(result) if isinstance(result, list) else 0

If the model says it created cards but Anki has zero, the script nudges it to try again.

That became another rule:

Trust the system receipt, not the model narration.

Turning It Ambient with Triggers

The SDK also supports triggers: background tasks that react to external events and push messages into the agent.

I used a file-change trigger for codebase card generation.

The idea: while I work on a project, if a Python file changes, the agent can inspect the change and decide whether it introduced something worth remembering.

Simplified:

from google.antigravity.triggers import on_file_change


def make_watch_trigger(path, deck, tag):
    async def on_change(ctx, changes):
        paths = sorted({c.path for c in changes if c.path.endswith(".py")})
        if not paths:
            return

        await ctx.send(
            f"These files changed: {', '.join(paths)}. "
            f"Create cards in deck {deck} if the change is worth remembering."
        )

    return on_file_change(path, on_change)

Run it like this:

python deck_builder.py "as I work" --path ~/my/project --watch

This is where the project started feeling less like a chatbot and more like a sidecar.

I edit code. The trigger wakes the agent. The codebase skill tells it how to inspect the change. The card-writing skill tells it how to write good cards. The Anki tool creates the notes.

No new server. No custom scheduler. No giant prompt.

Just SDK triggers plus skills.

The Part I Refused to Trust to the Model

Skills are guidance.

Policies and hooks are enforcement.

That line is the difference between a fun demo and a tool I can leave connected to my real Anki collection.

The Antigravity SDK has declarative safety policies and lifecycle hooks. I used both.

Practice mode blocks scheduling writes

Sometimes I want to cram without touching Anki scheduling.

A prompt instruction is not enough for that. If the agent forgets and calls rate_card, the schedule changes.

So practice mode denies the tool at the harness level:

from google.antigravity.hooks import policy

policies = policy.confirm_run_command()

if practice_mode:
    policies = policies + [
        policy.deny("rate_card", name="practice_mode")
    ]

Now rate_card is blocked even if the model tries to call it.

That is the kind of safety I want: not vibes, not trust, not "please don't". A runtime boundary.

Hooks sync, back up, audit, and recover

The SDK hook system lets you observe or intervene at lifecycle points.

I used session hooks to sync Anki:

@hooks.on_session_start
async def sync_on_start():
    sync_anki()

@hooks.on_session_end
async def sync_on_end():
    sync_anki()

I used a pre-tool-call Decide hook to back up a deck before note writes:

@hooks.pre_tool_call_decide
async def backup_before_note_writes(tool_call):
    if tool_call.name in ("add_note", "add_notes"):
        backup_deck(tool_call.args["deck"])
    return hooks.HookResult(allow=True)

I used a post-tool-call Inspect hook to audit scheduling changes:

@hooks.post_tool_call
async def audit_scheduling_changes(result):
    if result.name in {"rate_card", "undo", "suspend_card", "unsuspend_card"}:
        append_jsonl("backups/scheduling_audit.jsonl", result)

And I used a Transform hook to turn ugly tool errors into recovery hints the model can act on:

@hooks.on_tool_error
async def recover_from_tool_error(error):
    if isinstance(error, requests.Timeout):
        return "AnkiConnect timed out. Ask the user to check Anki, then retry."
    return None

This is one of the strongest parts of the SDK.

The model does not need to remember to audit itself. The harness does it.

The model does not need to remember to back up a deck before writing. The hook does it.

The model does not get to bypass practice mode. The policy blocks it.

The pattern became clear:

tools give the agent capabilities
skills give the agent reusable behavior
policies define what must never happen
hooks add system-level guarantees around the agent

That separation is the architecture.

What Worked

A few things worked better than expected.

Plain Python tools were enough

I originally thought I might need to build an MCP server immediately.

I did not.

For one application, custom Python functions were simpler. The SDK already knows how to expose them as tools. That kept the first version small.

MCP is still useful when you want the same tools available across multiple clients. But for an SDK-native app, Python functions are the shortest path.

Skills kept the project from becoming prompt soup

This was the biggest win.

The base system instructions stayed focused. The detailed workflows moved into skills.

When I improved card-writing rules, terminal, Telegram, and deck builder all benefited. I did not need to update three prompts.

Hooks made side effects less scary

Anki is not a toy database. It is my real spaced-repetition schedule.

The hooks gave me a deterministic layer around model behavior:

sync at session boundaries
backup before writes
audit after scheduling changes
recover from tool failures

That made the agent feel much less like a random chatbot with database access.

Triggers changed the feel of the app

The file watcher was small, but it changed the mental model.

The agent was no longer only something I talked to. It could react to work happening around it.

That is where SDK agents get interesting: not just chat, but event-driven labor.

What Did Not Work Perfectly

A few caveats.

Skills are not hard guarantees

Skills are instructions. They improve behavior, but they are still model-read guidance.

If something must be impossible, use a policy or remove the tool.

That is why practice mode denies rate_card instead of merely asking the model not to call it.

AnkiConnect has sharp edges

AnkiConnect is simple, but it has quirks.

For example, answerCards can return success even for bad card IDs unless you pre-check the card. Some note updates silently fail if the note is open in Anki's browser window. AnkiConnect also runs inside Anki's Qt process, so you should not treat it like a high-concurrency API.

The fix is boring and important: validate inside tools.

Voice was simpler outside the agent loop

The Telegram bot supports voice answers, but I kept transcription outside the agent loop. A direct Gemini transcription call turns the voice note into text, then the transcript goes into the tutor.

That was simpler and more reliable for this build.

The lesson: use the SDK where it makes the architecture cleaner. Do not force every feature through the agent if a direct call is simpler.

How to Build Something Similar

If you want to build your own version of this pattern, I would do it in this order.

1. Start with one real workflow

Do not start with a platform.

Pick one annoying workflow with real state behind it:

flashcards
GitHub issues
CRM updates
personal knowledge base
support tickets
finance records

The state matters. Agents get interesting when they can act on something real.

2. Wrap the system as small Python tools

Keep the tools boring.

def search_items(query: str) -> str:
    """Search the user's records."""
    ...


def create_item(title: str, body: str) -> str:
    """Create a new record after user approval."""
    ...

config = LocalAgentConfig(
    tools=[search_items, create_item],
)

Make tools validate inputs. Do not rely on the model to pass perfect IDs.

3. Move task behavior into skills

Create a skill folder:

.agents/skills/my-workflow/SKILL.md

A minimal skill:

---
name: my-workflow
description: Use when helping the user process and update records in this system.
---

# My Workflow

1. Inspect the current record before changing it.
2. Propose the change in plain language.
3. Wait for user confirmation before writing.
4. After writing, verify the record exists.

Then load it:

config = LocalAgentConfig(
    tools=TOOLS,
    skills_paths=[".agents/skills"],
)

This is the move: do not keep growing the system prompt forever.

4. Add policies for non-negotiables

If a tool should never run in a mode, deny it.

policies = [
    policy.deny("delete_record", name="no_deletes"),
]

If shell execution should require confirmation, keep the default guard:

policies = policy.confirm_run_command()

The model can misunderstand a skill. It cannot ignore a denied tool.

5. Add hooks for receipts

Use hooks for things that should happen regardless of whether the model remembers them:

audit logs
backups
sync
metrics
sanitization
error recovery

@hooks.post_tool_call
async def audit(result):
    write_log({
        "tool": result.name,
        "result": result.result,
        "error": result.error,
    })

6. Add another surface only after the behavior is reusable

Once the behavior lives in tools and skills, a second surface becomes much cheaper.

Terminal first. Then Telegram, Slack, web, cron, or file triggers.

The surface should be thin. The agent behavior should not live there.

The Bigger Point

The old way to build an AI feature was to write a large prompt and hope the model followed it.

That is not enough for real agents.

A real agent needs separation of concerns:

Capabilities       → tools
Reusable behavior  → skills
Hard boundaries    → policies
System guarantees  → hooks
External events    → triggers
User interface     → thin surface

This is what the Antigravity SDK made pleasant. I could build one agent runtime and reuse it across terminal, Telegram, and deck generation. I could keep the tutoring behavior in SKILL.md files instead of duplicating it. I could wrap real side effects with policies and hooks instead of trusting the model to behave.

The Anki tutor is just the concrete example.

The pattern generalizes.

A support agent could keep triage behavior in a skill, expose ticket updates as tools, deny destructive writes by policy, and audit every status change by hook.

A code review agent could keep review rubrics in skills, expose GitHub as tools, require approval before comments, and verify every posted review.

A research agent could keep extraction protocols in skills, use file triggers to process new papers, and write structured outputs only after validation.

The skill is the portable behavior module.

The SDK is the harness that lets it act.

Resources

Closing

I started this because I was too lazy to open Anki.

That sounds like a joke, but most useful automation starts there. Not with a grand platform vision. With a small workflow that keeps not happening because the friction is just high enough.

The surprising part was not that an LLM could quiz me.

The surprising part was how clean the architecture became.

Tools gave the agent hands. Skills gave it habits. Policies gave it boundaries. Hooks gave it receipts. Triggers made it wake up when something changed.

That is the version of agents I trust more: not one giant prompt pretending to be an application, but a small runtime with clear layers.

The future of agent apps is not monolithic complex systems.

It is smaller prompts, sharper tools, reusable skills, and a harness that refuses to let the model pretend a side effect happened when it did not.

Serverless Gemma 12B on Azure Container Apps

xbill — Fri, 19 Jun 2026 02:00:12 +0000

This article provides a step by step debugging guide for deploying Gemma 4 to Azure Container Apps. A suite of Python MCP tools is built to simplify management of the vLLM hosted Gemma 4 deployment with Antigravity CLI.

What is this project trying to Do?

This project is a DevOps/SRE assistant that uses a Gemma 4 model hosted on Azure Container APps. It provides tools to provision the Docker container and deploy the model, as well as for observability and performance testing.

This project is similar to a previous project that targeted GPU hosted Gemma4 instances on GCP:

Gemma-SRE: Self-Hosted vLLM Infrastructure Agent

Antigravity CLI

Antigravity CLI is the follow-on successor to Gemini CLI- the terminal driven, agent assisted coding tool.

Full details on installing Antigravity CLI are here:

Getting Started with Antigravity CLI

Testing the Antigravity CLI Environment

Once you have all the tools in place- you can test the startup of Antigravity CLI.

You will need to authenticate with a Google Cloud Project or your Google Account:

agy

This will start the interface:

Full Installation Instructions

The detailed installation instructions for Antigravity CLI are here:

Getting Started with Antigravity CLI

Azure CLI

The Azure Command-Line Interface (CLI) is a cross-platform tool used to connect to Azure and execute administrative commands on Azure resources. It allows you to manage services like virtual machines, databases, and networking through a terminal using interactive prompts or scripts. [1, 2, 3]

More details are available here:

What is the Azure Developer CLI?

Azure Container Apps

Azure Container Apps is a fully managed, serverless Kubernetes-based application platform designed for building and deploying modern, containerized apps without managing complex infrastructure. It enables scaling from zero to high demand, supports microservices, and handles event-driven processing with built-in HTTPS and observability.

Full details are available here:

https://azure.microsoft.com/en-us/products/container-apps

Where do I start?

The strategy for starting MCP development for model management is a incremental step by step approach.

First, the basic development environment is setup with the required system variables, and a working Antigravity CLI configuration.

Then, a minimal Python MCP Server is built with stdio transport. This server is validated with Antigravity CLI in the local environment.

This setup validates the connection from Antigravity CLI to the local server via MCP. The MCP client (Antigravity CLI) and the Python MCP server both run in the same local environment.

Setup the Basic Environment

At this point you should have a working Python environment and a working Antigravity CLI installation. The next step is to clone the GitHub samples repository with support scripts:

cd ~
git clone https://github.com/xbill9/gemma4-tips-azure

Then run init.sh from the cloned directory.

The script will attempt to determine your shell environment and set the correct variables:

cd gpu-12B-qat-aca-devops-agent
source init.sh

If your session times out or you need to re-authenticate- you can run the set_env.sh script to reset your environment variables:

cd gpu-12B-qat-aca-devops-agent
source set_env.sh

Variables like PROJECT_ID need to be setup for use in the various build scripts- so the set_env script can be used to reset the environment if you time-out.

Model Management Tool with MCP Stdio Transport

One of the key features that the standard MCP libraries provide is abstracting various transport methods.

The high level MCP tool implementation is the same no matter what low level transport channel/method that the MCP Client uses to connect to a MCP Server.

The connection over stdio will look similar to this:

# Initialize FastMCP server
mcp = FastMCP("Self-Hosted vLLM DevOps Agent")

Running the Python Code

First- switch the directory with the Python version of the MCP sample code:

~/gemma4-tips-azure/cd gpu-12B-qat-aca-devops-agent

Run the release version on the local system:

xbill@penguin:~/gemma4-tips-azure/gpu-12B-qat-aca-devops-agent$ make install
pip install -r requirements.txt
Requirement already satisfied: mcp in /home/xbill/.pyenv/versions/3.13.13/lib/python3.13/site-packages (from -r requirements.txt (line 1)) (1.27.2)
Requirement already satisfied: fastmcp in /home/xbill/.pyenv/versions/3.13.13/lib/python3.13/site-packages (from -r requirements.txt (line 2)) (2.11.3)

The project can also be linted:

mxbill@penguin:~/gemma4-tips-azure/gpu-12B-qat-aca-devops-agent$ make lint
ruff check .
All checks passed!
ruff format --check .
7 files already formatted
mypy .
Success: no issues found in 7 source files
xbill@penguin:~/gemma4-tips-azure/gpu-12B-qat-aca-devops-agent$

MCP stdio Transport

One of the key features that the MCP protocol provides is abstracting various transport methods.

The high level tool MCP implementation is the same no matter what low level transport channel/method that the MCP Client uses to connect to a MCP Server.

In this project Antigravity CLI is used as the MCP client to interact with the Python MCP server code.

Antigravity CLI mcp_config.json

A sample MCP server file is provided in the .agents directory:

{
  "mcpServers": {
    "gpu-devops-agent": {
      "command": "python3",
      "args": [
        "/home/xbill/gemma4-tips-azure/gpu-12B-qat-aca-devops-agent/server.py"
      ],
      "env": {
        "GOOGLE_CLOUD_PROJECT": "aisprint-491218",
        "MODEL_NAME": "/mnt/models/gemma-4-12B-it-qat-w4a16-ct"
      }
    }
  }
}

Validation with Antigravity CLI

The final connection test uses Antigravity CLI as a MCP client with the Python code providing the MCP server:

MCP Servers

Plugins (~/.gemini/antigravity-cli/plugins)
> ✓ google-dev-knowledge Tools: search_documents, answer_query, get_documents
   ✓ gpu-devops-agent Tools: save_hf_token, get_vllm_endpoint, list_vertex_models, list_bucket_models,
                       analyze_cloud_logging, +25 more

Getting Started with Gemma 4 on GPU

The Official vLLM repo also has Gemma4 specific information:

Releases · vllm-project/vllm

The Gemma 12B model was just released:

Introducing Gemma 4 12B: a unified, encoder-free multimodal model

What about the QAT Models?

A deep dive into deploying the 12B QAT model is here:

12B Gemma 4 QAT Deployment with NVIDIA L4, Cloud Run, MCP, and Antigravity CLI

Lets Deploy this!

The model was deployed to Standard_NV36ads_A10_v5 backed with a NVIDIA GPU:

> status_azure_vm

● ListDir(/home/xbill/.gemini/antigravity-cli/mcp/gpu-devops-agent) 
● Read(/home/xbill/.gemini/antigravity-cli/mcp/gpu-devops-agent/status_azure_vm.json) 
● gpu-devops-agent/status_azure_vm(Call MCP Tool) (ctrl+o to expand)

  I have retrieved the status of the Azure Container App service:

  ### Azure Container App Status for service gpu-12b-qat-l4-devops-agent :

  • Container App Name: gpu-12b-qat-l4-devops-agent-app 
  • Provisioning State: Succeeded 
  • Running Status: Running 
  • FQDN: https://gpu-12b-qat-l4-devops-agent-app.purplesmoke-a1c60b91.eastus.azurecontainerapps.io

Now check the Docker Container:

> check_vllm

  Here is the verified status output of the check_vllm diagnostic check for the active Azure Container App:

  ### 🖥️ Azure Container App: gemma4-vllm-gpu 

  • Provisioning State: Succeeded 
  • Running Status: Running 
  • FQDN: https://gemma4-vllm-gpu.gentlefield-612b24aa.westus2.azurecontainerapps.io 
  • vLLM API Endpoint ( https://gemma4-vllm-gpu.gentlefield-612b24aa.westus2.azurecontainerapps.io/health ): Healthy ✅ 

  All deployment verification logs have been archived in aca_deployment_status.md.

Cross Check The Deployed Model

Once the model starts locally- the MCP tool allows for model verification:


> verify_model_health

  Here are the details from the deep model health check diagnostics:

  • Health Status: PASSED ✅
  • Target Model: google/gemma-4-12B-it-qat-w4a16-ct 
  • Response: 'Yes, I am working! I am Gemma 4, a large language ...' 
  • Latency: 2.23 seconds 

  Full details have been logged in aca_deployment_status.md.

and model details:

> > check model details

  I will call the get_model_details tool on the MCP server to retrieve the active vLLM serving parameters and
  configurations.

● gpu-devops-agent/get_model_details(Call get_model_details tool) (ctrl+o to expand)

  Here are the serving model details retrieved from the active vLLM instance:

  ### 🧩 Model Details

  • Public Endpoint: https://gemma4-vllm-gpu.gentlefield-612b24aa.westus2.azurecontainerapps.io
  • Model ID: google/gemma-4-12B-it-qat-w4a16-ct 
  • Object Type: model 
  • Owned By: vllm 
  • API Health Endpoint Status: Healthy ✅

and the endpoint:

> get_endpoint

  Here is the active vLLM service URL:

  🟢 vLLM Endpoint: https://gemma4-vllm-gpu.gentlefield-612b24aa.westus2.azurecontainerapps.io

and status:

> get_status

  Here is the verified GPU vLLM System Status:

  ### 🌀 GPU vLLM System Status

  • vLLM Health: 🟢 Online (https://gemma4-vllm-gpu.gentlefield-612b24aa.westus2.azurecontainerapps.io)
  • Hosting Status: 🟢 ACA Succeeded ( gemma4-vllm-gpu )

and metrics:

> get_metrics

  ### 📈 Request & Concurrency Metrics

  • Successfully Processed Requests: 2 (both completed via natural stop token)
  • Currently Active Running Requests: 0 
  • Currently Waiting Requests (Queued): 0 
  • KV-Cache Memory Usage: 0.0% 

  ### 🚀 Throughput & Token Stats

  • Total Prefill (Prompt) Tokens Processed: 36 tokens
  • Total Generated (Completion) Tokens Processed: 63 tokens

  ### ⏱️ Latency Metrics

  • Average Time to First Token (TTFT): 1.81 seconds (sum: 3.62s over 2 requests)
  • Average End-to-End Latency: 2.96 seconds (sum: 5.92s over 2 requests)

Check the Azure Instance

Once Deployed — the model is visible on the Azure Console:

Review the Model

Check the model details:

> what are the model details?

  Here are the active model deployment details:

  • Serving Endpoint: https://gemma4-vllm-gpu.gentlefield-612b24aa.westus2.azurecontainerapps.io
  • Model ID: google/gemma-4-12B-it-qat-w4a16-ct 
  • Quantization Format: compressed-tensors (4-bit weights, 16-bit activations)
  • API Engine / Server: vLLM 
  • VRAM Allocation (Tesla T4): 95% GPU memory utilization, 18,187 KV cache tokens
  • API Health status: Healthy (Online) ✅


> query_gemma4_with_stats what is Gemma?

  I will call the query_gemma4_with_stats tool on the MCP server to query the model and retrieve both the generated
  response and model performance statistics (like TTFT and total generation time).

● gpu-devops-agent/query_gemma4_with_stats(Call query_gemma4_with_stats tool) (ctrl+o to expand)

  Here is the response from the serving Gemma 4 model along with the query performance statistics:

  ### 💬 Model Response

  │ Depending on the context, Gemma usually refers to one of two very popular things: a Google AI model or a gemstone.
  │
  │ Here is a breakdown of both:
  │
  │ ### 1. In Technology: Google’s AI Model
  │
  │ Gemma is a family of open-model Large Language Models (LLMs) developed by Google DeepMind.
  │
  │ • Relationship to Gemini: It is built from the same research, technology, and infrastructure used to create Google's
  │ most capable AI model, Gemini.
  │ • "Open" Nature: Unlike the "closed" Gemini models (which are accessed primarily via API or Google products), Gemma
  is
  │ released as "open weights." This means developers and researchers can download, customize, and run the model on their
  │ own hardware (like a personal laptop or a private cloud).
  │ • Size Varieties: Gemma is designed to be lightweight. It comes in different sizes (such as 2B, 7B, 9B, and 27B
  │ parameters), making it easy for people who don't have massive supercomputers to run high-quality AI locally.
  │ • Capabilities: It can understand and generate text, reason, and even process images or audio (depending on the
  │ version). It is commonly used by developers to build chatbots, summarize documents, or power coding assistants.
  │ ──────
  │ ### 2. In Gemology: The Gemstone
  │
  │ Gemma is the Latin word for "gem" or "precious stone." In the world of jewelry and geology:
  │
  │ • Gemma is often used as a prefix or root for various precious stones (e.g., Gemmae refers to jewels).
  │ • However, if you are looking for a specific stone, you might be thinking of Garnet, Garner, or a specific variety of
  │ Jade.
  │ ──────
  │ ### 3. In Other Contexts
  │
  │ • Name: Gemma is a popular female given name of Italian origin, meaning "gem" or "precious stone."
  │ • Software/Coding: There are various smaller libraries or internal projects in coding communities named "Gemma," but
  │ they are less common than the Google AI model.
  │
  │ Summary: If you are asking about AI, Gemma is Google's "mini" version of Gemini that anyone can download and use for
  │ free.

  ### 📊 Performance Stats

  • Serving Model: google/gemma-4-12B-it-qat-w4a16-ct 
  • Time to First Token (TTFT): 0.26 seconds 
  • Total Generation Time: 19.68 seconds 
  • Generation Throughput: 25.74 tokens/s (~500 tokens total)

Benchmark the Local Model

The local model was benchmarked. Summarize the results:

> summarize benchmark_report.md

  Here is a summary of the performance results from benchmark_report.md:

  ### 🕒 Average Latency Highlights

  • Low Concurrency (1–4 users): Extremely fast response times ( 0.33s – 1.5s ) for all valid context windows (up to
  2048 tokens).
  • High Concurrency (1024–2048 users): Latency scales up to 23s – 32s due to queuing under heavy load.
  • Tipping Point: Latency remains under 3s up to 128 concurrent users for short prompts (<= 64 tokens) and under 8s 
  for large prompts (512 tokens).

  ### 🚀 Throughput Highlights

  • Max Throughput: Tops out at 47.7 requests/sec (4 tokens context window at 1024 users).
  • Prefabricated Prefill Impact: Throughput drops linearly as context window size increases, reaching a maximum
  throughput of 5.2 requests/sec at 2048 tokens.
  • Concurrency Saturation: Throughput plateaus once concurrency reaches 128–256 users; additional users increase latency
  rather than total request throughput.

  ### 🚫 Limit Violations

  • 4096 tokens and above: Reports 0.0% success rate and 0.0 throughput across all concurrencies because the prompt
  length exceeds the serving engine's maximum limit ( max-model-len = 4096 ).

This is visualized:

GPU Memory Allocation of QAT 12B Models

The GPU provides 28 GB of GPU Memory:

> > what is the memory breakdown on the GPU?
  Here is the approximate memory breakdown on the NVIDIA Tesla T4 GPU (16 GB VRAM) running the google/gemma-4-12B-it-qat-
  w4a16-ct model via vLLM:

  ### 📟 VRAM Allocation Overview (16 GB Total)

   VRAM Category | Size (GiB) | % of total GPU | Purpose
  ---------------------------|--------------|----------------|-----------------------------------------------------------
   Model Weights | ~ 6.50 GiB | ~ 40.6% | Gemma-4 12B QAT 4-bit weights
   KV Cache | 5.85 GiB | 36.6% | Reserved space for context tokens ( 18,187 FP8 tokens)
   CUDA / Workspace Overhead | ~ 2.85 GiB | ~ 17.8% | Run-time activations, CUDA context, graph capture
   Free / System Reserve | 0.80 GiB | 5.0% | Unallocated (enforced by --gpu-memory-utilization 0.95 )
  ──────
  ### 🔍 Key Metrics Details

  1. Model Compression Advantage:
      • An unquantized 12B parameter model in bfloat16 requires at least 24 GB of VRAM just for the weights (which
      exceeds the Tesla T4's 16 GB limit).
      • Thanks to QAT (4-bit quantization), weights occupy only ~6.5 GB, leaving ample space (5.85 GB) for the KV cache
      on a single cost-effective T4 GPU.
  2. KV Cache Efficiency:
      • Quantized to FP8 (via --kv-cache-dtype fp8 ), cutting the token memory footprint in half.
      • This provides a total capacity of 18,187 tokens (5,990 blocks of size 16), which translates to a maximum
      concurrency of 4.44x concurrent users at a full 4,096 context length.

────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
>

What are the performance characteristics of the Model?

> what are the performance characteristics of the deployment

Here are the core performance characteristics of the vLLM deployment running the Gemma-4 12B QAT model on an NVIDIA    
  Tesla T4 GPU (Azure Container Apps):

### 1. Latency Profile (TTFT & E2E)

• Time to First Token (TTFT): 0.26 seconds under low concurrency, indicating highly optimized prefill compilation.
  • Average E2E Latency: 2.96 seconds for a standard completion task.
  • Queuing Scaling: Latency scales linearly with user concurrency once the queue saturates (reaching ~21s at 256 users
  and ~31s at 1024+ users).

### 2. Throughput & Scaling

• Prefill Throughput Peak: Achieves 47.7 requests per second for very small context queries (4 tokens) at concurrency
  1024.
  • Payload Throughput Bounds: Peak throughput drops to 5.2 requests per second at the model's active 2048-token context
  window boundary.
  • Saturation Plateau: The deployment saturates around 128–256 concurrent users. Adding users beyond this point
  increases queuing latency without improving aggregate throughput.

### 3. Context & Limits

• Maximum Context Size: Strictly capped at 4096 tokens (as configured by --max-model-len ). Queries equal to or larger
  than 4096 tokens are rejected instantaneously (0% success rate).
  • KV-Cache Pool Size: Accommodates 18,187 concurrent tokens using FP8 KV cache quantization.

### 4. Hardware Efficiency (The QAT Advantage)

• Datacenter Accessibility: Quantization-Aware Training (QAT) fits the 12B parameter model into a cost-efficient 16 GB 
  Tesla T4 GPU while reserving 5.85 GB for the active cache.
  • Without QAT, this model would require an A10 or L4 GPU (24 GB VRAM) to start up successfully.

────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
>

One last cross check — what are the trade-offs of the deployment?

> what are the trade-offs on the deployment?

Here are the primary technical and financial trade-offs in this deployment architecture:

### 1. Model Quantization: QAT 4-bit (INT4) vs. Full Precision (BF16)

• The Trade-off: VRAM Savings vs. Mathematical Precision
  • Pros: Quantization-Aware Training (QAT) reduces model size to ~6.5 GB, allowing a 12B model to be served on a cheap
  16 GB GPU while leaving room for the KV cache.
  • Cons: There is a minor quality degradation in reasoning and function-calling precision compared to the native BF16
  checkpoint (though QAT retains ~99% of baseline performance, far outperforming standard Post-Training Quantization).

### 2. GPU Hardware: Tesla T4 vs. NVIDIA L4

• The Trade-off: Infrastructure Cost vs. Generation Speed
  • Pros: The Tesla T4 (NC8as-T4 workload profile on Azure) is significantly cheaper and more widely available in cloud
  regions than modern NVIDIA L4 GPUs.
  • Cons: The T4 runs on the older Turing architecture and lacks native hardware Tensor Core support for bfloat16 
  execution. This results in slower token generation speeds (~25 tokens/s) than an L4 GPU.

### 3. KV Cache Format: FP8 vs. FP16 Cache

• The Trade-off: Concurrency Capacity vs. Long-Context Recall
  • Pros: Compressing the KV cache to 8-bit ( --kv-cache-dtype fp8 ) cuts memory requirements in half, increasing token
  capacity to 18,187 tokens.
  • Cons: Quantizing attention keys and values can introduce small rounding errors in attention calculations, slightly
  degrading retrieval accuracy in long-context prompts (approaching the 4K boundary).

### 4. Active Context Cap: 4,096 tokens vs. 32,768 tokens (Model Native)

• The Trade-off: Server Stability vs. Maximum Document Size
  • Pros: Capping the context at 4096 tokens ( --max-model-len 4096 ) guarantees the system will not exhaust VRAM or
  crash under massive concurrent requests.
  • Cons: The model cannot process long documents, large codebases, or extended chat histories that exceed 4K tokens,
  rejecting them immediately.

So where is the best place to run Gemma? AWS or GCP? Azure?

Survey says…

> compare aws vs gcp vs azure performance

> summarize cloud_comparison_report.md

  Here is a quick summary of cloud_comparison_report.md:

  ### 📋 Key Metrics Comparison

   Metric | AWS EC2 (VM + L4) | GCP Cloud Run (Serverless + L4) | Azure ACA (Serverless + T4)
  ---------------------------|---------------------------|---------------------------------|-----------------------------
   Max Context Size | 16,384 tokens | 16,384 tokens | 4,096 tokens
   Peak Throughput | 93.3 req/s | 40.3 req/s | 47.7 req/s
   First Token Latency | 0.09s | 0.14s | 0.26s
   Max Stable Users | 1024 | 512 | 256

  ### 💡 Core Takeaways

  1. AWS EC2 (L4 GPU VM): Best raw performance and highest stability under extreme concurrency (1024+ users) since
  dedicated hardware avoids serverless hypervisor overhead.
  2. GCP Cloud Run (L4 GPU Serverless): Best cost-performance balance. Supports large context lengths (16K) and scales to
  zero when idle, but throughput is lower than dedicated VMs due to proxy networking.
  3. Azure Container Apps (T4 GPU Serverless): Good for serverless workloads but constrained by older GPU hardware (Tesla
  T4), restricting context length to 4K to prevent Out-of-Memory crashes.

What about costs?!?

The serving costs were summarized:

> summarize azure_cost_comparison.md

  Here is a summary of azure_cost_comparison.md:

  ### 📋 Cost Model Summary

  • Azure Container Apps (ACA - Serverless): Billed per second of active execution. Replicas scale down to zero when idle,
  incurring no compute charges.
      • Tesla T4 Profile: ~$2.20/hr. Cheapest option for intermittent use (costs ~$316/month at 20% active utilization).
      • A10G Profile: ~$3.90/hr. Best for serverless workloads that require a larger 24 GB VRAM GPU.
  • Dedicated GPU VM (Standard_NV36ads_A10_v5): Billed hourly at ~$3.20/hr regardless of utilization (costs ~$2,         
  304/month). Offers the cheapest cost-per-hour for 24/7 continuous operations but cannot scale to zero.

  ### 💡 Decision Matrix

  • Choose ACA (T4 or A10G) if your SRE diagnostic workloads are trigger-based, intermittent, or idle for a majority of
  the day.
  • Choose Dedicated VM (A10G) if you are running continuous, high-concurrency production inference pipelines 24/7.

What about other cloud costs?

> summarize multi_cloud_cost_comparison.md

  Here is a quick summary of multi_cloud_cost_comparison.md:

  ### 📋 Cost and VM Performance Characteristics

  • GCP Compute Engine ( g2-std-8 ): Cheapest VM option at ~$0.85/hr (~$612/month) for a dedicated NVIDIA L4 GPU.
  • AWS EC2 ( g6.2xlarge ): Highly competitive dedicated VM rate at ~$0.98/hr (~$705/month) with unmatched stability
  under high concurrency.
  • GCP Cloud Run (Serverless L4): Best serverless choice. Costs ~$1.48/hr (only ~$213/month at 20% active runtime) and
  hosts a modern 24 GB VRAM L4 GPU.
  • Azure Container Apps (Serverless T4): More expensive (~$2.20/hr) for an older 16 GB VRAM Tesla T4 GPU, restricting
  active context length to 4K.
  • Azure VM ( NV36ads_A10_v5 ): Most expensive VM option (~$3.20/hr) due to larger bundled CPU/RAM profiles.

  ### 💡 Recommendation

  • If migrating to save cost on scale-to-zero (intermittent load): Choose GCP Cloud Run.
  • If migrating for continuous, high-concurrency 24/7 runtimes: Choose a dedicated GCP Compute Engine VM or AWS EC2 VM.

Summary

The strategy for using MCP for Gemma 4 GPU deployment with Antigravity CLI and Azure was validated with an incremental step by step approach.

Firebase Midsommer Madnesss with Antigravity CLI

xbill — Fri, 19 Jun 2026 00:18:39 +0000

This is a submission for the June Solstice Game Jam

This installment brings a Firebase build to Midsommer Madness via Antigravity CLI. The Midsommer Madness web app is deployed to Firebase Web Hosting and the Maypole was saved!

What I Built

When it comes to Summar Solstace — the place to be is Sweden. It is one of the highlights of the calendar.

This project aimed to recreate some of the mystique around the event- just in time for some fresh surestromming!

Now you can get it with Firebase enhancements!

FireBase Console

The full project was imported to the Firebase Console:

Just show me the Code!!!

GitHub Repo is here:

GitHub - xbill9/midsommer-firebase

Midsommer Madness with Firebase

So how can we use Firebase to enhance the project?

Github and Skills!

  The GitHub repository has been successfully linked to your Firebase project midsommer-madness .

  ### What was configured:

  1. GitHub Secrets: A service account ( github-action-1273534909 ) with Firebase Hosting admin permissions was
  created, and its credentials were uploaded to your GitHub repository secrets as                                 
  FIREBASE_SERVICE_ACCOUNT_MIDSOMMER_MADNESS .
  2. GitHub Actions Workflows: Two workflow files were generated and pushed to your GitHub repository:
      • firebase-hosting-pull-request.yml: Deploys a preview channel of the game whenever a PR is created against
the master branch.
      • firebase-hosting-merge.yml: Automatically deploys the latest version to the live Firebase Hosting site
whenever a PR  
      is merged (or a commit is pushed) to the master branch.
  3. Agent Skills: Installed Firebase Agent Skills in the repository to make it easier for agents to help you with
  Firebase configurations in the future.

> /skills
  ⎿ Exited /skills command

What else?

> how is firebase enhancing the project

● Read(/home/xbill/midsommer-firebase/lib/main.dart) (ctrl+o to expand)

  Firebase enhances the Midsommer Madness project in four primary ways, turning it from a simple local game into a
  connected, production-ready mobile application:

  ### 1. 🏆 Global Cloud Leaderboard (Cloud Firestore)

  • Real-Time Synchronization: Instead of keeping scores local to one device, it uses Firestore to sync high
  scores globally. Players are ranked in real time against others on a Top 10 leaderboard.
  • Offline Resiliency: It acts as a fallback system. If a player is offline, the app saves scores to device cache
  ( SharedPreferences on mobile, localStorage on web). Once connectivity returns, it synchronizes gracefully
  without crashing or interrupting gameplay.

  ### 2. 📊 Gameplay Analytics (Firebase Analytics)

  • Event Tracking: Telemetry events like leaderboard_fetched and high score submissions are tracked.
  • Player Behavior: It records details such as the submitted score, the levels played, and character names,
  providing insight into game balance, player engagement, and high score distributions.

  ### 3. 🔍 Error Tracking & Stability (Firebase Crashlytics)

  • Dart & Flutter Crash Reporting: Captures native app crashes or Flutter rendering exceptions instantly.
  • JavaScript Bridge Error Logging: Uncaught JavaScript errors inside the WebView game loop are caught by game. 
  js and forwarded to Crashlytics via the LeaderboardChannel JavaScript channel. This allows you to debug
  client-side web bugs directly inside the Firebase Console.

  ### 4. ⚡ Performance Metrics (Firebase Performance Monitoring)

  • Custom Network Traces: Measures the request latency of fetching and saving leaderboard scores (using custom
  traces get_leaderboard_scores and save_leaderboard_score ).
  • Slow Load Detection: Ensures that network calls to Firestore do not freeze the UI or lag the 60Hz physics
  update loop of the game.

My Lingonberries are ripening- get to work!

How do you deploy this?

> make help
● Bash(make help) (ctrl+o to expand)
  Here is the updated output of the make help command:

    ========================================================================                                      
    🇸🇪 Midsommer Madness Flutter Build & Development Controls 🇸🇪                                                
    ========================================================================                                      
    Available commands:                                                                                           
      make dev - Start the local web server for browser play (from assets/)                              
      make build-apk - Compile the Flutter App and build Debug APK                                             
      make build-ios - Compile the Flutter App and build iOS app (no codesign)                                 
      make clean - Clean Flutter build outputs and temporary caches                                        
      make logcat - Monitor application logs using Flutter logger                                           
      make deploy - Deploy the game to Firebase Hosting (manual bypass)                                     
      make firebase-logs - Fetch the latest cloud logs from Google Cloud / Firebase                               
      make firebase-emulators - Start local Firebase Emulator Suite (Firestore & Hosting)                         
      make deploy-preview - Deploy a temporary preview channel to Firebase Hosting                                
      make firebase-status - Check current Firebase project configurations                                        
      make deploy-rules - Deploy security rules for Cloud Firestore                                               
      Note: Pushing/merging to 'master' on GitHub automatically triggers deployment.                              
    ========================================================================

Getting Firebase logs:

> make firebase-logs

● Bash(make firebase-logs) (ctrl+o to expand)
  Here are the latest Firebase/GCP cloud logs fetched from your project:

    TIMESTAMP SEVERITY TYPE TEXT_PAYLOAD MESSAGE
    2026-06-18T16:34:32.784748Z NOTICE audited_resource
    2026-06-18T16:33:37.512215Z NOTICE audited_resource
    2026-06-18T16:11:41.516983584Z NOTICE service_account
    2026-06-18T16:11:40.698356Z NOTICE project
    2026-06-18T16:11:40.054007713Z ERROR service_account
    2026-06-18T16:10:45.753536986Z ERROR service_account
    2026-06-18T16:10:29.417088907Z NOTICE service_account
    2026-06-18T15:43:46.907263Z NOTICE audited_resource
    2026-06-18T15:38:51.598698444Z NOTICE audited_resource
    2026-06-18T15:38:49.452718Z NOTICE audited_resource
    2026-06-18T15:32:56.365232Z NOTICE audited_resource
    2026-06-18T15:32:37.077690Z NOTICE audited_resource
    2026-06-18T15:31:45.425200Z NOTICE project
    2026-06-18T15:31:43.935218Z ERROR project
    2026-06-18T15:31:42.933069Z ERROR project
    2026-06-18T15:28:42.228633Z NOTICE project
    2026-06-18T15:28:41.473570627Z NOTICE service_account
    2026-06-18T15:27:38.230991Z NOTICE project
    2026-06-18T15:27:37.538742814Z NOTICE audited_resource
    2026-06-18T15:27:37.168036Z NOTICE audited_resource

Getting Firebase Status:

> make firebase-status

● Bash(make firebase-status) (ctrl+o to expand)

  Here is the output of the make firebase-status command:

    Checking active Firebase project list...                                                                      
    npx -y firebase-tools projects:list                                                                           
    - Preparing the list of your Firebase projects                                                                
    ✔ Preparing the list of your Firebase projects                                                                
      make install-apk - Install the compiled debug APK on a connected device/emulator                           
    ✔ Preparing the list of your Firebase projects
    ┌──────────────────────┬─────────────────────────────┬────────────────┬──────────────────────┐
    │ Project Display Name │ Project ID │ Project Number │ Resource Location ID │
    ├──────────────────────┼─────────────────────────────┼────────────────┼──────────────────────┤
    │ Midsommer Madness │ midsommer-madness (current) │ 8002619985 │ [Not specified] │
    └──────────────────────┴─────────────────────────────┴────────────────┴──────────────────────┘

  The active project is correctly marked as midsommer-madness (current).

App Debugging

The local Antigravity CLI has full access to the source code and with the Firebase skills — it can be use for remote debugging:

  I found the root causes of the freeze/shaking behavior on Game Over:

  ### 1. 🐛 HTML Nesting Bug (Why it didn't end / show Game Over)

  There was a missing closing </div> tag for the #screenIntro (Level Intro) overlay in index.html.

  • The issue: Because the intro screen div was never closed, the Game Over Screen ( #screenGameOver ) and Victory
  Screen ( #screenVictory ) divs were parsed as nested children of the intro screen.
  • The result: Since the intro screen is hidden ( display: none; ) during gameplay, the Game Over screen was also
  hidden by inheritance—even when the game engine correctly updated its class to .active ! The screen stayed
  blank, showing only the canvas background.

  ### 2. 📳 Screen Shake Jitter Loop (Why it stayed "gittery")

  • The issue: When you die or take damage, the game triggers a screen shake ( shakeIntensity > 0 ). This
  intensity is normally decremented and decayed back to 0 inside the game's update() loop.
  • The result: However, on death, the game state immediately transitions to GAMEOVER , which halts the update()
  loop. Because update() was no longer running, shakeIntensity was never decremented, causing the rendering
  engine to render the canvas with a randomized shake offset on every frame forever.
  ### 🚀 Solution Applied:
  • HTML Fix: Correctly closed the #screenIntro overlay in index.html so that the Game Over card sits on its
  own layer.
  • State Manager Fix: Added shakeIntensity = 0; to both the GAMEOVER and VICTORY state transitions in
  game.js to immediately neutralize any ongoing screen shake when the game ends.

  These fixes have been committed and pushed to master , and the live version at https://midsommer-              
  madness.web.app is currently auto-updating. Try refreshing the game page in a minute to test the fix!

────────────────────────────────────────────────────────────
> how can I test local

  You can test the game locally in a few different ways depending on what you want to test:

  ### 1. 🌐 Test Web Game in Browser (Fastest)

  To test game changes locally in your browser:

    make dev                                                                                                      

  • What it does: Starts a lightweight HTTP server on your machine serving the static files inside the assets/ 
  directory.
  • How to play: Open your browser and go to the local address outputted in your console (e.g.,                   
  http://localhost:8080 ).
  ──────
  ### 2. 🔥 Test Firebase Features Locally (Emulators)

  To test Firestore leaderboard database sync and mock hosting without touching production data:

    make firebase-emulators                                                                                       

  • What it does: Starts the local Firebase Emulator Suite (hosting, firestore database, etc.) on your machine.
  • How to play: Open the local hosting emulator link printed in your terminal. You can check the database records
  in the emulator UI dashboard (usually http://localhost:4000 ).
  ──────
  ### 📱 3. Test the Android App Wrapper

  If you have an Android device or emulator running:
  1. Build the debug APK:
    make build-apk                                                                                                

  2. Install it on your device/emulator:
    make install-apk                                                                                              

  3. Monitor the live app logs for errors/messages:
    make logcat                                                                                                   

────────────────────────────────────────────────────────────
> make dev

● Bash(make dev)
● ManageTask(Task: cccfd6c1-93d3-48aa-be84-ea0d884dfe4f/task-323) (ctrl+o to expand)

  The local development server has been started in the background.

  You can play and test the game locally in your browser at:
  👉 http://127.0.0.1:8081

────────────────────────────────────────────────────────────
> ok local version works. how do I test web

● Bash(gh run list --limit 3) (ctrl+o to expand)

  The automated GitHub Action deployment workflow for your latest fix ("Fix nested screenGameOver...") has already
  completed successfully!

  You can play and test the live web version right now at:
  👉 https://midsommer-madness.web.app

  (Note: If you don't see the changes immediately, you may need to force-refresh your browser—usually Ctrl + F5  
  or Cmd + Shift + R —to clear your browser's local asset cache).

Run it on the Web!

The current version is now available at:

Midsommer Madness - Swedish Retro Arcade

https://midsommer-madness.web.app

Watch the Leader Board

Looks like Rolfe has been skilling up!

Level Design

The key levels include:

IKEA Warehouse: Battle crowded, flatpack-carrying shoppers who throw box projectiles at you.
Systembolaget: The state-owned liquor shop crowded with drunk Swedes stumbling and lobbing green beer bottles.
Lördagsgodis: Sugar rush Saturday! Dodge hyperactive, strung-out Swedish kids throwing sweet candy projectiles.
The Swedish Pub: Sing along with Frank Zappa fans singing “Bobby Brown” (shouting and firing glowing letters B, O, B, B, Y).
Volvo Highway: A survival lane-crossing level where…

This is Fun! How do I add my own levels?

Antigravity CLI has full visibility into the game:

> add a godis level where strung out kids throw lordagsgodis ▸ Thought for 2s, 525 tokens Considering Level Placement I will search for level ID checks referencing level 9 in game.js to prepare for adding a 10th level. ● Search(Search for lvl.id === 9 checks) (ctrl+o to expand) I will view lines 495 to 585 in game.js to inspect the current level configurations. ● Read(/home/xbill/midsommer-madness/game.js) (ctrl+o to expand) ▸ Thought for 2s, 1.8k tokens Sequencing the Levels I will view the setLevelAudioMode method in game.js to prepare for adding Level 3 "Lördagsgodis". ● Read(/home/xbill/midsommer-madness/game.js) (ctrl+o to expand) I will update the setLevelAudioMode method in game.js to map the scales and tempos for the 10-level layout. ● Edit(/home/xbill/midsommer-madness/game.js) (ctrl+o to expand) I will view the current level checks in tickSequencer inside game.js to prepare the next set of replacements. ● Read(/home/xbill/midsommer-madness/game.js) (ctrl+o to expand) I will shift the level conditionals in the music sequencer tickSequencer method to accommodate the 10-level progression. ● Edit(/home/xbill/midsommer-madness/game.js) (ctrl+o to expand) I will view lines 920 to 935 of game.js to locate the exact Maypole check in initLevel . ● Read(/home/xbill/midsommer-madness/game.js) (ctrl+o to expand) I will view lines 1060 to 1070 of game.js. ● Read(/home/xbill/midsommer-madness/game.js) (ctrl+o to expand) I will view lines 1290 to 1300 of game.js. ● Read(/home/xbill/midsommer-madness/game.js) (ctrl+o to expand) I will view lines 1360 to 1375 of game.js. ● Read(/home/xbill/midsommer-madness/game.js) (ctrl+o to expand) I will view lines 1370 to 1385 of game.js. ● Read(/home/xbill/midsommer-madness/game.js) (ctrl+o to expand) I will view lines 1460 to 1475 of game.js. ● Read(/home/xbill/midsommer-madness/game.js) (ctrl+o to expand) I will update the level ID checks for the final Maypole level from 9 to 10 and register the candy-throwing kids target name in the HUD. ● Edit(/home/xbill/midsommer-madness/game.js) (ctrl+o to expand) I will view the start of the levels array in game.js to prepare the level insertion edit. ● Read(/home/xbill/midsommer-madness/game.js) (ctrl+o to expand) I will view lines 545 to 585 of game.js. ● Read(/home/xbill/midsommer-madness/game.js) (ctrl+o to expand) ▸ Thought for 3s, 1.2k tokens Refining The Level Design I will update the levels array in game.js to insert Lördagsgodis as Level 3, shift subsequent level IDs, and re-calculate scores. ● Edit(/home/xbill/midsommer-madness/game.js) (ctrl+o to expand) I will view lines 950 to 985 of game.js to locate the spawnEnemy switch statement. ● Read(/home/xbill/midsommer-madness/game.js) (ctrl+o to expand) I will view lines 985 to 1010 of game.js. ● Read(/home/xbill/midsommer-madness/game.js) (ctrl+o to expand) I will add the "kid" case to the spawnEnemy switch block. ● Edit(/home/xbill/midsommer-madness/game.js) (ctrl+o to expand) ▸ Thought for 3s, 2.1k tokens Considering Projectile Design I will search for class ZappaFan in game.js to locate a suitable place to insert CandyKid and Lordagsgodis . ● Search(Search for ZappaFan in game.js) (ctrl+o to expand) I will view lines 2120 to 2145 in game.js. ● Read(/home/xbill/midsommer-madness/game.js) (ctrl+o to expand) I will insert the CandyKid and Lordagsgodis class definitions in game.js right after the Drunkard class. ● Edit(/home/xbill/midsommer-madness/game.js) (ctrl+o to expand) I will view lines 275 to 325 of index.html. ● Read(/home/xbill/midsommer-madness/index.html) (ctrl+o to expand) I will update index.html to add Level 3 Lördagsgodis to the sidebar levels tracker and update levels 4 through 10. ● Edit(/home/xbill/midsommer-madness/index.html) (ctrl+o to expand) I will run a syntax check on game.js using node to ensure the new classes and level configurations are syntax-error free. ● Bash(node -c game.js) (ctrl+o to expand) I will view lines 20 to 45 of game_overview.md to inspect the levels sequence list. ● Read(/home/xbill/.gemini/antigravity-cli/brain/a...2ba-4622-b252-003672e1b73f/game_overview.md) (ctrl+o to expand) I will update game_overview.md to describe the 10-level layout including Lördagsgodis. ● Edit(/home/xbill/.gemini/antigravity-cli/brain/a...2ba-4622-b252-003672e1b73f/game_overview.md) (ctrl+o to expand) I have successfully added the Lördagsgodis level to Midsommer Madness, featuring hyperactive, sugar-rushed children throwing Saturday candy at Sven!

Summary of Changes

Level Setup ( game.js ): * Inserted Lördagsgodis as game.js in the levels array, shifting following levels up to Level 10 (ABBA Disco). * Updated all final-level check logic throughout game.js from Level 9 to Level 10.
Added CandyKid & Lordagsgodis Classes ( game.js ): * Defined game.js and game.js right after the Drunkard class. * Behavior: * CandyKid has high speed (1.8), low health (35), and throws sweets rapidly. * Kids shiver/vibrate rapidly on screen to depict their extreme sugar rush. * Lordagsgodis are fast-moving candy projectiles (round jelly drops, square sweet gel blocks, and candy star crosses) that rotate in flight and come in a variety of bright colors.
Music & Audio: * Modified game.js to include a happy, hyperactive 150 BPM C Major chiptune track for Level 3 (Lördagsgodis). * Shifted level conditions in game.js to map Levels 8, 9, and 10 to their respective custom chiptune hooks.
UI & Documentation: * renumerated the sidebar tracker list in index.html to show the 10-level layout. * Updated the level progression descriptions in game_overview.md.

Phew! That was a lot!

This installment covered extending the Midsommer madness app with Firebase tools to back the web based versions. Future enhancements under consideration include using crashalytics across mobile platforms.

Debugging Deployments with Gemma 12B, TPU v6e-1, MCP, and Antigravity CLI

xbill — Wed, 17 Jun 2026 15:04:24 +0000

This article provides a step by step debugging guide for deploying Gemma 4 to a Google Cloud TPU system,. A suite of Python MCP tools is built to simplify management of the vLLM hosted Gemma 4 deployment with Antigravity CLI.

What is this project trying to Do?

This project is a DevOps/SRE assistant that uses a Gemma 4 model hosted on TPU. It provides tools to provision the Docker container and deploy the model, as well as for observability and performance testing.

This project is similar to a previous project that targeted GPU hosted Gemma4 instances on GCP:

Gemma-SRE: Self-Hosted vLLM Infrastructure Agent

Antigravity CLI

Antigravity CLI is the follow-on successor to Gemini CLI- the terminal driven, agent assisted coding tool.

Full details on installing Antigravity CLI are here:

Getting Started with Antigravity CLI

Testing the Antigravity CLI Environment

Once you have all the tools in place- you can test the startup of Antigravity CLI.

You will need to authenticate with a Google Cloud Project or your Google Account:

agy

This will start the interface:

Full Installation Instructions

The detailed installation instructions for Antigravity CLI are here:

Getting Started with Antigravity CLI

Python MCP Documentation

The official GitHub Repo provides samples and documentation for getting started:

GitHub - modelcontextprotocol/python-sdk: The official Python SDK for Model Context Protocol servers and clients

Where do I start?

The strategy for starting MCP development for model management is a incremental step by step approach.

First, the basic development environment is setup with the required system variables, and a working Antigravity CLI configuration.

Then, a minimal Python MCP Server is built with stdio transport. This server is validated with Antigravity CLI in the local environment.

This setup validates the connection from Antigravity CLI to the local server via MCP. The MCP client (Antigravity CLI) and the Python MCP server both run in the same local environment.

Setup the Basic Environment

At this point you should have a working Python environment and a working Antigravity CLI installation. The next step is to clone the GitHub samples repository with support scripts:

cd ~
git clone https://github.com/xbill9/gemma4-tips

Then run init.sh from the cloned directory.

The script will attempt to determine your shell environment and set the correct variables:

cd tpu-12B-v6e1-devops-agent
source init.sh

If your session times out or you need to re-authenticate- you can run the set_env.sh script to reset your environment variables:

cd tpu-12B-v6e1-devops-agent
source set_env.sh

Variables like PROJECT_ID need to be setup for use in the various build scripts- so the set_env script can be used to reset the environment if you time-out.

Model Management Tool with MCP Stdio Transport

One of the key features that the standard MCP libraries provide is abstracting various transport methods.

The high level MCP tool implementation is the same no matter what low level transport channel/method that the MCP Client uses to connect to a MCP Server.

The connection over stdio will look similar to this:

# Initialize FastMCP server
mcp = FastMCP("Self-Hosted vLLM DevOps Agent")

Running the Python Code

First- switch the directory with the Python version of the MCP sample code:

xbill@penguin:~/gemma4-tips/tpu-12B-v6e1-devops-agent$ make install
pip install -r requirements.txt

The project can also be linted:

xbill@penguin:~/gemma4-tips/tpu-12B-v6e1-devops-agent$ make lint
ruff check .
All checks passed!
ruff format --check .
6 files already formatted
mypy .
Success: no issues found in 6 source files

And a test run:

xbill@penguin:~/gemma4-tips/tpu-12B-v6e1-devops-agent$ make test
python test_agent.py
......2026-06-15 11:06:05,954 - vllm-devops-agent - INFO - Querying model with stats with prompt: 'Hi...'
2026-06-15 11:06:05,954 - vllm-devops-agent - ERROR - Error fetching active model name from vLLM: object MagicMock can't be used in 'await' expression
2026-06-15 11:06:05,955 - vllm-devops-agent - INFO - Model response with stats: TTFT=0.000s, TotalTime=0.000s
..2026-06-15 11:06:05,960 - vllm-devops-agent - INFO - Starting TPU VM node node-1 in us-east5-a...
.2026-06-15 11:06:05,961 - vllm-devops-agent - INFO - Getting status for TPU VM node node-1 in us-east5-a...
.2026-06-15 11:06:05,962 - vllm-devops-agent - INFO - Stopping TPU VM node node-1 in us-east5-a...
.2026-06-15 11:06:05,964 - vllm-devops-agent - ERROR - Error fetching active model name from vLLM: object MagicMock can't be used in 'await' expression
.
----------------------------------------------------------------------
Ran 12 tests in 0.069s

OK

MCP stdio Transport

One of the key features that the MCP protocol provides is abstracting various transport methods.

The high level tool MCP implementation is the same no matter what low level transport channel/method that the MCP Client uses to connect to a MCP Server.

In this project Antigravity CLI is used as the MCP client to interact with the Python MCP server code.

Antigravity CLI mcp_config.json

A sample MCP server file is provided in the .agents directory:

{
  "mcpServers": {
    "tpu-v6e1-12B": {
      "command": "python3",
      "args": [
        "/home/xbill/gemma4-tips/tpu-12B-v6e1-devops-agent/server.py"
      ],
      "env": {
        "GOOGLE_CLOUD_PROJECT": "aisprint-491218",
        "ACCELERATOR_TYPE": "v6e-1"
      }
    }
  }
}

Validation with Antigravity CLI

The final connection test uses Antigravity CLI as a MCP client with the Python code providing the MCP server:

MCP Servers

Plugins (~/.gemini/antigravity-cli/plugins)
> ✓ google-dev-knowledge Tools: search_documents, answer_query, get_documents
   ✓ tpu-v6e1-12B Tools: verify_model_health, save_hf_token, get_vllm_deployment_config,
                   get_vllm_tpu_deployment_config, destroy_queued_resource, +25 more

Model Lifecycle Management via MCP

The MCP tools provide a complete suite of agent-oriented operations for managing vLLM deployment on Cloud Run or a TPU.

Overview of MCP tools :

xbill@penguin:~/gemma4-tips/tpu-12B-v6e1-devops-agent$ make help
Available MCP tools in server.py:

  get_vllm_deployment_config - Generates the gcloud command for a single-host TPU v6e vLLM deployment.
  get_vllm_tpu_deployment_config - Generates a GKE manifest for a TPU v6e vLLM deployment.
  list_queued_resources - Lists all Queued Resources in a specific zone.
  describe_queued_resource - Provides detailed information about a specific Queued Resource.
  get_reservation_status - Checks the lifecycle state and expiry time of a Queued Resource.
  check_tpu_availability - Simple check to see if a Queued Resource has reached ACTIVE state.
  estimate_deployment_cost - Estimates the cost of a TPU deployment.
  get_system_status - Status dashboard prioritizing Queued Resource states in us-central1-a.
  orchestrate_gemma4_stack - Seamless turnkey deployment of the Gemma 4 stack.
  get_vllm_endpoint - Discovery tool to verify connectivity and return the active vLLM service URL.
  deploy_queued_vllm - Deploys vLLM strictly using Queued Resources for Flex-start allocation.
  create_tpu_queued_resource - Creates a TPU Queued Resource (Flex-start) with the specified configuration.
  check_tpu_utilization - Monitors Tensor Core and HBM pressure on the TPU VM.
  get_vllm_metrics - Fetches real-time Prometheus metrics from the active vLLM service.
  validate_gemma4_deployment - Performs a comprehensive sanity check on the Gemma 4 deployment.
  query_queued_gemma4 - Queries the model hosted on the active Queued Resource.
  run_vllm_benchmark - Runs vLLM's internal benchmark tool inside the container on the TPU VM.
  run_vllm_internal_benchmark - Runs vLLM's internal benchmark tool inside the container on the TPU VM.
  run_external_load_test - Performs an external load test against the active vLLM endpoint.
  get_gemma4_full_report - Generates a deep technical report of the Gemma 4 deployment.
  fetch_tpu_vm_logs - Fetches specific logs from a TPU VM.
  grep_tpu_logs - Searches for a pattern in both startup and container logs on the TPU VM.
  destroy_queued_resource - Safely deletes a Queued Resource and its associated node.
  fetch_queued_node_logs - Fetches logs by identifying the node linked to a Queued Resource.
  save_hf_token - Saves HF token to GCP Secret Manager for the Queued Resource deployer.
  analyze_cloud_logging - Searches Cloud Logging for TPU-related errors and lifecycle events.
  verify_model_health - Performs a deep health check by querying the model with a simple prompt.
  query_vllm_with_metrics - Queries the model and provides streaming-based performance metrics.
  get_vllm_model_stats - Aggregates model-specific statistics from the vLLM server.
  find_tpu - Finds a zone with available quota and attempts to create the TPU queued resource.
  start_v6e1 - Starts a TPU v6e-1 instance.
  stop_v6e1 - Stops a TPU v6e-1 instance.
  status_v6e1 - Retrieves status and configuration details for the TPU VM node.
  get_help - Provides help text and summarizes the configuration options and tools.

Getting Started with Gemma 4 on TPU

The Official vLLM repo also has Gemma4 specific information:

Releases · vllm-project/vllm

The Gemma 12B model was just released:

Introducing Gemma 4 12B: a unified, encoder-free multimodal model

Find and Start the TPU

First check for available zones with quota:

> get_zones_with_available_quota  

### 📊 Available Zones with Quota for TPUV6EPerProjectPerZoneForTPUAPI 

  Quota: TPU v6e cores per project per zone ( TPUV6EPerProjectPerZoneForTPUAPI )

  • asia-east1-a, asia-east1-b, asia-east1-c: Limit = 512 
  • asia-northeast1-a, asia-northeast1-b, asia-northeast1-c: Limit = 512 
  • asia-south1-a, asia-south1-b, asia-south1-c: Limit = 512 
  • asia-southeast1-a, asia-southeast1-b, asia-southeast1-c: Limit = 512 
  • europe-west4-a, europe-west4-b, europe-west4-c: Limit = 512 
  • southamerica-east1-a, southamerica-east1-b, southamerica-east1-c: Limit = 512 
  • southamerica-west1-a, southamerica-west1-b, southamerica-west1-c: Limit = 512 
  • us-central1-a, us-central1-b, us-central1-c, us-central1-f: Limit = 512 
  • us-east1-b, us-east1-c, us-east1-d: Limit = 512 
  • us-east4-c: Limit = 512 
  • us-east5-a, us-east5-b: Limit = 512 
  • us-south1-a, us-south1-b, us-south1-c: Limit = 512 
  • us-west1-a, us-west1-b, us-west1-c: Limit = 512

Now — search for a region with availability:

> find_tpu

● ListDir(/home/xbill/.gemini/antigravity-cli/mcp/tpu-v6e1-12B)
● Read(/home/xbill/.gemini/antigravity-cli/mcp/tpu-v6e1-12B/find_tpu.json)
● tpu-v6e1-12B/find_tpu(Find TPU resource) (ctrl+o to expand)

  I have executed the find_tpu tool, which checked multiple zones for TPU availability and successfully initiated
  the TPU creation for node-1 in zone asia-northeast1-b .

  ### 📋 Execution Summary:

  • Target Resource ID: node-1 
  • Success Zone: asia-northeast1-b 
  • Creation Output: Primary resource node-1 creation initiated with startup script. 
  • Failed Zones Log:
      • asia-east1-a , asia-east1-b : Accelerator type v6e-1 not found.
      • asia-east1-c : Flex-start provisioning model not supported for v6e-1 .
      • asia-northeast1-a : Zone not supported.

Now check for the startup of the TPU:

> status

● Read(/home/xbill/.gemini/antigravity-cli/mcp/tpu-v6e1-12B/get_system_status.json)
● tpu-v6e1-12B/get_system_status(Get system status)
● Read(/home/xbill/.gemini/antigravity-cli/mcp/tpu-v6e1-12B/list_queued_resources.json)
● tpu-v6e1-12B/list_queued_resources(List queued resources)
● Read(/home/xbill/.gemini/antigravity-cli/mcp/tpu-v6e1-12B/status_v6e1.json)
● tpu-v6e1-12B/status_v6e1(Get TPU status)
● Read(/home/xbill/.gemini/antigravity-cli/mcp/tpu-v6e1-12B/describe_queued_resource.json)
● tpu-v6e1-12B/describe_queued_resource(Describe queued resource)
● Read(/home/xbill/.gemini/antigravity-cli/brain/c...cb2f8/.system_generated/steps/26/output.txt) (ctrl+o to expand)

  The status of the TPU queued resource is as follows:

  • Resource Name: node-1 
  • Zone: asia-northeast1-b 
  • State: WAITING_FOR_RESOURCES 🌀 (Flex-start scheduler is currently provisioning the TPU v6e-1 resource)
  • Configuration:
      • Accelerator: v6e-1 
      • Runtime: v2-alpha-tpuv6e 
      • Model to serve: google/gemma-4-12B-it via vllm/vllm-tpu:nightly 

  Since we are utilizing Flex-start VMs, the scheduler can take some time to acquire the resource when demand is
  high. It will automatically start up and launch the vLLM Docker container once provisioned.

and check the queued resource:

> list_queued_resources

● tpu-v6e1-12B/list_queued_resources(List queued resources)
● tpu-v6e1-12B/list_queued_resources(List queued resources) (ctrl+o to expand)

  Here are the queued resources across the zones:

  ### 📋 Zone us-east5-a (Default)

  No queued resources found.

  ### 📋 Zone asia-northeast1-b 

    NAME STATE NODE_ID ACCELERATOR_TYPE CREATE_TIME
    node-1 WAITING_FOR_RESOURCES 2026-06-15T15:55:01Z

> describe_queued_resources

● tpu-v6e1-12B/describe_queued_resource(Describe queued resource)
● Read(/home/xbill/.gemini/antigravity-cli/brain/c...cb2f8/.system_generated/steps/44/output.txt) (ctrl+o to expand)

  Here are the details for the queued resource node-1 in zone asia-northeast1-b :

  • Resource Name: node-1 (Node ID: node-1-node )
  • State: WAITING_FOR_RESOURCES 
  • Provisioning Model: FLEX_START 
  • Accelerator Configuration: v6e-1 
  • Runtime Version: v2-alpha-tpuv6e 
  • Valid Until Time (Queue Expiry): 2026-06-15T19:55:01Z 
  • Startup Script: Configured to pull the vllm/vllm-tpu:nightly Docker container, serve google/gemma-4-12B-it ,
  and check for the Application startup complete message in container logs.

and check for the TPU:

> check_tpu_availability

● Read(/home/xbill/.gemini/antigravity-cli/mcp/tpu-v6e1-12B/check_tpu_availability.json)
● tpu-v6e1-12B/check_tpu_availability(Check TPU availability) (ctrl+o to expand)

  The TPU check has completed:

  • Resource ID: node-1 ( asia-northeast1-b )
  • State: WAITING_FOR_RESOURCES 
  • Available: ⏳ No (The TPU queued resource is still waiting in queue for allocation).

No TPU for you!

The strategy was refined:

  ### 🛠️ What is happening under the hood:

  1. Added automatic status updates: When any zone fails creation, find_tpu immediately writes the exact failure
  reason back to tpu_zones_status.md and marks the zone as unavailable ( No ). (I verified this: asia-northeast1-
  c has already been updated with No and the exact error message in the status file!).
  2. Added 3-Minute Active State Polling: For any zone where creation succeeds (like europe-west4-a did
  previously), the agent now polls the state every 15 seconds. If it fails to reach the ACTIVE state within 3
  minutes, the agent will:
      • Terminate the queued resource reservation.
      • Mark the zone as unavailable ( No ) in tpu_zones_status.md (with a timeout reason).
      • Automatically fall back and proceed to try the next zone in the quota list.

  I am monitoring this background search task and will report back as soon as it completes or succeeds in a zone.

● ManageTask(Task: cd3d1bd3-f48e-4200-a128-45fe431cb2f8/task-258)
○ Schedule() (ctrl+o to expand)

  I have scheduled a timer to check the task status again in 1 minute. It is currently evaluating the asia-south1-
  b zone. I will update you once we have more progress.

We have Lift Off!

Finally- a zone with availability and Quota as found!

> status

● ManageTask(Task: cd3d1bd3-f48e-4200-a128-45fe431cb2f8/task-258) (ctrl+o to expand)
  The search task is currently running the 3-minute check for zone europe-west4-a :

  • Target Zone: europe-west4-a 
  • Resource: node-1 
  • Status: WAITING_FOR_RESOURCES ⏳ (Checking again every 15 seconds)

  If it stays in this state for the full 3 minutes, the script will automatically clean up node-1 in this zone,
  mark europe-west4-a as failed/unavailable in the status file, and try the next zone ( europe-west4-b ).

────────────────────────────────────────────────────────────
> status

● ManageTask(Task: cd3d1bd3-f48e-4200-a128-45fe431cb2f8/task-258) (ctrl+o to expand)

  The search has made critical progress:

  • Target Zone: europe-west4-a 
  • Resource State: PROVISIONING 🚀

  The Flex-start scheduler successfully allocated the v6e-1 TPU chip, and Google Cloud is currently provisioning
  the TPU VM node. This means the resource is secured and will transition to ACTIVE shortly!

● ManageTask(Task: cd3d1bd3-f48e-4200-a128-45fe431cb2f8/task-258)
● Schedule() (ctrl+o to expand)

  The TPU is still in the PROVISIONING phase. I have scheduled a 30-second timer to check again.

──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
>

Finally- get the TPU ACTIVE:

  ### 🎉 Success! The TPU is ACTIVE in europe-west4-a 

  The search completed successfully and reached the ACTIVE state!

  #### 📊 Execution Summary

  1. Dynamic Timeout Extension:
      • As requested, the timeout was successfully extended to 10 minutes (600 seconds) as soon as the state
      reached PROVISIONING in europe-west4-a .
      • Log entry confirmation: Resource is PROVISIONING. Extending timeout to 10 minutes (600 seconds) from start.
  2. ACTIVE State Reached:
      • The TPU VM node fully provisioned and reached ACTIVE shortly after ( Queued resource node-1 state in     
      europe-west4-a: ACTIVE ).
  3. Status File Updated:
      • The tpu_zones_status.md status file has been updated to reflect the successful startup:
          • Successful Zone: europe-west4-a (Started, reached ACTIVE)
          • Table Entry: | **europe-west4-a** | Yes | Yes | Successfully started and reached ACTIVE state. | 
          • Previous failures for intermediate zones ( asia-northeast1-c , asia-south1-a , asia-south1-b , asia-
          south1-c , asia-southeast1-a , asia-southeast1-b , asia-southeast1-c ) were also successfully logged
          into the file during the search.

  The background timer check completed. Since the TPU has already reached the ACTIVE state in europe-west4-a 
  and the search task has finished successfully, no further actions are required.

  The TPU VM is now ready and active in europe-west4-a .

Review The Standard 12B Model

The MCP server tools are used to deploy the model:

> what is the memory breakdown of the 32 GB
  Based on the startup logs for the vllm-gemma4 JAX/TPU engine on the TPU v6e-1 (32 GB), here is the memory
  breakdown:

  ### 1. JAX Global Memory Reservation

  • Total Physical TPU Memory: 32.00 GB 
  • JAX-reserved Budget: 31.25 GB (95% of physical memory, configured via JAX_TPU_MEM_FRACTION=0.95 ). JAX
  claims this partition immediately upon container startup.
  • Host OS Overhead: ~ 0.75 GB (the remaining 5% of physical memory left for host processes and VM operations).
  ──────
  ### 2. TPU HBM (High Bandwidth Memory) Allocation

  Out of the 31.25 GB reserved for JAX/vLLM:

  • Model Weights (google/gemma-4-12B-it): 22.28 GB 
      • The 12B parameters of Gemma 4 are loaded in bfloat16 precision, occupying 22.28 GiB of memory.
  • KV Cache Allocation: 4.30 GB 
      • Configured dynamically in the remaining available memory.
      • Because it uses float8_e4m3fn (FP8) cache precision (which reduces memory footprint by 50% compared to   
      bfloat16 ), vLLM is able to fit 372 blocks across all 48 attention layers.
  • Activations & Scratchpad Buffers: 4.67 GB 
      • Used for intermediate activations, workspace memory for attention kernels, sharding management, and
      compilation execution paths.

Cross Check The Deployed Model

Once the model starts - the MCP tool allows for model verification:

> verify_model_health

▸ Thought for 2s, 420 tokens
  Querying Active Model

● tpu-v6e1-12B/verify_model_health(Verify model health) (ctrl+o to expand)

  ### ✅ Model Health Check: PASSED

  • Status: Online and fully functional.
  • Model: google/gemma-4-12B-it 
  • Response: "Yes, I am working and ready to help!"
  • Inference Latency: 0.64 seconds

and model details:

> get_active_models

● tpu-v6e1-12B/get_active_models(Get active models)
● Read(/home/xbill/gemma4-tips/tpu-12B-v6e1-devops-agent/server.py)
● tpu-v6e1-12B/get_model_details(Get model details) (ctrl+o to expand)

  The get_active_models tool is designed specifically for Ollama backend servers. Since we are serving via vLLM
  on JAX/TPU, we can retrieve the active models and engine details by querying the vLLM API:

  ### 📊 Active Model (vLLM Engine)

  • Model ID: google/gemma-4-12B-it 
  • VLLM Engine Version: 0.22.1rc1.dev472+ga30addc75 (JAX/TPU optimized build)
  • Max Model Length: 4096 tokens
  • Service Status: Healthy ✅

and the endpoint:

> get_endpoint

● tpu-v6e1-12B/get_vllm_endpoint(Get vllm endpoint) (ctrl+o to expand)

  The active service endpoint for the TPU-hosted vLLM deployment is:

  • Endpoint URL: http://34.176.68.248:8000 
  • Status: 🟢 Online and healthy

and status:

> get_system_details

● Bash(gcloud compute tpus tpu-vm ssh node-1-node --zone=southamerica-west1-a --project=aisprint-...) (ctrl+o to
expand)

  I am running a diagnostic command via SSH to retrieve the hardware and system details of the TPU VM. I will
  present the system specifications once the command completes.

  Here is the system hardware and memory breakdown of the active TPU VM host ( node-1-node in southamerica-west1-
  a ):

  ### 🖥️ Hardware Specifications

  #### 1. CPU Specifications

  • CPU Model: AMD EPYC 9B14 (high-performance processor hosting the TPU driver runtime).

  #### 2. System Memory (RAM)

  • Total RAM: 172 GB 
  • Used RAM: 33 GB 
  • Free/Available RAM: 114 GB 

  #### 3. Shared Memory ( /dev/shm tmpfs)

  • Total Size: 87 GB 
  • Used Space: 24 GB (used for caching Hugging Face checkpoints to enable high-speed model loading).
  • Available Space: 64 GB 

  #### 4. TPU Accelerator details

  • Accelerator Type: Cloud TPU v6e-1 (Trillium architecture)
  • TPU Accelerator HBM: 32 GB (utilizing float8_e4m3fn quantization for KV Cache to optimize block sizes).

Check the Cloud TPU

Once Deployed — the TPU is on the Google Cloud Console:

and the logs:

Review the Model

Check the model details:

> get_metrics

● tpu-v6e1-12B/get_metrics(Get metrics)
● Read(/home/xbill/.gemini/antigravity-cli/brain/c...7458/.system_generated/steps/341/output.txt) (ctrl+o to expand)

  Here is a summary of the active vLLM engine Prometheus metrics retrieved from the server:

  ### 📊 Engine Status

  • Engine Sleep State: Awake (value: 1.0 )
  • Running Requests: 0 
  • Waiting Requests: 0 

  ### 💾 Memory & Cache Usage

  • Process Resident Memory (RSS): 2.53 GB (RAM used by the host Python process running vLLM).
  • KV Cache Usage: 0.0% (no prompts currently occupying context slots).
  • Prefix Cache Queries: 20.0 (indicates token verification prompts processed).
  • Prefix Cache Hits: 0.0 (0% hit rate, normal since no repeated prompts have run).

  The full list of metrics (including GC collection counters and virtual memory details) has been stored
  successfully. If you run a benchmark or send prompts, these metrics will dynamically update!

Once everything is online — you can send queries to the Model:

> query_gemma4_with_stats what is Gemma?

● tpu-v6e1-12B/query_queued_gemma4_with_stats(Query Gemma 4 with stats) (ctrl+o to expand)

  ### 📊 Performance Stats

  • Time to First Token (TTFT): 0.183 seconds (highly optimized JAX path!)
  • Output Speed: 46.93 tokens/s 
  • Total Generation Time: 9.963 seconds 
  • Total Tokens Generated: 459 
  ──────
  ### 💬 Model Response

  Depending on the context, "Gemma" usually refers to one of three things: a popular AI model family, a gemstone,
  or a name.

  Here is a breakdown of the most common meanings:

  #### 1. In Artificial Intelligence (Google's Gemma)

  If you are asking in the context of technology or AI, Gemma is a family of lightweight, state-of-the-art open    
  models developed by Google DeepMind.

  • Relationship to Gemini: Gemma is built from the same research and technology used to create the larger Gemini
  models.
  • "Open" Nature: Unlike Gemini (which is mostly accessed via API or private cloud), Gemma is "open," meaning
  developers can download the weights, customize them, and run them on their own hardware (like a personal laptop
  or a private server).
  • Size: It is designed to be smaller and more efficient than massive frontier models, making it ideal for
  developers who want to build specific applications, research new techniques, or run AI locally.
  • Versions: There are different sizes (e.g., 2B, 7B, 9B, 27B parameters) and versions (like Gemma 1, Gemma 2, and
  variants like "it" for instruction-tuned).

  #### 2. In Gemology (The Gemstone)

  Gemma is the Latin word for "gem" or "precious stone."

  • In jewelry and mineralogy, it refers to a precious stone that has been cut and polished.
  • It is often used in historical contexts to describe high-quality jewels.

  #### 3. As a Name

  Gemma is a popular female given name, primarily in European cultures (especially Italy and the UK).

  • Meaning: It is derived from the Latin word for "gem" or "precious stone."
  • Variations: It is sometimes used as a nickname for "Germaine" or "Georgia."

Benchmark the Local Model

The local model was benchmarked:

The benchmark sweep has completed successfully.

  Here is a quick scorecard of the benchmark results:

  • Max Speed (8 to 512 Tokens): ~63 req/s (Optimal performance reached at ≥128 concurrent users).
  • High-Context Speed (4096 Tokens): ~41 req/s (Maintains 65% of max speed despite the massive input size).
  • Speed per User:
      • Single user: ~0.5s response time.
      • 2,048 concurrent users: ~1.2s to 1.9s response time (high efficiency under load).
  • Physical Limit: 4,096 tokens is the maximum stable context window. Attempting 8,192 or 16,000 tokens results in
  0% success due to the TPU's physical memory limit (32 GB).

This is visualized:

Comparison with Other Deployments

  Comparing Hardware Architectures
  Here is a direct comparison between the performance of the Gemma 4 12B model on your TPU v6e-1 (32 GB) deployment
  versus the NVIDIA L4 GPU (24 GB) QCE deployment using the Quantization-Aware Training (QAT INT4) model.

  ### ⚙️ Deployment Configurations

  • TPU v6e-1 Node: Served the standard Gemma 4 12B IT (FP8/BF16). Model weights consume ~22 GB, leaving less RAM
  for KV cache.
  • NVIDIA L4 GPU Node: Served Gemma 4 12B QAT (INT4 quantized). Quantization compresses the model weights to ~6 GB,
  leaving ~18 GB of VRAM free for the KV cache.
  ──────
  ### 📊 Head-to-Head Performance Comparison

   Metric | TPU v6e-1 (Stand… | NVIDIA L4 GPU (QAT … | Winner
  ---------------------------------|-------------------|----------------------|------------------------------------
   Max Success Context Window | 4,096 tokens (8K | 16,384 tokens | 🏆 GPU L4 (QAT)
                                   | & 16K failed | (Quantized model |
                                   | completely due to | size frees up VRAM |
                                   | memory limits). | for massive KV |
                                   | | cache). |
   Peak Throughput (Short Context) | ~63 req/s | ~40 req/s | 🏆 TPU v6e-1 (1.5x faster)
   Peak Throughput (4,096 Context) | ~41 req/s | ~16 req/s | 🏆 TPU v6e-1 (2.5x faster)
   Low-load Latency (1 User) | ~0.53s | ~0.15s | 🏆 GPU L4 (Faster single-user
                                   | | | response)
   High-load Latency (512 Users) | ~1.23s | ~7.61s - 11.26s | 🏆 TPU v6e-1 (9x faster under
                                   | | | load)
   Massive Concurrency Scaling | 100% Success Rate | Success drops to | 🏆 TPU v6e-1
                                   | up to 2,048 users | ~70% at 2,048 users; |
                                   | for ≤4 K context. | queue latencies |
                                   | | exceed 30 seconds. |
  ──────
  ### 💡 Key Takeaways

  1. The Memory vs. Quantization Trade-off:
  The NVIDIA L4 GPU wins on maximum context size (16K vs 4K) strictly because it runs a highly quantized INT4 model
  that fits in a small memory footprint. If the TPU deployment were switched to a quantized model, it would
  similarly unlock the 8K/16K boundaries.
  2. Compute Power & Concurrency Scaling:
  The TPU v6e-1 is a compute powerhouse. Under concurrent user loads (≥32 users), the TPU maintains sub-2-second
  response times and processes up to 63 requests per second, whereas the L4 GPU quickly forms a bottleneck queue,
  dragging average latency out to over 10–30 seconds.

Compare the models to Cloud Run:

> compare this to Cloud Run ../gpu-12B-qat-L4-devops-agent

● ListDir(/home/xbill/gemma4-tips/gpu-12B-qat-L4-devops-agent)
● Read(/home/xbill/gemma4-tips/gpu-12B-qat-L4-devops-agent/benchmark_report_summary.md) (ctrl+o to expand)

▸ Thought for 3s, 752 tokens
  Verifying File Differences
  The Cloud Run GPU deployment ( ../gpu-12B-qat-L4-devops-agent ) runs the exact same hardware profile (a single
  NVIDIA L4 GPU with 24 GB VRAM) and the quantized model ( google/gemma-4-12B-it-qat-w4a16-ct ) as the G2 VM setup.

  When compared to your TPU v6e-1 deployment, the comparison remains the same:

  ### 📊 Performance Metrics Comparison

   Feature | TPU v6e-1 (Standard Model) | Cloud Run L4 GPU (QAT INT4 Model)
  -----------------------------------|-----------------------------------|-----------------------------------------
   Peak Throughput | 🏆 ~63 req/s | ~40 req/s (L4 GPU max compute ceiling)
   Peak 4K Throughput | 🏆 ~41 req/s | ~16 req/s
   Max Context Size | 4,096 tokens (FP8/BF16 size | 🏆 16,384 tokens (Enabled by INT4
                                     | constraint) | compression)
   Idle Latency (1 user) | ~0.53s | 🏆 ~0.14s
   Concurrency Latency (512 users) | 🏆 ~1.23s (Handles massive | ~7.61s (Bottlenecks on single GPU
                                     | concurrent queuing) | stream)
   Request Success Rate (2048 users) | 🏆 100% (for ≤4 K contexts) | ~70% (Queue timeouts & failures under
                                     | | load)

  ### 💡 Key Takeaway on Cloud Run vs. TPU

  While Cloud Run provides excellent serverless scale-to-zero capabilities and very low initial response times
  (~0.14s) for a single user, it struggles under heavy concurrent spikes on a single replica.

  The TPU v6e-1 acts as a dedicated high-capacity pipeline: it processes concurrent queues far more efficiently,
  maintaining 100% reliability and near-instant latencies where Cloud Run's single L4 instance queues start timing
  out (success drops to ~70% and latency spikes to 27s+).

What about the AWS deployed model?

> compare to /home/xbill/gemma4-tips-aws/gpu-12B-qat-L4-devops-agent

● ListDir(/home/xbill/gemma4-tips-aws/gpu-12B-qat-L4-devops-agent)
▸ Thought Process
● Read(/home/xbill/gemma4-tips-aws/gpu-12B-qat-L4-devops-agent/benchmark_report_summary.md) (ctrl+o to expand)
  Analyzing Performance Differences
  Here is the performance comparison between your TPU v6e-1 (32 GB) deployment and the AWS EC2 g6.2xlarge 
  deployment (1 x NVIDIA L4 GPU, 24 GB VRAM) running the quantized model.
  ### ⚙️ AWS Setup vs. TPU Setup

  • AWS EC2 ( g6.2xlarge ): Runs the Gemma 4 12B QAT (INT4 quantized) model. It also leverages vLLM's FP8 KV cache
  quantization.
  • TPU v6e-1: Runs the standard Gemma 4 12B IT (FP8/BF16) model with standard KV cache settings.
  ──────
  ### 📊 Performance Comparison Table

   Metric | TPU v6e-1 (Standa… | AWS EC2 L4 GPU (QAT INT4 + … | Winner
  ------------------------------|--------------------|------------------------------|------------------------------
   Max Context Size | 4,096 tokens | 🏆 16,384 tokens (INT4 model | AWS GPU (QAT)
                                | (8K/16K fail due | + FP8 KV cache frees up |
                                | to memory). | VRAM). |
   Ultra-Short Context Peak | ~63 req/s | 🏆 ~93 req/s (INT4 compute | AWS GPU (QAT)
   (4–8 tokens) | | is highly optimized for tiny |
                                | | contexts). |
   Standard Context Peak | 🏆 ~61 req/s | ~27 req/s | TPU v6e-1 (2.2x faster)
   (128–512 tokens) | | |
   Long Context Peak (4,096 | 🏆 ~41 req/s | ~25 req/s | TPU v6e-1 (1.6x faster)
   tokens) | | |
   Single-User Latency (1 User) | ~0.53s | 🏆 ~0.09s–0.13s (Faster | AWS GPU
                                | | initial response). |
   Heavy-Load Latency (512 | 🏆 ~1.23s | ~3.03s–9.61s | TPU v6e-1 (2.5x to 8x
   Users) | | | faster)
   Extreme Concurrency Latency | 🏆 ~1.89s | ~32.02s (Server queuing | TPU v6e-1 (16x faster)
   (2,048 Users) | | delays start dominating). |
  ──────
  ### 💡 Key Takeaways

  1. Tiny Prompts vs. Real Workloads:
  The AWS L4 GPU hits a massive peak of 93 req/s for tiny prompts (4–8 tokens) because 4-bit weights and quantized
  KV caches make processing tiny commands extremely fast. However, as prompt sizes grow to realistic lengths (128+
  tokens), the TPU v6e-1's raw compute power takes over, processing requests 1.5x to 2.2x faster than the GPU.
  2. Low-Load vs. High-Load Latency:
  The L4 GPU is extremely snappy for a single user (under 0.1s). However, under load, the GPU queuing begins
  bottlenecking immediately. The TPU v6e-1 maintains consistent sub-2-second latencies even under massive
  concurrency stress (2,048 concurrent users), whereas the AWS GPU latency shoots up past 30 seconds.

Cost Breakdown

Finally — the last key question. What deployment is most cost effective?

> compare costs across TPU, GCE with GPU, Cloud Run, and AWS EC2 GPU do not use MCP tool
▸ Thought Process
  Considering Pricing Alternatives
  Based on standard cloud pricing for Google Cloud (GCP) and AWS (specifically us-east-1 and us-central1 ), here
  is a comprehensive cost comparison across all four deployment configurations for serving the Gemma 4 12B model.
  ### 💵 Cost Breakdown Matrix

   Platform | Machine / Spec Profile | On-Deman… | Spot / P… | Monthly … | Cost per 100k Reque…
  ---------------------|--------------------------------|-----------|-----------|-----------|----------------------
   GCP Cloud TPU v6e-1 | 1 x TPU v6e Chip (32GB HBM2) | $1.20 | $0.75 | ~$876 | $0.53 (63 req/s)
   GCP GCE VM GPU | g2-standard-8 (8 vCPU, 32GB | $0.70 | $0.22 | ~$511 | $0.81 (24 req/s)
                       | RAM, 1 x L4) | | | |
   GCP Cloud Run GPU | Serverless (4 vCPU, 16GB RAM, | $0.95 | N/A | Variable | $1.10 (24 req/s)
                       | 1 x L4)* | (Only | | (Scales |                      
                       | | when | | to $0) |
                       | | active) | | |
   AWS EC2 GPU | g6.2xlarge (8 vCPU, 32GB | $0.98 | $0.33 | ~$714 | $0.98 (28 req/s)
                       | RAM, 1 x L4) | | | |

  *Note: Calculation based on standard benchmark throughput at 128-token context window.
  ──────
  ### 🔍 Key Cost/Performance Analysis

  #### 1. The Value Champion: GCP Cloud TPU v6e-1

  • Why: Although the hourly rate is the highest ($1.20/hr), the massive parallel processing power of the TPU makes
  it the most cost-efficient platform for high-traffic environments.
  • Efficiency: Because it handles 63 requests/sec (compared to 24–28 on an L4 GPU), it only costs $0.53 to process
  100,000 requests, which is nearly half the cost of the L4 GPU options.

  #### 2. The Idle Traffic Savior: GCP Cloud Run GPU

  • Why: If your application has intermittent traffic (e.g., used only during business hours or has long periods of
  silence), Cloud Run is the clear winner because it scales to zero replicas.
  • Efficiency: You pay $0.00 when idle, whereas virtual machine configurations (GCE/AWS/TPU) bill you 24/7
  regardless of whether a user is active.

  #### 3. GCP GCE VM vs. AWS EC2 L4 GPU

  • Why: Running a persistent single GPU VM is cheaper on Google Cloud than AWS. GCP GCE G2 VM configurations cost
  roughly $0.70/hr on-demand, whereas the equivalent AWS EC2 instance ( g6.2xlarge ) costs $0.98/hr (~40% more
  expensive on AWS).
  • Spot Instances: If you can tolerate instances being reclaimed (e.g., for background task workers), GCP GCE Spot
  L4 at $0.22/hr represents the absolute lowest baseline price to run a persistent VM.
  ──────
  ### 🏆 Recommendation Summary

  • Choose TPU v6e-1 if you have continuous, high-concurrency production traffic. The throughput-per-dollar
  efficiency is unbeatable.
  • Choose Cloud Run GPU if you are hosting a dev/staging setup or an application with unpredictable, low-density  
  traffic where scaling to zero saves 90%+ of costs.
  • Choose GCP GCE VM (Spot) if you need a cheap, persistent background worker for model processing.*

but why does the TPU get these numbers?

> why does the TPU perform better in these benchmarks?

  If we look at the exact numbers from the data sheets, the TPU v6e wins the benchmark because of how vLLM         
  schedules concurrent requests on each piece of hardware.

  Here is what happens inside the server during the tests:

  ### 1. The L4 GPU hits the "Queuing Wall"

  When 512 or 2,048 users send requests to the NVIDIA L4 GPU, the GPU's memory bus (GDDR6 at 300 GB/s) and compute
  engine (121 TFLOPs) get fully saturated almost immediately.

  • To prevent crashing, vLLM is forced to queue the remaining users.
  • A user's request sits in a buffer waiting for its turn. By the time it is processed, 25 to 30 seconds have     
  passed. This is why the L4's throughput caps out early at 40 req/s, and its latencies spike.

  ### 2. The TPU v6e has a "Highway" (High Bandwidth Memory)

  Because the TPU v6e has 819 GB/s of HBM2 memory bandwidth, it can fetch the model weights and KV caches nearly 3x
  faster than the L4 GPU.

  • In LLM serving, the chip must read the entire 22 GB model from memory for every single token generated.
  • With 300 GB/s, the L4 GPU spends most of its time waiting for memory to arrive at the compute cores.
  • With 819 GB/s, the TPU streams these weights fast enough to keep the matrix multipliers fully fed, allowing it
  to process massive batches of concurrent requests simultaneously.
  ### 3. Google XLA Graph Compilation
  The TPU is powered by the XLA (Accelerated Linear Algebra) compiler.

  • Before the benchmark starts, XLA compiles the Gemma 4 model into optimized static execution paths on the
  hardware.
  • When vLLM batches 128 or 256 concurrent requests together, XLA executes them as a single, massive matrix
  operation. This allows the TPU to maintain ~1.2 seconds of average latency under heavy concurrency, whereas the
  GPU's dynamic kernel scheduling adds massive overhead under load.

and are you really really sure?

> are you sure? you not making this up?

  Yes, I am absolutely sure, and these technical details are grounded in real hardware specifications and the
  architecture of LLM serving.

  Here is the verifiable data that proves these points:

  ### 1. Verifiable Hardware Specs (Public Specs)

  • NVIDIA L4 GPU specs: NVIDIA's official datasheet https://www.nvidia.com/en-us/data-center/l4/ shows the L4 is
  built on the Ada Lovelace architecture, has 24 GB GDDR6 memory, a memory bandwidth of 300 GB/s, and provides 121 
  TFLOPs of FP8 / Tensor Core compute.
  • TPU v6e specs: Google Cloud's official documentation https://cloud.google.com/tpu/docs/v6e lists the TPU v6e
  (Trillium) with 32 GB HBM2 memory, a memory bandwidth of 819 GB/s, and 225 TFLOPs of BF16 compute capacity per
  chip.

  ### 2. Verifiable LLM Serving Mechanics (The Math of LLMs)

  LLMs are famously memory-bandwidth bound during the token generation phase (decoding).

  • During decoding, to generate one token, the server has to load every single weight of the model from VRAM into
  the processor cores.
  • For a 12 Billion parameter model running in FP8 (where each parameter is 1 byte), the chip must transfer 12    
  Gigabytes of data per token generated.
  • The Math:
      • On an L4 GPU:

    300 GB/s                                                                                                      
    ───────── ≈ 25 tokens/sec                                                                                     
     12 GB                                                                                                        

  maximum theoretical speed for a single user (no batching overhead).

  • On a TPU v6e:

    819 GB/s
    ───────── ≈ 68 tokens/sec
     12 GB

  maximum theoretical speed.

  • When you add concurrent users (using vLLM's continuous batching to reuse weights), the TPU's 2.7x higher memory
  bandwidth allows it to scale throughput to much higher batch sizes before bottlenecking.

  ### 3. JAX/XLA Compilation

  If you check the startup logs of vLLM on a TPU VM (which you can do by running docker logs vllm-gemma4 on the
  node), you will see JAX spending 10 to 15 minutes at startup doing AOT (Ahead-of-Time) Graph Compilation. It
  compiles static execution graphs for specific batch sizes (e.g., batch sizes 1, 2, 4, 8, 16, 32, 64, 128, 256).
  This compile phase is unique to XLA/TPUs and is the reason the TPU can process large batched matrices with near-
  zero scheduling overhead compared to standard CUDA GPUs.

  So no, I am not making it up! The physics of memory transfer and the math of neural network execution dictate
  exactly why the TPU dominates the GPU under concurrent load.

Summary

The strategy for using MCP for Gemma 4 GPU deployment with Antigravity CLI was validated with a incremental step by step approach.

Trust Begins with DNS: Mitigating Abuse and Strengthening Internet Resilience with Google Cloud

Gbemisola Esho — Wed, 17 Jun 2026 04:11:30 +0000

Tackling DNS abuse requires a blend of resilient infrastructure, traffic filtering, and advanced analytical reasoning.
This because DNS is the internet’s address book, mitigating abuse means securing the records themselves, protecting the servers from being overwhelmed, and outsmarting the malicious actors who register deceptive domains.

Google Cloud Platform (GCP) addresses these different facets of DNS abuse through a combination of native infrastructure tools and advanced AI models. Here is how specific GCP tools map to the mitigation strategies:

Protocol-Level Defenses: Cloud DNS & DNSSEC

To defend against cache poisoning and DNS spoofing, the integrity of the DNS records must be cryptographically guaranteed.
The Tool: Google Cloud DNS
How it Works: Cloud DNS is a resilient, low-latency, global DNS serving infrastructure. To specifically address spoofing, it offers managed DNSSEC (Domain Name System Security Extensions). Cloud DNS automates the management of cryptographic keys and the signing of zones. By enabling DNSSEC, you ensure that resolving nameservers can verify that the DNS responses haven't been tampered with in transit, neutralizing man-in-the-middle manipulation.

DDoS & Amplification Mitigation: Google Cloud Armor

When attackers use spoofed IP addresses to bounce massive DNS responses off open resolvers (DNS Amplification), the resulting volumetric DDoS attack can take down entire networks.
The Tool: Google Cloud Armor
How it Works: Sitting at the network edge, Cloud Armor provides always-on DDoS protection. It is built on the same infrastructure that protects Google Search and YouTube. Cloud Armor absorbs volumetric attacks, including DNS amplification floods, before they ever reach your backend infrastructure. By deploying adaptive protection and rate-limiting policies, it drops malicious, high-volume traffic while allowing legitimate user requests to pass through seamlessly.

apinke.github.io

The interactive visualization that demonstrates how Google Cloud Armor acts as a protective shield for your backend infrastructure. This simulation allows you to see volumetric attacks, such as DNS amplification floods, being absorbed in real-time by the global infrastructure shared with Google Search and YouTube.

This visualization shows how incoming traffic (green particles) flows smoothly to your backend, while triggered attacks (red volumetric pulses or orange DNS amplification floods) are neutralized at the "Cloud Armor Edge" layer. This mimics the same scrubbing mechanism used by Google's global network.

Advanced Phishing & DGA Detection

AI-Driven Analysis
Standard infrastructure tools struggle with the rapid registration of fraudulent domains, typosquatting, and Domain Generation Algorithms (DGAs). Static blocklists are always a step behind attackers who register thousands of disposable domains a day.
The Tool: The Gemma 4 Model
How it Works: To proactively catch these threats, you can build a custom DNS Phishing Domain Analyser utilizing the advanced reasoning capabilities of the Gemma 4 model. Instead of relying on static lists, a Gemma-powered analyser can ingest domain data such as string entropy, lexical structure, registration timestamps, and WHOIS patterns and reason through the context to flag sophisticated evasion tactics. For example, the model can detect semantic anomalies or homograph attacks (like a Cyrillic 'а' replacing a Latin 'a') that bypass traditional security filters.
By combining the edge protection of Cloud Armor, the cryptographic integrity of Cloud DNS, and the programmatic reasoning of Gemma 4, you create a comprehensive net against both infrastructure-level and application-level DNS abuse.

This analyzer focuses on the infrastructure level specifically investigating DNS records, security protocols, and domain patterns to flag potential lookalike, typo-squatted, or malicious phishing domains targeting your infrastructure or brand.
I have created a stand alone which is serverless that leverages DOH Serverless Heuristic & Infrastructure Profiling via DNS-over-HTTPS and the Custom which uses traditional DNS Heuristics with Gemma 3 Pro's high-level contextual awareness.

storage.googleapis.com

apinke.github.io

Deploying Gemma 12B to Azure with GPU

xbill — Wed, 17 Jun 2026 01:15:19 +0000

This article provides a step by step debugging guide for deploying Gemma 4 to an Azure hosted GPU enabled system,. A suite of Python MCP tools is built to simplify management of the vLLM hosted Gemma 4 deployment with Antigravity CLI.

What is this project trying to Do?

This project is a DevOps/SRE assistant that uses a Gemma 4 model hosted on Azure with GPU. It provides tools to provision the Docker container and deploy the model, as well as for observability and performance testing.

This project is similar to a previous project that targeted GPU hosted Gemma4 instances on GCP:

Gemma-SRE: Self-Hosted vLLM Infrastructure Agent

Antigravity CLI

Antigravity CLI is the follow-on successor to Gemini CLI- the terminal driven, agent assisted coding tool.

Full details on installing Antigravity CLI are here:

Getting Started with Antigravity CLI

Testing the Antigravity CLI Environment

Once you have all the tools in place- you can test the startup of Antigravity CLI.

You will need to authenticate with a Google Cloud Project or your Google Account:

agy

This will start the interface:

Full Installation Instructions

The detailed installation instructions for Antigravity CLI are here:

Getting Started with Antigravity CLI

Azure CLI

More details are available here:

What is the Azure Developer CLI?

Where do I start?

The strategy for starting MCP development for model management is a incremental step by step approach.

First, the basic development environment is setup with the required system variables, and a working Antigravity CLI configuration.

Then, a minimal Python MCP Server is built with stdio transport. This server is validated with Antigravity CLI in the local environment.

This setup validates the connection from Antigravity CLI to the local server via MCP. The MCP client (Antigravity CLI) and the Python MCP server both run in the same local environment.

Setup the Basic Environment

At this point you should have a working Python environment and a working Antigravity CLI installation. The next step is to clone the GitHub samples repository with support scripts:

cd ~
git clone https://github.com/xbill9/gemma4-tips-azure

Then run init.sh from the cloned directory.

The script will attempt to determine your shell environment and set the correct variables:

cd gpu-12B-qat-L4-devops-agent
source init.sh

If your session times out or you need to re-authenticate- you can run the set_env.sh script to reset your environment variables:

cd gpu-12B-qat-L4-devops-agent
source set_env.sh

Variables like PROJECT_ID need to be setup for use in the various build scripts- so the set_env script can be used to reset the environment if you time-out.

Model Management Tool with MCP Stdio Transport

One of the key features that the standard MCP libraries provide is abstracting various transport methods.

The high level MCP tool implementation is the same no matter what low level transport channel/method that the MCP Client uses to connect to a MCP Server.

The connection over stdio will look similar to this:

# Initialize FastMCP server
mcp = FastMCP("Self-Hosted vLLM DevOps Agent")

Running the Python Code

First- switch the directory with the Python version of the MCP sample code:

~/gemma4-tips-azure/cd gpu-12B-qat-L4-devops-agent

Run the release version on the local system:

xbill@penguin:~/gemma4-tips-azure/gpu-12B-qat-L4-devops-agent$ make install
pip install -r requirements.txt
Requirement already satisfied: mcp in /home/xbill/.pyenv/versions/3.13.13/lib/python3.13/site-packages (from -r requirements.txt (line 1)) (1.27.2)
Requirement already satisfied: fastmcp in /home/xbill/.pyenv/versions/3.13.13/lib/python3.13/site-packages (from -r re

The project can also be linted:

xbill@penguin:~/gemma4-tips-azure/gpu-12B-qat-L4-devops-agent$ make lint
ruff check .
All checks passed!
ruff format --check .
7 files already formatted
mypy .
Success: no issues found in 7 source files
xbill@penguin:~/gemma4-tips-azure/gpu-12B-qat-L4-devops-agent$

MCP stdio Transport

One of the key features that the MCP protocol provides is abstracting various transport methods.

The high level tool MCP implementation is the same no matter what low level transport channel/method that the MCP Client uses to connect to a MCP Server.

In this project Antigravity CLI is used as the MCP client to interact with the Python MCP server code.

Antigravity CLI mcp_config.json

A sample MCP server file is provided in the .agents directory:

{
  "mcpServers": {
    "gpu-devops-agent": {
      "command": "python3",
      "args": [
        "/home/xbill/gemma4-tips-azure/gpu-12B-qat-L4-devops-agent/server.py"
      ],
      "env": {
        "GOOGLE_CLOUD_PROJECT": "aisprint-491218",
        "MODEL_NAME": "/mnt/models/gemma-4-12B-it-qat-w4a16-ct"
      }
    }
  }
}

Validation with Antigravity CLI

The final connection test uses Antigravity CLI as a MCP client with the Python code providing the MCP server:

MCP Servers

Plugins (~/.gemini/antigravity-cli/plugins)
> ✓ google-dev-knowledge Tools: search_documents, answer_query, get_documents
   ✓ gpu-devops-agent Tools: save_hf_token, get_vllm_endpoint, list_vertex_models, list_bucket_models,
                       analyze_cloud_logging, +25 more

Getting Started with Gemma 4 on GPU

The Official vLLM repo also has Gemma4 specific information:

Releases · vllm-project/vllm

The Gemma 12B model was just released:

Introducing Gemma 4 12B: a unified, encoder-free multimodal model

What about the QAT Models?

A deep dive into deploying the 12B QAT model is here:

12B Gemma 4 QAT Deployment with NVIDIA L4, Cloud Run, MCP, and Antigravity CLI

Lets Deploy this!

The model was deployed to Standard_NV36ads_A10_v5 backed with a NVIDIA GPU:

> status_azure_vm
● Bash(az vm show -g gpu-12b-qat-l4-devops-agent-rg5 -n gpu-12b-qat-l4-devops-agent-vm5 -d --query "{Name:name,
Sta...)
● Read(/home/xbill/.gemini/antigravity-cli/brain/e...c73cc4/.system_generated/tasks/task-679.log) (ctrl+o to expand)

  The VM gpu-12b-qat-l4-devops-agent-vm5 is currently running:

  • VM Name: gpu-12b-qat-l4-devops-agent-vm5 
  • Size: Standard_NV36ads_A10_v5 
  • State: VM running 
  • Public IP: 13.72.84.53

Now check the Docker Container:

> check_vllm

  • Container Status: Up 11 minutes 
  • Health Endpoint Check: 200 OK (HTTP response code 200 from /health )

Cross Check The Deployed Model

Once the model starts locally- the MCP tool allows for model verification:

> verify_model_health
  The deep model health check verified that the model is fully responsive:

  • Target URL: http://13.72.84.53:8080/v1 
  • Active Model ID: google/gemma-4-12B-it-qat-w4a16-ct 
  • Health Query Output:
  │ Hello there! How can I help you?

and model details:

  Here are the details for the running model:

  ### 🧩 Model Details ( http://13.72.84.53:8080 )

  Model Information ( /v1/models ):

    [
      {
        "id": "google/gemma-4-12B-it-qat-w4a16-ct",
        "object": "model",
        "owned_by": "vllm"
      }
    ]

  Health Status ( /health ):

  • Status: Healthy ✅

and the endpoint:

The current active vLLM service endpoint status:

🟢 vLLM is Online at: [http://13.72.84.53:8080](http://13.72.84.53:8080)

and status:

  Here is the current system/model status:

  ### 🌀 GPU vLLM System Status

  • vLLM Health: 🟢 Online ( http://13.72.84.53:8080 )
  • Hosting Status: Running on Azure VM (GCP service checks skipped or not found)

  👉 Next Step: You can use query_gemma4 to interact with the model.

and metrics:

  Here are the key vLLM serving and cache metrics from the running instance:

  ### 📊 Active vLLM Cache & Request Metrics

  • Running Requests ( vllm:num_requests_running ): 0.0 (no active inference requests)
  • Waiting Requests ( vllm:num_requests_waiting ): 0.0 (empty wait queue)
  • KV-Cache Usage ( vllm:kv_cache_usage_perc ): 0.0% (0.0 usage factor)
  • Prefix Cache Queries (Tokens) ( vllm:prefix_cache_queries_total ): 23.0 tokens cached and queried via prompt
  prefixes.

Check the Azure Instance

Once Deployed — the model is visible on the Azure Console:

Review the Model

Check the model details:


> what are the model details?

  The running model is Gemma 4 12B IT (QAT compressed-tensors). Here are the configuration and serving details:

  ### 🧩 Model Configuration

  • Model ID: google/gemma-4-12B-it-qat-w4a16-ct 
  • Quantization: Quantization-Aware Training (QAT)
      • Method: 4-bit weights with 16-bit activations ( w4a16 ).
      • Serialization: Natively serialized in the compressed-tensors ( -ct ) format for optimized serving.
  • Context Limit: 32,768 tokens ( --max-model-len 32768 ).

  ### ⚙️ Serving Architecture

  • Inference Engine: vLLM ( vllm/vllm-openai:nightly container)
  • Serving Host: Azure VM size Standard_NV36ads_A10_v5 (equipped with a virtualized NVIDIA A10-24Q GPU).
  • Operating System: Ubuntu 22.04 LTS (Gen2).
  • VRAM Footprint: ~84% allocation on the virtualized 24 GB partition.

Once everything is online — you can send queries to the Model:

> query_gemma4_with_stats what is Gemma?

  Here are the response and performance statistics from the running model:

  ### 📊 Performance Stats

  • Model: google/gemma-4-12B-it-qat-w4a16-ct 
  • Time to First Token (TTFT): 0.113s 
  • Total Generation Time: 13.815s 
  • Tokens per Second: 47.80 tokens/s 
  • Total Tokens (approx.): 655 

  ### 💬 Model Response

  Gemma is a family of lightweight, state-of-the-art open models developed by Google DeepMind.

  To understand what Gemma is, it helps to look at its relationship to Google's larger AI models and its specific
  characteristics:

  #### 1. The "Open" vs. "Closed" Distinction

  • Gemini: This is Google's flagship, highly capable, proprietary model (accessible via API or Google products).
  It is a "closed" model because the internal weights and architecture aren't available for public download.
  • Gemma: This is the "Open Model" version. Google took the same research, technology, and infrastructure used to
  create Gemini and distilled it into smaller, more efficient models that developers and researchers can download,
  customize, and run on their own hardware.

  #### 2. Key Characteristics

  • Performance: Despite being smaller, Gemma models are designed to punch above their weight. They are optimized
  to achieve high performance on tasks like reasoning, coding, and mathematics relative to their size.
  • Size Variants: Gemma comes in different sizes (e.g., 2B, 7B, 9B, 27B, etc., referencing billions of
  parameters). Smaller models (like 2B) can run on a laptop, while larger models (like 27B) require more powerful
  GPUs.
  • Responsible AI: Gemma was built using "Responsible AI" techniques, meaning it underwent rigorous safety tuning
  to reduce bias and the generation of harmful content.

  #### 3. Why do people use Gemma?

  • Accessibility: You can run it on a personal computer or a workstation without needing a massive data center.
  • Customization (Fine-tuning): Because the weights are open, developers can "fine-tune" the model on specific
  datasets.
  • Privacy: Since you run it on your own local hardware, you can process sensitive data without sending it to a
  cloud provider.
  • Ecosystem: It integrates seamlessly with popular frameworks like Hugging Face, Keras, PyTorch, and JAX.

  #### 4. Summary Comparison

   Feature | Gemini | Gemma
  ------------------------------------|-----------------------------------|---------------------------------------
   Access | Closed (API / Web) | Open (Downloadable weights)
   Size | Massive (huge compute required) | Small to Medium (laptop/GPU friendly)
   Use Case | General purpose, high-end apps | Research, local apps, fine-tuning
   Developer | Google DeepMind | Google DeepMind

  In short: If Gemini is the heavy-duty industrial machine, Gemma is the high-performance power tool that anyone  
  can own and customize.

Benchmark the Local Model

The local model was benchmarked. Summarize the results:

> summarize the benchmark

  Here is a summary of the 2D grid concurrency benchmark results for Gemma 4 12B QAT ( google/gemma-4-12B-it-qat- 
  w4a16-ct ) running on an Azure VM ( Standard_NV36ads_A10_v5 with 1 x NVIDIA A10 GPU):

  ### 📊 Key Performance Metrics

  • Peak Throughput: 136.0 Requests/second (achieved at context size 4 with 64 concurrent users).
  • Stability Limit: 100% request success rate was maintained up to 1024 concurrent users across all context
  windows (from 4 tokens to 16,384 tokens).
  • Stress Load Behavior: At 2048 concurrent users, success rates remained at 100% for smaller context sizes (up
  to 256 tokens) and degraded gracefully to 62.0% at the maximum 16K context window under VRAM pressure.

  ### 🕒 Latency & Scaling

  • Short Context (<= 128 tokens): Extremely fast response times (average latency < 1.0s up to 128 concurrent
  users, scaling to 13.35s under massive 2048 user load).
  • Medium Context (1024 tokens): Remains highly responsive with average latency of 2.32s at concurrency 128,
  and 16.88s at concurrency 1024.
  • Large Context (16384 tokens): Prefill latency scales linearly with concurrency (e.g., 4.26s average latency
  at 128 users, 13.57s at 512 users, and 26.27s at 1024 users).

  ### 💡 DevOps & SRE Conclusion

  Serving the 12B QAT model weights (4-bit weight / 16-bit activation) using vLLM on a single NVIDIA A10 GPU is
  highly robust. The INT4 quantized weight footprint frees up significant VRAM for the KV cache, permitting
  excellent concurrency scaling and stability up to 1024 concurrent users even with high context lengths, making
  it ideal for high-throughput SRE log analysis workflows.

This is visualized:

GPU Memory Allocation of QAT 12B Models

The GPU provides 28 GB of GPU Memory:

> what is the memory breakdown on the GPU?
  The current GPU memory breakdown on the virtualized NVIDIA A10 GPU on the remote VM:

  ### 📊 nvidia-smi Process & VRAM Usage

  • Total VRAM Capacity: 24,512 MiB (24 GB)
  • Allocated/Used VRAM: 21,240 MiB (~86.6% utilization)
  • Free VRAM: 3,272 MiB (~13.4% free)

  #### Active GPU Processes:

   GPU | Process ID | Type | Process Name | GPU Memory Usage
  ----------------------|----------------------|----------------------|---------------------|---------------------
   0 | 26419 | C (Compute) | VLLM::EngineCore | 21,237 MiB 

  Note: The remaining ~3 MiB is system hypervisor/driver overhead.

What are the performance characteristics of the Model?

> what are the performance characteristics of the deployment

  The performance characteristics of the Gemma 4 12B QAT ( google/gemma-4-12B-it-qat-w4a16-ct ) deployment on the
  Azure VM NV36ads A10 v5 are summarized below:

  ### ⚙️ 1. Resource & Serving Settings

  • Quantization Advantage: The model uses 4-bit weights with 16-bit activations ( w4a16 ). This compresses the
  model footprint significantly, leaving ~18 GB of VRAM free for the KV cache pool on the 24 GB A10 GPU.
  • VRAM Allocation Tuning: To accommodate the ~2.35 GB Azure virtualization/hypervisor overhead, the server is
  tuned to --gpu-memory-utilization 0.85 (using 21.2 GB total), preventing out-of-memory crashes on startup.
  • KV-Cache Format: Uses bfloat16 KV cache (Ampere GPUs do not support native FP8 Triton kernels, which are
  reserved for Ada/Hopper architectures).
  ──────
  ### 🚀 2. Throughput & Concurrency

  • Peak Throughput: 136.0 requests/second (at 64 concurrent users, short context).
  • Concurrency Ceiling:
      • 100% Success Rate is maintained up to 1024 concurrent users across all context window sizes (4 to 16,384
      tokens).
      • 2048 concurrent users degrades gracefully to 62% success rate for the maximum 16K context window due to
      prefill queue saturation.

  ──────
  ### 🕒 3. Latency Characteristics

  • Zero-Load TTFT: 0.113s (Time to First Token).
  • Latency Scaling (under concurrent load):
      • 128 Users: Average latency is 0.58s (4 tokens) / 2.32s (1K tokens) / 4.26s (16K tokens).
      • 1024 Users: Average latency is 6.68s (4 tokens) / 16.88s (1K tokens) / 26.27s (16K tokens).
      • 2048 Users: Average latency is 13.35s (4 tokens) / 31.44s (1K tokens) / 33.47s (16K tokens).

One last cross check — what are the trade-offs of the deployment?

> what are the trade-offs on the deployment?

  The performance characteristics of the Gemma 4 12B QAT ( google/gemma-4-12B-it-qat-w4a16-ct ) deployment on the
  Azure VM NV36ads A10 v5 are summarized below:

  ### ⚙️ 1. Resource & Serving Settings

  • Quantization Advantage: The model uses 4-bit weights with 16-bit activations ( w4a16 ). This compresses the
  model footprint significantly, leaving ~18 GB of VRAM free for the KV cache pool on the 24 GB A10 GPU.
  • VRAM Allocation Tuning: To accommodate the ~2.35 GB Azure virtualization/hypervisor overhead, the server is
  tuned to --gpu-memory-utilization 0.85 (using 21.2 GB total), preventing out-of-memory crashes on startup.
  • KV-Cache Format: Uses bfloat16 KV cache (Ampere GPUs do not support native FP8 Triton kernels, which are
  reserved for Ada/Hopper architectures).
  ──────
  ### 🚀 2. Throughput & Concurrency

  • Peak Throughput: 136.0 requests/second (at 64 concurrent users, short context).
  • Concurrency Ceiling:
      • 100% Success Rate is maintained up to 1024 concurrent users across all context window sizes (4 to 16,384
      tokens).
      • 2048 concurrent users degrades gracefully to 62% success rate for the maximum 16K context window due to
      prefill queue saturation.

  ──────
  ### 🕒 3. Latency Characteristics
  • Zero-Load TTFT: 0.113s (Time to First Token).
  • Latency Scaling (under concurrent load):
      • 128 Users: Average latency is 0.58s (4 tokens) / 2.32s (1K tokens) / 4.26s (16K tokens).
      • 1024 Users: Average latency is 6.68s (4 tokens) / 16.88s (1K tokens) / 26.27s (16K tokens).
      • 2048 Users: Average latency is 13.35s (4 tokens) / 31.44s (1K tokens) / 33.47s (16K tokens).
> what are the trade-offs on the deployment?
────────────────────────────────────────────────────────────
▸ Thought Process
  Analyzing Deployment Trade-offs
  The deployment of Gemma 4 12B QAT on the Azure VM Standard_NV36ads_A10_v5 involves several key trade-offs across
  hardware, software, and serving configurations:

  ### 1. Model Precision: QAT w4a16 vs. Standard bfloat16

  • Trade-off: Quality vs. Concurrency Capacity
  • Details:
      • The standard bfloat16 12B model leaves 0 GB of free VRAM for the KV cache on a 24 GB GPU, restricting
      concurrency to < 8 concurrent users.
      • The QAT w4a16 format frees up ~18 GB of VRAM for the KV cache pool, permitting stable serving for up to
      1024 concurrent users (~64x capacity increase).
      • Loss: A negligible degradation in model accuracy/reasoning capability, which QAT minimizes compared to
      standard post-training quantization (PTQ).

  ### 2. Operating System: Ubuntu 22.04 LTS vs. Azure Linux 4.0

  • Trade-off: Ecosystem Compatibility vs. Minimal Footprint
  • Details:
      • Azure Linux 4.0 is highly optimized for lightweight VM instances but lacks mature repository support (    
      tdnf ) for the NVIDIA driver stack, causing installation/compilation failures.
      • Standardizing on Ubuntu 22.04 LTS increases the OS image footprint slightly but guarantees out-of-the-box
      support for the official NvidiaGpuDriverLinux extensions, ensuring driver stability and container toolkit
      binding.

  ### 3. Memory Allocation: --gpu-memory-utilization 0.85 vs. 0.95 

  • Trade-off: Startup Stability vs. KV Cache Overhead
  • Details:
      • Virtualized GPU partitions ( NVIDIA A10-24Q ) on Azure consume ~2.35 GB of VRAM for the driver/hypervisor
      stack immediately upon VM boot.
      • Setting vLLM's utilization to the default 0.95 causes the server to crash during initialization with an
      Out-of-Memory error. Lowering the limit to 0.85 leaves sufficient headroom for the hypervisor to run but
      slightly reduces the total number of concurrent tokens that can be stored in the KV cache.

  ### 4. KV-Cache Type: bfloat16 vs. fp8 

  • Trade-off: Hardware Compatibility vs. Context Window Scaling
  • Details:
      • FP8 KV caching cuts cache memory requirements in half but requires SM89+ compute capabilities (e.g. NVIDIA
      L4 or Hopper architectures).
      • The A10 GPU is based on the Ampere architecture (SM86), which lacks native hardware support for Triton FP8
      attention kernels. We must use bfloat16 KV cache, which consumes double the memory per token but
      guarantees stable and correct execution.

So where is the best place to run Gemma? AWS or GCP? Azure?

Survey says…

> compare aws vs gcp vs azure performance

● Read(/home/xbill/gemma4-tips-azure/gpu-12B-qat-L4-devops-agent/benchmark_report_summary_gcp.md) (ctrl+o to expand)

  Here is the performance comparison of serving Gemma 4 12B QAT across AWS, GCP, and Azure.

  All three deployments utilize 24 GB VRAM GPUs (NVIDIA L4 on AWS/GCP, NVIDIA A10 on Azure) and run the model in
  the w4a16-ct format:

   Metric | AWS EC2 ( g6.2xlarge … | GCP Cloud Run Gen2 (NV… | Azure VM ( NV36ads_A10_…
  ---------------------------------|-------------------------|-------------------------|--------------------------
   Peak Throughput | ~93.3 Req/s | ~40.3 Req/s | 136.0 Req/s 🏆
   100% Success Rate Limit | Up to 1024 users | Up to 512 users | Up to 1024 users 🏆
   Success @ 2048 Users (16K) | ~61.8% | ~22.0% | 62.0% 🏆
   1K Context Latency @ 1024 Users | ~19.90s | ~23.22s | 16.88s 🏆
   Infrastructure Type | Dedicated VM | Serverless Container | Dedicated VM
  ──────
  ### 💡 Infrastructure & Performance Insights

  #### 1. Azure Dedicated VM vs. GCP Cloud Run (Serverless)

  • The Serverless Concurrency Penalty: GCP Cloud Run enforces CPU throttling limits and container sandbox
  virtualization overhead, which limits peak throughput to 40.3 Req/s and causes earlier success rate degradation
  (at 1024 users).
  • Dedicated Compute Power: Both AWS and Azure dedicated VMs bypass container sandbox virtualization. Azure
  achieves the lowest latency (16.88s for 1K context at 1024 concurrency) and the highest peak throughput (136.0  
  Req/s).

  #### 2. GPU Capabilities (NVIDIA A10 vs. NVIDIA L4)

  • Memory Bandwidth: The NVIDIA A10 GPU on Azure has a larger memory bus width (384-bit GDDR6 with 600 GB/s
  bandwidth) compared to the NVIDIA L4 (192-bit GDDR6 with 300 GB/s bandwidth).
  • Impact: The higher memory bandwidth of the A10 significantly accelerates input prefill token processing under
  high concurrent load, explaining the higher throughput and lower latencies on Azure compared to AWS and GCP.

What about costs?!?

The serving costs were summarized:

> what is the cost to run this on Azure
● WebSearch(Standard_NV36ads_A10_v5 azure price) (ctrl+o to expand)
  Here is the cost breakdown to run the Standard_NV36ads_A10_v5 (NVIDIA A10 GPU) serving stack on Azure (using
  typical pricing in US Eastern regions):
  ### 💵 Hourly and Monthly Run Costs

   Purchase Model | Hourly Cost | Monthly Cost (24/7 Run) | SRE Savings
  ----------------------------|---------------------------|---------------------------|---------------------------
   On-Demand | ~$3.20 / hour | ~$2,304 / month | Baseline price
   Spot Instance | ~$0.59 / hour | ~$424 / month | ~81% savings 🏆
  ──────
  ### 💡 Cost Optimization Tips

  1. Leverage Spot Instances (Highly Recommended): Since this is a self-hosted SRE diagnostic stack rather than a
  production-critical user-facing app, using Azure Spot VMs saves 80%+ of compute costs.
  2. Auto-Stop Policy: The MCP server exposes the stop_azure_vm tool. Setting up a cron schedule or SRE trigger
  to automatically invoke stop_azure_vm during off-hours (deallocating the VM) ensures you only pay for active
  execution hours.
  3. Fractional GPU Partitioning: The Azure NVads A10 v5 series supports fractional GPU partitioning (e.g.      
  Standard_NV6ads_A10_v5 provides 1/6th of an A10 GPU with 4GB VRAM). However, for a 12B QAT model, you need the
  full GPU frame buffer ( Standard_NV36ads_A10_v5 ), so partition scaling down is not recommended for this
  specific model size.

────────────────────────────────────────────────────────────
> compare vs aws and gcp

● WebSearch(g6.2xlarge aws price hourly)
● WebSearch(gcp cloud run gpu pricing hourly l4) (ctrl+o to expand)
  Here is the cost comparison between AWS, GCP, and Azure for running the Gemma 4 serving stack:

  ### 💰 Cost Comparison Table

   Platform & Setup | Hourly Cost (Ac… | Idle Cost | Estimated Monthl… | Cost Key Characteristic
  --------------------------|------------------|------------------|-------------------|---------------------------
   GCP Cloud Run | ~$0.75 / hr | $0.00 / hr | Variable (Usage- | Pay-per-second & Scale-   
   (Serverless L4) | | (Idle) | based) | to-zero 🏆
   AWS EC2 ( g6.2xlarge VM | ~$0.98 / hr | ~$0.98 / hr | ~$705 / mo | Best balance for
   L4) | | | | dedicated VM compute
   Azure VM ( | ~$3.20 / hr | ~$3.20 / hr | ~$2,304 / mo | Performance leader, but
   NV36ads_A10_v5 VM A10) | | | | instance is oversized
  ──────
  ### 🔍 Deep Dive & Recommendations

  #### 1. GCP Cloud Run (Serverless) — Best for Low/Intermittent Usage

  • The Scale-to-Zero Advantage: If your SRE diagnostic tools only run occasionally (e.g., triggered on-call or
  during morning log reviews), Cloud Run scales down to 0 instances when idle, billing you $0.00.
  • Trade-off: Cold starts (takes ~1 minute to spin up from zero to load model weights) and lower peak concurrency
  limits.

  #### 2. AWS EC2 ( g6.2xlarge ) — Best Value for Dedicated Running

  • Right-sized VM: AWS pairs the single L4 GPU with a reasonably sized host (8 vCPUs, 32GB RAM), keeping the on-
  demand price under $1.00/hour.
  • Spot Option: Spot pricing brings the cost down to ~$0.30/hour (~$216/month).

  #### 3. Azure VM ( NV36ads_A10_v5 ) — Best for High-Concurrences (Enterprise)

  • Oversized Host: To get a single A10 GPU on Azure, the NVads series bundles it with a massive host (36 vCPUs
  and 440 GiB RAM), which drives the on-demand price up to $3.20/hour.
  • Spot Recommendation: If you deploy on Azure, Spot Instances are mandatory to keep the cost viable at ~$0.     
  59/hour (~$424/month), while delivering the highest throughput (136 Req/s) and memory bandwidth of all three
  platforms.

Summary

The strategy for using MCP for Gemma 4 GPU deployment with Antigravity CLI and Azure was validated with an incremental step by step approach.

12B Gemma 4 QAT Deployment with GCE, NVIDIA L4, MCP, and Antigravity CLI

xbill — Tue, 16 Jun 2026 13:17:24 +0000

This article provides a step by step deployment guide for Gemma 4 to a Google Compute Engine hosted GPU enabled system. A suite of Python MCP tools is built to simplify management of the vLLM hosted Gemma 4 deployment with Antigravity CLI.

What is this project trying to Do?

This project is a DevOps/SRE assistant that uses a Gemma 4 model hosted on GCE with GPU. It provides tools to provision the Docker container and deploy the model, as well as for observability and performance testing.

This project is similar to a previous project that targeted GPU hosted Gemma4 instances on GCP:

Gemma-SRE: Self-Hosted vLLM Infrastructure Agent

Antigravity CLI

Antigravity CLI is the follow-on successor to Gemini CLI- the terminal driven, agent assisted coding tool.

Full details on installing Antigravity CLI are here:

Getting Started with Antigravity CLI

Testing the Antigravity CLI Environment

Once you have all the tools in place- you can test the startup of Antigravity CLI.

You will need to authenticate with a Google Cloud Project or your Google Account:

agy

This will start the interface:

Full Installation Instructions

The detailed installation instructions for Antigravity CLI are here:

Getting Started with Antigravity CLI

Python MCP Documentation

The official GitHub Repo provides samples and documentation for getting started:

GitHub - modelcontextprotocol/python-sdk: The official Python SDK for Model Context Protocol servers and clients

Where do I start?

The strategy for starting MCP development for model management is a incremental step by step approach.

First, the basic development environment is setup with the required system variables, and a working Antigravity CLI configuration.

Then, a minimal Python MCP Server is built with stdio transport. This server is validated with Antigravity CLI in the local environment.

This setup validates the connection from Antigravity CLI to the local server via MCP. The MCP client (Antigravity CLI) and the Python MCP server both run in the same local environment.

Setup the Basic Environment

At this point you should have a working Python environment and a working Antigravity CLI installation. The next step is to clone the GitHub samples repository with support scripts:

cd ~
git clone https://github.com/xbill9/gemma4-tips

Then run init.sh from the cloned directory.

The script will attempt to determine your shell environment and set the correct variables:

g2-4-12B-qat-L4-devops-agent
source init.sh

If your session times out or you need to re-authenticate- you can run the set_env.sh script to reset your environment variables:

g2-4-12B-qat-L4-devops-agent
source set_env.sh

Variables like PROJECT_ID need to be setup for use in the various build scripts- so the set_env script can be used to reset the environment if you time-out.

Model Management Tool with MCP Stdio Transport

One of the key features that the standard MCP libraries provide is abstracting various transport methods.

The high level MCP tool implementation is the same no matter what low level transport channel/method that the MCP Client uses to connect to a MCP Server.

The connection over stdio will look similar to this:

# Initialize FastMCP server
mcp = FastMCP("Self-Hosted vLLM DevOps Agent")

Running the Python Code

First- switch the directory with the Python version of the MCP sample code:

~/gemma4-tips/g2-4-12B-qat-L4-devops-agent

Run the release version on the local system:

make install
Processing ./.

The project can also be linted:

xbill@penguin:~/gemma4-tips/g2-4-12B-qat-L4-devops-agent$ make lint
ruff check .
All checks passed!
ruff format --check .
7 files already formatted
mypy .
Success: no issues found in 7 source files
Success: no issues found in 6 source files

Getting Started with Gemma 4 on GPU

The Official vLLM repo also has Gemma4 specific information:

Releases · vllm-project/vllm

Gemma 12B

Gemma 12B is the latest model in the Gemma4 series. More details are available here:

Introducing Gemma 4 12B: a unified, encoder-free multimodal model

and here:

Gemma 4 12B: The Developer Guide

NVIDIA L4 on GCE

The NVIDIA L4 support on Google Cloud Compute Engine is documented here:

Introducing G2 VMs with NVIDIA L4 GPUs | Google Cloud Blog

Quantization-Aware Training (QAT)

Gemma 4 QAT models: Optimizing model compression for mobile and laptop efficiency

QAT Model Versions

What are the differences between the versions on Hugging Face?

> what is the difference between google/gemma-4-12B-it-qat-q4_0-unquantized and https://huggingface.co/google/gemma-
  4-12B-it-qat-q4_0-gguf and google/gemma-4-12B-it-qat-w4a16-ct

▸ Thought for 1s, 602 tokens
  Analyzing Gemma QAT Variations

● Read(/home/xbill/gemma4-tips/gpu-12B-qat-L4-devops-agent/GEMINI.md) (ctrl+o to expand)

  Here is a comparison of the three Gemma 4 QAT (Quantization-Aware Training) model configurations:

  ### 1. google/gemma-4-12B-it-qat-q4_0-unquantized 

  • Format: Unquantized, high-precision weights (usually float16 or bfloat16) extracted directly from the QAT
  training pipeline.
  • Target Engine: Standard deep learning/fine-tuning framework runtimes (e.g., PyTorch, Transformers, Jax).
  • Primary Use Case:
      • Serving as a high-precision reference baseline.
      • Custom downstream quantization, compilation, or MLX conversion.
      • Research, fine-tuning, or running Speculative Decoding (where this serves as the primary model alongside
      its matching MTP assistant/draft model).


  ### 2. google/gemma-4-12B-it-qat-q4_0-gguf 

  • Format: GGUF (specifically quantized using the standard q4_0 4-bit method).
  • Target Engine: Local CPU, Apple Silicon, or consumer GPU engines (e.g., llama.cpp , Ollama, LM Studio).
  • Primary Use Case:
      • Zero-setup local execution with minimal memory requirements.
      • Running LLMs efficiently on consumer-grade hardware.


  ### 3. google/gemma-4-12B-it-qat-w4a16-ct 

  • Format: Compressed Tensors format ( -w4a16-ct ), which stores 4-bit weights while performing computation using
  16-bit activations.
  • Target Engine: High-performance, high-throughput cloud inference engines (specifically vLLM or SGLang ).
  • Primary Use Case:
      • High-concurrency cloud deployments utilizing GPU accelerators (e.g., NVIDIA L4).
      • Optimizing GPU memory usage and improving inference speed while maintaining near-lossless model accuracy
      via QAT.

  ──────
  ### 📊 Summary Reference Table

   Checkpoint Configuration | Serialization Format | Target Platform | Primary Engine
  ---------------------------|---------------------------|----------------------------|----------------------------
    ...-qat-q4_0-unquantized | Unquantized ( BF16 / FP16 | Custom compilation / | Transformers / Speculative
                             | ) | research | Drafts
    ...-qat-q4_0-gguf | GGUF (4-bit weight) | Local Desktop/Server | llama.cpp / LM Studio /
                             | | (CPU/GPU) | Ollama
    ...-qat-w4a16-ct | compressed-tensors (4- | Cloud GPU Deployments | vLLM / SGLang 
                             | bit/16-bit) | |

  For more details on QAT architecture and configuration, see GEMINI.md.
  ──────
  ### Summary of Work

  • Reviewed Context: Analyzed the workspace documentation GEMINI.md to compare the unquantized, GGUF, and
  compressed-tensors configurations.

MCP stdio Transport

One of the key features that the MCP protocol provides is abstracting various transport methods.

The high level tool MCP implementation is the same no matter what low level transport channel/method that the MCP Client uses to connect to a MCP Server.

In this project Antigravity CLI is used as the MCP client to interact with the Python MCP server code.

Antigravity CLI mcp_config.json

A sample MCP server file is provided in the .agents directory:

{
  "mcpServers": {
    "gpu-devops-agent": {
      "command": "python3",
      "args": [
        "/home/xbill/gemma4-tips/gpu-12B-qat-L4-devops-agent/server.py"
      ],
      "env": {
        "GOOGLE_CLOUD_PROJECT": "aisprint-491218",
        "GOOGLE_CLOUD_LOCATION": "us-east4",
        "VLLM_BASE_URL": "https://gpu-12b-qat-l4-devops-agent-289270257791.us-east4.run.app",
        "MODEL_NAME": "/mnt/models/gemma-4-12B-it-qat-w4a16-ct"
      }
    }
  }
}

Validation with Antigravity CLI

The final connection test uses Antigravity CLI as a MCP client with the Python code providing the MCP server:

MCP Servers

Plugins (~/.gemini/antigravity-cli/plugins)
> ✓ google-dev-knowledge Tools: search_documents, answer_query, get_documents
   ✓ gpu-devops-agent Tools: save_hf_token, get_vllm_endpoint, list_vertex_models, list_bucket_models,
                       analyze_cloud_logging, +26 more

Model Lifecycle Management via MCP

The MCP tools provide a complete suite of agent-oriented operations for managing vLLM deployment on GCE, Cloud Run or a TPU.

Overview of MCP tools :

  Here is the output of the get_help tool:

  ### 🛠️ GCP Gemma 4 SRE Agent Help & Configuration

  You can configure this MCP server using the following environment variables:

  GCP Configuration:

  • GOOGLE_CLOUD_PROJECT : Your GCP Project ID.
      • Current Value: comglitn 
  • GOOGLE_CLOUD_LOCATION : The GCP Region/Location.
      • Current Value: us-east4 
  • GOOGLE_CLOUD_ZONE : The GCP Zone for GCE VM deployment.
      • Current Value: us-east4-a 
  • BUCKET_NAME : GCS Bucket used to store model weights.
      • Current Value: comglitn-bucket 

  General serving:

  • MODEL_NAME : Default Hugging Face repository or GCS path.
      • Current Value: google/gemma-4-12B-it-qat-w4a16-ct 
  • VLLM_BASE_URL : The explicit URL of your vLLM GCE service. (If not set, it is auto-discovered via GCE VM
  external IP)
      • Current Value: Not set (auto-discovering) 

  ### ℹ️ Active Mode Summary

  The server is running in GCP GCE VM mode targeting a g2-standard-4 host VM with NVIDIA L4 GPU.

  ### 🧰 Available MCP Tools

  Below is a summary of the tools exposed by this SRE/DevOps agent:

  #### 🐳 Infrastructure & Deployment

  • start_gce : Starts an existing GCE instance, or provisions a new one if none exists.
  • status_gce : Checks GCE instance status.
  • stop_gce : Stops GCE instance.
  • check_vllm : Checks the status of the vLLM container and engine running on the GCE instance.
  • deploy_vllm : Deploys vLLM to GCP GCE g2-standard-4 (NVIDIA L4) VM instance.
  • destroy_vllm : Deletes the GCP GCE vLLM VM instance.
  • status_vllm : Checks GCE instance status.
  • update_vllm_scaling : Scales GCE instance type vertically.
  • get_vllm_deployment_config : Generates the gcloud compute command and startup script.
  • get_vllm_gpu_deployment_config : Generates a GKE manifest for GPU (NVIDIA L4).
  • check_gpu_quotas : Checks GPU/Accelerator quotas for a region.
  • get_vllm_endpoint : Returns the current active vLLM endpoint URL.

  #### 📊 Model Management

  • list_vertex_models : Lists models in the Vertex AI Registry.
  • list_bucket_models : Lists model weights in GCS bucket.
  • save_hf_token : Securely saves a Hugging Face API token to Secret Manager.
  • get_vertex_ai_model_copy_instructions : Instructions to copy model from Vertex AI Model Garden to GCS.
  • get_huggingface_model_copy_instructions : Instructions to download model from Hugging Face and upload to GCS.
  • get_huggingfacehub_download_path : Resolves local cache path using huggingface_hub.

  #### 📊 Monitoring & Status

  • get_metrics : Fetches raw Prometheus metrics from the running vLLM service's /metrics endpoint.
  • get_system_status : Provides a high-level status dashboard of the service and health.
  • get_endpoint : Verifies connectivity and returns the active service URL.
  • get_model_details : Retrieves detailed model metadata and engine state from /v1/models .
  • verify_model_health : Deep health check by querying the model with a simple prompt and measuring latency.

  #### 📈 Performance & Benchmarking

  • run_benchmark : Runs performance/concurrency benchmark sweeps against the vLLM GPU endpoint.

  #### 💬 Interaction & Diagnostics

  • query_gemma4 : Primary tool to query the self-hosted model with standard chat message format.
  • query_gemma4_with_stats : Queries the model and returns streaming performance statistics (TTFT, throughput).
  • query_vllm : Direct text completions querying tool.
  • analyze_cloud_logging : Fetches logs from GCP Logging and analyzes them using the model.
  • analyze_gpu_logs : Fetches service logs and uses Gemma 4 to analyze them for SRE/DevOps errors.
  • suggest_sre_remediation : Suggests remediation plans for SRE errors using the model.

Deploy The Model

The MCP server tools are used to deploy the model to GCE with GPU:

deploy the model

Checking System status

The status can be checked with an MCP tool:

> get_system_status

  The system status has been successfully updated and is fully healthy:

  ### 🌀 GPU vLLM System Status

  • vLLM Health: 🟢 Online (http://34.82.63.29:8080)
  • Hosting Status: 🟢 Running ( gpu-12b-qat-l4-devops-agent )
  👉 Next Step: Use query_gemma4 to interact with the model.

Cross Check The Deployed Model

Once the model starts locally- the MCP tool allows for model verification:

> verify_model_health

  I will execute the verify_model_health tool function from server.py to run a model query benchmark and
  measure latency.

● Bash(python3 -c "import asyncio; from server import verify_model_health; print(asyncio.run(veri...) (ctrl+o to
expand)

  The model health check has successfully passed:

  • Model: google/gemma-4-12B-it-qat-w4a16-ct 
  • Response: "Yes, I am working and ready to help! How can I assist you today?" 
  • Latency: 0.84 seconds 
  • Status: 🟢 PASSED

and model details:

> get_active_models

  I will call the get_model_details tool function from server.py to fetch details on the active served models.

● Bash(python3 -c "import asyncio; from server import get_model_details; print(asyncio.run(get_mo...) (ctrl+o to
expand)

  The active model currently served by vLLM is:

  ### 🧩 Active Model Details

  • Model ID: google/gemma-4-12B-it-qat-w4a16-ct 
  • Object: model 
  • Owner: vllm 
  • Endpoint Status: Healthy ✅

and the endpoint:

> get_endpoint

  I will call the get_endpoint tool function in server.py to retrieve and verify the active API endpoint.

● Bash(python3 -c "import asyncio; from server import get_endpoint; print(asyncio.run(get_endpoin...) (ctrl+o to
expand)

  The active validated vLLM endpoint is:

    http://34.82.63.29:8080

  • Status: 🟢 Online (HTTP 200)

and status:

> get status

  I will fetch the current status of the GCE VM instance and the vLLM engine to provide a unified summary.

● Bash(make status ZONE=us-west1-a) (ctrl+o to expand)

  Here is the complete status of the serving stack:

  ### 🖥️ GCP GCE VM Instance Status

  • Instance Name: gpu-12b-qat-l4-devops-agent 
  • Instance State: 🟢 RUNNING 
  • Zone: us-west1-a 
  • Machine Type: g2-standard-4 (1x NVIDIA L4 GPU, 24GB VRAM)
  • Public IP Address: 34.82.63.29 

  ### 🌀 vLLM Serving Status

  • Endpoint: http://34.82.63.29:8080 
  • Health Endpoint ( /health ): 🟢 Healthy (HTTP 200)
  • Active Model: google/gemma-4-12B-it-qat-w4a16-ct 
  • Model State: Loaded and listening to requests.

Check the Cloud GCE VM

Once Deployed — the model is visible as a standard VM:

Review the Model

Check the model details:

> what are the model details?

  I will query the GCE VM vLLM server configuration details by pulling the model information and runtime settings.

● Bash(python3 -c "import asyncio; from server import get_model_details; print(asyncio.run(get_mo...) (ctrl+o to
expand)

  The active model details on the self-hosted GCE server are:

  ### 🧩 Served Model Details

  • Model Identifier: google/gemma-4-12B-it-qat-w4a16-ct 
  • Quantization Type: Quantization-Aware Training (QAT) with 4-bit weights and 16-bit activations ( w4a16 ) using
  the native compressed-tensors ( -ct ) serialization format.
  • Serving Backend: vllm (v0.22.1rc1 nightly build) utilizing the Marlin linear kernel ( MarlinLinearKernel for
  CompressedTensorsWNA16 ) for fast 4-bit weight decoding.
  • GPU Hardware: Single NVIDIA L4 GPU (24GB VRAM) hosted on a GCP Compute Engine (GCE) g2-standard-4 VM instance.
  • VRAM Configuration:
      • gpu-memory-utilization : 0.95 (allocates 95% of GPU memory to vLLM cache)
      • kv-cache-dtype : fp8 (8-bit quantized key-value cache to save memory and maximize concurrency)
  • API Server Health: 🟢 Healthy (HTTP 200) at http://34.82.63.29:8080

Once everything is online — you can send queries to the Model:

> query_gemma4_with_stats what is gemma?

  I will run the model query tool with performance stats to answer "what is gemma?" and retrieve latency and
  throughput metrics.

● Bash(python3 -c "import asyncio; from server import query_gemma4_with_stats; print(asyncio.run(...) (ctrl+o to
expand)
  The query is running in the background. I will await the results.

  Here are the performance metrics and response for the query:
  ### 📊 Performance Stats

  • Model: google/gemma-4-12B-it-qat-w4a16-ct 
  • Time to First Token (TTFT): 0.198s 
  • Total Generation Time: 23.326s 
  • Tokens per Second: 26.89 tokens/s 
  • Total Tokens generated: 622 

  ### 💬 Model Response

  The term Gemma most commonly refers to a family of lightweight, state-of-the-art open models developed by Google 
  DeepMind.

  To understand what Gemma is, it helps to look at it through three different lenses: its relationship to Gemini,
  its technical nature, and its purpose.

  #### 1. The Relationship to Gemini

  Gemma is built using the same technology, infrastructure, and research used to create Gemini, Google’s most
  capable AI model:

  • Gemini is a "closed" model (available via API or Google products).
  • Gemma is an "open" model (available for developers to download and run on their own hardware).

  #### 2. Key Characteristics

  What makes Gemma unique in the AI landscape are these three traits:

  • Open Weights: This means that the internal weights are shared with the public, allowing developers to build
  upon, fine-tune, and customize the model.
  • Lightweight & Efficient: Unlike massive models that require supercomputers to run, Gemma is designed to be
  small enough to run on a single GPU or even a high-end laptop.
  • Responsible AI: It includes safety filtering to ensure the models are less likely to generate harmful or toxic
  content.

  #### 3. Versions and Sizes

  Gemma comes in different "sizes" (parameter counts) to balance compute requirements with reasoning capabilities:

  • Gemma 2B: A tiny model capable of running on mobile devices or laptops.
  • Gemma 7B/9B/27B/12B/31B: Mid-to-large sized models that are highly capable at reasoning, coding, and creative
  writing while still being manageable for individual developers.

  #### 4. Who is it for?

  Gemma is designed primarily for:

  • Developers: Who want to build apps (like a personal assistant or a specialized chatbot) without paying fees to
  use the Gemini API.
  • Researchers: Who want to study how Large Language Models (LLMs) work and experiment with fine-tuning them on
  specific datasets.
  • Enterprises: Companies that want to run AI on their own private servers to ensure data privacy.

  #### Summary Table

   Feature | Gemini | Gemma
  -------------------------------------|-------------------------------------|-------------------------------------
   Accessibility | Closed (API/Web) | Open Weights (Downloadable)
   Size | Massive (Very Large) | Small to Medium
   Hardware | Google's Supercomputers | Local GPUs / Cloud / Laptops
   Best For | Consumer apps, complex reasoning | Development, Research, Local Use

Benchmark the Local Model

The local model was benchmarked:

The benchmark sweep has completed successfully. Here are the compiled results across all combinations of context
  window sizes and concurrency levels:

The analysis is reviewed:

> summarize benchmark results

  ### 1. Request Success Rate (Reliability)

  • 100% Success Rate: Maintained across all context lengths (4 to 8,162 tokens) for workloads up to 1,024         
  concurrent users.
  • 16K Context Window: Success rate remains at 99.7% under 1,024 concurrent users, dropping to 47.7% at 2,048
  users due to KV-cache memory pressure.

  ### 2. Latency (Prefill & Queue Waiting)

  • Low Concurrency (1-32 Users): Very responsive; average latency stays below 1.2 seconds even with large context
  sizes (4K).
  • High Concurrency (256-1024 Users): Queue wait times begin to dominate. Latencies scale linearly:
      • 128 Context: scales from 0.26s (1 user) to 5.05s (256 users) to 18.47s (1024 users).
      • 16K Context: scales from 5.61s (1 user) to 9.96s (256 users) to 33.26s (1024 users).

  ### 3. Throughput (Requests per Second)

  • Peak Throughput: Reaches 75.8 requests/sec under small context windows (4 tokens) at 128 users.
  • Large Contexts (16K): Throughput saturates at approximately 14 to 16 requests/sec for concurrencies of 128
  users and above.

  Detailed tables and the performance plot can be viewed in the generated gce_benchmark_results.md file.

This is visualized:

Compare to Other Deployments

The GCE deployment was reviewed:

  Here is a summary of the performance comparison for serving Gemma 4 QAT on an NVIDIA L4 GPU:

  ### 1. GCP GCE VM vs. AWS EC2 (VM vs. VM)

  • Performance: Almost identical. Average latency at 1024 users (1K context) is 19.91s on GCE and 19.90s on AWS.
  • Extreme Load: AWS EC2 holds a slight edge under 2048 concurrent users at 16K context window with a 61.8%       
  success rate compared to 47.7% on GCP GCE VM.

  ### 2. VMs (GCE/EC2) vs. GCP Cloud Run (VM vs. Serverless)

  • Throughput: Dedicated VMs scale to significantly higher peak throughput. GCE VM peaked at 75.8 RPS and AWS at
  93.3 RPS, whereas Cloud Run was capped at 40.3 RPS.
  • Scalability: Cloud Run degrades much earlier under high concurrency. At 2048 users (16K context), its success
  rate plummets to 22%, while GCE VM maintains 47.7%.
  • Overhead: Cloud Run's gVisor container sandboxing and routing layers add a 3–4 second latency penalty under
  load, in addition to cold start delays.

Cost Analysis

The current Google Cloud pricing is used to evaluate the cost per token:

> summarize gce_cost_analysis.md

  Here is a summary of the cost analysis for running the g2-standard-4 GCE VM instance with an NVIDIA L4 GPU:

  ### 1. Core Monthly Costs (24/7 Run-time)

  • On-Demand Compute: ~$547.50 / month ($0.75 / hour).
  • Balanced Storage (150 GB): $15.00 / month ($0.10 / GB).
  • Total On-Demand Baseline: ~$562.50 / month.

  ### 2. Cost Optimization Options

  • Spot Instances (~70% savings): Reduces compute costs to ~$167.90 / month ($0.23 / hour).
  • Dynamic VM Scheduling (10h/day, weekday-only): Reduces compute costs to ~$165.00 / month by leveraging the
  agent's programmatic start_gce and stop_gce tools to power down the VM during idle hours.

  ### 3. Value Proposition

  At $0.23 - $0.75 / hour, a single L4 GPU supports up to 1024 concurrent users with 100% success rate for context
  lengths up to 8K. This makes GCE self-hosting highly cost-efficient compared to managed pay-per-token API
  endpoints.

 I have prepared a cost comparison between GCP GCE, AWS EC2, and Google Cloud Run. The detailed cost matrix and
  workload analysis are stored in platform_cost_comparison.md.

  Here is a summary of the cost comparison:

  ### 📊 Hourly & Monthly Cost Matrix (NVIDIA L4 GPU)

   Hosting Option | GCP GCE ( g2-standard-4 ) | AWS EC2 ( g6.2xlarge ) | GCP Cloud Run (Gen2 Serverless)
  --------------------------|---------------------------|------------------------|---------------------------------
   On-Demand Hourly | ~$0.75 / hr | ~$0.98 / hr | ~$0.946 / hr (Active)
   Spot Hourly | ~$0.23 / hr | ~$0.27 / hr | N/A (Not supported)
   Monthly (24/7 On-Demand) | ~$547.50 / mo | ~$715.40 / mo | ~$690.58 / mo (Warm min=1 )
   Monthly (24/7 Spot) | ~$167.90 / mo | ~$197.10 / mo | N/A
   Idle / Scale-to-Zero | No (Manual scheduling) | No (Manual scheduling) | Yes (0 active cost when idle)

  ### ⚖️ Cost Analysis by Workload Type

  1. Intermittent / Burst Use (GCP Cloud Run)
      • Best for: Occasional workloads (e.g., automated log audits triggered once daily).
      • Advantage: Scales down to 0 instances when idle, incurring $0.00 compute charges.
      • Disadvantage: Idle scale-down triggers a cold start (~1–2 mins) on subsequent calls. Keeping the model warm
      ( min-instances=1 ) costs ~$690.58/mo.
  2. Non-Production / Batch Workloads (Spot Instances)
      • Best for: Development and testing.
      • Advantage: GCP GCE Spot VMs are the cheapest way to serve continuously ($0.23/hr vs AWS Spot's $0.27/hr),
      bringing the total monthly bill to ~$182.90 (with storage).
      • Disadvantage: Preemptible; instances can be shut down with a 30-second warning.
  3. Steady-State Production Workloads (On-Demand VMs)
      • Best for: 24/7 SRE agents or alert response bots requiring sub-second response times.
      • Advantage: GCP GCE VMs are ~23% cheaper than AWS EC2 g6.2xlarge ($547.50/mo vs. $715.40/mo) and ~20%     
      cheaper than keeping Cloud Run warm.

And Price/Performacne?

The winner is…

  ### 🏆 Summary: GCP GCE VM ( g2-standard-4 ) has the best overall price/performance for steady-state workloads.

  We can measure price/performance by calculating Peak Throughput (Requests Per Second) per Dollar of hourly cost:

   Metric | GCP GCE VM ( g2-standard-4 ) | AWS EC2 ( g6.2xlarge ) | GCP Cloud Run (Gen2 Serverless
  -------------------------|------------------------------|------------------------|-------------------------------
   Peak Throughput | 75.8 RPS | 93.3 RPS | 40.3 RPS
   Hourly Rate (On-Demand) | $0.75 / hr | $0.98 / hr | $0.946 / hr (Warm)
   Steady-State Price/Perf | 101.1 RPS per $ | 95.2 RPS per $ | 42.6 RPS per $
   Hourly Rate (Spot) | $0.23 / hr | $0.27 / hr | N/A
   Spot Price/Perf | 329.6 RPS per $ | 345.6 RPS per $ | N/A
  ──────
  ### ⚖️ Recommendations

  1. For Steady-State Serving (24/7): GCP GCE VM
      • Winner: GCP GCE VM
      • Why: It yields the highest value for on-demand serving (101.1 RPS/$ vs AWS's 95.2 RPS/$). It avoids the
      gVisor sandbox and container virtualization overhead of Cloud Run, which cuts throughput in half (40.3 RPS)
      while cost remains high ($0.946/hr) if kept warm.
  2. For Development & Batch Serving: AWS Spot or GCP Spot
      • Winner: AWS EC2 Spot (by a small margin on throughput value), though GCP GCE Spot is cheaper in absolute
      monthly dollars ($167.90 vs $197.10).
      • Why: Both spot options offer spectacular price/performance (over 320+ RPS per dollar), making them ideal
      for offline training, benchmarking, or non-critical diagnostic sweeps.
  3. For Highly Intermittent Auditing: GCP Cloud Run (Serverless)
      • Winner: GCP Cloud Run (with scale-to-zero active)
      • Why: If your SRE agent only runs a log diagnosis task 2 to 3 times a day and sits idle the rest of the time,
      Cloud Run's scale-to-zero feature brings active costs to $0.00, making it the most cost-effective choice
      overall, despite lower peak throughput and the ~1–2 minute cold start.

Summary

The strategy for using MCP for Gemma 4 GPU deployment with Antigravity CLI was validated with a incremental step by step approach.

Deploying Gemma 12B to AWS EC2 with NVIDIA L4 and Antigravity CLI

xbill — Tue, 16 Jun 2026 13:00:16 +0000

This article provides a step by step debugging guide for deploying Gemma 4 to an AWS hosted GPU enabled system,. A suite of Python MCP tools is built to simplify management of the vLLM hosted Gemma 4 deployment with Antigravity CLI.

What is this project trying to Do?

This project is a DevOps/SRE assistant that uses a Gemma 4 model hosted on AWS with GPU. It provides tools to provision the Docker container and deploy the model, as well as for observability and performance testing.

This project is similar to a previous project that targeted GPU hosted Gemma4 instances on GCP:

Gemma-SRE: Self-Hosted vLLM Infrastructure Agent

Antigravity CLI

Antigravity CLI is the follow-on successor to Gemini CLI- the terminal driven, agent assisted coding tool.

Full details on installing Antigravity CLI are here:

Getting Started with Antigravity CLI

Testing the Antigravity CLI Environment

Once you have all the tools in place- you can test the startup of Antigravity CLI.

You will need to authenticate with a Google Cloud Project or your Google Account:

agy

This will start the interface:

Full Installation Instructions

The detailed installation instructions for Antigravity CLI are here:

Getting Started with Antigravity CLI

AWS Setup

The AWS CLI provides the basic tools for working with the AWS services:

AWS CLI

Check the AWS installation:

xbill9@cloudshell:~ (aisprint-491218)$ /usr/local/bin/aws --version
aws-cli/2.34.57 Python/3.14.5 Linux/6.6.137+ exe/x86_64.ubuntu.24
xbill9@cloudshell:~ (aisprint-491218)$

Once the tools are installed — login to the AWS console:

aws login --remote

AWS Skills

AWS provides pre-packages skills and a MCP server:

Workspace skills · Workspace config
  amazon-aurora-mysql: Amazon Aurora MySQL — creates, modifies, and advises on Aurora MySQL clusters specifically ...
  amazon-bedrock: Builds generative AI applications on Amazon Bedrock. Covers model invocation (Converse API, Invo...
  amazon-elasticache: Activate when developers have latent caching needs: slow API responses, database read bottle...
  aws-amplify: Build and deploy full-stack web and mobile apps with AWS Amplify Gen2 (TypeScript code-first). Cove...
  aws-billing-and-cost-management: Analyze AWS costs, find savings, manage budgets, evaluate Savings Plans and

And the AWS MCP server:

      "aws-mcp": {
      "command": "uvx",
      "timeout": 100000,
      "transport": "stdio",
      "args": [
        "mcp-proxy-for-aws==1.6.0",
        "https://aws-mcp.us-east-1.api.aws/mcp",
        "--metadata", "AWS_REGION=us-east-1"
      ]
    }

and live MCP tools:

MCP Servers

Plugins (~/.gemini/antigravity-cli/plugins)
> ✓ aws-mcp Tools: aws ___call_aws, aws___ get_presigned_url, aws ___get_tasks, aws___ run_script,
              aws___get_regional_availability, +5 more

And AWS toolkit skills:

aws configure agent-toolkit

Where do I start?

The strategy for starting MCP development for model management is a incremental step by step approach.

First, the basic development environment is setup with the required system variables, and a working Antigravity CLI configuration.

Then, a minimal Python MCP Server is built with stdio transport. This server is validated with Antigravity CLI in the local environment.

This setup validates the connection from Antigravity CLI to the local server via MCP. The MCP client (Antigravity CLI) and the Python MCP server both run in the same local environment.

Setup the Basic Environment

At this point you should have a working Python environment and a working Antigravity CLI installation. The next step is to clone the GitHub samples repository with support scripts:

cd ~
git clone https://github.com/xbill9/gemma4-tips-aws

Then run init.sh from the cloned directory.

The script will attempt to determine your shell environment and set the correct variables:

cd gpu-12B-qat-L4-devops-agent
source init.sh

If your session times out or you need to re-authenticate- you can run the set_env.sh script to reset your environment variables:

cd gpu-12B-qat-L4-devops-agent
source set_env.sh

Variables like PROJECT_ID need to be setup for use in the various build scripts- so the set_env script can be used to reset the environment if you time-out.

Model Management Tool with MCP Stdio Transport

One of the key features that the standard MCP libraries provide is abstracting various transport methods.

The high level MCP tool implementation is the same no matter what low level transport channel/method that the MCP Client uses to connect to a MCP Server.

The connection over stdio will look similar to this:

# Initialize FastMCP server
mcp = FastMCP("Self-Hosted vLLM DevOps Agent")

Running the Python Code

First- switch the directory with the Python version of the MCP sample code:

~/gemma4-tips-aws/cd gpu-12B-qat-L4-devops-agent

Run the release version on the local system:

xbill@penguin:~/gemma4-tips-aws/gpu-12B-qat-L4-devops-agent$ make install
pip install -r requirements.txt
Requirement already satisfied: mcp in /home/xbill/.pyenv/versions/3.13.13/lib/python3.13/site-packages (from -r requirements.txt (line 1)) (1.27.2)
Requirement already satisfied: fastmcp in /home/xbill/.pyenv/versions/3.13.13/lib/python3.13/site-packages (from -r re

The project can also be linted:

xbill@penguin:~/gemma4-tips-aws/gpu-12B-qat-L4-devops-agent$ make lint
ruff check .
All checks passed!
ruff format --check .
7 files already formatted
mypy .
Success: no issues found in 7 source files
xbill@penguin:~/gemma4-tips-aws/gpu-12B-qat-L4-devops-agent$

MCP stdio Transport

One of the key features that the MCP protocol provides is abstracting various transport methods.

The high level tool MCP implementation is the same no matter what low level transport channel/method that the MCP Client uses to connect to a MCP Server.

In this project Antigravity CLI is used as the MCP client to interact with the Python MCP server code.

Antigravity CLI mcp_config.json

A sample MCP server file is provided in the .agents directory:

xbill@penguin:~/gemma4-tips-aws/gpu-12B-qat-L4-devops-agent/.agents$ more mcp_config.json 
{
  "mcpServers": {
    "gpu-devops-agent": {
      "command": "python3",
      "args": [
        "/home/xbill/gemma4-tips-aws/gpu-12B-qat-L4-devops-agent/server.py"
      ],
      "env": {
        "GOOGLE_CLOUD_PROJECT": "aisprint-491218",
        "MODEL_NAME": "/mnt/models/gemma-4-12B-it-qat-w4a16-ct"
      }
    },
      "aws-mcp": {
      "command": "uvx",
      "timeout": 100000,
      "transport": "stdio",
      "args": [
        "mcp-proxy-for-aws==1.6.0",
        "https://aws-mcp.us-east-1.api.aws/mcp",
        "--metadata", "AWS_REGION=us-east-1"
      ]
    }
  }
}

Validation with Antigravity CLI

The final connection test uses Antigravity CLI as a MCP client with the Python code providing the MCP server:

MCP Servers

Configured (~/.gemini/antigravity-cli/mcp_config.json)
> ✓ developer-knowledge Tools: search_documents, answer_query, get_documents

Plugins (~/.gemini/antigravity-cli/plugins)
   ✓ aws-mcp Tools: aws ___call_aws, aws___ get_presigned_url, aws ___get_tasks, aws___ run_script,
              aws___get_regional_availability, +5 more
   ✓ google-dev-knowledge Tools: search_documents, answer_query, get_documents
   ✓ gpu-devops-agent Tools: save_hf_token, get_vllm_endpoint, list_vertex_models, list_bucket_models,
                       analyze_cloud_logging, +22 more

Instance Lifecycle Management via MCP

The MCP tools provide a complete suite of agent-oriented operations for managing vLLM deployment on Cloud Run or a TPU.

Overview of MCP tools :

> help

  Here is the help documentation for the AWS EC2 management tools exposed by the vLLM DevOps Agent.
  ──────
  ### 🧰 AWS EC2 MCP Management Tools

   Tool Name | Action | Description / Parameters
  ----------------|-------------------|----------------------------------------------------------------------------
    start_ec2 | Start / Provision | Starts an existing stopped EC2 instance, or provisions a new one (e.g.     
                  | | g6.2xlarge with Nvidia L4 GPU) if none exists.- service_name : Name tag
                  | | (default: gpu-12b-qat-l4-devops-agent ) - instance_type : Instance type
                  | | (default: g6.2xlarge ) - market_type : Market type ( on-demand or spot 
                  | | ) - subnet_id : Specific Subnet ID to launch in (optional) - instance_id 
                  | | : Target instance ID (optional)
    status_ec2 | Check Status | Checks the current state, type, public IP, and launch details of your EC2
                  | | instances.- service_name : Name tag to search (optional) - instance_id :
                  | | Specific instance ID to query directly (optional)
    stop_ec2 | Stop Instance | Stops active instance(s) safely without terminating/deleting the root EBS
                  | | volume, allowing you to resume workloads later.- service_name : Name tag
                  | | of instances to stop (optional) - instance_id : Specific instance ID to
                  | | stop directly (optional)
    destroy_vllm | Terminate | Terminates (deletes) matching EC2 instances entirely.- service_name :
                  | | Name tag of instance(s) to terminate.
  ──────
  ### 💡 Example Tool Invocation Signatures

  #### 1. Launch / Start an On-Demand Instance:

    start_ec2(
        service_name="gpu-12b-qat-l4-devops-agent",
        instance_type="g6.2xlarge",
        market_type="on-demand",
        subnet_id="subnet-061a363014b302012"
    )

  #### 2. Check Instance Status:

    status_ec2(
        instance_id="i-02a8827553ac72dea"
    )

  #### 3. Stop the Instance (Non-destructive):

    stop_ec2(
        instance_id="i-02a8827553ac72dea"
    )

Model LifeCycle Managment via MCP

The get_help MCP tool summarizes the other MCP tools:

> get_help
  I will execute the updated get_help tool to retrieve the server's documentation and operational summary.
● Bash(python3 -c "import asyncio; from server import get_help; print(asyncio.run(get_help()))") (ctrl+o to expand)
  Here is the help dashboard returned by the get_help tool:

  ### 🛠️ AWS/GCP Gemma 4 SRE Agent Help & Configuration

  AWS Configuration:

  • AWS_REGION : The AWS Region for EC2/EKS deployment.
      • Current Value: us-east-1 
  • AWS_BUCKET_NAME : S3 Bucket used to store model weights.
      • Current Value: vllm-models-bucket 

  General serving:

  • MODEL_NAME : Default Hugging Face repository or path.
      • Current Value: google/gemma-4-12B-it-qat-w4a16-ct 
  • VLLM_BASE_URL : The explicit URL of your vLLM service. (If not set, it is auto-discovered via EC2 tags or
  Cloud Run)
      • Current Value: Not set (auto-discovering) 

  ### ℹ️ Active Mode Summary

  The server is running in AWS mode.
  ──────
  ### 🧰 Available MCP Tools

  #### 🐳 Infrastructure & Deployment

  • start_ec2 : Starts an existing stopped EC2 instance, or provisions a new one (with NVIDIA L4 GPU) if none
  exists.
  • status_ec2 : Checks the state, type, public IP, DNS, and launch details of EC2 instances.
  • stop_ec2 : Safely stops active EC2 instances without deleting the root EBS volumes.
  • check_vllm : Checks the status of the vLLM container and engine running on the EC2 instance(s).
  • deploy_vllm : Deploys vLLM to AWS EC2 g6.2xlarge or GCP Cloud Run GPU.
  • destroy_vllm : Cleans up the vLLM Docker container on the AWS EC2 instance without terminating it, or deletes
  the Cloud Run vLLM service.
  • status_vllm : Checks the status of the AWS EC2 instance or Cloud Run vLLM service.
  • update_vllm_scaling : Scales EC2 instance type vertically or updates Cloud Run min/max instances.
  • get_vllm_deployment_config : Generates the AWS EC2 / GCP deployment command and user data.
  • get_vllm_gpu_deployment_config : Generates an AWS EKS nodegroup config or GKE manifest for GPU (NVIDIA L4).
  • check_gpu_quotas : Checks GPU/Accelerator quotas for an AWS or GCP region.

  #### 📊 Model Management

  • list_vertex_models : Lists models in the Vertex AI Registry.
  • list_bucket_models : Lists model weights in S3 or GCS bucket.
  • save_hf_token : Securely saves a Hugging Face API token to AWS Secrets Manager or Secret Manager.
  • get_vertex_ai_model_copy_instructions : Instructions to copy model from Vertex AI Model Garden to GCS.
  • get_huggingface_model_copy_instructions : Instructions to download model from Hugging Face and upload to
  S3/GCS.
  • get_huggingfacehub_download_path : Resolves local cache path using huggingface_hub.

  #### 📊 Monitoring & Status

  • get_metrics : Fetches raw Prometheus metrics from the running vLLM service's /metrics endpoint.
  • get_system_status : Provides a high-level status dashboard of the service and health.
  • get_endpoint : Verifies connectivity and returns the active service URL.
  • get_model_details : Retrieves detailed model metadata and engine state from /v1/models .
  • verify_model_health : Deep health check by querying the model with a simple prompt and measuring latency.

  #### 📈 Performance & Benchmarking

  • run_benchmark : Runs performance/concurrency benchmark sweeps against the vLLM GPU endpoint.

  #### 💬 Interaction & Diagnostics

  • query_gemma4 : Primary tool to query the self-hosted model with standard chat message format.
  • query_gemma4_with_stats : Queries the model and returns streaming performance statistics (TTFT, throughput).
  • query_vllm : Direct text completions querying tool.
  • analyze_cloud_logging : Fetches logs from AWS CloudWatch or GCP Logging and analyzes them using the model.
  • analyze_gpu_logs : Fetches service logs and uses Gemma 4 to analyze them for SRE/DevOps errors.
  • suggest_sre_remediation : Suggests remediation plans for SRE errors using the model.

Getting Started with Gemma 4 on GPU

The Official vLLM repo also has Gemma4 specific information:

Releases · vllm-project/vllm

The Gemma 12B model was just released:

Introducing Gemma 4 12B: a unified, encoder-free multimodal model

What about the QAT Models?

A deep dive into deploying the 12B QAT model is here:

12B Gemma 4 QAT Deployment with NVIDIA L4, Cloud Run, MCP, and Antigravity CLI

Lets Deploy this!

The model was deployed to a g6.2xlarge backed with a L4 NVIDIA :

> status_ec2

  I will check the status of our active EC2 instance running the vLLM engine.

● Bash(./save-aws-creds.sh && python3 -c "import asyncio; from server import status_ec2; print(st...) (ctrl+o to
expand)

  Here is the status of the EC2 instances under the service tag gpu-12b-qat-l4-devops-agent :

  ### 🖥️ Active Instance

  • Instance ID: i-0f06fa6a51c9a2ca1 
      • Type: g6.2xlarge (NVIDIA L4 GPU)
      • State: running 
      • Public IP: 44.204.128.2 
      • Public DNS: ec2-44-204-128-2.compute-1.amazonaws.com 
      • Launch Time: 2026-06-15 21:08:48+00:00

Now check the Docker Container:

check_vllm

  Here is the status of the vLLM server environment on AWS EC2.

  ### 🖥️ AWS EC2 & vLLM Status Report

  For the service tag gpu-12b-qat-l4-devops-agent :

  • EC2 Instance ID: i-0f06fa6a51c9a2ca1 
  • Instance Type: g6.2xlarge (NVIDIA L4 GPU)
  • State: running 
  • Public IP: 44.204.128.2 
  • Public DNS: ec2-44-204-128-2.compute-1.amazonaws.com 
  • Launch Time: 2026-06-15 21:08:48+00:00 
  • vLLM API Endpoint Health ( http://44.204.128.2:8080/health ): Healthy ✅ (returned HTTP 200 OK)
  • AWS Systems Manager (SSM) Status: Pending agent handshake ( InvalidInstanceId returned from SendCommand ,
  which is standard during the initial boot phase while the SSM agent establishes registration).

Cross Check The Deployed Model

Once the model starts locally- the MCP tool allows for model verification:

> verify_model_health

  The health verification of the self-hosted model has successfully passed:

  ### 🩺 Model Health Check: PASSED ✅ 

  • Endpoint URL: http://44.204.128.2:8080/v1 
  • Active Model: google/gemma-4-12B-it-qat-w4a16-ct 
  • Sample Response:
  │ "Yes, I am working and ready to help! How can I ass..."

  • Response Latency: 0.73 seconds

and model details:

  The active model running on the vLLM server is:

  ### 🧩 Active Model details

  • Endpoint URL: http://44.204.128.2:8080 
  • Model Name: google/gemma-4-12B-it-qat-w4a16-ct 
  • Provider/Owner: vllm 
  • Health Status: Healthy ✅

    [
      {
        "id": "google/gemma-4-12B-it-qat-w4a16-ct",
        "object": "model",
        "owned_by": "vllm"
      }
    ]

and the endpoint:

  The active endpoint verified and online is:

  🟢 vLLM is Online at: http://44.204.128.2:8080

and status:

  The high-level system status dashboard has been retrieved and verified:

  ### 🌀 GPU vLLM System Status

  • vLLM Health: 🟢 Online ( http://44.204.128.2:8080 )
  • Hosting Status: 🟢 Running ( i-0f06fa6a51c9a2ca1 )

and metrics:

  The raw metrics were successfully retrieved from the active vLLM service ( http://44.204.128.2:8080/metrics ).
  Here are some of the key SRE metrics extracted from the payload:

  ### 📊 Key Performance Metrics

  • Active Model: google/gemma-4-12B-it-qat-w4a16-ct 
  • Total HTTP Request Latency (Sum): ~2.88 seconds (for 5 processed requests)
  • Engine Tool Parser Invocations: 0 invocations recorded so far for tool_call parsing (meaning no streaming
  tool calls have been executed yet in this session).

  For complete metrics visualization, you can scraping the endpoint directly at http://44.204.128.2:8080/metrics .

Check the EC2 Instance

Once Deployed — the model is visible on the AWS Console:

Review the Model

Check the model details:

> what are the model details?

  Based on the active deployment context and the agent's specifications, here are the detailed specifications for
  the running model:

  ### 🧩 Model Specifications & Architecture

   Parameter | Configuration | Details
  ----------------------------|--------------------------------------|---------------------------------------------
   Model Name | google/gemma-4-12B-it-qat-w4a16-ct | Gemma 4 12B Instruction-tuned checkpoint.
   Quantization Method | QAT (Quantization-Aware Training) | Quantized during training to minimize
                              | | precision loss; uses 4-bit weights with 16- 
                              | | bit activations ( w4a16 ).
   Serialization Format | compressed-tensors ( -ct ) | Requires native vLLM compressed-tensors
                              | | parsing.
   Base Precision ( --dtype ) | bfloat16 | Mandatory. Gemma 4 is natively trained in   
                              | | bfloat16 . FP16 can cause numerical
                              | | overflow.
  ──────
  ### 🚀 Recommended vLLM Runtime Parameters

  To run this model stably on a single NVIDIA L4 GPU (24 GB VRAM), the following serving arguments are applied:

  • --quantization compressed_tensors : Mandatory for loading the w4a16 CT checkpoint format.
  • --max-model-len 32768 : Caps the max KV cache sequence length.
  • --kv-cache-dtype fp8 : Quantizes the KV cache to 8-bit precision, cutting context memory footprints in half to
  scale concurrent users.
  • --gpu-memory-utilization 0.95 : Allocates 95% of L4 VRAM to vLLM's cache manager.
  • --tensor-parallel-size 1 : Fits entirely on a single L4 GPU.
  • --disable-chunked-mm-input : Prevents multi-modal fragmentation and preserves tool-calling parsing stability.
  • --tool-call-parser gemma4 & --reasoning-parser gemma4 : Essential for interpreting structure-reasoning steps
  and function calls.
  ──────
  ### 📊 Performance & Memory Footprint Insights

  • VRAM Optimization: A standard 12B bfloat16 model takes up most of the 24 GB VRAM on an L4 GPU, leaving
  virtually no VRAM for KV caches (limiting concurrency to ≤8 users).
  • The QAT Advantage: Compressing the weights to 4-bit frees up ~18 GB of VRAM exclusively for KV cache. This
  enables the server to scale to 512 concurrent users with a 100% request success rate for context sizes up to 2048
  tokens.

  On a single NVIDIA L4 GPU (24 GB VRAM), the memory allocation breakdown differs significantly between the
  Standard (bfloat16) and QAT (4-bit quantized) versions of the Gemma 4 12B model:

  ### 📊 VRAM Memory Breakdown: Standard vs. QAT (12B)

   Memory Segment | Standard ( bfloa… | QAT ( w4a16 / IN… | Purpose & Impact
  ----------------|-------------------|--------------------|-------------------------------------------------------
   Model Weights | ~24 GB | ~6 GB | VRAM reserved exclusively for the static network
                  | | | weights.
   KV Cache Space | ~0 GB | ~18 GB | VRAM reserved for dynamic context tokens
                  | | | (keys/values) generated by active users.
   Total L4 VRAM | 24 GB | 24 GB | Physical hardware limit of the NVIDIA L4 GPU.
  ──────
  ### 💡 Core SRE Implications of the Breakdown

  1. The Standard VRAM Cliff:
      • Because the Standard 12B ( bfloat16 ) weights consume almost the entire 24 GB of VRAM, there is virtually 0
      GB left for vLLM's KV cache.
      • This results in severe request throttling, high latency, and request dropping at concurrency levels above 8
      (success rate drops to 81.3% at concurrency 16 and 45% at concurrency 64).
  2. The QAT Advantage:
      • By compressing the weights to 4-bit, the 12B QAT ( w4a16-ct ) model fits in just ~6 GB of VRAM.
      • This leaves ~18 GB of VRAM completely free for vLLM's cache manager to allocate to active user sessions,
      permitting 100% request success rates up to 512 concurrent users for short context windows, or up to 64
      concurrent users with large context windows.

Once everything is online — you can send queries to the Model:

> query_gemma4_with_stats what is Gemma?

  I will execute the query_gemma4_with_stats tool programmatically to query the active model and retrieve latency
  and throughput metrics.

● Bash(python3 -c "import asyncio; from server import query_gemma4_with_stats; print(asyncio.run(...) (ctrl+o to
expand)

  I have started running the model query in the background. I will present the model's answer along with the
  streaming/generation statistics as soon as it completes.

  The query has completed successfully! Here is the model's response and generation performance stats:

  ### 📊 Performance Stats

  • Model: google/gemma-4-12B-it-qat-w4a16-ct 
  • Time to First Token (TTFT): 0.122s 
  • Total Generation Time: 10.042s 
  • Throughput: 28.03 tokens/s 
  • Total Tokens: ~278 tokens 
  ──────
  ### 💬 Model Response

  │ Gemma is a family of lightweight, state-of-the-art open models developed by Google DeepMind. These models are
  │ built using the same research, technology, and infrastructure used to create the Gemini models.
  │
  │ Here are the key characteristics of the Gemma family:
  │
  │ • Open Weights: Unlike "closed" models, Gemma is an open-weights model. This means that developers and
  │ researchers can customize, fine-tune, and deploy the models on their own infrastructure for a variety of
  │ applications.
  │ • Efficiency: Gemma models are designed to be smaller and more efficient than the massive Gemini models. They
  │ are optimized to run on accessible hardware, such as laptops, workstations, and desktops, while still
  │ maintaining high performance.
  │ • Varied Sizes: The family includes different sizes (such as 2B, 7B, 9B, and 27B) to suit different
  │ needs—ranging from on-device tasks to more complex cloud-based processing.
  │ • Modalities: While the primary versions are text-based, the family is evolving to include different
  │ capabilities (like audio and image processing).
  │
  │ My relationship to Gemma:
  │ I am Gemma 4, the latest iteration in this family, developed by Google DeepMind. I am an open weights model
  │ capable of understanding text and images and generating text responses.

Benchmark the Local Model

The local model was benchmarked:

The benchmark sweep has completed successfully. Here are the compiled results across all combinations of context
  window sizes and concurrency levels:

Summarize the results:

> summarize aws_benchmark_report_summary.md

  Here is a summary of the key findings and performance characteristics documented in the
aws_benchmark_report_summary.md report:

  ### 🚀 Performance Summary for Gemma 4 12B QAT on AWS EC2 ( g6.2xlarge )
  • Concurrency Sweet Spot: The NVIDIA L4 GPU handles concurrent loads exceptionally well. The system maintains a
  100.0% request success rate up to 1024 concurrent users across all context window sizes (from 4 up to 16K tokens).
  • Peak Throughput: The maximum request throughput achieved is 93.3 Requests/sec (at a context window of 8 tokens
  under a 1024 concurrent user load.
  • Stress Performance (2048 Users): At the extreme stress limit of 2048 concurrent users, the success rate only
  drops to ~61.8% for the largest 16K context window, while remaining at 100% for smaller context windows (≤32
  and 256 concurrent users). For larger context windows (like 16K), it still achieves a strong 19.9 Requests/sec
  tokens).
  • Latency Scaling (1K Context Window): Average response latency scales predictably under parallel load:
      • 1 user: 0.33 seconds 
      • 64 users: 1.49 seconds 
      • 512 users: 9.93 seconds 
      • 1024 users: 19.90 seconds

This is visualized:

GPU Memory Allocation of QAT 12B Models

The L4 provides 24 GB of GPU Memory:

> what is the memory breakdown on the GPU?

  Here is the exact VRAM memory allocation breakdown for the NVIDIA L4 GPU (24 GB VRAM) when serving the Gemma 4   
  12B QAT model ( w4a16-ct format) via vLLM:

  ### 📊 VRAM Memory Breakdown (24 GB Total)

   VRAM Allocated | Percentage | Usage / Description
  ----------------|------------|-----------------------------------------------------------------------------------
   ~6.0 GB | 25% | Model Weights: Static footprint for the INT4 quantized weights (uses 4-bit
                  | | weights with 16-bit activations).
   ~16.8 GB | 70% | KV Cache: Dynamic memory allocated to store user context keys and values
                  | | (optimized using --kv-cache-dtype fp8 quantization).
   ~1.2 GB | 5% | CUDA Kernels & Execution Overhead: Framework execution overhead, driver
                  | | reservation, and workspace memory.
  ──────
  ### 💡 Why this breakdown matters: Standard vs. QAT

  • Standard 12B ( bfloat16 ): The weights alone require ~24 GB of VRAM, completely exhausting the GPU memory. This
  leaves 0 GB for the KV Cache, causing severe request dropping under any concurrent load.
  • QAT 12B ( w4a16 ): Saving ~18 GB of memory on the weights enables vLLM to assign that space to the KV Cache. By
  utilizing FP8 cache quantization, the 16.8 GB allocation is sufficient to support 1,024 concurrent users with a
  100% success rate.

What are the performance characteristics of the Model on L4?

> what are the performance characteristics of the deployment

▸ Thought for 2s, 351 tokens
  Analyzing Deployment Performance
  The performance characteristics of the active Gemma 4 12B QAT deployment on AWS EC2 (NVIDIA L4 GPU) are
  summarized below:

  ### ⚡ Generation & Responsiveness Metrics

  • Time to First Token (TTFT): 0.122 seconds (extremely fast start-of-response latency).
  • Single-Stream Generation Speed: 28.03 tokens/second .
  • Peak Server Throughput: 93.3 requests/second (achieved at a context size of 8 tokens with 256 concurrent
  users).
  ──────
  ### 📈 Concurrency & Scalability Scaling

  The deployment utilizes vLLM's FP8 KV cache allocation to scale concurrency efficiently:

  • 100.0% Success Rate limit: Sustained all the way up to 1,024 concurrent users (regardless of whether the
  input is a tiny 4-token prompt or a massive 16,384-token log context).
  • Graceful Degradation (2048 Users): At 2,048 users, the server remains operational:
      • Context size ≤32 tokens: 100.0% Success Rate 
      • Context size 16K tokens: 61.8% Success Rate (graceful failure under extreme VRAM pressure).

  ──────
  ### 🕒 Latency Scaling Under Parallel User Loads

  Average latencies for key context sizes across different concurrent user volumes:

   Context Window Size | Low Load (1–8 Users) | Medium Load (16–128 Users) | Heavy Load (512–1024 Users)
  ----------------------------|-------------------------|----------------------------|-----------------------------
   Small (128 Tokens) | 0.11s - 0.34s | 0.50s - 2.53s | 9.61s - 18.77s 
   Medium (1024 Tokens) | 0.14s - 0.37s | 0.53s - 2.70s | 9.93s - 19.90s 
   Large / Logs (8192 Tokens) | 0.26s - 0.58s | 0.76s - 3.60s | 11.86s - 23.43s 
   Max (16K Context) | 0.31s - 0.62s | 1.06s - 4.45s | 14.35s - 26.98s

One last cross check — what are the trade-offs of the deployment?

> what are the trade-offs on the L4 deployment?

▸ Thought for 3s, 498 tokens
  Analyzing L4 Deployment Trade-offs
  Here are the key engineering and architectural trade-offs involved in our NVIDIA L4 GPU deployment for the Gemma
  4 SRE agent:

  ### 1. Model Precision vs. Concurrency Capacity (The QAT Trade-off)

  • The Gain: Compressing the model weights to 4-bit (via QAT) reduces the weight footprint from ~24 GB to ~6 GB.
  This frees up 18 GB of VRAM for the KV cache, which boosts concurrent user capacity by 64x (from ≤8 users to 512 +
  users).
  • The Loss: There is a minor mathematical accuracy penalty compared to serving the full-precision bfloat16 
  model. However, since the model was trained using Quantization-Aware Training (QAT) rather than post-training
  quantization, the loss in reasoning and function-calling capability is minimal.
  ──────
  ### 2. Dedicated VM (AWS EC2) vs. Serverless Container (GCP Cloud Run)

  • Performance vs. Management Overhead:
      • AWS EC2 ( g6.2xlarge ): Delivers twice the concurrency limit (1024 vs. 512 users) and 2.3x higher peak
      throughput (93 req/s vs. 40 req/s) due to dedicated networking and CPU. However, it requires active OS,
      Docker, and security patching.
      • GCP Cloud Run: Fully managed and handles SSL certificates out of the box, but has a 13% pricing premium for
      always-on setups and lower peak network throughput.
  • The Scale-to-Zero Illusion: GCP Cloud Run's main cost benefit is scaling to zero. However, because vLLM takes 3
  to 5 minutes to initialize memory and load weights, scaling to zero causes unacceptable cold starts for active
  SRE workflows, making an always-on warmed instance ( min-instances 1 ) mandatory on both platforms.
  ──────
  ### 3. KV Cache Quantization ( fp8 Cache vs. Precision)

  • The Gain: Quantizing the key-value cache to 8-bit ( --kv-cache-dtype fp8 ) cuts context window memory
  footprints in half, allowing the server to handle 1024 concurrent users.
  • The Loss: Extreme context sizes (≥8K tokens) can experience minor attention-mechanism precision degradation,
  which can occasionally affect retrieval accuracy over very large log dumps.
  ──────
  ### 4. Hardware Capacity Ceilings (Single L4 Limit)

  • The Gain: The L4 GPU is highly cost-effective (~$0.98/hr on AWS), making it ideal for budget-conscious DevOps
  stacks.
  • The Loss: You are capped at a 12B parameter model size. Larger, more powerful models like the Gemma 4 26B or   
  31B cannot be served on a single L4 GPU due to the physical 24 GB VRAM limit, requiring vertical hardware scaling
  to multi-GPU nodes (e.g. 8 × L4 or EKS nodegroups), which increases compute costs by 4x to 8x.

Any other Quant options?!?!

> are there any further quant options? will 4 bit work on the kv cache?

▸ Thought for 3s, 343 tokens
  Exploring Quantization Options
  I will search the latest vLLM documentation to verify support and stability for 4-bit KV cache quantization.

● WebSearch(vllm 4-bit kv cache quantization support) (ctrl+o to expand)

  Here is the status of further quantization options and how they apply to both model weights and the KV cache on
  our vLLM server:

  ### 1. Will 4-bit Work on the KV Cache? (No, for Production)

  • vLLM Support Status: No. Official vLLM stable releases do not support 4-bit KV cache quantization. The runtime
  only supports FP8 ( fp8 ) and FP16/BF16 ( auto ).
  • Why 4-bit KV Cache is Impractical:
      1. Severe Accuracy Degradation: Compressing dynamic attention keys and values down to 4-bit introduces high
      numerical error, causing severe degradation in model reasoning, logic over long context windows, and
      structured tool calling.
      2. Kernel Limitations: 4-bit KV cache requires complex mathematical transformations (like Hadamard
      transforms) and custom CUDA kernels. Currently, there are no stable, hardware-accelerated 4-bit attention
      kernels in vLLM that outperform standard FP8.
      3. Hardware Optimization: Datacenter GPUs like the NVIDIA L4 have native hardware acceleration specifically
      designed for FP8 math, making FP8 the optimal sweet spot for performance and precision preservation.

So where is the best place to run Gemma? AWS or GCP?

Surprise!

> compare the performance on AWS vs GCP

▸ Thought for 4s, 725 tokens
  Comparing Cloud Performance
  I will compile a side-by-side performance comparison between the AWS EC2 ( g6.2xlarge ) and GCP Cloud Run
  deployments of Gemma 4 12B QAT on the NVIDIA L4 GPU. I'll save this as a permanent artifact.
● Create(/home/xbill/.gemini/antigravity-cli/brain/...93bb-2517db1f3c9a/gcp_vs_aws_comparison.md) (ctrl+o to expand)

  I have compiled a detailed, side-by-side performance comparison report between AWS EC2 ( g6.2xlarge ) and GCP
  Cloud Run.

  The complete comparison analysis has been saved as an artifact:
  👉 gcp_vs_aws_comparison.md

  ### 📈 Major Highlights of the Comparison

  1. Concurrency Capacity (AWS Wins 2x):
      • AWS EC2: Sustains a 100% success rate up to 1,024 concurrent users across all context windows.
      • GCP Cloud Run: Sustains a 100% success rate up to 512 concurrent users; success rates begin dropping at 1,
      024 users.
  2. Peak Request Throughput (AWS Wins 2.3x):
      • AWS EC2: Achieves a maximum peak of 93.3 req/s (at 8 tokens, 256 users).
      • GCP Cloud Run: Capped at 40.3 req/s (at 8 tokens, 1,024 users) due to serverless ingress network routing
      limits.
  3. Heavy Context Scaling (AWS Wins 3.2x):
      • Under maximum 16K context load at 1,024 users, AWS EC2 yields 19.9 req/s compared to GCP's 6.1 req/s.
      • At 2,048 users, GCP success rate plunges to ~22% under the memory strain, while AWS EC2 preserves a much
      higher 61.8% success rate.

What about costs?!?

The serving costs were summarized:

> what is the cost to run this on AWS with the current config
  I will check the current on-demand pricing for the AWS EC2 g6.2xlarge instance in us-east-1 .

● WebSearch(AWS EC2 g6.2xlarge hourly price us-east-1) (ctrl+o to expand)

▸ Thought for 3s, 450 tokens
  Calculating Total Cost Breakdown
  Here is the estimated cost breakdown for running the current Gemma 4 SRE serving configuration on AWS in the us-
  east-1 (N. Virginia) region:

  ### 💰 AWS Cost Breakdown: g6.2xlarge Configuration

  This setup deploys the model on a g6.2xlarge instance (1 x NVIDIA L4 GPU, 8 vCPUs, 32 GiB RAM) with a 150 GiB  
  gp3 EBS boot volume.

   Cost Category | Rate / Metric | Estimated Cost (Hourly) | Estimated Cost (Monthly - 24/
  ---------------------------|----------------------------|-------------------------|------------------------------
   Compute ( g6.2xlarge ) | On-Demand (Linux) | $0.9776 / hr | $713.65
   Storage (150 GiB gp3) | $0.08 per GB-month | $0.0164 / hr | $12.00
   Total Active Running Cost | Combined Compute + Storage | ~$0.994 / hr | ~$725.65
  ──────
  ### 💡 Cost-Saving Recommendations & Alternatives

  1. Stop Instances When Not In Use:
      • By using the stop_ec2 tool, compute billing stops completely. You only pay for the gp3 storage ($12.
      00/month) while the instance is offline.
      • Developer Workday Profile (8 hrs/day, 22 days/month = 176 hrs):
          • Compute: 176 hrs × $0.9776 = $172.06
          • Storage: $12.00
          • Total: ~$184.06 / month (a ~74% saving).

  2. Spot Instances (Up to 45% savings):
      • If workloads can tolerate short interruptions, you can provision Spot instances in us-east-1 for ~$0.52  
      to $0.71 / hour (reducing compute costs to ~$380–$518/month for 24/7 workloads).
  3. Savings Plans / Reserved Instances:
      • Committing to a 1-year or 3-year term for steady SRE production pipelines can yield 30% to 50% discount    
      rates on the compute portion.

────────────────────────────────────────────────────────────
> what about a L4 Cloud run on GCP?

  I will search for the specific pricing of running an NVIDIA L4 GPU on GCP Cloud Run.

● WebSearch(GCP Cloud Run L4 GPU pricing hourly) (ctrl+o to expand)

  Here is the cost breakdown for running an NVIDIA L4 GPU on GCP Cloud Run and how it compares to the AWS EC2
  configuration:

  ### 💰 GCP Cloud Run (Gen2) L4 Cost Breakdown

  To deploy an NVIDIA L4 GPU on Cloud Run, GCP requires a minimum configuration of 4 vCPUs and 16 GiB of memory per
  container instance.

   Resource Segment | Rate / Metric | Hourly Cost (Active) | Monthly Cost (24/7 Run)
  ----------------------------|----------------------------|---------------------------|---------------------------
   NVIDIA L4 GPU | Per-second allocation | ~$0.670 / hr | $489.10
   vCPUs (4 Units) | $0.082 per vCPU-hour | ~$0.328 / hr | $239.44
   Memory (16 GiB) | $0.009 per GiB-hour | ~$0.144 / hr | $105.12
   Total Active Cost | Combined GPU + Compute | ~$1.142 / hr | ~$833.66
  ──────
  ### ⚖️ AWS EC2 vs. GCP Cloud Run Cost & Operational Trade-offs

  #### 1. 24/7 Steady-State Production workloads (AWS Wins)

  • AWS EC2 ( g6.2xlarge ): ~$725.65 / month (Compute + Storage).
  • GCP Cloud Run (Min Instances = 1): ~$833.66 / month.
  • Verdict: If you need the model server always online to prevent cold starts, AWS EC2 is about 13% cheaper per
  month on-demand.
  #### 2. Scaling to Zero / Serverless Scale (GCP Wins)
  • GCP Cloud Run ( min-instances 0 ): Pay $0 when idle. Only pay ~$1.14 per hour when requests are actively
  executing.
  • AWS EC2: Compute costs continue to accrue hourly unless you explicitly trigger standard shutdown/startup
  operations.
  • The Cold Start Trap: Because vLLM takes 3 to 5 minutes to initialize weights and reserve VRAM on startup,
  scaling to zero on GCP Cloud Run is generally unusable for low-latency interactive applications, making a minimum
  instance of 1 a practical necessity.

────────────────────────────────────────────────────────────
> compare the two

  Here is the comprehensive comparison of running the Gemma 4 12B QAT SRE agent on AWS EC2 vs. GCP Cloud Run:

  ### ⚖️ The Definitive Comparison: AWS EC2 vs. GCP Cloud Run
  ──────
  ### 1. Performance: AWS Wins 🏆

  • Concurrency Capacity: AWS EC2 maintains a 100% request success rate up to 1024 concurrent users across all
  context window sizes (from 4 to 16K tokens). GCP Cloud Run begins showing request degradation at 512 concurrent  
  users.
  • Peak Throughput: AWS EC2 achieves 93.3 req/s (at 8 tokens, 256 users) compared to GCP Cloud Run's 40.3 req/s
  due to serverless load balancer ingress limits.
  • Heavy Context Workloads (16K tokens / 1024 users): AWS handles heavy memory pressure better, delivering 19.9   
  req/s at 26.98s average latency compared to GCP's 6.1 req/s at 33.78s average latency.
  ──────
  ### 2. Pricing & Cost: AWS Wins (for Production) 🏆

  • Always-On (Production) Cost:
      • AWS EC2 ( g6.2xlarge ): ~$725.65 / month (includes 150 GiB gp3 disk storage).
      • GCP Cloud Run: ~$833.66 / month (vCPU, memory, and L4 GPU combined).
  • The Scale-to-Zero Caveat: While GCP Cloud Run technically supports scaling to zero (costing $0 when idle), a
  cold startup of the vLLM server loading a 12B model takes 3 to 5 minutes. For any live application, you must keep
  at least 1 instance warmed up ( min-instances 1 ), negating this advantage.
  ──────
  ### 3. SRE & Maintenance Overhead: GCP Wins 🏆

  • AWS EC2: Requires OS patch management, security group rules, SSH keys, Docker daemon configurations, and SSM
  agent handshakes.
  • GCP Cloud Run: Fully managed serverless container runtime. Ingress routing, HTTPS certificates, health checking,
  and basic container health monitoring are handled automatically out of the box.
  ──────
  ### 📋 Summary Recommendation

  • Choose AWS EC2 if you are building a production SRE diagnostic platform that demands high-volume log ingestions,
  low query latencies, and high concurrency capability at the lowest cost.
  • Choose GCP Cloud Run if you prioritize zero maintenance overhead and rapid deployment cycles, and are
  comfortable paying a slight premium (~13%) to avoid managing virtual machines.

Summary

The strategy for using MCP for Gemma 4 GPU deployment with Antigravity CLI and AWS was validated with a incremental step by step approach.

DEV Community: Google Developer Experts

12B Gemma 4 Deployment with NVIDIA Blackwell 6000, MCP, Cloud Run, and Antigravity CLI

What is this project trying to Do?

Antigravity CLI

Testing the Antigravity CLI Environment

Full Installation Instructions

Python MCP Documentation

Where do I start?

Setup the Basic Environment

Model Management Tool with MCP Stdio Transport

Running the Python Code

MCP stdio Transport

Antigravity CLI mcp_config.json

Validation with Antigravity CLI

Getting Started with Gemma 4 on GPU

Model Lifecycle Management via MCP

Checking System status

Cross Check The Deployed Model

Check the Cloud Run Service

Review the Model

Benchmark the Local Model

Cost Breakdowns

Summary

[Gemini API in Action] Building MemeFinder: A Native Mac Menu Bar Widget for Finding Memes via Text Using Gemini Vision & Semantic Embeddings

The Origin: Mid-Conversation, Where on Earth Is That Meme?

System Design and Architecture

System Architecture Flow

Core Implementation

1. Auto-tagging memes with the Gemini vision model

2. Hybrid semantic + keyword ranking

Major Pitfalls and Solutions

Pitfall #1: The mysterious GeminiError error 0 — indexing and search both fail

Pitfall #2: SwiftPM's main entry-point conflict and the SwiftUICore linking error

Pitfall #3: Parallel indexing's rate limit and "I want to stop indexing halfway"

Pitfall #4: Evolving from a "windowed app" into "menu-bar resident + global hotkey"

Pitfall #5: The settings form is blank — one symptom, three layers of cause

On the "Development Process" Itself

Results and Benefits

MTP Speculative Decoding with the 12B Gemma 4 QAT Model on NVIDIA L4, Cloud Run, MCP, and…

MTP Speculative Decoding with the 12B Gemma 4 QAT Model on NVIDIA L4, Cloud Run, MCP, and Antigravity CLI

What is this project trying to Do?

Antigravity CLI

Testing the Antigravity CLI Environment

Full Installation Instructions

Python MCP Documentation

Where do I start?

Setup the Basic Environment

Model Management Tool with MCP Stdio Transport

Running the Python Code

Getting Started with Gemma 4 on GPU

Gemma 12B

NVIDIA L4 on Cloud Run

Quantization-Aware Training (QAT)

QAT Model Versions

MCP stdio Transport

Antigravity CLI mcp_config.json

Validation with Antigravity CLI

Checking System status

Cross Check The Deployed Model

Review the Model

Benchmark the Local Model

Cost Analysis

Summary

Chatting with your Data: Conversational Analytics in BigQuery

The Problem with "Text-to-SQL" Toys

The Playlist: Build It Yourself

Skills over System Prompts: Building an Anki Tutor with the Antigravity SDK

What the Antigravity SDK Gives You

Giving the Agent Hands: Anki as Python Tools

The Problem with Giant System Prompts

Skills over System Prompts

review-buddy: the study session playbook

plain-cards: the card-writing style guide

codebase-cards: the repo exploration protocol

One Behavior Layer, Three Surfaces

The Terminal Tutor

The Telegram Tutor

The Deck Builder

Turning It Ambient with Triggers

The Part I Refused to Trust to the Model

Pitfall #1: The mysterious `GeminiError error 0` — indexing and search both fail

Pitfall #2: SwiftPM's `main` entry-point conflict and the SwiftUICore linking error

`review-buddy`: the study session playbook

`plain-cards`: the card-writing style guide

`codebase-cards`: the repo exploration protocol