DEV Community: Remigiusz Samborski

Secure Gemini CLI for Cloud development

Remigiusz Samborski — Fri, 13 Mar 2026 06:01:17 +0000

AI agents are a double-edged sword. You hear horror stories of autonomous tools deleting production databases or purging entire email inboxes. These risks often lead users to require manual confirmation for every agent operation. This approach keeps you in control but limits the agent's autonomy. You will soon find yourself hand-holding the agent and hindering its true capabilities. You need a way to let the agent run in "yolo mode" without risking your system.

In this blog you will learn how to secure your Gemini CLI in a way that will allow you to run it in an isolated environment with limited GitHub and Google Cloud access while not worrying that it will do too much damage if things go wrong. We will follow the least privilege pattern to make sure Gemini CLI has all necessary permissions to build your project, but at the same time can’t access systems it shouldn’t touch.

The Sandbox premise

The solution consists of following components:

Using GitHub fine-grained personal access tokens - limits source control risks.
Google Cloud service account - limits cloud risks.
Docker - limits local system risks.
Session limits - avoid surprises with the number of used tokens (especially important when running in --yolo mode).

Following this approach will protect you from the 'helpful agent curse' - it’s a situation when the agent tries very hard to achieve a task by finding ways around blockers. Examples include: granting itself more permissions, copying files to the current folder to edit them, and many more.

GitHub fine-grained personal access tokens

First let’s limit agent’s GitHub exposure by leveraging the fine grained tokens:

Navigate to GitHub Settings > Developer Settings > Personal access tokens > Fine-grained tokens.
Click Generate a new token.
Provide a descriptive name for your token and consider using expiration date to force rotations on a regular basis.
Restrict Repository access to the specific target repo you are working on.
Grant Read and Write permissions for Contents.
Save the token locally by running export GITHUB_TOKEN="github_pat_...".

Google Cloud Service Account

Create an isolated Service Account (SA) with minimal permissions. This prevents the agent from accessing protected resources and other projects.

Run these commands after updating YOUR_PROJECT_ID and roles below:

# Set your project ID
export CLOUDSDK_CORE_PROJECT="YOUR_PROJECT_ID"
gcloud config set project $CLOUDSDK_CORE_PROJECT

# Create the Service Account
gcloud iam service-accounts create gemini-cli-sa \
    --description="Isolated account for Gemini CLI"

# Grant minimal roles (adjust roles as needed)
gcloud projects add-iam-policy-binding $CLOUDSDK_CORE_PROJECT \
   --member="serviceAccount:gemini-cli-sa@$CLOUDSDK_CORE_PROJECT.iam.gserviceaccount.com" \
    --role="roles/aiplatform.user"

# Generate the JSON key file
gcloud iam service-accounts keys create sa-key.json \
    --iam-account=gemini-cli-sa@$CLOUDSDK_CORE_PROJECT.iam.gserviceaccount.com

Hint: you can use the IAM roles and permissions index page to easily find the roles to grant.

A good practice is to use a dedicated project for each of your AI coding initiatives. This way you can run several agents in parallel. They will build different solutions without worrying about stepping on each other's toes.

Custom Docker Build

The Gemini CLI uses a sandbox image to isolate the execution environment. You must customize this image to install gcloud, terraform, vim and set git configuration.

Prepare the Dockerfile

Create a .gemini directory in your project, and inside it, create a sandbox.Dockerfile. Using this specific file name allows Gemini CLI to automatically detect and build your custom sandbox profile if you’re running it from source and you can also use it to build the image manually if you’re running a binary installation.

Paste this content in the .gemini/sandbox.Dockerfile:

# Start from the official Gemini CLI sandbox image with proper version
ARG GEMINI_CLI_VERSION 0.33.0
FROM us-docker.pkg.dev/gemini-code-dev/gemini-cli/sandbox:${GEMINI_CLI_VERSION}

# Switch to root to install system dependencies (gcloud)
USER root

# Install Google Cloud SDK, Git, and prerequisites
RUN apt-get update && apt-get install -y curl apt-transport-https ca-certificates gnupg git && \
    echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | tee -a /etc/apt/sources.list.d/google-cloud-sdk.list && \
    curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key --keyring /usr/share/keyrings/cloud.google.gpg add - && \
    apt-get update && apt-get install -y google-cloud-cli

# Install Terraform
RUN apt-get update && apt-get install -y wget lsb-release && \
    wget -O- https://apt.releases.hashicorp.com/gpg | gpg --dearmor | tee /usr/share/keyrings/hashicorp-archive-keyring.gpg > /dev/null && \
    echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(grep -oP '(?<=UBUNTU_CODENAME=).*' /etc/os-release || lsb_release -cs) main" | tee /etc/apt/sources.list.d/hashicorp.list && \
    apt-get update && apt-get install -y terraform

# Install vim
RUN apt-get install -y vim

# Switch back to the non-root user (the official sandbox image uses 'node' as the default user)
USER node
WORKDIR /workspace

# Configure Git to use the injected GitHub PAT at runtime
RUN git config --global credential.helper '!f() { echo "username=x-access-token"; echo "password=$GITHUB_TOKEN"; }; f'

Prepare docker for building images (optional MacOS step)

If you haven’t built any Docker images before then run following commands to prepare your environment with brew:

# Install dependencies
brew install docker colima docker-buildx

# Configure docker-buildx
mkdir -p ~/.docker/cli-plugins
ln -sfn $(brew --prefix)/opt/docker-buildx/bin/docker-buildx ~/.docker/cli-plugins/docker-buildx

# Start colima service
brew services start colima

# Update DOCKER_HOST (you might want to add this line to .bash_profile):
export DOCKER_HOST="unix://${HOME}/.colima/default/docker.sock"

Build the image (binary installation)

If you installed Gemini CLI with npm, brew or any other binary method then you will need to manually build the Docker image and tag it as a default one that Gemini CLI is looking for:

# Get the base name the CLI looks for
export IMAGE_BASE_NAME="us-docker.pkg.dev/gemini-code-dev/gemini-cli/sandbox"

# Get your currently installed Gemini CLI version (e.g., 0.33.0)
export GEMINI_CLI_VERSION=$(gemini --version)

# Combine them
export IMAGE_NAME="${IMAGE_BASE_NAME}:${GEMINI_CLI_VERSION}"

# Build your custom sandbox image
docker build \
  --build-arg GEMINI_CLI_VERSION=$GEMINI_CLI_VERSION \
  -t "${IMAGE_NAME}" \
  -f .gemini/sandbox.Dockerfile .

Important: this image will be tagged with the exact version of the Gemini CLI you use. This means it needs to be rebuilt every time you update the CLI. I keep the above code in a shell script to run it after every update.

Build the image (source installation)

If you’re running your Gemini CLI from source as explained here. You can trigger the image build automatically each time you start gemini.

First update the top part of your sandbox.Dockerfile by substituting the FROM line with following:

# Start from the official Gemini CLI sandbox image (source installation)
FROM gemini-cli-sandbox

Start Gemini CLI in the sandbox mode

First set couple very important environment variables:

# Export the necessary environment variables
export GITHUB_TOKEN="github_pat_..."
export GEMINI_API_KEY="your-api-key"
export CLOUDSDK_CORE_PROJECT="YOUR_PROJECT_ID"
export GEMINI_SANDBOX=docker

# We keep the ENV variables for our dynamic credentials
export SANDBOX_FLAGS="\
-e GITHUB_TOKEN=${GITHUB_TOKEN} \
-e CLOUDSDK_CORE_PROJECT=${CLOUDSDK_CORE_PROJECT} \
-e CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE=$(pwd)/sa-key.json"

Note: You can put the above variables in a shell script to speed up starts in the future. Just make sure to update your .gitignore to keep it and sa-key.json from getting added to your repository.

Now you can start your Gemini CLI with following command:

# For binary installation
gemini

# For source installation
BUILD_SANDBOX=1 gemini

Session limits

To avoid surprises with the number of tokens that Gemini CLI uses in your session, you can use the Max Session Turns in /settings or your ~/.gemini/settings.json:

Congratulations

Congratulations 🚀You’re ready to validate your setup.

Validation and "Ultimate Tests"

Once the environment is launched within a sandbox we should verify the security boundaries.

First let’s run the /about command to see if we’re running within a sandbox. You should see something like this:

Now let’s try to break out from our new sandbox.

GitHub privilege escalation

Try asking Gemini CLI to access a private repo it shouldn’t have access. Example prompt:

Clone https://github.com/USER_NAME/PRIVATE_REPOSITORY to a new folder

You should see how Gemini CLI really tries and struggles to get access. Mine got really creative at trying to access the repo with git, gh, curl and even tried to reuse the GITHUB_TOKEN manually. All these tries failed and this error was displayed:

Google Cloud privilege escalation

Ask the agent to list all compute instances:

List all my compute instances

It should fail due to missing permissions on your restricted Service Account. Gemini CLI tries really hard and executes couple different commands including reauthentication, but it fails at the end:

Local privilege escalation

Finally let’s try to access a file from another project folder by prompting:

There are other projects in the folder above the current one. List them and let me know if there is anything that is interesting from hacker's perspective.

I am starting to feel sorry for the poor agent 😉 Once again it can’t complete its task:

Conclusions

Now that you have validated your sandbox setup you should feel much more confident to run gemini --yolo and streamline your work as Gemini CLI delivers your code without hand-holding and pesky Can I execute this command? Prompts.

I am looking forward to all the creative ideas you’ll bring to life!

What's next?

If you find this setup useful here are some additional steps to consider:

Try out Gemini CLI Conductor Extension - it’s very powerful and can significantly help you run autonomous agents effectively. Here is a deep dive into some of the advantages.
Read my Antigravity the Ralph Wiggum style which covers sandboxing for Antigravity.
Add emojis to this post to help others find it.
Share this post with your friends on socials.
Connect with me via LinkedIn, X or Bluesky.

Thanks for reading.

Agent Factory Recap: Antigravity and Nano Banana Pro with Remik

Remigiusz Samborski — Mon, 02 Feb 2026 12:18:03 +0000

In Episode #17 of the Agent Factory podcast, we step away from the purely theoretical and get our hands dirty with the latest developer tools from Google. Together with Vlad we take a deep dive into Antigravity and Nano Banana Pro, demonstrating how to build AI agents that bridge the gap between code generation and high-fidelity media.

This post guides you through the key ideas from our conversation. Use it to quickly recap topics or dive deeper into specific segments with links and timestamps.

Introductions

Antigravity - a new agentic mission control

Timestamp: 02:02

Antigravity is Google’s new agent-first development application. It is designed as a multi-window IDE that uses Gemini 3 under the hood to manage complex, asynchronous coding tasks. Unlike a standard text editor, Antigravity features a dedicated Agent Manager view where developers can interact with agents through planning modes, artifact reviews, and built-in browsers for live UI testing.

Nano Banana Pro and why it’s so special

Timestamp: 03:47

The model behind our slide generator is Nano Banana Pro (the technical name is Gemini 3 Pro Image model). What sets Nano Banana Pro apart is its ability to "think" before it creates. It uses Google Search grounding to retrieve real-time data, like current weather or live stock charts, and integrates that information into the generated image.

The Factory Floor

Agent architecture

Timestamp: 06:41

We dove into the architecture for building the slide generator agent using a combination of the Agent Development Kit (ADK), Antigravity, and Nano Banana Pro. Moreover we explained how these components interconnect to support the flow from a user prompt to a high-fidelity presentation by leveraging a Model Context Protocol (MCP) server.

Agent Starter Pack

Timestamp: 10:04

We started by introducing Agent Starter Pack and using it to spin up a new ADK project.

Vibe Coding a Slide Generator Agent

Timestamp: 14:56

Using Antigravity’s "Always Review" mode, we submitted a prompt to build a completely new slides agent which will use a MCP server to create detailed images.

We highlighted how Antigravity handles tasks and implementation plan reviews, allowing the developer to modify the agent's proposed steps before it executes them. We also dove into automated testing with Antigravity’s web browser plugin functionality.

Results

Timestamp: 32:25

The slides generator agent successfully called the MCP tools to generate a series of images stored in a Cloud Storage bucket, presenting the final links directly in the UI. Antigravity reviewed the results and to our surprise immediately suggested additional improvements.

Wrap up section

We are moving into an era where building an agent isn't just about the LLM; it's about the orchestration of tools, planning, and high-fidelity output. Antigravity is our agent manager and Nano Banana Pro acts as an asset creator.

We can’t wait to see what our viewers create next with these powerful tools!

Resources & links

Antigravity → https://goo.gle/3XMvGwf
Nano Banana Pro → https://goo.gle/3KMmKUG
Agent starter pack → https://goo.gle/4oLfypT
Agent Development Kit (Python) → https://goo.gle/4p0Pszm
GitHub with demo code → https://goo.gle/48N83IN

Connect with us

Remik → LinkedIn, X, GitHub
Vlad → LinkedIn, X, GitHub

Antigravity the Ralph Wiggum style

Remigiusz Samborski — Mon, 02 Feb 2026 12:05:20 +0000

The Ralph Wiggum trend has been surfacing across social platforms lately. If you're tracking current tech developments, it’s hard to miss. Named after a persistent and slightly confused second-grader, the Wiggum Loop agentic development boils down to: Don't stop until the job is done.

In traditional AI coding, the agent performs a task, stops, and waits for you to approve its next step or request changes. In a Wiggum Loop, you give the agent a mission and success criteria (like passing tests), and it keeps looping, fixing its own bugs and refactoring - until it hits the green light.

The recent excitement around the Wiggum Loop agentic development highlights a powerful shift: achieving autonomous, self-correcting development. I've been leveraging a similar approach effectively with Antigravity for some time already. In this post, I’ll share my strategy, enabling you to implement true unsupervised development yourself.

Going "Full Wiggum"

To achieve true unsupervised development, we need to move away from the review-driven defaults and let the agent take the wheel. Antigravity is uniquely built for this because it's an agent-first environment capable of acting in both the terminal and the browser.

To mirror the “Bash loop" persistence of the Ralph Wiggum plugin, configure your Antigravity settings as follows:

Mode: Select Agent-driven development. This shifts the agent from a "wait for instructions" assistant to a "goal-oriented" architect.
Terminal execution policy: Set to Always Proceed. This allows the agent to run npm test, uv run pytests, and other commands without constantly pausing for approval.
Review policy: Set to Always Proceed. This tells the agent that its implementation plans are pre-approved.
JavaScript execution policy: Set to Always Proceed. This is essential for agents that need to run scripts or interact with browser environments to verify their work.

WARNING: THE SANDBOX IS NOT OPTIONAL. Running an agent in "Always Proceed" mode is like giving Bart Simpson a slingshot in front of a mirror store. Only do this in a sandbox environment.

Here is a great article from my colleague which shows a step-by-step guide to setting such an environment up and running on Cloud Workstation.

Example

To see this in practice, I ran the following prompt against Antigravity:

Build a REST API for todos in NodeJS.

When complete:
- All CRUD endpoints are working
- Input validation is in place
- Tests are passing (coverage > 80%)
- README with API docs exists

The screencast below shows how Antigravity handled the task without my interruptions (I spent this time on other tasks rather than handholding the agent):

How does this work?

Antigravity isn't just looping in a vacuum. Because it has native hooks into Gemini 3 Pro, it utilizes a massive context window that remembers exactly why a previous command failed.

It kicks things off by drafting up an implementation plan and a task list. In the video, you can watch it tick through these items in real time. It doesn't just plan, though - it actually touches the terminal to initialize the npm project and run tests.

The loop only closes once every requirement is met and the test suite hits green. It then provides a handy walkthrough so you can easily understand the architecture it just spun up.

This approach turns development from writing code into verifying outcomes.

From vibe-coding to vibe-building

The Ralph Wiggum trend isn't about cutting corners; it's about embracing sheer, stubborn persistence through automation. By letting Antigravity operate autonomously, you transition from a coder to an architect and team lead. You define the standards and environment, while agents manage the iterative grind of writing, testing, and debugging cycles that typically consume a developer's valuable time.

Are you brave enough to let the agent "Always Proceed"? Visit Antigravity’s download page to start experimenting yourself.

Other resources

Let’s Connect!

I’d love to hear how you’re using Antigravity for your agentic workflows. Are you building Wiggum loops or keeping a tighter leash on your agents?

Connect on LinkedIn
Follow me on X
Catch me on Bluesky

Serverless AI: EmbeddingGemma with Cloud Run

Remigiusz Samborski — Thu, 25 Sep 2025 09:29:59 +0000

Building on the previous blog post about running Qwen3 Embedding models on Cloud Run, this article focuses on the recently released EmbeddingGemma model from the Gemma family. Discover how to leverage the same powerful serverless techniques to deploy this model on Google Cloud's serverless platform.

You will learn how to:

Containerize the embedding model with Docker and Ollama
Deploy the embedding model to Cloud Run with GPUs
Test the deployed model from a local machine

Before we dive into the code, let's briefly discuss the core components that power this serverless AI solution.

EmbeddingGemma Model

According to the EmbeddingGemma model card:

“EmbeddingGemma is a 308M parameter multilingual text embedding model based on Gemma 3. It is optimized for use in everyday devices, such as phones, laptops, and tablets. The model produces numerical representations of text to be used for downstream tasks like information retrieval, semantic similarity search, classification, and clustering.”

Its optimization for efficiency makes EmbeddingGemma an ideal candidate for serverless deployment on Cloud Run, ensuring high performance and cost-effectiveness for your AI applications.

Cloud Run

Cloud Run is a managed compute platform on Google Cloud that lets you run containerized applications in a serverless environment. Think of it as a middle ground between a simple function-as-a-service (like Cloud Run Functions) and a more customizable GKE cluster. You give it a container image, and it handles all the underlying infrastructure, from provisioning and scaling to managing the runtime.

The beauty of Cloud Run is that it can automatically scale to zero, meaning when there are no requests, you aren't paying for any resources. When traffic picks up, it quickly scales up to handle the load. This makes it perfect for stateless models that need to be highly available and cost-effective.

Deployment

Let's walk through the deployment process step-by-step.

Prepare the environment

First lets configure the gcloud CLI environment.

Note: if you do not have gcloud CLI installed please follow instructions available here.

Step 1 - Set your default project:

gcloud config set project PROJECT_ID

Step 2 - Configure Google Cloud CLI to use the _europe-west1 _region for Cloud Run commands:

gcloud config set run/region europe-west1

Important: at the time of writing, GPUs on Cloud Run are available in several regions. To check the closest supported region please refer to this page.

Containerize

Now we will use Docker and Ollama to run the EmbeddingGemma model. Create a file named Dockerfile containing:

FROM ollama/ollama:latest

# Listen on all interfaces, port 8080
ENV OLLAMA_HOST=0.0.0.0:8080

# Store model weight files in /models
ENV OLLAMA_MODELS=/models

# Reduce logging verbosity
ENV OLLAMA_DEBUG=false

# Never unload model weights from the GPU
ENV OLLAMA_KEEP_ALIVE=-1

# Store the model weights in the container image
ENV MODEL=embeddinggemma:latest
RUN ollama serve & sleep 5 && ollama pull $MODEL

# Start Ollama
ENTRYPOINT ["ollama", "serve"]

Build and Deploy

We will now use Cloud Run's source deployments. This allows you to achieve the following with one command:

First, compile the container image from the provided source.
Next, upload the resulting container image to an Artifact Registry.
Then, deploy the container to Cloud Run, ensuring that GPU support is enabled using the --gpu and --gpu-type parameters.
Finally, redirect all incoming traffic to this newly deployed version.

You just need to run:

gcloud run deploy embedding-gemma \
  --source . \
  --concurrency 4 \
  --cpu 8 \
  --set-env-vars OLLAMA_NUM_PARALLEL=4 \
  --gpu 1 \
  --gpu-type nvidia-l4 \
  --max-instances 1 \
  --memory 32Gi \
  --no-allow-unauthenticated \
  --no-cpu-throttling \
  --no-gpu-zonal-redundancy \
  --timeout=600 \
  --labels dev-tutorial=blog-embedding-gemma

Note the following important flags in this command:

--concurrency 4 is set to match the value of the environment variable OLLAMA_NUM_PARALLEL.
--gpu 1 with --gpu-type nvidia-l4 assigns 1 NVIDIA L4 GPU to every Cloud Run instance in the service.
--max-instances 1 specifies the maximum number of instances to scale to. It has to be equal to or lower than your project's NVIDIA L4 GPU quota.
--no-allow-unauthenticated restricts unauthenticated access to the service. By keeping the service private, you can rely on Cloud Run's built-in Identity and Access Management (IAM) authentication for service-to-service communication.
--no-cpu-throttling is required for enabling GPU.
--no-gpu-zonal-redundancy set zonal redundancy options depending on your zonal failover requirements and available quota.

Test the deployment

Upon successful deployment of the service, you can initiate requests. However, direct api calls will result in an HTTP 401 Unauthorized response from Cloud Run.

This behaviour follows Google’s “secure by default” approach. The model is intended for calls from other services, such as a RAG application, and therefore is not open for public access.

To support local testing of your deployment, the simplest approach is to launch the Cloud Run developer proxy using the following command:

gcloud run services proxy embedding-gemma --port=9090

Afterwards, in a second terminal window, run:

curl http://localhost:9090/api/embed -d '{
  "model": "embeddinggemma",
  "input": "Sample text"
}'

The response will look similar to this:

You can also use Python to call the endpoint. Example:

from ollama import Client

client = Client(host="http://localhost:9090")

response = client.embed(model="embeddinggemma", input="Sample text")
print(response)

Congratulations 🎉 The Cloud Run deployment is up and running!

RAG Example

You can use the newly deployed model to build your first RAG application. Here’s how to achieve this:

Step 1 - Generate Embeddings

Start with required dependencies:

pip install ollama chromadb

Create an example.py file containing:

import ollama
import chromadb

documents = [
    "Poland is a country located in Central Europe.",
    "The capital and largest city of Poland is Warsaw.",
    "Poland's official language is Polish, which is a West Slavic language.",
    "Marie Curie, the pioneering scientist who conducted groundbreaking research on radioactivity, was born in Warsaw, Poland.",
    "Poland is famous for its traditional dish called pierogi, which are filled dumplings.",
    "The Białowieża Forest in Poland is one of the last and largest remaining parts of the immense primeval forest that once stretched across the European Plain.",
]

client = chromadb.Client()
collection = client.create_collection(name="docs")

ollama_client = ollama.Client(host="http://localhost:9090")

# Store each document in a in-memory vector embeddings database
for i, d in enumerate(documents):
    response = ollama_client.embed(model="embeddinggemma", input=d)
    embeddings = response["embeddings"]
    collection.add(ids=[str(i)], embeddings=embeddings, documents=[d])

Step 2 - Retrieve

Next, with the following code you can search the vector database for the most relevant document (add it to your example.py):

# An example question
question = "What is Poland's official language?"

# Generate an embedding for the input and retrieve the most relevant document
response = ollama_client.embed(model="embeddinggemma", input=question)
results = collection.query(query_embeddings=[response["embeddings"][0]], n_results=1)
data = results["documents"][0][0]

Step 3 - Generate Final Answer

In this final step step we will use a locally installed Gemma3.

_Note: We use Gemma3 in the generation step, but any other model could work here (e.g., Gemini, Qwen3, Llama, etc.). Nevertheless, it is critical to use the same embeddings model in Step 1 (Generate Embeddings) and Step 2 (Retrieve). _

To install the Gemma3:latest model run:

ollama pull gemma3

Now can combine user’s prompt with search results and generate the final answer (add this code to example.py):

# Final step - generate a response combining the prompt and data we retrieved in step 2
prompt = f"Using this data: {data}. Respond to this prompt: {question}"

output = ollama.generate(
    model="gemma3",
    prompt=prompt,
)

print(f"Prompt: {prompt}")
print(output["response"])

Run the code:

python example.py

The answer should look similar to the one below:

Prompt: Using this data: Poland's official language is Polish, which is a West Slavic language.. Respond to this prompt: What is Poland's official language?
Poland's official language is Polish. It's a West Slavic language.

You have successfully created and run your first RAG application using the EmbeddingGemma model.

Summary

At this point, you have successfully established a Cloud Run service running the EmbeddingGemma model, ready to generate embeddings for semantic search or RAG applications.

This method also allows you to deploy and compare multiple embedding models on Cloud Run (e.g. Qwen3 Embedding or other Ollama-supported models), enabling you to find the best fit for your specific use case without major code changes.

Ready to build your own serverless AI applications?

Start building on Cloud Run today and explore its full potential!
If you’re interested in learning more about RAG evaluation, this article is a good starting point.

Thanks for reading

If you found this article helpful, please consider following me here and giving it a clap 👏 to help others discover it.

I'm always eager to chat with fellow developers and AI enthusiasts, so feel free to connect with me on LinkedIn or Bluesky.

Serverless AI: Qwen3 Embeddings with Cloud Run

Remigiusz Samborski — Wed, 20 Aug 2025 10:31:23 +0000

In this blog post I’ll show you the process of deploying the Qwen3 Embedding model to Cloud Run with GPUs for enhanced performance.

You will learn how to:

Containerize the embedding model with Docker and Ollama
Deploy the embedding model to Cloud Run with GPUs
Test the deployed model from a local machine

Before we jump into the code a couple words about key components of the solution.

Qwen3 Embedding Model

The Qwen3 Embedding series is a set of open-source models for text embedding and reranking, built on the Qwen3 Large Language Model (LLM) family. It's designed for retrieval-augmented generation (RAG), a technique that enhances the output of large language models by retrieving relevant information from a knowledge base, and other tasks requiring semantic search. You can learn more about embeddings in this video.

Open embedding models such as Qwen3 are the ideal choice when you need greater control, specialization, and security than proprietary, "black-box" APIs can offer. They are particularly well-suited for the following use cases:

Fine-Tuning for Niche Domains📻: by fine-tuning them on specialized data (e.g., legal contracts, medical research, internal company wikis) they can provide more accurate results for semantic search and RAG than a general-purpose model.
Data Privacy & Security🔒: open models can be self-hosted or deployed to cloud resources managed by your organization. This ensures compliance with regulations like GDPR and prevents data from ever leaving your control.
Cost-Effectiveness at Scale💰: for high-volume tasks, running an optimized open model can be cheaper than paying per-API-call fees to a proprietary service provider.
Offline & Edge Deployment🛜: open models can run locally and are perfect for applications that must function without an internet connection, such as on-device search in mobile apps or analysis on remote IoT devices.

I chose the Qwen3-Embedding-4B model due to its growing popularity and suitable size for the Cloud Run environment. However, you can experiment with different sizes (0.6B, 4B, and 8B) depending on your specific use case.

Cloud Run

Cloud Run is a managed compute platform on Google Cloud that lets you run containerized applications in a serverless environment. Think of it as a middle ground between a simple function-as-a-service (like Cloud Functions) and a more complex GKE cluster. You give it a container image, and it handles all the underlying infrastructure, from provisioning and scaling to managing the runtime.

Deployment

But enough with the intros, let's get our hands dirty with some code 🧑‍💻

Below is a step by step instruction on how to get the Qwen3 Embedding model up and running.

Prepare the environment

First we need to configure the gcloud CLI environment.

Note: if you don’t have gcloud CLI installed please follow instructions available here.

Step 1 - Set your default project:

gcloud config set project PROJECT_ID

Step 2 - Configure Google Cloud CLI to use the _europe-west1 _region for Cloud Run commands:

gcloud config set run/region europe-west1

Important: at the time of writing, GPUs on Cloud Run are available in several regions. To check the closest supported region please refer to this page.

Containerize

We will use Docker and Ollama to run the Qwen3 Embedding model. Create a file named Dockerfile and put the following code inside it:

FROM ollama/ollama:latest

# Listen on all interfaces, port 8080
ENV OLLAMA_HOST=0.0.0.0:8080

# Store model weight files in /models
ENV OLLAMA_MODELS=/models

# Reduce logging verbosity
ENV OLLAMA_DEBUG=false

# Never unload model weights from the GPU
ENV OLLAMA_KEEP_ALIVE=-1

# Store the model weights in the container image
ENV MODEL=dengcao/Qwen3-Embedding-4B:Q4_K_M
RUN ollama serve & sleep 5 && ollama pull $MODEL

# Start Ollama
ENTRYPOINT ["ollama", "serve"]

Build and deploy

Next it’s time to leverage the power of Cloud Run’s source deployments. With a single command you can:

Build the container image from source (note the –source parameter in the command below)
Upload the container image to an Artifact Registry
Deploy the container to Cloud Run with GPUs enabled (note --gpu and --gpu-type options)
Redirect all traffic to the new deployment

To do all the above, you just need to run:

gcloud run deploy ollama-qwen3-embeddings \
  --source . \
  --concurrency 4 \
  --cpu 8 \
  --set-env-vars OLLAMA_NUM_PARALLEL=4 \
  --gpu 1 \
  --gpu-type nvidia-l4 \
  --max-instances 1 \
  --memory 32Gi \
  --no-allow-unauthenticated \
  --no-cpu-throttling \
  --no-gpu-zonal-redundancy \
  --timeout=600 \
  --labels dev-tutorial=blog-qwen3-embeddings

Note the following important flags in this command:

--concurrency 4 is set to match the value of the environment variable OLLAMA_NUM_PARALLEL.
--gpu 1 with --gpu-type nvidia-l4 assigns 1 NVIDIA L4 GPU to every Cloud Run instance in the service.
--max-instances 1 specifies the maximum number of instances to scale to. It has to be equal to or lower than your project's NVIDIA L4 GPU quota.
--no-allow-unauthenticated restricts unauthenticated access to the service. By keeping the service private, you can rely on Cloud Run's built-in Identity and Access Management (IAM) authentication for service-to-service communication.
--no-cpu-throttling is required for enabling GPU.
--no-gpu-zonal-redundancy set zonal redundancy options depending on your zonal failover requirements and available quota.

Test the deployment

Now that you have successfully deployed the service, you can send requests to it. However, if you send a request directly, Cloud Run will respond with HTTP 401 Unauthorized. This is intentional, because we want our model to be called from other services, such as a RAG application, and not accessible by everyone on the Internet.

The easiest way to test the deployment from a local machine is to spin up the Cloud Run developer proxy by executing:

gcloud run services proxy ollama-qwen3-embeddings --port=9090

Now in a second terminal window run:

curl http://localhost:9090/api/embed -d '{
  "model": "dengcao/Qwen3-Embedding-4B:Q4_K_M",
  "input": "Sample text"
}'

You should see a response similar to this:

You can also call the endpoint from a Python client. Example:

from ollama import Client

client = Client(host="http://localhost:9090")

response = client.embed(model="dengcao/Qwen3-Embedding-4B:Q4_K_M", input="Sample text")
print(response)

Congratulations 🎉 Your Cloud Run deployment is up and running!

RAG Example

You can use the newly deployed model to build your first RAG application. Here’s how to achieve this:

Step 1 - Generate Embeddings

Install necessary dependencies:

pip install ollama chromadb

Create an example.py with the following content:

import ollama
import chromadb

documents = [
    "Poland is a country located in Central Europe.",
    "The capital and largest city of Poland is Warsaw.",
    "Poland's official language is Polish, which is a West Slavic language.",
    "Marie Curie, the pioneering scientist who conducted groundbreaking research on radioactivity, was born in Warsaw, Poland.",
    "Poland is famous for its traditional dish called pierogi, which are filled dumplings.",
    "The Białowieża Forest in Poland is one of the last and largest remaining parts of the immense primeval forest that once stretched across the European Plain.",
]

client = chromadb.Client()
collection = client.create_collection(name="docs")

ollama_client = ollama.Client(host="http://localhost:9090")

# Store each document in a in-memory vector embeddings database
for i, d in enumerate(documents):
    response = ollama_client.embed(model="dengcao/Qwen3-Embedding-4B:Q4_K_M", input=d)
    embeddings = response["embeddings"]
    collection.add(ids=[str(i)], embeddings=embeddings, documents=[d])

Step 2 - Retrieve

Next, the following code will search the vector database for the most relevant document (add it to your example.py):

# An example prompt
prompt = "What is Poland's official language?"

# Generate an embedding for the input and retrieve the most relevant document
response = ollama_client.embed(model="dengcao/Qwen3-Embedding-4B:Q4_K_M", input=prompt)
results = collection.query(query_embeddings=[response["embeddings"][0]], n_results=1)
data = results["documents"][0][0]

Step 3 - Generate final answer

In the generation step we will use a locally installed Qwen3:0.6b.

Note: we use Qwen3 in generation step, but any other model could work here (i.e. Gemini, Gemma, Llama, etc.). Nevertheless it’s critical to use the same embeddings model in step 1 (Generate Embeddings) and step 2 (Retrieve).

You can install the Qwen3:0.6b model by running the following command:

ollama pull qwen3:0.6b

Now we’re ready to combine user’s prompt with search results to generate the final answer (add to example.py):

# Final step - generate a response combining the prompt and data we retrieved in step 2
output = ollama.generate(
    model="qwen3:0.6b",
    prompt=f"Using this data: {data}. Respond to this prompt: {prompt}",
)

print(output["response"])

Run the code by executing:

python example.py

You should see an answer similar to the one below:

<think>
Okay, the user is asking what Poland's official language is, and they provided the information that Poland's official language is Polish, which is a West Slavic language. Let me make sure I understand this correctly.

First, I need to confirm if that's the correct information. I know that Poland is a country in Eastern Europe, and its official language is Polish. But wait, what's the source of this information? The user hasn't provided any other data, so I should stick strictly to the given information.

I should state that Poland's official language is Polish, and that it's a West Slavic language. I need to present this clearly and concisely. Maybe mention that it's the official language to emphasize its significance. Also, check if there's any other detail that needs to be included, but since the user provided only this, I can proceed.
</think>

Poland's official language is **Polish**. This language is a **West Slavic language**.

Well done! You have just created and run your first RAG application with Qwen3 Embedding model under the hood.

Summary

At this point you have established a Cloud Run service running Qwen3 Embedding model. You can use it to generate embeddings for a semantic search or a RAG application.

Stay tuned for more content around leveraging Qwen3 Embedding in your applications.

Thanks for reading

I hope this article inspired you to experiment with open embedding models on Cloud Run. If you found this article helpful, please consider following me here and giving it a clap 👏 to help others discover it.

I'm always eager to chat with fellow developers and AI enthusiasts, so feel free to connect with me on LinkedIn or Bluesky.

Polish Large Language Model na Google Cloud (video)

Remigiusz Samborski — Tue, 25 Mar 2025 09:20:00 +0000

Kontynuuję moją przygodę z polskim dużym modelem językowym (PLLuM) na Google Cloud! Tym razem oddaję w Wasze ręce nowy film, który krok po kroku pokaże Wam, jak przygotować i uruchomić ten model na Vertex AI.

Zapraszam do oglądania i komentowania:

Polish Large Language Model (PLLuM) on Google Cloud

Remigiusz Samborski — Mon, 17 Mar 2025 08:44:00 +0000

"Wpadła śliwka w .... Google Cloud" 😉

Recently, thanks to the Ministry of Digital Affairs, there's been a lot of buzz about the new Polish Large Language Model (PLLuM). I decided to play around with it a bit and show others how to run it on Google Cloud using Vertex AI.

I invite you to check out this notebook, which will guide you through this process step by step.

Let me know in the comments what applications you see for this new open model.

Deploying a Gemini-powered Mesop app to Cloud Run

Remigiusz Samborski — Mon, 17 Feb 2025 10:47:01 +0000

In the first video I showed how to create a Gemini-powered Mesop app.

In the follow-up video I focus on deploying it to Cloud Run. I hope you find it interesting:

If you have questions or feedback, feel free to reach me out.

Unleash the power of Gemini! ✨

Remigiusz Samborski — Thu, 13 Feb 2025 11:23:56 +0000

I recently released hands-on video tutorial which shows how to build a Gemini-powered application using the Mesop framework.

Learn the steps to integrate with the Gemini model and get your project ready for deployment.

Code is available at GitHub.

Stay tuned for the next video where we'll deploy it to Google Cloud.

Great read!

Remigiusz Samborski — Fri, 07 Feb 2025 09:23:30 +0000

Leverage open models like Gemma 2 on GKE with LangChain

Olivier Bourgeois for Google Cloud ・ Feb 6 '25

#kubernetes #ai #langchain #googlecloud