DEV Community: Google Cloud

Inference on GKE Private Clusters

Maciej Strzelczyk — Thu, 12 Mar 2026 12:52:00 +0000

Setting up inference service without access to Internet

Deploying an inference service on your GKE cluster in 2026 is a fairly simple task. With a short Deployment definition making use of a vLLM image (TPU or GPU) and a Service definition, you have the basic setup ready to go! vLLM grabs the model of your choosing from Hugging Face during its startup. It’s all nicely automated. However, this setup requires your GKE nodes to have access to the Internet. What should you do when there’s no Internet connection? I will discuss the options in this article, but first, let’s start with a short analysis of how and why you may want to have no Internet connection for your nodes.

GKE Private Nodes

One situation where your vLLM pod might not be able to download a model from the Internet is when you decide to use GKE Private Cluster. When you choose this option, the nodes in your cluster are assigned only a private IP from your VPC network. With only a private IP address, it’s impossible to reach them from outside of your network, but they also lose the default way to communicate with the outside world. This feature is great for increasing the security of your system, but it has obvious drawbacks, like this lack of connectivity to the world.

One easy solution to the private nodes situation is to configure Cloud NAT for the region your cluster is in. That will create a way for the nodes and pods running on them to access the Internet, while keeping them protected from any attempt to establish new connections from outside of the network. However, if you want your pods to be unable to connect to the Internet, we need another way to get the model for vLLM to run.

Providing images to the pods

One other problem you might encounter when choosing to use Private Cluster without access to the Internet is the fact that your nodes won’t have access to the default source of Docker images: Docker Hub. The simple vllm/vllm-openai:latest image specification will not work. You will need to copy the images you want to use to the Artifact Registry—this way GKE Nodes will be able to download the images and run them. This gives you additional control over your environment; you can carefully control which versions of the images to download and allow cluster users to use.

Providing the LLM

vLLM can run a model stored in a local directory if you pass it as the --model argument value. To make use of this ability in your private GKE cluster, you will have to somehow provide the model to the vLLM through a mounted directory. The easiest way to do this is through GCS FUSE, which allows you to simply mount a GCS bucket as a folder in your Pod. You just need to remember that:

The GKE Cluster must have the GcsFuseCsiDriver add-on enabled.
You should use Workload Identity and a dedicated service account to allow the pod to access the bucket. The roles/storage.objectViewer role should work just fine for read-only access.
It’s important to host the model in the same region as the nodes of your cluster to ensure the fastest transfers.

Serving LLMs from a mounted directory speeds up the startup process of your inference service, as it doesn’t have to download the model each time a new pod is started.

Alternative to mounting GCS Bucket - persistent disks

An alternative to mounting a bucket is to use a zonal or regional persistent disk or hyperdisk. A single disk can be mounted by multiple pods at once if using read-only mode. Creating a disk to store a model is a bit more time consuming than using a GCS bucket, but might provide better performance (depending on the disk type) and be cheaper, as GCS and disk billing is structured differently.

To create a disk storing a model, you will need a temporary Compute Instance, where you will mount, format and fill the disk with data (hf download works just fine for this). Once the disk is ready, the VM can be deleted and the disk attached to the vLLM pods.

Summary

Using GKE without Internet access can be a good practice, providing you with additional security and control. As you can see, the additional work required to get your inference service running in this case is not negligible, but it is also not a deal-breaker. It’s up to you to decide if it’s a configuration you would like to use in your setup. Using a GCS Bucket or persistent disk to store a model is also a very good idea to simply cut down on the startup time of your services, especially with larger models.

The ecosystem of AI is changing at a rapid pace and it’s important to stay up to date with all the latest news. Follow the official Google Cloud blog, Google Developers blog and Google Cloud Tech YouTube channel to not miss any updates!

AI deployment: to host or not to host?

Maciej Strzelczyk — Tue, 10 Mar 2026 23:28:46 +0000

So you’ve built your AI application prototype. You used your own local GPU to run the AI model, or just used the free AI Studio tier to power your clever program. The app is ready, the world is ready, time to deploy your production instance! In the case of traditional, non-AI powered apps and services, the choice of deployment platform is based on personal preference, what you are familiar with, how much control over fine details you want to have etc. Cost is usually not the most important factor, as for a new service, that’s just going to start gaining a userbase, the first usage bills won’t be that high anyway. The situation is different when it comes to running services that make use of AI. Here, you need to make two separate decisions. First is how to deploy your application, this is the same as for a vanilla non-AI app. Second is how you are going to provision the AI capabilities. This second decision will most likely be responsible for a big chunk of your bill and it shouldn’t be made without proper consideration. In this article, I will try to help you make the right decision for your use case.

Serverless vs hosted inference service

There are two ways of provisioning AI for a production-grade application:

Serverless - where you pay for the tokens your application sends and receives.This is sometimes called Model as a Service (MaaS). In Google Cloud, this approach is available in Vertex AI and Google AI Studio (Gemini API).
Hosted - where you pay for the time you use the infrastructure running an LLM. In Google Cloud, this model is available through multiple services like: Compute (through certain machine types), Vertex AI, GKE or Cloud Run.

Depending on your situation, you may not have an option to choose between the two, because only one would be possible. For example, if you have to use one of the Gemini models, there’s no way to host it yourself and the MaaS (pay per token) approach is the only one available. Similarly, if you have to use a custom model that is not available as a service, you just have to go down the hosted path.

In cases where you do have a choice between the two paths you need to understand how they will affect your budget.

Serverless (pay per token)

Paying only for the tokens your application uses is a fair and easy to understand setup. It works exactly like any other paid service on Google Cloud - you pay for what you use.

Pros:

It scales to zero, when you don’t use the AI,
you don’t have to worry about scaling,
Configuration and maintenance are extremely simple,

Cons:

Less predictable for your budget
You may reach service quota, either when your application experiences a rush-hour or when you reach some total monthly usage quota
In case your application is hacked, your bill might skyrocket
Once your application gets popular, the bill will grow with your active userbase

Hosted (pay per second)

Hosting an LLM on infrastructure that you pay for is extremely predictable cost-wise. As long as you know how long you are going to hold on to that GPU or TPU accelerated instance, you know exactly how much you are going to pay.

Pros:

Extremely predictable cost
Many ways to lower your bill: CUDs, Spot Instances, choosing a cheaper zone or choosing the right instance and/or accelerator type
No quota on how much tokens your application consumes
Full control over hardware and software inference configuration

Cons:

Big initial cost
Doesn’t scale as smoothly as serverless
Configuration and maintenance is more complicated

Couple of considerations

To help you out a bit further, here are some questions you should ask yourself, before deciding on one of the deployment options.

How much traffic do I expect?

With low traffic, the choice is almost obvious - serverless is cheaper and easier. However, as your usage grows, the number of tokens consumed will add up to a considerable amount. In such a case, using a self-hosted solution might save you from unexpected bills at the end of the month.

Am I legally bound to keep user data in certain region?

In some cases, like with medical or financial data, you might be required by local regulations or your own contracts to ensure your user data doesn’t leave a certain location, or will not be sent to service you don’t control. This might be a situation where no matter the cost effectiveness self-hosting an AI model is the only possible solution.

Am I likely to hit the hourly/monthly quota?

All API Services have some usage quotas, that includes AI services. If you expect your application may reach this quota, it’s a big hint that you should consider self-hosting your model.

Mixed-approach

It is also worth noting that you don’t have to limit your architecture to using only one AI Model with one deployment option. Imagine your application offers multiple AI-powered features - some of them might be simple enough for a small model to handle, while others require full power of Gemini. It is perfectly fine to have for example a Gemma 3 running on a VM, handling the easier tasks, while you delegate the harder/bigger tasks to Gemini API.

This is not an irrevocable decision

Even after careful consideration, the decision might still not be a simple one, especially if you’re starting with a new idea and simply don’t know how popular it’ll get. Luckily, with good architecture of your application, it is not that difficult to prepare for changing the AI API endpoint. It’s reasonable to start with a serverless solution, where you will often make great use of the fact that no traffic = zero cost. Once your application takes off and the Vertex AI or AI Studio bill reaches levels comparable to running a self-hosted model, you should reevaluate your situation and perhaps switch to the more predictable approach.

Keep up!

P.S. Did you know that Google Cloud now offers Developer Knowledge API and MCP server that can give your AI Agents access to always up-to-date knowledge straight from the official Google Cloud, Firebase and Android documentation?!

Making Sure Your Prompt Will Be There For You When You Need It

Shawn Jones — Tue, 10 Mar 2026 17:44:00 +0000

At Google, our team (Google Cloud Samples) uses Gemini to produce thousands of samples in batches. In doing so, we've learned that the biggest hurdle isn't the AI, it's our own expectations about these tools. As developers, we are wired for deterministic systems: we call a function and it produces the same result for the same input every time. This predictability allows for standard unit tests.

Large Language Models (LLMs) however, are probabilistic and stochastic. They don't store facts; they store the likelihood of patterns and use a "sophisticated roll of the dice" to choose the next token. This is why the same prompt can yield a “sparkly” ✨ success one minute and a hallucination 🤪 the next. You aren't just testing code anymore; you are forecasting the weather of your system. To move to production, we must build containment structures (like quality gates and evaluators) that make the unpredictability manageable.

LLMs Can Make Mistakes

Trying to make samples in large batches is different from asking for a single sample from a tool like Gemini CLI. When producing many samples at once, we see more mistakes because the statistics catch up with us. A small percentage of bad samples becomes a large number once the overall number of samples gets higher, not unlike issues in manufacturing. Here are some examples of mistakes.

Sometimes we detect code with syntax issues, like the def def snippet below. Python uses only one def keyword to specify the start of a function definition.

def def create_secret_with_expiration(
    project_id: str, location: str, secret_id: str
):

Syntax issues like this can be detected with linting or other build tools. If we detect them in our pipeline, we can just regenerate the sample. Other times the issues are more subtle, like how this JSDoc below is 7 lines away from the function it is documenting, separated from its function by a use statement, imports, and an object instantiation.

/**
 * Get secret metadata.
 *
 * @param projectId Google Cloud Project ID (such as 'example-project-id')
 * @param secretId ID of the secret to retrieve (such as 'my-secret-id')
 */
'use strict';

const {SecretManagerServiceClient} = require('@google-cloud/secret-manager');
const {status} = require('@grpc/grpc-js');

const client = new SecretManagerServiceClient();

async function getSecretMetadata(projectId, secretId) {

Or other times the docstring is incorrect, like how the the docstring below is missing parameters used by the function it documents.

def create_secret_with_notifications(
    project_id: str, location: str, secret_id: str, topics: list[str]
) -> None:
    """Create Secret with Pub/Sub Notifications. Creates a new secret resource
    configured to send notifications to Pub/Sub topics. This enables external
    systems to react to secret lifecycle events.

    Args:
        project_id: The Google Cloud project ID. for example,
            'example-project-id'
        location: The location of the resource. for example, 'us-central1'
    """

Issues don’t always show up directly in code, either. We have Gemini generating build artifacts, like package.json. In the case below, it was so eager to include the gRPC package that it listed the package 3 times in different ways, including one that has been deprecated.

{
  "name": "example",
  "private": true,
  "description": "Google Cloud Platform Code Samples 🎒",
  "dependencies": {
    "@google-cloud/secret-manager": "latest",
    "@grpc/grpc-js": "latest",
    "@grpc/grpc-js": "^1.10.0",
    "grpc": "latest"
  },
  "scripts": {
    "test": "node --test"
  }
}

We have other, more subtle issues as well. Sometimes the code is correct, but not saved with the correct filename or in the correct folder structure. Issues like these lead to more manual evaluation and testing. By iterating on prompts with evaluation we have improved our results.

Prompt Templates as Functional Interfaces

Quality responses are guided by the three elements shown above: the input data, a prompt template, and the LLM itself. As part of prompting for production, we’re evaluating prompt templates, like those created with the dotprompt format. Below is a very simple example of a prompt template in dotprompt. Using the prompt template we can reuse the same prompt text over and over with different inputs. Prompt templates give us a function interface for interacting with the LLM.

---
model: gemini-3-flash-preview
input:
  schema:
    need: string
    language: string
output:
  schema:
    code: string
---

Generate code that satisfies the need of {{ need }} using language {{ language }}.

By using templates, we can run the same logic across hundreds of different inputs to see where the "weather" changes.

We've found that a successful workflow follows these phases:

Build a foundation with Ground Truth
Finding Your Candidate Prompt (Vibe Check)
Statistical Trials – Because Unit Tests Alone Don’t Work

Phase 1: Build a foundation with Ground Truth

In the prompt template world, the template is only part of the picture. We need the input values as well. We also need the matching expected output values. You may say “But this sounds like unit testing!” and you would be right; it is a similar idea. The amount of testing data you need depends on what question you want to answer. If your question boils down to “Is the prompt template bad?” then 5-10 records of input/output test data is good enough. This will help you eliminate a bad prompt template quickly. If your question is more “Will my prompt template work well?” then you need 50 - 100. The more edge cases you can insert into your test data, the better.

Fortunately, we have a golden set of samples we can use as known good testing data. We continue to iterate on our test data while also adding more samples to it.

Phase 2: Finding Your Candidate Prompt (Vibe Check)

Before you share with your team, start experimenting by using a tool like Google AI studio to develop some handmade prompts. Try them with different inputs and outputs. Build an intuition for what works and what doesn’t. Use Gemini to help in your evaluation.

AI Studio’s playground can be very helpful at this stage, including providing structured outputs that can then be used to help plan the outputs used in our dotprompt file. When you feel good about your results, you have anecdotal evidence that your prompt template might work, but not statistical evidence.

Phase 3: Statistical Trials – Because Unit Tests Alone Don’t Work

Does your candidate prompt template work with many different inputs? This is where things get more complex and we move from the familiar deterministic unit testing to probabilistic testing. Because the LLM could answer differently each time, we need to run multiple trials for each input/output test record. But how many is enough? For recent academic work, my previous team ran as many as 128 times per input/output pair for better statistical relevance, but this gets expensive fast. To balance cost, time, and effort, the community consensus is either four and five times per input/output test record. The argument for five over four is that we need an odd number to “break ties.”

But how do you know if the output of your prompt is working well? Use a deterministic metric. In the case of samples, we build the code, we lint it, we apply other static analysis tools, which all provide deterministic review and feedback. Finally, once we have something that passes those quality gates, we perform manual testing and human review. With this many quality gates and a large number of samples, we can begin to rely on the Law of Large Numbers to determine if a prompt template is working and not worry about four or five trials per sample.

Embracing Statistical Techniques For The Best Performance

Beyond prompt templates, we can evaluate other parts of our workflow. The scenarios below show how we can freeze some elements of the workflow while keeping others the same (freeze). We start by listing the question we want to answer and then list which elements to change and which elements to freeze.

How well does my new prompt template work?
1. Change: prompt template
2. Freeze: model, hyperparameters, ground truth input and output
How well does a different model or model version affect the results?
1. Change: model
2. Freeze: hyperparameters, ground truth input and output, prompt template
Is a new input value a useful addition to the ground truth?
1. Change: input value
2. Freeze: model, hyperparameters, ground truth output, prompt template
Is a new output value a useful addition to the ground truth?
1. Change: output value
2. Freeze: model, hyperparameters, ground truth input, prompt template
How will changing the hyperparameter values improve the results?
1. Change: hyperparameter value
2. Freeze: model, ground truth input and output, prompt template

Say a new model version is released and we have results from testing the previous model. We can keep the hyperparameters, ground truth, and prompt template the same as before. Then we change the model in the dotprompt file and rerun our evaluation. Now we have data to decide if we want to use the new model version. Likewise, we can alter the other items in the list above to answer other questions.

We might be able to sidestep the statistical testing by forcing Gemini behave more deterministically. We could set its hyperparameters to their most deterministic values – such as temperature at 0, top-k at 1, top-p at 0, or by using the same seed the same value every time. This creates its own issues, and does not rid us of the need for testing. What if a given prompt’s deterministic response is incorrect every time? How do we automatically correct things for which there are no deterministic tools? We want there to be some degree of creativity and stochasticity in its responses. We want the option of running the generation again with the probability of getting a better response. We embrace this power but we also need to be more statistics-minded about our testing to make sure our prompts are there for us when we need them.

Join the Conversation

I’m curious about what others are doing to help evaluate their prompts and prompt templates.

Are you just starting out? How do you do your vibe checks? How do you test before shipping?
Have you been evaluating prompts for a while? How many times do you evaluate a prompt template before putting it into production? How do you keep time and cost down?
What recommendations do you follow when testing prompts? Do you have sources to share? Can we do this better?
What workflows have you found to work?

Please share in the comments.

A paper on the “Budget 5”: Confidence Improves Self-Consistency in LLMs
Vertex AI’s advice on evaluation datasets
Anthropic’s Writing Effective tools for AI agents
Stanford’s and UC Santa Barbara’s With LIttle Power Comes Great Responsibility about how many NLP studies are underpowered in terms of statistical testing
7 Technical Takeaways from Using Gemini to Generate Code Samples at Scale
How My Team Aligns on Prompting for Production

Thanks to Jennifer Davis, Adam Ross, Nim Jayawardena, and Katie McLaughlin for feedback on this post.

How My Team Aligns on Prompting for Production

Adam Ross — Tue, 17 Feb 2026 16:53:46 +0000

My team at Google is automating sample code generation and maintenance. Part of that is using Generative AI to produce and assess instructional code. This introduces a challenge: How do we trust the system to meet our specific standards, when core components are non-deterministic?

Establishing trust requires isolating and understanding each large language model (LLM) request. We need to know exactly what goes into the model, and a guarantee of what comes out.

This challenge isn't different from other feature development. To succeed, we realized we had to stop treating prompting like chatting or guessing and start treating it like coding.

Natural Language is a Fuzzy Programming Language

Prompting an LLM is effectively "natural language programming": we are programming in English. The problem is that English is not the greatest language for programming. It is ambiguous. It is subjective. It is open to interpretation.

In C++, a missing semicolon breaks the build. In English, a missing comma changes the objective entirely: "I don't know, John" becomes "I don't know John". In a programming language, syntax is binary; it works or it doesn't. In English, the difference between "Ensure variables are immutable" and "Make sure variables never change" might yield different results based on the model’s training data.

When you combine the fuzziness of human language with the "black box" probabilistic processing of an LLM, you face a difficult question: What is the weather going to be today in the land of AI?

To answer that, you have to make the intentions behind your prompts explicit.

Be Efficient with Brains (Pair & Review)

Writing a prompt is an exploratory process of finding words that trigger the best response. However, a single writer is limited by their own understanding. This is risky with LLMs, which are suited for ambiguous problems but require strict guardrails.

Relying on one person to do this creates blind spots. We found that prompt quality benefits from pairing. A diversity of thought helps create a more complete definition of any problem. What one engineer considers a clear instruction, another might see as a loophole. Pairing covers the gaps that a single brain might miss.

Furthermore, you should also review every prompt. This isn't just checking for typos; it’s a logic check. Does this prompt align with business requirements? We’ve found that prompt reviews often uncover disagreements about the requirements themselves, forcing us to align as a team before we ship.

Document "The Why" and manage change

Because English is fuzzy, the intent behind a specific word choice isn't always obvious. Why did we use the passive voice here? Why did we specify "immutable"?

Even well-structured prompts can eventually obscure the original business requirements. As optimizations blur into the general text, we must take every opportunity to document "the why":

Documentation: Avoid relying on prompts as canonical business requirements. Our LLM requests are a combination of system instructions, user input, context, and deterministic post-processing; the prompt is not complete for onboarding a developer.
Comments: Comment on complex prompts just as you would complex code. Spotlight specific constraints or even punctuation to explain the problem they solve. The model is a moving target, so any unintentional changes can make troubleshooting hard.
Commit Messages: Use commit messages as an opportunity to explain what was wrong with the prompt. (for example, fixed: Missing comma lost John)

Separation of Concerns (Use Dedicated Files)

Writing code and writing prompts require distinct mindsets. One focuses on syntax and execution flow; the other on semantics and intent. Embedding long English instructions inside code creates a distraction.

We keep prompts in dedicated files to disentangle application logic from the LLM interaction configuration, which requires frequent tuning.

By treating the prompt as a standalone component, we can prototype and iterate on the LLM behavior independent of the application's control flow. Tools like dotprompt allow us to treat these files as first-class artifacts containing text, model parameters, and schema definitions. This highlights that invoking a model isn't just a function call; it’s an integration with a distinct system that requires its own configuration.

Use Structured Output

To build a reliable tool, you need a bridge between unpredictable LLM output and deterministic computers.

We rely on structured output¹ to guide the model to emit JSON according to a schema. Even if we only need a single field, defining a schema provides a guardrail that helps the model output conform to a shape we can validate programmatically. This is critical for code generation, where models often add unwanted preambles, conversational filler, or inconsistent markdown fences.

If the output doesn't match the schema, we fail fast or retry. This allows us to integrate the LLM output into our process with the same confidence we have in detecting a bad API response.

From Magic to Engineering

Moving from one successful prompting to a reliable system requires acknowledging that prompts are code. You need to manage, review, and test them with the same rigor applied to the rest of your stack. While we are still working on better ways to benchmark quality, treating our prompts as first-class codebase assets is our first step towards building confidence in our AI assisted automation.

Join the conversation

I'm curious how you are handling the fuzzy nature of LLMs in production.

How does your team review and test prompts?
Do you treat prompts as configuration, code, or something else entirely?
Share your workflow in the comments.

The lumberjack paradox: From theory to practice by Jennifer Davis
Google Sandwich Manager, and the hallucinated SDK by Katie McLaughlin and Brian Dorsey

Thanks to Jennifer Davis & Shawn Jones for review and contributions.
Cover Photo by Jeton Bajrami on Unsplash

This links to how structured output works in the Genkit framework, which my team is using. A succinct example. ↩

Why your `curl` logic just bit you 🐾

Jennifer Davis — Mon, 09 Feb 2026 05:01:07 +0000

It’s a common strategy to test a new API directly with curl. It feels intuitive, fast, and removes the overhead of a language runtime. For example, if you are testing out the Google Cloud Logging API, you might start with a simple request to list logs from a Kubernetes container.:

curl -X POST \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  -H "Content-Type: application/json" \
  https://logging.googleapis.com/v2/entries:list \
  -d '{
    "resourceNames": ["projects/your-project-id"],
    "filter": "resource.type=\"k8s_container\""
  }'

The JSON returns perfectly. But then, you move that logic into a Node.js application using the official @google-cloud/logging library.

import { Logging } from "@google-cloud/logging";

export async function readLogsAsync() {
  // Common approach: Initializing the client without explicit auth
  const logging = new Logging();

  const [entries, , response] = await logging.getEntries({
    filter: 'resource.type="k8s_container"',
    resourceNames: ['projects/your-project-id'],
  });

  console.log(entries.length, response.nextPageToken);
}

All of a sudden, the code fails with a cryptic error: Error: Could not load the default credentials.

This happens because of a disconnect between how the gcloud CLI manages session tokens and how the Google Cloud client libraries search for credentials.

While your curl command relies on the explicit token you provided via gcloud auth print-access-token, the client libraries are designed to look for Application Default Credentials (ADC).

Running gcloud auth login authenticates, but it does not create the specific credential file that the Node.js library requires to run locally.

🤫 Authenticate for local development

The most efficient way to solve this is to provide the library with the credentials it is looking for. Run the gcloud auth application-default login command in your terminal.

gcloud auth application-default login

This opens a browser window to authorize your account and saves a JSON file to your local configuration folder. Once this is done, your Node.js code will automatically find these credentials—no code changes required.

🏡 Moving to Production

Once you get your local environment running, you need to think about "productionizing" your code. Hardcoded project IDs and lack of robust error handling are common causes of production outages.

🛡️ Robust Error Handling

In production, your app should handle permission issues or network timeouts gracefully.

try {
  const [entries] = await logging.getEntries({
    filter: 'resource.type="k8s_container"',
    resourceNames: [`projects/${process.env.GOOGLE_CLOUD_PROJECT}`],
    pageSize: 10, 
  });
} catch (error) {
  if (error.code === 7) {
    console.error("Permission Denied: Check your Service Account roles.");
  } else {
    console.error("Unexpected Logging Error:", error.message);
  }
}

🎭 Service Accounts

In production, you shouldn't use your personal user credentials. Use a Service Account with the "Logs Viewer" role.

🌍 Environment Variables

Avoid hardcoding project IDs. Use an environment variable to make your code portable across staging and production environments.

const PROJECT_ID = process.env.GOOGLE_CLOUD_PROJECT || 'your-project-id';
...

export async function readLogsAsync() {
  const logging = new Logging({ projectId: PROJECT_ID });

Structured Logging

Instead of just reading logs, think about how you write them. Using structured JSON logs makes them much easier to query later in the Cloud Logging console.

If you are running on Google Kubernetes Engine (GKE), you don't even need to use the Logging library to write logs. Printing a JSON string to stdout allows GKE to parse your data into searchable fields automatically.

export function logStructuredStatus() {
  const logEntry = {
    severity: "INFO",
    message: "Container health check successful",
    container_info: { version: "v2.1.0", uptime: process.uptime() },
    http_stats: { active_connections: 42 }
  };

  // GKE picks this up and converts it to a structured log automatically
  console.log(JSON.stringify(logEntry));
}

Get started

I'll be sharing a more thorough walk-through on using the native observability features provided with GKE real soon—including how to automate this stuff so you can avoid the manual debugging of credential errors.

In the meantime, dive deeper here:

📖 How Application Default Credentials work – Understand the "magic" behind the lookup.
🛠️ Providing credentials to ADC – Setup guides for every environment.

The lumberjack paradox: From theory to practice

Jennifer Davis — Wed, 19 Nov 2025 00:42:14 +0000

Previously, I shared my thoughts on Neal Sample’s "lumberjack paradox" and the urgent need to build the systems thinkers of tomorrow. I argued that leaders must move beyond simple efficiency and focus on re-engineering the experience (Dr. Gary Klein) and creating context to ensure we don't lose the path to deep expertise.

But what does "leadership as context creator" look like in practice?

For us in Cloud DevRel Engineering, it isn't abstract. It comes down to how we manage the most fundamental unit of our developer experience: the code sample.

As Neal notes, AI will lead to the "industrialization of creativity"—an infinite supply of ideas and code. In this world, the premium shifts to discernment: the ability to distinguish the quality from the mediocre.

But this isn't a choice between the axe (manual craft) and the chainsaw (AI). The modern expert needs both.

If you only have the axe, you are restricted to the problems that fit within manual reach. It is the perfect tool for the campsite, but it cannot clear the forest.
But if you only have the chainsaw, without the judgment to guide it, you are dangerous. You lack the control to distinguish a clean cut from a destructive one.

You need the deep expertise of the axe to get the precise, consistent outcomes from the chainsaw.

From theory to practice: The catalog as ground truth

In my previous post, I mentioned Dr. Richard Cook's work on "building common ground" and Donella Meadows’ warnings about suboptimization.

In Cloud DevRel Engineering, we realized that our code samples are the primary tool for building this common ground. In Dr. Cook’s terms, they form the "Line of Representation"—the tangible surface that connects the human "above the line" to the complex system "below the line."

When a developer (the human) learns a new platform, the sample is their manual for the "axe." When an AI assistant generates a solution, the sample is the training data that guides the "chainsaw."

When we looked at our systems, we saw suboptimization. By treating samples as low-priority content maintained by individual contributors, we created a fractured reality.

We broke the Line of Representation.

We saw this failure hit on two fronts:

We break the human judgment loop: If samples are inconsistent, developers cannot learn "good" from "bad." We fail to re-engineer the experience (Dr. Klein) necessary to build expertise.
We poison the AI well: AI models ingest our official repositories. The AI learns them, scales them, and feeds them back to the user.

We are currently witnessing exactly how this hand-crafted approach fails at scale.

The high cost of "geological strata" in code

Without central standardization, our repositories accumulated "geological strata"—layers of outdated practices—because manual maintenance cannot keep up with language evolution. This makes it hard to know what is correct today.

Node.js' paradigm tax: Our Node.js repositories contain a mix of callbacks, raw promises, and async/await. A user learning Pub/Sub sees one era, while a user learning Cloud Storage sees another. The AI sees all of it and treats it all as valid, stripping away the context of "outdated" versus "modern."
Python: The contributor long tail: With over 650 contributors, our Python samples suffer from extreme fragmentation. The total cost of ownership (TCO) of manually bringing thousands of older snippets up to modern Python 3.10+ standards is astronomically high, so it simply doesn't happen. This leaves a massive surface area of "technical debt" that the AI happily recycles.

Inconsistent quality creates "false best practices"

When samples are hand-written by federated teams, personal "developer flair" masquerades as industry best practice. Users copy-paste these patterns, inadvertently adopting technical debt.

Java's Framework creep: Instead of teaching the core platform, contributors often introduce heavy frameworks for simple tasks. This increases the "time-to-hello-world" and teaches the AI that simple tasks require complex dependencies.
Python vs. Go: Most Go samples handle errors correctly because the language forces it. Many Python samples show only the "happy path," skipping critical distributed systems patterns like exponential backoff or retry logic. The AI then generates code that looks clean but fails in production.

The hidden cost of incoherence

This is the "suboptimization" Donella Meadows warned about. It is not enough for individual samples to be correct in isolation; they must function as a cohesive unit.

For a human developer, shifting between products that use different coding styles creates friction. They have to spend mental energy decoding the "dialect" of a specific product team rather than focusing on the logic.

For an AI, this lack of cohesion is even more dangerous.

The Context Gap: When our samples for Cloud Storage look structurally different from our samples for BigQuery, the AI treats them as unrelated entities. It fails to learn the underlying "grammar" of our platform.
The Integration Failure: When a user asks for a solution that combines these products, the AI struggles to bridge the gap. Lacking a consistent pattern to follow, it often hallucinates a messy, "glue code" solution that is brittle and insecure.

By allowing fragmentation, we aren't just impacting the docs; we are training the AI to misunderstand how our platform is supposed to fit together.

Get started

We cannot view code samples as static documentation. They are the active constraints of our system—the "environment" we design for our users. If we fail to maintain them, we dull the tools that build developer judgment, and we degrade the quality of the AI they trust.

Coming up next

Next in this series, I will share our structural solution: the "Golden Path." This approach moves us away from isolated automation and towards a human-led, AI-scaled system that improves consistency.

I’ll be focusing more on the strategy in this series, but the execution is its own journey. Using AI to write code is well-known, but relying on it to produce production-ready educational content? Adam Ross, and Nim Jayawardena have shared the technical reality of our team's shift in a post on their 7 takeaways from generating samples at scale with Gemini.

Until then, ask yourself:

Are you trying to automate away your documentation debt without first defining a standard of quality?
Are your samples strong enough to serve as the "ground truth" for the AI models your developers rely on?

Special thanks to Katie McLaughlin, Adam Ross, and Nim Jayawardena for reviewing early drafts of this post.

How to enable Secure Boot for your AI workloads

Maciej Strzelczyk — Mon, 21 Jul 2025 14:43:09 +0000

Written in cooperation with Aron Eidelman.

As organizations race to deploy powerful GPU-accelerated workloads, they might overlook a foundational step: ensuring the integrity of the system from the very moment it turns on.

Threat actors, however, have not overlooked this. They increasingly target the boot process with sophisticated malware like bootkits, which seize control before any traditional security software can load and grant them the highest level of privilege to steal data or corrupt your most valuable AI models.

Why it matters: The most foundational security measure for any server is verifying its integrity the moment it powers on. This process, known as Secure Boot, is designed to stop deep-level malware that can hijack a system before its primary defenses are even awake.

Secure Boot is part of Google Cloud’s Shielded VM offering, which allows you to verify the integrity of your Compute VM instances, including the VMs that handle your AI workloads. It’s the only major cloud offering of its kind that can track changes beyond initial boot out of the box and without requiring the use of separate tools or event-driven rules.

The bottom line: Organizations don't have to sacrifice security for performance. There is a clear, repeatable process to sign your own GPU drivers, allowing you to lock down your infrastructure's foundation without compromising your AI workloads.

Google Cloud’s Secure Boot capability can be opted into at no additional charge, and now there’s a new, easier way to set it up for your GPU-accelerated machines.

Understanding the danger of bootkits

It’s important to secure your systems from boot-level threats. Bootkits target the boot process, the foundation of an operating system. By compromising the bootloader and other early-stage system components, a bootkit can gain kernel-level control before the operating system and its security measures load. Malware can then operate with the highest privileges, bypassing traditional security software.

This technique falls under the Persistence and Defense Evasion tactics in the MITRE ATT&CK framework. Bootkits are difficult to detect and remove due to their low-level operation. They hide by intercepting system calls and manipulating data, persisting across reboots, stealing data, installing malware, and disabling security features.

Bootkits and rootkits pose a persistent, embedded threat, and have been observed as part of current threat actor trends from Google Threat Intelligence Group, the European Union Agency for Cybersecurity (ENISA), and the U.S. Cybersecurity and Infrastructure Security Agency (CISA). Google Cloud always works on improving the security of our solutions by strengthening our products and providing tools you can use yourself. In this article, we would like to demonstrate a new, easier way of setting up Secure Boot for your GPU-accelerated machines.

Limitations of Secure Boot with GPUs

Shielded VMs employ a TPM 2.0-compliant virtual Trusted Platform Module (vTPM) as their root of trust, protected by Google Cloud's virtualization and isolation powered by Titan chips. While Secure Boot enforces signed software execution, Measured Boot logs boot component measurements to the vTPM for remote attestation and integrity verification.

Limitations start when you want to use a kernel module that is not part of the official distribution of your operating system. That is especially problematic for AI workloads, which rely on GPUs whose drivers are usually not part of official distributions. If you want to manually install GPU drivers on a system with Secure Boot, the system will refuse to use them because they won’t be properly signed.

How to use Secure Boot on GPU-accelerated machines

There are two ways you can tell Google Cloud to trust your signature when it confirms the GPU driver validity with Secure Boot: with an automated script, or manually.

The script that can help you prepare a Secure Boot compatible image is open-source and is available in our GitHub repository. Here’s how you can use it:

# Download the newest version of the script:
curl -L https://storage.googleapis.com/compute-gpu-installation-us/installer/latest/cuda_installer.pyz --output cuda_installer.pyz

# Make sure you are logged in with gcloud
gcloud auth login

# Check available option for the build process
python3 cuda_installer.pyz build_image --help

# Use the script to build an image based on Ubuntu 24.04
PROJECT = your_project_name
ZONE = zone_you_want_to_use
SECURE_BOOT_IMAGE = name_of_the_final_image

python3 cuda_installer.pyz build_image --project $PROJECT --vm-zone $ZONE --base-image ubuntu-24 $SECURE_BOOT_IMAGE

The script will execute each of the five steps described below for you. It may take up to 30 minutes, as the installation process takes this much time. We’ve also detailed how to use the building script in our documentation.

To manually tell Google Cloud to trust your signature, follow these five steps (also available in our documentation):

Generate your own certificate to be used for signing the driver.
Create a fresh VM with the OS of your choice (Secure Boot disabled, GPU not required).
Install and sign the GPU driver (and optionally CUDA toolkit).
Create a new Disk Image based on the machine with a self-signed driver, adding your certificate to the list of trusted certificates.
The new image can be now used with Secure Boot enabled VMs.

Whether you used the script or performed the task manually, you’ll want to verify that the process worked.

Start a new GPU accelerated VM using the created image

To verify that everything worked, we can create a new VM using the new disk image with the following command (we enable the Secure Boot option to verify that our process worked).

# Create a new VM with T4 GPU to verify that everything works. Note that here ZONE needs to have T4 GPUs available.
TEST_INSTANCE_NAME = name_of_the_test_instance

gcloud compute instances create $TEST_INSTANCE_NAME \
--project=$PROJECT \
--zone=$ZONE \
--machine-type=n1-standard-4 \
--accelerator=count=1,type=nvidia-tesla-t4 \
--create-disk=auto-delete=yes,boot=yes,device-name=$TEST_INSTANCE_NAME,image=projects/$PROJECT/global/images/$SECURE_BOOT_IMAGE,mode=rw,size=100,type=pd-balanced \
--shielded-secure-boot \
--shielded-vtpm \
--shielded-integrity-monitoring \
--maintenance-policy=TERMINATE

# gcloud compute ssh to run nvidia-smi and see the output
gcloud compute ssh --project=$PROJECT --zone=$ZONE $TEST_INSTANCE_NAME --command "nvidia-smi"

# If you decided to also install CUDA, you can verify it with the following command
gcloud compute ssh --project=$PROJECT --zone=$ZONE $TEST_INSTANCE_NAME --command "python3 cuda_installer.pyz verify_cuda"

Clean up

When you verify that the new image works, there’s no need to keep the verification VM around. You can delete it with:

gcloud compute instances delete --zone=$ZONE --project=$PROJECT $TEST_INSTANCE_NAME

Enabling Secure Boot

Now that you have built a Secure Boot compatible base image for your GPU-based workloads, remember to actually enable Secure Boot on your VM instances when you use those images! Secure Boot is disabled by default, so it needs to be explicitly enabled for Compute Engine instances.

When creating new instances

If you create a new instance using Cloud Console, the checkbox to enable Secure Boot can be found in the Security tab of the creation page, under the Shielded VM section.

For the gcloud enthusiasts, there’s --shielded-secure-boot flag available for the gcloud compute instances create command.

Updating existing instances

You can also enable Secure Boot for instances that already exist, however, make sure that they are running a compatible system. If the driver installed on those machines is not signed with a properly configured key, the driver will not be loaded. To update Secure Boot configuration for existing VMs, you’ll have to follow the stop, update and restart procedure, as described in this documentation page.

Get started

Make sure to visit our documentation page to learn more about the process and follow our GitHub repository to stay up to date with other GPU automation news.

Understanding Google Cloud’s Dynamic Workload Scheduler

Maciej Strzelczyk — Tue, 01 Jul 2025 11:53:04 +0000

In the age of artificial intelligence and machine learning, there is a constant need for powerful hardware like GPUs and TPUs. Ideally, access to this hardware should be predictable and reliable. Resource availability shouldn’t be a blocker for your projects. If customers want to use a GPU, they should be provided with a GPU! After all, this is supposed to be one of the ideas behind cloud computing: to have resources available on demand. But with a limited supply of hardware, there is a need for a solution more sophisticated than simple “first come, first serve.”

Introducing DWS

Dynamic Workload Scheduler (DWS) is Google Cloud's innovative solution designed to optimize the allocation of high-demand, finite resources like GPUs and TPUs, ensuring that customer workloads can access the necessary hardware when needed. It directly addresses the supply and demand imbalance problem. On one hand, Google Cloud has customers asking for GPUs and TPUs to run their workloads. On the other hand, there’s a limited number of hardware resources that can be assigned to the customers. DWS is what balances customer demands against the finite resources of the cloud (which wants to feel infinite).

To the traditional model of on-demand provisioning, Spot instances and reservations, DWS adds two simple, yet powerful provisioning methods:

In this article, I’ll explain the benefits of each of these DWS methods and provide practical scenarios for when you might want to use them, helping you choose the best provisioning strategy for your specific workloads. Both methods are still in preview, so you can expect their availability and scope to improve once they enter general availability later this year.

If you’d rather watch a video about Dynamic Workloads Scheduler — I’ve got you covered:

Calendar mode

Let’s start with Calendar mode, which is a bit simpler to understand. DWS Calendar Mode allows you to create future reservations for the hardware you know you will need in advance. Booking rooms in a hotel is a great analogy here. You specify the range of dates, location, type and quantity of the hardware you need and you submit your request. Like a hotel, the system checks resource availability. It then books the resources you want to reserve. Once your future reservation is approved, all you need to do is wait for the starting date. Google Cloud creates a reservation for you on the start date that you can then consume however you want (GCE, GKE, Vertex AI, Vertex AI Workbench and Batch - they can all consume reservations).

Once the reservation time runs out, the system will reclaim the resources, so they can be allocated to other customers. Just like in a hotel, you pay for the time you had your reservation, even if you didn’t use it 100% of the time.

Here are some facts about the DWS Calendar Mode reservations:

The reservation period has a fixed length of 1 to 90 days.
Currently, GPUs require a 4 day lead time before the reservation can start. TPU reservations can be submitted 24 hours in advance of the desired start time.
Once your request is accepted, you will have to pay for the full reservation period, even if not used.
Once the reservation period ends, the resources are reclaimed.
Reserved resources are physically close to each other to minimize network latency.
Calendar Mode reservations can be shared with other projects.
DWS has its own pricing, separate from other provisioning methods. (Usually cheaper than on-demand pricing).
No quota is consumed while using resources booked through Calendar Mode reservations.

So, what are the best scenarios for Calendar mode? If you…

Know how much resources you need
Know how long you need them for
Know when you want to start and finish your project

…then DWS Calendar Mode is the solution for you. Whether it’s an ML training job, HPC simulation or expected spike in the number of inference requests (isn’t Black Friday great?) - the Calendar Mode has you covered.

So what’s the difference between regular future reservations and Calendar Mode?

You might have seen that in Google Cloud, there are also future reservations that are not related to DWS Calendar Mode. You can think of Calendar Mode reservations as a subset of the more generic future reservations. Every Calendar Mode reservation is a Future Reservation, but for a Future Reservation to be a Calendar Mode reservation, it needs to be:

Configured to auto-delete the reservation on expiry, even if it’s not consumed.
No longer than 90 days.
Limited to certain types of resources (see documentation for up to date list)

Additionally, Calendar Mode comes with a handy assistant that helps you find available capacity.

Flex start mode

With Calendar mode being so great, what more may you need? Well, you don’t always have a schedule you need to keep. Sometimes you want your job finished as soon as possible. At other times, you don’t know how long it will take to complete the work. This is where Flex Start mode comes in. If Calendar mode works similar to a hotel, you can compare Flex Start mode to a restaurant.

How does it work? You tell DWS that you need hardware, let’s say 10x A4 machines, to run a job that will take at most 6 days. With that knowledge, DWS goes out to the Cloud to get you your 10 A4 machines. After some time (this is where the “flex” part comes from - it’s a flexible process) the system has the 10 A4 machines you need and provides them to you all at once. This 'all-or-nothing' approach ensures you receive the full requested capacity simultaneously. This way, you don’t have to worry about paying for unused 7 machines while you wait to create 3 more. You get all 10 at the same time. Once they are delivered to you, they will be yours until the specified time runs out, or you’re done with your task. If you release the resources before the time runs out, you pay only for the time you actually used them. Since there is no provisioning notification, ensure your workloads can start automatically upon machine creation.

While Calendar mode was similar to booking rooms in a hotel, Flex Start is more akin to waiting for your order in a restaurant. You wait until your “order” is served and eat until you’re done, or the restaurant closes. If you change your mind before the order is fulfilled, you can cancel your request without any consequences.

To summarize:

Flex Start mode requests hardware for specified periods of time from 1 minute to 7 days.
Requests are fulfilled as soon as possible. (Shorter requests tend to be fulfilled quicker)
You can cancel your request at any time; you only pay for what you used.
DWS Flex Start pricing offers discounts compared to on-demand provisioning.
Once the time limit of your request is reached, the resources are reclaimed.
Resources acquired through Flex Start mode consume the preemptible quota, which is usually a lot higher than on-demand quota.
Works only for Accelerator-optimized machine series and N1 virtual machine (VM) instances with GPUs attached
You can't stop, suspend, or recreate the instances you create through Flex Start mode.

Flex start mode works best if:

You have a short (< 7 days) need for resources.
You want your job started as soon as possible.
You don’t know how long your task will take, and appreciate the flexibility to release resources early and only pay for actual usage.

How to use it?

Flex Start mode works a bit differently in every supported product.

For Compute Engine, it comes in the form of an all-or-nothing Managed Instance Group resize request with the maximum run duration specified.
For Google Kubernetes Engine (GKE), it’s specified for a workload or through scheduling tool.
For Cloud Batch, it’s available for jobs running on specific machine types.
For Vertex AI, specify FLEX_START as your scheduling strategy.

Happy computing!

When it comes to getting your hands on high-demand hardware for your advanced workloads, Google Cloud's Dynamic Workload Scheduler has you covered. With its Calendar and Flex Start modes, you get powerful and flexible solutions that truly fit your needs. By digging into these new provisioning methods, you can count on predictable, reliable, and efficient access to essential resources like GPUs and TPUs. This means your AI, ML, and HPC projects will run smoother than ever. Try booking some powerful machines for your next project now!

Developing in the (Google) Cloud

Maciej Strzelczyk — Thu, 26 Jun 2025 13:46:24 +0000

As I entered the office today, it was clear that physical desktop computers are becoming a rarity. Most desks were equipped only with monitors, reflecting a significant shift in how many organizations, including Google, are approaching employee workstations. Historically, developers might have received both a desktop and a laptop. However, the trend is now towards providing only high-tier laptops, with heavy workloads and software development tasks offloaded to virtual workstations hosted in the cloud. This approach offers enhanced control over assets, improved security, and streamlined management for organizations.

This cloud-centric approach offers substantial benefits for organizations aiming to equip their employees with powerful development environments without the complexities of procuring and maintaining physical desktops. Beyond the immediate advantage of remote work flexibility, where employees can be fully productive with just a laptop and a stable internet connection, cloud-based workstations offer significant scalability. They allow organizations to rapidly provision and de-provision resources as needed, ensuring developers always have access to the optimal computing power, including high-end GPU-accelerated environments that traditional laptops simply cannot provide for demanding industry needs.

There are two ways your organization can leverage this model using Google Cloud Platform.

Google Compute Engine

Google Compute Engine (GCE) provides an Infrastructure as a Service (IaaS) approach to creating virtual workstations through highly configurable virtual machines. This solution offers unparalleled flexibility, granting you complete control over virtually every aspect of your development environment. You can choose your preferred operating system, machine type (including CPU, memory, and specialized hardware), storage solutions, and install any software or tools required. This level of customization makes GCE an excellent choice for a variety of use cases, including:

Heavy graphics

Once you create a virtual machine equipped with a powerful GPU, you can work with demanding graphical applications. Designing complicated systems and models, programming games or rendering videos - all this heavy lifting can happen in the datacenter, while your computer only has to handle the decoding of the remote desktop stream. To fully leverage the remote desktop experience of those setups, you need to:

Pick a GPU that supports NVIDIA RTX Virtual Workstations (vWS) for graphics workloads. That means L4, T4, P4 or P100 accelerators. A new G4 machine type hosting NVIDIA RTX PRO 6000 Blackwell cards should be available by the end of 2025.
Install RTX-compatible GPU drivers.
Select the remote desktop software you want to use to access the machines. There are many available options like HP Anywhere, Parsec or Moonlight to name a few.
Ensure the Internet connection on your client side is fast and reliable.

Computation intensive (like AI)

Google Cloud offers really powerful GPUs that can empower your team to effortlessly tackle many AI challenges. With no need for high-quality graphical interface, access to machines in this category can be even limited to an SSH tunnel. The developer can run their favourite IDE on their laptop, while executing the code remotely in the cloud. Depending on the GPU you pick, the pricing of such workstations will vary greatly. Good news though, a single machine can be easily shared between multiple developers with proper configuration.

General development

Developers who don’t need GPU-powered machines to do their jobs, still can benefit from a remote, powerful environment. More RAM, CPUs and storage is really easy to obtain to exceed what even the best laptops can provide.

Considerations

When working with GCE VMs, it is crucial to pay special attention to both the security and cost optimization of these machines. Failing to properly configure these aspects can lead to vulnerabilities or unnecessary expenses. Here are some key considerations (this list is not exhaustive):

Security Best Practices

1) Service Accounts: Avoid using the default compute Service Account, which comes with an overly permissive Editor role. Instead, create new service accounts with the principle of least privilege, assigning only the minimal required permissions for your workloads. For individual users, consider creating dedicated service accounts.

2) Network Access: Consider disabling external IPs for your VMs. For internet access, configure Cloud NAT. For secure remote access, leverage Cloud VPN or Identity-Aware Proxy (IAP).

3) Firewall Policies: Implement stringent firewall policies to control inbound and outbound traffic, ensuring only necessary ports and protocols are open.

Cost Optimization Strategies

1) Commitment-based Discounts: Take advantage of Committed Use Discounts (CUDs) for predictable workloads, which can substantially reduce costs over long-term commitments.

2) Automated Scheduling: Implement VM instance scheduling to automatically stop workstations during off-hours (e.g., overnight or weekends), minimizing resource consumption when not in use.

Google Cloud Workstations

If all your team needs is the computation power of cloud instances and not a full graphical connection, then Cloud Workstations might be just for you (video explainer). It’s a managed solution, which allows you to create virtual workstations that your team can connect to and use for development. Those instances can be based on many different machine types, including GPU-accelerated ones. You can choose to use them through Code OSS (Visual Studio), multiple JetBrains IDEs through JetBrains Gateway or Posit Workbench (with RStudio Pro).

Workstations allow you to customize the developer environments, so that each new instance comes with all the necessary tools preinstalled. Users can be allowed to create and destroy their own environments, while you retain the control over the allowed configurations of those environments.

Despite being a bit more expensive than “raw” Compute Engine instances, the managed Workstations might turn out to be cheaper in use than Compute Instances, as they allow you to configure auto-sleep and auto-shutdown settings, so resources are not wasted when the workstations are not used.

Cloud Workstations offer a wide variety of customization options and security configurations. While not as flexible as simple Virtual Machines, the Workstations might be more attractive due to easier management, strict control and out-of-the-box compatibility with popular coding solutions.

In summary

Google Cloud offers virtual workstation solutions for all kinds of developer needs. Here’s a short summary table, highlighting various applications of GCE and Workstations.

	Compute Engine (unmanaged VMs)	Cloud Workstations
Graphical-heavy work Designing Gaming Game development Video editing	GPU-accelerated VMs offer great performance when paired with proper virtual workspace software.	N/A - Cloud Workstations don’t support this kind of work.
AI and HPC workloads AI training AI inference GPU-powered simulations	GPU-accelerated VMs can make use of every GPU-type available in Google Cloud. Sharing a big VM between multiple developers is a valid approach.	Cloud Workstations support GPU-accelerated machine types, allowing developers to work on software that requires GPU-acceleration.
General workloads	While regular VMs can work for hosting a workstation for this kind of applications, it might not be worth the management effort.	Cloud Workstations work great as a platform for developers who need a remote cloud-based environment to work on their projects. With the majority of management hassle taken care of, you are free to just work on your project.

Embrace the future of development today by exploring the powerful virtual workstation solutions offered by Google Cloud. While Compute Engine provides unbridled flexibility, Cloud Workstations offer streamlined efficiency. Unlock enhanced productivity and simplified asset management for your team. Start your cloud development journey now and discover the perfect environment for your needs.

Observability in Action: A Google Cloud Next demo

Olivier Bourgeois — Mon, 05 May 2025 18:17:47 +0000

It was only a few weeks ago that over 32,000 cloud practitioners from all over the world came together in Las Vegas to attend Google Cloud Next 2025. Beyond the keynotes, the workshops, and the multiple jam-packed tracks of talks and sessions, an entire expo hall offered attendees the opportunity to observe or play around with more than 500 live demos. Let’s check out one of these demos!

Overview of the demo

The main goals of the Observability in Action demo were twofold. We wanted to showcase various ways of interacting with metrics and logs. And we wanted to give attendees a little bit of an interactive experience. For the interactive part of the demo, we utilized various oversized physical buttons and pedals that could be used to select answers or confirm inputs.

The flow of the demo was as followed:

We ask the attendee to type in a prompt that they wanted sent to an AI model.
The prompt was sent in the background to three different models: Gemma 3 on Cloud Run, Gemini 2.0 Flash on Vertex AI, and Gemini 2.0 Flash-Lite on Vertex AI. This generated logs and metrics.
The attendee was then given a short quiz about these three models. Each quiz input also generated logs and metrics.
At the end of the quiz, we give the attendee a rundown of their answers, and then flip over to the Google Cloud Console.
In Cloud Monitoring, we showcase the various native metrics that Cloud Run offers, custom metrics implemented using OpenTelemetry, as well as the Cloud Trace functionality.
Finally, we turn to BigQuery to showcase how we can mirror logs to a database for further analysis using Jupyter Notebooks.

Architecture

While the demo frontend runs locally, the backend is deployed as a Cloud Run instance. This instance is then talking to Gemini through the Vertex AI SDK and to Gemma through its own Cloud Run instance. The persistent state of the demo resides in a Firestore database. All Cloud Run logs are mirrored to BigQuery using a simple sink.

Visualizing metrics using Cloud Monitoring

Cloud Monitoring provides visibility into the performance and health of your cloud applications and infrastructure. It collects metrics, events, and metadata from Google Cloud services and other sources, allowing you to visualize this data on dashboards and create alerts for critical issues. This is useful for proactively identifying and resolving problems, optimizing resource utilization, improving uptime, and understanding system behavior, ultimately leading to more reliable and cost-effective applications.

For services like Cloud Run which we’re using for the backend of this demo, Cloud Monitoring automatically collects a wide array of native metrics without any setup needed. This includes data points such as request latency, count, container CPU and memory usage, and instance counts. This out-of-the-box integration means developers get immediate insights into their serverless application's performance and resource consumption, simplifying troubleshooting and optimization efforts.

Cloud Trace is a distributed tracing system within Google Cloud that helps you understand request latency across your application and its services. It tracks how long different parts of your application take to process requests, visualizing the entire request flow. This is particularly valuable for identifying performance bottlenecks in microservices architectures by showing where time is spent during a request's lifecycle.

Here’s a real life example: In this demo we send a prompt to multiple models. We were sure we implemented concurrency correctly (so the calls to the three different models should’ve happened in parallel) yet the latency seems significantly higher than expected. When we dug into the trace of a call, we quickly realized that we were accidentally making those calls sequentially! These traces were made available to us via an OpenTelemetry instrumentation we added to our code.

Interact with your logs with BigQuery

BigQuery is a serverless enterprise data warehouse that enables super-fast SQL queries on large datasets without infrastructure management. It's built for scalable analytics, supports diverse data types, and integrates machine learning, offering a powerful platform for insights from real-time and historical data.

With a simple sink, you can directly stream logs from Cloud Logging into BigQuery, transforming it into a powerful, long-term log analytics platform. This allows you to run complex SQL queries across extensive historical log data, which is invaluable for in-depth security audits, compliance, and identifying subtle operational trends.

Connecting BigQuery to Jupyter Notebooks further enhances log analysis capabilities. This empowers users to leverage Python and data science libraries for advanced data exploration, custom visualizations, and machine learning on log data, facilitating deeper insights and shareable, interactive analysis beyond standard logging tools.

For this demo, we built a Jupyter Notebook that did analysis on the various interactive quiz events, cross-referenced answers with an external Firestore database, and built tables and charts of the resulting data.

Try it out!

Want to try this demo from home? The source code is available on GitHub.

Want to learn more about observability on Google Cloud? Check out these resources:

Getting started with Rust on Google Cloud

Karl Weinmeister — Thu, 27 Mar 2025 04:29:49 +0000

This post will guide you through deploying a simple “Hello, World!” application on Cloud Run. You’ll then extend the application by showing how to integrate with Google Cloud services with experimental Rust client libraries.

I’ll cover the necessary code, Dockerfile configuration, and deployment steps. I’ll also recommend a robust and scalable stack for building web services, especially when combined with Google Cloud’s serverless platform, Cloud Run.

Why Rust and Axum?

Rust has gained significant traction in backend development, earning the title of most-admired language in the StackOverflow 2024 Developer Survey. This popularity stems from its core strengths: performance, memory safety, and reliability. Rust’s low-level control and zero-cost abstractions enable highly performant applications. Its ownership system prevents common programming errors like data races and null pointer dereferences. In addition, Rust’s strong type system and compile-time checks catch errors early in the development process, leading to more reliable software.

The Rust web framework ecosystem is vibrant and evolving. Popular choices include Axum, Rocket, and Actix. In this post, I’ll showcase Axum, but you can apply what you’ve learned here to other Rust web frameworks. Axum’s API is clear and composable, making it easy to build web services. Its modular architecture allows developers to select only the necessary components. Axum is built on Tokio, a popular asynchronous runtime for Rust, which allows it to handle concurrency and I/O operations efficiently.

Hello World Application

Let’s start by exploring a basic “Hello, World!” example from the official Axum repository. In each section of this blog post, you will enhance the example to leverage Google Cloud capabilities. You can access the final code sample in the cloud-rust-example repository.

First, the Cargo.toml manifest file defines the project’s metadata and dependencies:

[package]
name = "example-hello-world"
version = "0.1.0"
edition = "2021"
publish = false

[dependencies]
axum = { path = "../../axum" }
tokio = { version = "1.0", features = ["full"] }

Within this file, you see:

[package]: Contains basic project information like name, version, and the Rust edition. publish = false prevents accidental publication.
[dependencies]: Lists the project’s dependencies — axum for the web framework and tokio for asynchronous capabilities.

Now, let’s examine the core application code, src/main.rs:

use axum::{response::Html, routing::get, Router};

#[tokio::main]
async fn main() {
    // build our application with a route
    let app = Router::new().route("/", get(handler));

    // run it
    let listener = tokio::net::TcpListener::bind("127.0.0.1:3000")
        .await
        .unwrap();
    println!("listening on {}", listener.local_addr().unwrap());
    axum::serve(listener, app).await.unwrap();
}

async fn handler() -> Html<&'static str> {
    Html("<h1>Hello, World!</h1>")
}

This code sets up a minimal web server using Axum and Tokio. The #[tokio::main] macro enables asynchronous execution. The main function creates a Router to handle requests, defines a single route / that responds with “Hello, World!”, binds the server to 127.0.0.1:3000, and starts the server. The handler function generates the HTML response for the root route.

Enhancements for Cloud Run

The basic example above works well for local development, but let’s make some improvements for deploying to Cloud Run. The official example notably does not include a Dockerfile, which is required for Cloud Run.

1. Standalone Deployment: To make the example standalone and deployable, modify the Cargo.toml file. Change the axum dependency from axum = { path = “../../axum” } to axum = “0.8” to use the published version of Axum from crates.io instead of the local path.

2. Dynamic Port Configuration:

Cloud Run dynamically assigns a port to your application, which is provided through the PORT environment variable. The original example hardcodes the port to 3000. To make our application Cloud Run-compatible, modify the main function to read the PORT environment variable and use it if available, falling back to a default port such as 8080 if the variable is not set.

The address should also be changed to 0.0.0.0 to listen on all network interfaces, which is generally preferred for containerized applications.

Here’s the modified main function:

#[tokio::main]
async fn main() {
    // Get the port from the environment, defaulting to 8080
    let port = std::env::var("PORT").unwrap_or_else(|_| "8080".to_string());
    let addr = format!("0.0.0.0:{}", port);

    // build our application with a route
    let app = Router::new().route("/", get(handler));

    // run it
    let listener = tokio::net::TcpListener::bind(addr)
        .await
        .unwrap();
    println!("listening on {}", listener.local_addr().unwrap());
    axum::serve(listener, app).await.unwrap();
}

3. Dockerfile:

To deploy to Cloud Run, you’ll need a Dockerfile. Here’s a simple one that works well for this example:

FROM rust:1.85.1
WORKDIR /app
COPY . .
RUN cargo build --release
EXPOSE 8080
CMD ["./target/release/example-hello-world"]

This Dockerfile uses the official Rust image as a base, copies the project files, builds the application in release mode, exposes port 8080 (the default port), and sets the command to run the compiled executable. You can upgrade to the latest Rust image if you’d like.

4. .gcloudignore file:

You can also add a .gcloudignore file to the project root to exclude unnecessary files (like the target directory containing build artifacts) from the deployment:

.git/
.gitignore
target/

Deploying to Cloud Run

Before deploying, ensure you have the Google Cloud SDK installed and configured, and you have enabled the Cloud Run API in your Google Cloud project. You’ll also need to be in the root directory of your Axum project (where the Cargo.toml file is located).

Before attempting your deployment, you can check the local package and deployment for errors:

cargo check

To deploy directly to Cloud Run from source, use the following command:

gcloud run deploy cloud-rust-example \
    --source . \
    --region us-central1 \
    --allow-unauthenticated

Here’s what each part of the command means:

gcloud run deploy cloud-rust-example: This is the base command to deploy a service to Cloud Run. cloud-rust-example is the name we’re giving to our service. You can choose a different name.
—-source .: This flag tells Cloud Run where to find the source code for your application. The . indicates the current directory. Cloud Run will use the Dockerfile in this directory to build a container image.
—-region us-central1: This specifies the Google Cloud region where your service will be deployed. In this case, we’re using us-central1. You can choose a region closer to your users for lower latency.
—-allow-unauthenticated: This flag makes your deployed service publicly accessible without requiring authentication. This is convenient for initial testing and simple public services. For production applications, you should remove this flag and implement proper authentication and authorization.

Cloud Run will automatically build and deploy your application. You will be provided with a service URL in the output. Accessing this URL in your browser will display the “Hello, World!” message.

Hello world output from / route

Integrating with Google Cloud Services

Let’s now show how to integrate our application with Google Cloud services. I’ve selected a straightforward scenario that doesn’t require any project configuration to work. You’ll add a new application route /project that will display information about your project.

To implement this, you’ll use the google-cloud-rust library to interact with the Resource Manager API and retrieve information about your Google Cloud project.

Note: The google-cloud-rust library is currently experimental. APIs may change, and it’s important to stay updated with the latest releases and documentation.

Add Dependencies

First, add the Resource Manager v3 API and reqwest HTTP client to your Cargo.toml file:

cargo add google-cloud-resourcemanager-v3 reqwest

Implement the handler

There are four key changes we’ll need to make in src/main.rs:

Add /project Route: A new route /project will display project information, implemented by project_handler()

async fn main() {
    // ...
    let app = Router::new()
        .route("/", get(handler))
        .route("/project", get(project_handler))
        .layer(Extension(std::sync::Arc::new(client)));
    // ...
}

project_handler function: The project handler will call get_project() to fetch project details. Finally, it formats the project information into an HTML response. Error handling is included to display any errors that occur during the API call.

async fn project_handler(Extension(client): Extension<Arc<Projects>>) -> Html<String> {
    let project_id = PROJECT_ID.get().expect("Project ID not initialized");
    let project_name = format!("projects/{}", project_id);

    match client.get_project(project_name).send().await {
        Ok(project) => {
            let project_number = project.name.strip_prefix("projects/").unwrap_or("Unknown");

            Html(format!(
                "<h1>Project Info</h1><ul><li>Name: <code>{}</code></li><li>ID: <code>{}</code></li><li>Number: <code>{}</code></li></ul>",
                &project.display_name,
                project_id,
                project_number
            ))
        }
        Err(e) => Html(format!("<h1>Error getting project info: {}</h1>", e)),
    }
}

Share client with handler: For best performance, any one-time configuration should not reside in the handler. The Projects client can be initialized in main() and then shared with the handler with Axum’s Extension.
Add helper function for project metadata : To find out the project ID the container is running in, you’ll need to access the metadata key. That project ID will then be used to call the Resource Manager API to get more information about the project, including its display name and creation time. You can use LazyLock to initialize the project only once.

static PROJECT_ID: OnceLock<String> = OnceLock::new();

async fn main() {
    // ...

    let project_id = get_project_id().await.expect("Failed to get project ID");
    PROJECT_ID.set(project_id).expect("Failed to set PROJECT_ID");

    // ...
}

async fn get_project_id() -> Result<String, String> {
    if let Ok(project_id) = var("GOOGLE_CLOUD_PROJECT") {
        return Ok(project_id);
    }

    let client = reqwest::Client::new();
    let url = "http://metadata.google.internal/computeMetadata/v1/project/project-id";

    let response = client
        .get(url)
        .header("Metadata-Flavor", "Google")
        .send()
        .await;

    match response {
        Ok(res) => {
            if res.status().is_success() {
                Ok(res.text().await.map_err(|e| e.to_string())?)
            } else {
                Err(format!("Metadata server returned error: {}", res.status()))
            }
        }
        Err(e) => Err(format!("Error querying metadata server: {}", e)),
    }
}

Set GOOGLE_CLOUD_PROJECT Environment Variable (Locally)

For local testing, you’ll need to set the GOOGLE_CLOUD_PROJECT environment variable to your Google Cloud project ID. You can do this in your terminal before running the application:

export GOOGLE_CLOUD_PROJECT=your-project-id

Replace your-project-id with your actual project ID. Cloud Run will automatically set this environment variable when deployed.

Enable the Resource Manager API

If you haven’t already, make sure to enable the Resource Manager API within your Google Cloud project.

Provide Resource Manager IAM access

You will need to provide the resourcemanager.projects.get role to the appropriate Cloud Run service account. The instructions here use the Compute Engine default service account. If you are running locally, you’ll also need to provide these permissions to your account.

Redeploy to Cloud Run

Use the same gcloud run deploy command as before to redeploy your updated application:

gcloud run deploy cloud-rust-example \
    --source . \
    --region us-central1 \
    --allow-unauthenticated

Now, when you visit the service URL provided by Cloud Run and navigate to the /project path, you should see information about your Google Cloud project.

Project information output from /project route

Conclusion

This guide demonstrates the process of deploying a Rust Axum application on Cloud Run. I started with a basic “Hello, World!” example from the Axum repository, explained its code, and then showed how to enhance it for Cloud Run compatibility by dynamically configuring the port and creating a Dockerfile. By combining Rust and Axum with Cloud Run’s serverless simplicity, you can efficiently build and deploy robust web services. The sample source code is available in the cloud-rust-example repository.

For more information about Cloud Run, I recommend the quickstart for building and deploying a web application in the documentation. Also, check out this video for a video walkthrough of running Rust on Cloud Run. Feel free to connect on LinkedIn, X, and Bluesky to continue the discussion!

Polish Large Language Model (PLLuM) on Google Cloud

Remigiusz Samborski — Mon, 17 Mar 2025 08:44:00 +0000

"Wpadła śliwka w .... Google Cloud" 😉

Recently, thanks to the Ministry of Digital Affairs, there's been a lot of buzz about the new Polish Large Language Model (PLLuM). I decided to play around with it a bit and show others how to run it on Google Cloud using Vertex AI.

I invite you to check out this notebook, which will guide you through this process step by step.

Let me know in the comments what applications you see for this new open model.

DEV Community: Google Cloud

Inference on GKE Private Clusters

Setting up inference service without access to Internet

GKE Private Nodes

Providing images to the pods

Providing the LLM

Alternative to mounting GCS Bucket - persistent disks

Summary

AI deployment: to host or not to host?

Serverless vs hosted inference service

Serverless (pay per token)

Pros:

Cons:

Hosted (pay per second)

Pros:

Cons:

Couple of considerations

How much traffic do I expect?

Am I legally bound to keep user data in certain region?

Am I likely to hit the hourly/monthly quota?

Mixed-approach

This is not an irrevocable decision

Keep up!

Making Sure Your Prompt Will Be There For You When You Need It

LLMs Can Make Mistakes

Prompt Templates as Functional Interfaces

Phase 1: Build a foundation with Ground Truth

Phase 2: Finding Your Candidate Prompt (Vibe Check)

Phase 3: Statistical Trials – Because Unit Tests Alone Don’t Work

Embracing Statistical Techniques For The Best Performance

Join the Conversation

Read More

How My Team Aligns on Prompting for Production

Natural Language is a Fuzzy Programming Language

Be Efficient with Brains (Pair & Review)

Document "The Why" and manage change

Separation of Concerns (Use Dedicated Files)

Use Structured Output

From Magic to Engineering

Join the conversation

Read More

Why your `curl` logic just bit you 🐾

🤫 Authenticate for local development

🏡 Moving to Production

Get started

The lumberjack paradox: From theory to practice

From theory to practice: The catalog as ground truth

The high cost of "geological strata" in code

Inconsistent quality creates "false best practices"

The hidden cost of incoherence

Get started

Recommended Reading

Coming up next

How to enable Secure Boot for your AI workloads

Understanding the danger of bootkits

Limitations of Secure Boot with GPUs

How to use Secure Boot on GPU-accelerated machines

Start a new GPU accelerated VM using the created image

Clean up

Enabling Secure Boot

When creating new instances

Updating existing instances

Get started

Understanding Google Cloud’s Dynamic Workload Scheduler

Introducing DWS

Calendar mode

So what’s the difference between regular future reservations and Calendar Mode?

Flex start mode

How to use it?

Happy computing!

Developing in the (Google) Cloud

Google Compute Engine

Heavy graphics

Computation intensive (like AI)

General development

Considerations

Security Best Practices

Cost Optimization Strategies

Google Cloud Workstations

In summary