DEV Community: Adam Ross

How My Team Aligns on Prompting for Production

Adam Ross — Tue, 17 Feb 2026 16:53:46 +0000

My team at Google is automating sample code generation and maintenance. Part of that is using Generative AI to produce and assess instructional code. This introduces a challenge: How do we trust the system to meet our specific standards, when core components are non-deterministic?

Establishing trust requires isolating and understanding each large language model (LLM) request. We need to know exactly what goes into the model, and a guarantee of what comes out.

This challenge isn't different from other feature development. To succeed, we realized we had to stop treating prompting like chatting or guessing and start treating it like coding.

Natural Language is a Fuzzy Programming Language

Prompting an LLM is effectively "natural language programming": we are programming in English. The problem is that English is not the greatest language for programming. It is ambiguous. It is subjective. It is open to interpretation.

In C++, a missing semicolon breaks the build. In English, a missing comma changes the objective entirely: "I don't know, John" becomes "I don't know John". In a programming language, syntax is binary; it works or it doesn't. In English, the difference between "Ensure variables are immutable" and "Make sure variables never change" might yield different results based on the model’s training data.

When you combine the fuzziness of human language with the "black box" probabilistic processing of an LLM, you face a difficult question: What is the weather going to be today in the land of AI?

To answer that, you have to make the intentions behind your prompts explicit.

Be Efficient with Brains (Pair & Review)

Writing a prompt is an exploratory process of finding words that trigger the best response. However, a single writer is limited by their own understanding. This is risky with LLMs, which are suited for ambiguous problems but require strict guardrails.

Relying on one person to do this creates blind spots. We found that prompt quality benefits from pairing. A diversity of thought helps create a more complete definition of any problem. What one engineer considers a clear instruction, another might see as a loophole. Pairing covers the gaps that a single brain might miss.

Furthermore, you should also review every prompt. This isn't just checking for typos; it’s a logic check. Does this prompt align with business requirements? We’ve found that prompt reviews often uncover disagreements about the requirements themselves, forcing us to align as a team before we ship.

Document "The Why" and manage change

Because English is fuzzy, the intent behind a specific word choice isn't always obvious. Why did we use the passive voice here? Why did we specify "immutable"?

Even well-structured prompts can eventually obscure the original business requirements. As optimizations blur into the general text, we must take every opportunity to document "the why":

Documentation: Avoid relying on prompts as canonical business requirements. Our LLM requests are a combination of system instructions, user input, context, and deterministic post-processing; the prompt is not complete for onboarding a developer.
Comments: Comment on complex prompts just as you would complex code. Spotlight specific constraints or even punctuation to explain the problem they solve. The model is a moving target, so any unintentional changes can make troubleshooting hard.
Commit Messages: Use commit messages as an opportunity to explain what was wrong with the prompt. (for example, fixed: Missing comma lost John)

Separation of Concerns (Use Dedicated Files)

Writing code and writing prompts require distinct mindsets. One focuses on syntax and execution flow; the other on semantics and intent. Embedding long English instructions inside code creates a distraction.

We keep prompts in dedicated files to disentangle application logic from the LLM interaction configuration, which requires frequent tuning.

By treating the prompt as a standalone component, we can prototype and iterate on the LLM behavior independent of the application's control flow. Tools like dotprompt allow us to treat these files as first-class artifacts containing text, model parameters, and schema definitions. This highlights that invoking a model isn't just a function call; it’s an integration with a distinct system that requires its own configuration.

Use Structured Output

To build a reliable tool, you need a bridge between unpredictable LLM output and deterministic computers.

We rely on structured output¹ to guide the model to emit JSON according to a schema. Even if we only need a single field, defining a schema provides a guardrail that helps the model output conform to a shape we can validate programmatically. This is critical for code generation, where models often add unwanted preambles, conversational filler, or inconsistent markdown fences.

If the output doesn't match the schema, we fail fast or retry. This allows us to integrate the LLM output into our process with the same confidence we have in detecting a bad API response.

From Magic to Engineering

Moving from one successful prompting to a reliable system requires acknowledging that prompts are code. You need to manage, review, and test them with the same rigor applied to the rest of your stack. While we are still working on better ways to benchmark quality, treating our prompts as first-class codebase assets is our first step towards building confidence in our AI assisted automation.

Join the conversation

I'm curious how you are handling the fuzzy nature of LLMs in production.

How does your team review and test prompts?
Do you treat prompts as configuration, code, or something else entirely?
Share your workflow in the comments.

The lumberjack paradox: From theory to practice by Jennifer Davis
Google Sandwich Manager, and the hallucinated SDK by Katie McLaughlin and Brian Dorsey

Thanks to Jennifer Davis & Shawn Jones for review and contributions.
Cover Photo by Jeton Bajrami on Unsplash

This links to how structured output works in the Genkit framework, which my team is using. A succinct example. ↩

[Boost]

Adam Ross — Wed, 19 Nov 2025 00:43:10 +0000

Jennifer Davis for Google Cloud

Nov 19 '25

The lumberjack paradox: From theory to practice

#cloud #ai #programming

Comments 1

5 min read

Write to Google Sheets from a local script via gcloud CLI authentication

Adam Ross — Wed, 25 Sep 2024 22:16:33 +0000

Recently I needed to pull data from the GitHub API and publish to a Google Sheet so I could share some charts about code review workload. This post is about how I got authentication working.

I wrote a Node.js script because my use case was too complex for BASH and seemed too temporary for Go. Initially, the script generated ad hoc CSV data that I could manually copy into Google Sheets using the Paste-as-CSV feature. After a few rounds of manually copying, I wanted to use my time wisely: I estimated that I could get a Sheets integration working within a couple hours and if so that would
probably pay off within a few months.

Aside: Sheets isn't my database. It's my data reporting UI. Don't use Sheets as your database.

It took me closer to 4 hours to get this working, because authentication is hard. This post shows the more direct path to a working solution.

It's easy to be distracted with the complexity of creating an oAuth application, and while I especially like using a service account from my Cloud services, that's
less convenient when running locally.

The trick I learned is that the gcloud CLI's setup of application default credentials can operate as a sort of OAuth proxy for Google Workspace code, by expanding how it authenticates your account with Google to include some more permissions (OAuth Scopes).

🔐 Local authentication with gcloud CLI

To make API requests to Google Sheets, enable the Sheets API in your Cloud project:

$> gcloud services enable sheets.googleapis.com

Operation "operations/acat.p2-480745230567-02564c8d-c6ba-4f60-90bd-13f33e41f0fe" finished successfully.

Set your application default credentials, also claiming some non-default OAuth scopes so the credential can be used with sheets:

$> gcloud auth application-default login --scopes \
   'https://www.googleapis.com/auth/cloud-platform,https://www.googleapis.com/auth/drive,https://www.googleapis.com/auth/spreadsheets'

This will trigger an OAuth flow that involves visiting a webpage in your browser. Depending on the terminal configuration this may display a URL or even open the page.

When gcloud is granted this scope, code that can access your local credentials will have read/write access to all your Google Sheets data. You can re-run this command without the customized scopes to toggle that access on and off as needed.

🗝️ Initialize a Node client for Google Sheets

import {google} from 'googleapis';

function sheetsClient() {
    const authConfig = new google.auth.GoogleAuth({
        scopes: [
            // Only 'spreadsheets' scope is needed in the code.
            // gcloud CLI also needs 'cloud-platform' and 'drive'.
            'https://www.googleapis.com/auth/spreadsheets'
        ],
    });
    const auth = await authConfig.getClient();
    return google.sheets({version: 'v4', auth});
}

🖇 Append data to the sheet

The sample from the docs on how to append data to the sheet
(click on Node.js tab) worked well. However, I couldn't understand how to get authentication working from there. The sample worked once I understood the trick above to add the missing OAuth scopes to my dev environment credentials.

I made a few changes to enable easier reuse of the client, parameterize the request differently, and emphasize how I wanted the data handled by Sheets.

My code to append data to the sheet, leveraging the client initialization code above:

let client;

async function appendDataToSheet(spreadsheetId, tab, values) {
    if (!client) {
        client = sheetsClient();
    }

    try {
        const result = await client.spreadsheets.values.append({
            spreadsheetId,
            range: `${tab}!A2:AG`,
            // Use my data as provided.
            valueInputOption: 'RAW',
            // Inserts rows as part of appending to reduce overwrites.
            insertDataOption: 'INSERT_ROWS',
            requestBody: { values },
        });
        console.log(`${result.data.updates.updatedCells} cells appended.`);
    } catch(e) {
        // Show the error, do not stop. Cross-reference the error with terminal output
        // and decide case-by-case to re-run the script or manually copy data.
        console.error(e);
    }
}

I'm using "append" because my script collects monthly metrics, and append allows me to add new rows without removing earlier rows.

Here's an example of how to call the appendDataToSheet() function:

const values = [
  // Each nested array is a spreadsheet row.
  [1, 2, 3, 4, 'luggage'],
  [4, 5, 6, 'N/A', 'sticks'],
];
appendDataToSheet(
  'HPDkfqdu6rfIq5-4uTGDqz2tvmPxDZMul27JFexample',
  'Exported Data Tab',
  values
);

There are some good tips about working with the Sheets API in the docs such as the suggestion not to send more than one API request per second per sheet. I found out the hard way: overwriting data from a first request with data from a second.

🚀 Ship it!

If I move forward to productionize this, I might switch to using
Cloud Scheduler and Cloud Run Jobs. Let me know if you'd like to read about that.

Cover Photo by Glib Albovsky on Unsplash

Where to Host My Static Tech Blog

Adam Ross — Tue, 13 Aug 2024 22:10:02 +0000

Context: I'm a decision record enthusiast and will absolutely write more about this in the future. This post format is inspired by Architectural Decision Records and is meant to show how I got to my hosting solution, not how to use it.

Status: [~~thinking~~ | ~~nope~~ | current | ~~superceded by~~ ]
Last Updated: Mar 9, 2024
Objective: Choose a web host for my static site

Context & Problem Statement

To have a website, there must be a web host. A website is the visual artifact that everyone sees, the pages, images and videos. The web host is a service provides the compute, storage, networking, and all the bits that make it usable. For my personal website, I need to choose a web host that adheres to the following priorities and constraints.

Priorities & Constraints

Every project has requirements, here are a few of mine:

Be low cost: My personal website is a personal expense I'd rather keep minimal. My goal is to fit within a free tier, but not to exceed a "dollars per month" cost.
Be focused on publishing over development: My time is immensely valuable. I want to publish content with some flexibility to tailor for my way of working. However, I do not want to generate (too many) new side projects in web development instead of writing.
Be low attention: I want to ignore this site for months at a time without incurring security risks or a backlog of infrastructure projects.
Minimal Service Providers: I prefer to use the fewest number of service providers for my personal IT. I have GitHub, DNS, and Google Cloud already in the mix. Each additional company is another credit card profile to update and set of corporate customer policies to track.
Learn from Everything: Balance existing skills with learning opportunities
Sourcing from GitHub: I've decided to keep my site source and content in a public GitHub repository. I want to see merges to my main branch deploy to a live site.
Constraint: I've previously chosen Hugo and Go with code hosted on GitHub as tools for building the site. Hugo is a static site generator flexible enough to be used across many (all?) hosting services.

Considered Options

Option 1: Use Firebase Hosting
Option 2: Use GitHub Pages
Option 3: Use Google Cloud Run
Option 4: Use Netlify

Decision

Chosen option Option 1: Firebase Hosting

Firebase Hosting is a specialized tool for static site hosting, providing ease of deployment, a global CDN, and a good free tier. I have a lot of familiarity with Google Cloud, but less familiarity with the Firebase products.

Working for Google means my day job pays me to be familiar with Google Cloud, which may lead me to use Google Cloud for more personal projects. This gives Firebase Hosting some advantage in the decision.

I'm choosing Firebase Hosting because it meets all my requirements and broadens my knowledge of Google Cloud.

Alternatives Considered

I have experience hosting sites on Option 2: GitHub Pages. However, I have been frustrated by the lack of performant redirects (redirects via HTML meta refresh lack wildcard configuration options and can't be cached compared to server-side options). Other solutions have server-side redirect features (firebase hosting, netlify)

I've found GitHub Pages to be a bit complex for PR Previews, with various advice around multiple repositories, random & niche GitHub Actions, or building your own custom preview solution.

I'm not choosing GitHub Pages because it's missing features I want.

Option 3: Use Google Cloud Run. With 6 years of experience working on the developer experience of Google Cloud Run, using it would be straightforward. I don't need (or want) the control over the serving details like which web server is running. The other options are better aligned with my preference for minimal maintenance.

💡 My teammate Brian Dorsey has a great post on hosting a static website using Google Cloud Run if you'd like a recipe.

I'm not choosing Cloud Run because I want more abstraction away from the technical details. I don't want to pick up a new infrastructure hobby.

Option 4: Use Netlify. Several folks recommended Netlify as their preferred static site host, with strong GitHub integration. Since I am already using Google Cloud for other projects, Netlify has a disadvantage as a new service provider.

If Firebase Hosting doesn't work out, Netlify would be my next choice. It was a close decision.

Expected Consequences

Any of these options will meet my goal of serving a few megabytes of data. I expect problems to come from pet peeves or curiousity about the alternatives. Either way, this is a low stakes decision not worth a deeper analysis. If I had to change my decision, I expect the switching cost to move to another static site host to be an evening of work.

Revisiting this Decision

I've scheduled a reminder for March 2025 to reflect on this decision, and decide if I'd like to change hosting. With the low expected switching cost, it might be worth trying another host just for the learning experience.

Cover Photo by Mike Enerio on Unsplash

Dependency Updates: More to Do Than Review

Adam Ross — Sat, 29 Jun 2024 17:05:36 +0000

I enjoy tools that automate dependency updates in my projects (looking at you, Renovate and Dependabot). They save my team a lot of time: with a quick glance and the push of a couple buttons, I can update dozens of dependencies in a repository. Even better, the work came to me, I didn't need to go looking for release schedules or subscribe to a newsletter. This is a big change to how I see teams handle 3rd party updates.

Merging an update just because the tests pass creates a mess.

Context: What do these dependency automation tools do?

They help in the following ways:

Notify project maintainers that a dependency has an update
Modify the project's package manifest to reference the updated code
Open a Pull Request and highlight some context on the change
and because you have CI set up, they trigger your automated tests

When I start my day and look at a Pull Request created by a tool like this, I may see a lot of encouraging green checkmarks✅ declaring merge-worthiness.

The tools did all the work, I can review & merge, right?

Seeing green checkmarks is necessary but not sufficient to merge. I trust automated checks to tell me where to prioritize my review time. As the human in the loop, I should verify:

Did the automated tests pass?
Does the library have a history of causing trouble? Maybe I should manually test the change.
Does the package follow semantic versioning? Is their a compatibility break from a major update? What if the break doesn't change the API specification, but critical details in the payload data?

With answers "Yes", "No Problem", and "No", I could just merge...

🛑 WAIT. I've left something out: I know that I don't have 100% test coverage. Are the tests passing because the change is good, or because there is a test coverage gap and I have no idea what will happen next?

The question "Did the automated tests pass?" becomes "Is there automated test coverage in my project and does it cover the use of this dependency? Did those tests pass?"

Let's see an example of untested dependencies slipping through

In this example, suppose a dependency called ThirdPartyAuth is an important part of the business logic. Below I'll show some of the potential misunderstandings from different testing strategies.

// Important business logic that takes a callback/function parameter.
doGoodThings(maybeAMockedRequestFunction) {
  maybeAMockedRequestFunction()
}

RealRequest() {
  return http_request(
    'https://example.com',
    ThirdPartyAuth.coolLibraryAuthV2()
  )
}

doGoodThings(RealRequest)

Test Example #1: No Test

Test success reported with no tests run. Do I remember during code review?

// A commented out test stub.
// Test_doGoodThings()

✔️ Success // No tests were run

Test Example #2: Mocked Test

End-to-end testing can be expensive to write and much more expensive to maintain. Instead, we could mock the responses from external APIs and scope the tests to local business logic.

This test does not cover the use of the ThirdPartyAuth library.

MockRequest() {
  return "OK"
}

Test_doGoodThings(MockRequest)

✔️ Success // What does that mean?

Test Example #3: End-to-End Test

The integration of ThirdPartyAuth.coolLibraryAuthV2() with the codebase is verified.

If the test fails it may be a breaking change in the library or perhaps https://example.com is down.

Test_doGoodThings(RealRequest)

✅ Success // Probability of false negative > false positive

The value of the ✅ passing test is limited by the depth of test coverage and my knowledge of the limitation. No only me: since my team handles maintenance on a rotating basis, the value of the tests depends on how well each teammate understands those limitations.

Functionality not broken, but update not complete

Just because a library works doesn't mean it is used effectively.

When a dependency has significant changes over time, it likely gains new methods, recommended usage patterns, and planned deprecations on old ways of working. Upgrading the dependency without understanding the new, happy paths sets up a future with a broken update.

Today's outdated practice is tomorrow's deprecated design.

Call to Action

Passing tests can be deceptive, and package manifests aren't the only spot in your codebase to need changes when a big update happens.

💡 The robots cannot do all the necessary work to update a dependency safely. You do not need deeper test coverage. You and your collaborators do need a common understanding of the limits of what a passing test means, and what needs to happen to keep shipping quality software after your automated testing has done what it could.

In terms of Renovate configuration, maybe add some extra guidance to those Pull Requests:

{
  "prBodyNotes": "Do tests cover this dependency? The PR needs you."
}

Why Automate?

Adam Ross — Tue, 30 Apr 2024 19:28:36 +0000

Automation saves time and toil when performing a task, and we all appreciate that. This is only one frame for justifying automation but it limits view of other opportunities.

I love how automation transforms the tedium of work into puzzle-solving. Early in my career, I learned that management would value puzzle time (automation work) by pointing out a greater benefit for the effort than the time spent. (Aim for minimal time investment without sacrificing opportunities or incurring hidden costs like a high maintenance burden.)

Folks enjoy citing XKCD 1205 in automation discussions because it does a great job illustrating the opportunity time cost of automation. The time spent to code a new solution for an already solved problem doesn't help solve business goals until after the investment pays off.

XKCD #1205: Is it worth the time?

What's automation anyway? I'd call it "software that reduces the amount of effort humans need to put in to get a job done". It replaces people work with CPU work.

As I continued my journey as a developer, I found that the cost/benefit time ratio around automation didn't line up with my thinking. Time trade-offs are essential, but there are more variables in the trade-off equation than time--other ways of recouping effort. A particular automation may have other goals than time-savings.

Planning for a big push

Perhaps the usual routine doesn't deserve automation. What if your team plans a sprint next month, focusing everyone's effort on a big change? The trade-offs are different: Increasing the frequency of a task and the number of people to onboard to a task both change the math, raising the value of automation. A bit of scripting here also improves the chances the team meets a time-bound goal.

XKCD #2685: 2045

This kind of automation makes the most sense when swarming on a specific task, such as updating code to use a new logging library or migrating data between systems. This may represent a one-time need, so build with minimal flexibility and worry more about good error messages than over-all maintainability. (This is a good use for a shell script shared, maybe not even committed to source control.)

Hiding complexity

Folks need to do perform complex tasks but workflow automation can simplify the details. You don't need to ask everyone to carry the full cognitive load when tools can do more of the lifting. It also provides paths to onboard folks to the team in stages. How fast can folks be productive in your environment? For "new person job satisfaction", it is best to measure this in hours or days instead of weeks.

More persistent automation like this is an abstraction layer to hide a mess.

XKCD #1579: Tech Loops

Reducing cognitive load helps makes a complex system more approachable. More than a simplification in performing the task, it provides the vocabulary for non-experts to discuss the process. When folks have the ability to communicate about a task, new opportunities for collaboration open.

XKCD #2501: Average Familiarity

What do you do every day?

In the last month, I've given thought to the developer superpower of "defining the job you want". Automation is often thought of as a way to create more space to do the job you already have, but it is also a way to redefine the job. You can reinvest the time gained from automation on improving the automation, perhaps even expanding the scope of what the task achieves for your team. As a software developer, maintaining evolving software is more satisfying than turning a crank every 5 minutes!

XKCD #1319: Automation

The Checklist

When I think of automation now, I consider these factors:

Will it save time? When will the up-front investment pay off?
Will it help others contribute in a complex process?
Will it make discussing a big, complex system easier, leading to better high-level decisions?
Will it create space for more interesting activities?
Will it replace existing activities with a style of work I prefer?

Don't limit your thinking about the time payoff of automating something, but do consider all the relevant trade-offs.

Do you have a favorite XKCD comic that helps you think about automation? Let's talk about it!.

Author's Context: I’ve worked on products that can be used to help automation projects, such as Google Cloud Run, Scheduler, and Workflows. Within my organization, I’m currently arguing for the need to use more automation to scale our code review and policy enforcement practices.

Additional Credits

Photo by British Library on Unsplash

Thanks to explain xkcd for XKCD alt text (not to be confused with XKCD hover text, which you get by visiting xkcd.com).

Modernizing cloudbuild.yaml for Container Builds

Adam Ross — Thu, 21 Mar 2024 23:18:39 +0000

tl;dr: Running bash scripts in the Cloud Build documentation tells you use the script property with the automapSubstitutions option and I give this recommendation a 💯.

One of my favorite things to do in writing YAML is minimizing the square brackets and hyphens. The frequency of these things really impacts readability for me, especially when I'm trying to glance at a whole block of markup and assess what it's trying to do.

Over the last couple years, Cloud Build has made improvements in reducing square brackets and enabling more concise configuration for running shell scripts.

I'm going to step through a couple "generations" of simplification I walked through today in overhauling a container image build pipeline.

This is what a lot of minimal Cloud Build configurations look like:

steps:
- id: 'Build Container Image'
  name: 'gcr.io/cloud-builders/docker:latest'
  args: ['build', '--tag', 'gcr.io/${PROJECT_ID}/${_IMAGE}:latest', '.']

- id: 'Push Container Image to Container Registry'
  name: 'gcr.io/cloud-builders/docker:latest'
  args: ['push', 'gcr.io/${PROJECT_ID}/${_IMAGE}:latest']

substitutions:
  _IMAGE: service

Starting character count: 335

Step #1: Remove square-bracketed [distractions]

Use the script property instead of args.

steps:
- id: 'Build Container Image'
  name: 'gcr.io/cloud-builders/docker:latest'
  script: docker build --tag gcr.io/${PROJECT_ID}/${_IMAGE}:latest .

- id: 'Push Container Image to Container Registry'
  name: 'gcr.io/cloud-builders/docker:latest'
  script: docker push gcr.io/${PROJECT_ID}/${_IMAGE}:latest

substitutions:
  _IMAGE: service

🛑 If you try to use this it won't work.

It turns out switching from args to script means the code no longer has substitutions, only environment variables.

A year ago, the next step would be to manually map the substitutions to environment variables:

steps:
- id: 'Build Container Image'
  name: 'gcr.io/cloud-builders/docker:latest'
  script: docker build --tag gcr.io/${PROJECT_ID}/${_IMAGE}:latest .

- id: 'Push Container Image to Container Registry'
  name: 'gcr.io/cloud-builders/docker:latest'
  script: docker push gcr.io/${PROJECT_ID}/${_IMAGE}:latest

options:
  env:
  - PROJECT_ID=$PROJECT
  - _IMAGE=$_IMAGE

substitutions:
  _IMAGE: service

Updated character count: 390 (sometimes shorter isn't clearer)

Step #2: Remove excess hyphenation

Use the automapSubstitutions global option and the substitutions are injected as environment variables to all step scripts.

steps:
- id: 'Build Container Image'
  name: 'gcr.io/cloud-builders/docker:latest'
  script: docker build . --tag gcr.io/${PROJECT_ID}/${_IMAGE}:latest

- id: 'Push Container Image to Container Registry'
  name: 'gcr.io/cloud-builders/docker:latest'
  script: docker push gcr.io/${PROJECT_ID}/${_IMAGE}:latest

options:
  automapSubstitutions: true

substitutions:
  _IMAGE: service

Updated character count: 373 (progress, but I'd be more satisfied if we got back to 335 somehow)

Simplify the Image Push

Since this Cloud Build configuration doesn't have any follow-up steps that need to use the image outside the build, I can simplify further, by replacing the step Push Container Image to Container Registry with the images property. (Read more about that in Remember images in your cloudbuild.yaml! by @glasnt).

steps:
- id: 'Build Container Image'
  name: 'gcr.io/cloud-builders/docker:latest'
  script: docker build . --tag gcr.io/${PROJECT_ID}/${_IMAGE}:latest

images:
- gcr.io/${PROJECT_ID}/${_IMAGE}:latest

options:
  automapSubstitutions: true

substitutions:
  _IMAGE: service

This doesn't reduce hyphens any further, but it does reduce two lines of configuration.

Updated character count: 263 (Success!)

Bonus Round: Migrating to Artifact Registry

You may have noticed this configuration ships the container image to a Google Container Registry URL! There's a good chance if you are modernizing your Cloud Build YAML this way you've got some GCR in place, if so, you need to migrate soon.

Check out Transition from Container Registry. There are options to migrate while keeping the gcr.io URL.

Say I've already created a new Artifact Registry instance for my container, what does that YAML look like?

(Let's say it's a multi-regional repository called container in the us location.)

steps:
- id: 'Build Container Image'
  name: 'gcr.io/cloud-builders/docker:latest'
  script: docker build . --tag "us-docker.pkg.dev/${PROJECT_ID}/container/${_IMAGE}:latest"

images:
- "us-docker.pkg.dev/${PROJECT_ID}/container/${_IMAGE}:latest"

options:
  automapSubstitutions: true

substitutions:
  _IMAGE: service

Since an Artifact Repository is a Cloud resource, a project might have more than one. This encourages me to parameterize the location and repository name, so that development & testing are easier.

steps:
- id: 'Build Container Image'
  name: 'gcr.io/cloud-builders/docker:latest'
  script: docker build . --tag "${_LOCATION}-docker.pkg.dev/${PROJECT_ID}/${_REPO}/${_IMAGE}:latest"

images:
- "${_LOCATION}-docker.pkg.dev/${PROJECT_ID}/${_REPO}/${_IMAGE}:latest"

options:
  automapSubstitutions: true

substitutions:
  _IMAGE: service
  _LOCATION: us
  _REPO: container

Updated character count: 358 (A little longer than 335, but Artifact Registry is more verbose than Container Registry, not much we can do about that.)

In a more complicated YAML block, automapSubstitutions would make a much bigger difference. I go out of my way to avoid using args, and this option helps me do that without needing to add a lot of boilerplate config.

Aside Info: Reader, maybe you'll suggest I delete all those pesky curly braces around my variables, getting me a win at 342 characters. This stackoverflow answer explains the value of the curly braces, and I'm one of the people that makes curly braced variables part of my "shell variables in strings" practice.

Migrating to Cloud Run's Secret Manager Integration from Libraries

Adam Ross — Thu, 27 May 2021 16:36:32 +0000

Secret Manager is easier to use with Cloud Run since I last wrote about "serverless mysteries". At the time of that writing, the best way to use a Secret in Cloud Run was to add code to your service to integrate with the Secret Manager API. Now Cloud Run does that for you!

In this post we'll migrate the "Loyalty Checker" service from using the Secret Manager client library to use Cloud Run's direct Secret Manager integration. (As of this writing, the integration is still in preview.)

Prepare to follow along

If you're still set up from last time, just open Cloud Shell and continue where we left off.

Otherwise, please walk through the last post then continue here to enjoy the feeling of deleting code.

Modify the Loyalty Checker code

Direct integration with Secret Manager makes the code simpler:

Remove code to initialize the client library
Remove code to retrieve the secret on the first request
Rename the SECRET_NAME variable to SECRET_VALUE, the service will have direct access to the secret from the environment variable.

After these changes the service has shrunk by 46 lines of code, and no longer has any dependencies on Secret Manager:

Re-deploy the Service

We'll combine the Secret integration with the new all-in-one deploy command:

gcloud beta run deploy loyalty \
  --source . \
  --remove-env-vars SECRET_NAME \
  --update-secrets SECRET_VALUE=you-know-who:1

The --source flag tells gcloud to check for a Dockerfile in the current directory and use Cloud Build under the hood to build a container image without a separate command. (Read more about deploying from source code if you're curious about the details.)
The --remove-env-vars flag removes the no-longer-needed reference to a secret for our code to load.
The --update-secrets flag integrates a new secret with the service without disrupting any existing secrets. (Use --set-secrets to simultaneously remove any the other secrets.)

One of the nice things about Cloud Run's secrets flag is how you can use just the "resource ID" part of the overall resource name. Secrets are normally referenced as "projects/PROJECT_ID/secrets/SECRET_ID/versions/VERSION" but all you need to reference a secret in the same project is SECRET_ID:VERSION.

No IAM Update?!

There's no need to change IAM settings. Last time we set up a dedicated service account and gave it access to the secret. The same configuration works for client libraries or for this direct integration.

Next Steps

If you want to be able to "live refresh" the secret in your service, check out the documentation to learn how to mount your secrets as a volume. The environment variable method in this post is configured when an instance is started and is never updated.

All code © Google w/ Apache 2 license

Serverless Mysteries with Secret Manager Libraries on Google Cloud

Adam Ross — Fri, 06 Mar 2020 18:10:03 +0000

Secrets are private information used by an application. They're often obfuscated passphrases used to authenticate with APIs or connect to databases: not something that should ever be in your code! Until Google Cloud's Secret Manager, many tutorials suggested using secrets on platforms such as App Engine or Cloud Run with the magic of plaintext (in-the-clear) environment variables.

Unencrypted environment variables aren't very secret, but at least they don't need to be committed to your codebase. Using one to carry a secret is simple but it's not very secure:

YOU_KNOW_WHO="Lord Voldemort" npm start

Environment variables are a gossipy approach to injecting data, some examples of weak security include:

developers type them into terminal history
configuration manifests reveal them
exfiltrating them from production is as easy as a stray bit of code like console.log(process.env) and access to view the logs

Secret Manager provides a better approach for managing your secrets:

Create a named secret
Grant your code access to it
Watch the records of everything that tries to access it

Secret manager works smoothly. My first experience was a bit magical (work with me, we're on a theme). And what secrets are there in magic? In myth and fiction, names are significant mysteries doubling as a magic spell or a homing beacon for evil ears. How many times in the Harry Potter by J.K. Rowling series did we hear the phrase "You-Know-Who" or "He-Who-Must-Not-Be-Named" as an alias for the big bad?

In this post we'll walk through creating a "Ministry of Magic Loyalty Checker" service on Cloud Run, using Secret Manager to supply the dark wizard's true name.

Prepare to follow along

If you'd like to follow along, you'll need a Google Cloud project, a working installation of the gcloud CLI, and enabled APIs for Cloud Run and Secret Manager.

Assuming you already have a project, the fastest way to get started is a quick 2-step:

Open the Cloud Shell to get a VM workspace provisioned with all the tools you'll need.
Enable the APIs with a CLI command:

gcloud services enable run.googleapis.com secretmanager.googleapis.com

Step 1: Keeping Secrets Close

Secret Manager is a Key-Value Store, one with encryption, versioning, access control, and audit logging around individual key-values. Each one is initialized before it can be assigned a value.

Create a Secret

Use gcloud to create a new secret. This is like a variable declaration: it's a placeholder for something to come.

gcloud secrets create you-know-who --replication-policy="automatic"

The replication policy flag instructs Google Cloud to manage storage locations of the key.

Create a Version

A version is the secret data itself: every secret value is going to change eventually. Versioning as a first-class concept makes rotation much easier.

Open secret.txt in your favorite editor
Type "Lord Voldemort"
Save the file

This keeps the secret out of your terminal history.

gcloud secrets versions add "you-know-who" --data-file secret.txt
Created version [1] of the secret [you-know-who].

Step 2: Configure the Cloud Run service identity

Cloud Run allows you to specify the "service identity" of each service. This allows you to attach IAM permissions to your service to limit it's access to other Cloud resources. I prefer to create one service identity for each Cloud Run service so my services run with minimal privileges.

This feature ensures only the loyalty-check service is able to access the secret because only it (and Project Owners) have read permission.

Create a service account for the loyalty-checker service:

gcloud iam service-accounts create loyalty-identity

Grant the service account access to the you-know-who secret:

gcloud secrets add-iam-policy-binding you-know-who \
 --member serviceAccount:loyalty-identity@example-project-id.iam.gserviceaccount.com \
 --role roles/secretmanager.secretAccessor

Step 3: Implement the "Loyalty Check" service

The code of this service is mostly interesting as an example of how to use the Secret Manager client libraries. We'll walk through it step by step.

Create a new `npm` package

The only dependencies are Express v4 and the Secret Manager v3 client library.

npm init loyalty
npm i express@4 @google-cloud/secret-manager@3

Create a Dockerfile

A Dockerfile is used to define how to create a container image for deployment to Cloud Run.

This Dockerfile uses a Node.js 14 base image, copies in the package.json manifest to install npm dependencies, and adds the custom source code. When the container is started npm start is executed to run the service.

Create the application code

Both code blocks are placed in a single index.js file for use.

The code uses the Secret Manager client library to dig up the secret:

The code above does the following:

Ensures the SECRET environment variable is present to name the secret.
Defines a getSecret() function that uses global state to keep the Secret Manager client library in memory for reuse.
Provides an escape valve if the code is running locally: just call the service with SECRET=dev npm start
Logs an error if the API interaction fails but allows the caller to decide what to do.

Next define a web server that takes requests to perform a loyalty check. The secret is only retrieved when first requested. No need to have it decrypted in active memory until it's needed!

By storing the secret value in global state, changes to the secret value or access revocation will not affect this instance of the service. On Cloud Run, this means this value will be used by a given container instance until it scales down.

Who checks the loyalty checker?

Everyone is loyal to the Ministry of Magic unless the Loyalty Checker knows the name of You-Know-Who. Seems like this secret has turned the service into a traitor!

Maybe this could be made a little smarter: according to the quota reference Secret Manager supports payloads up to 64 KiB per key, so we could encode JSON as a string and stash a lookup table:

{
  "Harry Potter": "Ministry of Magic",
  "Voldemort": "Lord Voldemort",
  "Severus Snape": "unknown",
  "Lucius Malfoy": "Lord Vodemort"
}

This works for several characters, but quickly reaches a point where using a database is more sensible. The secret's role would change from directly holding the mystery to holding the database credentials to look up a trove of PII.

Step 4: Ship the Cloud Run service

Now that we're code complete, let's get the Loyalty Checker into production.

Build the container

This could be done with docker, but today we'll use Cloud Build. This step builds the service into a container image and pushes it to Google Container Registry. From there we can deploy to Cloud Run.

gcloud builds submit --tag gcr.io/$GOOGLE_CLOUD_PROJECT/loyalty

Deploy the service

This step is a bit more complicated than the typical Cloud Run deployment, because we need to specify the service account and configure the full name of the secret:

gcloud run deploy loyalty \
  --image gcr.io/$GOOGLE_CLOUD_PROJECT/loyalty \
  --update-env-vars SECRET_NAME=projects/$GOOGLE_CLOUD_PROJECT/secrets/you-know-who/versions/1 \
  --service-account loyalty-identity@adamross-svls-kibble.iam.gserviceaccount.com \
  --allow-unauthenticated \
  --platform managed \
  --region us-central1

The 1 at the end of the SECRET_NAME value specifies the version of the secret to use. When new versions are created, the number increments. You can double-check what the most recent enabled version is by running gcloud secrets versions list you-know-who.

Step 5: Try out the "Loyalty Checker"

Use curl to send an HTTP request to the URL of your Cloud Run service. You'll see that URL on screen after you deploy.

curl https://loyalty-[HASH]-uc.a.run.app/loyalty
You serve the Lord Voldemort!

We've created a new Cloud Run service which pulls essential configuration from Secret Manager. Access has been carefully managed to limit it to a single service account, which is only used by a single Cloud Run service. That's the traditional way to keep a secret: by not sharing it with anyone. Unfortunately once *this service* unlocks the true name it tells anyone who asks.

Down with Environment Variables!?

What about the "Boy Who Lived," is the name "Harry Potter" a secret too? No, it was a newspaper headline. If folks didn't know the name, he wouldn't have been their hero.

Don't overuse secrets: they're not only more expensive than environment variables, but the more you hide production configuration the more mysteries there are when it's time to troubleshoot production failures.

Next Steps

I'm looking forward to how Secret Manager helps improve security practices on Google Cloud. What will your first secret be?

Learn how to leave the libraries behind with Cloud Run's built-in Secret Manager integration, perhaps by reading the follow-up blog post?

Generalize your product knowledge by reading the Secret Manager documentation.

Portable code migrating across Google Cloud's serverless platforms

Adam Ross — Thu, 05 Dec 2019 17:37:36 +0000

Google Cloud has a number of options to run your code. We'll focus on three serverless platforms: Cloud Functions, App Engine, and Cloud Run, all of which automatically scale by traffic and bill by usage. The biggest difference between these platforms is the level of abstraction. We can deploy a function to Cloud Functions, an app to App Engine, or an app with a custom runtime (a Docker container) to Cloud Run.

The most abstract serverless hosting option is a function. It has no code except the core business logic of a specialized service. Two challenges in working with serverless functions include:

Developing locally without a platform to convert requests to function calls
Finding hosting alternatives when the infrastructure, runtime, or cost constraints of your platform are reached

Using a "Hello World" function with an international twist: "Hello Translate", we'll explore a new way to mitigate these challenges on Google Cloud.

Standalone Functions with the Functions Framework

Like all Google Cloud Functions runtimes, the Node.js v10 runtime supports direct HTTP triggers while removing the overhead of HTTP routing from developers. Under the hood this magic is done by the open source Node.js functions framework. The framework installs as a normal dependency for your function and encapsulates express.js to provide clean and standardized HTTP request handling.

Because it provides an HTTP server you can run your function as a localhost web service, or deploy it to any web hosting platform.

Write business logic

Add a Function Framework

Deploy to a web host

Let's get to the code for "Hello Translate".

Writing a Function

Create a new npm package and install the functions framework and the translate library:

npm init
npm install @google-cloud/functions-framework@1 @google-cloud/translate@5

(The @1 and @5 bits instruct npm to retrieve specific major versions of the libraries. It makes the code in this post less timeless, but more likely to work as long as these versions are compatible with Google Cloud.)

Cook up the code for "Hello Translate!":

Before we're ready to deploy HelloTranslate, let's modify the package.json to define a start script:

 "scripts": {
    "start": "functions-framework --target=helloTranslate"
 }

When npm start is run, the Node process will start a web server in the function framework package which is configured to use the target function above. This enables local development with no other special dependencies. The same mechanism will be used in Cloud Functions, App Engine, and Cloud Run.

Getting setup to work with Google Cloud

Set up a development environment and configure your project to support the "Hello Translate" function:

Create or select a Google Cloud Project. Add it to your shell environment with export PROJECT=[Your Project ID] to streamline the commands below.
Install and authenticate with the Cloud SDK
Enable the the Translate API: gcloud services enable translate.googleapis.com

Testing on Cloud Functions

Start by confirming this is works as a Cloud Function, let's deploy it!

gcloud functions deploy helloTranslate \
    --runtime=nodejs10 --region=us-central1 --trigger-http

Wait a couple minutes and we're ready to make a request. Use curl to ask for "Hello World!" in the default Spanish language:

curl https://us-central1-${PROJECT}.cloudfunctions.net/helloTranslate
Hola Mundo!

To learn more about how to use Cloud Functions, check out the HTTP Functions how-to.

Deploying a Function to App Engine

Now that we've confirmed the code works with Cloud Functions, let's see it work as an app on App Engine. App Engine allows more control over the instance scaling and behavior, and the requirement to deploy an HTTP server is met by the function framework. This means we can deploy code to App Engine without stopping for code changes.

Adapt Hello Translate to App Engine by following these steps:

1) Add an app.yaml configuration file to declare the runtime:

runtime: nodejs10

2) Deploy the code

gcloud app deploy

3) Ask our production service for the latest translation of "Hello World!" in Yoruba:

curl https://${PROJECT}.appspot.com?lang=yo
Mo ki O Ile Aiye!

To learn more about how to use App Engine Standard, check out the Building an App Quickstart.

Deploying a Function to Cloud Run

Cloud Run is similar to App Engine in expecting a full HTTP app to be deployed. Unlike App Engine, Cloud Run is driven by containers. Cloud Run does not know if you are using javascript, java, or a web server written in BASH. Containers allow the developer to customize the runtime environment.

Adapt Hello Translate to Cloud Run by following these steps:

1) Add a Dockerfile. Copy the Node.js Dockerfile from the Cloud Run quickstart. Here's a streamlined copy for quick reading:

2) Build the Container.

gcloud builds submit --tag=gcr.io/${PROJECT}/hello-translate

3) Deploy the service.

gcloud run deploy hello-translate \
    --image=gcr.io/${PROJECT}/hello-translate

4) Ask our production service for the latest translation of "Hello World!" in Japanese:

curl https://hello-translate-j6jxwetqdq-uc.a.run.app?lang=ja
"こんにちは世界"

To learn more about how to use Cloud Run, check out the Build and Deploy Quickstart.

Helpful Abstraction!

The function framework is a good case of an abstraction. Code that makes sense to deploy as a function doesn't need much customization in HTTP handling, so why include the boilerplate anywhere that code is hosted? With an open source package, your functions-that-are-services code is more portable across compute platforms.

If your function is chafing at the runtime or concurrency limitations of Cloud Functions, consider using a Functions Framework to migrate your service to another hosting option such as Cloud Run. With a little more work, you can even bring over event-driven functions by configuring Pub/Sub push subscriptions.

Read Onward

For an example application using a Functions Framework to be simultaneously deployable to Cloud Functions and Cloud Run, check out Grant Timmerman's Pizza Map sample.

DEV Community: Adam Ross

How My Team Aligns on Prompting for Production

Natural Language is a Fuzzy Programming Language

Be Efficient with Brains (Pair & Review)

Document "The Why" and manage change

Separation of Concerns (Use Dedicated Files)

Use Structured Output

From Magic to Engineering

Join the conversation

Read More

[Boost]

The lumberjack paradox: From theory to practice

Write to Google Sheets from a local script via gcloud CLI authentication

🔐 Local authentication with gcloud CLI

🗝️ Initialize a Node client for Google Sheets

🖇 Append data to the sheet

🚀 Ship it!

Where to Host My Static Tech Blog

Context & Problem Statement

Priorities & Constraints

Considered Options

Decision

Alternatives Considered

Expected Consequences

Revisiting this Decision

Dependency Updates: More to Do Than Review

Context: What do these dependency automation tools do?

The tools did all the work, I can review & merge, right?

Let's see an example of untested dependencies slipping through

Test Example #1: No Test

Test Example #2: Mocked Test

Test Example #3: End-to-End Test

Functionality not broken, but update not complete

Call to Action

Why Automate?

Planning for a big push

Hiding complexity

What do you do every day?

The Checklist

Additional Credits

Modernizing cloudbuild.yaml for Container Builds

Step #1: Remove square-bracketed [distractions]

Step #2: Remove excess hyphenation

Simplify the Image Push

Bonus Round: Migrating to Artifact Registry

Migrating to Cloud Run's Secret Manager Integration from Libraries

Prepare to follow along

Modify the Loyalty Checker code

Re-deploy the Service

No IAM Update?!

Next Steps

Serverless Mysteries with Secret Manager Libraries on Google Cloud

Prepare to follow along

Step 1: Keeping Secrets Close

Create a Secret

Create a Version

Step 2: Configure the Cloud Run service identity

Step 3: Implement the "Loyalty Check" service

Create a new npm package

Create a Dockerfile

Create the application code

Who checks the loyalty checker?

Step 4: Ship the Cloud Run service

Build the container

Deploy the service

Step 5: Try out the "Loyalty Checker"

Down with Environment Variables!?

Next Steps

Portable code migrating across Google Cloud's serverless platforms

Standalone Functions with the Functions Framework

Writing a Function

Getting setup to work with Google Cloud

Testing on Cloud Functions

Deploying a Function to App Engine

Deploying a Function to Cloud Run

Helpful Abstraction!

Read Onward

Create a new `npm` package