DEV Community: Katie McLaughlin

Step up to the Sheets: AI Eval Export and Illustrating Data

Katie McLaughlin — Tue, 01 Sep 2026 07:00:00 +0000

Welcome to part three of our series about designing, analyzing and visualizing AI Evals! As with the previous entries you are more than welcome to simply read over my exploration or for those of you looking for extra credit, feel free to play along at home.

Note: This blog series contains AI-generated diagrams alongside actual screenshots and hand-drawn edits of both. AI also assisted in minor copy editing.

Implementation prerequisites

If you intend to (re)implement the walkthrough, all you need to do right now is complete Part 1 (Designing AI Evals: Clarity Now and Visualization Next) and then install pandas; Part 2 (AI Evals at a Glance: Heatmaps for Stakeholders) is illustrative and will give you a deeper understanding going forward both of AI evals and why the following may be useful but it’s more elective than anything else.

Back to the story at hand

While engineering teams might evaluate agents using test automations, non-technical stakeholders (product managers, business leads, or executive sponsors) often use spreadsheets as their lingua franca and as such it would make sense to speak to them in a way they are almost certain to understand and which enables you all to collaboratively make visualizations to further get out the word.

To make this a reality, today we’ll use a pandas pipeline that transforms raw Inspect AI evaluation logs into a rich CSV optimized for Google Sheets, allowing you or anyone on your team with a little spreadsheet knowledge to make boardroom-ready visuals.

Drawing again from previous blog entries, the evals we are working on have three dimensions: model x skill x sample [and epoch] that cross to form the following matrix:

In Part 2 we created a simplified heatmap reflecting a planar slice of this space as reduced over the sample dimension. While we will ultimately do something similar in a future blog post using Data Studio, for the time being we actually want to keep all 32 combinations separate and pull more of their underlying stats from the inspect eval files and derive further information using pandas. This information will be exported as a flat csv file we can then import and view in Google Sheets.

The extractor `tocsv.py`

This script is used to parse the eval logs using Inspect AI’s python pandas interface. The script is relatively self documenting, and you can read up more on pandas and pandas’ csv IO if you are unfamiliar.

Heavy lifting in Python: Custom telemetry columns

While most of the script deals with mapping scalar metrics the like of which we’ve already discussed, there are two that are purely logical/boolean. They represent specific statistical cases that we will allow end users to partition the data set over for analysis later. These boolean conditions are derived and since we want to pare away excess information and therefore not supply the full eval files to Google Sheets, tocsv.py does the heavy lifting upstream during log ingestion.

It pre-computes two key boolean columns to simplify downstream dashboard filtering:

completed: Flags whether a sample finished cleanly (no API quota 429 errors, timeout truncations, or container crashes) with a valid score, cleanly isolating model capability from infrastructure noise.
is_baseline_or_has_activated_skill: Distinguishes Baseline control cohorts (always TRUE) and Active Skill interventions (TRUE when the skill tool was actually “activated” which in inspect means uptaken) from Dormant runs (FALSE, where skills were available but never activated).

In Google Sheets or Data Studio non-technical stakeholders can simply filter/toggle these pre-computed boolean keys to isolate clean, active performance (or otherwise) instantly.

Exporting to Google Sheets

In order to execute the pipeline I called:

python3 tocsv.py

The output file data_mega_export.csv is a standard comma-separated values file where each cell is surrounded by commas. You can preview the raw and unformatted contents in a standard text editor. While most of this is normal spreadsheet data such as strings and integers, you will also see fairly complicated Google Sheets formulas resulting from interpolating eval metrics into formula templates.

The above formula for instance will be transformed into a bar graph normalized against the range of values in the same column as its source data and tiered into colors by tertile.

How do you take this text and get it to show its stuff? By opening sheets.new of course, and importing data_mega_export.csv. When presented with the import dialog check Convert text to numbers, dates, and formulas.

On clicking Import data Google Sheets will instantly display the standard cells and render active, color-coded visual sparklines directly inside data cells!

What conclusions can you draw from what you see? One obvious one in this example is that the correlation between score and latency (how long an agent takes to answer a question) may not be very strict. But what if you wanted to get something deeper?

Well, since this is a spreadsheet, obviously you can make pivot tables, and custom functions or sort on multiple keys… you know, the whole accountant’s playbook. Let’s say though that you want custom views into the data rather than to permanently mutate it. How would you go about doing that?

Google Sheets, unlike inspect view and inspect viz heatmaps, allows you to dynamically sort AND filter on more than one key. Not only that but you can sort on ANY key, so I elected to do so with the previously unavailable score and latency fields. As shown in the image above, use Data > Create filter view which presents you with a new set of controls at the top of the main columns.

Note: Filter views do not mutate or change the underlying contents of a sheet; Instead they are (optionally named and shareable) configurations for restricting what rows are displayed and in what order.

If, as in the picture, you click the cost column > Filter by condition, select the Less than option and fill the comparison with 0.01, you can narrow the displayed rows to only those with costs less than a cent. This can be used (for instance) to only look over the samples that were “less costly” to try to analyze why.

Filter views (despite their name) also allow you to sort the display of the data. Because the sort operations are cumulative, the “primary” filter needs to be applied last. Thus, after sorting the filter view first by latency and then by score (both ‘Z to A’ aka decreasing) gives us the following:

Optionally you can name and save this filter view to toggle it on and off as well as share it with other collaborators. To learn more about Filter views, you can read Sort & filter your data.

This ability to sort and filter the display live without changing the underlying information, makes presenting the data and analyzing it during discussions with teammates significantly easier and faster.

Why leave the cradle?

Now, in spite of this tremendous flexibility, sheets have some drawbacks as visualization tools:

Spreadsheet formulas are cumbersome for nested IF statements and filters unless you define custom macros or JavaScript functions leading to code that's harder and harder to keep connected in one's mental model.
Flattened eval runs need to be inspected under multiple filter views to begin to understand complex trends or correlations across all of them.

So, how can we take this to the next level and circumvent those issues? What about an environment, which while it still offers you the ability to custom script anything you need, comes with a lot of WYSIWYG controllable visualizations and graphs, enabling you, without code, to bring data to an even more vibrant life? Better yet, what if that environment comes with built in visualizations for 3 dimensions or more and has broad compatibility with multiple data source types?

In Part 4, we’ll connect this Google Sheet directly to Data Studio to build a codeless executive summary dashboard and be able to quickly and intuitively display our findings!

AI Evals at a Glance: Heatmaps for Stakeholders

Katie McLaughlin — Tue, 25 Aug 2026 07:00:00 +0000

Visualizing AI evals with Inspect Viz

Welcome back to our blog series on running, analyzing and visualizing AI evals. Last time we discussed how to design and run evals for analysis and visualization using inspect eval and inspect view. Many of these methods, such as rearranging the dashboard columns and sorting on them to determine model vs model and skill vs no-skill differences in metrics, can give you a broad overview of the field. While this can inspire further and deeper inquiry, it runs into issues with how predictive it is and how to communicate findings to other people. Imagine the following:

The 40-log gridlock

Imagine a data science lead needing to present a performance breakdown of four LLM candidates across ten internal tools during a high-stakes, time-crunched live meeting. Reordering and filtering the evals across multiple dimensions is hard to read on a presentation screen and requires doing mental math with an audience: riveting stuff that they definitely won’t fall asleep during of course.

What if instead you had a tool that allowed you to easily automate rendering a comparison of the different configurations? What if lining them up by model or by skill showed clear and legible patterns?

Well, let me introduce you to heatmaps! A heatmap is a 2D visual matrix where color intensity represents numerical magnitude—in our case, quadratic curved accuracy scores. By leveraging the human eye's natural ability to distinguish color gradients, heatmaps instantly reveal performance spectrums at a single glance. This makes trends across dependent and independent variables clear, intuitive, and highly legible.

While you can simply read along and learn about using inspect viz to generate heatmaps for inspect AI evals, if you intend to follow along and recreate the charts, please start by replicating the steps in the first entry and come back afterward.

Note: This blog series contains AI-generated diagrams alongside actual screenshots and hand-drawn edits of both. AI also assisted in minor copy editing.

Follow-along requirements

For those playing along at home, in order to perform the following, you will need the inspect_viz library installed in your environment (alongside inspect_ai and pandas).

From there, take a look at the heatmap script. While there are some intricacies to the script (such as establishing its CLI API) the script mainly wraps scores_heatmap(viz_data, orientation="horizontal", ...) and marshals our data to assemble a heatmap of it.

If you run the script

python3 inspect_viz_heatmap.py logs -o heatmap.html

you will get a visual that looks like the following.

This heatmap has two axes, one is the model (vertical) which allows you to visually break down your analysis to just skill inclusions for the same model, the other is the skills (horizontal) and going down the row allows you to demonstrate how changes to the model affect the evaluation.

Stakeholders (like investors or your boss) can now spot skill gains and model ceilings at a glance without reading line-by-line log values. Some quick takeaways:

Top-performing configurations (such as 3.6-flash + gemini_api_skill at 1.0) stand out in high-intensity shades
Within the aggregated 2D heatmap view of this dataset, 3.6-flash trended higher in score on average across the evaluated skill configurations compared to 3.5-flash-lite.
Skills tended to improve scores, all other things held equal.

Caption: The inspect viz heatmap projects our full Evaluation Matrix down to a 2D plane by collapsing individual task samples and epochs into aggregate accuracy cells along the Model and Skill axes.

Cognitive overhead and segue to BI analytics

While matrix scoreboards provide quick visibility into accuracy differences across models, raw heatmaps suffer from key limitations:

2D Constraint: Heatmaps map only two axes to a single color intensity (in our case Accuracy).
Missing Business Context: A single heatmap cannot simultaneously show token costs, system latency, or operational reliability.
Hidden Variance: Collapsing samples for a given configuration removes the information necessary to determine if a result is consistent or wildly erratic.

To bridge CLI logs with interactive, multidimensional and in-depth reporting, we need a pipeline that exports the raw telemetry metrics into spreadsheets and BI dashboards.

In Part 3, we’ll build a python script to convert output data for processing first in Google Sheets and ultimately Data Studio!

Designing AI Evals: Clarity Now and Visualization Next

Katie McLaughlin — Tue, 18 Aug 2026 07:00:00 +0000

AI evals and analysis

Let's say you're testing out new AI tools. Perhaps you implement and run analytics for an Ad Agency and hope to automate deploying your standard event schema, or are a podcast producer automating generating social copy from your newest ep. While modern, newly trained LLMs can likely one-shot a lot of these tasks – this specificity might necessitate wasting tokens and time repeatedly prompting them with the same resources, descriptions and scripts. With this in mind, you investigate tooling whether that be an MCP server, an agent skill or an agent plugin.

The problem is, how do you know if a skill (whether developed by you or open sourced by someone else) is worth your time or, perhaps more importantly, your tokens and quota? How can one go about designing these (more) objective evaluations of AI tools and, from there, collecting and analyzing relevant metrics? That is exactly the context for using open source eval frameworks, like Inspect AI and Harbor to evaluate agent skills using open source frameworks. But how do you "extend the evaluation", use visualizations to spot trends and collaboratively explore alternative paths forward using Google Sheets and Data Studio?

These questions and more are exactly what I’m hoping to demonstrate for you in this series!

While you are more than welcome to simply read about how and why I conducted my own investigation, you can alternatively follow my lead and run the benchmark scripts.

So, if you’re hoping to play along, take a moment to complete the aforementioned codelab and come back when you’re done. Don’t worry, we’ll still be here when you get back!

Note: This blog series contains AI-generated diagrams
alongside actual screenshots and hand-drawn edits of
both. AI also assisted in minor copy editing.

Level up your evals

In the codelab, we learned how to run evals with Gemini CLI, Inspect and Inspect SWE in an isolated Docker Sandbox to understand how well each skill aids the agent in answering the same question.

For those playing along at home, please additionally install inspect view now before you need it to run the evals below; be prepared to have to wait for a couple minutes for evals to finish depending on your machine and quota usage.

Regarding our new source files while they are heavily commented and hopefully written in a self describing way, I’ll explain further subsequently.

A note on models

For the purpose of my investigation demo, I used three different models: google/gemini-3.5-flash-lite and google/gemini-3.6-flash as "solvers" (the models under evaluation), and google/gemini-3.1-flash-lite as a "grader" (the model rating the runs). The three of them (and models more broadly) differ in many ways but more specifically on problem solving ability, speed and cost.

We intentionally offloaded grading for this demo to a previous-generation model because by changing rubric criteria to strict binary decisions and applying a reduction programmatically, it delivers sufficiently robust evaluations without burning through solver quota. For a production evaluation system, consider investigating using newer and more capable models as graders because they will likely have narrower confidence intervals.

For a full technical breakdown of this decoupled grader architecture and to find out how you could substitute in your own choice of models, see the README section on Decoupled Grader & Multidimensional Rubrics.

Setting up the eval

Reproducibility & Local Setup Note: If you are following along locally, clone the domain skill definitions into google-skills/ before running benchmark sweeps:

git clone https://github.com/google/skills.git google-skills

All evaluations in this series were benchmarked on Python 3.13 using inspect-ai (v0.3.247), inspect-swe (v0.2.66), inspect-viz (v0.4.1), and pandas (v3.0.3). If upstream PyPI releases introduce breaking changes, check the README's Environment & Dependencies specification for exact version pins and instructions on how to reproduce the configuration.

For this investigation I used a new eval script; while the specifics of the script are important for anyone who hopes to make their own evals or run the code as we go, if you're primarily interested in analysis and visualization, feel free to skip to the next section.

The original script ran a small batch of tests on a local machine. While this version can be and was run on a local machine, it prioritizes three architectural dimensions needed for running automated tests as part of a larger scale development flow:

External Configs (providing the evals and solver system prompt as external files for separation of concerns).
Quota Management (for the GenAI API and package management).
Multidimensional Evaluations (expanding to arbitrary fact counts for graders).

For a complete technical breakdown of these architectural pillars—including external configuration schemas (questions.json, thrifty_system_prompt.txt), solver rate-limiting defenses (version="0.51.0" pinning), grader quota decoupling, and score curving math formulas see the README's Evaluation Pipeline Architecture & Technical Reference.

Running the eval

These commands create a matrix of evals subject to model x skill condition x sample x epoch. For a quick visual (that I’ll repeatedly reference throughout this), take a look below.

Caption: Visualization of the eval matrix that will be scored and compared. In Inspect AI, rubric elements, represented here as Facts, are graded Correct (C) or Incorrect (I).

With all that said, it’s finally time for the rubber to meet the road. I ran our evals with the following command

inspect eval skills-eval.py \
  --model google/gemini-3.5-flash-lite,google/gemini-3.6-flash \
  --time-limit 300 \
  --epochs 2 \
  --max-tasks 4 \
  -T web_access=false

This command runs a 4-way parallel sweep across models and skill conditions while enforcing a 300-second task timeout and disabling web search to minimize token spend. See the README for a complete CLI Parameter Reference and details on External Configuration Template Mechanics.

Run and wait

If you intend to and haven't done so already, run the command line. If you're interested in how to read over and interact with the Terminal User Interface, please refer to the previous codelab.

Watch the running command line long enough to determine it hasn’t crashed or hit an obvious error and then maybe take a moment to make some food, grab some water or take a walk around the block. Enjoy the small things in life, okay? Worst case, you get a 20 minute walk out of it, not so bad eh!?

Viewing the evals

With that said and done, reading the terminal output is only the most basic way of understanding the evals. Now that you have the raw eval logs, what do you need to do to begin to use these analytically and, more importantly, persuasively?

To start off, I simply ran

inspect view

and clicked the link to open the browser to the GUI.

For step-by-step diagnostic trace analysis in the browser GUI, see the README section on Local Diagnostic Trace Analysis (inspect view).

Mine came out like this (after a little bit of column drag-and-drop):

While we covered the rudiments of looking over the inspect view GUI in Evaluate agent skills using open source frameworks, there is significantly more that can be done with it. Speaking of which, the TASK ARGS column (not pictured) serves as a quick reference for what arguments were passed to a task.

Macro view: Comparing runs

While LLM’s are inherently stochastic (and given floating point hardware, nondeterministic), when running evals we are trying to use sampling to characterize the average observable metrics (e.g. correctness, latency and token usage) which our particular configuration model x skill scores on relevant questions. As such, the eval tasks were set up to characterize how a change to an independent variable (model or skill) affect dependent variables (the aforementioned metrics).

A. Comparing models head to head

Grouping evaluation runs by model in inspect view allows for a direct "eyeball" inspection of how skill inclusion alters accuracy across baseline controls—given a specific model, what is the impact of adding a skill?

I have added some annotations to make metric change from baseline to skilled a little more apparent:

The above shows that

In all cases, the addition of a skill improved or tied scores relative to the same model’s baseline.
All of these cases also increased duration

Another simple measurement that can be done is to look at the TOKENS column where gemini-api actually led to a decrease in usage relative to its baseline in 3.6-flash.

While the general relationship between tokens and score is difficult to parse visually, one notable result: for 3 out of 4 task x model configurations, gemini-3.6-flash used more tokens than the comparable 3.5-flash-lite run.

B. Comparing skill conditions (skill vs. baseline)

Now to analyze the inverse; in more concrete terms: given a particular skill, what is the effect of changing the underlying model?

What was the effect of the change from 3.5-flash-lite to 3.6-flash?

For 3 out of 4 of these, the changes between the two models amounted to an up to 45% increase in accuracy. For whatever reason, however, changing the gcloud skill from using one model to another led to a slight decrease.

While these are both valuable findings, they warrant further questions:

Are these representative samples?
If these findings are repeatable, how can we characterize the relationship between the existing metrics?

For the first of those, if you want to make more representative samples, know that you should conduct further research using more questions, samples and epochs and compare those metrics with these. For the sake of this blog series though, I’ll leave that for you to do.

For the second though, we can and will endeavor to do so. That said, it might help in designing follow up analysis to think a little bit about common patterns encountered during evals. This is especially true for those running evals themselves (possibly on different skills or rubrics); you are likely to encounter very different metrics than mine, and thus I’ll lay out some common patterns and follow up actions to investigate them further.

Mental models for eval comparison (skill vs. baseline)

When analyzing evaluation runs, comparing skilled execution against baseline controls typically maps to five distinct diagnostic outcomes—ranging from High-Efficiency Capability Lift (best) to Context Overload & Skill Regression (worst).

For a complete breakdown of this diagnostic taxonomy and actionable audit steps for each outcome, see the README breakdown of Diagnostic Mental Models for Eval Comparison.

Micro view: Sample-level diagnostics

While high-level metric summaries alert you to outcomes like cost bloat or skill regression, opening an individual sample surfaces the Sample Details panel for granular trace analysis:

Transcript tab: How to read the LLM's conversation as it happened

This tab shows the exact multi-turn conversation between the solver agent, the sandbox shell, and external tools. Use it to diagnose model reasoning versus environment noise:

System Prompt Verification: Confirm that web search rules (-T web_access=false) and automated time limits (300 seconds) were correctly injected into the container environment.

Skill Ingestion Check: Verify whether the agent activated the skill. If activation didn’t occur, your eval is mostly testing baseline model knowledge rather than skill utility.

Here is an example of one case where the skill was activated.

This means: The agent actually ingested gemini-api instead of relying on baseline pre-training memory.

Of note though, some of the tasks where skills were made available did NOT activate the skill. If you’re a skill author, you may want to rewrite the “activation criteria” (aka what situation calls for using the skill) such that it's more applicable to the specifics of relevant tasks.

Errors and reasoning loops

Auditing transcript details allows you to distinguish model reasoning loops (e.g., repeated redundant tool calls) from sandbox environment noise (e.g., container timeouts or missing binary dependencies). For step-by-step diagnostic trace auditing procedures, see the README reference on Sandbox Noise vs. Model Reasoning.

Scoring tab: Multi-fact verification breakdown

Clicking the Scoring Tab displays the empirical breakdown of our custom multi_scorer on a sample:

The multi_scorer aggregates the results of checking the sample’s answer against individual binary yes/no Facts using model_graded_qa. The generated list of scores in the range [0.0, 1.0] is supplied to our custom_reducer which calculates their arithmetic mean. From there, it applies quadratic score curving ( $mean^{2}$ ) to ensure that the further from correct the mean of the supplied answers are, the lower the score is pulled. This allows the most correct answers to stand out immediately. In the Scoring Tab, this downward curve maps a raw 5/6 fact score ( $\approx 0.8333$ ) down to a normalized 0.65 sample score. For complete formulas and math derivations, see the README section on Atomic Fact Verification & Quadratic Curving.

Visual milestone: The transition to cohort analytics

While inspect view provides outstanding deep-dive diagnostics for individual sample traces, evaluating dozens of models across multiple skill domains gets confusing fast (as you may have seen above). Weighing the correlations between skill inclusion vs exclusion and model changes against each other can be confounding and more art than science if we don’t find better ways to reason over them; it requires a structured matrix overview.

Remember that mess of sectors of a 3d space I showed you earlier representing all the configs? Well, unless we find numeric ways to collapse that or quantify deeper comparison, you won’t get the granular information you need to choose between two similarly capable alternatives. Worse yet, you won’t be able to communicate this to the people with their hands at the purse strings (unless of course, that's you too).

To make this easier, next time we’ll start down the road of understanding and displaying these things visually: As visualized in the cohort matrix linked above, understanding agent capability factor-by-factor requires cohort slicing across multi-dimensional metrics.

In Part 2, we scale our analysis from single log UI inspection to aggregate scoreboards with inspect viz!

Codegolf: Build a container in Cloud Build

Katie McLaughlin — Wed, 27 Mar 2024 01:19:28 +0000

Banner Photo by Viktor Kiryanov on Unsplash

I read Adam Ross's Modernizing cloudbuild.yaml for Container Builds and saw all his mentions of "character counts" and it got me thinking:

How codegolf can we get with this?

Context: like regular golf, codegolf asks you to implement something in the least amount of character possible.

Starting with Adam's finishing iteration:

steps:
- id: 'Build Container Image'
  name: 'gcr.io/cloud-builders/docker:latest'
  script: docker build . --tag "${_LOCATION}-docker.pkg.dev/${PROJECT_ID}/${_REPO}/${_IMAGE}:latest"

images:
- "${_LOCATION}-docker.pkg.dev/${PROJECT_ID}/${_REPO}/${_IMAGE}:latest"

options:
  automapSubstitutions: true

substitutions:
  _IMAGE: service
  _LOCATION: us
  _REPO: container

Initial character count: 358

It appears his character count doesn't inline new lines, which is effectively the same as the wc -c minus the wc -l. I'll be using the following for my character counts:
expr $(wc -c cloudbuild.yaml | cut -d' ' -f1) - $(wc -l cloudbuild.yaml | cut -d ' ' -f1)

He notes that the introduction of Artifact Registry make this a bit longer than it would be in Container Registry, but we'll sit with that since it's the supported registry.

We'll also keep with this last iteration's requirements: a project might have more than one registry, so we'll keep some parameterization.

Each iteration I'll make sure is a valid build. For testing, I'll ensure gcloud builds submit runs clean in a folder containing just this cloudbuild.yaml file and my go-to codegolf Dockerfile (that's also a valid Cloud Run service, just for fun).

What's in an ID?

To start, Cloud Build steps don't require an ID, so we can remove that line entirely. Without an ID, the Cloud Build build details in the Google Cloud console will use the image name as the step details, so there is a slight usability disadvantage to this change, but nothing that affects functionality.

-- id: 'Build Container Image'
-  name: 'gcr.io/cloud-builders/docker:latest'
+- name: 'gcr.io/cloud-builders/docker:latest'

Character count: 329.

Deduplicate with dynamics

We've also got the same image ID used multiple times. We want to keep the dynamic nature of this, but we also want to use substitutions in the value itself. We can use dynamicSubstitutions to allow use of substitutions within other substitutions, but adding this adds ~27 characters to our file (and we're removing 45x2 characters so this is a win!)

steps:
- name: 'gcr.io/cloud-builders/docker:latest'
  script: docker build . --tag "$_IMAGE_URI"

images:
- "$_IMAGE_URI"

options:
  automapSubstitutions: true
  dynamicSubstitutions: true

substitutions:
  _IMAGE_URI: ${_LOCATION}-docker.pkg.dev/${PROJECT_ID}/${_REPO}/${_IMAGE}:latest
  _IMAGE: service
  _LOCATION: us
  _REPO: container

Character count: 326.

At this point we can also remove the superfluous quotation marks, saving 6 characters, bringing us down to 320.

Shortest values

Now we get to the part where we work out: What's the minimum value for the variables we're using?

The $PROJECT_ID is a default substitution, so we should keep that as is, and it will be dynamic for our project.

We're already using the shortest Artifact Registry region, the US multi-region (the longest Google Cloud region could be up to 24 characters, so 2 is great!)

But we can possibly do better with the Artifact Registry name and the Container Name.

d (for "Docker") is a valid Artifact Registry repository name, and i (for "image") are valid, so we can save ~15 characters there:

...
-  _IMAGE: service
+  _IMAGE: i
   _LOCATION: us
-  _REPO: container
+  _REPO: c

Character count: 306.

Minimal variables

Any of our underscore substitutions we can reduce down to two letters, the underscore and some useful character. (We're replacing "Image URL", "Image", "Repo", and "Location", respectively.)

steps:
- name: gcr.io/cloud-builders/docker:latest
  script: docker build . --tag $_U

images:
- $_U

options:
  automapSubstitutions: true
  dynamicSubstitutions: true

substitutions:
  _U: ${_L}-docker.pkg.dev/${PROJECT_ID}/${_R}/${_I}:latest
  _I: i
  _L: us
  _R: d

Character count: 254.

(We're now down below what Adam was before migrating to Artifact Registry! 😅)

Remove tags

Another reduction we can make is removing the :latest tags. "Latest" is used in Docker where there is a lack of tag, so this is implicit in our call (and it just so happens that the Cloud Builder we are referencing has a "latest" tag. (You can check this by going to gcr.io/cloud-builders/docker in the browser, which will show you information about this image in its registry. This image in particular will direct you to its Artifact Registry information, because this registry has been automatically redirected.)

steps:
- name: gcr.io/cloud-builders/docker
  script: docker build . --tag $_U

images:
- $_U

options:
  automapSubstitutions: true
  dynamicSubstitutions: true

substitutions:
  _U: ${_L}-docker.pkg.dev/${PROJECT_ID}/${_R}/${_I}
  _I: i
  _L: us
  _R: d

Character count: 240.

Putting the args back

The last step we can do to save space is to actually re-introduce the args parameter, rather than using script. In this case, we will be adding some characters for the separation of the arguments, but the biggest win is being able to remove the ~27 characters that the automapSubstitutions config setting adds.

steps:
- name: gcr.io/cloud-builders/docker
  args: ['build','.','--tag','$_U']

images:
- $_U

options:
  dynamicSubstitutions: true

substitutions:
  _U: ${_L}-docker.pkg.dev/${PROJECT_ID}/${_R}/${_I}
  _I: i
  _L: us
  _R: d

Character count: 213.

Tip: Make sure you check what your default entrypoint is for images! In this case, I have also removed docker, as my args are appended to the default entrypoint. The script needs the full command (I may have spent too long debugging this issue 😅)

Twist

Actually you only need 22 characters:

gcloud run deploy a
[ENTER]
[ENTER]

First time you run this you'll need 1 extra character (ENTER) to enable APIs, and also 1 extra character (ENTER) to create the Cloud Run Source Deploy image registry. (You can also use this method to automatically create the Artifact Registry for you, read more about this on my blog post: "Auto-provisioning Artifact Registry though Cloud Run Source Deploys (glasnt.com)").

Summary

You can make a very short Cloud Build configuration to build images, but you should consider the readability of your scripts as you go. Addressing any cruft and modernizing your scripts can be an interesting exercise for you and your infrastructure.

With thanks to Adam Ross for the review!

Migrating from Secret Manager API to built-in secrets

Katie McLaughlin — Sun, 19 Jun 2022 23:17:12 +0000

Secret Manager is directly integrated into a number of products, including Cloud Run and Cloud Build.

Many samples I've authored use the Secret Manager API by calling the API from within the code. This can be more secure, but variable leakage is something that needs to be considered in any deployment. Removing the direct API calls can help with portability, and reduce the amount of dependencies in the deployment, and the complexity of the code itself.

So let's look at how we can migrate to built-in secrets.

Integrating Secret Manager API

Screenshot: "Secret value: Input your secret value or import it directly from a file.", showing a file upload dialog, or a text area input field.

When you create secrets, you can create them from a file or from direct input. When you retrieve these secrets, you may presume they're a single value, or a file of content you have to process.

Once you create the secret, you can access it with the Secret Manager API using a client library for Secret Manager available in many different languages.

This post will use Python examples and packages, but the same patterns can be applied to other languages.

For example, this code snippet will retrieve the latest version of the secret mySecret in the current project:

import google.auth
from google.cloud import secretmanager as sm

# trick to detect current project
_, project = google.auth.default() 
client = sm.SecretManagerServiceClient()

secret = "mySecret"

name = f"projects/{project}/secrets/{secret}/versions/latest"
secret_value = client.access_secret_version(path).payload.data.decode("UTF-8")

The secret_value can then be used as required. This presumes that mySecret is a single value, for example an API key.

One way developers can include multiple secrets in one format is with .env files. These files contain one or more key/value pairs which mean you can store multiple secrets in one file, then use helper packages to parse these values into your code. This prevents you from having to define multiple separate secret values.

If mySecret was a .env file, I could load it using python-dotenv:

## pip install python-dotenv
import io
from dotenv import load_dotenv

load_dotenv(stream=io.StringIO(secret_value))

Or I could to similar with django-environ, for example:

## pip install django_environ
import io
import environ
env = environ.Env()
env.read_env(io.StringIO(secret_value))

In both of these examples, these packages presume that the values are in a .env file within the same directory as the running process. Using StringIO means that a string variable can be sent to these methods in a file-like object, so the values can be read as though they were a file.

When using the Secret Manager API, you can run the application in any place where Google Cloud authentication works. That is, any Google Cloud service, or your local machine if you have set up gcloud.

Consuming built-in secrets

If you have your secret configured as an environment variable, then you only have to retrieve it as an environment variable:

import os
secret_value = os.environ.get("MYSECRET", None)

This method prevents KeyErrors if the key is not found.

For products with built-in secrets, they may be made available in additional ways.

In Cloud Build you can connect secrets directly to environment variables. But Cloud Run additionally allows mounting the secret as a volume, which means it can be read as a file. Any time the file is read, if you specify latest, the current latest will be received! However, you must mount the file in a new volume so you can't mount it directly in the default .env.

If you want to adapt your code to handle a local .env file, or a separately mounted file, or an environment variable, you will have to ensure all methods are possible in your code.

You also need to consider which configuration takes priority, as by default both python-dotenv and django-environ accept the first declared value as the value they use. You can override this by using the --override or --overwrite respectively.

When developing applications, I might choose to say my local .env file takes priority, then any mounted secrets, and then any declared variables.

Another point to mention is that using python-dotenv, if a file is not found or a value is empty, it silently continues. This means you can include the method calls without having to explicitly handle errors:

import io
import os

from dotenv import load_dotenv

load_dotenv()
load_dotenv("/secrets/.env")
load_dotenv(stream=io.StringIO(os.environ.get("MYSECRET", None)))

The same code works for django-environ, where you can just import in the order of priority without having to worry about missing files.

import io
import os

import environ
env = environ.Env()

env.read_env()
env.read_env("/secrets/.env")
env.read_env(io.StringIO(os.environ.get("MYSECRET", None)))

Note that in these examples, I'm choosing /secrets/as my volume, and keeping the path the same name as the original file. You can choose any volume and path, as long as the volume is not already used by the application (for example, if you choose /app/ as your working directory, you cannot mount secrets there.)

Deploying built-in secrets

To run this code locally, you'd create a .env file with the contents MYSECRET=serkitValue.

If you're committing this code to git, ensure you're not committing the secret file! Make sure you add .env to your .gitignore file!

You can also choose to ignore any contents of your .gitignore file in your Google Cloud commands by adding .gitignore's contents to .gcloudignore:

.git
#!include:.gitignore

You can then create the secret from this file with gcloud:

gcloud secrets create mySecret --data-file .env

For Cloud Build, you will need to ensure the secret is available in the environment (which your script can then use):

steps: 
 - name: python:slim
   entrypoint: pip
   args: ['install', '-r', 'requirements.txt', '--user']

 - name: python:slim
   secretEnv: ['MYSECRET']
   entrypoint: python
   args: ['main.py']

availableSecrets:
  secretManager:
  - versionName: projects/$PROJECT_ID/secrets/mySecret/versions/latest
    env: 'MYSECRET'

You can also reference secrets in the args call itself, using bash variables.

For Cloud Run, you'll have to deploy the service specifying either a mount or an environment variable:

# for mounted volume
gcloud run deploy myservice --source .  \
  --update-secrets /secrets/.env=mySecret:latest

# for environment variable
gcloud run deploy myservice --source .  \
  --update-secrets MYSECRET=mySecret:latest

And never forget IAM!

Don't forget: you'll also need to make sure the service account you're using has permissions to access your secret!

Katie is a Developer Advocate for Google Cloud, focusing on Python and Serverless. She tweets @glasnt about clouds, crafts, and cats.

Using Workflows to dynamically update a Datasette website

Katie McLaughlin — Sun, 20 Feb 2022 22:04:32 +0000

A while ago I spoke with Simon Willison about the sorts of things he'd like to have in a cloud provider in order to deploy Datasette nicely and repeatedly. A few of the things mentioned were being able to attach to object storage, especially when databases were particularly large and it didn't make sense to mount them into the container image itself.

With the preview of the second generation execution environment with Cloud Run, one of the many new features is the ability to mount Cloud Storage as a network filesystem.

With this, you can attach any Cloud Storage bucket and have a Datasette web service serve data from that source.

This works very well when you have one database that you want to serve. However, one of the known limitations of Datasette, where if you serve a folder of content, the glob of files in that folder is loaded at server start time, and not refreshed unless the server is restarted.

There are few ways we can work around this issue. You could have the process within the container restart when the mounted filesystem changes, using something like inotify (fsevent on macOS). You could also consider periodically starting, but that may mean restarting when not required, or not often enough. For a sufficiently low traffic system, you'd end up automatically booting from cold after a time of inactivity, anyway.

The way I chose to solve this was to use the event of an object being uploaded to the bucket to trigger an update of the service, thus restarting Datasette (albeit dramatically).

And since I was now listening to the upload event, I thought, why not do more processing while I'm here?

I'd recently learnt from Lab 6 of the Pic-A-Daily Workshop how to use Workflows to handle events, so I thought I'd try developing one of my own.

The result is github.com/glasnt/dynamic-datasette, and once deployed it works like this:

you upload an new file to the upload bucket
the event of that file being uploaded triggers a function, that starts a Workflow
the workflow then follows through a number of steps:
- sends the file for processing:
  - if the file is already an SQLite database, then it sends it off to the serving bucket
  - if it's one of the types of files it knows about, it uses one of the many *-to-sqlite tools Simon has made. For example, csvs-to-sqlite.
  - Once converted, the resulting SQLite database file is uploaded to the serving bucket
- updates metadata for Datasette:
  - this step mostly just ensures that the Datasette service displays the datetime when this workflow was run, so users can know how new their data is that's being served
- restarts the web service
  - This step uses the replaceService API to force a 'restart', deploying a new revision of the Datasette service, ensuring that the latest information in the serving bucket is being relayed to users.

The Datasette container mounts the serving bucket, and metadata.json in that same bucket, then just calls "datasette serve" on the mounted folder.

You can try this out on your own project by deploying the code with Terraform, following the instructions in the README.md.

Try extending the functionality of the process-upload function to do more steps, or adding more metadata to the hosted service.

Photo by Victor on Unsplash

Open Source Credit Survey

Katie McLaughlin — Tue, 01 Feb 2022 21:31:08 +0000

The University of Vermont is conducting a survey to help understand how people receive credit for tasks in open source, and are seeking your participation!
Survey: https://qualtrics.uvm.edu/jfe/form/SV_1zUs19oVcZJ0SPA

Today's open source world has similarities to the world of open source when I first wrote about this problem in 2015. I wrote a tool that helps discover who is contributing to GitHub projects based on the contributions that aren't code. In 2022, we are still contributing countless hours of time towards communities to drive and improve open source across many industries, but we don't know how much of that work is credited beyond automated tools like this.

I'm a member of the OCEAN, a research team from the University of Vermont Complex Systems Center, and we're collaborating with industry experts from Google, OSI, IEEE, PSF, and OSSF, among others. You may have recently heard about our project on CHAOSSCast, or being shout-out on SustainOSS.

As we've written in our correspondence in Nature Computational Science, we believe that having a recognition model for the work done in any open source community can help more in just gaining credit for code, but providing clear signaling of the work done for the community.

As part of this work, the University of Vermont team is conducting a survey about how people receive credit for the tasks they do as part of open source projects.

The survey asks about your experiences with receiving credit on open source projects. The survey asks multiple choice questions about how often you received credit for the tasks you did, and written response questions about what did and did not go well on open source projects.

We hope that with this information, we can improve the equity of contributions across all of open source.

Click on this link to participate in the survey: https://qualtrics.uvm.edu/jfe/form/SV_1zUs19oVcZJ0SPA

OCEAN is a partnership between the Google Open Source Program Office and the Vermont Complex Systems Center. Google does not have any influence on the survey, and will be unable to see any individual responses.

When you're not around: trigger Cloud Run on a schedule

Katie McLaughlin — Fri, 22 Jan 2021 00:00:31 +0000

On this edition of Serverless Expeditions, we take a look at scheduling automated Cloud Run jobs with Cloud Scheduler.

Check out the video version of this blog post.

Source code for this blog post is available on GitHub under the "scheduled-cloud-run" folder.

GoogleCloudPlatform / serverless-expeditions

Cloud Run is designed to host a containerised web service -- any language you like, as long as the service listens on port 8080 -- but your code can run on events other than a user visiting your site.

An example of this design pattern would be a nightly billing job: you want to process billing data nightly at 1am, but you don't want to be the person to click a button in the middle of the night.

If you wanted to implement such a receiving service in NodeJS, you could implement it like in our example code: at the top of the code, create and initialize an Express app that listens for incoming HTTP calls. Note that Cloud Scheduler uses the “text/plain” content type, so you'll have to tell body-parser to parse more than the default “application/json” type. The POST handler parses the minimum balance that was sent to it from Cloud Scheduler, and then sends that off to the billing method later in the code.

Cloud Scheduler is a managed service offered on Google Cloud, where you can create jobs on a cron schedule that either makes a HTTPS call, or publishes a Pub/Sub message. In this example, you can setup a scheduled job that runs at 1am every night by entering the cron value 0 1 * * * in the Frequency field, and setting the HTTP Target to be the URL of your Cloud Run service you created earlier.

However, you need to make sure that this billing service is private, and not accessible by anyone on the internet. To convert a public service to private, you need to remove the allUsers member from the service -- that is, deny anyone from accessing it. To then allow Cloud Scheduler to access the service, create a new service account, assigning it the Cloud Run Invoker role. Then set that service account's email identifier in the Auth Header setting in Cloud Scheduler, under "Add OIDC token". OIDC, or OpenID Connect, is a small layer on top of OAuth that handles identity. This field tells Cloud Scheduler to use the new service account's identity for running the job.

The free tier of Cloud Scheduler allows you 3 jobs for free, no matter how many times the job is run. Each job after that is USD$0.10 each. You may also incur costs for excess Cloud Run execution time, database costs, etc.

By using Google Cloud managed services that connect to Cloud Run, you can build more complex architectures, and automate manual processes to make your developer live easier.

About Serverless Expeditions

Serverless Expeditions is a fun and cheeky video series that looks at what serverless means and how to build serverless apps with Google Cloud.

Follow these hosts on Twitter at @glasnt and @martinomander.

Running multiple versions of Python in Cloud Run

Katie McLaughlin — Fri, 11 Dec 2020 01:05:06 +0000

On this episode of Serverless Expeditions, we take a look at how to deploy multiple versions of Python in Cloud Run.

Check out the video version of this blog post.

Running multiple versions of Python in Cloud Run - Serverless Expeditions (YouTube.com)

As new versions of Python are released, trying to choose between the new and shiny or the old and stable can be an issue. But if you are deploying on Cloud Run, you don't have to choose between the two, you can run both.

Imagine this hypothetical scenario: you are an SRE for a company with a custom storefront, maintaining a fleet of virtual machines running various components, all with their own versions of Python. To improve reliability, you can upgrade all the components to use one version of Python, and move away from virtual machines, and into something more scalable, like serverless.

This upgrade won't happen overnight, and the storefront must keep running in the meantime. So your plan is multi-step: migrate from virtual machines to serverless first, then make the language upgrades. But before migrating the existing systems, a new service coming online could be deployed serverless first, skipping the virtual machine all together.

When choosing which serverless platform to upgrade to, limitations may exist on the languages available. For example, Cloud Functions only supports specific versions of Python.

However, the newest serverless offering from Google Cloud -- Cloud Run -- doesn't have this limitation. Cloud Run defines services as container images, which are defined by Dockerfiles, and can specify any base image. There is an entire repository of official Python images on Docker Hub which you can source from, by specifying the base image in the FROM line of your Dockerfile.

FROM python:2.7
...

FROM python:3.9
...

As you migrate each service from its original virtual machine, you can set the exact version of Python it was originally running. You can also upgrade each service in isolation, as your team has time and resources.

Once your dependencies have been upgraded and you want to test your service, you can deploy your new revision to only a limited number of users, using the Traffic Splitting feature of Cloud Run. You can set your new revision to serve, say, only 5% of requests, so if there is something wrong, only some of your users are affected, and you can quickly rollback if there are any issues.

By migrating each service in turn, you are able to control each service in isolation, allowing for a more incremental upgrade process, and a more reliable environment.

By being able to specify the exact version of Python being used in your project, you can run multiple services each with their own isolated Python environment, which allows you more time to upgrade each service to supported versions of Python.

It also allows you to try the newest versions of Python -- such as prereleases of Python 3.10 -- as soon as their images are available on Docker Hub.

About Serverless Expeditions

Serverless Expeditions is a fun and cheeky video series that looks at what serverless means and how to build serverless apps with Google Cloud.

Follow these hosts on Twitter at @glasnt and @martinomander.

"Point-and-click" continuous deployment with Cloud Run

Katie McLaughlin — Thu, 10 Dec 2020 20:36:47 +0000

On this edition of Serverless Expeditions, we take a look at how to deploy a GitHub repo on GitHub, though Cloud Run's new "point-and-click" integration, no Dockerfile required.

Check out the video version of this blog post.

Say you've created a website for a travel blog, and you want to host it online. You've got your code on GitHub, and you have a README of instructions of how a friend could deploy a copy of your website: they have to have your programming language installed on their machine, install some package dependencies, then run a command to start a web server.

If you want to host your website, you need a way that you can run these steps any time you want to update your code. This process is known as Continuous Deployment: when you commit a change to code a process is started that takes the new version of your code, performs the prerequisite tasks, and then starts a web server of your new code.

Cloud Run, a serverless platform from Google Cloud, allows you to to bring any container to serve your website, no matter the programming language used inside, as long as it follows the Runtime Container contract: a web server must be listening for requests on 0.0.0.0, on port 8080 (or the configured PORT environment variable).

So, if you have a website, how do you turn it into a container? You could write a Dockerfile, a set of instructions to build a container image based on your setup. But if you're using a common language like Python, Go, or Ruby, the instructions are going to be similar for many websites: copy your code into the container, install your dependencies, and start the web server.

You can setup your Cloud Run service through the Google Cloud Console using the new walkthrough integration with Cloud Build: you connect your GitHub repo with Cloud Run, and set up your website to be built any time your code updates.

However, instead of having to write a Dockerfile, you can now use Buildpacks, a set of common instructions for popular programming languages that help you build your code into a container image that Cloud Run can deploy.

If you're using a language like Go, Node.js, Python, Java or .NET Core, this will automatically be detected when building a container using Buildpacks. These common languages and their respective package installation processes are handled for you; all you need to do is provide the command to start your web server in a special file called Procfile.

In the example in the video, Martin is using Flask, a Python-based web framework. To start his application, he runs:

python3 -m flask run

So his Procfile will be:

web: python3 -m flask run

The "web" in this case means "web server", and is the special term Buildpack looks for in a Procfile to start the web server.

He should also configure his Flask app to listen on the expected IP and port. He can either add that to his app.run command, or in the Procfile itself as part of starting Flask:

web: python3 -m flask run --host 0.0.0.0 --port $PORT

As long as he adds this file to his codebase, he doesn't need to change anything else about his website to host it on Cloud Run.

Once configured, any commits made to the GitHub repo -- from the command line, VSCode, GitHub's website, or anywhere else -- are automatically detected and deployed.

To learn more about the concepts discussed in this video, read more:

You can try some of the sample buildpack apps at https://github.com/GoogleCloudPlatform/buildpack-samples

About Serverless Expeditions

Serverless Expeditions is a fun and cheeky video series that looks at what serverless means and how to build serverless apps with Google Cloud.

Follow these hosts on dev.to at @glasnt and @martinomander.

Deploying Django and Wagtail on Google Cloud

Katie McLaughlin — Thu, 01 Oct 2020 23:03:15 +0000

On this edition of Serverless Expeditions, we take a look at how to deploy more complex applications on Google Cloud, using Django as an example.

Check out the video version of this blog post.

When moving legacy applications into serverless infrastructure, there is going to be some sort of state involved: there will be a database, media, and other components. Traditionally these can all be run within the confines of a virtual machine, where all the services can talk to each other in that environment. But when moving to serverless, all these components need to be specifically provisioned and connected together.

While Cloud Run itself is stateless, it can be connected to other stateful services such as Cloud SQL for relational databases, or Cloud Storage for static media hosting.

In today's episode, we'll discuss containerizing and deploying a typical Django application using Cloud Run and other Google Cloud services. The steps described can be applied to any website whose CMS uses the Django web framework, such as Wagtail or Django CMS.

Django supports both PostgreSQL and MySQL, both of which are available in Cloud SQL, a managed database service. When connecting to a Cloud Run service, specifying --add-cloudsql-instances allows the service and the database to communicate over an automatically encrypted connection. The authentication from the application to the database is achieved by specifying a connection URL, stored securely in Secret Manager, loaded using the Python API. The database can then be populated by running ./manage.py migrate

Static and media assets can be migrated to use Cloud Storage, with help from the django-storages package. By creating a new storage bucket and specifying the name in the django settings.py, then running ./manage.py collectstatic, the images and assets are uploaded to the bucket.

However, those two manage.py commands -- migrate and collectstatic -- are typically run within an interactive terminal. Cloud Run has no such terminal, so these commands need to be run elsewhere. One method is to run these commands as part of the continuous integration process with Cloud Build. Adding a 'migrate' step using the app-engine-exec-wrapper between the typical build and deploy steps allows migration to happen as part of deployment.

If database migrations need to be manual, Cloud SQL proxy can be used to interact with a Cloud SQL database locally. Details of how to set this up with Docker Compose are available here.

With a managed database, highly-available static assets, automated deployment pipelines and robust containerisation solutions, you can migrate your Django applications to Google Cloud with confidence.

If you want to get started with these concepts, check out the Django Codelabs:

You can also find a sample implementation of these concepts at GoogleCloudPlatform/django-demo-app-unicodex.

About Serverless Expeditions

Serverless Expeditions is a fun and cheeky video series that looks at what serverless means and how to build serverless apps with Google Cloud.

Follow these hosts on Twitter at @glasnt and @martinomander.

Automating Python package releases by extending git

Katie McLaughlin — Mon, 10 Aug 2020 03:12:30 +0000

Like many automation workflows, you end up connecting different operations together that chain together.

In this post I'll describe how after initial setup, you can cut and release a new patch version of your GitHub hosted Python package by running:

git version-bump patch
git release

Link One

In Python's packaging ecosystem, versions are important. version must be declared in a package's setup.py. It doesn't have to be defined directly as a parameter of setup(), but it must be declared. You could setup a yourpackage/__version__.py, which allows you to at have the version stored outside of setup.py, but you still need to read from that file.

But why define the version in a file when you can dynamically query it from a git tag? How can you get setup.py to understand the version?

If you pip install versioneer and versioneer install it into your package, you can get versioneer to query git directly to get the current tag. This will work within the context of git. This is important later. Versioneer itself isn't a package dependency of your package, it's self-contained in a _versioneer.py file

Link Two

Git can execute external commands as if they were git commands. If you have an executable file in your PATH that starts with git-, e.g. git-thing, it also works as git thing (with a space rather than a hyphen). As of git v2.20.0, git help --all now lists these external commands. The implementation language of the executable doesn't matter; the interpreter used to run the executable is defined in the shebang.

Link Three

Git tags can be manually edited using git tag, but you can install a custom command to help automate the process of implementing say, semantic versioning, by allowing you to specify "bump patch" or "bump major" and the tag is updated based on the current version.

You can get this functionality as git version-bump if you gem install git-version-bump to automate semantic version git tag bumps. git-version-bump happens to be implemented in Ruby, but that doesn't matter as long as you have Ruby installed on your operating system.

Link Four

Git tags do appear in GitHub's web interface, but Releases are a richer experience. They are associated to a git tag, but allow attaching artefacts, and releases appear on the top level page of a repo.

You can automate publishing git tags as GitHub release by installing the github-release gem, which gives you the git release command. This uploads local tags to GitHub, using a custom API token setup for you within the command on first execution (including asking for your OTP). github-release is also written in Ruby.

Link Five

GitHub Actions are a now public feature which makes continuous deployment an integrated feature of GitHub. When you setup an Action on a Python repo, it's suggested that you could setup an action to publish to PyPI. This action is triggered when a GitHub release is created. Your PyPI username and password are stored as GitHub secrets. Within the scope of a running action, it is a git context. This was important from earlier.

Do you see where we're going with this?

By setting up versioneer to listen to git tags, and using git-version-bump to automate semver bumps of git tags, then github-release to convert git tags into GitHub Releases, which then trigger a GitHub Action to publish to PyPI, you can publish a new version of your package with just two git commands.

To get this setup on your existing package, install the local dependencies:

gem install git-version-bump
gem install github-release
pip install versioneer
versioneer install

Versioneer will guide you through the installation

And add a GitHub action for the publication of your package on GitHub release

Bonus Link One

If you setup versioneer and the GitHub Action, you can still trigger a new release by manually creating a GitHub Release from the website. Just be sure to git pull the new tags when you go back to the command line.

Bonus Link Two

You can follow the same release process for packages in other programming languages; you just need a way to query git tags into your package version declaration, and a workflow to perform publication. git-version-bump itself handles the git tag issue for ruby.

Bonus Link Three

If you're worried about installing random gems or ruby itself, you can first install rbenv (which pyenv was originally forked from), then rbenv install the latest version of ruby. Extending git only works if the executable is in your PATH, so you don't also need to install bundler just for this purpose.

Bonus Link Four (to be completed by the reader)

If you don't want worry about Ruby, you are more than free to create an executable shell scripts with the same functionality.

You could use a combination of git describe, semver-tool, and git tag to help with that part. It might even be a one-liner with enough pipes.

The creation of a GitHub release is a bit more complicated, but nothing you couldn't handle, I'm sure :)

Resources that helped me write this piece

Extending git, by Stefan Saasen, Atlassian.
github.com/git/git, a mirror of the git source code, and a URL that never stops being amazing.
My own post on this topic from 2014 from my Ruby days, which lead me to my own closed pull request on git-version-bump, wherein I was told the point "is to take this version information out of the filesystem", which lead me to find versioneer by searching "setup.py version from git tags" which sent me to this StackOverflow answer

This article was updated in August 2021 to ensure the line links to git source code were locked to a commit, and did not move over time.

DEV Community: Katie McLaughlin

Step up to the Sheets: AI Eval Export and Illustrating Data

Implementation prerequisites

Back to the story at hand

The extractor tocsv.py

Heavy lifting in Python: Custom telemetry columns

Exporting to Google Sheets

Why leave the cradle?

AI Evals at a Glance: Heatmaps for Stakeholders

Visualizing AI evals with Inspect Viz

The 40-log gridlock

Follow-along requirements

Cognitive overhead and segue to BI analytics

Designing AI Evals: Clarity Now and Visualization Next

AI evals and analysis

Level up your evals

A note on models

Setting up the eval

Running the eval

Run and wait

Viewing the evals

Macro view: Comparing runs

A. Comparing models head to head

B. Comparing skill conditions (skill vs. baseline)

Mental models for eval comparison (skill vs. baseline)

Micro view: Sample-level diagnostics

Transcript tab: How to read the LLM's conversation as it happened

Errors and reasoning loops

Scoring tab: Multi-fact verification breakdown

Visual milestone: The transition to cohort analytics

Codegolf: Build a container in Cloud Build

What's in an ID?

Deduplicate with dynamics

Shortest values

Minimal variables

Remove tags

Putting the args back

Twist

Summary

Migrating from Secret Manager API to built-in secrets

Integrating Secret Manager API

Consuming built-in secrets

Deploying built-in secrets

And never forget IAM!

Using Workflows to dynamically update a Datasette website

Open Source Credit Survey

When you're not around: trigger Cloud Run on a schedule

GoogleCloudPlatform / serverless-expeditions

About Serverless Expeditions

Running multiple versions of Python in Cloud Run

About Serverless Expeditions

"Point-and-click" continuous deployment with Cloud Run

About Serverless Expeditions

Deploying Django and Wagtail on Google Cloud

About Serverless Expeditions

Automating Python package releases by extending git

Link One

Link Two

Link Three

Link Four

Link Five

Do you see where we're going with this?

Bonus Link One

Bonus Link Two

Bonus Link Three

Bonus Link Four (to be completed by the reader)

Resources that helped me write this piece

The extractor `tocsv.py`