DEV Community: Katie McLaughlin

Interactive AI Eval Dashboards with Data Studio

Katie McLaughlin — Tue, 08 Sep 2026 07:00:00 +0000

Welcome to the final entry of our series about designing, analyzing and visualizing AI evals! As with the previous entries, you are more than welcome to simply read over my exploration or, for those of you looking for extra credit, feel free to play along at home. If this isn’t you, please skip to "Overview of New Pre-Configured Dashboard Components"

Note: This blog series contains AI-generated diagrams alongside actual screenshots and hand-drawn edits of both. AI also assisted in minor copy editing. Furthermore, the math is mine but the formula formatting is AI's.

Implementation prerequisites

If you intend to (re)implement the procedures in this walkthrough, all you need is the CSV export (data_mega_export.csv) generated during Part 3 (from evaluations performed in Part 1) and a Google Workspace account to access Data Studio. Part 2 will give you a deeper understanding of evals, trace diagnostics and back-of-the-napkin analysis but is optional if you just want to build dashboards.

Back to the story at hand

In this final part of our series, we connect our evaluation dataset to Data Studio to build codeless, interactive dashboards.

While inspect view provides developer-focused TUI trace diagnostics and Google Sheets gives us tabular sparklines, non-technical stakeholders (product managers, business leads, executive sponsors) need interactive dashboards where they can slice performance across our foundational 3D Evaluation Matrix on the fly without writing code. Beyond macro dimensions like model family and skill group, our dashboard empowers users to dynamically partition data over key operational booleans:

completed (Conditional Accuracy): Isolates Model Capability (accuracy on completed runs) from infrastructure noise (quota drops, timeouts, container crashes). Toggling between TRUE and ALL exposes Survivorship Bias in task execution.
is_baseline_or_has_activated_skill (Conditional Uptake Slicer): Separates baseline control cohorts from active skill interventions where the model invoked activate_skill (i.e. uptook the skill into context).
- Crucially, "activated" in Inspect AI telemetry specifically tracks skill uptake (executing the activate_skill tool call). Filtering for FALSE isolates dormant runs where a skill was available but never uptaken.

1-click Data Studio template setup

We provide a pre-configured Master Dashboard Template containing all pre-styled components used here.

Clone the master template

Open the Inspect AI Dashboard Template.
Click Make a copy

30-second setup and customization

Once cloned:

In the new report, press Add data
Select Google Sheets
Select the Spreadsheet and the Worksheet (likely the sole Worksheet) you created in the third entry and press Add
Select all the components and set their data source to that sheet
Finally, select the bubble chart and open the Style tab, find the Bubble color dropdown and set it to ‘model’

That's it! Your interactive dashboard is now live, fully bound to your evaluation data.

Before diving into the new charts, check out the bottom left corner of the report. There you will find a Pivot Table Heatmap. Like the heatmap I made in the second entry in this series, this heatmap is a 2D visual matrix where color intensity represents numerical magnitude—here, quadratic curved accuracy scores. Since I already discussed how to use these for analysis in the earlier blog entry, I will not be explaining it here. If, on the other hand, you are interested in implementation, feel free to inspect and adjust the settings on the Pivot Table Heatmap in your copied report.

Note: This screenshot was done on a later dataset than the rest of the report and the specific metrics should not be taken into account when comparing to the rest of the screenshots here.

Overview of new pre-configured dashboard components

Data Studio translates multi-dimensional evaluation CSV data into intuitive visual dimensions (spatial X/Y coordinates, bubble area scaling, color coding, and hover popovers). Here is how each pre-built dashboard element visually measures evaluation performance:

Chart A: Performance and cost (scatter/bubble chart)

This chart maps three core evaluation dimensions onto a single 2D visual canvas:

Horizontal Position (X-Axis ➔ Latency): Measures average task duration in seconds. Bubbles placed further to the right represent slower runs.
Vertical Position (Y-Axis ➔ Score / Accuracy): Measures average task accuracy from 0.0 to 1.0. Bubbles placed higher up represent better accuracy.
Bubble Size / Area (Visual Cost & Unit Economics): Circle size scales with average task cost for a quick visual gut-check.
Hover Tooltips & Sub-Cent Formatting: Hovering displays latency, accuracy, model, skill group, and cost formatted as Currency > USD (#,##0.0000) so sub-cent costs (e.g., $0.0044) don't round to $0.00.

Chart B: Accuracy and error bars (±1 SEM whiskers)

The second pre-configured chart visualizes accuracy scores alongside standard error of the mean (SEM) whiskers.

Visual Whisker Bounds: The bar height shows average accuracy (score), while floating dots above and below show upper (score_lower) and lower (score_upper) standard error bounds.
Design Note (Multi-Metric vs. Breakdown Dimension Constraint): Data Studio won't let you use a Breakdown Dimension if you're plotting multiple Y-axis metrics on the same chart. Because our whiskers need three Y-metrics (score, score_lower, score_upper), we put model on the X-axis and filter by skill_group using top-level drop-downs instead.

Statistical Rigor Note: Standard error tracks pass/fail score variation across runs. Our demo ( $N = 4$ ) covers a ~68% confidence range ( $\pm 1 SEM$ ), while production benchmarks require $N \geq 30$ for standard 95% confidence intervals.

The 4 dynamic cohort slicers (drop-down controls)

Located across the top control bar, these four drop-down lists act as interactive dynamic filters, immediately re-rendering both canvas charts in real time:

model Slicer: Filters results by model family (gemini-3.5-flash-lite vs gemini-3.6-flash).
skill_group Slicer: Filters by skill setup (gemini-api, gcloud, baseline control).
is_baseline_or_has_activated_skill Slicer (Conditional Uptake): Separates Baseline controls and Active Skill runs (TRUE via activate_skill) from Dormant runs (FALSE, offered but uncalled). Activation confirms tool uptake, not instruction compliance.
completed Slicer (Conditional Accuracy / Infrastructure Noise): Filters clean runs (TRUE) from system failures (FALSE for quota drops, timeouts, container crashes).
- Relevance Note: This is vital for production sweeps hit by rate limits or timeouts—selecting TRUE measures Model Capability (Conditional Accuracy), while FALSE or (All) shows the full audit trail.

View and interact

Click the blue View button in the top right corner to see the dashboard as a user would.
Use the model, skill_group, completed, and is_baseline_or_has_activated_skill drop-downs at the top.
Notice how both the Bubble Chart and the Bar Chart update instantly in unison to show only the models or skills you selected!

Slicer-based analyses

While the model slicer would be useful in a situation with more models, since we only have two and you can visualize them using the existing coloring. skill_group on the other hand is more interesting

On selecting gemini-api (Skill) and gemini-api (Baseline), there are a couple visually apparent trends. As discussed in previous sections, the gemini-api (Skill) run improved the evaluation score at the price of increased latency and cost.

Conditional uptake analysis: Separating skill discovery from capability

By filtering for runs where is_baseline_or_has_activated_skill is FALSE, we perform a Conditional uptake analysis: examining dormant runs where a skill was provided in context (Prior), but the agent chose not to call activate_skill (zero uptake).

Baseline or has activated skill = false (notice these both score relatively low)

Baseline or has activated skill = true

One thing to note immediately is that the total number of bubbles is GREATER than 8 (the count of Model x Skill combinations). This is because each of these bubbles are subsets of the Model x Skill x Sample x Epoch space in that this distinction is made at the samples/epochs level.

In Inspect AI telemetry, "activated" specifically measures uptake—whether the model executed the activate_skill tool call to load the skill into context. Interestingly, the samples of 3.5-flash-lite where the skills were activated have a much higher visual ceiling than the general case (Baseline or Has Activated Skill = Any) corresponding to the elimination of Baseline or Has Activated Skill = False.

Noting that only 3.5-flash-lite had samples where it didn’t activate a provided skill, it may be worth investigating if this is a more general trend.

Survivorship bias and joint operational uptime (`completed` analysis)

For the sake of further exploration, I reran the evals with web search enabled and the same time limit.

inspect eval skills-eval.py \
  --model google/gemini-3.5-flash-lite,google/gemini-3.6-flash \
  --time-limit 300 \
  --epochs 2 \
  --max-tasks 4 \
  -T web_access=true

Running all of these evals serially and in parallel causes them to use too much web search quota and reach the unforgiving time-limit. As such, the most realistic run configuration would be to increase the time limit and use the built-in adaptive rate limits. The problem is this would last a lot longer and take more resources than I care to ask you to emulate.

So let’s look instead at the results of the web search enabled, 300 second time limit run:

completed: ALL

Since this graph contains every model x skill x sample x epoch, you can eyeball the average metric values of a number of bubbles by imagining the centroid of the polygon they define on the space.

Some of the most obvious conclusions in our local web-search enabled eval sweep: 3.5-flash-lite recorded lower latency than 3.6-flash, while their average scores were comparable. To reason about what this might mean, refer to the Gemini API Model Documentation for official model latency, throughput, and performance specifications. Instinctually, this makes me want to check if the timed out runs (counted as score 0) affected 3.6-flash more (since it didn’t complete as quickly in these samples). To test this, let’s see how scores look when we only count completed runs.

completed: TRUE

When filtering the web-search rerun sweep strictly for completed runs (completed = TRUE), the average score for the 3.6-flash sample cohort trended slightly higher, converging between 0.5 and 0.6 for partially scoring runs (which earlier averaged between 0.35 and 0.60 when completed = ALL).

Isolating completed = TRUE calculates Conditional Accuracy (Model Capability on completed runs). However, if longer or more complex tasks disproportionately trigger time-limits, filtering out incompletes introduces Survivorship Bias, truncating the difficult tail of your test set. Comparing completed = ALL against completed = TRUE bridges the gap between pure model intelligence and total operational system reliability for given limits.

Are these findings generalizable to every scenario?

While we’ve uncovered a promising trend, the next step is to validate it with statistical rigor. Think of our current demo ( $N = 4$ ) as an exploratory pilot; it’s great for spotting trends, but we need more data to confirm them. To verify these results aren't coincidental, I encourage you to scale your runs to $N \geq 30$ . With this larger sample size, you can achieve statistical significance, a metric used to ensure that your findings aren't just a random outlier. This is supported by the p-value—a standard calculation that gives you the probability that your results are a fluke. Aiming for a low p-value is the final step in moving from exploratory analysis to building the mathematically sound, launch-ready benchmark required for production.

Conclusion and next steps: Where to go from here

We’ve come full circle in this series—from running terminal-based evaluation sweeps with Inspect AI in Part 1, to analyzing multi-turn LLM reasoning traces in Part 2, modeling tabular metrics with pandas in Part 3, and finally assembling interactive, codeless BI dashboards here in Data Studio.

Now that you have the end-to-end telemetry pipeline running, the floor is yours to explore! Here are a few directions you can take your evaluation harness next:

Custom Calculated Metrics: Experiment with custom formulas in Data Studio to track ROI metrics, cost per successful task, or normalized token efficiency ratios.
Expanded Matrix Sweeps: Scale your evaluation matrix across new LLM provider endpoints, prompt techniques, or tool playbooks.
Direct Database Connectivity: Replace the static CSV / Google Sheets ingestion layer with a direct connector to BigQuery, PostgreSQL, or Cloud Storage for automated, streaming evaluation updates.
Automated Alerting & Thresholds: Set up automated notifications when a new model deployment drops below baseline accuracy or exceeds latency/cost budgets.

Thank you for following along with this series, and happy evaluating!

Step up to the Sheets: AI Eval Export and Illustrating Data

Katie McLaughlin — Tue, 01 Sep 2026 07:00:00 +0000

Welcome to part three of our series about designing, analyzing and visualizing AI Evals! As with the previous entries you are more than welcome to simply read over my exploration or for those of you looking for extra credit, feel free to play along at home.

Note: This blog series contains AI-generated diagrams alongside actual screenshots and hand-drawn edits of both. AI also assisted in minor copy editing.

Implementation prerequisites

If you intend to (re)implement the walkthrough, all you need to do right now is complete Part 1 (Designing AI Evals: Clarity Now and Visualization Next) and then install pandas; Part 2 (AI Evals at a Glance: Heatmaps for Stakeholders) is illustrative and will give you a deeper understanding going forward both of AI evals and why the following may be useful but it’s more elective than anything else.

Back to the story at hand

While engineering teams might evaluate agents using test automations, non-technical stakeholders (product managers, business leads, or executive sponsors) often use spreadsheets as their lingua franca and as such it would make sense to speak to them in a way they are almost certain to understand and which enables you all to collaboratively make visualizations to further get out the word.

To make this a reality, today we’ll use a pandas pipeline that transforms raw Inspect AI evaluation logs into a rich CSV optimized for Google Sheets, allowing you or anyone on your team with a little spreadsheet knowledge to make boardroom-ready visuals.

Drawing again from previous blog entries, the evals we are working on have three dimensions: model x skill x sample [and epoch] that cross to form the following matrix:

In Part 2 we created a simplified heatmap reflecting a planar slice of this space as reduced over the sample dimension. While we will ultimately do something similar in a future blog post using Data Studio, for the time being we actually want to keep all 32 combinations separate and pull more of their underlying stats from the inspect eval files and derive further information using pandas. This information will be exported as a flat csv file we can then import and view in Google Sheets.

The extractor `tocsv.py`

This script is used to parse the eval logs using Inspect AI’s python pandas interface. The script is relatively self documenting, and you can read up more on pandas and pandas’ csv IO if you are unfamiliar.

Heavy lifting in Python: Custom telemetry columns

While most of the script deals with mapping scalar metrics the like of which we’ve already discussed, there are two that are purely logical/boolean. They represent specific statistical cases that we will allow end users to partition the data set over for analysis later. These boolean conditions are derived and since we want to pare away excess information and therefore not supply the full eval files to Google Sheets, tocsv.py does the heavy lifting upstream during log ingestion.

It pre-computes two key boolean columns to simplify downstream dashboard filtering:

completed: Flags whether a sample finished cleanly (no API quota 429 errors, timeout truncations, or container crashes) with a valid score, cleanly isolating model capability from infrastructure noise.
is_baseline_or_has_activated_skill: Distinguishes Baseline control cohorts (always TRUE) and Active Skill interventions (TRUE when the skill tool was actually “activated” which in inspect means uptaken) from Dormant runs (FALSE, where skills were available but never activated).

In Google Sheets or Data Studio non-technical stakeholders can simply filter/toggle these pre-computed boolean keys to isolate clean, active performance (or otherwise) instantly.

Exporting to Google Sheets

In order to execute the pipeline I called:

python3 tocsv.py

The output file data_mega_export.csv is a standard comma-separated values file where each cell is surrounded by commas. You can preview the raw and unformatted contents in a standard text editor. While most of this is normal spreadsheet data such as strings and integers, you will also see fairly complicated Google Sheets formulas resulting from interpolating eval metrics into formula templates.

The above formula for instance will be transformed into a bar graph normalized against the range of values in the same column as its source data and tiered into colors by tertile.

How do you take this text and get it to show its stuff? By opening sheets.new of course, and importing data_mega_export.csv. When presented with the import dialog check Convert text to numbers, dates, and formulas.

On clicking Import data Google Sheets will instantly display the standard cells and render active, color-coded visual sparklines directly inside data cells!

What conclusions can you draw from what you see? One obvious one in this example is that the correlation between score and latency (how long an agent takes to answer a question) may not be very strict. But what if you wanted to get something deeper?

Well, since this is a spreadsheet, obviously you can make pivot tables, and custom functions or sort on multiple keys… you know, the whole accountant’s playbook. Let’s say though that you want custom views into the data rather than to permanently mutate it. How would you go about doing that?

Google Sheets, unlike inspect view and inspect viz heatmaps, allows you to dynamically sort AND filter on more than one key. Not only that but you can sort on ANY key, so I elected to do so with the previously unavailable score and latency fields. As shown in the image above, use Data > Create filter view which presents you with a new set of controls at the top of the main columns.

Note: Filter views do not mutate or change the underlying contents of a sheet; Instead they are (optionally named and shareable) configurations for restricting what rows are displayed and in what order.

If, as in the picture, you click the cost column > Filter by condition, select the Less than option and fill the comparison with 0.01, you can narrow the displayed rows to only those with costs less than a cent. This can be used (for instance) to only look over the samples that were “less costly” to try to analyze why.

Filter views (despite their name) also allow you to sort the display of the data. Because the sort operations are cumulative, the “primary” filter needs to be applied last. Thus, after sorting the filter view first by latency and then by score (both ‘Z to A’ aka decreasing) gives us the following:

Optionally you can name and save this filter view to toggle it on and off as well as share it with other collaborators. To learn more about Filter views, you can read Sort & filter your data.

This ability to sort and filter the display live without changing the underlying information, makes presenting the data and analyzing it during discussions with teammates significantly easier and faster.

Why leave the cradle?

Now, in spite of this tremendous flexibility, sheets have some drawbacks as visualization tools:

Spreadsheet formulas are cumbersome for nested IF statements and filters unless you define custom macros or JavaScript functions leading to code that's harder and harder to keep connected in one's mental model.
Flattened eval runs need to be inspected under multiple filter views to begin to understand complex trends or correlations across all of them.

So, how can we take this to the next level and circumvent those issues? What about an environment, which while it still offers you the ability to custom script anything you need, comes with a lot of WYSIWYG controllable visualizations and graphs, enabling you, without code, to bring data to an even more vibrant life? Better yet, what if that environment comes with built in visualizations for 3 dimensions or more and has broad compatibility with multiple data source types?

In Part 4, we’ll connect this Google Sheet directly to Data Studio to build a codeless executive summary dashboard and be able to quickly and intuitively display our findings!

AI Evals at a Glance: Heatmaps for Stakeholders

Katie McLaughlin — Tue, 25 Aug 2026 07:00:00 +0000

Visualizing AI evals with Inspect Viz

Welcome back to our blog series on running, analyzing and visualizing AI evals. Last time we discussed how to design and run evals for analysis and visualization using inspect eval and inspect view. Many of these methods, such as rearranging the dashboard columns and sorting on them to determine model vs model and skill vs no-skill differences in metrics, can give you a broad overview of the field. While this can inspire further and deeper inquiry, it runs into issues with how predictive it is and how to communicate findings to other people. Imagine the following:

The 40-log gridlock

Imagine a data science lead needing to present a performance breakdown of four LLM candidates across ten internal tools during a high-stakes, time-crunched live meeting. Reordering and filtering the evals across multiple dimensions is hard to read on a presentation screen and requires doing mental math with an audience: riveting stuff that they definitely won’t fall asleep during of course.

What if instead you had a tool that allowed you to easily automate rendering a comparison of the different configurations? What if lining them up by model or by skill showed clear and legible patterns?

Well, let me introduce you to heatmaps! A heatmap is a 2D visual matrix where color intensity represents numerical magnitude—in our case, quadratic curved accuracy scores. By leveraging the human eye's natural ability to distinguish color gradients, heatmaps instantly reveal performance spectrums at a single glance. This makes trends across dependent and independent variables clear, intuitive, and highly legible.

While you can simply read along and learn about using inspect viz to generate heatmaps for inspect AI evals, if you intend to follow along and recreate the charts, please start by replicating the steps in the first entry and come back afterward.

Note: This blog series contains AI-generated diagrams alongside actual screenshots and hand-drawn edits of both. AI also assisted in minor copy editing.

Follow-along requirements

For those playing along at home, in order to perform the following, you will need the inspect_viz library installed in your environment (alongside inspect_ai and pandas).

From there, take a look at the heatmap script. While there are some intricacies to the script (such as establishing its CLI API) the script mainly wraps scores_heatmap(viz_data, orientation="horizontal", ...) and marshals our data to assemble a heatmap of it.

If you run the script

python3 inspect_viz_heatmap.py logs -o heatmap.html

you will get a visual that looks like the following.

This heatmap has two axes, one is the model (vertical) which allows you to visually break down your analysis to just skill inclusions for the same model, the other is the skills (horizontal) and going down the row allows you to demonstrate how changes to the model affect the evaluation.

Stakeholders (like investors or your boss) can now spot skill gains and model ceilings at a glance without reading line-by-line log values. Some quick takeaways:

Top-performing configurations (such as 3.6-flash + gemini_api_skill at 1.0) stand out in high-intensity shades
Within the aggregated 2D heatmap view of this dataset, 3.6-flash trended higher in score on average across the evaluated skill configurations compared to 3.5-flash-lite.
Skills tended to improve scores, all other things held equal.

Caption: The inspect viz heatmap projects our full Evaluation Matrix down to a 2D plane by collapsing individual task samples and epochs into aggregate accuracy cells along the Model and Skill axes.

Cognitive overhead and segue to BI analytics

While matrix scoreboards provide quick visibility into accuracy differences across models, raw heatmaps suffer from key limitations:

2D Constraint: Heatmaps map only two axes to a single color intensity (in our case Accuracy).
Missing Business Context: A single heatmap cannot simultaneously show token costs, system latency, or operational reliability.
Hidden Variance: Collapsing samples for a given configuration removes the information necessary to determine if a result is consistent or wildly erratic.

To bridge CLI logs with interactive, multidimensional and in-depth reporting, we need a pipeline that exports the raw telemetry metrics into spreadsheets and BI dashboards.

In Part 3, we’ll build a python script to convert output data for processing first in Google Sheets and ultimately Data Studio!

Designing AI Evals: Clarity Now and Visualization Next

Katie McLaughlin — Tue, 18 Aug 2026 07:00:00 +0000

AI evals and analysis

Let's say you're testing out new AI tools. Perhaps you implement and run analytics for an Ad Agency and hope to automate deploying your standard event schema, or are a podcast producer automating generating social copy from your newest ep. While modern, newly trained LLMs can likely one-shot a lot of these tasks – this specificity might necessitate wasting tokens and time repeatedly prompting them with the same resources, descriptions and scripts. With this in mind, you investigate tooling whether that be an MCP server, an agent skill or an agent plugin.

The problem is, how do you know if a skill (whether developed by you or open sourced by someone else) is worth your time or, perhaps more importantly, your tokens and quota? How can one go about designing these (more) objective evaluations of AI tools and, from there, collecting and analyzing relevant metrics? That is exactly the context for using open source eval frameworks, like Inspect AI and Harbor to evaluate agent skills using open source frameworks. But how do you "extend the evaluation", use visualizations to spot trends and collaboratively explore alternative paths forward using Google Sheets and Data Studio?

These questions and more are exactly what I’m hoping to demonstrate for you in this series!

While you are more than welcome to simply read about how and why I conducted my own investigation, you can alternatively follow my lead and run the benchmark scripts.

So, if you’re hoping to play along, take a moment to complete the aforementioned codelab and come back when you’re done. Don’t worry, we’ll still be here when you get back!

Note: This blog series contains AI-generated diagrams
alongside actual screenshots and hand-drawn edits of
both. AI also assisted in minor copy editing.

Level up your evals

In the codelab, we learned how to run evals with Gemini CLI, Inspect and Inspect SWE in an isolated Docker Sandbox to understand how well each skill aids the agent in answering the same question.

For those playing along at home, please additionally install inspect view now before you need it to run the evals below; be prepared to have to wait for a couple minutes for evals to finish depending on your machine and quota usage.

Regarding our new source files while they are heavily commented and hopefully written in a self describing way, I’ll explain further subsequently.

A note on models

For the purpose of my investigation demo, I used three different models: google/gemini-3.5-flash-lite and google/gemini-3.6-flash as "solvers" (the models under evaluation), and google/gemini-3.1-flash-lite as a "grader" (the model rating the runs). The three of them (and models more broadly) differ in many ways but more specifically on problem solving ability, speed and cost.

We intentionally offloaded grading for this demo to a previous-generation model because by changing rubric criteria to strict binary decisions and applying a reduction programmatically, it delivers sufficiently robust evaluations without burning through solver quota. For a production evaluation system, consider investigating using newer and more capable models as graders because they will likely have narrower confidence intervals.

For a full technical breakdown of this decoupled grader architecture and to find out how you could substitute in your own choice of models, see the README section on Decoupled Grader & Multidimensional Rubrics.

Setting up the eval

Reproducibility & Local Setup Note: If you are following along locally, clone the domain skill definitions into google-skills/ before running benchmark sweeps:

git clone https://github.com/google/skills.git google-skills

All evaluations in this series were benchmarked on Python 3.13 using inspect-ai (v0.3.247), inspect-swe (v0.2.66), inspect-viz (v0.4.1), and pandas (v3.0.3). If upstream PyPI releases introduce breaking changes, check the README's Environment & Dependencies specification for exact version pins and instructions on how to reproduce the configuration.

For this investigation I used a new eval script; while the specifics of the script are important for anyone who hopes to make their own evals or run the code as we go, if you're primarily interested in analysis and visualization, feel free to skip to the next section.

The original script ran a small batch of tests on a local machine. While this version can be and was run on a local machine, it prioritizes three architectural dimensions needed for running automated tests as part of a larger scale development flow:

External Configs (providing the evals and solver system prompt as external files for separation of concerns).
Quota Management (for the GenAI API and package management).
Multidimensional Evaluations (expanding to arbitrary fact counts for graders).

For a complete technical breakdown of these architectural pillars—including external configuration schemas (questions.json, thrifty_system_prompt.txt), solver rate-limiting defenses (version="0.51.0" pinning), grader quota decoupling, and score curving math formulas see the README's Evaluation Pipeline Architecture & Technical Reference.

Running the eval

These commands create a matrix of evals subject to model x skill condition x sample x epoch. For a quick visual (that I’ll repeatedly reference throughout this), take a look below.

Caption: Visualization of the eval matrix that will be scored and compared. In Inspect AI, rubric elements, represented here as Facts, are graded Correct (C) or Incorrect (I).

With all that said, it’s finally time for the rubber to meet the road. I ran our evals with the following command

inspect eval skills-eval.py \
  --model google/gemini-3.5-flash-lite,google/gemini-3.6-flash \
  --time-limit 300 \
  --epochs 2 \
  --max-tasks 4 \
  -T web_access=false

This command runs a 4-way parallel sweep across models and skill conditions while enforcing a 300-second task timeout and disabling web search to minimize token spend. See the README for a complete CLI Parameter Reference and details on External Configuration Template Mechanics.

Run and wait

If you intend to and haven't done so already, run the command line. If you're interested in how to read over and interact with the Terminal User Interface, please refer to the previous codelab.

Watch the running command line long enough to determine it hasn’t crashed or hit an obvious error and then maybe take a moment to make some food, grab some water or take a walk around the block. Enjoy the small things in life, okay? Worst case, you get a 20 minute walk out of it, not so bad eh!?

Viewing the evals

With that said and done, reading the terminal output is only the most basic way of understanding the evals. Now that you have the raw eval logs, what do you need to do to begin to use these analytically and, more importantly, persuasively?

To start off, I simply ran

inspect view

and clicked the link to open the browser to the GUI.

For step-by-step diagnostic trace analysis in the browser GUI, see the README section on Local Diagnostic Trace Analysis (inspect view).

Mine came out like this (after a little bit of column drag-and-drop):

While we covered the rudiments of looking over the inspect view GUI in Evaluate agent skills using open source frameworks, there is significantly more that can be done with it. Speaking of which, the TASK ARGS column (not pictured) serves as a quick reference for what arguments were passed to a task.

Macro view: Comparing runs

While LLM’s are inherently stochastic (and given floating point hardware, nondeterministic), when running evals we are trying to use sampling to characterize the average observable metrics (e.g. correctness, latency and token usage) which our particular configuration model x skill scores on relevant questions. As such, the eval tasks were set up to characterize how a change to an independent variable (model or skill) affect dependent variables (the aforementioned metrics).

A. Comparing models head to head

Grouping evaluation runs by model in inspect view allows for a direct "eyeball" inspection of how skill inclusion alters accuracy across baseline controls—given a specific model, what is the impact of adding a skill?

I have added some annotations to make metric change from baseline to skilled a little more apparent:

The above shows that

In all cases, the addition of a skill improved or tied scores relative to the same model’s baseline.
All of these cases also increased duration

Another simple measurement that can be done is to look at the TOKENS column where gemini-api actually led to a decrease in usage relative to its baseline in 3.6-flash.

While the general relationship between tokens and score is difficult to parse visually, one notable result: for 3 out of 4 task x model configurations, gemini-3.6-flash used more tokens than the comparable 3.5-flash-lite run.

B. Comparing skill conditions (skill vs. baseline)

Now to analyze the inverse; in more concrete terms: given a particular skill, what is the effect of changing the underlying model?

What was the effect of the change from 3.5-flash-lite to 3.6-flash?

For 3 out of 4 of these, the changes between the two models amounted to an up to 45% increase in accuracy. For whatever reason, however, changing the gcloud skill from using one model to another led to a slight decrease.

While these are both valuable findings, they warrant further questions:

Are these representative samples?
If these findings are repeatable, how can we characterize the relationship between the existing metrics?

For the first of those, if you want to make more representative samples, know that you should conduct further research using more questions, samples and epochs and compare those metrics with these. For the sake of this blog series though, I’ll leave that for you to do.

For the second though, we can and will endeavor to do so. That said, it might help in designing follow up analysis to think a little bit about common patterns encountered during evals. This is especially true for those running evals themselves (possibly on different skills or rubrics); you are likely to encounter very different metrics than mine, and thus I’ll lay out some common patterns and follow up actions to investigate them further.

Mental models for eval comparison (skill vs. baseline)

When analyzing evaluation runs, comparing skilled execution against baseline controls typically maps to five distinct diagnostic outcomes—ranging from High-Efficiency Capability Lift (best) to Context Overload & Skill Regression (worst).

For a complete breakdown of this diagnostic taxonomy and actionable audit steps for each outcome, see the README breakdown of Diagnostic Mental Models for Eval Comparison.

Micro view: Sample-level diagnostics

While high-level metric summaries alert you to outcomes like cost bloat or skill regression, opening an individual sample surfaces the Sample Details panel for granular trace analysis:

Transcript tab: How to read the LLM's conversation as it happened

This tab shows the exact multi-turn conversation between the solver agent, the sandbox shell, and external tools. Use it to diagnose model reasoning versus environment noise:

System Prompt Verification: Confirm that web search rules (-T web_access=false) and automated time limits (300 seconds) were correctly injected into the container environment.

Skill Ingestion Check: Verify whether the agent activated the skill. If activation didn’t occur, your eval is mostly testing baseline model knowledge rather than skill utility.

Here is an example of one case where the skill was activated.

This means: The agent actually ingested gemini-api instead of relying on baseline pre-training memory.

Of note though, some of the tasks where skills were made available did NOT activate the skill. If you’re a skill author, you may want to rewrite the “activation criteria” (aka what situation calls for using the skill) such that it's more applicable to the specifics of relevant tasks.

Errors and reasoning loops

Auditing transcript details allows you to distinguish model reasoning loops (e.g., repeated redundant tool calls) from sandbox environment noise (e.g., container timeouts or missing binary dependencies). For step-by-step diagnostic trace auditing procedures, see the README reference on Sandbox Noise vs. Model Reasoning.

Scoring tab: Multi-fact verification breakdown

Clicking the Scoring Tab displays the empirical breakdown of our custom multi_scorer on a sample:

The multi_scorer aggregates the results of checking the sample’s answer against individual binary yes/no Facts using model_graded_qa. The generated list of scores in the range [0.0, 1.0] is supplied to our custom_reducer which calculates their arithmetic mean. From there, it applies quadratic score curving ( $mean^{2}$ ) to ensure that the further from correct the mean of the supplied answers are, the lower the score is pulled. This allows the most correct answers to stand out immediately. In the Scoring Tab, this downward curve maps a raw 5/6 fact score ( $\approx 0.8333$ ) down to a normalized 0.65 sample score. For complete formulas and math derivations, see the README section on Atomic Fact Verification & Quadratic Curving.

Visual milestone: The transition to cohort analytics

While inspect view provides outstanding deep-dive diagnostics for individual sample traces, evaluating dozens of models across multiple skill domains gets confusing fast (as you may have seen above). Weighing the correlations between skill inclusion vs exclusion and model changes against each other can be confounding and more art than science if we don’t find better ways to reason over them; it requires a structured matrix overview.

Remember that mess of sectors of a 3d space I showed you earlier representing all the configs? Well, unless we find numeric ways to collapse that or quantify deeper comparison, you won’t get the granular information you need to choose between two similarly capable alternatives. Worse yet, you won’t be able to communicate this to the people with their hands at the purse strings (unless of course, that's you too).

To make this easier, next time we’ll start down the road of understanding and displaying these things visually: As visualized in the cohort matrix linked above, understanding agent capability factor-by-factor requires cohort slicing across multi-dimensional metrics.

In Part 2, we scale our analysis from single log UI inspection to aggregate scoreboards with inspect viz!

Codegolf: Build a container in Cloud Build

Katie McLaughlin — Wed, 27 Mar 2024 01:19:28 +0000

Banner Photo by Viktor Kiryanov on Unsplash

I read Adam Ross's Modernizing cloudbuild.yaml for Container Builds and saw all his mentions of "character counts" and it got me thinking:

How codegolf can we get with this?

Context: like regular golf, codegolf asks you to implement something in the least amount of character possible.

Starting with Adam's finishing iteration:

steps:
- id: 'Build Container Image'
  name: 'gcr.io/cloud-builders/docker:latest'
  script: docker build . --tag "${_LOCATION}-docker.pkg.dev/${PROJECT_ID}/${_REPO}/${_IMAGE}:latest"

images:
- "${_LOCATION}-docker.pkg.dev/${PROJECT_ID}/${_REPO}/${_IMAGE}:latest"

options:
  automapSubstitutions: true

substitutions:
  _IMAGE: service
  _LOCATION: us
  _REPO: container

Initial character count: 358

It appears his character count doesn't inline new lines, which is effectively the same as the wc -c minus the wc -l. I'll be using the following for my character counts:
expr $(wc -c cloudbuild.yaml | cut -d' ' -f1) - $(wc -l cloudbuild.yaml | cut -d ' ' -f1)

He notes that the introduction of Artifact Registry make this a bit longer than it would be in Container Registry, but we'll sit with that since it's the supported registry.

We'll also keep with this last iteration's requirements: a project might have more than one registry, so we'll keep some parameterization.

Each iteration I'll make sure is a valid build. For testing, I'll ensure gcloud builds submit runs clean in a folder containing just this cloudbuild.yaml file and my go-to codegolf Dockerfile (that's also a valid Cloud Run service, just for fun).

What's in an ID?

To start, Cloud Build steps don't require an ID, so we can remove that line entirely. Without an ID, the Cloud Build build details in the Google Cloud console will use the image name as the step details, so there is a slight usability disadvantage to this change, but nothing that affects functionality.

-- id: 'Build Container Image'
-  name: 'gcr.io/cloud-builders/docker:latest'
+- name: 'gcr.io/cloud-builders/docker:latest'

Character count: 329.

Deduplicate with dynamics

We've also got the same image ID used multiple times. We want to keep the dynamic nature of this, but we also want to use substitutions in the value itself. We can use dynamicSubstitutions to allow use of substitutions within other substitutions, but adding this adds ~27 characters to our file (and we're removing 45x2 characters so this is a win!)

steps:
- name: 'gcr.io/cloud-builders/docker:latest'
  script: docker build . --tag "$_IMAGE_URI"

images:
- "$_IMAGE_URI"

options:
  automapSubstitutions: true
  dynamicSubstitutions: true

substitutions:
  _IMAGE_URI: ${_LOCATION}-docker.pkg.dev/${PROJECT_ID}/${_REPO}/${_IMAGE}:latest
  _IMAGE: service
  _LOCATION: us
  _REPO: container

Character count: 326.

At this point we can also remove the superfluous quotation marks, saving 6 characters, bringing us down to 320.

Shortest values

Now we get to the part where we work out: What's the minimum value for the variables we're using?

The $PROJECT_ID is a default substitution, so we should keep that as is, and it will be dynamic for our project.

We're already using the shortest Artifact Registry region, the US multi-region (the longest Google Cloud region could be up to 24 characters, so 2 is great!)

But we can possibly do better with the Artifact Registry name and the Container Name.

d (for "Docker") is a valid Artifact Registry repository name, and i (for "image") are valid, so we can save ~15 characters there:

...
-  _IMAGE: service
+  _IMAGE: i
   _LOCATION: us
-  _REPO: container
+  _REPO: c

Character count: 306.

Minimal variables

Any of our underscore substitutions we can reduce down to two letters, the underscore and some useful character. (We're replacing "Image URL", "Image", "Repo", and "Location", respectively.)

steps:
- name: gcr.io/cloud-builders/docker:latest
  script: docker build . --tag $_U

images:
- $_U

options:
  automapSubstitutions: true
  dynamicSubstitutions: true

substitutions:
  _U: ${_L}-docker.pkg.dev/${PROJECT_ID}/${_R}/${_I}:latest
  _I: i
  _L: us
  _R: d

Character count: 254.

(We're now down below what Adam was before migrating to Artifact Registry! 😅)

Remove tags

Another reduction we can make is removing the :latest tags. "Latest" is used in Docker where there is a lack of tag, so this is implicit in our call (and it just so happens that the Cloud Builder we are referencing has a "latest" tag. (You can check this by going to gcr.io/cloud-builders/docker in the browser, which will show you information about this image in its registry. This image in particular will direct you to its Artifact Registry information, because this registry has been automatically redirected.)

steps:
- name: gcr.io/cloud-builders/docker
  script: docker build . --tag $_U

images:
- $_U

options:
  automapSubstitutions: true
  dynamicSubstitutions: true

substitutions:
  _U: ${_L}-docker.pkg.dev/${PROJECT_ID}/${_R}/${_I}
  _I: i
  _L: us
  _R: d

Character count: 240.

Putting the args back

The last step we can do to save space is to actually re-introduce the args parameter, rather than using script. In this case, we will be adding some characters for the separation of the arguments, but the biggest win is being able to remove the ~27 characters that the automapSubstitutions config setting adds.

steps:
- name: gcr.io/cloud-builders/docker
  args: ['build','.','--tag','$_U']

images:
- $_U

options:
  dynamicSubstitutions: true

substitutions:
  _U: ${_L}-docker.pkg.dev/${PROJECT_ID}/${_R}/${_I}
  _I: i
  _L: us
  _R: d

Character count: 213.

Tip: Make sure you check what your default entrypoint is for images! In this case, I have also removed docker, as my args are appended to the default entrypoint. The script needs the full command (I may have spent too long debugging this issue 😅)

Twist

Actually you only need 22 characters:

gcloud run deploy a
[ENTER]
[ENTER]

First time you run this you'll need 1 extra character (ENTER) to enable APIs, and also 1 extra character (ENTER) to create the Cloud Run Source Deploy image registry. (You can also use this method to automatically create the Artifact Registry for you, read more about this on my blog post: "Auto-provisioning Artifact Registry though Cloud Run Source Deploys (glasnt.com)").

Summary

You can make a very short Cloud Build configuration to build images, but you should consider the readability of your scripts as you go. Addressing any cruft and modernizing your scripts can be an interesting exercise for you and your infrastructure.

With thanks to Adam Ross for the review!

Migrating from Secret Manager API to built-in secrets

Katie McLaughlin — Sun, 19 Jun 2022 23:17:12 +0000

Secret Manager is directly integrated into a number of products, including Cloud Run and Cloud Build.

Many samples I've authored use the Secret Manager API by calling the API from within the code. This can be more secure, but variable leakage is something that needs to be considered in any deployment. Removing the direct API calls can help with portability, and reduce the amount of dependencies in the deployment, and the complexity of the code itself.

So let's look at how we can migrate to built-in secrets.

Integrating Secret Manager API

Screenshot: "Secret value: Input your secret value or import it directly from a file.", showing a file upload dialog, or a text area input field.

When you create secrets, you can create them from a file or from direct input. When you retrieve these secrets, you may presume they're a single value, or a file of content you have to process.

Once you create the secret, you can access it with the Secret Manager API using a client library for Secret Manager available in many different languages.

This post will use Python examples and packages, but the same patterns can be applied to other languages.

For example, this code snippet will retrieve the latest version of the secret mySecret in the current project:

import google.auth
from google.cloud import secretmanager as sm

# trick to detect current project
_, project = google.auth.default() 
client = sm.SecretManagerServiceClient()

secret = "mySecret"

name = f"projects/{project}/secrets/{secret}/versions/latest"
secret_value = client.access_secret_version(path).payload.data.decode("UTF-8")

The secret_value can then be used as required. This presumes that mySecret is a single value, for example an API key.

One way developers can include multiple secrets in one format is with .env files. These files contain one or more key/value pairs which mean you can store multiple secrets in one file, then use helper packages to parse these values into your code. This prevents you from having to define multiple separate secret values.

If mySecret was a .env file, I could load it using python-dotenv:

## pip install python-dotenv
import io
from dotenv import load_dotenv

load_dotenv(stream=io.StringIO(secret_value))

Or I could to similar with django-environ, for example:

## pip install django_environ
import io
import environ
env = environ.Env()
env.read_env(io.StringIO(secret_value))

In both of these examples, these packages presume that the values are in a .env file within the same directory as the running process. Using StringIO means that a string variable can be sent to these methods in a file-like object, so the values can be read as though they were a file.

When using the Secret Manager API, you can run the application in any place where Google Cloud authentication works. That is, any Google Cloud service, or your local machine if you have set up gcloud.

Consuming built-in secrets

If you have your secret configured as an environment variable, then you only have to retrieve it as an environment variable:

import os
secret_value = os.environ.get("MYSECRET", None)

This method prevents KeyErrors if the key is not found.

For products with built-in secrets, they may be made available in additional ways.

In Cloud Build you can connect secrets directly to environment variables. But Cloud Run additionally allows mounting the secret as a volume, which means it can be read as a file. Any time the file is read, if you specify latest, the current latest will be received! However, you must mount the file in a new volume so you can't mount it directly in the default .env.

If you want to adapt your code to handle a local .env file, or a separately mounted file, or an environment variable, you will have to ensure all methods are possible in your code.

You also need to consider which configuration takes priority, as by default both python-dotenv and django-environ accept the first declared value as the value they use. You can override this by using the --override or --overwrite respectively.

When developing applications, I might choose to say my local .env file takes priority, then any mounted secrets, and then any declared variables.

Another point to mention is that using python-dotenv, if a file is not found or a value is empty, it silently continues. This means you can include the method calls without having to explicitly handle errors:

import io
import os

from dotenv import load_dotenv

load_dotenv()
load_dotenv("/secrets/.env")
load_dotenv(stream=io.StringIO(os.environ.get("MYSECRET", None)))

The same code works for django-environ, where you can just import in the order of priority without having to worry about missing files.

import io
import os

import environ
env = environ.Env()

env.read_env()
env.read_env("/secrets/.env")
env.read_env(io.StringIO(os.environ.get("MYSECRET", None)))

Note that in these examples, I'm choosing /secrets/as my volume, and keeping the path the same name as the original file. You can choose any volume and path, as long as the volume is not already used by the application (for example, if you choose /app/ as your working directory, you cannot mount secrets there.)

Deploying built-in secrets

To run this code locally, you'd create a .env file with the contents MYSECRET=serkitValue.

If you're committing this code to git, ensure you're not committing the secret file! Make sure you add .env to your .gitignore file!

You can also choose to ignore any contents of your .gitignore file in your Google Cloud commands by adding .gitignore's contents to .gcloudignore:

.git
#!include:.gitignore

You can then create the secret from this file with gcloud:

gcloud secrets create mySecret --data-file .env

For Cloud Build, you will need to ensure the secret is available in the environment (which your script can then use):

steps: 
 - name: python:slim
   entrypoint: pip
   args: ['install', '-r', 'requirements.txt', '--user']

 - name: python:slim
   secretEnv: ['MYSECRET']
   entrypoint: python
   args: ['main.py']

availableSecrets:
  secretManager:
  - versionName: projects/$PROJECT_ID/secrets/mySecret/versions/latest
    env: 'MYSECRET'

You can also reference secrets in the args call itself, using bash variables.

For Cloud Run, you'll have to deploy the service specifying either a mount or an environment variable:

# for mounted volume
gcloud run deploy myservice --source .  \
  --update-secrets /secrets/.env=mySecret:latest

# for environment variable
gcloud run deploy myservice --source .  \
  --update-secrets MYSECRET=mySecret:latest

And never forget IAM!

Don't forget: you'll also need to make sure the service account you're using has permissions to access your secret!

Katie is a Developer Advocate for Google Cloud, focusing on Python and Serverless. She tweets @glasnt about clouds, crafts, and cats.

Using Workflows to dynamically update a Datasette website

Katie McLaughlin — Sun, 20 Feb 2022 22:04:32 +0000

A while ago I spoke with Simon Willison about the sorts of things he'd like to have in a cloud provider in order to deploy Datasette nicely and repeatedly. A few of the things mentioned were being able to attach to object storage, especially when databases were particularly large and it didn't make sense to mount them into the container image itself.

With the preview of the second generation execution environment with Cloud Run, one of the many new features is the ability to mount Cloud Storage as a network filesystem.

With this, you can attach any Cloud Storage bucket and have a Datasette web service serve data from that source.

This works very well when you have one database that you want to serve. However, one of the known limitations of Datasette, where if you serve a folder of content, the glob of files in that folder is loaded at server start time, and not refreshed unless the server is restarted.

There are few ways we can work around this issue. You could have the process within the container restart when the mounted filesystem changes, using something like inotify (fsevent on macOS). You could also consider periodically starting, but that may mean restarting when not required, or not often enough. For a sufficiently low traffic system, you'd end up automatically booting from cold after a time of inactivity, anyway.

The way I chose to solve this was to use the event of an object being uploaded to the bucket to trigger an update of the service, thus restarting Datasette (albeit dramatically).

And since I was now listening to the upload event, I thought, why not do more processing while I'm here?

I'd recently learnt from Lab 6 of the Pic-A-Daily Workshop how to use Workflows to handle events, so I thought I'd try developing one of my own.

The result is github.com/glasnt/dynamic-datasette, and once deployed it works like this:

you upload an new file to the upload bucket
the event of that file being uploaded triggers a function, that starts a Workflow
the workflow then follows through a number of steps:
- sends the file for processing:
  - if the file is already an SQLite database, then it sends it off to the serving bucket
  - if it's one of the types of files it knows about, it uses one of the many *-to-sqlite tools Simon has made. For example, csvs-to-sqlite.
  - Once converted, the resulting SQLite database file is uploaded to the serving bucket
- updates metadata for Datasette:
  - this step mostly just ensures that the Datasette service displays the datetime when this workflow was run, so users can know how new their data is that's being served
- restarts the web service
  - This step uses the replaceService API to force a 'restart', deploying a new revision of the Datasette service, ensuring that the latest information in the serving bucket is being relayed to users.

The Datasette container mounts the serving bucket, and metadata.json in that same bucket, then just calls "datasette serve" on the mounted folder.

You can try this out on your own project by deploying the code with Terraform, following the instructions in the README.md.

Try extending the functionality of the process-upload function to do more steps, or adding more metadata to the hosted service.

Photo by Victor on Unsplash

Open Source Credit Survey

Katie McLaughlin — Tue, 01 Feb 2022 21:31:08 +0000

The University of Vermont is conducting a survey to help understand how people receive credit for tasks in open source, and are seeking your participation!
Survey: https://qualtrics.uvm.edu/jfe/form/SV_1zUs19oVcZJ0SPA

Today's open source world has similarities to the world of open source when I first wrote about this problem in 2015. I wrote a tool that helps discover who is contributing to GitHub projects based on the contributions that aren't code. In 2022, we are still contributing countless hours of time towards communities to drive and improve open source across many industries, but we don't know how much of that work is credited beyond automated tools like this.

I'm a member of the OCEAN, a research team from the University of Vermont Complex Systems Center, and we're collaborating with industry experts from Google, OSI, IEEE, PSF, and OSSF, among others. You may have recently heard about our project on CHAOSSCast, or being shout-out on SustainOSS.

As we've written in our correspondence in Nature Computational Science, we believe that having a recognition model for the work done in any open source community can help more in just gaining credit for code, but providing clear signaling of the work done for the community.

As part of this work, the University of Vermont team is conducting a survey about how people receive credit for the tasks they do as part of open source projects.

The survey asks about your experiences with receiving credit on open source projects. The survey asks multiple choice questions about how often you received credit for the tasks you did, and written response questions about what did and did not go well on open source projects.

We hope that with this information, we can improve the equity of contributions across all of open source.

Click on this link to participate in the survey: https://qualtrics.uvm.edu/jfe/form/SV_1zUs19oVcZJ0SPA

OCEAN is a partnership between the Google Open Source Program Office and the Vermont Complex Systems Center. Google does not have any influence on the survey, and will be unable to see any individual responses.

When you're not around: trigger Cloud Run on a schedule

Katie McLaughlin — Fri, 22 Jan 2021 00:00:31 +0000

On this edition of Serverless Expeditions, we take a look at scheduling automated Cloud Run jobs with Cloud Scheduler.

Check out the video version of this blog post.

Source code for this blog post is available on GitHub under the "scheduled-cloud-run" folder.

GoogleCloudPlatform / serverless-expeditions

Cloud Run is designed to host a containerised web service -- any language you like, as long as the service listens on port 8080 -- but your code can run on events other than a user visiting your site.

An example of this design pattern would be a nightly billing job: you want to process billing data nightly at 1am, but you don't want to be the person to click a button in the middle of the night.

If you wanted to implement such a receiving service in NodeJS, you could implement it like in our example code: at the top of the code, create and initialize an Express app that listens for incoming HTTP calls. Note that Cloud Scheduler uses the “text/plain” content type, so you'll have to tell body-parser to parse more than the default “application/json” type. The POST handler parses the minimum balance that was sent to it from Cloud Scheduler, and then sends that off to the billing method later in the code.

Cloud Scheduler is a managed service offered on Google Cloud, where you can create jobs on a cron schedule that either makes a HTTPS call, or publishes a Pub/Sub message. In this example, you can setup a scheduled job that runs at 1am every night by entering the cron value 0 1 * * * in the Frequency field, and setting the HTTP Target to be the URL of your Cloud Run service you created earlier.

However, you need to make sure that this billing service is private, and not accessible by anyone on the internet. To convert a public service to private, you need to remove the allUsers member from the service -- that is, deny anyone from accessing it. To then allow Cloud Scheduler to access the service, create a new service account, assigning it the Cloud Run Invoker role. Then set that service account's email identifier in the Auth Header setting in Cloud Scheduler, under "Add OIDC token". OIDC, or OpenID Connect, is a small layer on top of OAuth that handles identity. This field tells Cloud Scheduler to use the new service account's identity for running the job.

The free tier of Cloud Scheduler allows you 3 jobs for free, no matter how many times the job is run. Each job after that is USD$0.10 each. You may also incur costs for excess Cloud Run execution time, database costs, etc.

By using Google Cloud managed services that connect to Cloud Run, you can build more complex architectures, and automate manual processes to make your developer live easier.

About Serverless Expeditions

Serverless Expeditions is a fun and cheeky video series that looks at what serverless means and how to build serverless apps with Google Cloud.

Follow these hosts on Twitter at @glasnt and @martinomander.

Running multiple versions of Python in Cloud Run

Katie McLaughlin — Fri, 11 Dec 2020 01:05:06 +0000

On this episode of Serverless Expeditions, we take a look at how to deploy multiple versions of Python in Cloud Run.

Check out the video version of this blog post.

Running multiple versions of Python in Cloud Run - Serverless Expeditions (YouTube.com)

As new versions of Python are released, trying to choose between the new and shiny or the old and stable can be an issue. But if you are deploying on Cloud Run, you don't have to choose between the two, you can run both.

Imagine this hypothetical scenario: you are an SRE for a company with a custom storefront, maintaining a fleet of virtual machines running various components, all with their own versions of Python. To improve reliability, you can upgrade all the components to use one version of Python, and move away from virtual machines, and into something more scalable, like serverless.

This upgrade won't happen overnight, and the storefront must keep running in the meantime. So your plan is multi-step: migrate from virtual machines to serverless first, then make the language upgrades. But before migrating the existing systems, a new service coming online could be deployed serverless first, skipping the virtual machine all together.

When choosing which serverless platform to upgrade to, limitations may exist on the languages available. For example, Cloud Functions only supports specific versions of Python.

However, the newest serverless offering from Google Cloud -- Cloud Run -- doesn't have this limitation. Cloud Run defines services as container images, which are defined by Dockerfiles, and can specify any base image. There is an entire repository of official Python images on Docker Hub which you can source from, by specifying the base image in the FROM line of your Dockerfile.

FROM python:2.7
...

FROM python:3.9
...

As you migrate each service from its original virtual machine, you can set the exact version of Python it was originally running. You can also upgrade each service in isolation, as your team has time and resources.

Once your dependencies have been upgraded and you want to test your service, you can deploy your new revision to only a limited number of users, using the Traffic Splitting feature of Cloud Run. You can set your new revision to serve, say, only 5% of requests, so if there is something wrong, only some of your users are affected, and you can quickly rollback if there are any issues.

By migrating each service in turn, you are able to control each service in isolation, allowing for a more incremental upgrade process, and a more reliable environment.

By being able to specify the exact version of Python being used in your project, you can run multiple services each with their own isolated Python environment, which allows you more time to upgrade each service to supported versions of Python.

It also allows you to try the newest versions of Python -- such as prereleases of Python 3.10 -- as soon as their images are available on Docker Hub.

About Serverless Expeditions

Serverless Expeditions is a fun and cheeky video series that looks at what serverless means and how to build serverless apps with Google Cloud.

Follow these hosts on Twitter at @glasnt and @martinomander.

"Point-and-click" continuous deployment with Cloud Run

Katie McLaughlin — Thu, 10 Dec 2020 20:36:47 +0000

On this edition of Serverless Expeditions, we take a look at how to deploy a GitHub repo on GitHub, though Cloud Run's new "point-and-click" integration, no Dockerfile required.

Check out the video version of this blog post.

Say you've created a website for a travel blog, and you want to host it online. You've got your code on GitHub, and you have a README of instructions of how a friend could deploy a copy of your website: they have to have your programming language installed on their machine, install some package dependencies, then run a command to start a web server.

If you want to host your website, you need a way that you can run these steps any time you want to update your code. This process is known as Continuous Deployment: when you commit a change to code a process is started that takes the new version of your code, performs the prerequisite tasks, and then starts a web server of your new code.

Cloud Run, a serverless platform from Google Cloud, allows you to to bring any container to serve your website, no matter the programming language used inside, as long as it follows the Runtime Container contract: a web server must be listening for requests on 0.0.0.0, on port 8080 (or the configured PORT environment variable).

So, if you have a website, how do you turn it into a container? You could write a Dockerfile, a set of instructions to build a container image based on your setup. But if you're using a common language like Python, Go, or Ruby, the instructions are going to be similar for many websites: copy your code into the container, install your dependencies, and start the web server.

You can setup your Cloud Run service through the Google Cloud Console using the new walkthrough integration with Cloud Build: you connect your GitHub repo with Cloud Run, and set up your website to be built any time your code updates.

However, instead of having to write a Dockerfile, you can now use Buildpacks, a set of common instructions for popular programming languages that help you build your code into a container image that Cloud Run can deploy.

If you're using a language like Go, Node.js, Python, Java or .NET Core, this will automatically be detected when building a container using Buildpacks. These common languages and their respective package installation processes are handled for you; all you need to do is provide the command to start your web server in a special file called Procfile.

In the example in the video, Martin is using Flask, a Python-based web framework. To start his application, he runs:

python3 -m flask run

So his Procfile will be:

web: python3 -m flask run

The "web" in this case means "web server", and is the special term Buildpack looks for in a Procfile to start the web server.

He should also configure his Flask app to listen on the expected IP and port. He can either add that to his app.run command, or in the Procfile itself as part of starting Flask:

web: python3 -m flask run --host 0.0.0.0 --port $PORT

As long as he adds this file to his codebase, he doesn't need to change anything else about his website to host it on Cloud Run.

Once configured, any commits made to the GitHub repo -- from the command line, VSCode, GitHub's website, or anywhere else -- are automatically detected and deployed.

To learn more about the concepts discussed in this video, read more:

You can try some of the sample buildpack apps at https://github.com/GoogleCloudPlatform/buildpack-samples

About Serverless Expeditions

Serverless Expeditions is a fun and cheeky video series that looks at what serverless means and how to build serverless apps with Google Cloud.

Follow these hosts on dev.to at @glasnt and @martinomander.

Deploying Django and Wagtail on Google Cloud

Katie McLaughlin — Thu, 01 Oct 2020 23:03:15 +0000

On this edition of Serverless Expeditions, we take a look at how to deploy more complex applications on Google Cloud, using Django as an example.

Check out the video version of this blog post.

When moving legacy applications into serverless infrastructure, there is going to be some sort of state involved: there will be a database, media, and other components. Traditionally these can all be run within the confines of a virtual machine, where all the services can talk to each other in that environment. But when moving to serverless, all these components need to be specifically provisioned and connected together.

While Cloud Run itself is stateless, it can be connected to other stateful services such as Cloud SQL for relational databases, or Cloud Storage for static media hosting.

In today's episode, we'll discuss containerizing and deploying a typical Django application using Cloud Run and other Google Cloud services. The steps described can be applied to any website whose CMS uses the Django web framework, such as Wagtail or Django CMS.

Django supports both PostgreSQL and MySQL, both of which are available in Cloud SQL, a managed database service. When connecting to a Cloud Run service, specifying --add-cloudsql-instances allows the service and the database to communicate over an automatically encrypted connection. The authentication from the application to the database is achieved by specifying a connection URL, stored securely in Secret Manager, loaded using the Python API. The database can then be populated by running ./manage.py migrate

Static and media assets can be migrated to use Cloud Storage, with help from the django-storages package. By creating a new storage bucket and specifying the name in the django settings.py, then running ./manage.py collectstatic, the images and assets are uploaded to the bucket.

However, those two manage.py commands -- migrate and collectstatic -- are typically run within an interactive terminal. Cloud Run has no such terminal, so these commands need to be run elsewhere. One method is to run these commands as part of the continuous integration process with Cloud Build. Adding a 'migrate' step using the app-engine-exec-wrapper between the typical build and deploy steps allows migration to happen as part of deployment.

If database migrations need to be manual, Cloud SQL proxy can be used to interact with a Cloud SQL database locally. Details of how to set this up with Docker Compose are available here.

With a managed database, highly-available static assets, automated deployment pipelines and robust containerisation solutions, you can migrate your Django applications to Google Cloud with confidence.

If you want to get started with these concepts, check out the Django Codelabs:

You can also find a sample implementation of these concepts at GoogleCloudPlatform/django-demo-app-unicodex.

About Serverless Expeditions

Serverless Expeditions is a fun and cheeky video series that looks at what serverless means and how to build serverless apps with Google Cloud.

Follow these hosts on Twitter at @glasnt and @martinomander.

DEV Community: Katie McLaughlin

Interactive AI Eval Dashboards with Data Studio

Implementation prerequisites

Back to the story at hand

1-click Data Studio template setup

Clone the master template

30-second setup and customization

Overview of new pre-configured dashboard components

Chart A: Performance and cost (scatter/bubble chart)

Chart B: Accuracy and error bars (±1 SEM whiskers)

The 4 dynamic cohort slicers (drop-down controls)

View and interact

Slicer-based analyses

Conditional uptake analysis: Separating skill discovery from capability

Baseline or has activated skill = false (notice these both score relatively low)

Baseline or has activated skill = true

Survivorship bias and joint operational uptime (completed analysis)

completed: ALL

completed: TRUE

Are these findings generalizable to every scenario?

Conclusion and next steps: Where to go from here

Step up to the Sheets: AI Eval Export and Illustrating Data

Implementation prerequisites

Back to the story at hand

The extractor tocsv.py

Heavy lifting in Python: Custom telemetry columns

Exporting to Google Sheets

Why leave the cradle?

AI Evals at a Glance: Heatmaps for Stakeholders

Visualizing AI evals with Inspect Viz

The 40-log gridlock

Follow-along requirements

Cognitive overhead and segue to BI analytics

Designing AI Evals: Clarity Now and Visualization Next

AI evals and analysis

Level up your evals

A note on models

Setting up the eval

Running the eval

Run and wait

Viewing the evals

Macro view: Comparing runs

A. Comparing models head to head

B. Comparing skill conditions (skill vs. baseline)

Mental models for eval comparison (skill vs. baseline)

Micro view: Sample-level diagnostics

Transcript tab: How to read the LLM's conversation as it happened

Errors and reasoning loops

Scoring tab: Multi-fact verification breakdown

Visual milestone: The transition to cohort analytics

Codegolf: Build a container in Cloud Build

What's in an ID?

Deduplicate with dynamics

Shortest values

Minimal variables

Remove tags

Putting the args back

Twist

Summary

Migrating from Secret Manager API to built-in secrets

Integrating Secret Manager API

Consuming built-in secrets

Deploying built-in secrets

And never forget IAM!

Using Workflows to dynamically update a Datasette website

Open Source Credit Survey

When you're not around: trigger Cloud Run on a schedule

GoogleCloudPlatform / serverless-expeditions

About Serverless Expeditions

Running multiple versions of Python in Cloud Run

About Serverless Expeditions

"Point-and-click" continuous deployment with Cloud Run

About Serverless Expeditions

Deploying Django and Wagtail on Google Cloud

About Serverless Expeditions

Survivorship bias and joint operational uptime (`completed` analysis)

The extractor `tocsv.py`