DEV Community: Tilde A. Thurium

Harness engineering doesn't mean building your own harness

Tilde A. Thurium — Tue, 15 Sep 2026 01:04:48 +0000

It means giving the agent enough context to do the right thing

For this week's Agent Factory podcast, I interviewed Ryan Lopopolo, the engineer who coined the term "agentic harness."

What's in the video

How to Stop Writing Code Manually: Ryan explains how to sculpt smaller feedback loops into long-horizon agentic workflows via harness engineering
Harness, Deconstructed: Billy Jacobson delivers a code walkthrough of a simple harness, exploring linear and closed-loop styles.
The Three-Layer Stack: Smitha Kolan breaks down how to use Gemini 3.8 Flash, Antigravity Boost, and the Google Skills GitHub repo to configure memory, tools, and guardrails.

The thing that struck me was how many best practices for scaling software engineering teams also apply to coding agents.

How much code are you writing manually these days? Let me know in the comments!

is Graph Engineering just reinventing systems architecture for the AI age?

Tilde A. Thurium — Thu, 10 Sep 2026 19:23:39 +0000

Bringing backend predictability to multi-agent systems

I've been hearing the phrase "graph engineering" thrown around a lot lately. I originally assumed it had something to do with knowledge graphs, or maybe GraphRAG. Not quite!

I sat down with Annie Wang and asked her to break it down from the ground up: what graph engineering is, and how it gives developers control over complex AI systems.

What's in the video

Demystifying the terms: The actual differences between an agent harness, a loop, and a graph.
A real-world example: Walking through an automated pull request (PR) review pipeline using fan-out, join, and conditional router patterns.
What it isn't: Why graph engineering is fundamentally different from knowledge graphs and agent swarms.
Shared state: How to orchestrate shared state and strict schemas using Google's Agent Development Kit (ADK).

The point that stuck with me is that graph engineering basically feels like modern backend architecture —like microservices— applied to agents. Instead of letting LLMs wander in a black-box loop and hoping for the best, you treat nodes with strict schemas and fan-out/join pipelines. It gives us back the predictability and debuggability that software engineers actually need.

(You can also follow along with the hands-on ADK Codelab).

Did your mental model of "graph engineering" match up? Are you still running straight loops for your agents, or have you started breaking them into graph patterns? Let me know in the comments!

4 pitfalls of loop engineering (and how to fix them)

Tilde A. Thurium — Wed, 09 Sep 2026 17:37:13 +0000

Perhaps you've heard the term Loop Engineering: instead of solving a problem by hand, you build a system, set a measurable goal, and let an agent keep iterating until it gets there.

It sounds great until something goes wrong.

So I sat down with Annie Wang to talk through the four most common ways Loop Engineering breaks down, and how to fix each one.

What's in the video

What Loop Engineering actually means: building an agentic system that retries toward a definable goal
Failure #1 - runaway loops: you need a hard stop rule because tokens cost real $$$.
Failure #2 - unverified autonomy: why letting an agent grade its own work is like asking a kindergartner to grade its own homework, and why you want agent A checking agent B's work instead
Failure #3 - vague or uncheckable goals: why "make this better" breaks an LLM, and how to write criteria that are actually non-negotiable
Failure #4- complexity overflow: when a single loop chokes on a big task, and why that's the moment to move from Loop Engineering to Graph Engineering

Have you hit any of these failure modes yourself? Tell me which one (or more) got you.

Building a viral Imax ticketing app that never crashes

Tilde A. Thurium — Thu, 20 Aug 2026 21:29:37 +0000

When 150,000 tickets went on sale for The Odyssey in 70mm IMAX, they sold out almost instantly. But plans change, cancellations happen, and good seats randomly open up at odd hours.

To solve this, Andrew Baker from Temporal built IMAXXING: a service that monitors every 70mm IMAX showing across the US and alerts subscribers the moment great seats become available. What started as a fun weekend project quickly scaled, now over 9,000 users.

I sat down with Andrew to break down the architecture: how durable execution keeps long-running workflows alive, how to debounce alerts so you don't spam users, and how serverless workers on Google Cloud Run handle sudden spikes in demand without overprovisioning.

What's in the video

Durable execution 101: How Temporal allows you to rewind history to the point of failure.
The Entity Workflow pattern: Why there is one persistent workflow per user subscription and separate monitoring workflows per showing across the country.
Signals & smart debouncing: How showing workflows send signals to wake up subscription workflows, and how a 60-second in-workflow timer batches multiple theater alerts into a single digest—without consuming active CPU while sleeping.
Serverless workers on Cloud Run: How running Temporal workers as serverless containers lets compute autoscale directly with task queue depth rather than generic CPU metrics.
AI agents for ops: How modern coding agents paired with Terraform and the gcloud CLI accelerated the deployment and operational dashboard setup.

The point that stuck with me is how durable execution fundamentally changes how you think about long-lived state and retries. Instead of building complex cron jobs, custom retry databases, and alert queues, the workflow state itself is the queue and the timer.

Have you experimented with entity workflows or running workflow workers on serverless infrastructure? How do you handle debouncing and noisy downstream APIs in your own apps?

Skills vs MCP: How AI tools have evolved

Tilde A. Thurium — Thu, 30 Jul 2026 17:50:23 +0000

Eighteen months ago, MCP was the thing. Every demo and chatbot connector was running on MCP under the hood. When I started seeing everyone talk about Skills, I got curious. How are MCP and Skills connected? Are they different lenses on the same problem?

I brought these questions to @greggyb again in our latest video.

What's in the video

What MCP actually does: giving a model programmatic access to APIs and the file system through natural language
Downsides of MCP: context bloat from loading hundreds of lines of tool definitions for tools you're not even using
What a Skill actually is (spoiler: pretty much a markdown file containing a reusable prompt)
Progressive disclosure: why an agent can "know about" 20 skills without loading the full instructions for all of them
Why MCP and Skills aren't competitors, and how Skills can even teach your agent how to use MCP tools

My favorite line from this one: the hottest programming language right now is English. Skills let non-developers write, iterate on, and share instructions that make an entire team's agents faster, no code required.

Are you using Skills yet? Curious where everyone's at.

AI builder essentials: tokens, context windows and RAG 101

Tilde A. Thurium — Fri, 24 Jul 2026 19:16:22 +0000

Tokens are an important foundational concept that underlie how we interact with LLMs. @greggyb and myself recorded a quick video to demystify what a token actually is and unpack several other related concepts.

What's in the video

What is a token? Tokens are the basic unit of input and output that LLMs operate on.
What is retrieval-augmented generation (AKA RAG)? Providing a model with extra data that isn't already in the model's training dataset, and is relevant to a specific query.
What are context windows? Context windows are how many tokens a model can use for input and output. Context window size varies from model to model, but it has increased a lot in recent years.
What is tokenmaxxing? Does using more tokens really mean you're being more productive? (Spoiler alert: no.)

To paint a bigger picture: more context you give a large language model, the better your responses are likely to be. The design patterns for providing that context have evolved at a rapid pace, and are likely to continue doing so.

What AI terms should we demystify next? Let us know!

What is an "agentic harness," actually?

Tilde A. Thurium — Thu, 16 Jul 2026 21:19:20 +0000

I've been hearing the word "harness" thrown around a lot lately. I assumed it just meant "the IDE" or "whatever app is running your agent." Turns out, it goes a little deeper than that.

So I sat down with @greggyb and asked him to explain it from the ground up: what an LLM actually does, what turns it into an agent, and where the harness fits into all of it.

What's in the video

Simon Willison's definition of an agent: an LLM with tools, running in a loop to accomplish a goal
What "tools" really means under the hood, and how function calling lets an LLM pull in context it wasn't trained on
The loop part: how an agent programmatically checks its own output to decide if it's actually done
Why the harness is everything after the LLM, and why that's a different thing than the interface
Why you can swap interfaces without touching the underlying harness, and why some agents don't need a UI at all

The point that stuck with me is the harness isn't the app you're looking at. It's the invisible plumbing deciding whether the agent keeps going or calls it done.

Did your mental model of "harness" match up? Or were you picturing something else too?

I built a native Android app in an afternoon, and I've never written a line of Kotlin

Tilde A. Thurium — Wed, 01 Jul 2026 18:32:34 +0000

I’ve always thought building a mobile app required climbing a massive learning curve just to get a basic environment set up. To test that theory, I tried building my very first Android app using Google AI Studio.

Five minutes later, I had a working prototype.

The coolest part about this isn't just the speed: it’s that anyone can do this. The traditional barriers to building software are disappearing, making it incredibly easy to just start creating.

I recorded the whole 5-minute process here if you want to see what it looks like in practice:

What's in the video

Prompting AI Studio to build a native Android app from scratch
Progressive Webapp (PWA) vs Android Native App in 2026: feature comparison
Sideloading the app onto an Android device via USB-C cable. No Play Store required
What happens when the AI gets something wrong? Fixing bugs in a vibe coded app

3 takeaways from the IO '26 developer keynote

Tilde A. Thurium — Tue, 19 May 2026 21:53:04 +0000

My top 3 takeaways from the IO 26’ developer keynote, in no particular order:

Gemini 3.5 Flash GA is a breakthrough for developers worldwide. In plain English, this model offers flagship-tier coding brains that run at lightning speeds for a fraction of the cost of frontier models. This is a massive win for international devs, as the ultra-low cost levels the playing field against steep USD exchange rates, and the raw speed easily offsets regional network latency. I tested it out myself by vibe coding a vaporwave-themed pet store app with Antigravity. I was impressed with how snappy the code generation felt.

Google AI Studio new features can make anyone a builder. One-click deployment to Firebase and Cloud Run makes it easier than ever to go from demo to serving real users. Custom asset generation with Nano Banana makes your vibe coded apps look sharp. And when you’re ready to add next-level complexity, now it’s easier than ever to export your project to Antigravity.
Antigravity 2.0 unlocks full-stack agentic development. You can now use always-on agents that build for you while you sleep, while the new Antigravity SDK lets you richly customize your agents and deploy them to private infrastructure.

For more details, check out the official announcement blog and stay tuned for upcoming Dev.To challenges where you can build with our new models. What are you most excited about?

How to use your network to get a job

Tilde A. Thurium — Mon, 13 Oct 2025 17:26:06 +0000

Common advice for job seekers is to "use your network" but what does that mean, exactly? Let's break it down.

In 2025, job referrals are everything

As a hiring manager, I got hundreds of applicants within 24 hours of posting a new job. Most of the applications were AI slop. Recruiters and managers are overwhelmed.

A personalized referral from someone you know will get you further with a hiring manager than the perfect resume.

Use your network to get referrals. By network, I mean people you already have a relationship with, who can vouch for your awesome work.

Going to meetups to chat up strangers who are probably also job seeking is likely not the highest-value use of your time.

How to ask for referrals

People want to help you. But first, you need to ask. To the LinkedIn mobile, Batman.

Identify jobs that are a good fit with what you're seeking.
Do some LinkedIn snooping to see if anyone you know works at those companies.
Write those people a quick message asking if they'd be comfortable referring you.

Isn't it weird to ask for a favor when you haven't seen someone in years?

In a professional context, it's totally okay to message an old co-worker you haven't talked to since 2019 and ask for a referral.

Example note to ask for a referral

Hi Tilde!

Long time no see, hope you're doing well.

I saw that LaunchDarkly is hiring for a Backend Engineer, Metrics. [INSERT LINK TO SPECIFIC POSTING]. I have 2 years of experience working on Go backends in my previous company. Would you mind referring me for this role? My resume is attached. Thanks so much!

What works about this letter:

It's brief
It specifies why you're a good fit for the role
It includes all the information needed to make the referral. Nobody has time for a scavenger hunt.

Caveats

Using your network is much harder for early-career engineers. There are many less junior positions available than there used to be. Also, juniors generally have smaller networks to begin with.

Norms vary geographically. I've seen this approach work in North America and European companies. Outside those zones, your mileage may vary.

Parting thoughts

The best time to grow your network is before you need it. If you are currently employed, take the time to celebrate your teammates' wins. Ask people how they are doing. Talk about the excellent work you are doing. Smash that connect button on LinkedIn. Small actions pay off over time in the long run.

The job market is incredibly tough, but people want to help. Go forth and ask.

P.S. LaunchDarkly is hiring.

Snowflake Cortex Completion API + LaunchDarkly SDK Integration

Tilde A. Thurium — Mon, 25 Aug 2025 18:18:23 +0000

This tutorial walks through an integration between the Snowflake Cortex Completion API and LaunchDarkly's AI SDKs. We'll be using a Snowflake Personal Access Token to query the Cortex API and receive completion responses.

Leveraging Snowflake's gateway approach to completions alongside LaunchDarkly's ability to make runtime changes to AI Configs, you can update the models, prompts, and parameters that you're using in the Snowflake endpoints in real-time.

This tutorial is presented in TypeScript, but since we're using Snowflake's REST API it's universal to any language in which you can access Snowflake. Snowflake additionally has a Python package that can be used to access their AI and ML functions.

Authenticate with Snowflake

If you are new to Snowflake, there is some setup you'll need to do to get an application running, like setting up a user that is able to access the API.

Head into your Snowflake instance and follow the guide provided by Snowflake for authenticating against the REST API, and the guide for authenticating against the Cortex REST API.

Pay attention to:

It's recommended to create a new user to access the API so that its permissions, privileges and access can be limited to the necessary scope
Make sure to grant the role of the user you're authenticating as a SNOWFLAKE.CORTEX_USER database role if it's not already present
If you are using a Personal Access Token, make sure to apply a Network Policy that allow-lists the IP you'll be accessing from

Admin Privileges Required for Network Policies

You need admin privileges to create Network Policies for Personal Access Token authentication. If you don't have admin access:

Create a fresh Snowflake trial account (where you'll have admin access)
Or contact your Snowflake administrator for help with authentication setup
Enterprise/work accounts typically don't grant these privileges to regular users
Capture your account identifier, which can be found by accessing the lower-left button on the UI that contains your name and account role.
- Click your name
- Hover over your active account
- In the popover menu, select "View account details"
- Copy the field labeled "Account/Server URL"

Set up an AI Config

Before we write any code, we'll go into LaunchDarkly and create an AI Config to be used in the integration.

Navigate to your LaunchDarkly instance and follow these steps:

Navigate to "AI Configs" and click "Create AI Config" on the top-right side

Give your Config a name, and then select "Cortex" from the provider dropdown.

Now that Cortex is selected, the model dropdown will be filtered to the available models. For this first variation, we'll select claude-3-5-sonnet

Make sure that the region you're accessing from has support for the model you select. You can view model availability on this page.

Add your messages for the completion. We'll add a single system message, as well as a template for where the user message will go:

{{variables}} signify a variable that will be replaced at the time you retrieve your config. This is how you provide dynamic content to your Configs such as contextual user information

Click "Review and save". You'll be given a chance to review your changes before committing them.

Your AI Config is now saved so it's time to serve our new variation to our users. Click the "Targeting" tab on the top of the AI Config:

By default your Config will be serving the disabled variation which is used to signal that a Config is turned off. We'll revisit this aspect later in the code.

Click "Edit" on the default rule and select your variation from the dropdown, click "Review and save", and then confirm the changes:

You've now targeted a variation which can be served in the SDK. We'll come back to this later once we've got some code set up. For now, just copy the key in your sidebar for later:

Set up the server

Next, we'll set up our sample application so that we can see LaunchDarkly AI Configs and the Snowflake REST API interacting in real-time. This section assumes some knowledge with TypeScript and the NodeJS ecosystem, but can be accomplished in any language with AI SDK support.

For the following sections, these are instructions to set it up as a new application. If you're not concerned about which piece does what or having a clean slate, you can also just clone this repository and run npm install and then npm run start after filling out the .env section.

Set up an ExpressJS application

Follow the ExpressJS installation guide to set up a new project leveraging Express.

Basic setup

Let's create some of the structure we'll need for the app:

mkdir app views
touch index.ts package.json views/index.html app/launchdarklyClient.ts app/completions.ts
cp .env.example .env

`.env`

The last command created a .env file that we'll use to register our secrets so they can be securely loaded by the application.

Within this file, fill out the following values:

SNOWFLAKE_ACCOUNT_IDENTIFIER=<Snowflake account identifier>.snowflakecomputing.com
SNOWFLAKE_PAT=<Snowflake Personal Access Token>

LAUNCHDARKLY_SDK_KEY=<LaunchDarkly SDK key>
LAUNCHDARKLY_AI_CONFIG_KEY=<LaunchDarkly AI Config key>
PORT=3000

The Snowflake account identifier and Personal Access Token should be available from following the authentication instructions for Snowflake.

If you do not know how to get your LaunchDarkly SDK key, you can follow this guide.

`package.json`

Grab the contents of the package.json file from the repository and replace your local package.json file.

Now run npm install to install our dependencies. Once that finishes, run typescript --init from the project folder to create a tsconfig.json file. You'll need the dependencies in here to process TypeScript files and run your local application.

The dependencies in this file do the following:

Add TypeScript support to ExpressJS (@types/express, typescript)
Add utilities to run the application (nodemon, ts-node, dotenv)
Initialize the LaunchDarkly SDKs (@launchdarkly/node-server-sdk, @launchdarkly/server-sdk-ai)

We are using default TypeScript settings. Feel free to edit these to match your project's needs.

`index.ts`

The index.ts file is responsible for initializing the application. We'll be including two routes; one to render an HTML page and one to respond to the completion.

Grab the index.ts file from the repo and replace this content. The file has comments explaining the functionality.

`index.html`

Replace the index.html file in views/index.html with the same content from the repository, or use it as a guideline to build your own interface for the chat. This file is also commented, but outside of the HTML structure, you'll want to pay attention to the <script> at the bottom of the page which handles making the API call.

Make our completion calls

Now that we have an application, we can start wiring up LaunchDarkly to the Snowflake API.

We'll set up the LaunchDarkly clients in app/launchdarklyClient.ts:

import { LDClient, init } from "@launchdarkly/node-server-sdk";
import { LDAIClient, initAi } from "@launchdarkly/server-sdk-ai";

let ldClient: LDClient;
function getSDKClient() {
    if (!ldClient) {
        ldClient = init(process.env.LAUNCHDARKLY_SDK_KEY!);
    }

    return ldClient;
}

let aiClient: LDAIClient;
function getAIClient() {
    if (!aiClient) {
        aiClient = initAi(getSDKClient());
    }
    return aiClient;
}

// Initialize and return the LaunchDarkly client
export async function getLaunchDarklyClients() {
    const ldClient = getSDKClient();
    const aiClient = getAIClient();

    try {
        await ldClient.waitForInitialization({ timeout: 10 });
    } catch (err) {
        // log your error.
    }

    return { ldClient, aiClient };
}

export async function closeLaunchDarklyClients() {
    await ldClient.flush();
    ldClient.close();
}

The LaunchDarkly AI SDK allows you to use the features of AI Configs within the LaunchDarkly SDK.

Within app/completions.ts let's go ahead and set up the Snowflake call:

import { getLaunchDarklyClients } from './launchdarklyClient';
// Base URL
const SNOWFLAKE_BASE_URL = `https://${process.env.SNOWFLAKE_ACCOUNT_IDENTIFIER}`
// Completion endpoint
const SNOWFLAKE_COMPLETE_URL = `${SNOWFLAKE_BASE_URL}/api/v2/cortex/inference:complete`

const snowflakeCompletionClient = async (body: Record<string, any>) => {
    const headers = {
        'Content-Type': 'application/json',
        // Includes the authorization token on the request
        'Authorization': `Bearer ${process.env.SNOWFLAKE_PAT}`,
        'Accept': 'application/json'
    }
    // Run a fetch on the Snowflake completion URL
    return fetch(SNOWFLAKE_COMPLETE_URL, {
        method: 'POST', 
        headers,
        // We are not going to stream our responses, so pass `stream:false` to all instances of this invocation
        body: JSON.stringify({...body, stream: false}),
    })
}

export async function runSnowflakeCompletion(userInput: string) {
    // Retrieve the AI Client from LaunchDarkly
    const { aiClient } = await getLaunchDarklyClients();
    // Set up the user's context; this can be used to control which variations the users receive. You can target against any attribute passed in this context.
    const userContext = {
        type: 'user',
        name: 'John Doe',
        key: `user-${Math.random().toString(36).substring(2, 15)}`
    }
    // Retrieve the AI Config from the LD SDK
    const config = await aiClient.config(
        // This is the key of our config
        process.env.LAUNCHDARKLY_AI_CONFIG_KEY!, 
        // Context
        userContext, 
        // Defaults - can be left empty unless you want to provide
        // default values if the AI Config is not found.
        {}, 
        // These variables are automatically interpolated into the 
        // messages returned from the SDK. Here, we're providing the
        // user's actual query, but this can be used for other runtime data augmentation purposes
        { userInput }
    )
    // Check that the config is enabled and conforms to the shape we would expect.
    // Some calls may omit things like the `messages` array when just changing models.
    if(!config.enabled || !config.model || !config.messages) {
        throw new Error("Malformed config")
    }
    // Make the call to the Snowflake API
    const run = await snowflakeCompletionClient({
        // The model name is provided dynamically, which means we
        // can change this at runtime with AI Configs!
        model: config.model.name,
        messages: config.messages,
    })

    const result = await run.json();
    // Extract the top choice of message and return it to the client
    const response = result.choices?.[0]?.message.content ?? "No response from Snowflake"

    return { response, model: config.model.name };
}

Run the app

Navigate to your root directory and run npm run start to start the application.

When you navigate to localhost:3000 (or whichever port you changed it to) you should see a simple screen that looks like this:

Enter a query into the textarea, such as "How do I create a user in Snowflake?" and after a few moments a response will be generated:

We can see in the response that it lists the model we're using to generate the completion.

Make runtime changes

Now that we have a completion endpoint set up, let's create a new variation and change the model at runtime. You can minimize your code editor for now; we'll only need to make changes in the LaunchDarkly UI!

Navigate back to your AI Config in the LaunchDarkly UI
Click "Add another variation" and then repeat the steps from earlier, but this time select a different model. We'll use llama3.1-8b and edit the system message slightly for tone:

Click save and head back over to targeting
On the targeting page, edit the default rule and select the new variation you created:

Click save and confirm the changes

Now, let's head back over to our app on the localhost URL. Without refreshing the page or restarting the server, go ahead and re-submit the request.

Our output was now generated by our llama3.1-8b model rather than the Sonnet model we were leveraging earlier:

And that's it! We've now made a runtime model change on an active AI Config.

We didn't have to change any code to change the model, and there's more we can tweak such as the model's parameters and messages. We can add messages and have them automatically inserted at runtime, test different models on different customers, and tweak parameters in a live system to see how the models respond in real-world conditions.

The last point we'll touch on is how we can optionally capture data about our AI Config invocations and send those over to LaunchDarkly.

Setting up monitoring

To set up monitoring, we need to extract the tracker object from our AI Config and call some track methods that will communicate the metrics to LaunchDarkly.

Let's update the app/completions.ts file to include tracking:

import { LDAIConfigTracker } from '@launchdarkly/server-sdk-ai';
import { getLaunchDarklyClients } from './launchdarklyClient';
// Base URL
const SNOWFLAKE_BASE_URL = `https://${process.env.SNOWFLAKE_ACCOUNT_IDENTIFIER}`
// Completion endpoint
const SNOWFLAKE_COMPLETE_URL = `${SNOWFLAKE_BASE_URL}/api/v2/cortex/inference:complete`

const snowflakeCompletionClient = async (body: Record<string, any>) => {
    const headers = {
        'Content-Type': 'application/json',
        // Includes the authorization token on the request
        'Authorization': `Bearer ${process.env.SNOWFLAKE_PAT}`,
        'Accept': 'application/json'
    }
    // Run a fetch on the Snowflake completion URL
    return fetch(SNOWFLAKE_COMPLETE_URL, {
        method: 'POST', 
        headers,
        // We are not going to stream our responses, so pass `stream:false` to all instances of this invocation
        body: JSON.stringify({...body, stream: false}),
    })
}

export async function runSnowflakeCompletion(userInput: string) {
    // Retrieve the AI Client from LaunchDarkly
    const { aiClient } = await getLaunchDarklyClients();
    // Set up the user's context; this can be used to control which variations the users receive. You can target against any attribute passed in this context.
    const userContext = {
        type: 'user',
        name: 'John Doe',
        key: `user-${Math.random().toString(36).substring(2, 15)}`
    }
    // Retrieve the AI Config from the LD SDK
    const config = await aiClient.config(
        // This is the key of our config
        process.env.LAUNCHDARKLY_AI_CONFIG_KEY!, 
        // Context
        userContext, 
        // Defaults - can be left empty unless you want to provide
        // default values if the AI Config is not found.
        {}, 
        // These variables are automatically interpolated into the 
        // messages returned from the SDK. Here, we're providing the
        // user's actual query, but this can be used for other runtime data augmentation purposes
        { userInput }
    )
    // Extract the tracker from the AI Config
    const { tracker } = config;
    // Check that the config is enabled and conforms to the shape we would expect.
    // Some calls may omit things like the `messages` array when just changing models.
    if(!config.enabled || !config.model || !config.messages) {
        // Track an error if the config is not enabled or does not conform to the shape we would expect.
        tracker.trackError();
        throw new Error("Malformed config")
    }
    try {
        const durationStart = Date.now();

        // Make the call to the Snowflake API
        const run = await snowflakeCompletionClient({
            // The model name is provided dynamically, which means we
            /// can change this at runtime with AI Configs!
            model: config.model.name,
            messages: config.messages,
        })

        const durationEnd = Date.now();

        const result = await run.json();
        // Track a successful completion
        tracker.trackSuccess();
        // Track the duration of the completion
        tracker.trackDuration(durationEnd - durationStart);
        // Track the tokens used in the completion
        if (result.usage) {
            tracker.trackTokens({
                total: result.usage.total_tokens,
                input: result.usage.prompt_tokens,
                output: result.usage.completion_tokens,
            });
        }

        // Extract the top choice of message and return it to the client
        const response = result.choices?.[0]?.message.content ?? "No response from Snowflake"

        return { response, model: config.model.name };
    } catch(error) {
        // An error occurred while making the completion call
        tracker.trackError();
        throw error;
    }

}

We now capture a duration by setting a timer before and after the call, and use the usage parameter returned from Snowflake to capture the token usage. These metrics will appear in your dashboard for your AI Config under the "Monitoring" tab:

You'll receive additional information on this page about when different variations released and when changes were made to the configs so that you can keep track of what changes have what impact on your completions.

Wrapping up

This is a simple example but demonstrates how you can use the power of Snowflake's Cortex completion gateway in conjunction with AI Configs. Together, they allow you to tweak models in real-time by selecting any models available in your region and having them update seamlessly without requiring any code changes.

Additionally, with the power of LaunchDarkly's targeting you can serve different models and prompts to different users, and even run experiments against your AI Configs to see which model and prompts best fit your features.

To get started with AI Configs, sign up for a free trial. You can also contact us at aiproduct@launchdarkly.com with any questions.

Originally published by Andrew Klatzke on docs.launchdarkly.com.

Using LaunchDarkly's AI Configs to review database changes

Tilde A. Thurium — Thu, 21 Aug 2025 19:43:01 +0000

Overview

At LaunchDarkly, we’re constantly pushing the boundaries of what it means to move fast without breaking things. We ship frequently, serve quadrillions of events per day, and operate with zero tolerance for downtime. To keep pace, we need systems that help us ship confidently—even when change is happening at a rapid, “vibe coding” pace.

But what happens when that rapid change reaches your database?

From an SRE’s perspective, the database is sacred. It’s the source of truth—and one of the riskiest areas to touch without deep context. Even if your code is reviewed in a pull request:

Does the reviewer understand the query access patterns?
Could this schema change hurt index performance?
Is the change touching critical production tables?
Will the new model scale with usage?

With LaunchDarkly AI Configs, we finally have a way to automate this kind of insight — reviewing database changes before they become production issues.

Prerequisites

To follow along, you'll need:

A LaunchDarkly account with AI Configs enabled. Sign up for free here..
Access to a database system. These code samples are written to be compatible with a database using the PostgreSQL wire protocol. However, you could adapt them to suit other flavors of databases.
Basic familiarity with SQL and database schema management
A development environment where you can test database changes

What are AI Configs?

LaunchDarkly AI Configs allow you to customize, test, and roll out new large language models (LLMs) within your generative AI applications.

🧠 What the AI Reviewer Checks

Our system uses LaunchDarkly’s internal AI Configs to analyze your schema and query changes directly from a CI build. It checks for:

Are the queries optimized for access patterns?
Are the right indexes in place?
Are we modifying high-risk tables?
Will the schema scale and evolve over time?

This isn’t just a linter. It’s an AI-powered reviewer trained on your environment.

If you want to skip right to reading the code, a complete example can be found here.

## 🛠 Step 1: Collect the Right Data

The AI needs a complete snapshot of your system to make a meaningful review:

Full schema (full-schema.json)
Schema diff (schema-diff.json)
Full set of SQL queries (sql-queries.json)
Query diff (queries-diff.json)

📦 sql-proxy container

To evaluate database changes, we need to observe real SQL queries your application runs during CI.

We do this by inserting a lightweight PostgreSQL proxy between your app and the database. It logs and deduplicates queries, then exposes them via an API for analysis.

Here’s the setup in Docker Compose / GitHub Action Service Container:

services:
  postgres:
    image: postgres
    env:
      POSTGRES_PASSWORD: postgres
    ports:
      - 5432:5432

  proxy:
    image: ghcr.io/${{ github.repository_owner }}/sql-proxy:latest
    env:
      LISTEN_PORT: 5433
      BACKEND_HOST: postgres
      BACKEND_PORT: 5432
      API_PORT: 8080
      DB_CONNECTION_STRING: host=postgres port=5432 user=postgres password=postgres sslmode=disable
    ports:
      - 5433:5433
      - 8080:8080

Every query is deduplicated and exposed via:
GET http://localhost:8080/queries

Connecting to the database via 5433 will now pass the queries through the proxy.

🏗 Dumping the Schema

To give the AI full context, we also need a snapshot of the database schema—including table definitions, columns, indexes, and relationships.

This can be triggered early in the CI pipeline to run in parallel with your other steps:

- name: Migrate Database
  run: |
    cd schema_test
    go run .

- name: Start Schema Dump
  run: |
    curl -X POST http://localhost:8080/schema_dump

⏱️ Note: For large schemas, this can take a minute or two. Triggering it early(but after the migrations) avoids blocking downstream jobs.

📊 Step 2: Compare Against Main

After your tests or migrations run through the proxy, it now holds:

The full set of SQL queries the app executed
The current state of the database schema

With this data captured, you can run a GitHub Action that:

Calls the proxy’s API to fetch the captured data
Downloads artifacts from the main branch (last known good state)
- Note: it needs to run on the main branch at least once to have generated a proper artifact for comparison
Compares current vs. main to generate:
- schema-diff.json
- queries-diff.json

Here’s what that looks like in CI:

- name: Get SQL Data
  uses: droptableifexists/recon@main
  id: get-sql-data
  with:
    SQL_PROXY_API_ADDRESS: localhost:8080
    GITHUB_REPOSITORY: ${{ github.repository }}
    GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

🤖 Step 3: Run the AI Review

After you have the four key files, you pass them into the AI Config system.

Save input to a file:

- name: Save SQL data to file
  run: |
    cat << EOF > analysis_input.json
    {
      "sql_queries": ${{  toJSON(steps.get-sql-data.outputs.sql-queries) }},
      "queries_diff": ${{  toJSON(steps.get-sql-data.outputs.queries-diff) }},
      "schema": ${{  toJSON(steps.get-sql-data.outputs.schema) }},
      "schema_diff": ${{  toJSON(steps.get-sql-data.outputs.schema-diff) }}
    }
    EOF

Run the DB analysis tool:

- name: Run Database Analysis
  uses: launchdarkly-labs/lddbai@main
  with:
    message: "Analyzing database changes..."
    input_file: analysis_input.json
    openai_api_key: ${{ secrets.OPENAI_API_KEY }}
    launchdarkly_sdk_key: ${{ secrets.LAUNCHDARKLY_SDK_KEY }}
    github_token: ${{ secrets.GITHUB_TOKEN }}
    pull_request_number: ${{ github.event.pull_request.number }}

Under the hood, here's what the code looks like:

def get_ai_config(payload: dict) -> tuple[AIConfig, LDAIConfigTracker]:
   aiclient = Deps().get_launchdarkly_ai()
   context = Context.builder('cockroachdb').kind('database').name('cockroachdb').build()
   fallback_value = AIConfig(
       enabled=True,
       model=ModelConfig(name="gpt-4o-mini", parameters={"temperature": 0.8}),
       messages=[LDMessage(role="system", content="")],
       provider=ProviderConfig(name="my-default-provider"),
   )
   return aiclient.config('evaluate-database-changes', context, fallback_value, {
       'schema': payload.get('schema', []),
       'schema_diff': payload.get('schema_diff', []),
       'sql_queries': payload.get('sql_queries', []),
       'queries_diff': payload.get('queries_diff', [])
   })

And to get the AI’s recommendation:

def get_ai_recommendation(payload: dict) -> str:
   config, tracker = get_ai_config(payload)
   messages = [] if config.messages is None else config.messages
   response = tracker.track_openai_metrics(
       lambda: client.chat.completions.create(
           model=config.model.name,
           messages=[message.to_dict() for message in messages],
       )
   )
   Deps().get_launchdarkly().flush()
   return response.choices[0].message.content

🧠 Step 4: Add Context That Only You Know

Once the model has your queries and schema, you can make it smarter by adding business context:

Database engine and version 🧱
Critical tables to tread carefully around 🚨
Average query volume and server specs 📈
Team-specific data modeling principles 📐
History of past incidents or patterns 📜

This context transforms the AI from a generic reviewer into a tailored risk advisor for your system.

🚀 Why This Matters

With AI Configs, you can:

Catch performance and scaling issues before they hit production
Share SRE intuition across your whole engineering team
Shorten feedback loops without blocking deploys
Scale database expertise without bottlenecks

You’re no longer at the mercy of “who reviewed the PR.” Every change gets a consistent, context-aware review.

🏁 Final Thoughts

Database changes don’t have to be scary anymore.

By plugging into LaunchDarkly AI Configs, you can automate reviews, enforce data modeling best practices, and de-risk your deploys—without slowing anyone down. To get started, sign up for a free trial today or email us at aiproduct@launchdarkly.com if you have questions.

So yeah, go ahead. Vibe out. Ship confidently. And let the AI handle the hard stuff.

Originally published by Kevin Kruger on the LaunchDarkly blog.