DEV Community: Michael Kaminski

Statistical Gating for Agent Instruction Changes

Michael Kaminski — Sun, 09 Aug 2026 19:16:23 +0000

An edit to an agent's instruction file is a deploy, and I stopped letting mine ship on a hunch.
The daily tuning job that maintains my assistant's CLAUDE.md now requires a statistically
significant improvement — Welch's t-test, p < 0.10, at least a 5% lift — measured against a
14-day rolling baseline, before an instruction change is allowed to stay.

Then I ran the power calculation on my own gate and found the 5% threshold is decorative. At
14 days per window the test can only detect a shift of roughly 0.97 standard deviations.
Anything smaller is invisible regardless of what the lift threshold claims.

Both halves of that are this post. The mechanism is worth copying. The window size is the part
I got wrong, and the arithmetic is short enough to check.

The prompt is the least-tested code in most agent systems

Every other artifact in an agent pipeline has a gate. Application code gets a test suite and a
review. Infrastructure gets a plan and a diff — I have written enough Terraform modules and
managed enough remote state to know nobody merges those blind.

The instruction file gets none of that. Someone notices the agent did something annoying, adds
a line telling it not to, and ships. There is no baseline, no holdout, and no record of whether
the previous eleven lines are still earning their tokens.

That is how instruction files rot. They accumulate rules that were true about one bad afternoon
and have been costing context ever since.

You cannot test what you do not score

The gate needs a dependent variable, so the first build was a rubric, not a test. Five
dimensions, each scored 0–10 per conversation: goal clarity, rework rate, context hit rate,
scope discipline, and response density. The daily score is the average across that day's
conversations, weighted by conversation length.

The rubric is frozen. Changing it invalidates every historical comparison, so a rubric change is
itself a tracked meta-edit that resets all baselines.

Scores are anchored to observable friction rather than self-assessment. The scorer parses
transcripts for five pattern classes — corrections ("no, that", "actually", "don't"), rework
("redo", "start over"), scope drift ("I didn't ask", "just do"), missed context ("I told you
before", "check memory"), and praise ("exactly", "nailed it"). Friction pulls the dimension
score down, praise pulls it up.

This is the load-bearing decision. An agent grading its own transcripts will drift toward
generosity. Regex over the human's actual words will not.

Welch, not Student, and why that is not pedantry

Each edit gets a 14-day pre-window and a 14-day post-window of daily scores, compared with
Welch's t-test. Three outcomes: p < 0.10 and lift ≥ 5% marks the edit KEPT; p < 0.10 and
lift ≤ −5% triggers an automatic revert; everything else is INCONCLUSIVE and the edit
stays on probation.

Welch rather than Student's t because the two windows should not have equal variance. A good
instruction usually works by removing a failure mode, which compresses the bad tail — the
variance drops as much as the mean rises.

Student's t assumes equal variance and over-rejects when the smaller-variance group is the
larger sample. That is precisely the case you care about, so the pooled test would hand you your
most confident false positives on your best edits. Welch costs a few degrees of freedom and
removes the assumption.

The arithmetic that broke the design

A two-sample test at α = 0.10 and 80% power detects a minimum effect of roughly
(t_α/2 + t_β) × √(2/n) standard deviations. For a 14-day window that is:

Days per window	Minimum detectable effect
7	1.42 SD
14	0.97 SD
21	0.78 SD
28	0.67 SD
42	0.55 SD
60	0.46 SD

The gate can only see effects above the yellow line. The 5% threshold sits inside the shaded band — under it the whole time, doing no work.

Now convert to the units the gate actually uses. Assume a 7.0 baseline on the 0–10 rubric and a
day-to-day standard deviation of 0.6 points, which is unremarkable for a metric averaged over a
handful of conversations.

A 5% lift is 0.35 points, or 0.58 SD. The 14-day window detects 0.97 SD, which is 0.58 points —
8.3% of baseline. The statistical test is roughly 1.7× stricter than the lift threshold
sitting next to it.

So the 5% number never binds. It is doing no work. Every edit that clears the t-test has already
cleared 5% by a wide margin, and every edit that fails does so on power, not on effect size.

Detecting a genuine 5% lift at that variance takes 38 days per window, not 14. And the
requirement moves fast with variance:

Daily score SD	5% lift, in SD	Days/window needed	What 14 days actually detects
0.4	0.88	17	5.5%
0.5	0.70	27	6.9%
0.6	0.58	38	8.3%
0.8	0.44	66	11.1%
1.0	0.35	102	13.8%

The honest label on the gate. Every bar right of the dashed line is a threshold the config never mentions.

Read the last column as the honest label on the gate. At SD 1.0 a "5% threshold" is really a 14%
threshold, and the difference is entirely hidden from whoever reads the config file.

The window has to be sized to the variance, not to a calendar. Fourteen days was chosen because
two weeks is a tidy number, which is not a reason.

There are two ways out and only one of them is cheap. Lengthen the window, and you wait longer
per lesson. Or shrink the variance by scoring more conversations per day — the daily mean's
standard error falls with √n, so tripling daily volume cuts the SD by about 42% and pulls the
required window from 38 days down to roughly 17.

Put the same arithmetic the other way around. If an edit truly delivers a 5% lift, a 14-day
window calls it significant 44% of the time. That is a coin flip on your own best changes.
At 38 days it is 81%.

Same arithmetic, run forward. Each frame is a window length; the shaded tail is how often a
genuinely good edit gets called significant.

What the gate cannot do, stated plainly

It does not control for multiple comparisons. At the configured cadence of one low-risk edit
every three days, that is about 122 evaluations a year. At α = 0.10 two-sided, noise alone
produces roughly six spurious KEPT verdicts and six spurious auto-reverts per year. The
auto-revert side is the one that stings: the system will occasionally roll back a good edit with
statistical confidence.

It is also not a randomized experiment. Pre and post windows are consecutive calendar time, so a
model version change, a vacation, or a month of unusually messy work lands entirely in one window
and gets attributed to the edit.

That makes this a noise filter, not a causal claim. It stops the obviously-worse edits and the
obviously-imaginary wins. It will not tell you why anything moved.

The safeguards do more work than the test

Three rules keep the loop from eating itself, and they matter more than the p-value.

Cooldowns. One low-risk auto-edit per three days, one high-risk proposal per week. Without a
cooldown the windows overlap so badly that no edit is ever cleanly attributable.

Plateau detection. If the 14-day rolling score has not improved 2% over the prior 14 days and
no friction is firing, the day's edit is skipped. The easy wins arrive early; after that, editing
is mostly a way to add variance.

A risk split with a human in it. Wording and formatting changes auto-apply. Anything that
adds a section or changes how the agent decides gets written to a proposals folder and waits.
Every edit backs up the prior file first and lands in a revertible log, tagged, so a bad call is
a one-line rollback rather than an archaeology project.

The config file has a section titled "when NOT to edit," and the last line of it is the most
useful thing in the whole system: doing nothing is always a valid action.

The cost

Latency. That is the whole bill, and it is larger than I estimated when I built this.

A gated instruction file learns on a 14-day clock at best and, if the variance numbers above are
right, a 38-day clock in practice. An ungated one learns in an afternoon and is wrong in ways
nobody measures. I would still take the slow version, but I would not pretend the tradeoff is
free, and I would not build this at all for a system I was still prototyping.

I have no KEPT verdicts to report yet, because the first honest one cannot exist until a full
post-window closes. Publishing the design before the results is the point — the design is the
part that is checkable, and the arithmetic above is the part I would want someone to argue with.

If you are gating prompt or instruction changes statistically at your shop, I want to know what
window you landed on and what your daily variance looks like. That number is the whole ballgame
and almost nobody publishes it.

Originally published at michael-kaminski.io. I write field notes on agent infrastructure — evals, MCP servers, and what it costs to run agents in production.

🚀 Boosting Our Project with Strategic Unit Testing: A Closer Look at Our Latest Code 🛠️

Michael Kaminski — Sun, 14 Jan 2024 19:20:58 +0000

Introduction

Hello Devs! 👋 We've just rolled out an exciting update in our project, integrating a series of unit tests that are set to elevate our application's reliability and maintainability to new heights. Let’s dive deep into this new code block and explore its impact on our SDLC. Check out our repo here for the full codebase!

The New Code Block
Here's a snippet from our latest addition:

import unittest
from unittest.mock import patch, MagicMock, Mock
from PIL import Image
from io import BytesIO
import main

This segment sets the stage for our unit testing, importing essential modules and our main module where our business logic resides.

The Tests 🔍

Testing User Input Prompt

@patch('main.input', return_value='test')
def test_prompt_user_for_theme(self, input):
    self.assertEqual(main.prompt_user_for_theme(), 'test')

👉 This test ensures that our prompt_user_for_theme function correctly captures and returns user input.

Fetching Images from Unsplash

def test_fetch_images_from_unsplash(self):
    # Mock setup...
    with patch('main.requests.get', return_value=mock_response) as mock_get:
        images = main.fetch_images_from_unsplash('test', {'match_aspect_ratio': False})
        self.assertEqual(len(images), 1)

👉 By mocking a response, this test verifies if the correct number of images is fetched from Unsplash.

Image Conversion to Grayscale

def test_convert_to_grayscale_and_contrast(self):
    img = Image.new('RGB', (64, 64))
    converted_img = main.convert_to_grayscale_and_contrast(img)
    self.assertEqual(converted_img.mode, 'L')

👉 This snippet checks if our image conversion functionality is up to the mark.

Saving Icons to Directory

@patch('main.os.path.exists', return_value=False)
@patch('main.os.makedirs')
@patch('PIL.Image.new')
def test_save_icons_to_directory(self, mock_new, mock_makedirs, mock_exists):
    # Test implementation...

👉 It tests the creation of directories and saving of icons, ensuring our file system interactions are flawless.

The Value of These Tests 💡

Reliability Boost: Ensures each component works correctly in isolation.
Speedy Debugging: Pinpoints issues, cutting down on troubleshooting time.
Encourages Better Design: Leads to a more modular and maintainable codebase.
CI Integration: Ready for automated testing in CI pipelines.

Impact on SDLC 🔄

Planning & Analysis
1.: Our tests provide clarity on functionality, aiding in precise planning.

Design: They encourage a design that's modular and easy to test.

Implementation: Immediate feedback on code changes reduces bugs.
Testing & Integration: Continuous testing elevates quality and integration.
Maintenance: They make updates and extensions safer and easier.

Conclusion

🌟 The addition of these unit tests marks a significant leap in our quest for a robust and maintainable application. By ensuring each component's functionality, we not only prevent bugs but foster a development culture centered around quality and efficiency.

🔗 Explore More: Dive into our repository here for the full code and more insights!

📢 Stay tuned for more updates and let's code towards excellence together! 🚀💻

Introducing Unsplash Image Fetcher: A Practical Approach to Image Processing

Michael Kaminski — Sun, 14 Jan 2024 18:48:28 +0000

Overview

Welcome to the first iteration of the Unsplash Image Fetcher, a Python script tailored for developers, designers, and content creators. This script efficiently sources and processes images based on your input theme, simplifying a task that often consumes valuable time.

Version 1.0 Features

Theme-Based Image Fetching: Select a theme and the script fetches corresponding images from Unsplash.
Resizing: Automatically resizes images to 64x64 pixels.
Aspect Ratio Selection: Option to fetch images with equal aspect ratios.
Grayscale and Contrast Enhancement: Converts images into high-contrast grayscale versions.
Organized Saving: Saves original and transformed images in separate directories for easy access.

Code Snippet: Fetching and Resizing Images

def fetch_images_from_unsplash(theme, config):
    """
    Fetches images from Unsplash based on the provided theme and configuration.
    If 'match_aspect_ratio' in the configuration is True, only fetches images that have an equal aspect ratio.
    Resizes the fetched images to 64x64 pixels and returns them as a list of PIL Image objects.
    """
    print(f"Fetching images for theme: {theme}")
    response = requests.get(f'https://api.unsplash.com/search/photos?query={theme}&client_id={UNSPLASH_ACCESS_KEY}')

    if response.status_code == 200:
        data = response.json()
        images = []
        for i, result in enumerate(data['results']):
            img_url = result['urls']['small']
            img_response = requests.get(img_url)
            img = Image.open(BytesIO(img_response.content))

            # Check if the image has an equal aspect ratio
            if not config['match_aspect_ratio'] or img.width == img.height:
                # Resize the image to 64x64 pixels
                img = img.resize((64, 64))
                images.append(img)
                print(f"Image {i+1} fetched and resized")
            else:
                print(f"Image {i+1} skipped due to unequal aspect ratio")

        print(f"Finished fetching images for theme: {theme}")
        print(f"Number of images retrieved: {len(images)}")
        if len(images) == 0:
            print(colored('No images were retrieved.', 'red'))
        print('-'*50)
        return images
    else:
        print(f"Failed to get images for theme '{theme}' from Unsplash API. Status code: {response.status_code}")
        return []

Current Limitations and Upcoming Version 2.0

While effective, version 1.0 has its limits. It's a standalone script without a user interface, making it more suitable for those comfortable with command-line tools. Recognizing this, version 2.0 aims to include:

React App Integration: Launching a user-friendly front-end.
Enhanced Customization: More configuration options for image processing.
User Feedback Integration: Implementing features based on user suggestions.

Ideation for Version 2.0 Features

Batch Processing: Allow multiple themes in one go for bulk image handling.
Image Filtering Options: Introduce filters like brightness, contrast, and saturation adjustments.
Download Quality Selection: Choose the resolution for downloaded images.
Interactive UI: A simple and intuitive interface for non-programmers.
Social Media Integration: Directly upload edited images to social platforms.

Code Example: Grayscale Conversion

def convert_to_grayscale_and_contrast(image):
    """
    Converts the provided image to grayscale and applies high contrast.
    Returns the converted image.
    """
    grayscale_image = ImageOps.grayscale(image)
    high_contrast_image = ImageOps.autocontrast(grayscale_image)
    return high_contrast_image

Repository and Collaboration

The script is available on GitHub, inviting collaboration and improvements. You're encouraged to contribute ideas, report bugs, or suggest enhancements. Check out the repo here.

Conclusion

Version 1.0 of the Unsplash Image Fetcher marks the beginning of a journey towards more streamlined image processing. It's a functional tool, but with your feedback and contributions, version 2.

Embark on a Journey Towards Your Goals with VisionQuestFrameworks Pt 1 of X

Michael Kaminski — Mon, 18 Dec 2023 06:08:03 +0000

TLDR;
Repo Quick-Access
Primary App: VisionQuest
Side-Project: data_analysis
Extension: Python Resource Monitor

Hey there, tech enthusiasts and dream chasers! Welcome to the inaugural chapter of our weekly series on VisionQuest, a project that's about to transform the way you chase your dreams and goals. In this journey, we'll dive into the exciting world of VisionQuest, from its grand vision to the progress we've made so far.

The Grand Vision: Your Personal AI Coach 🤖
At the core of VisionQuest lies a bold aspiration: to create a Personal AI Coach, omni-scient, and ever present - basically clippy back from windows 95. Ok, but we make some enhancements – an AI mentor that not only comprehends your goals but also tracks your journey and offers timely guidance to ensure your success. Whether you're striving for personal growth, professional achievements, or a healthier lifestyle, VisionQuest is your trusted companion on this path to greatness.

In my case the goal of VisionQuest is to keep me guided towards my goals, with auto-coaching.

Progress Report 🚀

Ideation Phase

Here's what we're initially setting out to accomplish:

Goal Setting: Our journey began with the simple act of setting and tracking goals. VisionQuest empowers you to remain laser-focused on what matters most to you.

Customizable Coaching Intervals: We introduced coaching intervals of 2 minutes, 5 minutes, 10 minutes, or 20 minutes, letting you tailor the guidance to your pace.

AI-Powered Coaching: Our AI algorithms are in a constant state of evolution, delivering personalized coaching that aligns with your goals and progress.

The actual progress we make week-by-week will vary depending on anything we get hung-up on. This is an educational journey, or perhaps a case-study in how VisionQuest may be able to keep a wandering mind on track.

Diving into the VisionQuest Codebase

The codebase of VisionQuest, a PyQt5-based application that interacts with the OpenAI API to generate responses to user prompts. You can check out the full code on Github, but I'll be sharing some juicy snippets right here.

The Heart of the App: MainWindow
At the heart of the application is the MainWindow class. This is where the magic happens. It's responsible for the user interface and the main functionality of the application.

class MainWindow(QMainWindow):
    def __init__(self):
        super().__init__()
        self.setWindowTitle("VisionQuest")
        self.userName = self.getUserName()
        self.initUI()

Setting the Stage: initUI
The initUI method is where we set up the stage for our application. We're talking window flags, background, position, and transparency. We also create the chat log and send prompt sections here.

def initUI(self):
    self.setWindowFlags(Qt.FramelessWindowHint)
    self.setWindowOnTop()
    self.setGradientBackground()
    self.moveWindowToTopRight()
    self.setTransparency()

The Pulse: Frequency and Countdown
The createMenuBar method is where we create a menu bar with frequency options. This frequency determines how often a prompt is sent to the OpenAI API. We also have a countdown timer that ticks down to the next prompt.

def createMenuBar(self):
    menu_bar = self.menuBar()
    self.createFrequencyMenu(menu_bar)
    self.frequency = 5  # Default frequency is 5 seconds

The Conversation Starter: sendPrompt
The sendPrompt method is where we send prompts to the OpenAI API and receive responses. We select a random prompt from a list, send it to the API, log the response, and update the chat log.

def sendPrompt(self):
    prompt = random.choice(prompts)
    response = oapi.get_openai_response(prompt)
    self.log_to_db("OpenAI_Model", response, "receiver")
    self.updateChatLog()

The Memory: Chat Log
The createChatLog and updateChatLog methods are where we create and update the chat log. The chat log is stored in a SQLite database, because who doesn't love a good SQLite database?

def createChatLog(self):
    conn = sqlite3.connect('chat_log.db')
    c = conn.cursor()
    c.execute('''CREATE TABLE IF NOT EXISTS chat_log
                (id INTEGER PRIMARY KEY AUTOINCREMENT, time TEXT, type TEXT, username TEXT, message TEXT)''')

Interacting with OpenAI's GPT-3.5 Turbo Model

In this section, we will look at a Python script that uses OpenAI's GPT-3.5 Turbo model to generate responses to user questions. This can be useful in a variety of applications, such as chatbots, virtual assistants, and more.

Code Explanation

from openai import OpenAI

vision_quest_api = 'OpenAI-Key'
client = OpenAI(api_key=vision_quest_api)

We start by importing the OpenAI module and initializing a client with our OpenAI API key.

def get_openai_response(user_question):
    completion = client.chat.completions.create(
        model="gpt-3.5-turbo",
        messages=[
            {"role": "system", "content": "You are a productivity enhancer, evaluating what people are doing at fixed intervals and recommending specific paths of action to help them reach their ultimate goal."},
            {"role": "user", "content": user_question}
        ]
    )

    return completion.choices[0].message.content

The get_openai_response function takes a user question as an argument. It sends a chat completion request to the GPT-3.5 Turbo model with two messages: a system message that sets the behavior of the assistant, and a user message that contains the user's question.

The system message tells the model that it is a "productivity enhancer" that recommends actions to help people reach their goals. This sets the context for the conversation and guides the model's responses.

The user message is the question that we want the model to answer.

The function then returns the content of the model's response. This is the text that the model generated in response to the user's question.

Convert Images to Icons with PyQt5 and PIL

We wanted to ensure we had a neat looking logo that matched our theme, we prompted DALE for an image - which was saved as .jpeg, then required conversion to .ico - we decided to create a utility.

In this section, we will explore a simple Python script that uses PyQt5 and PIL (Python Imaging Library) to convert images to icons. This can be useful in various scenarios, such as when you're developing a desktop application and need to create an icon for it.

Code Explanation

from PyQt5.QtWidgets import QApplication, QFileDialog
from PIL import Image
import sys

We start by importing the necessary modules. QApplication and QFileDialog from PyQt5 are used to create a simple GUI for selecting the image file. Image from PIL is used to handle the image processing.

def convert_to_icon(filename):
    img = Image.open(filename)
    img = img.resize((32, 32))  # resize image to 32x32 pixels
    ico_filename = filename.rsplit('.', 1)[0] + '.ico'
    img.save(ico_filename, format='ICO', sizes=[(32,32)])

The convert_to_icon function takes a filename as an argument. It opens the image, resizes it to 32x32 pixels (the standard size for icons), and saves it in ICO format. The new icon file will have the same name as the original image file, but with the .ico extension.

if __name__ == "__main__":
    app = QApplication([])
    filename, _ = QFileDialog.getOpenFileName()
    if filename:
        convert_to_icon(filename)
    sys.exit(app.exec_())

In the main part of the script, we create a QApplication instance and use QFileDialog.getOpenFileName() to open a file dialog for the user to select an image file. If a file is selected (i.e., if filename is not an empty string), we call convert_to_icon to convert the image to an icon. Finally, we call sys.exit(app.exec_()) to ensure a clean exit when the application is closed.

This script provides a simple way to convert images to icons using PyQt5 and PIL. It can be easily extended or integrated into other projects.

Round-Up & Kickers

A Nod to Efficiency
Here's the kicker: I've integrated a resource monitor to keep an eye on memory usage. Why, you ask? Well, I'm also working on this cool data analysis project (check it out) and being resource-efficient is my jam!

Name: Python Resource Monitor

The Look and Feel
Right now, VisionQuest rocks a transparent blue UI. It's like looking through a futuristic window where you can chat with AI. Pretty neat, huh?

What’s Next? Oh, The Plans I Have!
I’m planning to split the code into neat little compartments (because organization is key), separate the UI from business logic (they need their own space), and tuck away those API keys in a safe config file.

And here’s a list of touch-ups and future fireworks I'm planning:

1) Get that countdown number to shine on its own in the toolbar – no more pyqt5.qtcore.qttimer gatecrashing the party.
2) Add some cool transparency to all text boxes – it's all about that sleek look.
3) Dress up the 'Send Prompt' button in a dapper dark blue.
4) Fix the time calculations – let's keep it simple, no more math gymnastics.
5) Modify frequency options to seconds in all locations
6) On startup, VisionQuest will ask for your goals – a chat app that cares about your ambitions!
7) And the big one: a Lucid chart mapping out multi-goals, progress measurement, and comparing your forecasted performance with actual progress.

It's going to be like a control center for your aspirations!

Wrapping Up
So, that's the scoop on VisionQuest. It's more than a chat app; it's my personal lab experiment where I get to mesh code, design, and AI chats into something unique.

I'd love to hear your thoughts, suggestions, or any cool ideas you might have. Let's make coding not just about typing away in solitude but about sharing, learning, and growing together!