DEV Community: Geoff Cox

I stopped re-explaining myself to AI

Geoff Cox — Tue, 03 Nov 2026 00:20:00 +0000

Image credit: author, drawn with code

For a while, every new AI session started with me re-explaining things: who I am, what I’m working on and which documents matter, often by uploading the same PDFs again. Now any AI I use starts with all of that.

This is a story about how I moved my notes out of Google Drive and into a folder of plain text files that every AI I use can read and update, and about the handful of rules that keep that folder from turning into a mess. I call it my memex.

If you read my last post, you’ve already met it. The bot that reads our kids’ school email keeps its rules there. This post is about the memex itself: what’s in it, how it stays accurate when several AIs are writing to it, and how you could start one.

What Was Wrong With Google Drive?

In short, AIs kept tripping over my documents. When I pointed Claude at my Google Drive and asked it to edit a Google Doc, it would create a new copy instead of changing the one I had. Claude has its own documents that it can edit, but other AIs can’t easily read them. And most AIs couldn’t search my Drive well enough to find things, so I often ended up exporting documents to PDF and uploading them by hand.

I wanted one place that any AI could read, search and change, without me copying things around. A note-taking app wasn’t the answer either, since you’re limited to the structure and features the app gives you.

What Is the Memex?

The name comes from Vannevar Bush’s 1945 essay “As We May Think”, where a memex is a desk that stores a person’s books, records and communications so they can be consulted quickly. Mine is a git repo of markdown files. Markdown is plain text with a little formatting, e.g. a # in front of a heading. A git repo is a folder that records every change, so you can see what changed, when and why, and undo any of it. Mine lives on my laptop and in a private repo on GitHub.

The files are organized by what they’re about:

knowledge/ holds the notes: my career history, projects, reference material, the household (school schedules, the cars, recipes) and a daily log.
context/ holds a few files that tell an AI who I am and how I like to work, e.g. “ask me instead of guessing” and “never commit directly to main”.
bots/ holds a profile for each bot, which is what my last post was about.
scripts/ and skills/ hold the small programs that check and maintain everything, with their tests.

I created the repo on March 1, 2026 from a template and then mostly forgot about it. The real work started on October 13, 2026. As of October 31, it has 131 notes and about 157,000 words.

Words across all notes at the end of each day. Most of the jump came in the first two days.

How Did My Drive Get Into It?

On October 13, I downloaded my Google Drive to my laptop and had Claude read it. It extracted the text from about 780 files: PDFs, Word documents, slide decks, spreadsheets and plain text. Google Docs were the one snag. They sync to a laptop as pointer files with no text in them, so I downloaded those folders again as Word and Excel files. Claude then read the converted files too and summarized everything it had read into notes, mostly about my career and past projects, rather than copying the documents in.

Some things I didn’t want in the memex at all, so before anything was summarized we agreed on what to leave out: tax and financial records, anything that looked like it held a password, medical and identity documents, and the folders about our kids. The memex keeps a ledger of the import that lists what was read, what was skipped and why. That way a later session doesn’t redo the work, or worse, import something I had already decided to exclude.

The import surfaced something I hadn’t expected. When you read 25 years of someone’s documents side by side, they disagree with each other. For example, one of my old self-reviews said our hotfixes had dropped from 8 a week to 1.5, while my other records said 7. Rather than pick one, Claude listed every conflict like that and asked me. I answered them in batches that day, and each answer became an entry in a decisions log.

One conflict from the import, from my documents to a rule the checker enforces.

How Does It Stay Accurate?

A pile of notes that several AIs write to will drift. One session restates a fact a little differently, another copies an old number from a document, and a month later nobody knows which version is right. So the memex has a few rules, and a program checks most of them on every change.

An example header. The index is built from these lines.

Every note starts with a short header: when it was created and last updated, some tags, whether it’s active or a draft, a one-line summary and how private it is. Private notes never leave the memex. Usable notes can inform things I write, like this post. Public notes are already public.
A script builds an index from those headers, one line per note. An AI reads the index first and then opens only the notes it needs. That matters because an AI can only hold so much text at once (its “context window”), and 157,000 words is far more than you’d want it to load for every question.
The decisions log records each of my rulings with its date, 42 of them so far. Old decisions are never edited. A new one says which one it replaces.
Some decisions are backed by a rule in the checker, so a corrected fact can’t creep back in. If a note says hotfixes went from 8 to 1.5 a week, the check fails and points at the decision.
The checker, or linter, also makes sure every note has its header, every link points to a real note, file names follow one pattern, nothing looks like a password or an access key, and the index is current. It runs on every change, along with 196 automated tests for the scripts.
Every change arrives as a pull request, which is a proposed change I can read before it’s accepted. Nothing is committed straight to the main copy. For the memex, I let the AI merge its own pull request once the checks pass.
Nothing is deleted. Notes that are out of date move to an archive folder.

None of this is clever. The rules are simple enough that any AI can follow them, and the checker catches it when one doesn’t.

What Is the Dream?

The checker can tell you a link is broken, but it can’t tell you that two notes say the same thing in different words, or that a plan from three weeks ago is stale. For that, there is a cleanup pass called a “dream”, loosely after the way sleep is thought to consolidate memories.

I didn’t come up with the idea. Anthropic addeddreaming to Claude Managed Agents in May 2026: a scheduled process that reviews an agent’s past sessions and memory, pulls out patterns and tidies the memory, either on its own or with you reviewing the changes first. Their Code with Claude talk,“Memory and dreaming for self-learning agents”, covers how they got there. OpenClaw has its own version that runs in light, REM and deep phases and writes a diary of what it changed, and Hermes Agent has a community plugin modeled on OpenClaw’s. Mine borrows from all of them. The difference is that my memory is a git repo, so a dream’s changes arrive as a pull request I can read.

A dream has two halves. The mechanical half runs the checker and rebuilds the index. The reflective half is an AI working through a written checklist: promote anything durable from the daily log into the right note, find facts stated in more than one place and keep one owner for each, flag notes that have gone stale, and write down the questions only I can answer. It lands as a pull request with a short report.

What the AI is allowed to do on its own depends on the change:

Mechanical fixes that don’t change meaning, like a broken link or a stale index, it just does.
Moves, merges and new notes, it makes on the branch and calls out in the report, so I can accept or revert them.
Anything that changes what a fact means, which version of something is right, or a privacy rule, it asks me about first and doesn’t touch until I answer.

What a dream may do on its own, what it proposes and what it asks about.

There have been four dreams so far: October 13, October 15, October 25 and November 1. The last two were run by a bot on its own weekly schedule. On October 25, lint was already clean, so the main results were notes for the two people mentioned most often who didn’t have one yet (notes on people are private and stay in the memex), and one place where the repo’s instructions contradicted a newer decision. The November 1 dream was lighter: one lint warning fixed, the bot’s own log brought up to date and no questions for me.

Who Writes to It?

More than one AI, which was the whole idea. From October 13 to 31, the memex had 190 commits and 122 merged pull requests. Most came from Claude Code sessions on my laptop. Others came from Grok Bot, the agent I use to run bots on a cloud computer, and from the bots themselves, e.g. the school bot updating its profile and a memex bot that does the weekly upkeep.

Two things fall out of that. First, I’m not tied to any one AI. If I switch tools tomorrow, the notes, the rules and the history come with me, because they’re just files. Second, every session starts with context. I don’t paste my background into a prompt or explain how something works, because the AI reads the index and finds it.

In practice, I start most things with that same memex bot. When I want to add to something or work on it, like a new project idea, a blog post I should write or a book I should read, I ask the memex bot to add the details. It already has the context of many things in my life, both current and old. And when I’m working in another AI and I think my memex would help, I give that AI read and write access to the memex’s GitHub repo, and it has the same context right away. A bot gets its own access token that works only on that one repo, can only read and write files and open pull requests, and expires within 90 days.

Pull requests turned out to be useful for more than catching an AI’s mistakes. When I want feedback on a document, I open it as a pull request and leave comments on specific lines in GitHub, the way you’d review code. The AI answers each comment and pushes a new version. This post was reviewed that way.

Is It Safe?

I think it’s relatively safe, as long as you don’t have a lot of automations that send out or post content unattended. The only bot of mine that emails anyone sends to one family list, from its own account; the others open pull requests or message me in a private Slack channel. That said, there are risks, and you certainly don’t want to store anything sensitive in your memex, e.g. credit card numbers. The checker flags anything that looks like a password or an access key, but it can’t catch everything, so the better habit is to leave those things out in the first place.

What Would I Do Differently From Day One?

Not much, and I wouldn’t try to design it all up front. Everyone’s preferences differ, and mine weren’t clear until I’d used it for a while. A memex changes shape as you use it, and the goal is that over time it holds all of your context and preferences, written down.

That said, three things would have saved me some cleanup:

Start the decisions log on day one, not after the import.
Decide what stays out, e.g. financial and medical records, before an AI reads anything.
Add the index before the notes pass a few dozen.

How Could You Start One?

You don’t need my scripts to get most of the benefit. A folder of markdown files in a git repo, a short header on every note and an AI that can read files will get you a long way. I’d add the rest in roughly the order it paid off for me:

A context file that says who you are and how you like to work.
The header on every note, and an index built from the headers.
A decisions log, as soon as you catch yourself correcting the same fact twice.
A checker that runs on every change.
Changes by pull request, so you see what an AI did before it sticks.
A cleanup pass once the notes outgrow what you can review by hand.

If you’d like a head start, I’ve put the memex’s structure and tooling on GitHub as a template, with none of my notes in it:memex-template. It has the folders, the note header, the index builder, the checker, a decisions log with an example rule, and the dream protocol with its /dream command. The bot template from my last post lives there too, since the bots deliver into the memex.

Thanks for reading.

I built an AI bot to read our kids' school email. Here's what I'd tell you before you build one.

Geoff Cox — Mon, 26 Oct 2026 22:33:00 +0000

Image credit: author, drawn with code

This is a story about how my wife and I went from several school emails and texts a day to one short digest every afternoon, and about what it took to trust an AI bot to write it. Getting the bot to summarize was the easy part. Most of the work went into deciding what it is not allowed to do, and into catching the times it got something wrong without saying so.

If you read my first post, this is the same idea in a very different setting. There, automated tests were the gate that let AI agents write code for a system that trades real money. Here, the gate is a short list of things the bot can never do, plus one place where every rule it follows is written down.

Why Build a Bot for School Email?

We have two kids at two different schools, a 5th grader and a 2nd grader. Between the district, both schools, the teachers, the parent groups, the after-school club and the flyers, the messages never really stop. We were getting multiple emails and texts a day, and I figured a bot would be a good way to consolidate the noise.

I counted afterwards. In the six weeks from September 9 to October 21, 2026, roughly 200 emails came in that were actually about school, or a bit over 30 a week. Over the same weeks, the kids’ sports team app sent another 120. Some of the school mail is duplicated, too, e.g. the after-school club often sends the same notice once for each kid.

On top of that, the schools use TalkingPoints, an app for school announcements and teacher messages, and those never reach email at all. In one sample week it had about 13 announcements and 3 active conversations with teachers. In other words, keeping up meant reading all of that email and then remembering to open a separate app every day.

Six weeks of school messages. TalkingPoints isn’t in the count, because it never reaches email.

What Does the Bot Actually Do?

Every afternoon, the bot reads the new school mail in Gmail and the new messages in TalkingPoints. It follows the links to the full newsletters instead of trusting the preview in the email, and then it writes one digest. Everything is grouped by school, with the must-dos first. Every event has a time, or says “time not listed” rather than a guess. A notice that was sent once per kid shows up once, and the sports team mail is left out. On Saturday evenings, a second email looks ahead at the next few weeks.

An illustrative digest with made-up items, to show the format. The real ones are private.

Before it writes, the bot also checks my Google Calendar. If a must-do is already on it, the digest says “Already in calendar”. If it isn’t, the item gets an “Add to calendar” link, so adding it takes one click. And if nothing new came in, the bot sends nothing at all, so an email from it always means there’s something to read.

The digest goes from the bot’s own Gmail account to a mailing list that Mimi and I share. From September 9 to October 20 it sent 44 digests, 40 daily and 4 weekly. That’s about 7 emails a week in place of 30-plus emails and an app. I’d estimate it saves me 30 to 45 minutes of reading a week.

We both read the digest every day, and it has cut down the time I spend keeping up with all the notifications. Another benefit is that my day has far fewer interruptions. Instead of switching over every time an email or a message comes in, I let them pile up and read them all at once in the digest. It has also made it less likely that something falls through the cracks, and the calendar links have made it easy to keep my calendar up to date. Mimi and I share our calendars with each other, so whatever I add, she can see too.

Here are a few of its catches from the first six weeks:

A school event that was posted only in TalkingPoints, with the details on a linked newsletter page. Reading email alone would have missed it.
Two notices gave different dates for the same family night, and the later date was the same evening as an event at the other school. The digest flagged it so we could check.
A calendar entry labeled with the wrong school, and another that ran half an hour longer than the school said.
A newsletter that printed last year’s date for this year’s parade.
A reading test on the same day as two appointments. A later teacher message showed it had already been handled, so the digest told us there was nothing to reschedule.

What Is the Bot Not Allowed to Do?

The bot can read a lot: my Gmail, my calendar and TalkingPoints. If it could also write to all of those, one bad run could send an email in my name to a teacher, move an event on my calendar or reply in a class conversation. So most of its rules are about what it can’t do.

It never sends from my account. It has its own Gmail account, and it only sends to our shared list. It can only read my Gmail.
It never creates, changes or deletes a calendar event. It reads the calendar, and one of us clicks “Add to calendar”.
It never replies to the school, by email or in TalkingPoints.
It never invents a detail. If a time or a place isn’t in the source, the digest says so.
It works out every day of the week itself from the date. It never copies one from a message or trusts its own guess.

What the bot can read, where it can send, and what it never does.

TalkingPoints needed the most care. It has no API, i.e. no official way for a program to read it, so the bot reads it the way a person would, in a web browser. I run the bot onGrok Bot, which runs its bots on a cloud computer with a web browser, and the bot stays signed in to TalkingPoints there between runs.

Signing in to TalkingPoints takes a code sent by text message to a phone number that the school manages. So, when the session expires, the bot stops and asks me to sign in. I click “Open computer” in Grok Bot, take over the bot’s screen, type in the code from my phone and hand control back. I looked into giving the bot a virtual phone number so it could read its own codes, and decided against it. The number is managed by the school, and a number the bot could read would be a key to the account. So for now, I type in the code myself, and I’ll revisit that if it starts happening often.

What Went Wrong?

Quite a bit, and the failures that matter most are the quiet ones. A crash is easy to spot, since no digest shows up. A digest that looks complete but isn’t is much worse, because you trust it.

On September 11, the bot took the day of the week from the text of a message instead of from when the message was sent. By the end of September it had mixed up weekdays on its own too, so now it calculates every weekday from the date.
On October 10, the weekly email skipped TalkingPoints without saying so. Now the bot has to check TalkingPoints on every run, and if it can’t, a warning goes at the very top of the digest.
On October 11, a digest arrived full of raw formatting code, e.g. <h3> where a heading should have been. Now a small program checks every email before it’s sent, and the bot isn’t allowed to send until the check passes.
On October 15, the bot reported that TalkingPoints had signed it out. It hadn’t. The bot had judged from the app’s front page alone. A corrected digest went out the same day, and now the bot confirms a sign-out by opening a page that only a signed-in user can reach.

The four quiet failures, and the rule each one became.

If you recall from my post on random seeds, the dangerous results there didn’t crash either. They looked convincing. A digest that quietly skips a source is the same kind of bug, and the fix was the same kind of fix: a check that runs every time, whether or not anyone remembers it.

There were plumbing problems too. In early October, the bot’s Gmail connection lost the ability to send, so it got a backup way to send email, and a week later the cloud computer blocked that backup, by which point the Gmail connection worked again. All told, 5 of the 44 digests were corrections or second versions.

Every one of those lessons became a written rule. But written where?

Why Do the Bots Live in a Git Repo?

For the first five weeks, the bot’s setup, its instructions and its rules lived only on the cloud computer. If that computer had gone away, I would have had to rebuild the bot from memory. On October 14 the bot exported all of it into my memex, and that is where its rules live now.

My memex is a git repo of markdown files. If you haven’t used git, a repo is a folder that records every change, so you can see what changed, when, and undo any of it. Markdown is plain text with a little formatting. The name comes from Vannevar Bush’s 1945 essay “As We May Think”, where a memex is a device in which a person “stores all his books, records, and communications”, so that it “may be consulted with exceeding speed and flexibility.” Mine is less grand than that.

I started it because AIs kept tripping over my documents. I had pointed Claude at my Google Drive, and when it edited a Google Doc, it would create a new version of the document instead of changing the one I had, which got messy and tedious fast. Claude has its own documents that it can edit, but other AIs can’t easily read them. Most AIs also couldn’t search my Drive well enough to find things, so I often ended up exporting documents to PDF and uploading them by hand. A note-taking app wasn’t the answer either, since you’re limited to the structure and features the app gives you. A folder of markdown files, on the other hand, seems to be something an AI can work with natively. So I downloaded my Drive, had the AI summarize the non-sensitive parts into the memex, and now any AI I use can read and update the same notes. I also have the full history of every change.

Every note starts with a short header that says what it is, when it changed and how private it is. A checker reads every note, a script builds an index of all of them, and a decisions log records each ruling with its date. 185 automated tests run on every change. There is even a cleanup pass I call “dream”, which checks, re-indexes and tidies up the notes.

Each bot has a profile in the repo called BOT.md. It records where the bot runs, its instructions word for word, what it has access to, the decisions it must follow and every lesson above. Changes to it arrive as a pull request, which is a proposed change I can read before I accept it, and the bot is expected to update its profile whenever its setup changes. So, if the cloud computer disappeared tomorrow, I could rebuild the bot from what’s in the repo.

Having the profile has one more advantage. Because the bot already knows how it works and what was in the latest newsletters, I don’t have to explain anything when I ask it a question. I just ask, e.g. whether anything changed for a field trip, and it has the context.

What Would I Do Differently From Day One?

The school bot came first, and it grew its rules one failure at a time. The bots I’ve built since start from a template, and they go a step further: everything they produce, not just their profile, arrives in the memex as a pull request. I cleaned up that template and published it as memex-bots. This part is more technical, so feel free to skim. These are its rules.

The AI finds and judges, and ordinary code decides. The AI searches with its own tools and writes down what it found. A small Python program checks that, scores it and writes the digest. Because the scores come from code, the same input always gets the same result.
Every item needs a source. An item without a link and a short quote is dropped, and the digest says how many were dropped.
Every run is one pull request, never a direct change. GitHub enforces that, and it also requires CI to pass first. CI, short for continuous integration, is the set of automated checks that run on every proposed change.
A file named PAUSED stops a bot, and a cap on unreviewed pull requests stops a bot that’s running away.
A failed run still opens a pull request, with a “Run problems” section that says what went wrong. That’s the October 10 lesson, built into the template.
Every bot starts as a written plan: what it’s for, how much of my time it may take each week, what’s off-limits, when to shut it down and the open questions, all before any code.
Every bot gets its own GitHub access token, a key that only works on the one repo and expires on a set date.
Every bot keeps its own BOT.md up to date.

You can try it without an AI. Download the repo and run the example, and the program runs on sample data: it writes a digest, adds a line to a log, and lists the one item with no link under “Run problems” instead of quietly using it.

What Does It Take to Run Them?

I run my bots on Grok Bot for now, because it comes with my Cursor subscription and has a web browser built in. The template doesn’t depend on Grok Bot, though. Any AI agent that can run on a schedule, read and write files, run a program and open a pull request can run these bots, and the repo has a short guide to controlling a bot from your phone through a private Slack channel.

Grok Bot isn’t the only option anymore. OpenAI’sdots and Meta’s Muse are newer services that also give each AI agent a cloud computer with a browser, and each one has its own advantages when it comes to data. For example, Grok Bot appears to be able to search posts on X, and Muse can search and monitor Facebook. I’m currently using Muse to watch a Buy Nothing group for anyone giving away a golf bag.

Before Grok Bot, I built “automations” with OpenClaw, and then with Hermes because it handled failures better, using OpenRouter to choose the AI model. You can do some pretty powerful things that way, but you’re ultimately limited by the model. I’ve found that useful bots still need one of the most capable models, and the best bang for your buck right now seems to be the monthly subscriptions from the leading AI labs. Open-weight models like DeepSeek, which anyone can download and run, are promising, but in my experience they weren’t really cheaper than something like Grok Bot. My guess is that the AI companies are covering some of the cost today, so that could change.

Whichever service you use, I’d encourage you to build a bot for whatever keeps piling up in your inbox. Run it by hand for a few days and check every digest against the original messages before you put it on a schedule, and every time it gets something wrong, add a rule. I’m planning a follow-up post with step-by-step directions for building a school digest bot like this one. And if a bot turns into something you rely on, that’s the time to write its rules and profile down in a repo, so its memory belongs to you and it can move to the next service with you.

Thanks for reading.

I spent 21 months building a trading system, then had AI models tell me not to start the company

Geoff Cox — Tue, 20 Oct 2026 15:00:00 +0000

Image credit: author, drawn with code

This is a story about how I spent 21 months building a trading system, wrote down what it would take for it to become a company, and then had two AI models tell me that it hadn’t.

If you’ve read my last three posts, you know TopSet. It’s the system I built, mostly with AI coding agents, to pick a portfolio of stocks with machine-learning models and trade it with real money. The first post was about the automated tests that made agent-written code safe to run,the second was about the ways information from the future leaked into my models, andthe third was about noise, and ended with a promise to explain what I decided to do once I knew that my model’s results couldn’t be told apart from luck. This is that post, and you don’t need to have read the others to follow it. There are no charts of returns in it, and that’s on purpose.

Why Did I Give Myself a Deadline?

I started TopSet because I saw promise in using machine learning to find patterns in the stock market, and my early results looked promising enough that I thought I had something. I also knew that a project like this can absorb years without ever answering the question it was started to answer. So, I gave myself a kill criterion, i.e., a condition, decided ahead of time, under which I would stop: about two years full time to find out whether there was a business here. If there wasn’t, I’d go back to leading engineering teams.

The first live trading bot started on December 2, 2024, and I went full time in January 2025. By September 2026, the system had run continuously on live capital for almost two years, through about 1,600 pull requests (each one a reviewed change to the code), and I was close enough to the deadline that I needed an answer I could trust.

To be honest, I already suspected what the answer would be. Around the beginning of the summer of 2026, I’d started to think that, measured over the long run, my results weren’t statistically strong enough to justify starting a company. But a suspicion is easy to argue yourself out of, especially when you want the other answer, so I wanted a process that would either confirm it or talk me out of it.

How Do You Ask an AI Whether to Start a Company?

Most of the work happened before I asked anything. The obvious way would have been to open a chat and type “here’s my trading system, should I start a company?” I’d have gotten an encouraging answer, because I’d have described the system the way founders describe their systems. Instead, on September 17, 2026, I wrote two documents first, working with a Cursor agent.

The first was an index of everything in the repository: what was built, what was live, and, just as importantly, what had already been tried and abandoned. Every closed experiment was listed with its verdict and a link to where it was decided. The point was that the evaluation would start from the record, not from my pitch.

The second was the evaluation itself, written as a set of gates. Each gate was a question with a written condition for failing it, so the evaluator couldn’t decide what counted as failure after seeing the answer. There were five:

Differentiation. What does a customer get here that a cheap index fund, i.e., a fund that simply holds every stock in an index like the S&P 500, doesn’t?
Evidence. Does the case for a company rest on anything unproven, or on anything I’d already tested and abandoned?
Regulatory fit. Can the product be marketed honestly within the rules for investment advisers? In the US, anyone paid to manage or advise on other people’s investments generally has to register with regulators, and there are strict rules about what they can say about performance.
Economics. Can it work for one person without raising money?
The personal path. Found a company, run a 90-day trial to see whether anyone would pay, keep it for my own money only, or stop and look for a job?

The differentiation gate, for example, failed if the honest pitch collapsed to “levered / concentrated market beta with model complexity attached.” Beta is how much a portfolio moves with the market as a whole. In other words, if the product was really just the stock market with extra risk and a complicated model on top, it failed. That matters because extra risk is cheap to buy: anyone can get more of the market’s ups and downs by adding a little leverage to a low-cost index fund, so a product built on that would be charging for something customers can already get cheaply.

I also wrote down what the evaluator was not allowed to do. The brief at the top says “Do not treat years invested … as evidence of a company.” And a list titled “Anti-patterns (do not do these)” includes “Soften a STOP / JOB SEARCH verdict to protect sunk cost.” Sunk cost is the time and money already spent, which can’t be recovered whatever you decide next. The person most likely to want that verdict softened was me.

There’s a small irony in that brief. Under go-to-market, i.e., how customers would find the product, I’d written that I would attract them by blogging about building AI-powered investment products. Here I am, blogging about it.

What Did the Memos Say?

About half an hour after those documents were committed, Claude Fable 5.1, running in Claude Code, worked through the evaluation against the index, the issues and pull requests it pointed to, and its own web research on competitors and regulation. The next morning, Cursor’s agent, Composer, did the same evaluation independently.

How the evaluation ran, from the record to the verdict. Dates are 2026.

Both came back with the same verdict: “personal only.” Keep running the system on my own money, and don’t turn it into a company. The first memo spelled it out: don’t found one now, and don’t raise money. It summed up the honest pitch as “concentrated, high-turnover S&P exposure with ML attached, viable only inside an IRA,” which is very close to the failure condition I’d written for the first gate. It also recommended that I start the job search right away, rather than waiting for my two-year mark.

That phrase “viable only inside an IRA” needs some context. My system rebalances roughly once a month, which means it sells stocks it has held for less than a year. In an ordinary brokerage account, those profits are short-term capital gains, which in the US are taxed at the same rate as income, every year, and that tax drag erases a lot of the advantage of trading actively. In a retirement account like an IRA, trades aren’t taxed as they happen, so that drag disappears. So, any product built on my system would have had to be sold into retirement accounts.

I was heartbroken. I was 21 months in, and I felt close to something big. The answer was that launching a financial product on what I had wasn’t justified.

Can Two AI Models Be Wrong in the Same Way?

Yes, and that was the first thing I learned once I started pushing back.

On September 20, I had Claude Opus review both memos. It found the same two mistakes in each. The first was that both memos described the strategy as riskier than the version I actually run, calling it levered or high-beta when it moves with the market about one for one.

The second mistake was about those retirement accounts. Both memos said they were small by nature, because the government caps how much you can contribute to an IRA each year. That’s true for new contributions, but it misses rollover IRAs. When people leave a job, they often move their whole 401(k), i.e., their workplace retirement savings, into an IRA, and a rollover isn’t subject to the yearly cap. So, the accounts the product would have been sold into could be much larger than the memos assumed. The correction added to both memos says so plainly.

Neither mistake changed the verdict. But they changed how I read the agreement. Two models that read the same index and the same prompt are not two independent opinions. They share the blind spots of the record they were given, and, as the IRA mistake showed, they can also share a wrong assumption about the world that wasn’t in the record at all. Agreement between them is worth something, just much less than it feels like when you first read it.

So, I did with the memos what I’d started doing with my research: I reviewed them again with a different model, and I kept going until the review stopped finding anything major.

How Many Ways Can You Argue With a Verdict?

I didn’t accept it. I spent days trying to poke holes in it, and every time I thought of a reason the memos might be wrong, I had it investigated. I didn’t sit down with a list of counterarguments. Each one was a follow-up thought I had while I was digesting the idea that there wasn’t going to be a company.

Each one also got written down, in a section of the memo called “Arguments tried and retired,” with a rule at the top: “A new argument should say which of these it is not.” In other words, before I could spend another afternoon on an idea, I had to check that I hadn’t already lost that argument once.

By October 10, there were 25 entries. Here are a few of them, so you get a feel for it:

Sell to financial advisers instead of to individual investors. It would fix the cost of finding customers one at a time, but professional buyers screen on exactly the long, audited track record that I didn’t have, so it raises the bar instead of lowering it.
Charge only a performance fee, i.e., a share of the profits instead of a flat fee. US rules generally allow performance fees only for clients who are already wealthy, which excludes the very customers every other idea targeted.
Wrap the strategy in an actively managed ETF, an exchange-traded fund that trades on the stock market like a single stock, to solve the tax problem. ETFs can avoid passing capital gains on to their investors through in-kind redemptions: when a large investor cashes out, the fund hands over shares of the stocks it holds rather than selling them for cash, so the gains are never realized inside the fund. The structure works, and it’s how established funds avoid distributing gains, but it costs hundreds of thousands of dollars a year to run, so it only makes sense for a fund that’s already large.
Registering as an adviser turned out to be much cheaper than the memos assumed, if I did the paperwork myself. That was true, and it was kept as a correction, but cheaper registration doesn’t create any evidence that the strategy is better than an index fund.
Instead of waiting years for the evidence, change the portfolio so that the evidence arrives sooner. Holding more stocks would make the portfolio swing less relative to the market, and with less noise, a real edge would show up in fewer years. The arithmetic worked, but the data didn’t cooperate: the model’s picks only stood out at the very top of its list, so adding more stocks diluted the edge as fast as it cut the noise.

All 25 arguments from the memo, in the order I raised them. The blue ones fixed something the memos had gotten wrong; none of them changed the verdict.

A handful were kept in part, because they fixed something the memos had gotten wrong. A few are still open, including two that only a securities lawyer can answer. None of them moved the verdict.

What Happened on the Day the Verdict Almost Flipped?

On September 22, I ran an evidence review that went my way, four times in a row. Each of the four was a correction, and each one made the project look better than the memos had said. One of them confirmed that my execution costs, i.e., what I lose between deciding to trade and actually getting filled at the broker, were essentially zero, which is a measurement I’d spent months building the tooling to make.

The pull request I opened that day was drafted to change the verdict from “personal only” to “90-day discovery,” i.e., go and find out whether real customers would pay. Then I checked the biggest of the four results on a second, longer set of price history. The first set covered about 8 years and the second about 24. On the 24 years, the result didn’t hold. It was the second favorable result in three days to fail on the longer history. The pull request that merged says, near the top, “This PR was originally drafted as a revision to 90-DAY DISCOVERY on the eight-year evidence. Checking the 24-year pool reversed it.”

If you haven’t spent time with market data, this pattern is worth knowing about, because it’s the rule rather than the exception. Over a few years, it’s surprisingly easy to find a way of picking stocks that beats the market, partly because there are so many ways to try and some of them will win by chance. Over longer periods, most of those advantages shrink or disappear. One well-known study, by David McLean and Jeffrey Pontiff in the Journal of Finance in 2016, looked at 97 stock-market patterns that academic papers had found to predict returns. After the papers were published, the patterns earned, on average, 58% less than they had in the original research.

It was incredibly disappointing. As an entrepreneur, it’s very exciting when you feel like you’ve discovered something significant, and when it turns out not to be significant, it’s hard to come down from that high. Higher highs and lower lows are the life of an entrepreneur.

So, What Actually Decides It?

Looking back at all 25 arguments, the pattern is simple. Packaging, pricing, partners and the sales channel can all change the business questions. Only time, with real money and results that can’t be revised after the fact, changes the evidence question, and nothing I could build or buy makes that time pass faster.

The reason it takes so long comes down to signal and noise. If a strategy’s average advantage is small compared with how much its results swing from year to year, then a few good years look the same as a few lucky ones. At the size of advantage on offer here, and with how much a concentrated portfolio, one that holds relatively few stocks, swings around the market, telling skill from luck with live money would take decades, not quarters. That’s the same wall the momentum index in my last post hit, even after I reconstructed 24 years of its history.

The textbook arithmetic, not my system’s numbers: the smaller the edge compared with its swings, the more years it takes before luck can be ruled out.

My system has now run continuously on live capital for almost two years, and on the evidence so far, its results are not statistically distinguishable from cheaper alternatives. In other words, the difference is too small, compared with the noise, to rule out luck. Even the most celebrated stock pickers go through long stretches of trailing the index. A non-professional launching a financial product on that evidence just didn’t hold up.

Somewhere in the middle of the arguing, after the first memo was drafted, I read Burton Malkiel’s A Random Walk Down Wall Street. I’d read a lot of finance books, but never this one. Malkiel’s argument, first published in 1973 and updated through many editions since, is that markets absorb new information so quickly that consistently beating them is extremely hard, so that an investor who buys and holds a low-cost, diversified index fund is likely to do better than most portfolios carefully picked by professionals. I generally knew that broad index funds are near impossible to beat, but hearing it laid out with that much research behind it was very helpful. I felt like I’d spent nearly two years proving the thesis of the book!

Was It 21 Months Wasted?

Some people will read this as 21 months spent proving a negative. I don’t regret it. I went deep on the principles behind quantitative trading: finding and storing data you can trust, building machine-learning pipelines that are both fast and reliable, and building execution that’s fast and reliable with real money on the line. I learned a ton. That being said, I do wish the outcome had been different.

It’s always better if you can validate an idea up front, so that you don’t spend more time than you need to chasing something that won’t work. The kill criterion and the gates were my attempt at that. But I also don’t think I could have been confident either way without doing the work. Sometimes failing along the way, and the hindsight that comes with it, is the only way to know.

What Happens Now?

I stopped building toward a company. The system keeps running on my own accounts, and I still do research on the side in between other things. The evidence question stays open, and if it’s ever answered, it will be answered by time rather than by another memo.

Next, I’m going back to leading engineering teams, and I’m taking with me the habits that made this decision possible: writing the gates before looking at the result, holding my own ideas to the same evidence bar I’d hold anyone else’s, and treating a pre-committed stop as something to carry out rather than renegotiate.

Thanks for reading.

This describes software I run on my own accounts. It is not investment advice and I do not offer investment services.

Changing only the random seed made my trading model look twice as good

Geoff Cox — Mon, 12 Oct 2026 21:27:00 +0000

Image credit: author, drawn with code

This is a story about how I found out that most of the decisions in one of my research logs were coin flips, and about what I changed so that it wouldn’t happen again.

In September 2026, I ran the same research model eleven times. The code, the data and the settings were identical each time, and the only thing I changed was the random seed. The best of those eleven runs scored about twice as well as the worst.

That would just be a curiosity if I hadn’t spent months making decisions by comparing a single run against a single other run. When we went back through the research log for that model, 14 of the 21 decisions it recorded, mostly about whether to keep a feature or throw it out, rested on a difference smaller than what the seed alone could produce.

If you’re new to this, here’s a bit of background. I build models that pick stocks, and I can’t test one by waiting ten years to see how it does. So, I backtest it: I replay history, let the model make its picks as if it were living through each date, and score those picks against what actually happened next. I also run hyperparameter searches, which backtest hundreds or thousands of versions of a model with different settings and compare them. Over the last couple of years, that has added up to a lot of models.

In my last post, I wrote about leakage, where information from the future sneaks into the replay and makes a backtest look better than anything you could have done for real. This post is about two other ways of fooling yourself: noise, and the choices you make as the researcher. They’re harder to catch, because neither of them is a bug in the code.

Where Did This Start?

It started with a rerun. Earlier in 2026, I had measured what my live trades actually cost to execute, and the answer was close to nothing. That made me wonder whether a few research lines that I had already closed deserved a second look, since some of them had been judged with a cost assumption that was now clearly too pessimistic. So, an agent reran some of the old experiments.

When I reviewed the agent’s work, the conclusions held up, but several of the supporting numbers were wrong. Fixing those meant actually running a check that had been skipped, and that check reran a configuration from the old log with the identical command. The log had recorded it as a solid result. The rerun came back negative.

The first question was whether the engine was random at all, and it wasn’t. Two runs with the same seed on the same machine produced results that were identical down to the byte. So, whatever had changed between the original run and the rerun, it wasn’t chance, and the data changes since then became their own investigation.

However, it raised a question that I had never asked. If the engine gives the same answer every time for a given seed, how much does the answer move when the seed changes?

How Much Does a Result Move When Nothing Changes?

First, a little background on why a seed matters at all. The models in that research line were gradient-boosted trees, and like a lot of tree models, each tree is trained on a random subset of the features and a random subset of the rows. The seed decides which subsets. If you change it, you get a different set of trees from the same data, which is supposed to be a detail.

On September 8, 2026, Claude and I held one configuration fixed and ran it with eleven different seeds over the full four-year research window. The best seed scored about twice as well as the worst. From that spread, you can work out the smallest difference between two single runs that you can trust, i.e. a difference that you’d only expect to see by chance about one time in twenty.

Then we went back through the log. It recorded 21 decisions that had been made by comparing one run against one other run, and 14 of them rested on a smaller difference than that. Some were features I had thrown out as harmful, and four were features I had kept because they seemed to help. In other words, the evidence on file couldn’t tell any of those 14 variants apart from the run they were compared against, in either direction.

Each bar is one of the 21 keep-or-drop decisions in the research log, measured by the difference it rested on, as a multiple of what the random seed alone can produce. Fourteen fall short of the line.

One of those four soon confirmed it. A later rerun, done properly this time with the same seed on both sides, found that a feature the log had recorded as helping was actually hurting. That’s exactly what you’d expect from a difference inside the noise, since it can come out either way.

Fortunately, none of those features were in production, so nothing needed reverting. But the same habit of comparing one run against one other run was also in my more active research lines. When we measured the noise for the daily research engine the same way, one recorded decision stood out. In a long/short research line, which buys some stocks and bets against others by shorting them, the time of day to trade had been chosen on a difference between 17 and 29 times smaller than that engine’s noise. In other words, the backtest couldn’t separate the candidate times at all. The same measurement also showed that the worst loss from peak to trough was even more sensitive to the seed than the score was, and more than doubled between the gentlest seed and the harshest.

So, when does my live system trade? Shortly after the open rather than at the opening bell, and I’d defend that without any backtest. The first stretch after the market opens is the most volatile part of the day, which makes it hard to hit a price target or to model the price you’ll actually get. That reasoning is about how the market works, and no seed can flip it.

The rules that came out of all this are short:

No feature is declared harmful, or kept, on a difference smaller than the measured noise.
Comparisons either use the same seed on both sides, or run several seeds per side and compare the averages.
The seed, the number of threads and where the data came from are stamped into every recorded result, so that any number in the log can be reproduced.

There was one more surprise in the details. The library we were using doesn’t pick a random seed when you don’t give it one. It uses fixed defaults, so every “unseeded” run in the log had been perfectly reproducible all along, and had been sitting on one arbitrary draw that nobody had recorded or varied. Moreover, setting the seed to 0 doesn’t reproduce those runs, because it overrides the defaults with different values. The agent had assumed that it would, and there’s now a test that pins the fact that it doesn’t.

This was tough to learn, as it had taken me a lot of time to engineer myself into those conclusions. Once the coin flips came to light, I kept pushing to see if there was a way to improve the statistical significance of my results. That question, how long it takes before you can tell a real edge from luck, runs through the rest of this post.

Can I Rebuild an Index Fund From Scratch?

A couple of weeks later, I wanted a fair comparison for my own model: something cheap and simple that I could hold it up against, over long stretches of history that included real bear markets. The obvious candidate was a public momentum index fund, which holds the stocks that have risen the most over the past year. It’s one of the cheapest and best-known ways to buy a simple stock-picking rule, which makes it a fair bar for any model that picks stocks. The problem was that the fund only goes back to 2015, and its index’s history before that is the index provider’s own backtest.

So, the plan was to rebuild the index from its published rules, using my own data, and run it back through history myself.

The catch is that a rebuilt index involves a lot of choices, e.g. how far back to measure momentum, how to size each position and when to rebalance (reshuffle the portfolio), and every one of them is a knob. If I build the benchmark, look at how my model compares, and then adjust the benchmark, I’ve tuned my own yardstick.

To avoid that, the spec was written first and merged before any construction code existed. It included a gate that the rebuild had to pass over the years where the real fund exists: its annual return had to land within three percentage points of the fund’s, its risk-adjusted return (how much it earned for how bumpy the ride was) had to be within a fixed margin, and its returns had to move with the fund’s, period by period, with a correlation of at least 0.8, where 1 means moving in lockstep. All three had to pass or the rebuild was disqualified. The spec even explains why it was written first: every parameter is a researcher degree of freedom, and my research log already had four results on record that had been retracted after a closer look.

Writing the gate first wasn’t a new habit for me. When working with LLMs, I’ve found that you have to have very clear acceptance criteria, especially if you want an agent to run unsupervised for long periods of time. I set the initial rules and then refined them with the models.

There was one more guard. The benchmark was deliberately not registered as one of my models, as registered models can be fed into the hyperparameter search. Otherwise, someone, or more likely an agent, could have swept the benchmark’s own parameters and kept the best-scoring version. Changing it requires editing the code, which shows up in the history.

On September 23, 2026, the first build failed the gate by a wide margin. The leading suspect was that it weighted every stock equally, while the real index weights by company size. So, the second build pulled historical share counts from EDGAR, the SEC’s filing database, and weighted by size. That closed about a tenth of the gap, and it failed again.

That second failure was actually useful. A pass would have justified buying a commercial dataset to reach further back in time, but closing only a tenth of the gap meant that better share counts weren’t the problem. So, I found out that the data wasn’t worth buying without having to buy it.

Was That Moving the Goalposts?

This is the part I’ve thought the hardest about how to tell honestly, because the spec said that if the gate failed, the work stopped. It failed twice, and I didn’t stop.

What changed was the diagnosis. The benchmark had been built to rebalance on my model’s dates, roughly once a month, so that a comparison between the two would only differ in how they picked stocks. Meanwhile, the gate asked it to track a fund that rebalances twice a year. Those two requirements contradict each other, so the gate was close to impossible to pass by construction, and both failures had to be read in that light.

So, the work split into two builds with different jobs. The first is a faithful replica on the fund’s own twice-a-year schedule, and its only purpose is to pass the gate and prove that the index can be recreated from my data at all. The second is the monthly build that my model gets compared against, which is then justified by the replica passing. The thresholds never moved, and each new attempt was written up as its own issue, with the reason it might close the gap, before it was run.

On the fund’s own schedule alone, the replica still failed. Then it was weighted by the shares available to public investors (the free float, which is what the index uses), rather than every share a company has issued, and it passed. That happened in the third pull request of the day, about an hour and a quarter after the first failure was merged.

The gap between the rebuilt index and the real fund’s annual return, step by step, against the pre-registered limit of three percentage points. The first two were scored on my model’s monthly schedule, the rest on the fund’s.

That being said, I don’t think “the thresholds never moved” settles it. A gate that you’re allowed to keep attempting is a search of its own, and with enough attempts, something will eventually pass. What convinced me that this one hadn’t just been searched into passing was the next step.

What If You Fit to Holdings Instead of Returns?

A replica that matches the fund’s return could still be holding the wrong stocks, as lots of different portfolios produce similar returns over a few years, especially when they’re all tilted the same way. However, the fund has to file its actual holdings with the SEC, so there’s a much harder target available: the list of roughly 100 companies it held at each rebalance, and how much of each it held.

So, with the gate already passed, Claude compared the replica against those filings. The first result said that where the replica held the same company as the fund, it sized it almost exactly right, but about a third of the companies were different.

Two bugs in the comparison itself came out along the way. It had been pitting a freshly rebalanced replica against holdings from months after the fund’s last rebalance, and the code that matched company names to tickers had been quietly dropping as much as a third of the fund’s holdings. For example, a check for " CO" ate the “Cos.” in “Marsh & McLennan Cos., Inc.,” which then matched nothing. After both fixes, the replica still disagreed with the fund on about a third of the companies, so the difference was real and it was in how the replica ranked stocks.

Part of that ranking was still a guess, and the agent couldn’t read the index provider’s methodology document, because the provider’s site blocked automated downloads. So, it swept the uncertain parameters and scored each setting by how many of the fund’s actual holdings the replica matched, never by return.

One sweep found that the index measures momentum about 42 trading days before each rebalance, rather than on the rebalance date itself. That moved the share of matching companies from about two thirds to about 80%, and it did better on the filings that were held out of the sweep, which it never saw while choosing, than on the ones it was fitted to.

Another sweep, over how many weeks of volatility (how much a stock’s price swings) to measure, came back flat across a tenfold range. That turned out to be the clue, as the form was wrong rather than the length. A line in the fund’s own documentation described volatility as measured on daily returns, not weekly, and on the holdings, one year of daily returns beat the old assumption on the held-out filings. A shorter window won on the filings it was fitted to and lost on the held-out ones, which is what overfitting looks like: a setting that fits the quirks of the data it was tuned on rather than the real pattern. So, the agent took the round one-year value instead.

Neither change was chosen by the gate, and the commit messages say so when the gate didn’t cooperate. The lag narrowed the return gap, while two of the gate’s other measures got slightly worse. Switching to daily volatility didn’t move the return gap at all, and it was adopted on the strength of the documentation and the held-out holdings.

Then I downloaded the methodology document by hand, and its appendix confirmed both. Volatility is the “standard deviation of daily price returns,” and momentum is measured between month-ends ending two months before the rebalance, which from a late-March rebalance is about 42 trading days. That’s exactly the lag the sweep had found! The document also corrected a couple of things we had guessed wrong, including the cap on how much any one company can weigh and the order of the rules for keeping existing holdings, and those brought the gap down to under one percentage point.

I think this is the most useful thing I took from the whole exercise. Fitting the same parameters to returns would have picked a volatility window that, measured against the holdings, didn’t matter at all, and I would have declared the spec identified. On the other hand, matching about 100 specific companies across 13 real rebalances isn’t something that a single free parameter does by luck. So, if you’re validating a model or a reconstruction, look for the target that is hardest to hit by accident, and fit to that.

The Gate Was Green, but Were the Numbers Right?

The replica still only started in 2011, because the free source of company sizes it used begins around then. To reach back further, Claude and I used the same trick as in my last post. An S&P 500 index fund holds every company in the index in proportion to its size, and it has filed those holdings with the SEC going back to 1995. So, the value of each holding in those filings tells you the company’s size on that date.

The longer replica passed the gate again on the years where the real fund exists, with about 81% of its companies in common with the fund and an average weight error of 0.13 percentage points against the fund’s own filings.

Then I reviewed it, and the review still found three wrong numbers:

The daily return series silently dropped one real day of returns in every rebalance period. After the fix, it agreed with the period-by-period returns, and a regression test now pins that.
For a company that joined the index between two filings, a fallback took its size from the next filing. That’s leakage, since it uses a filing nobody could have seen yet, which is my last post in a single sentence. The effect was small, but the default is now strictly point-in-time, i.e. it only uses what was known on each date.
Prices jumped at the point where one data vendor’s history hands over to another’s, so the long runs were redone on a single source, and the seam is filed as its own bug, which is still open.

The way I do these reviews isn’t sophisticated. On delicate or complex work, I ask a model to go through the result and find what’s wrong, often switching to a different model from the one that did the work, and I repeat this until the review stops finding anything major. All three of these turned up after every check had already passed.

Two more findings from that review were about noise rather than bugs. The first was that leaving out the companies that later failed, the survivorship bias from my last post, flattered the result by a measurable amount. It flattered it the most during the dot-com bust, because the missing companies were the ones that collapsed. The second was that shifting every rebalance date by a single week, earlier or later, moved the result about as much as fixing the survivorship bias had. Nobody chooses the calendar date their backtest happens to rebalance on, yet it moved the answer as much as a real data fix.

So, Could It Tell Itself From Luck?

No. After all of that, with a gate written in advance, passed on returns, checked against holdings and reviewed until the review went quiet, the reconstruction gave me 24 years of a momentum index, most of them from before the fund existed. Over those 24 years, its edge over the plain S&P 500 isn’t statistically established, meaning that it isn’t large enough, compared with the noise, to rule out luck.

The reconstruction did what it was built to do. When the edge you’re measuring is small and markets are noisy, telling skill from luck takes decades, not quarters, and even 24 years can fall short. My own model runs into the same wall: on the evidence so far, it isn’t statistically distinguishable from cheaper alternatives. What I decided to do about that is the subject of my next post.

What I’d Do From Day One

If I were starting a research project like this again, this is the list I’d hand to myself and to every agent working with me:

Measure the noise before the first comparison. Run one configuration across several seeds and work out the smallest difference you can trust.
Write the gate before the build. Freeze the spec and the pass/fail criteria first, with the reason for each, and make the thing you’re testing hard to tune by accident.
Never widen a threshold. If the gate itself turns out to be wrong, say so in writing, split the job and keep the threshold.
Fit to the hardest target you can find, e.g. holdings rather than returns.
Review the work again after the gate passes, with a different model if you can, until the review stops finding things.
Shift your dates. If moving the calendar by a week moves the answer, the answer is smaller than it looks.
Keep the graveyard. Every closed experiment stays in the repo with the gate it was judged against and its verdict, so that the next person, or the next agent, can see what was already tried and why it failed.

Thanks for reading.

This describes software I run on my own accounts. It is not investment advice and I do not offer investment services.

My first trading models knew which companies would survive

Geoff Cox — Mon, 05 Oct 2026 22:38:00 +0000

Image credit: author, drawn with code

My first live trading bot started trading real money on December 2, 2024. The models behind it had been trained on years of stock-market history, and every company in that history had something in common: it was in the S&P 500 on the day I trained the model.

That sounds harmless until you think about what it leaves out. A company that was in the index in 2016 and went bankrupt in 2019 wasn’t in the training data at all, and a company that joined the index in 2025 was in it all the way back. In other words, the models learned from a version of the market where nothing ever failed and every stock was already a future winner.

That was a leak, and it went live. It wasn’t the only one either.

If you haven’t built one of these systems, here’s what I mean by a leak. You can’t test a trading model by waiting ten years to see how it does, so instead you backtest it: replay history, let the model pick stocks as if it were living through each date, and score those picks against what actually happened next. A leak is any information that sneaks into that replay which the model couldn’t have had on the date in question. It doesn’t have to be anything as obvious as tomorrow’s price. Knowing which companies will still be around in five years is enough, and the result is a backtest that looks better than anything you could have done for real.

The standard defence is to train walk-forward. For each rebalancing date (the days the portfolio is reshuffled, roughly monthly for me), the model trains only on data from before that date, picks its stocks, holds them until the next rebalancing date, and then the whole thing steps forward and repeats. That way, no single model is ever scored on data it was trained on. Walk-forward is necessary, but as you’ll see, every leak in this post got through it, because each one smuggled the future in somewhere other than the training dates: the list of stocks, the shape of the data, or the data itself, through prices adjusted the wrong way or figures dated before anyone could have known them.

So, this post is about the leaks I shipped, the ones I caught in research before they shipped, and the times an AI was sure it had found a leak and was wrong. Every one of them is a reason that backtests are hard to trust, including your own. Along the way I’ve ended up with one rule for all three: you can’t reason your way to the answer. You need a test that makes a leak show up as a number.

Survivors

What gave it away was a backtest that was too good. I tune the model by running hyperparameter searches, where I backtest hundreds of versions of the model with different settings and compare them. In August 2025, one of those runs came back with a 2020 that was simply impossible, and most of that year’s profit came from a single holding. When I went to look it up, neither Yahoo Finance nor Google Finance had any prices for its ticker before 2021, and my note from the investigation reads: “Perhaps the symbol was renamed?”

It had been. It was Chesapeake Energy, which had been dropped from the S&P 500 in 2018, did a 1-for-200 reverse split in April 2020, went bankrupt later that year, and eventually merged, took on a new name and rejoined the index in 2025. My backtest was holding it in 2020, two years after it had left the index, because the training code only knew about the index as it stood today. (Keep that reverse split in mind, as it shows up again later in this post.)

Fixing survivorship bias turned out to be mostly a data problem. First, I needed the S&P 500’s membership for every day of the training period, not just the current list. I built this from the public record of index changes and had a couple of AI models double-check the list. Then I needed price history for the companies that had left the index, many of which no longer trade. Finally, the training loop had to resolve the set of stocks it was allowed to pick from (its universe) date by date, so that a model trained on 2016 data only ever saw companies that were actually in the index in 2016.

When the fix first ran, the backtest fell apart. The note I left myself in the pull request was, in all caps, “THE ANNUALIZED RETURN IS NEAR 0!!!” That’s what removing a leak is supposed to do, and it’s a very unpleasant thing to watch.

The fix reached the production training path in late September 2025, about ten months after the first live trade.

There was one experiment from that period that I’m still glad I ran. I wondered whether a small, controlled leak might be acceptable: train the first stretch of the model on a later snapshot of the index, so that it learned from today’s large companies, and then walk forward point-in-time from there. It gave no reliable improvement. My conclusion at the time was that a model trained on a fixed list of companies never sees a stock enter or leave the index, which leaves it less prepared for the real index, where that happens all the time.

Unfortunately, that wasn’t the end of it. A year later, in October 2026, a copy of the equal-weight S&P 500 that an agent and I had rebuilt from my research data beat the real equal-weight index fund by a margin that a copy has no business beating it by. The two moved almost in lockstep, so the copy was tracking the right thing, but it was consistently doing better. That’s the same “too good” signature, and it turned out to be survivorship again, coming from two places.

The first was the membership list itself. The public record of index changes I’d built it from is sparse before about 2007, so the early lists were short and slightly wrong. To check, we rebuilt membership from the holdings of an S&P 500 index fund, which has to file its complete list of holdings with the SEC. Against those filings, my list was 74 companies short in 2001 and still 5 short in 2009, and it even contained a few companies that hadn’t gone public yet. The AI models I’d asked to double-check the list in 2025 hadn’t caught any of this.

How many S&P 500 members my list knew about, against the holdings SPY filed with the SEC. The gap is the least my 2025 fix still left out, since the list also held a few companies that hadn’t gone public yet.

The second was prices. Many of the companies that failed outright, like Lehman Brothers, Bear Stearns, Enron and WorldCom, have no price data in either of my data sources, so even with a correct list they can never be picked or held in a backtest.

This one mostly affects my long-history research backtests rather than the live system, since the data the live configuration was chosen from covers 99.6% of index members. But it means that for my backtests before about 2016, the honest reading is “probably an upper bound”, and fixing it properly needs a data source that still sells prices for companies that no longer exist.

So, What Did It Cost?

I don’t know, and I think the reason is worth explaining.

First, a correction to the intuition. A live model can’t see the future, because on the day it makes a prediction there’s no future data to see. Today’s S&P 500 list is the correct list for today. So, the live models were never peeking at tomorrow’s prices. The damage was in two other places:

The backtests I used to choose the model and its settings were inflated by the leak, so the configuration that went live was selected on a flattering number.
The live model had been trained on a history with no failures in it, so whatever it had learned, it had learned from survivors.

Putting a price on that requires comparing what the model expected with what the account actually did, period by period, and in 2025 I had no mechanism for tracking that gap. Moreover, the model worked from end-of-day closing prices while the orders filled the next morning, so any gap I computed mixed model error with hours of ordinary market movement.

That was a consequence of how the early system worked. It ran at a fairly leisurely pace after the market closed: download the day’s prices, retrain the model, decide what to buy and sell, and queue the orders for the next morning. Since then, a lot of optimization and engineering has gone into doing all of that during the trading day. The system pulls a price snapshot at a fixed time in the morning, trains the model on it and starts trading, with a median of about 12 minutes from the snapshot to the first fill. For bots that run this way, the model’s prices and the fill prices are now minutes apart instead of a night apart.

There are also now reports that split a bot’s live result into what the model predicted, what the portfolio’s composition delivered and what execution cost. When I finally ran those reports across the live history in September 2026, every period I analyzed had run without a fixed price time, so the execution numbers came with a caveat that they couldn’t really be read as execution at all. The measurement that would have priced the leak simply didn’t exist yet.

The Leak the LLMs Couldn’t Find

Survivorship was actually the second leak to reach production. The first was subtler, and it’s where most of the testing in the rest of this post came from.

In March 2025, while analyzing the returns of a newer model, I started to worry that there was leakage in the older one. So, I compared two ways of training the same model. One trained incrementally, one rebalancing period at a time, exactly as it would run live and exactly as walk-forward requires. The other trained over the whole period at once. Given the same data and the same settings, they should have produced the same results, but the annualized returns were different.

I asked the LLMs I was working with to explain the difference and they couldn’t. The pull request I opened to debug it says so directly: “The LLMs appear to be unable to determine why the annualized returns for leakage_check.py and train_and_predict.py differ. Let’s roll up our sleeves and figure this out.”

Rolling up my sleeves meant writing a logger. I logged every call to fit() andpredict(), along with the shape and columns of every input, to a JSON file without timestamps, ran both paths and diffed the files. Then I shortened the run until I had the smallest period that still showed a difference and stepped through it in the debugger.

The difference was in the shape of the training matrix. Which columns it had, and which rows survived a filter for missing values, depended on the data loaded for the whole run rather than just the training window. When training all at once, the model for an early period was shaped by companies whose data only began years later. When training incrementally, it didn’t know those companies existed yet. Neither path was reading a future price directly, but the future was deciding what the training data looked like, and that turned out to be enough to change the results.

The comparison that found the leak became a test: train incrementally on part of the data, continue on the rest, and the picks must be bit-for-bit identical to training on all of it from scratch. It started as a script that I ran by hand, then became the code path that production uses to train, and it now runs every week in the automated test suite. If I could only keep one test from this whole project, it would be this one.

The Fix That Switched Off Most of the Model

Unfortunately, the March 2025 refactor fixed the leak and broke something else, and nothing noticed for thirteen months.

Near the end of the data-preparation step, the refactored code selected the training columns by matching column names against a list of ticker symbols. The raw return column for each stock is named after its ticker, so it matched. However, the engineered features (moving averages and volatility measures built from those returns) have the ticker as a prefix rather than as the whole name, so they didn’t match and were dropped. A later step, which lines up every iteration’s columns with the model’s expected list of features, then quietly put them back, filled with zeros.

The model ran, the tests passed and the features were all present in the matrix. Every one of them was zero, and that version went into production along with the leak fix.

It finally came out in April 2026, while an LLM and I were wiring a new family of features into the training pipeline. Earlier that same day we had found a different silent drop, where three new settings were decoded from the model’s configuration and then never passed to the model. When we looked at how much the trained models relied on each input (their feature importance), every engineered feature had an importance of exactly zero.

The fix went in behind a flag so that it could be compared with the old behaviour, and a week later it became the default. If you read my last post, you may recall a change that did nothing, where two runs came back bit-for-bit identical. This is the same smell from a different angle, and I now treat an exact zero with the same suspicion as an exact match.

Corporate Actions Keep Coming Back

Most of the leaks I found in 2026 came from the same place: splits, reverse splits and dividends, which are known as corporate actions.

A quick primer, since everything below depends on it. A raw price is what a stock actually traded at on a given day. When a company does a 20-for-1 split, every share becomes 20 shares worth a twentieth as much, so the raw price drops 95% overnight even though no one lost a cent. A reverse split is the opposite: 200 shares become one, and the raw price jumps 200-fold. An adjusted price rewrites the history before each event so that the series is continuous, e.g. by dividing every price before a 20-for-1 split by 20, and it does something similar for dividends. Neither series is wrong. They answer different questions, and the bugs below all came from asking one series a question meant for the other.

I told the stock-dividend story inmy last post, where a test asserted a data convention that never occurs in practice. Here are three more.

The Model Learned to Short Stock Splits

In a long/short research line, which buys some stocks and bets against others by shorting them, so that it profits when they fall, one data source’s features were computed from raw prices rather than adjusted ones. So, when Amazon split 20:1 in June 2022, the raw series showed a one-day move of about −95%. Alphabet’s 20:1 split about six weeks later looked the same, and Tesla’s 3:1 split read as −67%.

A −95% day dominates any ranking of stocks against each other, and the model’s momentum-style features, which look at how much a stock has risen or fallen recently, carried it for one to three months afterwards. The model learned that stocks with that shape made good shorts, but they weren’t failing companies. They had just split.

A Copilot agent found it while comparing the same strategy across two data vendors. One vendor’s prices came pre-adjusted, the other’s feature panel had been built from raw prices, and the two short sides disagreed by far more than data noise should explain. A simple count shows the problem: in a 2019 sample, 9.7% of the symbols in the raw panel had a single-day move bigger than 50%, against 6.1% in the other vendor’s adjusted data.

What I find interesting is why raw prices had been chosen in the first place. Adjusted prices rewrite the past, e.g. a 2022 split changes the adjusted price for 2018. That looks exactly like lookahead, so raw prices seemed like the safe choice. However, these features were ratios: a price today divided by a price from some days ago. For any window that doesn’t straddle a split, the split factor cancels out and raw and adjusted prices give identical values. For a window that does straddle a split, the adjusted ratio is the true economic return and the raw one is an artifact. In other words, the thing that looked like lookahead wasn’t, and the thing chosen to avoid it was the bug.

Amazon’s 2022 split as raw and adjusted prices. The raw one-month return reads as a 95% crash for a month: the shape the model learned to treat as a good short.

Ratios Want Adjusted, Levels Want Raw

The mirror image turned up about a month later. The same research line had a minimum-price filter so that it wouldn’t trade anything under $10, and it was being applied to adjusted prices.

After a large reverse split, adjustment inflates a stock’s earlier history. A shell company that really traded at a fraction of a cent, and then did a reverse split of a few hundred to one, ends up with an adjusted history of tens to thousands of dollars. So, those names passed the $10 floor, were trained on and were shorted, and the backtest booked wins on positions that nobody could have actually held. Claude found this one while investigating short-side results that still looked wrong after the earlier fixes. Gating on the raw price instead took the number of candidate instances priced under $10 from 11,194 to zero.

The rule I took from the pair is that a ratio wants adjusted prices, because it’s asking how much a holding changed, while a price level wants raw prices, because it’s asking what the stock actually traded at. Neither series is a safe default.

After four layers of price corruption had been removed from that research line, its log summed it up plainly: the earlier positive results had been “largely the corruption we removed.”

The Screen in Front of Everything Was Leaking

This last one is the one I find most instructive, because it was in the tool I use to decide what is worth testing in the first place.

I screen new signal ideas (candidate inputs for the model) with a quick check that runs in seconds, before committing to a hyperparameter search that runs for days. It asks a simple question: on each past date, did ranking stocks by this signal put them in roughly the right order of what they did next? In September 2026, an agent using that screen to check a textbook momentum signal for another project found that the screen itself had two problems.

The first was that it loaded prices through a function whose adjusted flag defaults toFalse, and which returns its price column under the key "Adj Close" either way. The variable was named for adjusted prices, the column was named for adjusted prices and every statistic downstream said adjusted, but the data was raw.

The second was that it built its universe once, for the whole run, from every company that had ever been in the index. On the dates checked, 42 to 47% of the names it was ranking on a given date weren’t index members on that date. Some joined later, which is lookahead, since companies are added to the index because they grew. Others had left years before.

The tell had been sitting in the documentation for months, as a statistic with a note beside it saying “investigate”. It traced back to Chesapeake’s 1-for-200 reverse split in April 2020, the same reverse split from the survivorship investigation a year earlier. Here, raw prices turned it into a gain of more than 10,000%, and it was counted twice because the universe carried both the old and the new ticker. The problem also generalizes. Reverse splits tend to happen to companies in trouble, and companies in trouble are exactly what sits at the bottom of a ranking by recent performance, so any signal that sorts on losers is contaminated in the same direction.

Fortunately, the decisions I had made with that screen were based on rank correlation, which only asks whether the stocks came out in the right order, not by how much, so a few enormous outliers barely move it. When the agent recomputed a signal on corrected data, its rank correlation hardly changed, so those decisions probably still stand. Two other columns in the screen’s output didn’t survive. A leak doesn’t always invalidate everything it touches, and most of the work is finding out which numbers it actually reached.

Time Has Two Dates

Point-in-time doesn’t just mean the date on the row.

Some data describes a period but only becomes public later. One data vendor only keys its quarterly company financials by the end of the quarter they describe. A company’s December quarter isn’t public until its annual report is filed, around the start of March for the largest companies, and the vendor then needs time to pick it up. A reporting lag of 45 trading days after the period end made those numbers “known” a week or two before the vendor would actually have had them. That setting had already stood out in my hyperparameter search results, and the vendor’s own documentation settled it, as the period end is the only date it provides. So, I removed the option.

EDGAR, the SEC’s filing database, does record the filing date, so when I later needed historical share counts, I keyed them on the date they were filed, typically about five weeks after the quarter they describe. Keying on the period end would have used numbers that nobody could have known yet.

One vendor field had no code bug at all. An earnings-surprise figure (how far a company’s reported earnings beat or missed what analysts expected) was being computed against the analysts’ average forecast as it stood after later revisions, rather than as it stood on the day the earnings came out. The fix was to rebuild it from point-in-time estimate snapshots, which meant dropping every year before the snapshots begin and about a fifth of the symbols, which had no snapshots at all. Sometimes an honest fix makes your dataset smaller.

The False Alarms

Leakage can also go wrong in the other direction.

In August 2025, in the middle of the survivorship work, two issues drafted with an AI’s help claimed there was lookahead in my volume filter and in how the code decided which symbols were available. Both read convincingly. When I traced the dates, though, the rebalancing decision happens at the end of the window the code was reading, so all of that data was already known. My reply on the first one starts, “Actually, I believe we DO NOT have a lookahead bias.”

A year later, an agent flagged code that was, in its words, “target-leakage-shaped”: the training target also appears among the input features. It does, and at a glance it looks alarming. However, tracing every date boundary showed no lookahead, as the training window ends where the test window begins and returns are measured forward from the pick date. The agent’s follow-up claim, that the model must therefore be a simple momentum ranker, didn’t survive a measurement either. That’s two confident answers, each corrected by measuring rather than arguing.

The repo’s instruction file for agents now says that any leakage claim must name the specific date boundary that it violates and cite the code path, and those boundaries are pinned by tests named test_no_lookahead_*.

Who Found What?

When I went back through the issues and pull requests to write this post, the pattern was clearer than I’d remembered. I found the early leaks, the ones that reached production, from numbers that were too good or didn’t match: an impossible year and two training paths that disagreed. The 2026 leaks were mostly found by agents, and almost never while they were looking for leaks. They turned up while we were wiring new features, building a new test harness, comparing two data vendors or checking a rebuilt index against the real one, with me reviewing what came back. The agents also raised most of the false alarms.

The tests mostly arrived after the leak that justified them. I don’t think that’s a failure. A test written before you’ve been bitten is a guess about where the bugs are, whereas a test written afterwards is a record of where they were.

Making Leaks Show Up as Numbers

Here’s what actually caught things, roughly in the order that I’d add them to a new project:

Invariance. A result for a past window must not depend on the future. Training incrementally must match training all at once, and moving a run’s end date must not change an earlier window. The March 2025 leak broke exactly this, and so did a similar one I found later in a different research line, where extending a run’s end date changed results for years that were already over.
Counts of the impossible. Single-day moves beyond 50%, prices under a cent passing a $10 filter, feature importance of exactly zero, or a year of returns that no strategy has ever produced. Several of the leaks in this post would have shown up in a count like that well before they showed up in a result.
Point-in-time by construction. Resolve index membership per date, from the best source you can find, and key vendor fields on when they became knowable, not on the period they describe.
Compare against something real. A copy of an index fund that beats the fund, or two vendors that disagree, is a leak or a data bug until proven otherwise.
A pure-noise null. Run the whole pipeline on random-walk prices, where there’s no signal by construction. If it finds one, it leaks. In one research line, after four leak fixes, the null finally sat on zero, and that was the result I trusted most.
Replication on independent data. When an agent hands you an encouraging statistic, rerun it on data it hasn’t seen before you believe it.

This describes software I run on my own accounts. It is not investment advice and I do not offer investment services.

I let AI agents write most of a live-trading codebase. Here's the gate that made it safe.

Geoff Cox — Mon, 28 Sep 2026 23:50:00 +0000

Image credit: AI-generated

For the last 21 months I have been the only engineer on TopSet, a system that trades real money. It picks a portfolio with machine-learning models, rebalances on a schedule, and executes through live broker APIs on infrastructure I own end to end. It is my own capital, not anyone else’s.

Agents wrote most of the code. Claude Code, Cursor, Copilot. About 1,600 pull requests in those 21 months, which is more than I could have typed in three times the span.

That leaves you with a choice everyone adopting these tools runs into. You can read every line the agent produces, in which case you have bought yourself a slower version of writing it yourself. Or you can skim, merge, and move fast, in which case you accumulate code that looks right and isn’t. On a system that places orders with real money, the second option ends with trades you didn’t intend.

I took a third option: make something other than my attention the thing that decides what ships. Tests became the gate. Nothing merges or deploys without a green build, and my own reading moved up a level, to design, to test coverage, and to the handful of results I refuse to let change quietly.

That worked. It also failed in ways I did not expect, and the failures are the interesting part, so I’ll spend most of this post there.

What I actually read

“Tests are the gate” is not the same as “I stopped looking.” What changed is where I spend attention.

Most pull requests get a light review. Research code I skim. Changes to the parts that decide what to buy or how to trade it, the model training and the execution paths, I read properly, looking for whether the approach is reasonable rather than whether every line is correct. The tests cover the lines. I cover the idea.

Then there is one rule I do not bend. If a change moves a snapshot, or shifts the result of an end-to-end test, that is a stop. Snapshots and integration results are supposed to be stable, so a diff in one of them means either the change did something I didn’t intend, or it did something I did intend and I now owe an explanation. An agent will happily regenerate a snapshot to make a build go green. That is the single most dangerous thing these tools do, and it is why the snapshot files live in the repo where a diff is visible in review.

For anything involving a calculation I have not verified before, the tests are not enough either. I have the agent write a report script, and there are now 161 of them, that dumps the pipeline to CSV. I open the CSV in a spreadsheet and check the arithmetic by hand: return per trade, slippage, annualized return. Then I spot-check inputs, picking a few prices and going back to the raw data directory to confirm those were the prices the system actually used.

It is slow and it is not clever, and it has caught things nothing else would. A test asserts that a function returns what the author believed it should return. A spreadsheet asks whether the number is true.

What the gate is

Every pull request runs:

lint (ruff) and type checking (mypy)
about 7,500 tests across 331 test modules
an assertion that the test database is empty afterward
a migration rollback-and-reapply, all the way to base and back
terraform validate against both AWS accounts

If any of that fails, nothing merges. The build is the merge gate, and it is not advisory.

Once a week, on a schedule, a heavier set runs:

two end-to-end rebalancing runs against a mock broker
the same integration test 100 times in a row, looking for flakes
nine deterministic model-training snapshot runs, pinned inside Docker
two leakage checks on the training pipeline

Running the same integration test 100 times only makes sense because the test is not the same each time. The mock broker fills orders probabilistically: a 20% chance per 10-second cycle, 70% of those complete and the rest partial at a random size, at random prices. Every iteration produces a different interleaving of fills, partial fills and timing. A hundred runs is a search for race conditions, not a hundred repeats of one path.

What it finds are real concurrency bugs wearing flaky-test costumes. Worker threads leaking between tests. A queued task waking up after teardown had already deleted the row it wanted. A data loader that wasn’t thread-safe and corrupted a shared index under load. Each one showed up first as “that test fails sometimes.”

The weekly set exists because a flaky gate is worse than no gate. If a test fails once every thirty runs, and twenty agent PRs land in a week, that one test goes red most weeks. Have a few like it and something red is always on screen, and you start ignoring red. Hunting flakes is not hygiene here. It is what keeps the gate meaningful.

The tests that matter are about money, not functions

Unit tests are cheap and agents write them happily. They are not what I trust.

What I trust are end-to-end tests that drive a full rebalancing against a mock broker, with the awkward parts turned on. They’re named after the ways money goes wrong:

orders that have to be resubmitted
a rebalancing that resumes from the middle, with partial fills already on the books
a resume that has both buys and sells outstanding
smart orders cancelled mid-flight, on the sell side and then the buy side
deposits and repeating withdrawals arriving while a rebalancing is in progress

These are exactly the control flows a language model gets plausibly wrong. Interrupted work, partial state, money arriving in the middle of an operation: the code for each of these reads fine in isolation. It’s the interaction that bites.

Here is one that made it past review and got caught by the constraint at the bottom of the stack. When the last pending buy of a rebalancing had its amount adjusted for pending withdrawals, the adjusted value could land on exactly zero. The existing guard raised an error on a negative amount, but zero isn’t negative. The affordability check, available_cash >= amount, was 0 >= 0, so it passed. The order was promoted, and the database rejected it: amount > 0 is a check constraint on that table.

The interesting part is what that did to the retry. The integrity error fired inside the same transaction that carried the completing order’s status update, so the rollback discarded that too. The scheduler came back around, found identical inputs, and did it again. Not a self-healing retry: a stuck rebalancing.

The fix took three attempts, and the second one is the useful story. Gating the promotion on amount > 0 looks correct and does nothing, because the code had already assigned the adjusted value to a live ORM object. That marked it dirty, and the next autoflush in the same transaction persisted the zero regardless of whether the order was ever promoted. The condition guarded the decision, not the mutation.

I only know that because the failure was reproduced live against main before any fix was written, and the first fix was re-run against that same repro rather than checked by reading the code. When an agent is writing the patch, “I read it and it looks right” is the single least reliable signal available to you.

Three ways a green build lied to me

This is the part I’d want to read, so here are the three worst cases, all real, all from this codebase.

1. The test asserted a convention that does not exist

Corporate actions have to be applied to historical prices, and stock dividends are the fiddly ones. My broker’s API returns a rate for each. The code treated that rate as a bare fraction: a 5% stock dividend arrives as 0.05.

It isn’t. It’s the ratio of new shares to old. Every one of the 101 stock-dividend records across 43 symbols in my data has a rate of at least 1.0016. Not one is a bare fraction. The convention the code assumed does not occur in the data at all.

The unit test passed, because the test asserted the same wrong convention with a fixture value of rate: 0.05, a number that cannot appear in production. The test and the code were wrong in exactly the same way, which is what you get when both are written in the same sitting from the same misunderstanding.

The effect was that each stock dividend roughly halved the adjusted price, and it compounded across events. One symbol with nine of them came out deflated by about 490×, turning a real price history into a fake ramp of more than a thousand-fold over the period. And because every price stayed between a few cents and a few hundred dollars, a check for implausible price levels would not have flagged it either.

Southern Copper’s real daily prices, adjusted twice by the same code. The only difference is whether the stock-dividend factor is 1 / rate or 1 / (1 + rate).

The fix was a corrected fixture, plus a regression test that runs a nine-event sequence and asserts the cumulative adjustment factor stays sane. The lesson I took: a fixture is a claim about the world, and nobody reviews fixtures.

2. The test passed on NaN

In a research tool, an agent added a second way to select a model configuration and reported the result in the PR: 100% agreement with the existing selector, at every sample size tried.

That number was not real. The tool built its pool from a source that was missing the columns the default scoring version needs, so every score came out NaN. With all scores NaN, the selector fell through to a fixed tie-break order, which of course agrees with itself 100% of the time at every sample size.

The test that was supposed to cover this had been passing on those same NaN scores for as long as it had existed. It never checked anything. Green, meaningless, and quoted in a PR description as a finding.

The follow-up PR made the condition a hard error rather than a silent NaN, pinned the test to a scoring version where the columns exist, and added guards. It also flagged that an older result produced the same way might carry the same artifact. That is the right instinct: when you find a lie, go and check what else it told you.

3. The change did nothing at all

An agent wired a new training target through what looked like the right code path. The first screen ran clean and produced numbers bit-for-bit identical to the baseline. Every metric, to four decimal places.

Identical results are not a pass. They’re a smell. Two different configurations producing byte-equal output usually means one of them didn’t happen.

It hadn’t. The runner always sets a parameter that routes execution to a completely separate function, which builds its own training target and never looked at the new field. Unpickling both runs and diffing the actual selections showed 238 of 238 identical picks for the test year. After the real fix, 0 of 238 were identical, and a regression test now asserts the two modes produce different output.

Nothing failed. The compute was spent. The result would have gone into the research log as “no effect,” which is a conclusion, and it would have been wrong.

The shape of all three

In every case the build was green, and the number was wrong. Tests prove that the code does what the tests say. They say nothing about whether what the tests say is true, whether the code under test ran at all, or whether the data going in is what you think.

That is the agent-era failure mode, and it is worse with agents than without them, for a simple reason: an agent writes the code and the test together, from one reading of the problem. If the reading is wrong, both artifacts are wrong, and they agree. A human pair would at least have had two readings.

What I do about it now, concretely:

Distrust results that are identical, unchanged, or perfect. Check the output, not the exit code.
Treat fixtures as claims. If a fixture value can’t occur in production, that’s a bug in the test.
Make impossible states loud. NaN, missing data, and empty inputs should raise, not flow through and produce a number.
When something is found to be wrong, go back and check what else that thing told you.

The reviewer that isn’t a test

Some mistakes are not code-shaped and no test will ever catch them.

So I added a second kind of review: a deliberately skeptical reviewer persona, run against a finished result rather than a diff, with instructions to argue that the result is not real. I iterated on the persona with Claude until it stopped being agreeable.

It found things I am glad it found. That a configuration had been tuned by repeatedly checking results on data I had promised myself I would only look at once, so my holdout wasn’t a holdout any more. That a cost model was running with zero financing cost while the strategy it priced used nearly four times leverage.

Neither is a bug. Both are the kind of error that invalidates a conclusion, and no test suite in the world was going to tell me.

The gate after the gate

CI stops caring the moment something merges. After that the system runs unattended, on a schedule, against a live broker, and the question changes from “is this code correct” to “is this thing behaving right now.”

Every ERROR the system logs in production becomes a CloudWatch alarm that posts to a Discord channel on my phone. That is the whole alerting stack, and for a one-person operation it is enough: I am not running a rotation, I just need to know.

The loop from there is the same one I use for everything else. Point an agent at the production logs, hand it a local copy of the database from around the failure, ask for a diagnosis and a fix with a test. The logs and the data are usually enough for it to find the problem, and the test it writes is the thing that stops the problem coming back. Several of the bugs in this post arrived that way rather than through CI.

Worth saying plainly: the tests did not catch those. The tests caught the next occurrence.

What stays human

Architecture. Test design. What “done” means. Merge judgment when the tests pass and something still feels off.

And one more thing that turned out to matter: the repo’s own instruction file. Every time an agent reached a wrong conclusion in a way another agent would repeat, I wrote the correction down there. One rule says that any claim of data leakage must name the specific date boundary it violates, because two different agents had asserted leakage from reasoning alone, and both were wrong. That file is now a list of the mistakes this codebase invites. It’s the closest thing I have to institutional memory for a team of one.

What it cost

The per-PR suite takes about ten minutes. That is the tax on every change, and it is the number I would defend hardest: ten minutes is short enough that I never route around it, and long enough to run tests that touch a database and a fake broker.

The machine bill is every one of the 3,000 GitHub Actions minutes my plan includes, plus roughly $20 a month for extra minutes on top. The agents themselves are Claude Pro and Cursor Pro, $20 each. I also paid $40 a month for Copilot until GitHub moved it to per-token billing, which would have raised my bill more than tenfold, so that work moved to Claude Code and Cursor. So the tools that wrote most of this codebase cost tens of dollars a month, like the compute spent checking them, and neither is what the project actually cost.

What it actually cost is fixtures and flakes. Writing a mock broker that fails in realistic ways, maintaining a self-contained dataset so model training can be tested at all, and then the long tail of chasing tests that fail once in thirty runs. None of that is glamorous and all of it is load-bearing. Every hour I skipped there came back later as an hour of not being able to trust a result.

And sometimes the gate itself is the thing that’s broken. The 100-run soak lives in a shell script rather than inline in CI, because the inline version ran under a POSIX shell where {1..100} isn’t brace expansion. It was a literal string. The loop ran exactly once, and reported success, for longer than I’d like to admit.

The tests are not where I over-built. Every gate in this post exists because something broke, or nearly did, and the existing checks let it through. I would add all of them again.

Where I over-built is everything the tests had to cover.

I started TopSet expecting to turn it into a product, so I built it for customers who never existed. A bot lifecycle with eleven kinds of change request, each one a queued, locked, asynchronous workflow object, so that deposits and withdrawals and stops could be requested safely by someone who wasn’t me. Two broker integrations instead of one. A staging environment that mirrors production, behind a VPN. Cross-account backups with compliance-grade retention locks on a database that holds my own trades.

For one person running their own accounts, several of those could have been a script I ran by hand while watching it.

That is the part that compounds, and it is why I am putting it in a post about testing. Every one of those surfaces is something the gate has to cover. The change requests need end-to-end tests for money arriving mid-rebalance. Two brokers means two of everything, including the mocks. The staging environment is a second place for migrations to fail. I did not pay for the product shape once, in the building. I have paid for it every week since, in the size of the harness that has to keep it honest.

If I were starting again for a book of one, I would build the trading and the research properly and leave the product machinery until somebody asked for it.

If you’re rolling this out to a team

I’ve done this alone, with tests I wrote and one person’s judgment. That is not the same problem as a team of fifteen, and I’d be careful about anyone who tells you otherwise. But some of it transfers, and the ordering I’d insist on comes from a team I did run: at a healthcare company, requiring automated tests on every PR is what took active medium and high bugs down 72% and weekly hotfixes from 7 to 1.5. That was before agents. It’s the same first move.

Fix the gate before you raise throughput. Coverage on the paths that lose money or data, and no tolerated flakes. Agents multiply whatever your CI already is.
The author owns the change, whoever typed it. “The agent wrote it” is not a review comment, and it is not a defense in a postmortem.
Review the result, not the diff. For anything that produces a number, ask what would look different if the change had done nothing.
Measure escaped defects and hotfix rate. Not lines, not PR counts. Agents make those metrics meaningless overnight.

The part I would not change: the gate is what let one person move at this speed on a system where mistakes cost real money. The part I underestimated: a gate is a claim about correctness, and claims need checking too.

This describes software I run on my own accounts. It is not investment advice and I do not offer investment services.

Turnkey Data Analysis Using DuckDB and Metabase for PostgreSQL & CloudWatch

Geoff Cox — Fri, 01 Mar 2024 13:05:39 +0000

How we used DuckDB and Metabase to streamline our data analysis workflow, enabling us to build an email list to contact our frequent users.

Data Analysis with DuckDB and Metabase. Image credit: Author

Back in July 2023, my friend and I were excited about building something with a Large Language Model (LLM). At that time, we noticed ChatGPT had no native internet access. ChatGPT did roll out a beta feature providing Bing search access, but it was retracted shortly after its release. We observed several ChatGPT plugins offering internet access, but none were free. Anticipating OpenAI’s reintroduction of Bing search, we decided to quickly develop our own web search plugin for ChatGPT to attract some users.

After spending a few days coding and setting up AWS infrastructure, we successfully launched our MindfulDataAI Search Plugin in the ChatGPT Plus store. Impressively, OpenAI approved our plugin within hours. Within days, we observed thousands of web searches being conducted through our plugin. This success spurred us to explore other ideas, such as developing business tools for ChatGPT. We knew our plugin’s relevance might be short-lived, but we were hopeful that we could identify a business-specific problem before then. Indeed, this was underscored in September 2023 when ChatGPT reintroduced Bing search.

Surprisingly, even after the reintroduction of ChatGPT’s native Bing search, a significant number of users continued using our plugin. Intrigued by this preference, we sought to understand the underlying reasons.

Our challenge was to analyze user data efficiently. User activities were logged, and email addresses were stored in a PostgreSQL database. Initially, we used CloudWatch Insights to identify the most active users, followed by PostgreSQL queries to look up their email addresses. This method was effective but disjointed. I was also looking for an opportunity to experiment more with DuckDB, so I decided to create an open-source repository, called QuackMeUp, along with a guide to help others perform this type of data analysis more efficiently.

In the following sections, we’ll discuss how we used DuckDB and Metabase to streamline our data analysis workflow, enabling us to build an email list so that we could contact our users to understand why they still preferred our plugin.

DuckDB & Metabase

DuckDB is an in-memory, columnar database optimized for quick analytical queries. Its main advantages are speed, ease of setup, and SQL compatibility. It’s particularly useful for fast data analysis without heavy resource requirements.

Metabase is a user-friendly, open-source BI tool. It allows easy data querying and visualization, even for non-technical users. Metabase supports various data sources and is known for its intuitive interface and collaboration features.

Why Use Them?

For MindfulDataAI, DuckDB offers the speed we need for analyzing logs and PostgreSQL data, while Metabase makes data visualization and sharing straightforward. This combination aims to streamline our data analysis, merging different data sources efficiently and reducing complexity.

Next, we’ll explore how we integrated these tools into our workflow and the insights we gained.

High-Level Overview of the Solution

Let’s start with an overview of the key components of our solution:

Modify the app’s code so that it logs in JSON format.
Deploy a scheduled AWS Lambda function that will automate the export of CloudWatch logs to an S3 bucket.
Download the logs from the S3 bucket using the aws s3 sync command.
Run a Python script to extract JSON segments from the logs and save them as .json files.
Configure DuckDB to connect to your PostgreSQL instance and to access these .json files.
Use Metabase for business intelligence. This setup will enable you to join datasets and create useful analytics dashboards.

This is how the pipeline works:

End-to-End Data Pipeline for Business Analytics. Image credit: Author

The Detailed Solution

Download the GitHub repo

To make it faster to get started, I’ve packaged all the code in a GitHub repo that you can download with:

git clone https://github.com/redgeoff/QuackMeUp

Then, cd QuackMeUp so that you are ready for the next steps.

Install the dependencies

Install Docker
Install the awscli and run aws configure
Install Python, e.g. brew install python@3.11
python3.11 -m pip install poetry
poetry install
Install the Git hooks for code formatting with black: poetry run pre-commit install
Install DuckDB, e.g. brew install duckdb

Configure the .env file

cp .env.example .env and then edit .env
Choose a value for TERRAFORM_BUCKET_NAME that will be unique across all AWS users, e.g. myorg-quackmeup-terraform-bucket, where myorg is a unique name for your organization. This bucket will be used to remotely store your Terraform state.
Choose a value for LOGS_BUCKET_NAME that will be unique across all AWS users, e.g. myorg-quackmeup-logs, where myorg is a unique name for your organization. This bucket will be used to store logs exported from CloudWatch.
Be sure to substitute the value of PG_CONNECTION_STRING with the connection string for your Postgres instance. Ideally, you’d use the connection string for a read replica.
You can leave the rest of the values in .env intact unless you prefer them to be different.

Deploy the infrastructure

Install Terraform

Load the AWS Console and manually create an S3 bucket with the same name that you specified for TERRAFORM_BUCKET_NAME.

Create S3 bucket for Terraform state. Image credit: Author

Initialize the terraform environment: ./terraform/init.sh

Make sure the terraform plan doesn’t error out: ./terraform/plan.sh

The following apply.sh script will then:

Create an S3 bucket for LOGS_BUCKET_NAME
Create an ECR repository for the Lambda code
Build and push a Docker image to this ECR repository
Schedule a Lambda (using the Docker image) that will export the latest CloudWatch logs to the S3 bucket every 4 hours
Create IAM policies and roles as needed

Deploy the infrastructure with ./terraform/apply.sh

Tag target logs for export

Open the AWS Console and navigate to the CloudWatch Log Groups page. For each log that you wish to be exported to S3, simply add a tag with Key=ExportToS3 and Value=true. The log exporter Lambda runs every 4 hours so you’ll have to wait at least 4 hours before you start seeing any data in the S3 bucket.

Tag CloudWatch logs for export. Image credit: Author

Install QuackMeUp

Create the DuckDB instance: ./scripts/duckdb/create_db.sh
Attach the Postgres instance to DuckDB: ./scripts/duckdb/attach_postgres.sh
Build the Docker image that will run Metabase with DuckDB support: ./scripts/metabase/build_docker.sh
Run the Docker containers: docker-compose up -d
Seed the Metabase config so that you can just log in: ./scripts/metabase/import_db.sh

Execute the pipeline

After the logs have been updated, run the following pipeline to ingest the updated data into DuckDB:

Use s3 sync to download the logs from S3: ./scripts/s3/download_logs.sh. Note: if you have a lot of data, this can take a while when it is first run.
Run a Python script to extract the JSON payloads in the logs so that they can be analyzed with DuckDB: make format_logs
Create a DuckDB table pointing to the extracted logs: ./scripts/duckdb/import_logs.sh
Restart Metabase so that it loads the updated DuckDB: docker restart quackmeup-metabase-1

Accessing Metabase

You can then access Metabase by visiting http://localhost:3000 and logging in with:

Email: quackmeup@example.com
Password: E!v_#nc$48pqfZJ

Troubleshooting

DuckDB is incredibly fast because it stores a substantial amount of data in memory. If you encounter errors related to insufficient memory, it’s advisable to open the Docker console and increase the memory allocation. In our experience, for processing 3GB of zipped logs, we needed to increase the memory allocation to 10GB. Alternatively, you can modify the format_logs.py script so that it filters out any JSON data that you don’t need.

Example Analysis

Now that we’ve got QuackMeUp running, we can start doing some data Analysis. The following section is specific to our data, but it will help you to visualize the power of the QuackMeUp Analytics pipeline.

Recall that in the case of MindfulDataAI, we want to get the email addresses of all the frequent users. In our particular case, the search requests are logged in payloads like the following:

{
 "json": {
 "asctime": "2024-01-08 23:57:51,020",
 "name": "my_logger",
 "levelname": "INFO",
 "message": "Response",
 "code": "RESPONSE",
 "request": {
 "method": "GET",
 "path": "/search",
 "query": {
 "query": [
 "number of tourists in Orlando 2022"
 ]
 },
 "body": …
 }
 }
}

To start, we can look at things like the number of searches each month:

SELECT
 SUBSTRING(json->>'asctime', 1, 7) AS month,
 COUNT(*) as num_searches
FROM logs
WHERE JSON_EXTRACT(json, '$.code')='"RESPONSE"'
 AND JSON_EXTRACT(json, '$.request.path')='"/search"'
GROUP BY month
ORDER BY month ASC

You can run this query in Metabase by creating a new Question:

Metabase query for searches per month. Image credit: Author

And then you can render a nice-looking visualization:

Monthly searches. Image credit: Author

Note: You may notice from our data that it contrasts with the typical “up and to the right” trend. We launched the plugin in late July, which led to a significant increase in searches in August. However, in September, OpenAI reintroduced native Bing search support in ChatGPT. Since then, the number of searches using our plugin has nearly halved each month! 😱. Nonetheless, we still retain a user base, and we are interested in contacting them to understand why they prefer our plugin over the native Bing search.

We utilize Supabase and its OAuth integrations, which means the users’ email addresses are stored in a PostgreSQL table named users. By joining the searches and users using the user_id, we can compile a list of users with the highest number of searches:

SELECT
 s.num_searches,
 s.user_id,
 s.first_search,
 s.last_search,
 u.raw_user_meta_data->>'name' as name,
 u.email
FROM (
 SELECT
 COUNT(*) as num_searches,
 json->>'request'->>'headers'->>'MINDFULDATAAI_USER_ID' as user_id,
 MIN(SUBSTRING(json->>'asctime', 1, 11)) as first_search,
 MAX(SUBSTRING(json->>'asctime', 1, 11)) as last_search
 FROM logs
 WHERE JSON_EXTRACT(json, '$.code')='"RESPONSE"'
 AND JSON_EXTRACT(json, '$.request.path')='"/search"'
 AND user_id IS NOT NULL
 GROUP BY user_id
) s JOIN users u ON u.id=s.user_id
ORDER BY s.num_searches DESC
LIMIT 10

And when run, Metabase will give you a table of data that can easily be exported to a CSV file:

Image credit: Author

We just joined the JSON log data (searches) with the Postgres data (users) in a single query using DuckDB. Isn’t that cool?! 😎

Although this solution was somewhat more complex than necessary, it has provided us with a robust pipeline that enables further analysis through the crafting of new SQL queries.

Alternative Solutions

While the combination of DuckDB and Metabase presents an efficient and effective approach for data analysis in our scenario, it’s important to consider alternative solutions that might better align with different requirements or preferences. For example, you may want to consider a hosted version of Metabase that can be shared with your coworkers. Alternatively, you might prefer not to have to re-execute the entire pipeline whenever there are changes. Here are some noteworthy alternatives:

1. Using Presto or Athena for Querying:

Presto, an open-source distributed SQL query engine, and Athena, the AWS managed version of Presto, are powerful tools for querying data across various sources like S3 and PostgreSQL. By employing either of these, you can execute SQL queries across your data lake and database without the need to load data into an analytics tool. This approach can significantly reduce the memory footprint of your Metabase instance, as it doesn’t require storing large datasets for querying purposes.

2. Analytics Postgres Database for Reporting:

Another approach is to set up a pipeline that selectively copies key log events and PostgreSQL data into a dedicated analytics database. This strategy can streamline the data analysis process, particularly for creating an analytics dashboard aimed at end-users. By isolating the analytics data, you can optimize the database schema for reporting and ensure that your production databases are not burdened by heavy analysis queries.

3. Exploring Other BI Tools:

While Metabase is known for its user-friendliness and simplicity, other Business Intelligence tools like Superset, Looker, or Tableau might offer additional features or better integration with existing systems. Each of these tools has its unique strengths — for example, Superset is highly customizable, Looker excels in creating complex data models, and Tableau is renowned for its data visualization capabilities. Depending on the specific needs and existing tech stack of your organization, one of these alternatives might be a more suitable choice.

4. Utilizing Snowflake, BigQuery, or Redshift:

For more robust data warehousing solutions, Snowflake, Redshift, or BigQuery can be considered. These platforms are designed to handle large volumes of data and complex queries efficiently. They can directly query data stored in S3 and integrate with PostgreSQL, similar to DuckDB. However, they offer additional features like automatic scaling, extensive support for different data formats, and advanced security features. These platforms are particularly beneficial for organizations dealing with massive datasets and requiring a comprehensive data warehousing solution.

In conclusion, while our implementation with DuckDB and Metabase offers a streamlined and effective solution for our needs, these alternative solutions provide different benefits that could be more aligned with other specific requirements or preferences. The choice of tools and architecture depends greatly on the scale of data, specific use cases, and the existing technical infrastructure of the organization.

Conclusion

In wrapping up our journey with DuckDB and Metabase, the key takeaway is the value of simplicity and efficiency in data analysis. Our experience with the MindfulDataAI Search Plugin highlighted how the right tools, like DuckDB and Metabase, can simplify the process of merging different data sources and extracting meaningful insights quickly.

These tools enabled us to analyze log data and PostgreSQL records effectively, leading to a better understanding of user preferences and improving our services. However, it’s essential to recognize that different scenarios may call for different tools. Alternatives like Presto, Athena, or other BI tools might be more suitable in varying contexts.

Ultimately, the goal is to choose a solution that is not only effective but also aligns with your project’s specific needs and technical environment. This approach ensures that data analysis remains a practical and valuable asset in decision-making processes.

Want to use ChatGPT in Slack?

At MindfulDataAI.com, we offer Arty, an app that integrates ChatGPT with Slack. Arty can be used in group settings or one-on-one, seamlessly incorporating ChatGPT and web search capabilities directly into Slack.

Running MosaicML's MPT-7B, a ChatGPT Competitor, on AWS SageMaker

Geoff Cox — Tue, 18 Jul 2023 15:48:01 +0000

In this blog post, I’m going to take you step-by-step through the process of running MosaicML’s ChatGPT competitor, MPT-7B, on your own AWS SageMaker instance.

MPT-7B and AWS SageMaker. Image credit: MosaicML and Author

Are you excited about the capabilities of ChatGPT, but have concerns about exposing your sensitive data to OpenAI? Luckily, there are alternatives that you can run on your own infrastructure. One such alternative is MosaicML’s MPT-7b, a competitor to ChatGPT, which we’ll explore in this blog post.

Introduction to MosaicML and MPT-7B

MosaicML, recently acquired by Databricks for $1.3 billion, has been making waves in the ML community with their MPT-7B model, a supposed competitor to ChatGPT. Despite its promise, running this model can be daunting due to sparse documentation and its heavy resource requirements. However, one can run MPT-7B on AWS SageMaker in a Jupyter notebook, an environment that is beginner-friendly and highly flexible for rapid iteration. This setup allows you to test the model’s feasibility and hardware requirements before deciding to move into production.

Running MPT-7B on AWS SageMaker

Running MPT-7B in a Jupyter notebook on AWS SageMaker provides several benefits. Not only can you pay just for what you use and turn it off when you’re done, but the ability to easily rerun portions of your code without having to reload the model saves time during iterative development. But do beware! If you forget to stop your notebook instances, the charges can quickly add up.

While this method is relatively convenient, there are some considerations you must take into account. Firstly, loading the model can take up to 20 minutes even on a high-performance GPU, making this process somewhat time-consuming. Also, the cost is a factor to consider, as the running cost is at least $4 per hour. You’ll need to run MPT-7B on at least a p3.2xlarge instance; anything smaller does not seem feasible. If you opt for EC2 instead of SageMaker, you’ll have to ask AWS for permission to use a p3.2xlarge instance.

In the next sections I’ll take you step-by-step through how to run the MPT-7B model in your very own SageMaker jupyter notebook:

Step 1 - Open the SageMaker Console

Fire up the AWS Console and search for SageMaker:

Search for SageMaker. Image credit: Author

Step 2 - Create a notebook instance

From the left-side menu, select Notebook->Notebook instances:

Notebook instances. Image credit: Author

Click the Create notebook instance button:

Create notebook instance. Image credit: Author

Specify an instance name. Choose the instance type m1.p3.2xlarge. Unfortunately, it seems that an instance as powerful as m1.p3.2xlarge is required, or else your instance may run out of memory or take an excessive amount of time to respond to even the simplest questions. However, please note that this instance will cost approximately $4/hr, so it’s important to monitor your usage carefully.

Specify notebook instance details. Image credit: Author

Create a new IAM role:

Create a new role. Image credit: Author

If your test environment doesn’t have any particularly sensitive data in it then you can grant access to Any S3 bucket. Otherwise, you’ll need to be more explicit.

Specify S3 access. Image credit: Author

Click the Create notebook instance button:

Create notebook instance. Image credit: Author

The notebook will then be in a Pending status. This will likely last for about 10 minutes:

Pending notebook instance. Image credit: Author

In the meantime, we’ll download a notebook so that we can upload it after the AWS SageMaker instance has finished provisioning.

Step 3 - Download the notebook

Head on over to the notebook at MPT-7B on AWS SageMaker.ipynb and download it:

The notebook on Google Colab. Image credit: Author

Download the notebook. Image credit: Author

In this notebook you’ll notice two main code blocks. The first block loads the MPT-7B tokenizer and model:

from torch import cuda, bfloat16
from transformers import AutoTokenizer, AutoModelForCausalLM, AutoConfig

device = f'cuda:{cuda.current_device()}' if cuda.is_available() else 'cpu'

tokenizer = AutoTokenizer.from_pretrained("mosaicml/mpt-7b-chat",
 trust_remote_code=True)

config={"init_device": "meta"}
model = AutoModelForCausalLM.from_pretrained("mosaicml/mpt-7b-chat",
 trust_remote_code=True,
 config=config,
 torch_dtype=bfloat16)

print(f"device={device}")
print('model loaded')

The tokenizer is used to encode the question sent to the model and decode the response from the model. Additionally, we obtain the device specification for our GPU so that we can configure the model to utilize it later:

import time
from IPython.display import Markdown
import torch
from transformers import StoppingCriteria, StoppingCriteriaList

# mtp-7b is trained to add "<|endoftext|>" at the end of generations
stop_token_ids = [tokenizer.eos_token_id]

# Define custom stopping criteria object.
# Source: https://github.com/pinecone-io/examples/blob/master/generation/llm-field-guide/mpt-7b/mpt-7b-huggingface-langchain.ipynb
class StopOnTokens(StoppingCriteria):
def __call__ (self, input_ids: torch.LongTensor,scores: torch.FloatTensor,
 **kwargs) -> bool:
 for stop_id in stop_token_ids:
 if input_ids[0][-1] == stop_id:
 return True
 return False

stopping_criteria = StoppingCriteriaList([StopOnTokens()])

def ask_question(question, max_length=100):
 start_time = time.time()

 # Encode the question
 input_ids = tokenizer.encode(question, return_tensors='pt')

 # Use the GPU
 input_ids = input_ids.to(device)

 # Generate a response
 output = model.generate(
 input_ids,
 max_new_tokens=max_length,
 temperature=0.9,
 stopping_criteria=stopping_criteria
 )

 # Decode the response
 response = tokenizer.decode(output[:, input_ids.shape[-1]:][0],
 skip_special_tokens=True)

 end_time = time.time()
 duration = end_time - start_time

 display(Markdown(response))

 print("Function duration:", duration, "seconds")

Note the use of stopping_critera, which is needed or else the model will just start babbling on, even after it has answered our question.

See model generate parameters if you want to explore the different options.

Now, let’s upload this notebook to SageMaker.

Step 4 - Upload the notebook

Hopefully by this time your SageMaker notebook instance has finished being provisioned. When it has, click the Open Jupyter link:

Open Jupyter. Image credit: Author

Then, click the Upload button in the top-right corner of your screen and select the notebook that you just downloaded:

Upload the notebook. Image credit: Author

Set the kernel to conda_python3:

Set the kernel. Image credit: Author

Step 5 - Run the notebook

Select Cell -> Run All:

Run all cells. Image credit: Author

An hourglass logo will then appear in the browser tab:

Wait for notebook. Image credit: Author

You’ll then need to wait about 10 minutes for the model to be downloaded:

Download model. Image credit: Author

After it runs, you’ll see the answer to the question Explain to me the difference between nuclear fission and fusion:

Explain to me the difference between nuclear fission and fusion. Image credit: Author

Since the model and tokenizer have already been loaded above, you can simply modify the ask_question code block and click the Run button to ask any other questions. This will save you from spending 10 minutes each time you want to test a new question.

What is the capital of France?. Image credit: Author

Step 6 - Stop the notebook

As soon as you have finished testing the model, you’ll want to head back to your list of notebook instances and stop it. If you don’t, $4/hr will add up very quickly 💸

Stop the notebook. Image credit: Author

Performance Comparison

In terms of performance, my preliminary tests suggest that MPT-7B’s results might not be as good as ChatGPT’s. It does a decent job answering questions like What is the capital of France?, Explain to me the difference between nuclear fission and fusion, and Write python code that converts a csv into pdf. But for questions like_What is the capital of Belize?_ it fails pretty horribly:

What is the capital of Belize? Image credit: Author

I am currently collecting more data and will conduct a comprehensive comparative analysis in a follow-up blog post. In that post, I will compare the question and answer performance of MPT-7B, MPT-30B, Falcon-40b, and ChatGPT using actual conversation history.

From Testing to Production

Once you’re ready to transition from testing to production, SageMaker offers an additional benefit - the ability to create an endpoint for your model. With SageMaker, you can auto-scale based on demand to the endpoint, optimizing your resources.

Additional Tips

Be mindful that it’s easy for your process to get forked while running in a Jupyter notebook and run out of memory. If this happens, simply shut down the kernel and run all the commands again.

If you’re curious about running this model on a platform other than AWS, Google Colab Pro is another viable option at $9/month. However, based on our testing, we found that we exhausted the provided credits within just a few hours. 😳

Another challenge you may face is the inability to utilize the Triton optimization on SageMaker due to a CUDA version incompatibility. Unfortunately, AWS’s current P3 instances do not include a recent CUDA version. Therefore, if you wish to utilize the Triton optimization, you will need to create an EC2 container with command line access. However, it’s important to note that you will also require special permission from AWS Support to run an instance with 8 VCPUs. In a future post, I will provide a detailed guide on how to integrate Triton and utilize a more cost-effective GPU cloud provider, such as Lambda Labs.

Final Thoughts

While MosaicML’s MPT-7B offers a viable alternative to OpenAI’s ChatGPT, it presents its own set of challenges. Running the model can be time-consuming, expensive, and the available documentation is lacking. However, the ability to keep the model in-house and protect your data from being exposed to OpenAI can be compelling for certain use cases.

SageMaker offers great convenience for quickly testing the model and provides the flexibility to transition to production when you’re ready. Whether you’re just starting with MPT-7B or have been using it for a while, we hope this guide has provided valuable insights.

Stay tuned for our next blog post, where we’ll delve deeper into the performance comparisons between MPT-7B, MPT-30B, Falcon-40b, and ChatGPT.

See the following links if you are interested in learning more about MPT-7B or its larger variant, MPT-30B.

And remember, whether you’re working with ChatGPT or MPT-7B, the key is to ensure your use case is served without compromising data privacy and cost-effectiveness. Happy tinkering!

Want a Turn-Key Solution for Using MPT or ChatGPT with Your Data?

At MindfulDataAI.com, we offer ChatGPT for businesses. If you’re interested in leveraging ChatGPT, MPT, or other models with your company’s data, please get in touch with us.

My Journey with ChatGPT for Business: Unleashing the Power of AI for Software Engineers

Geoff Cox — Mon, 10 Jul 2023 13:46:28 +0000

Introduction

As a software engineer, I'm constantly searching for tools and technologies that can revolutionize the way I work. Today, I want to share a prototype app: ChatGPT for Business. I believe if we follow through on building this, it has the potential to transform our industry. Join me as I take you through my experiences, challenges, and the immense possibilities that lie ahead.

From Crypto Trading to AI-Assisted Productivity

The story begins with me leaving my Director of Engineering job at a fast-growing startup that was about to be acquired. I stumbled into the world of crypto trading after attending a web3 event in early 2022, where I heard about arbitrage opportunities. Eager to explore further, I began experimenting with free bots at Pionex and discovered the passive income potential of trading bots. That's when everything changed.

Driven by curiosity, I reverse-engineered a promising bot, conducted extensive backtesting using historical price data from Binance, and optimized its parameters using genetic mutation techniques. Deploying it on AWS, I witnessed consistent daily returns, achieving a remarkable 18% annual return on a sizable investment. But as the market landscape shifted, I realized that luck played a significant role in my success, and a long-term buy-and-hold strategy would yield better results. When the SEC's lawsuit against Binance and Coinbase loomed, I secured my crypto in wallets and closed my trading operations.

Discovering the Power of ChatGPT

During this transformative period, I found myself increasingly relying on ChatGPT, a powerful generative AI model. What began as a tool for trivial questions quickly evolved into something much more. ChatGPT gradually replaced around 80% of my Google searches, becoming indispensable for coding technical analysis algorithms, troubleshooting Docker configurations, proofreading emails, and even generating pitch deck outlines. I realized that ChatGPT had become my go-to resource, surpassing Google searches in many aspects. However, it was missing the capability to search my private data and access any information released after 2021.

The Spark of Innovation

One day, over coffee, a friend shared an interesting challenge: scraping information from multiple mom-and-pop websites to create a comprehensive activities-for-kids marketplace. It was at that moment that I had a revelation—what if ChatGPT could help? Instead of writing custom web scrapers for each site, could ChatGPT extract the desired information? Intrigued, I stumbled upon an informative post titled "How to Train an AI Chatbot With Custom Knowledge Base Using ChatGPT API". Inspired by its possibilities, I pointed ChatGPT to a Boys and Girls Club website, asking questions about activities in Seattle, their schedules, and costs. To my delight, the results were remarkably accurate, and the cost of using OpenAI’s API was surprisingly affordable.

Unlocking the Power of ChatGPT for Business

Fueled by this success, I couldn't help but wonder how I could further enhance my productivity using ChatGPT. To my surprise, I discovered that very few business products allowed users to leverage ChatGPT with their own data. Existing solutions fell short in terms of integrations, customization, and scalability. MyAskAI catered to individuals with one-off files and only allowed two questions per message thread. AiSera seemed more focused on building chatbots with custom data. CustomGPT lacked the necessary integrations with data sources like Google Drive, Salesforce, and Confluence. Gerev.ai, although open-source and appealing, failed to index a 5-page PDF on a brand-new MacBook Pro. Klu had an impressive turnkey integration with sources like Google Drive, but its "Beta AI" spun endlessly when asked a basic question about a single 5-page PDF file.

This realization sparked a vision in me—I wanted to build a product specifically designed to empower all businesses, not just the ones with the funds to train new AI models, with ChatGPT's capabilities. Excited by the possibilities, I reached out to my friend Arty, a talented engineer, and we began brainstorming ideas together.

Seeking Validation and Refining the Vision

To refine our concept and gain support, we sought guidance from a local venture capitalist (VC). We were told that our idea might be too broad and that we should instead focus on a specific industry. However, we remained unconvinced that there isn't a place for a general-purpose ChatGPT tool for businesses. Therefore, we decided to rely on data to guide us in the right direction. Undeterred, we developed a playbook, crafted surveys, optimized keywords for SEO and advertising, and created compelling website mock-ups. But we knew that true validation would come from our fellow software engineers—the very people who could benefit from ChatGPT for Business—YOU!

Imagine, if you will…

A ChatGPT-powered teammate, seamlessly integrated with your data sources and the web, providing real-time insights, securely linked to source documents, and offering turnkey integrations with your company's tools and systems. We envision automated document synchronization, real-time web searches, single sign-on (SSO) integration, access controls, conversation sharing, onboarding and training, and robust security and compliance measures.

Imagine being able get the status of a Jira ticket in real-time:

Or, what about keeping tabs on the competition?

Or, even synthesizing reports from raw data. Sharing and explaining insights (like a goalie, answering questions before you need to!). The more documents you share with the agent, the more time it will save for you as you could even let this AI agent handle basic questions on your behalf!

The possibilities are endless.

Join the Journey

And now, dear reader, we invite you to embark on this incredible journey with us. We think we’re on to something, but we want your thoughts! Your insights and feedback are essential as we shape the future of ChatGPT for Business. Together, we can unleash the potential of ChatGPT, transforming the way we work, and empowering ourselves and our teams with a tool that will revolutionize communication. Please take a moment to share your thoughts by participating in our quick survey. Additionally, you can also register for the beta release.

Conclusion

Our journey with ChatGPT for Business has been one of passion, innovation, and relentless pursuit. We have witnessed firsthand the immense potential of AI in our field, and we’re determined to harness that power to create a tool that will revolutionize software engineering. Thank you for reading, and we eagerly await your input as we shape the future of ChatGPT for Business together.

Run a Free VPN Server on AWS (2022 Edition)

Geoff Cox — Tue, 15 Nov 2022 18:48:50 +0000

This tutorial will take you through the steps of setting up an EC2 instance running OpenVPN server. It will then cover how to grant and revoke access to your VPN server.

Image credit: Privecstasy on Unsplash

AWS has an awesome firewall built into its core services, which can easily be used to make sure that only certain ports are open to the outside world. One extra step that we can take is to run a VPN server that serves as the gateway to our protected AWS resource, e.g. EC2, RDS, etc… We can then shutdown direct access to our AWS resources just by revoking access via our VPN server. This is very useful if you need to revoke access for a former employee.

AWS now offers a managed VPN Service, but this service costs at least $72 a month and is even more expensive when your VPN serves a lot of traffic. A lot of smaller organizations don’t need all the features of the managed service and instead can run their own VPN server for just the cost of an EC2 instance. You can even use the free tier so that you don’t have to pay anything for the EC2 instance for the 1st year.

Step 1 — Create the VPN Security Group

Overview: security groups allow your servers to communicate with each other in a private cloud while exposing specific ports to the world. We are going to create a security group to allow VPN access to our VPN server. We will assume that all your other AWS resources are members of the default security group and that the default security group does not allow access from the outside world.

From the EC2 dashboard, click Security Groups:

Image credit: Author

Click Create security group:

Image credit: Author

Enter a name and description of vpn and specify the following inbound rules on ports 22, 443, 943, and 1194. Note: the protocol for port 1194 is UDP.

Image credit: Author

Note: if the IP addresses that your team uses are static then you can add yet another layer of security by specifying an IP address range in the Source of your rules. However, you’ll want to leave the Source open to anywhere if you want your team to be able to connect from any IP as they may be working from a hotel, home, cafe, etc…

Step 2 — Create the EC2 Instance

Return to the EC2 Dashboard and then click Launch instances:

Image credit: Author

Select Ubuntu (you can of course select almost any other OS that runs OpenVPN, but this tutorial is tailored to Ubuntu)

Image credit: Author

Select t2.micro:

Image credit: Author

Note: you can use a nano instance instead of a micro instance, but nano instances are not eligible for the free tier.

In the Network settings section, select your default VPC and disable the auto-assign public IP option. Then, select both your default security group and the security group that you created above for the VPN.

Image credit: Author

Click Launch instance

Step 3 — Disable Source/Destination Check

From the list of instances, select the VPN instance and then Actions -> Networking -> Change source/destination check from the drop down menu.

Image credit: Author

Select Stop and click Save. This is needed as otherwise, your VPN server will not be able to connect to your other AWS resources.

Image credit: Author

Step 4 — Create an Elastic IP Address

Overview: when an EC2 instance is stopped and restarted, the Public IP address changes. We want the IP address of our VPN server to remain static so we’ll use an Elastic IP Address.

From the E2C Dashboard, select Elastic IPs:

Image credit: Author

Click Allocate Elastic IP address:

Image credit: Author

Make a note of your new Elastic IP address as this will be the Public IP Address of your VPN server. We will refer to this address later as PUBLIC-IP-OF-VPN-SERVER.

Select the IP address you just created and click Associate Elastic IP address:

Image credit: Author

Then, select the Elastic IP and click Associate address from the drop down menu.

Select the EC2 instance you created above and click Associate:

Image credit: Author

Step 5— Install and Configure the OpenVPN Server

SSH into your VPN server:

$ ssh ubuntu@PUBLIC-IP-OF-VPN-SERVER

Download our helper scripts and set up a default config:

$ git clone https://github.com/redgeoff/openvpn-server-vagrant
$ cd openvpn-server-vagrant
$ cp config-default.sh config.sh

Edit config.sh and enter in your configuration. Note: PUBLIC_IP should be equal to the Elastic IP Address that you created above.

$ nano config.sh

Switch to root:

$ sudo su -

You’ll now update Ubuntu. Note: you will be prompted several times and when you do, just press the Enter key.

$ /home/ubuntu/openvpn-server-vagrant/ubuntu.sh

Now, install OpenVPN. Note: you will be prompted several times and when you do, just press the Enter key.

$ /home/ubuntu/openvpn-server-vagrant/openvpn.sh

At this point, the OpenVPN server is running.

Step 6 — Add the Route

Routes must be added to the server so that your team’s clients know which traffic to route to the VPN server.

You can determine the proper subnet by returning to your list of EC2 instances, clicking on a target instance and identifying the Private IP.

Image credit: Author

Your network will be the first 2 parts of the Private IP appended with zeros, e.g. 172.31.0.0

On the VPN server edit /etc/openvpn/server/server.conf and add something like the following:

push "route 172.31.0.0 255.255.0.0"

Then, restart the VPN server with:

$ systemctl restart openvpn-server@server.service

Step 7 — Grant Access to Your VPN

Note: We assume that you are still SSH’d into the VPN and logged in as root.

Run the following command and be sure to replace client below with a unique name for your user/client.

$ /home/ubuntu/openvpn-server-vagrant/add-client.sh client

You’ll then find a configuration file at

~/client-configs/files/client-name.ovpn

You will want to provide this file to the individual on your team who will be connecting to your VPN. SCP is handy for downloading this .ovpn file from your VPN server.

Your team can use one of various VPN clients such as Tunnelblick (OS X) and OpenVPN (Linux, iOS, Android and Windows). After installing one of these clients, they should be able to set up the VPN config just by double clicking on the .ovpn file.

Note: once connected to the VPN, your users will want to use the Private IPs of your AWS resources. You may want to use nslookup to lookup the IP address from the AWS custom domain names provided by AWS. You’ll probably want to use Route 53 to create subdomain records that route to the Private IPs.

Step 8 — Revoke Access to Your VPN

Note: We assume that you are still SSH’d into the VPN and logged in as root.

Run the following command and be sure to replace client below with a unique name for your user/client.

$ /home/ubuntu/openvpn-server-vagrant/revoke-full.sh client

Troubleshooting

If your VPN client reports a TLS handshake failed error then this is most likely because your VPN security group (Step 1) is incorrect. Make sure that you have the correct ports and protocols specified — a common problem is not specifying UDP for port 1194.

Grocery Hacking: Automate Your Grocery Shopping Using the Robot Framework

Geoff Cox — Wed, 09 Mar 2022 15:32:52 +0000

I used Robotic Process Automation (RPA) to automate my grocery shopping so that I can spend less time shopping.

Image credit: Author

My family and I are lucky enough to live in a Seattle neighborhood with 7 supermarkets that are each within walking distance. So, it wasn’t until the pandemic hit, and we were wishing to minimize our contact with others, that we started experimenting with grocery delivery. Like many others, we started working from home and we could now be available to receive deliveries at any time, it just seemed like a no-brainer to give grocery delivery a try.

At the time, the first grocery store in our area to offer delivery, which wasn’t the megacorp that is Amazon, was Fred Meyer, through a partnership with Instacart. Our initial experience was pretty good and we figured that the delivery cost of an extra 15% of the total bill was worth the extra safety of a contact-free delivery. We used Fred Meyer’s delivery service weekly for about 9 months and during this time we learned that it had its shortcomings. We found that Fred Meyer’s online inventory was often out of sync with its actual inventory, which meant that we could easily receive substitutions, or items would just be missing from the delivery as they were out of stock. It became pretty frustrating to have to physically go to the store after a delivery just to pick up the 20% of the items that were not delivered.

Fortunately, in mid 2021, our regular pre-COVID supermarket, Safeway, started offering a delivery service. With a delivery fee of just $3.95, the products we were used to, and substantial savings, this service seemed promising. And, indeed it was — so much so that we’ve been using it for the last 9 months and plan to continue to use the service.

COVID is still very much a thing, but most of my family is vaccinated (we’re still waiting for a vaccine for my 2-year old) and with masks we find it an acceptable risk to shop indoors in well-ventilated spaces. So what keeps us coming back to use a grocery delivery service? Convenience and cost. As much as going to the supermarket with my kids can be fun, as we get to buy a whole bunch of yummy food and we’re all a bunch of goofballs, it’s also tough for my kids to stay focused for 45 minutes during a big shop. This means they’ll be running all over the place and are going to slow down the shopping to a crawling speed (see what I did there?). Instead, my wife and I can do some meal planning on Sunday night, order the groceries, and then have them delivered for just $3.95!

What’s more? Safeway delivery drivers are employees of the store so Safeway doesn’t ask for a tip. With competing services, like those from Instacart, you can easily fork over an additional 10–20% of your bill for a tip and delivery fee. That adds up when you talk about a large shop. Moreover, our Safeway shopping actually leads to significant savings at the gas pump, which is a nice added bonus!

OK, enough background already… let’s talk about how I used to do my shopping before I automated it.

How I used to do my shopping

How I used to do my grocery shopping. Image credit: Author

Here’s how it would go:

During the week, my wife and I (and sometimes our kids) would ask Alexa to add an item to our shopping list. This is easy to do if you just add the item to the list as soon as you know you’re going to need more of it.
There is then an integration, provided by Todoist, that syncs the items from Alexa to Todoist. We use Todoist instead of the Alexa app to manage our list, as during the early days of Alexa, we found that it was much easier for my wife and I to share lists via Todoist. (This may no longer be the case, but Todoist works well for us and is free, so we keep using it).
On Sunday night, my wife and I sit down and do some meal planning, survey the refrigerator and cupboard, and then add items to the list via Alexa.
I fire up my computer and load Todoist and Safeway.com in different browser tabs.
I then go one by one through the items in my Todoist list. First, I look at the previously purchased items on Safeway, AKA Buy It Again items and if there is a match, I add the item to my cart. Safeway does a pretty good job at recalling the last 300 or so items that I’ve purchased and there is a darn good chance that my item will be in this list.
If I’ve never purchased the item before or my previous purchase is out of stock, I use the Safeway search feature to search for the item and then add it to the cart.
I mark the item as completed on my Todolist
I check out my Safeway cart and schedule a delivery

What’s important to note is that the repetitive tasks of steps 4–7 can take up to 30 minutes. You may ask how this is possible? Well, each item takes about 1 minute to process manually and I can easily have about 30–40 items on my list. This is partly because browsing the Safeway website is a bit slow. On many occasions, I’ve thought that it is a bummer that I have to spend 30 minutes each week ordering my groceries. After all, I’m usually ordering the same stuff and it doesn’t require much intelligence to do this shopping. There must be a better way, right?

You may be thinking:

OK, but doesn’t Alexa already provide a way to shop for products? Surely, this is why Amazon created Alexa?

Actually, it does! Fairly recently, the Alexa app started allowing you to Shop your list. You simply add items to your Alexa list and then use the Alexa app to select from a list of Fresh or Whole Foods products:

Alexa’s Shop Your List feature. Image credit: Author

This interface is pretty slick, but there’s one problem. My wife says:

I’ve already thought about boycotting Amazon as they aren’t the most socially responsible company. We already buy too much on Amazon and we need to support other grocery stores.

As much as a I love the convenience, speed, and prices on Amazon, she’s right! So, for now, we’ll stick to shopping at Safeway. Sure, Safeway (a subsidiary of Albertsons) is one of the 5 largest grocery chains in the US, but at least it isn’t Amazon.

I was still left wondering if Safeway or Amazon is cheaper. So, I did a quick cost comparison between Safeway and Amazon Fresh by building a shopping cart of 32 similar items on each of their websites. In doing this, I found that their pricing was roughly the same for my basket of products, e.g. around $150:

Safeway vs Amazon Fresh pricing. Image credit: Author

Knowing that we wanted to continue using Safeway for grocery delivery and Alexa for our shopping list, I began to wonder how I could automate my shopping? Enter: Robotic Process Automation (RPA)

Robot Process Automation (RPA)

If you haven’t heard of it, RPA is simply a way of automating repetitive tasks, that are normally done by a human, with a software robot. These are often tasks done in Graphical User Interfaces (GUIs) like web browsers, MS Excel, email clients, etc…

I had been casually reading about RPA for the past few years and even pitched it to my non-technical leadership peers at work, when they were looking for a way to automate operational processes without distracting our already-spread-thin engineers. It seemed like the perfect opportunity for me to jump in and build an RPA bot.

After doing some quick research, I quickly learned that there are a number of expensive proprietary RPA vendors like UIPath and Automation Anywhere. I don’t doubt that these systems are worth what you pay for them as they can ultimately reduce your labor costs. But, there are also a few open-source solutions like Robot Framework and TagUI, and as a frequent contributor to open source, I’m a big fan of open source.

Currently, Robot Framework has the most GitHub stars and is supported by an extensive community. So, I fired up a quick Getting Started with Robot Framework Tutorial and found it pretty straightforward to build a bot that could navigate a website. I found it very cool that Robot Framework is a layer on top of Python and therefore, it is easy to insert custom Python code into the bot. And, what I found particularly cool is that there are community provided libraries for all types of things.

Goals of the bot

Before we get into how I built my bot, let’s discuss my goals. As a software engineer, I knew I could craft some code that fired off API requests to the Todoist and Safeway backends. However, this would sidestep my desire to play with the Robot Framework. Instead, I opted to use the SeleniumLibrary with Robot Framework, which under the hood would allow me to build a bot that uses Selenium to control a browser in a way that is very similar to how I would otherwise manually use a browser to do my grocery shopping.

Later, I’ll mention some of the pitfalls I found with RPA and why in the future, I would choose to make an API-based bot instead of a GUI-based bot. In fact, Todoist appears to have a supported developer API. Safeway/Albertsons does not appear to have a developer API, but its website does make backend calls and therefore you could reverse engineer the backend API.

My other goal was to simplify the deployment architecture by not introducing another datastore. In other words, I wanted my bot to just use the data in Todoist and Safeway and not have to store anything in a separate datastore. Ultimately, I’m able to do my grocery shopping without writing down anything extra so why can’t the bot function without an additional database?

Building the bot

In the next few sections, I’m going to be walking through excerpts of my bot code. Please feel free to download the full source code at github.com/redgeoff/grocery-shopping-bot

If you are new to the Robot Framework, please read through Robot Framework Quick Start: UI Automation Tutorial.

Reusable keywords

The Robot Framework refers to your typical programming language function as a keyword. We’re going to define some keywords so that we can abstract some of the rough edges that you’ll encounter when working with a website in a browser. In particular, websites are beasts that load many elements asynchronously, which means that elements may not yet be visible or enabled. Bots are inherently much faster than humans so if you aren’t careful, it is easy for your bot to error out from the UI not yet being ready or because other race conditions occur. Of course, you could insert Sleep statements in your code, which would simulate the timing that a human would encounter, but this is an anti-pattern because:

Adding Sleep statements will make your bot much slower
Timing can vary based on the network, the backend servers, and even your compute power.

Instead, we’ll create some helper keywords that will wait for elements to be visible and enabled, and even introduce a retry loop for when the page takes an exorbitant amount of time to load or an unexpected error occurs.

Our first keyword will be Wait Until Element Ready:

Wait Until Element Ready Inner
 [Arguments] ${locator} ${timeout}=None

 Wait Until Page Contains Element ${locator} ${timeout}
 Wait Until Element Is Visible ${locator}
 Wait Until Element Is Enabled ${locator}
 Scroll Element Into View ${locator}

 Wait Until Element Is Visible ${locator}
 Wait Until Element Is Enabled ${locator}
 Scroll Element Into View ${locator}

Wait Until Element Ready
 [Arguments] ${locator} ${timeout}=None
 Wait Until Keyword Succeeds 3 1s Wait Until Element Ready Inner ${locator} ${timeout}

Next, we’ll define Click Element When Ready, which uses Wait Until Element Ready and then clicks the element.

Click Element When Ready Inner
 [Arguments] ${locator}
 Wait Until Element Ready ${locator}
 ${element}= Get WebElement ${locator}
 Click Element ${element}
 [Return] ${element}

Click Element When Ready
 [Arguments] ${locator}
 Wait Until Keyword Succeeds 3 1s Click Element When Ready Inner ${locator}

To make it so that we can resume the same browser session, which will help us to avoid Todoist sending us an email each time we log in, we’ll borrow a trick from this post.

Open Browser Profiled
 [Arguments] ${url}
 ${options}= Evaluate sys.modules['selenium.webdriver'].ChromeOptions() sys, selenium.webdriver
 Call Method ${options} add_argument --user-data-dir\=${USER_DATA_PATH}
 Create WebDriver Chrome chrome_options=${options}
 Go To ${url}

Finally, using a trick from this post, we’ll define a way to get child elements. This will help us to do things like get a list of groceries and then pluck out the nested attributes.

Get Child WebElements
 [Arguments] ${locator} ${child_xpath}

 ${element} Get WebElement ${locator}
 ${children} Call Method
 ... ${element}
 ... find_elements
 ... by=xpath value=.${child_xpath}

 [Return] ${children}

The actual bot, has a number of other reusable keywords, but they are omitted from this post for brevity.

We’ve now abstracted some of the less-fun stuff, let’s jump into making the bot do it’s thang!

Get the list from Todoist

The first thing we want the bot to do is log in to Todoist, select our desired list, and then save the list to memory. The Todoist page that lists our items, looks like this:

Todoist shopping list. Image credit: Author

To avoid making extra log-ins and triggering emails from Todoist, we’ll make it so that the bot only logs in when we aren’t already logged in. This is particularly helpful during testing when you need to run the script MANY times to find all the race conditions that result from navigating the website in the browser. Naturally, you could log in (and log out) each time the script runs if you’d prefer.

***Variables***
${TODOIST_TODAY} https://todoist.com/app/today
${TODOIST_TODAY_TITLE} Today: Todoist

***Keywords***

Log In To Todoist
 [Arguments] ${email} ${password}
 Wait Until Element Ready email
 Wait Until Element Ready password
 Input Text email ${email}
 Input Password password ${password}
 Click Button Log in
 Wait Until Page Contains Today timeout=30s
 Update Settings If Needed

Log In Or Open Todoist Inner
 [Arguments] ${email} ${password}
 Go To ${TODOIST_TODAY}
 Wait Until Page Contains Element xpath=//*[.='Log in' or .='Today'] # Wait for page to settle
 ${logged_in}= Run Keyword And Return Status Title Should Be ${TODOIST_TODAY_TITLE}
 IF not ${logged_in}
 Log In To Todoist ${email} ${password}
 END

Log In Or Open Todoist
 [Arguments] ${email} ${password}
 Wait Until Keyword Succeeds 5x 2 sec Log In Or Open Todoist Inner ${email} ${password}

Select Todoist List
 [Arguments] ${list_name}
 Update Settings If Needed
 Click Element When Ready xpath:(.//span[contains(., '${list_name}')])[1]

Get Todoist List Items
 @{locators}= Get WebElements xpath=//li[contains(@class,'task_list_item')]
 ${items}= Create List
 FOR ${locator} IN @{locators}
 ${list_item} Get Child WebElements ${locator} //div[contains(@class,'task_list_item__content')]
 ${name}= Get Text ${list_item}
 ${id}= Get Element Attribute ${locator} data-item-id
 ${item}= Create Dictionary id=${id} name=${name}
 Append To List ${items} ${item}
 END
 [Return] ${items}

If you take a closer look at the Get Todoist List Items keyword, you’ll see that we are using an XPath query to get all the li elements with a task_list_item class. And then we pluck the list_item child element so that we can get the item name and id. (Yes, this is how Todoist identifies the items in their HTML and if you think this looks brittle… well, yup)

You can find the complete code at todoist.robot.

Buy it again

After we get the items from Todoist, the next step is to add corresponding items to our Safeway cart. Now, we could just go to the Safeway search page and search for the item, but the search page orders by Best Match and doesn’t allow you to order by the most recently purchased items._

Safeway’s search page. Image credit: Author

The problem is that sometimes we buy items just because others are out of stock, even though we have a preference for the items that are out of stock. Instead, I want to first give preference to the items that we frequently purchase. This way, we’ll always attempt to first grab our preferred item, even if it is not in stock until the next time we shop. Sure as a human, I can simply pick what I want from the list, but the bot isn’t that smart. For the bot, we can do this by going to the Buy It Again page and sorting by Frequently Purchased:

Sort by Frequently Purchased. Image credit: Author

Our bot will then iterate through the list of items that we got from Todoist and look for the first match in Safeway’s Buy It Again list. It will then add that item to the Safeway cart.

***Variables***
${SAFEWAY_HOMEPAGE} https://www.safeway.com
${SAFEWAY_BUY_IT_AGAIN_URL} https://www.safeway.com/shop/purchases/buy-it-again.html

***Keywords***

Capture List
 @{locators}= Get WebElements xpath=//product-item-v2
 ${items}= Create List
 FOR ${locator} IN @{locators}
 ${title} Get Child WebElements ${locator} //*[contains(@class,'product-title')]
 ${name}= Get Text ${title}
 ${id}= Get Element Attribute ${title} data-bpn
 ${url}= Get Element Attribute ${title} href
 ${add_button} Get Child WebElements ${locator} //*[@data-qa='addbutton']
 ${available}= Run Keyword And Return Status Page Should Contain Element ${add_button}
 ${remove_button} Get Child WebElements ${locator} //*[@data-qa='prdctdcrmntr']
 ${in_cart}= Run Keyword And Return Status Page Should Contain Element ${remove_button}
 ${item}= Create Dictionary id=${id} name=${name} url=${url} available=${available} in_cart=${in_cart}
 Append To List ${items} ${item}
 END
 [Return] ${items}

Sort Buy It Again Items
 Click Element When Ready //*[@data-qa='srt-ptns-dflt']
 Click Link Frequently Purchased

Get Buy It Again Items
 [Arguments] ${max_pages}
 Go To ${SAFEWAY_BUY_IT_AGAIN_URL}
 Dismiss Popups If Needed
 Sort Buy It Again Items
 Load More Items ${max_pages}
 ${items}= Capture List
 [Return] ${items}

Load More Items
 [Arguments] ${max_pages}
 ${result} ${condition}= Run Keyword And Ignore Error Wait Until Page Contains Load more error=false
 FOR ${i} IN RANGE 1 ${max_pages}
 Exit For Loop If '${result}'=='FAIL'
 Click Button Load more
 ${result} ${condition}= Run Keyword And Ignore Error Wait Until Page Contains Load more error=false
 END

Add To Cart
 [Arguments] ${item_id}

 Wait Until Element Ready //*[@id='addButton_${item_id}']

 # Safeway requires a mouse over before the item can be added to the cart
 Set Focus To Element //*[@id='addButton_${item_id}']
 Mouse Over //*[@id='addButton_${item_id}']

 Click Element When Ready //*[@id='addButton_${item_id}']

 # Wait for loading spinner to disappear
 Wait Until Page Does Not Contain Element //*[@class='quantity-loading']

 # Wait enough time for cart addition to settle
 Wait Until Page Does Not Contain Element //*[@id='addButton_${item_id}']
 Wait Until Element Ready //*[@id='dec_qtyInfo_${item_id}'] timeout=10s

Buy It Again
 [Arguments] ${items_to_buy} ${max_pages}
 ${buy_it_again_items}= Get Buy It Again Items ${max_pages}
 ${items_in_cart}= Create List
 ${items_not_found}= Create List
 FOR ${item_to_buy} IN @{items_to_buy}
 Set Local Variable ${added_to_cart} False
 FOR ${buy_it_again_item} IN @{buy_it_again_items}
 ${name_matches}= Run Keyword And Return Status Should Contain ${buy_it_again_item.name} ${item_to_buy.name} ignore_case=True
 IF '${name_matches}'=='True' and ('${buy_it_again_item.available}'=='True' or '${buy_it_again_item.in_cart}'=='True')
 # Look up the availability again as we may have just added the item to the cart so
 # we cannot use ${buy_it_again_item.available}
 ${available}= Run Keyword And Return Status Page Should Contain Element //*[@id='addButton_${buy_it_again_item.id}']

 # If the item is already in the cart then we just ignore it as this allows us to
 # retry the script without it adding any additional items to the cart
 IF '${available}'=='True'
 Log To Console Adding to cart: ${buy_it_again_item.name} (Todoist item name: ${item_to_buy.name})
 Wait Until Keyword Succeeds 10x 2 sec Add To Cart ${buy_it_again_item.id}
 ELSE
 Log To Console Already in cart: ${buy_it_again_item.name} (Todoist item name: ${item_to_buy.name})
 END
 Set To Dictionary ${item_to_buy} name_in_store=${buy_it_again_item.name}
 Append To List ${items_in_cart} ${item_to_buy}
 Set Local Variable ${added_to_cart} True
 Exit For Loop
 END
 END
 IF '${added_to_cart}'=='False'
 Append To List ${items_not_found} ${item_to_buy}
 END
 END

 [Return] ${items_in_cart} ${items_not_found} ${buy_it_again_items}

Did you notice the fun stuff happening in Add To Cart? Well, the Safeway website requires that you mouse over an item and then click it before it can be added to the cart. As for the Wait Until Page Does Not Contain Element, the website flashes a loading icon when an item is added to the cart and we want to wait until this icon is no longer displayed as this confirms that the item is in our cart. (Pro Tip: to inspect a disappearing element in Chrome, open the Sources tab, and hit F8 to stop the browser execution and keep the icon displayed)

You can view the complete Safeway robot code at safeway.robot.

Search for items not purchased before

If you are like me, and most of your grocery shopping includes the same 300 or so items, the bot will have completed almost your entire list just by shopping from the Buy It Again (recently purchased) page. The remaining handful of items on your list will likely be those that you have never purchased before, e.g. you are trying a new recipe or your typical brand is out of stock.

There are also other reasons why you’ll have remaining items in your list:

Safeway truncates long product names, e.g. Signature Care Wipes Sensitive…
The formatting of the items in your list may not match, e.g. nutrigrain vs nutri-grain
The items in your list have names that don’t match the name of the Safeway item. As you might expect, Safeway associates hidden keywords with each product. For example, you may have pepper jack cheese on your list, but the Safeway product may be called Cheese Slices Pepper Jack

To handle these cases, our bot, visits the Safeway search page and simply picks the first item that is in stock to add to the cart:

Find Item And Buy It Again
 [Arguments] ${item_name} ${buy_it_again_items}
 Go To https://www.safeway.com/shop/search-results.html?q=${item_name}
 Load More Items 2

 ${added_to_cart} ${name_in_store}= Add Found Items With Buy It Again Badge

 [Return] ${added_to_cart} ${name_in_store}

Find And Buy It Again
 [Arguments] ${items_to_buy} ${buy_it_again_items}
 ${items_in_cart}= Create List
 ${items_not_found}= Create List
 FOR ${item_to_buy} IN @{items_to_buy}
 ${added_to_cart} ${name_in_store} Find Item And Buy It Again ${item_to_buy.name} ${buy_it_again_items}
 IF '${added_to_cart}'=='True'
 Log To Console Found and adding to cart: ${item_to_buy.name} (Name in store: ${name_in_store})
 Set To Dictionary ${item_to_buy} name_in_store=${name_in_store}
 Append To List ${items_in_cart} ${item_to_buy}
 ELSE
 Log To Console Not found: ${item_to_buy.name}
 Append To List ${items_not_found} ${item_to_buy}
 END
 END
 [Return] ${items_in_cart} ${items_not_found}

Mark Todolist items completed

By now, our bot created a cart with every item it could find. The next step is to mark these items as completed on our Todoist list.

Mark Todoist Item Completed
 [Arguments] ${id}
 Click Element When Ready //button[@aria-describedBy='task-${id}-content']

Mark Todoist Items Completed
 [Arguments] ${items}
 Set Local Variable ${i} 1
 FOR ${item} IN @{items}
 Log To Console Marking Todoist item #${i} as completed: ${item.name} (Name in Store: ${item.name_in_store})
 Wait Until Keyword Succeeds 10x 2 sec Mark Todoist Item Completed ${item.id}
 IF ${i} % 10 == 0
 Sleep 10
 ELSE
 Sleep 2
 END
 ${i} Evaluate ${i}+1
 END

Hold on, what’s up with those Sleep statements? Shouldn’t we avoid them? Yes, but Todoist appears to have some sort of rate limiting in place. In other words, if our bot it too fast, the website will error out as humans don’t complete items that quickly. In my testing, I found that when I took 2 seconds in between items and waited 10 seconds after each batch of 10 items I avoided this rate limiting. (You may be saying once again, but this looks brittle… yup! 😵‍💫)

Send an email with the list of items added to cart

The last thing that I wanted my bot to do was to send an email that lists all the items that were added to the cart. The email should list the name of the item as it appeared on my Todoist list and as it appears in my Safeway shopping cart. I’m then able to inspect this email quickly to see if anything looks out of place. I’ve typically found that at least a couple items look funky.

One rare reason is that Safeway’s search can be a little buggy. For example, if you search for bananas, the first result is a cabbage:

Searching for “banana”, finds a cabbage. Image credit: Author

Another thing that can occur is that a simple Todoist item like onion can actually match both Green Onion and Yellow Onion in the Buy It Again list. One way that I’ve thought about improving this in the future is to have a Replacements list that I can use to train the bot so that whenever I say onion, I mean Yellow Onion. For the MVP of my bot however, a simple email like the following is enough to empower me to make a few manual tweaks to my cart:

Example grocery shopping bot email. Image credit: Author

From what I can tell, sending an email with Robot Framework isn’t as out-of-the-box as you would hope. Robocorp, a pretty cool hosting company for Robot Framework bots, maintains an RPA Library for Email, but I found that it didn’t play well with the ppodgorsek/robot-framework Docker image that I was using to run my robot in a container. In addition, I was excited to use a custom Python script in my bot and knew that sending an email with Python is trivial. So, I simply created a custom Email.py file:

import smtplib, ssl
from email.message import EmailMessage

def send_html_email(smtp_username, smtp_password, smtp_server, smtp_port, email_recipient, subject, body):
 msg = EmailMessage()
 msg['Subject'] = subject
 msg['From'] = smtp_username
 msg['To'] = email_recipient
 msg.add_alternative(body, subtype='html');

 context = ssl.create_default_context()
 with smtplib.SMTP_SSL(smtp_server, smtp_port, context=context) as server:
 server.login(smtp_username, smtp_password)
 server.send_message(msg);

and exposed it via a Send Email keyword:

***Settings***
Documentation Send Email
Library DateTime
Library Email.py

***Keywords***

Send Email
 [Arguments] ${smtp_username} ${smtp_password} ${smtp_server} ${smtp_port} ${email_recipient} ${items_in_cart}
 ${date}= Get Current Date UTC exclude_millis=yes
 ${converted_date}= Convert Date ${date} result_format=%a %B %d %H:%M:%S UTC %Y

 Set Local Variable ${i} 1
 Set Local Variable ${body} <table border="1"><tr><th>#</th><th>Todoist Item</th><th>Name in Safeway Cart</th></tr>
 FOR ${item} IN @{items_in_cart}
 ${body}= Catenate ${body} <tr><td>${i}</td><td>${item.name}</td><td>${item.name_in_store}</td></tr>
 ${i} Evaluate ${i}+1
 END
 ${body}= Catenate ${body} </table>

 Send Html Email ${smtp_username} ${smtp_password} ${smtp_server} ${smtp_port} ${email_recipient} Grocery shopping bot ${converted_date} ${body}

The Email.py script connects to an SMTP server and sends an HTML email. I simply set up a new Gmail account and enabled SMTP access by enabling the allow less secure apps option.

Note: it is very likely that emails sent this way will end up in your Spam folder so you may need to create a filter to keep them out of your Spam folder. Sending non-spamy emails from a proper mail host is a rather involved process and is out of the scope of this post.

Running the bot

Here’s a quick video that demos how to run the bot: watch the demo on Loom.

To run the bot locally, follow the steps here.

After you run the bot, you’ll see that an output directory was created and it contains some new artifacts:

Output directory contains Robot Framework logs. Image credit: Author

You can open the log.html to view a detailed log from the bot, e.g.

Robot Framework logs. Image credit: Author

This log, and the embedded screenshots, can be quite helpful to troubleshoot any issues.

Useful tooling

If you are a Visual Studio (VS) Code user, I found that the Robot Framework Intellisense extension can be very helpful.

If you are not already acquainted with the Inspect feature in the browser console, I suggest that you give it a try as it allows you to discover the structure of the elements on a webpage:

Inspect HTML elements in Firefox. Image credit: Author

As you may have noticed in the code above, because Todoist’s and Safeway’s websites aren’t really the most accessible, the bot uses XPath to navigate the websites. For example, we select all the Todoist items with:

Get WebElements xpath=//li[contains(@class,'task_list_item')]

A lesser known feature of Chrome and Firefox is that you can actually test out your XPath query in the console with the $x() function:

Test XPath queries in Firefox. Image credit: Author

Running the bot in headless mode using Docker

What if you want to run the bot without a UI (headless) and in the background? Easy, just follow the directions at Run Locally with Docker.

What if you want to deploy the bot to the cloud? I’ll cover this in detail in a later post, but for now, take a look at the steps at Deploy to AWS for how I run the bot for just pennies a month.

Possible Enhancements

This bot is just a Minimal Viable Product (MVP). Here are some things that I think could be added to make it even more useful:

As mentioned above, a list of Replacements could be used to substitute items like onion with Yellow Onion so that it doesn’t just match Green Onion.
Assuming that a human will perform minor tweaks to the list after the bot has run, it should be possible for the bot to learn from these tweaks. In other words, if the bot determines that a Yellow Onion was purchased, when the original item on the list was onion, it could add onion => yellow onion to the Replacements list on your behalf.
Keyword expansion could be used so that a single word can represent a basket of items. For example, your family may eat a lot of fruit and want the fruit keyword to include strawberries, raspberries, and blueberries. This feature could also be good for ordering all the groceries for a dish. Perhaps, this is implemented as part of the Replacements list, i.e. lasagne => ricotta, beef, eggs, ...
Support for item counts. For example, 7 bananas would add 7 bananas to the shopping cart. For now, the bot adds just one banana to the cart and a human has to increase the count manually after the bot has finished executing.
On Safeway’s product pages there is sometimes an option to clip a coupon for a specific product, i.e. you can save some money on the purchase. The bot should do this for you.
As shown above, Alexa’s shop your list feature provides a UI that allows you to select from a list of the top search results that match the item on your shopping list. There could be value in displaying such an interface after the bot has run so that you can make any needed tweaks quickly.
The bot could analyze your purchasing habits and make recommendations to buy certain items in bulk when you consume items fast enough and when purchasing in bulk would result in saving money.
Use machine learning to suggest products that have never been shopped for (new products) based on your existing preferences for brand, price, flavor, and size.
Support for other grocery stores. The existing bot already supports any smart speaker that integrates with Todoist, but it only supports the grocery store, Safeway.

Is RPA the best way to implement this bot?

Probably not. As much fun as it was playing with the Robot Framework, the use of the framework, through the browser, led to a fairly brittle solution. If Todoist or Safeway decide to change their websites, the bot will most likely break.

I’d argue that the best way to implement such a bot is to use an officially supported developer API, such as what is provided by Todoist (or Alexa). The main reason is that this API is a contract with which Todoist will attempt to adhere. Or, in other words, it would make it a lot less likely that Todoist would introduce a change that would break the bot.

Safeway on the other hand, doesn’t appear to have a supported developer API. Instead, you could use the browser’s Network tab to reverse engineer their backend API. Using this API would significantly improve the bot in that it would:

Make it less dependent on the presentation layer (HTML) of the website. In other words, the look of the website could change, but as long as the backend API doesn’t change, the bot wouldn’t break. Even though Safeway doesn’t expose the developer API officially, they likely have teams of engineers working on the backend and frontend and they would most likely avoid breaking the API as it could cause instability in their website and reduce their team’s velocity.
Make it much faster as navigating the GUI is SLOW.
Reduce the amount of resources that the bot consumes as making a RESTful API call takes a lot less resources than running a browser
Reduce the possibility for race conditions that are encountered in the browser when a bot navigates the GUI faster than a human
Avoid the complexity of working with popups, which may be added at any time to the GUI and may have little to do with the bot’s primary objective

How has the bot been working for me?

I’ve been using the bot now for several weeks. I’ve found that it accurately builds a cart for about 90% of the items on my list. I then go in and make some minor tweaks to my shopping cart via the Safeway mobile app, but overall, I’ve cut down the weekly time I spend ordering groceries from about 30 minutes to 5 minutes. So, yeah, it works pretty well for me!

I host the bot on AWS Batch for pennies a month and have even created a custom Alexa skill that kicks off the bot. This means that after my Alexa list is done, I can just say open grocery shopping bot and the bot does its magic. In a future post, I’ll dive into how this deployment works, but for now, you can play with it here.

Enjoyed this article? If this bot were turned into an easy-to-use service and supported your preferred grocery store and smart speaker, would you pay for it? Please consider taking 3 minutes to answer these 3 quick questions

Thanks for reading this post.

Resources

Here is a list of helpful resources mentioned in this article:

We Migrated From AWS Lambda to ECS, but Hope to Eventually Migrate Back

Geoff Cox — Tue, 17 Aug 2021 14:16:09 +0000

The pitfalls of Lambda that made us switch away

Image credit: Attentie Attentie on Unsplash

Author’s note: The views expressed in this post are my own and not necessarily those of my employer.

This is a story about how we migrated from Lambda to ECS because of a 2% error rate and limited concurrency controls. And, about how AWS has more recently fixed their error-rate issues and how enhancements to concurrency controls could win us back to Lambda. When I first started at Knock in 2018, we were running our entire stack on AWS EC2 instances that we were manually managing.

As a result, it was painful to scale and maintain our infrastructure, and we were wasting money as we were unable to distribute our resource consumption. We were considering a managed container service like AWS Elastic Container Service (ECS) and AWS’s most recent serverless option, Lambda.

AWS claimed that Lambda was a reliable, feature-rich environment that abstracts away any notion of having to manage a server, and we were excited about the prospect of just being able to focus on the code.

Unfortunately, after a year of using Lambda in production, we observed Lambda was a relatively unreliable technology and when it was used to deliver customer-facing features, a 2% error rate in the AWS layer resulted in a very poor user experience. So, we decided to migrate some of our customer-facing Lambdas to ECS.

Before we dive into the details, I’d like to clarify that this post is not a hate letter to AWS about Lambda.

As you will read later, in 2020, AWS made improvements to Lambda networking, which has almost completely eroded the error rates we were seeing.

This leaves just the concern of concurrency, which is something that can be mitigated in certain use cases, but not all.

Because there is one shared concurrency pool for all your Lambdas, a burst of activity in any of your Lambdas can easily lead to significant throttling of mission-critical functionality, which results in an unacceptable user experience.

Overall, I’m hopeful that AWS will eventually enhance the concurrency model for Lambda and remain committed to the promise of a world where engineers spend less time managing servers.

What Were We Running On Lambda?

Let’s start by describing the types of services we had running on Lambda, as after all, we can sometimes tolerate latency and failures in certain use cases.

The first class of services comprises a backend of RESTful APIs fronted by API Gateway and backed by a Lambda function for each route.

We had both Node.js (Typescript) and Python code running in this layer. Although the programming language choices here have little impact on what we discuss in this post, they do impact cost and overall Lambda concurrency and latency.

The main callout here is that most of these APIs were being called by frontends synchronously and thus, significant latency and failure is unacceptable as it manifests as downtime to the user.

The second class of services mostly consists of background tasks that read messages from SQS queues and then perform IO operations on RDS instances and S3 buckets.

In our case, nothing in this pipeline is too time-sensitive, so if these background tasks fail occasionally, a retry mechanism is enough to keep our system healthy and everyone happy.

Up to 2% Error Rates in Lambda From 2019–2020

It wasn’t until we launched RESTful APIs in production that we began to notice that transient errors in the AWS layer were common for AWS Lambda. In particular, we started noticing errors in CloudWatch where a Lambda would timeout before it even ran any of our application-layer code. We knew the application code was not running as the very first thing our application code does is log the incoming request and these log statements from our application layer were missing. In other words, there was nothing that we could tune to prevent this from happening as the issue was occurring in the AWS layers. For example:

START RequestId: 56a12bf8–43eb-4b6c-8a20-c73fa2152c6b Version: $LATEST

[Missing entries from application logs]

END RequestId: 56a12bf8–43eb-4b6c-8a20-c73fa2152c6b

REPORT RequestId: 56a12bf8–43eb-4b6c-8a20-c73fa2152c6b Duration: 30019.42 ms Billed Duration: 30000 ms Memory Size: 1024 MB Max Memory Used: 77 MB

2019–08–22T20:10:31.270Z 56a12bf8–43eb-4b6c-8a20-c73fa2152c6b Task timed out after 30.02 seconds

On August 28th, 2019, We reached out to AWS via a support ticket and they confirmed that a 1–2% failure rate was somewhat normal:

Just to summarize the conversation, we talked about a 1%-2% failure rate not being outside of the realm of possibility in spikes, and the recommended implementation is backoff and retry on the client side.

AWS recommended a retry mechanism, but in practice, the only way to detect these problems was by setting a more aggressive timeout and then retrying.

Some of our legacy code in the critical code path can take several seconds to run during heavy times of load and Lambda cold starts used to be fairly significant.

So, let’s say that we implemented a five-second timeout and then retried at this point. This would result in the user having to wait at least five seconds, which is unacceptable when it happens 2% of the time!

Moreover, our backend is distributed, and therefore operations are not always atomic, meaning that there would be an increased possibility of duplicate requests when aborting the Lambda invocation and retrying the request.

This was a major deal-breaker for us, so we decided to start migrating away from Lambda at the end of 2019.

Shared Concurrency Leads To Throttled Execution and Frustrated Users

One could argue that the concurrency control capabilities of Lambda are severely lacking as there is a single per-region cap on Lambda currency per AWS account.

This means that any bursting Lambda activity can cause customer-facing Lambdas to be throttled. Here, I’m defining customer-facing Lambdas as those fielding synchronous requests from our users via a UI.

In other words, Lambda-powered background tasks can starve these customer-facing Lambdas and can essentially result in your app becoming unresponsive. This manifests as an outage to your users and is completely unacceptable when it occurs frequently.

As illustrated in the next set of graphs, you can see that spikes in customer-facing errors roughly correlate to the spikes in our account’s Lambda concurrency:

Errors from hitting max concurrency. Image credit: Author

This cannot be easily prevented as there is a limit of 1,000 concurrent Lambdas on our account, and the only way to bound Lambda executions is by reserving part of this pool of 1,000. In other words, each time you reserve concurrency for a Lambda, you forever take away potential concurrency for your other Lambdas, resulting in an awkward juggling act where you have some Lambdas that are statically bounded and others that are not.

If you are familiar with AWS Elastic Container Service (ECS) or Elastic Kubernetes Service (EKS), you’ll know that it is fairly trivial to control the number of containers (worker pool) by scaling up or down the number of tasks running for a service. In other words, you have full control over how you allocate your resources and aren’t locked into a primitive per-function concurrency model — like you are with Lambda. In a later section in this post, we’ll investigate how I think AWS could improve its concurrency controls to facilitate better resource sharing.

Given Lambda’s basic concurrency controls, is there anything we can do to prevent throttling our user-facing Lambdas? Yes, you can:

temporarily constrain the concurrency of more Lambdas with reserved concurrency
file a request with AWS support to bump up your concurrency limit from 1,000
increase the batch size on your SQS reads to a maximum of 10. This means that it takes fewer Lambdas to service the same SQS traffic. Let’s just hope you structured your code to work this way before your Lambdas were throttled
set the reserved or provisioned concurrency for user-facing Lambdas. This is fine if you only have a few Lambdas, but if you have one for each RESTful API route, you could have hundreds of Lambdas and you’ll quickly get into a situation where you actually lock up all of your concurrency. For example, imagine you have just 10 Lambdas/routes and set the reserved concurrency at 20 for each of these Lambdas, resulting in a total reserved concurrency of 20 x 10 = 200 (recall you have a max of 1,000 for your entire AWS account). That number can quickly add up and what’s more is that each of these Lambdas is now bound to a max concurrency of 20, meaning that your user-facing Lambdas aren’t going to scale as your traffic increases, which is a big reason why you even use Lambda in the first place.
stand up a SQS queue in front of API endpoints for things that are resource-intensive and/or can be processed asynchronously, e.g., email sending and report generation. This allows the customer-facing Lambdas to complete their execution faster as the resource-intensive work is deferred. Although this is typically a good pattern as it reduces latency to your user, it’s not always enough to keep your Lambda concurrency low enough to prevent throttling and it may just increase the concurrency among your background tasks if you are using Lambda to run your background tasks.
Move your user-facing Lambdas to a different region so that you have access to a different concurrency pool of 1,000. This can greatly increase the complexity of your deployments and it may even mean that other services like Cognito, DynamoDB, SQS, etc., would need to be moved to another region.

A lot of these mitigations are good things to consider when using Lambda, but they don’t solve the root problem of very limited concurrency controls leading to throttling. If you use these mitigations, you’re basically just incurring tech debt that you’ll be on the hook for later.

Poor error rates and unavoidable throttling? These were both deal-breakers, so we had no choice but to migrate away from Lambda.

Refactoring Our Code So That It Can Run On Both Lambda And ECS

Now that we had decided to migrate away from Lambda, we wondered if there was a way to make a minimal set of changes to our code so that it could also run in a Docker environment.

We chose AWS’s Elastic Container Service (ECS) to run our containers as it is relatively easy to manage and something that our DevOps team had the bandwidth to support. Fortunately, we noticed that the Lambda handlers we already implemented utilize a nice abstraction, which could be used to translate from a Docker-supported framework to a Lambda one and vice-versa. All we needed to do was create a wrapper that would:

translate the ExpressJS request payload to the event payload needed by the Lambda handler
translate the Lambda response to an ExpressJS response
use a wrapper that would allow us to reuse all our existing Lambda handler code and run it on top of the ExpressJS framework! We chose to go with ExpressJS to build the RESTful API as we were using Node.js and ExpressJS is a battle-tested framework. The following implementation is in JS but the pattern will work with any web framework and any programming language. (Our actual implementation is in TypeScript.)

Let’s take a look at some code to see how we did this. Consider a simplified version of a Lambda handler that implements a POST /lead/:leadId route, which accepts a JSON body of { name, email }:

import leadService from './lead-service';

// Define a handler for `PUT /lead/:leadId` that accepts the request body
// payload `{ name, email }`
const handler = async (event) => {
 // Parse the request parameters
 const { name, email } = JSON.parse(event.body);
 const { leadId } = event.pathParameters;

 // Assume this is the service-layer code that actually performs the
 // updating of the lead
 await leadService.update(leadId, name, email);

 // Return a 200 success code and the full lead
 return {
 statusCode: 200,
 body: JSON.stringify({
 id: leadId,
 name,
 email
 })
 };
}

export default handler;

We created a lambdaToExpress function that translates from ExpressJS to Lambda and back again, using the following code:

// Convert from a Lambda to an ExpressJS handler
const lambdaToExpress = async (handler) => {
 return async (req, res) => {
 // Repackage the express parameters as an event object for the Lambda
 const event = {
 body: JSON.stringify(req.body),
 headers: req.headers,
 queryStringParameters: req.query,
 pathParameters: req.params
 };

 const handlerRes = await handler(event);

 // Instruct express to use the response from the Lambda 
 res.status(handlerRes.statusCode).json(JSON.parse(handlerRes.body))
 };
}

export default lambdaToExpress;

Then, we registered our ExpressJS routes using our new wrapper. Here is its code:

import express from 'express';
import cors from 'cors';
import handler from './lambda';
import lambdaToExpress from './lambda-to-express';

const PORT = 3000;
const app = express();

app.use(cors());
app.use(express.json()); // Parse JSON bodies

app.put('/lead/:leadId', lambdaToExpress(handler));

app.listen(PORT);

With just a few lines of code, we enhanced our single codebase to support both Lambda and ExpressJS. One thing to note is that both the lambdaToExpress and handler functions are performing a JSON.stringify() and JSON.parse(), which adds a tiny bit of extra overhead. That being said, these operations are very quick and unnoticeable in the grand scheme of things.

The code was all set, but what about anything provided by AWS’s API Gateway?

As part of our API is external-facing, we needed to support rate-limiting. To achieve this, we incorporated node-rate-limiter-flexible (backed by ElasticCache) to implement rate-limiting. And, we simply added this rate-limiting check to our lambdaToExpress wrapper to implement this in a single place.

We were using the Serverless Framework to deploy our Lambdas, and you could probably use this framework to deploy code to ECS, but we find Terraform to be a slightly more versatile tool for configuring non-serverless infrastructure and were already using Terraform to deploy other microservices.

So, we created Terraform Modules alongside our serverless.yml configuration and deployed them with Jenkinsfiles.

And now, we were running on ECS!

We have been happily running a portion of our RESTful API backend on AWS Lambda for over a year and have found that we have reduced our compute costs by 90%! In a future post, I’ll dive into a cost comparison between Lambda and ECS.

Lambda Reduced Its Error Rate From 2% To 0.0000625%

If you recall from earlier, in August 2019, we found that AWS Lambda could suffer from an error rate of up to 2%!

In September 2019, AWS released a blob post touting improvements to networking for Lambda that would reduce latency incurred with cold starts. We were very excited about this news as it meant that AWS had acknowledged the problem and was actively working on a fix.

We had migrated some of our backends from Lambda to ECS but had a lot of remaining functionality on Lambda.

In 2021, we performed another analysis and found that the latest version of Lambda was resulting in an error rate of only 0.0000625%. This is a significant improvement from 2%!

Here is the analysis we did for our auth-service, a RESTful API responsible for authentication and authorization functionality in our system. In a seven-day period, the auth-service had about 1.6 million invocations and only one resulted in a timeout.

auth-service invocations. Image credit: Author

With error rates as low as this, our error rate concern with user-facing Lambdas has been eliminated! But what about the lingering concern with concurrency controls?

How Could AWS Improve Its Concurrency Controls?

If you recall, we ended up performing many of the concurrency mitigations mentioned above and moved a number of our Lambdas to ECS.

Afterward, we were left with a relatively comfortable set of about 200 Lambdas, and as a result, bursting concurrency is now less of a concern.

That being said, we now avoid using Lambda for any new user-facing services. And, we generally avoid using it for substantial background jobs as we are worried that excessive use will result in throttling our few remaining user-facing Lambdas.

Despite this current strategy, I’m hoping the day will come when AWS implements a more feature-rich implementation of Lambda concurrency controls that better mimics that of an ECS environment. Here are some enhancements that would make a huge difference:

Max Concurrency

Let’s define the Max Concurrency of a Lambda as a setting for the max concurrent invocations of a specific Lambda. This value is not the same as Reserved Concurrency in that the value of Max Concurrency wouldn’t subtract from the overall concurrency available for the AWS account.

Here’s an example: assume Lambda A has a Max Concurrency of 100 and your account has a global max concurrency of 1,000 (the AWS default).

Lambda A would not be throttled until there are over 100 instances of Lambda A running concurrently. Other Lambdas, e.g., Lambda B/C, can still consume up to 1,000 instances, if Lambda A has 0 running instances.

This in essence means that no matter what, Lambda A cannot consume more than 100 instances of the total 1,000, but doesn’t require us to set aside 100 instances from our total 1,000 for Lambda A.

This setting is somewhat akin to having an ECS service that can burst to an upper bound of max tasks but doesn’t require us to reserve a portion of our cluster, even if that service is not running all the instances.

Group Max Concurrency

We consider a service to be a group of related Lambdas. For example, consider 10 Lambdas that comprise 10 RESTful endpoints of a service. Or, 10 Lambdas that define the workers of an ETL pipeline. Often, we’d like to define an upper bound on concurrency for a group of Lambdas, e.g., we don’t want our ETL pipeline to consume more than 100 out of our total of 1,000 current Lambda invocations.

A Group Max Concurrency could effectively be accomplished by first allowing you to tag Lambdas to be part of a group. Second, you could then set the Group Max Concurrency for that group. Or AWS could even reuse the concept of a “stack” and have a Stack Max Concurrency.

This Group Max Concurrency is similar to how you may have an ECS service where each container is performing multiple (a group of) operations, e.g., a web server fielding requests for multiple RESTful API endpoints.

Ability to increase account-wide Max Concurrency without contacting AWS support

Some will point out that you can always increase your total Lambda concurrency just by contacting AWS support.

In fact, we actually did this to bump our Full account concurrency from 1,000 to 2,000. AWS support was pretty amenable to this, but it took a couple of days for them to make the change and we needed to provide justification.

All the while, our user-facing Lambdas were throttled, and our users were upset. I’ve even heard of companies being able to get AWS to increase their max concurrency to hundreds of thousands of Lambdas.

That’s great, but why is special approval even needed? Isn’t the fact that a company is willing to pay for the increased usage enough?

To compare with ECS, this max-concurrency-per-account setting is similar to how you can easily customize the number of nodes in an ECS cluster, which effectively defines a max number of ECS tasks that can be run concurrently.

And of course, ECS has autoscaling capabilities that allow you to scale your cluster based on your own needs. None of this scaling requires any special approval from AWS.

If AWS were able to implement these controls, it would bring AWS Lambda’s concurrency controls to parity with those of ECS and EKS, and I’d once again advocate for using Lambda in many more areas of our system.

When Does Lambda Shine?

Even with its limited concurrency controls, Lambda shines in the following areas:

For background tasks, e.g., ones that read off SQS. Sure, you can stand up something like celery or bull to get something similar, but wiring up a Lambda to an SQS queue or one of the many other AWS triggers, is such a beautifully simple thing. Note: Lambda has a max timeout of 15 minutes, so it is only suitable for running relatively quick tasks.
When you have other Lambdas that don’t burst to consume all of the available concurrency, such as when you have a relatively small amount of Lambda activity.
When you wish to optimize autoscaling so that you only pay for what you use. Lambda automatically scales up and down at the single invocation layer so that you can save a ton of money during low traffic periods
As a place to run your code without the need to manage infrastructure

Conclusion

The promise of serverless — where developers don’t need to manage infrastructure — is very compelling, and I don’t want to let this dream die.

AWS has fixed its error-rate problem with Lambdas, but Lambda is still lacking better concurrency controls.

Until these concurrency controls are enhanced, you’ll probably want to use ECS or EKS for your user-facing services, if your Lambda usage is significant and will lead to throttling.

I suspect that AWS will continue to evolve the Lambda technology to the point where it becomes a strong alternative to ECS and when it does, I’ll once again tout the virtues of using Lambda everywhere.

Thanks for reading.