DEV Community: Geoff Cox

Changing only the random seed made my trading model look twice as good

Geoff Cox — Mon, 12 Oct 2026 21:27:00 +0000

Image credit: author, drawn with code

This is a story about how I found out that most of the decisions in one of my research logs were coin flips, and about what I changed so that it wouldn’t happen again.

In September 2026, I ran the same research model eleven times. The code, the data and the settings were identical each time, and the only thing I changed was the random seed. The best of those eleven runs scored about twice as well as the worst.

That would just be a curiosity if I hadn’t spent months making decisions by comparing a single run against a single other run. When we went back through the research log for that model, 14 of the 21 decisions it recorded, mostly about whether to keep a feature or throw it out, rested on a difference smaller than what the seed alone could produce.

If you’re new to this, here’s a bit of background. I build models that pick stocks, and I can’t test one by waiting ten years to see how it does. So, I backtest it: I replay history, let the model make its picks as if it were living through each date, and score those picks against what actually happened next. I also run hyperparameter searches, which backtest hundreds or thousands of versions of a model with different settings and compare them. Over the last couple of years, that has added up to a lot of models.

In my last post, I wrote about leakage, where information from the future sneaks into the replay and makes a backtest look better than anything you could have done for real. This post is about two other ways of fooling yourself: noise, and the choices you make as the researcher. They’re harder to catch, because neither of them is a bug in the code.

Where Did This Start?

It started with a rerun. Earlier in 2026, I had measured what my live trades actually cost to execute, and the answer was close to nothing. That made me wonder whether a few research lines that I had already closed deserved a second look, since some of them had been judged with a cost assumption that was now clearly too pessimistic. So, an agent reran some of the old experiments.

When I reviewed the agent’s work, the conclusions held up, but several of the supporting numbers were wrong. Fixing those meant actually running a check that had been skipped, and that check reran a configuration from the old log with the identical command. The log had recorded it as a solid result. The rerun came back negative.

The first question was whether the engine was random at all, and it wasn’t. Two runs with the same seed on the same machine produced results that were identical down to the byte. So, whatever had changed between the original run and the rerun, it wasn’t chance, and the data changes since then became their own investigation.

However, it raised a question that I had never asked. If the engine gives the same answer every time for a given seed, how much does the answer move when the seed changes?

How Much Does a Result Move When Nothing Changes?

First, a little background on why a seed matters at all. The models in that research line were gradient-boosted trees, and like a lot of tree models, each tree is trained on a random subset of the features and a random subset of the rows. The seed decides which subsets. If you change it, you get a different set of trees from the same data, which is supposed to be a detail.

On September 8, 2026, Claude and I held one configuration fixed and ran it with eleven different seeds over the full four-year research window. The best seed scored about twice as well as the worst. From that spread, you can work out the smallest difference between two single runs that you can trust, i.e. a difference that you’d only expect to see by chance about one time in twenty.

Then we went back through the log. It recorded 21 decisions that had been made by comparing one run against one other run, and 14 of them rested on a smaller difference than that. Some were features I had thrown out as harmful, and four were features I had kept because they seemed to help. In other words, the evidence on file couldn’t tell any of those 14 variants apart from the run they were compared against, in either direction.

Each bar is one of the 21 keep-or-drop decisions in the research log, measured by the difference it rested on, as a multiple of what the random seed alone can produce. Fourteen fall short of the line.

One of those four soon confirmed it. A later rerun, done properly this time with the same seed on both sides, found that a feature the log had recorded as helping was actually hurting. That’s exactly what you’d expect from a difference inside the noise, since it can come out either way.

Fortunately, none of those features were in production, so nothing needed reverting. But the same habit of comparing one run against one other run was also in my more active research lines. When we measured the noise for the daily research engine the same way, one recorded decision stood out. In a long/short research line, which buys some stocks and bets against others by shorting them, the time of day to trade had been chosen on a difference between 17 and 29 times smaller than that engine’s noise. In other words, the backtest couldn’t separate the candidate times at all. The same measurement also showed that the worst loss from peak to trough was even more sensitive to the seed than the score was, and more than doubled between the gentlest seed and the harshest.

So, when does my live system trade? Shortly after the open rather than at the opening bell, and I’d defend that without any backtest. The first stretch after the market opens is the most volatile part of the day, which makes it hard to hit a price target or to model the price you’ll actually get. That reasoning is about how the market works, and no seed can flip it.

The rules that came out of all this are short:

No feature is declared harmful, or kept, on a difference smaller than the measured noise.
Comparisons either use the same seed on both sides, or run several seeds per side and compare the averages.
The seed, the number of threads and where the data came from are stamped into every recorded result, so that any number in the log can be reproduced.

There was one more surprise in the details. The library we were using doesn’t pick a random seed when you don’t give it one. It uses fixed defaults, so every “unseeded” run in the log had been perfectly reproducible all along, and had been sitting on one arbitrary draw that nobody had recorded or varied. Moreover, setting the seed to 0 doesn’t reproduce those runs, because it overrides the defaults with different values. The agent had assumed that it would, and there’s now a test that pins the fact that it doesn’t.

This was tough to learn, as it had taken me a lot of time to engineer myself into those conclusions. Once the coin flips came to light, I kept pushing to see if there was a way to improve the statistical significance of my results. That question, how long it takes before you can tell a real edge from luck, runs through the rest of this post.

Can I Rebuild an Index Fund From Scratch?

A couple of weeks later, I wanted a fair comparison for my own model: something cheap and simple that I could hold it up against, over long stretches of history that included real bear markets. The obvious candidate was a public momentum index fund, which holds the stocks that have risen the most over the past year. It’s one of the cheapest and best-known ways to buy a simple stock-picking rule, which makes it a fair bar for any model that picks stocks. The problem was that the fund only goes back to 2015, and its index’s history before that is the index provider’s own backtest.

So, the plan was to rebuild the index from its published rules, using my own data, and run it back through history myself.

The catch is that a rebuilt index involves a lot of choices, e.g. how far back to measure momentum, how to size each position and when to rebalance (reshuffle the portfolio), and every one of them is a knob. If I build the benchmark, look at how my model compares, and then adjust the benchmark, I’ve tuned my own yardstick.

To avoid that, the spec was written first and merged before any construction code existed. It included a gate that the rebuild had to pass over the years where the real fund exists: its annual return had to land within three percentage points of the fund’s, its risk-adjusted return (how much it earned for how bumpy the ride was) had to be within a fixed margin, and its returns had to move with the fund’s, period by period, with a correlation of at least 0.8, where 1 means moving in lockstep. All three had to pass or the rebuild was disqualified. The spec even explains why it was written first: every parameter is a researcher degree of freedom, and my research log already had four results on record that had been retracted after a closer look.

Writing the gate first wasn’t a new habit for me. When working with LLMs, I’ve found that you have to have very clear acceptance criteria, especially if you want an agent to run unsupervised for long periods of time. I set the initial rules and then refined them with the models.

There was one more guard. The benchmark was deliberately not registered as one of my models, as registered models can be fed into the hyperparameter search. Otherwise, someone, or more likely an agent, could have swept the benchmark’s own parameters and kept the best-scoring version. Changing it requires editing the code, which shows up in the history.

On September 23, 2026, the first build failed the gate by a wide margin. The leading suspect was that it weighted every stock equally, while the real index weights by company size. So, the second build pulled historical share counts from EDGAR, the SEC’s filing database, and weighted by size. That closed about a tenth of the gap, and it failed again.

That second failure was actually useful. A pass would have justified buying a commercial dataset to reach further back in time, but closing only a tenth of the gap meant that better share counts weren’t the problem. So, I found out that the data wasn’t worth buying without having to buy it.

Was That Moving the Goalposts?

This is the part I’ve thought the hardest about how to tell honestly, because the spec said that if the gate failed, the work stopped. It failed twice, and I didn’t stop.

What changed was the diagnosis. The benchmark had been built to rebalance on my model’s dates, roughly once a month, so that a comparison between the two would only differ in how they picked stocks. Meanwhile, the gate asked it to track a fund that rebalances twice a year. Those two requirements contradict each other, so the gate was close to impossible to pass by construction, and both failures had to be read in that light.

So, the work split into two builds with different jobs. The first is a faithful replica on the fund’s own twice-a-year schedule, and its only purpose is to pass the gate and prove that the index can be recreated from my data at all. The second is the monthly build that my model gets compared against, which is then justified by the replica passing. The thresholds never moved, and each new attempt was written up as its own issue, with the reason it might close the gap, before it was run.

On the fund’s own schedule alone, the replica still failed. Then it was weighted by the shares available to public investors (the free float, which is what the index uses), rather than every share a company has issued, and it passed. That happened in the third pull request of the day, about an hour and a quarter after the first failure was merged.

The gap between the rebuilt index and the real fund’s annual return, step by step, against the pre-registered limit of three percentage points. The first two were scored on my model’s monthly schedule, the rest on the fund’s.

That being said, I don’t think “the thresholds never moved” settles it. A gate that you’re allowed to keep attempting is a search of its own, and with enough attempts, something will eventually pass. What convinced me that this one hadn’t just been searched into passing was the next step.

What If You Fit to Holdings Instead of Returns?

A replica that matches the fund’s return could still be holding the wrong stocks, as lots of different portfolios produce similar returns over a few years, especially when they’re all tilted the same way. However, the fund has to file its actual holdings with the SEC, so there’s a much harder target available: the list of roughly 100 companies it held at each rebalance, and how much of each it held.

So, with the gate already passed, Claude compared the replica against those filings. The first result said that where the replica held the same company as the fund, it sized it almost exactly right, but about a third of the companies were different.

Two bugs in the comparison itself came out along the way. It had been pitting a freshly rebalanced replica against holdings from months after the fund’s last rebalance, and the code that matched company names to tickers had been quietly dropping as much as a third of the fund’s holdings. For example, a check for " CO" ate the “Cos.” in “Marsh & McLennan Cos., Inc.,” which then matched nothing. After both fixes, the replica still disagreed with the fund on about a third of the companies, so the difference was real and it was in how the replica ranked stocks.

Part of that ranking was still a guess, and the agent couldn’t read the index provider’s methodology document, because the provider’s site blocked automated downloads. So, it swept the uncertain parameters and scored each setting by how many of the fund’s actual holdings the replica matched, never by return.

One sweep found that the index measures momentum about 42 trading days before each rebalance, rather than on the rebalance date itself. That moved the share of matching companies from about two thirds to about 80%, and it did better on the filings that were held out of the sweep, which it never saw while choosing, than on the ones it was fitted to.

Another sweep, over how many weeks of volatility (how much a stock’s price swings) to measure, came back flat across a tenfold range. That turned out to be the clue, as the form was wrong rather than the length. A line in the fund’s own documentation described volatility as measured on daily returns, not weekly, and on the holdings, one year of daily returns beat the old assumption on the held-out filings. A shorter window won on the filings it was fitted to and lost on the held-out ones, which is what overfitting looks like: a setting that fits the quirks of the data it was tuned on rather than the real pattern. So, the agent took the round one-year value instead.

Neither change was chosen by the gate, and the commit messages say so when the gate didn’t cooperate. The lag narrowed the return gap, while two of the gate’s other measures got slightly worse. Switching to daily volatility didn’t move the return gap at all, and it was adopted on the strength of the documentation and the held-out holdings.

Then I downloaded the methodology document by hand, and its appendix confirmed both. Volatility is the “standard deviation of daily price returns,” and momentum is measured between month-ends ending two months before the rebalance, which from a late-March rebalance is about 42 trading days. That’s exactly the lag the sweep had found! The document also corrected a couple of things we had guessed wrong, including the cap on how much any one company can weigh and the order of the rules for keeping existing holdings, and those brought the gap down to under one percentage point.

I think this is the most useful thing I took from the whole exercise. Fitting the same parameters to returns would have picked a volatility window that, measured against the holdings, didn’t matter at all, and I would have declared the spec identified. On the other hand, matching about 100 specific companies across 13 real rebalances isn’t something that a single free parameter does by luck. So, if you’re validating a model or a reconstruction, look for the target that is hardest to hit by accident, and fit to that.

The Gate Was Green, but Were the Numbers Right?

The replica still only started in 2011, because the free source of company sizes it used begins around then. To reach back further, Claude and I used the same trick as in my last post. An S&P 500 index fund holds every company in the index in proportion to its size, and it has filed those holdings with the SEC going back to 1995. So, the value of each holding in those filings tells you the company’s size on that date.

The longer replica passed the gate again on the years where the real fund exists, with about 81% of its companies in common with the fund and an average weight error of 0.13 percentage points against the fund’s own filings.

Then I reviewed it, and the review still found three wrong numbers:

The daily return series silently dropped one real day of returns in every rebalance period. After the fix, it agreed with the period-by-period returns, and a regression test now pins that.
For a company that joined the index between two filings, a fallback took its size from the next filing. That’s leakage, since it uses a filing nobody could have seen yet, which is my last post in a single sentence. The effect was small, but the default is now strictly point-in-time, i.e. it only uses what was known on each date.
Prices jumped at the point where one data vendor’s history hands over to another’s, so the long runs were redone on a single source, and the seam is filed as its own bug, which is still open.

The way I do these reviews isn’t sophisticated. On delicate or complex work, I ask a model to go through the result and find what’s wrong, often switching to a different model from the one that did the work, and I repeat this until the review stops finding anything major. All three of these turned up after every check had already passed.

Two more findings from that review were about noise rather than bugs. The first was that leaving out the companies that later failed, the survivorship bias from my last post, flattered the result by a measurable amount. It flattered it the most during the dot-com bust, because the missing companies were the ones that collapsed. The second was that shifting every rebalance date by a single week, earlier or later, moved the result about as much as fixing the survivorship bias had. Nobody chooses the calendar date their backtest happens to rebalance on, yet it moved the answer as much as a real data fix.

So, Could It Tell Itself From Luck?

No. After all of that, with a gate written in advance, passed on returns, checked against holdings and reviewed until the review went quiet, the reconstruction gave me 24 years of a momentum index, most of them from before the fund existed. Over those 24 years, its edge over the plain S&P 500 isn’t statistically established, meaning that it isn’t large enough, compared with the noise, to rule out luck.

The reconstruction did what it was built to do. When the edge you’re measuring is small and markets are noisy, telling skill from luck takes decades, not quarters, and even 24 years can fall short. My own model runs into the same wall: on the evidence so far, it isn’t statistically distinguishable from cheaper alternatives. What I decided to do about that is the subject of my next post.

What I’d Do From Day One

If I were starting a research project like this again, this is the list I’d hand to myself and to every agent working with me:

Measure the noise before the first comparison. Run one configuration across several seeds and work out the smallest difference you can trust.
Write the gate before the build. Freeze the spec and the pass/fail criteria first, with the reason for each, and make the thing you’re testing hard to tune by accident.
Never widen a threshold. If the gate itself turns out to be wrong, say so in writing, split the job and keep the threshold.
Fit to the hardest target you can find, e.g. holdings rather than returns.
Review the work again after the gate passes, with a different model if you can, until the review stops finding things.
Shift your dates. If moving the calendar by a week moves the answer, the answer is smaller than it looks.
Keep the graveyard. Every closed experiment stays in the repo with the gate it was judged against and its verdict, so that the next person, or the next agent, can see what was already tried and why it failed.

Thanks for reading.

This describes software I run on my own accounts. It is not investment advice and I do not offer investment services.

My first trading models knew which companies would survive

Geoff Cox — Mon, 05 Oct 2026 22:38:00 +0000

Image credit: author, drawn with code

My first live trading bot started trading real money on December 2, 2024. The models behind it had been trained on years of stock-market history, and every company in that history had something in common: it was in the S&P 500 on the day I trained the model.

That sounds harmless until you think about what it leaves out. A company that was in the index in 2016 and went bankrupt in 2019 wasn’t in the training data at all, and a company that joined the index in 2025 was in it all the way back. In other words, the models learned from a version of the market where nothing ever failed and every stock was already a future winner.

That was a leak, and it went live. It wasn’t the only one either.

If you haven’t built one of these systems, here’s what I mean by a leak. You can’t test a trading model by waiting ten years to see how it does, so instead you backtest it: replay history, let the model pick stocks as if it were living through each date, and score those picks against what actually happened next. A leak is any information that sneaks into that replay which the model couldn’t have had on the date in question. It doesn’t have to be anything as obvious as tomorrow’s price. Knowing which companies will still be around in five years is enough, and the result is a backtest that looks better than anything you could have done for real.

The standard defence is to train walk-forward. For each rebalancing date (the days the portfolio is reshuffled, roughly monthly for me), the model trains only on data from before that date, picks its stocks, holds them until the next rebalancing date, and then the whole thing steps forward and repeats. That way, no single model is ever scored on data it was trained on. Walk-forward is necessary, but as you’ll see, every leak in this post got through it, because each one smuggled the future in somewhere other than the training dates: the list of stocks, the shape of the data, or the data itself, through prices adjusted the wrong way or figures dated before anyone could have known them.

So, this post is about the leaks I shipped, the ones I caught in research before they shipped, and the times an AI was sure it had found a leak and was wrong. Every one of them is a reason that backtests are hard to trust, including your own. Along the way I’ve ended up with one rule for all three: you can’t reason your way to the answer. You need a test that makes a leak show up as a number.

Survivors

What gave it away was a backtest that was too good. I tune the model by running hyperparameter searches, where I backtest hundreds of versions of the model with different settings and compare them. In August 2025, one of those runs came back with a 2020 that was simply impossible, and most of that year’s profit came from a single holding. When I went to look it up, neither Yahoo Finance nor Google Finance had any prices for its ticker before 2021, and my note from the investigation reads: “Perhaps the symbol was renamed?”

It had been. It was Chesapeake Energy, which had been dropped from the S&P 500 in 2018, did a 1-for-200 reverse split in April 2020, went bankrupt later that year, and eventually merged, took on a new name and rejoined the index in 2025. My backtest was holding it in 2020, two years after it had left the index, because the training code only knew about the index as it stood today. (Keep that reverse split in mind, as it shows up again later in this post.)

Fixing survivorship bias turned out to be mostly a data problem. First, I needed the S&P 500’s membership for every day of the training period, not just the current list. I built this from the public record of index changes and had a couple of AI models double-check the list. Then I needed price history for the companies that had left the index, many of which no longer trade. Finally, the training loop had to resolve the set of stocks it was allowed to pick from (its universe) date by date, so that a model trained on 2016 data only ever saw companies that were actually in the index in 2016.

When the fix first ran, the backtest fell apart. The note I left myself in the pull request was, in all caps, “THE ANNUALIZED RETURN IS NEAR 0!!!” That’s what removing a leak is supposed to do, and it’s a very unpleasant thing to watch.

The fix reached the production training path in late September 2025, about ten months after the first live trade.

There was one experiment from that period that I’m still glad I ran. I wondered whether a small, controlled leak might be acceptable: train the first stretch of the model on a later snapshot of the index, so that it learned from today’s large companies, and then walk forward point-in-time from there. It gave no reliable improvement. My conclusion at the time was that a model trained on a fixed list of companies never sees a stock enter or leave the index, which leaves it less prepared for the real index, where that happens all the time.

Unfortunately, that wasn’t the end of it. A year later, in October 2026, a copy of the equal-weight S&P 500 that an agent and I had rebuilt from my research data beat the real equal-weight index fund by a margin that a copy has no business beating it by. The two moved almost in lockstep, so the copy was tracking the right thing, but it was consistently doing better. That’s the same “too good” signature, and it turned out to be survivorship again, coming from two places.

The first was the membership list itself. The public record of index changes I’d built it from is sparse before about 2007, so the early lists were short and slightly wrong. To check, we rebuilt membership from the holdings of an S&P 500 index fund, which has to file its complete list of holdings with the SEC. Against those filings, my list was 74 companies short in 2001 and still 5 short in 2009, and it even contained a few companies that hadn’t gone public yet. The AI models I’d asked to double-check the list in 2025 hadn’t caught any of this.

How many S&P 500 members my list knew about, against the holdings SPY filed with the SEC. The gap is the least my 2025 fix still left out, since the list also held a few companies that hadn’t gone public yet.

The second was prices. Many of the companies that failed outright, like Lehman Brothers, Bear Stearns, Enron and WorldCom, have no price data in either of my data sources, so even with a correct list they can never be picked or held in a backtest.

This one mostly affects my long-history research backtests rather than the live system, since the data the live configuration was chosen from covers 99.6% of index members. But it means that for my backtests before about 2016, the honest reading is “probably an upper bound”, and fixing it properly needs a data source that still sells prices for companies that no longer exist.

So, What Did It Cost?

I don’t know, and I think the reason is worth explaining.

First, a correction to the intuition. A live model can’t see the future, because on the day it makes a prediction there’s no future data to see. Today’s S&P 500 list is the correct list for today. So, the live models were never peeking at tomorrow’s prices. The damage was in two other places:

The backtests I used to choose the model and its settings were inflated by the leak, so the configuration that went live was selected on a flattering number.
The live model had been trained on a history with no failures in it, so whatever it had learned, it had learned from survivors.

Putting a price on that requires comparing what the model expected with what the account actually did, period by period, and in 2025 I had no mechanism for tracking that gap. Moreover, the model worked from end-of-day closing prices while the orders filled the next morning, so any gap I computed mixed model error with hours of ordinary market movement.

That was a consequence of how the early system worked. It ran at a fairly leisurely pace after the market closed: download the day’s prices, retrain the model, decide what to buy and sell, and queue the orders for the next morning. Since then, a lot of optimization and engineering has gone into doing all of that during the trading day. The system pulls a price snapshot at a fixed time in the morning, trains the model on it and starts trading, with a median of about 12 minutes from the snapshot to the first fill. For bots that run this way, the model’s prices and the fill prices are now minutes apart instead of a night apart.

There are also now reports that split a bot’s live result into what the model predicted, what the portfolio’s composition delivered and what execution cost. When I finally ran those reports across the live history in September 2026, every period I analyzed had run without a fixed price time, so the execution numbers came with a caveat that they couldn’t really be read as execution at all. The measurement that would have priced the leak simply didn’t exist yet.

The Leak the LLMs Couldn’t Find

Survivorship was actually the second leak to reach production. The first was subtler, and it’s where most of the testing in the rest of this post came from.

In March 2025, while analyzing the returns of a newer model, I started to worry that there was leakage in the older one. So, I compared two ways of training the same model. One trained incrementally, one rebalancing period at a time, exactly as it would run live and exactly as walk-forward requires. The other trained over the whole period at once. Given the same data and the same settings, they should have produced the same results, but the annualized returns were different.

I asked the LLMs I was working with to explain the difference and they couldn’t. The pull request I opened to debug it says so directly: “The LLMs appear to be unable to determine why the annualized returns for leakage_check.py and train_and_predict.py differ. Let’s roll up our sleeves and figure this out.”

Rolling up my sleeves meant writing a logger. I logged every call to fit() andpredict(), along with the shape and columns of every input, to a JSON file without timestamps, ran both paths and diffed the files. Then I shortened the run until I had the smallest period that still showed a difference and stepped through it in the debugger.

The difference was in the shape of the training matrix. Which columns it had, and which rows survived a filter for missing values, depended on the data loaded for the whole run rather than just the training window. When training all at once, the model for an early period was shaped by companies whose data only began years later. When training incrementally, it didn’t know those companies existed yet. Neither path was reading a future price directly, but the future was deciding what the training data looked like, and that turned out to be enough to change the results.

The comparison that found the leak became a test: train incrementally on part of the data, continue on the rest, and the picks must be bit-for-bit identical to training on all of it from scratch. It started as a script that I ran by hand, then became the code path that production uses to train, and it now runs every week in the automated test suite. If I could only keep one test from this whole project, it would be this one.

The Fix That Switched Off Most of the Model

Unfortunately, the March 2025 refactor fixed the leak and broke something else, and nothing noticed for thirteen months.

Near the end of the data-preparation step, the refactored code selected the training columns by matching column names against a list of ticker symbols. The raw return column for each stock is named after its ticker, so it matched. However, the engineered features (moving averages and volatility measures built from those returns) have the ticker as a prefix rather than as the whole name, so they didn’t match and were dropped. A later step, which lines up every iteration’s columns with the model’s expected list of features, then quietly put them back, filled with zeros.

The model ran, the tests passed and the features were all present in the matrix. Every one of them was zero, and that version went into production along with the leak fix.

It finally came out in April 2026, while an LLM and I were wiring a new family of features into the training pipeline. Earlier that same day we had found a different silent drop, where three new settings were decoded from the model’s configuration and then never passed to the model. When we looked at how much the trained models relied on each input (their feature importance), every engineered feature had an importance of exactly zero.

The fix went in behind a flag so that it could be compared with the old behaviour, and a week later it became the default. If you read my last post, you may recall a change that did nothing, where two runs came back bit-for-bit identical. This is the same smell from a different angle, and I now treat an exact zero with the same suspicion as an exact match.

Corporate Actions Keep Coming Back

Most of the leaks I found in 2026 came from the same place: splits, reverse splits and dividends, which are known as corporate actions.

A quick primer, since everything below depends on it. A raw price is what a stock actually traded at on a given day. When a company does a 20-for-1 split, every share becomes 20 shares worth a twentieth as much, so the raw price drops 95% overnight even though no one lost a cent. A reverse split is the opposite: 200 shares become one, and the raw price jumps 200-fold. An adjusted price rewrites the history before each event so that the series is continuous, e.g. by dividing every price before a 20-for-1 split by 20, and it does something similar for dividends. Neither series is wrong. They answer different questions, and the bugs below all came from asking one series a question meant for the other.

I told the stock-dividend story inmy last post, where a test asserted a data convention that never occurs in practice. Here are three more.

The Model Learned to Short Stock Splits

In a long/short research line, which buys some stocks and bets against others by shorting them, so that it profits when they fall, one data source’s features were computed from raw prices rather than adjusted ones. So, when Amazon split 20:1 in June 2022, the raw series showed a one-day move of about −95%. Alphabet’s 20:1 split about six weeks later looked the same, and Tesla’s 3:1 split read as −67%.

A −95% day dominates any ranking of stocks against each other, and the model’s momentum-style features, which look at how much a stock has risen or fallen recently, carried it for one to three months afterwards. The model learned that stocks with that shape made good shorts, but they weren’t failing companies. They had just split.

A Copilot agent found it while comparing the same strategy across two data vendors. One vendor’s prices came pre-adjusted, the other’s feature panel had been built from raw prices, and the two short sides disagreed by far more than data noise should explain. A simple count shows the problem: in a 2019 sample, 9.7% of the symbols in the raw panel had a single-day move bigger than 50%, against 6.1% in the other vendor’s adjusted data.

What I find interesting is why raw prices had been chosen in the first place. Adjusted prices rewrite the past, e.g. a 2022 split changes the adjusted price for 2018. That looks exactly like lookahead, so raw prices seemed like the safe choice. However, these features were ratios: a price today divided by a price from some days ago. For any window that doesn’t straddle a split, the split factor cancels out and raw and adjusted prices give identical values. For a window that does straddle a split, the adjusted ratio is the true economic return and the raw one is an artifact. In other words, the thing that looked like lookahead wasn’t, and the thing chosen to avoid it was the bug.

Amazon’s 2022 split as raw and adjusted prices. The raw one-month return reads as a 95% crash for a month: the shape the model learned to treat as a good short.

Ratios Want Adjusted, Levels Want Raw

The mirror image turned up about a month later. The same research line had a minimum-price filter so that it wouldn’t trade anything under $10, and it was being applied to adjusted prices.

After a large reverse split, adjustment inflates a stock’s earlier history. A shell company that really traded at a fraction of a cent, and then did a reverse split of a few hundred to one, ends up with an adjusted history of tens to thousands of dollars. So, those names passed the $10 floor, were trained on and were shorted, and the backtest booked wins on positions that nobody could have actually held. Claude found this one while investigating short-side results that still looked wrong after the earlier fixes. Gating on the raw price instead took the number of candidate instances priced under $10 from 11,194 to zero.

The rule I took from the pair is that a ratio wants adjusted prices, because it’s asking how much a holding changed, while a price level wants raw prices, because it’s asking what the stock actually traded at. Neither series is a safe default.

After four layers of price corruption had been removed from that research line, its log summed it up plainly: the earlier positive results had been “largely the corruption we removed.”

The Screen in Front of Everything Was Leaking

This last one is the one I find most instructive, because it was in the tool I use to decide what is worth testing in the first place.

I screen new signal ideas (candidate inputs for the model) with a quick check that runs in seconds, before committing to a hyperparameter search that runs for days. It asks a simple question: on each past date, did ranking stocks by this signal put them in roughly the right order of what they did next? In September 2026, an agent using that screen to check a textbook momentum signal for another project found that the screen itself had two problems.

The first was that it loaded prices through a function whose adjusted flag defaults toFalse, and which returns its price column under the key "Adj Close" either way. The variable was named for adjusted prices, the column was named for adjusted prices and every statistic downstream said adjusted, but the data was raw.

The second was that it built its universe once, for the whole run, from every company that had ever been in the index. On the dates checked, 42 to 47% of the names it was ranking on a given date weren’t index members on that date. Some joined later, which is lookahead, since companies are added to the index because they grew. Others had left years before.

The tell had been sitting in the documentation for months, as a statistic with a note beside it saying “investigate”. It traced back to Chesapeake’s 1-for-200 reverse split in April 2020, the same reverse split from the survivorship investigation a year earlier. Here, raw prices turned it into a gain of more than 10,000%, and it was counted twice because the universe carried both the old and the new ticker. The problem also generalizes. Reverse splits tend to happen to companies in trouble, and companies in trouble are exactly what sits at the bottom of a ranking by recent performance, so any signal that sorts on losers is contaminated in the same direction.

Fortunately, the decisions I had made with that screen were based on rank correlation, which only asks whether the stocks came out in the right order, not by how much, so a few enormous outliers barely move it. When the agent recomputed a signal on corrected data, its rank correlation hardly changed, so those decisions probably still stand. Two other columns in the screen’s output didn’t survive. A leak doesn’t always invalidate everything it touches, and most of the work is finding out which numbers it actually reached.

Time Has Two Dates

Point-in-time doesn’t just mean the date on the row.

Some data describes a period but only becomes public later. One data vendor only keys its quarterly company financials by the end of the quarter they describe. A company’s December quarter isn’t public until its annual report is filed, around the start of March for the largest companies, and the vendor then needs time to pick it up. A reporting lag of 45 trading days after the period end made those numbers “known” a week or two before the vendor would actually have had them. That setting had already stood out in my hyperparameter search results, and the vendor’s own documentation settled it, as the period end is the only date it provides. So, I removed the option.

EDGAR, the SEC’s filing database, does record the filing date, so when I later needed historical share counts, I keyed them on the date they were filed, typically about five weeks after the quarter they describe. Keying on the period end would have used numbers that nobody could have known yet.

One vendor field had no code bug at all. An earnings-surprise figure (how far a company’s reported earnings beat or missed what analysts expected) was being computed against the analysts’ average forecast as it stood after later revisions, rather than as it stood on the day the earnings came out. The fix was to rebuild it from point-in-time estimate snapshots, which meant dropping every year before the snapshots begin and about a fifth of the symbols, which had no snapshots at all. Sometimes an honest fix makes your dataset smaller.

The False Alarms

Leakage can also go wrong in the other direction.

In August 2025, in the middle of the survivorship work, two issues drafted with an AI’s help claimed there was lookahead in my volume filter and in how the code decided which symbols were available. Both read convincingly. When I traced the dates, though, the rebalancing decision happens at the end of the window the code was reading, so all of that data was already known. My reply on the first one starts, “Actually, I believe we DO NOT have a lookahead bias.”

A year later, an agent flagged code that was, in its words, “target-leakage-shaped”: the training target also appears among the input features. It does, and at a glance it looks alarming. However, tracing every date boundary showed no lookahead, as the training window ends where the test window begins and returns are measured forward from the pick date. The agent’s follow-up claim, that the model must therefore be a simple momentum ranker, didn’t survive a measurement either. That’s two confident answers, each corrected by measuring rather than arguing.

The repo’s instruction file for agents now says that any leakage claim must name the specific date boundary that it violates and cite the code path, and those boundaries are pinned by tests named test_no_lookahead_*.

Who Found What?

When I went back through the issues and pull requests to write this post, the pattern was clearer than I’d remembered. I found the early leaks, the ones that reached production, from numbers that were too good or didn’t match: an impossible year and two training paths that disagreed. The 2026 leaks were mostly found by agents, and almost never while they were looking for leaks. They turned up while we were wiring new features, building a new test harness, comparing two data vendors or checking a rebuilt index against the real one, with me reviewing what came back. The agents also raised most of the false alarms.

The tests mostly arrived after the leak that justified them. I don’t think that’s a failure. A test written before you’ve been bitten is a guess about where the bugs are, whereas a test written afterwards is a record of where they were.

Making Leaks Show Up as Numbers

Here’s what actually caught things, roughly in the order that I’d add them to a new project:

Invariance. A result for a past window must not depend on the future. Training incrementally must match training all at once, and moving a run’s end date must not change an earlier window. The March 2025 leak broke exactly this, and so did a similar one I found later in a different research line, where extending a run’s end date changed results for years that were already over.
Counts of the impossible. Single-day moves beyond 50%, prices under a cent passing a $10 filter, feature importance of exactly zero, or a year of returns that no strategy has ever produced. Several of the leaks in this post would have shown up in a count like that well before they showed up in a result.
Point-in-time by construction. Resolve index membership per date, from the best source you can find, and key vendor fields on when they became knowable, not on the period they describe.
Compare against something real. A copy of an index fund that beats the fund, or two vendors that disagree, is a leak or a data bug until proven otherwise.
A pure-noise null. Run the whole pipeline on random-walk prices, where there’s no signal by construction. If it finds one, it leaks. In one research line, after four leak fixes, the null finally sat on zero, and that was the result I trusted most.
Replication on independent data. When an agent hands you an encouraging statistic, rerun it on data it hasn’t seen before you believe it.

This describes software I run on my own accounts. It is not investment advice and I do not offer investment services.

I let AI agents write most of a live-trading codebase. Here's the gate that made it safe.

Geoff Cox — Mon, 28 Sep 2026 23:50:00 +0000

Image credit: AI-generated

For the last 21 months I have been the only engineer on TopSet, a system that trades real money. It picks a portfolio with machine-learning models, rebalances on a schedule, and executes through live broker APIs on infrastructure I own end to end. It is my own capital, not anyone else’s.

Agents wrote most of the code. Claude Code, Cursor, Copilot. About 1,600 pull requests in those 21 months, which is more than I could have typed in three times the span.

That leaves you with a choice everyone adopting these tools runs into. You can read every line the agent produces, in which case you have bought yourself a slower version of writing it yourself. Or you can skim, merge, and move fast, in which case you accumulate code that looks right and isn’t. On a system that places orders with real money, the second option ends with trades you didn’t intend.

I took a third option: make something other than my attention the thing that decides what ships. Tests became the gate. Nothing merges or deploys without a green build, and my own reading moved up a level, to design, to test coverage, and to the handful of results I refuse to let change quietly.

That worked. It also failed in ways I did not expect, and the failures are the interesting part, so I’ll spend most of this post there.

What I actually read

“Tests are the gate” is not the same as “I stopped looking.” What changed is where I spend attention.

Most pull requests get a light review. Research code I skim. Changes to the parts that decide what to buy or how to trade it, the model training and the execution paths, I read properly, looking for whether the approach is reasonable rather than whether every line is correct. The tests cover the lines. I cover the idea.

Then there is one rule I do not bend. If a change moves a snapshot, or shifts the result of an end-to-end test, that is a stop. Snapshots and integration results are supposed to be stable, so a diff in one of them means either the change did something I didn’t intend, or it did something I did intend and I now owe an explanation. An agent will happily regenerate a snapshot to make a build go green. That is the single most dangerous thing these tools do, and it is why the snapshot files live in the repo where a diff is visible in review.

For anything involving a calculation I have not verified before, the tests are not enough either. I have the agent write a report script, and there are now 161 of them, that dumps the pipeline to CSV. I open the CSV in a spreadsheet and check the arithmetic by hand: return per trade, slippage, annualized return. Then I spot-check inputs, picking a few prices and going back to the raw data directory to confirm those were the prices the system actually used.

It is slow and it is not clever, and it has caught things nothing else would. A test asserts that a function returns what the author believed it should return. A spreadsheet asks whether the number is true.

What the gate is

Every pull request runs:

lint (ruff) and type checking (mypy)
about 7,500 tests across 331 test modules
an assertion that the test database is empty afterward
a migration rollback-and-reapply, all the way to base and back
terraform validate against both AWS accounts

If any of that fails, nothing merges. The build is the merge gate, and it is not advisory.

Once a week, on a schedule, a heavier set runs:

two end-to-end rebalancing runs against a mock broker
the same integration test 100 times in a row, looking for flakes
nine deterministic model-training snapshot runs, pinned inside Docker
two leakage checks on the training pipeline

Running the same integration test 100 times only makes sense because the test is not the same each time. The mock broker fills orders probabilistically: a 20% chance per 10-second cycle, 70% of those complete and the rest partial at a random size, at random prices. Every iteration produces a different interleaving of fills, partial fills and timing. A hundred runs is a search for race conditions, not a hundred repeats of one path.

What it finds are real concurrency bugs wearing flaky-test costumes. Worker threads leaking between tests. A queued task waking up after teardown had already deleted the row it wanted. A data loader that wasn’t thread-safe and corrupted a shared index under load. Each one showed up first as “that test fails sometimes.”

The weekly set exists because a flaky gate is worse than no gate. If a test fails once every thirty runs, and twenty agent PRs land in a week, that one test goes red most weeks. Have a few like it and something red is always on screen, and you start ignoring red. Hunting flakes is not hygiene here. It is what keeps the gate meaningful.

The tests that matter are about money, not functions

Unit tests are cheap and agents write them happily. They are not what I trust.

What I trust are end-to-end tests that drive a full rebalancing against a mock broker, with the awkward parts turned on. They’re named after the ways money goes wrong:

orders that have to be resubmitted
a rebalancing that resumes from the middle, with partial fills already on the books
a resume that has both buys and sells outstanding
smart orders cancelled mid-flight, on the sell side and then the buy side
deposits and repeating withdrawals arriving while a rebalancing is in progress

These are exactly the control flows a language model gets plausibly wrong. Interrupted work, partial state, money arriving in the middle of an operation: the code for each of these reads fine in isolation. It’s the interaction that bites.

Here is one that made it past review and got caught by the constraint at the bottom of the stack. When the last pending buy of a rebalancing had its amount adjusted for pending withdrawals, the adjusted value could land on exactly zero. The existing guard raised an error on a negative amount, but zero isn’t negative. The affordability check, available_cash >= amount, was 0 >= 0, so it passed. The order was promoted, and the database rejected it: amount > 0 is a check constraint on that table.

The interesting part is what that did to the retry. The integrity error fired inside the same transaction that carried the completing order’s status update, so the rollback discarded that too. The scheduler came back around, found identical inputs, and did it again. Not a self-healing retry: a stuck rebalancing.

The fix took three attempts, and the second one is the useful story. Gating the promotion on amount > 0 looks correct and does nothing, because the code had already assigned the adjusted value to a live ORM object. That marked it dirty, and the next autoflush in the same transaction persisted the zero regardless of whether the order was ever promoted. The condition guarded the decision, not the mutation.

I only know that because the failure was reproduced live against main before any fix was written, and the first fix was re-run against that same repro rather than checked by reading the code. When an agent is writing the patch, “I read it and it looks right” is the single least reliable signal available to you.

Three ways a green build lied to me

This is the part I’d want to read, so here are the three worst cases, all real, all from this codebase.

1. The test asserted a convention that does not exist

Corporate actions have to be applied to historical prices, and stock dividends are the fiddly ones. My broker’s API returns a rate for each. The code treated that rate as a bare fraction: a 5% stock dividend arrives as 0.05.

It isn’t. It’s the ratio of new shares to old. Every one of the 101 stock-dividend records across 43 symbols in my data has a rate of at least 1.0016. Not one is a bare fraction. The convention the code assumed does not occur in the data at all.

The unit test passed, because the test asserted the same wrong convention with a fixture value of rate: 0.05, a number that cannot appear in production. The test and the code were wrong in exactly the same way, which is what you get when both are written in the same sitting from the same misunderstanding.

The effect was that each stock dividend roughly halved the adjusted price, and it compounded across events. One symbol with nine of them came out deflated by about 490×, turning a real price history into a fake ramp of more than a thousand-fold over the period. And because every price stayed between a few cents and a few hundred dollars, a check for implausible price levels would not have flagged it either.

Southern Copper’s real daily prices, adjusted twice by the same code. The only difference is whether the stock-dividend factor is 1 / rate or 1 / (1 + rate).

The fix was a corrected fixture, plus a regression test that runs a nine-event sequence and asserts the cumulative adjustment factor stays sane. The lesson I took: a fixture is a claim about the world, and nobody reviews fixtures.

2. The test passed on NaN

In a research tool, an agent added a second way to select a model configuration and reported the result in the PR: 100% agreement with the existing selector, at every sample size tried.

That number was not real. The tool built its pool from a source that was missing the columns the default scoring version needs, so every score came out NaN. With all scores NaN, the selector fell through to a fixed tie-break order, which of course agrees with itself 100% of the time at every sample size.

The test that was supposed to cover this had been passing on those same NaN scores for as long as it had existed. It never checked anything. Green, meaningless, and quoted in a PR description as a finding.

The follow-up PR made the condition a hard error rather than a silent NaN, pinned the test to a scoring version where the columns exist, and added guards. It also flagged that an older result produced the same way might carry the same artifact. That is the right instinct: when you find a lie, go and check what else it told you.

3. The change did nothing at all

An agent wired a new training target through what looked like the right code path. The first screen ran clean and produced numbers bit-for-bit identical to the baseline. Every metric, to four decimal places.

Identical results are not a pass. They’re a smell. Two different configurations producing byte-equal output usually means one of them didn’t happen.

It hadn’t. The runner always sets a parameter that routes execution to a completely separate function, which builds its own training target and never looked at the new field. Unpickling both runs and diffing the actual selections showed 238 of 238 identical picks for the test year. After the real fix, 0 of 238 were identical, and a regression test now asserts the two modes produce different output.

Nothing failed. The compute was spent. The result would have gone into the research log as “no effect,” which is a conclusion, and it would have been wrong.

The shape of all three

In every case the build was green, and the number was wrong. Tests prove that the code does what the tests say. They say nothing about whether what the tests say is true, whether the code under test ran at all, or whether the data going in is what you think.

That is the agent-era failure mode, and it is worse with agents than without them, for a simple reason: an agent writes the code and the test together, from one reading of the problem. If the reading is wrong, both artifacts are wrong, and they agree. A human pair would at least have had two readings.

What I do about it now, concretely:

Distrust results that are identical, unchanged, or perfect. Check the output, not the exit code.
Treat fixtures as claims. If a fixture value can’t occur in production, that’s a bug in the test.
Make impossible states loud. NaN, missing data, and empty inputs should raise, not flow through and produce a number.
When something is found to be wrong, go back and check what else that thing told you.

The reviewer that isn’t a test

Some mistakes are not code-shaped and no test will ever catch them.

So I added a second kind of review: a deliberately skeptical reviewer persona, run against a finished result rather than a diff, with instructions to argue that the result is not real. I iterated on the persona with Claude until it stopped being agreeable.

It found things I am glad it found. That a configuration had been tuned by repeatedly checking results on data I had promised myself I would only look at once, so my holdout wasn’t a holdout any more. That a cost model was running with zero financing cost while the strategy it priced used nearly four times leverage.

Neither is a bug. Both are the kind of error that invalidates a conclusion, and no test suite in the world was going to tell me.

The gate after the gate

CI stops caring the moment something merges. After that the system runs unattended, on a schedule, against a live broker, and the question changes from “is this code correct” to “is this thing behaving right now.”

Every ERROR the system logs in production becomes a CloudWatch alarm that posts to a Discord channel on my phone. That is the whole alerting stack, and for a one-person operation it is enough: I am not running a rotation, I just need to know.

The loop from there is the same one I use for everything else. Point an agent at the production logs, hand it a local copy of the database from around the failure, ask for a diagnosis and a fix with a test. The logs and the data are usually enough for it to find the problem, and the test it writes is the thing that stops the problem coming back. Several of the bugs in this post arrived that way rather than through CI.

Worth saying plainly: the tests did not catch those. The tests caught the next occurrence.

What stays human

Architecture. Test design. What “done” means. Merge judgment when the tests pass and something still feels off.

And one more thing that turned out to matter: the repo’s own instruction file. Every time an agent reached a wrong conclusion in a way another agent would repeat, I wrote the correction down there. One rule says that any claim of data leakage must name the specific date boundary it violates, because two different agents had asserted leakage from reasoning alone, and both were wrong. That file is now a list of the mistakes this codebase invites. It’s the closest thing I have to institutional memory for a team of one.

What it cost

The per-PR suite takes about ten minutes. That is the tax on every change, and it is the number I would defend hardest: ten minutes is short enough that I never route around it, and long enough to run tests that touch a database and a fake broker.

The machine bill is every one of the 3,000 GitHub Actions minutes my plan includes, plus roughly $20 a month for extra minutes on top. The agents themselves are Claude Pro and Cursor Pro, $20 each. I also paid $40 a month for Copilot until GitHub moved it to per-token billing, which would have raised my bill more than tenfold, so that work moved to Claude Code and Cursor. So the tools that wrote most of this codebase cost tens of dollars a month, like the compute spent checking them, and neither is what the project actually cost.

What it actually cost is fixtures and flakes. Writing a mock broker that fails in realistic ways, maintaining a self-contained dataset so model training can be tested at all, and then the long tail of chasing tests that fail once in thirty runs. None of that is glamorous and all of it is load-bearing. Every hour I skipped there came back later as an hour of not being able to trust a result.

And sometimes the gate itself is the thing that’s broken. The 100-run soak lives in a shell script rather than inline in CI, because the inline version ran under a POSIX shell where {1..100} isn’t brace expansion. It was a literal string. The loop ran exactly once, and reported success, for longer than I’d like to admit.

The tests are not where I over-built. Every gate in this post exists because something broke, or nearly did, and the existing checks let it through. I would add all of them again.

Where I over-built is everything the tests had to cover.

I started TopSet expecting to turn it into a product, so I built it for customers who never existed. A bot lifecycle with eleven kinds of change request, each one a queued, locked, asynchronous workflow object, so that deposits and withdrawals and stops could be requested safely by someone who wasn’t me. Two broker integrations instead of one. A staging environment that mirrors production, behind a VPN. Cross-account backups with compliance-grade retention locks on a database that holds my own trades.

For one person running their own accounts, several of those could have been a script I ran by hand while watching it.

That is the part that compounds, and it is why I am putting it in a post about testing. Every one of those surfaces is something the gate has to cover. The change requests need end-to-end tests for money arriving mid-rebalance. Two brokers means two of everything, including the mocks. The staging environment is a second place for migrations to fail. I did not pay for the product shape once, in the building. I have paid for it every week since, in the size of the harness that has to keep it honest.

If I were starting again for a book of one, I would build the trading and the research properly and leave the product machinery until somebody asked for it.

If you’re rolling this out to a team

I’ve done this alone, with tests I wrote and one person’s judgment. That is not the same problem as a team of fifteen, and I’d be careful about anyone who tells you otherwise. But some of it transfers, and the ordering I’d insist on comes from a team I did run: at a healthcare company, requiring automated tests on every PR is what took active medium and high bugs down 72% and weekly hotfixes from 7 to 1.5. That was before agents. It’s the same first move.

Fix the gate before you raise throughput. Coverage on the paths that lose money or data, and no tolerated flakes. Agents multiply whatever your CI already is.
The author owns the change, whoever typed it. “The agent wrote it” is not a review comment, and it is not a defense in a postmortem.
Review the result, not the diff. For anything that produces a number, ask what would look different if the change had done nothing.
Measure escaped defects and hotfix rate. Not lines, not PR counts. Agents make those metrics meaningless overnight.

The part I would not change: the gate is what let one person move at this speed on a system where mistakes cost real money. The part I underestimated: a gate is a claim about correctness, and claims need checking too.

This describes software I run on my own accounts. It is not investment advice and I do not offer investment services.

My Journey with ChatGPT for Business: Unleashing the Power of AI for Software Engineers

Geoff Cox — Mon, 10 Jul 2023 13:46:28 +0000

Introduction

As a software engineer, I'm constantly searching for tools and technologies that can revolutionize the way I work. Today, I want to share a prototype app: ChatGPT for Business. I believe if we follow through on building this, it has the potential to transform our industry. Join me as I take you through my experiences, challenges, and the immense possibilities that lie ahead.

From Crypto Trading to AI-Assisted Productivity

The story begins with me leaving my Director of Engineering job at a fast-growing startup that was about to be acquired. I stumbled into the world of crypto trading after attending a web3 event in early 2022, where I heard about arbitrage opportunities. Eager to explore further, I began experimenting with free bots at Pionex and discovered the passive income potential of trading bots. That's when everything changed.

Driven by curiosity, I reverse-engineered a promising bot, conducted extensive backtesting using historical price data from Binance, and optimized its parameters using genetic mutation techniques. Deploying it on AWS, I witnessed consistent daily returns, achieving a remarkable 18% annual return on a sizable investment. But as the market landscape shifted, I realized that luck played a significant role in my success, and a long-term buy-and-hold strategy would yield better results. When the SEC's lawsuit against Binance and Coinbase loomed, I secured my crypto in wallets and closed my trading operations.

Discovering the Power of ChatGPT

During this transformative period, I found myself increasingly relying on ChatGPT, a powerful generative AI model. What began as a tool for trivial questions quickly evolved into something much more. ChatGPT gradually replaced around 80% of my Google searches, becoming indispensable for coding technical analysis algorithms, troubleshooting Docker configurations, proofreading emails, and even generating pitch deck outlines. I realized that ChatGPT had become my go-to resource, surpassing Google searches in many aspects. However, it was missing the capability to search my private data and access any information released after 2021.

The Spark of Innovation

One day, over coffee, a friend shared an interesting challenge: scraping information from multiple mom-and-pop websites to create a comprehensive activities-for-kids marketplace. It was at that moment that I had a revelation—what if ChatGPT could help? Instead of writing custom web scrapers for each site, could ChatGPT extract the desired information? Intrigued, I stumbled upon an informative post titled "How to Train an AI Chatbot With Custom Knowledge Base Using ChatGPT API". Inspired by its possibilities, I pointed ChatGPT to a Boys and Girls Club website, asking questions about activities in Seattle, their schedules, and costs. To my delight, the results were remarkably accurate, and the cost of using OpenAI’s API was surprisingly affordable.

Unlocking the Power of ChatGPT for Business

Fueled by this success, I couldn't help but wonder how I could further enhance my productivity using ChatGPT. To my surprise, I discovered that very few business products allowed users to leverage ChatGPT with their own data. Existing solutions fell short in terms of integrations, customization, and scalability. MyAskAI catered to individuals with one-off files and only allowed two questions per message thread. AiSera seemed more focused on building chatbots with custom data. CustomGPT lacked the necessary integrations with data sources like Google Drive, Salesforce, and Confluence. Gerev.ai, although open-source and appealing, failed to index a 5-page PDF on a brand-new MacBook Pro. Klu had an impressive turnkey integration with sources like Google Drive, but its "Beta AI" spun endlessly when asked a basic question about a single 5-page PDF file.

This realization sparked a vision in me—I wanted to build a product specifically designed to empower all businesses, not just the ones with the funds to train new AI models, with ChatGPT's capabilities. Excited by the possibilities, I reached out to my friend Arty, a talented engineer, and we began brainstorming ideas together.

Seeking Validation and Refining the Vision

To refine our concept and gain support, we sought guidance from a local venture capitalist (VC). We were told that our idea might be too broad and that we should instead focus on a specific industry. However, we remained unconvinced that there isn't a place for a general-purpose ChatGPT tool for businesses. Therefore, we decided to rely on data to guide us in the right direction. Undeterred, we developed a playbook, crafted surveys, optimized keywords for SEO and advertising, and created compelling website mock-ups. But we knew that true validation would come from our fellow software engineers—the very people who could benefit from ChatGPT for Business—YOU!

Imagine, if you will…

A ChatGPT-powered teammate, seamlessly integrated with your data sources and the web, providing real-time insights, securely linked to source documents, and offering turnkey integrations with your company's tools and systems. We envision automated document synchronization, real-time web searches, single sign-on (SSO) integration, access controls, conversation sharing, onboarding and training, and robust security and compliance measures.

Imagine being able get the status of a Jira ticket in real-time:

Or, what about keeping tabs on the competition?

Or, even synthesizing reports from raw data. Sharing and explaining insights (like a goalie, answering questions before you need to!). The more documents you share with the agent, the more time it will save for you as you could even let this AI agent handle basic questions on your behalf!

The possibilities are endless.

Join the Journey

And now, dear reader, we invite you to embark on this incredible journey with us. We think we’re on to something, but we want your thoughts! Your insights and feedback are essential as we shape the future of ChatGPT for Business. Together, we can unleash the potential of ChatGPT, transforming the way we work, and empowering ourselves and our teams with a tool that will revolutionize communication. Please take a moment to share your thoughts by participating in our quick survey. Additionally, you can also register for the beta release.

Conclusion

Our journey with ChatGPT for Business has been one of passion, innovation, and relentless pursuit. We have witnessed firsthand the immense potential of AI in our field, and we’re determined to harness that power to create a tool that will revolutionize software engineering. Thank you for reading, and we eagerly await your input as we shape the future of ChatGPT for Business together.

Run a Free VPN Server on AWS (2022 Edition)

Geoff Cox — Tue, 15 Nov 2022 18:48:50 +0000

This tutorial will take you through the steps of setting up an EC2 instance running OpenVPN server. It will then cover how to grant and revoke access to your VPN server.

Image credit: Privecstasy on Unsplash

AWS has an awesome firewall built into its core services, which can easily be used to make sure that only certain ports are open to the outside world. One extra step that we can take is to run a VPN server that serves as the gateway to our protected AWS resource, e.g. EC2, RDS, etc… We can then shutdown direct access to our AWS resources just by revoking access via our VPN server. This is very useful if you need to revoke access for a former employee.

AWS now offers a managed VPN Service, but this service costs at least $72 a month and is even more expensive when your VPN serves a lot of traffic. A lot of smaller organizations don’t need all the features of the managed service and instead can run their own VPN server for just the cost of an EC2 instance. You can even use the free tier so that you don’t have to pay anything for the EC2 instance for the 1st year.

Step 1 — Create the VPN Security Group

Overview: security groups allow your servers to communicate with each other in a private cloud while exposing specific ports to the world. We are going to create a security group to allow VPN access to our VPN server. We will assume that all your other AWS resources are members of the default security group and that the default security group does not allow access from the outside world.

From the EC2 dashboard, click Security Groups:

Image credit: Author

Click Create security group:

Image credit: Author

Enter a name and description of vpn and specify the following inbound rules on ports 22, 443, 943, and 1194. Note: the protocol for port 1194 is UDP.

Image credit: Author

Note: if the IP addresses that your team uses are static then you can add yet another layer of security by specifying an IP address range in the Source of your rules. However, you’ll want to leave the Source open to anywhere if you want your team to be able to connect from any IP as they may be working from a hotel, home, cafe, etc…

Step 2 — Create the EC2 Instance

Return to the EC2 Dashboard and then click Launch instances:

Image credit: Author

Select Ubuntu (you can of course select almost any other OS that runs OpenVPN, but this tutorial is tailored to Ubuntu)

Image credit: Author

Select t2.micro:

Image credit: Author

Note: you can use a nano instance instead of a micro instance, but nano instances are not eligible for the free tier.

In the Network settings section, select your default VPC and disable the auto-assign public IP option. Then, select both your default security group and the security group that you created above for the VPN.

Image credit: Author

Click Launch instance

Step 3 — Disable Source/Destination Check

From the list of instances, select the VPN instance and then Actions -> Networking -> Change source/destination check from the drop down menu.

Image credit: Author

Select Stop and click Save. This is needed as otherwise, your VPN server will not be able to connect to your other AWS resources.

Image credit: Author

Step 4 — Create an Elastic IP Address

Overview: when an EC2 instance is stopped and restarted, the Public IP address changes. We want the IP address of our VPN server to remain static so we’ll use an Elastic IP Address.

From the E2C Dashboard, select Elastic IPs:

Image credit: Author

Click Allocate Elastic IP address:

Image credit: Author

Make a note of your new Elastic IP address as this will be the Public IP Address of your VPN server. We will refer to this address later as PUBLIC-IP-OF-VPN-SERVER.

Select the IP address you just created and click Associate Elastic IP address:

Image credit: Author

Then, select the Elastic IP and click Associate address from the drop down menu.

Select the EC2 instance you created above and click Associate:

Image credit: Author

Step 5— Install and Configure the OpenVPN Server

SSH into your VPN server:

$ ssh ubuntu@PUBLIC-IP-OF-VPN-SERVER

Download our helper scripts and set up a default config:

$ git clone https://github.com/redgeoff/openvpn-server-vagrant
$ cd openvpn-server-vagrant
$ cp config-default.sh config.sh

Edit config.sh and enter in your configuration. Note: PUBLIC_IP should be equal to the Elastic IP Address that you created above.

$ nano config.sh

Switch to root:

$ sudo su -

You’ll now update Ubuntu. Note: you will be prompted several times and when you do, just press the Enter key.

$ /home/ubuntu/openvpn-server-vagrant/ubuntu.sh

Now, install OpenVPN. Note: you will be prompted several times and when you do, just press the Enter key.

$ /home/ubuntu/openvpn-server-vagrant/openvpn.sh

At this point, the OpenVPN server is running.

Step 6 — Add the Route

Routes must be added to the server so that your team’s clients know which traffic to route to the VPN server.

You can determine the proper subnet by returning to your list of EC2 instances, clicking on a target instance and identifying the Private IP.

Image credit: Author

Your network will be the first 2 parts of the Private IP appended with zeros, e.g. 172.31.0.0

On the VPN server edit /etc/openvpn/server/server.conf and add something like the following:

push "route 172.31.0.0 255.255.0.0"

Then, restart the VPN server with:

$ systemctl restart openvpn-server@server.service

Step 7 — Grant Access to Your VPN

Note: We assume that you are still SSH’d into the VPN and logged in as root.

Run the following command and be sure to replace client below with a unique name for your user/client.

$ /home/ubuntu/openvpn-server-vagrant/add-client.sh client

You’ll then find a configuration file at

~/client-configs/files/client-name.ovpn

You will want to provide this file to the individual on your team who will be connecting to your VPN. SCP is handy for downloading this .ovpn file from your VPN server.

Your team can use one of various VPN clients such as Tunnelblick (OS X) and OpenVPN (Linux, iOS, Android and Windows). After installing one of these clients, they should be able to set up the VPN config just by double clicking on the .ovpn file.

Note: once connected to the VPN, your users will want to use the Private IPs of your AWS resources. You may want to use nslookup to lookup the IP address from the AWS custom domain names provided by AWS. You’ll probably want to use Route 53 to create subdomain records that route to the Private IPs.

Step 8 — Revoke Access to Your VPN

Note: We assume that you are still SSH’d into the VPN and logged in as root.

Run the following command and be sure to replace client below with a unique name for your user/client.

$ /home/ubuntu/openvpn-server-vagrant/revoke-full.sh client

Troubleshooting

If your VPN client reports a TLS handshake failed error then this is most likely because your VPN security group (Step 1) is incorrect. Make sure that you have the correct ports and protocols specified — a common problem is not specifying UDP for port 1194.

How to autogenerate forms in React and Material-UI with MSON

Geoff Cox — Tue, 09 Feb 2021 14:38:33 +0000

Implementing great forms can be a real time-waster. With just a few lines of JSON, you can use MSON to generate forms that perform real-time validation and have a consistent layout. And, MSON comes with a bunch of cool stuff like date pickers, masked fields and field collections.

Disclaimer: this post is geared towards those wishing to use Material-UI with React. Future versions of MSON will support other rendering layers.

What the heck is MSON?

MSON is a declarative language that has the capabilities of an object-oriented language. The MSON compiler allows you to generate apps from JSON. The ultimate goal of MSON is to allow anyone to develop software visually. You can also use pieces of MSON to turbo charge your form development.

A Basic Form Generated with MSON

Simply declare your form in JSON. Then let the MSON compiler and UI-rendering layer autogenerate your UI:

Did you try submitting the form without filling in any values? Did you notice how real-time validation is automatically baked in?

Now, let’s take a closer look at what is happening. The first block of code contains a JSON definition that describes the fields in the form:

const definition = {
  component: "Form",
  fields: [
    {
      name: "heading",
      component: "Text",
      text: "# Form using [MSON](https://github.com/redgeoff/mson)"
    },
    {
      name: "fullName",
      component: "PersonFullNameField",
      required: true
    },
    {
      name: "birthday",
      component: "DateField",
      label: "Birthday",
      required: true
    },
    {
      name: "phone",
      component: "PhoneField",
      label: "Phone"
    },
    {
      name: "submit",
      component: "ButtonField",
      label: "Submit",
      type: "submit",
      icon: "Send"
    }
  ]
};

This code adds the following fields to your form:

The Text component displays some markdown
The PersonNameField is used to capture a person’s first and last names
The DateField allows a user to choose a date using a slick date picker
The PhoneField uses an input mask and country codes to guide the user when entering a phone number
The SubmitField contains a Send icon and allows the user to submit the form via a click or by pressing enter

Now, let’s take a look at the code that renders the component and handles the submit event:

ReactDOM.render(
  <Component
    definition={definition}
    onSubmit={({ component }) => {
      alert(JSON.stringify(component.getValues()));
    }}
  />,
  document.getElementById("root")
);

That’s it!? Yup! The mson-react layer automatically knows how to render the form component. It uses pub/sub and Pure Components to keep the rendering up-to-date.

When there is no validation error and the user clicks the submit button, an event with the name equal to the button’s name is emitted. In our case, this event is called submit. Therefore, we can define a handler using the onSubmit property. To keep things simple, we just alert the user of the entered values. Typically you want to do something like contact an API, redirect, etc…

Basic Form 2

Now, let’s go a bit deeper into the CRUD with a different example:

The first thing you may notice are the validators in the definition:

validators: [
  {
    where: {
      "fields.email.value": "nope@example.com"
    },
    error: {
      field: "email",
      error: "must not be {{fields.email.value}}"
    }
  }
]

Each field has a default set of validators, e.g. the EmailField ensures that email addresses are in a valid format. You can also extend these validators for a particular field or even for an entire form. For example, you can prevent the user from entering nope@example.com.

Next, let’s take a look at the code that loads the initial values when the component is mounted:

onMount={({ component }) => {
  // Load any initial data, e.g. from an API
  component.setValues({
    id: "abc123",
    firstName: "Bob",
    lastName: "Marley",
    email: "bob@example.com"
  });
}}

This code could just as easily be replaced by code that retrieves the values from some API asynchronously.

Finally, we use a more sophisticated event listener to handle the submit action. In a real-world app, you’d probably want to send a request to an API to save the data. You would receive a response from this API. If you receive an error, e.g. the email address is already in use, you can present this error to the user:

onSubmit={({ component }) => {
  // TODO: Contact some API with the data
  console.log("submitting", component.getValues());

  // Simulate response from API saying that email address is already in use and report this
  // error to the user
  if (component.get("fields.email.value") === "taken@example.com") {
    component.set({ "fields.email.err": "already in use" });
  } else {
    // Everything was successful so redirect, show confirmation, etc...
  }
}}

Live Demo

This post only touches on a small piece of what you can do using MSON. Because MSON is a full-featured language, you can declare all types of cool components. If you are interested in seeing more live examples, check out the live demo.

Wrap It Up!

If you are using React and Material-UI, you can speed up your app development by autogenerating your forms from JSON. This can be particularly useful if you need to bootstrap an app quickly and don’t want to have to worry about building a UI from scratch.
If you liked this post, please give it a clap or two. Happy autogenerating!

About the Author

Geoff Cox is the creator of MSON, a new declarative programming language that will allow anyone to develop software visually. He loves taking on ambitious, yet wife-maddening, projects like creating a database and distributed data syncing system. You can read more of his posts at redgeoff.com or reach him at github.com/redgeoff or @CoxGeoffrey.

Creating a New Programming Language That Will Allow Anyone to Make Software

Geoff Cox — Mon, 16 Nov 2020 00:02:24 +0000

Originally Posted on Medium

I spent 7 months working on a new programming language called MSON. Here is a nerdy post about why I did it, how it works and where I want to take it.

There are already a bazillion programming languages, why do we need another?

Software continues to eat the world and software development tools are evolving at lightning speeds, but it stills takes an in-depth knowledge of a programming language to develop software. Sure, there are millions of open-source projects that can be mashed together and robust cloud platforms that make the process easier, but it takes considerable skill to get all of these technologies working together. When I envision the future, I imagine the average person being able to make software, just like how the average person can create a spreadsheet today.

Creating a spreadsheet has become commonplace and this is largely a result of the intuitive user interface (UI) in MS Excel. Developing software; however, still remains something that is only done by a relative few and using complex programming languages. One could argue that it is somewhat trivial to create a UI that generates code, but it is almost infeasible to reverse the process and digest coded logic, as most programming languages have an almost infinite number of ways of representing logic. In other words, we don’t usually know how a piece of software will execute until we actually execute it. Creating a truly intuitive way of developing software requires a new paradigm, a new language that is easily read, modified and written by a UI.

It was back in 2003, when PHP was the hotness, that I started constructing a framework called FORMfields. At the time, FORMfields was pretty cool as you could create forms with PHP objects and then FORMfields would automatically generate the corresponding HTML, CSS and JS. Soon after creating FORMfields, I co-founded a company called GoExpo that produces event management software that is powered by FORMfields. GoExpo grew to process millions of dollars in booth sales annually for some of the largest trade shows in the world and this was a testament to the effectiveness of FORMfields.

Over the years, FORMfields continued to be essential in allowing our team to create a consistent UI that could be modified quickly (remember, these were the days before Twitter Bootstrap and more modern UI frameworks). However, as browsers were becoming more capable, more logic was moving into the front end; and FORMfields, which is written in PHP, only runs in the back end.

Fast forward to 2018, where JavaScript is the workhorse of the web and the difference between the front end and back end can feel like night and day. GoExpo has been acquired and I now have time to revisit the notion of FORMfields with a more modern stack. I want the next generation of FORMfields to run in both the browser and the back end, be open source and become the foundation of something that can be used by the masses.

This line of thinking has given birth to MSON, an extensible declarative language that can be easily manipulated by a UI. MSON is comprised of just a few building blocks, yet it is as powerful as its non-declarative counterparts.

The purpose of this initial post is to introduce the core principles of the MSON programming language and elaborate on its unique design. I plan on writing a follow-up post later that will explain MSON from a layman’s perspective. This post; however, is rather technical as it is about the developer-layer stuff in MSON and not about the designed-for-all UI layer I’ll be writing on top. I will be elaborating on the following topics:

Declarative Syntax
Components, Validators, Events & Listeners
Access Control
Inheritance, Template Parameters & Composition
Aggregate Components
Schemas and Self Documentation
User-Defined JavaScript Components
Using MSON in Any JavaScript Code
Reusing MSON Code Throughout the Full Stack
In/Out Properties

I will then finish with a statement about how I plan to monetize MSON and make it a self-sustaining project. I’d love to hear what you think.

Language Principles

Declarative Syntax

MSON is short for Model Script Object Notation, which is intentionally similar to JSON (JavaScript Object Notation). In fact, MSON is a subset of JSON, so if you know JSON then you know the syntax of MSON!

Declarative languages are much easier for software to read and write as they define what the software must do without stating exactly how to do it. And JSON is a great foundation on which to build. It contains just a few main constructs, is ubiquitous and supported by a vast ecosystem.

Components

The smallest building block in MSON is called a component. Components maintain state and can also control presentation and are very similar to the components now commonplace in most web frameworks. Components can inherit, contain or wrap other components. The rendering layer supports plugins for different environments and the default plugin supports React and Material-UI. Use of the rendering layer is optional, so components can be used on both the front end and back end.

A simple form component used to collect a name and email address could look like:

{
  name: 'MyForm',
  component: 'Form',
  fields: [
    {
      name: 'name',
      component: 'TextField',
      label: 'Name',
      required: true
    },
    {
      name: 'email',
      component: 'EmailField',
      label: 'Email'
    },
    {
      name: 'submit',
      component: 'ButtonField',
      label: 'Submit',
      icon: 'CheckCircle'
    }
  ]
}

And when it renders it looks like:

And, real-time validation is automatically baked in:

The majority of the remaining examples in this post will focus on form components, as they are simple to visualize, but MSON can support any type of component, e.g. menus, snackbars, redirects, etc… In addition, you can use JavaScript to create user-defined components that can pretty much do anything else you can imagine.

Validators

Each field has a default set of validators, e.g. the EmailField ensures that email addresses are in a valid format. You can also extend these validators for a particular field or even for an entire form.

For example, you can prevent the user from entering nope@example.com:

{
  name: 'MyForm',
  component: 'Form',
  fields: ...,
  validators: [
    {
      where: {
        fields: {
          email: {
            value: 'nope@example.com'
          }
        }
      },
      error: {
        field: 'email',
        error: 'must not be {{fields.email.value}}'
      }
    }
  ]
}

Template parameters like {{fields.email.value}} can be used to inject the values of fields. And, you can use any MongoDB-style query in the where. For example, if you had password and retypePassword fields, you could ensure that they are equivalent with:

where: {
  retypePassword: {
    fields: {
      value: {
        $ne: '{{fields.password.value}}'
      }
    }
  },
  error: ...
}

Events & Listeners

Changes to properties in a component generate events and you can create listeners that respond to these events with actions. There are basic actions that set, emit, email, contact APIs, etc… and custom actions can also be built using JavaScript.

The following example sets the value of the email field based on the value supplied in the name field when the user clicks the submit button:

{
  name: 'MyForm',
  component: 'Form',
  fields: ...,
  validators: ...,
  listeners: [
    {
      event: 'submit',
      actions: [
        {
          component: 'Set',
          name: 'fields.email.value',
          value: '{{fields.name.value}}@example.com'
        }
      ]
    }
  ]
}

We can also make this action conditional, e.g. only set the email if it is blank:

listeners: [
  {
    event: 'submit',
    actions: [
      {
        component: 'Set',
        if: {
          fields: {
            email: {
              $or: [
                {
                  value: null
                },
                {
                  value: ''
                }
              ]
            }
          }
        },
        name: 'fields.email.value',
        value: '{{fields.name.value}}@example.com'
      }
    ]
  }
]

And sometimes we want to nest actions so that a condition is met before all actions are executed:

listeners: [
  {
    event: 'submit',
    actions: [
      {
        component: 'Action',
        if: {
          fields: {
            email: {
              $or: [
                {
                  value: null
                },
                {
                  value: ''
                }
              ]
            }
          }
        },
        actions: [
          {
            component: 'Set',
            name: 'fields.email.value',
            value: '{{fields.name.value}}@example.com'
          },
          {
            component: 'Set',
            name: 'fields.name.value',
            value: '{{fields.name.value}} Builder'
          }
        ]
      }
    ]
  }
]

Access Control

Unlike most programming languages, access control is a first-class citizen in MSON, so its easy to use without a lot of work. Access can be restricted at the form or field layers for the create, read, update and archive operations. (MSON is designed to encourage data archiving instead of deletion so that data can be restored when it is accidentally archived. You can, of course, permanently delete data when needed).

Each user can have any number of user-defined roles and access is then limited to users with specified roles. There is also a system role of owner that is defined for the owner of the data. Field-layer access is checked first and if it is missing it will cascade to checking the form-layer access. When the access is undefined at the form layer (and not defined at the field-layer), all users have access.

Here is an example configuration:

{
  name: 'MyForm',
  component: 'Form',
  fields: ...,
  validators: ...,
  listeners: ...,
  access: {
    form: {
      create: ['admin', 'manager'],
      read: ['admin', 'employee'],
      update: ['admin', 'owner', 'manager'],
      archive: ['admin']
    },
    fields: {
      name: {
        create: ['admin'],
        update: ['owner']
      }
    }
  }
}

Among other things, only users with the admin or manager roles can create records. In addition, only owners of a record can modify the name.

Inheritance

Inheritance is used to add additional functionality to a component. For example, we can extend MyForm and add a phone number:

{
  name: 'MyFormExtended',
  component: 'MyForm',
  fields: [
    {
      name: 'phone',
      component: 'PhoneField',
      label: 'Phone Number',
      before: 'submit'
    }
  ]
}

We can define new validators, listeners, access, etc… at this new layer. For example, we can pre-populate some data, lay out all fields on the same line and disable the email field by creating a listener for the create event:

{
  name: 'MyFormExtended',
  component: 'MyForm',
  fields: ...,
  listeners: [
    {
      event: 'create',
      actions: [
        {
          component: 'Set',
          name: 'value',
          value: {
            name: 'Bob Builder',
            email: 'bob@example.com',
            phone: '(206)-123-4567'
          }
        },
        {
          component: 'Set',
          name: 'fields.name.block',
          value: false
        },
        {
          component: 'Set',
          name: 'fields.email.block',
          value: false
        },
        {
          component: 'Set',
          name: 'fields.email.disabled',
          value: true
        }
      ]
    }
  ]
}

which would render as:

Template Parameters

Template parameters are helpful when creating reusable components as they allow you to make pieces of your component dynamic. For example, let’s say that we want our first field and the label of our second field to be dynamic:

{
  name: 'MyTemplatedForm',
  component: 'Form',
  fields: [
    '{{firstField}}',
    {
      name: 'secondField',
      label: '{{secondFieldLabel}}',
      component: 'EmailField'
    }
  ]
}

we can then extend MyTemplatedForm and fill in the pieces:

{
  name: 'MyFilledTemplatedForm',
  component: 'MyTemplatedForm',
  firstField: {
    name: 'firstName',
    component: 'TextField',
    label: 'First Name'
  },
  secondFieldLabel: 'Email Address'
}

Composition

The componentToWrap property allows you to wrap components, enabling your reusable components to transform any component. For example, we can use composition to create a reusable component that adds a phone number:

{
  name: 'AddPhone',
  component: 'Form',
  componentToWrap: '{{baseForm}}',
  fields: [
    {
      name: 'phone',
      component: 'PhoneField',
      label: 'Phone Number',
      before: 'submit'
    }
  ]
}

and then pass in a component to be wrapped:

{
  name: 'MyFormWithPhone',
  component: 'AddPhone',
  baseForm: {
    component: 'MyForm'
  }
}

You can even extend wrapped components, paving the way for a rich ecosystem of aggregate components comprised of other components.

Aggregate Components

MSON ships with a number of aggregate components such as the RecordEditor and RecordList, which make it easy to turn your form components into editable UIs with just a few lines of code.

Let’s define a user component:

{
  name: 'MyAccount',
  component: 'Form',
  fields: [
    {
      name: 'firstName',
      component: 'TextField',
      label: 'First Name'
    },    
    {
      name: 'lastName',
      component: 'TextField',
      label: 'Last Name'
    },
    {
      name: 'email',
      component: 'EmailField',
      label: 'Email'
    }
  ]
}

we can then use a RecordEditor to allow the user to edit her/his account:

{
  name: 'MyAccountEditor',
  component: 'RecordEditor',
  baseForm: {
    component: 'MyAccount'
  },
  label: 'Account'
}

rendering as:

and if you were to click the Edit button you’d see:

You can also use the RecordList to display an editable list of these accounts:

{
  name: 'MyAccountsList',
  component: 'RecordList',
  label: 'Accounts',
  baseFormFactory: {
    component: 'Factory',
    product: {    
      component: 'MyAccount'
    }
  }
}

rendering as:

Schemas and Self Documentation

Schemas must be defined for all components, which means that MSON is strongly typed. For example, a schema that defines boolean and date properties may look like:

{
  name: 'MyComponent',
  component: 'Component',
  schema: {
    component: 'Form',
    fields: [
      {
        name: 'hidden',
        component: 'BooleanField',
        help: 'Whether or not the component is hidden'
      },
      {
        name: 'updatedAt',
        component: 'DateTimeField',
        required: true, 
        help: 'When the component was updated'
      }
    ]
  }
}

Schemas can also contain documentation via help properties, which means that components are self-documenting! In addition, schemas are inherited and can be overwritten to allow for more or even less constraints.

User-Defined JavaScript Components

The MSON compiler is written in JavaScript and can run in both the browser and in Node.js. As such, you can use any custom JS, including external JS libraries, to create your own components.

For example, here is a component that uses Moment.js to set a currentDay property to the current day:

import compiler from 'mson/lib/compiler';
import Component from 'mson/lib/component';
import Form from 'mson/lib/form';
import { TextField } from 'mson/lib/fields';
import moment from 'moment';
class MyComponent extends Component {
  _create(props) {
    super._create(props);
    this.set({
      // Define a currentDay property
      schema: new Form(
        fields: [
          new TextField({
            name: 'currentDay'
          })
        ]
      ),
      // Default currentDay
      currentDay: moment().format('dddd')
    });
  }
}
compiler.registerComponent('MyComponent', MyComponent);

And then MyComponent can be used in any MSON code.

You can also do things like define custom asynchronous actions, e.g. one that POSTs form data:

import compiler from 'mson/lib/compiler';
import Action from 'mson/lib/actions/action';
import Form from 'mson/lib/form';
import { TextField } from 'mson/lib/fields';
class MyAction extends Action {
  _create(props) {
    super._create(props);
    this.set({
      schema: new Form(
        fields: [
          new TextField({
            name: 'foo'
          })
        ]
      )
    });
  }
  async act(props) {
    const form = new FormData();
    form.append('foo', this.get('foo'));
    const account = props.component;
    form.append('firstName', account.get('firstName');
    form.append('lastName', account.get('lastName');
    form.append('email', account.get('email');
    return fetch({
      'https://api.example.com',
      {
        method: 'POST',
        body: form
      }
    })
  }
}
compiler.registerComponent('MyAction', MyAction);

And then you can use this in your MSON code:

{
  name: 'MyAccountExtended',
  component: 'MyAccount',
  listeners: [
    {
      event: 'submit',
      actions: [
        {
          component: 'MyAction',
          foo: 'bar'
        }
      ]
    }
  ]
}

Using MSON in Any JavaScript Code

There is always parity between compiled and uncompiled components so that the same feature set is supported by both compiled and uncompiled code. For example, our same MyAccount component can also be defined as:

import Form from 'mson/lib/form';
import { TextField, Email } from 'mson/lib/fields';
class MyAccount extends Form {
  _create(props) {
    super._create(props);
    this.set({
      fields: [
        new TextField({
          name: 'firstName',
          label: 'First Name'
        }),
        new TextField({
          name: 'lastName',
          label: 'Last Name'
        }),
        new EmailField({
          name: 'email',
          label: 'Email'
        })
      ]
    })
  }
}

In fact, converting MSON code to this type of code is basically what the compiler does. Although, the compiler doesn’t actually transpile MSON to JS, it merely instantiates JS code based on the MSON definitions.

Since all MSON code can be compiled to JS code, you can use MSON components in any JS code. For example, you can set some fields and validate the data:

import compiler from 'mson/lib/compiler';
// Compile the MyAccount component
const MyAccount = compiler.compile({
  component: 'MyAccount'
});
// Instantiate the JS class with a default value
const myAccount = new MyAccount({
  // Default values
  value: {
    firstName: 'Bob'
  }
});
// Set the remaining data
myAccount.set({
  lastName: 'Builder',
  email: 'invalid-email@'
});
// Make sure the values are valid
myAccount.validate();
if (myAccount.hasErr()) {
  console.log(myAccount.getErrs());
}

In other words, you can use MSON in your existing JS code to save time writing complex code. By declaring components in MSON, you’ll remove a lot of boilerplate code and reduce the possibility of bugs. You’ll also have code that has a standard structure and is framework agnostic. And this code doesn’t add any unneeded frameworks or back-end dependencies to your codebase.

Reusing MSON Code Throughout the Full Stack

MSON components can be shared by both the front end and back end, allowing for key logic to be written once and then reused. For example, the same form validation rules can be enforced in the browser and by your back-end API.

Moreover, actions can be limited to the backEnd or frontEnd, so that the same component can adjust according to the host environment. For example, you may want a contact form to send an email to the user when it is used on the back end, but only display a snackbar on the front end:

{
  component: 'Form',
  fields: [
    {
      name: 'email',
      component: 'EmailField',
      label: 'Email'
    },
    {
      name: 'message',
      component: 'TextField',
      label: 'Message'
    },
    {
      name: 'Submit',
      component: 'ButtonField',
      label: 'Submit'
    }
  ],
  listeners: [
    {
      event: 'submit',
      actions: [
        {
          // Send an email on the back end
          component: 'Email',
          layer: 'backEnd',
          from: '{{fields.email.value}}',
          to: 'noreply@example.com',
          subject: 'My message',
          body: '{{fields.message.value}}',
          // Detach so that user doesn't have to wait for email
          // to send
          detach: true
        },
        {
          // Display a message to the user on the front end
          component: 'Snackbar',
          layer: 'frontEnd',
          message: 'Thanks for the message'
        }
      ]
    }
  ]
}

In/Out Properties

Sometimes you want the presence of data, but don’t want it to be written or read from the back end. For example, your default user component may not allow for the password to be read or edited:

{
  name: 'MyUser',
  component: 'Form',
  fields: [
    {
      name: 'name',
      component: 'TextField',
      label: 'Name'
    },
    {
      name: 'email',
      component: 'EmailField',
      label: 'Email'
    },
    {
      name: 'password',
      component: 'PasswordField',
      label: 'Password',
      hidden: true,
      in: false,
      out: false
    }
  ]
}

However, your EditPasswordForm may need to allow such access:

{
  name: 'EditPasswordForm',
  component: 'MyUser',
  listeners: [
    {
      event: 'create',
      actions: [
        {
          // Hide all fields
          component: 'Set',
          name: 'hidden',
          value: true
        },
        {
          // Show password field
          component: 'Set',
          name: 'fields.password.hidden',
          value: false
        },
        {
          // Allow user to write password to the back end
          component: 'Set',
          name: 'fields.password.out',
          value: true
        }
      ]
    }
  ]
}

Next Steps

Before publishing this post, I had showed MSON to only a handful of people. This may seem odd for an open-source project, but it was intentional, as I wanted to flesh out the language before doing a wider release. I didn’t want people to start using something where the building blocks would be changing drastically. Therefore, I started with a basic prototype of the compiler and then started building a sample app written in MSON. This allowed me to use an iterative approach to see what works and what doesn’t work. And after several rewrites of the compiler, thank goodness for unit tests and a patient wife, I’ve been able to implement a full-featured app. I’m sure the language will continue to evolve, but I now feel confident that MSON is a solid foundation on which to build.

The MSON compiler is completely written in JS and runs in both the browser and on Node.js. The repo is hosted on github at https://github.com/redgeoff/mson — if you like the idea please give it a star.

I also plan on monetizing this idea by developing out mson.co, which will host a UI that anyone can use to make software. The UI will read, modify and write MSON behind the scenes and a back end will automatically be created based on your component definitions. And for developers, you’ll have the option of using as much of the mson.co stack as you wish, e.g. you could bootstrap your app using the MSON UI Builder and then replace/customize the front end with your favorite technologies (the mson.co back end is exposed via a GraphQL API). Or, you can use MSON on just the front end and create your own custom back-end API.

I hope you enjoyed reading this post and I would love to hear your thoughts.