Check out StackCircuit
Deploying a website now is almost stupidly easy. You push something to GitHub, Vercel picks it up and a few minutes later its live. Thats basically it. A few years ago getting something into production was a whole process, now half the time I don't even think about it. But the weird part is that when a deployment actually breaks something, a lot of that convenience just kind of disappears.
And a deployment doesn't even have to completely fail for this to happen. It can build fine, deploy fine, and Vercel can tell you everything worked. Meanwhile an API route is broken or some important part of the site just doesn't work anymore. Technically the deployment succeeded, but obviously something still went wrong.
We kept running into versions of this while working on our own projects. Most of what we build uses GitHub and Vercel, so usually deploying is the easiest part. Its when something breaks after that where it gets annoying. You start checking what you changed, then logs, then the last deployment, and eventually your probably just trying to figure out what version was working before you messed something up. Then you rollback and check the site again.
This isn't that huge of a problem if it happens once. But we started wondering why we were even doing some of this manually. Vercel already knows when we deployed. GitHub knows exactly what changed. We know what production looked like before and after a deployment. So why does someone still have to notice the problem and then go backwards through all of it themselves?
That was basically where StackCircuit started. If deploying code can be automated this much, we thought at least some of the recovery process probably could be too.
*The Idea Behind StackCircuit
*
The basic idea behind StackCircuit is pretty simple. It keeps watching your production app after a deployment instead of assuming everything is good just because the deployment succeeded.
If something starts going wrong, it looks at what was recently deployed and whether the problem actually lines up with it. From there it can figure out what the last healthy deployment was and, depending on how you set it up, either recommend going back or actually recover it.
Originally we thought the rollback part was going to be the difficult part of building this. It really wasn't. The harder problem was figuring out when a rollback should happen in the first place.
Because production is messy. Sometimes an API goes down for 30 seconds. Sometimes something times out for basically no reason. Other times your app is broken but the deployment had nothing to do with it. If we just made StackCircuit rollback every time it saw something fail, it would be pretty useless and probably make things worse.
So a lot more of the problem became figuring out whether a deployment actually caused the issue, instead of just detecting that an issue exists. Was the app healthy before the deployment? When did the problem actually start? Did it happen once or is it still happening? And do we actually have a previous deployment that we know was working?
The actual rollback is only one part of it. Knowing when you should do it ended up being the much more interesting problem.
*Giving Software Access to Production Is Kind of Scary
*
There was another issue we knew we had to deal with pretty early. Asking someone to connect a new tool to their production environment and then saying "don't worry, it can rollback your deployments automatically" isn't exactly comforting.
We wouldn't immediately trust that either.
So instead of making StackCircuit either fully automatic or completely manual, we ended up with three different modes.
Observe Only is basically what the name says. StackCircuit watches production and tells you what its seeing, but it can't actually change anything. Approval Required lets it detect what it thinks is a bad deployment and recommend recovery, but someone still has to approve it. Then theres Auto-Recover, where StackCircuit can actually take action when the conditions you've configured are met.
The idea is that you don't have to trust it all at once. You could run it in Observe Only for awhile and see whether what its telling you actually makes sense. Then maybe move to Approval Required. If you eventually trust the system and want it to handle certain failures automatically, you can turn on Auto-Recover.
We felt like this made a lot more sense than asking people to hand over control immediately. Especially because this is production we're talking about. If StackCircuit makes a bad decision there, thats obviously a much bigger problem than just sending a wrong notification or something.
*Rolling Back Doesn't Mean You Fixed It
*
This was probably one of the more obvious things we realized, but it changed quite a bit about how we thought about recovery.
Imagine deployment 42 is working fine. Then you push deployment 43 and the app starts failing. StackCircuit determines that 43 is probably the problem and restores 42.
Sounds like its fixed.
But what if deployment 43 actually wasn't the problem? Maybe an external service happened to go down right after the deployment. Maybe theres another issue somewhere that has nothing to do with your code. StackCircuit could successfully restore deployment 42 and the app would still be broken.
So we couldn't really treat "rollback completed" as "problem solved."
After a recovery, StackCircuit needs to check the application again and see if things actually went back to normal. If the same problem is still happening, then clearly the rollback didn't solve whatever was going on.
That ended up changing how we looked at the product in general. We originally thought about StackCircuit mostly as an automatic rollback tool. Now we think of it more as a loop around the deployment itself.
Something gets deployed. StackCircuit watches what happens. If production starts having problems, it tries to figure out whether the new deployment is actually responsible. If it decides recovery makes sense, it goes back to a known healthy version. Then it checks again.
It sounds like a small difference but its pretty important. The goal isn't to rollback deployments. The goal is to get production healthy again.
*Why We Only Started With GitHub and Vercel
*
Right now StackCircuit is pretty focused. Its mainly built around GitHub and Vercel instead of trying to support every cloud platform possible.
Obviously we could have tried adding AWS, Azure, Cloudflare and a bunch of other platforms from the beginning. But that would've meant spending a lot of time building integrations before we even knew if people cared about the main idea.
Also, GitHub and Vercel is what we use ourselves. So it made sense to start there.
There are a ton of small teams using basically this exact setup. GitHub, Vercel, Next.js, TypeScript, and maybe a couple other services. They can push updates incredibly fast and sometimes deploy multiple times a day. But a three person startup probably doesn't have someone sitting there specifically watching production every time they deploy something.
And they probably shouldn't need to.
Large companies can have entire teams dedicated to reliability, monitoring and incident response. Smaller teams obviously don't have that luxury. When something breaks, the person dealing with it could literally be the same person who wrote the code an hour earlier.
We wanted StackCircuit to be useful for those teams first.
*The Part That Became More Interesting to Us
*
The more we worked on StackCircuit, the more we started thinking about something slightly bigger than just rollbacks.
Where does a deployment actually end?
Right now, most deployment pipelines are really good at everything leading up to production. You write the code, open a pull request, run tests, run type checking, build the project, generate a preview and eventually deploy it.
Then production goes live and, for the most part, the deployment pipeline is done.
But just because the build succeeded doesn't mean the release was actually successful.
Thats the gap we found interesting.
Before production theres a ridiculous amount of automation. We have automated tests, linting, builds, previews, security tools, type checking and whatever else a team wants to add. But after the code actually reaches users, a lot of teams still rely on someone noticing that something looks wrong.
Obviously monitoring tools already exist, and we're definitely not pretending we invented production monitoring. But monitoring and actually doing something about a failed release are different things.
If a system knows a new deployment happened at 3:04, production was healthy before 3:04, and immediately afterward an important endpoint started consistently failing, thats useful context. If it also knows exactly what deployment was running before the failure, then it has even more context.
At some point it starts to feel weird that the system can know all of that and still just wait for a person to come fix it.
*What We Think Deployment Could Look Like
*
We're not saying every production error should cause an automatic rollback. That would be a disaster. There are way too many reasons something can go wrong, and sometimes doing nothing is safer than making an automated decision.
But there are also failures where the evidence is pretty clear.
If an application was healthy, a new deployment goes out, it immediately starts consistently failing, and theres a verified healthy deployment sitting right behind it, recovery becomes a lot less ambiguous.
Thats the kind of situation we're interested in.
Over time we think deployment tooling will probably start moving further in this direction. Instead of the pipeline only asking "did the deployment succeed?" it could also ask "should this deployment stay in production?"
Those are two completely different questions.
A build passing tells you the code could be deployed. It doesn't guarantee that users are actually getting a working application.
*StackCircuit Is Live
*
We made StackCircuit publicly available at stackcircuit.dev.
Its still really early, and we're not pretending otherwise. There are things we know need to get better and probably a bunch of things we don't even know are problems yet. Detection can improve, we want to support more setups eventually, and theres a lot we can do around how recovery decisions get made.
Thats also kind of the reason we decided to launch it now.
We could keep building features based on what we think developers want, but eventually you need people actually using the thing. Production environments are weird and every team does something slightly different. We're probably going to learn more from real usage than another few months of us trying to imagine every possible situation ourselves.
StackCircuit isn't going to make production incidents disappear. Things will still break and there will always be failures that require an actual person to investigate what happened.
But some failures aren't that complicated.
Sometimes you deployed something, everything immediately broke, and the version you were running five minutes ago worked perfectly fine.
Developers have spent years making it ridiculously easy to push code into production. We think recovering from the code that shouldn't have been pushed there should probably be easier too.
Top comments (0)