DEV Community: Brandon Bayer

9 ways to improve how you ship software

Brandon Bayer — Tue, 16 Jan 2024 23:58:12 +0000

10x developers are real, and there’s more than one way to be a 10x developer.

Perhaps the most approachable is to make 10 other people 10% more productive.

In this article, I cover things to improve your individual and team productivity. This is valuable to your company, so it will likely help level up your career.

First, we’ll cover 7 principles and then 9 tactics.

7 principles for shipping software

1. Speed

Let’s face it. As developers, we are speed junkies. If we find a tool that’s 10 ms faster, we’re inclined to immediately rewrite our entire application.

But when it comes to how often we deploy our code, we tend to be much more cautious. We’d rather deploy tomorrow or next week, so we don’t have to stay late today to fix whatever problems might happen.

Like Charity says, it is a deep seated biological impulse to slow down when we feel cautious. For humans, slow is safety.

But software physics is different. It's more like riding a bicycle. The slower you go, the more dangerously you wobble.

Speed is safety. Small is speed.

When it comes to deploys speed is safety, and small is speed.

The speed of shipping has massive compounding effects. If your deploy takes hours, you’re going to want to make damn sure there are no bugs, because if you find a critical bug right after a deploy goes live, then you have to wait another couple hours for your fix to go live.

We must aim for speed in every part of the software lifecycle.

Fast feedback loops
Fast builds
Fast tests
Fast deploys
Fast fixes
Fast rollbacks

2. Reliability

The second principle is reliability. Our systems must be dependable.

Flaky CI/CD issues have to be one of the worst possible things in the universe.

We must avoid flakiness at all costs.

3. Reproducibility

In order to build and maintain complex systems, we need to know that if we do X, we get Y every time. If one time you do X, you get Y, but another time you get Z, you will lose confidence in your systems.

Basically, we want to use scripts and code to control everything we do instead of manual clicking, manual commands, etc.

4. Calm on-call rotations

Somewhat self-explanatory, no one wants to dread on-call rotations. If they are hectic, it will degrade morale and cause all sorts of issues. So it’s something we should explicitly prioritize. It can serve as a backstop of sorts to make sure what we’re doing is truly working well.

5. Easy to debug

Everyone causes bugs. Even seniors. Even when you have tests. Even when you have 100% test coverage. You will ship bugs to production.

So you must be prepared to fix production bugs. This is an area where speed is important. And as part of that, it must be easy to debug.

If it’s hard to debug and slow to deploy fixes, it’s going to slow you down because you are going to add lengthy QA processes, and before you know it, you’ll be deploying code only once every few weeks.

But even with lots of QA, you’ll still have production bugs.
Production is the only tried-and-true way to find all the bugs.

Our systems must be easy to debug.

6. Self-serve infrastructure and deployments

In the old way of doing infrastructure and code deployments you had two separate teams. Devs and Ops. If devs needed a new database, they filed a ticket with ops. If they need a new service, file a ticket with ops. Need to deploy your code? File a ticket with ops.

Naturally, this has several problems:

Devs are often blocked, waiting on ops to get to some ticket.
Devs aren’t monitoring infrastructure, and likely won’t get feedback on whether their code is fast or slow, efficient or inefficient.
The whole organization is slowed down.

We, as an industry, figured out it’s much better for devs to deploy and run their own code. This is more efficient, with fewer blockages and less communication overhead, and devs get more feedback from the real world on how their code is performing.

7. Ship on Fridays

Healthy engineering organizations deploy multiple times per day, including on Friday!

If you are afraid to deploy on Friday, it’s likely from one or more of the following:

You don’t have reliable deploys.
You have slow deploys.
You don’t have good monitoring and alerting.
Your app is hard to debug.
You don’t have tests.

Those are all things you must fix.

9 tactics to improve how you ship software

With the above principles in place, let’s look at 9 specific tactics to implement them in practice.

1. Decouple deploys from releases

Deploying software is a terrible, horrible, no good, very bad way to go about the process of changing user-facing code. \
—Charity Majors

Fundamentally, there are two possible actions for changing code in production: deploys and releases.

Releasing refers to the process of changing the user experience in a meaningful way.

Deploying refers to the process of building, testing, and rolling out changes to your production software.

The traditional process is that you change the code on a branch, and when it’s ready, you merge and deploy it. Once it’s deployed, users will see the new code.

But today’s modern engineering organizations use feature flags.

What is a feature flag?

It allows you to write new code alongside the old code, and then, based on a runtime feature flag, your app decides whether to run the old or new code.

This is huge! Remember how speed is one of the key principles? Separating deploys from releases speeds up both. It speeds up deploys because engineers can deploy their code as soon as it’s ready without waiting for product management to be ready to release it. And it speeds up releases because now you can release a feature instantly. And rollback a feature instantly. Without needing a deploy. Product managers can do releases on their own schedule.

Furthermore, it unlocks advanced methods like progressive rollout, where you enable the feature flag for a percentage of your users. You could start out with 5% of users. If everything is working, then increase to 10%, 20%, etc. This is the best way to roll out changes at scale because if there are problems, only a subset of your users will experience them instead of all.

And feature flags are superior to long-lived feature branches for development flow. You have fewer merge conflicts, less stale code, more team momentum, better team morale, and more flexibility for testing things with real users.

I encourage you to introduce this idea at work. And next time someone has a long-lived feature branch instead of a feature flag, use this:

2. Continuous deployment (CD)

Deployments are the heartbeat of your engineering organization. And like a human heart, you want a steady rhythm of simple heartbeats. And with this steady rhythm of deployments, you get momentum. And momentum is life for organizations.

Along with momentum, you get efficiency, being able to deliver code to production as quickly as possible after it’s written.

You want to deploy as often as possible, like multiple times per day. Every commit should be deployed. Small, understandable changes mean you have debuggable and understandable deploys. The engineer who writes the code should merge their own PR and then monitor production for any unexpected issues.

Feature flags are basically a prerequisite for this because they allow your team to ship small changes continuously instead of having long-lived feature branches.

3. Set up alerts

Make sure you set up alert notifications for the following anomalies:

Failed builds
Failed deploys
Service downtime
Unhealthy servers
Unexpected errors
Unusual amount of traffic
Statuses of third-party services (many have a status page that you can subscribe to in Slack)

The benefit is relatively obvious, but it does take at least a bit of effort to set them up, so this is something many teams can do as a quick win.

And if you already have alerts, you should audit them:

Are they useful?
Are you getting too many false alerts so that you ignore them?
Are there things you should add new alerts for?

4. Optimize deploy speed — 15 minutes or bust

Your target for a happy team is 15 minute deploys or less.
Of course, we want them to be as fast as possible, but the neighborhood of 15 minutes is the baseline target. Significantly longer than this is significantly painful. But significantly faster than this could take more work than it’s worth, depending on your circumstances.

There are a number of ways to improve your speed of deploys.

a. Track and measure your current deploy times

You want to understand your starting point and which parts of the process are taking a long time. That’s where you want to focus initially.

b. Speed up dependency installation

Dependency installation is often time consuming, so it can be a good first step.

Switch to a faster package manager.

If using javascript, switch to pnpm. It’s a drop-in replacement for npm and yarn that has significant performance and caching improvements.

Cache the entire install step.

Use your build system to cache this entire step, so this step doesn’t even run if your dependencies haven’t changed.

Example with Docker:

Example with Nixpacks:

c. Improve your build speed

The main way to improve your build speed is to switch to a faster bundler.

For example if you are using create-react-app which uses webpack, you can switch to Vite which is way faster.

If using Next.js, upgrade to the latest and make sure you don’t have a .babelrc file; this will use SWC instead of babel which is a lot faster.

If you like living on the edge, you can try Turbopack by passing the --turbo flag to the next commands, but it’s still experimental and doesn’t work for all setups.

d. Set up build caching

If you are using Docker, there are several ways to improve caching:

Optimize your build layers.
Use multi-stage builds.
Set up remote caching.

When building locally, Docker caches by default and is fast, but in CI you usually don't have a permanent machine. You have to manually set up remote caching. Refer to the Docker documentation for more information.

e. Globally cache everything

If you want to really live with your hair on fire, in a good or maybe bad way, there is another style of caching that can give you big wins, as long as you have a javascript project.

This caching style works by having a single shared cache between your entire team and also CI.

This means if you build a commit locally and then push that commit to CI, the CI build will download the previous cached build from your local machine and be done in mere seconds. Or if someone else on your team already built a commit, and then you run the build locally, it will again download the cache and be done in mere seconds instead of building from scratch again.

You might be thinking, “ok Brandon that’s cool and all, but usually we’re changing code so it won’t be cached”. But the trick is that this caching works on a package level, so assuming you have a monorepo, you are likely only working in one package at a time. So this enables you to iterate faster and not have to build things you aren’t working on.

There are two tools that do this: NX and Turborepo.

There is a gotcha. You must be careful with how you handle environment variables. I’ve seen numerous people have production issues because a dev ran the production build locally but with dev or staging environment variables. And then CI uses the cached build with incorrect environment variables. Make sure you set up different commands for dev, staging, and production.

5. Replace staging with preview environments

Preview environments are temporary environments, tied to the lifecycle of a pull request. When you open a pull request, infrastructure can be automatically provisioned for that PR.

This enables stakeholders to see the changes in a production-like environment before it’s merged. And then, when the pull request is merged or closed, its environment will be automatically cleaned up.

Preview environments are a better replacement for long-lived staging environments. Because staging is running all the time, whether it needs to or not. And you have to merge a PR before anyone can verify the change.

Preview environments are a companion to feature flags. Larger changes should use feature flags and will have many PRs for that feature. But small changes are easier to review in a preview environment instead of going through the hassle of managing a feature flag for them.

6. Infrastructure-as-code

Hopefully, you have automated deployments, but you might not have automated infrastructure. Without infrastructure-as-code (IaC), you typically define your infrastructure config by clicking through a dashboard.

Bringing automation to your infrastructure config in the form of infrastructure-as-code has a huge number of benefits, including:

Repeatability or reproducibility
Consistency and standardization
Predictability
Version controlled
Collaboration and review

IaC can take many forms. If using AWS, it can be low level like CloudFormation or Terraform. Or it can be at a higher level of abstraction, designed for product developers. Examples are Flightcontrol’s flightcontrol.json or Render’s render.yaml.

7. Platform Engineering

Platform engineering might be a new term for you, so let me give you the back story that got us to this point.

Back in the old days, you had to rack your own servers in a data center. The abstraction level of infrastructure was extremely low. Then came AWS which introduced the cloud and raised the abstraction level. But it was still too low for the average developer.

This led to the creation of Heroku, to the glee of developers all around the world. But that excitement was not to last because ops were not happy. It turns out abstractions like Heroku do not scale for large companies. So ops take over and create a new abstraction over AWS, infrastructure as code and Terraform. This works well, but it’s still too tedious for developers.

Additionally, company leadership wants to increase operational efficiency, and one way to do that is to provide devs with self-serve infrastructure, so we, as an industry, started creating internal platforms that provide a better developer experience. Through this, we found a way for both ops and devs to be mostly happy. This has come to be known as platform engineering.

Now many larger companies are building an internal platform, sometimes known as an internal developer platform. This platform can look more like an internal Heroku or more like plain Terraform.

Either way, the three key concepts are:

Deploys to your own AWS/GCP account
Self-serve infrastructure with good developer experience
Ops can dictate and customize the underlying primitives.

There are many tools in the space, one of which is Flightcontrol.

8. Work as a team

Back in the beginning of Flightcontrol, it was only my cofounder and me. We had a clear separation of duties aligned with our skillsets. He did all the backend/infra stuff, and I did all the web app and frontend things. It worked great!

But then we started hiring more engineers. We initially kept the same work model where each person was off working on their own project or feature.

It started to feel like something wasn’t right.

We weren’t benefiting from collaboration. Since each person was working mostly alone, we weren’t benefiting from others’ expertise on everything from architecture design to code quality.

We would review each other’s code, but since no one else had the full context of the project, they were only able to give superficial reviews. LGTM!

Then, somewhere during this time, I ran across a few blog posts by Swizec Teller, talking about how most teams don’t actually work as a team, and if you actually work as a team, you can be way more effective.

When we first started working as a team instead of a group of soloists, every story felt like a million gerbils running through my brain.
We moved faster. We got more done. We kept everyone in the loop. Shared more ideas. Found more edge cases. Created fewer bugs. And heard more opinions.

—Swizec

I read this, and my heart was saying, YES YES, but my mind was saying, I don’t understand!

As a team, we decided to fundamentally shift how we worked. We’d no longer have multiple projects in progress at the same time. Instead, we’d have one project, and the entire team would work on it together from start to finish.

Initially, it was scary, awkward, and felt like it wouldn’t work smoothly. And it was rough for a couple months until we got things ironed out. But now it’s many months later, and we can’t imagine working any other way.

We can emphatically say that teams should work as a team. Everything has improved. We ship better features, have fewer bugs, and cover more edge cases. Job satisfaction and team morale increased. Because now they get true collaboration, not a “hi” as you pass them in the slack channel.

How does this work?

Small team (2-6 people, 4 is ideal)
Entire team works on 1 project at a time, together.
Kickoff meeting (deep discussions on how to build)
Break project into small tasks (usually in kickoff)
Work on subtasks in parallel.
Small PRs, at least 1 per day.
Quick reviews. I.e. don’t be a blocker.
Unblock yourself; merge your own safe PR if no one is available to review and you are blocked.

9. Trunk based development

The next level of team productivity is to get rid of pull requests and commit straight to the main branch.

I’m serious.

It’s called trunk based development. If this is a new concept to you, I know you’re thinking this is impossible; there is no way this can work. Even if you’ve heard of this concept before, you are probably thinking, “I have no idea how this could work”.

Think of it like a cafeteria. Your tray is the main branch. When the burger is done, it goes on the tray. When the fries are ready, they go on the tray. When the milk shake is poured, it goes on the tray. Once the tray is full, you’re done.

That's how trunk-based development works. Every feature goes straight to main when it's ready. Whether subtasks or an entire project, it doesn’t matter because they're all working, independent, and deployable.

If committing straight to main is too radical for you, then do the next best thing, which is short-lived branches.

And by short-lived branches, I mean, short lived branches.

You want to be as close as possible to pure trunk-based development as possible.

PRs shouldn’t last more than about a day. For a project, you should have many short PRs that are quick for others to review and easy to merge.

That’s how you get momentum.

Conclusion

We’ve covered seven principles and ten tactics for improving how you ship software.

What do you think of these? Have things to add or take away?

🎉 Announcing Blitz.js: Rails-like framework for full-stack React apps — built on Next.js

Brandon Bayer — Fri, 24 Apr 2020 15:23:30 +0000

Today we're announcing the first alpha release of Blitz.js!

Blitz is a Rails-like framework for monolithic, full-stack React apps.

Blitz brings back the simplicity and conventions of server-rendered frameworks like Ruby on Rails while preserving everything we love about React and client-side rendering!

Blitz is the framework for the 99% of us at companies with <100 employees. This means we don't force you to use advanced technologies like GraphQL. We let you add advanced technologies on your terms and at your pace.

Blitz maximizes your productivity both when starting an app and when scaling it to lots of code and users.

Features:

⚡️ Built on Next.js
⚡️ Don't have to build an API for client-side rendering
⚡️ Client-side rendering, Server-side rendering, and fully static pages all in the same app
⚡️ Full Typescript support with static, end-to-end typing (no code generation step needed like with GraphQL)
⚡️ React Concurrent Mode enabled
⚡️ Database/ORM agnostic, but Prisma 2 is default
⚡️ CLI with code scaffolding, Rails-style console REPL, etc
⚡️ GraphQL Ready
⚡️ Deploy serverless or serverful

Other key features coming:

⚡️ Highly secure authentication
⚡️ Authorization you can use on both server and client
⚡️ Model validation you can use on both server and client
⚡️ Plugins for easily adding libraries like Tailwind, CSS-in-JS, etc.
⚡️ React native support
⚡️ GUI so you don't have to use the CLI

Try Blitz Yourself

npm i -g blitz
blitz new myapp

Read the Alpha User Guide to learn more!

The Blitz.js Manifesto (A New Fullstack React Framework)

Brandon Bayer — Tue, 10 Mar 2020 11:12:21 +0000

Blitz.js is a new Javascript framework for building monolithic, full-stack, serverless React apps with zero data-fetching and zero client-side state management.

Background

Technology follows a repeating cycle of bundling and unbundling. Created in 2005, Ruby on Rails became a major bundling force, making web application development easier and more accessible than ever before. This benefited everyone, from those learning programming to seniors building production systems.

A major unbundling happened in 2013 with the release of React, a hyper focused rendering layer without any opinions for things like styling, state management, and routing. As React grew in popularity, so did the choices for all the other parts, leaving developers with hundreds of decisions to make when starting a new app. While this has contributed to JavaScript Fatigue, it's been a powerful driving force for rapid frontend innovation.

Now, in 2020, is the perfect time for another major bundling. Developers are yearning for an easier, simpler way to build web applications. Beginners want a guiding hand for building a robust app, and seniors want a framework that removes mundane tasks while providing a highly scalable architecture with all the right escape hatches.

Hence the creation of Blitz.

What is Blitz For?

Blitz is for building tiny to large database-backed web applications (and in the future, mobile apps). It's not for building extremely large web apps, like Facebook.com. It's not for building content sites, although you can easily add fully static pages to a Blitz app so you don't need a separate host for your marketing site.

Foundational Principles

Fullstack & Monolithic
Backend API Optional
Convention over Configuration
Loose Opinions
Easy to Start, Easy to Scale
Stability
Community over Code

1. Fullstack & Monolithic

A fullstack, monolithic application is simpler than an application where frontend and backend are developed and deployed separately. Monolithic doesn't mean it will be slow or hard to scale to large teams. Monolithic doesn't mean there isn't separation of concerns. Monolithic means you can reason about your app as a single entity.

2. Backend API Optional

Choosing React for your view layer should not force you to build a backend API. Usually you only need an API to get data to your frontend, unless you need a public API for third-parties or a mobile app. It's extremely expensive to build and maintain an API that's only used by your frontend because it adds a lot of unnecessary complexity, making development slower, maintenance harder, and deployment more complex.

In a Blitz app, you write server-side controllers for all your CRUD operations. These controllers have direct access to your database. You connect them to the pages that need that data, and then Blitz automatically and securely gets your data from the backend controller to your frontend components. You don't make any client-side fetch calls, deal with global state management, or mess with client-side data caching. Mutations work in a similar fashion.

You develop Blitz apps similar to a traditional server rendered framework like Rails, but the end-user experience is as a React Single Page App.

If at some point you actually do need an API, you can easily add a GraphQL API with auto generated resolvers. Or if REST is your jam, you can add that to your Blitz app instead.

3. Convention over Configuration

When starting a new app, you should be able to immediately start developing core app features instead of spending days to configure eslint, prettier, husky git hooks, jest, cypress, typescript, deciding on a file structure, setting up a database, adding authentication and authorization, setting up a router, defining routing conventions, setting up your styling library, and all the other menial tasks for initial app setup.

Blitz will make as many decisions and do as much work for you as possible. This makes it lightning fast to start real development. It also greatly benefits the community. Common project structure and architectural patterns make it easy to move from Blitz app to Blitz app and immediately feel at home.

Convention over configuration doesn't mean no configuration. It means configuration is optional. Blitz will provide all the escape hatches and configuration options you need for bespoke customization.

4. Loose Opinions

Blitz is opinionated. The out-of-the-box experience guides you on a path perfect for most applications. However, Blitz isn't arrogant. It totally understands there are very good reasons for deviating from convention, and it allows you to do so easily. For example, Blitz has a conventional file structure, but, with few exceptions, doesn't enforce it.

And when there's not consensus among the React community for a thing, blitz new will prompt you to choose the approach you want, such as Tailwind CSS, Theme UI, or styled-components.

5. Easy to Start, Easy to Scale

A framework that's only easy for one end of an application lifecycle is not a good framework. Both starting and scaling must be easy.

Easy to start includes being easy for beginners and being easy to migrate existing Next.js apps to Blitz.

Scaling is important in all forms: lines of code, number of people working in the codebase, and code execution in production.

6. Stability

In the fast-paced world of Javascript, a stable, predictable release cycle is a breath of fresh air. A stable release cycle ensures minimal breaking changes and ensures you know exactly what and when a breaking change will occur. It also minimizes bugs in stable releases by ensuring features are in beta for a minimum amount of time. Ember is the model citizen in this regard.

The exact details of the Blitz release cycle are to be determined, but we'll follow a pattern similar to Ember which strictly follows SemVer with stable releases every 6 weeks and LTS releases every 6 months.

Blitz will follow a public RFC (request for comments) process so all users and companies can participate in proposing and evaluating new features.

If a Blitz API needs to be removed, it will be deprecated in a minor release. Major releases will simply remove APIs already deprecated in a previous release.

7. Community over Code

The Blitz community is the most important aspect of the framework, by far.
We have a comprehensive Code of Conduct. LGBTQ+, women, and minorities are especially welcome.

We are all in this together, from the youngest to the oldest. We are all more similar than we are different. We can and should solve problems together. We should learn from other communities, not compete against them.

Now Accepting Sponsorships and Donations

Funds will be used to replace my consulting revenue so I can work more on Blitz and get it production ready asap (probably late Q2). With sufficient funds, other contributors will be supported too!

This is a great chance to get your business in front of early adopters!