DEV Community: John Mitchell

Deploys should be obvious

John Mitchell — Fri, 16 Jan 2026 05:05:15 +0000

I always use a timestamp as a postfix. Makes issues super easy to track.

One time I did a deploy, and checked the service, and the deploy worked!... once. Then I reloaded the page, and the page was wrong, it was an old version. Then I reloaded and the new version came back!

I'd added a running service. If it was originally running version 1, and I deployed version 2: first load would return v1 page, 2nd load would take v2, then it would flip back to v1, v2, v1...

That was... exciting.

Tests vs Business Value

John Mitchell — Wed, 17 Dec 2025 15:06:29 +0000

Tests are in a higher level language, much simpler than app code. But, they are NOT free!

A test which is wrong, or doesn't match business expectations, is worse than no test at all. With no test, you can tell you don't understand how the code works in some situations. With a wrong test, the bad info gets lost among the other tests.

One time as a DevOps person I tried to help the App Devs with their work. A test failed. But I didn't understand the feature nor code nor the test well enough to know which one to fix! I did NOT want to "just make it work", that would be much worse than doing nothing. So I did nothing and moved on. I was... very salty that an expert Dev like myself couldn't help a simple code/test problem, but that was the best value I could provide: do nothing, there's definitely a bug, I can't fix it.

Junior Broken Feedback Loop

John Mitchell — Wed, 17 Dec 2025 15:05:41 +0000

It's a thinking challenge, not an AI challenge.

A while back, a junior asked me a question. They wanted to do X, they had code, with error Y. So they searched for it, got a page on Stack Overflow, pasted "the answer", then got a new and different error.

They:

didn't understand the original code
didn't understand the original error

This is fine. They then searched for the error and found a relevant page.

This is also fine. However, they:

cut-pasted "an answer" from SO without understanding if it was relevant or not

The junior was hoping to work with a Puzzle: adding information will gradually give them a solution. In practice they are working with a Mystery: more information makes the task harder since they can't distinguish between different aspects.

I focused them on a few relevant details and let them go to it.

<3 RSS for learning

John Mitchell — Fri, 29 Aug 2025 13:10:57 +0000

I adore RSS! Use it literally every single day. I have many feeds on Feedly.com, and add to it every week or two.

Tip: use a service to stream quality content to your RSS feed reader. For Hacker News, http://hnapp.com/ does the trick for me.

I subscribe to a couple dozen authors on Hacker News. If someone has great ideas, and writes well, I'm very happy to learn from them.

Example: in hnapp, search for author:bob1029, there's an RSS link, paste that into your RSS feed reader to see that person's Hacker News comments.

I have an entire "Hacker News" section in Feedly, just with author's comments. Very useful!

WIP notebooks

John Mitchell — Mon, 14 Jul 2025 13:46:49 +0000

I have two types of journals for work: a small-ish disc-bound one for time-based task planning, and a second for notes and research.

The first one is a 5x7 I just got from the dollar store. It turns out they have a bunch that work just fine. Each morning I write out the hours and mark in the meetings I have. I wrote down what I'll say at the Standup (= what I'm working on). During the day I plot each task as it's being done in 15-minute increments, so I have a history of what I'm spending my time on.

This has worked incredibly well! By tracking my time carefully I take breaks and celebrate my wins. Before things got mashed together and work could be a drag sometimes.

The second journal is just a cheap 8.5 x 11 one. It contains: 1) notes "in the moment" when I'm working on something, 2) meeting notes, and 3) research notes.

Capturing notes on paper makes a big difference. I focus more, and re-reading it helps cement the info into my brain. If I have questions or unique ideas they get highlighted so I can discuss with the team later or another time.

AI x Quality

John Mitchell — Thu, 10 Jul 2025 15:15:19 +0000

https://martinfowler.com/articles/exploring-gen-ai/i-still-care-about-the-code.html

I'm a fan of AI. However at the end of the day what we do is build features.

Martin notes that focusing on tests first is a great way to go in the AI age, and I strongly agree.

If feature code doesn't have tests, that's fine. As a team we don't pay for writing and maintaining the test. However if we get "bad" tests, that's SO MUCH WORSE than no tests at all!

If AI writes a test, and a human merges it without thinking, or without thinking enough, that's bad. The entire feature is now at risk. Business and the tech team are lulled into a false sense of safety and security.

The way we as developers know what and how to test, is by writing tests. We slowly gain experience and more deeply understand the business and their requirements. If we delegate easy tests to AI, we're letting our testing and understanding skills weaken. We're at risk of not understanding the test, not understanding the code, and not understanding the business requirements.

Recently a client complained that their test suite had too many "flaky" tests. They were spending a lot of time debugging the tests. Fortunately I had a simple solution: delete flaky tests with prejudice. See my previous post.

Tests only value is in critiquing the feature code. It has no other value nor function. Delete it if it's not creating value.

delete flaky tests with prejudice

John Mitchell — Fri, 04 Jul 2025 17:07:49 +0000

(In response to "What's your biggest challenge in proving your automated tests are truly covering everything important?" on Reddit)

As a Software Engineer, your job is not writing tests. In fact your job is not writing code. It's delivering features reliably and quickly. Tests are just one way to prove to yourself, the team, and the business, that the quality is high enough.

It's a feedback loop.

The best CICD "pipeline" I've ever used was just a shim which automatically runs the project-based tests. If you run the full suite locally, the pipeline won't do anything surprising and it's just a backstop.

Learn your test tool very well, with an eye towards narrowing the scope of tests which run after a code change. This increases the feedback speed.

If you're doing Python: pytest has options like "run this test starting with the last-failing test, then continue" which make it stupid simple to have a super fast dev loop. (Please comment on how to do this with your language/tool, I'm curious)

One tool I use on 100% of my projects is a little thing that runs a script when a file changes. Get to know it and love it, or find a replacement. https://jvns.ca/blog/2020/06/28/entr/

My core dev loop is:
1) write a little test with high-level thoughts about the feature
2) write a little code that implements some of the feature
3) execute "run tests when files change" in a terminal

Then the feedback loop is very fast: edit the high-level test, save the file to immediately see if it worked. Or, add code to the implementation, save the file to immediately see if it worked.

Very often I'm not sure about what to do so I put a "drop into debugger" command into the test or code and then rerun the test. It does some stuff then gives me an interactive prompt. I can single-step the code/test, examine variables, even make API calls. So much fun!

Bootstrapping clarification

John Mitchell — Wed, 04 Jun 2025 14:25:38 +0000

My reviewers pointed out the "Bootstrapping an Infrastructure in 2025" article could use some clarification.

The first part of setting up a cluster has these parts:

Version Control - CVS, track who made changes, backout
Gold Server - only require changes in one place
Host Install Tools - install hosts without human intervention
Ad Hoc Change Tools - 'expect', to recover from early or big problems

"Version Control" these days is Git.

"Host Install Tools" are tools so that when a new computer is booted, it's setup with a base operating system, so it can become a functioning member of the cluster. In other words, PXE. In cloud world it's like AMI or Packer or Docker images.

A "Gold Server" is a server that's central to managing the cluster. Instead of making changes to each individual service machine, an admin registers the change centrally, then lets the cluster make the changes happen. "Ad Hoc Change Tools" is ssh (manual changes) vs the standard path. Ad hoc changes are flexible but dangerous.

When the paper was written, computers were individual little snowflakes. To fix a database server, you'd connect using ssh to the server, figure out what's wrong, then run commands or edit files on the server to fix the issue. This method is fun, effective, and flexible, but breaks down almost instantly. You don't remember what you changed. Other people can change things randomly, and also forget. The system doesn't crash per se, but mostly works. This is worse. The system works except sometimes it acts really strangely and causes an enormous amount of effort to fix.

The Bootstrapping paper recommends another way to make changes:
1) setup a change in the central, "gold" server. Example:

database servers should have "postgres" process running

2) from the gold server, trigger some or all other servers to check for changes

3) when a database server checks the central server, it'll find the "make sure postgres is running" change, and execute that change.

This has a lot of advantages. The major one is "eventual consistency". Changes eventually make it out to all the correct machines.

In a medium or large cluster, very often changes fail. The server isn't up, or is too busy, or something else is going on. A centrally-pushed change is applied to only a subset of servers.

In the "pull" style, each server periodically polls the central gold server for changes. Changes set up once, in the central server, eventually are applied to the appropriate machines.

I despise bash but...

John Mitchell — Wed, 04 Jun 2025 13:44:35 +0000

... use it constantly. It's just so useful.

Two tips:

1) first line of ALL SCRIPTS is:

set -euo pipefail # strict mode

This makes the script crash so you can fix it if any command gets an error, or a segment of a pipe gets an error. It'll also crash if a variable gets used before being set.

A program that does the wrong thing then silently continues, is a bad bad program.

2) rewrite the script in a real language (Python?) if it has more than 3 conditionals or loops.

Peronally I find conditionals to be do-able in Bash, but loops tend to be problematic.

I've written thousands of lines of Perl and Awk and other things in my day, but Bash and Python cover 100% of my work these days.

BONUS:

3) set -o xtrace also known as set +x is also great.

Print each command before it's executed, making code run really obvious. We love obvious.

Bootstrapping an Infrastructure in 2025

John Mitchell — Tue, 03 Jun 2025 14:16:35 +0000

Notes on "Bootstrapping an Infrastructure"

My job is Cloud Ops at a large media company. We're moving a ton of users and other resources to a new cloud tenant. I enjoyed the opportunity to re-visit the classic paper Bootstrapping an Infrastructure by Steve Traugott and Joel Huddleston, published all the way back in 1998. They compare booting a cluster to booting a computer - each is composed of a large set of services, each one supporting the following ones.

Steps

bootstrap diagram

Summary of paper

They model a cloud made of many machines, as a single machine, not as a collection of "pet" computers.
By following a specific series of steps, each one supporting the others, a single cloud is constructed.

The paper has a whole section on "Infrastructure Thinking":

Providing capable, reliable infrastructures which grant easy access to applications makes users happier and tends to raise the sysadmin's quality of life.
The "virtual machine" concept simplified how we maintained individual hosts. Upon adopting this mindset, it immediately became clear that **all nodes in a "virtual machine" infrastructure needed to be generic, each providing a commodity resource to the infrastructure.** It became a relatively simple operation to add, delete, or replace any node.

Commentary

The 16 steps are in four layers, each one building atop the layers that came before. Each layer focuses on a single audience, and delivers a specific feature in the cluster to that audience.

The four layers are:

Infrastructure
Support
Client Hosts
Cluster services

Infrastucture

Version Control - CVS, track who made changes, backout
Gold Server - only require changes in one place
Host Install Tools - install hosts without human intervention
Ad Hoc Change Tools - 'expect', to recover from early or big problems

These tools support the cluster management, and are designed for the cluster admins only. Like all layers, they support the layers above.

Coming from a modern/cloud perspective, this is very familiar and very different. Version control and central "server" makes sense. Host Install means PXE: machine boots, asks central server what OS distribution and customization to install, and does that over minutes/hours. The modern equivalent would be an AWS AMI (machine image) or Hashicorp Packer or Docker image.

This is great: start with nothing, install a full blob all at once. If it doesn't work as expected, iterate. Tweaking individual machines is fine for experimentation but acknowledges that local data is ephemeral and will be reset soon.

Support

Directory Servers - DNS, NIS, LDAP
Authentication Servers - NIS, Kerberos
Time Synchronization - NTP
Network File Servers - NFS, AFS, SMB
File Replication Servers - SUP

These services provide low-level data to the cluster and to users. DNS (cluster-wide network names) and LDAP (~ shared user, printer, other resource info) provide trusted low-level data to the cluster. Authorization requires two way trust: a server only allows access if person knows a secret password. NFS (Network File System) provides raw storage to the cluster, to be used by higher-level services.

The Support services make different types of data available to the cluster.

Surprise: the concept of "replication", where data is centrally managed but then copied to local machines, isn't something I've seen much. I guess it makes sense. In the world of physical machines, being able to provide apps to local users even if the network is gone, is a great idea.

Client

Client File Access - automount, AMD, autolink
Client OS Update - rc.config, configure, make, cfengine
Client Configuration Management - cfengine, SUP, CVSup
Client Application Management - autosup, autolink

Unlike modern cloud, the paper talks about apps running locally on each physical machine.

The Client services manage app support at the single, cluster level:

Automount: each machine makes available specific parts of the shared network file system for the local user(s) and services.
OS Update: machine operating systems are managed centrally.
Client Configuration and Application Management: at this layer, individual machine differences are managed centrally. If a user wants to make a local change, it's setup and managed centrally, so the entire machine can be replaced without concern.

Cluster Services

Mail and Printing: these are user-level services that are managed and maintained centrally, but are understandable and directly usable by the end users.

Monitoring: Another cluster-level service, this one's audience is the admins themselves

More Commentary

Reading this paper from the late 1990s was enlightening and also surreal. Many details have changed (Perl! Cfengine! brrrrrr), but the overall flow of the ideas is 100% solid. In the modern world, when a new cloud provider or "tenant" is onboarded, the sequence of layers is extremely similar.

Surprise: of the dozens of tools/services mentioned only two are still in common use: DNS and NTP. And admins still love to complain about DNS breaking things.

Surprise: the authors didn't divide things into layers, nor did they mention "audience" except for "client" as in user-facing apps.

Surprise: no security services! I guess a Web Application Firewall would be a big ask in the 1990s, but a central "these things are happening on those machines by these users" service would be valuable. E.g. AWS CloudTrail or CloudWatch Logs or Splunk.

Similarly, no app dev services: application logs or traceback collectors. ~ New Relic, Datadog. Years ago as a dev we lived by our Sentry app showing us where our app was crashing.

Surprise: authors put "cluster monitoring" at the very end of the process. They mentioned never getting around to central logging! This was shocking to me: they spend a huge amount of time controlling each layer, without the support of cluster-level feedback mechanisms. "Cluster Admin" is an important audience. Cluster-wide services can be divided into "Infra" (for the admins), or "Common" (for end users). Central logging and networking and security services are "Infra", CICD pipelines are "Common".

I study and teach Feedback Loops. The central idea is: 1) make a change, 2) receive feedback, 3) adjust the next change loop based on feedback. Presumably the authors would ssh into each machine, make a change from the central server, then watch on the local machine what happened. This is fine: it's easy to get multiple high quality logs and other data locally. However some problems only show up at the cluster level, over larger time scales.

Developers talk about "Test Driven Development". Instead of developing a feature by writing code, a feature is developed by 1) writing a test which fails, 2) writing "just enough" feature code to get the test to pass, then 3) refactor the test and code. Tests are an artifact that require investment but give value forever. Test automation gives devs and the business the confidence that new changes don't break business-critical features.

For a cluster (or cloud tenant), this helps tremendously. Build the cluster-wide feedback services first. This gives rapid, reliable, actionable feedback to the whole boostrap process.

Test Driven Development hasn't reached all of the Cloud / DevOps world for some reason. Maybe it's time for me to publish more articles and videos...

Starting Tests from Zero

John Mitchell — Fri, 27 Dec 2024 17:19:42 +0000

BEER

Often developers and businesses can't see the value that testing gives. Devs see it as a "tax", an extra thing that slows them down from doing their "real job". Businesses go into magical thinking mode, that Devs are always creating flawless features that deliver whatever the Business thinks it does and fits into the system.

Tests don't slow down development, then pull the development forward. Writing "happy path" Unit Tests shows Devs what to work on next. Investing in high level End to End tests gives the Business confidence the system is working reliably and new changes aren't dangerous.

Here's how to get started with tests if you have non.

Top-down vs Bottom-up: two different Audiences

Write one or more end-to-end tests (E2E). This gives confidence that the entire app is solid, at the cost of being slow and not very detailed. The main benefit is for the Business: they can see and understand the results.

Back in the day, for a shopping site, our E2E test was "create user, add item to cart, checkout, validate there's a total cost" kind of thing. It was slow but had really obvious business value. Nowadays you'd do something like Playwright for this.

Write lots of "happy path" unit tests, generally one per non-obvious function. These are fast and give a lot of info, but for a tiny scope: one function. Generally any time I have to think, I write a test.

Also write a unit test per bug, even if it's just "this doesn't work" or "expect some sort of Exception to happen here". Tests can be vague if that gives you actionable feedback as a developer.

publish Test Coverage! This is great as you and your team -- and more importantly the Business -- can see this number and how it changes over time. New code makes number go down? Write tests for the new code (or old code, if that makes sense for your team). Bug happens? Write test -- number goes up -- then fix the bug.

The overall goal is getting into a Value Feedback Cycle. For the Business audience, they want confidence the app is functioning as expected, that a new deploy won't bring down the site. For the Dev audience, they want fast actionable feedback for them to complete the features. The two types of tests: End to End and Unit Tests, give this feedback to the two audiences.

Advanced: unit tests don't cover much code. Consider adding API-level tests like with Postman. These are cheaper in that a single test checks more code, even though at a less detailed level. They're also fun for the Business and Devs to see because a single API test makes the Test Coverage go up more than from a single Unit Test.

Starting Tests from Zero

John Mitchell — Fri, 27 Dec 2024 17:02:24 +0000

Here's how to get started with tests if you have non.

Top-down vs Bottom-up: two different Audiences

Write one or more end-to-end tests (E2E). This gives confidence that the entire app is solid, at the cost of being slow and not very detailed. The main benefit is for the Business: they can see and understand the results.

Write lots of "happy path" unit tests, generally one per non-obvious function. These are fast and give a lot of info, but for a tiny scope: one function. Generally any time I have to think, I write a test.

Also write a unit test per bug, even if it's just "this doesn't work" or "expect some sort of Exception to happen here". Tests can be vague if that gives you actionable feedback as a developer.

publish Test Coverage! This is great as you and your team -- and more importantly the Business -- can see this number and how it changes over time. New code makes number go down? Write tests for the new code (or old code, if that makes sense for your team). Bug happens? Write test -- number goes up -- then fix the bug.