Posted on Jul 26

Programming rituals nobody warned you about

#ai #webdev #discuss #programming

Bash silently lies to you. Django ships your secret key. Python ghosts your async task. Nobody put this in the onboarding doc.

You passed the interview. You shipped the feature. Maybe you even got the senior title. And then, somewhere between your third production incident and your fifth “wait, that’s just how it works?” moment, you realized something: a huge chunk of this job is memorizing a list of traps that nobody officially told you about.

Not edge cases. Not obscure bugs. Traps baked into the tools themselves sitting there, patient, waiting for the one time you forget the ritual.

I’ve been collecting mine for years. Some came from incidents I caused. Some from watching teammates debug something for hours only to find the fix was four characters typed at the top of a file. A few came from reading post-mortems written by engineers at companies way bigger than mine, who made the exact same mistake for the exact same reason: the default behavior was broken, and nobody warned them either.

The good news: once you know the ritual, you never get burned by that particular fire again. The bad news: there’s always another one. The list never stops growing.

TL;DR: This is a tour through five rituals every working developer eventually collects Bash’s silent failure mode, the many flavors of dumb substitution, Django’s charming habit of hardcoding your secret key, and Python quietly garbage-collecting your async task mid-flight. None of these are your fault. All of them will cost you if you forget.

Bash’s “optimistic nihilism” mode

Here’s a fun game. Write a Bash script. Make one command in the middle fail. Don’t add any error handling. Run it.

Bash will finish the script. Every line after the failure. Cheerfully. Like nothing happened.

That’s not a bug report I’m filing that’s the default behavior. Bash, by design, treats a failed command as a suggestion. You said run these things. It ran them. Whether they succeeded is kind of your problem.

It gets better. If a command in a pipeline fails, the pipeline’s exit code is determined by the last command not the one that actually broke. So you can have garbage data flowing through three pipes, the whole thing producing nonsense, and Bash will look you in the eye and return exit code 0. Success. Great job everyone.

And if you reference a variable you forgot to define? Bash assumes it’s an empty string. No warning. No error. Just silently substitutes nothing and keeps going. Which is fine until that variable was part of a path, and now you’re running rm -rf / instead of rm -rf /some/actual/directory.

That last one isn’t hypothetical. In 2021, a Kyoto University backup script wiped 77TB of research data because an undefined variable expanded to empty, turning a targeted delete into a root-level one. The researchers lost years of work. The script ran to completion without complaint.

Cloudflare had their own version of this. A deployment script piped the output of a failed command into the next stage, which consumed the garbage and cascaded into a global outage. The kind of thing that ends up on the front page of Hacker News with 800 comments, half of them saying “just use set -euo pipefail."

Which brings us to the ritual.

set -euo pipefail

Four options. One line. You put it at the top of every Bash script, no exceptions:

-e stop the script if any command returns a non-zero exit code
-u treat unset variables as errors, not empty strings
-o pipefail fail the pipeline if any command in it fails, not just the last one

That’s it. That’s the whole ritual. Three decades of Bash, and “stop if something goes wrong” is still opt-in.

The reason it’s not default is mostly historical old scripts relied on Bash’s permissive behavior, and flipping the default would have broken them. Which is a completely reasonable engineering decision made in the 1980s that the rest of us have been paying for ever since.

The real tell is when you join a new codebase and grep the shell scripts. If half of them are missing set -euo pipefail, you know exactly how the team learned about it or didn't. It's a pretty reliable proxy for "has this project had a bad day yet."

Add it. Every time. Before anything else. The one day you forget is the day a variable comes up undefined on a prod server and you spend the next four hours figuring out what it deleted.

Django said “just commit the secret key, it’s fine”

There’s a specific kind of betrayal that hits different when it comes from a framework you actually like.

Django is genuinely good. Batteries included, sensible defaults, mature ecosystem, excellent documentation. It’s the kind of framework that makes you feel productive fast, which is exactly why the first thing it does to a new project is so magnificently at odds with everything else it stands for.

django-admin startproject myproject
cat myproject/settings.py | grep -i secret

# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = 'django-insecure-m&!$#k(56_this_is_literally_in_your_repo_now'

There it is. Your secret key. In your source file. With a comment telling you to keep it secret, immediately above the line where it isn’t.

The django-insecure- prefix is Django's way of flagging that this key was auto-generated and shouldn't go to production. It's a reasonable idea. The problem is that it's just a string prefix in a settings file it doesn't actually stop anything. It doesn't warn you at deploy time. It doesn't block you from pushing it to GitHub. It doesn't trigger any runtime error if you forget to replace it. It's a comment with ambition.

Meanwhile, the rest of the industry has been building infrastructure specifically to avoid this moment. GitHub has secret scanning that’ll flag exposed credentials in pushed commits. AWS has Secrets Manager. Every CI platform has encrypted environment variables. HashiCorp built an entire product around the idea that secrets should never touch source code. And then you run django-admin startproject and there's your key, sitting in settings.py, one git push away from being public forever.

The number of public GitHub repositories containing django-insecure- in a committed settings.py is not small. It's the kind of thing you can find with a five-second search, and people do.

The ritual is straightforward and should honestly be the default:

import os
SECRET_KEY = os.environ.get('DJANGO_SECRET_KEY')

Or if you want something slightly more ergonomic, python-decouple and django-environ both handle this cleanly pull the key from a .env file locally, from an environment variable in prod, and never let it touch version control at all.

The .env file gets added to .gitignore immediately. The production secret lives in your deployment environment, not your codebase. This is not advanced security practice it's the baseline, and it's been the baseline for years.

What makes the Django situation frustrating isn’t that the framework is insecure it isn’t, overall. It’s that this one specific default runs exactly counter to the secure habits the rest of the ecosystem has spent a decade trying to build. A junior developer running startproject for the first time has no particular reason to know that the generated file is a trap. The warning comment is easy to skim past. The key looks like configuration, not a credential.

Fix it in the first commit. Add it to your project template. Make it the first thing in your team’s Django onboarding doc. Because the alternative is finding out about it the way most people do after the push, after the scan, after the “your repository may have exposed credentials” email from GitHub at an unreasonable hour.

Python’s garbage collector will silently fire your async task

At some point in your Python career you will write something like this:

async def notify_user(user_id):
    await send_email(user_id)

async def handle_request(user_id):
    asyncio.create_task(notify_user(user_id))
    return {"status": "ok"}

Clean. Async. Non-blocking. The request returns immediately, the notification fires in the background. Exactly what you wanted.

Except sometimes it doesn’t fire. Not always just occasionally, unpredictably, in a way that doesn’t produce any error, any warning, or any log entry. The task simply doesn’t run. The email doesn’t send. The user never hears back. And your monitoring shows zero exceptions because nothing actually failed the task just ceased to exist before it had a chance to start.

What happened is that Python’s garbage collector looked at your task, noticed nothing in the program held a reference to it, and collected it. “No reference” means “not needed” in GC logic, and GC logic doesn’t know or care that your task was mid-flight. From the runtime’s perspective, you created an object, immediately dropped it, and the cleanup crew did its job.

This isn’t a bug in asyncio. The Python docs actually document it, in a note that’s easy to read past the first time:

“Important: Save a reference to the result of this function, to avoid a task disappearing mid-execution.”

That’s the whole warning. One italicized paragraph. For behavior that will silently drop work in production.

The ritual is keeping a reference alive for as long as the task needs to run:

tasks = set()

async def handle_request(user_id):
    task = asyncio.create_task(notify_user(user_id))
    tasks.add(task)
    task.add_done_callback(tasks.discard)
    return {"status": "ok"}

The add_done_callback is the clean part once the task finishes, it removes itself from the set automatically. You're not leaking memory, you're just giving the task somewhere to live until it's done. That's all it needed.

If you’re on Python 3.11 or newer, asyncio.TaskGroup is the more structured solution:

async def handle_request(user_id):
    async with asyncio.TaskGroup() as tg:
        tg.create_task(notify_user(user_id))
    return {"status": "ok"}

TaskGroup manages lifetimes explicitly, handles exceptions properly, and generally makes background task management feel like something the language actually supports rather than something you’re working around.

The reason this one is particularly brutal to debug is the failure mode. A missing SQL parameterization blows up loudly. A hardcoded secret key shows up in a GitHub scan. Bash without set -e at least leaves evidence in the logs if you know where to look. But a GC'd async task produces nothing. No stack trace, no error code, no entry in your observability platform. Work just quietly doesn't happen, and you only find out when a user complains or a metric drifts in a direction that takes a while to connect back to the right cause.

I’ve seen this burn a team running a background job processor for three days before anyone noticed the completion rate had dropped. Every individual request looked fine. The service was healthy by every dashboard metric. The tasks were just evaporating before they ran, silently, at a rate that depended on GC timing and load which meant it was inconsistent enough to look like a data issue at first.

The fix took twenty minutes once they found it. The finding took three days.

Add the reference. Use TaskGroup if you can. And maybe add a completion counter to any background task that actually matters, so you know when the numbers stop adding up.

The ritual tax compounds

Here’s the thing nobody tells you when you’re starting out: the job doesn’t get easier as you get more experienced. It gets differently hard. Early on, the hard part is understanding how things work. Later, the hard part is remembering everything that secretly doesn’t.

Every incident adds a line to the list. Every post-mortem ends with “and going forward, we’ll always remember to…” and then someone writes it in a Confluence doc that gets read once and slowly buried under newer Confluence docs. The ritual exists. The knowledge exists. Whether it survives the next team rotation is a different question.

That’s the compounding problem. It’s not that any individual ritual is complicated set -euo pipefail is four characters, parameterized queries are a one-line swap, moving a secret key to an environment variable takes about ninety seconds. The complexity is in the sheer volume of them. Five years in, your mental checklist for "things that will silently wreck you if you forget" is long enough that forgetting one isn't a question of skill. It's just probability.

And the list keeps growing. Every language adds entries. Every framework has its own charming defaults. Every new tool comes with its own set of rituals that aren’t in the README, only in the GitHub issues filed by people who hit the wall before you.

The better-designed tools are trying to fix this. Rust makes a whole category of memory mistakes structurally impossible. Go forces you to handle errors explicitly you can ignore them, but you have to do it on purpose. Some newer frameworks ship with secure defaults and make the insecure path the harder one. That’s the right direction: push the ritual into the language, not into the developer’s memory.

But most of the stack the average engineer works with wasn’t built that way. It was built to be flexible, to be backwards-compatible, to not break the scripts people wrote in 1994. Which is fine. It just means the ritual tax is real, it’s ongoing, and it falls on you.

None of this is your fault. The defaults were set before you got here. The design decisions made sense in context. You inherited a codebase full of traps that were laid by reasonable people under reasonable constraints, and your job now includes knowing where they are.

What I’d push back on is the idea that collecting rituals is the same as getting better. It’s not. It’s just getting older in the industry. The devs who actually level up are the ones who, after getting burned, ask why the trap was there at all and either document it properly, automate the check, add it to the linter, or at minimum write the post-mortem that means the next person doesn’t spend three days finding a twenty-minute fix.

AI tooling is starting to catch some of these Claude Code will flag a missing set -e, Copilot will suggest parameterized queries, static analysis has been doing this for years in some languages. It helps. It's not complete coverage, and it's not a substitute for understanding why the ritual exists, but the gap between "things that will silently break your code" and "things your tooling catches before prod" is narrowing.

Until it closes: keep the list. Add to it every time something burns you. Share it with the next person who joins your team before they find out the hard way.

It’s not our fault the list exists. But it is our job to pass it on.