Adrian Rinnus

Posted on Sep 17

Production Is Where Bugs Go to Party 🐛🎉

#webdev #startup #cicd

This Time I’m Doing Everything Right

I’m sitting in the office, staring at my screen. Dinner is almost ready.

It’s 5:00 PM.

This time I’m doing everything right.

Merge early, leave time for proper testing, prepare my demo script for tomorrow’s presentation.

Not just click through my own features, but also understand the ones my teammates built.

Last time I just merged to production, loaded the data and thought:

“It’ll be fine.”

It wasn’t fine.

The presentation the next day was pure chaos – improvised workarounds and awkward explanations.

But not this time.

I merge the new code into production, open the site – everything looks good.

I lean back, take a deep breath, feel my body relax.

It feels like the calm after the storm.

I head to dinner.

While I’m eating, my buddy calls.

“So, are we still on for sports tonight?”

I look out the window, still optimistic, and say:

“Let’s postpone. I want to use the time to prepare the presentation.”

I get back to my desk, open my demo script.

While clicking through, I test the app again.

Everything works fine in the admin role.

Then I switch roles.

Click → loading spinner → nothing.

My stomach knots up.

My neck tightens.

My pulse jumps.

“Not again…”

I message my teammate:

“Dashboard doesn’t load for user role. Can you check?”

We start debugging together.

Minutes turn into hours.

My plan to go to bed early is long gone.

At 11:30 PM we’re still staring at server logs.

Finally, we find the culprit:

A query that was lightning-fast on my laptop had turned into a snail on the server.

This time, though, we made a conscious decision:

We shipped with the known performance issue and communicated it clearly in the client presentation.

And guess what – it wasn’t a big deal.

My confidence during the presentation mattered more than the bug itself.

I still spent the weekend fixing it –

but this time with a clear plan, not panic.

Now we have a staging environment, realistic data –

and I can merge without my stomach doing somersaults.

How We Walked Right Into the Trap

Looking back, the bug wasn’t really a surprise.

If I’m honest, we almost invited it.

We had been cranking out new code for weeks –

and a lot of it was AI-generated.

And I have to admit: I was excited.

The first suggestions looked great, everything worked locally,

and I felt like I was moving ten times faster than before.

And that’s when I let go of the reins.

I stopped reviewing as carefully.

“If it works, it can’t be that bad,” I told myself.

But with every merge, the code got more complex.

Our RLS policies became more and more nested,

and we had multiple joins across several tables.

On my laptop it all ran like butter.

Even the complex queries took virtually no time.

And that was the trap.

We never tested on the server –

no network latency, no real load, no production-scale data.

In production, these queries suddenly took so long

that the dashboard just sat there loading.

The fix was surprisingly simple – and honestly, liberating:

We eliminated the joins completely by storing the relevant IDs directly in the tables.

The result: 96% runtime reduction.

And the code is now simpler, cleaner, and less error-prone.

The big insight:

We had let complexity creep in without validating it under real conditions.

🎯 My Key Takeaways:

AI code isn’t a free pass. It can speed you up, but it can also lead you into a dead end.
Complexity eats performance. Joins aren’t evil, but they have a cost.
Local ≠ real. My laptop is not the truth. The real world has latency, big data, and nasty edge cases.

Now I know: if everything works perfectly on my machine,

that just means the real test is still ahead.

The New Solution: Staging Born Out of Necessity

After that release, it was clear:

We couldn’t risk shipping slow code to production again.

The system was already in use – one more major issue could break trust.

So a staging environment went from “nice-to-have” to mandatory.

Our challenge:

We don’t have revenue yet – that comes after the successful release.

We needed a way to deploy staging cheaply but with realistic conditions.

Here’s what we did:

Second Supabase account:

The free tier only gives you two databases,

which were already used by our production products.

So we spun up a second account.

It runs a copy of the production schema with anonymized data.
Deployment on AWS Amplify:

Not because Vercel is bad – we actually like Vercel –

but we have AWS Startup Credits.

This means our staging deploys are effectively free.

The only “extra” step compared to Vercel:

you need to create an amplify.yml and add all your env variables there.

Once done, it just works.
Logs:

They’re not as pretty as Vercel’s,

but they get the job done.

The best part:

Every developer can use staging.

We can log in, switch roles, run through real scenarios –

and do it all without touching production data.

This setup made all the difference.

We were able to test our performance optimizations against staging data

and, in just one weekend, deliver a solution that made our queries 96% faster –

without shipping more bugs to production.

🎯 My Key Takeaway:

Staging isn’t just a safety net –

it’s a turbocharger for structured debugging.

It gave us realistic conditions, let us isolate problems,

and helped us deploy with a clear head.

A New Release Process – Finally Calm

Releases used to feel like Russian roulette.

Merge straight to main, test quickly, cross fingers, deploy.

If it blew up, we’d fix it at midnight or sweat it out in front of the client.

Now it’s a whole different story.

We start by creating a release branch.

This gets deployed to staging with anonymized production data.

Then we do what we never did before:

we test together.

Everyone on the team clicks through their features,

switches roles, tries real user flows.

The result:

We catch issues before they reach the client.

We can even run performance tests safely,

without risking downtime.

When everything looks good, we merge to main, deploy to production –

and breathe easy.

No more heart racing, no more stomach knots, no more late-night log marathons.

Just a clear, repeatable process.

🔑 Key Takeaway:

Having a clear release process with staging already removed a lot of stress.

The next step is to add automated tests –

so we can keep shipping fast without fear of breaking things.

Want to follow along as we add more automation,

refine our release process, and learn more lessons the hard way?

Follow me – I share everything I learn as we go.

And if you’re up for it:

Tell me your worst production bug story

or how you handle staging and testing in your projects.

I genuinely reply – I probably learn as much from you as you do from me.