DEV Community: Najeen Nepali

Fell in Love: Ephemeral Environments and DevEx

Najeen Nepali — Tue, 20 Oct 2026 11:25:47 +0000

How a conference talk, a hardcoded branch, and too many deployment requests led me to ephemeral environments.

Remember how it started?
I was at a conference, volunteering.
Someone said "ephemeral," and I got hooked.
Remember when I fell in love?

I do.

April. NDC Sydney.

I wasn't on stage, or even there as a regular attendee.

I was volunteering.

And somewhere in between, I ended up in a talk called:

"Ephemeral Environments: Easier Said Than Spun"

And I fell for it.

Not a person.

An idea that could kill the most annoying sentence in my job.

To explain why it hit so hard, I need to go back a bit.

Before we met

For a long time, my favourite word at work was:

"Not now."

Not because I didn't want to help.

Because I was the only one who could do it.

Our pipeline was simple.

Build.

Deploy.

That's it.

Two stages, and only for the branches that mattered. develop went to dev. main went to prod.

It worked.

Until a developer walked up to me with this:

"Can we deploy my branch?"

And I said it.

"Not now."

But I'm not completely heartless.

So, a bit later, I added a rule.

If the branch is this one, deploy it.

Something like:

deploy:
  rules:
    - if: $CI_COMMIT_BRANCH == "feature/the-one-you-asked-for"

A branch name, hardcoded in the pipeline.

Done.

Next.

Then came the variables

Hardcoding one branch was fine.

Hardcoding the next one was annoying.

So I got a little smarter.

I swapped the branch name for a variable.

If DEFAULT_BRANCH equals this branch, deploy.

deploy:
  rules:
    - if: $CI_COMMIT_BRANCH == $DEFAULT_BRANCH

Now anyone could trigger the pipeline by hand, pass in the branch, and deploy it.

No new commit, no new rule.

And this time I was smart enough not to do it myself.

"Go to Run pipeline, set the variable to your branch, and do it yourself."

At the time, it felt like engineering.

It was also about when I started hearing the word DevEx.

And I really believed I was doing it.

Halfway, maybe.

Then it spread

It wasn't one developer, or one project.

The messages kept coming in.

But the question had changed.

It wasn't "Can you deploy my branch?" anymore.

It was:

"How can I deploy my branch?"

Progress.

Kind of.

So every time, the same little routine.

Which project?

Does that one have the variable yet?

Go to CI/CD, Run pipeline, set DEFAULT_BRANCH, wait for green.

Then the next one.

I'd stopped being the deploy button.

Now I was the manual for the deploy button.

Back to that talk

So there I was at NDC, listening to a talk about ephemeral environments.

The gist: everyone loves the idea, few teams make it stick, and it's a trade between the coordination overhead of static environments and the cost of ephemeral ones.

Coordination overhead.

That one landed.

That was me, explaining Run pipeline to one developer after another.

And one more line I'd remember later: you'll need parameters and variables everywhere, and they'll frustrate you.

I walked out thinking about my pipeline.

None of the idea was new to the world.

It was new to me.

Every merge request gets its own environment.

It comes up when the MR opens.

It goes away when the MR closes.

Nobody asks me.

Nobody waits for me.

Nobody types a branch name into a variable.

And that's when it clicked.

I'd been solving the right problem the wrong way.

I'd stopped deploying for them.

But they still needed me to tell them how.

Remember when I fell in love?

That was it.

Easier said than spun

So I tried it.

One project.

And I got lucky with the first one.

It's a monorepo. Backend and frontend, in the same place.

Nothing else to drag along.

Before I wrote a single line of YAML, I had to check something with the developer.

"Is the API URL hardcoded in a NEXT_ variable anywhere?"

Because if the frontend only knows how to talk to one specific backend, every preview frontend ends up calling the same API.

That's not a preview.

That's a costume.

The talk did warn me about variables.

Once that was sorted, the setup was pretty simple.

GitLab builds the images for the MR.

Argo CD notices the MR and spins up a preview for it.

Its own namespace, its own URL with the MR number in it, and its own fresh database on the shared dev Postgres server, with some demo data so it isn't empty.

When the MR is merged or closed, Argo deletes the preview, and a cleanup step drops its database. The empty namespace Argo leaves behind gets swept up by a small hourly job.

Open the MR. Test it. Close it. Gone.

Nobody asks me.

Honestly, I haven't fully tested it yet.

But it exists, and it's already more honest than the variable trick.

And yes, I know. I can already hear it.

"What if you have 10 MRs a day?"

"What if one MR stays open for a week? A month?"

Fair.

I don't have the answer yet.

I know there are levers.

I can cap how many MR environments run at the same time.

I can look for stale branches and clean them up.

Whether that's enough for my setup?

Still finding out.

The bit I'm actually confused about

The monorepo was the easy one.

And even that one gets its own database, not its own Postgres server.

Most of my apps aren't like that.

I haven't built this part yet, but here's the scenario I keep staring at.

An app that, to run at all, needs:

a backend, a frontend, MinIO, a database, and sometimes a CMS.

And the CMS usually brings its own backend and frontend, plus variables hardcoded for that specific app.

Now say I'm in the frontend repo, and I open an MR.

What does the preview spin up?

Just the frontend, pointed at the backend that's already running in dev?

That's fast.

But now my "isolated" preview shares a backend, a database and a bucket with everyone else. And whatever the CMS has hardcoded still points wherever it points.

Or all of it?

Backend, MinIO, a fresh database, the CMS and all its pieces, for one frontend change?

That's properly isolated.

It's also slow and expensive.

Exactly the kind of spin-up the talk warned about.

I don't know yet.

And I'd rather say that than pretend I've figured it out because a conference talk made it look clean.

What DevEx means to me

Somewhere along the way, I stopped thinking of DevEx as tooling.

For me, it's simple.

It's control.

Control given to the developer to deploy their own branch and test it before it ever reaches a customer.

A real environment where QA can poke at it.

Where a PM can click through it and say "yes, that's what I meant."

Without anyone having to ask me first.

And, honestly?

It's getting rid of the shitty daily jobs.

The ones that aren't hard, or interesting.

The ones that only exist because one person is standing between the developer and the deploy.

That person was me.

Where I am now

One project, not fully tested.

It's not done.

It's not even close.

But the direction finally feels right.

For a long time, the answer to "can you deploy my branch?" was:

"Not now."

Then it was a hardcoded rule.

Then it was a variable, and me explaining how to use it.

What I want it to be is no answer at all.

Because nobody had to ask.

Easier said than spun.

But I'm spinning it anyway.

🎧 Fell in Love — Blink-182

Originally published at najeennp.com.np/devex.

Playing By Ear

Najeen Nepali — Sat, 03 Oct 2026 16:44:55 +0000

FEELING THIS

Why moving to GitOps was the best infrastructure decision I've made: scattered Docker hosts, on-prem Kubernetes, a bash script talking to Portainer, and the day I stopped being the source of truth.

Before GitOps, our deployments were a bit like a band playing by ear.

Everyone kind of knew the song.

Nobody had the sheet music.

And if you wanted to know how it actually went, you had to ask me.

How it used to work

At first, our apps lived on Docker.

The docker-compose.yml sat in the same repo as the application code.

The pipeline built the image.

We usually had two of them: a develop image and a main image.

Then a script, triggered from CI/CD, told Portainer to redeploy.

Green pipeline.

Done.

Next.

It worked.

Until I needed to change one environment variable.

One.

And to do that, I had to run the whole pipeline again.

Build the image.

Push the image.

Trigger the script.

Redeploy.

All of that for a single line of config that had nothing to do with the code.

And that's if everything went right.

Now imagine a developer messages me:

"Hey, can you add these two environment variables?"

I add one.

Run the pipeline.

Build. Push. Trigger. Redeploy.

Wait.

Green.

Done.

A little later:

"Hey, the second one is still missing."

Right.

Back to the pipeline.

Build the same image again.

Push it again.

Trigger the script again.

Redeploy again.

And if I made a typo in the value?

Same thing.

Again.

The code hadn't changed once.

The image was identical every single time.

Then came Kubernetes

We had two on-prem clusters, dev and prod.

And somewhere around them, 10 to 20 Docker hosts scattered across the infrastructure, each doing its own thing.

On the Kubernetes side, the pipeline ran helm directly.

Which meant the pipeline needed cluster credentials.

Which meant the kubeconfig lived in GitLab CI/CD variables.

So our CI system could reach straight into the cluster and do whatever it liked.

Where was the truth?

This was the real problem.

Not the scripts.

Not Portainer.

Not Helm.

The question was simple:

"What is actually supposed to be running right now?"

And the honest answer was:

It depends who you ask.

The cluster had one version.

Some wiki page had another.

The compose file in the repo had a third.

And the rest of it lived in my head.

Anyone with access could change things.

Sometimes someone hotfixed something directly.

To be fair, drift wasn't a huge problem yet, because we were still early with Kubernetes. There wasn't much to drift.

But the configuration was a mess.

And onboarding a new service took a lot of work: copying files, writing scripts, wiring up credentials, remembering which wiki page was the correct one.

We weren't reading from the sheet music.

We were all playing by ear.

Fleet first

Funny thing is, GitOps didn't start with my own projects.

It started with a message from another team, on January 29, 2024.

I'd already used Kubernetes the way I described above: kubectl, helm upgrade straight from the pipeline, credentials in GitLab.

This team was about to move a customer from Docker Swarm to Kubernetes, on a fresh on-prem setup with Rancher, RKE2 and Ceph.

And then this line:

"What's important to him is that all deployment processes follow gitops principles and are well documented to make debugging easier."

They also wanted a separate infra repo for all the configs not directly related to the application itself: reverse proxy, pipelines, the lot.

A separate infra repo.

I didn't even have that concept yet.

Honestly, they wanted more than what I gave them.

I was lazy.

My knowledge was limited.

And AI wasn't really a thing yet, so I couldn't just ask it to explain the hard parts to me.

So I didn't go looking for the best tool.

I went looking for the simplest one.

That was Rancher Fleet.

It came built into Rancher, and it was easy to set up without a lot of work.

At least, I thought so.

Because it didn't work.

The repo was there.

Fleet was there.

Nothing deployed.

Meanwhile, the developer on that team just wanted to ship to dev:

"If you give me a quick introduction on how to update the deployment on dev, I can also do it."

"Or do you think the pipelines will be ready by then?"

I didn't think so.

So I offered the fallback.

Connect Azure DevOps to the cluster.

Run helm upgrade from the pipeline.

Yes.

The exact thing the customer had asked us not to do.

Then came the fair question:

"But what exactly is your issue with the GitOps approach again? Is it that the Azure repos don't work but the GitLab ones do?"

Honestly?

I didn't fully know.

So I took the same repo, the same process, and pointed another cluster at it.

It worked.

So the problem wasn't my repo.

It wasn't Azure.

It wasn't GitOps.

It was the Fleet version in Rancher.

All that time, and the fix was an upgrade.

That's where it all started.

After that, I brought it to the rest of our projects.

No big-bang migration.

One project at a time, based on priority.

Then Argo CD

I stayed on Fleet for a little over two years.

What finally moved me wasn't a blog post or a comparison table.

It was NDC Sydney, in April 2026.

I met the Octopus Deploy team there, and they were all in on Argo.

So I tried it.

And I'll be honest about what hooked me.

It wasn't some deep architectural reason.

It was the deployment notifications.

That's it. That was the hook.

But once I was in, I started to understand what I'd actually been missing.

Configuration drift. Argo shows you when the cluster doesn't match Git, which Fleet, at least the way I was using it, never really made obvious.

Templating. The way Argo templates applications is one of the best things about it.

And then there was ApplicationSet.

What it looks like now

The manifests live in a separate repo.

One repo per project, a small monorepo holding the manifests and infrastructure definitions that project needs to run.

It evolved over time:

compose → Helm → infrastructure → and gradually, catalogs.

Dev and prod are just folders.

Mostly Helm. Some raw YAML. Kustomize has started showing up recently.

The flow is simple:

CI builds the image and runs the checks.

A deployment stage commits the new image tag to the manifest repo.

Argo sees the change and syncs it.

No kubeconfig in GitLab.

No script talking to Portainer.

No pipeline reaching into the cluster.

The pipeline writes to Git.

Argo reads from Git.

The cluster follows.

And no, we don't "promote" images from dev to prod.

Dev and prod can run genuinely different code, and some frontends bake environment variables in at build time, so the same image can't simply move forward.

I'm still looking at whether we can make that work.

But I'm not going to pretend we're doing it just because a conference talk says we should.

Things I had to unlearn

Moving to GitOps wasn't just new tools.

It was breaking a lot of old habits.

kubectl edit doesn't work anymore.

One day I wanted to quickly test something.

Temporarily change an image tag.

So I did what I'd always done:

kubectl edit deployment <app>

Changed the tag.

Saved.

Felt productive.

A minute later, Argo reverted it.

WTF.

Then it clicked.

Argo wasn't broken.

It was doing exactly what I'd asked it to do: make the cluster look like Git.

I'd just never had anything that actually held me to that before.

The code repo isn't the only repo.

Before, all I knew was that whatever an app needed should live in the app's repo.

Code, compose file, scripts, config. All of it.

But the code and the configuration that runs the code are not the same thing.

They change at different speeds.

They're changed by different people.

They break for different reasons.

A separate repo for manifests sounded like extra work.

It turned out to be the whole point.

Stop answering from memory.

Before, I knew where things were.

That felt useful.

It was actually the problem.

And it's a hard habit to break, because answering from memory is faster.

Every single time.

So now when someone asks, I have to stop myself.

Open the repo.

Find the line.

Send the link:

"The answer you want is on this line."

Slower for me today.

Faster for everyone after.

Stop building around Portainer.

I was so obsessed with Portainer that I ended up writing Ansible playbooks for Portainer.

They created stacks.

They redeployed the stacks.

And all the credentials were in the repo itself, encrypted with Ansible Vault.

At the time, it felt like engineering.

Looking back, I had built a GitOps tool by hand.

A worse one.

The gains

I can see everything.

One glance shows the whole project.

What's deployed.

Which version.

Whether it matches Git.

Remember that one environment variable that needed a full rebuild?

Now it's a one-line commit.

ApplicationSet.

We run a multi-tenant setup.

Before, if something changed, I had to run the same helm command once for every tenant.

And keep the tenant list somewhere.

Usually a mix of the pipeline, my notes and my memory.

Now the tenant list is in Git.

One trigger, and every tenant gets deployed.

Want to bump the chart version?

I don't open each tenant's Argo file anymore.

I change one line.

Every tenant gets it.

Image Updater.

This one I never found in Fleet.

For one add-on repo, CI doesn't commit the image tag at all.

It just builds the image and pushes it.

Argo CD Image Updater watches the registry, sees the new image and updates the deployment itself.

No tag commit.

No extra pipeline stage.

Push the image, and it shows up.

Anyone can deploy.

Developers can update ConfigMaps and check logs themselves.

They don't have to wait for me.

The tradeoffs

It's not free.

It's slower.

Before, a green pipeline was the deployment. No waiting.

Now there's roughly a 4–5 minute gap while Argo picks up the change and syncs it.

When you want something changed right now, those minutes feel long.

It hides the deployment.

This is the one I didn't see coming.

Developers see the deployment stage go green and assume it's deployed.

But the green tick only means the tag was committed to Git.

The sheet music changed.

The band hasn't played it yet.

It's not deployed until Argo says the app is healthy and the pods are actually running.

To be fair, I had to learn that one too. On Docker, if the container was running, I called it healthy.

GitOps made deploying so easy that a lot of people stopped wondering how it actually happens.

And honestly, most people want to know how it works.

Most of them just have no idea yet.

That one's on me to fix.

Secrets: the bit I'm still fixing

Secrets didn't move into Git.

For a long time, they were applied by hand, with kubectl apply.

And I paid for that.

When I rebuilt the dev cluster, we were still on Fleet. Re-adding the projects to Fleet was easy, and the workloads came back.

The secrets didn't.

Finding them, rebuilding them, working out which values were current: that was the painful part.

What was in Git came back easily.

What wasn't was a hunt.

That says more about GitOps than anything else in this post.

So now we're slowly moving to Infisical.

Partly because developers want to see the current value without asking me.

Partly because I'm too lazy to log in and check for them.

And mostly because I don't want to be the one person who has to be around every time a credential changes.

What's next

Once the configuration lives in Git, the next step gets much easier.

Backstage templates.

An internal developer platform.

And, honestly, the fun part: being able to ask an AI to build a manifest based on our company standard, and getting back something that actually fits.

That only works because the standard exists.

And the standard only exists because it's written down.

Was it worth it?

So far, it's the best infrastructure decision I've made.

Not because Argo CD is magic.

Not because GitOps is fashionable.

Because for the first time, there's one answer to:

"What is supposed to be running?"

And the answer isn't me.

It's in Git.

We used to be a band playing by ear.

Everyone knew roughly how the song went.

Nobody could tell you exactly.

Now there's sheet music.

Git is the sheet music. Argo is the one who notices when someone plays the wrong note.

And I finally get to stop humming the tune for everyone.

I got no regret right now.

I'm feeling this.

Originally published on najeennp.com.np.