DEV Community: drtobbyas

EP3: Native Kubernetes deployment is officially working in Coolify. But getting there meant wrestling with vicious race conditions.

drtobbyas — Mon, 11 May 2026 07:19:14 +0000

If you missed Episode 2, we realized that Coolify's SSH-native engine is surprisingly cluster-friendly. The architecture wasn't locked to Docker; it simply lacked a translation layer.

In Episode 3, it was time to build that translation layer and prove the concept. But turning theory into reality required two massive, distinct phases of implementation and this led to one of the most stressful race conditions I've ever debugged, all while I was backpacking across 4 countries.

Here is the story of how the Kubernetes Proof of Concept (POC) came to life.

🏗️ Phase 1: The Struggle for a Cluster

Before you can deploy an application to a Kubernetes cluster, you must actually have a cluster to deploy to.

This was the first major hurdle. The official Coolify deployment currently has zero built-in Kubernetes infrastructure. I knew that I couldn't just build a deployment script without giving users a way to actually spin up an environment to test it on without leaving the UI.

I spent days searching around, reviewing so many different options for how to bootstrap a cluster natively. Ultimately, I settled on K3s. It is an incredibly lightweight, production-ready Kubernetes distribution that is perfectly suited for the types of servers Coolify normally runs on.

I integrated it directly into the frontend. I built out the UI and the underlying backend logic so that users can now do two things straight from the dashboard:

Spin up a brand new K3s cluster from scratch on a server.
Link securely to an existing Kubernetes cluster.

Phase 1 was a resounding success. I had the foundation.

🌍 Phase 2: Racing Across Borders

As I moved into Phase 2, life happened. I had to pause active work while I took a cross-country tour across 4 different countries.

But while I was navigating borders, the codebase I had just written to handle my deployments was locked in an intense race condition of its own.

I was literally racing across borders while trying to stop my codebase from "racing" and locking up the UI.

Here is what went wrong.

🐉 Deploying the Docker Image & Fighting the Clock

The second phase of the POC was the grand finale: taking a standard Docker image (I used Nginx), deploying it as a service directly to the K3s cluster I had just created, and ensuring it was accessible via an Ingress route.

The translation script worked flawlessly. My Docker manifests were effortlessly converted into K8s Deployments, Services, and Traefik Ingress rules. But making the status UI sync was a nightmare.

To ensure the deployment succeeded, I initially set the core action to run synchronously. The deployment worked! But because it waited for the cluster to finish, it held the PHP process hostage and completely locked up the Coolify frontend.

"Simple," I thought. "Just revert the deployment to an asynchronous background job."

The UI immediately became snappy again. But this introduced the ultimate syncing problem. The moment the async deployment fired, Coolify's Application Status Checker instantly polled the K8s API.

Because Kubernetes is eventually-consistent, it takes a few seconds to pull the image and schedule the pods. The API responded, accurately, that there were zero pods running. Instead of understanding that the app was just booting up, the Coolify orchestrator aggressively flagged the perfectly healthy deployment as "Exited" or "Failed."

A fully functional deployment was showing a glaring red error state.

🚀 The Resolution: The POC is Alive

You cannot force an intrinsically asynchronous system (Kubernetes scheduling) to behave linearly against a strict synchronous status check. The absence of a resource immediately after creation is an expected state, not a failure.

I solved the "racing codebase" by injecting an intelligent, two-minute graceful memory window into the status pipeline. If the checker polls an application within two minutes of an update and finds zero pods, it simply holds the UI status at "Starting" until the pods are scheduled. The moment the K8s API confirms the pods are healthy, it seamlessly flips the interface to "Running."

The end-to-end "Docker Image -> K8s" flow is now incredibly fast, fully observable, and completely robust. I conquered the K3s installation, I defeated the deployment race conditions, and I proved that Native Kubernetes fits perfectly inside Coolify.

⏭️ Next in the Investigation

Letting the automated systems handle the eventual consistency of Kubernetes meant I could actually close my laptop and enjoy the rest of my tour across countries. But the work is far from over.

Next up, I dive deeper into linking external production clusters and polishing the features for robust availability. You will not want to miss what's coming next!

GitHub Issue: https://github.com/coollabsio/coolify/issues/2390

Connect with me: Twitter/X, Linkedin, Telegram

This is the third post in a series documenting my investigation into building native Kubernetes support for Coolify. Next up: Connecting robust external clusters.

EP2: Mapping the Labyrinth: How Coolify Deploys Your Apps (and Why K8s Fits)

drtobbyas — Wed, 22 Apr 2026 13:27:36 +0000

If you missed Episode 1, we established the goal: Investigate whether native Kubernetes support in Coolify is actually impossible.

Now, the investigation moves from the "Why" to the "How." I spent the last few days inside the Coolify source code, trying to map exactly how it moves code from a repository into a running container.

Here is the technical reality of the engine.

🏗️ Part 1: Finding the Heartbeat

To understand how Coolify works, you have to find its "Engine Room." In this codebase, that room is located at app/Jobs/ApplicationDeploymentJob.php.

It is a massive, 4,000-line procedural job.

In some circles, a 4k-line file is a "code smell." But in an orchestrator, it’s actually a map. Because it's written procedurally, you can read it like a script. I spent hours tracing the flow:

The Setup: Cloning the repo and establishing the build environment.
The Network: Creating the Docker bridge networks.
The Deployment: Building the images and running docker compose up.

The audit confirmed my first hunch: The logic isn’t hardcoded to Docker. It’s a sequence of commands. If we can swap those commands, we can change the engine.

🗺️ The Map of the Territory

To find the path to Kubernetes, I first had to map the Labyrinth. Here is the simplified structure of the Coolify engine:

coolify/
├── app/
│   ├── Actions/        # Reusable deployment logic
│   ├── Jobs/           # The heart: ApplicationDeploymentJob.php (4k lines)
│   └── Models/         # Data structures (Server, Destination, Service)
├── bootstrap/
│   └── helpers/        # The heavy lifters: remoteProcess.php & proxy.php
├── config/             # Global platform settings
└── docker-compose.dev.yml

Key Discovery Points:

app/Jobs: This is where the linear deployment sequence lives.
bootstrap/helpers/remoteProcess.php: This is the "SSH Tunnel" that makes everything possible.
app/Models: This is where we’ll define the new KubernetesDestination.

🐉 Part 2: The Fedora Sidequest

Before I could dive deeper, I had to fix my own "Engine Room."

I develop on Fedora, which means I’m running a security-hardened stack with SELinux. As soon as I tried to spin up a basic service like Dashy or Homepage in my local Coolify dev environment, I hit a stone wall.

Permission Denied.

The proxy container (Traefik/Caddy) couldn’t talk to the Docker socket. Everything was 404ing.

I spent a few hours patching bootstrap/helpers/proxy.php to handle this "hardened" reality. The fix required two key adjustments:

Adding the :z flag for volume relabeling (/var/run/docker.sock:/var/run/docker.sock:z).
Setting privileged: true for the local proxy.

The Lesson: Local dev is never as simple as docker compose up. But solving these "gatekeeper" bugs gave me a deeper understanding of how Coolify handles its proxy logic. Such knowledge I'll need when we move to K8s Ingress.

🚪 Part 3: The SSH Backdoor

While auditing the engine, I found the most important piece of the puzzle: remote_process.

Coolify doesn't rely on complex, vendor-locked SDKs to manage your servers. It does something much simpler and more powerful: it uses SSH to run shell commands.

This is the "Kubernetes Backdoor."

Right now, the ApplicationDeploymentJob sends strings like:
docker compose up -d

But because it’s just a CLI pipeline over SSH, there is no architectural reason it can't send:
kubectl apply -f manifest.yaml

The engine treats servers as SSH-ready shell endpoints. If your server has kubectl installed, Coolify can already talk to it. The "impossible" barrier isn't the architecture,it's just a translation problem.

🚀 The Phase 2 Conclusion: It’s a Translation Problem

They said Kubernetes isn't coming. I've found that the door is already wide open.

The challenge ahead isn't rewriting the core engine. It's building the Translator. We need to take the configuration you provide in the Coolify UI and turn it into Kubernetes YAML instead of Docker Compose labels.

Next in the Investigation:
I’m moving on to building the KubernetesDestination model, the foundation for a cluster-native Coolify experience.

Follow along as we start building the bridge.

GitHub Issue: https://github.com/coollabsio/coolify/issues/2390

Connect with me: Twitter/X, Linkedin, Telegram

This is the second post in a series documenting my investigation into Kubernetes support for Coolify. Next up: Building the first Kubernetes Destination model.

EP1: They Said Kubernetes Isn't Coming to Coolify. I'm Going to Find Out If That's True.

drtobbyas — Tue, 14 Apr 2026 13:08:46 +0000

22 months open. $250 bounty. 46 reactions. One unexpected answer.

The Reach-Out

On March 14, 2026, I did something that felt important.

I commented on Coolify issue #2390 — the Kubernetes support request that has been sitting open since June 2024. I offered to help. I shared my background (8 years DevOps, deep K8s specialization). I asked if they were open to collaboration.

Then I took it to Discord and asked the team directly: "What do you think about this?"

The response came from Peak, a core maintainer:

"We will use our own custom solution (for v5) that integrates directly with Docker Compose and uses Docker in the background, as it is more flexible and tightly integrated into Coolify, like Swarm but better."

Kubernetes support, it seemed, was not coming to Coolify. Not in v5. Maybe not ever.

The Gap in the Answer

I read that response multiple times. Something struck me.

They never said Kubernetes was technically impossible.

They said their solution was:

"More flexible"
"Tightly integrated"
"Like Swarm but better"

These are comparative advantages, not absolute constraints.

The maintainers made a product decision: "We will build X instead of Y."

They did not make a technical claim: "Y cannot be built."

The Unanswered Question

This distinction matters because of what sits behind issue #2390:

46 community reactions (36 hearts, 10 eyes watching)
$250 in community bounties
22 months of sustained interest
Comments from users explaining why this is the blocker preventing them from adopting Coolify

The community clearly wants native Kubernetes support.

The maintainers have chosen a different path.

And no one has investigated whether the community's request is actually possible to fulfill.

The Investigation

So I'm asking a different question than "Will you accept my PR?"

Is native Kubernetes support in Coolify actually impossible?

With 8 years of Kubernetes experience — designing clusters, integrating K8s clients into applications, debugging production deployments — I'm in a position to find out.

Here's what I'm going to do:

Phase 1: Architecture Mapping

Clone Coolify and understand its current deployment model
Identify where K8s integration would logically live
Document the current "Docker-only" architecture

Phase 2: Integration Point Analysis

Map the API boundaries where K8s could interface
Identify potential blockers (technical, architectural, philosophical)
Research how other PaaS platforms handle dual Docker/K8s support

Phase 3: Proof-of-Concept

Build a minimal K8s integration for one deployment type
Document what works, what breaks, what's hard
Share working code or detailed failure analysis

Phase 4: Community Decision

Publish findings: "Here's what I found when I tried"
If it's possible: offer the solution to maintainers and community
If it's impossible: document exactly why (valuable for future reference)

Three Possible Outcomes

I'm not approaching this with a predetermined conclusion. There are three ways this investigation ends:

Outcome 1: It's Possible

I build working K8s support that proves the architecture can accommodate it. The maintainers can choose to merge it or not — but the community will have a working solution (either in mainline or as a maintained fork).

Outcome 2: It's Impossible

I discover genuine technical blockers that make K8s integration infeasible. I document these blockers in detail. The community finally has a definitive answer instead of an open question.

Outcome 3: It's Complicated

The integration is technically possible but comes with significant trade-offs. I map these trade-offs clearly so the community and maintainers can make an informed decision.

Any of these outcomes serves the community better than the current state: an open issue with demand but no investigation.

Why I'm Doing This

Let's be honest about the risks:

The maintainers might see this as undermining their vision
I could invest significant time and prove it's actually impossible
The community might not care about the findings
I could end up maintaining a fork that no one uses

But here's what I know:

The question deserves an answer. 46 reactions and $250 in bounties suggest real demand. Someone should investigate if that demand can be met.
The investigation has value regardless of outcome. Even a documented failure teaches us something about Coolify's architecture.
Building in public creates connections. The process of investigation — sharing blockers, breakthroughs, architecture insights — is valuable content for the DevOps community.
The maintainer decision isn't the end of the story. Open source means the community can explore paths the core team chooses not to. That's not hostile — it's how open source evolves.

The Bigger Picture

This isn't just about Coolify and Kubernetes.

It's about what happens when:

A community clearly wants a feature
The maintainers choose a different direction
No one investigates if the community's request is technically feasible

Someone has to ask: "Is this actually impossible? Or just not prioritized?"

I'm asking that question. And I'm documenting the answer.

Follow Along

I'll be sharing every step of this investigation:

Architecture deep-dives
Integration point analysis
Blockers and breakthroughs
Working code or documented failures
Honest assessments of what's possible

If you're interested in:

Kubernetes architecture
How PaaS platforms handle deployment abstraction
The gap between maintainer vision and community demand
What "impossible" actually looks like in practice

Connect with me: Follow me here on Dev.to for updates. You can also find me on Twitter/X, Linkedin and Telegram

GitHub Issue: https://github.com/coollabsio/coolify/issues/2390

Questions? Drop them in the comments below.