DEV Community: Yu Ting Chen

A Team Asked for One More Service, and My Kubernetes Platform Quietly Refused

Yu Ting Chen — Thu, 01 Oct 2026 04:03:32 +0000

I'm building a small internal developer platform on Kubernetes. The idea is simple. A team writes a short request that says "this is my team, this is my service, run it". The platform does the rest: it gives the team its own space in the cluster, the right access, a resource budget, and it deploys the service.

It worked for the first service. Then I added a second service for the same team. Kubernetes accepted the request. No error came back. The new service just never became ready.

This post is about what was going on, how I thought it through, and what I changed. If you build things on Kubernetes that create other things, you may run into the same wall.

What the platform does

Kubernetes lets you add your own kinds of objects, called custom resources. You then write a controller: a program that watches those objects and makes the cluster match what they describe.

My platform had one custom resource, a ServiceClaim. A team's request looked like this:

kind: ServiceClaim
metadata:
  name: payments          # the service name
spec:
  team: payments
  image: nginx:1.27-alpine   # a stand-in for the real service; any image works
  replicas: 2
  resources:              # the team's resource budget
    cpu: "2"
    memory: 4Gi
    pods: 10

For each claim, the controller created four things:

a namespace for the team, team-payments (a namespace is a separate space inside the cluster)
a RoleBinding that gives the team edit access in that namespace
a ResourceQuota that caps what the team can use
an ArgoCD Application that deploys the service (ArgoCD is a tool that keeps the cluster in sync with manifests in Git)

Why the second service got stuck

In Kubernetes, an object can record who owns it. One of those owners can be marked as the controller: the one object responsible for it. Kubernetes allows only one controller owner per object. The library my controller uses, controller-runtime, checks this rule before it even sends an update, and returns an AlreadyOwnedError if the object already has one.

My controller made the ServiceClaim the controller owner of all four objects. That's fine for one service. But look at the list again. Only the last one belongs to a service. The namespace, the access and the budget belong to the team.

So the second claim for the same team asked to own a namespace that the first claim already owned. Kubernetes' rule said no. The controller stopped at step one and retried, over and over. The claim's status showed it:

NamespaceReady=False (NamespaceError) Object /team-payments is already owned by another ServiceClaim controller payments
Ready=False (ResourcesNotReady) one of the claim's resources is not ready

Nothing crashed. The request was accepted, the first service kept running, and the second one waited forever. That kind of failure is easy to miss, because nothing looks broken until someone asks where their service is.

How I thought about it

The error message was clear. The harder question was why my design asked for this at all. So before fixing anything, I listed the obvious fixes and asked what each one would lock in.

Block the second claim. A validation check could reject any second claim for the same team. It's the simplest fix, and it's the wrong one. It turns a bug into a rule. Teams still can't run two services, they just get told so earlier.

Let claims share the team's objects. Each claim could add itself to the team's objects and keep a count. The namespace would go away only when the last claim did. But the controller would then have to add up every claim's budget into one quota, and track who is still around. That's a lot of code to work around the shape of one object.

Make one object per team, with a list of services inside. No conflict, since there is only one owner. But changing one service would mean editing the whole team's list. That's not how teams think. They want to "deploy a service", not "re-declare my team".

There was also a second problem hiding behind the first. Even if the ownership conflict were solved another way, each claim carried its own resources budget. Two claims would keep writing two different budgets into the same quota. Nobody had hit that yet, because the second claim never got past the namespace. But on paper, the design was already wrong.

Looking at the three options side by side made the real issue easier to see. One object was carrying two different sizes of thing: team things and service things. The fix was to split it along that line.

The fix: one object for the team, one for each service

I added a second custom resource, the Tenant, for the team:

kind: Tenant
metadata:
  name: payments          # the team name
spec:
  resources:
    cpu: "2"
    memory: 4Gi
    pods: 10

The Tenant owns the namespace, the access and the budget. Its name is the team name, so "one Tenant per team" is guaranteed by Kubernetes itself. A Tenant is cluster-wide, and two cluster-wide objects of the same kind can't share a name. I didn't have to write any code for it.

The ServiceClaim became smaller. It keeps only what a service needs, team, image and replicas, and it owns only its own ArgoCD Application. A team can now have as many claims as it wants, all pointing at the same Tenant.

The split raised new questions

Two objects that depend on each other bring questions that one object never had. I answered two of them before calling the fix done.

What if the service arrives before the team? Teams often apply a folder with a Tenant and several ServiceClaims in one go, and Kubernetes doesn't promise any order. If a claim were rejected because its Tenant wasn't there yet, a valid folder would fail or succeed by luck. So a claim without a ready Tenant doesn't fail. It reports TenantReady=False and waits, and it continues on its own when the Tenant is ready. "Not yet" and "wrong" are different states, and only "wrong" deserves an error.

What happens on delete? Kubernetes cleans up owned objects automatically when their owner is deleted, but it doesn't do it in any order. Deleting a Tenant could remove the namespace while services were still running in it. To control the order, I used finalizers. A finalizer is a marker on an object that makes Kubernetes wait, before deleting it, until a controller has done its cleanup. Three of them work together:

The ServiceClaim finalizer deletes the claim's ArgoCD Application, and waits until it's really gone.
The Application carries ArgoCD's own finalizer. That's what makes ArgoCD remove the running service before the Application itself disappears. I found this out while building the delete path: without it, deleting the Application only makes ArgoCD forget the service, and the service keeps running with nothing managing it.
The Tenant finalizer refuses to delete the team while any claim still points at it. So the namespace stays alive long enough for every service to be removed first.

One gap is known and written down rather than fixed: deleting a Tenant with kubectl delete --cascade=foreground removes the namespace before the finalizer runs. The default delete is the supported path.

What it cost

The split isn't free. One simple object became two objects with a relationship: a reference, a waiting state, and finalizers that have to run in the right order. That's more to reason about and more to test. I think it's the right trade here, but it's a trade, not a pure win.

If you run into something similar

This is the approach I'd take again:

Treat the error as a question about the design. AlreadyOwnedError wasn't the bug. It was Kubernetes pointing at an object that owned things at two different levels.
For each quick fix, ask what it would lock in. The simplest fix here would have made the limitation permanent.
Split objects by what they really belong to. Team things and service things have different lifetimes. Putting them in one object is where the conflict came from.
Answer the questions the fix creates. Order of arrival and order of deletion were new problems that only existed after the split.
Write down the questions you're not solving yet. A month before this bug, I had noted that "who owns what" between my custom resources would need real design once there was more than one. The note didn't predict this bug, but when it came, the question was already on paper.

Try it yourself

The original code that failed isn't in the repository's history, because the history was squashed. So I rebuilt that earlier version to show the failure. It runs in its own small local cluster, and it prints the same status you saw above.

The failing version: scenarios/ownership-conflict. The README there has the steps. You need Docker, k3d, kubectl, the task runner and Go.
The current design: idp-platform-lab. The README's quick start brings up the whole platform, and docs/verification.md walks from a request to a running service.

I used AI to help draft and edit the English. I checked the technical claims against the code and a live reproduction.

When Celery Chains Get Hard to Maintain: Introducing CeleryFlow

Yu Ting Chen — Thu, 06 Aug 2026 07:48:35 +0000

If you run Celery in production and your multi-step jobs have turned
into a pile of chain() calls nobody wants to touch, this post is
about that specific problem.

I ran into this problem at a previous job and built a small internal library to solve it. Earlier this year, I released an open-source version as CeleryFlow. It lets you describe Celery workflows in YAML instead of assembling them in Python.

The honest disclaimer first: it's not the right tool for most new projects, and it is not a durable execution engine. If you're starting fresh in 2026, there are better places to look, and I'll point at them near the end. What follows is written for people already running
Celery.

When you outgrow `chain()`

At that job, we had dozens of multi-step workflows in an e-commerce platform: order placement, refund, plan upgrade, subscription renewal, etc. Each was 5-10 steps. They shared common sub-flows ("validate order", "charge card", "send receipt"). Conditions like "only run the loyalty-points step if buyer is a returning customer" appeared all over.

Celery's chain(), group(), and chord() primitives are powerful,
and none of this was impossible. But this approach had two specific pain points:

Composing chains was painful. When place_order_flow and refund_flow needed the same opening sequence, you'd either duplicate the chain construction or wrap it in helper functions that ballooned over time.
Conditionals were inline. "Only run step X when payload field Y looks like Z" became if statements inside the task body, mixing business logic with flow control.

Same two flows, two approaches. On the left, the steps are duplicated across flows, and the condition is implemented as an inline if statement. On the right, the sub-flow is defined once, and the condition is defined in config.

The natural response was to pull the workflow structure out of the code and into config files. We did, and it stuck. CeleryFlow brings that design to open source.

What it looks like

work-flows:
  - name: checkout
    tasks:
      - order.validate
      - order.charge

main-flows:
  - name: PlaceOrder
    flows:
      - flow: checkout
      - task: order.send_receipt
      - task: order.award_loyalty_points
        condition:
          customer_type: { $eq: "returning" }

That's the whole workflow definition, but not the whole setup. To wire it in, use the CeleryFlow app class instead of Celery, give participating tasks base=EventTask, and register the file with FlowBuilder.from_yaml(). Once registered, the flow runs on your existing Celery workers and broker. There is no new worker model or orchestration server, and your existing retry behavior, routing, Flower setup, and monitoring continue to work.

One important caveat: in v0.2.0, a condition that doesn't match raises ConditionFailed before the task is queued. There is no built-in quiet skip; unless the exception is handled explicitly, the chain fails. A missing field counts as passing, so conditions only gate fields that are present in the payload.

pip install "celeryflow[yaml]"

The yaml extra pulls in PyYAML, which FlowBuilder.from_yaml()
needs. Plain pip install celeryflow works too if you'd rather keep
your flow definitions as Python dicts or JSON.

Quickstart and full docs: https://github.com/ChenYuTingJerry/CeleryFlow

When you don't need it

You don't need any workflow framework if:

You only have single tasks or short chains with one or two follow-ups, and you're happy using Task.s() | Task.s() for the chains.
You don't have many of them.
The chains rarely change shape.

For a handful of simple workflows, just use the Celery primitives. Adding a layer on top is overhead you don't need.

What it isn't

The most important question before choosing an approach is:

If a worker process dies in the middle of a multi-step workflow,
what happens?

With Celery, and therefore with CeleryFlow, individual tasks may be redelivered after certain failures, depending on acks_late, the broker, and how the worker failed. A task may also run more than once. What Celery doesn't provide is durable workflow state from which the whole flow can resume.

If that's not acceptable, look at durable execution platforms:
Temporal, Hatchet, Restate, AWS Step Functions. They persist workflow
progress, so execution can recover after a failure and wait between
steps. Adopting one means moving to a different execution and operational model, whether self-hosted or managed. CeleryFlow has no equivalent persistence or durable timers, so it is a poor fit for flows that wait for hours or days between steps. It is also Python-only, while most of those platforms support several languages.

Durable workflow recovery does not mean that every side effect happens exactly once. Individual operations may still be retried, so anything involving money or an external API needs to be idempotent. The exact guarantees vary by platform.

And if your "workflow" is really a data pipeline (ETL, batch ML
training, scheduled report generation), Prefect and Dagster are built
for that, with first-class scheduling, observability UIs, and
integrations with warehouses and dbt. CeleryFlow can run scheduled
tasks via Celery Beat, but that's not what it's for.

Where it sits

My first-hand experience here is with Celery and CeleryFlow. I've also operated Airflow at the infrastructure level. For the other tools, I'm relying on their intended design rather than direct use.

Tool	Type	When to use it
Celery	Task queue	You need workers that run individual tasks. Mature, battle-tested.
Celery + `chain()`/`group()`	Manual workflow	You have Celery and a handful of multi-step jobs. Works, gets messy at scale.
CeleryFlow	Config-driven workflow on Celery	You already use Celery, want declarative workflows, don't want a separate service.
Temporal	Durable execution platform	You need workflows that survive worker crashes, run for days, and resume from where they stopped.
Prefect	Modern data orchestration	Data pipelines, ML workflows, want a UI, want hybrid cloud / on-prem.
Dagster	Asset-oriented orchestration	Data engineering teams who think in terms of "data assets" not "tasks."
Airflow	Classic DAG scheduler	You're at a company that already runs Airflow, or your team is comfortable with it.
arq / TaskIQ / Procrastinate	Async-native task queues	Brand-new project, async-first, don't need Celery's surface area.

Two questions capture most of the trade-off: do you need durable execution, and how much additional infrastructure are you willing to run?

So is it for you?

I'll be specific. CeleryFlow is a good fit if:

You have a Python backend that already runs Celery in production.
You have between 5 and 50 multi-step workflows.
Each workflow is seconds to minutes long, not days.
Workflows have conditions that gate a step on the payload, or share common sub-flows.
You don't want a separate service for workflow orchestration.
Re-triggering the whole flow from the upstream event is an acceptable recovery strategy.

The whole article as one picture. CeleryFlow is the leaf at the end of a fairly narrow path, and that's on purpose.

What I'd like feedback on

I'm trying to understand whether this design still fits real-world needs in 2026 or whether teams have moved on. If you've used Celery at any scale, I'd love to hear your perspective:

Have you ever written something like CeleryFlow? Why or why not?
Did you switch from Celery to Temporal / Prefect / something else? What pushed you?
If you tried CeleryFlow and didn't keep it, what made you bounce?

GitHub issues and comments here on DEV are both welcome.

I used AI to help draft and edit the English. I checked the technical claims against the CeleryFlow code.