DEV Community: Aroua KABOUBI

Kubernetes Controllers: The Cache-Staleness Bug at Scale

Aroua KABOUBI — Wed, 14 Oct 2026 09:40:37 +0000

Key takeaways

Structure reconcilers as ordered, independently testable steps, each reporting its own status condition, over one monolithic Reconcile function.

Informer caches lag the API server by tens of milliseconds at scale; a stale read inside a MergeFrom patch can clobber a condition you set 49ms earlier.

Fix status patching with an uncached reader inside RetryOnConflict; Kubernetes v1.36 (April 2026) now lets controllers detect and skip behind-cache reconciles.

Even a logging change is a production change: a duplicate log key once dropped every log line from the busiest controller in an environment.

Everything below comes from operating a major European automotive manufacturer's internal Kubernetes platform, running a fleet of custom controllers on managed GKE. Names and identifiers are generic on purpose. The failure modes are exact.

What is the step-based reconciler pattern?

The step-based reconciler pattern splits a controller's reconcile loop into an ordered sequence of small, composable steps. Each step reports its own status condition, instead of one monolithic function doing everything. In practice, this keeps provisioning logic table-testable and makes partial progress visible. Across our controller suite, it turned a 400-line reconcile body into a readable chain. It is the same reconciliation discipline behind the Kubernetes as a Service control plane we operate, where dozens of resource types share one contract.

A monolithic Reconcile becomes unreadable fast. Picture one custom resource that has to provision networking, IAM, namespaces, quotas, and DNS. Have you ever tried to unit-test a function like that? So we organize each controller as controller.go plus steps/creation/ and steps/deletion/ packages. Every step embeds a shared BaseStep. It holds the controller-runtime client, an uncached reader, the scheme, a logger, and an event recorder. Each step also exposes explicit Ready and Reconciling conditions.

An Environment resource, for example, runs a chain like this: EnsureFinalizer, ResolveClusterConfigRef, ComputeTargetNamespaceName, EnsureNamespace, ApplyResourceQuotas, ApplyLimitRanges, ApplyServiceAccounts, ApplyRoleBindings, ApplyNetworkPolicies, DeleteFinalizer. Each step is small, ordered, and independently testable. As a result, partial-failure states show up as per-step conditions. You can read them straight from kubectl get print columns.

The single most valuable primitive we built was not the step interface itself. It was the shared status-patching helper underneath it. Every step calls one function to update conditions. So that function centralizes the conflict-safe read-then-patch dance in exactly one place. When we later found a cache bug, we fixed it once, not in fifteen call sites.

The shared conflict-safe status patch

A status helper re-fetches the latest object, mutates it, and patches it with client.MergeFrom inside a retry.RetryOnConflict loop. So it avoids two reconciling steps racing each other's condition writes. And the generation-aware comparison means unchanged state does not re-patch. That keeps the API server and your audit log quiet.

func (r *StepRuntime) UpdateStatusFields(ctx context.Context, obj client.Object,
    mutate func(client.Object) error) error {
    return retry.RetryOnConflict(retry.DefaultRetry, func() error {
        // Read through the UNCACHED reader, not the informer cache.
        fresh := obj.DeepCopyObject().(client.Object)
        if err := r.UncachedClient.Get(ctx, client.ObjectKeyFromObject(obj), fresh); err != nil {
            return err
        }
        base := fresh.DeepCopyObject().(client.Object)
        if err := mutate(fresh); err != nil {
            return err // may be ErrNoOp; caller short-circuits
        }
        return r.Client.Status().Patch(ctx, fresh, client.MergeFrom(base))
    })
}

Two small details carry weight. The ErrNoOp sentinel lets a step's mutate function say "nothing changed", distinct from a real error. So callers short-circuit cleanly. And event correlation only emits a Kubernetes event when the visible condition state actually changed, not on every generation-only refresh. Without that guard, a controller reconciling every 20 seconds floods the event stream with duplicates.

Gotcha: The uncached read in the snippet above was not the original design. It was a live-incident fix. The first version read through the informer cache, and that is exactly where the next section starts.

Why does a fresh status condition get overwritten?

A fresh status condition gets overwritten when a step reads the object through the informer cache instead of the API server. The cache can lag by tens of milliseconds within a single reconcile. That stale read then rides along in a MergeFrom patch and clobbers a condition another step set moments earlier (controller-runtime #741, 2021).

Here is the incident, exactly as it bit us. A routine release introduced new condition types on the StandardCluster resource, replacing an older readiness condition with a more specific one. A fresh reconcile did three things in sequence. First, MarkReconciling patched Ready=False at T=0. Second, the next step, seeing no prior value for its new condition type, re-read the object to update status. Third, the informer cache still returned the pre-restart object carrying a stale Ready=True.

That stale Ready=True traveled inside the same MergeFrom patch. So it overwrote the Ready=False written 49 milliseconds earlier. Net effect: clusters never converged to Ready. The maddening part? Running the controller locally fixed it every time. A local run used a fresh client that bypassed the shared cache. It looked non-deterministic. It was not. It was a read-your-own-write violation hiding in the cache.

Gotcha: If a bug disappears when you run the controller locally but persists in-cluster, suspect informer-cache staleness before you suspect a logic error. Local runs often bypass the shared cache path that production hits.

Most articles explain the reconcile loop as a clean read-diff-write cycle and stop there. They never connect the informer cache to a correctness bug. The cache is usually framed as a pure performance optimization. But at scale it becomes a correctness hazard, much like a stale eBPF program on GKE Dataplane V2 can keep enforcing a NetworkPolicy you already deleted. "Read" and "write" no longer target the same view of the world. The upstream root cause is a DeltaFIFO backlog plus read-write mutex contention in the informer cache. Under load, that produces stale reads (kubernetes/kubernetes #130767, 2025).

The fix in our code was to read through an uncached reader inside the retry loop, exactly as the earlier snippet shows. Upstream, the fix arrived in Kubernetes v1.36 in April 2026. Controllers can now check the cache's resourceVersion. Then they skip reconciliation when the cache is behind their own writes. That closes the read-your-own-write gap at the framework level (kubernetes.io blog, 2026).

When should OperationResultUpdated requeue?

OperationResultUpdated should almost never mean "requeue and come back later." It means something changed, and you must still fall through to the readiness check before deciding to requeue. Treating Updated as terminal produced an infinite roughly 20-second reconcile loop across seven step files in our service-exposure controller.

The shape of the bug is worth memorizing, because it is easy to reintroduce. A step's mutate function toggled an annotation on every reconcile. For example, a fast-reconcile-interval hint derived from current readiness. So the annotation changed each pass. As a result, controllerutil.CreateOrPatch returned OperationResultUpdated almost every time. Several steps then treated Updated as "done for now, requeue". So they never reached the check for whether the underlying cloud resource is actually ready.

switch result {
case controllerutil.OperationResultCreated,
    controllerutil.OperationResultUpdated,
    controllerutil.OperationResultNone:
    // All three fall through to the SAME readiness check.
    if !cloudResourceReady(obj) {
        return ctrl.Result{RequeueAfter: 20 * time.Second}, nil
    }
    return ctrl.Result{}, nil // ready: stop requeuing
}

We fixed this in seven sibling files. We collapsed the Created, Updated, and unchanged cases into one readiness check. Then we requeue only when the resource is still not ready. If you review a new step that wraps CreateOrPatch, look for the anti-pattern case OperationResultUpdated: return ctrl.Result{RequeueAfter: ...}, nil with no readiness check ahead of it. That single line is a self-perpetuating loop waiting to happen.

How do you handle read-after-create NotFound races?

Handle a read-after-create NotFound by retrying the transient error a few times, not by widening a cache window. The object exists on the API server but is not yet visible to your very next Get. This is an API-server-side visibility gap, the mirror image of client-side cache staleness, so the fixes point in opposite directions.

We hit this when a workspace-creation step patched owner references onto namespaces right after creating them. Occasionally the next Get returned NotFound, even though the namespace clearly existed. The instinct is to assume a longer cache-consistency wait would help. But the gap here is eventual server-side visibility, not a stale client cache. So a longer wait does nothing. Instead, the correct pattern is a short bounded retry on the transient NotFound. Treat it as expected, not as step failure.

Gotcha: A stale-cache bug and a read-after-create bug feel identical from a stack trace. One is fixed by reading past the cache; the other by retrying the read. Diagnose which side of the client-server boundary you are on before you reach for a fix.

How do you scale controllers with environment-based sharding?

Environment-based sharding partitions reconciliation by a business dimension: deployment environment. It uses controller-runtime predicates, rather than splitting the CRD, API, or codebase. One controller instance reconciling every environment couples their fate. So a reconcile storm in dev can degrade ope. Sharding decouples them, while keeping a single schema.

The mechanism is a release topology, not a code fork. So we deploy one Helm chart as several releases of the same version. The same fleet-shaped coordination shows up when you promote add-ons across many clusters, which we handle with Sveltos-driven fleet add-on promotion. A common release runs environment-agnostic singletons like the platform API and tenant controller. One release per environment runs only the environment-scoped controllers. Each gets a CLI argument declaring which single environment it owns.

Each environment-scoped controller composes its existing predicates with one extra environment-matching predicate. That predicate admits an object into the work queue only when its environment matches the shard's. And it derives that environment from the existing source of truth, like a label or namespace. Filtering happens before objects enter the queue. So out-of-scope objects never trigger a reconcile, and the existing logic stays untouched.

The design's real cost is not code. It is an operational invariant that lives outside the type system: the union of environment releases must cover every environment exactly once, no overlap and no gaps. The type checker cannot enforce this. Your release automation must. We also give each shard an independent leader-election identifier so releases never contend for the same lock. A missing environment argument falls back to non-sharded mode, which makes rollout incremental and reversible.

Why is even a logging change a production change?

A logging change is a production change. Why? Logs flow through parsers and agents that fail in ways your unit tests never see. In March 2026, two upgrades on our control plane both traced back to the same mistake: code duplicating behavior the framework already provided. And one of them silently deleted logs from the busiest controller in an environment.

The first incident was an apimachinery upgrade. A dependency bump removed a generated ProtoMessage() method from core meta types. Our generated proto file embedded those Kubernetes types directly. So the first time the gRPC server tried to marshal any response, the protobuf runtime panicked at lazy descriptor init. That took down the entire process. Because the panic fires at file-descriptor init rather than per field, one bad request crashed everything. The fix moved the HTTP gateway from endpoint-based registration to in-process registration. That bypassed the binary codec entirely, by reusing an existing JSON marshaler.

The second incident is the one I retell most. We added a reconcileID correlation field to all eight controllers with a .WithValues("controller", ..., "reconcileID", uuid.New()) call at the top of every Reconcile. Two days later, every log line from the busiest controller in one environment had silently vanished from Cloud Logging.

The root cause was a duplicate JSON key. Would your test suite have caught that? Ours did not. Controller-runtime already injects controller, name, namespace, and reconcileID into the context logger automatically. So our added call re-added the same keys. That produced JSON objects with duplicate keys. GKE's fluentbit log agent parses those logs via msgpack. And msgpack's Go implementation cannot hash a duplicate-key object. So it dropped the entire line, rather than erroring loudly. One environment escaped: its busiest controller used a logger variant that replaced keys instead of appending them. The fix was simple. We removed the redundant call, because the framework already carried every field we were trying to add.

Gotcha: Before you add fields to a structured logger, check what your framework already injects. Duplicate keys are valid JSON to your program and poison to your log pipeline.

What keeps controllers reliable at scale

Custom controllers fail in ways the tutorials rarely mention. And almost all of those failures live at one boundary: between your reconcile logic and the framework's caching. So structure reconcilers as ordered, independently testable steps. That keeps each unit of provisioning logic small and partial progress visible. Then centralize status patching in one conflict-safe helper. Read through an uncached reader in that path, so a stale cache read cannot overwrite a condition you just set.

Watch the framework, not just your code. OperationResultUpdated is not a signal to requeue. It is a signal to re-check readiness. A read-after-create NotFound wants a retry, not a wider cache window. And a logging change can be as destructive as a schema change. So if you run controllers at scale, adopt the Kubernetes v1.36 cache-staleness detection early. And treat every framework upgrade as the production change it is.

Local Kubernetes Development with Tilt: Fast Inner Loop

Aroua KABOUBI — Tue, 06 Oct 2026 09:53:04 +0000

Most Kubernetes teams debug their platform in the one place it does not run: their laptop. Production is 82% likely to be Kubernetes (CNCF Annual Cloud Native Survey, 2025, up from 66% in 2023), yet the local dev loop is often a Docker Compose file that quietly disagrees with the real cluster on namespaces, RBAC, and service discovery. We rebuilt our inner loop around a real local Kubernetes cluster driven by Tilt, and it changed how fast — and how honestly — we could iterate on a multi-service backend. This is the setup, the pattern, and the one bug it caught that a mock never would have.

Key takeaways

Run the actual deployment target locally: a real Kubernetes cluster on Kind reproduces namespaces, RBAC, and service discovery that Docker Compose only approximates.

Tilt's Live Update syncs code into a running container "in seconds, not minutes" (tilt.dev, 2026), which is what makes a heavier local substrate viable.

Ship one self-contained Tiltfile per service that runs standalone or composes into a root Tiltfile, so a new contributor can spin up just the service they touch.

Test auth against the real identity stack with curl, not a mock — it caught a client-config bug that mocked auth would have hidden.

Skip Bitnami: its free stable images were deprecated in 2025, so pinning to them is silent rot.

Why run a real cluster locally instead of Docker Compose?

Because the platform you ship to is Kubernetes, and Kubernetes has surfaces Docker Compose cannot fake. Namespaces, RBAC bindings, service accounts, DNS-based service discovery, and admission control are all first-class in a cluster and simply absent in Compose. When they break, a Compose-based local loop stays green and the bug surfaces only after you deploy — the most expensive place to find it.

Kubernetes is now the default production substrate: the CNCF (2025) puts production use at 82% and cloud-native adoption at 98%. If that is your target, local-vs-prod parity is not a luxury; it is the difference between a fast inner loop and a slow, deploy-driven outer loop. The same platform-engineering discipline that standardises production is worth applying to the laptop.

The honest tradeoff: a real cluster is heavier than docker compose up. You need a fast local cluster and a tool that keeps the edit-run cycle tight despite the extra substrate. That is exactly the gap Kind and Tilt fill.

What is Tilt, and how does it fix the Kubernetes inner loop?

Tilt is an open-source toolkit "for fixing the pains of microservice development" (tilt-dev/tilt, 9.9k GitHub stars, v0.37.5, 2026). It watches your files, rebuilds container images, and brings your cluster up to date automatically — turning a manual build-push-deploy sequence into a single tilt up that stays live as you edit.

The feature that earns its keep is Live Update. Instead of rebuilding an image and rolling out a new pod on every save, Live Update syncs the changed files straight into the running container and re-runs the build step in place. Tilt's own framing is deploying code to running containers "in seconds not minutes" (tilt.dev, 2026). For a compiled language like Go, that means the .go files land in the pod and the binary recompiles there — no image round-trip.

The Tiltfile itself is written in Starlark, a Python dialect, so it has real functions, loops, and conditionals rather than YAML templating. That matters once your local environment has more than one service and a few shared dependencies to wire together.

How do you bootstrap a local cluster with ctlptl and Kind?

Start with a cluster that creates itself. We use Kind — "local Kubernetes clusters using Docker container nodes" (kubernetes-sigs/kind, 15.4k stars, a CNCF-certified conformant installer, 2026) — fronted by ctlptl to attach a local image registry. A small helper script makes the whole thing idempotent, creating the cluster only if one is not already running:

#!/usr/bin/env bash
# scripts/ensure-cluster.sh — idempotent local cluster + registry
set -euo pipefail

CLUSTER_NAME="klocal"

if ctlptl get cluster "kind-${CLUSTER_NAME}" >/dev/null 2>&1; then
  echo "cluster kind-${CLUSTER_NAME} already up — reusing"
  exit 0
fi

# ctlptl creates the Kind cluster AND a local registry Tilt can push to
ctlptl create registry ctlptl-registry --port=5005
ctlptl create cluster kind --name="${CLUSTER_NAME}" --registry=ctlptl-registry

Wiring this in as a Tilt local_resource means a contributor runs tilt up and gets a cluster, a registry, and their service — no README full of prerequisite commands to run by hand first.

Gotcha: Let ctlptl own both the cluster and the registry. If you create a Kind cluster without a registry Tilt can push to, every build falls back to kind load, which is far slower than a pushed image and defeats the point of a tight loop.

What makes one Tiltfile per service the right pattern?

Each service ships its own self-contained Tiltfile, designed to run standalone or be folded into a root Tiltfile later via Tilt's include(). This is the reusable core of the whole setup, and it rests on three rules.

First, path independence: every path in the Tiltfile is resolved relative to the Tiltfile's own directory, so it behaves identically whether run from the service root or included from a parent. Second, a dependency toggle: a boolean flag groups shared infrastructure — Postgres, an identity stack, a gateway — so a future root Tiltfile can provide them once instead of each service starting its own copy. Third, standalone-first: the default tilt up in any service directory brings up a working environment for that service alone.

# Tiltfile — self-contained, includable, path-independent
config.define_bool('skip_deps')  # a root Tiltfile sets this to own shared deps
cfg = config.parse()

# ensure a cluster exists before anything else
local_resource('cluster', cmd='./scripts/ensure-cluster.sh',
               labels=['00-bootstrap'])

if not cfg.get('skip_deps', False):
    k8s_yaml('deploy/postgres.dev.yaml')     # plain upstream YAML, not Bitnami
    k8s_yaml('deploy/ory.dev.yaml')          # identity stack for local auth

docker_build('local/klastro-api', '.',
             dockerfile='Dockerfile.dev')
k8s_yaml('deploy/api.dev.yaml')

The payoff is onboarding: a new contributor touching one service clones it, runs tilt up, and is productive without standing up the entire platform. When they need the full stack, a root Tiltfile include()s each child and sets skip_deps=True so shared dependencies come up exactly once. It mirrors the structure of a submodule monorepo without coupling the services to each other.

How does Live Update keep the Go loop fast?

By syncing source and recompiling in the pod instead of rebuilding the image. The docker_build step declares a live_update block that copies changed .go files into the container and runs the compile there, so a save-to-running-binary cycle drops from a full image build to a few seconds.

docker_build(
    'local/klastro-api', '.',
    dockerfile='Dockerfile.dev',
    live_update=[
        sync('.', '/app'),                      # push changed source into the pod
        run('go build -o /app/server ./cmd/api',# recompile in place
            trigger=['./**/*.go']),
    ],
)

# Forward Delve so you can attach a debugger to the in-cluster process
k8s_resource('klastro-api', port_forwards=['8080:8080', '2345:2345'])

Port-forwarding Delve on 2345 means you attach your IDE's debugger to a process running inside the cluster, against real service DNS and real config — not a locally-linked binary that skips half the platform. You get breakpoints on the code path that will actually run in production. When the failure you are chasing is a non-deterministic one — the kind we hunt in controller cache staleness — debugging against the real substrate is the difference between reproducing it and guessing.

Why avoid Bitnami images and charts?

Because the free, stable Bitnami catalog is going away. Effective August 28, 2025, Bitnami moved its versioned Docker Hub images to a bitnamilegacy repository that receives no further updates, fixes, or support, with the public catalog narrowed to a latest-only community subset (bitnami/charts #35164, 2025). The deletion of the old catalog was postponed to late September 2025 to give teams time to migrate (Northflank, 2025).

For a local environment, pinning Postgres or an identity stack to a Bitnami tag now means one of two failure modes: a frozen legacy image that silently stops getting patches, or a "latest-only" image that shifts under you between rebuilds. Neither is reproducible. We deploy shared dependencies from plain upstream YAML or the projects' own charts instead, which keeps the local stack pinnable and free of a subscription dependency.

Gotcha: Bitnami charts and images were the path of least resistance for years, so they are baked into countless tutorials and starter repos. Audit your local deploy/ manifests for docker.io/bitnami/* references before they turn into a legacy image that never updates again.

How do you test auth flows before a UI exists?

With curl against the real identity stack — and it is worth doing precisely because it catches bugs a mock cannot. Before any frontend existed, we exercised protected API endpoints by minting a real session token from the identity provider and passing it as an X-Session-Token header through the gateway, straight to the API:

# 1. Mint a real session token from the identity stack (Ory Kratos)
SESSION_TOKEN=$(curl -s -X POST \
  "http://localhost:4433/self-service/login/api" \
  -H 'Content-Type: application/json' \
  -d '{"method":"password","identifier":"dev@example.test","password":"…"}' \
  | jq -r '.session_token')     # ory_st_…

# 2. Hit a protected endpoint THROUGH the gateway with that token
curl -s "http://localhost:8080/v1/orgs" \
  -H "X-Session-Token: ${SESSION_TOKEN}" | jq

Running this against the real authorization stack rather than a stub surfaced a genuine integration bug: the permission-check client library was resolving the authorization service's address as though the host:port needed a URL scheme, and failing the lookup. A mocked-auth local setup would have returned a cheerful 200 and hidden the defect until it hit a shared environment. Testing the real chain — the same reason we take privileged-access flows seriously — turned a would-be production incident into a five-minute local fix.

What does this cost, and when is it worth it?

The cost is real: a local cluster consumes more laptop RAM than Compose, and the initial Tiltfile-plus-ctlptl plumbing is an afternoon of work. The parity is worth it when your production target is Kubernetes and your bugs live in the platform surfaces — auth, RBAC, service discovery, admission — that Compose cannot represent. For a single stateless service with no cluster-specific behaviour, Compose may still be the right, lighter choice.

The break-even arrives fast for a multi-service backend. Once you have more than two services, shared dependencies, and any auth in the request path, the "works on my machine" gap between Compose and the cluster stops being an annoyance and starts being the source of your integration incidents. A per-service Tiltfile that composes into a whole-stack root gives each engineer the smallest environment they need and the full one when they want it.

Conclusion

Local Kubernetes development is not about running the whole platform on your laptop for its own sake — it is about closing the gap between where you write code and where it runs. Tilt and Kind make a real cluster fast enough to be your inner loop, Live Update keeps the Go cycle in seconds, and a self-contained-but-includable Tiltfile per service keeps onboarding to a single tilt up.

Two habits carry most of the value. Test against the real thing — real cluster, real identity stack, real curl — because mocks hide exactly the integration bugs that cost the most later. And keep your dependencies pinnable and free of surprise deprecations, which in 2025 means walking away from Bitnami's legacy catalog. Invest the afternoon in the Tiltfile; you buy back the parity every day after.

How to Build Kubernetes as a Service with Custom Controllers

Aroua KABOUBI — Tue, 29 Sep 2026 08:28:07 +0000

Key takeaways

Kubernetes as a Service turns a management cluster into a universal control plane: developers declare infrastructure as custom resources, controllers reconcile the real cloud.

The platform is modelled as cloud-agnostic KRM CRDs — LandingZone, KubernetesCluster, ServiceExposure — each backed by a custom Go controller on controller-runtime.

Crossplane plus the hyperscaler operators (Config Connector, ACK, Azure Service Operator) translate those resources into real cloud APIs, while continuous reconciliation corrects drift automatically.

Multi-tenancy (isolated tenants), aggregated RBAC, and federated identity let teams self-serve without the platform team losing governance.

Operationally, the control plane scales through per-environment controller sharding and an API-first surface (REST, gRPC, SDKs, and a Kubernetes API proxy).

In a previous article we argued why Terraform falls short for platform engineering and why Kubernetes wins as the engine of a self-service platform. That post answered why. This one answers how: the concrete architecture behind a production Kubernetes as a Service platform — the one pattern that lets product teams create and operate managed clusters on their own, while security, cost, and reliability stay centrally governed.

Everything below is drawn from building real control planes. Names and identifiers are generic on purpose; the architecture is what matters.

What "Kubernetes as a Service" actually means

Kubernetes as a Service is a self-service platform where a central management cluster is the control plane for all infrastructure. A developer declares a cluster, a database, or an exposed service as a Kubernetes custom resource; a controller reconciles that declaration into real cloud resources. No tickets, no manual apply, governance enforced in one place.

The shift is subtle but decisive. Instead of treating Kubernetes as a place to run containers, you treat it as the place to describe and reconcile everything else. The Kubernetes Resource Model (KRM) becomes the contract between the people who consume infrastructure and the platform that provisions it. That is the same idea we explored across our decade-long Kubernetes journey — pushed to its logical conclusion.

The management cluster: one control plane to rule them all

The management cluster is a dedicated Kubernetes cluster that owns no application workloads. Its only job is to host the platform API, authentication, custom controllers, and the resources that describe every downstream workload cluster. It is the single point where requests arrive, identities are verified, and reconciliation happens.

Centralising on one control plane pays off in four ways:

One API surface — teams reach the platform through kubectl, a CLI, an API, or an internal portal, all hitting the same endpoint.
One reconciliation engine — every resource is continuously driven toward its declared state by controllers running in the same place.
One policy boundary — RBAC, admission policies (for example Kyverno), and tenancy rules live together, not scattered across clusters.
One lifecycle — upgrades, add-on rollout, and drift correction for many clusters are orchestrated centrally.

Workload clusters — typically managed Kubernetes such as GKE — are outputs of the control plane, registered back into a multi-cluster manager once they become ready.

Modelling the platform as custom resources

The platform's public API is a small set of cloud-agnostic Custom Resource Definitions. Each CRD is a deliberate abstraction: it exposes the intent a team should control and hides the dozens of provider-specific primitives underneath. Two resources form the backbone of the model — LandingZone and KubernetesCluster.

`LandingZone`: an insulated multi-tenant boundary

A LandingZone is an enterprise-grade structural blueprint. It provisions a secure, insulated, compliant tenant boundary for a project, and its declarative controller loop performs the full multi-cloud reconciliation natively — abstracting away provider differences to stamp out and enforce organizational policy across three concerns at once:

Resource containment — the top-level isolation unit for the target cloud: a GCP Project, an AWS Account, or an Azure Resource Group.
Networking topology — VPCs, isolated subnets, firewall rules, and routing matrices.
Identity & security guardrails — IAM roles, policy attachments, and principal-group bindings.

The spec declares intent, never a vendor SKU. The same manifest targets any provider by changing one field:

apiVersion: platform.example.com/v1alpha1
kind: LandingZone
metadata:
  name: payments-dev
  namespace: proj-1234-tenant-mgmt
spec:
  provider: gcp                    # gcp | aws | azure
  env: dev
  resourceContainer:
    displayName: payments-dev      # → GCP Project / AWS Account / Azure Resource Group
  networking:
    cidrBlock: 10.24.0.0/16
    subnets:
      - name: nodes
        cidr: 10.24.0.0/20
      - name: pods
        cidr: 10.24.16.0/20
    firewall:
      denyAllIngress: true
  identity:
    guardrails:
      - policy: enforce-cmek
      - policy: restrict-public-ip
    roleBindings:
      - principalGroup: payments-platform-admins
        role: platform-admin

The controller reconciles that intent into the right provider objects — a Project plus VPC on GCP, an Account plus VPC on AWS, a Resource Group plus VNet on Azure — and continuously enforces the guardrails as policy drifts.

`KubernetesCluster`: managed clusters as declarative intent

A KubernetesCluster is the headline abstraction — a whole managed cluster reduced to the handful of decisions that matter, with everything else defaulted to a secure baseline. Crucially, the spec is runtime-agnostic: it describes what the team wants, not which managed engine backs it.

apiVersion: platform.example.com/v1alpha1
kind: KubernetesCluster
metadata:
  name: cluster-1
  namespace: proj-1234-dev
spec:
  landingZoneRef:
    name: payments-dev           # inherits provider, network, identity
  exposure: internal             # immutable: security boundary
  location: europe-west1         # immutable: physical placement
  version:
    channel: REGULAR
  gatewayApi:
    channel: standard

The control-plane engine parses this high-level intent and dynamically orchestrates the low-level managed primitive behind the scenes: GKE when the referenced LandingZone targets Google, EKS on AWS, AKS on Azure. The team never writes a provider-shaped cluster manifest — they declare a KubernetesCluster, and the composer picks the engine.

Two design rules keep the abstraction safe. Immutable fields (exposure, location, network) are rejected on update by an admission webhook, because changing them would silently recreate or endanger the cluster. Secure defaults — hardened dataplane networking, shielded nodes, sane maintenance windows — are applied automatically, so a minimal spec still produces a hardened cluster. The user makes five decisions; the controller makes five hundred.

The reconciliation loop: how a controller provisions the cloud

The engine behind every CRD is a custom controller running a reconciliation loop on controller-runtime. The loop watches a resource, compares desired state (spec) against observed state (status), and creates or updates the downstream objects that map to real cloud APIs. Then it runs again — forever — correcting drift.

At its core a reconciler is one idempotent function:

func (r *KubernetesClusterReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    var cluster platformv1alpha1.KubernetesCluster
    if err := r.Get(ctx, req.NamespacedName, &cluster); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    // 1. Render downstream provider objects (Config Connector / ACK / ASO / Crossplane).
    // 2. Wait for readiness; mirror observed state into .status.
    // 3. On ready: mint a kubeconfig secret, register the cluster with the
    //    multi-cluster add-on manager, and trigger add-on delivery.

    if !clusterReady(&cluster) {
        return ctrl.Result{RequeueAfter: 30 * time.Second}, nil
    }
    return ctrl.Result{}, r.Status().Update(ctx, &cluster)
}

The controller never calls a cloud SDK to build a cluster directly. Instead it renders infrastructure-operator resources and lets those controllers talk to the cloud. This layering is what makes the platform declarative end to end: the reconcile function is pure Kubernetes bookkeeping, and the actual API calls are delegated to purpose-built providers.

The infrastructure controller ecosystem

Under a LandingZone or KubernetesCluster sits a layer of infrastructure controllers that turn Kubernetes objects into real cloud API calls. There are two families, and a serious multi-cloud platform understands both.

Cloud-specific operators (the hyperscaler-native layer)

Each hyperscaler ships its own operator that maps its services one-to-one to CRDs. These are faithful, high-fidelity representations of a single provider's API — ideal when the composer needs a direct, provider-native primitive:

Google Cloud — Config Connector (KCC): maps GCP services (Projects, subnets, IAM policies, GKE clusters, BigQuery) to CRDs, reconciled by the cnrm controllers.
AWS — AWS Controllers for Kubernetes (ACK): a family of per-service controllers (EKS, VPC, IAM, S3, RDS) that manage AWS resources directly from Kubernetes.
Azure — Azure Service Operator (ASO): exposes Azure resources (Resource Groups, VNets, AKS, role assignments) as CRDs reconciled against Azure Resource Manager.

Each is authoritative for its own cloud, but each speaks only its own dialect. Build directly on them and your abstractions fragment along provider lines.

Provider-independent engines: Crossplane

Crossplane is the unifying layer. Its open-source provider ecosystem — provider-gcp, provider-aws, provider-azure (and the newer provider families generated from each cloud's API) — wraps the same underlying cloud APIs the native operators expose. On top of that, Compositions and Composite Resource Definitions (XRDs) let you fold those fragmented, per-cloud managed resources into a single, uniform, high-level abstraction.

This is exactly the mechanism behind our LandingZone and KubernetesCluster models: one composite claim fans out into a Project + VPC + IAM binding on GCP, or an Account + VPC + role on AWS, without the consumer ever seeing the provider seam. The platform controller orchestrates these engines — native operators for direct fidelity, Crossplane for cross-cloud composition — rather than replacing them. That separation keeps each layer testable and lets you onboard a new cloud by adding a provider, not by rewriting reconcile logic.

Multi-tenancy and RBAC without losing control

Multi-tenancy is enforced by the LandingZone boundary described above — the resource container, network topology, and identity guardrails that insulate each project — combined with an in-cluster tenant (for example Capsule) that scopes namespaces, quotas, and network policies. On top of that boundary, RBAC decides who can do what, and the clever part is that it extends itself as the platform grows.

The control plane defines three base roles — platform-admin, platform-editor, platform-viewer — as aggregated ClusterRoles. Every time a new service API is added, it ships its own role labelled to aggregate into the base ones, and Kubernetes merges the rules automatically:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: platform-admin
aggregationRule:
  clusterRoleSelectors:
    - matchLabels:
        rbac.platform.example.com/aggregate-to-platform-admin: "true"
rules: []   # filled automatically by the control plane

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: gke-platform-admin
  labels:
    rbac.platform.example.com/aggregate-to-platform-admin: "true"
rules:
  - apiGroups: ["platform.example.com"]
    resources: ["kubernetesclusters"]
    verbs: ["create", "get", "update", "delete", "list"]

Deploy a new service and its permissions flow into the existing personas — no central role edited by hand. Identities themselves come from your OIDC provider (Okta, Entra ID, Keycloak) and are federated into the cloud through Workload and Workforce Identity Federation, then mapped to least-privilege bindings. A project owner administers their own tenant; the platform team never becomes the bottleneck.

Add-ons as a service

A bare cluster is not a useful cluster. Every workload cluster needs security agents, policy engines, observability, cost optimisation, secret management, and ingress before a team can ship. Delivering that consistently across dozens of clusters is a first-class platform concern, not an afterthought.

The pattern is GitOps-driven multi-cluster delivery. When the controller registers a new cluster with an add-on manager such as Sveltos, the manager matches the cluster against a set of ClusterProfiles and rolls out the required Helm charts and manifests — endpoint security, policy enforcement, external secrets, autoscaling, backup, and gateways — in dependency order.

The subtle challenge is visibility. Add-on state lives on the management cluster as controller objects that ordinary users can't read. A production platform closes that gap by surfacing add-on status through the platform API: a read-only endpoint reads the underlying summaries on demand and returns a clean model — provisioned, provisioning, failed, waiting-for-dependencies — so a UI can render live health and even a dependency graph. The platform doesn't just install add-ons; it makes their state a supported, self-service API.

Exposing the platform as an API, not just YAML

CRDs are a great internal contract, but not every consumer wants to write YAML. Because a CRD is an API definition, you can generate a typed API surface on top of it: derive protobuf from the resources, then expose REST and gRPC endpoints and ship SDKs in Go, Python, JavaScript, and Java. Teams call the platform like any other product API.

For consuming live cluster state, a powerful addition is a transparent Kubernetes API proxy hosted by the control plane. The user authenticates once to the platform; the platform mirrors a workload cluster's API server so standard tools work by changing only the server URL:

kubectl --server=https://platform.example.com/tenants/t1/landingzones/lz1/kubernetesclusters/c1/proxy \
  get pods -n t1-lz1-dev

Critically, this is not an open pass-through. A mandatory, Kubernetes-aware policy engine classifies every request (group, version, resource, subresource, verb, namespace) and decides allow or deny before a single byte is forwarded. It runs deny-by-default, with a compiled-in deny floor — secrets, token minting, exec, attach, port-forward — that no misconfiguration can override. Compatibility comes from mirroring the API path; safety comes from the policy engine in front of it. Users get native kubectl ergonomics without ever holding a workload cluster's credentials.

Scaling the control plane with controller sharding

A single controller reconciling every environment at once couples their fate together: a reconcile storm or bad rollout in dev can degrade int and ope too. As the platform grows, that shared blast radius becomes the main operational risk.

The fix is horizontal partitioning by environment. The same controller binary is deployed as separate releases, each pinned to one environment through a startup flag, and each admits only its own resources using a controller-runtime predicate:

func EnvironmentPredicate(assigned string) predicate.Predicate {
    return predicate.NewPredicateFuncs(func(o client.Object) bool {
        return landingZoneEnvOf(o) == assigned
    })
}

Filtering at the predicate layer keeps out-of-scope objects out of the work queue entirely, so reconcile logic stays untouched. Environment-agnostic components — the API server, the tenant controller — run once in a common release, while each environment gets its own controllers, its own resource limits, and its own independent lifecycle. One environment can be upgraded, paused, or rolled back without touching the others. Leader election is scoped per shard so releases never contend for the same lease.

What the pattern buys you

Put together, these pieces deliver the promise of platform engineering: autonomy for teams, control for the platform. Product teams request a cluster, an exposed service, or a cloud resource with a single declarative call and get it in minutes. The platform team enforces security baselines, cost policy, and reliability standards centrally, and corrects drift automatically through reconciliation rather than through review meetings.

The engineering discipline that makes it work is worth restating: model a small, opinionated CRD surface; back each CRD with an idempotent reconciler; delegate cloud calls to hyperscaler operators (KCC, ACK, ASO) and Crossplane; make RBAC self-extending; deliver add-ons via GitOps and surface their state through the API; and shard controllers so scale never means shared failure. None of these are exotic — they are the boring, durable choices that keep a control plane running for years.

Conclusion: how Edixos can support you

At Edixos, building control planes like this is our core craft. A cloud platform is more than a stack of tools — it must be built to last, to evolve, and to fit your organisation's real constraints. Our work goes well beyond deploying clusters:

We write custom Kubernetes controllers that model your provisioning workflows and business rules as reconciliation loops.
We integrate composition engines like Crossplane and Kro to turn Kubernetes into a multi-cloud provisioning engine.
We work with existing providers such as Config Connector (KCC) to expose cloud services natively through CRDs.
We build complete, API-first platforms — REST, gRPC, and multi-language SDKs — with multi-tenancy, federated identity, and self-extending RBAC baked in.

This is exactly the kind of platform we described in why Kubernetes beats Terraform for platform engineering, made concrete. If your ambition is a robust, self-service Kubernetes platform that aligns innovation, agility, and governance, we have the building blocks and the field experience to make it real.

Let's talk about your Kubernetes as a Service platform

Are you looking to give your teams true self-service infrastructure while keeping security and cost under central control? Let's discuss your challenges and see how a custom control plane can bring you speed, reliability, and scale.

provider-ovh 2.17.0: full OVHcloud schema coverage for Crossplane

Aroua KABOUBI — Tue, 15 Sep 2026 13:10:38 +0000

Key takeaways

provider-ovh 2.17.0 wires the complete OVHcloud Terraform provider schema: 165 managed resources per API scope and 335 CRDs, up from 138 and 281.

Four database engines disappeared upstream. Migrate or delete objects of the removed kinds before you upgrade — the CRDs are gone, not deprecated.

The 31 newly wired kinds are beta: generated, built and unit-tested, but not yet exercised end to end against a live account.

provider-ovh is the Crossplane provider for OVHcloud that we build and maintain at Edixos. Version 2.17.0, released 25 July 2026, upgrades to OVHcloud Terraform provider 2.17.0 and — for the first time — exposes every resource in the upstream schema as a Crossplane managed resource.

That is the headline. The part that needs your attention before you upgrade is the removals.

What 2.17.0 actually contains

	v2.13.2	v2.17.0
OVHcloud Terraform provider	2.13.1	2.17.0
Managed resources per API scope	138	165
CRDs	281	335

Every resource is available in both API groups the provider ships: the cluster-scoped *.ovh.edixos.io and the namespaced *.ovh.m.edixos.io. That doubling is why the CRD count is roughly twice the resource count.

Breaking changes: read this before upgrading

Upstream 2.17.0 removed the deprecated cassandra, m3db, m3aggregator and redis database engines, along with the long-deprecated ovh_cloud_project_database_ip_restriction resource. The corresponding CRDs are gone.

Removed kind	Migration path
`ProjectDatabaseIPRestriction`	Declare restrictions inline on `ProjectDatabase.spec.forProvider.ipRestrictions`
`ProjectDatabaseRedisUser`	`ProjectDatabaseValkeyUser` against a `valkey` engine cluster
`ProjectDatabaseM3DbNamespace`	None — engine discontinued upstream
`ProjectDatabaseM3DbUser`	None — engine discontinued upstream

Migrate or delete any objects of the removed kinds before upgrading. Removing a CRD while managed objects still reference it leaves you with resources Crossplane can no longer reconcile, and finalizers that need clearing by hand.

ProjectDatabase now validates engine client-side. The supported set is clickhouse, grafana, kafka, kafkaConnect, kafkaMirrorMaker, mongodb, mysql, opensearch, postgresql and valkey — anything else is rejected before it reaches the OVHcloud API, which is a faster failure than waiting for a remote error.

Moving IP restrictions inline

apiVersion: databases.ovh.edixos.io/v1alpha1
kind: ProjectDatabase
metadata:
  name: example-postgres
spec:
  forProvider:
    serviceName: <service-name>
    engine: postgresql
    version: "16"
    plan: business
    ipRestrictions:
      - ip: 10.0.0.0/24
        description: platform-team

Delete the standalone ProjectDatabaseIPRestriction objects first, then apply the parent database with the rules inline.

31 new resources

The full-schema pass adds coverage across most of the OVHcloud surface:

Group	New kinds
`cloud`	`FloatingIP`, `Quota`, `SecurityGroup`, `SSHKey`
`gateway`	`CloudGateway`
`network`	`PrivateVrackNetwork`, `PrivateVrackSubnet`
`kms`	`KeyManagerContainer`, `KeyManagerContainerConsumer`, `KeyManagerSecret`, `KeyManagerSecretConsumer`
`storage`	`EFS`, `BlockVolume`, `BlockVolumeBackup`, `BlockVolumeSnapshot`, `FileShare`, `FileShareNetwork`, `FileShareSnapshot`, `ProjectFileStorageShare`, `ProjectFileStorageShareNetwork`, `ProjectStorageLifecycleConfiguration`, `ProjectStorageReplicationJob`
`databases`	`ProjectDatabaseClickhouseUser`, `ProjectDatabaseLogSubscription`
`kube`	`LogSubscription`
`logs`	`LogsEncryptionKey`, `LogsOutputGraylogStream`
`me`	`IdentityUserToken`
`email` (new group)	`DomainAccount`
`vrack`	`PublicRoutingPriority`, `VrackServicesOrder`

One deliberate design decision worth calling out: IdentityUserToken writes its token to the connection secret rather than to status, because the upstream attribute is marked sensitive. A credential in status is readable by anything with get access on the resource; a connection secret is a separate object you can RBAC independently.

Honest scope note

These 31 kinds are generated, built, linted and unit-tested, and their Terraform identifiers follow the upstream import documentation. They have not been exercised end to end against a live OVHcloud account. Treat them as beta, and open an issue if you hit something — that feedback is what promotes them out of beta.

Schema improvements inherited from 2.14.0–2.17.0

Four upstream releases folded in at once, so several fields appear that were never announced here individually:

ovh_cloud_project_kube: ip_allocation_policy, customization_cilium
ovh_cloud_project_volume: encryption, for customer-managed-key encrypted volumes
ovh_cloud_project_storage: tags
ovh_dbaas_logs_output_graylog_stream: encryption_keys_ids

Behavioural fixes in the same span: in-place node scaling for ovh_cloud_project_database; idempotent database creation on 409 and 500 responses; desired_nodes capped when shrinking a nodepool; UPDATING accepted as a pending Kubernetes state; and weight: 0 treated as a real value on load-balancer routes rather than as unset.

That last one is the kind of bug that only shows up in production: a route weighted zero is a deliberate drain, and treating it as absent silently re-enables traffic.

Upgrading

cat <<EOF | kubectl apply -f -
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-ovh
spec:
  package: xpkg.upbound.io/edixos/provider-ovh:v2.17.0
EOF

Order of operations that avoids trouble:

Inventory objects of the four removed kinds across every namespace.
Move IP restrictions inline on the parent ProjectDatabase, and Redis users to Valkey.
Delete the leftover objects of removed kinds.
Apply the provider upgrade above.
Confirm the package became healthy: kubectl get providers.pkg.crossplane.io provider-ovh.

If you skip step 3, the CRDs vanish underneath live objects, and you clean up finalizers by hand afterwards. It is recoverable, but it is an outage-shaped afternoon.

Full changelog: v2.13.2…v2.17.0

Where to go next

New to the provider? Start with the step-by-step provider-ovh guide — install on kind, wire credentials, deploy a managed cluster. Then read provider-ovh 2.18.0, which removes inline accessRules from FileShare in a way that fails quietly.

We maintain provider-ovh, and we build OVHcloud control planes on it for a living: Crossplane consulting · platform engineering · talk to us.

provider-ovh 2.18.0: FileShare access rules become their own resource

Aroua KABOUBI — Wed, 09 Sep 2026 10:14:24 +0000

Key takeaways

provider-ovh 2.18.0 adds FileShareACL and removes inline accessRules from FileShare.

The removed field is silently pruned, not rejected. An untouched FileShare manifest keeps applying cleanly while its access rules quietly stop being managed. This is the one thing to check before you upgrade.

Rules already provisioned in OVHcloud are not deleted by the upgrade — adopt them with crossplane.io/external-name.

Version 2.18.0 of provider-ovh, our Crossplane provider for OVHcloud, was released on 1 August 2026. It tracks OVHcloud Terraform provider 2.18.0 and brings the provider to 166 managed resources per API scope and 337 CRDs.

It is a small release with one sharp edge, so we are leading with the edge.

The breaking change that does not announce itself

Upstream removed the inline access_rules attribute from ovh_cloud_storage_file_share and replaced it with a standalone ACL resource. FileShare therefore loses four paths, in both API groups:

spec.forProvider.accessRules
spec.initProvider.accessRules
status.atProvider.accessRules
status.atProvider.currentState.accessRules

Here is why this deserves more care than a normal removal. Once the CRDs are upgraded, accessRules is not rejected. The Kubernetes API server prunes unknown fields, so a FileShare manifest you never touched will still apply, still report ready, and still look entirely healthy in kubectl — while the access rules it used to declare are no longer managed by anything.

A breaking change that errors is a nuisance. A breaking change that succeeds is a trap. Audit your FileShare objects for accessRules before you upgrade, not after.

kubectl get fileshares.storage.ovh.edixos.io -A \
  -o jsonpath='{range .items[?(@.spec.forProvider.accessRules)]}{.metadata.namespace}{"\t"}{.metadata.name}{"\n"}{end}'

Migrating to FileShareACL

Declare one FileShareACL per rule that was previously inline:

apiVersion: storage.ovh.edixos.io/v1alpha1
kind: FileShareACL
metadata:
  name: example-acl
spec:
  forProvider:
    serviceName: <service-name>
    shareIdRef:
      name: example-share
    accessTo: 10.0.0.0/24
    accessLevel: READ_WRITE

accessTo takes a single IP or CIDR. accessLevel is READ_WRITE or READ_ONLY. shareId resolves by reference or selector to a FileShare, so the ACL follows the share rather than hard-coding an ID.

One property to design around: every configurable field forces replacement. There is no in-place update of an ACL — changing accessTo or accessLevel deletes and recreates it. For a rule governing live traffic, that is a brief window with no ACL in place, so schedule changes accordingly rather than discovering it during an incident.

Adopting rules that already exist

The upgrade does not delete access rules already provisioned in OVHcloud. They keep working; they are simply unmanaged. To bring one under Crossplane without creating a duplicate, set the external name:

metadata:
  name: example-acl
  annotations:
    crossplane.io/external-name: <serviceName>/<shareId>/<aclId>

Given that every field forces replacement, adoption is strictly better than recreate-and-hope: it avoids a delete/create cycle on a rule that is currently allowing production traffic.

Also in 2.18.0

BlockVolume gains an optional availabilityZone, plus status.atProvider.currentState.location.availabilityZone. If you run across OVHcloud AZs, you can now pin a volume rather than accepting placement.

BlockVolume no longer replaces the volume when encryption is left unchanged. Upstream now keeps the value from state. Previously an unchanged encryption field could trigger a replacement — on a block volume, meaning data movement. If you had a BlockVolume that seemed to want recreating for no reason, this was likely why.

The provider image now downloads the native Terraform provider binary from the upstream GitHub release assets instead of releases.hashicorp.com. The reason is concrete: that mirror only carries some versions of the OVHcloud provider and has no 2.18.0 at all. Sourcing from the upstream release assets means the provider is never blocked on a mirror that lags or skips a version.

Upgrading

cat <<EOF | kubectl apply -f -
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-ovh
spec:
  package: xpkg.upbound.io/edixos/provider-ovh:v2.18.0
EOF

Sequence that avoids the trap:

Audit FileShare objects for accessRules and record every rule — IP or CIDR, and access level.
Collect the corresponding aclId values from the OVHcloud API for adoption.
Upgrade the provider.
Apply one FileShareACL per recorded rule, with crossplane.io/external-name set so each adopts its existing rule.
Confirm each ACL reaches SYNCED and READY, and that no duplicate rule appeared on the share.

Coming from 2.13.x rather than 2.17.0? Read provider-ovh 2.17.0 first — it removes four database engines and their CRDs, which is a larger migration than this one.

Full changelog: v2.17.0…v2.18.0

Where to go next

If you are new to the provider, the step-by-step provider-ovh guide covers installation, credentials and a first managed cluster.

We maintain provider-ovh and build OVHcloud control planes on it: Crossplane consulting · platform engineering · talk to us.

Provider-ovh 2.19.1: we removed the Terraform CLI from the provider

Aroua KABOUBI — Fri, 04 Sep 2026 11:54:20 +0000

Key takeaways

provider-ovh 2.19.1 drops the Terraform CLI entirely. All 168 managed resources per API scope reconcile in-process, in the controller, as Go function calls.

The image no longer carries the terraform binary, the native OVHcloud provider binary, bash, or a writable /terraform tree. No terraform init, no workspace on disk, no registry egress at runtime.

It is a patch number carrying breaking CRD changes. Read the migration section before you upgrade — one of them will reject existing objects.

Version 2.19.1 of provider-ovh, our Crossplane provider for OVHcloud, was released on 3 September 2026. It tracks the same OVHcloud Terraform provider 2.19.0 as the release before it, and brings the provider to 168 managed resources per API scope and 341 CRDs.

The upstream surface did not move. What moved is how the provider talks to OVHcloud at all — and it is the largest architectural change the project has had.

What a Crossplane provider built on upjet actually did

upjet generates a Crossplane provider from a Terraform provider. Historically it did that by running Terraform. For every managed object, on every reconcile, the controller would:

Create a workspace directory on disk and write a .tf file and a state file into it.
Fork and exec terraform init, resolving the OVHcloud provider from a local filesystem mirror baked into the image.
Fork and exec terraform plan, then parse the JSON output to decide whether the object was up to date.
Fork and exec terraform apply when it was not.

That works. It also means a Kubernetes controller whose real execution engine is a subprocess, whose real state is a directory tree, and whose image has to ship two extra binaries to function.

Here is what the 2.19.0 image needed just to boot:

FROM alpine:3.17.1
RUN apk --no-cache add ca-certificates bash

ENV PLUGIN_DIR=/terraform/provider-mirror/registry.terraform.io/${TERRAFORM_PROVIDER_SOURCE}/...
ENV TF_CLI_CONFIG_FILE=/terraform/.terraformrc

ADD https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_..._${TARGETOS}_${TARGETARCH}.zip /tmp
ADD ${TERRAFORM_PROVIDER_DOWNLOAD_URL_PREFIX}/..._${TARGETOS}_${TARGETARCH}.zip /tmp

RUN unzip ... -d /usr/local/bin \
  && chmod +x /usr/local/bin/terraform \
  && unzip ... -d ${PLUGIN_DIR} \
  && chown -R ${USER_ID}:${USER_ID} /terraform

A shell, a Terraform distribution, a vendored provider plugin, and a writable tree owned by the runtime user.

What 2.19.1 does instead

upjet has supported two in-process reconciliation modes for a while: one that calls a Terraform Plugin SDK v2 provider's CRUD functions directly, and one that speaks the Plugin Framework's interfaces in the same process. 2.19.1 moves every resource onto them.

106 resource types go through the SDKv2 path. 70 go through the Plugin Framework path. Zero go through the CLI.

There is no subprocess. Observe, Create, Update and Delete are Go calls into the OVHcloud provider's own code, linked into the controller binary. The whole Dockerfile is now this:

FROM alpine:3.24.1
RUN apk --no-cache add ca-certificates

ADD "bin/${TARGETOS}_${TARGETARCH}/provider" /usr/local/bin/provider

## Terraform provider metadata consumed by the controller at startup.
## All resources are reconciled in-process (Terraform Plugin SDK / Framework);
## no Terraform CLI or native provider binary is shipped in this image.
ENV TERRAFORM_VERSION=${TERRAFORM_VERSION}
ENV TERRAFORM_PROVIDER_SOURCE=${TERRAFORM_PROVIDER_SOURCE}
ENV TERRAFORM_PROVIDER_VERSION=${TERRAFORM_PROVIDER_VERSION}

USER ${USER_ID}
EXPOSE 8080
ENTRYPOINT ["provider"]

One Go binary and a CA bundle. Even bash is gone.

What that buys you in production

No writable working directory. The reason the provider needed one was the workspace tree. It doesn't have one now, which removes the obstacle to running the controller with a read-only root filesystem, one of the baseline hardening steps we apply as part of platform engineering work.

No registry egress at runtime. The CLI path resolved a provider plugin on every init, from a mirror that had to exist in the image. There is nothing to resolve any more. On a cluster with egress policy on the control plane namespace, that is one fewer exception to justify.

A smaller thing to scan. Two third-party binaries left the image. Every CVE either of them accumulates used to be your CVE, whether or not the code path was reachable.

No fork/exec per reconcile. A terraform init plus a plan for every object on every reconcile loop is process creation, filesystem I/O and JSON parsing you were paying for on top of the API call you actually wanted. We are not publishing a benchmark here — a fair one needs a real fleet and real OVHcloud rate limits — but the work removed is not subtle.

The bug this shook out, and why no test caught it

Wiring the SDKv2 path exposed something worth writing down, because it is a trap for anyone doing the same migration on their own upjet provider.

On the Plugin Framework path, upjet configures the provider itself from terraform.Setup.Configuration. On the SDKv2 path it does not. It passes terraform.Setup's Meta field straight through as the provider meta to Diff, RefreshWithoutUpgrade and Apply, and never populates it. Every OVHcloud SDKv2 resource begins with a meta.(*Config) type assertion.

Leave Meta nil, and all 106 SDK-backed resources panic on their first Observe:

interface conversion: interface {} is nil, not *ovh.Config

Not a crash loop — controller-runtime recovers the panic — just every SDK resource permanently failing to reconcile, quietly, with a stack trace in the logs. No CI job in the repository would have caught it: the CRD diff job compares schemas, the build job compiles, and the deploy job installs the provider without credentials, so it never reconciles anything.

2.19.1 configures the provider and sets ps.Meta, and caches the result per ProviderConfig and per effective configuration. The caching is not premature optimisation: configuring the OVHcloud provider runs loadAndValidate and calls GET /auth/details, and Connect runs on every reconcile of every object. A rotated OAuth access token produces a different cache key, so a cached client is never reused with credentials that have since changed.

Migrating: the patch number is lying to you

2.19.1 is a patch release with breaking CRD changes in it. That was a judgement call about not stranding people on a vulnerable image, and it means the version number will not warn you. These are the three changes that can bite.

1. Integer fields are now `integer`, not `number` (506 properties)

terraform providers schema -json renders Terraform's TypeInt as number. That is what the generated CRDs carried, across 506 properties. None of those fields is ever fractional, so declaring them as integer is a correction rather than a behaviour change.

Manifests and stored objects are unaffected: 3 was valid as a number and is valid as an integer.

The real break is in the generated Go types, where the corresponding fields move from *float64 to *int64. This matters only if you import apis/{cluster,namespaced}/... directly — in a composition function, an operator, or a test harness. If you write YAML, there is nothing to do.

2. Two string maps become atomic for server-side apply

x-kubernetes-map-type: granular is no longer set on User.openstackRc (in the cloud group) and PrivateNetwork.regionsOpenstackIds (in the network group) — across forProvider, initProvider and atProvider, in both API groups, eight paths in total.

Those maps are now atomic. A client that applies the map takes ownership of it whole. Two appliers can no longer each own individual keys, and a partial apply replaces the map instead of merging into it. If you have a controller and a human both writing keys into one of those maps, that arrangement stops working the way you expect.

3. `vrack` `CloudProject` now requires `projectId` — and will reject existing objects

This is the one that will actually page someone. CloudProject in the vrack group gains a CEL validation rule making spec.forProvider.projectId required whenever managementPolicies includes Create, Update or *.

Objects that do not set it and carry either policy are rejected on their next update. They keep running until something touches them, which is exactly the kind of latency that turns an upgrade into an incident three weeks later.

Find them before you upgrade:

# cluster-scoped
kubectl get cloudprojects.vrack.ovh.edixos.io -o json \
  | jq -r '.items[]
      | select((.spec.forProvider.projectId // .spec.initProvider.projectId) == null)
      | .metadata.name'

# namespaced
kubectl get cloudprojects.vrack.ovh.m.edixos.io -A -o json \
  | jq -r '.items[]
      | select((.spec.forProvider.projectId // .spec.initProvider.projectId) == null)
      | "\(.metadata.namespace)/\(.metadata.name)"'

Anything printed needs projectId set before the upgrade, or it will fail its next write.

Security

2.19.1 also remediates everything Trivy reported against the 2.19.0 image. Go moves from 1.25.6 to 1.25.13, and the Alpine base from 3.17.1 — which is end of life — to 3.24.1, along with the affected Go dependencies.

A Trivy scan of the resulting image reports no HIGH or CRITICAL findings. govulncheck reports the provider code as affected by none.

One advisory stays visible and is worth naming rather than hiding: golang.org/x/crypto/openpgp is flagged module-wide as GO-2026-5932. There is no fixed version, and the affected package is neither imported nor called by this provider.

The build-time Terraform pin moves from 1.8.1 to 1.15.9. It is only used to generate config/schema.json and is not shipped; regenerating with 1.15.9 produces byte-identical output, so no CRD change follows from it.

Also worth knowing, from 2.19.0

If you are coming from 2.18.0, 2.19.0 added two resources in both API groups:

Instance wraps the new upstream ovh_cloud_instance, covering flavorId (which resizes in place), imageId, powerState, networks, securityGroupIds, volumeIds and shares. Two sharp edges: changing imageId rebuilds the instance and wipes the root disk, and omitting securityGroupIds applies the project's default group while an explicit empty list applies none — an instance that accepts no inbound traffic at all.

InstanceGroup wraps ovh_cloud_instance_group: a placement group with an AFFINITY or ANTI_AFFINITY policy. It is immutable — changing name, region or policy replaces it — and membership is set only through an Instance's groupId.

This is the first of the two OVHcloud instance APIs exposed as a distinct CRD. The pre-existing ProjectInstance is untouched and keeps its own schema.

Upgrading

cat <<EOF | kubectl apply -f -
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-ovh
spec:
  package: xpkg.upbound.io/edixos/provider-ovh:v2.19.1
EOF

A sequence that avoids surprises:

Run the two kubectl/jq commands above and fix every CloudProject they print.
Check whether anything applies keys into User.openstackRc or PrivateNetwork.regionsOpenstackIds alongside another writer.
If you import the generated Go types, rebuild against 2.19.1 — the compiler will show you every *float64 that is now *int64.
Upgrade the provider.
Confirm your managed objects return to SYNCED and READY. Because reconciliation moved in-process, this is the step that validates the new path against your real credentials.

If your cluster policy pinned the provider's working directory or allowed egress specifically for the Terraform registry, both can now be removed.

Full changelog: v2.19.0…v2.19.1

Thanks

Every change in this release came from the community rather than from me.

@ekarlso contributed #65 — the move to in-process reconciliation, the removal of the Terraform CLI from the image, and the SDKv2 provider meta fix and its cache. It is a large, careful change to the most load-bearing part of the provider.

@VeSeWe made their first contribution with #63 — the Go, Alpine and dependency updates that take the image to zero HIGH and CRITICAL findings. Welcome, and thank you.

If you run provider-ovh, issues and pull requests are the fastest way to shape where it goes next.

Where to go next

If you are new to the provider, the step-by-step provider-ovh guide covers installation, credentials and a first managed cluster. Upgrading from 2.17.0 or earlier? Read 2.18.0 and 2.17.0 first — both remove fields, which is a heavier migration than this one.

We maintain provider-ovh and build OVHcloud control planes on it: Crossplane consulting · platform engineering · talk to us.

Kubernetes FinOps: A Field Guide to Cutting Cluster Cost Without Breaking Prod

Aroua KABOUBI — Tue, 01 Sep 2026 11:20:22 +0000

Key takeaways

Kubernetes bills balloon because the two numbers that drive them, pod requests and idle node capacity, are invisible by default. You pay for reserved capacity, not usage.

FinOps is a loop, not a project: Inform (make spend visible per team), Optimize (rightsize, spot, autoscale), Operate (keep it from drifting back).

The biggest single lever is rightsizing. Cast AI's 2026 report puts CPU overprovisioning near 69% — most of that is pure waste.

Spot plus autoscaling takes 70–90% off the compute bill for the right workloads; governance-as-code stops savings from eroding the moment attention moves on.

Teams without a FinOps practice waste an estimated 32–40% of cloud spend; mature teams pull that down to 15–20%.

Kubernetes made compute elastic. It also made the bill opaque. A cluster will happily schedule pods that ask for eight times the CPU they use, keep half-empty nodes running through the night, and never once tell finance which team is responsible. The result is a line item that grows faster than traffic, and a quarterly conversation nobody enjoys.

This is not a tooling gap you buy your way out of. It's a discipline: FinOps applied to the specific mechanics of Kubernetes. According to CNCF's FinOps for Kubernetes research, 68% of organizations saw their Kubernetes costs rise, with roughly half reporting increases above 20%. The teams that reverse this don't find a magic dashboard. They make spend visible, act on the waste, and build guardrails so it stays gone.

Why the Kubernetes bill is invisible by default

You are billed for capacity you reserve, not capacity you use. A pod's resources.requests reserves CPU and memory on a node whether the workload touches them or not. Set those requests as a comfortable safety margin, multiply across hundreds of pods, and the scheduler packs nodes around numbers that have little to do with reality. The gap between requested and used is the waste, and nothing in a default cluster surfaces it.

The second invisible number is idle node capacity. When requests are inflated, the scheduler can't bin-pack tightly, so you run more nodes than the real workload needs. Each node is a full instance on the bill. Neither of these shows up in a cloud provider's invoice, which reports instances and disks, not namespaces and teams. That accounting mismatch is the core problem: the cloud bill speaks the language of infrastructure, while your cost owners think in the language of applications.

FinOps is a loop, not a one-off cleanup

FinOps, as defined by the FinOps Foundation, runs on three phases that repeat: Inform, Optimize, Operate. Inform is visibility: allocate spend to the teams and applications that generate it, so cost has an owner. Optimize is action: rightsize requests, move fault-tolerant work to spot, eliminate idle resources, apply commitment discounts. Operate is durability: track cost against business goals continuously so the savings don't decay.

Most cost-cutting efforts fail because they stop after a single Optimize pass. Someone spends a sprint rightsizing, the graph dips, and six months later it's back where it started because nothing changed the behavior that created the waste. The loop is the point. A one-time cleanup is a snapshot; FinOps is the practice that holds the line.

Rightsizing: stop paying for requests you don't use

Rightsizing is the highest-leverage move available, because overprovisioned requests are the root of both invisible costs. Cast AI's 2026 Kubernetes cost report found CPU overprovisioning reached roughly 69% across the clusters it analyzed: for every three cores reserved, two sit idle and billed. Bring requests down toward real usage and two things happen at once: the reservation waste disappears, and the scheduler bin-packs onto fewer nodes.

The mechanism is straightforward but easy to get wrong by hand. The Vertical Pod Autoscaler observes actual consumption and recommends requests from data instead of a nervous guess. Run it in recommendation mode first, read the numbers, then apply. Set requests to the workload's real steady-state, keep limits sane to protect noisy neighbors, and revisit as traffic patterns shift. This is unglamorous work, and it is where most of the money is.

Spot and autoscaling: 70–90% off the right workloads

Spot instances deliver the largest per-unit discount in the cloud, typically 70–90% below on-demand, and in 2026 the interruption handling is mature enough to use in production. The catch is that the provider can reclaim the capacity on short notice, so spot is for work that tolerates a node vanishing: stateless services behind a load balancer, batch and CI, anything that reschedules cleanly. Stateful singletons and latency-critical paths stay on-demand.

The tool that makes this practical is a workload-aware autoscaler. Karpenter provisions nodes to fit pending pods, mixes spot and on-demand by policy, and drains interrupted nodes gracefully so a reclaim is a non-event rather than an incident. Pair spot with tight requests and you compound the wins: fewer nodes, and each node far cheaper. This is the combination that turns a scary bill into a boring one.

Governance as code: keep the savings from drifting back

Optimization erodes the moment attention moves on, unless the guardrails live in the cluster. This is where policy engines earn their place: Kyverno and Open Policy Agent let you encode cost rules as admission policy, so the cluster refuses waste before it's ever scheduled. Reject pods with no requests, cap oversized requests, block workloads that ask for a whole node's worth of memory without justification, require the cost-center label that makes allocation possible.

The shift is from cleanup to prevention. A rightsizing sprint fixes yesterday's waste; an admission policy stops tomorrow's from being created. Policy-as-code is also what lets a small platform team hold the line across dozens of clusters and hundreds of developers without becoming a manual gate. The guardrail runs on every apply, forever, and never gets tired.

The metrics that actually move the bill

Total cloud spend is a lagging number that tells you nothing actionable. The metrics that drive decisions are cost per namespace, cost per application, CPU and memory utilization, and idle-resource percentage. Cost per namespace gives waste an owner. Utilization tells you whether requests match reality. Idle percentage is the headroom you're paying for and not using.

OpenCost, a CNCF project, allocates real cloud spend down to the namespace, controller, and pod, translating the infrastructure invoice into the application language your teams think in. That translation is the whole game. Once a team can see its own number and is measured against it, behavior changes without anyone issuing a mandate. Visibility with ownership does more than any single optimization, because it makes the optimization someone's job.

Where teams get stuck

The pattern we see repeatedly: a team knows the bill is too high, runs one heroic rightsizing pass, saves 30% for a quarter, and watches it climb back because nothing operationalized the discipline. The gap is rarely knowledge of what to do. It's the wiring that makes it durable: allocation every team trusts, autoscaling and spot configured for the actual workload mix, and policy that holds the line without a human in the loop.

That wiring is platform engineering work, and it's what we build. We treat cost as a property of the platform, reconciled continuously alongside reliability and security, not a spreadsheet someone reconciles at quarter-end. The goal is a cluster where the cheap way and the right way are the same way, enforced by the system rather than by vigilance.

Kubernetes expertise: how to choose a consulting partner in 2026

Aroua KABOUBI — Tue, 18 Aug 2026 14:02:09 +0000

Key takeaways

Five delivery models all answer “yes” to “do you do Kubernetes?”. They are not selling the same thing, and three of them will be wrong for your problem.
The deciding criterion is neither day rate nor firm size. It is who actually writes the code on your cluster, and whether that is the person you met in pre-sales.
One question reveals incentive alignment: “what happens if we take operations back in two years?”

The market is not short of Kubernetes providers. It is short of ways to tell them apart, because five different businesses present themselves under the same phrase — “Kubernetes expertise” — and none will volunteer which one they actually are.

This guide sets out those five models, what each is really selling, and the questions that separate a pitch from a capability. It names no companies: the model is the right level of analysis, because two firms sharing a delivery model resemble each other far more than two firms sharing a headcount.
Disclosure :Edixos is a specialist cloud-native consultancy — model 3 — and also sells managed cloud operations. So we sit in two of the five models, including the one whose conflict of interest we flag. Put the same questions to us as to anyone else; we answer them at the end.

The five Kubernetes delivery models

1. The large systems integrator

The large technology consultancies, from several hundred to several thousand people, covering cloud, data, security and mobile, with framework agreements at most large enterprises.

What you are buying: capacity and contractual coverage. If you need forty people staffed next quarter, or procurement only permits pre-approved suppliers, this is the only workable model.

The trade-off : pyramid staffing. The architect who convinces you in pre-sales is not the person who writes your controllers — the margin comes precisely from that gap. On work where depth beats volume, the gap is paid in months of drift.

2. The cloud-native managed provider

Firms that build the platform and then run it, with 24/7 on-call and sometimes their own sovereign cloud.

What you are buying: the problem going away. No platform team to hire, no rotation to staff, no upgrade window to plan. For many organisations this is the right call, and it deserves saying plainly: if Kubernetes is not a strategic asset for you, operating it yourself is a cost with no return.

The trade-off: incentive alignment. A provider whose recurring revenue depends on running your platform has no economic reason to make you self-sufficient. Not bad faith — the structure of the model, and it can be contracted around. Ask directly: “contractually and technically, what happens if we want to take over operations in two years?“

3. The specialist cloud-native consultancy

Small to mid-sized firms where Kubernetes and platform engineering are the business, not one line in a catalogue of thirty expertises.

What you are buying : depth, and a handover. These teams write controllers, publish code, and work on CRDs, Crossplane, multi-tenancy and bare metal — where generalists stop. The normal deliverable is a platform your team then operates.

The trade-off: capacity. Ten people will not staff forty roles, will not clear procurement demanding three years of minimum revenue, and will not cover your security workstream in parallel. If your binding constraint is volume or process, this model wastes your time.

4. The freelancer

An independent expert, often excellent, with no overhead.

What you are buying: one precise skill, immediately. For an audit, an unblocking, or reinforcement on a team that already exists, this is frequently the best value in the market.

The trade-off: bus factor. No continuity, nobody to review the work. A control plane designed by one person without review becomes an asset you can neither evolve nor hand on. The risk is not competence — that is usually present.

5. Vendor-managed Kubernetes

GKE Autopilot, EKS, AKS, OVHcloud Managed Kubernetes, Scaleway Kapsule, with vendor support.

What you are buying: the control plane, operated by the vendor, at a cost nowhere near a human engagement. Many organisations need nothing else, and paying a consultancy for what a managed service already does is a classic procurement mistake.

The trade-off: support stops at the edge of the product. It will not design your golden path, will not write your abstractions, and will not say no to a team deploying whatever it likes. Everything that decides whether a platform gets adopted stays with you.

The decision table

	Large SI	Managed Provider	Specialist Consultancy	Freelancer	Vendor-managed
Who writes the code	Supervised juniors	Their engineers	The people you met	One person	Nobody (product)
Seniority guaranteed in contract	Rarely	Often	By construction	N/A	N/A
Scale to 40 people	✅	⚠️ Limited	❌	❌	N/A
Depth (CRDs, Crossplane, bare metal)	⚠️ Varies	✅	✅	✅ Narrow	❌
Your team inherits the platform	⚠️ Varies	❌ Rarely the goal	✅	⚠️ Unreviewed	N/A
24/7 on-call	⚠️ Optional	✅	⚠️ Varies	❌	✅ Product scope
Fits enterprise procurement	✅	⚠️ Varies	⚠️ Often not	❌	✅
Contract rewards finishing	❌ Billed by time	⚠️ Recurring revenue	✅ Defined deliverable	❌ Billed by time	N/A
Bus factor	Low	Low	Medium	High	Low

This table describes models, never named companies. A single firm may sit in two models — Edixos included, which does both consulting and managed operations. When that is the case, ask which of the two carries the margin.

Seven questions to ask in pre-sales

1. “Who specifically will do the work, and can I speak to them before signing?” The one question whose answer cannot be rehearsed. A polite refusal, or “we’ll confirm staffing after signature”, is itself an answer.

2. “Show me public code.” Controllers, operators, Crossplane providers, CNCF contributions, maintained Helm charts. Real Kubernetes expertise leaves public traces; slide-deck expertise leaves none.

3. “Tell me about a production incident you caused.” An unfair question, deliberately. A team that has operated clusters has one, tells it precisely, and explains what they changed afterwards. A team without one has never been on call.

4. “What exactly is the deliverable, and what does the end of the engagement look like?” A platform engagement with no exit milestone quietly becomes open-ended staffing. Insist on the Git repository, documentation, runbooks, a handover session, and a date.

5. “What happens if we take operations back in two years?” The question that reveals alignment better than any reversibility clause.

6. “What are you not good at?” A provider claiming to cover everything has disqualified itself. Across a surface as wide as cloud-native, no admitted boundary means no actual specialism.

7. “What does the deliverable cost, not the day?” An isolated day rate informs nothing. A cheaper profile who takes three times as long costs more, and costs you a quarter on top. The full calculation is in what each model really costs.

Five red flags

A cloud partner tier presented as Kubernetes expertise . A partnership measures resale volume, not engineering capability.
The senior CV in the appendix, with no staffing commitment in the contract.
No public trace at all — no repository, no talk, no technical writing, no post-mortem. In an ecosystem as open as the CNCF, that is abnormal.
A proposal that opens with a tool — “we will deploy platform X” — before anyone has understood your golden path.
A quote with no discovery phase. Nobody prices a Kubernetes migration without looking at what exists. An instant number means a change request is already planned.

Running the selection in three steps

1- Qualify the model before the firms. Half a day internally: do we need capacity, operations, depth, reinforcement, or just a managed cluster? This eliminates two-thirds of the market and stops you comparing offers that were never comparable.

2- Buy an audit before you buy a project. Two to five days of architecture review from two firms in the same model costs a fraction of the project and shows how they reason. It is the only test drive available.

3- Contract the exit at the start. Repository on your side, documentation as a deliverable, a dated handover session. An aligned partner agrees without argument.

Edixos against its own checklist

We are model 3: a small, senior consultancy building Kubernetes control planes, golden-path internal platforms, reusable Crossplane APIs and SRE agents. Here are our answers to the seven questions, in order

Who will do the work? Named engineers, with their record and certifications published on the team page. From the first conversation you are talking to the person who will write the code — there is no bench behind us.
Public code? A Crossplane provider for OVHcloud that we maintain, 15,000+ downloads, running in production for French social ministries. Sveltos fleet work, contributions to the Tilt ecosystem.
A production incident? Our field notes on Talos and bare metal and on a stale eBPF network policy on GKE are published, causes and fixes included.
Production tenure? Kubernetes in production since version 1.2, in 2015 — before CRDs, before managed offerings existed.
Real systems? Over a thousand applications migrated for an automotive manufacturer, 150+ CRDs for a bespoke PaaS, self-service platforms in production.
A clean end of engagement? Repository, documentation and runbooks yours from day one, and a dated handover session written into the contract.
What are we not good at? Volume. We will not staff forty roles, will not clear heavyweight procurement, and for operations at very large scale on a provider’s own sovereign cloud another model will serve you better. We say so in the first meeting rather than cost you a quarter.

And to answer our own alignment question: we also sell managed operations , as Managed Cloud Operations. The conflict of interest described in model 2 therefore applies to us too. What you can verify in the contract rather than take on trust: it is contracted separately, never as the obligatory sequel to a build, and everything you need in order to do without us is yours from day one.

The lowest-risk next step is a two-to-five-day architecture review. You leave with an assessment you can act on even if you never work with us afterwards.

Frequently asked questions

What kind of Kubernetes consulting partner should I choose?
The one whose model matches your binding constraint. Need headcount: a large systems integrator. Need never to operate the platform: a managed provider. Need technical depth and autonomy afterwards: a specialist consultancy. Vendor size is not the criterion; the delivery model is.

How do I verify a partner's real Kubernetes expertise?
Ask for public code: controllers, Crossplane providers, CNCF contributions, published post-mortems. Then ask to speak to the engineer who will do the work, not the pre-sales architect. The gap between those two people is the most useful signal in the whole procurement.

What questions should I ask a Kubernetes provider before signing?
Who specifically will do the work, and can I speak to them? Where is your public code? Tell me about a production incident you caused. What exactly is the deliverable and what does the end of the engagement look like? What happens if we take operations back in two years?

Does a certified cloud partnership prove Kubernetes expertise?
No. A partner tier measures resale volume and commercial certification counts, not the ability to design a control plane or write a controller. It is a procurement signal, not an engineering one, and it is routinely presented as the second.

Specialist consultancy or vendor-managed Kubernetes?
If you need a reliable cluster and nothing more, vendor-managed is enough and costs far less than a human engagement. A consultancy earns its place when the value sits in the layer above: golden paths, abstractions, governance, and product-team autonomy.

Why Terraform Falls Short for Platform Engineering (and Kubernetes Wins)

Aroua KABOUBI — Tue, 11 Aug 2026 15:26:03 +0000

Key takeaways:

Terraform is not the problem — it’s the wrong tool for a self-service platform: manual apply per change, no native API, no continuous reconciliation.
Kubernetes + KRM + CRDs turn infrastructure into a self-service API: request a database, bucket, or cache with a single YAML — no tickets.
Even Kelsey Hightower has stated Terraform isn’t suited for building cloud-provider-style control planes . -Field-reported KPIs from teams that made the switch: +50% developer productivity, −36% lead time, +40% software quality (Puppet State of DevOps, Gartner).

Do you practice DevOps or Cloud today? Then you’ve lived this: you’re shipping a critical feature, but everything stops because you need a Jira ticket to provision a database, an S3 bucket, or any other cloud service. Welcome to TicketOps — a pattern that creates bottlenecks, silos, and quietly kills the agility your teams spent years building.

Let’s be blunt: the celebrated principle “you build it, you run it” too often becomes “you build it, someone else queues it, and maybe one day you get to run it.” The result is wasted time, frustrated developers, and stalled innovation.

Why Terraform is not suited for Platform Engineering

Terraform is popular and works well for provisioning resources occasionally — but it was never designed to be the engine of a self-service platform. Three structural gaps explain why:

Manual cycle — a terraform apply is required for every change. Not viable for dynamic, constantly evolving resources.
No native API — you can’t easily expose infrastructure as self-service through a simple API.
No continuous reconciliation — Terraform doesn’t automatically drive infrastructure back to its desired state when it drifts.

In short, for a genuine platform-as-a-service, Terraform hits its ceiling fast.

Kubernetes + KRM + CRDs: the new self-service model

Kubernetes is no longer just a container orchestrator — it’s a universal control plane for your entire infrastructure. The mechanism is the Kubernetes Resource Model (KRM) plus Custom Resource Definitions (CRDs), which let you manage anything declaratively.

Imagine this:

Need a database? Apply a YAML.
Need a Cloud Storage bucket? Another YAML.
Want a Redis cache? That same YAML pattern again.

This is pure self-service, with zero tickets, powered by Kubernetes and projects like Crossplane, AWS Controllers for Kubernetes (ACK), and Google Config Connector.

During his talk at Control Plane Day with Crossplane 2023, Kelsey Hightower — Developer Advocate at Google and a defining voice in the cloud-native community — put it plainly:

“Terraform is not suited for building control planes similar to those of major public cloud providers.”

Coming from an engineer whose talks and live demos have guided modern infrastructure practice for a decade, that carries real weight.

Level up: expose everything via REST, gRPC, and SDKs

The goal of this architecture is simple: give teams autonomy without losing control. Kubernetes acts as the central control plane on top of KRM, and product, DevOps, and development teams provision what they need through a single API — exposed over REST, gRPC, or even WebSocket.

This API isn’t a gadget — it becomes the standard entry point, usable from client libraries (Go, Python, JavaScript, Java), CLI tools, or an internal portal like Backstage. Teams stop waiting on tickets, and delivery cycles accelerate.

In the background, the platform continuously reconciles resources against your cloud providers and key tools (Google Cloud, GitLab, Kubernetes, Vault, Okta…). While teams self-serve, your Platform Engineering, SRE, and DevSecOps groups enforce company-wide security rules, reliability standards, and best practices. You industrialize the cloud at scale without friction: governance stays central, teams gain speed — the best of both worlds.

Better still, you can expose your services as a first-class API. From your CRDs you can auto-generate protobuf definitions, then expose REST or gRPC endpoints and ship SDKs for every language your developers use. They won’t even need to write YAML — they’ll call your platform API directly. The modern developer’s dream becomes real:

Zero tickets
Instant provisioning
Transparent GitOps integration

The numbers speak for themselves

Here are concrete KPIs reported by organizations that made the leap from TicketOps to self-service platforms:

🚀 +50% developer productivity
🕒 −36% lead time between commit and production
🧘 +40% improvement in software quality
📈 75% of large enterprises expected to adopt this model by end of 2025 (Gartner)

These figures come from the field — notably the Puppet State of DevOps report and Gartner analysis. Your mileage will vary, but the direction is consistent: less waiting, faster delivery, higher quality.

Goodbye TicketOps, hello self-service

The era of endless Jira tickets for simple provisioning is over. Adopt Kubernetes as the foundation of your Platform Engineering — with KRM and CRDs — and give your teams a genuinely self-service experience. Your infrastructure becomes agile, scalable, and API-first, and your developers will thank you (maybe even with cookies 🍪).

So — ready to leave Terraform behind for the parts it was never built for, and step into an API-first future with Kubernetes? You won’t regret it. 🚀

Conclusion: how Edixos can support you

At Edixos, we know a cloud platform is more than a stack of tools: it must be built to last, evolve, and fit each organization’s specific needs. Our expertise goes far beyond deploying Kubernetes clusters:

We write custom Kubernetes controllers that automate your business processes and specific needs.
We integrate powerful composition engines like Crossplane and Kro to turn Kubernetes into a true multi-cloud provisioning engine.
We work with existing controllers such as Config Connector (KCC), which natively expose major cloud providers’ services through Kubernetes CRDs.
We build complete, API-first platforms — exposing your services via REST, gRPC, and multi-language SDKs — to unify access and accelerate adoption.

What we offer is unique know-how to industrialize your cloud platforms: governance, security, and self-service at scale. This is exactly the kind of work behind our decade-long Kubernetes journey. If your ambition is a robust platform that aligns innovation, agility, and control, we have the building blocks and the experience to make it happen.

Let’s talk about your advanced Kubernetes challenges

Are you looking to automate operations, model your business processes, or industrialize deployments on Kubernetes? Let’s discuss your challenges and see how the API Machinery can bring you speed, reliability, and scalability.

Crossplane and OVHcloud: A provider-ovh Step-by-Step Guide

Aroua KABOUBI — Thu, 06 Aug 2026 13:46:11 +0000

Key takeaways:

Crossplane turns your Kubernetes cluster into a control plane that provisions cloud infrastructure through custom resources — no terraform apply, just YAML that a controller reconciles continuously.

provider-ovh is an open-source Crossplane provider built by Edixos, generated with Upjet on top of the official OVHcloud Terraform provider. It exposes 135+ OVHcloud resources as CRDs and is published on the Upbound Marketplace

This guide is end to end: a kind cluster, Crossplane, the provider, credential wiring, and a real managed Kubernetes cluster provisioned on OVHcloud.

Introduction

Most teams still provision OVHcloud the imperative way: a Terraform run here, a console click there, a script nobody wants to touch. It works until you need self-service, drift correction, and RBAC — the things a platform team actually has to deliver.

Crossplane flips the model. Your Kubernetes cluster becomes the control plane, and infrastructure becomes a set of custom resources that a controller keeps reconciled. At Edixos we built provider-ovh to bring that model to OVHcloud, and this article walks through it from an empty machine to a running managed Kubernetes cluster on OVH.

If you have used Crossplane with AWS or GCP, this will feel familiar. If you have not, you will have a working control plane in about fifteen minutes.

What is Crossplane?

Crossplane is a CNCF open-source control plane that extends Kubernetes so it can provision and manage external infrastructure. You install a provider, which registers Custom Resource Definitions (CRDs) for a cloud’s resources, and from then on you declare that infrastructure as Kubernetes objects. A controller reconciles each object against the provider’s API, correcting drift and cleaning up on delete.

The payoff is that infrastructure inherits everything Kubernetes already gives you: declarative YAML, RBAC, GitOps, admission control, and a single API surface across clouds. Instead of teams filing tickets, they apply a manifest — and the platform enforces the guardrails.

What is provider-ovh?

provider-ovh is the Crossplane provider for OVHcloud, built and maintained by Edixos. It exposes OVHcloud services — public cloud projects, managed Kubernetes, databases, private networks, IAM, object storage, and more — as Kubernetes managed resources. It is available on the Upbound Marketplace and the source lives on GitHub.

The story behind it is worth telling, because it explains why the project exists at all. When we first looked, there was no official or community Crossplane provider for OVHcloud. We built the first one and offered to donate it — the response at the time was that community providers could not be considered official, and it would not be maintained internally. Facing other commitments, the project went dormant for nearly a year.

Then three engineers — Nicolas, Kevin, and Ferréol from Theseus — got in touch. They were already running the provider in production, powering a Database-as-a-Service offering for the French public sector. Shortly after, Fabien opened a pull request bumping the provider to Crossplane 2.0. That was the catalyst. The provider was reborn as v2.9.0, with Endre Karlson and Alegrowin joining as co-maintainers, and it is now an actively maintained community project covering 135+ OVH resources and 90%+ of the Terraform provider surface, on both Crossplane 1.x and 2.x.

The lesson we keep repeating: real users in production are what keep infrastructure projects alive.

How provider-ovh is built?

provider-ovh is generated, not hand-written. It uses Upjet, the Crossplane code-generation toolkit, on top of the official OVHcloud Terraform provider — pinned in this release to version 2.13.1. Upjet reads the Terraform provider’s schema and generates XRM-conformant managed resources: Go API types, deepcopy functions, controllers, and CRD manifests.

That approach is why coverage is so broad. Instead of implementing every OVH endpoint by hand, we regenerate the provider whenever OVHcloud ships a new Terraform version, and the new resources come along for free. The current release ships 281 CRD manifests — the 135+ resources appear in two flavors, cluster-scoped and namespaced, thanks to Crossplane v2 (more on that below). Bumping the underlying Terraform provider is a documented, four-command routine: update the Makefile and go.mod, run make generate, and commit the regenerated schema, types, and CRDs.

Prerequisites

Before you start, install the following tools locally. Everything runs on your machine except the OVHcloud API calls, which need a real account.

Docker (or another container runtime kind supports)
kind — to run a throwaway Kubernetes cluster
kubectl — to talk to the cluster
Helm — to install Crossplane
An OVHcloud account with API credentials (we generate them in Step 5)

Step 1: Create a local kind cluster

Crossplane is a lightweight control plane, so a single-node kind cluster is plenty for this walkthrough. Create one:

Confirm the cluster is up and kubectl points at it:

This cluster never runs your OVH workloads — it only hosts the Crossplane control plane that provisions them.

Step 2: Install Crossplane

Install Crossplane with its official Helm chart into a dedicated crossplane-system namespace. This deploys the core Crossplane controllers and the package manager that installs providers.

Wait for the pods to become ready, then verify the installation:

You should see the crossplane and crossplane-rbac-manager pods running. Crossplane has also registered its own core CRDs — providers, configurations, providerconfigs — which the next step builds on.

Step 3: Install provider-ovh

Installing a provider is itself a Kubernetes resource: you apply a Provider object pointing at the package image, and Crossplane’s package manager pulls it and registers its CRDs. Apply provider-ovh from the Upbound registry:

Pin an explicit version tag in production — check the Marketplace for the latest release rather than using :latest. Watch the provider become healthy:

Once the INSTALLED and HEALTHY columns both read True, the provider’s controllers are running and every OVH CRD is registered in your cluster.

Step 4: Discover the managed resources

Because the provider registers its resources as standard CRDs, you explore them with plain kubectl — no special tooling. This is one of the quiet advantages of the Crossplane model: your infrastructure catalog is queryable from the same API as everything else.

List the OVH resource types the provider added:

You will see resources grouped by service — kube.ovh.edixos.io, network.ovh.edixos.io, cloud.ovh.edixos.io, databases.ovh.edixos.io, iam.ovh.edixos.io, and more. To inspect the fields of any resource, use kubectl explain:

That prints the full spec for a managed OVHcloud Kubernetes cluster — every forProvider field you can set — straight from the CRD’s OpenAPI schema. The complete API reference is also published at doc.crds.dev.

Step 5: Configure OVHcloud credentials

The provider needs OVHcloud API credentials to reconcile anything real. You supply them through a Kubernetes Secret, then point a ProviderConfig at that Secret. First, generate an application key, application secret, and consumer key from the OVH API console, granting the API rights your resources require.

Create the Secret in the crossplane-system namespace:

Now create a ProviderConfig named default that tells the provider where to read those credentials:

Never commit real credentials to Git. In production, source the Secret from an external secret store — provider-ovh supports Crossplane’s external-secret-store integration for exactly this.

Step 6: Provision your first OVHcloud resources

With credentials wired up, provisioning is now a kubectl apply. Let’s build a real stack: a private network, a subnet inside it, and a managed Kubernetes cluster attached to both. This is the same flow that I walk through in the Platform Engineering with Crossplane & OVHcloud talk. Each manifest references the default ProviderConfig from the previous step.

Start with a private network. Replace serviceName with your OVH public cloud project ID:

Add a subnet inside that network — this is where your cluster’s nodes get their IP addresses. It references the private network by name with networkIdRef, so Crossplane waits for the network and wires the ID in for you:

Now a managed Kubernetes cluster that references both the network and the subnet by name — Crossplane resolves each reference and injects the resulting IDs automatically:

Apply them and watch Crossplane reconcile against the OVHcloud API:

The kubectl get managed command lists every external resource Crossplane is tracking, with READY and SYNCED columns. When both read True, your cluster exists on OVHcloud — provisioned entirely through the Kubernetes API. Add a NodePool the same way, referencing the cluster with kubeIdRef, and you have a complete managed Kubernetes environment described as YAML.

Cluster-scoped vs namespaced resources

provider-ovh ships every resource in two variants because it targets Crossplane 2.x. Cluster-scoped resources use the classic network.ovh.edixos.io group; namespaced resources use the *.m.ovh.edixos.io groups (the m marks the modern, namespaced API). Namespaced resources let you scope OVH infrastructure to a Kubernetes namespace, which is the foundation for safe multi-tenancy — each team gets its own namespace, its own RBAC, and its own slice of infrastructure. That doubling is why the release contains 281 CRD manifests for roughly 135 logical resources.

Beyond raw resources: Compositions

Applying individual managed resources is powerful, but it is not yet a platform. The real leverage comes from Compositions: you define a Composite Resource Definition (an XRD) that presents a simple, opinionated API — say, EdixosCluster — and a Composition that expands it into the private network, subnet, cluster, and node pool underneath. Developers request one high-level resource; the platform assembles the details with your standards baked in.

The provider-ovh repository ships a complete Composition example that does exactly this. This is where Crossplane stops being a Terraform replacement and becomes a self-service platform — the pattern we build for clients at Edixos every week.

Troubleshooting

When a resource will not become ready, the reconcile loop tells you why. Describe the managed resource and read its conditions and events:

Most failures fall into three buckets: credential errors (the Secret JSON is malformed, or the OVH token lacks the required API rights), reference errors (a *Ref points at a resource that does not exist or is not yet ready), and quota or region errors returned straight from the OVHcloud API. For controller-level issues, install the provider with the --debug flag via a DeploymentRuntimeConfig and check the provider pod logs in crossplane-system.

Conclusion

You now have the full loop: a kind cluster running Crossplane, provider-ovh installed and healthy, credentials wired through a ProviderConfig, and real OVHcloud infrastructure provisioned as Kubernetes resources. The same manifests drop straight into a Git repository and a GitOps pipeline — which is the whole point.

provider-ovh started as a project OVHcloud declined to adopt and survived because engineers ran it in production and pushed it forward. It is open source, community-maintained, and built for teams who want OVHcloud to behave like every other cloud in a Crossplane control plane. Star it on GitHub, try it on your own project, and open an issue if something is missing — that feedback is exactly what keeps it moving.

Crossplane and provider-ovh are one piece of a larger platform-engineering practice. For the wider context, read why Kubernetes beats Terraform for platform engineering and our field-tested blueprint for building Kubernetes as a Service — both grounded in the decade of production Kubernetes behind this provider.

Let’s talk about your Crossplane platform

Do you want to expose OVHcloud — or any cloud — as self-service infrastructure your teams consume through the Kubernetes API? Let’s discuss your platform goals and see how a custom Crossplane control plane can bring you speed, governance, and scale.

Frequently asked questions:

What is provider-ovh?
provider-ovh is a Crossplane provider built by Edixos that exposes OVHcloud resources as Kubernetes custom resources. It is generated with Upjet on top of the official OVHcloud Terraform provider, covers 135+ resources, and is published on the Upbound Marketplace under edixos/provider-ovh for Crossplane 1.x and 2.x.

Do I need OVHcloud API credentials to use provider-ovh?
Yes. The provider authenticates to the OVHcloud API using an application key, application secret, and consumer key. You generate these tokens from the OVH API console, store them in a Kubernetes Secret, and reference that Secret from a ProviderConfig so Crossplane can reconcile your resources.

Can I run Crossplane and provider-ovh on a local kind cluster?
Yes. Crossplane is a lightweight control plane and runs fine on a local kind cluster for learning and development. It only provisions real OVHcloud infrastructure once you supply valid API credentials through a ProviderConfig, so the control plane itself has no special hosting requirement.

How is provider-ovh different from the OVHcloud Terraform provider?
provider-ovh reuses the OVHcloud Terraform provider's schema under the hood but exposes it as Kubernetes CRDs reconciled continuously by Crossplane. Instead of running terraform apply, you declare resources as YAML and a controller keeps live infrastructure matching that declared state, with RBAC and GitOps built in.

Our Kubernetes Journey: 10 Years Building Cloud-Native Platforms

Aroua KABOUBI — Mon, 27 Jul 2026 18:21:28 +0000

Key takeaways:

Edixos ran Kubernetes 1.2 in production in 2015, using Third Party Resources — the predecessors of today’s CRDs — well before managed Kubernetes was mainstream.
Our edge is the platform layer: custom Go controllers, GitOps, and Crossplane that turn Kubernetes into an API-driven, self-service control plane.
Real production systems, not slideware: 1,000+ apps migrated for an automotive leader, 150+ custom CRDs for a bespoke PaaS, plus smart-city and utility platforms.

Introduction:

Some tools permanently change how you design infrastructure. For Edixos, that tool is Kubernetes — and our story with it started in 2015, when container orchestration was still in its infancy. A decade of trials, failures, and shipped production systems laid the foundation for the company we are today.

We are not another consulting firm renting out slides. We are practitioners who have run Kubernetes in production since version 1.2, written the controllers, and operated the clusters at 3 a.m. This article takes you behind the scenes: from our first master node to the platforms that now run thousands of applications.

How did Edixos start with Kubernetes?

It started in 2015 with kubeup.sh and Kubernetes 1.2 in production — a bold bet at a time when almost no one ran containers at scale. We used Third Party Resources (TPR), the ancestors of CRDs, installed clusters on AWS with kops, and migrated a SaaS platform onto this new architecture.

I still remember launching the kubeup.sh script and watching our first master node appear. It was more than a technical success — it was a defining moment.

To automate environment creation, we even wrote our own Kubernetes controller in Python. It was demanding, hand-crafted work, but incredibly formative. In 2016 I moved into Kubernetes consulting, helping major companies across France — from bare-metal setups to managed services. Every mission widened the expertise Edixos is built on.

What makes Edixos different?

We don’t just promise excellence — we build it daily, line by line, cluster by cluster. What sets us apart is turning complex problems into elegant, durable solutions: automating Kubernetes environments, designing custom controllers, and industrializing GitOps practices with a relentless focus on impact.

Working with Edixos means partnering with seasoned practitioners who are genuinely passionate about cloud-native best practices — engineers who write production code, not PowerPoints, and who level up your teams as they go.

Real projects, not PowerPoints :

At Edixos we write code, Kubernetes manifests, robust APIs, and custom controllers — not slides. Every mission is a real technical challenge and a complex business problem. Here are a few of the adventures that forged our reputation.

🚗Cloud transformation for a French automotive leader
For a major French car manufacturer, we orchestrated the migration of more than 1,000 applications to the public cloud. Kubernetes became the backbone, with a centralized control plane built on Crossplane and KRM and multi-tenant custom controllers written by our team. The result: teams now consume infrastructure as a service through CRDs — with traceability, RBAC, and governance built in, and zero friction.

🌐A Kubernetes-first Platform-as-a-Service
Inspired by KRM and GitOps, we designed a bespoke PaaS on Kubernetes for a client. Key features:

A self-service REST and gRPC API.
Over 150 custom CRDs.
In-house Go controllers automating the creation of clusters, workloads, and managed services.
Built-in automatic reporting, policy enforcement, and FinOps.

This is not an off-the-shelf product. It’s a living platform, engineered for scalability, control, and team autonomy.

🏙️Kubernetes serving smart cities
For a public-sector organization, we built a Kubernetes platform hosting critical components: web services, databases, Kafka brokers, and urban data collectors. Every microservice and every real-time pipeline is orchestrated resiliently. It was never just a “cluster” — it was a digital city operated continuously.

💧Innovation flowing like water
Using Kubernetes and Kafka, we built a real-time water-consumption monitoring solution for a local operator: leak detection, live dashboards, and optimized storage. Proof that cloud-native and sustainability can go hand in hand.

These projects show that at Edixos we build cloud-native platforms to engineering standards. If you’re looking for a partner who truly knows API Machinery, controllers, and GitOps — and speaks YAML and Go better than PowerPoint — you’re in the right place.

Conclusion:

Cloud-native is a field of execution, not theory. At Edixos we don’t sell promises; we ship architectures that run, pipelines that deploy, platforms that evolve, and APIs that simplify. What we build is designed to last, to scale, and to serve your teams.

And when the official documentation isn’t enough, that usually means we need to write it ourselves — in Go, with tests, and a solid controller-runtime. Whether you want to industrialize GitOps, create custom CRDs, stand up an enterprise PaaS, or adopt Crossplane for infrastructure as code, our engineers have already been there. And they still are.

Curious how these platforms actually work under the hood? Read why Terraform falls short for Platform Engineering and how Kubernetes + CRDs deliver true self-service.

Let’s talk about your advanced Kubernetes challenges:

Straight answers:

Frequently asked questions:

When did Edixos start working with Kubernetes?

Edixos' founder ran Kubernetes 1.2 in production in 2015, using Third Party Resources (the ancestors of CRDs) and installing clusters on AWS with kops. That decade of hands-on experience predates most managed Kubernetes offerings and shaped the company's platform-engineering practice.

What is a Kubernetes custom controller and why does it matter?

A custom controller is code that watches Custom Resources and continuously reconciles real infrastructure to match a declared state. It matters because it turns Kubernetes into an API-driven platform: teams request databases, clusters, or workloads through YAML instead of tickets, with RBAC and governance built in.

Does Edixos only deploy Kubernetes clusters, or build platforms on top?

Both, but the value is in the platform layer. Edixos writes custom controllers, integrates Crossplane and Config Connector, and exposes infrastructure as self-service APIs (REST, gRPC, SDKs) — so internal teams provision what they need without friction while SRE and security keep central governance.

What industries has Edixos built Kubernetes platforms for?

Edixos has delivered cloud-native platforms across automotive (migrating 1,000+ applications to public cloud), the public sector (smart-city services on Kubernetes), and utilities (real-time water-consumption monitoring with Kafka), among others — each a production system, not a proof of concept.

DEV Community: Aroua KABOUBI

Kubernetes Controllers: The Cache-Staleness Bug at Scale

What is the step-based reconciler pattern?

The shared conflict-safe status patch

Why does a fresh status condition get overwritten?

When should OperationResultUpdated requeue?

How do you handle read-after-create NotFound races?

How do you scale controllers with environment-based sharding?

Why is even a logging change a production change?

What keeps controllers reliable at scale

Local Kubernetes Development with Tilt: Fast Inner Loop

Why run a real cluster locally instead of Docker Compose?

What is Tilt, and how does it fix the Kubernetes inner loop?

How do you bootstrap a local cluster with ctlptl and Kind?

What makes one Tiltfile per service the right pattern?

How does Live Update keep the Go loop fast?

Why avoid Bitnami images and charts?

How do you test auth flows before a UI exists?

What does this cost, and when is it worth it?

Conclusion

How to Build Kubernetes as a Service with Custom Controllers

What "Kubernetes as a Service" actually means

The management cluster: one control plane to rule them all

Modelling the platform as custom resources

LandingZone: an insulated multi-tenant boundary

KubernetesCluster: managed clusters as declarative intent

The reconciliation loop: how a controller provisions the cloud

The infrastructure controller ecosystem

Cloud-specific operators (the hyperscaler-native layer)

Provider-independent engines: Crossplane

Multi-tenancy and RBAC without losing control

Add-ons as a service

Exposing the platform as an API, not just YAML

Scaling the control plane with controller sharding

What the pattern buys you

Conclusion: how Edixos can support you

Let's talk about your Kubernetes as a Service platform

provider-ovh 2.17.0: full OVHcloud schema coverage for Crossplane

What 2.17.0 actually contains

Breaking changes: read this before upgrading

Moving IP restrictions inline

31 new resources

Honest scope note

Schema improvements inherited from 2.14.0–2.17.0

Upgrading

Where to go next

provider-ovh 2.18.0: FileShare access rules become their own resource

The breaking change that does not announce itself

Migrating to FileShareACL

Adopting rules that already exist

Also in 2.18.0

Upgrading

Where to go next

Provider-ovh 2.19.1: we removed the Terraform CLI from the provider

What a Crossplane provider built on upjet actually did

What 2.19.1 does instead

What that buys you in production

The bug this shook out, and why no test caught it

Migrating: the patch number is lying to you

1. Integer fields are now integer, not number (506 properties)

2. Two string maps become atomic for server-side apply

3. vrack CloudProject now requires projectId — and will reject existing objects

Security

Also worth knowing, from 2.19.0

Upgrading

Thanks

Where to go next

Kubernetes FinOps: A Field Guide to Cutting Cluster Cost Without Breaking Prod

Why the Kubernetes bill is invisible by default

FinOps is a loop, not a one-off cleanup

Rightsizing: stop paying for requests you don't use

Spot and autoscaling: 70–90% off the right workloads

Governance as code: keep the savings from drifting back

The metrics that actually move the bill

Where teams get stuck

Related reading

Kubernetes expertise: how to choose a consulting partner in 2026

Key takeaways

The five Kubernetes delivery models

1. The large systems integrator

`LandingZone`: an insulated multi-tenant boundary

`KubernetesCluster`: managed clusters as declarative intent

1. Integer fields are now `integer`, not `number` (506 properties)

3. `vrack` `CloudProject` now requires `projectId` — and will reject existing objects