Kubernetes Operators Explained: What They Are, How They Work, and How to Build One

What Is a Kubernetes Operator?

A Kubernetes Operator is a method of packaging, deploying, and managing a Kubernetes application using custom resources and custom controllers. In concrete terms, an operator is a controller that watches a Custom Resource Definition (CRD) and continuously works to make the actual state of the cluster match the desired state declared in that custom resource.

The idea is straightforward: take the operational knowledge that a human engineer would use to run a complex application -- how to deploy it, how to scale it, how to recover from failures, how to handle upgrades -- and encode that knowledge into software that runs inside the cluster. Instead of writing runbooks and hoping someone follows them at 3 AM, you write a controller that handles it automatically.

Operators extend the Kubernetes API itself. When you install an operator, you get new resource types (like EtcdCluster or PostgresCluster) that you can manage with kubectl just like built-in resources such as Deployments and Services. The operator watches these custom resources, reacts to changes, and takes whatever actions are necessary to maintain the desired state.

If you are running Kubernetes in production -- and the adoption numbers suggest most organizations are -- understanding operators is essential. They are the standard mechanism for running stateful and complex workloads on Kubernetes, from databases and message queues to monitoring stacks and certificate management.

Origin of the Operator Pattern

The Operator pattern was invented at CoreOS in November 2016. Engineers Brandon Philips and the CoreOS team published a blog post titled "Introducing Operators" that laid out the concept: use the Kubernetes control loop mechanism to automate the management of complex, stateful applications.

The problem they were solving was real. Kubernetes was already good at running stateless applications -- deploy a container, scale it horizontally, replace failed instances. But stateful applications like databases, distributed key-value stores, and monitoring systems required specialized knowledge to operate. Scaling an etcd cluster, for example, is not the same as scaling a web server. You need to add new members in a specific order, update peer URLs, handle quorum implications, and manage data migration. These are operational tasks that Kubernetes Deployments and StatefulSets alone cannot handle.

CoreOS shipped two operators alongside the announcement: the etcd Operator for managing etcd clusters and the Prometheus Operator for managing Prometheus monitoring deployments. Both demonstrated the core value proposition -- instead of following a multi-step runbook to scale your etcd cluster, you edit a single field in a YAML file and the operator handles the rest.

CoreOS was acquired by Red Hat in 2018, and Red Hat was subsequently acquired by IBM. The operator concept survived these transitions and became a cornerstone of the OpenShift platform. More importantly, the broader Kubernetes community adopted operators as the standard pattern for running complex workloads.

The Problem Operators Solve

To understand why operators exist, consider what it takes to run a PostgreSQL database cluster on Kubernetes without one.

You need to:

1. Deploy the primary instance with the correct storage configuration and initialization scripts.
2. Set up streaming replication to one or more standby instances.
3. Configure connection pooling (PgBouncer or similar).
4. Manage automated backups to object storage on a schedule.
5. Handle failover when the primary goes down -- promote a standby, reconfigure replication, update connection endpoints.
6. Perform minor version upgrades with rolling restarts in the correct order (standbys first, then primary).
7. Perform major version upgrades with pg_upgrade, which requires a completely different procedure.
8. Monitor replication lag, connection counts, and query performance.
9. Scale read replicas based on load.

Each of these tasks requires domain-specific knowledge about PostgreSQL. Kubernetes knows nothing about replication lag or WAL archiving. A StatefulSet can create pods with persistent storage and stable network identities, but it cannot orchestrate a failover or run a backup.

This is the gap that operators fill. A PostgreSQL operator (such as CloudNativePG, Crunchy PGO, or Zalando's postgres-operator) encodes all of this operational knowledge into a controller. You declare what you want in a PostgresCluster custom resource, and the operator handles the how.

How Operators Work: Custom Resources + Custom Controllers

Every operator consists of two components working together: a Custom Resource Definition (CRD) that extends the Kubernetes API with new resource types, and a custom controller that watches those resources and acts on them.

Custom Resource Definitions (CRDs)

A CRD tells the Kubernetes API server about a new resource type. Once you apply a CRD to a cluster, you can create, read, update, and delete instances of that resource type just like any built-in resource.

Here is a simplified CRD that defines an EtcdCluster resource:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: etcdclusters.etcd.database.coreos.com
spec:
  group: etcd.database.coreos.com
  names:
    kind: EtcdCluster
    listKind: EtcdClusterList
    plural: etcdclusters
    singular: etcdcluster
    shortNames:
      - etcd
  scope: Namespaced
  versions:
    - name: v1beta2
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                size:
                  type: integer
                  minimum: 1
                version:
                  type: string

After applying this CRD, users can create EtcdCluster resources:

apiVersion: etcd.database.coreos.com/v1beta2
kind: EtcdCluster
metadata:
  name: my-etcd-cluster
  namespace: default
spec:
  size: 3
  version: "3.5.17"

The CRD alone does nothing. The Kubernetes API server will store this resource, but nothing happens until a controller is watching for it.

The Custom Controller (Reconciliation Loop)

The controller is the brain of the operator. It runs inside the cluster (typically as a Deployment) and implements a reconciliation loop -- the same control loop pattern that drives every built-in Kubernetes controller.

The reconciliation loop works like this:

1. Observe -- Watch the API server for changes to the custom resource (and any related resources like Pods, Services, or ConfigMaps).
2. Analyze -- Compare the desired state (what the custom resource spec says) with the actual state (what currently exists in the cluster).
3. Act -- Take whatever actions are needed to make the actual state match the desired state. Create pods, update configurations, trigger failovers, run backups.

This is a level-triggered system, not edge-triggered. The controller does not react to individual events ("a pod was deleted"). Instead, it reacts to the current state ("the spec says 3 replicas but only 2 exist"). This makes operators inherently resilient to missed events, network partitions, and controller restarts -- the next reconciliation will always pick up where things left off.

Here is a minimal reconciliation function in Go using the controller-runtime library:

func (r *EtcdClusterReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    log := log.FromContext(ctx)

    // Fetch the EtcdCluster resource
    var cluster etcdv1beta2.EtcdCluster
    if err := r.Get(ctx, req.NamespacedName, &cluster); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    // List existing pods for this cluster
    var pods corev1.PodList
    if err := r.List(ctx, &pods, client.InNamespace(req.Namespace),
        client.MatchingLabels{"app": "etcd", "cluster": cluster.Name}); err != nil {
        return ctrl.Result{}, err
    }

    currentSize := len(pods.Items)
    desiredSize := cluster.Spec.Size

    if currentSize < desiredSize {
        log.Info("Scaling up", "current", currentSize, "desired", desiredSize)
        if err := r.addMember(ctx, &cluster, currentSize); err != nil {
            return ctrl.Result{}, err
        }
        return ctrl.Result{RequeueAfter: 30 * time.Second}, nil
    }

    if currentSize > desiredSize {
        log.Info("Scaling down", "current", currentSize, "desired", desiredSize)
        if err := r.removeMember(ctx, &cluster, pods.Items[currentSize-1]); err != nil {
            return ctrl.Result{}, err
        }
        return ctrl.Result{RequeueAfter: 30 * time.Second}, nil
    }

    cluster.Status.ReadyMembers = countReadyPods(pods.Items)
    if err := r.Status().Update(ctx, &cluster); err != nil {
        return ctrl.Result{}, err
    }

    return ctrl.Result{}, nil
}

The key thing to notice: this function is idempotent. You can call it a hundred times and it will always converge toward the desired state without creating duplicates or causing conflicts.

The Operator Framework

Building an operator from scratch requires significant boilerplate. The Operator Framework, originally created by CoreOS and now maintained by Red Hat and the community, provides tools to streamline this process.

Operator SDK

The Operator SDK (currently at v1.42.x) is the primary tool for building operators. It supports three approaches:

• Go-based operators -- Write the controller in Go using controller-runtime. Full control, standard for production.
• Ansible-based operators -- Write reconciliation logic as Ansible playbooks and roles.
• Helm-based operators -- Wrap an existing Helm chart as an operator. Fastest path from chart to operator.

Scaffolding a new Go-based operator project:

# Initialize a new operator project
operator-sdk init --domain example.com --repo github.com/example/my-operator

# Create a new API (CRD + controller)
operator-sdk create api --group cache --version v1alpha1 --kind Memcached --resource --controller

Operator Lifecycle Manager (OLM)

The Operator Lifecycle Manager (OLM) handles the installation, upgrade, and lifecycle management of operators themselves. It provides dependency resolution, update channels, RBAC scoping, and CRD upgrade safety.

The community is currently transitioning to OLM v1, which simplifies the API surface with the new ClusterExtension resource.

Real-World Operator Examples

Prometheus Operator (kube-prometheus-stack)

The Prometheus Operator introduces CRDs like Prometheus, ServiceMonitor, PodMonitor, and AlertmanagerConfig that let you declaratively define your monitoring stack.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-app
  labels:
    release: kube-prometheus-stack
spec:
  selector:
    matchLabels:
      app: my-app
  endpoints:
    - port: metrics
      interval: 30s
      path: /metrics

cert-manager

cert-manager (currently at v1.19.x) automates the management of TLS certificates in Kubernetes.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: my-app-tls
  namespace: default
spec:
  secretName: my-app-tls-secret
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  dnsNames:
    - app.example.com
    - www.example.com
  duration: 2160h
  renewBefore: 360h

CloudNativePG

CloudNativePG is a PostgreSQL operator that manages the full lifecycle of PostgreSQL clusters: provisioning, high availability with automated failover, continuous backup, point-in-time recovery, rolling updates, and monitoring integration.

Strimzi (Apache Kafka Operator)

Strimzi manages Apache Kafka clusters on Kubernetes. Running Kafka on Kubernetes is notoriously complex -- Strimzi encodes all of this into operators that manage Kafka, KafkaTopic, KafkaUser, and KafkaConnect resources.

Operator Maturity: The Capability Model

The Operator Framework defines a five-level capability model:

1. Level 1 -- Basic Install: Automated deployment and configuration.
2. Level 2 -- Seamless Upgrades: The operator handles version upgrades including migrations.
3. Level 3 -- Full Lifecycle: Backup, restore, and failure recovery.
4. Level 4 -- Deep Insights: Metrics, alerts, and log processing.
5. Level 5 -- Auto Pilot: Automatic scaling, tuning, and self-healing.

Most Helm-based operators top out at Level 2. Reaching Levels 3-5 requires a Go-based or Ansible-based operator with substantial domain logic.

When to Use Operators (and When Not To)

Use an Operator When:

• You are running a stateful application that requires domain-specific operational procedures.
• Operations require multi-step orchestration -- the sequence of actions matters.
• You need automated Day-2 operations -- backups, certificate rotation, version upgrades.
• You want to provide a self-service API for a platform team.
• A mature operator already exists for your workload.

Do Not Use an Operator When:

• A Deployment or StatefulSet is sufficient.
• A Helm chart solves the problem.
• You do not have the capacity to maintain it.
• The operational logic is trivial.

A useful rule of thumb: if the operational runbook for your application is more than a page long and involves conditional logic ("if replica lag exceeds X, then do Y"), it is a good candidate for an operator. If the runbook is "run helm upgrade," it is not.

Building a Basic Operator with Operator SDK

Step 1: Scaffold the Project

mkdir memcached-operator && cd memcached-operator
operator-sdk init --domain example.com --repo github.com/example/memcached-operator
operator-sdk create api --group cache --version v1alpha1 --kind Memcached --resource --controller

Step 2: Define the API

Edit api/v1alpha1/memcached_types.go:

type MemcachedSpec struct {
    // +kubebuilder:validation:Minimum=1
    // +kubebuilder:validation:Maximum=10
    Size int32 `json:"size"`

    // +kubebuilder:default="64m"
    MemoryLimit string `json:"memoryLimit,omitempty"`
}

type MemcachedStatus struct {
    ReadyReplicas int32              `json:"readyReplicas"`
    Conditions    []metav1.Condition `json:"conditions,omitempty"`
}

Step 3: Implement the Controller

The controller checks if a Deployment exists, creates one if not, ensures replica count matches the spec, and updates status. Key patterns:

• Owner references -- The operator sets the CR as the owner of the Deployment. When the CR is deleted, Kubernetes garbage collection cleans up.
• Idempotent reconciliation -- Every call checks current state and only acts on drift.
• Requeue -- After making a change, trigger another reconciliation to verify.
• Status subresource -- Update .status separately from the spec.

Step 4: Build, Deploy, and Test

# Build the operator image
make docker-build IMG=my-registry/memcached-operator:v0.1.0

# Push to a registry
make docker-push IMG=my-registry/memcached-operator:v0.1.0

# Install the CRD
make install

# Deploy the operator to the cluster
make deploy IMG=my-registry/memcached-operator:v0.1.0

# Create a sample Memcached resource
kubectl apply -f config/samples/cache_v1alpha1_memcached.yaml

# Watch the operator create the Deployment
kubectl get memcached
kubectl get deployments
kubectl get pods

Writing Production-Grade Operators: Best Practices

Handle Finalizers

If your operator creates resources outside the cluster, use finalizers to ensure cleanup when the custom resource is deleted.

Use Server-Side Apply

Instead of Get-Modify-Update patterns that are prone to conflict errors, use Server-Side Apply (SSA) for managing owned resources.

Expose Metrics

Your operator should expose Prometheus metrics about its own health: reconciliation duration, errors, managed resource counts, and queue depth.

Write Integration Tests

The envtest package lets you run the Kubernetes API server and etcd locally for integration testing -- significantly more valuable than unit tests that mock the Kubernetes client.

Practical Advice for Getting Started

1. Use existing operators before building your own. Install cert-manager, the Prometheus Operator, or CloudNativePG. Study how they work.
2. Read the Kubernetes documentation on controllers. Understanding how built-in controllers work will make custom controllers intuitive.
3. Start with the Operator SDK tutorial. Work through it hands-on with a local kind or minikube cluster.
4. Study production operators. Read the source code of mature operators like cert-manager.
5. Build your first operator for a real (but low-risk) use case.

The operator pattern is one of the most important abstractions in the Kubernetes ecosystem. It bridges the gap between "Kubernetes can run containers" and "Kubernetes can run anything" by encoding the operational knowledge that complex applications require. Whether you are adopting existing operators or building your own, understanding how they work will make you a more effective Kubernetes engineer.