OpenShift: Build a Homegrown Operator with the Ansible Operator SDK in Under a Day

Most platform teams end up with the same artifact graveyard: a dozen Helm charts, a values-prod.yaml nobody dares to diff, and a Jenkins job that runs helm upgrade --install whenever someone remembers. It works on day 1. It collapses on day 2, when someone edits a Deployment by hand, when a certificate needs rotating, when a schema migration must run before the rollout, or when a namespace is deleted and leaves orphaned external resources behind.

An Operator fixes the structural problem: it is a control loop, not a one-shot templating engine. And with the Ansible Operator SDK you do not need to write a line of Go. If you already have Ansible roles, you can ship a production-grade Operator, installable through OLM, in a single working day.

This article walks through the whole chain: CRD design, the reconciliation role, the watches.yaml, the bundle, the ClusterServiceVersion, and the OLM install on an OpenShift cluster — plus the RBAC and SCC traps that will bite you in a hardened cluster.

Why an Operator and not another Helm chart

Helm renders templates and applies them. That is a push model: the desired state exists only at the moment of the helm upgrade. Between two pipeline runs, nothing reconciles anything. If an SRE scales a StatefulSet manually at 3 a.m., the drift persists until the next release.

An Operator inverts the model. The desired state lives in a custom resource stored in etcd, and a controller continuously compares it to the actual state. Three concrete consequences:

• Drift correction is free. Any modification to a child object triggers a reconcile through the owner reference watch.
• Day-2 verbs become first-class API. Backup, restore, version upgrade, credential rotation — they are fields in a CRD spec, not pipeline parameters.
• Deletion is handled. Finalizers let you deregister a DNS entry, drop a database, or revoke a Vault role before the namespace disappears.

The Ansible flavour of the Operator SDK gives you this reconcile loop while your business logic stays in YAML tasks executed by ansible-runner. The trade-off is honest: slower reconciles than Go (a few hundred milliseconds of Ansible startup per run), no typed client, no deep caching. For an internal application managed by a platform team, that trade-off is almost always worth it.

Prerequisites

operator-sdk version
# operator-sdk version: "v1.37.0", commit: "...", kubernetes version: "v1.30.0", go version: "go1.22"

oc version --client
oc whoami -c
podman login quay.io

You need cluster-admin (CRDs are cluster-scoped), a registry the cluster can pull from, and OpenShift 4.12+. Everything below was validated on 4.16.

Scaffolding the project

mkdir internalapp-operator && cd internalapp-operator

operator-sdk init \
  --plugins=ansible \
  --domain acme.io \
  --owner "Platform Team" \
  --project-name internalapp-operator

operator-sdk create api \
  --group apps \
  --version v1alpha1 \
  --kind InternalApp \
  --generate-role

You now have roles/internalapp/, watches.yaml, config/ (Kustomize bases for CRD, RBAC, manager) and a Makefile that drives the whole build.

Designing a real CRD, not a free-form blob

The scaffolded CRD ships with x-kubernetes-preserve-unknown-fields: true. Never ship that to users: there is no validation, no oc explain, and no safe evolution path. Write a structural schema instead.

# config/crd/bases/apps.acme.io_internalapps.yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: internalapps.apps.acme.io
spec:
  group: apps.acme.io
  names:
    kind: InternalApp
    listKind: InternalAppList
    plural: internalapps
    singular: internalapp
    shortNames: [iapp]
  scope: Namespaced
  versions:
    - name: v1alpha1
      served: true
      storage: true
      subresources:
        status: {}
      additionalPrinterColumns:
        - name: Replicas
          type: integer
          jsonPath: .spec.replicas
        - name: Version
          type: string
          jsonPath: .spec.image.tag
        - name: Ready
          type: string
          jsonPath: .status.conditions[?(@.type=="Ready")].status
        - name: Age
          type: date
          jsonPath: .metadata.creationTimestamp
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              required: [image]
              properties:
                replicas:
                  type: integer
                  minimum: 0
                  maximum: 20
                  default: 2
                image:
                  type: object
                  required: [repository, tag]
                  properties:
                    repository:
                      type: string
                    tag:
                      type: string
                      pattern: '^v[0-9]+\.[0-9]+\.[0-9]+$'
                route:
                  type: object
                  properties:
                    enabled:
                      type: boolean
                      default: false
                    host:
                      type: string
                databaseSecretRef:
                  type: object
                  required: [name]
                  properties:
                    name:
                      type: string
                backup:
                  type: object
                  properties:
                    schedule:
                      type: string
                      pattern: '^(\S+\s+){4}\S+$'
                    retentionDays:
                      type: integer
                      minimum: 1
                      default: 7
            status:
              type: object
              x-kubernetes-preserve-unknown-fields: true

Two things matter here. First, status keeps preserve-unknown-fields because the operator_sdk.util.k8s_status module writes an open-ended condition structure. Second, the regex on image.tag is your first line of defence against someone deploying :latest in production.

The reconciliation role

The role receives the CR spec as Ansible variables (snake-cased by default: database_secret_ref, retention_days). Keep it strictly idempotent, and never use command or shell where a module exists.

# roles/internalapp/tasks/main.yml
---
- name: Resolve the database secret
  kubernetes.core.k8s_info:
    api_version: v1
    kind: Secret
    name: "{{ database_secret_ref.name }}"
    namespace: "{{ ansible_operator_meta.namespace }}"
  register: db_secret
  no_log: true

- name: Fail fast when the secret is missing
  ansible.builtin.fail:
    msg: "Secret {{ database_secret_ref.name }} not found in {{ ansible_operator_meta.namespace }}"
  when: db_secret.resources | length == 0

- name: Apply the application workload
  kubernetes.core.k8s:
    state: present
    apply: true
    definition: "{{ lookup('template', item) | from_yaml }}"
    wait: true
    wait_timeout: 300
  loop:
    - deployment.yaml.j2
    - service.yaml.j2
    - pdb.yaml.j2

- name: Expose the application through a Route
  kubernetes.core.k8s:
    state: "{{ 'present' if route.enabled | default(false) else 'absent' }}"
    definition: "{{ lookup('template', 'route.yaml.j2') | from_yaml }}"

- name: Schedule the logical backup
  kubernetes.core.k8s:
    state: "{{ 'present' if backup.schedule is defined else 'absent' }}"
    definition: "{{ lookup('template', 'backup-cronjob.yaml.j2') | from_yaml }}"

- name: Publish status
  operator_sdk.util.k8s_status:
    api_version: apps.acme.io/v1alpha1
    kind: InternalApp
    name: "{{ ansible_operator_meta.name }}"
    namespace: "{{ ansible_operator_meta.namespace }}"
    status:
      observedGeneration: "{{ ansible_operator_meta.annotations['acme.io/generation'] | default(0) }}"
    conditions:
      - type: Ready
        status: "True"
        reason: ReconcileSucceeded
        message: "Deployment reconciled at version {{ image.tag }}"

The Deployment template must carry owner references so that garbage collection and the watch mechanism work:

# roles/internalapp/templates/deployment.yaml.j2
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ ansible_operator_meta.name }}
  namespace: {{ ansible_operator_meta.namespace }}
  labels:
    app.kubernetes.io/name: {{ ansible_operator_meta.name }}
    app.kubernetes.io/managed-by: internalapp-operator
spec:
  replicas: {{ replicas | default(2) }}
  selector:
    matchLabels:
      app.kubernetes.io/name: {{ ansible_operator_meta.name }}
  template:
    metadata:
      labels:
        app.kubernetes.io/name: {{ ansible_operator_meta.name }}
    spec:
      securityContext:
        runAsNonRoot: true
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: app
          image: "{{ image.repository }}:{{ image.tag }}"
          imagePullPolicy: IfNotPresent
          envFrom:
            - secretRef:
                name: {{ database_secret_ref.name }}
          ports:
            - containerPort: 8080
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
          resources:
            requests:
              cpu: 100m
              memory: 256Mi
            limits:
              memory: 512Mi
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080

The SDK injects owner references automatically for resources created by the kubernetes.core.k8s module when manageStatus and the ownership mechanism are active — but the apply: true path requires the objects to live in the same namespace as the CR. For cluster-scoped children, set ownerReferences explicitly or clean them up in the finalizer.

watches.yaml: reconcile period and finalizer

# watches.yaml
---
- version: v1alpha1
  group: apps.acme.io
  kind: InternalApp
  role: internalapp
  reconcilePeriod: 5m
  manageStatus: true
  watchDependentResources: true
  snakeCaseParameters: true
  finalizer:
    name: apps.acme.io/deprovision
    role: internalapp-deprovision

The default reconcilePeriod for Ansible-based Operators is 10 hours. For drift correction on an internal app, drop it to a few minutes — but remember every tick starts an ansible-runner process. With 300 CRs and a 1-minute period, you will saturate the manager. Measure before tuning, and raise --max-concurrent-reconciles through the manager args rather than hammering the period.

The finalizer role is where day-2 deletion logic belongs: dropping a schema, deregistering from an external service registry, deleting an S3 bucket. Make it tolerant of already-deleted resources, otherwise a failing finalizer will wedge the namespace in Terminating forever.

Build, run and iterate

Local run against a live cluster, without deploying the manager:

export KUBECONFIG=~/.kube/config
make install            # applies the CRD only
make run                # runs ansible-operator locally against the cluster

Then in another shell:

oc new-project demo-internalapp
oc create secret generic app-db --from-literal=DB_HOST=pg.internal --from-literal=DB_USER=app
oc apply -f - <<'EOF'
apiVersion: apps.acme.io/v1alpha1
kind: InternalApp
metadata:
  name: billing
  namespace: demo-internalapp
spec:
  replicas: 3
  image:
    repository: quay.io/acme/billing
    tag: v2.4.1
  route:
    enabled: true
  databaseSecretRef:
    name: app-db
  backup:
    schedule: "0 2 * * *"
    retentionDays: 14
EOF

oc get internalapps -n demo-internalapp
# NAME      REPLICAS   VERSION   READY   AGE
# billing   3          v2.4.1    True    42s

When satisfied, build and push the Operator image. On OpenShift, base it on the Red Hat–supported image rather than the upstream one:

# Dockerfile
FROM registry.redhat.io/openshift4/ose-ansible-operator:v4.16

COPY requirements.yml ${HOME}/requirements.yml
RUN ansible-galaxy collection install -r ${HOME}/requirements.yml --force \
 && chmod -R ug+rwx ${HOME}/.ansible

COPY watches.yaml ${HOME}/watches.yaml
COPY roles/ ${HOME}/roles/
COPY playbooks/ ${HOME}/playbooks/

export IMG=quay.io/acme/internalapp-operator:0.1.0
make docker-build docker-push IMG=$IMG

Packaging for OLM

OLM is what turns a Deployment into an installable, upgradable, discoverable operator. The bundle is an OCI image containing the CRDs, the ClusterServiceVersion and metadata.

make bundle IMG=$IMG VERSION=0.1.0 CHANNELS=stable DEFAULT_CHANNEL=stable
make bundle-build bundle-push BUNDLE_IMG=quay.io/acme/internalapp-operator-bundle:0.1.0
operator-sdk bundle validate quay.io/acme/internalapp-operator-bundle:0.1.0 \
  --select-optional suite=operatorframework

The generated CSV needs curation. The parts that matter:

# bundle/manifests/internalapp-operator.clusterserviceversion.yaml (excerpt)
apiVersion: operators.coreos.com/v1alpha1
kind: ClusterServiceVersion
metadata:
  name: internalapp-operator.v0.1.0
  annotations:
    categories: "Application Runtime"
    containerImage: quay.io/acme/internalapp-operator:0.1.0
    support: [e-mail]
    alm-examples: |-
      [{"apiVersion":"apps.acme.io/v1alpha1","kind":"InternalApp",
        "metadata":{"name":"billing"},
        "spec":{"replicas":3,"image":{"repository":"quay.io/acme/billing","tag":"v2.4.1"},
                "databaseSecretRef":{"name":"app-db"}}}]
spec:
  displayName: InternalApp Operator
  version: 0.1.0
  maturity: alpha
  installModes:
    - type: OwnNamespace
      supported: true
    - type: SingleNamespace
      supported: true
    - type: MultiNamespace
      supported: true
    - type: AllNamespaces
      supported: false
  customresourcedefinitions:
    owned:
      - name: internalapps.apps.acme.io
        version: v1alpha1
        kind: InternalApp
        displayName: Internal Application
        description: An internally packaged application with day-2 automation.

Setting AllNamespaces: false is a deliberate blast-radius decision: the Operator will only ever be granted namespaced RBAC.

For a durable install, publish a file-based catalog and point a CatalogSource at it:

apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: acme-operators
  namespace: openshift-marketplace
spec:
  sourceType: grpc
  image: quay.io/acme/acme-catalog:v4.16
  displayName: ACME Internal Operators
  publisher: Platform Team
  updateStrategy:
    registryPoll:
      interval: 30m

---
apiVersion: v1
kind: Namespace
metadata:
  name: internalapp-operator
---
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: internalapp-operator
  namespace: internalapp-operator
spec:
  targetNamespaces:
    - demo-internalapp
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: internalapp-operator
  namespace: internalapp-operator
spec:
  channel: stable
  name: internalapp-operator
  source: acme-operators
  sourceNamespace: openshift-marketplace
  installPlanApproval: Manual
  startingCSV: internalapp-operator.v0.1.0

oc apply -f olm/
oc get installplan -n internalapp-operator
oc patch installplan install-xxxxx -n internalapp-operator \
  --type merge -p '{"spec":{"approved":true}}'
oc get csv -n internalapp-operator -w
# internalapp-operator.v0.1.0   InternalApp Operator   0.1.0   Succeeded

For a quick dev loop, operator-sdk run bundle quay.io/acme/internalapp-operator-bundle:0.1.0 -n internalapp-operator creates an ephemeral catalog and subscription in one shot; operator-sdk cleanup internalapp-operator removes it.

Security best practices

Trim the generated RBAC. operator-sdk create api writes a permissive config/rbac/role.yaml with wildcards. Replace it with exactly the verbs your role uses. A wildcard here becomes a cluster-wide privilege escalation path once the CSV is installed, because OLM creates the ServiceAccount and binds it for you.

rules:
  - apiGroups: ["apps.acme.io"]
    resources: ["internalapps", "internalapps/status", "internalapps/finalizers"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
  - apiGroups: [""]
    resources: ["services", "configmaps"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["route.openshift.io"]
    resources: ["routes", "routes/custom-host"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
  - apiGroups: ["batch"]
    resources: ["cronjobs"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

Note that secrets is read-only. An Operator that can create Secrets cluster-wide can mint ServiceAccount tokens indirectly in many designs — avoid it unless you truly need it.

Never put secrets in the CR spec. A spec.databasePassword ends up in etcd unencrypted-by-default, in oc get -o yaml, in GitOps repositories, and in ansible-runner artifacts on disk. Use a secretRef and read it at reconcile time with no_log: true.

Keep the manager under restricted-v2. The manager Deployment generated by the SDK may still carry legacy fields. Enforce explicitly:

# config/manager/manager.yaml (excerpt)
      securityContext:
        runAsNonRoot: true
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: manager
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: false   # ansible-runner needs /tmp writes
            capabilities:
              drop: ["ALL"]
          env:
            - name: ANSIBLE_GATHERING
              value: explicit
            - name: ANSIBLE_DEBUG_LOGS
              value: "false"
          resources:
            requests:
              cpu: 100m
              memory: 256Mi
            limits:
              memory: 768Mi

ANSIBLE_DEBUG_LOGS=true dumps full task results — including resolved Secret content — into the pod logs. Never leave it on outside a debugging session.

Pin and sign images. Reference the Operator image by digest in the CSV, and gate it with a ClusterImagePolicy or Sigstore verification if your cluster enforces it. A floating tag in a CSV means OLM can silently run different code after a registry push.

Use installPlanApproval: Manual in production. Automatic upgrades of a homegrown Operator mean an untested reconcile loop can roll across every tenant namespace at 2 a.m.

Restrict the Operator namespace with a NetworkPolicy. The manager only needs egress to the API server and your registry; deny the rest.

Validate at the API boundary. CEL validation rules (x-kubernetes-validations) in the CRD are cheaper and safer than defensive fail tasks, because they reject bad input before anything reconciles.

Debugging the reconcile loop

oc logs -n internalapp-operator deploy/internalapp-operator-controller-manager -c manager -f
oc get events -n demo-internalapp --sort-by=.lastTimestamp
oc describe internalapp billing -n demo-internalapp

Failures appear as a Failure condition on the CR with the Ansible task name and message. For deeper inspection, exec into the manager and look at the runner artifacts under /tmp/ansible-operator/runner/<group>/<version>/<kind>/<namespace>/<name>/artifacts/. Remember those artifacts contain task output: treat the pod as sensitive.

Key takeaways

• Helm installs; an Operator reconciles. If your day-2 story involves humans running scripts, you need a control loop, not a chart.
• The Ansible Operator SDK gets you from operator-sdk init to a working reconcile in under an hour. The rest of the day goes into the CRD schema, the finalizer, and the RBAC — which is exactly where it should go.
• Write a real structural schema with validation, printer columns and a status subresource. Your CRD is a public API; preserve-unknown-fields is not an API.
• Set reconcilePeriod deliberately and measure the cost: every tick forks an ansible-runner process.
• Package with OLM. The ClusterServiceVersion gives you install modes, upgrade channels, RBAC provisioning and a UI entry for free, and installModes is your blast-radius control.
• Harden by default: trimmed RBAC, read-only Secret access, secretRef instead of inline credentials, restricted-v2-compatible pod security, manual install plan approval, and debug logging off.

Start with one internal application. Once the pattern is in place, the second Operator takes an afternoon — and your Helm chart graveyard starts shrinking.