controller-runtime & kubebuilder

• code
• refs

Resources

Client

RW client for k8s objects

Cache

RO client from a local cache

Manager

1. Creating a Controller
2. Provides shared dependencies (clients, caches, schemes ...)
3. **Controllers should be started by calling Manager.Start()

&controllerManager{
        stopProcedureEngaged:          ptr.To(int64(0)),
        cluster:                       cluster,
        runnables:                     runnables,
        errChan:                       errChan,
        recorderProvider:              recorderProvider,
        resourceLock:                  resourceLock,
        metricsServer:                 metricsServer,
        controllerConfig:              options.Controller,
        logger:                        options.Logger,
        elected:                       make(chan struct{}),
        webhookServer:                 options.WebhookServer,
        leaderElectionID:              options.LeaderElectionID,
        leaseDuration:                 *options.LeaseDuration,
        renewDeadline:                 *options.RenewDeadline,
        retryPeriod:                   *options.RetryPeriod,
        healthProbeListener:           healthProbeListener,
        readinessEndpointName:         options.ReadinessEndpointName,
        livenessEndpointName:          options.LivenessEndpointName,
        pprofListener:                 pprofListener,
        gracefulShutdownTimeout:       *options.GracefulShutdownTimeout,
        internalProceduresStop:        make(chan struct{}),
        leaderElectionStopped:         make(chan struct{}),
        leaderElectionReleaseOnCancel: options.LeaderElectionReleaseOnCancel,
    }

• cluster: Cluster provides various methods to interact with a cluster.

type Cluster interface {
    // GetHTTPClient returns an HTTP client that can be used to talk to the apiserver
    GetHTTPClient() *http.Client

    // GetConfig returns an initialized Config
    GetConfig() *rest.Config

    // GetCache returns a cache.Cache
    GetCache() cache.Cache

    // GetScheme returns an initialized Scheme
    GetScheme() *runtime.Scheme

    // GetClient returns a client configured with the Config. This client may
    // not be a fully "direct" client -- it may read from a cache, for
    // instance.  See Options.NewClient for more information on how the default
    // implementation works.
    GetClient() client.Client

    // GetFieldIndexer returns a client.FieldIndexer configured with the client
    GetFieldIndexer() client.FieldIndexer

    // GetEventRecorderFor returns a new EventRecorder for the provided name
    GetEventRecorderFor(name string) record.EventRecorder

    // GetRESTMapper returns a RESTMapper
    GetRESTMapper() meta.RESTMapper

    // GetAPIReader returns a reader that will be configured to use the API server directly.
    // This should be used sparingly and only when the cached client does not fit your
    // use case.
    GetAPIReader() client.Reader

    // Start starts the cluster
    Start(ctx context.Context) error
}

type cluster struct {
    // config is the rest.config used to talk to the apiserver.  Required.
    config *rest.Config

    httpClient *http.Client
    scheme     *runtime.Scheme
    cache      cache.Cache
    client     client.Client

    // apiReader is the reader that will make requests to the api server and not the cache.
    apiReader client.Reader

    // fieldIndexes knows how to add field indexes over the Cache used by this controller,
    // which can later be consumed via field selectors from the injected client.
    fieldIndexes client.FieldIndexer

    // recorderProvider is used to generate event recorders that will be injected into Controllers
    // (and EventHandlers, Sources and Predicates).
    recorderProvider *intrec.Provider

    // mapper is used to map resources to kind, and map kind and version.
    mapper meta.RESTMapper

    // Logger is the logger that should be used by this manager.
    // If none is set, it defaults to log.Log global logger.
    logger logr.Logger
}

Manager.Start source code

Controller

Reconcile loop

Webhook

Admission Webhooks are a mechanism for extending kubernetes APIs.

Target event type (object Create/Update/Delete)
--> API server
--> Handler
--> Send event to webhook

There are two types of admission webhook

1. Mutating webhook
   • mutate a core API object
   • CRD instance
2. Validating webhook
   • validate if an object meet certain requirements.

Reconciler

Ensure that the state of the system matches what is specified in the obj at the time Reconciler is called.

• Reconciler is a function provided to a Controller.
• It may be called at anytime with the Name and Namespace of an object.
• It contains all of the business logic of a Controller.

Source

It provides a stream of events for k8s objects through the Watch API.

• resource.Source is an argument to Controller.Watch(Source, EventHandler, Predicate)
• Events typically come from watching k8s APIs.
• Users SHOULD only use the provided Source implementations instead of implementing their own for nearly all cases.

EventHandler

It enqueues reconcile.Requests in response to events.

• handler.EventHandler is an arg to Controller.Watch

• Users SHOULD only use the provided EventHandler implementations instead of implementing their own for almost all cases.

Predicate

Filter for CRD Events

func (p TypedFuncs[object]) Create(e event.TypedCreateEvent[object]) bool {
    if p.CreateFunc != nil {
        return p.CreateFunc(e)
    }
    return true
}

• If true? Enqueue

Wrap up

Controller Writing Tips

• Reconciler Runtime Complexity: It is better to write Controllers to perform an O(1) reconcile N times

If you need to update all Services in response to a Node being added - reconcile Services but Watch Nodes (transformed to Service object name / Namespaces) instead of Reconciling Nodes and updating Services

• Event Multiplexing: reconcile.Requests for the same Name / Namespace are batched and deduplicated when they are enqueued. This allows Controllers to gracefully handle a high volume of events for a single object. Multiplexing multiple event Sources to a single object Type will batch requests across events for different object types.

Pod events for a ReplicaSet are transformed to a ReplicaSet Name / Namespace, so the ReplicaSet will be Reconciled only 1 time for multiple events from multiple Pods.