The controller-runtime package provides a Cache mechanism to users so that they don need to be aware of Informer's presence. By default, when you Get/List Kubernetes Objects, the Cache will automatically start Informers in the background and cache all Objects with the same GVK (GroupVersionKind).
This (somewhat naïve) behavior of the Cache usually doesn't pose an issue in small Kubernetes Clusters. However, as the cluster scales up, this can lead Controllers to unexpectedly consume a significant amount of memory. To address this, the Cache comes equipped with Options to finely control the behavior of Informers.
However, there is a lack of documentation regarding these Options, making it challenging to understand the intricate behaviors. Therefore, we will first delve into the Cache code and then consider the behavior of the Options. When we refer to "Cache" below, it encompasses the Cache interface, as well as its implementations, namely informerCache, multiNamespaceCache, and delegatingByGVKCache.
Get/List and Cache
Before getting into the details of Cache, let's check the general flow of Get/List Kubernetes Objects using Cache. When using controller-runtime, you typically start by obtaining a Client to interact with the Kubernetes API using the GetClient method.
The actual GetClient method is controllerManager's GetClient method, which internally calls the [cluster's GetClient method](https://github.com/kubernetes-sigs/controller-runtime/blob/4fd4f6e7116453ea4e07a2cdd8ba0d38ad64680f/pkg/cluster/internal.go#L70 This method simply returns the value of the cluster's client field, so we'll now look at what is set in this client field.
What is set in the client field is clientWriter. The clientWriter is initialized using the options.NewClient function, with the client package's New function set as the default. Here, Cache is also initialized using the options.NewCache function, with the cache package's New function set as the default. The initialized Cache is passed as an argument to options.NewClient12.
The cache package's New function then calls the newClient function to initialize the Client. It is clear that the Client's cache field is set with the Cache we initialized earlier (link).
All we need to do now is to look at the client's Get and List methods. After calling the shouldBypassCache method, if the Kubernetes Object is allowed to be cached, the Get/List methods of Cache are called.
Although this was rather extensive, we've now determined that calling Get/List using a client obtained from the GetClient method actually calls the Get/List methods of Cache.
Types of Cache
The following types of Cache implement the Cache interface:
informerCache is the most basic implementation of Cache, and it is directly used as Cache without specifying the Cache Options DefaultNamespaces or ByObject. multiNamespaceCache is used when setting the DefaultNamespaces option, and internally has a map of namespace name to Cache (= informerCache), allowing you to specify the behavior of Informers for particular namespaces.
delegatingByGVKCache is used when setting the ByObject option, and internally a map of GVK to Cache. It allows you to specify the behavior of Informers for particular GVKs. If the Namespaces option of ByObject is set3, then multiNamespaceCache is used as Cache for the corresponding GVK; otherwise, informerCache is used. Let's take a closer look at the behavior of each Cache.
informerCache
Using the informerCache's Get method as an example, let's confirm the flow of dynamically created Informers. The Get method internally calls the getInformerForKind method of informerCache, which in calls the Get method of Informers. If the Informer does not exist, Get method of Informers invokes the addInformerToMap method to create an Informer.
The created Informer is packed into a Cache entry (this Cache is not the Cache interface of the cache package, but rather the Cache struct of the internal package), and is then saved to the tracker field of Informers via the informersByType method4.
The Get method of informerCache eventually obtains the Object through the Get method of CacheReader set in the Reader field of this Cache entry. The Get method of CacheReader retrieves the Object from the Indexer.
multiNamespaceCache
For multiNamespaceCache as well, let's quickly look at its behavior. The multiNamespaceCache uses the newMultiNamespaceCache function to create an informerCache for each namespace and saves them in the namespaceToCache field. The Get method of multiNamespaceCache extracts the corresponding informerCache from the namespaceToCache field based on the namespace of the given Object, and calls the Get method of that informerCache.
delegatingByGVKCache
The delegatingByGVKCache generates a Cache for each GVK based on the ByObject option and saves them in the caches field of delegatingByGVKCache. As mentioned before, the Cache created here will be either an informerCache or a multiNamespaceCache depending on the option. The Get method of delegatingByGVKCache extracts the Cache corresponding to the given Object's GVK and calls the Get method of that Cache.
Builder's For, Watches, and Owns, and Cache
In addition to calling Client's Get/List, you can also register Informers through Builder's For, Watches, and Owns methods. Let's check the behavior of Cache in these cases as well. If we look at the Builder's doWatch method, whether you use For, Watches, or Owns, you ultimately call the Watch method of the Controller, which then uses the Start method of Kind to register the Informer using the GetInformer method of Cache.
Cache Options
Let's take a closer look at the behaviors of Cache Options.
ReaderFailOnMissingInformer
ReaderFailOnMissingInformer is an option that prevents the dynamic start-up of Informers through Get/List actions. By looking at the getInformerForKind method of informerCache, we can see that it will return an error if there is no corresponding Informer for an Object's GVK. You can preemptively start Informers either by calling Cache's GetInformer method through Builder's For, Watches, and Owns methods, or by directly invoking Cache's GetInformer method.
DefaultNamespaces
DefaultNamespaces is an option for using multiNamespaceCache to specify Informer behavior per namespace. To see what options can be specified with DefaultNamespaces, check out the Config. Let's consider a few edge cases when DefaultNamespaces is specified.
Get/List for Cluster-scoped Object
Upon reviewing the Get method of multiNamespaceCache, it is clear that if an Object is cluster-scoped, then it calls the Get of clusterCache. The clusterCache is an informerCache without namespace restriction5. This means that even if DefaultNamespaces is specified, Informers for cluster-scoped Objects can be started using clusterCache, and Get/List operations can be performed.
Get/List for Objects Outside the Specified Namespace
When we revisit the Get method of multiNamespaceCache, the first step is to search namespaceToCache using the namespace of the provided Object. If a Cache is not found, it searches namespaceToCache again using metav1.NamespaceAll (=""). This indicates that for Get/List of Objects outside the specified Namespace, if the empty string namespace is specified in DefaultNamespaces, informerCache is utilized; otherwise, an error is returned.
DefaultTransform
DefaultTransform specifies the default Transform function to pass to Informers. The Transform function allows modifications to be made to an Object before it is processed by the Informer. A common use case includes deleting fields from an Object (such as Metadata) to reduce memory consumption6.
DefaultUnsafeDisableDeepCopy
DefaultUnsafeDisableDeepCopy is an option to conserve memory usage when calling Get/List by allowing the CacheReader to return Kubernetes Objects cached by the Informer without performing a DeepCopy. As stated in the comment, enabling this option requires you to perform a DeepCopy when making changes to the Object. This option is unlikely to be enabled unless there are special circumstances, such as listing a large number of Objects in a single reconcile.
ByObject
ByObject is an option to use delegatingByGVKCache to specify the behavior of Informers for each GVK. The available options that can be specified with ByObject can be found in the ByObject struct. This option is similar to DefaultNamespaces, but the behavior is different when an Object with a GVK not specified by ByObject is passed to Get/List.
Unlike DefaultNamespaces, which returns an error for most unspecified Namespace Get/List operations, the delegatingByGVKCache's cacheForGVK method returns dbt.defaultCache when the provided GVK is not in dbt.caches.
The defaultCache will be either a multiNamespaceCache or an informerCache depending on other options. Thus, with ByObject, if an Object with an unspecified GVK is provided, an error typically does not return, and an Informer may be dynamically started.
In addition to the above, there are other Cache Options such as SyncPeriod, DefaultLabelSelector, DefaultFieldSelector, as well as the Config from DefaultNamespaces and the ByObject option. However, we won't delve into these here as their behaviors can be easily understood from the discussion so far.
-
The initialized Cache is set to the cluster's cache field, from where it is started through the cluster's Start method. The cluster itself is added as a Runnable to the controllerManager's runnables and started. The concrete implementation of the Start method held by the Cache interface is the Informers' Start method for the informerCache, whereas multiNamespaceCache and delegatingByGVKCache each have their own distinct Start methods. ↩
-
The Cache itself also implements the Start method through the Informers interface, so it can be directly added to the controllerManager as a Runnable. By creating a Client using this Cache, you can utilize a Client with Cache Options that are completely different from those of the controllerManager. ↩
-
The multiNamespaceCache is also employed when the DefaultNamespaces option is specified. As noted in the comment, to avoid using the DefaultNamespaces option, one must specify an empty list in the Namespaces option of ByObject. ↩
-
For more details about client-go's Informer and Indexer, refer to other documents like client-go under the hood. ↩
-
The globalConfig, when the multiNamespaceCache is generated from DefaultNamespaces, is produced using the optionDefaultsToConfig function, but when the multiNamespaceCache is created through the delegatingByGVKCache originating from ByObject, it results in nil. ↩
-
Another method to limit memory usage is by using PartialObjectMetadata. By utilizing PartialObjectMetadata, only the metadata of an Object is cached, which can help to reduce the amount of memory used. ↩
Top comments (0)