controller-runtime: What Happens When You Do Partial Server-Side Apply?

In this post, I'll explore what happens when you repeatedly apply partial manifests using Kubernetes Server-Side Apply (SSA). Specifically, I'll investigate through experiments what occurs when you omit fields that were previously managed by a field manager in subsequent applies.

While developing controllers, I found myself wondering: "What happens to fields I don't include when applying partial manifests with SSA?" If only the submitted portions are merged as a diff, we could simplify our code by sending only the fields we want to manage. On the other hand, if we need to send all managed fields every time, we need to be conscious of this when coding, or we might accidentally create bugs that unintentionally delete fields.

Developing controllers with incorrect assumptions could lead to bugs that unintentionally destroy resources. While this is about understanding the basic specifications of SSA, I felt it was essential to have a clear understanding, which is why I'm writing this post.

The experimental code referenced in this article is available in the following repository:

https://github.com/suinplayground/controller-runtime/tree/main/02-server-side-apply-partials

What You'll Learn

• The behavior when applying partial manifests with Server-Side Apply
• Field ownership and the role of field managers
• The deletion mechanism that occurs when managed fields are omitted
• Collaborative management by multiple field managers

Purpose of the Experiment

The goal of this experiment is to understand what happens when a field manager omits a field in a subsequent apply that was included in a previously applied manifest.

In real controller development, you might encounter scenarios like: "Initially I was managing the breed field, but later I only want to update the color field." What happens to the breed field in this case?

Experiment 1: What Happens When You Omit Fields Managed by the Same Manager?

In this experiment, I'll use a custom resource called Cat. I'll specify cat-owner as the "field manager" that manages field ownership.

Step 1: First Apply (breed field only)

First, I'll apply a partial manifest containing only spec.breed to the Cat resource my-cat.

cat := applycatv1.Cat("my-cat", "default").
    WithSpec(applycatv1.CatSpec().
        WithBreed("Maine Coon"))

err := cl.Apply(ctx, cat, client.FieldOwner("cat-owner"), client.ForceOwnership)

After this operation, the resource state looks like this. You can see that cat-owner managing spec.breed is recorded in managedFields.

"spec": {
  "breed": "Maine Coon"
},
"managedFields": [
  {
    "manager": "cat-owner",
    ...
    "fieldsV1": {
      "f:spec": {
        "f:breed": {}
      }
    }
  }
]

Step 2: Second Apply (color field only, breed omitted)

Next, I'll apply a manifest containing only spec.color to the same my-cat resource, using the same cat-owner. The crucial point is that this manifest doesn't include spec.breed.

cat := applycatv1.Cat("my-cat", "default").
    WithSpec(applycatv1.CatSpec().
        WithColor("Black")) // no breed

err := cl.Apply(ctx, cat, client.FieldOwner("cat-owner"), client.ForceOwnership)

Observation

After the second Apply, while spec.color was added to the resource, the spec.breed field completely disappeared.

"spec": {
  "color": "Black"
},
"managedFields": [
  {
    "manager": "cat-owner",
    ...
    "fieldsV1": {
      "f:spec": {
        "f:color": {}
      }
    }
  }
]

Looking at managedFields, we can see that cat-owner's managed target has been updated from spec.breed to spec.color.

Why Was the Field Deleted?

This behavior relates to Server-Side Apply's "declarative ownership management." The Kubernetes official documentation explains exactly this case:

If you remove a field from a manifest and apply that manifest, Server-Side Apply checks if there are any other field managers that also own the field. If the field is not owned by any other field managers, it is either deleted from the live object or reset to its default value, if it has one. The same rule applies to associative list or map items.

-- Field management - Kubernetes

This explanation perfectly describes our experimental results:

1. In the first Apply, cat-owner became the sole owner of the spec.breed field
2. In the second Apply, cat-owner didn't include spec.breed in the manifest. This is interpreted as "I no longer want to manage spec.breed"
3. Since there were no other owners of spec.breed, Server-Side Apply deleted this field from the live object

This is a cleanup feature that prevents unmanaged fields from persisting indefinitely. To avoid unintentionally deleting fields, you need to understand that you must include all fields you want to manage in every Apply request.

Supplementary Experiment: What Happens When Another Field Manager Appears?

So what happens when another field manager enters the picture? Let's look at the ownership aspect of SSA.

Step 3: Third Apply by a Different Manager

From the previous state (where only spec.color exists), a new field manager called age-controller applies a manifest containing only spec.age.

Observation

After applying, the resource has both spec.color and spec.age. The spec.color managed by cat-owner remains unaffected.

"spec": {
  "color": "Black",
  "age": 3
},
"managedFields": [
  {
    "manager": "age-controller",
    ...
    "fieldsV1": {
      "f:spec": {
        "f:age": {}
      }
    }
  },
  {
    "manager": "cat-owner",
    ...
    "fieldsV1": {
      "f:spec": {
        "f:color": {}
      }
    }
  }
]

managedFields records two entries for age-controller and cat-owner, clearly tracking which fields each owns.

age-controller isn't omitting any fields it previously owned (none existed in this case), so it simply claimed ownership of spec.age. It didn't interfere with fields owned by cat-owner.

Summary

From this experiment, we can understand two key aspects of using Server-Side Apply:

1. Manifest Completeness = Include All Fields: When a field manager omits a field it was managing from the next Apply manifest, that field will be deleted (if no other owners exist). Therefore, always include fields under management in your manifest to maintain manifest completeness.

2. Completeness is Per-Owner: Multiple field managers can safely manage different fields of the same resource without interfering with each other, so you don't need to include fields managed by others in your manifest.

Server-Side Apply is a crucial feature in Kubernetes environments where multiple controllers need to collaboratively manage a single resource. The phenomenon of fields "disappearing" isn't a bug—it's a powerful "feature" that makes declarative configuration management more robust. Understanding this behavior enables safer resource management.

Thank you for reading to the end!