In June we released version 2.0 of our worker autoscaler, and for the next nine nights the order consumers of four teams scaled to zero. The first orders each morning waited up to four minutes while a pod started.
The autoscaler is ours, an operator that watches queue depth and sets replica counts for anything described by a ScaledWorker resource. Version 1 kept a floor of two replicas built in. Version 2 added scale to zero, through a new field, spec.minReplicas, which defaults to zero when it is absent. The release notes asked every team to set minReplicas to two in their charts before the upgrade, and all of them did. The operator's chart carried the new schema in its crds directory.
Helm 3 installs the CRDs in a chart's crds directory when a release is first installed and never touches them again. Its documentation is direct about it: there is no support for upgrading or deleting CRDs. helm upgrade rolled out the new controller and left the cluster with the version 1 schema. That schema was structural, and the API server prunes any field a structural schema does not declare. So each team's resource arrived with minReplicas set to two, was stored without it, and the new controller read an absent field and applied its default of zero. The API server did warn, once for each resource, unknown field spec.minReplicas, and the warning landed in deploy logs nobody reads when the job is green.
We applied the new CRD by hand that morning and every floor came back within a reconcile. Since then CRDs are deployed by their own pipeline step, with a server side kubectl apply, before any chart that depends on them, and our deploy jobs run kubectl with warnings treated as errors. The operator compares the installed CRD with the fields it knows about at startup and refuses to start if anything is missing, so the mismatch is a failed rollout instead of a quiet one. And version 2.1 changed the default: without the field the floor is two again, and scaling to zero is something a team asks for, not something it gets by omission.
We shipped a new controller with an old contract. Helm upgraded the code, and nothing upgraded what the cluster was willing to store.
– Sergey Shinder
Top comments (0)