At twenty past four on a Thursday a release failed thirty three minutes after it started, at the final stage, because a values file had a resource block indented one level too deep. The templating engine produced a manifest with a key the API server has never heard of, the apply was rejected, and the engineer fixed it in four seconds and started the whole thing again.
Our pipeline ran in the order the delivery happens. Lint and formatting, forty seconds. Unit tests, six minutes. Image build and push, five minutes. Integration tests against ephemeral dependencies, eighteen minutes. Then deploy, which is where the chart is rendered for the first time, because rendering needs the environment's values and the environment is a property of the deploy stage. Every mistake in the deployment description, and every mistake in the six values files that feed it, was therefore discovered at the far end of half an hour of work that had nothing to do with it.
I pulled eight weeks of pipeline history. A hundred and twelve runs had failed in the deploy stage. Sixty one of those were schema or syntax errors in a template or a values file, with a median of thirty four minutes burnt before the failure, and a person attached to most of them. That is a working week of waiting a month, to be told about mistakes that a parser can find instantly.
There is a preflight job now and it is the first thing in the pipeline. It renders the chart for all six environments, validates every resulting manifest against the schema for the API version our clusters actually run, and compares the key sets of the six values files against a reference so that a key added to staging and forgotten in production is a failure rather than a surprise. It takes eleven seconds, and in its first fortnight it caught the same class of mistake three times, each within a minute of the push instead of half an hour after it.
A pipeline is a list of questions, and ours had been ordered by how a deployment flows rather than by how quickly each question can answer. Put everything that can say no in the first minute. Waiting half an hour to learn about a space is a choice somebody made, even if nobody remembers making it.
– Sergey Shinder
Top comments (0)