Even deployed together there's a window where old workers are still draining jobs enqueued by the new web version, or the reverse, so the payload format question doesn't fully disappear. Versioning the job payload and keeping workers able to read the previous version for one release covers both setups. Splitting deploys then becomes a scaling and blast radius question rather than a correctness one. Where do you land on a job enqueued by the new version that gets picked up after a rollback?
Hosted cron job service. Schedule HTTP requests on a cron, after a delay or right away. Every run logged, failed jobs retried, alerts by email or Slack.
Agreed, the mixed-version window exists either way, so payload versioning isn't really optional. On a rollback I'd want old workers to either read the new payload (if the change was additive) or park those jobs in a holding queue until a fixed version rolls forward, rather than failing them and burning retries. That's a big part of why I like keeping payload changes additive for a release or two.
Parking jobs in a holding queue on rollback is a nice detail, since burned retries are the part that does real damage. How do you decide when a parked job is safe to release: a version tag on the payload, or a manual flip once the fix is out?
Hosted cron job service. Schedule HTTP requests on a cron, after a delay or right away. Every run logged, failed jobs retried, alerts by email or Slack.
A version tag on the payload, mostly. Each parked job records the minimum worker version that can handle it, and a release that meets that bar drains the queue on its own, so nobody has to remember the manual flip. I'd still keep a manual release as a fallback for the odd case where the fix ships under a different shape.
A minimum worker version on the payload is neat, since the release that meets the bar drains the queue by itself and nobody has to remember the flip.
What happens on a rollback: does a job that a newer worker already picked up and half processed get parked again, or do you only park before first pickup?
Hosted cron job service. Schedule HTTP requests on a cron, after a delay or right away. Every run logged, failed jobs retried, alerts by email or Slack.
Only before first pickup. Once a newer worker has started on a job, an older worker shouldn't touch it, so a half-finished job stays with its claim until the lease runs out and then goes back into the parked queue for the next compatible version. That only works if each step inside the job is safe to repeat, which is the real cost of the whole approach.
SRE and founder of Runtimez, building tools to detect Kubernetes deployment risks before production. Writing about Kubernetes, reliability, platform engineering, and observability.
On Kubernetes, the "finish or safely abandon within a timeout" part already has a number, and plenty of worker Deployments never change it. terminationGracePeriodSeconds defaults to 30 seconds. After SIGTERM the kubelet waits that long and then kills whatever is still running. So a 20-minute job on a default worker gets cut off half a minute into every rollout.
Quick way to see it across a cluster:
kubectl get deploy -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,GRACE:.spec.template.spec.terminationGracePeriodSeconds
A worker showing 30 is running on the default. A preStop hook runs inside that same window, so it doesn't buy extra time. And raising the number only helps if the worker stops pulling new jobs as soon as SIGTERM arrives, which ties straight back to your point about graceful restarts.
Hosted cron job service. Schedule HTTP requests on a cron, after a delay or right away. Every run logged, failed jobs retried, alerts by email or Slack.
Good catch, that 30-second default is exactly the kind of number nobody looks at until a long job dies halfway through a rollout. Handy one-liner too. For anything that can run longer than a grace period you'd realistically set, I'd rather have the job checkpoint and resume than keep raising the timeout.
함께 배포해도 큐 안에는 서로 다른 버전의 작업이 남는다는 지적이 중요하네요. 장기 예약에서 ID만 전달하는 안도, 실행 시점의 최신 값을 읽어야 하는 항목과 예약 당시의 의도를 유지해야 하는 항목을 나눠보면 좋겠습니다. 배포 토폴로지와 별도로 이 정책을 테스트해두면 “형식은 호환되지만 다른 내용으로 실행된 작업”도 점검할 수 있겠어요.
Hosted cron job service. Schedule HTTP requests on a cron, after a delay or right away. Every run logged, failed jobs retried, alerts by email or Slack.
Even deployed together there's a window where old workers are still draining jobs enqueued by the new web version, or the reverse, so the payload format question doesn't fully disappear. Versioning the job payload and keeping workers able to read the previous version for one release covers both setups. Splitting deploys then becomes a scaling and blast radius question rather than a correctness one. Where do you land on a job enqueued by the new version that gets picked up after a rollback?
Agreed, the mixed-version window exists either way, so payload versioning isn't really optional. On a rollback I'd want old workers to either read the new payload (if the change was additive) or park those jobs in a holding queue until a fixed version rolls forward, rather than failing them and burning retries. That's a big part of why I like keeping payload changes additive for a release or two.
Parking jobs in a holding queue on rollback is a nice detail, since burned retries are the part that does real damage. How do you decide when a parked job is safe to release: a version tag on the payload, or a manual flip once the fix is out?
A version tag on the payload, mostly. Each parked job records the minimum worker version that can handle it, and a release that meets that bar drains the queue on its own, so nobody has to remember the manual flip. I'd still keep a manual release as a fallback for the odd case where the fix ships under a different shape.
A minimum worker version on the payload is neat, since the release that meets the bar drains the queue by itself and nobody has to remember the flip.
What happens on a rollback: does a job that a newer worker already picked up and half processed get parked again, or do you only park before first pickup?
Only before first pickup. Once a newer worker has started on a job, an older worker shouldn't touch it, so a half-finished job stays with its claim until the lease runs out and then goes back into the parked queue for the next compatible version. That only works if each step inside the job is safe to repeat, which is the real cost of the whole approach.
On Kubernetes, the "finish or safely abandon within a timeout" part already has a number, and plenty of worker Deployments never change it. terminationGracePeriodSeconds defaults to 30 seconds. After SIGTERM the kubelet waits that long and then kills whatever is still running. So a 20-minute job on a default worker gets cut off half a minute into every rollout.
Quick way to see it across a cluster:
A worker showing 30 is running on the default. A preStop hook runs inside that same window, so it doesn't buy extra time. And raising the number only helps if the worker stops pulling new jobs as soon as SIGTERM arrives, which ties straight back to your point about graceful restarts.
Good catch, that 30-second default is exactly the kind of number nobody looks at until a long job dies halfway through a rollout. Handy one-liner too. For anything that can run longer than a grace period you'd realistically set, I'd rather have the job checkpoint and resume than keep raising the timeout.
함께 배포해도 큐 안에는 서로 다른 버전의 작업이 남는다는 지적이 중요하네요. 장기 예약에서 ID만 전달하는 안도, 실행 시점의 최신 값을 읽어야 하는 항목과 예약 당시의 의도를 유지해야 하는 항목을 나눠보면 좋겠습니다. 배포 토폴로지와 별도로 이 정책을 테스트해두면 “형식은 호환되지만 다른 내용으로 실행된 작업”도 점검할 수 있겠어요.
좋은 포인트예요. 실행 시점의 최신 값을 읽어야 하는 필드와 예약 당시의 의도를 그대로 지켜야 하는 필드를 나누는 건 페이로드 설계에서 꼭 필요한 구분 같아요. 예를 들어 이메일 주소는 최신 값을 읽고, 금액이나 할인율은 예약할 때 값을 고정하는 식으로요.