Short answer: Stop admitting videos when a batch is cancelled, but let each already-leased video stop only at a boundary where its moderation result can be discarded or committed as one unit. For a B2B SaaS media library, that two-level design is the safer choice whenever search tags must never outrun moderation coverage.
One cancellation flag cannot express both decisions. A batch flag answers whether the scheduler may start more work. A video token answers whether a worker may publish the result of work it already owns. Mixing them creates the awkward cases: a cancelled batch keeps consuming inference capacity, or a half-processed video becomes searchable with tags that imply it passed checks it never completed.
The operational recommendation is strict: stop new leases immediately, check cancellation between bounded units of video work, and make the final metadata write conditional on the same generation of the job. Fast aborts matter. Correct visibility matters more.
How Should You Compare Batch Cancellation With Video Cancellation Boundaries?
Compare them by the state each boundary protects, not by how quickly an API reports cancelled. Batch cancellation protects queue admission and shared capacity. Video cancellation protects one asset's decode, inference, moderation, and metadata transaction. They overlap in time, but they have different owners and different proof obligations.
| Decision | Batch cancellation boundary | Video cancellation boundary |
|---|---|---|
| Primary owner | Scheduler or coordinator | Worker processing one asset |
| Stops | New leases and queued dispatch | Further work on an active video |
| Safe checkpoint | Before lease issuance or renewal | After a bounded segment and before publication |
| Durable proof | Batch generation and cancellation timestamp | Asset attempt, checkpoint, and commit status |
| Main failure if omitted | Cancelled work continues to consume fleet capacity | Partial or stale tags become visible |
| Recovery unit | Remaining queue | One video attempt |
This distinction sets the SLO language. Batch cancellation latency measures how long new work continues to enter the fleet after the control-plane decision. Video cancellation latency measures how long an active worker takes to reach a checkpoint and acknowledge the request. Neither metric proves moderation integrity, so add a third invariant: no asset becomes searchable unless the required moderation policy version has a complete result for the same processing attempt.
Suppose a customer cancels a 40,000-video import after discovering the wrong library was selected. The scheduler can reject the next lease at once, yet hundreds of workers may already hold videos. Killing every process at that instant looks decisive, but it can strand multipart uploads, leave ambiguous checkpoints, and force the recovery path to guess whether tags were committed. Letting every worker finish the entire video has the opposite defect: cancellation latency now tracks the longest asset, and scarce decode or model capacity remains assigned to a batch the customer no longer wants. The practical middle is a bounded checkpoint such as a segment, frame window, or inference request, followed by a generation check before any durable write. The exact window is a capacity-planning choice: smaller units reduce abort latency while increasing coordination and checkpoint traffic; larger units do the reverse.
That's the core comparison.
Put the Boundary Before Visibility
Auto-tagging is a derived-data pipeline. A useful run may decode a video, sample frames, classify content, evaluate moderation policy, aggregate candidate tags, and update a search document. Those stages should not share one vague processing bit. Record an attempt identifier and policy version, write intermediate output under that attempt, and expose tags only through a final conditional commit.
The conditional part is important. Cancellation can arrive after the worker's last poll but before its write. A worker that merely checks a boolean and then performs an unconditional update has a time-of-check/time-of-use race. The commit should succeed only when the video's active attempt and the batch generation still match, and when the moderation decision is complete. If the condition fails, intermediate output remains invisible and can be expired later by ordinary retention policy.
Do not equate cancellation with deletion. Cancellation stops future computation; deletion is a separate data-lifecycle action with different authorization, retention, and audit requirements. Keeping those commands distinct also makes rollback legible: an operator can resume or supersede a cancelled processing attempt without reconstructing source media.
A small worker loop can make the contract explicit. The interfaces below are generic, and the implementation deliberately treats context cancellation as a request to stop at the next safe checkpoint rather than as permission to publish partial results.
package tagging
import (
"context"
"errors"
)
var ErrCancelled = errors.New("processing cancelled")
type Lease struct {
BatchID string
VideoID string
AttemptID string
Generation int64
}
type Segment struct {
Number int
}
type Result struct {
Segment int
Tags []string
ModerationComplete bool
}
type Store interface {
StillCurrent(context.Context, Lease) (bool, error)
SaveCheckpoint(context.Context, Lease, Result) error
CommitVisibleTags(context.Context, Lease) error
}
type Classifier interface {
Classify(context.Context, Lease, Segment) (Result, error)
}
func ProcessVideo(ctx context.Context, lease Lease, segments []Segment, store Store, model Classifier) error {
for _, segment := range segments {
if err := ctx.Err(); err != nil {
return ErrCancelled
}
current, err := store.StillCurrent(ctx, lease)
if err != nil {
return err
}
if !current {
return ErrCancelled
}
result, err := model.Classify(ctx, lease, segment)
if err != nil {
return err
}
if err := store.SaveCheckpoint(ctx, lease, result); err != nil {
return err
}
}
current, err := store.StillCurrent(ctx, lease)
if err != nil {
return err
}
if !current {
return ErrCancelled
}
return store.CommitVisibleTags(ctx, lease)
}
Go's context package defines cancellation propagation across API boundaries, but the context alone does not choose a durable media boundary. That is application policy. I wouldn't pass the request context from an HTTP cancellation call directly into unrelated workers; persist the batch decision, propagate it through the scheduler, and derive worker cancellation from durable state so a coordinator restart does not erase intent.
Size Checkpoints From an Abort Budget
Start with an explicit abort-latency objective. If active video work must acknowledge cancellation within 30 seconds, a checkpoint that can occupy a worker for several minutes cannot meet the objective, regardless of how often the worker polls before and after it. Bound the unit's wall time, include queue and state-propagation delay, and reserve margin for a slow dependency. These are design inputs, not benchmark claims; measure them on representative codecs, durations, and resolutions before setting a production SLO.
Capacity planning follows from the same boundary. Track active leases by batch, checkpoint writes per second, bytes of intermediate output, and useful work discarded after cancellation. A very fine checkpoint may look responsive in a demo while moving the bottleneck into the metadata store. A coarse checkpoint reduces coordination overhead but expands the amount of work at risk. I'm not sure there is a universal sweet spot because scene density, codec seek behavior, and model request size vary; a replay using the library's actual media distribution resolves that uncertainty better than a generic default.
Moderation coverage constrains the optimization. If policy requires audio, sampled frames, and generated text to pass before tags are searchable, then completion means all three signals are present for the same policy version. A cancelled attempt with two signals is incomplete, even when its candidate tags look plausible. Keep it out of the search index.
Short and blunt.
The buy-versus-build decision should focus on control, not fashion:
| Approach | Prefer it when | The catch is |
|---|---|---|
| Managed media pipeline | Standard stages fit, cancellation state is durable, and conditional publication is exposed | Not suitable when the service cannot represent your moderation-complete invariant or bound active-work cancellation |
| Self-hosted workers on a general queue | Policy changes often and the team needs precise lease, generation, and checkpoint control | You own worker draining, retry storms, storage cleanup, capacity forecasts, and the on-call load |
| Hybrid control plane and replaceable workers | The scheduler contract is stable but codecs or classifiers need independent evolution | More interfaces and conformance tests are required, especially around stale attempts |
Stick with a managed path when its documented cancellation and commit boundaries match the invariant and the reduced operational surface matters more than custom scheduling. Build the worker path when moderation evidence, cancellation latency, or data placement cannot be expressed otherwise. The limitation is real: custom semantics buy control by putting queue behavior and recovery correctness on your pager.
Verify Cancellation as a Race, Not a Happy Path
A unit test that cancels before processing begins proves very little. Exercise cancellation at every state transition: before lease, after lease, during a bounded classification call, after the final checkpoint, and immediately before the visible-tag commit. For each case, assert the durable state and the search view separately. The last case catches stale writers that a cooperative in-memory token cannot prevent.
Use an injected barrier rather than timing sleeps. Pause the worker just before commit, cancel the batch and increment its generation, release the barrier, then verify that the old attempt cannot publish. Repeat with lease expiry and reassignment: attempt B may complete, while a late write from attempt A must fail its condition. This is where an attempt ID earns its keep.
The runbook needs observable signals, not a green dashboard built from process exits. Record cancellation decision time, last lease time, acknowledgement time by active attempt, checkpoint age, and conditional-commit rejections. Alert on an SLO window, not one slow video. Also reconcile durable state: cancelled batches should stop gaining leases, incomplete attempts should have no visible search document, and every visible document should point to complete moderation evidence.
Rollback means disabling new dispatch for the affected processor version, preserving source media and checkpoints, and starting a new generation under the last known-good policy. Do not revive an old generation; that makes late workers current again. If a boundary change increases load on the checkpoint store, widen the segment window or reduce concurrency while keeping the conditional publication rule intact. The integrity rule is the part you do not relax under pressure.
Operational Decision Rule
Choose both cancellation layers for any batch-oriented media tagging system. The batch boundary is the economical stop: it protects admission, queue depth, and shared capacity. The per-video boundary is the correctness stop: it bounds active work and prevents an incomplete moderation attempt from becoming searchable. A single global kill is not suitable when work must drain cleanly, and finish-everything cancellation is not suitable when long videos can consume the entire abort budget.
Ship only after fault-injection tests prove that an old generation cannot commit, dashboards separate admission latency from worker acknowledgement, and the rollback procedure creates a new attempt rather than reopening the cancelled one. That decision rule remains valid as codecs and classifiers change because it is anchored to ownership and visibility, not to a particular vendor.
Top comments (0)