Laravel image jobs need idempotency before they need new APIs

Laravel getting first-party image processing is useful, but it also creates the wrong instinct in a lot of teams. They start thinking the hard part is finally solved because the resize and transform layer is now closer to the framework.

It is not.

If your app handles uploads through queues, the dangerous part is still retries, partial completion, duplicate writes, and stale state. A cleaner API for cropping or encoding does not fix a worker running twice, a timeout after two variants, or a replace flow that leaves old files live behind the CDN.

My recommendation is blunt: treat Laravel image jobs as idempotent workflow code first and image manipulation code second. If you get that order wrong, the pipeline will look fine in demos and slowly become operational debt in production.

The Real Bug Is Usually Workflow Drift

Most broken image systems are not broken because the JPEG library failed. They are broken because the surrounding workflow had no durable memory.

A typical naive implementation looks like this:

1. Accept upload
2. Save original with a random name
3. Dispatch a queued job
4. Generate a few variants
5. Update the database at the end

That sounds reasonable until reality arrives.

What happens if the worker dies after writing the thumbnail but before writing the hero image? What happens if the queue retries the job automatically? What happens if the user re-uploads a replacement while the first job is still running? What happens if S3 accepts one object write but the DB transaction rolls back? What happens if Horizon runs the same job again after a timeout even though some files are already present?

Those are not weird edge cases. They are normal production behavior once your queue, storage, and HTTP boundary stop behaving like one process.

Laravel's queue docs already push you toward this mental model: jobs may be retried, may time out, and may be processed asynchronously by separate workers. That is exactly why image work must be retry-safe in the first place: https://laravel.com/docs/13.x/queues.

If your image pipeline assumes every job runs exactly once from start to finish, you do not have a resilient pipeline. You have a best-case script.

The Failure Modes Worth Designing For

The recurring failure modes are boring, which is exactly why teams under-design them:

• duplicate original files for the same logical image
• half-generated variants with no reliable way to resume
• a DB row marked processed while one or more files are missing
• an older job overwriting paths after a newer upload already replaced the image
• retries generating the same derivatives repeatedly and inflating storage cost
• cleanup code deleting active files because it cannot distinguish versions

The common thread is simple: the system cannot tell what work is intended, what work is complete, and what work has been superseded.

That is an idempotency problem, not an imaging problem.

Start With A Durable Image Record, Not With Storage::put()

The first design move should be a database record that represents the lifecycle of a logical image. Do that before you worry about which driver or image library you prefer.

You want one place that can answer these questions at any moment:

• what is the canonical original for this image?
• which processing version is current?
• what status is the workflow in right now?
• which variants are already durable?
• was this image replaced, failed, or completed?
• do two uploads contain the same original bytes?

A minimal schema can carry most of that:

Schema::create('media_images', function (Blueprint $table) {
    $table->uuid('id')->primary();
    $table->morphs('imageable');

    $table->string('disk', 32);
    $table->string('status', 24)->default('pending')->index();
    $table->unsignedInteger('version')->default(1);

    $table->string('original_path');
    $table->string('original_extension', 16);
    $table->string('original_checksum', 64)->index();
    $table->unsignedBigInteger('original_bytes');
    $table->string('mime_type', 100);

    $table->json('variants')->nullable();
    $table->json('meta')->nullable();

    $table->timestamp('processing_started_at')->nullable();
    $table->timestamp('processed_at')->nullable();
    $table->timestamp('failed_at')->nullable();
    $table->text('failure_reason')->nullable();
    $table->timestamp('superseded_at')->nullable();

    $table->timestamps();
});

That table is doing real work.

The status field gives the queue something explicit to reconcile against. The version field gives replacements a clean upgrade path. The original_checksum gives you content identity instead of guessing from filenames. The variants JSON gives you durable progress tracking without having to scan storage on every decision.

Status Columns Matter More Than People Admit

A lot of teams try to infer state from file existence. If thumb.webp exists, they treat the image as processed. That is fragile.

File existence alone cannot tell you whether:

• the current version is the one on disk
• all required variants are present
• the database metadata matches the actual artifacts
• the image was replaced and the old path is now stale
• the workflow failed after partial success

Use explicit workflow states instead. Keep them small and operationally meaningful:

• pending
• processing
• processed
• failed
• superseded

That is enough for most Laravel apps. Resist the urge to invent a state machine with 15 statuses unless you genuinely need it.

Deterministic Paths Are The Backbone Of Retry Safety

Randomized filenames are fine for raw uploads at the edge. They are bad as the long-term identity of a processing workflow.

If every retry produces a different target path, you cannot safely answer basic questions like "did we already write this variant?" or "which files belong to version 2 of this image?" You may avoid collisions, but you create a bigger mess: hidden duplicates, expensive cleanup, and no stable namespace.

A better rule is this: each logical image should own a deterministic directory, and each version should own deterministic child paths.

For example:

final class ImagePaths
{
    public static function original(string $imageId, int $version, string $extension): string
    {
        return "images/{$imageId}/v{$version}/original.{$extension}";
    }

    public static function variant(string $imageId, int $version, string $name, string $extension = 'webp'): string
    {
        return "images/{$imageId}/v{$version}/{$name}.{$extension}";
    }
}

That path scheme is boring, which is exactly what you want.

It gives you stable targets across retries, clean version isolation across replacements, and predictable cleanup scopes. It also plays nicely with Laravel's filesystem abstraction because the framework can switch disks without changing your naming contract: https://laravel.com/docs/13.x/filesystem.

Why Versioned Paths Beat Overwrites

Many teams try to overwrite original.jpg and regenerate the same variant paths in place when an image is replaced. That looks simpler until concurrency shows up.

Imagine this timeline:

1. User uploads image A
2. Job A starts generating variants
3. User replaces it with image B
4. Job B starts
5. Job A finishes late and writes over shared paths

Now you have stale data winning because the path contract allowed two generations to compete for the same filenames.

Versioned paths stop that class of bug. Job A writes only under v1. Job B writes only under v2. The database decides which version is current. Cleanup can remove v1 later, but only after v2 is fully durable.

That is a much cleaner system boundary than trying to make timing guarantees you do not actually control.

Checksums Turn Upload Chaos Into Something You Can Reason About

The checksum is the underrated field in this whole pipeline.

Without it, repeated uploads are guesswork. With it, you can make concrete decisions.

If the user retries the same request, or your frontend double-submits, or an API client replays an upload after a network wobble, the checksum lets you tell whether this is genuinely new content or the same file arriving twice.

That creates practical rules:

• same image record + same checksum: do not regenerate everything
• same image record + different checksum: increment version and reprocess
• different record + same checksum: maybe deduplicate later, but only if the added complexity is worth it

Per-record idempotency is usually enough. Global deduplication across the whole app sounds clever, but it adds reference counting, ownership questions, and cleanup complexity fast. Most teams should earn that complexity only after local idempotency is solid.

Compute The Checksum At Ingress

You want the checksum as early as possible, ideally when the original file is first accepted:

$stream = fopen($uploadedFile->getRealPath(), 'rb');
$checksum = hash_file('sha256', $uploadedFile->getRealPath());
$extension = strtolower($uploadedFile->getClientOriginalExtension() ?: 'bin');

$image = MediaImage::create([
    'id' => (string) Str::uuid(),
    'imageable_type' => $post::class,
    'imageable_id' => $post->getKey(),
    'disk' => 's3',
    'status' => 'pending',
    'version' => 1,
    'original_extension' => $extension,
    'original_checksum' => $checksum,
    'original_bytes' => $uploadedFile->getSize(),
    'mime_type' => $uploadedFile->getMimeType(),
    'original_path' => ImagePaths::original($id, 1, $extension),
]);

Storage::disk('s3')->put($image->original_path, $stream);

This is also where you want to make a judgment call about request-level idempotency. If your API surface can receive the same upload twice from a flaky client, it is often worth pairing the checksum with an application-level idempotency key so you can return the existing image record instead of creating a new one.

The Job Should Reconcile Progress, Not Rebuild Blindly

The core mistake in most image jobs is that they act like a one-shot script. They should act like a reconciler.

A reconciler asks:

• what state should exist?
• what state already exists?
• what is missing?
• what has become obsolete?

That mental model is better than "run all transforms and hope it finishes" because retries become safe by construction.

Here is the shape I prefer:

final class ProcessImageVariants implements ShouldQueue
{
    public int $timeout = 120;

    public function __construct(
        public string $imageId,
        public int $version,
    ) {}

    public function handle(): void
    {
        $image = MediaImage::query()->findOrFail($this->imageId);

        if ($image->version !== $this->version || $image->status === 'superseded') {
            return;
        }

        if ($image->status === 'processed' && $this->allRequiredVariantsExist($image)) {
            return;
        }

        $image->forceFill([
            'status' => 'processing',
            'processing_started_at' => $image->processing_started_at ?? now(),
            'failed_at' => null,
            'failure_reason' => null,
        ])->save();

        $written = $image->variants ?? [];

        foreach ($this->variantDefinitions() as $name => $definition) {
            if ($this->variantAlreadyDurable($image, $written, $name)) {
                continue;
            }

            $binary = app(ImageVariantBuilder::class)->build(
                image: $image,
                width: $definition['width'],
                height: $definition['height'],
                fit: $definition['fit'],
            );

            $path = ImagePaths::variant($image->id, $image->version, $name, 'webp');

            Storage::disk($image->disk)->put($path, $binary);

            $written[$name] = [
                'path' => $path,
                'width' => $definition['width'],
                'height' => $definition['height'],
                'format' => 'webp',
            ];

            $image->forceFill(['variants' => $written])->save();
        }

        $image->forceFill([
            'status' => 'processed',
            'processed_at' => now(),
            'variants' => $written,
        ])->save();
    }
}

There are three important ideas here.

First, the job exits early if it is no longer the current version. That prevents stale work from winning late.

Second, the job persists progress after each successful variant. That means a crash after two variants is annoying, not catastrophic.

Third, completion is earned only after the required set is actually present.

Incremental Persistence Beats Fake Atomicity

You cannot make storage writes and database writes one perfect transaction across systems. Pretending otherwise just hides the truth.

The pragmatic answer is incremental honesty:

1. mark processing
2. write one artifact
3. persist that artifact's metadata
4. continue
5. mark processed only when the full required set exists

That pattern gives retries something real to work with. It also makes operator debugging much easier because the record reflects partial progress instead of collapsing everything into a binary success/failure fantasy.

Concurrency And Replacement Flows Need Explicit Rules

Image systems usually get messy when the same logical image can be edited, replaced, or reprocessed while workers are still active.

Do not solve that with hope. Solve it with rules.

Rule 1: Jobs Must Carry Version Context

Never dispatch a generic "process this image" job without the version it is supposed to process. Pass both image ID and version.

That way, when the worker wakes up, it can immediately verify whether it still owns relevant work. If the DB row has already advanced to a newer version, the old job should do nothing.

Rule 2: Replacements Should Supersede, Not Mutate In Place

When the original changes, do not quietly rewrite the existing artifacts. Increment version, write the new original under the new namespace, reset status to pending, and dispatch a fresh job.

The old version can remain on disk for a short time. That is fine. Temporary duplication is much safer than cross-version corruption.

Rule 3: Cleanup Should Be Separate From Processing

Do not mix destructive cleanup into the happy-path processing job unless you absolutely need to. A failed cleanup should not poison image generation.

Use a later job that removes obsolete versions only after the new one is fully processed and any publish/CDN rules are satisfied.

That separation keeps the hot path smaller and prevents one category of failure from cascading into another.

Storage Writes Need To Be Retry-Safe Too

A surprising number of pipelines are "idempotent" in theory but still perform wasteful or dangerous writes in practice.

The simplest rule is this: before writing a variant, know the exact target path and know whether an existing durable file already satisfies the contract.

That often means checking both:

• the DB metadata says the variant exists
• the storage disk confirms the path exists

If both are true, skip the write. If the DB says it exists but storage disagrees, repair it. If storage says it exists but the DB row never recorded it, verify it belongs to the current version before trusting it.

That sounds like extra bookkeeping because it is. That bookkeeping is what turns retries into recovery instead of duplication.

S3 And Local Disk Fail Differently

Laravel makes local disk and S3 look similar from the app code, which is a good abstraction. But operationally, the tradeoffs are still different.

Local disk gives you lower latency and simple existence checks but ties durability to host topology. S3 gives you better durability and easier horizontal scale but makes object writes, verification, and cleanup more distributed in feel.

The idempotency rules do not change across disks. What changes is how much you trust path existence, how expensive repeated writes are, and how carefully you want to stage cleanup.

That is why designing around deterministic paths and explicit workflow state matters so much. It survives infrastructure changes better than path conventions invented ad hoc in controllers.

What I Would Ship In A Real Laravel App

If I were implementing image jobs in a production Laravel codebase today, the baseline would be:

• one DB row per logical image
• explicit status and version columns
• checksum captured at upload time
• deterministic original and variant paths
• queued jobs that receive image ID plus version
• per-variant progress persistence
• completion only after required variants are verified
• replacement flows that create new versions instead of overwriting paths
• cleanup handled asynchronously after successful supersession

That is not over-engineering. That is the minimum structure that keeps image work understandable once retries and replacements show up.

New first-party image APIs are a good addition. They should reduce glue code, make transforms more Laravel-native, and clean up the manipulation layer. Use them.

Just do not confuse a better transform API with a reliable image pipeline.

Practical Decision Rule

If an image job can run twice, then it must be idempotent. That means stable identity, deterministic paths, explicit workflow state, and retry-safe writes. Everything else is secondary.

In other words: the best Laravel image pipeline is not the one with the nicest resize syntax. It is the one that can crash halfway through, run again, and still converge on the same correct result.

---

Read the full post on QCode: https://qcode.in/laravel-image-jobs-need-idempotency-more-than-new-apis/