DEV Community

Cover image for Who Tells My Laravel App That a File Was Uploaded to S3 (When the Browser Leaves Before Delivering the Message)
Hermann D. Schimpf
Hermann D. Schimpf

Posted on Originally published at hds-solutions.net

Who Tells My Laravel App That a File Was Uploaded to S3 (When the Browser Leaves Before Delivering the Message)

Versión en español aquí.


Banner Image


So far in this series, we've laid the groundwork piece by piece. In chapter 3, we made files travel directly from the browser to S3 using pre-signed URLs; and in chapter 4, we configured SQS and a Lambda to process jobs in the background. But if the file never passes through our application, one question remains:

Who tells Laravel that a file has finished uploading so it can enqueue that job?

I have to confess something: until now, the browser was the one doing the notifying in my app. The actual flow has three phases: Laravel creates a signed upload URL and sends it to the browser; the browser sends the file directly to S3; and when it finishes, the browser itself makes a request to the app saying "I've uploaded it" so Laravel can enqueue the job. It works... until someone closes the tab between phases two and three, leaving us with an uploaded photo and a record waiting for confirmation that will never come.

I was asking the browser to tell me what was happening in my own bucket. Like asking the mail carrier whether the letter they delivered had arrived.

On a traditional server, this problem doesn't exist: the upload goes through PHP ($request->file()->store()) and dispatch() happens in the same controller. The application was the mandatory tollbooth for every upload.

In this chapter, we'll close the loop.


1. The Serverless Treasure Map

Let's revisit the map, this time to connect two pieces we already know:

AWS Serverless architecture

Today, a new guest appears in the diagram: S3 Event Converter (Lambda), a Lambda function that acts as a translator between what the bucket says and what the Laravel worker understands.


2. S3 Event Notifications: When the Bucket Has Something to Say

a) The Building Doorman

Amazon S3 is a storage service, but it also knows how to notify us when something happens to its objects 1. Think of the bucket as the building's doorman: they don't go knocking on every door on every floor, but if you leave instructions like "if a package arrives for 4B, call this number", they do it without our application having to keep asking whether anything has arrived.

The configuration lives on the bucket and is defined by two things: which events it should notify us about and where it should send those notifications. The events we're interested in are:

  • s3:ObjectCreated:*: A new object was created. We use * to cover all operations: Put, Post, Copy, and CompleteMultipartUpload (for multipart uploads).
  • s3:ObjectRemoved:*: An object was deleted. Again, * catches all operations: Delete and DeleteMarkerCreated.

There are more events 2, but they aren't relevant to us for now.

And what are the possible destinations? SNS, SQS, Lambda, or EventBridge 2. For our case, we'll use the Lambda destination.

b) We Don't Care About the Whole Bucket

We don't need S3 to notify us about everything that happens in the bucket: we can filter by the object's prefix and/or suffix 3. For example, "notify me only when an object is created under uploads/", and "only if it's a .jpg file":

rules:
    -   prefix: uploads/
    -   suffix: .jpg
Enter fullscreen mode Exit fullscreen mode

Three details to keep in mind:

  • No wildcards: We can't use *, only literal text.
  • Spaces and special characters: a space must be replaced with +, and the other special characters must be URL-encoded (event photos @ 2026/ → event+photos+%40+2026/).
  • No overlapping filters: for the same event type, an object can't match two overlapping notification rules 3.

c) What S3 Tells Us

The event content S3 publishes is JSON that looks like this (version 2.x, trimmed) 4:

{
    "Records": [
        {
            "eventSource": "aws:s3",
            "awsRegion": "us-east-1",
            "eventTime": "2026-11-06T14:32:11.000Z",
            "eventName": "ObjectCreated:Put",
            "s3": {
                "bucket": { "name": "my-app-storage" },
                "object": {
                    "key": "inbox/ProcessImage/9f8d7c6b-2aab-4c12-a11a-00f4c3d2e1f0.jpg",
                    "size": 20481234,
                    "eTag": "b21b84d653bb07b05b1e6b33684dc11b",
                    "sequencer": "0C0F6F405D6ED209E1"
                }
            }
        }
    ]
}
Enter fullscreen mode Exit fullscreen mode

Everything we need is there: the bucket name (s3.bucket.name), the file path (s3.object.key), its size in bytes, and a sequencer for ordering events on the same key.

Important: the filename in s3.object.key is URL-encoded, following the same prefix and suffix rules. If someone uploads me & she photo.jpg, the key will contain me+%26+she+photo.jpg.

d) What S3 Guarantees (and What It Doesn't)

S3 notifies us, but there are three conditions our code must account for 1 5:

  • At-least-once delivery: S3 guarantees the notification arrives, but in rare cases it can arrive twice.
  • Fast, almost always: typically seconds, but it can take a minute or more.
  • No ordering guarantee: two events may arrive out of order relative to when they occurred.

S3 may send the same message more than once; our job shouldn't repeat the disaster. And as we saw in the previous chapter, that means our processing must be idempotent.

And how much does the notification cost? S3 doesn't charge to emit these notifications; the destination that receives them may incur charges.


3. S3 Already Notified Us. Now What?

First, one fact that makes everything that follows possible: with bref/laravel-bridge, a Lambda function's handler can be a PHP class directly, and Bref resolves it from Laravel's container 6. If we look at the HandlerResolver that pulls off the trick (simplified) 7:

private function laravel(): Application
{
    // require bootstrap/app.php
    $application = require LaravelPathFinder::app();

    // Laravel booted and ready
    $kernel = $application->make(Kernel::class);
    $kernel->bootstrap();

    return $application;
}
Enter fullscreen mode Exit fullscreen mode

It boots all of Laravel, then calls $this->laravel()->get($id) with our class name. The result: everything works inside our class: Storage, Log, dispatch(), even dependency injection. It's real Laravel, not a cut-down version.

One design decision before we continue, because it simplifies everything that follows: uploads don't land just anywhere in the bucket; they go into an inbox directory we'll call inbox/, organized by task type:

inbox/ProcessImage/{media_uuid}.jpg    → photo to process
inbox/ImportData/{list_uuid}.csv       → spreadsheet to import
inbox/GeneratePreview/{doc_uuid}.pdf   → document to convert
Enter fullscreen mode Exit fullscreen mode

The first segment after inbox/ defines who should process it (the Job name), and the second segment defines what to process (the filename). Results are always written outside the inbox/ directory (to prevent infinite loops).

inbox/ is the building doorman's inbox: it holds the originals until a job takes them over.

a) Option A: a Lambda that extends S3Handler and processes files directly

The most direct approach: write a class that extends Bref's S3Handler 6 and do the work right there. Read the uploaded file, process it, and you're done:

namespace App\Lambda;

use Bref\Context\Context;
use Bref\Event\S3\S3Event;
use Bref\Event\S3\S3Handler;
use Illuminate\Support\Facades\Storage;
use Intervention\Image\Format;
use Intervention\Image\Laravel\Facades\Image;

final class GenerateThumbnail extends S3Handler
{
    public function handleS3(S3Event $event, Context $context): void
    {
        foreach ($event->getRecords() as $record) {
            // get the original filename,
            // urldecode() because the key is URL-encoded
            $key = urldecode($record->getObject()->getKey());

            // process the image ...
            $image = Image::decodeStream(Storage::readStream($key));
            Storage::put(
                str_replace('inbox/', 'media/thumbnails/', $key),
                $image->coverDown(600, 600)->encodeUsingFormat(Format::WEBP, quality: 75),
            );

            // and delete the original, which has done its job,
            // to keep the inbox clean
            Storage::delete($key);
        }
    }
}
Enter fullscreen mode Exit fullscreen mode

With our GenerateThumbnail class in place, we assign it in serverless.yml as the function that will listen for the s3:ObjectCreated:* event on the bucket we already created with Lift's storage construct 8:

functions:
    laravel:
        handler: Bref\LaravelBridge\Http\OctaneHandler
        runtime: php-85
        # ...

+   thumbnails:
+       handler: App\Lambda\GenerateThumbnail
+       runtime: php-85
+       layers:
+           - ${bref-extra:gd-php-85}
+       timeout: 60
+       events:
+           - s3:
+               bucket: ${construct:storage.bucketName}
+               existing: true
+               event: s3:ObjectCreated:*
+               rules:
+                   -   prefix: inbox/

constructs:
    storage:
        type: storage
        allowAcl: true
        cors: ${construct:website.url}
+       lifecycleRules:
+           -   prefix: inbox/
+               expirationInDays: 7   # orphan cleanup (section 5.d)
Enter fullscreen mode Exit fullscreen mode

Two important details we've added to serverless.yml:

  • functions.thumbnails.events.s3.existing: true: without this flag, Serverless Framework would try to create a new bucket with that name and fail, because the bucket already exists. Enabling the flag points it to the existing bucket Lift already created.

Note: this adds a Lambda custom resource and an extra IAM role to the stack to configure the notification, and only allows one existing bucket per function 9.

  • functions.thumbnails.events.s3.rules.prefix: inbox/: we run the function only when something new lands in the inbox. And as we recommended earlier, results are written outside this directory.

The advantages:

  • One function receives the event and processes the file,
  • Minimal latency between upload and processing,
  • And code anyone can understand.

For trivial tasks such as validating a file, writing a log, or moving it to another prefix, this works beautifully.

The honest drawbacks, and there are a few:

  • Less flexible retries: S3 invokes Lambda asynchronously 10. If the function returns an error, Lambda retries by default up to 2 additional times (with 1- and 2-minute delays) and, if it still fails, the event is discarded 11. There's no #[Tries(3)], #[Backoff], failed(), failed_jobs, or queue:retry. You could configure an onFailure destination 12, but that would add unnecessary complexity (Option B simplifies this).
  • Concurrency is controlled another way: we can limit this Lambda with reservedConcurrency 13, but we don't have SQS's per-queue consumption control. If more events arrive than it can process, they remain pending in Lambda's internal asynchronous queue, subject to its retry and expiration policy 11.
  • More functions: each "task type" is an additional function. If you want to process images and CSV files, that's two functions, two custom resources, and two subscriptions to the same bucket (and remember: overlapping prefixes aren't allowed).
  • Duplicated tuning: additional layers (GD for images, etc.), memorySize, timeout... the same work we already did for the worker in the previous chapter, all over again for each function.
  • Outside the jobs ecosystem: we can't use Queue::fake() in tests, or queue:failed or serverless jobs:failed to access the list of failed jobs. Processing is not a Laravel Job: it's a separate class with its own lifecycle.

Option A is like asking the doorman not only to notify you, but also to develop, crop, and frame the photos: they can do it, but we've changed the job description quite a bit.

b) Option B: S3 Event Converter; translate the event and delegate

The second option is the one that convinced me: a thin Lambda that doesn't process anything. It only receives the S3 event, figures out which file it is, and dispatches the Laravel Job that already exists. A translator between two languages: from S3 JSON to a Laravel Queue payload.

namespace App\Lambda;

use Bref\Context\Context;
use Bref\Event\S3\S3Event;
use Bref\Event\S3\S3Handler;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Support\Facades\Log;

final class S3EventConverter extends S3Handler
{
    public const INBOX_KEY = '#^inbox/(?<job>[A-Za-z]+)/(?<file>[^/]+)$#';

    public function handleS3(S3Event $event, Context $context): void
    {
        foreach ($event->getRecords() as $record) {
            // original filename (URL-decoded)
            $key = urldecode($record->getObject()->getKey());

            // perform some basic validation:
            // 1. the file must be in the inbox ^/inbox,
            // 2. it must contain a Job name (?<job>[A-Za-z]+),
            // 3. it must contain a filename (?<file>[^/]+)$
            if (! preg_match(self::INBOX_KEY, $key, $m)) {
                Log::debug('S3 event ignored', ['key' => $key]);
                continue;
            }

            $jobClass = "App\\Jobs\\{$m['job']}";

            // finally, validate that the Job can be queued
            if (! is_a($jobClass, ShouldQueue::class, true)) {
                Log::warning('Unknown job for S3 inbox key', ['key' => $key, 'job' => $jobClass]);
                continue;
            }

            // finally, dispatch the Job to the SQS queue
            $jobClass::dispatch($key);
        }
    }
}
Enter fullscreen mode Exit fullscreen mode

Three decisions packed into those ~23 lines:

  1. The directory is the job: the inbox/{JobName}/{filename} convention maps the first segment of the key to a class in App\Jobs:
    • inbox/ProcessImage/abc.jpg arrives → dispatches App\Jobs\ProcessImage,
    • inbox/ImportGuests/xyz.csv arrives → dispatches App\Jobs\ImportGuests,
    • Need to add a new process type tomorrow? Just create the job: the converter stays untouched.
  2. Validate against ShouldQueue: is_a(..., ShouldQueue::class) resolves the class and checks that it implements the queueable-job interface. If someone or something uploads a file to inbox/Whatever/x.bin, the converter ignores it with a warning log instead of blowing up.
  3. Dispatch the full key: each job receives $key and handles it itself. A uniform contract for all inbox jobs: "one constructor, one key".

The converter is a translator with one job: listen for ObjectCreated and respond with dispatch().

And the serverless.yml configuration is much simpler than Option A: we listen on a single prefix and run a single Lambda:

functions:
    laravel:
        handler: Bref\LaravelBridge\Http\OctaneHandler
        runtime: php-85
        # ...

+   s3-event-converter:
+       handler: App\Lambda\S3EventConverter
+       runtime: php-85
+       memorySize: 256
+       timeout: 10
+       events:
+           - s3:
+               bucket: ${construct:storage.bucketName}
+               existing: true
+               event: s3:ObjectCreated:*
+               rules:
+                   -   prefix: inbox/
Enter fullscreen mode Exit fullscreen mode

The converter, which only reads S3 JSON and makes a SendMessage call to SQS, runs comfortably with 256 MB of memory (measured in my application). If you want to fine-tune it, aws-lambda-power-tuning can help you find the sweet spot between duration and cost 14. Since this is a background hop, a cold start doesn't keep anyone up at night.

Everything we didn't have to configure:

  • No IAM permissions: Lift already grants all functions in the stack sqs:SendMessage permission through the jobs construct 15, and read/write access to the bucket through the storage construct 16.
  • No queue variables: QUEUE_CONNECTION and SQS_QUEUE are already defined in provider.environment from the previous chapter.
  • No additional layers: the GD layer lives only in the jobs function (the queue:work from the previous chapter). The converter is a glorified SendMessage.
  • No new logic: ProcessImage is the same old job, #[Tries(3)], #[Backoff(30)], Intervention Image, etc.; only who dispatches it changes.

Why is it better than Option A?

It takes advantage of all the configuration from chapter 4:

  • Laravel handles retries with #[Tries] and #[Backoff],
  • exhausted jobs go to failed_jobs in DynamoDB with a TTL,
  • invocations that die halfway through end up in the DLQ with 14-day retention and an email alert,
  • batchSize and maxConcurrency hold back the avalanches,
  • the job can be tested with Queue::fake() and runs locally with QUEUE_CONNECTION=sync.

And it has a subtle architectural benefit: the converter doesn't write to S3, so it doesn't generate new events itself. To avoid a loop in the full flow, jobs must also write outside inbox/.

What about the downsides?

Well, yes, there are some:

  • Latency from one extra hop: S3 → converter → SQS → worker, instead of S3 → Lambda. We add conversion, enqueueing, and waiting latency.
  • It's async just like Option A: if everything fails in the converter, the event is discarded.
  • More things to monitor: an upload can reach S3 and fail before it gets into SQS. We need to distinguish a converter failure from a job failure.
  • Dispatch can also be duplicated: if SendMessage succeeds but the invocation fails afterward, a retry can enqueue the same work again. Idempotency is still necessary.
  • The converter needs its own failure policy and observability: if the converter fails before enqueuing the Job, there is no Laravel Job to recover with queue:retry.

Comparing the two options:

A: S3 → S3Handler directly B: S3 → Converter → SQS → worker
New pieces N Lambdas, one per task type (in this implementation) One Lambda; everything routes to the existing SQS queue
Retries Up to 2 additional retries by default for function errors #[Tries] + #[Backoff] + messages re-enqueued by Bref
If it fails, where does it end up? Discarded (or onFailure if configured) Worker: failed_jobs / DLQ. Converter: its own failure destination, if configured
Concurrency limit reservedConcurrency, without SQS consumption control maxConcurrency + reservedConcurrency
Infinite-loop risk Low if it writes outside inbox/ None (the converter doesn't write to S3), as long as the job doesn't write to inbox/
Add an upload type? New function + filter Just create the Job; the converter stays untouched
Tests / local Invoke the class manually Queue::fake(), QUEUE_CONNECTION=sync
Extra latency None Conversion, enqueueing, and waiting in SQS
When to choose it? Direct processing when we don't need to reuse Laravel's queue When we want to reuse Laravel jobs and its queue system

c) Why Not Point S3 Directly at the Existing Queue?

Since we already have a queue that works perfectly, the obvious question is: "what if S3 sent notifications directly to the jobs queue Lift created?" After all, SQS is a valid notification destination 2. Two variants, two short answers: neither works.

Variant 1: S3 → worker function (Lambda destination). Bref's QueueHandler doesn't listen for S3 events: it extends SqsHandler, and every message goes through SqsRecord, which validates the source in its constructor 17:

public function __construct(mixed $record)
{
    if (! is_array($record) || ! isset($record['eventSource']) || $record['eventSource'] !== 'aws:sqs') {
        throw new InvalidArgumentException;
    }
    $this->record = $record;
}
Enter fullscreen mode Exit fullscreen mode

An S3 event has eventSource: "aws:s3", so an InvalidArgumentException is thrown on the first record. The Lambda dies on the first "hello".

Variant 2: S3 → jobs queue (SQS destination). This one is more interesting, because technically it could work... if the message body were Laravel's payload. But S3 sends its own JSON (the one in section 2.c.), while Laravel expects a payload with fields such as displayName, job, data, and uuid, generated when the job is dispatched 18. Job::fire() starts like this 19:

public function fire()
{
    $payload = $this->payload();

    [$class, $method] = JobName::parse($payload['job']);
    // ...
}
Enter fullscreen mode Exit fullscreen mode

$payload['job'] doesn't exist in S3's JSON: the message doesn't have the structure needed to run a Laravel job. We can't assume it will follow the normal retry cycle and end up in failed_jobs; we're handing the worker a format it doesn't understand.

The attempts counter is another piece: when a valid job is released for a retry, Bref increments that field and resends the payload to SQS 20. That mechanism doesn't turn an S3 notification into a Laravel job.

Bonus: S3 requires a queue policy authorizing s3.amazonaws.com to call SendMessage 2, which Lift doesn't add; and when you enable the notification, S3 publishes an s3:TestEvent event in a different format 4 that would also break the worker.

It's like putting the letter in the right mailbox but writing it in another language: the mail carrier delivers it, the recipient can't understand it, and throws it away.

That's why we added the converter: it adapts the S3 event to the format our worker expects, without changing how jobs are processed.


4. A Photo's Complete Journey

Let's put it all together. Here's the complete journey of a photo with this design, from beginning to end, and how it differs from the "browser notification":

Journey

a) Laravel creates the signature; the browser uploads to S3

The controller creates the photo record and returns the pre-signed URL 21. The file is uploaded directly to S3 under inbox/{JobName}/{file}. The directory tells us which job will process it:

// MediaController::prepareFileUpload()
$media = Media::create([
    'status'   => MediaStatus::Uploading,
    'filename' => $filename = sprintf('%s.%s', Str::uuid()->toString(), $fileInfo['extension']),
    /* ... */
]);

$signedUrl = Storage::temporaryUploadUrl(
    path: "inbox/ProcessImage/{$filename}",
    expiration: now()->addMinutes(15),
    options: ['Content-Type' => $fileInfo['mime_type']],
);
Enter fullscreen mode Exit fullscreen mode

Security detail: the pre-signed URL only allows an upload to that exact key. We choose the directory and filename, not the client; no one can PUT to inbox/DeleteDatabase/x unless we sign that key.

The browser uploads the file directly to S3:

xhr.open('PUT', uploadData.upload_url);
xhr.send(file);
Enter fullscreen mode Exit fullscreen mode

b) S3 Confirms the Upload

When the browser upload finishes, S3 emits the ObjectCreated:Put event (or CompleteMultipartUpload if the frontend uses multipart), the prefix: inbox/ filter lets it through, and the notification invokes the s3-event-converter Lambda asynchronously.

c) The converter translates and dispatches

The Lambda reads the key (inbox/ProcessImage/9f8d7c6b-...jpg), extracts ProcessImage from the directory, checks that it's a queueable job, and dispatches App\Jobs\ProcessImage::dispatch($key) to the queue.

The converter's responsibility ends when it dispatches the job to SQS. Processing happens in the worker.

d) The worker does the heavy lifting

SQS invokes the Queue Worker from chapter 4, and Laravel's Worker runs ProcessImage. The job reads the original image from inbox/, generates the 2048px version and the 600px thumbnail, stores the generated versions outside the inbox (media/), updates the status from processing → completed, and finishes by emptying the inbox:

Simplified job for readability.

final class ProcessImage implements ShouldQueue, ShouldBeUnique {
    use Queueable;

    private readonly string $filename;

    private Media $media;

    public function __construct(
        public readonly string $key,
    ) {
        preg_match(S3EventConverter::INBOX_KEY, $key, $matches);
        $this->filename = $matches['file'];
    }

    public function uniqueId() {
        return $this->getMedia()->uuid;
    }

    public function handle(): void {
        // ..., check whether the image has already been processed,
        // validate that the original file exists, ...
        if (! Storage::exists($media_path = "inbox/ProcessImage/{$this->filename}")) {
            throw new Exception("The original file does not exist");
        }

        // ... Process, write to media/, ...
        $image = Image::decodeStream(Storage::readStream($media_path));
        Storage::put("media/{$this->getMedia()->uuid}.webp", $image->encodeUsingFormat(Format::WEBP, quality: 85));

        // ..., update the Media record's status, ...
        $this->getMedia()->update([ 'status' => MediaStatus::Completed ]);

        // ..., cleanup: the original image is no longer needed
        Storage::delete($this->key);
    }

    private function getMedia(): Media {
        return $this->media ??= Media::ofFilename($this->filename)->query()->first();
    }
}
Enter fullscreen mode Exit fullscreen mode

Cleaning up only at the end matters: if the job fails and Laravel retries it, the original is still waiting there.

e) Events That Don't Trigger the Converter

This is where the inbox/ prefix design pays off twice. The job writes to media/, outside the prefix, so those writes don't invoke the converter, preventing infinite loops.

This avoids invocations triggered by the generated versions. We expect one converter invocation per upload, though duplicates and retries are still possible.

The final Storage::delete() emits an ObjectRemoved, but we only listen for ObjectCreated.

f) What about the browser's PATCH?

The status endpoint still exists, but it's no longer the processing trigger: it remains to record the failed event if the browser couldn't upload the photo.

The browser went from being the main witness to a courtesy notification.


5. What Tutorials Don't Tell You (Caveats and Risks)

a) The infinite loop: a bucket calling itself

AWS warns about this in the official docs, in bold and everything 1 10: if the notification destination writes to the same bucket that triggers it, you can create an infinite loop. upload → process → write thumbnail → trigger same event → ∞. The inbox/ design cuts it off at the source: only uploads land there, results live in other directories, and the final delete produces an ObjectRemoved that no one listens for.

That said, the loop is still possible if we don't follow the convention. To prevent it, we combine prevention by design with one last safety net:

  1. Filter in code: the converter validates the key before dispatching and won't let anything through unless it matches inbox/{Job}/{filename}.
  2. Dedicated input prefix: inbox/ receives uploads only; no job should write there. If a job ever needs to leave a processed output, it goes in media/ or another appropriate directory, never in the inbox.
  3. AWS's safety net: Lambda has recursive loop detection, enabled by default: if it detects a repeated S3 → Lambda → S3 chain about 16 times, it stops the invocation and notifies you by email and in the Health Dashboard 22.

Heads up: recursive loop detection is the airbag, not the seat belt. The cutoff comes after approximately 16 invocations in the same request chain, so it doesn't prevent those invocations from consuming resources. It only covers loops involving Lambda, S3, SQS, SNS, and EventBridge custom event buses (with AWS SDK for PHP ≥ 3.232.0) 22. A loop that passes through DynamoDB, for example, is outside this protection.

b) The key is URL-encoded

The key detail comes up again: mi foto.jpg arrives as mi+foto.jpg 4. And Bref doesn't decode it. BucketObject::getKey() returns the name exactly as it appears in the JSON 23. If you call Storage::get('mi+foto.jpg'), you'll get a glorious 404 that ends up in failed_jobs three attempts later. Always call urldecode() as the handler's first step.

c) At-least-once delivery + Lambda retries

Two sources of duplicates: S3 may publish the same event twice (rare, but documented) 1, and Lambda may invoke the converter more than once with the same event 11. As a result, ProcessImage may be dispatched twice for the same photo. The recipe I used is ShouldBeUnique, using the Media uuid as uniqueId() 24, and checking whether the destination file exists when the job starts.

d) Inbox orphans (and S3's Roomba)

Storage::delete($key) at the end of the job keeps inbox/ clean, but it only happens on the happy path.

But what if the job uses up all 3 attempts and ends up in failed_jobs?

In that case, the original file stays in the inbox (which is exactly what we want). It's available for a queue:retry, unless it's never retried and inbox/ turns into a trash dump billed by the GB-month.

That's what the lifecycleRules we added to the storage construct in section 3 are for: S3 applies an expiration rule to inbox/ originals starting at 7 days 8. If the bucket has versioning enabled, expiration creates a delete marker and the previous version is subject to its own cleanup policy; it doesn't delete every version of the file at that point 25. It's the same Roomba that cleans our DynamoDB cache table from chapter 3, but for a bucket.

e) events.s3.existing: true has some quirks

  • Only one existing bucket per function; but we can declare several events from that same bucket for the function, as long as their filters are compatible.
  • If CloudFormation doesn't update the triggers on subsequent deployments, you can add forceDeploy: true to force the update 9.
  • The notification takes about 5 minutes to activate after deployment; if you test by uploading a file and nothing happens, wait a bit before blaming the code 26.

f) Multipart uploads: the Put that isn't a Put

If the frontend uploads large files using multipart, the event isn't ObjectCreated:Put but ObjectCreated:CompleteMultipartUpload 2. That's why we chose to listen for s3:ObjectCreated:* in serverless.yml: the wildcard covers every way of creating an object.

g) Filters must not overlap

If you later add a second function listening to the same bucket for the same event, the prefix/suffix filters can't overlap: if you define two functions with prefix: inbox/ and ObjectCreated:*, S3 will reject the deployment 3. That's another reason we chose to consolidate all routing in the converter.

h) Test locally without AWS

The converter is a regular PHP class, so it's easy to test. Use a sample JSON (trimmed from a real event) in a test with Queue::fake(): instantiate the handler, call handleS3() with a manually constructed S3Event, and verify that the job was sent to the queue with Queue::assertPushed(ProcessImage::class).

The real ProcessImage class runs locally with QUEUE_CONNECTION=sync, as always.

i) sequencer: the number that puts chaos in order

S3 doesn't guarantee order, but Put and Delete events include an increasing hexadecimal sequencer for each key 4. If it ever matters that "the latest version wins" (e.g. re-uploading the same file), save the last processed sequencer and discard lower ones. I don't need it in my case, but it's good to know.

j) Cleaning up the SDK

S3 was already on the removeUnusedServices list in chapter 2, so there's nothing to add. But if your SDK is trimmed down, check that S3 and Sqs are included before blaming the code.


6. How Much Does Connecting the Pieces Cost?

Let's update the numbers, as we do in every chapter.

  • S3 Event Notifications: 100% free. There's no charge for publishing, filtering, or the s3:TestEvent 27.
  • The converter: one invocation per event; and thanks to the inbox/ prefix, one invocation per upload (excluding duplicates and retries). For an estimate of 100,000 uploads/month:
    • 100,000 invocations at 256 MB for ~100 ms
    • 100k requests ($0.02) + 2,500 GB-s ($0.04) ≈ $0.06/month ($0.00 within the free tier) 28
  • SQS: we estimate ~$0.12 for operations on 100,000 messages and $0.25 for polling, using the same scenario as the previous chapter: $0.37 total 29.
  • The worker: the same cost we calculated in chapter 4 (~$0.44/100k jobs total) 28. The heavy lifting hasn't changed.
~100k uploads/month Free tier Full rate
S3 Event Notifications (~100k notifications) $0.00 $0.00
Converter (~100k invocations, 256MB, 100 ms) $0.00 ~$0.06
SQS (300k message requests + 630k polling) $0.00 ~$0.37
Worker (100k invocations, 512MB, 500 ms) $0.00 ~$0.44
Monthly total $0.00 ~$0.87

S3 doesn't charge to send the notification. The bill starts when we put the rest to work.


7. What's Next?

Today we connected the direct-upload flow from chapter 3—"the file travels from the browser to S3 without touching Lambda"—to the worker from chapter 4—"the heavy lifting with all its retries, DLQ, and alarms"—through a Lambda that acts as a translator.

So far, Laravel wakes up because someone makes a request or an event arrives. But some tasks don't wait for an event: reservations need to expire, reminders need to be sent, or data needs to be cleaned up at a specific time. For all these cases, who wakes up our Laravel application to run them if we no longer have a server with cron? In the next chapter, we'll see how to run the scheduler with EventBridge and Artisan without keeping a server watching the clock.


One question for you: How does your application currently find out that an upload has finished: does the browser tell it, or does storage notify it? Let me know in the comments.


  1. https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventNotifications.html ↩

  2. https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-how-to-event-types-and-destinations.html ↩

  3. https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-how-to-filtering.html ↩

  4. https://docs.aws.amazon.com/AmazonS3/latest/userguide/notification-content-structure.html ↩

  5. https://aws.amazon.com/blogs/storage/manage-event-ordering-and-duplicate-events-with-amazon-s3-event-notifications/ ↩

  6. https://bref.sh/docs/use-cases/s3 ↩

  7. https://github.com/brefphp/laravel-bridge/blob/a62196d5cb3efe95861fc6b78f7e5606bd2ad09d/src/HandlerResolver.php#L68-L94 ↩

  8. https://github.com/getlift/lift/blob/45ec61c6fb36e2a220326e62e4e543e5b11fbdbc/docs/storage.md ↩

  9. https://www.serverless.com/framework/docs/providers/aws/events/s3#using-existing-buckets ↩

  10. https://docs.aws.amazon.com/lambda/latest/dg/with-s3.html ↩

  11. https://docs.aws.amazon.com/lambda/latest/dg/invocation-async-error-handling.html ↩

  12. https://www.serverless.com/framework/docs/providers/aws/guide/functions#asynchronous-invocation ↩

  13. https://docs.aws.amazon.com/lambda/latest/dg/configuration-concurrency.html ↩

  14. https://github.com/alexcasalboni/aws-lambda-power-tuning ↩

  15. https://github.com/getlift/lift/blob/45ec61c6fb36e2a220326e62e4e543e5b11fbdbc/docs/queue.md#permissions ↩

  16. https://github.com/getlift/lift/blob/45ec61c6fb36e2a220326e62e4e543e5b11fbdbc/docs/storage.md#permissions ↩

  17. https://github.com/brefphp/bref/blob/3b1dbda5680930187bd44b7c863429e570b35692/src/Event/Sqs/SqsRecord.php#L16-L17 ↩

  18. https://github.com/laravel/framework/blob/c42f9abd4a3d2f9a3cd6be109255ddce8092ceb8/src/Illuminate/Queue/Queue.php#L176-L215 ↩

  19. https://github.com/laravel/framework/blob/c42f9abd4a3d2f9a3cd6be109255ddce8092ceb8/src/Illuminate/Queue/Jobs/Job.php#L96-L103 ↩

  20. https://github.com/brefphp/laravel-bridge/blob/a62196d5cb3efe95861fc6b78f7e5606bd2ad09d/src/Queue/SqsJob.php#L13-L45 ↩

  21. https://laravel.com/framework/docs/13.x/filesystem#temporary-upload-urls ↩

  22. https://docs.aws.amazon.com/lambda/latest/dg/invocation-recursion.html ↩

  23. https://github.com/brefphp/bref/blob/3b1dbda5680930187bd44b7c863429e570b35692/src/Event/S3/BucketObject.php#L24-L27 ↩

  24. https://laravel.com/framework/docs/13.x/queues#unique-jobs ↩

  25. https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecycle-expire-general-considerations.html ↩

  26. https://docs.aws.amazon.com/AmazonS3/latest/userguide/how-to-enable-disable-notification-intro.html ↩

  27. https://aws.amazon.com/s3/faqs/ ↩

  28. https://aws.amazon.com/lambda/pricing/ ↩

  29. https://aws.amazon.com/sqs/pricing/ ↩

Top comments (0)