DEV Community: Jakub Stanisławczyk

DynamoDB Professional - part 2 - sparse index

Jakub Stanisławczyk — Tue, 26 Aug 2025 09:25:29 +0000

In the previous part, I talked about indexing mechanisms in DynamoDB. I mentioned that there are two types of indexes: Local Secondary Index (LSI) and Global Secondary Index (GSI). I lied to you a little, because there are actually three types. But wait, how is that possible? Is the AWS documentation wrong?

How does a GSI work?

Before I answer that question, let's first go back to the basics. Let's recall how a GSI works. I covered its mechanism in the first part of this series. In short, this index operates under the hood like a separate table. Data is copied into it only when needed. And it is precisely this mechanism that forms the foundation of what can be called a Sparse Index.

Sparse Index

A sparse index is a type of database index where we don't create an entry for every single record in the table - only for the ones we care about. In DynamoDB, this isn't something that's directly supported. But, using the mechanisms we've already talked about, we can simulate it pretty easily.

Standard GSI Index example

Let's take an example. Imagine we're building an app that processes events from temperature sensors. Say each sensor emits an event every second. Each event has an ID, the sensor ID, a createdAt, a value, and status. The status is just a simple enum: OK if the value is within the expected range, or ALARM if it's outside.

Now, suppose we want to add a section to our dashboard that shows only the alarms for a given sensor. It's easy, we create a new GSI called SENSOR_STATUS_GSI, where the partition key is {sensorId}#{status} and the sort key is {createdAt}.

This is how it looks in the code. We create separate SENSOR_STATUS_GSI_PK property.

With this setup, querying events by status becomes super straightforward. When we scan index we can see all the events. In this example I created 10 events for 2 sensors (5 events per sensor)

When we want to fetch all the alarms for the sensor, we only need to provide expected GSI PK

But this isn't really the best solution. After all, what we actually need is to query only the events that are alarms. Now we're storing everything. In a system that might generate millions of events, this would quickly increase the cost of our DynamoDB table.

Instead, we can tweak our GSI a little bit.

Sparse Index example

Let's create new ALARM_GSI index. This one uses {sensorId} as PK and {createdAt} as SK.

I also updated the code with the new GSI PK.

When we query our index by sensor ID, we'll only get corresponding alarms.

The magic really starts to happen when we scan our index.

The difference is immediate. It turns out the index now stores only the alarms, no events with an OK status in sight. The index already contains only the data you want.

Summary

A Sparse Index in DynamoDB is basically a clever hack. Even though DynamoDB doesn't give us native support for this feature, with a bit of creativity we can simulate it using GSIs.

This solution is very useful for large data sets that need to be searched binarily. For example:

A job queue where we need to search for jobs with a status of PENDING to process
Searching only for available products

The downside of this solution is that future changes must be anticipated. If the business also wants to display recent events with a status of OK, our sparse index won't be able to handle it. This will require either adding another index or modifying the current index and migrating the data.

Sparse indexes are a neat DynamoDB trick. They're not always the right fit, but in scenarios where you only care about a subset of your data, they can save you money and speed up your queries.

Links

👋 LinkedIn
💻 Github

DynamoDB Professional - part 1 - indexing

Jakub Stanisławczyk — Mon, 07 Jul 2025 13:24:43 +0000

Amazon DynamoDB is a fully managed, NoSQL database designed for high performance and scalability. With the ability to handle massive numbers of requests per second and automatic scaling, DynamoDB is widely used in systems that require reliability and low latency - from mobile applications to IoT systems and microservices.

However, despite its many advantages, DynamoDB has one key limitation - it doesn't forgive mistakes made during the design of your solutions. Unlike relational databases or document databases like MongoDB, here the structure of data and queries must be carefully planned. DynamoDB requires not only knowledge of its indexing mechanisms, but more importantly, a deliberate approach to data modeling and an understanding of the application query patterns. This is crucial, as some decisions cannot be easily reversed without creating a new table and migrating data.

In this article, I will explain you how the DynamoDB indexing mechanism works and show you how to plan your data structure to fully harness the potential of this powerful yet demanding technology.

A bit of theory

Before we dive into the internal mechanisms of DynamoDB, we need to start with the basics. It is the understanding of general database techniques that will help us achieve maximum performance. One of the fundamental topics in any distributed database is the data consistency. In the case of DynamoDB, there are two available read modes: Eventually Consistent Read and Strongly Consistent Read.

What's the difference between them? With Strong Consistency, you are always guaranteed to receive the most up-to-date data. In the other hand, if you choose Eventual Consistency, there's a chance that the returned data may be outdated. Its value will eventually be updated to the expected state, but this will happen with some delay.

The read mode is selected at the query level. Let's see how this works with an example:

export const getUserById = async (id, consistentRead = true) => {
    const ddbClient = initDDBClient();
    const command = new GetCommand({
        TableName: usersTable,
        Key: {
            id,
        },
        ConsistentRead: consistentRead, // Set true/false
    });

    try {
        const response = await ddbClient.send(command);
        return response.Item;
    } catch (error) {
        console.error("Error getting item:", error);
        throw error;
    }
}

Now we can use this function to fetch our data.

const user = {
    id: crypto.randomUUID(),
    name: "John Doe",
    email: "jdoe@examplemail.com",
}

await saveUser(user);

const [eventualUser, strongUser] = await Promise.all([
    // Read with eventual consistency.
    getUserById(user.id, false),
    // Read with strong consistency
    getUserById(user.id, true)
]);

console.log("Read with eventual consistency:", eventualUser);
console.log("Read with strong consistency:", strongUser);

At this point Read with strong consistency will always return correct data. Read with eventual consistency may show empty result.

Now you might be thinking, 'So the solution to all problems is to always choose Strong Consistency by default?'. Not quite. Unfortunately, such reads consume twice as many Read Capacity Units (RCUs). What does that mean? If you're using the On-Demand mode, it means you'll pay twice as much for your queries. If you're using the Provisioned mode, you'll reach the RCU limit of your table much more easily.

So, if your application can tolerate slightly outdated data, it's worth considering Eventual Consistency to optimize costs.

Another important point is that not all queries can support Strong Consistency. But more on that in a moment :)

What can a record consist of?

First and foremost: the primary key. This is what uniquely identifies each entity in our table. It can consist of two components:

Partition Key (always required) - Its value determines which partition a given record will be placed into. This works through a hashing function that distributes entities across physical nodes. As a result, the load is evenly balanced and the data access is faster. Each partition, except of the size (10 GB), also has a maximum of 3000 Read Capacity Units and 1000 Write Capacity Units. If your table grows beyond that, DynamoDB automatically adds more partitions. However, if too much data is stored under a single partition key, it cannot be split. That's why it's crucial for the partition key (PK) to be as granular as possible. Otherwise, we might end up with all records landing in a single partition - a situation known as a hot partition, which significantly slows down queries. Let's visualize it. We have a users identified by a given ID. ID is hashed using an internal algorithm, which determines which partition the item will be stored in.

Sort Key (optional) - Adds powerful capabilities to our table:
- Enables ordering of data within a single partition, so we can retrieve records in a specific sequence - either from the beginning or the end of the table.
- Allows us to associate multiple records with a single partition key. A good example would be documents with multiple versions. Caution⚠️: If we associate too many items with a single partition key, it can still result in a hot partition.
- It also enables more advanced querying capabilities. With only a partition key, we can retrieve data by exact match. But once we introduce a sort key, we can perform range queries, search by prefix, and many more.

Let's see how it looks in practice. A common use case for sort keys is storing versioned data. For example, imagine a system that manages files. Each file has a unique identifier which we use as the partition key. Files can have multiple versions, so we use the version number as the sort key.

This way, all versions of a given file are stored under the same partition, and the sort key allows us to query them efficiently - whether we want to fetch all versions or only specific one.

If you use only the partition key in your query, DynamoDB will return all items that share that partition key (e.g. all versions of a specific file).

Query document by PK = "abc123"
[
    { id: "abc123", version: 1, ...},
    { id: "abc123", version: 2, ...},
    { id: "abc123", version: 3, ...},
]

If you provide both the partition key and the sort key (PK + SK), DynamoDB will return exactly one item - the one that matches both values. This is ideal when you want to fetch a specific version of a file.

Query document by PK = "abc123" and SK = "1"
[
    { id: "abc123", version: 1, ...},
]

Types of Operations

Now that we understand what our data might look like, let’s take a look at the different ways we can retrieve it:

GetItem
BatchGetItem
Query
Scan
ParallelScan
TransactGetItem

Whoa - that's quite a few options just for reading data! So, which one is the best?
Well, it depends on what you're trying to retrieve and how your table is structured. Let's briefly go over each one:

GetItem - The most basic read operation. It retrieves a single item based on the partition key. Note: If your table has a sort key defined, you must provide it as well.
BatchGetItem - Retrieves multiple items in a single batch (up to 100 items or 16MB per request). This is much more efficient than fetching them one at a time.
Query - Offers more advanced search capabilities (e.g. sorting, filtering, range queries). It can also be used to query secondary indexes, which I'll cover shortly.
Scan - Reads every item in the table. Because of its brute-force nature, this operation is slow and expensive. You can apply filters to reduce the number of returned results - but keep in mind that RCU (Read Capacity Units) usage remains the same. It just reduces the size of the received object.
ParallelScan - Works like Scan, but splits the work across multiple parallel workers. It's faster, but can quickly consume your table's available RCU.
TransactGetItem - Reads up to 25 items across one or more tables, with all-or-nothing guarantees. If one item can't be retrieved, none will be returned.

As you can see, the most efficient and preferred options for retrieving data are GetItem, BatchGetItem, and Query.
Scan operations should be your last resort - only use them when none of the other methods fit your use case.

Indexes

A primary key alone often isn't enough - we frequently need more flexibility when querying our data than just using an ID.
That's where indexes come in. Just like in other database engines, indexes in DynamoDB are used to optimize and speed up queries on your tables.
It supports two types of indexes. We can visualize them as follows:

Local Secondary Index (LSI)

An LSI provides an alternative sort key while still using the same partition key. Since it's stored within the same partition as the base table, it supports both eventual and strongly consistent reads. However, LSIs come with several limitations that are important to consider during the design phase:

You can define up to 5 LSIs per table, which can limit your query flexibility.
LSIs share the table's provisioned throughput and storage, so heavy usage of the index may impact performance of other operations.
They must be created at the time the table is created - you can't add LSIs to an existing table.
You'll incur additional costs for storing data in the index.

These constraints mean that you need a clear understanding of your query patterns before designing your schema.

Earlier, we discussed a document metadata table that stores versioned file metadata, where the partition key is the file ID and the sort key is the version number. While this model is effective for accessing different versions of a file, you might encounter situations where you want to query versions of a file by their name - for example, to find all versions with a specific name pattern or to retrieve a renamed file history.

This is where a Local Secondary Index (LSI) becomes useful. By defining an LSI on the name attribute, you can query the same set of items (same partition key) but sort and filter them based on their name instead of version number.

Query LSI "name" index by PK = "abc123" sorted in reverse order
[
    { id: "abc123", version: 2, name: "myfile.txt", ... },
    { id: "abc123", version: 3, name: "final.txt", ... },
    { id: "abc123", version: 1, name: "draft.txt", ... },
]

Global Secondary Index (GSI)

A GSI allows you to define a completely different partition key and sort key, giving you far greater flexibility in querying your data. DynamoDB supports up to 20 GSIs per table, making them a powerful tool for advanced query requirements.

Since GSIs are stored in a separate partition space, they:

Have independent read/write capacity limits
Can be added after the table is created
Support only eventual consistency for reads

As with LSIs, storing data in a GSI incurs additional storage and throughput costs. To control these costs, you can configure attribute projections, which determine what data is copied to the index:

KEYS_ONLY - Only the primary key and index keys are projected
INCLUDE - Keys and selected non-key attributes
ALL - All attributes from the base table

Let's say we have a table that stores user data. Each user has a unique id, which we use as the primary partition key. However, we also want to be able to query all users who belong to a specific team - something that isn't possible with the base table's key structure alone.

To support this access pattern, we can define a GSI with teamId as the partition key. What's important to understand is that a GSI behaves like a separate table under the hood. It has its own partition key and (optionally) sort key, independent from the base table. In the case of our user table, this means that teamId becomes the primary key of the index, allowing us to efficiently query all users assigned to a specific team.

Examples

Testing Eventually Consistent Reads

A common challenge is properly testing code that uses Eventually Consistent Reads. What does a typical integration test verifying database reads look like?

Insert test data
Read the data
Validate its correctness

This ensures each test is independent and can be run in isolation. Unfortunately, as mentioned earlier, Eventually Consistent data can have a delay, which may result in inconsistency during read operations. This, in turn, can lead to failed assertions.

Let's imagine our TEAM_ID GSI in our users table.

The first idea might be to wait and poll periodically until the data is updated.

...Other tests

it('should read users by team', async () => {
    // Arrange
    await saveUser(user);

    // Act
    let attempts = 15;
    let usersByTeam = [];

    for (let i = 0; i < attempts; i++) {
        usersByTeam = await getUsersByTeam(teamId);

        if (usersByTeam.length > 0) {
            break;
        }

        await setTimeout(300); // Wait for eventual consistency
    }

    // Assert
    expect(usersByTeam.length).toBe(1);
});

...Other tests

However, this results in significantly longer test execution times.
A better solution might be to insert the data before all tests and run the eventual consistency tests at the very end. Of course, we still need to wait to be 100% sure that the data has eventually become consistent. However, part of that waiting time will be offset by other tests executing in the meantime.

beforeAll(async () => {
    // Init GSI data before all tests
    await saveUser(user);
})

... Other tests

// GSI tests below
it('should read users by team', async () => {
    // Act
    let attempts = 15;
    let usersByTeam = [];

    for (let i = 0; i < attempts; i++) {
        usersByTeam = await getUsersByTeam(teamId);

        if (usersByTeam.length > 0) {
            break;
        }

        await setTimeout(300); // Wait for eventual consistency
    }

    // Assert
    expect(usersByTeam.length).toBe(1);
});

Prefixing

Prefixing is a common technique used in DynamoDB to structure sort keys in a way that enables more flexible and efficient query patterns.

Instead of using a raw value as a sort key (e.g. just a timestamp), you prefix the sort key with a constant label or category to create a composite value. This allows you to distinguish between different types of data stored under the same partition key and query them accordingly.

Let's consider the following structure:

A company has multiple teams
Each team has multiple users
Each user can upload multiple documents

We can store all of this in a single DynamoDB table using the following pattern:

PK - company ID
SK - TEAM#{teamId}#USER#{userId}#DOCUMENT#{documentId}

Why this Works?

You can query all teams of a company: SK begins_with(SK, 'TEAM#')
You can query all users in a team: SK begins_with(SK, 'TEAM#teamA#USER#')
You can query all documents for a user: SK begins_with(SK, 'TEAM#teamA#USER#user1#DOCUMENT#')

However, it's important to note that parsing the composite sort key is our responsibility. Since DynamoDB stores the sort key as a simple string, it's up to the application logic to split and interpret its components (e.g. extracting teamId, userId, or documentId from a key like TEAM#teamA#USER#user1#DOCUMENT#doc1).

This design also introduces a potential risk of hot partitions. Since all nested data (teams, users, documents) lives under a single partition key (company ID). Let's see this in the next example.

Hot Partition

We're building software for factories that collects sensor events. Since the system is multi-tenant, we decided to model multi-tenancy directly at the DynamoDB level. Specifically, the partition key is the company ID, and the sort key is a composite value in the format {sensorId}#{eventId}. This design allows us to efficiently query for all events from a specific sensor or retrieve a specific event by ID.

However, we've encountered a scaling issue: one of our clients has thousands of active sensors, each sending event once per second. Since all this data shares the same partition key (their company ID), DynamoDB directs all writes to a single partition, which quickly becomes a hot partition. This causes problems such as throttled writes due to exceeding partition throughput limits and poor horizontal scalability.

This example highlights how seemingly clean and logical data models can run into physical limitations in high-throughput environments, especially when many high-frequency entities (like IoT sensors) share the same partition key.

There are several techniques to mitigate this problem but those are more advanced strategies that deserve a dedicated article :)

Summary

As we've seen through the examples, DynamoDB offers immense power, but only when used with a clear understanding of how it works under the hood.

One of the most critical aspects of working with DynamoDB is data modeling, and at the heart of that lies indexing. Choosing the right partition key, designing efficient sort keys and leveraging Global or Local Secondary Indexes is the key to the efficient database.

DynamoDB can be incredibly fast, scalable, and reliable - but only if you know exactly what you're doing. That's why investing time upfront in understanding your access patterns and indexing strategy is not just helpful - it's mandatory.

Getting indexing right is a foundational step - but truly mastering DynamoDB requires diving much deeper. We'll explore those advanced techniques in the next parts of this guide. Stay tuned 🎉.

Links

👋 LinkedIn
💻 Github

Become the Serverless DJ. How to process audio using AWS?

Jakub Stanisławczyk — Mon, 02 Jun 2025 16:55:56 +0000

Intro

As software developers, we work with different types of files. These are often formats like JSON, XML, or CSV. Data engineers, on the other hand, use more specialized tools such as Parquet. Beyond text files, we also process images - resizing them, adjusting colors, or altering shapes.

However, there is one type of medium that seems to be somewhat overlooked, despite surrounding us everywhere. After all, who doesn't enjoy listening to music to relax or using it as background sound while working?

But how do we work with audio files? How can we process them in the AWS cloud? What aspects should we consider to ensure our architecture is both scalable and cost-effective? I'll answer these and many other questions in this article.

Full code can be found on my Github. I used Terraform to describe all AWS resources. This will help you set up this project without having to configure it manually. I decided not to use any architectures or design patterns because I wanted to keep things as simple as possible. Feel free to extend it using layers, interfaces, hexagonal architecture etc.

Technologies used

In the world of modern cloud solutions, we are constantly looking for ways to increase efficiency, reduce costs and eliminate unnecessary infrastructure. That is why choosing serverless technology seemed like a natural step. Thanks to the model in which I do not have to worry about managing servers, I can focus on what is most important - processing audio in a fast, scalable and cost-optimized way. The following services will help me with this:

Amazon S3 - The storage, where we will save our files. It scales very well and has high availability (99.999999999%). Mechanisms such as Presigned URL or S3 Events will be an important part of our architecture.

AWS Lambda - This will be our main working tool that allows us to run our Node.js code. The main advantage is that we only pay for the time it takes to run, making it ideal for reacting to events. Cons? Maximum 15 minutes of runtime and 10GB of RAM.

AWS API Gateway - This service allow us to expose our Lambdas as REST API endpoints.

DynamoDB - A NoSQL database that provides very good performance and scalability. To be able to talk about its capabilities would require a separate (and not so short) article. TLDR: DDB is for you if you need an efficient and scalable database, and at the same time you know exactly what the query patterns will be.

Amazon SQS - A simple AWS queue that allows us to send events in two modes:
- Normal - where we have virtually unlimited scaling but duplicate messages are possible.
- FIFO - where duplicates are automatically removed but with a limit of 300 operations per second.

FFmpeg - This tool allows us to customize and modify the audio file to suit our needs. It is a CLI tool designed for multimedia processing and consists of two subtools:
- FFmpeg – enables conversion to different formats, trimming and merging files, changing the sampling rate… and that’s just the tip of the iceberg.
- FFprobe – allows us to analyze the file, including checking size, format, and other attributes..
Well, a CLI tool. Won't that be a problem in the case of Lambda? After all, it is a serverless solution, which is very high-level. Fortunately, there is a way to solve this problem, but more on that later.

Architecture

When developing our solution, we need to cover three fundamental aspects:

Upload – How can we efficiently deliver new files?
Audio Processing – Similar to text files, audio files come in a wide range of formats. Additionally, each file can have different sampling rates and channel configurations. Standardizing them will simplify further processing.
Metadata – It’s essential to ensure that files can be easily searched and sorted later.

Here's how the final process looks like.

Upload

Uploading a file seems like the least of your problems. After all, it's just sending a file to our backend and throwing it into an S3 bucket? Right?

Well, not really. Of course, it can be done this way, but it will be inefficient. After all, in such an architecture, our Lambda will work a bit like a shovel that has to move a large amount of data and that is its only task. What if our client could upload a file directly to the Bucket?

Fortunately, AWS provides us with a Presigned URL mechanism that allows for direct upload to the S3 Bucket. It's very simple:

In the first step, we ask the bucket to generate an URL that allows us to upload file directly.
After receiving the response from the URL, we do a redirect to it. We place our file in the body of the new request.

Another factor to consider is the file size - larger files may take longer to upload and are more susceptible to network errors. The solution is to use the Multipart Upload. It allows parallelization of requests and increases resilience, for example by enabling re-sending. For what files should it be used?

>100MB - you should consider using this mechanism.
>5GB - AWS requires the use of Multipart Upload for files larger than 5GB.

In this example I will stick to the PutObject for simplicity purposes.

Audio Processing

With the raw audio file already saved, the next step is to pass it to Lambda to initiate processing. The easiest way is to use the S3 Events mechanism. It allows us to listen for the changes in directories and files. We specify the event type, prefix, suffix, and the destination service for the notification. From now on, each time a file is added that fits the rules given above, an event is triggered that will start our Lambda. Of course, it doesn't contain the file itself - only the metadata needed to download it from our S3 bucket.

Alternatively, you can use EventBridge, which is a more general solution that supports a much wider range of services and events.

As you can see, this is a very simple architecture and unfortunately not fully functional. There is one detail that may be problematic and it is "At least one delivery" of S3 events. This means that duplicates may occur, which will cause us to process the same file twice. The simplest way is to set up an additional SQS FIFO queue, which will automatically reject duplicates and save us some computing resources. Note that this will only work if we fit into the SQS FIFO limit (300 messages per second). To achieve higher throughput, we can use a solution like DynamoDB to track whether a given event has already been processed.

Okay, but how should our actual audio processing look like? How to use the FFmpeg in Node.js? There is two ways:

You can directly call CLI commands using Node child_process

import { exec } from 'child_process';
import { promisify } from 'util';

...

const execAsync = promisify(exec);

try {
  const { stderr } = await execAsync(`ffmpeg -i ${audioFilePath} -b:a ${bitrate} ${transformedAudioPath}.${format}`);

  if (stderr) {
    console.warn('stderr:', stderr);
  }
} catch (err) {
  console.error('Error:', err.message);

  if (err.stderr) {
    console.error('stderr:', err.stderr);
  }
}

You can also use fluent-fmmpeg. It's the NPM package that wraps ugly CLI commands in a beautiful chain of functions. I know it's deprecated but it can still be useful for most operations. Here's how we use it:

await new Promise((resolve, reject) => {
  ffmpeg(audioFilePath)
    .toFormat(format) // Change format
    .audioBitrate(bitrate) // Change bitrate
    .save(transformedAudioPath)
    .on('end', () => {
      console.log('File has been transformed successfully');
      return resolve(transformedAudioPath);
    })
    .on('error', (error: Error) => {
      console.log('Failed to transform audio file: ', error.message);
      return reject(error);
    });
});

Simple and easy to use. No matter what are the input parameters, we still get the unified and predictable output. This helps us in further processing. E.g. we no need to worry if given format is supported by the browser or not. We can also degrade the audio quality to save some space on S3.

"But wait. You mentioned that FFmpeg is a CLI tool. Can we just install the library and expect it to work?" Well, unfortunately, it is not that easy. We still need to have FFmpeg installed on our system. But how can we do this? Do we need to put it in a ZIP with the Lambda code? This is where Lambda Layers comes in handy. This mechanism allows us to pack our dependencies into archives, which can then be used in our functions. We can include predefined dependencies, as well as include external tools. In our case, FFmpeg and FFprobe will be packaged in this way. We only need to zip the binaries, create the new layers and attach them to our Lambda function. We also need to remember to set the appropriate FFMPEG_PATH and FFPROBE_PATH values using Lambda environment variables.

From now on we can use FFmpeg CLI commands.

Metadata

The last element is the metadata needed for later filtering and searching of audio files, e.g. for frontend purposes. Here, I will use DynamoDB as a database, which provides very good scalability and on demand payment (only for used resources). During the entire flow, we will update the current state of file processing, which looks as follows:

And this is how it looks like in the code:

Create new metadata record

export const createAudioMetadataRecord = async (audioMetadata: AudioMetadata): Promise<void> => {
  const documentClient = initDocumentClient();
  const putCommand = new PutCommand({
    TableName: process.env.AUDIO_TABLE_NAME,
    Item: audioMetadata,
  });

  await documentClient.send(putCommand);
}

Update record with new status

  const updateCommand = new UpdateCommand({
    TableName: process.env.AUDIO_TABLE_NAME,
    Key: {
      id: audioId,
    },
    UpdateExpression: 'SET #status = :status',
    ExpressionAttributeNames: {
      '#status': 'status',
    },
    ExpressionAttributeValues: {
      ':status': 'UPLOADED' satisfies FileStatus,
    },
  });

Summary

The final architecture looks as follows:

Let's test it!
First, we need to generate new Presigned URL. We get it in the response body of our POST /api/files endpoint.

Then we use it in the PUT method. We can also leverage HTTP 301 code to automatically redirect after successful response. This starts the whole processing flow. We can get updated metadata with GET /api/files endpoint that lists all the uploaded files.

As you can see, even such a simple task should be well planned with attention to details such as scaling, duplicates or low operating cost of our solution. Of course, such architecture is only a base for more complex business cases, so I encourage you to experiment.