Supabase Storage is receiving a major upgrade, implementing many of the most requested features from our users: Resumable Uploads, Quality Filters, Next.js support, and WebP support.
The key feature: Resumable Uploads! With Resumable Uploads, you can continue uploading a file from where you left off, even if you lose internet connectivity or accidentally close your browser tab while uploading.
Resumable uploads divides the file into chucks before uploading them, emitting progress events during upload.
With this release, you can now upload files as large as 50GB! (Previously the limit was 5GB).
To build this feature, we implemented Postgres Advisory locks which solved some gnarly concurrency problems. We can now handle edge-cases, like two clients uploading to the same location. We’ll deep dive into how we implemented Advisory locks later in the post.
New features
Storage v3 introduces a number of new features.
More image transformations options
We introduced image resizing last Launch Week. This time, we’ve added the ability to specify quality and format filters when downloading your image. When you request images via the transform endpoint, by default we render it as Webp, if the client supports it.
supabase.storage.from('bucket').download('image.jpg', {
transform: {
width: 800,
height: 300,
quality: 75,
format: 'origin',
},
})
Next.js loader
You can serve images from Storage with a simple Next.js loader for the Image component. Check out our docs on how to get started.
// supabase-image-loader.js
const projectId = '<SUPABASE_PROJECT_ID>'
export default function supabaseLoader({ src, width, quality }) {
return `https://${projectId}.supabase.co/storage/v1/render/image/public/${src}?width=${width}&quality=${
quality || 75
}`
}
// nextjs.config.js
module.exports = {
images: {
loader: 'custom',
loaderFile: './supabase-image-loader.js',
},
}
// Using Next Image
import Image from 'next/image'
const MyImage = (props) => {
return <Image src="bucket/image.png" alt="Picture of the author" width={500} height={500} />
}
Presigned upload URLs
Authenticated users can now generate presigned URLs.
These URLs can then be shared with other users who can then upload to storage without further authorization. For example, you can generate a presigned URL on your server (ahem, Edge Function).
Shoutout to community members @abbit and @MagnusHJensen who implemented this feature on the Storage server and @Rawnly for the client library bindings 🎉.
// create a signed upload url
const filePath = 'users.txt'
const { token } = await storage.from(newBucketName).createSignedUploadUrl(filePath)
// this token can then be used to upload to storage
await storage.from(newBucketName).uploadToSignedUrl(filePath, token, file)
Size and file type limits per bucket
You can now restrict the size and type of objects on a per-bucket basis. These features make it easy to upload to Storage from the client directly, without requiring validation from an intermediary server.
For example, you can restrict your users a 1 MB and image/* files when uploading their profile images:
Deep Dive into Resumable Uploads
Let’s get into the nuts-and-bolts of how we implemented Resumable Uploads.
First, why do we need Resumable Uploads, when the HTTP protocol has a standard method for uploading files - multipart/form-data ? This approach works well for small files, since the file is streamed to the server in bytes over the network. For medium to large files this method becomes problematic, especially on spotty connections like mobile networks. Uploads that are interrupted need to be restarted from the beginning.
TUS - Resumable Protocol
We use S3 to store your files and it implements a proprietary protocol for resumable uploads. At Supabase, we support existing open source communities when possible and so, instead of exposing the S3 protocol to our users, we implemented TUS (historically an acronym for Transloadit Upload Server, later renamed to The Upload Server). TUS is an open protocol for resumable file uploads. By leveraging an open protocol, developers can use existing libraries with Supabase Storage.
TUS is a powerful protocol. It’s built on top of HTTP, making it easy to integrate your browser and mobile applications. Because of its open nature, a variety of powerful, drop-in clients and open-source libraries have been built around it. For example, at Supabase, we love Uppy.js, a multi-file uploader for TUS.
Using Uppy with Supabase Storage looks like this:
import { Uppy, Dashboard, Tus } from 'https://releases.transloadit.com/uppy/v3.6.1/uppy.min.mjs'
const token = 'anon-key'
const projectId = 'your-project-ref'
const bucketName = 'avatars'
const folderName = 'foldername'
const supabaseUploadURL = `https://${projectId}.supabase.co/storage/v1/upload/resumable`
var uppy = new Uppy()
.use(Dashboard, {
inline: true,
target: '#drag-drop-area',
showProgressDetails: true,
})
.use(Tus, {
endpoint: supabaseUploadURL,
headers: {
authorization: `Bearer ${token}`,
},
chunkSize: 6 * 1024 * 1024,
allowedMetaFields: ['bucketName', 'objectName', 'contentType', 'cacheControl'],
})
uppy.on('file-added', (file) => {
file.meta = {
...file.meta,
bucketName: bucketName,
objectName: folderName ? `${folderName}/${file.name}` : file.name,
contentType: file.type,
}
})
uppy.on('complete', (result) => {
console.log('Upload complete! We’ve uploaded these files:', result.successful)
})
And there you have it, with a few lines of code, you can support parallel, resumable uploads of multiple files, with progress events!
Implementing TUS inside Supabase Storage
There were a few technical challenges we faced while implementing TUS in Supabase Storage.
Storage is powered by our Storage-API service, a Node.js server that interfaces with different storage backends (like AWS S3). It is fully integrated with the Supabase ecosystem, making it easy to protect files with Postgres RLS policies.
To implement the TUS protocol, we use tus-node-server, which was recently ported to Typescript. It was only missing a few features we needed:
- Ability to limit the upload to files of a certain size
- Ability to run multiple instances of TUS (more on this later)
- Ability to expire upload URLs after a certain amount of time
We will be contributing these features back to TUS with discussions and PRs after Launch Week.
Scaling TUS
One of the biggest challenges we faced was the ability to scale TUS by running multiple instances of the server behind a load balancer. The protocol divides the file into chunks and sends it to any arbitrary server. Each chunk can be processed by a different server. Cases like these can lead to corrupted files with multiple servers trying to buffer the same file to S3 concurrently.
The TUS documentation gives 2 work-arounds:
- Use Sticky sessions to direct the client to the same server the upload was originally started.
- Implement some sort of distributed locking to ensure exclusive access to the storage backend.
Option 1 would have affected the even distribution of requests across servers. We decided to go with option 2 - Distributed Locking. Storage uses Postgres as a database, a queue, and now as a lock manager.
Enter Postgres Advisory Locks
Postgres advisory locks offer a way for defining locking behaviour of resources outside of the database. These are called advisory locks because Postgres does not enforce their use - it is up to the application to acquire and release the locks when accessing the shared resource. In our case, the shared resource is an object in S3. Advisory locks can be used to mediate concurrent operations to the same object.
const key = `/bucket-name/folder/bunny.jpg`
const hashedKey = hash(key)
await db.withTransaction(() => {
// try acquiring a transactional advisory lock
// these locks are automatically released at the end of every transaction
await db.run('SELECT pg_advisory_xact_lock(?)', hashedKey);
// the current server can upload to s3 at the given key
await uploadObject();
if (isLastChunk) {
// storage.objects stores the object metadata of all objects
// It doubles up as a way to enforce authorization.
// If a user is able to insert into this table, they can upload.
await db.run('insert into storage.objects(..) values(..)')
}
});
// the advisory lock is automatically released at this point
With advisory locks, we’ve been able to utilize Postgres as a key part of the Supabase Stack to solve difficult concurrency problems.
Roll out
Because this is a major update, we’re rolling it out gradually over the next month. You will receive a notification in your dashboard when the feature is available for project. Reach out to us if you want early access to this feature.
Coming up next
We’ve got an exciting roadmap for the next few Storage releases:
- Presigned upload URLs for TUS
- Increasing max file size limit to 500 GB
- Transform images stored outside Supabase Storage
- Smart CDN v2 with an even higher cache hit rate
Top comments (0)