DEV Community: Ajit Singh

Understanding TUS and imgProxy in supabse/storage

Ajit Singh — Wed, 15 Jan 2025 08:22:35 +0000

TUS and Imgproxy

Today we will learn at how external elements like TUS and Imgproxy are used in the Supabase Storage engine. TUS is used for resumable uploads and Imgproxy is used for image transformations. Now let's dive into the details.

1. TUS Protocol for Resumable Uploads

Purpose: The TUS protocol is implemented to handle resumable uploads, particularly useful for large files or unreliable network connections. This ensures that uploads can be paused and resumed without losing progress.
Key Use Cases:
- Large Files: When a client uploads large files through TUS, the upload is broken down into smaller, manageable chunks, allowing for resumable transfers.
- Unreliable Networks: It is used in the event of connectivity issues, as the upload can be continued after the network becomes available.
- Multipart Upload Abstraction: For large files, TUS acts as a high-level protocol abstraction over S3's multipart uploads.
Lifecycle:
1. Initialization (TUS Client):
  - A TUS client, such as tus-js-client, is used by the client application and it initiates an upload by making a POST request to the /upload/resumable endpoint.
  - The POST request includes a Upload-Length header to specify the file's total size and additional metadata about the file.
2. Create Request (/upload/resumable, HTTP Layer):
  - Extraction: The TUS middleware detects a new upload using the Upload-Length and Upload-Metadata headers in the request and calls the namingFunction, which does the following:
  - Parses the metadata from the header, including the bucketName and objectName.
  - Validates that the bucket and key are valid using mustBeValidBucketName and mustBeValidKey.
  - The system creates a new UploadId with tenantId, bucketName, objectName, and a version, which is a UUID generated on every request.
  - The new UploadId is encoded into a base64url string and is added to the response Location header to create the unique upload ID URL.
  - Logging: The TUS server logs when the upload begins, as well as the request headers and parameters using Pino.
  - Security: The request also checks if the user has the authorization to create this object and the request is authenticated.
3. Pre-Upload Checks (onCreate hook):
  - The onCreate hook (in src/http/routes/tus/lifecycle.ts) is called before the upload happens; it performs the following:
  - The TUS upload data is saved into the file system or S3.
  - Validates the content type if available.
  - If the max file size exceeds the configured value, then it aborts the upload.
  - The onCreate hook returns the headers for the responses.
4. Uploading Parts (PUT/PATCH /upload/resumable/{uploadId}, TUS Layer):
  - Request Check: Each individual PUT or PATCH request is checked against the onIncomingRequest middleware, which validates if:
  - The request is for a signed upload URL.
  - The request has valid authentication.
  - Validates the file size limit using the headers Upload-Offset, Upload-Length, or by checking the content-length header, also checking the configuration or tenant limit.
  - Data Handling: The TUS middleware (in @tus/server) handles each PUT/PATCH request to upload a part or a chunk of the file.
  - Data Storage:
  - If using S3Store, parts are uploaded to the S3 compatible object storage.
    - The UploadPartCommand is used to send upload part requests.
    - Tracing: Each of the calls in the S3Store has a trace using ClassInstrumentation.
  - If using FileStore, parts are stored locally in the server's file system.
  - Metrics: The S3UploadPart histogram is used to record the time taken to complete the upload process.
  - Metadata Update: If the used adapter is S3 storage, the updateMultipartUploadProgress is used in the database to persist information about the multipart upload state.
  - The uploadPartCopy is used when a copy operation is done, and the metadata attributes are used to ensure the file was properly copied.
  - Concurrency Control: To avoid concurrency issues during multi-part uploads, a mutex is created by the PgLocker class to handle the concurrent uploads and avoid race conditions.
  - When the lock is created, a pg_try_advisory_xact_lock is created using Postgres, which allows the backend to block and wait if a particular ID is being uploaded.
  - The advisory lock ensures that concurrent uploads to the same file are done one at a time.
  - Error Handling: If any errors occur during part uploads or on the mutex locking, appropriate errors are handled, and a 4xx response is sent back; the error is also logged.
5. Upload Completion (PUT /upload/resumable/{uploadId}, TUS Layer):
  - Once all parts of the file have been uploaded, the client sends a final PUT request which triggers the onUploadFinish hook.
  - Object Creation: The completeMultipartUpload from the storage backend is called to assemble the parts into the final object in the file system or S3.
  - Metadata Storage: The system saves the file metadata into the storage.objects table.
  - Tracing: A new span called Storage.completeUpload is created by the ClassInstrumentation plugin.
  - Webhook Trigger: The ObjectCreated event is fired using the queue.
6. Error Handling:
  - If there is any failure during any phase of the upload, the server will return an error to the client.
  - If a fatal error happens, the upload will be cleaned up using the ObjectAdminDelete event.
7. Cleanup: Once the upload is completed (either finished or aborted), temporary data or the upload folder will be cleaned up.
Asynchronous Nature: The TUS protocol is asynchronous and allows the server to handle large uploads concurrently. When an S3 upload finishes, the system relies on a background task to clean up any remaining temporary files.

2. Imgproxy for Image Transformations

Purpose: Imgproxy is an external service used for on-demand image processing and transformations. It allows the Supabase Storage system to serve resized, cropped, and optimized images without requiring pre-generated versions of each object.
Key Use Cases:
- Dynamic Image Resizing: When an image is requested with specified width and height parameters, imgproxy applies these transformations.
- Image Format Conversion: Imgproxy is used to convert images to optimized formats such as WebP or AVIF.
- Watermarks and Other Effects: Can handle other image transformations such as watermarks.
Lifecycle:
1. Image Request (HTTP Layer):
  - A client requests a particular image via the /render/image or a signed URL from the /render/sign endpoint with a variety of transformations specified via URL parameters.
  - Request Parameters: Query parameters such as width, height, and resize are parsed from the URL string by the ImageRenderer class.
2. Private Object URL Creation (Storage Layer):
  - The system calls the S3 storage backend to generate a signed presigned URL to allow imgproxy to read from the bucket.
  - Tracing: A new span named S3Backend.privateAssetUrl is created using the ClassInstrumentation plugin.
  - Performance: If there is a local file, this path is returned instead of the S3 one.
3. Imgproxy Request (HTTP Layer):
  - The system constructs a URL to the imgproxy service based on transformation parameters. The URL includes parameters to configure imgproxy to perform operations such as resizing, cropping, and format conversion, as well as the signed internal URL to fetch the original image data.
  - The HTTP client Axios is used to call imgproxy with the constructed URL.
  - Tracing: A new span named axios.get is created with all the requested parameters.
  - Request Timeout: The connection is configured to have a timeout to avoid long-running calls and also to make use of connection pooling.
  - Error Handling: In case of an error on the response, the body response is streamed and added as an error message.
4. Image Transformation (Imgproxy Service):
  - Imgproxy, using the signed URL, fetches the object and applies the requested transformations, storing it in memory before streaming it back.
5. Response Streaming (HTTP Layer):
  - The imgproxy response stream (with the transformed image) is streamed back to the client. If a response is not successful, an error is raised.
  - Caching: The caching headers such as cache-control and etag are extracted from the original request or the backend.
  - Tracing: The axios.get span will be closed, either with a status of OK or with an error.
Asynchronous Nature: The integration with imgproxy involves making an asynchronous network call, but the use of HTTP streams makes the process non-blocking.

Usage Details:

TUS Workflow:
1. A client sends a POST request to /upload/resumable to initiate a new upload with the total file size.
2. The server creates a TUS upload resource and responds with a unique upload ID (URL).
3. The client makes multiple PUT or PATCH requests to the Location URL returned in step 2, uploading the file in smaller parts.
4. The client makes a final PUT request to the Location URL; the server then stitches the parts into one final object in S3 or on disk.
Imgproxy Workflow:
1. A client requests an image through the /render/image endpoint, specifying transformation parameters in the query string.
2. The server generates a signed URL for access to the original image in object storage using privateAssetUrl.
3. The server calls imgproxy with the transformation options and the signed URL.
4. Imgproxy processes the image, and the server returns the processed image via a stream to the client.

Key Takeaways:

TUS for Resumable Uploads: This feature enhances file upload resilience and efficiency. It provides support for uploading big files and for unreliable connections.
Imgproxy for Dynamic Transformations: This service enables flexible image transformations without altering original files and without the need for pre-generated files, saving costs and reducing complexity.
Independent Services: Both TUS and Imgproxy are implemented in a way that these services are external and not a core dependency for the application.
Streaming Data: Streams are used to process large files efficiently and to have non-blocking I/O operations.
Tracing: Both of these steps are instrumented with OpenTelemetry and can be visualized by using a compatible backend.

Tenants in supabase/storage

Ajit Singh — Wed, 15 Jan 2025 08:21:58 +0000

Tenant Management in Supabase/Storage

Let's learn how tenant data is managed within this Supabase Storage repository, focusing on how it's kept separate and how they are set up in the multi-tenant environment.

Tenant Data Separation Lifecycle

This system implements a multi-tenant architecture, which means that a single instance of the application serves multiple tenants (organizations, projects) while ensuring data isolation and security. Here's the lifecycle of how tenant data is kept separate:

Tenant Creation (Admin API):

Admin Request: An administrator makes a request using the multi-tenant admin API to create a new tenant. This typically involves sending a POST request to /admin/tenants/{tenantId}.
- Authentication: The request requires a valid admin API Key, which is validated using the apiKey plugin.
- Tenant Data: The request body usually includes data such as:
  - tenantId: A unique identifier for the tenant (often a UUID).
  - anonKey: The anonymous public key for the tenant.
  - databaseUrl: A specific URL to be used for a particular tenant.
  - jwtSecret: A tenant-specific JWT secret to generate and validate JWTs.
  - serviceKey: A service key used for the tenant to bypass row-level security for database reads and writes.
  - features: Features for the tenant such as imageTransformation and s3Protocol.
- Database Entry: The request handler calls the multitenantKnex client and inserts a new record in the tenants table, located in the multitenant database. Note that the sensitive values are encrypted before being persisted in the database.
- Tracing: If tracing is enabled for the admin server, a trace will be created named tenants.create using the ClassInstrumentation plugin.

Tenant Request (HTTP Layer):

x-forwarded-host Header: When a request enters the main Storage service, the tenantId plugin checks for the x-forwarded-host header. This header indicates the specific tenant to whom the request belongs. If the header is not provided, a 400 error is returned.
- RegExp Extraction: The REQUEST_X_FORWARDED_HOST_REGEXP configuration is used to extract a valid tenant ID from the header; if the header doesn't match a regexp, a 400 error is thrown.
  - Tenant Id: The extracted tenant ID is stored in request.tenantId for use in subsequent operations.
  - Context: Every request has the tenantId available in the request object that can be used to retrieve the data for a particular tenant.
  - Database Connection: When a database connection is created, the multitenant database configuration is used by multitenantKnex, which in turn allows for database connection pooling at the application server level.
  - Tenant-specific database pool: Every request will create a tenant-specific database pool, which is also used for each database operation.
  - In case a database connection pool is set for a given tenant, each request for that tenant will use its own database connection pool.

Tenant Configuration Lookup (Internal Layer):

Cache: The getTenantConfig function first tries to read from a cache, identified using the tenantId from the request. If the entry is not found, it will continue to the next step.
- Database Fetch: The system uses the tenant ID to query the tenants table in the multi-tenant database and fetch the configuration for that tenant.
  - Mutual Exclusion: To prevent multiple simultaneous lookups that could lead to thundering herd issues, a mutex is implemented using the createMutexByKey function. This ensures that only one request can perform the lookup at a time, thereby reducing contention and improving efficiency.
- Data Decryption: The data, including the databaseURL, serviceKey, jwtSecret, etc., are decrypted using the configured encryption key. This ensures that no sensitive data is visible in the database.
- In-memory cache: The tenant config, once obtained, is saved in the in-memory cache for fast access in the next requests that come for the same tenant.
- Tracing: Every time the tenant is fetched, if tracing is enabled, a span named tenant.fetch is created with the tenantId.
- Pooling: The retrieved configuration is used to configure the database connection, or if set, the database connection pool.

Resource Handling with Tenant Specific Context (Storage, Database Layer):

Tenant Specific Connections: The created TenantConnection from getPostgresConnection sets up the connection with the tenant data and also the custom settings, which allow using RLS with the setScope method.
- RLS Authorization: Once an object is created or read in the storage layer, the underlying SQL calls will automatically respect RLS policies.
- Scoped Operations: The operations within Storage, ObjectStorage, and the internal database operate within the tenant's context using the tenantId, ensuring the correct credentials and settings are used for the given resources.

S3 Credentials Management
- S3 Credentials: Each tenant can also have a specific S3 key and secret, which is managed in the tenants_s3_credentials table.
- Unique Access Key: Each credential for the tenant is tied to a unique access key.
- Rotation: These credentials can be rotated when creating new credentials for a particular tenant.
- Access Restrictions: The service key or anon key cannot be used to sign a request to a particular tenant.
- Caching: The getS3CredentialsByAccessKey caches all the S3 credentials in memory for a specific amount of time.
- If there is an update or deletion, the cache is automatically invalidated via a PubSub listener in listenForTenantUpdate.
- Automatic Scope: The getS3CredentialsByAccessKey will also return a claims payload that can be used to create a scoped JWT token.

Key Elements for Data Isolation

Multi-Tenant Database: All tenant-specific configuration information such as database URLs, JWT secrets, and service keys is managed in a centralized multi-tenant database, separated from the actual storage data.
Request Context: The tenant ID and configuration are part of the request context, which is used for all subsequent operations.
In-Memory Cache: Frequent tenant configurations are cached in memory for quick retrieval.
Data Encryption: Sensitive data, such as passwords and secrets, are encrypted at rest in the multi-tenant database.

How It Works in Practice:

Tenant Setup: An administrator creates a new tenant using the multi-tenant API, which stores the tenant configurations.
Client Request: A client attempts to access a resource, providing a JWT or presigned S3 token in the Authorization header, or using an upload signed URL.
Tenant Identification: The system extracts the tenant ID from the request using the x-forwarded-host header sent by the load balancer.
Configuration Lookup: The getTenantConfig function retrieves the specific configuration for the tenant, including the database URL, secrets, etc.
Database Connection: A database connection is made using the tenant-specific configuration, including tenant-specific database pool options if any.
RLS Enforcement: Queries against the storage database use the custom functions auth.uid and auth.role using the extracted JWT from the header.
Response Generation: The system responds to the client with the requested data, ensuring data isolation and security at all stages.

This is how the multi-tenant application achieves strong data isolation.

Deployement in supabase storage

Ajit Singh — Wed, 15 Jan 2025 08:21:12 +0000

Supabase Storage how deployment works

Let's break down the deployment aspects of this Supabase Storage repository, focusing on how the various tools connect in the docker-compose.yml files, their interactions. We'll try to cover the full operations side of the repository.

Deployment Overview

The repository provides several Docker Compose files to manage different deployment scenarios, primarily focusing on:

docker-compose.yml: This file sets up a single-tenant environment with a PostgreSQL database, a connection pooler (pgBouncer), a MinIO S3-compatible storage, and the Storage API itself.
docker-compose-multi-tenant.yml: This file sets up a multi-tenant environment, adding a multi-tenant database, Supavisor (a connection pooler and proxy for multi-tenant Postgres), MinIO, and the Storage API (configured for multi-tenancy).
./.docker/docker-compose-infra.yml: Defines the basic infrastructure components shared between the single-tenant and multi-tenant setups.
./.docker/docker-compose-monitoring.yml: Defines the basic monitoring components shared between the single-tenant and multi-tenant setups.

Component Interaction in `docker-compose.yml` (Single-Tenant):

Here's a breakdown of the services defined in docker-compose.yml and their interactions:

storage (Supabase Storage API):

Image: supabase/storage-api:latest
Purpose: This is the core service that provides the Storage API, handling object storage logic, auth, etc.
Ports: Exposes port 5000 for incoming HTTP requests.
Dependencies: Depends on tenant_db, pg_bouncer, and minio_setup. It requires the tenant database to be ready and migrations to be run, also requires a bucket on the S3 compatible provider before starting; that's why minio_setup is a dependency here.

Environment Variables
- SERVER_PORT: 5000: the exposed port on the container
- AUTH_JWT_SECRET: JWT secret to validate requests
- AUTH_JWT_ALGORITHM: Algorithm to use with the JWT library
- DATABASE_URL: Connection URL to PostgreSQL
- DATABASE_POOL_URL: Connection URL to pgBouncer connection pooler
- DB_INSTALL_ROLES: true: Indicates if roles need to be installed on the database (if it is set to false, it needs to be managed outside of the application)
- STORAGE_BACKEND: s3: which type of backend to use (s3 or file)
- STORAGE_S3_BUCKET: Name of the S3 bucket to use
- STORAGE_S3_ENDPOINT: S3 endpoint to use
- STORAGE_S3_FORCE_PATH_STYLE: true: If true, uses the path instead of subdomains
- STORAGE_S3_REGION: S3 bucket region
- AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY: MinIO credentials
- UPLOAD_FILE_SIZE_LIMIT, UPLOAD_FILE_SIZE_LIMIT_STANDARD: limits for file uploads
- TUS_URL_PATH, TUS_URL_EXPIRY_MS: TUS settings
- IMAGE_TRANSFORMATION_ENABLED: "true": Enables or disables image transformations
- IMGPROXY_URL, IMGPROXY_REQUEST_TIMEOUT: Configuration for the imgproxy URL and timeout
- S3_PROTOCOL_ACCESS_KEY_ID and S3_PROTOCOL_ACCESS_KEY_SECRET: If the access key or secret are set on the request, those values will be used instead of the static AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY
- S3_PROTOCOL_ALLOWS_SERVICE_KEY_AS_SECRET: if true, allows using service keys as secrets
Functionality: This is the core of the storage engine; it uses various libraries from the repo, in particular:
- The database classes such as TenantConnection to connect to the database
- The Storage class to interact with the database and the underlying storage
- All the auth validations using jwt and apikey plugins
- Registers all the TUS, object, bucket, S3, and render HTTP endpoints

tenant_db (PostgreSQL Database):

Image: postgres:15
Purpose: Provides the PostgreSQL database that stores the buckets, objects, and file metadata
Ports: Exposes port 5432 for database access
Healthcheck: Check if the service is healthy using pg_isready
Environment:
- POSTGRES_DB, POSTGRES_USER, POSTGRES_PASSWORD: PostgreSQL credentials

pg_bouncer (pgBouncer Connection Pooler):

Image: bitnami/pgbouncer:latest
Purpose: Manages a pool of connections to the PostgreSQL database. This helps in preventing database connection overload when a large number of connections come at once and also allows for transaction-level pooling
Ports: Exposes port 6432 for connections from the storage service
Environment:
- POSTGRESQL_USERNAME, POSTGRESQL_HOST, POSTGRESQL_PASSWORD: PostgreSQL connection details
- PGBOUNCER_POOL_MODE: Sets the pool mode to transaction, which is recommended for serverless or edge environments
- PGBOUNCER_IGNORE_STARTUP_PARAMETERS: List of parameters to ignore on startup
- PGBOUNCER_STATS_USERS: Users that are able to query stats

minio (MinIO S3-Compatible Storage):

Image: minio/minio
Purpose: Provides a local S3-compatible object storage that the Storage API will use
Ports: Exposes ports 9000 (API) and 9001 (console)
Healthcheck: Check if the service is healthy via TCP connection
Environment:
- MINIO_ROOT_USER, MINIO_ROOT_PASSWORD: MinIO credentials
Command: server --console-address ":9001" /data starts the server in the defined port, exposing the console and also where the files are located

minio_setup (MinIO Setup):

Image: minio/mc
Purpose: Configures MinIO by setting up an alias and creating a bucket
Dependencies: Depends on minio
Entrypoint: Runs the mc command to create a bucket in MinIO

imgproxy:
- Image: darthsim/imgproxy
- Purpose: Handles image transformation using the imgproxy project
- Ports: Exposes port 8080 for incoming requests
- Volumes: The local folder data is mapped to the Docker folder /images/data
- Environment:
  - IMGPROXY_WRITE_TIMEOUT and IMGPROXY_READ_TIMEOUT: Sets the read and write timeout for image operations
  - IMGPROXY_REQUESTS_QUEUE_SIZE: The requests to the queue size
  - IMGPROXY_LOCAL_FILESYSTEM_ROOT: The root folder where images are stored
  - IMGPROXY_USE_ETAG: Enable the usage of ETag when available
  - IMGPROXY_ENABLE_WEBP_DETECTION: Enable detection of WebP format images

Component Interaction in `docker-compose-multi-tenant.yml` (Multi-Tenant):

This configuration includes all of the above plus additional services for multi-tenancy:

multitenant_db (PostgreSQL Database):

Image: postgres:15
Purpose: Manages the database schema for multiple tenants. It stores metadata and configuration information for each tenant
Ports: Exposes port 5433 for database access
Healthcheck: Check if the service is healthy using pg_isready
Environment:
- POSTGRES_DB, POSTGRES_USER, POSTGRES_PASSWORD: PostgreSQL credentials
Configs: Loads the SQL schema on /docker-entrypoint-initdb.d/init.sql

supavisor (Supavisor Connection Pooler and Proxy):

Image: supabase/supavisor:latest
Purpose: Acts as a connection pooler and proxy for routing database connections in a multi-tenant setup. It will have a pool of connections for each tenant
Ports: Exposes ports 4000 (API), 5452 (session), and 6543 (transaction)
Dependencies: multitenant_db, tenant_db
Healthcheck: Check if the service is healthy by pinging http://localhost:4000/api/health

Environment
- PORT, PROXY_PORT_SESSION, PROXY_PORT_TRANSACTION: Supavisor port configuration

- DATABASE_URL: Connection URL to the multi-tenant database

- CLUSTER_POSTGRES: "true": Enable clustered PostgreSQL configuration

- SECRET_KEY_BASE, VAULT_ENC_KEY, API_JWT_SECRET, METRICS_JWT_SECRET, REGION: Other environment variables needed for the application
Command: Migrates the Supavisor data structure with /app/bin/migrate and then
starts the server with /app/bin/server

supavisor_setup (Supavisor Setup):
- Image: supabase/supavisor:latest
- Purpose: Sets up an initial tenant in the Supavisor database
- Dependencies: Depends on supavisor
- Command: Creates the tenant inside the Supavisor database via an HTTP PUT request

The Role of MinIO, pgBouncer, and Supavisor

MinIO: Acts as a local object storage system that is used by the Storage Service; it allows easy setup and use without the need for a cloud provider. The Storage Service uses a local bucket supa-storage-bucket as a default to store files.
pgBouncer: Reduces the number of direct connections to the PostgreSQL database using connection pooling, which helps in preventing database overload and allows for maintaining a healthy database connection. It also enables transaction-level pooling. This is a single connection pooler, which is used only for the storage application.
Supavisor: Supavisor is a multi-tenant connection pooler that connects to the multi-tenant database and acts as a proxy for all the tenant databases. It uses a database to handle connection pooling and manage tenants.

Monitoring

The repository also contains additional configurations and tooling that are important for monitoring:

.docker/docker-compose-monitoring.yml: This file sets up monitoring tools such as:
- pg_bouncer_exporter: Exports metrics from pgBouncer
- postgres_exporter: Exports metrics from PostgreSQL
- prometheus: Collects all the metrics from services like storage, PostgreSQL, and Supavisor
- grafana: Used for visualizing collected data from Prometheus, and it has a set of dashboards that you can use for monitoring storage, PostgreSQL, and Supavisor
- jaeger: This is a collector for traces that are created on the application
- otel-collector: This is a collector for traces
.github/workflows/: This directory includes GitHub Actions workflow to automate build, test, release, deployment, and documentation tasks
.releaserc: This file defines release configurations for semantic-release

Deployment Process Summary

Local Development: For local development docker-compose-infra.yml is used to create all the shared services.
Production Deployment: In production, the storage service should be deployed in a cloud environment using a database and an S3-compatible object store.
- The Docker images used for the deployments are built using the workflow defined on .github/workflows/release.yml, which are then published to Docker Hub and GHCR.
- mirror.yml is used to mirror the newly published versions on different providers.
Multi-Tenant Configuration: When setting up multi-tenancy:
- Create a multitenant database.
- Set up a Supavisor instance.
- Use the admin API to create a new tenant specifying all configurations needed.
- The infra scripts can be used to restart and deploy the basic infrastructure.
Monitoring:
- The docker-compose-monitoring.yml is used to create the monitoring services.

How migrations work in supabase/storage

Ajit Singh — Wed, 15 Jan 2025 08:18:19 +0000

Supabse storage migrations

Let's learn about database migrations in this Supabase Storage repository, covering both single-tenant and multi-tenant setups, and exploring the three different migration strategies: "on request," "progressive," and "full fleet."

Single-Tenant Migrations

In a single-tenant setup, we have one admin application interacting with a single database. Here's how migrations are handled:

Migration Files:

Migrations are stored in the migrations/tenant directory.
- Each migration file is a SQL file and has a numbered prefix, e.g., 0001-initialmigration.sql, 0002-storage-schema.sql, that indicates in which order they should be applied.
  - They may contain operations to create tables, indices, change columns, or define functions, etc.
- internal/database/migrations/types.ts: This file defines a mapping of the different migration names with their associated id.

runMigrationsOnTenant Function (src/internal/database/migrations/migrate.ts):

Connection: This function creates a database connection by using the databaseUrl and creates a Client instance from pg library.
- connectAndMigrate Function: This helper function is invoked with the database configuration and path for the migrations, and it takes care of applying the migrations to that given database.
  - It reads all the files in a particular directory using loadMigrationFilesCached.
  - If the migrations table is not present in the database it creates it.
  - It reads the applied migrations using the migrations table.
    - If there are applied migrations, it refreshes the migration position. This is for backport compatibility.
  - It compares the applied migrations with the available migrations in a directory using validateMigrationHashes, and if there are some issues then it will throw an error and stop migrations unless dbRefreshMigrationHashesOnMismatch option is set to true, this will update migration hashes on the migrations table if there is a mismatch.
  - It will filter migrations that are pending to be run using filterMigrations.
  - For each pending migration, the system runs runMigration with the SQL file that will perform the schema updates.
  - It sets the search_path for every session, by using SET search_path TO ....

How It's Triggered:

When the main storage application starts up (src/start/server.ts) the function runMigrationsOnTenant is called using the databaseURL.
- On every request the migrations are verified to be up to date, if not, the db-init will trigger the method to make sure the database is up to date.
  - This is only enabled when the application is not in multi-tenant mode.

Multi-Tenant Migrations

In a multi-tenant setup, we have multiple tenants sharing the same application instance, each with its own data but potentially sharing some database schemas. Here's how migrations are handled:

Multi-Tenant Database Migrations:

Files: These migrations are stored in the migrations/multitenant directory and they have the same schema as single-tenant migrations.
- Purpose: These migrations are used for database changes that affect the multi-tenant database schema.
- Execution: The runMultitenantMigrations function is called when the server starts to apply changes to the database which is used to manage the different tenants (docker-compose-multi-tenant.yml).
- Connection: The function uses the multitenantKnex database client to connect to the multitenant database.

Tenant-Specific Migrations:
- Tenant migrations Table: Each tenant has their own private database schema and the system tracks migration state in a migrations table for every tenant.
- runMigrationsOnTenant Function:
  - It uses the databaseUrl from each tenant to connect to each database.
  - The process from applying those migrations is identical to the single tenant migrations.
- Tenant Tracking:
  - The tenants table in the multi-tenant database, holds the migrations_version and migrations_status which indicates up to what migration has been applied to a particular tenant and if the migration was successful or not.
  - The system uses the cursor_id to paginate through the available tenants.
  - The listTenantsToMigrate returns an async iterator that lists all the tenants where migrations_status is not set to "COMPLETED" or migrations_version is different than the latest migration.

Migration Strategies (Multi-Tenant)

In a multi-tenant environment, the repository uses three different strategies to apply pending migrations:

MultitenantMigrationStrategy.ON_REQUEST (On Request):

How it works: - Every request is checked using the db plugin for pending migrations using the tenant information present on the request. - The plugin uses the function hasMissingSyncMigration which checks if the migrations have been completed or not on the database, before proceeding with the request. - If the migration status is not set to COMPLETED or migrations_version is not the latest then all pending migrations will be executed for the specific tenant. This operation happens every time an application does a database query for this specific tenant. - To avoid concurrent migrations on the same tenant the migrations use createMutexByKey to execute the migration sequentially and only once per tenant at a time.
- Use Case: This is useful if you want every request to have all the migrations applied, and you want to prioritize safety instead of performance.

MultitenantMigrationStrategy.PROGRESSIVE (Progressive):

How it Works: - The progressiveMigrations class (in src/internal/database/migrations/progressive.ts) is configured to run in a loop at a specific interval checking for any tenants that need to be migrated, this check will be done in the background on a specific server. - If a new tenant is added, or there is a new migration available, then the tenant will be added to a list which is used by createJobs to perform the migrations asynchronously. - If a request is made and it has pending migrations, the plugin will trigger the migration immediately using the runMigrationsOnTenant and use updateTenantMigrationsState to set the migrations as completed, in this case, the migrations will block the requests, so you don't have a version inconsistency. - It checks on each request if there is a SYNC migration pending, if there is, it means that a migration will have to be run during the main thread. - The list is limited by the maxSize property which if reached will trigger the creation of migration jobs. - The background task will use the list of tenants and use the RunMigrationsOnTenants class to send messages to the queue for each tenant that has pending migrations. The queue workers will then pick those jobs and apply the required migrations. - In this case the main request will be blocked until the migrations are completed. - The system will set the status to failed if the migrations failed multiple times.
- Use Case: Suitable for applications that want to apply migrations in the background.

MultitenantMigrationStrategy.FULL_FLEET (Full Fleet):

How it Works: - The runMigrationsOnAllTenants is called asynchronously on start of the application. - It uses the advisory locks to prevent concurrent executions. - It iterates through the tenants that require a migration using listTenantsToMigrate and sends a queue message to the RunMigrationsOnTenants queue using RunMigrationsOnTenants.batchSend, so each tenant migration is performed by the queue workers asynchronously.
- Use Case: This method is useful if you have a large fleet of tenants, and you want the migrations to be handled asynchronously.

Code Flow

Initial Setup:
- On the application startup, depending if it is a multi-tenant environment or not, the migration strategy will be determined.
- Single tenant migrations are run when the server boots up using runMigrationsOnTenant
- Multi-tenant migration are run in the multi tenant database. runMultitenantMigrations.
- If the strategy is set to PROGRESSIVE or FULL_FLEET, an async migration process will be started in the background using startAsyncMigrations, to continue to migrate all the tenants while the application is running.
Request Processing (Multi-Tenant):
- When a request is made, depending on the selected strategy, one of three possible approaches to trigger the migrations is used.
- If migrations are run on request, then the request processing will be blocked while migrations are being executed.
- If migrations are run in a PROGRESSIVE mode, and there is a pending ---SYNC--- migration, then the requests are also blocked until migrations are done, else they will be queued to run in background.
- The system uses the tenantId to verify if migrations have been done in the current tenant.
- If FULL_FLEET is being used, then a queue is used to migrate the tenants in background.
Database Updates: All changes are performed to the specific tenant's database, and the migrations info on the tenants table are updated.
Async Process: In the progressive case, the background process will send messages to the queue so that worker instances can pick them up.

Key Takeaways

Single vs. Multi-Tenant: Single-tenant deployments use a straightforward approach, running migrations on server start. Multi-tenant deployments are more flexible, allowing migrations on each request, progressively, or in full fleet.
Flexibility and Control: Three different types of migration strategies are offered to fit different environments.
Data Integrity: The use of version tracking helps ensure the integrity of the schema migrations.
Automated Migration: This system uses a combination of row level security and migrations to enforce the structure of the data.

This should provide a solid understanding of how migrations are handled in single and multi-tenant modes.

Tracing and Metrics in supabase/storage

Ajit Singh — Tue, 14 Jan 2025 07:59:35 +0000

Tracing and Metrics

Lets learn more about how tracing and metrics are collected and used within the system. From a request perspectivewe will check they are added and implemented.

Tracing Lifecycle

Tracing is used to observe the path a request takes through the system, providing a holistic view of the processing flow, including execution times, dependencies, and errors along the way. Here’s a detailed breakdown:

Request Ingress (HTTP Layer):
- Instrumentation: OpenTelemetry (OTEL) HttpInstrumentation automatically creates a new root span when a request comes in. The span's name is http.request. The span's context (traceId, spanId) can be accessed through the trace.getActiveSpan() function.
- Attributes: The instrumentation also sets relevant attributes to the span including the tenant id (if available), method, url, status code, route, headers, and more, based on applyCustomAttributesOnSpan, headersToSpanAttributes configuration in the HttpInstrumentation.
- Logging: The logRequest plugin is executed, capturing the request information in a log line.
- Context Propagation: The span's context (including traceId and spanId) is attached to the current asynchronous execution context, ensuring consistent propagation to downstream operations.
Authentication and Authorization (HTTP Layer):
- Plugin: The jwt plugin is activated.
- Span Creation: The ClassInstrumentation plugin detects jwt.verify calls, creates, and ends a new span named jwt.verify, which represents authentication.
- Span Attributes: The span has metadata such as the role that comes from the payload.
Database Operations (Internal Layer):
- Plugin: The db plugin initializes the database connection pool and retrieves a connection.
- Span Creation: The ClassInstrumentation plugin detects StorageKnexDB.runQuery calls and creates and ends a new span named StorageKnexDB.runQuery.{{queryName}}.
- Span Attributes: The spans store the query name if present.
- Context Propagation: The database query is instrumented using @opentelemetry/instrumentation-knex, and any calls to the database or database pool will be automatically linked to the main trace context.
Business Logic (Storage Layer):
- Span Creation: When Storage, ObjectStorage, Uploader, etc. methods are called, ClassInstrumentation creates a span such as Storage.createBucket or Uploader.upload, using the methodsToInstrument configuration.
- Span Attributes: Span's name and attributes are configured by setName and setAttributes functions when available.
- Chaining: If nested calls within the same trace are made, those spans will automatically be children of the parent operation; this also happens with database calls using the knex instrumentation.
Backend Interactions (Storage/Backend Layer)
- Span Creation: When a function like S3Backend.getObject or S3Backend.uploadObject is called, a new span with the name S3Backend.getObject or S3Backend.uploadObject is created by ClassInstrumentation, respectively.
- Span Attributes: Span contains attributes such as operation: command.constructor.name.
- Context Propagation: When an API call is made using aws-sdk, the request is automatically added to the OTEL context to ensure all spans are connected; this happens because of @opentelemetry/instrumentation-aws-sdk instrumentation.
Async Operations:
- Context Propagation: If async operations are performed, and a new span is to be created, the context should be carried forward using the trace.getTracer().startActiveSpan function to create and automatically activate the span, so that the new span will correctly be associated with the request.
- Span Attributes: It might contain attributes depending on what method is calling it.
Response Phase (HTTP Layer):
- Plugin: traceServerTime is used to measure the server time it took to complete the response, capturing time spent in queue, database, HTTP operations, etc.
- Span Attributes: If the tracing mode is set to debug, spans collected using the TraceCollector are serialized as JSON and added as a stream attribute to the main HTTP span.
- Span End: All spans created using the active context, including the root http.request span, are ended. This finalizes the span, collects metrics, and prepares it for export.
- Logging: The logRequest plugin logs the request, including the status code and response time, and the serverTimes that were captured with the traceServerTime plugin.
Exporting Spans:
- OTLP Exporter: If configured, the OpenTelemetry Collector BatchSpanProcessor batches up the created spans using the OTLPTraceExporter and sends them to the OTEL endpoint.
  - This is done asynchronously in the background.

Metrics Collection

Metrics provide a numerical representation of system behavior, such as request rates, duration, etc. This system exposes metrics via a Prometheus endpoint, and here's how they are collected:

Request Level (HTTP Layer):
- Counters: The fastify-metrics plugin collects HTTP request-related metrics such as request counts, duration, and error counts. It also stores the request data into the storage_api_http_request_duration_seconds metrics.
- Labels: Metrics collected by the fastify-metrics plugin include method, route, and status_code.
- Label Aggregation: All Prometheus labels for HTTP are stored in fastify-metrics as tags.
Database Operations (Internal Layer):
- Histograms: The DbQueryPerformance histogram records the time it takes for a database query to complete. It captures the time spent waiting for connection and the database query time as well as labels region and the method name, stored as name.
- Connections: The metrics DbActivePool and DbActiveConnection track the pool connection counts and active connections.
S3 Operations (Storage/Backend Layer):
- Histograms: The S3UploadPart histogram records the time it takes to upload a part of a large file to the object storage service. It has one label, region.
Uploads:
- Gauges: FileUploadStarted is incremented when the file upload process starts. FileUploadedSuccess is incremented when an upload completes successfully.
- Labels: The upload-related metrics include labels for region and upload type (standard or multipart).
Queue Operations (Internal Layer):
- Histograms: QueueJobSchedulingTime records the time it took to schedule the message to the queue, labeled with name, which is usually the queue name and region.
- Gauges: QueueJobScheduled for the messages scheduled to be processed by the queue, labeled with name and region, QueueJobCompleted for the number of completed messages, QueueJobRetryFailed for the number of failed retries on each message, and QueueJobError, which is the total count of errored messages. The labels used here are the queue names and region.
HTTP Agent Metrics
- Gauges: HttpPoolSocketsGauge for the number of active sockets, HttpPoolFreeSocketsGauge for the number of free sockets, HttpPoolPendingRequestsGauge for the pending requests, HttpPoolErrorGauge for the errors, each one of them having name, region, protocol, and type as labels.
Supavisor Metrics
- Custom Exporter: The Supavisor has a custom Prometheus exporter that collects information about pool sizes, tenant status, connected clients, etc. These are collected by the Prometheus config file.

Key Takeaways:

End-to-End Visibility: Tracing provides a complete view of the request lifecycle, including HTTP, DB, and file I/O operations.
Resource-Specific Metrics: Metrics provide an overview of request performance with different labels.
Integration with OpenTelemetry: The use of OpenTelemetry allows traces to be sent to observability backends.
Integration with Prometheus: The usage of Prometheus makes it easy to collect and visualize metrics.

Understanding Storage backends in supabase/storage

Ajit Singh — Tue, 14 Jan 2025 07:24:00 +0000

Storage Backends

A storage backend is an abstraction layer. Instead of directly interacting with specific storage systems (like a local file system, AWS S3, Google Cloud Storage, etc.), the repository interacts with a generic interface. This interface defines a set of operations (like getObject, putObject, deleteObject, etc.) that any conforming storage backend must implement.

There are two storage backends used in this Supabase Storage repository. They are FileBackend and S3Backend. lets learn more about them and how they interact with the file system and the database, and the key differences between them.

1. `FileBackend` Storage Adapter

Purpose: The FileBackend adapter is designed to store and retrieve data using the local file system.
Key Features:
- Local File Storage: Files are stored directly on the server's disk, organized within a directory structure based on the tenant, bucket, key (object path), and version information.
- Metadata Storage: The file metadata is stored within the extended file attributes of the file system. Extended attributes are part of the file system and are a better approach than using a metadata file since it avoids having multiple files in the same location for just the file and the metadata.
- Direct File Access: Operations read and write to disk directly, without any intermediate layer or external service.
- Multi-Part Uploads: Creating parts in a temporary directory and then concatenating them into a single file when the upload is complete.
Implementation Details:
- Initialization: The constructor fetches the storage path from the configuration or from the FILE_STORAGE_BACKEND_PATH env variable.
- getObject Method:
- It constructs the file path using the bucket name, object key, and version.
- It reads the file from the disk using fs.createReadStream.
- It retrieves metadata (content-type, cache-control) from the extended file attributes, using the fs-xattr package to get those values.
- If the request is using range headers, it will stream data from disk at a specified range.
- Returns a stream along with the file metadata.
- uploadObject Method:
- The system will ensure a specific folder exists using fs-extra.ensureFile.
- Constructs the file path from given information.
- Writes the incoming stream to the file system using fs.createWriteStream.
- Sets metadata on the disk using the xattr package.
- Returns the metadata by reading using the headObject method.
- deleteObject Method:
- Constructs the file path using the bucket name, object key, and version.
- Uses fs.remove to delete the file from the file system.
- copyObject Method:
- Constructs source and destination file paths.
- Uses fs.copyFile to copy the file and ensures the folder where the files are being copied exists.
- It copies the metadata information from extended attributes.
- deleteObjects Method:
- Uses fs.rm to recursively remove folders and files with the given prefixes.
- headObject Method:
- Constructs the file path based on the request.
- Uses fs.stat to retrieve file system metadata.
- Reads file metadata using the xattr package.
- Calculates the checksum using the md5-file package.
- Returns relevant file metadata (size, content-type, cache-control, etag, last modified date).
- createMultiPartUpload: Creates a folder where the file parts will be stored for multipart uploads, and a metadata file with the extra configurations.
- uploadPart: Creates a file part where the data is saved into, and sets the Etag metadata attribute.
- completeMultipartUpload: Once all file parts are available, it concatenates all the files together using multistream and calls the uploadObject method to finalize the upload. It also removes all the temporary file parts.
- abortMultipartUpload: Removes the folder where the parts are stored.
- uploadPartCopy: Copies parts from already uploaded files and stores them into a newly created temporary file that it later uses in completeMultipartUpload.
- privateAssetUrl Method:
- Returns a special local file path for internal processing purposes.
- Dependencies: Uses packages like fs-extra, path, md5-file, fs-xattr, and multistream.
Interaction with the Database:
- The FileBackend adapter uses the database to retrieve tenant and bucket information. All file operations are performed on the disk and data is stored using StorageKnexDB.
Pros:
- Simplicity: It's easy to set up and reason about; it's simple to debug since everything lives on the file system.
- No External Dependency: It does not rely on an external service and can be started quickly on a local environment without the need for external dependencies.
- Lower Latency: File reads and writes are performed locally, avoiding extra network calls.
Cons:
- Not Scalable: It doesn't scale well beyond a single server.
- Limited Reliability: Data is at risk if the server is damaged.
- Inadequate for Production: It is not suitable for production environments if they need reliability and high availability.

2. `S3Backend` Storage Adapter

Purpose: The S3Backend adapter interacts with an S3-compatible object storage service (like AWS S3, MinIO), abstracting away the specifics of the API. This is the preferred implementation if you want to achieve better scalability and high availability.
Key Features:
- Remote Object Storage: Uses an S3-compatible object store for the long-term storage of object data.
- AWS SDK: It uses the AWS SDK for NodeJS to interact with S3 services.
- Authentication & Authorization: It uses AWS SDK's authentication mechanisms based on access keys, secret keys, session tokens, and policies.
- Multi-Part Uploads: It supports multi-part uploads for large files using uploadPart and createMultiPartUpload operations from the AWS SDK, and also has support for copying parts using uploadPartCopy.
Implementation Details:
- Initialization: The constructor gets configuration from the environment, such as storage bucket name, access key, secret key, region, and endpoint. The constructor also creates a custom HTTP agent and monitors it if tracing is enabled to collect HTTP socket usage information.
- getObject Method:
- Constructs an AWS GetObjectCommand using the provided bucket name and key (path).
- Uses the AWS SDK to send this command to the S3 endpoint.
- The response, including body and metadata, is returned. It also adds all metadata from the S3 response to the metadata object. If a range header is provided, it gets a partial response using ranged requests.
- uploadObject Method:
- Constructs an AWS PutObjectCommand with the request body and metadata.
- If tracing is enabled, it uses the monitorStream function to keep track of upload speeds. If the file is larger than the allowed uploadFileSizeLimit, the upload will be aborted.
- Returns the object metadata.
- deleteObject Method:
- Constructs a DeleteObjectCommand using bucket name and key.
- Sends the request to S3 to delete the object.
- copyObject Method:
- Constructs a CopyObjectCommand using bucket name, source key, and destination key.
- Uses the AWS SDK to send the copy command.
- deleteObjects Method:
- Constructs a DeleteObjectsCommand using bucket name and keys.
- Sends the request to S3 to delete multiple objects at once.
- Returns a list of deleted or failed delete operations.
- headObject Method:
- Constructs an AWS HeadObjectCommand using the provided bucket name and key.
- Returns metadata information from the AWS S3 object.
- privateAssetUrl Method:
- Creates a private URL using the getSignedUrl function.
- createMultiPartUpload: Creates a multi-part upload using the CreateMultipartUploadCommand and returns the UploadId.
- uploadPart: Creates a part of the upload using UploadPartCommand and returns the eTag of the upload.
- completeMultipartUpload: Commits a multipart upload by using CompleteMultipartUploadCommand, which expects the part information. If no parts are provided, it will fetch the list of parts before completing the upload.
- abortMultipartUpload: Aborts an upload using AbortMultipartUploadCommand.
- uploadPartCopy: Copies a specific byte range of an existing object and uploads it as part of a multipart upload using UploadPartCopyCommand.
- Dependencies: Depends on the @aws-sdk/client-s3, @aws-sdk/lib-storage, @aws-sdk/s3-request-presigner, and @smithy/node-http-handler packages for interacting with S3 services, and also @internal/http and @internal/streams for stream monitoring and handling.
Interaction with the Database:
- Like the FileBackend, the S3Backend doesn't interact directly with the database; instead, it uses the StorageKnexDB class to manage the database operations. It only saves metadata information about the object in the database, such as size, mime-type, date of upload, etc.
Pros:
- Scalability and High Availability: Leverages the scalability and reliability of the S3 service.
- Durability and Redundancy: Data is stored across multiple data centers for redundancy and durability.
- Better for Production: Production applications should use an S3-compatible backend.
Cons:
- Complexity: Requires configuration with an S3 provider.
- Latency: Might be prone to higher latency as every request goes through the network.

Key Differences Summarized

Feature	`FileBackend`	`S3Backend`
Storage Location	Local file system	Remote object storage (S3-compatible)
Scalability	Not scalable	Highly scalable
Reliability	Limited	High
Complexity	Simpler to set up	Requires S3 API configuration
Data Handling	Direct file I/O	API calls via AWS SDK
Metadata	Extended file attributes	S3 metadata
Ideal Use Cases	Development, testing, local use if you are self hosting.	Production, scalable, and reliable storage
Authentication	No authentication	Supports AWS S3 authentication
Instrumentation	Limited	Support for custom HTTP agent and tracing

Key Takeaways

FileBackend is for local development if you do self hosting or simple scenarios, while S3Backend is for production usage or wherever scalability, high availability, or data durability is required. The way both adapters interact with the database is similar by using StorageKnexDB, as the storage layer acts as an abstraction between those adapters and the data persistence layer. The adapters just focus on reading and writing data on the storage backend.

Auth in Supabase storage

Ajit Singh — Tue, 14 Jan 2025 06:25:52 +0000

Authentication and Authorization

Now we will go through the authentication and authorization lifecycle. How authorization and authentication work in the storage repository. We will understand the various parts involved in authentication and authorization and how they work together.

Authentication and Authorization Components

The Supabase storage repository employs a combination of JWTs (JSON Web Tokens) and row-level security (RLS) policies to ensure that only authorized users and services can access resources. It uses JWT for authentication and RLS for authorization. Here's a step-by-step breakdown:

Authentication:

JWT Acquisition: Clients obtain JWTs through the Supabase Auth service or through the storage service for a signed upload. These JWTs contain claims that specify the user's identity, roles, and permissions. With these JWTs, the client can access the storage service. Sometimes, when signed URLs are generated by the storage service, the JWT also contains the resource to which it has access.
Request Headers: Clients making requests to the Storage API include the JWT in the Authorization header, typically using the Bearer <token> format (or via query params for upload signed URLs).
JWT Extraction (HTTP Layer):
* The auth-jwt plugin extracts the JWT from the request's Authorization header or query param.
* It removes the Bearer prefix if present.
* The extracted JWT is stored in request.jwt.
- JWT Verification (Internal Layer):
  - The verifyJWT function from src/internal/auth/jwt.ts is called.
  - It uses the server's JWT secret or JWKS to verify the signature and validity of the JWT.
  - If signature validation fails, a 401 error is returned.
- Retrieving the Key for Signature Verification:
  - The function getJWTVerificationKey determines which key to use for verification based on the kid on the token or the static secret.
  - If the alg header from the token is using RSA or ECC, the key will be determined using kty and kid, or the static secret will be returned.
  - If there is an issue while extracting the public key from JWKS, the system will fall back to use the configured static key if the algorithm is HS256 or fail.
- JWT Payload Parsing: If the JWT is valid, the payload is parsed and saved in request.jwtPayload. The role key is assigned to request.jwtPayload.role if available.
- Request Decoration: If the JWT is valid, the system also sets request.isAuthenticated to true.
  - If an invalid or no JWT is provided but the route has the allowInvalidJwt option, request.jwtPayload and request.isAuthenticated are set to anon and false, respectively.
- Authorization Through RLS: The RLS will check the authorization on specific database queries as explained in the next section.

Authorization and Row Level Security (RLS):

Database Queries: When performing database operations (reading, creating, updating, deleting records), queries go through RLS policies.
- RLS Policies:
  - RLS policies are defined in the database using PostgreSQL's row-level security mechanism. These policies are configured by using the storage.install_roles flag; the roles that you set when running the migrations are set on the database as well.
  - RLS policies define rules for which users can access which rows of the database. These policies are based on the context of the request, such as auth.uid() (user ID) and auth.role() (user role) claims inside a JWT.
  - For example, a simple policy can be written as USING(owner = auth.uid()), which means only the objects with the same owner can be viewed, modified, or deleted.
- Dynamic RLS: The system will extract the current authenticated user information from the JWT token and add it to the current database context. The RLS policy will then use these values to validate the user permissions against the database.
- Enforcement: The database automatically applies the RLS policies on queries and operations to restrict data access.
- Error Handling: If an RLS policy violation occurs, PostgreSQL returns an error. The application catches this error using the DatabaseError error handler and returns a 403 (Unauthorized) response.
- Bypass with Service Key: For server operations, the system uses the service key that is generated and passed on the JWT token; this can be used to bypass any RLS rules in the database.
- Scope Setting: Before each database query, using the TenantConnection.setScope, the system sets the required options to the current_setting, allowing RLS to extract the required information from it.
- Tracing: If tracing is enabled, a span named knex.query is created using the @opentelemetry/instrumentation-knex, which shows the database query that was performed.

Presigned S3 URLs
- The presigned S3 URL follows the same flow as a normal GET object request.
- The URL is signed on the server side to allow access to the protected resources; it is used when uploading files or accessing private files using the S3 protocol.
  - signJWT is used to sign the URL using the server secrets.
- The authentication process happens using the x-signature header, which has the value of the signed JWT.
  - The verifyObjectSignature will verify if the provided token is valid using the verifyJWT function, and the function will check if the URL on the JWT matches the route.
Admin API Key:
- Any request to the /admin routes needs to send an apikey in the header to pass the validation; if this header is missing or invalid, the request will be refused.

Role of Schemas in Validation

JSON Schemas are used here to validate the request and response data.

Request Validation:

Definition: JSON Schema is used to define the structure and data types of incoming HTTP requests (both params, body, and headers).
- Enforcement: Fastify uses the ajv library to validate all incoming HTTP requests against the defined schemas before the route handler is called.
- Error Handling: If a request doesn't conform to the schema, Fastify automatically returns a 400 error with information about the validation failure.

Data Structure:
- Data Definition: Schemas also help define the properties of the returned objects, which are later used to define interfaces using the json-schema-to-ts library such as Obj, Bucket, UploadMetadata, and more.
Metadata Validation:
- User metadata is validated as a generic JSONB, so it doesn't have specific validation of its content to avoid having schema errors during the upload, as the content can be anything.

How a request is authenticated and authorized

Client Request: A client attempts to upload a file, create a bucket, download an object, or any other action.
JWT Verification: The server verifies the client's JWT, extracting authentication information.
Schema Validation: The server validates the client's request payload against the API schemas.
Database Interaction: If authentication and validation are successful, a database transaction is created, and the data is sent to the database.
RLS Enforcement: The database applies row-level security policies using data extracted from the JWT, restricting access to resources.
Operation Execution: If authorization is successful, then the operation is executed.
Response Generation: The system returns a response after completing all validation, auth, and permission steps.

Benefits

Improved Data Integrity: Schemas enforce consistency and correctness of input data, preventing invalid data from entering the system.
Declarative Approach: Using schemas in code makes it easier to see what the expected parameters are for every API call.
Developer Experience: The process provides a better developer experience by adding type safety and auto-validation features.
Extensible: Since the system only works on two levels—one is JWT and the other is RLS—any system that uses JWT and RLS can be used with the storage repository. So, you only need JWT and Postgres to use the storage repository.

This ends our exploration of the auth/authentication, RLS, and schema workflow. You should now have a clear view of how those pieces play an important role in every request, making the Supabase Storage Engine a reliable and secure system. Let me know if you have more questions or if you want to explore any topic in more detail.

Overview of supabase storage repository

Ajit Singh — Tue, 14 Jan 2025 04:37:28 +0000

Supabase storage Repository Overview

To understand the supabase storage repository, we need to understand the various components and how they interact with each other. For that we will break down how the various components in this Supabase Storage repository interact with each other. It has many components, but I first list and discuss the major ones:

Core Concepts:

Storage Backend: This is the service, responsible for managing object storage. It handles 2 storage backends (S3 and local filesystem), manages metadata in a database, enforces access control (RLS), and provides APIs for clients.
Fastify: This is the web framework used to create the HTTP API for the Storage service.
PostgreSQL: It is the primary data storage for metadata (objects, buckets, user defined metadata, multipart uploads, etc..)
S3: It is one of the supported storage backends where the actual files are stored in object stores like AWS S3, Digital Ocean Spaces, Cloudflare R2, etc..
TUS: It is the protocol implemented for resumable uploads.
Image Proxy ([Imgproxy)[]: An external service used for transforming images (resizing, formatting).
Metrics: Prometheus used for tracking the performance of the application and monitoring all operations.
Authentication/Authorization: Enforces access control using JWTs and Postgres Row Level Security (RLS).
Tenants: Used in multi-tenant configurations to isolate data and configurations for different customers. This is how supabase manages multiple clients in a single instance.
Tracing: OpenTelemetry (OTel) for distributed tracing, used to debug performance issues.
Streaming & Events: Streams and message queue systems (PgBoss, pubsub) for asynchronous operations, creating, and reacting to events (webhooks).
Schemas: JSON schemas for request and response validation.

Interaction from a request lifecycle perspective

Here we will take a look at how the various components interact with each other when a request is made to the storage service so that we can understand the flow of data and how the various components are used to perform the request and interact with the request.

Interactions in Detail:

Client Request:

A client sends an HTTP request to the Storage API (using Fastify).
- The request can be for various operations like:
  - Creating a bucket.
  - Uploading/downloading objects.
  - Transforming images.
  - Listing objects in a bucket.
  - Initiating or completing a TUS upload.
  - S3 protocol operations

Authentication and Authorization:

JWT Authentication: The jwt plugin verifies the request's Authorization header for a valid JWT.
- Tenant ID Extraction: The tenant-id plugin extracts the tenant ID from the subdomain of the x-forwarded-host header (in multi-tenant setups) or from an environment variable for single-tenant.
- API Key Authentication: Admin requests (e.g. multi-tenant management) use the apikey plugin. Which checks for the apikey header in the request.
- Request Context: After the JWT validation, it adds tenant specific information to the request object.
  - request.tenantId is set with the extracted or default tenant ID.
  - request.isAuthenticated is set to true if the JWT is valid, false otherwise.
  - request.jwtPayload contains the decoded JWT claims.
  - request.owner contains the subject (sub) claim from the JWT.

Request Handling and Validation:

Fastify Routing: Fastify routes the request to the appropriate handler function.
- Schema Validation: Each route is associated with a JSON schema. The body, params, and querystring of the request are validated using ajv against the defined schema, to ensure that request are proper.
- Plugins: Several plugins are attached to the request object to perform certain action.
  - db: It's an authentication specific connection to the database, that has all necessary information to be able to set the correct role in Postgres.This is used to perform all the database operations and helps in managing RLS(Row Level Security) for the tenant.
  - storage: Instance of the storage class with all the required backend/database specific instance of the classes.
  - metrics: Collects request level metrics like number of requests, timing to provide performance data on the application.

Storage Layer Interaction:

Storage class: The storage class is an orchestrator for all operations, and interacts with backend and database abstractions to fetch the data or perform an action.
- Object Storage:
  - S3 Backend: When the storage configuration is S3 then storage operations will be interacting with an S3 compatible object store. The s3 plugin creates a new S3 client for each request, with the tenant specific settings (region, endpoint etc..).
  - File Backend: When the storage configuration is local then the storage operations will be interacted with a local disk to perform all the file operation.
  - Upload Class: It handles all the multipart or single part upload to the backend, this class also handles the logic when upload limits are defined in the bucket configuration.
- Database Interaction:
  - The db plugin creates an authenticated database connection (TenantConnection) using knex, allowing for database operations.
  - Row-Level Security (RLS) is used to enforce access control on the database level, based on JWT claims and the tenant ID.

Image Transformation:

If the request is for image transformation, the ImageRenderer(This is a class that handles the creation of actual image) is used, and a secure URL is generated for image proxy to fetch the image from object stores and then transform it as requested.
- Rate limiting is applied to prevent abuse of the image transformation service when enabled.

TUS Protocol

When the TUS protocol is used to upload files, the request is routed to TusServer, this manages the different operation in the TUS protocol and orchestrates all operations * FileStore or S3Store manages the storage of the upload chunks (when uploading resumable files) * PgLock manages the concurrency of upload operations. This makes it such that 2 users are not performing operations on a file at the same time. * AlsMemoryKV manages the metadata storage in the TUS context. This is an async local storage instance that TUS uses to make resumable uploads.

Event System
- When an operation succeeds, a queue message might be generated. Events are used for the following operations. This queue is made in Postgres using PgBoss.
- Webhooks: Used to notify external services about file events (upload/delete).
- Migrations: Handles async migrations of tenant databases, when new migrations are available.
- Object Admin Delete: Asynchronously deletes the previous versions of an object when it gets updated.
Asynchronous Operations:
- Some operations like webhooks, background migrations, file processing, and deletion can be processed asynchronously through the queue system.
- PgBoss: Used as the queue mechanism for asynchronous operations, persisting tasks to the database.
  - RunMigrationsOnTenants queue handles the migrations of tenant databases.
  - ObjectAdminDelete queue handles the deletion of an object.
  - Webhook queue handles the sending of webhooks to external services.
- Postgres Pubsub: For real-time notifications, used for cache invalidation when the multi-tenant setup is in use. This uses node EventEmitter to create the pubsub system.
  - Invalidate the tenant config cache when the tenant information is changed or deleted.
  - Invalidate the S3 credential cache when the credentials are changed or deleted.
Metrics and Monitoring:

Prometheus: Used for gathering metrics.
- fastify-metrics: collects http requests.
- pino-logflare: is used as a custom pino logging strategy to push logs to logflare.
- Custom metrics are exposed through the /metrics endpoint to track database pool usage, queue sizes, s3 object transfers, http request latencies etc...

Tracing:
- OpenTelemetry: Used to trace the execution of requests across different components.
- Spans are added to method calls for a better way to track executions.
- When debug or logs tracing level is set, OTel data is added to the logs.
- The OTel collector collects traces in various formats, that are exported to Jaeger or other external consumers.

Data Flow Summary:

Request: Client makes an HTTP request.
Authentication: JWT, API Key, and tenant extraction.
Authorization: RLS and other code based authorization.
Data Access: Data is read/written from/to the backend or database.
- If files are needed to be transferred then files are downloaded from S3 or local system.
- If only metadata is needed, then data is fetched from database.
Image Transformation: If transformation is required then Imgproxy url is built and the asset is fetched from it.
Asynchronous Task: If needed, a task is enqueued for async processing.
Response: Data is returned to client.
Monitoring & Tracing: Metrics and traces are generated along the way, and exported to the corresponding consumers.

Key Interaction Points:

HTTP Layer (Fastify) <-> Storage: Fastify routes requests to the Storage class or s3protocolHandler based on the request parameters or headers. The Storage class uses the DB and S3 client to implement the required action.
Storage <-> Database (PostgreSQL): The Storage class utilizes the database interface for metadata persistence and RLS enforcement.
Storage <-> Storage Backend: The Storage class relies on a backend adapter (either S3Backend or FileBackend) to perform the object storage operations.
Storage <-> Imgproxy: The ImageRenderer class communicates with the imgproxy for image transformations by generating presigned urls.
Fastify <-> Queue (PgBoss): Queue jobs are created by Fastify routes and handled by workers in a separated pool, enabling async behaviour.
Fastify <-> Authentication: Fastify utilizes auth plugins to authorize all the requests and set the request.jwtPayload, request.isAuthenticated, request.tenantId, and request.owner variables.
Fastify <-> Logging: Fastify registers a pino logger in the request context and sets up structured logging for all the requests.
Fastify <-> Monitoring: Fastify registers prometheus metrics middleware, to track the HTTP metrics and adds OTel spans for tracing purposes.
Internal Components: The core components use shared utility libraries and classes, to do perform core operations, like streaming, concurrency control, or error handling.

The Supabase Storage repository is a highly flexible object storage service. It combines a core with a set of features to store and manipulate files. It can be deployed in several ways, either a single instance, or a multi-tenant instance with tenant configurations.

Key takeaways:

Pluggable Architecture: The system is highly modular and built to enable multiple backend and integration points (like S3 or local disk, HTTP, TUS, external image transformers),
Performance: Caching, efficient connection pooling, and async mechanisms.
Observability: OTel and prometheus to provide visibility into application's health and performance.
Multi-tenancy support: Tenant separation with DB isolation using connection pooling and RLS.

This detailed overview should help you understand how the components interact in this system.

How strong are browsers for file Conversions

Ajit Singh — Sun, 07 Jul 2024 03:26:07 +0000

Introduction: The Need for Browser-Based Conversions

File conversions are a common necessity today. We frequently need to transform files from one format to another for various reasons:

Compatibility: Different applications support different file formats.
Size reduction: Some formats offer better compression for sharing or storage.
Feature support: Certain formats provide additional features or metadata.

Traditionally, file conversions were handled by desktop applications or server-side processes. However, server-based conversions come with several significant drawbacks:

Privacy Concerns:
- Data Exposure: Uploading files to a server increases the risk of sensitive information leaking.
- Data Retention: We often have no control over how long their data is stored on the server or how it's used. People can use your data for AI training with the lack of high quality data.
- Trust Issues: We must trust the service provider not to misuse their data or fall victim to data breaches.
Lack of Control:
- Limited Customization: Server-based tools often offer one-size-fits-all solutions with limited customization options.
- Dependency on Service: We rely on the availability and reliability of the conversion service.
- Potential for Data Loss: Network issues during upload or download can result in data loss or corruption.
Speed and Bandwidth Limitations:
- Upload Time: Large files can take significant time to upload, especially on slower connections.
- Server Processing Time: High server load can lead to delays in processing.
- Download Time: Converted files need to be downloaded, adding more time to the process.
Costs and Limitations:
- Service Fees: Many online conversion tools charge fees, especially for larger files or frequent use.
- File Size Limits: Free services often impose strict limits on file sizes.
- Conversion Quotas: There may be restrictions on the number of conversions allowed.

Given these challenges, browser-based conversions offer several compelling advantages:

Privacy: Files don't need to be uploaded to a server, reducing security risks.
Control: We have full control over their data and the conversion process.
Speed: Client-side processing can be faster, especially for large files.
Offline capability: Conversions can work without an internet connection.
Accessibility: We can convert files from any device with a modern web browser.
Reduced server load: Processing happens on the user's device, saving server resources.
Cost-effective: No need for expensive server infrastructure or bandwidth costs.

Browser-based conversions leverage the power of modern web technologies to perform complex file manipulations directly in the user's browser. This approach not only addresses the privacy and control issues associated with server-based conversions but also offers a more seamless and efficient user experience.

I think all the online converters doing server conversions should ditch the server conversions and give us browser based conversions as there is more control. Sometimes you need to convert sensitive documents and browser based conversions are a very safe way to do them.

Let's explore how modern web technologies enable these powerful in-browser conversions.

JavaScript Example: Converting HEIC to JPEG

We'll start with a common use case: converting HEIC images (common in iOS devices) to the more widely supported JPEG format.

import heic2any from 'heic2any';

async function convertHeicToJpeg(heicFile) {
  try {
    const jpegBlob = await heic2any({
      blob: heicFile,
      toType: 'image/jpeg',
      quality: 0.8
    });
    return URL.createObjectURL(jpegBlob);
  } catch (error) {
    console.error('Conversion failed:', error);
  }
}

Explanation:

We import the heic2any library, which handles the conversion.
The convertHeicToJpeg function is asynchronous, allowing non-blocking operation.
We use heic2any to convert the HEIC file to a JPEG blob.
The quality parameter (0.8) balances image quality and file size.
We create a URL for the resulting JPEG blob, which can be used to display or download the image.

Usage example:

const fileInput = document.getElementById('fileInput');
fileInput.addEventListener('change', async (event) => {
  const heicFile = event.target.files[0];
  const jpegUrl = await convertHeicToJpeg(heicFile);
  const img = document.createElement('img');
  img.src = jpegUrl;
  document.body.appendChild(img);
});

This code sets up a file input and displays the converted JPEG image when a HEIC file is selected.

JavaScript Example: PDF to PNG Conversion

Next, let's look at converting a PDF page to a PNG image, which can be useful for previews or sharing.

import * as pdfjsLib from 'pdfjs-dist';

async function convertPdfToImage(pdfFile, pageNumber = 1) {
  const arrayBuffer = await pdfFile.arrayBuffer();
  const pdf = await pdfjsLib.getDocument({ data: arrayBuffer }).promise;
  const page = await pdf.getPage(pageNumber);

  const scale = 1.5;
  const viewport = page.getViewport({ scale });

  const canvas = document.createElement('canvas');
  const context = canvas.getContext('2d');
  canvas.height = viewport.height;
  canvas.width = viewport.width;

  const renderContext = {
    canvasContext: context,
    viewport: viewport
  };

  await page.render(renderContext).promise;

  return new Promise((resolve) => {
    canvas.toBlob((blob) => {
      resolve(URL.createObjectURL(blob));
    }, 'image/png');
  });
}

Explanation:

We use the pdf.js library to handle PDF parsing and rendering.
The function takes a PDF file and an optional page number.
We create a canvas element to render the PDF page.
The scale factor (1.5) determines the resolution of the output image.
After rendering the page to the canvas, we convert it to a PNG blob.
Finally, we create a URL for the PNG image.

This conversion is particularly useful for generating thumbnails or previews of PDF documents directly in the browser, improving user experience in document management systems or file sharing platforms.

JavaScript Example: CSV to JSON Conversion

Converting CSV to JSON is a common task in data processing and analysis. Here's how we can do it in the browser:

import Papa from 'papaparse';

function convertCsvToJson(csvFile) {
  return new Promise((resolve, reject) => {
    Papa.parse(csvFile, {
      complete: (results) => {
        const jsonData = results.data.map((row) => {
          const obj = {};
          results.meta.fields.forEach((field, index) => {
            obj[field] = row[index];
          });
          return obj;
        });
        resolve(jsonData);
      },
      header: true,
      error: (error) => {
        reject(error);
      }
    });
  });
}

Explanation:

We use the Papa Parse library, which is excellent for CSV parsing.
The header: true option tells Papa Parse to use the first row as field names.
We transform the parsed data into an array of objects, where each object represents a row.
The resulting JSON data can be easily manipulated or displayed in the browser.

This conversion is valuable for data analysis tools, allowing We to upload CSV files and work with the data in a more structured JSON format without server involvement.

WebAssembly Example: WebP to PNG Conversion

WebAssembly allows us to use high-performance libraries compiled from languages like C or C++. Here's an example using libwebp:

import { WEBP } from '@saschazar/wasm-webp';

let webpModule;

async function initWebPModule() {
  webpModule = await WEBP();
}

async function convertWebPToPNG(webpFile) {
  if (!webpModule) {
    await initWebPModule();
  }

  const arrayBuffer = await webpFile.arrayBuffer();
  const webpData = new Uint8Array(arrayBuffer);

  const { width, height } = webpModule.getInfo(webpData);
  const rgbaData = webpModule.decode(webpData);

  const canvas = document.createElement('canvas');
  canvas.width = width;
  canvas.height = height;
  const ctx = canvas.getContext('2d');
  const imageData = ctx.createImageData(width, height);
  imageData.data.set(rgbaData);
  ctx.putImageData(imageData, 0, 0);

  return new Promise((resolve) => {
    canvas.toBlob((blob) => {
      resolve(URL.createObjectURL(blob));
    }, 'image/png');
  });
}

Explanation:

We use a WebAssembly module wrapping libwebp for high-performance decoding.
The module is initialized asynchronously when needed.
We decode the WebP image to raw RGBA data using the WebAssembly module.
The decoded data is then drawn onto a canvas and converted to a PNG.

This WebAssembly-based conversion showcases how browsers can leverage native-speed libraries for complex tasks like image processing, providing performance comparable to desktop applications.

Conclusion

Browser-based file conversions offer us the convenience of performing complex file manipulations without leaving their browser or installing additional software. This approach not only enhances user experience but also reduces server load and addresses privacy concerns by keeping sensitive data on the client-side.
As web technologies continue to evolve, we can expect even more powerful and diverse file conversion capabilities directly in the browser.

I also implemented a few browser based conversions in my website HEIC Converter you can check it out. Once the website loads you can even turn off the internet still everything will work.

Launch Announcement: Optimize Your AI Experience with "Prompt Engineering Patterns" Email Course 🚀

Ajit Singh — Thu, 06 Jul 2023 12:55:13 +0000

Launch Announcement: Optimize Your AI Experience with "Prompt Engineering Patterns" Email Course 🚀

Today, I'm thrilled to introduce the latest addition to the world of AI learning: the "Prompt Engineering Patterns" email course. This in-depth 14-day email course is specifically designed to help you optimize your experience with AI, with a concentrated focus on OpenAI's ChatGPT.

Here's why this course is an essential tool in your AI toolbox:

1️⃣ Empower your ChatGPT Interactions: Learn how to unlock the full potential of advanced language models like ChatGPT.

2️⃣ AI Optimization Techniques: Gain practical knowledge on understanding, manipulating, and optimizing AI models to enhance their performance.

3️⃣ AI Learning for All: The course is crafted to cater to everyone, from beginners to experienced AI professionals.

The best part? This comprehensive AI learning resource is absolutely FREE! 🎁

In the evolving digital world, it's not the advent of AI that might challenge us, but the inability to operate and optimize AI could leave us behind. Therefore, enhancing your AI skills is crucial for both personal and professional growth.

Eager to dive in? You can sign up for the "Prompt Engineering Patterns" email course right here: https://prompting.aicygnus.com/

Feel free to reach out if you need more information or have any queries. Let's venture into the exciting future of AI together!

I'm looking forward to welcoming you to the course. Let's optimize our AI learning and ensure we stay at the forefront of technological advancement!

If you've already signed up or have experiences with similar AI courses, please share your thoughts and experiences in the comments below. Let's get a conversation going about AI learning, ChatGPT, and the future of AI.

Caching for performance with Amazon DocumentDB and Amazon ElastiCache

Ajit Singh — Sun, 28 Aug 2022 12:33:00 +0000

Overview of My Submission

I move the AWS sample app from using two databases to a single database. Earlier they were using Elasticache for cacheing and DynamoDb for data persistence bu adding Redis as the primary DB I have removed one file of code and made it super simple to read and manage.

Submission Category:

Minimalism Magicians

Language Used

Node.js

Link to Code

ajitsinghkaler / amazon-documentdb-and-amazon-elacticache-caching-for-performance-example

Caching for performance with Amazon DocumentDB and Amazon ElastiCache

How it works

It saves songs to redis database on the /cd endpoint. Using the createEntity and save function

It can also search songs from the database on the /cd/:title endpoint. USing redis search, where and equal function.

How the data is stored:

Data is stored using the create entity function. Schema is the following

{
    title: { type: 'string' },
    singer: { type: 'string' },
    text: { type: 'text' },
}

How the data is accessed:

The data is access using Redis Seach using the search function and the where…

View on GitHub

Check out Redis OM, client libraries for working with Redis as a multi-model database.
Use RedisInsight to visualize your data in Redis.
Sign up for a free Redis database.

AWS Elastic Beanstalk - Hands On

Ajit Singh — Wed, 29 Sep 2021 03:18:30 +0000

To create a stack on AWS bean stalk follow the steps below

Search for beanstalk in the Search bar after logging in
Click on create application on the homepage
Select an application name I selected test-beanstalk-app and tags you want to attach to this app
After that on platform select

Platform- Language with which you want to deploy your app
Select version of node.js I selected the automatically selected one
Platform version- Version of AWS platform that you want to select as different versions support deifferent node.js engines

In application code select sample application for a test app. If you want to upload your own code select upload your code
Click on create application and wait for a few minutes after a few logs you will see the following screen
Click on the app link highlighted in the previous image to have a look at your demo app.
After that in configuration you can check all the things that were setup by AWS beanstalk
You can create various environments in the environments like creating develop, production etc.
Finally delete your app using the applications tab and select application and after that click on actions and click delete application.

We are done with creating a platform in Beanstalk next will study how to create your own pipelines in AWS

DEV Community: Ajit Singh

Understanding TUS and imgProxy in supabse/storage

TUS and Imgproxy

1. TUS Protocol for Resumable Uploads

2. Imgproxy for Image Transformations

Usage Details:

Key Takeaways:

Tenants in supabase/storage

Tenant Management in Supabase/Storage

Tenant Data Separation Lifecycle

Deployement in supabase storage

Supabase Storage how deployment works

Deployment Overview

Component Interaction in docker-compose.yml (Single-Tenant):

Component Interaction in docker-compose-multi-tenant.yml (Multi-Tenant):

The Role of MinIO, pgBouncer, and Supavisor

Monitoring

Deployment Process Summary

How migrations work in supabase/storage

Supabse storage migrations

Single-Tenant Migrations

Migration Strategies (Multi-Tenant)

Tracing and Metrics in supabase/storage

Tracing and Metrics

Tracing Lifecycle

Metrics Collection

Key Takeaways:

Understanding Storage backends in supabase/storage

Storage Backends

1. FileBackend Storage Adapter

2. S3Backend Storage Adapter

Key Differences Summarized

Key Takeaways

Auth in Supabase storage

Authentication and Authorization

Authentication and Authorization Components

Role of Schemas in Validation

How a request is authenticated and authorized

Benefits

Overview of supabase storage repository

Supabase storage Repository Overview

Core Concepts:

Interaction from a request lifecycle perspective

Key Interaction Points:

Key takeaways:

How strong are browsers for file Conversions

Introduction: The Need for Browser-Based Conversions

JavaScript Example: Converting HEIC to JPEG

JavaScript Example: PDF to PNG Conversion

JavaScript Example: CSV to JSON Conversion

WebAssembly Example: WebP to PNG Conversion

Conclusion

Launch Announcement: Optimize Your AI Experience with "Prompt Engineering Patterns" Email Course 🚀

Launch Announcement: Optimize Your AI Experience with "Prompt Engineering Patterns" Email Course 🚀

The best part? This comprehensive AI learning resource is absolutely FREE! 🎁

Caching for performance with Amazon DocumentDB and Amazon ElastiCache

Overview of My Submission

Submission Category:

Language Used

Link to Code

ajitsinghkaler / amazon-documentdb-and-amazon-elacticache-caching-for-performance-example

Caching for performance with Amazon DocumentDB and Amazon ElastiCache

Caching for performance with Amazon DocumentDB and Amazon ElastiCache

How it works

How the data is stored:

How the data is accessed:

AWS Elastic Beanstalk - Hands On

Component Interaction in `docker-compose.yml` (Single-Tenant):

Component Interaction in `docker-compose-multi-tenant.yml` (Multi-Tenant):

1. `FileBackend` Storage Adapter

2. `S3Backend` Storage Adapter