DEV Community

Cover image for Combine multiple S3 storage
somen das
somen das

Posted on

Combine multiple S3 storage

#aws #cloudstorage #node #googlecloud

I currently have some files stored in AWS S3 and others in Cloudflare R2. That made me realize—it would be really convenient to combine everything into a single API. By doing so, I can take advantage of the free tiers offered by multiple cloud storage providers. As soon as I reach the limit on one provider, I can simply create another account or switch to another service and add it to my unified system.

This approach allows me to maintain a single source of truth for all my file storage and retrieval needs. Behind the scenes, it intelligently manages connections to multiple S3-compatible storage providers. This would significantly simplify how I handle file uploads and access across services.

🎯 Why I'm Building This Package:

I'm building a package that acts as a unified interface for multiple S3-compatible cloud storage APIs. The goal is to:

  • Simplify File Management: Upload and access files through a single API, regardless of the underlying provider (e.g., AWS S3, Cloudflare R2, Wasabi, Backblaze B2, etc.).

  • Optimize Cost: Leverage free tiers across various providers by intelligently routing uploads or downloads to the least costly or most available storage bucket.

  • Seamless Scalability: As storage needs grow, I can add more cloud accounts or services without changing how the API is used.

  • Abstraction of Complexity: Hide provider-specific quirks and authentication behind a single clean interface.

  • Failover and Redundancy: If one provider fails or reaches a quota, the system automatically falls back to another available provider.

This package will make it easy to build apps or services that need reliable and affordable cloud storage without locking into a single provider or dealing with vendor-specific APIs.

MultiBucket

github

A Node.js library for generating presigned URLs for multiple object storage providers (AWS S3, Cloudflare R2) with automatic load balancing.

Features

  • Support for multiple storage providers (AWS S3 and Cloudflare R2)
  • Automatic load balancing between providers using various strategies
  • Live configuration updates from file or remote URL
  • Rate limiting and error handling
  • RESTful API endpoints for generating presigned URLs
  • Monitoring and statistics

Installation 

npm install multibucket
Enter fullscreen mode Exit fullscreen mode

Dependencies

This library requires the following dependencies:

npm install @aws-sdk/client-s3 @aws-sdk/s3-request-presigner express axios chokidar
Enter fullscreen mode Exit fullscreen mode

Basic Usage 

const MultiBucket = require('multibucket');

// Create an instance with initial providers
const storagePresigner = new MultiBucket({
  providers: [
    {
      id: 's3-main',
      type: 's3',
      bucket: 'my-main-bucket',
      region: 'us-east-1',
      accessKeyId: 'YOUR_AWS_ACCESS_KEY',
      secretAccessKey: 'YOUR_AWS_SECRET_KEY',
      weight: 3,
      rateLimit: 100
    },
    {
      id: 'r2-cloudflare',
      type: 'r2',
      bucket: 'my-r2-bucket',
      endpoint: 'https://account-id.r2.cloudflarestorage.com',
      accessKeyId: 'YOUR_R2_ACCESS_KEY',
      secretAccessKey: 'YOUR_R2_SECRET_KEY',
      publicUrlBase: 'https://cdn.example.com'
    }
  ],
  loadBalanceStrategy: 'round-robin',
  defaultExpiry: 3600
});

// Start the API server
storagePresigner.createServer(3000);
Enter fullscreen mode Exit fullscreen mode

Configuration Options

Constructor Options

  • providers: Array of storage provider configurations
  • configSource: Path or URL to a config file (optional)
  • loadBalanceStrategy: Strategy for load balancing (default: 'round-robin')
  • defaultExpiry: Default expiry time for presigned URLs in seconds (default: 3600)

Provider Configuration

Each provider object requires the following properties:

Common Properties:

  • id: Unique identifier for the provider
  • type: Provider type ('s3' or 'r2')
  • bucket: Bucket name
  • accessKeyId: Access key ID
  • secretAccessKey: Secret access key
  • weight (optional): Weight for weighted-random load balancing
  • rateLimit (optional): Maximum requests per second
  • publicUrlBase (optional): Base URL for public access

S3-specific Properties:

  • region: AWS region
  • endpoint (optional): Custom endpoint for S3-compatible services
  • forcePathStyle (optional): Use path-style addressing

R2-specific Properties:

  • endpoint: R2 endpoint URL

Load Balancing Strategies

  • round-robin: Cycle through providers sequentially
  • least-used: Select the provider with the fewest requests
  • least-errors: Select the provider with the lowest error rate
  • weighted-random: Select providers randomly based on their weight

External Configuration

You can provide a path to a JSON file or a URL in the configSource option:

const storagePresigner = new MultiBucket({
  configSource: './storage-config.json'
});
Enter fullscreen mode Exit fullscreen mode

The library will watch for changes to the file or poll the URL to update the configuration dynamically.

API Endpoints

When you start the server with createServer(), the following endpoints are available:

Generate Upload URL 

POST /generate-upload-url
Enter fullscreen mode Exit fullscreen mode

Request body:

{
  "filename": "example.jpg",
  "contentType": "image/jpeg",
  "path": "uploads/images",
  "expiry": 1800,
  "providerId": "s3-main"
}
Enter fullscreen mode Exit fullscreen mode

Response:

{
  "uploadUrl": "https://my-main-bucket.s3.us-east-1.amazonaws.com/uploads/images/uuid-example.jpg?...",
  "publicUrl": "https://my-main-bucket.s3.us-east-1.amazonaws.com/uploads/images/uuid-example.jpg",
  "key": "uploads/images/uuid-example.jpg",
  "bucket": "my-main-bucket",
  "provider": "s3-main",
  "expires": "2023-06-01T12:30:00.000Z"
}
Enter fullscreen mode Exit fullscreen mode

Generate Read URL 

POST /generate-read-url
Enter fullscreen mode Exit fullscreen mode

Request body:

{
  "key": "uploads/images/uuid-example.jpg",
  "providerId": "s3-main",
  "expiry": 3600
}
Enter fullscreen mode Exit fullscreen mode

Response:

{
  "readUrl": "https://my-main-bucket.s3.us-east-1.amazonaws.com/uploads/images/uuid-example.jpg?...",
  "key": "uploads/images/uuid-example.jpg",
  "bucket": "my-main-bucket",
  "provider": "s3-main",
  "expires": "2023-06-01T13:00:00.000Z"
}
Enter fullscreen mode Exit fullscreen mode

Get Stats 

GET /stats
Enter fullscreen mode Exit fullscreen mode

Response:

{
  "providerCount": 2,
  "totalRequests": 150,
  "providerStats": [
    {
      "id": "s3-main",
      "type": "s3",
      "requestCount": 100,
      "errorCount": 2,
      "errorRate": "0.0200"
    },
    {
      "id": "r2-cloudflare",
      "type": "r2",
      "requestCount": 50,
      "errorCount": 0,
      "errorRate": "0.0000"
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

Health Check 

GET /health
Enter fullscreen mode Exit fullscreen mode

Response:

{
  "status": "ok",
  "providers": 2,
  "timestamp": "2023-06-01T11:00:00.000Z"
}
Enter fullscreen mode Exit fullscreen mode

Programmatic Usage

You can also generate presigned URLs programmatically:

// Generate upload URL
const uploadUrlInfo = await storagePresigner.generateUploadUrl({
  filename: 'example.jpg',
  contentType: 'image/jpeg',
  path: 'uploads/images',
  expiry: 1800
});

// Generate read URL
const readUrlInfo = await storagePresigner.generateReadUrl({
  key: 'uploads/images/uuid-example.jpg',
  providerId: 's3-main',
  expiry: 3600
});
Enter fullscreen mode Exit fullscreen mode

License

MIT
"presigned",
"url",
"s3",
"r2",
"cloudflare",
"load-balancing",
"storage",
"upload",
"multi-storage",
"multi-bucket",
"aws-sdk",
"cloudflare-r2",
"express",
"axios",
"chokidar",
"dotenv",
"multi-provider",
"multi-provider-presigner",
"multi-storage-presigner",
"multi-bucket-presigner",
"multi-storage-url",
"multi-bucket-url",
"multi-storage-upload",
"multi-bucket-upload",
"s3-presigner",
"r2-presigner",
"s3-upload",
"r2-upload",
"s3-url",
"r2-url",
"s3-multi-provider",
"r2-multi-provider",
"s3-multi-storage",
"r2-multi-storage"

Top comments (0)