DEV Community

Ramiro Estrella
Ramiro Estrella

Posted on

File storage in Node.js without a framework — a cleaner approach

#webdev #node #opensource #typescript

blobpipe cover

If you're using Next.js, NestJS, or a similar framework, you probably already have a storage story. There are packages built around those ecosystems that handle the integration for you, and they're worth using.

But what if you're not?

GitHub · npm · MIT License

Imagine you're building a CLI tool that processes and uploads files. Or a background worker that handles document ingestion. Or an API service you scaffolded with Express and nothing else. There's no framework magic here — just Node.js and whatever you reach for.

At some point you need to store a file somewhere. So you install the AWS SDK, write some code, and move on.

That works. Until it doesn't.

The problem shows up later

Maybe you start on S3, then a client needs Azure Blob. Maybe you want local disk in development and a real cloud in production. Maybe you just want to run your tests without setting up credentials or spinning up a Docker container.

At that point you look at your upload code and realize it's tangled up with the SDK you chose on day one. The validation logic, the error handling, the logging — all of it is woven into AWS-specific calls. Untangling it is the kind of work that feels pointless because the end result is the same behavior, just reorganized.

That's the problem blobpipe tries to solve.

The core idea

Your application code shouldn't know which cloud provider it's talking to. It should just say "store this file at this key" and let something else figure out where that actually goes.

Blobpipe gives you a StorageClient that works the same regardless of the underlying provider. You pick a driver when you wire things up, and from that point on your code doesn't change:

import { StorageClient } from '@restrella/blobpipe';
import { S3Driver }      from '@restrella/blobpipe-s3';
import { LocalDriver }   from '@restrella/blobpipe-local';

const driver = process.env.NODE_ENV === 'production'
  ? new S3Driver({ bucket: 'my-bucket', region: 'us-east-1' })
  : new LocalDriver({ rootDir: './uploads' });

const storage = new StorageClient(driver);
Enter fullscreen mode Exit fullscreen mode

From here, storage.put(), storage.get(), storage.delete() — none of that changes when you swap the driver. You can move from S3 to GCS to Azure without touching the code that actually uses storage.

There are drivers for S3 (and anything S3-compatible: R2, MinIO, DigitalOcean Spaces), Google Cloud Storage, Azure Blob, local disk, and an in-memory driver for tests.

Validation and logging without copy-pasting

Most upload code ends up needing the same few things: check the file type, check the size, log what came in. If you have more than one place in your app that uploads files, you've probably either copy-pasted this or built some helper that's grown over time.

Blobpipe has a middleware pipeline that runs before every put():

const storage = new StorageClient(driver)
  .use(validateMimeType({ allowed: ['image/png', 'image/jpeg', 'image/webp'] }))
  .use(maxFileSize({ maxBytes: 10 * 1024 * 1024 }))
  .use(logUploads());
Enter fullscreen mode Exit fullscreen mode

use() returns a new client — it doesn't mutate the original. So if you have different upload contexts in the same app, you can derive them from a shared base:

const base   = new StorageClient(driver).use(logUploads());
const images = base.use(validateMimeType({ allowed: ['image/png', 'image/jpeg'] }));
const docs   = base.use(maxFileSize({ maxBytes: 50 * 1024 * 1024 }));
Enter fullscreen mode Exit fullscreen mode

base still only has logging. images and docs are independent. No side effects.

You can also write your own middleware when you need something custom — it's just an async function:

const addVersionMetadata: Middleware = async (ctx, next) => {
  ctx.options = {
    ...ctx.options,
    metadata: { ...ctx.options.metadata, appVersion: process.env.APP_VERSION ?? 'unknown' },
  };
  await next();
};
Enter fullscreen mode Exit fullscreen mode

Testing is the part I care about most

In framework-less Node.js projects, testing file upload code is often the thing that gets skipped. You either need real credentials, a running emulator, or a pile of mocks that don't really test anything.

The MemoryDriver is a complete in-process implementation — backed by a Map, no network, no Docker:

import { StorageClient } from '@restrella/blobpipe';
import { MemoryDriver }  from '@restrella/blobpipe-memory';

beforeEach(() => {
  storage = new StorageClient(new MemoryDriver());
});

it('rejects oversized uploads', async () => {
  const limited = storage.use(maxFileSize({ maxBytes: 100 }));
  await expect(limited.put('big.txt', 'x'.repeat(101))).rejects.toThrow();
});
Enter fullscreen mode Exit fullscreen mode

Because MemoryDriver implements the same interface as every other driver, you're testing real behavior — not a mock. The same test works against LocalDriver or S3Driver if you want to run it against a real provider in CI.

Errors you can actually catch cleanly

Cloud SDKs throw their own error types. If your app is doing catch (err) and checking err instanceof S3ServiceException, that's provider knowledge leaking into your application layer.

Blobpipe normalizes errors into its own hierarchy:

try {
  await storage.get('missing-file.pdf');
} catch (err) {
  if (err instanceof ObjectNotFoundError) return res.status(404).send();
  if (err instanceof AccessDeniedError)   return res.status(403).send();
  throw err;
}
Enter fullscreen mode Exit fullscreen mode

This works the same whether the file is on S3, GCS, or local disk. The underlying SDK error is attached as cause if you need it, but you don't have to.

Who this is actually for

If you're on Next.js, Remix, NestJS, or another framework with storage integrations — check what's already available in that ecosystem first. Those integrations are probably a better fit because they're built around how the framework handles requests, auth, and config.

Blobpipe is useful when you're outside of that:

  • CLI tools that upload or download files
  • Background workers or queue processors
  • Custom API servers without a framework
  • Open-source tools that need to support multiple cloud backends
  • Any project where you want to keep provider details out of your business logic

It's on npm as @restrella/blobpipe. The source is on GitHub at github.com/balance3840/blobpipe. Each driver is its own package so you only install the SDK you actually need.

If you're building something where this fits, give it a try. And if the middleware design doesn't make sense or something's missing for your use case, open an issue — still early days.

Top comments (0)