When we horizontally scale an application, adding more instances is only part of the solution.
We also need to consider where the application stores its data and how other instances access it.
While exploring horizontal scaling, I came across MinIO, an S3-compatible object storage system, and wanted to understand how it helps address one of these challenges.
The problem: Local storage in a distributed application
Imagine a backend service running on three instances behind a load balancer.
If App-1 stores a video on its local disk, App-2 cannot automatically access that file when a subsequent request reaches it.
Each instance has its own filesystem, creating a dependency between the data and the instance that stores it.
The solution: Shared object storage
MinIO provides a separate storage layer that multiple application instances can access through S3-compatible APIs.
Instead of storing files exclusively on their local disks, application instances can store and retrieve objects from shared storage.
This helps decouple file storage from individual application instances and makes horizontal scaling easier to manage.
Why does this matter?
The same principle applies to many systems:
- Video streaming: Recordings, HLS segments, and thumbnails.
- Computer vision: Images, datasets, and inference artifacts.
- Backend services: User uploads, documents, and generated reports.
MinIO doesn't automatically make an application horizontally scalable. Database connections, sessions, distributed coordination, and other instance-specific state still need to be addressed.
However, choosing the right storage architecture is an important step.
What's next?
Horizontal scaling introduces several engineering challenges beyond file storage.
In the next parts of this series, I'll explore concepts such as distributed coordination, leader election, shared state, and database considerations when running multiple service instances.
One takeaway: Scaling application instances is easier when shared data and instance-specific responsibilities are deliberately separated.
Top comments (0)