DEV Community

Aman Kumar
Aman Kumar

Posted on

I Built an S3-Compatible Storage Server in Go — Here's What I Learned

#backend #distributedsystems #go #showdev

Why Build Another S3 Clone?

Two reasons:

  1. I wanted to understand distributed storage — not just read about it, but actually build it
  2. AWS bills are scary — paying ₹25k/month for storage hurts when you're a solo developer

So I spent 2 weeks building my own S3-compatible storage server in Go. Here's what happened.

The Stack

Component What I Used
Language Go (1.22+)
Database PostgreSQL (10 tables, auto-migrations)
Caching Redis
Container Docker + Docker Compose
Orchestration Kubernetes manifests
Metrics Prometheus
API S3-compatible REST

Features I Built

🔐 JWT Auth + RBAC

I implemented JWT with access/refresh token rotation. Users can have different roles with different permissions — admin, read-only, bucket-specific access. IAM-style policy engine built from scratch.

💾 Erasure Coding (Reed-Solomon)

Your file is split into 4 data shards + 2 parity shards. Even if 2 disks die, your data survives. No expensive RAID hardware needed.

📤 Multipart Upload

Large files are uploaded in parallel parts. Each part is independently stored and reassembled on download. Essential for files > 100MB.

📊 Prometheus Metrics

Every request is tracked — latency histograms, error rates, storage usage, active connections. Grafana dashboard ready.

🔍 Content Deduplication

SHA-256 hashing ensures duplicate files are stored once. Saves significant space in real-world usage.

🚦 Graceful Shutdown

SIGTERM/SIGINT handling with connection draining. Zero-downtime deployments.

The Architecture 

┌──────────────────────────────────────────┐
│            HTTP/REST API                 │
├──────────────────────────────────────────┤
│      Auth & Authorization Layer          │
├──────────────────────────────────────────┤
│        Business Logic Layer              │
├──────────────────────────────────────────┤
│           Storage Layer                  │
│  Chunking · Dedup · Erasure · Replication│
├──────────────────────────────────────────┤
│     Infrastructure                       │
│  PostgreSQL │ Redis │ Local Disk Store   │
├──────────────────────────────────────────┤
│     Observability                        │
│  Metrics │ JSON Logs │ Audit Trails      │
└──────────────────────────────────────────┘
Enter fullscreen mode Exit fullscreen mode

Quick Start 

git clone https://github.com/aman179102/distributed-file-storage-service.git
cd distributed-file-storage-service
docker compose up -d

curl http://localhost:8080/health
Enter fullscreen mode Exit fullscreen mode

That's it. Your own S3. Running. Free.

What I Learned Building This

  1. Distributed systems aren't magic — they're just good engineering decisions stacked together
  2. Go is perfect for this — concurrency, performance, simple deployment
  3. PostgreSQL can handle more than you think — 10 tables, smooth migrations
  4. Building > Reading — I learned more in 2 weeks of building than 2 months of tutorials

What's Next

  • [ ] S3 API compatibility (ListObjects, DeleteObjects, etc.)
  • [ ] Web UI for file management
  • [ ] S3-to-S3 replication (backup to real AWS)
  • [ ] MinIO client compatibility

Get The Code

🔗 GitHub: Link

If you found this useful, star the repo ⭐ — it helps more people discover it.

Drop a comment if you've built something similar or have questions. Let's learn together. 🚀

Top comments (0)