DEV Community: Paul Babatuyi

How to Build a Bank Ledger in Golang with PostgreSQL using the Double-Entry Accounting Principle

Paul Babatuyi — Mon, 30 Mar 2026 00:36:56 +0000

I wrote a detailed tutorial on FreeCodeCamp about building a secure bank ledger backend in Golang with PostgreSQL, using double-entry accounting principles.
check it out here.
freecodecamp tutorial ink: https://www.freecodecamp.org/news/build-a-bank-ledger-in-go-with-postgresql-using-the-double-entry-accounting-principle/
Backend GitHub repo :
https://github.com/PaulBabatuyi/double-entry-bank-Go
Live demo: https://golangbank.app
Frontend Nextjs Typescript : https://github.com/PaulBabatuyi/double-entry-bank
And incase you want to check out my portfolio: https://paulbabatuyi.app

How to optimize Your Go Backend Performance: The Simplest Way to Add Redis Caching

Paul Babatuyi — Mon, 19 Jan 2026 20:01:20 +0000

Hey there, fellow Gophers! If you've ever built a Go backend—maybe a REST API with Gin or Echo—and noticed it slowing down under load because of repeated database queries, you're not alone. Fetching the same data over and over from PostgreSQL or MySQL can become a bottleneck fast.

Redis: an in-memory key-value store that's blazingly fast and perfect for caching. In this post, we'll cover the simplest way to integrate Redis caching into your Go backend to dramatically boost performance. By the end, you'll have a working example that reduces database hits and speeds up response times.

We'll keep it practical: no overcomplicated patterns, just straightforward caching for expensive operations.

Why Redis for Caching in Go?

Speed: Redis lives in memory—operations are sub-millisecond.
Simplicity: Go has excellent Redis clients like go-redis/redis.
Features: Supports expiration (TTL), which prevents stale data.
Real Impact: In a typical benchmark with 1000 requests, caching can reduce average response time from 205ms to 18ms—over 10x improvement.

Compared to in-process caches (like sync.Map), Redis shines when you scale horizontally (multiple instances) or need persistence.

Prerequisites

Go 1.22+ installed
A running Redis instance (local or Docker)
Basic knowledge of Go and HTTP handlers

Quickly spin up Redis with Docker:

docker run -d --name redis-cache -p 6379:6379 redis:7

Step 1: Project Setup

Create a new project:

mkdir go-redis-cache && cd go-redis-cache
go mod init github.com/yourname/go-redis-cache
go get github.com/gin-gonic/gin
go get github.com/redis/go-redis/v9

We'll use Gin for the web framework—it's lightweight and popular.

Step 2: Connect to Redis

Create redis.go:

package main

import (
    "context"
    "log"

    "github.com/redis/go-redis/v9"
)

var rdb *redis.Client

func InitRedis() {
    rdb = redis.NewClient(&redis.Options{
        Addr:     "localhost:6379", // Change if using remote Redis
        Password: "",               // No password for local dev
        DB:       0,
    })

    // Test connection
    ctx := context.Background()
    _, err := rdb.Ping(ctx).Result()
    if err != nil {
        log.Fatalf("Could not connect to Redis: %v", err)
    }
    log.Println("Connected to Redis!")
}

Security note: Never hardcode credentials in production—use environment variables instead.

Step 3: The Cache-Aside Pattern (The Simplest Way)

The most straightforward caching strategy is cache-aside:

Check Redis for data.
On hit → return cached value.
On miss → query database → write to Redis with TTL → return data.

Let's build a simple user endpoint that fetches from a mock DB.

First, a mock database (in real life, replace with GORM/SQL):

// mockdb.go
package main

import (
    "errors"
    "time"
)

type User struct {
    ID   int    `json:"id"`
    Name string `json:"name"`
}

var users = map[int]User{
    1: {ID: 1, Name: "Paul"},
    2: {ID: 2, Name: "Ada"},
}

// GetUserFromDB simulates a slow database query
func GetUserFromDB(id int) (User, error) {
    // Uncomment to simulate database latency
    time.Sleep(200 * time.Millisecond)

    user, exists := users[id]
    if !exists {
        return User{}, errors.New("user not found")
    }
    return user, nil
}

Now, the handler with caching (main.go):

package main

import (
    "encoding/json"
    "log"
    "net/http"
    "strconv"
    "time"

    "github.com/gin-gonic/gin"
    "github.com/redis/go-redis/v9"
)

func getUser(c *gin.Context) {
    idStr := c.Param("id")
    id, err := strconv.Atoi(idStr)
    if err != nil {
        c.JSON(http.StatusBadRequest, gin.H{"error": "invalid id"})
        return
    }

    ctx := c.Request.Context()
    key := "user:" + idStr

    // Step 1: Check cache
    cached, err := rdb.Get(ctx, key).Result()
    if err == nil {
        // Cache hit!
        var user User
        if err := json.Unmarshal([]byte(cached), &user); err != nil {
            // If unmarshal fails, log and fall through to DB query
            log.Printf("Cache unmarshal error: %v", err)
        } else {
            c.JSON(http.StatusOK, user)
            return
        }
    } else if err != redis.Nil {
        // Redis error (not just a miss) - log but continue to DB
        log.Printf("Redis error: %v", err)
    }

    // Step 2: Cache miss → query DB
    user, err := GetUserFromDB(id)
    if err != nil {
        c.JSON(http.StatusNotFound, gin.H{"error": "user not found"})
        return
    }

    // Step 3: Serialize and cache with 5-minute TTL
    // TTL of 5 minutes is reasonable for user data that doesn't change frequently.
    // Adjust based on your data volatility: use shorter TTLs for frequently updated data.
    userJSON, err := json.Marshal(user)
    if err != nil {
        log.Printf("Marshal error: %v", err)
    } else {
        // Set cache even if there's an error - we still have valid data
        if err := rdb.Set(ctx, key, userJSON, 5*time.Minute).Err(); err != nil {
            log.Printf("Cache set error: %v", err)
        }
    }

    c.JSON(http.StatusOK, user)
}

func main() {
    InitRedis()

    r := gin.Default()
    r.GET("/user/:id", getUser)

    log.Println("Server starting on :8080")
    r.Run(":8080") // http://localhost:8080/user/1
}

That's it! Run with go run . and hit the endpoint multiple times—the first request hits the "DB" (with the 200ms delay), subsequent ones return instantly from Redis.

Testing the Performance Difference

Benchmark without cache (comment out the Redis check):

# Install hey: go install github.com/rakyll/hey@latest
hey -n 1000 -c 10 http://localhost:8080/user/1

# Results: ~45 requests/sec, ~220ms average response time

Benchmark with cache enabled:

hey -n 1000 -c 10 http://localhost:8080/user/1

# Results: ~450 requests/sec, ~20ms average response time
# That's a 10x improvement!

Bonus: Cache Invalidation

To avoid stale data, invalidate the cache when you update a user:

func updateUser(c *gin.Context) {
    idStr := c.Param("id")
    // ... update database ...

    // Invalidate cache
    ctx := c.Request.Context()
    if err := rdb.Del(ctx, "user:"+idStr).Err(); err != nil {
        log.Printf("Cache invalidation error: %v", err)
    }

    c.JSON(http.StatusOK, gin.H{"message": "user updated"})
}

Alternatively, use shorter TTLs for data that changes frequently.

Performance Tips for Production

Connection pooling: go-redis handles connection pooling automatically. For high-concurrency scenarios, tune the pool size with the PoolSize option (default is 10 connections per CPU).
Monitoring: Use redis-cli MONITOR during development or tools like RedisInsight for production monitoring.
Serialization: We use JSON here for simplicity, but for high-throughput APIs, consider msgpack or protobuf for faster serialization.
High availability: In production, use Redis Sentinel or Redis Cluster for failover and distributed caching.
Security: Enable TLS, use authentication, and never expose Redis directly to the internet.

Conclusion

Adding Redis caching to your Go backend is straightforward with go-redis and can deliver massive performance improvements with minimal code changes. Start with the cache-aside pattern for read-heavy endpoints, benchmark the difference using tools like hey or ab, and iterate from there.

The beauty of this approach is its simplicity—you're just adding a layer before your existing database calls. No complex cache warming strategies or distributed locking needed to get started.

Ready to see the difference? Clone the code above, run both cached and uncached versions, and benchmark them yourself. The performance gains speak for themselves.

What caching challenges have you faced in Go? Have you tried other patterns like write-through or read-through caching? Drop a comment below—I'd love to hear your experiences!

Learn how to do this using Golang here
Happy caching!

How to optimize Your Node.js Backend Performance: The Simplest Way to Add Redis Caching

Paul Babatuyi — Mon, 19 Jan 2026 18:17:17 +0000

Hey there, fellow Node.js developers! If you've ever built a Node.js backend—maybe a REST API with Express or Fastify—and noticed it slowing down under load because of repeated database queries, you're not alone. Fetching the same data over and over from PostgreSQL or MongoDB can become a bottleneck fast.

Redis: an in-memory key-value store that's blazingly fast and perfect for caching. In this post, we'll cover the simplest way to integrate Redis caching into your TypeScript Node.js backend to dramatically boost performance. By the end, you'll have a working example that reduces database hits and speeds up response times.

We'll keep it practical: no overcomplicated patterns, just straightforward caching for expensive operations.

Why Redis for Caching in Node.js?

Speed: Redis lives in memory—operations are sub-millisecond.
Simplicity: Node.js has excellent Redis clients like ioredis.
Features: Supports expiration (TTL), which prevents stale data.
Real Impact: In a typical benchmark with 1000 requests, caching can reduce average response time from 205ms to 18ms—over 10x improvement.

Compared to in-process caches (like node-cache), Redis shines when you scale horizontally (multiple instances) or need persistence across server restarts.

Prerequisites

Node.js 18+ installed
A running Redis instance (local or Docker)
Basic knowledge of TypeScript and Express

Quickly spin up Redis with Docker:

docker run -d --name redis-cache -p 6379:6379 redis:7

Step 1: Project Setup

Create a new TypeScript project:

mkdir nodejs-redis-cache && cd nodejs-redis-cache
npm init -y
npm install express ioredis
npm install --save-dev typescript @types/node @types/express ts-node nodemon

Create tsconfig.json:

{
  "compilerOptions": {
    "target": "ES2020",
    "module": "commonjs",
    "lib": ["ES2020"],
    "outDir": "./dist",
    "rootDir": "./src",
    "strict": true,
    "esModuleInterop": true,
    "skipLibCheck": true,
    "forceConsistentCasingInFileNames": true,
    "resolveJsonModule": true
  },
  "include": ["src/**/*"],
  "exclude": ["node_modules"]
}

Update package.json scripts:

{
  "scripts": {
    "dev": "nodemon --exec ts-node src/server.ts",
    "build": "tsc",
    "start": "node dist/server.js"
  }
}

Create the src directory:

mkdir src

We'll use Express for the web framework and ioredis as our Redis client—it has excellent TypeScript support out of the box.

Step 2: Connect to Redis

Create src/redis.ts:

import Redis from 'ioredis';

// Create Redis client
const redis = new Redis({
  host: process.env.REDIS_HOST || 'localhost',
  port: parseInt(process.env.REDIS_PORT || '6379'),
  password: process.env.REDIS_PASSWORD || '', // No password for local dev
  // Automatic reconnection on connection loss
  retryStrategy: (times: number): number | null => {
    const delay = Math.min(times * 50, 2000);
    return delay;
  }
});

// Test connection
redis.on('connect', () => {
  console.log('Connected to Redis!');
});

redis.on('error', (err: Error) => {
  console.error('Redis connection error:', err);
});

export default redis;

Security note: We're using environment variables for configuration—never hardcode credentials in production.

Step 3: The Cache-Aside Pattern (The Simplest Way)

The most straightforward caching strategy is cache-aside:

Check Redis for data.
On hit → return cached value.
On miss → query database → write to Redis with TTL → return data.

Let's build a simple user endpoint that fetches from a mock DB.

First, define our types and mock database (src/mockdb.ts):

export interface User {
  id: number;
  name: string;
}

const users: Record<number, User> = {
  1: { id: 1, name: 'Paul' },
  2: { id: 2, name: 'Ada' }
};

// Simulate a slow database query
export async function getUserFromDB(id: number): Promise<User> {
  // Uncomment to simulate database latency
  await new Promise(resolve => setTimeout(resolve, 200));

  const user = users[id];
  if (!user) {
    throw new Error('User not found');
  }
  return user;
}

Now, the main application with caching (src/server.ts):

import express, { Request, Response } from 'express';
import redis from './redis';
import { getUserFromDB, User } from './mockdb';

const app = express();
const PORT = process.env.PORT || 3000;

// Middleware
app.use(express.json());

app.get('/user/:id', async (req: Request, res: Response): Promise<void> => {
  try {
    const { id } = req.params;

    // Validate ID
    const userId = parseInt(id);
    if (isNaN(userId)) {
      res.status(400).json({ error: 'Invalid ID' });
      return;
    }

    const key = `user:${id}`;

    // Step 1: Check cache
    const cached = await redis.get(key);

    if (cached) {
      // Cache hit!
      try {
        const user: User = JSON.parse(cached);
        res.json(user);
        return;
      } catch (parseError) {
        // If parse fails, log and fall through to DB query
        console.error('Cache parse error:', parseError);
      }
    }

    // Step 2: Cache miss → query DB
    const user = await getUserFromDB(userId);

    // Step 3: Serialize and cache with 5-minute TTL
    // TTL of 300 seconds (5 minutes) is reasonable for user data that doesn't change frequently.
    // Adjust based on your data volatility: use shorter TTLs for frequently updated data.
    try {
      await redis.setex(key, 300, JSON.stringify(user));
    } catch (cacheError) {
      // Log cache error but still return the data
      console.error('Cache set error:', cacheError);
    }

    res.json(user);

  } catch (error) {
    if (error instanceof Error && error.message === 'User not found') {
      res.status(404).json({ error: 'User not found' });
      return;
    }
    console.error('Server error:', error);
    res.status(500).json({ error: 'Internal server error' });
  }
});

app.listen(PORT, () => {
  console.log(`Server running on http://localhost:${PORT}`);
  console.log(`Test with: curl http://localhost:${PORT}/user/1`);
});

// Graceful shutdown
process.on('SIGTERM', async () => {
  console.log('SIGTERM received, closing Redis connection...');
  await redis.quit();
  process.exit(0);
});

That's it! Run with npm run dev and hit the endpoint multiple times—the first request hits the "DB" (with the 200ms delay), subsequent ones return instantly from Redis.

Testing the Performance Difference

Benchmark without cache (comment out the Redis check):

# Install autocannon: npm install -g autocannon
autocannon -c 10 -d 10 http://localhost:3000/user/1

# Results: ~45 requests/sec, ~220ms average latency

Benchmark with cache enabled:

autocannon -c 10 -d 10 http://localhost:3000/user/1

# Results: ~450 requests/sec, ~20ms average latency
# That's a 10x improvement!

Bonus: Cache Invalidation

To avoid stale data, invalidate the cache when you update a user:

interface UpdateUserBody {
  name?: string;
}

app.put('/user/:id', async (req: Request<{ id: string }, {}, UpdateUserBody>, res: Response): Promise<void> => {
  try {
    const { id } = req.params;
    const updates = req.body;

    const userId = parseInt(id);
    if (isNaN(userId)) {
      res.status(400).json({ error: 'Invalid ID' });
      return;
    }

    // ... update database ...

    // Invalidate cache
    const key = `user:${id}`;
    await redis.del(key);

    res.json({ message: 'User updated successfully' });
  } catch (error) {
    console.error('Update error:', error);
    res.status(500).json({ error: 'Failed to update user' });
  }
});

Alternatively, use shorter TTLs for data that changes frequently.

Advanced: Type-Safe Cache Wrapper

For cleaner, reusable code, create a generic cache wrapper with full TypeScript support (src/cacheWrapper.ts):

import redis from './redis';

interface CacheOptions {
  ttl: number; // Time to live in seconds
}

export async function cacheWrapper<T>(
  key: string,
  options: CacheOptions,
  fetchFunction: () => Promise<T>
): Promise<T> {
  // Try cache first
  const cached = await redis.get(key);

  if (cached) {
    try {
      return JSON.parse(cached) as T;
    } catch (error) {
      console.error('Cache parse error:', error);
      // Fall through to fetch
    }
  }

  // Cache miss - fetch data
  const data = await fetchFunction();

  // Cache the result
  try {
    await redis.setex(key, options.ttl, JSON.stringify(data));
  } catch (error) {
    console.error('Cache set error:', error);
    // Don't throw - we have the data
  }

  return data;
}

Then use it in your routes with full type safety:

import { cacheWrapper } from './cacheWrapper';
import { User } from './mockdb';

app.get('/user/:id', async (req: Request, res: Response): Promise<void> => {
  try {
    const { id } = req.params;
    const userId = parseInt(id);

    if (isNaN(userId)) {
      res.status(400).json({ error: 'Invalid ID' });
      return;
    }

    const user = await cacheWrapper<User>(
      `user:${id}`,
      { ttl: 300 }, // 5 minutes
      () => getUserFromDB(userId)
    );

    res.json(user);
  } catch (error) {
    if (error instanceof Error && error.message === 'User not found') {
      res.status(404).json({ error: 'User not found' });
      return;
    }
    res.status(500).json({ error: 'Internal server error' });
  }
});

The generic <T> ensures your cached data maintains its type throughout the application!

Performance Tips for Production

Connection pooling: ioredis handles connection pooling automatically. For high-concurrency scenarios, consider using a Redis cluster.
Monitoring: Use Redis Commander or RedisInsight for visual monitoring in production.
Serialization: We use JSON here for simplicity, but for high-throughput APIs, consider msgpack for faster serialization.
High availability: In production, use Redis Sentinel or Redis Cluster for failover and distributed caching.
Security: Enable TLS, use authentication, and never expose Redis directly to the internet. Always use environment variables for sensitive configuration.
Error handling: Always handle Redis errors gracefully—your app should continue working even if Redis is down.
Type safety: Leverage TypeScript's type system to catch cache-related bugs at compile time.

Environment Variables Setup

Create a .env file for local development:

PORT=3000
REDIS_HOST=localhost
REDIS_PORT=6379
REDIS_PASSWORD=

Install dotenv for environment variable management:

npm install dotenv

Load it at the top of src/server.ts:

import 'dotenv/config';
import express, { Request, Response } from 'express';
// ... rest of imports

Conclusion

Adding Redis caching to your TypeScript Node.js backend is straightforward with ioredis and can deliver massive performance improvements with minimal code changes. Start with the cache-aside pattern for read-heavy endpoints, benchmark the difference using tools like autocannon or artillery, and iterate from there.

The beauty of this approach is its simplicity—you're just adding a layer before your existing database calls. With TypeScript, you get the added benefit of type safety throughout your caching logic, catching potential bugs before they reach production.

Ready to see the difference? Clone the code above, run both cached and uncached versions, and benchmark them yourself. The performance gains speak for themselves.

What caching challenges have you faced in Node.js? Have you tried other patterns like write-through or read-through caching? Drop a comment below—I'd love to hear your experiences!

Happy caching!

Complete Project Structure

nodejs-redis-cache/
├── package.json
├── tsconfig.json
├── .env
├── .gitignore
└── src/
    ├── server.ts
    ├── redis.ts
    ├── mockdb.ts
    └── cacheWrapper.ts

Don't forget to add .gitignore:

node_modules/
dist/
.env

Run the complete example:

npm run dev  # Development with hot reload
npm run build && npm start  # Production build

Learn how to do this using Golang here

# How to Build a Clean REST API in Go: From Spaghetti Code to Clean Architecture

Paul Babatuyi — Sat, 17 Jan 2026 00:45:03 +0000

I'll be honest—my first REST API in Go was a disaster. Everything lived in main.go: database queries, business logic, HTTP handlers, all tangled together in a 1,500-line monstrosity. Adding a feature meant scrolling through endless code, hoping I wouldn't break something unrelated.

Then I discovered Clean Architecture, and it changed everything. This is the story of how I rebuilt my inventory management API with proper separation of concerns, making it actually maintainable. No fluff, no theory-only content—just the real lessons from building BMG-Go-Backend.

The Problem: Why I Needed Better Architecture

Picture this: you need to add OAuth login to your API. In my original spaghetti code, I'd have to:

Hunt through main.go for where users are created
Hope the password hashing wasn't hardcoded somewhere random
Add OAuth logic... where? Next to the database code? In a new file? Who knows!
Cross your fingers that you didn't break existing email/password login

Sound familiar? That pain drove me to redesign everything with layers.

The Solution: Clean Architecture in Go

Here's the key insight that clicked for me: separate what something does from how it does it.

Instead of:

// ❌ BAD: Everything mixed together
func CreateUser(w http.ResponseWriter, r *http.Request) {
    var user User
    json.NewDecoder(r.Body).Decode(&user)  // HTTP stuff

    if user.Email == "" {                  // Validation
        http.Error(w, "bad email", 400)
        return
    }

    db.Exec("INSERT INTO users...")        // Database stuff

    json.NewEncoder(w).Encode(user)        // More HTTP stuff
}

We do this:

// ✅ GOOD: Each layer does ONE thing
// Handler: HTTP concerns only
func (h *UserHandler) Create(w http.ResponseWriter, r *http.Request) {
    var dto CreateUserDTO
    parseJSON(r.Body, &dto)

    user, err := h.userService.Create(ctx, dto)
    writeJSON(w, 201, user)
}

// Service: Business logic only  
func (s *UserService) Create(ctx context.Context, dto CreateUserDTO) (*User, error) {
    if err := validateEmail(dto.Email); err != nil {
        return nil, err
    }
    return s.repo.Create(ctx, userFromDTO(dto))
}

// Repository: Database only
func (r *UserRepo) Create(ctx context.Context, user *User) error {
    _, err := r.db.Exec("INSERT INTO users...", user.Email, user.Name)
    return err
}

Now, when PM asks for OAuth? I just add a new method in UserService. The handler and repository don't even know it happened. Beautiful.

The Architecture: A Tour Through the Layers

Let me walk you through how requests flow through BMG. I'll use a real example: creating an inventory item.

Layer 1: Handler (The HTTP Bouncer)

Job: Convert HTTP requests into something the business logic can understand.

func (app *application) createItemHandler(w http.ResponseWriter, r *http.Request) {
    // 1. Parse JSON from HTTP request
    var input CreateItemDTO
    err := app.readJSON(w, r, &input)
    if err != nil {
        app.badRequestResponse(w, r, err)
        return
    }

    // 2. Ask the service to do the work
    item, err := app.itemService.Create(r.Context(), input)
    if err != nil {
        app.serverErrorResponse(w, r, err)
        return
    }

    // 3. Convert back to JSON and send HTTP response
    app.writeJSON(w, http.StatusCreated, item, nil)
}

Notice what's NOT here:

❌ No business rules ("quantity can't be negative")
❌ No SQL queries
❌ No password hashing or complex logic

Just: receive HTTP → call service → return HTTP. That's it.

Layer 2: Service (The Brain)

Job: Enforce business rules, orchestrate complex operations.

func (s *ItemService) Create(ctx context.Context, dto CreateItemDTO) (*Item, error) {
    // Business rule: can't create items with negative quantity
    if dto.Quantity < 0 {
        return nil, ErrInvalidQuantity
    }

    // Business rule: prices must be positive
    if dto.Price <= 0 {
        return nil, ErrInvalidPrice
    }

    // Transform DTO into domain model
    item := &Item{
        Name:        dto.Name,
        Description: dto.Description,
        Quantity:    dto.Quantity,
        Price:       dto.Price,
        CreatedAt:   time.Now(),
    }

    // Ask repository to save it
    return s.repo.Create(ctx, item)
}

This layer knows what should happen, but not how it happens. It doesn't care if we're using PostgreSQL, MongoDB, or a text file—that's the repository's problem.

Layer 3: Repository (The Database Whisperer)

Job: Talk to the database. Nothing else.

func (r *ItemRepository) Create(ctx context.Context, item *Item) (*Item, error) {
    query := `
        INSERT INTO items (name, description, quantity, price, created_at)
        VALUES ($1, $2, $3, $4, $5)
        RETURNING id, created_at
    `

    err := r.db.QueryRowContext(ctx, query,
        item.Name,
        item.Description,
        item.Quantity,
        item.Price,
        item.CreatedAt,
    ).Scan(&item.ID, &item.CreatedAt)

    return item, err
}

All the SQL lives here. If we switch from PostgreSQL to MySQL tomorrow, we only change this file. The service and handler don't even know we use a database.

The Secret Sauce: DTOs (Data Transfer Objects)

Here's something that confused me for months: why not just use domain models everywhere?

Bad idea. Here's why:

// Domain model: internal representation
type User struct {
    ID           string
    Email        string
    PasswordHash string  // ⚠️ We DO NOT want this in API responses!
    CreatedAt    time.Time
    LastLoginAt  *time.Time
}

// DTO: what we actually send over the wire
type UserResponseDTO struct {
    ID        string    `json:"id"`
    Email     string    `json:"email"`
    CreatedAt time.Time `json:"created_at"`
    // Notice: no password hash!
}

DTOs give you:

Security: Don't accidentally leak sensitive fields
Flexibility: API shape doesn't force database schema
Versioning: Support multiple API versions easily

I learned this the hard way when I accidentally returned password hashes in /users endpoint. Good times.

Middleware: The Pipeline Pattern

Middleware in Go is elegant. Each request passes through a chain of functions before hitting your handler:

Request → Logger → CORS → RateLimit → Auth → Handler → Response

Here's how simple it is with Chi router:

router := chi.NewRouter()

// Apply to ALL routes
router.Use(middleware.Logger)
router.Use(middleware.CORS)
router.Use(middleware.RateLimiter)

// Protected routes only
router.Group(func(r chi.Router) {
    r.Use(middleware.Auth)  // JWT validation

    r.Post("/items", app.createItemHandler)
    r.Put("/items/{id}", app.updateItemHandler)
})

// Public routes (no auth needed)
router.Get("/healthcheck", app.healthcheckHandler)

The Middleware I Wish I'd Built Earlier

1. Request Logger

func Logger(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        start := time.Now()

        next.ServeHTTP(w, r)

        log.Printf("%s %s %v", r.Method, r.URL.Path, time.Since(start))
    })
}

Seeing "POST /items 245ms" in logs saved my bacon when debugging slow requests.

2. Rate Limiter

func RateLimiter(next http.Handler) http.Handler {
    limiter := rate.NewLimiter(10, 20) // 10 req/s, burst of 20

    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        if !limiter.Allow() {
            http.Error(w, "Rate limit exceeded", http.StatusTooManyRequests)
            return
        }
        next.ServeHTTP(w, r)
    })
}

Prevented a junior dev's infinite loop from killing the server. True story.

Error Handling: The Go Way

Go's error handling gets mocked, but I've grown to love it. Here's my approach:

// Domain errors (business rules violated)
var (
    ErrInvalidQuantity = errors.New("quantity cannot be negative")
    ErrInvalidPrice    = errors.New("price must be positive")
    ErrNotFound        = errors.New("item not found")
)

// Service layer
func (s *ItemService) Create(ctx context.Context, dto CreateItemDTO) (*Item, error) {
    if dto.Quantity < 0 {
        return nil, ErrInvalidQuantity  // Business rule
    }

    item, err := s.repo.Create(ctx, item)
    if err != nil {
        return nil, fmt.Errorf("create item: %w", err)  // Wrap for context
    }

    return item, nil
}

// Handler layer
func (app *application) createItemHandler(w http.ResponseWriter, r *http.Request) {
    item, err := app.itemService.Create(r.Context(), input)
    if err != nil {
        switch {
        case errors.Is(err, ErrInvalidQuantity):
            app.badRequestResponse(w, r, err)  // 400
        case errors.Is(err, ErrNotFound):
            app.notFoundResponse(w, r)         // 404
        default:
            app.serverErrorResponse(w, r, err) // 500
        }
        return
    }

    app.writeJSON(w, http.StatusCreated, item, nil)
}

The key: errors flow up, decisions flow down. Services return errors, handlers decide HTTP status codes.

Database: Connection Pooling That Actually Works

This took me embarrassingly long to get right:

// ❌ BAD: Creating connection per request
func createItem(w http.ResponseWriter, r *http.Request) {
    db, _ := sql.Open("postgres", "...")  // Creates new connection!
    defer db.Close()
    db.Exec("INSERT...")
}

// ✅ GOOD: Pool created once at startup
func main() {
    db, err := sql.Open("postgres", connectionString)
    if err != nil {
        log.Fatal(err)
    }

    // Configure the pool
    db.SetMaxOpenConns(25)    // Max 25 connections
    db.SetMaxIdleConns(5)     // Keep 5 idle
    db.SetConnMaxLifetime(5 * time.Minute)

    // Share across all handlers
    app := &application{
        db: db,
    }
}

Went from 200ms queries to 15ms just by fixing this. Connection overhead is real.

Testing: What Actually Gets Tested

I don't test everything. Hot take, I know. Here's what I do test:

1. Business Logic (Service Layer)

func TestItemService_Create_InvalidQuantity(t *testing.T) {
    service := &ItemService{repo: &mockRepo{}}

    _, err := service.Create(ctx, CreateItemDTO{
        Name:     "Widget",
        Quantity: -5,  // Invalid!
    })

    if !errors.Is(err, ErrInvalidQuantity) {
        t.Errorf("expected ErrInvalidQuantity, got %v", err)
    }
}

2. Repository Integration Tests

func TestItemRepo_Create(t *testing.T) {
    db := setupTestDB(t)
    defer db.Close()

    repo := NewItemRepository(db)
    item := &Item{Name: "Test", Quantity: 10}

    created, err := repo.Create(context.Background(), item)

    assert.NoError(t, err)
    assert.NotEmpty(t, created.ID)
}

What I DON'T test: Handlers. They're just glue code. If the service works and the repository works, the handler will work.

Performance: The Numbers That Matter

Here's what I learned from production:

Before Optimization

Avg Response Time: 450ms
P95: 1.2s
Throughput: ~100 req/s

After Optimization

Avg Response Time: 45ms (10x improvement!)
P95: 180ms
Throughput: ~800 req/s

What made the difference:

Connection pooling (biggest win)
Context timeouts (prevents slow queries from piling up)
Proper indexing (added indexes on frequently queried columns)
Middleware ordering (auth before expensive operations)

Deployment: From Local to Production

The beauty of this architecture? It deploys anywhere.

Local development:

export DATABASE_URL="postgres://localhost/bmginventory"
go run cmd/api/main.go

Docker:

FROM golang:1.25-alpine AS builder
WORKDIR /app
COPY . .
RUN go build -o api cmd/api/main.go

FROM alpine:latest
COPY --from=builder /app/api /api
EXPOSE 4000
CMD ["/api"]

Kubernetes/Cloud:
Same binary, different config. That's the power of the 12-factor app.

Lessons Learned (The Hard Way)

1. Start with layers from day one
Refactoring spaghetti code is 10x harder than starting clean.

2. DTOs are worth the boilerplate
Yes, it's extra typing. No, it's not premature optimization. Saved me from leaking sensitive data.

3. Middleware ordering matters

// ✅ GOOD: Auth after rate limiting
router.Use(RateLimit)
router.Use(Auth)

// ❌ BAD: Auth before rate limiting
// Attackers can spam your auth DB!
router.Use(Auth)
router.Use(RateLimit)

4. Context is your friend

ctx, cancel := context.WithTimeout(r.Context(), 5*time.Second)
defer cancel()

item, err := app.itemService.Create(ctx, input)

Prevents runaway queries from killing your server.

5. Don't over-abstract
My first attempt had 5 layers. Ridiculous. Three is plenty: handler, service, repository.

What's Next?

This is just v1. The architecture makes it easy to add:

[ ] OAuth2 integration (just add a new auth method in service)
[ ] Caching with Redis (add a cache layer, repository doesn't change)
[ ] Event-driven features (publish events from service layer)
[ ] gRPC endpoints (reuse the service layer!)

That last one is key: good architecture is protocol-agnostic. Want both REST and gRPC? Just add handlers. Your business logic stays the same.

The Code

Everything's on GitHub: BMG-Go-Backend

Clone it. Break it. Make it better. That's how we all learn.

The folder structure:

cmd/api/          # HTTP server entrypoint
internal/
  ├── handler/    # HTTP → Service
  ├── service/    # Business logic
  ├── repository/ # Service → Database
  ├── domain/     # Core entities
  └── dto/        # API contracts

Quick start:

make migrate-up  # Setup DB
make run         # Start server
curl localhost:4000/v1/healthcheck

Final Thoughts

Clean architecture isn't about following rules religiously. It's about making your future self's life easier.

When you get that 3 AM support call because something broke, you want to know exactly where to look. Handler? Service? Repository? Clear boundaries = faster debugging.

When PM wants "just a small feature" that turns into refactoring half the codebase, you want layers that prevent cascading changes.

When you're onboarding a new dev, you want a structure so obvious they can contribute on day one.

That's what this architecture gave me. Hope it helps you too.

Questions? Disagree with my approach? Drop a comment. I'm especially curious about how others handle testing—I know my approach is minimal, and I'd love to hear alternatives.

Found this useful? Star the repo and follow me for more backend deep dives. Next up: adding OAuth2 to this exact API.

Want the gRPC version? Check out my UploadStream project where I use the same layered approach for high-performance file streaming.

Learn how to Build a Production-Ready File Upload Service with gRPC Streaming in Go

Paul Babatuyi — Thu, 15 Jan 2026 15:18:07 +0000

Last month, I uploaded a 400MB video to a service, and while watching the progress bar crawl, I thought: "What if the connection drops at 95%?" That frustration sparked this project—UploadStream, a file upload/download service built with gRPC streaming that handles large files gracefully, without eating all your server's memory.

Here's what makes this interesting: instead of buffering entire files in memory (the classic rookie mistake that crashes your server), we stream them in chunks. Think of it like a water pipe rather than a bucket—data flows through continuously without ever being held all at once.

The "Why gRPC?" Question Everyone Asks

When I told my friend I was building this with gRPC instead of REST, he looked at me like I'd chosen to write in assembly. Fair question though—REST is everywhere, why complicate things?

Here's the thing: imagine you're uploading a 500MB file. With traditional REST:

Your client reads the entire file into memory
Sends it in one massive POST request
The server receives it all at once
Both sides pray nothing crashes

With gRPC streaming:

Client reads 64KB chunks
Sends each chunk immediately
Server writes chunks as they arrive
Both sides breathe easy

The difference? REST feels like carrying 100 grocery bags in one trip (heroic but risky). gRPC streaming is like making multiple trips—less dramatic, way more reliable.

The Core Architecture: Not Your Typical File Upload

Let me walk you through what happens when someone uploads a file to UploadStream:

1. The Upload Flow (Client Streaming)

// First message: metadata
stream.Send(&UploadFileRequest{
    Metadata: &FileMetadata{
        Filename:    "vacation.mp4",
        ContentType: "video/mp4",
        Size:        524288000, // 500MB
        UserId:      "user-123",
    },
})

// Then: stream chunks
buffer := make([]byte, 64*1024) // 64KB
for {
    n, err := file.Read(buffer)
    if err == io.EOF { break }

    stream.Send(&UploadFileRequest{
        Chunk: buffer[:n],
    })
}

Notice what we're not doing? We're not loading the entire file first. Each 64KB chunk is read, sent, and forgotten. Memory usage stays flat even if you're uploading gigabyte-sized files.

On the server side, something cool happens:

// Receive metadata first
firstMsg, _ := stream.Recv()
metadata := firstMsg.GetMetadata()

// Create file and start writing immediately
fileID := uuid.New().String()
writer, _ := storage.CreateFile(fileID)

// Stream chunks directly to disk
for {
    msg, err := stream.Recv()
    if err == io.EOF { break }

    writer.Write(msg.GetChunk())
}

We're writing to disk as chunks arrive. No buffering. No memory bloat. Just a continuous flow from client → network → server → disk.

2. The Download Flow (Server Streaming)

Downloads work in reverse. The server reads the file in chunks and streams them back:

// Send file info first
stream.Send(&DownloadFileResponse{
    Info: &FileInfo{
        Filename: "vacation.mp4",
        Size: 524288000,
    },
})

// Stream chunks
buffer := make([]byte, 64*1024)
for {
    n, _ := reader.Read(buffer)
    stream.Send(&DownloadFileResponse{
        Chunk: buffer[:n],
    })
}

The client receives chunks and writes them to disk immediately. Again, no massive memory buffers. This is how services like Google Drive and Dropbox can handle massive files without imploding.

The Devil's in the Details: Production-Ready Features

Building a toy upload service is easy. Making it production-ready is where things get spicy. Here's what I learned the hard way:

Content Type Validation (Or: How I Stopped Trusting Users)

Early on, I trusted whatever content_type the client sent. Bad idea. Someone uploaded a JavaScript file claiming it was an image. Security nightmare.

Now we do magic byte validation:

// Read first 512 bytes
buffer := make([]byte, 512)
n, _ := reader.Read(buffer)

// Detect actual type from content
actualType := http.DetectContentType(buffer[:n])

// Compare with declared type
if !isContentTypeMatch(actualType, declaredType) {
    return errors.New("type mismatch")
}

The http.DetectContentType function is fascinating—it looks at file signatures (magic bytes). For example, PNG files always start with \x89PNG\r\n\x1a\n. If someone claims they're uploading a PNG but the bytes don't match, we reject it.

Size Limits (Memory Safety First)

You must enforce size limits, both per-chunk and total:

const (
    maxFileSize  = 512 * 1024 * 1024 // 512MB
    maxChunkSize = 4 * 1024 * 1024   // 4MB (gRPC limit)
)

// Check each chunk
if chunkLen > maxChunkSize {
    return status.Errorf(codes.InvalidArgument,
        "chunk too large: %d bytes", chunkLen)
}

// Check total doesn't exceed declared
if totalSize + chunkLen > metadata.Size {
    return status.Error(codes.InvalidArgument,
        "size mismatch")
}

Without these checks, a malicious client could declare a 1KB file then send 10GB. Your server would happily write all 10GB to disk before realizing something's wrong.

Graceful Cancellation

Here's a subtle bug that bit me: what if the client disconnects mid-upload? Without proper context handling, the server keeps writing chunks that will never complete.

for {
    select {
    case <-ctx.Done():
        storage.DeleteFile(fileID) // Clean up partial file
        return status.Errorf(codes.Canceled, 
            "upload canceled: %v", ctx.Err())
    default:
        msg, err := stream.Recv()
        // ... process chunk
    }
}

That select with ctx.Done() is crucial. It lets us detect cancellations immediately and clean up. Without it, you get orphaned partial files littering your storage.

Background Processing: The Async Magic

Once a file is uploaded, we don't just store it and call it a day. For images, we generate thumbnails. For videos, we could extract metadata. This happens asynchronously using a background worker pattern:

// After successful upload
db.CreateProcessingJob(ctx, fileID)

// Worker polls for jobs
for {
    job := db.GetNextPendingJob()
    if job == nil { 
        time.Sleep(2 * time.Second)
        continue 
    }

    // Process image
    processImage(job.FileID)
}

This is a simple polling approach. In production, you'd probably use a proper job queue (RabbitMQ, Redis Streams, etc.), but this illustrates the pattern. The key insight: never block the upload waiting for processing. Accept the file, return success, process later.

Observability: Because "It Works On My Machine" Doesn't Cut It

I shipped the first version without proper observability. When users reported slow uploads, I had zero visibility into what was happening. Don't make this mistake.

Structured Logging with Zap

logger.Info("upload started",
    zap.String("file_id", fileID),
    zap.String("user_id", userID),
    zap.Int64("size", metadata.Size),
    zap.String("content_type", metadata.ContentType),
)

Those structured fields are gold for debugging. You can now query logs like:

grep "upload started" | grep "user_id=problem-user"

Metrics with Prometheus

grpc_server_handled_total{grpc_method="UploadFile",grpc_code="OK"} 1523
grpc_server_handling_seconds_bucket{le="1.0"} 1201
grpc_server_handling_seconds_bucket{le="5.0"} 1523

These metrics tell stories:

"We've handled 1,523 uploads, all successful"
"1,201 completed in under 1 second"
"322 took 1-5 seconds (investigate these)"

Distributed Tracing

When a request spans multiple services, tracing shows you the whole journey:

Client → gRPC Server → Database → Storage → Worker
  50ms      20ms         10ms      300ms     2000ms
                                    ^ Found the bottleneck!

Without tracing, you'd be guessing. With it, you know storage writes are slow.

The Database Dance: PostgreSQL Patterns

File metadata lives in PostgreSQL. Here's a non-obvious decision: we use soft deletes instead of hard deletes.

CREATE TABLE files (
    id UUID PRIMARY KEY,
    user_id TEXT NOT NULL,
    filename TEXT NOT NULL,
    size BIGINT NOT NULL,
    uploaded_at TIMESTAMPTZ DEFAULT NOW(),
    deleted_at TIMESTAMPTZ  -- NULL = active, set = deleted
);

CREATE INDEX idx_files_user_id 
ON files(user_id) 
WHERE deleted_at IS NULL;  -- Partial index FTW

Why soft delete?

Recovery: "I accidentally deleted my thesis!" → We can restore it
Audit trails: Who deleted what, when?
Analytics: Understand deletion patterns

That partial index (WHERE deleted_at IS NULL) is clever—it only indexes active files, making queries faster and saving space.

Deployment: Docker Compose to Kubernetes

Development uses Docker Compose:

services:
  uploadstream:
    build: .
    ports:
      - "50051:50051"
    environment:
      UPLOADSTREAM: "postgres://..."
    depends_on:
      - postgres

Production uses Kubernetes with:

Horizontal Pod Autoscaling: Scale 2-5 pods based on CPU
Persistent Volumes: File storage survives pod restarts
StatefulSet for PostgreSQL: Stable network identity
Health checks: Liveness and readiness probes

The Kubernetes manifests were painful to write but worth it. Auto-scaling alone saved us during a traffic spike—pods scaled from 2 to 5 automatically when CPU hit 70%.

What I'd Do Differently Next Time

1. Use S3 from the start

Filesystem storage works for prototypes, but S3 (or equivalent) gives you:

Infinite scaling
Built-in redundancy
CDN integration
Better security

2. Implement resumable uploads

If a 2GB upload fails at 99%, the user shouldn't start over. I'd add:

Upload session IDs
Chunk checksums
Resume from last successful chunk

3. Add rate limiting

Nothing stops a user from uploading 1000 files simultaneously and crushing the server. I'd add per-user rate limits using token buckets.

4. Better error messages

codes.InvalidArgument is vague. Users need: "File size 600MB exceeds limit of 512MB" not "invalid request."

Key Takeaways

If you're building something similar:

Stream everything – Don't buffer large data in memory
Validate aggressively – Trust nothing from clients
Fail gracefully – Handle cancellations, timeouts, errors
Observe everything – You can't fix what you can't see
Plan for async – Long-running tasks should be background jobs

The full code is on GitHub. Clone it, break it, improve it. That's how we all learn.

gRPC streaming isn't magic—it's just a really elegant way to handle continuous data flows. Once you wrap your head around the client-stream and server-stream patterns, a whole new world of possibilities opens up: live video feeds, real-time analytics, progressive data processing.

Now go build something cool with it. And when your upload hits 99% and the connection drops, you'll smile knowing your service handles it gracefully.

Questions? Thoughts? Disagreements? Drop a comment below. I'm particularly interested if you've solved the resumable upload problem elegantly—I'm still researching best practices there.

Found this helpful? Star the repo and follow me for more deep dives into backend systems.