DEV Community: Lakshya Negi

5 Things I Learned Building a Database File Format from Scratch

Lakshya Negi — Thu, 19 Jun 2025 06:17:25 +0000

Last month, I decided to build a key-value database from scratch. Not because the world needs another database, but because I wanted to understand what actually happens behind the scenes when you call db.get() or db.put().

After weeks of wrestling with file formats, serialization, and the surprisingly complex world of "simple" storage systems, I've learned some hard lessons that no textbook quite prepared me for. Here are the five biggest insights that changed how I think about databases.

1. Your File Format Design Choices Haunt You Forever

When I started, I thought file format design would be the easy part. "Just throw some bytes in a file, right?" Wrong. Every single decision you make in your file format becomes permanent baggage that you'll carry for the life of your database.

I initially designed a complex header with 15 different fields:

Creation timestamp, last modified time, record count,
page count, configuration flags, user metadata,
version numbers, checksums...

It felt thorough and professional. Then I tried to implement it.

The reality: Most of those fields were never used, and maintaining them added complexity everywhere. Worse, I realized I'd committed to this format forever—any change would break compatibility with existing files.

The lesson: Start with the absolute minimum. For my key-value database, I ended up with just 16 bytes:

Bytes 0-7:   File signature ("KVDB2024")
Bytes 8-11:  First freelist page pointer
Bytes 12-15: First data page pointer

That's it. Everything else can be added later if you actually need it. Your future self will thank you for keeping it simple.

2. Endianness Will Bite You When You Least Expect It

I'm embarrassed to admit how long it took me to figure out why my database worked perfectly on my laptop but produced garbage on my friend's ARM-based server.

The culprit? I was storing integers without specifying byte order:

// This breaks on different architectures
func writePageNumber(buf []byte, pageNum uint32) {
    *(*uint32)(unsafe.Pointer(&buf[0])) = pageNum
}

The number 305419896 was being read as 2018915346 on the ARM machine. Same bits, different interpretation.

The fix was simple but crucial:

func writePageNumber(buf []byte, pageNum uint32) {
    binary.LittleEndian.PutUint32(buf, pageNum)
}

The lesson: Always, always, ALWAYS specify your byte order explicitly. Even if you only plan to run on one type of machine, you'll eventually want to share database files or deploy somewhere else. Make endianness a conscious choice from day one.

3. Error Handling Is More Important Than the Happy Path

My first implementation focused entirely on making things work correctly. Reading files, writing data, parsing headers—when everything went right, it was beautiful.

Then I started testing edge cases:

What if the file gets truncated?
What if someone tries to open a JPEG as a database?
What if the disk runs out of space mid-write?
What if the process crashes during a header update?

My database crashed, corrupted data, or silently accepted garbage input in every single scenario.

The turning point came when I realized that error handling isn't just about making your code robust—it's about making your database trustworthy. A database that sometimes loses data is worse than no database at all.

func openExistingDatabase(file *os.File) (*Database, error) {
// Read header
    headerData := make([]byte, 16)
    if _, err := file.ReadAt(headerData, 0); err != nil {
        return nil, fmt.Errorf("failed to read header: %w", err)
    }

// Validate every field
    if len(headerData) < 16 {
        return nil, fmt.Errorf("header too short")
    }

// Check signature
    expectedSig := [8]byte{'K', 'V', 'D', 'B', '2', '0', '2', '4'}
    if !bytes.Equal(headerData[0:8], expectedSig[:]) {
        return nil, fmt.Errorf("invalid file signature")
    }

// More validation...
}

The lesson: Write your error handling first, then implement the happy path. If your database can't fail gracefully, it can't be trusted with real data.

4. File I/O Is Asynchronous (Even When It Looks Synchronous)

This one nearly gave me a heart attack during testing.

I was running a simple test: write some data, immediately cut power to the machine, then check if the data survived. It didn't. Even though my file.Write() calls returned successfully, the data never made it to disk.

The problem: Operating systems buffer writes for performance. When you call write(), the OS says "sure, I'll get to that" and immediately returns success. Your data might sit in a buffer for seconds before actually hitting the disk.

For most applications, this is fine. For databases, it's catastrophic.

The solution: Learn to love fsync():

func writeHeader(file *os.File, header *DatabaseHeader) error {
    headerData := header.Serialize()

// Write to OS buffer
    if _, err := file.WriteAt(headerData, 0); err != nil {
        return err
    }

// Force write to disk
    return file.Sync()// This is the magic
}

The lesson: If you care about durability, you must explicitly force data to disk. Every critical operation should end with a sync. Yes, it's slower. No, you can't skip it if you want your database to survive power failures.

5. Simplicity Is a Feature, Not a Bug

Throughout this project, I constantly felt pressure to add features. "Real databases have indexing, so I need indexing." "Production systems need compression, so I need compression." "Enterprise databases support transactions, so I need transactions."

This feature creep nearly killed my project.

The breakthrough came when I stepped back and asked: "What's the simplest thing that could possibly work?"

For a key-value database, that turned out to be surprisingly minimal:

A file header pointing to the start of data
Fixed-size pages containing variable-length records
A simple append-only storage model

No fancy indexing (yet). No compression (yet). No complex transactions (yet). Just a system that can reliably store and retrieve key-value pairs.

The surprise: This simple design was faster, more reliable, and easier to debug than any of my complex attempts. More importantly, it actually worked.

The lesson: Every feature you don't implement is a feature that can't break. Build the simplest thing first, then add complexity only when you actually need it. Your simple database that works is infinitely better than your complex database that doesn't.

What I'd Tell My Past Self

If I could go back and give myself advice before starting this project:

Start with file format design, but keep it minimal
Specify endianness explicitly from day one
Write error handling before implementing features
Always sync critical writes to disk
Resist the urge to add features until the basics work perfectly

Building a database taught me that the hardest part isn't the algorithms or data structures—it's handling all the ways things can go wrong in the real world. Files get corrupted, processes crash, disks fill up, and users try to open the wrong files.

A good database isn't just a system that works when everything goes right. It's a system that fails gracefully when everything goes wrong.

Database Internals: Understanding Data Serialization

Lakshya Negi — Mon, 26 May 2025 11:30:52 +0000

"Programs must be written for people to read, and only incidentally for machines to execute." — Harold Abelson

Picture this: you've just created an amazing in-memory data store that works flawlessly during your program's execution. But here's the catch—every restart sends your precious data into oblivion. Sound familiar? This is where data serialization becomes your lifeline.

Today, we're exploring the foundation that makes database persistence possible. While it might seem like a detour from actual database construction, mastering serialization is essential for any storage system that needs to survive beyond a single program execution.

Understanding Data Serialization

Think about moving houses with fragile antiques. You have two options: risk carrying everything intact (and potentially losing it all with one mistake), or carefully wrap each piece, label it, and reassemble everything at your destination.

Data serialization follows the second approach for your program's information.

Serialization transforms your program's data structures into a storable format. Deserialization reverses this process, recreating your original structures from the stored representation.

Your program's data exists in memory as interconnected structures with references and relationships. Memory, however, is ephemeral—it vanishes when your program terminates. To achieve persistence, you must convert this living data into a format suitable for long-term storage.

Why Database Systems Prioritize Serialization

Database systems are persistence engines. Users expect their saved documents, updated profiles, and stored preferences to remain accessible across sessions, days, and years. Without effective serialization, databases would lose all information between runs.

Different serialization approaches offer varying benefits. Human-readable formats like JSON excel for debugging but consume more space and processing time. Binary formats prioritize efficiency over readability—a sensible choice since computers handle most database operations.

Effective database serialization requires:

Speed: Operations occur continuously, making performance critical
Compact storage: Efficient space usage reduces costs and improves I/O performance
Platform independence: Databases should function across different systems
Data integrity: Corruption detection prevents silent failures

Building Custom Binary Serialization

For our educational database project, we're implementing a custom binary format. While production systems typically use established libraries like Protocol Buffers or Apache Avro, building from scratch reveals the underlying principles.

Our scenario is intentionally straightforward: storing key-value pairs in a single file. This simplicity eliminates complex requirements like nested structures or schema evolution, allowing us to focus on core serialization concepts.

Length-prefixed binary approach

Our strategy centers on length-prefixed encoding for variable-sized data.

Consider reading a magazine without page numbers or a table of contents. Finding a specific article requires scanning from the beginning. However, if each article began with "This article contains 237 words," you could navigate directly to any content.

Length-prefixed encoding applies this principle to data storage, enabling efficient random access without sequential parsing.

Practical Implementation

Let's implement serialization for a representative data structure:

type UserProfile struct {
    Username string
    UserID   uint32
    Email    string
    Active   bool
}

Our binary layout follows this pattern:

[4 bytes: Username length][Username data][4 bytes: UserID]
[4 bytes: Email length][Email data][1 byte: Active flag]

Implementation example:

func (u *UserProfile) ToBytes() ([]byte, error) {
    usernameData := []byte(u.Username)
    emailData := []byte(u.Email)

    requiredSize := 4 + len(usernameData) + 4 + 4 + len(emailData) + 1
    result := make([]byte, requiredSize)

    position := 0

    // Store username length and content
    binary.LittleEndian.PutUint32(result[position:], uint32(len(usernameData)))
    position += 4
    copy(result[position:], usernameData)
    position += len(usernameData)

    // Store user ID
    binary.LittleEndian.PutUint32(result[position:], u.UserID)
    position += 4

    // Store email length and content
    binary.LittleEndian.PutUint32(result[position:], uint32(len(emailData)))
    position += 4
    copy(result[position:], emailData)
    position += len(emailData)

    // Store active status
    if u.Active {
        result[position] = 1
    } else {
        result[position] = 0
    }

    return result, nil
}

Byte order considerations: Using binary.LittleEndian ensures consistent interpretation across different processor architectures. Without this specification, identical data might represent completely different values on different systems.

Core Principles Demonstrated

This implementation showcases several fundamental database concepts:

Deterministic structure: Every field's location is calculable without parsing preceding data, enabling efficient random access.

Memory efficiency: Pre-calculating storage requirements allows single allocation, avoiding fragmentation and reallocations.

Error resilience: Comprehensive bounds checking during deserialization protects against corrupted or malicious input.

Cache-friendly access: Sequential reading and writing patterns optimize both disk I/O and processor cache utilization.

Scaling to Database Operations

The techniques demonstrated with UserProfile records directly apply to database functionality:

Variable usernames translate to variable-length keys
Email storage becomes arbitrary value storage
Fixed integers handle metadata and timestamps
Boolean flags manage record state information

Length-prefixed encoding that efficiently handles user profiles will equally well manage application configurations, user preferences, or any key-value data our database stores.

Moving Forward

We've established a robust serialization foundation using principles that power real-world database systems. Our next step involves designing the complete architecture for our single-file key-value database, demonstrating how serialization integrates with other storage system components.

The principles we've explored—predictable parsing, efficient encoding, comprehensive error handling—scale from simple data structures to enterprise-grade database systems. These fundamentals will guide every architectural decision in our database construction journey.

Understanding serialization transforms abstract storage concepts into concrete, implementable solutions. With this foundation, we're ready to build a complete persistence system that reliably stores and retrieves data across program executions.

This article is part of an ongoing series about constructing a key-value database from fundamental principles. Join us as we progress from basic concepts to a fully functional storage system.

How To Implement JWT Authentication In Command-Line Applications

Lakshya Negi — Thu, 24 Apr 2025 04:53:21 +0000

If you've ever built or used a command-line interface (CLI) tool, you know that authentication can be a tricky puzzle to solve. Getting this authentication flow right makes the difference between a tool people love using and one they constantly fight with.

In this post, I'll explore what seems to be the most practical approach to CLI authentication: combining initial authentication (either through email/password or OAuth) with a dual-token system. While other methods certainly work for specific use cases, this approach appears to be a sweet spot for many applications.

🛡️ An Effective Authentication Pattern for CLI Apps

After researching various authentication methods for CLI applications, this approach seems most balanced for modern tools:

The Dual-Token Pattern:

Initial Authentication via email + password or OAuth

Token-Based Authorization with two distinct tokens:

Short-lived access token (15-60 minutes)

Longer-lived refresh token (days to weeks)

This dual-token approach provides solid security while not constantly interrupting users with annoying re-authentication prompts.

🔄 How It Works: The Authentication Flow

Let's explore how this pattern works in a typical CLI application:

📝 1. Initial Authentication

When a user first runs an authenticated command, they're prompted to authenticate:

# User attempts to access protected resource
$ myapp resources list
Not authenticated. Please log in first.

# User authenticates with email and password
$ myapp login
Email: user@example.com
Password: ********
Login successful!

Behind the scenes, the CLI sends these credentials to the authentication server, which validates them and returns both an access token and a refresh token.

💡 Pro Tip: You could also use OAuth here if the service supports it. In that case, the CLI would open a browser window where the user can complete authentication through a standard web login flow.

💾 2. Token Storage

After authentication, both tokens need to be stored securely. A common approach is to store them in a config file in the user's home directory:

~/.config/myapp/credentials.json

The structure typically looks something like this:

{
  "access_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...",
  "refresh_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...",
  "access_expires": 1682349600
}

⚠️ Security Note: Always set appropriate file permissions (0600) to prevent other users from accessing these tokens!

For increased security, other options include:

🔒 Storing tokens only in memory (though this requires keeping the CLI running)
⏱️ Setting shorter refresh token expiration (e.g., 12 hours instead of days)
🗝️ Using the system's secure keychain rather than a plain file

🌐 3. Making Authenticated Requests

With tokens securely stored, the CLI can now make authenticated requests to the API:

Before making any request, check if the access token has expired
If expired, automatically use the refresh token to get a new access token
If refresh fails (e.g., refresh token expired), prompt for login
Otherwise, include the valid access token in the API request

This automatic refresh mechanism is what makes this approach so user-friendly. When functioning correctly, users rarely need to manually log in. The CLI handles token management behind the scenes, and everything just works.

🎯 Conclusion

Combining email/password (or browser-based OAuth) authentication with the dual-token approach seems to provide a good balance of security and usability for most applications. This approach offers several benefits:

🔒 Limited security exposure through token expiration
😌 Seamless user experience with behind-the-scenes token refreshes
🎮 Server-side control over session management
💻 Consistent behaviour across MacOS, Windows, and Linux

This is just a starting point, though. Depending on security requirements, this basic flow could be enhanced by using system keychains instead of config files, implementing MFA for sensitive operations, or adding device registration capabilities.

Key Takeaway: The beauty of the dual-token approach is how adaptable it is - starting simple and adding security layers as needs evolve seems like a sensible path.

Have you tried implementing CLI authentication? Did you find storing tokens in .config directories sufficient for your needs? Let me know in the comments!

P.S. Remember that authentication is just one piece of a secure CLI. No matter which approach you choose, always use HTTPS, implement proper error handling, and follow secure coding practices throughout your application.

8 Memory-Efficient Go Coding Techniques For Better GC Performance

Lakshya Negi — Mon, 10 Mar 2025 19:09:43 +0000

Go’s garbage collector is impressively engineered for low latency and minimal pauses, but even the best GC can become a bottleneck when your code generates excessive garbage. After optimizing numerous Go applications, these eight practical techniques consistently help reduce GC pressure while maintaining clean, readable code.

1. Minimize Allocations

Every allocation creates work for the garbage collector. The Go GC must trace through all reachable objects using its tri-colour marking system, and each allocation increases this workload.

During a collection cycle, Go’s GC must:

Mark all new allocations as “white” (potentially collectable)
Trace through object graphs to find reachable objects
Move objects between white→gray→black states as it determines reachability

Each additional object increases this workload linearly, and complex object graphs increase it even more.

I’ve seen significant performance improvements simply by reusing variables:

// Instead of this:
for _, item := range items {
    result := process(item)  // Creates a new variable each iteration    
    // use result
}

// Do this:
var result SomeType
for _, item := range items {    
    result = process(item)  // Reuses the same variable    
    // use result
}

This simple change can dramatically reduce GC pressure in hot paths processing thousands of items.

2. Pre-allocate Slices and Maps

When a slice or map grows beyond its capacity, Go allocates a new, larger backing array and copies the data. The old backing array becomes garbage.

Go’s slice growth strategy typically doubles the capacity when more space is needed. This means:

A slice growing from 1,000 to 1,001 elements will allocate a new 2,000-element array
The original 1,000-element array becomes garbage
This creates a “sawtooth pattern” in memory usage that triggers the GC more frequently

The GC pacing algorithm detects these allocation spikes and may trigger collection cycles in response, causing more frequent pauses.

When a data processing pipeline was triggering GC cycles too frequently, pre-allocation solved the problem:

// Instead of this:
users := []User{}  // Will grow and reallocate multiple times

// Do this:
users := make([]User, 0, 1000)  // Pre-allocate capacity for 1000 users

This simple change reduced GC frequency by 30% in our services.

3. Object Pooling for Frequently Used Objects

Short-lived objects like buffers in a web server handling thousands of requests create constant allocation pressure.

The Go GC uses a target heap growth formula to decide when to collect:

next_gc = live_heap * (1 + GOGC/100)

Where GOGC is typically 100 (meaning the heap can double before collection).

By pooling objects, you:

Lower the rate at which this target is approached
Reduce “marking” work since pooled objects stay marked as “black” (in-use)
Lower the heap fragmentation that occurs when objects are frequently allocated and freed

var bufferPool = sync.Pool{    
    New: func() interface{} {        
        return make([]byte, 4096)    
    },
}

func processRequest() {    
    buffer := bufferPool.Get().([]byte)    
    defer bufferPool.Put(buffer)    
    // Use buffer...
}

This pattern reduced p99 latency by 15% in a high-throughput API gateway.

4. Avoid String Concatenation in Loops

Strings in Go are immutable. Each concatenation creates a new string, leaving the old ones for garbage collection.

Internally, strings in Go are represented as:

type stringStruct struct {    
    str unsafe.Pointer // Pointer to the backing byte array    
    len int            // Length of the string
}

Each concatenation:

Allocates a new backing array
Copies both original strings into it
Creates a new string header
Leaves the original strings to be collected

This is particularly problematic for the GC because strings often have short lifetimes during concatenation, which can trigger the “generational hypothesis” problem that Go’s non-generational collector doesn’t handle optimally.

In a report generator, switching from concatenation to strings.Builder made a dramatic difference:

// Instead of this:
var result string
for i := 0; i < 1000; i++ {    
    result += fmt.Sprintf("%d,", i)  // Creates garbage on each iteration
}

// Do this:
var builder strings.Builder
builder.Grow(10000)  // Pre-allocate approximate size
for i := 0; i < 1000; i++ {    
    fmt.Fprintf(&builder, "%d,", i)
}
result := builder.String()

This change turned report generation from seconds to milliseconds.

5. Be Careful with Closures

Closures capture variables by reference, potentially keeping entire object graphs alive.

When the Go GC determines reachability, it follows all pointers from root objects. A closure is implemented as a struct containing:

A function pointer
References to all captured variables

The GC must mark everything reachable from these captured variables, which can create “accidental retention” where large structures remain in memory because a small part is referenced by a closure.

When debugging a memory leak, I found a seemingly innocent event handler was capturing a large data structure:

// Instead of this:
largeObject := createLargeObject()
handler := func(w http.ResponseWriter, r *http.Request) {    
// largeObject is captured and can't be garbage collected    
    fmt.Fprintf(w, "Using %v", largeObject.Name)
}

// Do this:
type Handler struct {    
    name string  // Store just what you need
}

largeObject := createLargeObject()
handler := &Handler{
    name: largeObject.Name,
}
// Now largeObject can be garbage collected

This pattern reduced memory usage by hundreds of megabytes in a large application.

6. Use Value Types for Small Objects

Value types can live on the stack rather than the heap, bypassing the GC entirely.

Go uses escape analysis during compilation to determine where values should be allocated:

Stack allocations have zero GC overhead as they’re automatically reclaimed when functions return
Heap allocations are subject to the GC’s mark-and-sweep algorithm

The compiler’s escape analysis has specific rules for determining when values escape to the heap:

Values whose addresses are taken and might outlive the function
Values too large for the stack
Values with unknown size at compile time

// Instead of this:
func createUser() *User {    
    return &User{Name: "Bob"}  // Escapes to the heap
}

// Do this when possible:
func createUser() User {    
    return User{Name: "Bob"}  // May stay on the stack
}

Applying this pattern throughout a codebase reduced GC overhead by 20%.

7. Batch Processing for High-Volume Operations

Processing items one by one often leads to more allocations and poor memory locality.

Go’s GC performance is directly tied to working set size and memory access patterns:

The GC must scan all objects to determine liveness
Better memory locality leads to better CPU cache utilization during GC scans
Batch processing reduces the “mark” phase duration by localizing memory access

Additionally, batching can reduce the total number of write barriers executed. Write barriers are extra instructions that the GC inserts to track pointers written during concurrent collection, and they add overhead to normal execution.

// Instead of this:
for _, item := range hugeList {    
    processItem(item)  // May allocate each time
}

// Do this:
batchSize := 100
for i := 0; i < len(hugeList); i += batchSize {    
    end := i + batchSize
  if end > len(hugeList) {        
      end = len(hugeList)
    }    
    processBatch(hugeList[i:end])  // Process many items with fewer allocations
}

For large datasets, this approach reduced processing time by 40%.

8. Minimize Interface Conversions

Interface values in Go consist of both a type pointer and a data pointer, requiring heap allocations in many cases.

Internally, interfaces are represented as:

type iface struct {    
    tab  *itab          // Contains type information    
    data unsafe.Pointer // Points to the actual data
}

Each interface conversion potentially allocates this structure. Furthermore, boxing values into interfaces often forces stack-allocated values to escape to the heap, where they become subject to garbage collection.

The GC must trace both the interface structure and the underlying value, doubling the work compared to direct value usage.

// Instead of this:
func process(items []interface{}) {  // Requires boxing each item    
    for _, item := range items {        
        // ...    
    }
}

// Do this:
func process[T any](items []T) {  // Uses generics, no boxing    
    for _, item := range items {        
        // ...    
    }
}

Refactoring to use generics instead of empty interfaces significantly reduced allocations and GC overhead.

Conclusion

Go’s garbage collector is remarkably efficient, but understanding how it works and writing code that generates less garbage can dramatically improve performance. These techniques aren’t about fighting the garbage collector—they’re about working harmoniously with it by generating less garbage in the first place.

Always measure before and after applying these optimizations. Use Go’s built-in profiling tools to identify where GC is becoming a bottleneck, then apply these patterns strategically. Not every application will need all of these optimizations, but understanding them will make you a more effective Go developer, especially when working on performance-critical systems.

What GC-friendly techniques have you found effective in your Go applications? Share your experiences in the comments below!

The Ultimate Guide To Custom Error-Handling System In Go APIs

Lakshya Negi — Sun, 09 Feb 2025 11:20:26 +0000

Imagine you’re using an API, and you get this response:

{
  "error": "Something went wrong."
}

Helpful, right? (Spoiler: It’s not.)

A good error response tells you:

• What went wrong: “The resource you’re looking for doesn’t exist.”

• What to do next: “Check the URL or resource ID.”

• That the API was built with care: “Oh hey, this team really thought this through.”

That’s why I designed a system that provides consistent, actionable, and meaningful error responses. Let’s dive in!

Designing an Actionable Error Handling System

At the heart of this system is the CustomError struct. Think of it as a single, organized “package” for all the details you need to communicate an error.

CustomError Struct Breakdown

Here’s what it looks like:

type CustomError struct {
    BaseErr     error                  // Underlying Base error
    StatusCode  int                    // HTTP status code
    Message     string                 // Detailed error message
    UserMessage string                 // User-friendly error message
    ErrType     string                 // Error type
    ErrCode     string                 // Unique error code
    Retryable   bool                   // Retryable flag
    Metadata    map[string]interface{} // Additional metadata
}

In plain English:

• BaseErr: The raw, underlying error.

• StatusCode: HTTP status codes like 404 (Not Found) or 500 (Internal Server Error).

• UserMessage: The “don’t scare the user” version of the error message.

• ErrType & ErrCode: Help with error categorization (handy when debugging).

• Retryable: Indicates whether it’s worth trying the request again.

Creating Your Own Errors

The system includes a factory function to simplify the creation of CustomError instances. This ensures all errors are initialized with a consistent structure:

func New(statusCode int, message, userMessage, errType, errCode string, retryable bool) *CustomError {
    return &CustomError{
        BaseErr:     fmt.Errorf("error: %s", message),
        StatusCode:  statusCode,
        Message:     message,
        UserMessage: userMessage,
        ErrType:     errType,
        ErrCode:     errCode,
        Retryable:   retryable,
        Metadata:    make(map[string]interface{}),
    }
}

Let’s make this more concrete. Here’s how you can create a “Resource Not Found” error using the CustomError:

func NewNotFoundError(resource string) *CustomError {
    return New(
        404,
        fmt.Sprintf("%s not found", resource),
        "The requested resource could not be found.",
        "NOT_FOUND",
        "ERR_NOT_FOUND",
        false,
    )
}

This makes sure that any “not found” error returns a consistent, meaningful response.

While the New function is ideal for creating errors from scratch, there are cases where you may need to wrap an existing error into a CustomError. For this purpose, the NewFromError function is provided.

The NewFromError function takes an existing error to create a CustomError:

func NewFromError(err error, statusCode int, userMessage, errType, errCode string, retryable bool) *CustomError {
    return &CustomError{
        BaseErr:     err,
        StatusCode:  statusCode,
        Message:     err.Error(),
        UserMessage: userMessage,
        ErrType:     errType,
        ErrCode:     errCode,
        Retryable:   retryable,
        Metadata:    make(map[string]interface{}),
    }
}

Sometimes, you’re working with functions that already return errors—like parsing a file:

func parseFile(filename string) error {
    // Simulating a parsing error
    return fmt.Errorf("failed to parse file: %s", filename)
}

You don’t want to lose that original error message. Here’s how you wrap it in a CustomError:

func HandleParseError(filename string) *CustomError {
    err := parseFile(filename)
    if err != nil {
        return NewFromError(
            err,
            400,
            "Unable to process the file you uploaded.",
            "BAD_REQUEST",
            "ERR_FILE_PARSE",
            false,
        )
    }
    return nil
}

This approach keeps the original error intact for debugging while providing a user-friendly message.

The Error Handler: Turning Errors into Responses

Here’s where the magic happens. The NewErrHandler function is like the ultimate “translator” between internal errors and client-facing API responses.

Key Steps in NewErrHandler:

The NewErrHandler function processes errors as follows:

Check for CustomError: Determines if the error can be mapped to a CustomError.
Create an API-Specific Error: Uses the APIErrorCreator interface to dynamically create an API-specific error response struct.
Transform the Error: Maps the CustomError fields into the API-specific error using the FromCustomError method of the APIError interface.
Handle Fallbacks: If the error transformation fails or the error isn’t a CustomError, the handler defaults to returning an internal server error.

Here’s the core logic:

func NewErrHandler(creator APIErrorCreator, writerFactory func() ResponseWriter) func(error) {
    return func(werr error) {
        var customErr *CustomError
        writer := writerFactory()

        // Check if the error is a CustomError
        if errors.As(werr, &customErr) {
            apiErr := creator.New()
            if err := apiErr.FromCustomError(customErr); err != nil {
                _ = writer.WriteResponse(http.StatusInternalServerError, NewParseErrorError(werr, err))
                return
            }

            // Write the transformed error response
            _ = writer.WriteResponse(customErr.StatusCode, apiErr)
            return
        }

        // Handle generic errors
        _ = writer.WriteResponse(http.StatusInternalServerError, NewInternalServerErrorError())
    }
}

Interfaces That Keep Things Flexible

The error handler leverages three critical interfaces: APIError, APIErrorCreator, and ResponseWriter.

APIError: Defines how your API formats errors.

// APIError defines the interface for custom API errors.
type APIError interface {
    FromCustomError(customErr *CustomError) error
}

Example:

type MyAPIError struct {
    StatusCode  int    `json:"status_code"`
    Message     string `json:"message"`
    UserMessage string `json:"user_message"`
    TraceID     string `json:"trace_id"`
}

func (e *MyAPIError) FromCustomError(customErr *CustomError) error {
    e.StatusCode = customErr.StatusCode
    e.Message = customErr.Message
    e.UserMessage = customErr.UserMessage
    e.TraceID = "generated-trace-id" // can be added from the metadata also
    return nil
}

APIErrorCreator: Responsible for creating new APIError instances. Ensures fresh instances of APIError are made for each error, supporting concurrent requests.

// APIErrorCreator defines the interface for creating custom API errors.
type APIErrorCreator interface {
    New() APIError
}

Example:

type MyAPIErrorCreator struct{}

func (c MyAPIErrorCreator) New() APIError {
    return &MyAPIError{}
}

ResponseWriter: Abstracts the response-writing mechanism, making the system framework-agnostic and decoupling the error-handling logic from specific frameworks like net/http, Echo, or Gin.

// ResponseWriter abstracts response writing for different frameworks.
type ResponseWriter interface {
    WriteResponse(statusCode int, body interface{}) error
}

Example:

type MyResponseWriter struct {
    writer http.ResponseWriter
}

func (rw MyResponseWriter) WriteResponse(statusCode int, body interface{}) error {
    rw.writer.WriteHeader(statusCode)
    return json.NewEncoder(rw.writer).Encode(body)
}

Complete Workflow

Here’s how all components work together in an example:

Define the Error Response: Implement APIError and APIErrorCreator to structure the response.
Handle Requests: Use the error handler to process errors dynamically.
Write Responses: Use the ResponseWriter to send consistent error responses to clients.

This design allows developers to define custom error response structures and formats. The same handler logic works across multiple modules with different APIErrorCreator implementations. We can easily extend the system to handle new types of errors or integrate with new frameworks.

Error Generation Using YAML And Templates

Defining errors manually for every scenario can be a pain. That’s where code generation comes in. Instead of writing each error from scratch, I use YAML configurations and Go templates to automate the process.

The YAML Configuration

Errors are defined in a YAML file that outlines their attributes, such as the HTTP status code, error message, and other arguments.

Here’s an example:

errors:
  - name: BadRequest
    description: The request is invalid or malformed.
    err_type: BAD_REQUEST
    err_code: ERR_BAD_REQUEST
    err_msg: Invalid request: %s
    display_msg: The request is invalid.
    status_code: 400
    retryable: false
    args:
      - name: details
        arg_type: string
  - name: NotFound
    description: The requested resource could not be found.
    err_type: NOT_FOUND
    err_code: ERR_NOT_FOUND
    err_msg: %s not found
    display_msg: The requested resource could not be found.
    status_code: 404
    retryable: false
    args:
      - name: resource
        arg_type: string

The Go Template

The YAML configuration is transformed into Go code using a template. The template generates error structs, constructors, and helper methods for each error.

Here’s the core structure of the template:

// Code generated by error-gen. DO NOT EDIT.

package generr

import (
    "apierr"
    "fmt"
)

{{range .Errors}}
// {{.Name}}Error represents {{.Description}}.
type {{.Name}}Error struct {
    apierr.CustomError
    {{- range .Args}}
    {{.Name}} {{.ArgType}} // {{.Name}} is a custom argument for the error
    {{- end}}
}

// New{{.Name}}Error creates a new {{.Name}}Error.
func New{{.Name}}Error(
    {{- range .Args}}{{.Name}} {{.ArgType}},{{- end}}
) *apierr.CustomError {
    return apierr.New(
        {{.StatusCode}},
        fmt.Sprintf(
            "{{.ErrMsg}}",
            {{range .Args}}{{.Name}},{{end}}
        ),
        "{{.DisplayMsg}}",
        "{{.ErrType}}",
        "{{.ErrCode}}",
        {{if .Retryable}}true{{else}}false{{end}},
    )
}
{{end}}

Generated Code Example

For the BadRequest and NotFound errors defined earlier, the following Go code would be generated:

// Code generated by error-gen. DO NOT EDIT.

package generr

import (
    "apierr"
    "fmt"
)

// BadRequestError represents The request is invalid or malformed.
type BadRequestError struct {
    apierr.CustomError
    Details string // Details is a custom argument for the error
}

// NewBadRequestError creates a new BadRequestError.
func NewBadRequestError(details string) *apierr.CustomError {
    return apierr.New(
        400,
        fmt.Sprintf("Invalid request: %s", details),
        "The request is invalid.",
        "BAD_REQUEST",
        "ERR_BAD_REQUEST",
        false,
    )
}

// NotFoundError represents The requested resource could not be found.
type NotFoundError struct {
    apierr.CustomError
    Resource string // Resource is a custom argument for the error
}

// NewNotFoundError creates a new NotFoundError.
func NewNotFoundError(resource string) *apierr.CustomError {
    return apierr.New(
        404,
        fmt.Sprintf("%s not found", resource),
        "The requested resource could not be found.",
        "NOT_FOUND",
        "ERR_NOT_FOUND",
        false,
    )
}

Advantages of Code Generation

Consistency: All error definitions follow the same structure and conventions.
Efficiency: Eliminates repetitive coding for error structs and constructors.
Scalability: Adding new errors is as simple as updating the YAML file and regenerating the code.
Customizability: Supports application-specific fields and error attributes through the args property.

Predefined Errors

Predefined errors are generated dynamically from the YAML configuration using the error generation mechanism. These errors provide a consistent and reusable way to represent common API issues, such as bad requests, unauthorized access, or resource not found.

Putting It All Together

The strength of this error-handling system lies in how its components—CustomError, error handlers, interfaces, and predefined errors—work together seamlessly to provide a robust and extensible solution.

Here’s an example of the workflow:


// Creator implements apierr.APIErrorCreator for custom API errors.
type Creator struct{}

func (c *Creator) New() apierr.APIError {
    return NewAPIError()
}

func NewCreator() apierr.APIErrorCreator {
    return &Creator{}
}

// APIError represents a custom API error structure.
type APIError struct {
    Code    int    `json:"code"`
    Message string `json:"message"`
}

// Parse maps a CustomError to the APIError structure.
func (e *APIError) FromCustomError(err *apierr.CustomError) error {
    e.Code = err.StatusCode
    e.Message = err.UserMessage
    return nil
}

func NewAPIError() apierr.APIError {
    return &APIError{}
}

func main() {
    // Generate error definitions from YAML (if required)
    err := apierr.Generate("example/errors.yml", "example/errors.gen.go")
    if err != nil {
        log.Fatal(err)
        return
    }

    // Initialize Echo
    e := echo.New()

    // Middleware
    e.Use(middleware.Logger())
    e.Use(middleware.Recover())

    // Create a generic error handler
    creator := NewCreator()
    e.HTTPErrorHandler = func(err error, c echo.Context) {
        handler := apierr.NewErrHandler(creator, func() apierr.ResponseWriter {
            return &EchoResponseWriter{ctx: c}
        })
        handler(err)
    }

    // Routes to test various error scenarios
    e.GET("/simple", func(c echo.Context) error {
        return errors.New("simple error")
    })

    e.GET("/error", func(c echo.Context) error {
        return apierr.NewFromError(
            errors.New("complex error"),
            http.StatusUnauthorized,
            "Unauthorized",
            "AUTH_ERROR",
            "AUTH_401",
            false,
        )
    })

    e.GET("/new", func(c echo.Context) error {
        return simulateError()
    })

    e.GET("/wrap", func(c echo.Context) error {
        return fmt.Errorf("wrap: %w", simulateError())
    })

    e.GET("/double", func(c echo.Context) error {
        return fmt.Errorf(
            "outer error: %w",
            fmt.Errorf("inner error: %w", simulateError()))
    })

    e.Logger.Fatal(e.Start(":8080"))
}

// EchoResponseWriter adapts Echo's context to the apierr.ResponseWriter interface.
type EchoResponseWriter struct {
    ctx echo.Context
}

func (w *EchoResponseWriter) WriteResponse(statusCode int, body interface{}) error {
    return w.ctx.JSON(statusCode, body)
}

// Simulate an error for testing purposes.
func simulateError() error {
    return apierr.New(
        http.StatusNotFound,
        "Resource not found",
        "The requested resource does not exist",
        "NOT_FOUND",
        "NF_404",
        false,
    )
}

Final Thoughts

Building this system transformed how I handle errors in my Go APIs. No more vague logs or generic “server error” responses—just clear, actionable, and developer-friendly feedback.

It might seem like a lot of work upfront, but once you’ve set it up, you’ll wonder how you ever lived without it. And the next time an error pops up, you’ll have everything you need to diagnose and fix it—without breaking a sweat.

Ready to give it a try? You’ve got this!

You can check out the full project HERE