Building a Production-Ready File Download System with GoFrame

Introduction 👋

Hey devs! If you're building web applications with Go, you've probably encountered the need to handle file downloads. While it might seem straightforward at first, implementing production-ready file downloads comes with its own set of challenges. In this guide, I'll share practical insights on building robust file download functionality using GoFrame.

TL;DR: We'll cover everything from basic file downloads to production-ready implementations with GoFrame, including handling large files, implementing resume support, and optimizing performance.

What We'll Cover 📝

• Setting up basic file downloads
• Handling large files without memory issues
• Implementing resume/partial downloads
• Adding security measures
• Optimizing for production
• Real-world examples and gotchas

Why GoFrame for File Downloads? 🤔

Before we dive in, you might be wondering why choose GoFrame for handling file downloads. Here's the deal: while you could implement file downloads with standard Go libraries, GoFrame provides some nice abstractions that make our lives easier:

• Stream-based processing (goodbye memory issues!)
• Built-in middleware support
• Clean error handling
• Progress tracking capabilities

Getting Started 🌱

First things first, let's set up our environment. Make sure you have Go installed (1.16+), then grab GoFrame:

go get -u github.com/gogf/gf/v2@latest

Basic Implementation

Let's start with a simple file download handler. This is your entry point to understanding how GoFrame handles downloads:

package handler

import (
    "github.com/gogf/gf/v2/net/ghttp"
)

func SimpleDownload(r *ghttp.Request) {
    filePath := r.Get("file").String()

    // Quick security check
    if !isValidFile(filePath) {
        r.Response.WriteStatus(403)
        return
    }

    // Set headers and serve
    r.Response.Header().Set("Content-Type", "application/octet-stream")
    r.Response.ServeFileDownload(filePath)
}

Pretty straightforward, right? But wait - there's more to consider when building for production! 🛠️

Security First! 🔒

Before we get too excited about serving files, let's talk security. Here's a robust validation function to prevent common security issues:

var allowedExtensions = map[string]bool{
    ".txt":  true,
    ".pdf":  true,
    ".doc":  true,
    ".xlsx": true,
}

func isValidFile(filePath string) bool {
    // 🚫 Path traversal check
    if strings.Contains(filePath, "..") {
        return false
    }

    // 📁 File existence & type check
    fileInfo, err := os.Stat(filePath)
    if err != nil || !fileInfo.Mode().IsRegular() {
        return false
    }

    // 📏 Size check (100MB limit)
    if fileInfo.Size() > 100*1024*1024 {
        return false
    }

    // 🔍 Extension check
    ext := strings.ToLower(filepath.Ext(filePath))
    return allowedExtensions[ext]
}

Handling Large Files Like a Pro 🏋️‍♂️

Now, here's where things get interesting. When dealing with large files, you don't want to load everything into memory. Here's a streaming implementation that won't make your server cry:

func StreamDownload(r *ghttp.Request) {
    const bufSize = 32 * 1024 // 32KB chunks

    file := r.Get("file").String()
    fd, err := os.Open(file)
    if err != nil {
        r.Response.WriteStatus(500)
        return
    }
    defer fd.Close() // Don't forget this! 😉

    info, _ := fd.Stat()
    r.Response.Header().Set("Content-Length", gconv.String(info.Size()))

    // Stream in chunks
    buf := make([]byte, bufSize)
    for {
        n, err := fd.Read(buf)
        if n > 0 {
            r.Response.Write(buf[:n])
        }
        if err == io.EOF {
            break
        }
    }
}

Resume Support: Because Downloads Sometimes Fail 😅

Let's be real - downloads can fail, especially for large files. Here's how to implement resume support:

func ResumeDownload(r *ghttp.Request) {
    rangeHeader := r.Header.Get("Range")
    if rangeHeader != "" {
        // Handle range request
        // ... (previous range parsing code)

        r.Response.Header().Set("Content-Range",
            fmt.Sprintf("bytes %d-%d/%d", start, end, fileSize))
        r.Response.WriteStatus(206) // Partial Content

        // Serve the requested chunk
        streamFileContent(r, fd, end-start+1)
    }
}

Real-World Scenarios and Solutions 🎯

Let's look at some common scenarios you might encounter and how to handle them:

1. Handling Different File Types 📁

Different file types often need different handling. Here's a practical example:

func SmartDownload(r *ghttp.Request) {
    filePath := r.Get("file").String()
    fileExt := strings.ToLower(filepath.Ext(filePath))

    switch fileExt {
    case ".pdf":
        // For PDFs, we might want to allow preview
        r.Response.Header().Set("Content-Type", "application/pdf")
        r.Response.Header().Set("Content-Disposition", "inline")
    case ".csv":
        // For CSVs, we might want to add BOM for Excel compatibility
        r.Response.Header().Set("Content-Type", "text/csv")
        r.Response.Write("\xEF\xBB\xBF") // Add BOM
    case ".mp4":
        // For videos, support range requests for streaming
        handleVideoStream(r)
    default:
        // Default download behavior
        r.Response.Header().Set("Content-Type", "application/octet-stream")
        r.Response.Header().Set("Content-Disposition", "attachment")
    }

    r.Response.ServeFileDownload(filePath)
}

2. Browser Compatibility Handling 🌐

Different browsers handle downloads differently. Here's how to handle it:

func BrowserAwareDownload(r *ghttp.Request) {
    filename := r.Get("file").String()
    userAgent := r.Header.Get("User-Agent")

    // Handle filename encoding for different browsers
    if strings.Contains(userAgent, "MSIE") || 
       strings.Contains(userAgent, "Edge") {
        // URL encode for IE/Edge
        filename = url.QueryEscape(filename)
    } else {
        // RFC 5987 encoding for others
        filename = fmt.Sprintf("UTF-8''%s", 
            url.QueryEscape(filename))
    }

    disposition := fmt.Sprintf("attachment; filename*=%s", filename)
    r.Response.Header().Set("Content-Disposition", disposition)
}

3. Download Progress Monitoring 📊

Here's how to implement progress monitoring with WebSocket updates:

type Progress struct {
    Total     int64   `json:"total"`
    Current   int64   `json:"current"`
    Speed     float64 `json:"speed"`
    Remaining int     `json:"remaining"`
}

func ProgressDownload(r *ghttp.Request) {
    ws, err := r.WebSocket()
    if err != nil {
        return
    }

    file := r.Get("file").String()
    info, _ := os.Stat(file)
    total := info.Size()

    // Create a custom reader that reports progress
    reader := &ProgressReader{
        Reader: bufio.NewReader(file),
        Total:  total,
        OnProgress: func(current int64) {
            progress := Progress{
                Total:     total,
                Current:   current,
                Speed:    calculateSpeed(current),
                Remaining: calculateRemaining(current, total),
            }
            ws.WriteJSON(progress)
        },
    }

    io.Copy(r.Response.Writer, reader)
}

Production Tips and Best Practices 💡

After implementing file downloads in numerous production environments, here are some battle-tested tips:

Always Rate Limit 🚦

var downloadLimiter = rate.NewLimiter(rate.Limit(100), 200)

Implement Caching 🗄️

func CachedDownload(r *ghttp.Request) {
    if data := memCache.Get(file); data != nil {
        return serveContent(r, data)
    }
    // ... fallback to disk
}

Monitor Everything 📊

Here's a practical monitoring implementation:

type DownloadMetrics struct {
    ActiveDownloads    *atomic.Int64
    TotalBytes        *atomic.Int64
    ErrorCount        *atomic.Int64
    DownloadDurations *metrics.Histogram
}

func NewDownloadMetrics() *DownloadMetrics {
    return &DownloadMetrics{
        ActiveDownloads:    atomic.NewInt64(0),
        TotalBytes:        atomic.NewInt64(0),
        ErrorCount:        atomic.NewInt64(0),
        DownloadDurations: metrics.NewHistogram(metrics.HistogramOpts{
            Buckets: []float64{.1, .5, 1, 2.5, 5, 10, 30},
        }),
    }
}

func (m *DownloadMetrics) Track(r *ghttp.Request) func() {
    start := time.Now()
    m.ActiveDownloads.Add(1)

    return func() {
        m.ActiveDownloads.Add(-1)
        duration := time.Since(start).Seconds()
        m.DownloadDurations.Observe(duration)
    }
}

Implement Circuit Breakers 🔌

Protect your system with circuit breakers:

type DownloadBreaker struct {
    breaker *gobreaker.CircuitBreaker
}

func NewDownloadBreaker() *DownloadBreaker {
    return &DownloadBreaker{
        breaker: gobreaker.NewCircuitBreaker(gobreaker.Settings{
            Name:        "download-breaker",
            MaxRequests: 100,
            Interval:    10 * time.Second,
            Timeout:     30 * time.Second,
            ReadyToTrip: func(counts gobreaker.Counts) bool {
                failureRatio := float64(counts.TotalFailures) / float64(counts.Requests)
                return counts.Requests >= 10 && failureRatio >= 0.6
            },
        }),
    }
}

Implement Graceful Shutdown 🛑

Handle shutdowns properly:

type DownloadManager struct {
    activeDownloads sync.WaitGroup
    shutdownCh     chan struct{}
}

func (dm *DownloadManager) HandleDownload(r *ghttp.Request) {
    dm.activeDownloads.Add(1)
    defer dm.activeDownloads.Done()

    // Create download context with shutdown signal
    ctx, cancel := context.WithCancel(r.Context())
    defer cancel()

    go func() {
        select {
        case <-dm.shutdownCh:
            // Save progress and cleanup
            cancel()
        case <-ctx.Done():
            return
        }
    }()

    // Proceed with download...
}

func (dm *DownloadManager) Shutdown(timeout time.Duration) error {
    close(dm.shutdownCh)

    // Wait for active downloads with timeout
    c := make(chan struct{})
    go func() {
        dm.activeDownloads.Wait()
        close(c)
    }()

    select {
    case <-c:
        return nil
    case <-time.After(timeout):
        return errors.New("shutdown timeout")
    }
}

Advanced Features and Examples 🚀

1. Zip Download on the Fly 📦

Need to create ZIP archives dynamically? Here's how:

func ZipDownload(r *ghttp.Request) {
    files := r.GetArray("files")

    r.Response.Header().Set("Content-Type", "application/zip")
    r.Response.Header().Set("Content-Disposition", 
        "attachment; filename=archive.zip")

    zw := zip.NewWriter(r.Response.Writer)
    defer zw.Close()

    for _, file := range files {
        // Add file to zip
        f, err := os.Open(file)
        if err != nil {
            continue
        }
        defer f.Close()

        // Create zip entry
        header := &zip.FileHeader{
            Name:   filepath.Base(file),
            Method: zip.Deflate,
        }

        writer, err := zw.CreateHeader(header)
        if err != nil {
            continue
        }

        io.Copy(writer, f)
    }
}

2. Cloud Storage Integration 🌥️

Here's an example integrating with S3-compatible storage:

func CloudDownload(r *ghttp.Request) {
    bucket := "my-bucket"
    key := r.Get("key").String()

    // Get object from S3
    input := &s3.GetObjectInput{
        Bucket: aws.String(bucket),
        Key:    aws.String(key),
    }

    result, err := s3Client.GetObject(input)
    if err != nil {
        r.Response.WriteStatus(500)
        return
    }
    defer result.Body.Close()

    // Set headers
    r.Response.Header().Set("Content-Type", *result.ContentType)
    r.Response.Header().Set("Content-Length", 
        fmt.Sprintf("%d", *result.ContentLength))

    // Stream the response
    io.Copy(r.Response.Writer, result.Body)
}

3. Download Queue Implementation 📋

For managing large numbers of downloads:

type DownloadQueue struct {
    queue    chan DownloadJob
    workers  int
    metrics  *DownloadMetrics
}

type DownloadJob struct {
    ID       string
    FilePath string
    Priority int
    Callback func(error)
}

func NewDownloadQueue(workers int) *DownloadQueue {
    dq := &DownloadQueue{
        queue:   make(chan DownloadJob, 1000),
        workers: workers,
        metrics: NewDownloadMetrics(),
    }

    for i := 0; i < workers; i++ {
        go dq.worker()
    }

    return dq
}

func (dq *DownloadQueue) worker() {
    for job := range dq.queue {
        // Process download job
        err := processDownload(job)
        if job.Callback != nil {
            job.Callback(err)
        }
    }
}

func (dq *DownloadQueue) Enqueue(job DownloadJob) error {
    select {
    case dq.queue <- job:
        return nil
    default:
        return errors.New("queue full")
    }
}

Comprehensive Error Handling Patterns 🎯

Let's look at how to handle various error scenarios robustly:

// Custom error types for better error handling
type DownloadError struct {
    Code    int
    Message string
    Err     error
}

func (e *DownloadError) Error() string {
    if e.Err != nil {
        return fmt.Sprintf("%s: %v", e.Message, e.Err)
    }
    return e.Message
}

// Error codes
const (
    ErrFileNotFound = iota + 1000
    ErrPermissionDenied
    ErrQuotaExceeded
    ErrInvalidFileType
    ErrFileTooLarge
)

// Robust download handler with error handling
func RobustDownload(r *ghttp.Request) {
    defer func() {
        if err := recover(); err != nil {
            // Log the stack trace
            debug.PrintStack()
            // Return 500 error to client
            r.Response.WriteStatus(500)
        }
    }()

    file := r.Get("file").String()

    // Validate request
    if err := validateDownloadRequest(r); err != nil {
        handleDownloadError(r, err)
        return
    }

    // Check user quota
    if err := checkUserQuota(r); err != nil {
        handleDownloadError(r, &DownloadError{
            Code:    ErrQuotaExceeded,
            Message: "Download quota exceeded",
            Err:     err,
        })
        return
    }

    // Attempt file download
    if err := streamFileWithRetry(r, file); err != nil {
        handleDownloadError(r, err)
        return
    }
}

// Error handler for different scenarios
func handleDownloadError(r *ghttp.Request, err error) {
    var downloadErr *DownloadError
    if errors.As(err, &downloadErr) {
        switch downloadErr.Code {
        case ErrFileNotFound:
            r.Response.WriteStatus(404)
            r.Response.WriteJson(g.Map{
                "error": "File not found",
                "details": downloadErr.Message,
            })
        case ErrPermissionDenied:
            r.Response.WriteStatus(403)
            r.Response.WriteJson(g.Map{
                "error": "Access denied",
                "details": downloadErr.Message,
            })
        case ErrQuotaExceeded:
            r.Response.WriteStatus(429)
            r.Response.WriteJson(g.Map{
                "error": "Quota exceeded",
                "details": downloadErr.Message,
            })
        default:
            r.Response.WriteStatus(500)
            r.Response.WriteJson(g.Map{
                "error": "Internal server error",
                "reference": uuid.New().String(),
            })
        }
        return
    }

    // Handle generic errors
    r.Response.WriteStatus(500)
}

// Retry mechanism for transient failures
func streamFileWithRetry(r *ghttp.Request, file string) error {
    const maxRetries = 3
    const baseDelay = 100 * time.Millisecond

    var lastErr error
    for attempt := 0; attempt < maxRetries; attempt++ {
        err := streamFile(r, file)
        if err == nil {
            return nil
        }

        lastErr = err

        // Don't retry on certain errors
        if errors.Is(err, os.ErrNotExist) || 
           errors.Is(err, os.ErrPermission) {
            return err
        }

        // Exponential backoff
        delay := baseDelay * time.Duration(math.Pow(2, float64(attempt)))
        time.Sleep(delay)
    }

    return fmt.Errorf("failed after %d attempts: %w", maxRetries, lastErr)
}

Specific Use Case Examples 🔍

1. Video Streaming with HLS Support 🎥

func VideoStreamHandler(r *ghttp.Request) {
    videoPath := r.Get("video").String()

    // Check if requesting manifest
    if strings.HasSuffix(videoPath, ".m3u8") {
        serveHLSManifest(r, videoPath)
        return
    }

    // Check if requesting segment
    if strings.HasSuffix(videoPath, ".ts") {
        serveHLSSegment(r, videoPath)
        return
    }

    // Serve video file directly with range support
    serveVideoWithRange(r, videoPath)
}

func serveHLSManifest(r *ghttp.Request, path string) {
    r.Response.Header().Set("Content-Type", "application/vnd.apple.mpegurl")
    r.Response.Header().Set("Cache-Control", "max-age=5")

    // Read and serve manifest
    content, err := ioutil.ReadFile(path)
    if err != nil {
        r.Response.WriteStatus(404)
        return
    }

    r.Response.Write(content)
}

func serveVideoWithRange(r *ghttp.Request, path string) {
    info, err := os.Stat(path)
    if err != nil {
        r.Response.WriteStatus(404)
        return
    }

    file, err := os.Open(path)
    if err != nil {
        r.Response.WriteStatus(500)
        return
    }
    defer file.Close()

    rangeHeader := r.Header.Get("Range")
    if rangeHeader != "" {
        ranges, err := parseRange(rangeHeader, info.Size())
        if err != nil {
            r.Response.WriteStatus(416)
            return
        }

        if len(ranges) > 0 {
            start, end := ranges[0][0], ranges[0][1]
            r.Response.Header().Set("Content-Range",
                fmt.Sprintf("bytes %d-%d/%d", start, end, info.Size()))
            r.Response.Header().Set("Content-Length",
                fmt.Sprintf("%d", end-start+1))
            r.Response.WriteStatus(206)

            file.Seek(start, 0)
            io.CopyN(r.Response.Writer, file, end-start+1)
            return
        }
    }

    r.Response.Header().Set("Content-Length", fmt.Sprintf("%d", info.Size()))
    r.Response.Header().Set("Content-Type", "video/mp4")
    io.Copy(r.Response.Writer, file)
}

2. Large Excel File Generation 📊

func ExcelDownloadHandler(r *ghttp.Request) {
    // Create a new file
    f := excelize.NewFile()
    defer f.Close()

    // Create buffered writer
    buf := new(bytes.Buffer)
    writer := bufio.NewWriter(buf)

    // Start progress tracking
    progress := 0
    total := 1000000 // Example: 1 million rows

    // Stream data writing
    for i := 0; i < total; i++ {
        // Write row data
        row := []interface{}{
            fmt.Sprintf("Data %d", i),
            time.Now(),
            rand.Float64(),
        }
        cell, _ := excelize.CoordinatesToCellName(1, i+1)
        f.SetSheetRow("Sheet1", cell, &row)

        // Update progress every 1%
        currentProgress := (i * 100) / total
        if currentProgress > progress {
            progress = currentProgress
            // Send progress through WebSocket if needed
            sendProgress(r, progress)
        }
    }

    // Set headers for Excel file
    r.Response.Header().Set("Content-Type", 
        "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet")
    r.Response.Header().Set("Content-Disposition", 
        "attachment; filename=large-report.xlsx")

    // Save to response writer
    if err := f.Write(r.Response.Writer); err != nil {
        r.Response.WriteStatus(500)
        return
    }
}

func sendProgress(r *ghttp.Request, progress int) {
    // Implementation depends on your WebSocket setup
    // Example using gorilla/websocket
    if ws, ok := r.GetCtxVar("ws").(*websocket.Conn); ok {
        ws.WriteJSON(map[string]interface{}{
            "progress": progress,
        })
    }
}

3. PDF Generation and Download 📄

func PDFDownloadHandler(r *ghttp.Request) {
    // Create PDF document
    pdf := gofpdf.New("P", "mm", "A4", "")
    pdf.AddPage()

    // Add content
    pdf.SetFont("Arial", "B", 16)
    pdf.Cell(40, 10, "Generated Report")

    // Add table
    pdf.SetFont("Arial", "", 12)
    data := [][]string{
        {"Column 1", "Column 2", "Column 3"},
        {"Data 1", "Data 2", "Data 3"},
        // ... more rows
    }

    for i, row := range data {
        for j, col := range row {
            pdf.Cell(40, 10, col)
            if j == len(row)-1 {
                pdf.Ln(-1)
            }
        }
        if i == 0 {
            pdf.Ln(-1)
        }
    }

    // Set headers
    r.Response.Header().Set("Content-Type", "application/pdf")
    r.Response.Header().Set("Content-Disposition", 
        "attachment; filename=report.pdf")

    // Write to response
    if err := pdf.Output(r.Response.Writer); err != nil {
        r.Response.WriteStatus(500)
        return
    }
}

Common Gotchas to Avoid 🚨

1. Memory Leaks

   • Always use defer for cleanup
   • Don't read entire files into memory

2. Security Issues

   • Validate file paths
   • Check file permissions
   • Limit file types

3. Performance Problems

   • Use buffered reading
   • Implement proper caching
   • Don't forget to close files

Real-World Example: A Complete Download Service 🌟

Here's a production-ready download service combining all the concepts we've discussed:

type DownloadService struct {
    limiter  *rate.Limiter
    cache    *Cache
    metrics  *Metrics
}

func (s *DownloadService) ServeDownload(r *ghttp.Request) {
    // 1. Rate limiting
    if err := s.limiter.Wait(r.Context()); err != nil {
        r.Response.WriteStatus(429)
        return
    }

    // 2. Try cache
    if data := s.cache.Get(r.Get("file").String()); data != nil {
        s.metrics.RecordHit()
        return s.serveContent(r, data)
    }

    // 3. Stream file with resume support
    s.streamWithResume(r)
}

Testing Your Implementation 🧪

Don't forget to test! Here's a quick test case to get you started:

func TestDownload(t *testing.T) {
    s := g.Server()
    s.BindHandler("/download", DownloadHandler)

    client := g.Client()
    r, err := client.Get("/download?file=test.txt")

    assert.Nil(t, err)
    assert.Equal(t, 200, r.StatusCode)
}

Wrapping Up 🎁

Building production-ready file downloads with GoFrame isn't just about calling ServeFileDownload. It's about:

• Handling security properly
• Managing resources efficiently
• Supporting resume capabilities
• Monitoring and maintaining performance

What's Next?

• Add progress tracking for downloads
• Implement cloud storage integration
• Add support for file preprocessing
• Explore streaming compression

Resources 📚

• GoFrame Documentation
• HTTP Range Requests Spec
• Go Rate Limiting Guide

Let me know in the comments if you have any questions or want to share your own experiences with file downloads in Go! 💬

---

Like this article? Follow me for more Go development tips and tutorials! 🚀