Introduction 👋
Hey devs! If you're building web applications with Go, you've probably encountered the need to handle file downloads. While it might seem straightforward at first, implementing production-ready file downloads comes with its own set of challenges. In this guide, I'll share practical insights on building robust file download functionality using GoFrame.
TL;DR: We'll cover everything from basic file downloads to production-ready implementations with GoFrame, including handling large files, implementing resume support, and optimizing performance.
What We'll Cover 📝
- Setting up basic file downloads
- Handling large files without memory issues
- Implementing resume/partial downloads
- Adding security measures
- Optimizing for production
- Real-world examples and gotchas
Why GoFrame for File Downloads? 🤔
Before we dive in, you might be wondering why choose GoFrame for handling file downloads. Here's the deal: while you could implement file downloads with standard Go libraries, GoFrame provides some nice abstractions that make our lives easier:
- Stream-based processing (goodbye memory issues!)
- Built-in middleware support
- Clean error handling
- Progress tracking capabilities
Getting Started 🌱
First things first, let's set up our environment. Make sure you have Go installed (1.16+), then grab GoFrame:
go get -u github.com/gogf/gf/v2@latest
Basic Implementation
Let's start with a simple file download handler. This is your entry point to understanding how GoFrame handles downloads:
package handler
import (
"github.com/gogf/gf/v2/net/ghttp"
)
func SimpleDownload(r *ghttp.Request) {
filePath := r.Get("file").String()
// Quick security check
if !isValidFile(filePath) {
r.Response.WriteStatus(403)
return
}
// Set headers and serve
r.Response.Header().Set("Content-Type", "application/octet-stream")
r.Response.ServeFileDownload(filePath)
}
Pretty straightforward, right? But wait - there's more to consider when building for production! 🛠️
Security First! 🔒
Before we get too excited about serving files, let's talk security. Here's a robust validation function to prevent common security issues:
var allowedExtensions = map[string]bool{
".txt": true,
".pdf": true,
".doc": true,
".xlsx": true,
}
func isValidFile(filePath string) bool {
// 🚫 Path traversal check
if strings.Contains(filePath, "..") {
return false
}
// 📁 File existence & type check
fileInfo, err := os.Stat(filePath)
if err != nil || !fileInfo.Mode().IsRegular() {
return false
}
// 📏 Size check (100MB limit)
if fileInfo.Size() > 100*1024*1024 {
return false
}
// 🔍 Extension check
ext := strings.ToLower(filepath.Ext(filePath))
return allowedExtensions[ext]
}
Handling Large Files Like a Pro 🏋️♂️
Now, here's where things get interesting. When dealing with large files, you don't want to load everything into memory. Here's a streaming implementation that won't make your server cry:
func StreamDownload(r *ghttp.Request) {
const bufSize = 32 * 1024 // 32KB chunks
file := r.Get("file").String()
fd, err := os.Open(file)
if err != nil {
r.Response.WriteStatus(500)
return
}
defer fd.Close() // Don't forget this! 😉
info, _ := fd.Stat()
r.Response.Header().Set("Content-Length", gconv.String(info.Size()))
// Stream in chunks
buf := make([]byte, bufSize)
for {
n, err := fd.Read(buf)
if n > 0 {
r.Response.Write(buf[:n])
}
if err == io.EOF {
break
}
}
}
Resume Support: Because Downloads Sometimes Fail 😅
Let's be real - downloads can fail, especially for large files. Here's how to implement resume support:
func ResumeDownload(r *ghttp.Request) {
rangeHeader := r.Header.Get("Range")
if rangeHeader != "" {
// Handle range request
// ... (previous range parsing code)
r.Response.Header().Set("Content-Range",
fmt.Sprintf("bytes %d-%d/%d", start, end, fileSize))
r.Response.WriteStatus(206) // Partial Content
// Serve the requested chunk
streamFileContent(r, fd, end-start+1)
}
}
Real-World Scenarios and Solutions 🎯
Let's look at some common scenarios you might encounter and how to handle them:
1. Handling Different File Types 📁
Different file types often need different handling. Here's a practical example:
func SmartDownload(r *ghttp.Request) {
filePath := r.Get("file").String()
fileExt := strings.ToLower(filepath.Ext(filePath))
switch fileExt {
case ".pdf":
// For PDFs, we might want to allow preview
r.Response.Header().Set("Content-Type", "application/pdf")
r.Response.Header().Set("Content-Disposition", "inline")
case ".csv":
// For CSVs, we might want to add BOM for Excel compatibility
r.Response.Header().Set("Content-Type", "text/csv")
r.Response.Write("\xEF\xBB\xBF") // Add BOM
case ".mp4":
// For videos, support range requests for streaming
handleVideoStream(r)
default:
// Default download behavior
r.Response.Header().Set("Content-Type", "application/octet-stream")
r.Response.Header().Set("Content-Disposition", "attachment")
}
r.Response.ServeFileDownload(filePath)
}
2. Browser Compatibility Handling 🌐
Different browsers handle downloads differently. Here's how to handle it:
func BrowserAwareDownload(r *ghttp.Request) {
filename := r.Get("file").String()
userAgent := r.Header.Get("User-Agent")
// Handle filename encoding for different browsers
if strings.Contains(userAgent, "MSIE") ||
strings.Contains(userAgent, "Edge") {
// URL encode for IE/Edge
filename = url.QueryEscape(filename)
} else {
// RFC 5987 encoding for others
filename = fmt.Sprintf("UTF-8''%s",
url.QueryEscape(filename))
}
disposition := fmt.Sprintf("attachment; filename*=%s", filename)
r.Response.Header().Set("Content-Disposition", disposition)
}
3. Download Progress Monitoring 📊
Here's how to implement progress monitoring with WebSocket updates:
type Progress struct {
Total int64 `json:"total"`
Current int64 `json:"current"`
Speed float64 `json:"speed"`
Remaining int `json:"remaining"`
}
func ProgressDownload(r *ghttp.Request) {
ws, err := r.WebSocket()
if err != nil {
return
}
file := r.Get("file").String()
info, _ := os.Stat(file)
total := info.Size()
// Create a custom reader that reports progress
reader := &ProgressReader{
Reader: bufio.NewReader(file),
Total: total,
OnProgress: func(current int64) {
progress := Progress{
Total: total,
Current: current,
Speed: calculateSpeed(current),
Remaining: calculateRemaining(current, total),
}
ws.WriteJSON(progress)
},
}
io.Copy(r.Response.Writer, reader)
}
Production Tips and Best Practices 💡
After implementing file downloads in numerous production environments, here are some battle-tested tips:
Always Rate Limit 🚦
var downloadLimiter = rate.NewLimiter(rate.Limit(100), 200)
Implement Caching 🗄️
func CachedDownload(r *ghttp.Request) {
if data := memCache.Get(file); data != nil {
return serveContent(r, data)
}
// ... fallback to disk
}
Monitor Everything 📊
Here's a practical monitoring implementation:
type DownloadMetrics struct {
ActiveDownloads *atomic.Int64
TotalBytes *atomic.Int64
ErrorCount *atomic.Int64
DownloadDurations *metrics.Histogram
}
func NewDownloadMetrics() *DownloadMetrics {
return &DownloadMetrics{
ActiveDownloads: atomic.NewInt64(0),
TotalBytes: atomic.NewInt64(0),
ErrorCount: atomic.NewInt64(0),
DownloadDurations: metrics.NewHistogram(metrics.HistogramOpts{
Buckets: []float64{.1, .5, 1, 2.5, 5, 10, 30},
}),
}
}
func (m *DownloadMetrics) Track(r *ghttp.Request) func() {
start := time.Now()
m.ActiveDownloads.Add(1)
return func() {
m.ActiveDownloads.Add(-1)
duration := time.Since(start).Seconds()
m.DownloadDurations.Observe(duration)
}
}
Implement Circuit Breakers 🔌
Protect your system with circuit breakers:
type DownloadBreaker struct {
breaker *gobreaker.CircuitBreaker
}
func NewDownloadBreaker() *DownloadBreaker {
return &DownloadBreaker{
breaker: gobreaker.NewCircuitBreaker(gobreaker.Settings{
Name: "download-breaker",
MaxRequests: 100,
Interval: 10 * time.Second,
Timeout: 30 * time.Second,
ReadyToTrip: func(counts gobreaker.Counts) bool {
failureRatio := float64(counts.TotalFailures) / float64(counts.Requests)
return counts.Requests >= 10 && failureRatio >= 0.6
},
}),
}
}
Implement Graceful Shutdown 🛑
Handle shutdowns properly:
type DownloadManager struct {
activeDownloads sync.WaitGroup
shutdownCh chan struct{}
}
func (dm *DownloadManager) HandleDownload(r *ghttp.Request) {
dm.activeDownloads.Add(1)
defer dm.activeDownloads.Done()
// Create download context with shutdown signal
ctx, cancel := context.WithCancel(r.Context())
defer cancel()
go func() {
select {
case <-dm.shutdownCh:
// Save progress and cleanup
cancel()
case <-ctx.Done():
return
}
}()
// Proceed with download...
}
func (dm *DownloadManager) Shutdown(timeout time.Duration) error {
close(dm.shutdownCh)
// Wait for active downloads with timeout
c := make(chan struct{})
go func() {
dm.activeDownloads.Wait()
close(c)
}()
select {
case <-c:
return nil
case <-time.After(timeout):
return errors.New("shutdown timeout")
}
}
Advanced Features and Examples 🚀
1. Zip Download on the Fly 📦
Need to create ZIP archives dynamically? Here's how:
func ZipDownload(r *ghttp.Request) {
files := r.GetArray("files")
r.Response.Header().Set("Content-Type", "application/zip")
r.Response.Header().Set("Content-Disposition",
"attachment; filename=archive.zip")
zw := zip.NewWriter(r.Response.Writer)
defer zw.Close()
for _, file := range files {
// Add file to zip
f, err := os.Open(file)
if err != nil {
continue
}
defer f.Close()
// Create zip entry
header := &zip.FileHeader{
Name: filepath.Base(file),
Method: zip.Deflate,
}
writer, err := zw.CreateHeader(header)
if err != nil {
continue
}
io.Copy(writer, f)
}
}
2. Cloud Storage Integration 🌥️
Here's an example integrating with S3-compatible storage:
func CloudDownload(r *ghttp.Request) {
bucket := "my-bucket"
key := r.Get("key").String()
// Get object from S3
input := &s3.GetObjectInput{
Bucket: aws.String(bucket),
Key: aws.String(key),
}
result, err := s3Client.GetObject(input)
if err != nil {
r.Response.WriteStatus(500)
return
}
defer result.Body.Close()
// Set headers
r.Response.Header().Set("Content-Type", *result.ContentType)
r.Response.Header().Set("Content-Length",
fmt.Sprintf("%d", *result.ContentLength))
// Stream the response
io.Copy(r.Response.Writer, result.Body)
}
3. Download Queue Implementation 📋
For managing large numbers of downloads:
type DownloadQueue struct {
queue chan DownloadJob
workers int
metrics *DownloadMetrics
}
type DownloadJob struct {
ID string
FilePath string
Priority int
Callback func(error)
}
func NewDownloadQueue(workers int) *DownloadQueue {
dq := &DownloadQueue{
queue: make(chan DownloadJob, 1000),
workers: workers,
metrics: NewDownloadMetrics(),
}
for i := 0; i < workers; i++ {
go dq.worker()
}
return dq
}
func (dq *DownloadQueue) worker() {
for job := range dq.queue {
// Process download job
err := processDownload(job)
if job.Callback != nil {
job.Callback(err)
}
}
}
func (dq *DownloadQueue) Enqueue(job DownloadJob) error {
select {
case dq.queue <- job:
return nil
default:
return errors.New("queue full")
}
}
Comprehensive Error Handling Patterns 🎯
Let's look at how to handle various error scenarios robustly:
// Custom error types for better error handling
type DownloadError struct {
Code int
Message string
Err error
}
func (e *DownloadError) Error() string {
if e.Err != nil {
return fmt.Sprintf("%s: %v", e.Message, e.Err)
}
return e.Message
}
// Error codes
const (
ErrFileNotFound = iota + 1000
ErrPermissionDenied
ErrQuotaExceeded
ErrInvalidFileType
ErrFileTooLarge
)
// Robust download handler with error handling
func RobustDownload(r *ghttp.Request) {
defer func() {
if err := recover(); err != nil {
// Log the stack trace
debug.PrintStack()
// Return 500 error to client
r.Response.WriteStatus(500)
}
}()
file := r.Get("file").String()
// Validate request
if err := validateDownloadRequest(r); err != nil {
handleDownloadError(r, err)
return
}
// Check user quota
if err := checkUserQuota(r); err != nil {
handleDownloadError(r, &DownloadError{
Code: ErrQuotaExceeded,
Message: "Download quota exceeded",
Err: err,
})
return
}
// Attempt file download
if err := streamFileWithRetry(r, file); err != nil {
handleDownloadError(r, err)
return
}
}
// Error handler for different scenarios
func handleDownloadError(r *ghttp.Request, err error) {
var downloadErr *DownloadError
if errors.As(err, &downloadErr) {
switch downloadErr.Code {
case ErrFileNotFound:
r.Response.WriteStatus(404)
r.Response.WriteJson(g.Map{
"error": "File not found",
"details": downloadErr.Message,
})
case ErrPermissionDenied:
r.Response.WriteStatus(403)
r.Response.WriteJson(g.Map{
"error": "Access denied",
"details": downloadErr.Message,
})
case ErrQuotaExceeded:
r.Response.WriteStatus(429)
r.Response.WriteJson(g.Map{
"error": "Quota exceeded",
"details": downloadErr.Message,
})
default:
r.Response.WriteStatus(500)
r.Response.WriteJson(g.Map{
"error": "Internal server error",
"reference": uuid.New().String(),
})
}
return
}
// Handle generic errors
r.Response.WriteStatus(500)
}
// Retry mechanism for transient failures
func streamFileWithRetry(r *ghttp.Request, file string) error {
const maxRetries = 3
const baseDelay = 100 * time.Millisecond
var lastErr error
for attempt := 0; attempt < maxRetries; attempt++ {
err := streamFile(r, file)
if err == nil {
return nil
}
lastErr = err
// Don't retry on certain errors
if errors.Is(err, os.ErrNotExist) ||
errors.Is(err, os.ErrPermission) {
return err
}
// Exponential backoff
delay := baseDelay * time.Duration(math.Pow(2, float64(attempt)))
time.Sleep(delay)
}
return fmt.Errorf("failed after %d attempts: %w", maxRetries, lastErr)
}
Specific Use Case Examples 🔍
1. Video Streaming with HLS Support 🎥
func VideoStreamHandler(r *ghttp.Request) {
videoPath := r.Get("video").String()
// Check if requesting manifest
if strings.HasSuffix(videoPath, ".m3u8") {
serveHLSManifest(r, videoPath)
return
}
// Check if requesting segment
if strings.HasSuffix(videoPath, ".ts") {
serveHLSSegment(r, videoPath)
return
}
// Serve video file directly with range support
serveVideoWithRange(r, videoPath)
}
func serveHLSManifest(r *ghttp.Request, path string) {
r.Response.Header().Set("Content-Type", "application/vnd.apple.mpegurl")
r.Response.Header().Set("Cache-Control", "max-age=5")
// Read and serve manifest
content, err := ioutil.ReadFile(path)
if err != nil {
r.Response.WriteStatus(404)
return
}
r.Response.Write(content)
}
func serveVideoWithRange(r *ghttp.Request, path string) {
info, err := os.Stat(path)
if err != nil {
r.Response.WriteStatus(404)
return
}
file, err := os.Open(path)
if err != nil {
r.Response.WriteStatus(500)
return
}
defer file.Close()
rangeHeader := r.Header.Get("Range")
if rangeHeader != "" {
ranges, err := parseRange(rangeHeader, info.Size())
if err != nil {
r.Response.WriteStatus(416)
return
}
if len(ranges) > 0 {
start, end := ranges[0][0], ranges[0][1]
r.Response.Header().Set("Content-Range",
fmt.Sprintf("bytes %d-%d/%d", start, end, info.Size()))
r.Response.Header().Set("Content-Length",
fmt.Sprintf("%d", end-start+1))
r.Response.WriteStatus(206)
file.Seek(start, 0)
io.CopyN(r.Response.Writer, file, end-start+1)
return
}
}
r.Response.Header().Set("Content-Length", fmt.Sprintf("%d", info.Size()))
r.Response.Header().Set("Content-Type", "video/mp4")
io.Copy(r.Response.Writer, file)
}
2. Large Excel File Generation 📊
func ExcelDownloadHandler(r *ghttp.Request) {
// Create a new file
f := excelize.NewFile()
defer f.Close()
// Create buffered writer
buf := new(bytes.Buffer)
writer := bufio.NewWriter(buf)
// Start progress tracking
progress := 0
total := 1000000 // Example: 1 million rows
// Stream data writing
for i := 0; i < total; i++ {
// Write row data
row := []interface{}{
fmt.Sprintf("Data %d", i),
time.Now(),
rand.Float64(),
}
cell, _ := excelize.CoordinatesToCellName(1, i+1)
f.SetSheetRow("Sheet1", cell, &row)
// Update progress every 1%
currentProgress := (i * 100) / total
if currentProgress > progress {
progress = currentProgress
// Send progress through WebSocket if needed
sendProgress(r, progress)
}
}
// Set headers for Excel file
r.Response.Header().Set("Content-Type",
"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet")
r.Response.Header().Set("Content-Disposition",
"attachment; filename=large-report.xlsx")
// Save to response writer
if err := f.Write(r.Response.Writer); err != nil {
r.Response.WriteStatus(500)
return
}
}
func sendProgress(r *ghttp.Request, progress int) {
// Implementation depends on your WebSocket setup
// Example using gorilla/websocket
if ws, ok := r.GetCtxVar("ws").(*websocket.Conn); ok {
ws.WriteJSON(map[string]interface{}{
"progress": progress,
})
}
}
3. PDF Generation and Download 📄
func PDFDownloadHandler(r *ghttp.Request) {
// Create PDF document
pdf := gofpdf.New("P", "mm", "A4", "")
pdf.AddPage()
// Add content
pdf.SetFont("Arial", "B", 16)
pdf.Cell(40, 10, "Generated Report")
// Add table
pdf.SetFont("Arial", "", 12)
data := [][]string{
{"Column 1", "Column 2", "Column 3"},
{"Data 1", "Data 2", "Data 3"},
// ... more rows
}
for i, row := range data {
for j, col := range row {
pdf.Cell(40, 10, col)
if j == len(row)-1 {
pdf.Ln(-1)
}
}
if i == 0 {
pdf.Ln(-1)
}
}
// Set headers
r.Response.Header().Set("Content-Type", "application/pdf")
r.Response.Header().Set("Content-Disposition",
"attachment; filename=report.pdf")
// Write to response
if err := pdf.Output(r.Response.Writer); err != nil {
r.Response.WriteStatus(500)
return
}
}
Common Gotchas to Avoid 🚨
-
Memory Leaks
- Always use
deferfor cleanup - Don't read entire files into memory
- Always use
-
Security Issues
- Validate file paths
- Check file permissions
- Limit file types
-
Performance Problems
- Use buffered reading
- Implement proper caching
- Don't forget to close files
Real-World Example: A Complete Download Service 🌟
Here's a production-ready download service combining all the concepts we've discussed:
type DownloadService struct {
limiter *rate.Limiter
cache *Cache
metrics *Metrics
}
func (s *DownloadService) ServeDownload(r *ghttp.Request) {
// 1. Rate limiting
if err := s.limiter.Wait(r.Context()); err != nil {
r.Response.WriteStatus(429)
return
}
// 2. Try cache
if data := s.cache.Get(r.Get("file").String()); data != nil {
s.metrics.RecordHit()
return s.serveContent(r, data)
}
// 3. Stream file with resume support
s.streamWithResume(r)
}
Testing Your Implementation 🧪
Don't forget to test! Here's a quick test case to get you started:
func TestDownload(t *testing.T) {
s := g.Server()
s.BindHandler("/download", DownloadHandler)
client := g.Client()
r, err := client.Get("/download?file=test.txt")
assert.Nil(t, err)
assert.Equal(t, 200, r.StatusCode)
}
Wrapping Up 🎁
Building production-ready file downloads with GoFrame isn't just about calling ServeFileDownload. It's about:
- Handling security properly
- Managing resources efficiently
- Supporting resume capabilities
- Monitoring and maintaining performance
What's Next?
- Add progress tracking for downloads
- Implement cloud storage integration
- Add support for file preprocessing
- Explore streaming compression
Resources 📚
Let me know in the comments if you have any questions or want to share your own experiences with file downloads in Go! 💬
Like this article? Follow me for more Go development tips and tutorials! 🚀
Top comments (0)