Uploading and Serving Images from MongoDB in Golang

👨🏻‍💻 Introduction

In this blog, we will delve into the fascinating realm of handling images in a Golang application, leveraging the power of the Gin framework for RESTful API development, MongoDB as a robust NoSQL database, and the mongo-driver library for seamless interaction with MongoDB. To store images efficiently, we'll explore the intricacies of GridFS, a specification within MongoDB for storing large files as separate chunks.

1. Gin Framework:

Gin, a high-performance web framework for Golang, will serve as the backbone of our RESTful API. Known for its speed and minimalistic design, Gin provides essential routing and middleware functionalities, making it an ideal choice for building scalable and efficient web applications. We'll explore how to set up routes, handle HTTP requests, and integrate middleware for image-related operations.

2. MongoDB and mongo-driver:

MongoDB, a document-oriented NoSQL database, will be our data powerhouse. We'll utilize the mongo-driver library to seamlessly connect our Golang application to MongoDB. This section will cover essential database interactions, including creating collections, storing metadata, and efficiently querying for image-related data. Understanding these fundamentals is crucial for building a robust image storage and retrieval system.

3. GridFS for Image Storage:

GridFS, an integral part of MongoDB, enables us to store large files, such as images, by dividing them into smaller, manageable chunks. Each chunk is stored as a separate document, allowing for efficient retrieval and storage. We'll delve into the details of using GridFS with the mongo-driver in Golang, including uploading images, managing chunks, and retrieving images seamlessly.

🔭 Methodology:

In this section, we'll walk through the step-by-step process of creating REST endpoints for uploading and serving images in a Golang application using Gin, MongoDB, and GridFS.

1. Image Uploading Endpoint:

• Receive Image via Form File:
  To initiate the image uploading process, our Golang application will expose a REST endpoint that accepts image files sent as a Form file from the client.

• Connect to MongoDB:
  We'll establish a connection to MongoDB using the mongo-driver library, ensuring seamless interaction with our database.

• Initialize GridFS Upload Stream:
  To handle large file uploads efficiently, we'll open an upload stream in the MongoDB GridFS bucket. This stream will enable us to upload image data in manageable chunks.

• Write Image Data in Chunks:
  As the image data is received, we'll write it into the GridFS upload stream in chunks. This method optimizes the storage and retrieval of large files in MongoDB.

• Return FileId in Response:
  Upon successful upload, the response to the client will include the MongoDB primitive ID (FileId) associated with the uploaded image. This ID will serve as a reference for later retrieval.

2. Image Serving Endpoint:

• Receive FileId in Request:
  The serving endpoint will expect a request containing the FileId, which uniquely identifies the desired image in the MongoDB GridFS bucket.

• Retrieve Image Data from MongoDB:
  Using the received FileId, we'll fetch the corresponding image data from the MongoDB GridFS bucket.

• Send Binary Data in Response:
  The retrieved binary image data will be sent as the response body with appropriate content-type headers, ensuring compatibility with various clients.

⚙️ Project Setup:

Step 1: Setting Up the Project

Create a Golang project and initialize it with Go modules:

go mod init image-server

Install the required packages using the following commands:

go install github.com/gin-gonic/gin
go install github.com/joho/godotenv
go install go.mongodb.org/mongo-driver

Step 2: Environment Configuration

Create a .env file in the root of your project and define two variables:

MONGOURI=<your_mongodb_connection_string>
PORT=8001

🚀 Endpoints:

Certainly! Here are the explanations for both endpoints in Markdown format:

1. Upload Image:

http://localhost:8001/upload

Method:

POST

Purpose:

Accepts a POST request to upload an image to the MongoDB database using GridFS.

Functionality:

1. Form File Extraction:

   • The endpoint expects a POST request with an image file included in the form data. The file is retrieved using c.Request.FormFile("image").

2. GridFS Bucket Creation:

   • A GridFS bucket named "photos" is created within the "image-server" database using the gridfs.NewBucket() function.

3. Image Data Buffering:

   • The image data is read into a buffer using io.Copy(buf, file).

4. Filename Generation:

   • A unique filename is generated based on the current timestamp and the original filename.

5. Upload Stream Opening:

   • An upload stream is opened in the GridFS bucket for the specified filename using bucket.OpenUploadStream(filename,).

6. Image Data Writing:

   • The image data in the buffer is written to the upload stream in chunks using uploadStream.Write(buf.Bytes()).

7. Response Construction:

   • The function constructs a JSON response containing the FileId and fileSize and sends it back to the client using c.JSON(http.StatusOK, map[string]interface{}{"fileId": fileId, "fileSize": fileSize}).

---

2. Get Image:

http://localhost:8001/image/:imageId

Method:

GET

Purpose:

Accepts a GET request to serve an image based on the provided imageId.

Functionality:

1. ImageId Extraction:

   • The endpoint expects a GET request with the imageId as a path parameter, which is extracted using imageId := strings.TrimPrefix(c.Request.URL.Path, "/image/").

2. ObjectID Conversion:

   • The extracted imageId is converted to a MongoDB ObjectID using primitive.ObjectIDFromHex(imageId).

3. GridFS Bucket Creation:

   • A GridFS bucket named "photos" is created within the "image-server" database using gridfs.NewBucket().

4. Image Data Retrieval:

   • The image data associated with the ObjectID is retrieved from the GridFS bucket using bucket.DownloadToStream(objID, &buf).

5. Content-Type Determination:

   • The content-type header is set based on the detected content type of the image using contentType := http.DetectContentType(buf.Bytes()).

6. Response Construction:

   • The function sets appropriate headers (Content-Type and Content-Length) and writes the image data to the HTTP response using c.Writer.Write(buf.Bytes()), effectively serving the image.

---

These explanations provide a detailed understanding of the both the endpoints for image upload and serving in from the server. You can also checkout the PostMan documentation where you can better understand the working of both these endpoints.

🤖 Code:

Paste the below code in main.go file.

package main

import (
    "bytes"
    "context"
    "encoding/json"
    "io"
    "log"
    "net/http"
    "os"
    "strconv"
    "strings"
    "time"

    "github.com/gin-gonic/gin"
    "github.com/joho/godotenv"
    "go.mongodb.org/mongo-driver/bson/primitive"
    "go.mongodb.org/mongo-driver/mongo"
    "go.mongodb.org/mongo-driver/mongo/gridfs"
    "go.mongodb.org/mongo-driver/mongo/options"
)

// To get mongodb connection URI from .env file
func EnvMongoURI() string {
    err := godotenv.Load()
    if err != nil {
        log.Fatal("Error loading .env file")
    }
    return os.Getenv("MONGOURI")
}

// To connect to mongodb
func ConnectDB() *mongo.Client {
    ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
    defer cancel()

    // mongo.Connect return mongo.Client method
    uri := EnvMongoURI()
    client, err := mongo.Connect(ctx, options.Client().ApplyURI(uri))
    if err != nil {
        log.Fatal("Error: " + err.Error())
    }

    //ping the database
    err = client.Ping(ctx, nil)
    if err != nil {
        log.Fatal("Error: " + err.Error())
    }
    log.Println("Connected to MongoDB")
    return client
}

// Client instance
var DB *mongo.Client = ConnectDB()
var name = "photos"
var opt = options.GridFSBucket().SetName(name)

// Getting database collections
func GetCollection(client *mongo.Client, collectionName string) *mongo.Collection {
    collection := client.Database("image-server").Collection(collectionName)
    return collection
}

// Upload image handler
func uploadImage() gin.HandlerFunc {
    return func(c *gin.Context) {
        file, header, err := c.Request.FormFile("image")
        if err != nil {
            log.Fatal(err)
            c.JSON(http.StatusBadRequest, err.Error())
            return
        }
        defer file.Close()

        bucket, err := gridfs.NewBucket(
            DB.Database("image-server"), opt,
        )
        if err != nil {
            log.Fatal(err)
            c.JSON(http.StatusBadRequest, err.Error())
            return
        }

        buf := bytes.NewBuffer(nil)
        if _, err := io.Copy(buf, file); err != nil {
            log.Fatal(err)
            c.JSON(http.StatusBadRequest, err.Error())
            return
        }

        filename := time.Now().Format(time.RFC3339) + "_" + header.Filename
        uploadStream, err := bucket.OpenUploadStream(
            filename,
        )
        if err != nil {
            log.Fatal(err)
            c.JSON(http.StatusBadRequest, err.Error())
            return
        }
        defer uploadStream.Close()

        fileSize, err := uploadStream.Write(buf.Bytes())
        if err != nil {
            log.Fatal(err)
            c.JSON(http.StatusBadRequest, err.Error())
            return
        }

        fileId, _ := json.Marshal(uploadStream.FileID)
        if err != nil {
            log.Fatal(err)
            c.JSON(http.StatusBadRequest, err.Error())
            return
        }
        c.JSON(http.StatusOK, map[string]interface{}{"fileId": strings.Trim(string(fileId), `"`), "fileSize": fileSize})
    }
}

// Serving image over http REST handler
func serveImage() gin.HandlerFunc {
    return func(c *gin.Context) {
        imageId := strings.TrimPrefix(c.Request.URL.Path, "/image/")
        objID, err := primitive.ObjectIDFromHex(imageId)
        if err != nil {
            log.Fatal(err)
            c.JSON(http.StatusBadRequest, err.Error())
            return
        }

        bucket, _ := gridfs.NewBucket(
            DB.Database("image-server"), opt,
        )

        var buf bytes.Buffer
        dStream, err := bucket.DownloadToStream(objID, &buf)
        if err != nil {
            log.Fatal(err)
            c.JSON(http.StatusBadRequest, err.Error())
            return
        }

        log.Printf("File size to download: %v\n", dStream)
        contentType := http.DetectContentType(buf.Bytes())

        c.Writer.Header().Add("Content-Type", contentType)
        c.Writer.Header().Add("Content-Length", strconv.Itoa(len(buf.Bytes())))

        c.Writer.Write(buf.Bytes())
    }
}

// Home endpoint to check server status
func homePage() gin.HandlerFunc {
    return func(c *gin.Context) {
        c.JSON(http.StatusOK, "Home page of Image Server")
    }
}

func Route(router *gin.Engine) {
    //All routes will be added here
    router.GET("/", homePage())
    router.POST("/upload", uploadImage())
    router.GET("/image/:imageId", serveImage())
}

func main() {
    port := os.Getenv("PORT")
    if port == "" {
        port = "8001"
    }
    router := gin.Default()
    Route(router)

    router.Run(":" + port)
}

😄 About Me

I’m Sitaram Rathi, a Full-stack Developer focused on back-end development. I'm pursuing my B.Tech+M.Tech Dual Degree in Computer Science & Engineering from NIT Hamirpur. I love working on projects and problems which make me push my limits and learn something new. You can connect with me here.