loading...

Handling file uploads simple and memory friendly in Go with Baraka

xis profile image Enes Furkan Olcay Updated on ・4 min read

Hi, in this article im going to explain how to handle a file upload simply and memory friendly in Go.

story, problem, baraka

I was handling files with ParseMultipartForm before then i decided to create a mini library to minimize the code because i was creating APIs and rewriting some file handling codes when there is a file upload endpoint. I was coding the library, examining ParseMultipartForm function and i saw something wrong.
When you use ParseMultipartForm, you keep ALL the files from the http requests in memory or disk.
Even if there is a maximum size parameter on ParseMultipartForm, this parameter stands for maximum memory to allocate, not for disk.
For example, if you set maximum size to 32 mb, you can keep files up to 32mb in memory. If there is a 50mb file in the request, you can still keep it but just in the disk.
So, you can limit the request size and fix it too or you can just use MultipartReader. Baraka uses it and makes it simple to use.

dependencies

gin-gonic (optional)

go get -u github.com/gin-gonic/gin

baraka

go get -u github.com/xis/baraka

handling files

firstly, we create a file storage

func main() {
    // create a storage
    storage, err := baraka.NewStorage("./pics/", baraka.Options{})
    if err != nil {
        fmt.Println(err)
    }
}

Then, we create http router. I used gin here but you don't have to.

func main() {
    // create a storage
    storage, err := baraka.NewStorage("./pics/", baraka.Options{})
    if err != nil {
        fmt.Println(err)
    }
    router := gin.Default()
    router.POST("/upload", func(c *gin.Context) {

    })
    router.Run()
}

Let's fill our handler function

router.POST("/upload", func(c *gin.Context) {
    // parsing
    maxFileSize := 5 << 20 // 5 MB
    maxFileCount := 5
    p, err := storage.ParseButMax(maxFileSize, maxFileCount, c.Request)
    if err != nil {
        fmt.Println(err)
    }
    // saving
    p.Store("product_1_")
})

Firstly, i defined the max limits then i used storage.ParseButMax(). This function parses the request and gets the files from it. Finally i used p.Store("product_1_"), this function stores the file to disk and it's parameter sets a prefix for filenames.
For example, i have product images and that product's id is 1 and got 4 file from request. So, i will see files in the disk like that;

product_1_1.jpg
product_1_2.jpg
product_1_3.jpg
product_1_4.jpg

so, we simply parsed and stored files.

other

You can do some other things with baraka, like you can get json data with it, or files count, content types, filenames and filtering.

filtering

Filtering is cool, because you can prevent unwanted files from getting into memory, so you got a lot of memory advantage than ParseMultipartForm.

Let's get just JPG files into memory.
firstly, when declaring a storage pass a filter function to its options.


func main() {
    storage, err := baraka.NewStorage("./pics/", baraka.Options{
        // passing filter function
        Filter: func(file *multipart.Part) bool {
            // file bytes and file information reachable here
        },
    })
    ...codes below...

You can pass a function to options. Filter function can reach file bytes, so you can make a lot of things here.
Let's identify files and filter them.

func main() {
    storage, err := baraka.NewStorage("./pics/", baraka.Options{
        // passing filter function
        Filter: func(file *multipart.Part) bool {
            // create a byte array
            b := make([]byte, 512)
            // get the file bytes to created byte array
            file.Read(b)
            // detect the content type
            fileType := http.DetectContentType(b)
            // if it is jpeg then pass the file
            if fileType == "image/jpeg" {
                return true
            }
            // if not then don't pass
            return false
        },
    })
    ...codes below...

I created a byte array for http.DetectContentType function so we can identify file's bytes with it, then a simple if condition to check if it is a JPG file. If it is then we return true to pass to the memory, if not we return false, and non-JPG files can't get into memory. cool.

getting file's information

You can get file's information too. Simple than Filtering.

... codes above ...
    p, err := storage.Parse(c.Request)
    if err != nil {
        fmt.Println(err)
    }
    // prints filenames
    fmt.Println(p.Filenames())
    // prints total files count
    fmt.Println(p.Length())
    // prints content types of files
    fmt.Println(p.ContentTypes())
... codes below ...

I think you got it.

additional data with files, json

so, what if i need some json data with files? Just be sure that you added a json file into your request's multipart form and set it's Content-Type to application/json then use process.JSON() function. lets see it on code.

... codes above ...
   process, err := storage.Parse(c.Request)
   if err != nil {
    fmt.Println(err)
   }
   b, err := process.JSON()
   if err != nil {
    fmt.Println(err)
   }
   var foo Foo
   err := json.Unmarshal(b, foo)
   if err != nil {
    return err
   }
... codes below ...

you get bytes of json data from process.JSON(), unmarshal it to your struct or you can make another things with it.

end & contribute

This is the end of the article, baraka is open to contributors. open an issue to discuss, suggest a feature, make a bugfix etc.
bye!

GitHub logo xis / baraka

a tool for handling file uploads, simple, memory-friendly

baraka

Go Report Card codecov Build Status go.dev reference

a tool for handling file uploads for http servers

makes it easier to save multipart files from http request and to filter them prevents unwanted files from getting into memory, extracts json data with files.

install

go get -u github.com/xis/baraka

using

func main() {
    // create a storage
    storage, err := baraka.NewStorage("./pics/", baraka.Options{})
    if err != nil {
        fmt.Println(err)
    }
    router := gin.Default()
    router.POST("/upload", func(c *gin.Context) {
        // parsing
        p, err := storage.Parse(c.Request)
        // or you can use ParseButMax if you need limited size 
        p, err := storage.ParseButMax(32<<20, 5, c.Request)
        if err != nil {
            fmt.Println(err)
        }
        

Posted on by:

xis profile

Enes Furkan Olcay

@xis

full stack dev. focused on backend. curious for real-time communication and rockets.

Discussion

pic
Editor guide