DEV Community

loading...

Uploading Files to MongoDB with GridFS and Multer Using NodeJS

shubhambattoo profile image Shubham Battoo Originally published at shubhambattoo.in Updated on ・7 min read

Hello, in this tutorial we will learn how to upload files directly to MongoDB using GridFS specification.

If you think TLDR; just check finish code here.

The official docs explain when to use this specification for uploading files. Which is summarized in the following:

  • If your filesystem limits the number of files in a directory, you can use GridFS to store as many files as needed.

  • When you want to access information from portions of large files without having to load whole files into memory, you can use GridFS to recall sections of files without reading the entire file into memory.

  • When you want to keep your files and metadata automatically synced and deployed across a number of systems and facilities, you can use GridFS. When using geographically distributed replica sets, MongoDB can distribute files and their metadata automatically to a number of mongod instances and facilities.

Since, GridFS stores files in chunks. Following are the collections created:

  • chunks stores the binary chunks.
  • files stores the file’s metadata.

Prerequisites

  1. NodeJS LTS
  2. MongoDB installed on your local machine
  3. a Code Editor

Setting up a local NodeJS server

Go to your command line, and type

npm init -y
Enter fullscreen mode Exit fullscreen mode

This will generate an package.json file with default values.

Then install all the dependencies required for this project

npm install express mongoose ejs multer multer-gridfs-storage 
Enter fullscreen mode Exit fullscreen mode

Create a file named app.js in the root of the project. Require the necessary packages for creating a server.

const express = require("express");
const app = express();

app.use(express.json());
app.set("view engine", "ejs");

const port = 5001;

app.listen(port, () => {
  console.log("server started on " + port);
});
Enter fullscreen mode Exit fullscreen mode

It will be better for us to create scripts to run the web app from the command line, go to your package.json file and on the scripts key, add the following:

  "scripts": {
    "start": "node app.js",
    "dev": "nodemon app.js"
  }
Enter fullscreen mode Exit fullscreen mode

then run, npm start and the server should start on the port 5001. You should see one log on the command line stating that server started on 5001.

Connecting to Database, Initializing GridFsStorage and Creating a Storage

Require all the necessary packages

const crypto = require("crypto");
const path = require("path");
const mongoose = require("mongoose");
const multer = require("multer");
const GridFsStorage = require("multer-gridfs-storage");
Enter fullscreen mode Exit fullscreen mode

Mongoose is an ORM for MongoDB which will be used for this tutorial. Multer is a NodeJS middleware which facilitates file uploads. And GridFsStorage is GridFS storage engine for Multer to store uploaded files directly to MongoDB. Crypto and Path will be used to create unique name for the file uploaded.

// DB
const mongoURI = "mongodb://localhost:27017/node-file-upl";

// connection
const conn = mongoose.createConnection(mongoURI, {
  useNewUrlParser: true,
  useUnifiedTopology: true
});
Enter fullscreen mode Exit fullscreen mode

Now, Initializing the GridFsStorage

// init gfs
let gfs;
conn.once("open", () => {
  // init stream
  gfs = new mongoose.mongo.GridFSBucket(conn.db, {
    bucketName: "uploads"
  });
});
Enter fullscreen mode Exit fullscreen mode

Here we are using the native nodejs-mongodb-drive which the mongoose uses and creating a GridFSBucket, we are passing the db to the bucket, you can see we are giving one bucket name, this bucket name will be used a name of a collection.

// Storage
const storage = new GridFsStorage({
  url: mongoURI,
  file: (req, file) => {
    return new Promise((resolve, reject) => {
      crypto.randomBytes(16, (err, buf) => {
        if (err) {
          return reject(err);
        }
        const filename = buf.toString("hex") + path.extname(file.originalname);
        const fileInfo = {
          filename: filename,
          bucketName: "uploads"
        };
        resolve(fileInfo);
      });
    });
  }
});

const upload = multer({
  storage
});
Enter fullscreen mode Exit fullscreen mode

Now we are initializing the storage as per Multer GridFS and creating random bytes using the randomBytes method present on the crypto library.

Here we are using the Promise constructor to create a promise, which then resolves with the fileInfo object. This step is optional as you can only pass a url key and the bucket will work just fine and not change the file name. For example you can use like the following :

const storage = new GridFsStorage({ url : mongoURI})
Enter fullscreen mode Exit fullscreen mode

Next lets set up our frontend with a template engine and configure express to render the template.

Creating the view

Create a new folder named views in the root of the folder, and inside it create a file named index.ejs. Here we will store our front end view. I will not bore you guys will the HTML creation and just post the code for it. I am using bootstrap for fast prototyping.

<!DOCTYPE html>
<html lang="en">

<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <meta http-equiv="X-UA-Compatible" content="ie=edge">
    <link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css">
    <title>Mongo File Upload</title>
</head>

<body>
    <div class="container">
        <div class="row">
            <div class="col-md-6 m-auto">
                <h1 class="my-4">Lets upload some stuff</h1>
                <form action="/upload" method="post" enctype="multipart/form-data">
                    <div class="custom-file mb-3">
                        <input type="file" class="custom-file-input" name="file" id="file1" onchange="readSingleFile(this.files)">
                        <label class="custom-file-label" for="file1" id="file-label">Choose file</label>
                    </div>
                    <input type="submit" value="Submit" class="btn btn-primary btn-block">
                </form>
            </div>
        </div>
    </div>

    <script src="https://code.jquery.com/jquery-3.3.1.slim.min.js"></script>
  <script src="https://cdnjs.cloudflare.com/ajax/libs/popper.js/1.14.7/umd/popper.min.js"></script>
    <script src="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/js/bootstrap.min.js"></script>
    <script>
        function readSingleFile(e) {
            const name = e[0].name;
            document.getElementById("file-label").textContent = name;
        }
    </script>
</body>

</html>
Enter fullscreen mode Exit fullscreen mode

Setting up the express app to render the view. Set up the view engine middleware to ejs

....
app.use(express.json());
app.set("view engine", "ejs");
....

app.get("/", (req, res) => {
 res.render("index")
})
Enter fullscreen mode Exit fullscreen mode

Then start the server again, go to the browser and open http://localhost:5001, and you should see one page rendered with the view we just created.

Alt Text

Create Request to handle the form submit and upload the file

app.post("/upload", upload.single("file"), (req, res) => {
  res.redirect("/");
});
Enter fullscreen mode Exit fullscreen mode

As we already did most of our heavy lifting while creating a storage bucket and multer take cares of the rest. We just need to pass the middleware and then just redirect to the same url.

The tricky part is to download or in this case stream the data from the GridFS storage bucket and render the image, for that we will create a route for showing an image that will take the name of the file as an argument or passed as a route param.

app.get("/image/:filename", (req, res) => {
  // console.log('id', req.params.id)
  const file = gfs
    .find({
      filename: req.params.filename
    })
    .toArray((err, files) => {
      if (!files || files.length === 0) {
        return res.status(404).json({
          err: "no files exist"
        });
      }
      gfs.openDownloadStreamByName(req.params.filename).pipe(res);
    });
});
Enter fullscreen mode Exit fullscreen mode

On the gridfs bucket we get access to many methods one such is find, which is very similar to normal find in MongoDB and accepts a filename as a first argument and then we are converting the result to an array and check if there is any file with such filename and if there is we use another method which is present on the gridfs bucket called openDownloadStreamByName which then again takes the filename and then we use the pipe to return the response to the client.

Now up until now, we can get the image with the above route but no way to render it on our view, so let's create a method inside the route where we were rending our index.ejs page.

....
app.get("/", (req, res) => {
  if(!gfs) {
    console.log("some error occured, check connection to db");
    res.send("some error occured, check connection to db");
    process.exit(0);
  }
  gfs.find().toArray((err, files) => {
    // check if files
    if (!files || files.length === 0) {
      return res.render("index", {
        files: false
      });
    } else {
      const f = files
        .map(file => {
          if (
            file.contentType === "image/png" ||
            file.contentType === "image/jpeg"
          ) {
            file.isImage = true;
          } else {
            file.isImage = false;
          }
          return file;
        })
        .sort((a, b) => {
          return (
            new Date(b["uploadDate"]).getTime() -
            new Date(a["uploadDate"]).getTime()
          );
        });

      return res.render("index", {
        files: f
      });
    }
  });
});
....
Enter fullscreen mode Exit fullscreen mode

Here you can see a lot of optional code like the sorting of the array and you can skip those.

Now, On the template, we loop over the files sent and then show the images below the form. We will only render the files which are of type jpg or png, that check can be upgraded by using a regex and depends on the personal preference.

        <hr>
                <% if(files) { %>
                <% files.forEach(function(file) {%>
                <div class="card mb-3">
                    <div class="card-header">
                        <div class="card-title">
                                <%= file.filename %>
                        </div>
                    </div>
                    <div class="card-body">
                        <% if (file.isImage) { %>
                    <img src="image/<%= file.filename %>" width="250" alt="" class="img-responsive">
                        <%} else { %>
                        <p><% file.filename %></p>
                        <% } %>
                    </div>
                    <div class="card-footer">
                        <form action="/files/del/<%= file._id %>" method="post">
                            <button type="submit" class="btn btn-danger">Remove</button>
                        </form>
                    </div>
                </div>
                <%}) %>
                <% } else { %>
                <p>No files to show</p>
                <% } %>
Enter fullscreen mode Exit fullscreen mode

You can see there is one remove button on the above code, so let us create one delete route to remove the file from the database.

Alt Text

// files/del/:id
// Delete chunks from the db
app.post("/files/del/:id", (req, res) => {
  gfs.delete(new mongoose.Types.ObjectId(req.params.id), (err, data) => {
    if (err) return res.status(404).json({ err: err.message });
    res.redirect("/");
  });
});
Enter fullscreen mode Exit fullscreen mode

Here we get the id as a string so that needs to be converted into a mongodb objectid and then only the bucket method can delete the file with the corresponding id. I kept things simple by not using the delete HTTP method here you are free to use it if you feel like, a post request just works fine here.

Conclusion

As we can see MongoDB provides a nice solution to store files on the database and can come handy while creating WebApps with less storage facility, but keep in mind you can only store documents up to 16mb.

Give the post a like and star the repo if it helped you.

Alt Text

Discussion (24)

pic
Editor guide
Collapse
davidpalomeque profile image
DavidPalomeque

The only thing that i can do is thank you for such amazing post . Today I was learning how to manage the images files with nodejs and mongodb , I saw some tutorials but some of them didn´t work and there were others that I couldn´t understand , so I was a little stuck . But your post solved me a lot of doubts and luckily I figure out and could do it .

The only thing that i didn´t get was this part : function readSingleFile(e) {const name = e[0].name;document.getElementById("file-label").textContent = name;}

I´ve not get why is this for . Anyway , the post is amazing , thank you .

Collapse
shubhambattoo profile image
Shubham Battoo Author

Thanks glad this could help.

Basically what that part is doing is that i am getting the file name and displaying it on the input element, its not required just some code to complete the tutorial, can be skipped.

Collapse
davidpalomeque profile image
DavidPalomeque

Thanks for the answer !

Collapse
krishnay2000 profile image
Krishnaraj Yadav

Error: Cannot find module 'ejs '
Thanks alot. This is very helpful tutorial for me.I am new in this field. I am facing above error and have installed ejs by npm and following line by line your code.
I can upload files but can't retrieve it . After writing ejs code of retrieving files i am getting this error.I need your help.

Collapse
shubhambattoo profile image
Shubham Battoo Author

did you try installing ejs? it can be done using npm, you can run npm install ejs to install ejs

Collapse
krishnay2000 profile image
Krishnaraj Yadav

Yes I have done that already and even tried to install it globally.Actually, it just works fine if I pass only json data to index.ejs but error occurs only after typing the below code in index.ejs file:

Code link : dev-to-uploads.s3.amazonaws.com/i/...
Error link : dev-to-uploads.s3.amazonaws.com/i/...

Thread Thread
shubhambattoo profile image
Shubham Battoo Author

Can you share your GitHub repo, if you have one where you where trying it, i can maybe take a look for you 😀

Thread Thread
krishnay2000 profile image
Krishnaraj Yadav

Sure . Here it is
github.com/krishna-y2000/Uploading...
Server file of above code is in /routes/uploadFiles.js and /routes/upload.js.
Client file is in /views/HomePage.ejs and /views/upload.ejs

Thread Thread
krishnay2000 profile image
Krishnaraj Yadav

Update in error :
Please refer this image because the comment box is not responding properly .
dev-to-uploads.s3.amazonaws.com/i/...
You can mail me if you want to text me in detail.
Email : krishnay.75676@gmail.com
Thanks alot for your effort.

Collapse
abskmj profile image
abskmj

Amazing post! I referred to this to implement one of my projects.

Eventually, I put together a reusable Mongoose Schema which further simplifies Mongoose and MongoDB GridFS setup. It is available as an NPM module called GridFile. It also supports the Mongoose schema association and query population.

Here is an example of how to use GridFile to upload/download files to/from MongoDB GirdFS.

Collapse
phillipdacosta profile image
Phillip DaCosta

Hello! Thank you for this helpful post! I have an issue I need some help with though.
I was able to successfully upload the images into MongoDB. However, I am not having much success retrieving the images and sending it to the frontend. The error I am currently getting is TypeError: Cannot read property 'find' of undefined ..this is happening at my ' gfs.find().toArray((err, files) => ' line under my 'app.get('/profilePicture' )' request. ..

`
const express = require('express')
const path = require('path')
const crypto = require('crypto')//to generate file name
const mongoose = require('mongoose')
const multer = require('multer')
const GridFsStorage = require('multer-gridfs-storage')
const Grid = require('gridfs-stream')
const app = express()

//CORS Middleware
app.use(function (req, res, next) {
//Enabling CORS
res.header('Access-Control-Allow-Origin', '*');
res.header('Access-Control-Allow-Methods', 'GET,HEAD,OPTIONS,POST,PUT');
res.header('Access-Control-Allow-Headers', 'Origin, X-Requested-With, Content-Type, Accept, x-client-key, x-client-token, x-client-secret, Authorization');
next();
});

const uri = "mongodb+srv://fakeaccount:password@xxxxx.mongodb.net/x_app?retryWrites=lkngdndfg"

let conn = mongoose.connection
var gfs
conn.once('open', () => {
//initialize our stream
gfs = Grid(conn.db, mongoose.mongo)
gfs.collection('employee')
})

let storage = new GridFsStorage({
url: uri,
file: (req, file) => {
return new Promise(
(resolve, reject) => {
const fileInfo = {
filename: file.originalname,
bucketName: "imageUpload"
}
resolve(fileInfo)

        })  }
Enter fullscreen mode Exit fullscreen mode

})

const upload = multer({ storage })

app.post("/uploadImg",upload.single("profilePic"),(req,res)=>{

console.log(req.body)
Enter fullscreen mode Exit fullscreen mode

res.json({file:req.file})
})

app.get('/proficPicture', (req, res) => {

    gfs.find().toArray((err, files) => {
        //check if files exist
        if (!files || files.length == 0) {
            return res.status(404).json({
                err: "No files exist"
            }) }
        // files exist
        return res.json(files)
    })


})

const PORT =5000
app.listen(PORT,()=>console.log(`Server started on port ${PORT}`))
Enter fullscreen mode Exit fullscreen mode
Collapse
zubairdroid profile image
Syed Zubair(zubi-droid)

Thank you for this post sir.. Now I can upload my files to my mongo-atlas cluster successfully. Its very kind of you on my reach to this. I can now see the uploaded collections binary data on the collections (upl.files , upl.chunks) in mongo-atlas.

But I can't find a way to download a pdf file that i uploaded sir...Can u please help me on how to download a uploaded pdf file to my database sir. :{ Help sir.

Collapse
shubhambattoo profile image
Shubham Battoo Author

Hi Syed, you can do something like this to download a pdf or any other file for this matter

<a href="image/<%= file.filename %>" download><%= file.filename %></a>

I have created a route "image/:filename" which then downloads the image which is not limited to a image and can handle any other file types also, but for this demo I was only downloading images.

app.get("/image/:filename", (req, res) => {
  // console.log('id', req.params.id)
  const file = gfs
    .find({
      filename: req.params.filename
    })
    .toArray((err, files) => {
      if (!files || files.length === 0) {
        return res.status(404).json({
          err: "no files exist"
        });
      }
      gfs.openDownloadStreamByName(req.params.filename).pipe(res);
    });
});
Collapse
zubairdroid profile image
Syed Zubair(zubi-droid)

Thank you so much for your help sir. What I have to do if i need to download the file that the user uploads from his end, and I have to download that from the database collection but not from the user side itself. if i'm not wrong, Like in google classroom app., we upload our assignments as .pdf to our professor and those attachments reaches him/her in their database(provided by that app.) and they download them on their end..... Is there any way to implement like this, sir. Any help would be appreciated.

Collapse
staszeksoup profile image
staszeksoup

Absolutely Marvelous. Thank you so much sir, I've been really scratching my head with this. My question is, how do you add other parameters? So the image has a name?

Collapse
shubhambattoo profile image
Shubham Battoo Author

Image does have a name, when we where setting up the GridFS as per Multer I used crypto and randomBytes to create a filename for the file, you can have your own strategy for naming.

following code was written if you missed it:

// Storage
const storage = new GridFsStorage({
  url: mongoURI,
  file: (req, file) => {
    return new Promise((resolve, reject) => {
      crypto.randomBytes(16, (err, buf) => {
        if (err) {
          return reject(err);
        }
        const filename = buf.toString("hex") + path.extname(file.originalname);
        const fileInfo = {
          filename: filename,
          bucketName: "uploads"
        };
        resolve(fileInfo);
      });
    });
  }
});

const upload = multer({
  storage
});
Collapse
staszeksoup profile image
staszeksoup

Yes, thanks but I mean when the user uploads the image, how can they add other fields? The image will need a unique filename, but can there be a title and description?

Thread Thread
shubhambattoo profile image
Shubham Battoo Author

Yes sure why not, but then that needs to be in other collection and you can reference the image objectid in the collection where you are storing the meta information for the images

Thread Thread
staszeksoup profile image
staszeksoup

Yes I see, so you have to save the file location and file name and add that as a string to the other collection. Thanks for your help

Thread Thread
shubhambattoo profile image
Shubham Battoo Author

File is being saved in its own collection as per gridfs, you can get an objectid for the image uploaded you can reference this objectid

Collapse
stevegroom profile image
Steve Groom

Hi,
thanks' for the blog post - its good to have a working example, however I want to take it a little further as I need to add file uploads to my own project and so must refactor your code to fit my app structure. Somewhere along the way I am missing some synchronisation. Upload triggers the file create and immediately shows the home page. If I press refresh the missing image is shown.

I think the error will be in my files.js /upload route - probably the next() is in the wrong place or the fact that I have an app.use nested inside my app.post...

I opened a Stack Overflow question and then saw I could perhaps ask here too :-)
stackoverflow.com/questions/628608...

My ‘fork’ of your repo is here: github.com/stevegroom/redogridfsst...

regards
Steve

Collapse
sumukha210 profile image
Sumukha210

I am using react as a front end and I want to upload profile image with other information such as name, email etc. How to do this stuff, Please help me

Collapse
shaheryarshaikh1011 profile image
Shaheryar Shaikh

why cant we add data more than 16 MB

Collapse
shubhambattoo profile image
Shubham Battoo Author

It is limited from MongoDb end. There are some cases listed down by mongodb where you can use this.