Discussion on: How to use node.js streams for fileupload

[aderchox]

This article is really valuable, I also want to ask you, just as @longbotton_dev did, to write more of such amazing articles on Node.js core/fundamentals. I have a few questions though:
Question 1. When I do the above, the file, which is an image, is uploaded successfully, and the size of the upload file matches the original file as well, but when I open the uploaded image (the one on the server) it's just blank. I tried with a few different images to make sure it's not an issue of a certain image.
My code:
Client:

        fileinput.onchange = (e) => {
            const formData = new FormData();
            console.log({ file: e.target.files[0] });
            formData.append('file', e.target.files[0]);
            upload(formData);
        }
        async function upload(data) {
            const response = await fetch("http://localhost:1234/upload", {
                method: "POST",
                body: data
            })
        }

Server:

app.post("/upload", (req, res) => {
  const stream = fs.createWriteStream(path.resolve(__dirname, "file.png")); // name is hard-coded
  stream.on("open", () => req.pipe(stream));
});

Question 2. Towards the end of the article you've said:

To make it more interesting, try using the browser's filereader API and send the file asynchronously, instead of using a form.

But according to this SO answer:

When doing a file upload from a File on disk, the browser doesn't load the full file in memory but streams it through the request. This is how you can upload gigs of data even though it wouldn't fit in memory. This also is more friendly with the HDD since it allows for other processes to access it between each chunk instead of locking it.
When reading the File through a FileReader you are asking the browser to read the full file to memory, and then when you send it through XHR the data from memory is being used. You are thus limited by the memory available, bloating it for no good reasons, and even asking the CPU to work here while the data could have gone from the disk to the network card almost directly.

So it seems that using FileReader is not interesting at all... or maybe I'm getting it wrong.
I hope you're still checking dev.to 😁 Thanks a lot.

[tq-bit]

Hi there. Thank you for your reply :-)

I'm still here, will try and replicate your first case.

Good point on the filereader as well. This one was one of the first posts I made when learning Javascript & Node.js and wasn't very familiar with what's good for performance and what's not. If you have an input field available, you don't necessarily need the file reader. I just figured it'd be helpful to include because it'd be the next thing I took a look at.

PS: Out of curiousity: What about Node.js core content would you like to read? I thought about writing an article on how the http module works, but I feel like that'd be a bit trivial.

[aderchox]

1. This might be ridiculous (or funny?), but after I added this comment here, I read a chapter on streams from a node.js book and now that I check node.js docs again, I don't see much more that I personally need to learn about node.js itself. But I think the main reason I asked for more is because the first part of your article (the theory) was so well explained that excited me :D and I thought your articles will be valuable for future readers about whatever they should be.

2. Did you manage to replicate the issue (the first case)?

3. I also want to recommend a few things about the article:

   • Mention that this is exactly what packages like Multer and Formidable use under the hood.
   • Mention that <input type="file"> does not load the files in RAM and it is things like fetch that create the stream automatically internally as soon as they are making the request (so no need to use the FileReader API explicitly). I know it's not directly related to your article, but realizing this connected some vagueness dots for me personally.
   • Explain a bit more clearly where the "open" event is documented, I couldn't find it on Writeable Streams docs on Node.js docs. Maybe this was for an older version of Node.js?

Thanks for your response.

[tq-bit]

1. Noted. I do try to make my articles easily graspable. Sometimes it works, sometimes it doesn't.

2. Yes. All you have to do is to leave the form data out. Or implement a form parser on the backend. You're basically handling the raw binary data without the form wrapper. I'm not exactly sure how form data is parsed, but I did change your code so it looks like so and it worked (same server code). I attached the full staticially served index.html file:
   

<!DOCTYPE html>
<html lang="en">
    <head>
        <meta charset="UTF-8" />
        <meta http-equiv="X-UA-Compatible" content="IE=edge" />
        <meta name="viewport" content="width=device-width, initial-scale=1.0" />
        <title>Document</title>
    </head>
    <body>
        <h1>Hello World!</h1>
        <input type="file" />

        <script>
            const input = document.querySelector('input');
            input.onchange = (e) => {
                upload(e.target.files[0]);
            };
            async function upload(data) {
                const response = await fetch('http://localhost:1234/upload', {
                    method: 'POST',
                    body: data,
                });
            }
        </script>
    </body>
</html>

1. Formidable works a bit differently. It's a form parser, more standardised than what's going on here. Since I discovered fastify, I favour Busboy over Multer, but I believe it serves the same purpose.

[aderchox]

I found the answer to my third point too, I'll add here for future readers:
Based on the documentation, createWriteStream returns an instance of <fs.WriteStream> that has an 'open' event which is emitted when the <fs.WriteStream>'s file is opened: nodejs.org/api/fs.html#event-open_1
(btw, this is weird, I did the exact same thing as you and passed the data leaving form data out, but still not working for me, but thanks anyways).

[satya nishanth]

Just to be on the same page. In many upload file kinda websites they preview the input . I believe that has load into ram to preview right? ( I mean if the preview is enabled)

[tq-bit]

Yes. Images are always loaded into memory when rendering a page. Instead of instantly uploading the img and providing a link, some pages store images as base64 on the user's computer and permit uploading only after a confirmation.