Background
Click this if you want to skip to main part
I recently came across a problem in Next.js, I was just working at some api endpoints at my job, at one of the endpoints, files from user needed to be accepted at this endpoint.
Simple enough, I thought I had done file uploads before, this would be no different, I can just use multipart form data and get the files inside a field, easy enough.
But that's when a thought came across my mind as this endpoint was supposed to accept relatively large files, maybe Gigabytes in size, would the server be able to handle all these large files in concurrent?
The first thought was of course HTTP Servers are made for this stuff and I had done this with Express before as well, but as I remembered this, I knew that in Express file uploads are handled by multer which is built on **busboy **which basically streams the data into your selected location on disk.
I thought I can just use multer with Next.js as well, everything will work out fine. That's where the deep dive began.
See I'm no code expert, I'm just a developer who has delved head first into things that interested me and learn the basics stuff while learning the more advanced stuff on the way, so this was a genuine challenge for me.
I know I could use other approached like Pre-Signed Url as the content needs to be uploaded to a bucket eventually, still it has it's issues and I really wanted to dive into using these libraries with Next.js.
I researched through some solutions online and did find some but something or another always had some type issues, sometimes I hate Typescript.
Anyways, a few days deep diving into everything I did figure it out, a research there some help with ChatGPT there. So I'm happy to share this with everyone so that not everyone has to deep dive like me.
Main Article Starts Here
So why won't busboy and multer work out of the box of Next.js
This issue lies with how Next.js is implemented, first up you can't provide your own handlers in route.ts anymore, Named Exports are the way to go.
NextRequest which is provided in handlers in Next.js as the Request object doesn't extend the Node.js request object, there are multitudes of reasons behind this, one of them is to support multiple runtimes like the Edge runtime that Next.js provides (Maybe an expert can shed some more light on this), it instead extends the Request object from Web Fetch API.
This is a problem if you're working with Typescript as busboy and multer expect Request of Node.js type, which contains a type IncomingMessage which contains the required headers.
So what is the solution?
Important note all the code next is only applicable in Node.js runtime, edge doesn't have these functions available
For safety purposes you can use this code in your route.ts file if you want export const runtime = 'nodejs';
The solution is surprisingly simple when you understand how these libraries work.
Step:1 Convert the Web API Request headers to Node.js Request Headers
busboy requires headers in the form of string to string object key pairs or in Typescript terms a Record type object or {
[k: string]: string;
} interface if you prefer that.
We can achieve this simply by converting using it to a normal object as the headers are in a Map form using Object.fromEntries(req.headers), then pass these to busboy.
Step:2 Piping the request to busboy
If you have worked with busboy before you know we have to pipe the request busboy instance via request, but when does that actually work, as we don't have any pipe method in Request object from Web API.
The answer is busboy internally requires request body in the form of a ReadableStream, so that it can check if a file or field is being sent and then process them in chunks accordingly, also this is different from the ReadableStream provided by Web API.
This can be easily done
So the question becomes how can we send the Request body as a ReadableStream to busboy/multer for it to process the files, for our convenience Node.js already provides a method for this.
import { Readable } from "node:stream"
const nodeStream = Readable.fromWeb(req.body as any);
This code first uses the req.body Instance Property available on the Request object from Web API, which Provides us a ReadableStream from Web API, now you may checkout the link or know that this method has limited availability, like in Firefox, but don't worry it's a bit misleading, this only happens if you create the request in a certain way, like this to be exact:
const request = new Request("/myEndpoint", {
method: "POST",
body: JSON.stringify({ name: "John Wick", password: "P@ssw0rd!" }),
});
which is not common nowdays, if you just use even the fetch api like:
fetch("/myEndpoint", {
method: "POST",
body: JSON.stringify({ name: "John Wick", password: "P@ssw0rd!" }),
});
body will not be empty, now I do not know the exact reason behind this, so it would be helpful if someone could explain in detail, but from what I understood, reading the documentation and some research, the fetch function does create the body instanceMethod to Request when making a request, but the standalone Request constructor object does not get the body instanceMethod in Firefox.
Step:3 Using the modified Request in the route handler
Back to the point now we can safely pipe the busboy instance to request by converting it to a ReadableStream of Node.js type, and make functions for different events that busboy may encounter, so the final route.ts file looks like
import { NextRequest, NextResponse } from "next/server";
import busboy from 'busboy'
import { Readable } from "node:stream"
import fs from 'fs'
import { cwd } from "node:process";
export const runtime = 'nodejs';
export async function POST(req: NextRequest) {
return new Promise<NextResponse>((resolve, reject) => {
// Convert to plain object
const headers = Object.fromEntries(req.headers);
const bb = busboy({ headers });
bb.on('file', (name, file, info) => {
console.log(`File field [${name}]: filename=${info.filename}, mime=${info.mimeType}`);
// Stream file to disk
const savePath = `${cwd()}/tempuploads/${info.filename}`;
file.pipe(fs.createWriteStream(savePath));
});
bb.on('field', (name, value) => {
console.log(`Field [${name}]: value=${value}`);
});
// Can be both finish/close
// bb.on('finish', () => {
bb.on('close', () => {
resolve(NextResponse.json(({ message: 'Files and fields received' }), { status: 200 }));
});
bb.on('error', (err) => {
reject(err);
});
// Convert Fetch API stream → Node stream
const nodeStream = Readable.fromWeb(req.body as any);
nodeStream.pipe(bb);
});
}
Multer Implementation
Now this works totally fine and you can use this as you want and process your files, how you like maybe and write any other logic. But let's delve one step further what if you want to use multer instead and use it's capability like fields to better handle your files, well you can do that as well, as multer as mentioned before uses busboy in the internals.
The implementation is a bit lengthy but for this to work we basically create a custom parsing function that parses our request and we have to follow the same steps as busboy first, we create two files, first uploader.ts which will have the custom parser function, and the handler route.ts file which will consume it. I have also created a basic interface called MulterFile which are the properties that a file will contain after being parsed. I have also added some filters as an example for processing. Also I have also installed Express for it's Request type as multer is a middleware for Express, you can just use normal Web API Request as well but Typescript is going to have some more issues, the code still works fine although.
uploader.ts
import { Request as ExpressRequest } from 'express';
import multer from 'multer';
import { cwd } from 'process';
import { Readable } from 'stream';
import { v4 as uuidV4 } from 'uuid'
export interface MulterFile {
fieldname: string,
originalname: string,
encoding: string,
mimetype: string,
destination: string,
filename: string,
path: string,
size: number,
}
const storage = multer.diskStorage({
destination: function (req, file, cb) {
cb(null, `${cwd()}/tempuploads/`)
},
filename: function (req, file, cb) {
const uniquePrefix = uuidV4();
cb(null, uniquePrefix + '-' + file.originalname)
}
})
const upload = multer({
storage: storage,
fileFilter: (req, file, cb) => {
if(file.fieldname=='avatar' && !file.mimetype.startsWith('image')){
return cb(new Error("Unexpected file type"))
}
return cb(null, true)
}
});
export async function parseMultipart(req: Request, fields: { name: string; maxCount: number }[]): Promise<ExpressRequest>{
const nodeReq: any = Readable.fromWeb(req.body as any)
nodeReq.headers = Object.fromEntries(req.headers)
nodeReq.method = req.method;
nodeReq.url = req.url;
return new Promise((resolve, reject) => {
upload.fields(fields)(nodeReq, {} as any, (err: any) => {
if (err) return reject(err);
resolve(nodeReq as ExpressRequest);
});
});
}
The difference here is we process the nodeReq as any and attach some required properties to the Request object which are needed by multer.
Then in the route handler file we use parseMultipart function.
route.ts
import { MulterFile, parseMultipart } from "@/multer/uploader";
import { unlinkSync } from "fs";
import { MulterError } from "multer";
import { NextRequest, NextResponse } from "next/server";
export async function POST(req: NextRequest){
try {
try {
const parsedReq = await parseMultipart(req, [
{ name: 'avatar', maxCount: 1 },
]);
const avatar: MulterFile = (parsedReq.files as any)?.avatar?.[0]
const userName = parsedReq.body?.username as string
console.log("avatar is: ",avatar)
console.log("userName is: ", userName)
setTimeout(() => {
unlinkSync(avatar.path)
}, 3000);
} catch (error) {
console.error("Error while getting files", error)
const errorMessage = error instanceof Error?
error.message:error instanceof MulterError?
error.message:"Unexpected Error"
return NextResponse.json(
{error: errorMessage},
{status: 400}
)
}
return NextResponse.json(
{message: "Files and fields received"},
{status: 200}
)
} catch (error) {
console.error("Error while uploading file in file2", error)
return NextResponse.json(
{error: "Internal Server Error"},
{status: 500}
)
}
}
Now you are able to handle files using multer as well!!
So I hope this article helped you out and you can parse files with busboy or multer all you want in Next.js.
Do let me know your feedback on the article and any suggestions you may have.
If you have any questions feel free to ask away, I'll be glad to help.
Happy Coding
Signing Off
Grimshinigami
Top comments (0)