DEV Community 👩‍💻👨‍💻

ldsrogan
ldsrogan

Posted on

AWS SDK with Javascript: Multi-Files Download from S3

If you want to download multiple files as zipped from AWS S3, and you have your server to take care of the file metadata handlings, then this article may help you understand how the process works. AWS doesn't provide the default multi-files download, so in order to achieve you may want to add lambda function or to use your own implemented service. This article uses the custom service to support multi-files download function.

To know the basic setup of the code (AWS keys, the meaning of Key in the code, bucket, etc.), please refer to another article that I worte AWS SDK with Javascript: Download File from S3.

import * as AWS from 'aws-sdk';
import archiver from 'archiver';
import path from 'path'
import { PassThrough } from 'stream';

const s3bucket = new AWS.S3({
  accessKeyId: process.env.AWS_ACCESS_KEY,
  secretAccessKey: process.env.AWS_SECRET_KEY,
  signatureVersion: 'v4',
  region: process.env.AWS_REGION, // ex) us-west-2
});
Enter fullscreen mode Exit fullscreen mode

Here is the main function that allows to receive file infos as an input parameter and to create stream object that can be used to write or pass the data as zipped format.

const multiFilesStream = (infos) => {
  // using archiver package to create archive object with zip setting -> level from 0(fast, low compression) to 10(slow, high compression) 
  const archive = archiver('zip', { zlib: { level: 5 } });

  for (let i = 0; i < infos.length; i += 1) {
    // using pass through stream object to wrap the stream from aws s3
    const passthrough = new PassThrough();
    s3bucket
      .getObject({
        Bucket: process.env.AWS_BUCKET_NAME,
        Key: path.join(infos[i].path, infos[i].filename);
      })
      .createReadStream()
      .pipe(passthrough);
    // name parameter is the name of the file that the file needs to be when unzipped.
    archive.append(passthrough, { name: infos[i].filename });
  }
  return archive;
};

Enter fullscreen mode Exit fullscreen mode
const files = [
   {path: '/drive', filename: 'file1.jpg'},
   {path: '/drive', filename: 'file2.jpg'},
   {path: '/drive', filename: 'file3.jpg'},
   {path: '/drive', filename: 'file4.jpg'},
];

const mfStream = multiFilesStream(files);

// res is the response object in the http request. You may want to create your own write stream object to write files in your local machine
mfStream.pipe(res);

// use finalize function to start the process
mfStream.finalize();
Enter fullscreen mode Exit fullscreen mode

Top comments (4)

Collapse
navdesign profile image
Navninder Benipal

Hey thanks for posting this amazing solution. I was getting an error (0, archiver_1.default) is not a function. Would you care to help me out resolve it???

Collapse
ng_speedster profile image
Yuriy
import * archiver  from "archiver";
Enter fullscreen mode Exit fullscreen mode
Collapse
navdesign profile image
Navninder Benipal

Hey thanks, I figured it out long ago. I was developing an app and needed this file archiving and download feature. I used this archiver utility for my app dockefy. Take a look and let me know if you like it at dockefy.com

Thread Thread
ng_speedster profile image
Yuriy

Hi, I had such a feeling...)
I left that comment in case someone might run into the same issue)
dockefy.com looks awesome, great job!

🌚 Friends don't let friends browse without dark mode.

Sorry, it's true.