DEV Community: Ray Lassiter

Download Multiple Files Using FormData

Ray Lassiter — Fri, 07 Mar 2025 19:21:23 +0000

If you're like most web developers, you probably associate multipart/form-data encoding with file uploads. What if I told you that the same thing can be done in reverse? Sure, it's slightly less straightforward, but doing so allows us to download several files with one request. Totally worth it, right? The secret, as we'll see here, is the form-data NPM library. By importing it into our node.js project, we can serve all sorts of data to the client, including booleans, strings, and even binary content such as MP3s.

The Server Code

I'm using express, but you can go with any type of server you prefer. You'll want to install the form-data module so that you can use it: npm i form-data.

Then import it into your server like so:

const FormData = require('form-data');

In your request handler, you'll need to instantiate the FormData object before we can append data to it. For file content, we need to supply the title, content (as a buffer in this case), and the file details. That should include the full file name - including the extension - along with the content type and known length.

app.post("/", 
  async (req, res) => {
    // instantiate the form
    const form = new FormData();
    const mp3Titles = req.titles;

    getAudioAsBuffer(mp3Titles)
      .then(buffers => {
        buffers.forEach((buffer, i) => {
          const title = mp3Titles[i];
          form.append(title, buffer, {
            filename: `${title}.mp3`,
            contentType: 'audio/mpeg',
            knownLength: buffer.length,
          });
        });

        writeForm(res, form);
    }).catch(error => {
      // handle error
    });
});

Perhaps the trickiest part is sending the multipart/form-data response. To do that, we need to set some headers via the writeHead() method - most notably the Content-Type and the Content-Length. Luckily, the FormData object has just the methods we need: getBoundary() and getLengthSync() respectively.

Then it's just a matter of writing the form's buffer to the client.

const writeForm = (res, form) => {
  // boundary must match the one generated by form-data
  res.writeHead(200, {
    'Content-Type': `multipart/form-data; boundary=${form.getBoundary()}`,
    'Content-Length': form.getLengthSync(),
    'Access-Control-Allow-Credentials': 'true',
    'Access-Control-Allow-Origin': '*',
  });

  res.write(form.getBuffer());
  res.end();
};

The Client-side Code

To make the server call I'm using axios. It returns the response as a promise.

The response.data property is the FormData object. It provides a few methods for obtaining its payload. I used values() which returns an iterator. From there I converted it to an array so that I could inspect its size. Each value should be a Blob instance assuming that all went well. In that case the array is passed along to the downloadBlob() method for processing.

axios
  .post(
    "http://localhost:5000/",
    { 
      // files to fetch
    },
    {
      headers: { Accept: "multipart/form-data" },
      adapter: 'fetch', 
      responseType: 'formdata'
    }
  )
  .then((response) => { 
    const values = Array.from(response.data.values());
    if (values.length > 0) {
      if (values[0] instanceof Blob) {
        this.downloadBlob(values);
      } else {
        //Check for errors
      }
    }
  });

Thanks to the <a> element's download attribute, it's now possible to download a file from the browser. Of course, it's limited to the downloads folder, but it's still extremely useful. Luckily, there are no restrictions regarding the use of JavaScript to either dynamically create the <a> element or to programmatically invoke its onclick event.

To set the download attribute to the Blob, we can use the URL.createObjectURL() method. It can accept a File, image, or any other MediaSource object for which the object URL is to be created. Since we stored each MP3 Blob as a file, we can pass it directly to createObjectURL() without having to apply any further transformations.

Note that most browsers will present a popup dialog asking you to confirm that the page is attempting to download more than one file. Once you click OK, the process should be fully automatic.

downloadBlob = (files) => {
  const a = window.document.createElement('a');
  a.href = window.URL.createObjectURL(files[0]);
  a.download = files[0].name;
  // Append anchor to body.
  document.body.appendChild(a);
  a.click();
  // Remove anchor from body
  document.body.removeChild(a);
  files = files.slice(1);
  if (files.length > 0) {
    setTimeout(() => this.downloadBlob(files), 200);
  }
}

A few things to notice in the above code:

The downloadBlob() method is recursive; it slices all but the first element off the array and proceeds to invoke itself again only if there are any more files to process.
There is a setTimeout() of 200 milliseconds. This is necessary to allow time for the download process to kick off.
A new <a> element is created for every file. It may be possible to recycle the same one by passing it to the downloadBlob() method, but I'll leave it up to you to explore that strategy.

Conclusion

In this article we learned how send multiple files to a client from an express server in a single response using a few NPM libraries. One thing we didn't cover here is how to convert binary file data into a buffer to append to the form. That is a whole other challenge that is best left for a separate article...

Follow me on Substack
Follow me on Bluesky
Connect with me on Linkedin

Converting CSV Data To JSON In Node.js

Ray Lassiter — Wed, 27 Nov 2024 19:16:31 +0000

A short time ago, I wrote a service that fetches historical data from Yahoo! Finance in order to generate Monte Carlo simulations based on a security's performance. For those of you who may be unfamiliar with Monte Carlo simulations, they are a probabilistic modeling technique that assesses potential outcomes in complex scenarios involving random variables. By simulating multiple probability scenarios, Monte Carlo simulations help analyze risk and uncertainty across various fields like investing, business, physics, and engineering.

One of the great things about Yahoo! Finance is that it lets you download historical data in CSV format. The only problem is that my service - like many - serves its response in JSON format. That required performing a transformation on the incoming data. This article will cover exactly how I achieved that in Node.js using the excellent Node.js csvtojson library.

Comparing the CSV and JSON Formats

Let's quickly go over both formats and establish what needs to happen in order to go from one to the other.

CSV (Comma-Separated Values) is a simple file format used to store tabular data. In a CSV file, each line represents a row of data, and values within each row are separated by commas. It's one of the most popular formats for importing data into a database. For example, here's a sample of a Yahoo! Finance response:

Date,Open,High,Low,Close,Adj Close,Volume
2004-08-19,2.490664,2.591785,2.390042,2.499133,2.499133,897427216
2004-08-20,2.515820,2.716817,2.503118,2.697639,2.697639,458857488
2004-08-23,2.758411,2.826406,2.716070,2.724787,2.724787,366857939
2004-08-24,2.770615,2.779581,2.579581,2.611960,2.611960,306396159
2004-08-25,2.614201,2.689918,2.587302,2.640104,2.640104,184645512
2004-08-26,2.613952,2.688672,2.606729,2.687676,2.687676,142572401
2004-08-27,2.692408,2.705360,2.632383,2.643840,2.643840,124826132

Meanwhile, JSON (JavaScript Object Notation) is a lightweight, text-based data interchange format that is easy for humans to read and write and simple for machines to parse and generate. It consists of two primary structures: objects (key-value pairs enclosed in curly braces {}) and arrays (ordered lists enclosed in square brackets []). To get an idea what it looks like, here is a partial response from my API service:

{
  "stockData": [
    {
      "Date": "2019-01-02T00:00:00.000Z",
      "Close": 52.233059
    },
    {
      "Date": "2019-01-03T00:00:00.000Z",
      "Close": 50.745255
    },
    {
      "Date": "2019-01-04T00:00:00.000Z",
      "Close": 53.474648
    },
    {
      "Date": "2019-01-07T00:00:00.000Z",
      "Close": 53.35878
    },
    {
      "Date": "2019-01-08T00:00:00.000Z",
      "Close": 53.752831
    },
    // etc...
  ]
}

Getting Started With csvtojson

There are a few similar libraries, with csvtojson being one of the most active. To install it using the Node Package Manager (npm), run the following in the terminal:

npm i csvtojson

Next, add the following import statement to the top of your Node script:

const csvtojson = require("csvtojson");

And just like that, we're ready to use the library!

Transforming the Data

To transform the data, all we need to do is invoke the csvtojson constructor with a few options and then chain that to the fromString() method:

try {
    const response = await fetch(url);
    if (response.status !== 200) {
      throw new Error('Fetch failed. Received a response of '
                     + response.status);
    }
    const data = await csvtojson({
      checkType: true,
      colParser: { 
        "Date": dt => new Date(dt),
        "Open": "omit",
        "High": "omit",
        "Low": "omit",
        "Volume": "omit"
      }
    }).fromString(await response.text());

    return data;
  } catch (err) {
    // handle errors
  }

There is also a fromStream() method, but I found it easier to handle fetch-related errors by separating the calls.

A bit about the options:

By default, csvtojson sets every value to a string. To override this behaviour, we can set the checkType property to true. Doing so tells csvtojson to attempt to find a proper type parser according to the cell value. That is, if cell value is "5", a numberParser will be used and all values under that column will use the numberParser to transform data. There are built-in parsers for strings and numbers, but not for dates. That's why the "Date" column employs a custom conversion method.
There is one other built-in parser: a value of "omit" omits the whole column.

We can access our transformed data using dot accessor notation, i.e., data.Date or data['Adj Close'].

Changing a Property Name

The "Close" attribute in the API response object is actually the "Adj Close", but I didn't want to call it that, so I changed the attribute name. To transform the result that is sent to downstream, we can use the .subscribe() method for each json object, i.e., for each transformed row.

To do that, I assign the "Adj Close" value to the new attribute and then delete the existing column. This is all done within a Promise due to the asynchronous nature of the transformation process:

const data = await csvtojson({
      checkType:true,
      colParser: { 
        "Date": dt => new Date(dt),
        "Open": "omit",
        "High": "omit",
        "Low": "omit",
        "Volume": "omit"
      }
    }).fromString(await response.text()).subscribe(jsonObj => 
      new Promise(resolve => {
        // use the adj close
        jsonObj.Close = jsonObj['Adj Close'];
        delete jsonObj['Adj Close'];
        resolve();
      })
    );

Conclusion

In this tutorial we learned how to transform Yahoo! Finance's historical price information of a security from CVS to JSON using the Node.js csvtojson library.

You can try the Monte Carlo simulator API on RapidAPI. I also wrote an API to Get the Stock Buy Price to Earn Desired Returns and a Total Credit Card Interest Calculator. All are free to use up to a certain threshold and then require a paid subscription beyond that.

If you have any questions about my APIs or would like to inquire about having a custom one built, feel free to email me at raylsstr(AT)gmail(DOT)com.