DEV Community

Streaming files from AWS S3 using NodeJS Stream API with Typescript

about14sheep on January 02, 2022

Checkout the new npm package! This simple solution just got easier! I also added some upgrades like the ability to adjusted the size of ...

Read full post

Yegair • Jun 30 '22

I had some trouble getting this to work with AWS SDK v3 (@aws-sdk/s3-client). To be precise, I always got an error in this line:

// We push the data into the stream buffer
this.push(data.Body);

The error message told me, that data.Body is of type http.IncomingMessage which cannot be used as an argument for push.

After some trial and error I found a solution which works for me, maybe this helps someone who is facing a similar problem.

import Stream from 'stream';
import { S3 } from '@aws-sdk/client-s3';

type S3DownloadStreamOptions = {
  readonly s3: S3;
  readonly bucket: string;
  readonly key: string;
  readonly rangeSize?: number;
};

const DEFAULT_DOWNLOAD_CHUNK_SIZE = 512 * 1024;

class S3DownloadStream extends Stream.Transform {
  private options: S3DownloadStreamOptions;
  private _currentCursorPosition = 0;
  private _maxContentLength = -1;

  constructor(options: S3DownloadStreamOptions, nodeReadableStreamOptions?: Stream.ReadableOptions) {
    super(nodeReadableStreamOptions);
    this.options = options;
    this.init();
  }

  async init() {
    const res = await this.options.s3.headObject({ Bucket: this.options.bucket, Key: this.options.key });
    this._maxContentLength = res.ContentLength;
    await this.fetchAndEmitNextRange();
  }

  async fetchAndEmitNextRange() {
    if (this._currentCursorPosition > this._maxContentLength) {
      this.end();
      return;
    }

    // Calculate the range of bytes we want to grab
    const range = this._currentCursorPosition + (this.options.rangeSize ?? DEFAULT_DOWNLOAD_CHUNK_SIZE);

    // If the range is greater than the total number of bytes in the file
    // We adjust the range to grab the remaining bytes of data
    const adjustedRange = range < this._maxContentLength ? range : this._maxContentLength;

    // Set the Range property on our s3 stream parameters
    const rangeParam = `bytes=${this._currentCursorPosition}-${adjustedRange}`;

    // Update the current range beginning for the next go
    this._currentCursorPosition = adjustedRange + 1;

    // Grab the range of bytes from the file
    this.options.s3.getObject({ Bucket: this.options.bucket, Key: this.options.key, Range: rangeParam }, (error, res) => {
      if (error) {
        // If we encounter an error grabbing the bytes
        // We destroy the stream, NodeJS ReadableStream will emit the 'error' event
        this.destroy(error);
        return;
      }

      console.log(`fetched range ${this.options.bucket}/${this.options.key} | ${rangeParam}`);

      const data = res.Body;

      if (!(data instanceof Stream.Readable)) {
        // never encountered this error, but you never know
        this.destroy(new Error(`unsupported data representation: ${data}`));
        return;
      }

      data.pipe(this, { end: false });

      let streamClosed = false;

      data.on('end', async () => {
        if (streamClosed) {
          return;
        }
        streamClosed = true;
        await this.fetchAndEmitNextRange();
      });
    });
  }

  _transform(chunk, _, callback) {
    callback(null, chunk);
  }
}

Note that I moved the s3.headObject call into the S3DownloadStream class, so the usage of this class is a little different from the article:

const s3 = new S3({ /* ... */ });
const bucket = 'my-fancy-s3-bucket';
const key = 'some/file/in/my/bucket.csv';
const downloadStream = new S3DownloadStream({ s3, bucket, key });

about14sheep • Sep 3 '22

Nice! Yeah a major version change would bring in some breaking changes.

srikanth597 • Jan 8 '22

I couldn't understand the calculations entirely can u please make me understand?

Since 64kb is _s3DataRange, S3 file size is let's say 128kb, then u will fetch first 64kb
But when modifying the S3StreamParams Range its caculated as bytes = -64kb ( in minus)
That's bit strange for me understand.

My second question is on next iteration it will start from 65th KB. Where exactly this iteration begins? There's no loop here that instructs it to keep repeating till completion of last bit In 128kb file.

about14sheep • Jan 8 '22 • Edited

For your first question, the Range parameter isn't a calculation. So the (-) hyphen there can be read as, 'grab a range of bytes starting at byte 65(up too)128'. So it wouldn't be negative.

Your second questions is really good. This is where the simplicity of NodeJS Readable Stream API comes in. When this stream is in the data flowing mode, it will call the _read() method whenever there is room in the buffer (and the stream is not paused). So you don't need to write in a loop, the super class Readable handles all this for you!

Hope this helps! If you have any other questions please let me know!

Arpitkumar • Sep 2 '22

How can we handle seeking of the video?

  <video
        onContextMenu={(e) => e.preventDefault()}
        id="videoPlayer"
        width="650"
        height="auto"
        controls
        <source src={`/api/videos?videoId=${id}`} type="video/mp4" />
      </video>

I have created a simple video element but I am not able to seek video. Is there anything we have to do in S3DownloadStream class to make it work?

I am using Next.js for UI.

@about14sheep

about14sheep • Sep 3 '22

That would need to be handled on the frontend. Once the S3DownloadStream grabs a range, it just pushes it through to the output. Seeking would be handled by the video player

Sandip Basnet • Mar 8 '22

Awesome!

Horacio Rivero • Jan 2 '22

Thanks for sharing, what would be the best way to send the transfer progress percentage to the browser client?

about14sheep • Jan 2 '22

Thank you for the question!

Since the SmartStream class is just like any NodeJS readStream, you can throw a 'data' event handler on it.

You can then store the data.ContentLength value returned from the s3.headObject call and subtract the chunk length returned from the 'data' event. You can use this value to determine how much data is left, which you can then send to the frontend.

The neat thing about NodeJS streams is all of this can be done without editing the SmartStream class!

Thanks for reading and let me know if I can help any more!

Horacio Rivero • Jan 2 '22

sure, but how would you send this information to a client browser, using websockets?, I'm trying to use fetch streams for its simplicity but they don't work very well for me.

fetch-upload-streaming

about14sheep • Jan 2 '22

You can pipe this stream into the response from a http request. I found this article in a quick search that might help. Anywhere you see a fs.createReadStream you can substitute in this readStream!

There are numerous ways, including websockets, that this can be done. I would have to put in more research. If this article gets enough traction I could do a part 2 where I send the data to a frontend.

about14sheep • Jan 6 '22

I ended up making an extremely simple front end, not sure if it will help in your case, but it might be a good starting point. You can fork it from my github if you like.

github.com/about14sheep/awsstreaming