DEV Community: Eduardo Marques

Node.Js - Stream from S3 with Partial Content support (206)

Eduardo Marques — Thu, 05 Oct 2023 17:15:39 +0000

In this post I will show you how you can stream videos from a S3 bucket with Partial Content support (HTTP 206).

To give a bit of context, I have a service where users can upload their files, which are stored in a AWS S3 bucket. For security, the files in the bucket can not be accessed directly, only through my API.

When a user requests a file I stream the entire file directly from S3.

This approach is good enough for the majority of use cases when the user simply wants to download a file. But things don't work so well when the user wants to stream the file, and that's usually the case for videos - if the user wants to display the video on their webpage using the HTML5 video tag it will have limited functionality, for example, the user won't be able to use the "seeking" feature of the video player.

To solve that the server needs to support Partial Content requests.

HTTP 206 Partial Content

In the context of video streaming, we don't want to download the entire file at once. We want to be able to tell the server what portion of the video we want. When the server is able to fulfill such a request it will respond with the status code 206.

To request a portion of the file the Range header is used by the client, specifying the content range. The range is specified in bytes.

Example:
Range: bytes=0-10000

Note that the video player does this automatically for us.

In the response, the server will include the Content-Range header to inform the client the range of the content being sent and Content-Length informing the size of the file.

Example:
Content-Range: bytes 0-10000/10001 Content-Length: 10001

Show me the code!

In this example I'm using the version 3 of the AWS SDK and will take advantage of the Range property when retrieving the file from S3.

So here's how the controller looks like!

const { S3 } = require('@aws-sdk/client-s3');
const s3Config = require('...YOUR_S3_CONFIG_FILE...')

// #1
const s3AwsClient = new S3({
    credentials: {
        accessKeyId: s3Config.accessKeyId,
        secretAccessKey: s3Config.secretAccessKey
    },
    region: s3Config.region,
    endpoint: 'https://' + s3Config.endpoint
});

async (req, res, next) => {
  // #2
  const s3Options = {
    Bucket: s3Config.bucket,
    Key: objectKey // key to your object in S3
  };

  // #3
  const { ContentLength } = await s3AwsClient.headObject(s3Options);
  const videoSize = ContentLength;

  // #4
  const FILE_PORTION_SIZE = 5000000; //bytes = 5MB
  const requestedRange = req.headers.range || '';
  const start = Number(requestedRange.replace(/\D/g, ''));
  const end = Math.min(start + FILE_PORTION_SIZE, videoSize - 1);
  const contentLength = end - start + 1;

  // #5
  res.status(206);
  res.setHeader('Cache-Control', 'no-cache');
  res.setHeader('Accept-Ranges', 'bytes');
  res.setHeader('Content-Range', `bytes ${start}-${end}/${videoSize}`);
  res.setHeader('Content-Length', contentLength);

  // #6
  const streamRange = `bytes=${start}-${end}`;
  const s3Object = await s3AwsClient.getObject({
    ...s3Options,
    Range: streamRange
  });

  // #7  
  s3Object.Body.pipe(res);
}

Let's analyse what's happening in the code above:

#1
Create the S3 client. If the code above doesn't work you can find more information here.

#2
s3Options contain the bucket name and the key of the file you want to access.

#3
The headObject function allows us to get information about the file. In our case we are interested in the ContentLength of the file.

#4
In this step we calculate the ranges of the file portion we want to return. After this we will have start, end and contentLength. Note that in the FILE_PORTION_SIZE variable we define the size we want to return. You can change this to your needs.

#5
Since we want to support Partial content requests, we need to set the correct response code (206), Content-Range and Content-Length headers.

#6
Now it's time to get the object from S3! For that we use getObject function. Note that we are passing the range here, if you forget this you will get the entire file.

#7
The property Body of the object returned by getObject is a stream, so we will pipe it to the response (res) object.

And this is all you need!

Using the video stream

  <video controls>
    <source
      src="url-to-your-video.mp4"
      type="video/mp4"
    />
  </video>

The HTML5 video tag does all the work for you. Once it recognises that the server supports Partial Content, it will provide the correct Range headers and make the sequential requests to get the video.

BONUS! Are you using CloudFlare?

After implementing this everything worked perfectly on my local machine, but once it was deployed to the server it wouldn't work. The video would unexpectedly stop and the video player would reload randomly.

All the requests to my server are proxied through CloudFlare, and I was suspicious that the problem could be here.

And I was right, after some googling I discovered a thread where this issue was discussed. In summary, CloudFlare ignores the Range headers in the requests to cache them. As we explained above, the server expects this header to be able to calculate the range of the next portion of the video.

To go around this issue you need to create a page exception in CloudFlare to not cache the request. To do this go to page rules and add an exception. This is what I added:

URL: *app.rested.dev*.mp4*
Cache Level: Bypass

This has the downside that the requests won't be cached, but you can stream your videos without issues!

Conclusion

I hope this will help you with the implementation of Partial Content support in your. Keep in mind that this is not the most sophisticated implementation, but is good enough to get you started!

If you are curious about my service you can visit rested.dev.

And that's all for today! Thank you for reading!

Working with JSON in JS, the easy way! JSONata

Eduardo Marques — Wed, 10 Feb 2021 21:02:06 +0000

Data structures are one of the most important elements in any programming language. It allows us to structure the data in a consistent way and to perform operations that require some patterns. Without a pattern it is hard for any developer to figure out an efficient way to extract or manipulate the data.

As an example, JSON is a very well known format that most of us uses every day.
When working with it, there are operations we do regularly, such as querying, filtering, sorting, grouping and others.

Recently, while working on a project, I needed two things, proper functions for all of these operations, and some sort of expressions for external users. I quickly realised that this wasn’t a small task.

We, as Software Developers, always get excited with these tasks, we can get creative and explore a lot, but we should not try to reinvent the wheel. I recently read a great book called Getting real, it really helped me focus on my goals and, in this particular case, quickly realise that before trying to implement this I should look for existent solutions.

And that’s when I came across JSONata, a really powerful library that did everything what I was looking for and more!

In this article I would like to give you a quick introduction to JSONata and show how easy it is to work with JSON.

Usage

const jsonata = require('jsonata');

const data = [
  {
     field: 'some-field',
     createdAt: '2021-01-03T21:58:31.000Z'
  }
];

const expression = jsonata('$[createdAt="2021-01-03T21:58:31.000Z"]');
const result = expression.evaluate(data);

Expressions for your day to day needs

JSONata helps us solve the most common cases when it comes to extracting data from JSON. In this article I cover some basic expressions that will help you get started with the library.

We will use the following JSON to test our expressions.

[
  {
    "uniqueId": "9157c740159e4ff18b4a42d4a0c39622",
    "data": {
      "name": "Eduardo",
      "phone": 999888727,
      "age": 30
    },
    "createdAt": "2021-02-03T21:54:37.000Z",
    "updatedAt": "2021-02-03T21:54:37.000Z"
  },
  {
    "uniqueId": "66a1212c55ce47f9bc055e6c331b7af3",
    "data": {
      "name": "Miguel",
      "age": 17,
      "phone": 999888777
    },
    "createdAt": "2021-02-03T21:56:54.000Z",
    "updatedAt": "2021-02-03T21:56:54.000Z"
  },
  {
    "uniqueId": "86011b52ebc640aeba53a6e09a9d9773",
    "data": {
      "name": "Marco",
      "age": 18
    },
    "createdAt": "2021-02-04T21:57:59.000Z",
    "updatedAt": "2021-02-04T21:57:59.000Z"
  },
  {
    "uniqueId": "0f6586aa7ccf4901a3f066eeeea69116",
    "data": {
      "name": "Sergio",
      "age": 28,
      "jobDetails": {
        "company": "SpaceY"
      }
    },
    "createdAt": "2021-02-07T22:02:50.000Z",
    "updatedAt": "2021-02-07T22:02:50.000Z"
  }
]

You can try out these expressions at https://try.jsonata.org.

Simple queries

data.name

Gets an array of names.

Result

[
  "Eduardo",
  "Miguel",
  "Marco",
  "Sergio"
]

$[1]

Gets the second element.

Result

{
  "uniqueId": "66a1212c55ce47f9bc055e6c331b7af3",
  "data": {
    "name": "Miguel",
    "age": 17,
    "phone": 999888777
  },
  "createdAt": "2021-02-03T21:56:54.000Z",
  "updatedAt": "2021-02-03T21:56:54.000Z"
}

$[0]data.name

Gets the name of the first element.

Result

"Eduardo"

Filtering

$[data.name="Eduardo"]

Filters records in the array containing the path data.name and matching Eduardo.

Result

{
  "uniqueId": "9157c740159e4ff18b4a42d4a0c39622",
  "data": {
    "name": "Eduardo",
    "phone": 999888727,
    "age": 30
  },
  "createdAt": "2021-02-03T21:54:37.000Z",
  "updatedAt": "2021-02-03T21:54:37.000Z"
}

$[data.jobDetails]

Filters records in the array containing the path data.jobDetails.

Result

{
  "uniqueId": "0f6586aa7ccf4901a3f066eeeea69116",
  "data": {
    "name": "Sergio",
    "age": 28,
    "jobDetails": {
      "company": "SpaceY"
    }
  },
  "createdAt": "2021-02-07T22:02:50.000Z",
  "updatedAt": "2021-02-07T22:02:50.000Z"
}

$[data.age > 16 and data.age < 20]

Filters records in the array with data.age between 16 and 20.

Result

[
  {
    "uniqueId": "66a1212c55ce47f9bc055e6c331b7af3",
    "data": {
      "name": "Miguel",
      "age": 17,
      "phone": 999888777
    },
    "createdAt": "2021-02-03T21:56:54.000Z",
    "updatedAt": "2021-02-03T21:56:54.000Z"
  },
  {
    "uniqueId": "86011b52ebc640aeba53a6e09a9d9773",
    "data": {
      "name": "Marco",
      "age": 18
    },
    "createdAt": "2021-02-04T21:57:59.000Z",
    "updatedAt": "2021-02-04T21:57:59.000Z"
  }
]

Pagination

$[[0..1]]

Gets the first 2 elements.

Result

[
  {
    "uniqueId": "9157c740159e4ff18b4a42d4a0c39622",
    "data": {
      "name": "Eduardo",
      "phone": 999888727,
      "age": 30
    },
    "createdAt": "2021-02-03T21:54:37.000Z",
    "updatedAt": "2021-02-03T21:54:37.000Z"
  },
  {
    "uniqueId": "66a1212c55ce47f9bc055e6c331b7af3",
    "data": {
      "name": "Miguel",
      "age": 17,
      "phone": 999888777
    },
    "createdAt": "2021-02-03T21:56:54.000Z",
    "updatedAt": "2021-02-03T21:56:54.000Z"
  }
]

$[[2..4]]

Gets the second 2 elements.

Result

[
  {
    "uniqueId": "86011b52ebc640aeba53a6e09a9d9773",
    "data": {
      "name": "Marco",
      "age": 18
    },
    "createdAt": "2021-02-04T21:57:59.000Z",
    "updatedAt": "2021-02-04T21:57:59.000Z"
  },
  {
    "uniqueId": "0f6586aa7ccf4901a3f066eeeea69116",
    "data": {
      "name": "Sergio",
      "age": 28,
      "jobDetails": {
        "company": "SpaceY"
      }
    },
    "createdAt": "2021-02-07T22:02:50.000Z",
    "updatedAt": "2021-02-07T22:02:50.000Z"
  }
]

Sorting

$[] ~> $sort(function($a,$b){ $a.data.age > $b.data.age })

Sorts the results by ascending age. In this expression we use the $sort function to sort the results.

Result

[
  {
    "uniqueId": "66a1212c55ce47f9bc055e6c331b7af3",
    "data": {
      "name": "Miguel",
      "age": 17,
      "phone": 999888777
    },
    "createdAt": "2021-02-03T21:56:54.000Z",
    "updatedAt": "2021-02-03T21:56:54.000Z"
  },
  {
    "uniqueId": "86011b52ebc640aeba53a6e09a9d9773",
    "data": {
      "name": "Marco",
      "age": 18
    },
    "createdAt": "2021-02-04T21:57:59.000Z",
    "updatedAt": "2021-02-04T21:57:59.000Z"
  },
  {
    "uniqueId": "0f6586aa7ccf4901a3f066eeeea69116",
    "data": {
      "name": "Sergio",
      "age": 28,
      "jobDetails": {
        "company": "SpaceY"
      }
    },
    "createdAt": "2021-02-07T22:02:50.000Z",
    "updatedAt": "2021-02-07T22:02:50.000Z"
  },
  {
    "uniqueId": "9157c740159e4ff18b4a42d4a0c39622",
    "data": {
      "name": "Eduardo",
      "phone": 999888727,
      "age": 30
    },
    "createdAt": "2021-02-03T21:54:37.000Z",
    "updatedAt": "2021-02-03T21:54:37.000Z"
  }
]

Working with dates

Combining $filter and $toMillis

$[] ~> $filter(function($value){
    $toMillis($value.createdAt)>$toMillis("2021-02-07T22:00:00.000Z")
})

$toMillis converts a ISO 8601 date to milliseconds. In this example we filter all the elements created after 2021-02-07 22:00:00.

Result

{
  "uniqueId": "0f6586aa7ccf4901a3f066eeeea69116",
  "data": {
    "name": "Sergio",
    "age": 28,
    "jobDetails": {
      "company": "SpaceY"
    }
  },
  "createdAt": "2021-02-07T22:02:50.000Z",
  "updatedAt": "2021-02-07T22:02:50.000Z"
}

And this is all for now. I hope you find this article useful, and don't forget to give JSONata a go!

PS: In case you are wondering, I’m not the author and I'm not part of the team, I just liked it so much that I thought it would be worth sharing with others! All the Kudos goes to the JSONata team!

Terms and conditions

Eduardo Marques — Tue, 20 Oct 2020 06:57:11 +0000

Hi, is there a platform or service I can use to generate Terms and conditions for my project?

I've seen recently some sites without terms & conditions, and made me wonder, is it okay not to have them?

Could you help clarify this?