DEV Community: Daniel Cruz

Content Delivery with Node.js Streams

Daniel Cruz — Tue, 27 Jun 2023 18:42:35 +0000

The problem

In a project I'm currently working on, I have an API endpoint that retrieves the content of a file. However, this endpoint requires a private token for authentication, making it unsuitable to call directly from the browser due to the token exposure risk.

Fortunately, I'm using Next.js, which allows me to leverage API endpoints to safely make the call on the backend and then access the endpoint from the front end.

However, the external endpoint can return content of any type, making it challenging to parse the response and send it to the browser. One possible but undesirable solution would be manually checking the Content-Type header and using different parsing methods based on its value. A sample of that implementation would look like

// ...
const response = fetch("https://example.com/api/v1/content/" + file);

const contentType = response.headers.get("content-type");

if (contentType.includes("json")) {
  const parsedResponse = await response.json();
  return res.send(parsedResponse);
}

if (contentType.includes("text")) {
  const parsedResponse = await response.text();
  return res.send(parsedResponse);
}

// etc

This approach is brittle and relies on imperative programming.

Furthermore, it involves having the entire response in memory before sending it to the client, which can lead to memory overhead, slow response times, and even application crashes when handling large files.

A better approach is to forward the Content-Type header from the external API endpoint and let the browser handle the response's content based on that. This approach means that I need to pass the response body as-is, without parsing it on the backend. Fortunately, Node.js streams provide an elegant solution to this problem.

Implementation of the solution

To address this challenge, I have implemented a solution using Node.js streams and Next.js API endpoints. Here's the code, and below it, a step-by-step explanation:

// api/content/[filename].ts
import { fetchWithAuth } from "./fetch-with-auth";
import { NextApiRequest, NextApiResponse } from "next";
import { pipeline } from "stream/promises";
import { Readable } from "stream";

export default async function handler(
  req: NextApiRequest,
  res: NextApiResponse
) {
  if (req.method === "GET") {
    const { filename: _filename = "" } = req.query;
    const file = Array.isArray(_filename) ? _filename[0] : _filename;

    const response = await fetchWithAuth(
      "https://example.com/api/v1/content/" + file,
      {
        headers: {
          Accept: "*/*",
        },
      }
    );

    const contentType = response.headers.get("content-type");

    if (!contentType) {
      console.error("Missing content type");
      return res.status(500).end();
    }

    const readable = response.body;

    if (!readable) {
      console.error("Missing body from API");
      return res.status(500).end();
    }

    const nodeReadable = Readable.fromWeb(readable);

    res.setHeader("Content-Type", contentType);

    await pipeline(nodeReadable, res);

    res.end();
  }

  return res.status(405);
}

Here's how the solution works:

The code checks if the request method is GET; otherwise, I return 405 status (Method Not Allowed).
Extract the filename from the request query, allowing dynamic content retrieval based on the requested filename.
Since I'm using Node.js 18, I have access to the fetch function, which I use in a custom fetchWithAuth that abstracts away my auth logic, my custom implementation has the same function signature as fetch.
The code retrieves the Content-Type header from the API response. If the content type is missing, an error is logged, and a 500 status (Internal Server Error) is returned to the client. This approach ensures consistency as the API should always include the header, even though modern browsers perform content sniffing, and I could still send the response to it.
response.body is a readable stream. If the body is missing, an error is logged, and a 500 status (Internal Server Error) is returned since that means there's nothing to stream back to the client.
According to the spec of fetch implemented in undici, despite it being used in a Node.js environment the response.body is a web readable stream so I have to convert it into a Node.js stream using Readable.fromWeb()
I forward the Content-Type header for the response using res.setHeader.
The pipeline function is used to stream the content from the readable stream to the response.
Finally, when the pipeline finishes, I end the response with res.end().

With this approach, I can securely utilize the external API and confidently include my own endpoint as the source for any HTML tag (<img>, <video>, <iframe>, etc.), without concerns about passing incorrect content.

Using Intl.NumberFormat to transform number notation

Daniel Cruz — Wed, 03 May 2023 02:04:19 +0000

Problem

You want to transform thousand numbers into their compact version (1,000 -> 1k).

Solution

Use Intl.NumberFormat with the compact notation as option

const formatter = new Intl.NumberFormat("en-US", { notation: "compact" });

const number = 12_000;

const formattedNumber = formatter.format(number) // -> will return '12k'

MDN Reference

How to re-sync auto-increment in PostgreSQL

Daniel Cruz — Mon, 17 Oct 2022 21:11:05 +0000

First, to check if your auto-increment is out of sync, run the following:

select max(id) from table;

select nextval('table_id_seq')

If the result from nextval is not greater than the result of your max(id), your auto-increment is out of sync.

table_id_seq is the name of the sequence, which is composed of table name _ column with auto-increment _ seq. If you have a table purchases where the id is the auto-increment column, then the sequence name would be purchases_id_seq.

If you have your table inside a schema, following the previous example, having your table inside of ecommerce schema, the name would be ecommerce."purchases_id_seq".

You can check all your sequences with the following statement :

select * from information_schema.sequences

Here's the statement to fix it, using our example names:

select setval('ecommerce."purchases_id_seq"', coalesce(
(select max(id)+1 from ecommerce.purchases), 1), false)

Hope this was helpful!

How to find the delta between rows in PostgreSQL

Daniel Cruz — Mon, 17 Oct 2022 20:50:38 +0000

Imagine we have the following data:

+--------------------------+----------+
|       "timestamp"        | "price"  |
+--------------------------+----------+
| "2021-10-14 21:40:00+00" | 57681.01 |
| "2021-10-14 21:35:00+00" | 57698.41 |
| "2021-10-14 21:30:00+00" | 57575.77 |
| "2021-10-14 21:25:00+00" | 57573.92 |
+--------------------------+----------+

And we want to calculate the delta between prices, to see how much the current price increased or decreased relative the the previous one. For that we can use lead().

Having the following SQL query:

SELECT
    timestamp,
    price,
    lead(price) OVER (ORDER BY timestamp DESC) AS previous_price
FROM
    prices;

We get this data:

+--------------------------+----------+------------------+
|       "timestamp"        | "price"  | "previous_price" |
+--------------------------+----------+------------------+
| "2021-10-14 21:40:00+00" | 57681.01 |         57698.41 |
| "2021-10-14 21:35:00+00" | 57698.41 |         57575.77 |
| "2021-10-14 21:30:00+00" | 57575.77 |         57573.92 |
| "2021-10-14 21:25:00+00" | 57573.92 |         NULL     |
+--------------------------+----------+------------------+

Dissecting a little bit the lead part we can see that we have to use the OVER keyword and tell it how to order the data, and over that data we will get the lead price field.

Now to get the relative change we need to apply the following formula:

(b - a) / a

Where b is the current value and a the previous one.

So we will end with the following query; note we are already multiplying by 100 to get it as a percentage.

SELECT
    timestamp,
    price,
    (( price - lead(price) OVER (ORDER BY timestamp DESC)) / lead(price) OVER (ORDER BY timestamp DESC)) * 100 as percentage_change
FROM
    prices;

Giving us the data we want:

+--------------------------+----------+--------------------------+
|       "timestamp"        | "price"  |   "percentage_change"    |
+--------------------------+----------+--------------------------+
| "2021-10-14 21:40:00+00" | 57681.01 | -0.030156810213661       |
| "2021-10-14 21:35:00+00" | 57698.41 | 0.213006269824962827     |
| "2021-10-14 21:30:00+00" | 57575.77 | 0.0032132604484808399359 |
| "2021-10-14 21:25:00+00" | 57573.92 | NULL                     |
+--------------------------+----------+--------------------------+

Hope it was useful!

P.S. Here is a more detailed explanation of the lead function.