DEV Community: Sebastien Filion

Handling Binary Data — Building a HTTP Server from scratch

Sebastien Filion — Wed, 14 Jul 2021 12:50:28 +0000

On the last post of BTS: HTTP Server series.
I wrote a barebone HTTP server that can handle requests and respond appropriately.
I think I covered the basics, but that server is limited in what it can do.
It can only handle text-based Requests and Responses... That means no image or other media exchange.
And then, if the Request or the Response is larger than a KB, I'm out of luck. Again, not great for media...

This article is a transcript of a Youtube video I made.

Oh, hey there...

That's my challenge for today, refactor my server to handle arbitrarily sized Requests and avoid treating everything as
text...

If I want to be able to handle large requests, the first thing I can do is to read the stream in chunks, 1KB at a time
until there's nothing left to read.
Once I have all of my chunks, I can concatenate them together into one Typed Array. And voila, arbitrarly sized Request!

const concat = (...chunks) => {
  const zs = new Uint8Array(chunks.reduce((z, ys) => z + ys.byteLength, 0));
  chunks.reduce((i, xs) => zs.set(xs, i) || i + xs.byteLength, 0);
  return zs;
};

const chunks = [];
let n;
do {
    const xs = new Uint8Array(1024);
    n = await r.read(xs);
    chunks.push(xs.subarray(0, n));
} while (n === 1024);

const request = concat(...chunks);

The second challenge is to figure out how much of the data stream is the Request line and the Headers versus the body...
I want to avoid reading too far into the body, since it might be binary data.
I know that the body starts after the first empty line of the Request.
So I could technically, search for the first empty line and then I'll know that the rest is the body and only parse the first part.

So I wrote this function that will try to find a sequence within the array. First tries to find the first occurence of
a byte, and then I can just test the following bytes until I have a match.
In our case, I want to find a two CRLF sequences. So I try to find the first CR, then check if it is followed by LF, CR
and LF... And, I repeat this until I find the empty line.

export const findIndexOfSequence = (xs, ys) => {
  let i = xs.indexOf(ys[0]);
  let z = false;

  while (i >= 0 && i < xs.byteLength) {
    let j = 0;
    while (j < ys.byteLength) {
      if (xs[j + i] !== ys[j]) break;
      j++;
    }
    if (j === ys.byteLength) {
      z = true;
      break;
    }
    i++;
  }

  return z ? i : null;
};

🐙 You will find the code for this post here: https://github.com/i-y-land/HTTP/tree/episode/03

The problem with this approach is that I have to traverse the whole request, and it might end up that the request doesn't
have a body, and therefore I wasted my time.

Instead, I will read the bytes one line at a time, finding the nearest CRLF and parse them in order.
On the first line, I will extract the method and the path.
Whenever I find an empty line, I will assume the is body is next and stop.
For the remaining lines, I will parse them as header.

// https://github.com/i-y-land/HTTP/blob/episode/03/library/utilities.js#L208
export const readLine = (xs) => xs.subarray(0, xs.indexOf(LF) + 1);

export const decodeRequest = (xs) => {
  const headers = {};
  let body, method, path;
  const n = xs.byteLength;
  let i = 0;
  let seekedPassedHeader = false;
  while (i < n) {
    if (seekedPassedHeader) {
      body = xs.subarray(i, n);
      i = n;
      continue;
    }

    const ys = readLine(xs.subarray(i, n));

    if (i === 0) {
      if (!findIndexOfSequence(ys, encode(" HTTP/"))) break;
      [method, path] = decode(ys).split(" ");
    } else if (
      ys.byteLength === 2 &&
      ys[0] === CR &&
      ys[1] === LF &&
      xs[i] === CR &&
      xs[i + 1] === LF
    ) {
      seekedPassedHeader = true;
    } else if (ys.byteLength === 0) break;
    else {
      const [key, value] = decode(
        ys.subarray(0, ys.indexOf(CR) || ys.indexOf(LF)),
      ).split(/(?<=^[A-Za-z-]+)\s*:\s*/);
      headers[key.toLowerCase()] = value;
    }

    i += ys.byteLength;
  }

  return { body, headers, method, path };
};

On the other hand, the function to encode the Response is absurdly simpler, I can pretty much use the function I already made
and just encode the result. The biggest difference, is that I have to be aware that the body might not
be text and should be kept as a Typed Array. I can encode the header and then concat the result with the body.

// https://github.com/i-y-land/HTTP/blob/episode/03/library/utilities.js#L248
export const stringifyHeaders = (headers = {}) =>
  Object.entries(headers)
    .reduce(
      (hs, [key, value]) => `${hs}\r\n${normalizeHeaderKey(key)}: ${value}`,
      "",
    );

export const encodeResponse = (response) =>
  concat(
    encode(
      `HTTP/1.1 ${statusCodes[response.statusCode]}${
        stringifyHeaders(response.headers)
      }\r\n\r\n`,
    ),
    response.body || new Uint8Array(0),
  );

From there, I have enough to write a simple server using the serve function I've implemented previously.
I can decode the request... then encode the response.

...
serve(
  Deno.listen({ port }),
  (xs) => {
    const request = decodeRequest(xs);

    if (request.method === "GET" && request.path === "/") {
      return encodeResponse({ statusCode: 204 })
    }
  }
).catch((e) => console.error(e));

I could respond to every requests with a file. That is a good start to a static file server.

    ...
    if (request.method === "GET" && request.path === "/") {
      const file = Deno.readFile(`${Deno.cwd()}/image.png`); // read the file
      return encodeResponse({
        body: file,
        headers: {
          "content-length": file.byteLength,
          "content-type": "image/png"
        },
        statusCode: 200
      });
    }

I can start my server and open a browser to visualize the image.

With a bit more effort, I can serve any file withing a given directory.
I would attempt to access the file and cross-reference the MIME type from a currated list using the extension.
If the system can't find the file, I will return 404 Not Found.

const sourcePath =
    (await Deno.permissions.query({ name: "env", variable: "SOURCE_PATH" }))
            .state === "granted" && Deno.env.get("SOURCE_PATH") ||
    `${Deno.cwd()}/library/assets_test`;
...
    if (request.method === "GET") {
      try {
        const file = await Deno.readFile(sourcePath + request.path); // read the file
        return encodeResponse({
          body: file,
          headers: {
            "content-length": file.byteLength,
            ["content-type"]: mimeTypes[
              request.path.match(/(?<extension>\.[a-z0-9]+$)/)?.groups?.extension
                .toLowerCase()
              ].join(",") || "plain/text",
          },
          statusCode: 200
        });
      } catch (e) {
        if (e instanceof Deno.errors.NotFound) { // if the file is not found
          return encodeResponse({
            body: new Uint8Array(0),
            headers: {
              ["Content-Length"]: 0,
            },
            statusCode: 404,
          });
        }

        throw e;
      }
    }

With a broadly similar approach, I can receive any file.

const targetPath =
    (await Deno.permissions.query({ name: "env", variable: "TARGET_PATH" }))
            .state === "granted" && Deno.env.get("TARGET_PATH") ||
    `${Deno.cwd()}/`;
...
    if (request.method === "GET") { ... }
    else if (request.method === "POST") {
      await Deno.writeFile(targetPath + request.path, request.body); // write the file
      return encodeResponse({ statusCode: 204 });
    }

Now, you can guess if you look at the position of your scrollbar that things can't be that simple...

I see two problems with my current approach.
I have to load whole files into memory before I can offload it to the File System which that can become a bottle neck at
scale.
Another surprising issue is with file uploads...
When uploading a file, some clients, for example curl will make the request in two steps... The first request is
testing the terrain stating that it wants to upload a file of a certain type and length and requires that the server
replies with 100 continue before sending the file.
Because of this behaviour I need to retain access to the connection, the writable resource.
So I think I will have to refactor the serve function from accepting a function that takes a Typed Array as an
argument, to a function that takes the connection.
This could also be positive change that would facilitate implementing powerful middleware later on...

export const serve = async (listener, f) => {
  for await (const connection of listener) {
    await f(connection);
  }
};

There's two ways that my server can handle file uploads.
One possibility is that the client tries to to post the file directly,
I have the option to read the header and refuse the request if it's too large. The other possibility is that the
client expects me to reply first.
In both case I will read the first chunk and then start creating the file with the data processed. Then I want to
to read one chunk at a time from the connection and systematically write them to the file. This way, I never hold
more than 1KB in memory at a time... I do this until I can't read a whole 1KB, this tells me that the file has been
completely copied over.

export const copy = async (r, w) => {
  const xs = new Uint8Array(1024);
  let n;
  let i = 0;
  do {
    n = await r.read(xs);
    await w.write(xs.subarray(0, n));
    i += n;
  } while (n === 1024);

  return i;
};
...
    let xs = new Uint8Array(1024);
    const n = await Deno.read(r.rid, xs);
    const request = xs.subarray(0, n);
    const { fileName } = request.path.match(
      /.*?\/(?<fileName>(?:[^%]|%[0-9A-Fa-f]{2})+\.[A-Za-z0-9]+?)$/,
    )?.groups || {};

    ...

    const file = await Deno.open(`${targetPath}/${fileName}`, {
      create: true,
      write: true,
    });

    if (request.headers.expect === "100-continue") {
      // write the `100 Continue` response
      await Deno.write(connection.rid, encodeResponse({ statusCode: 100 }));

      const ys = new Uint8Array(1024);
      const n = await Deno.read(connection.rid, ys); // read the follow-up
      xs = ys.subarray(0, n);
    }

    const i = findIndexOfSequence(xs, CRLF); // find the beginning of the body

    if (i > 0) {
      await Deno.write(file.rid, xs.subarray(i + 4)); // write possible file chunk
      if (xs.byteLength === 1024) {
        await copy(connection, file); // copy subsequent chunks
      }
    }

    await connection.write(
      encodeResponse({ statusCode: 204 }), // terminate the exchange
    );
...

From there, I can rework the part that responds with a file.
Similarly to the two-step request for receiving a file, a client may opt to request the headers for a given file
with the HEAD method.
Because I want to support this feature, I can first get information from the requested file, then I can start writing
the headers and only if the request's method is GET -- not HEAD -- I will copy the file to the connection.

...
    try {
      const { size } = await Deno.stat(`${sourcePath}/${fileName}`);

      await connection.write(
        encodeResponse({
          headers: {
            ["Content-Type"]: mimeTypes[
              fileName.match(/(?<extension>\.[a-z0-9]+$)/)?.groups?.extension
                .toLowerCase()
              ].join(",") || "plain/text",
            ["Content-Length"]: size,
          },
          statusCode: 200,
        }),
      );

      if (request.method === "GET") {
        const file = await Deno.open(`${sourcePath}/${fileName}`);
        await copy(file, connection);
      }
    } catch (e) {
      if (e instanceof Deno.errors.NotFound) {
        Deno.write(
          connection.rid,
          encodeResponse({
            headers: {
              ["Content-Length"]: 0,
            },
            statusCode: 404,
          }),
        );
      }

      throw e;
    }
...

Wow. At this point I have to be either very confident with my programming skills or sadistic...
I need to implement a slew of integrations tests before going any further.
I created four static files for this purpose, a short text file, less than a KB, a longer text file, an image and
music...
For that purpose, I wrote a higher-order-function that will initialize the server before calling the test function.

// https://github.com/i-y-land/HTTP/blob/episode/03/library/integration_test.js#L6
const withServer = (port, f) =>
  async () => {
    const p = await Deno.run({ // initialize the server
      cmd: [
        "deno",
        "run",
        "--allow-all",
        `${Deno.cwd()}/cli.js`,
        String(port),
      ],
      env: { LOG_LEVEL: "ERROR", "NO_COLOR": "1" },
      stdout: "null",
    });

    await new Promise((resolve) => setTimeout(resolve, 1000)); // wait to be sure

    try {
      await f(p); // call the test function passing the process
    } finally {
      Deno.close(p.rid);
    }
  };

With that, I generate a bunch of tests to download and upload files; this ensures that my code is working as expected.

// https://github.com/i-y-land/HTTP/blob/episode/03/library/integration_test.js#L58
[...]
  .forEach(
    ({ headers = {}, method = "GET", path, title, f }) => {
      Deno.test(
        `Integration: ${title}`,
        withServer(
          8080,
          async () => {
            const response = await fetch(`http://localhost:8080${path}`, {
              headers,
              method,
            });
            await f(response);
          },
        ),
      );
    },
  );

When I got to that point, I realized that my serve function was starting to be very... long.
I knew I needed to refactor it into two functions receiveStaticFile and sendStaticFile.
But, because I need to be able to check the Request line to route to the right function, and I can only read the request
once...
I knew that I was in trouble.

I need something that can keep part of the data in memory while retaining access to the raw connection...

...
    if (method === "POST") {
      return receiveStaticFile(?, { targetPath });
    } else if (method === "GET" || method === "HEAD") {
      return sendStaticFile(?, { sourcePath });
    }
...

I could have decoded the request and shove the connection in there and call it a day...
But it didn't feel right aaaand I guess I love making my life harder.

const request = decodeRequest(connection);
request.connection = connection;

...
    if (method === "POST") {
      return receiveStaticFile(request, { targetPath });
    } else if (method === "GET" || method === "HEAD") {
      return sendStaticFile(request, { sourcePath });
    }
...

The solution I came up with was to write a buffer. It would hold in memory only a KB at a time, shifting the bytes
each time I read a new chunk. The advantage of that is I can move the cursor back to the beginning of the buffer
and read-back parts that I need.
Best of all, the buffer has the same methods as the connection; so the two could be used interchangeably.
I won't go into the details because it's a bit dry, but if you want to checkout the code, it's currently on Github.

// https://github.com/i-y-land/HTTP/blob/episode/03/library/utilities.js#L11
export const factorizeBuffer = (r, mk = 1024, ml = 1024) => { ... }

With this new toy I can read a chunk from the connection, route the request, move the cursor back to the beginning and
pass the buffer to the handler function like nothing happened.

The peek function specifically has a similar signature to read, the difference is that it will move the cursor
back, read a chunk from the buffer in memory and then finally move the cursor back again.

serve(
  Deno.listen({ port }),
  async (connection) => {
    const r = factorizeBuffer(connection);

    const xs = new Uint8Array(1024);
    const reader = r.getReader();
    await reader.peek(xs);
    const [method] = decode(readLine(xs)).split(" ");

    if (method !== "GET" && method !== "POST" && method !== "HEAD") {
      return connection.write(
        encodeResponse({ statusCode: 400 }),
      );
    }

    if (method === "POST") {
      return receiveStaticFile(r, { targetPath });
    } else {
      return sendStaticFile(r, { sourcePath });
    }
  }
)

To finish this, like a boss, I finalize the receiveStaticFile (https://github.com/i-y-land/HTTP/blob/episode/03/library/server.js#L15) and sendStaticFile (https://github.com/i-y-land/HTTP/blob/episode/03/library/server.js#L71) functions, taking care of all
the edge cases.
Finally, I run all the integration tests to confirm that I did a good job. And uuugh. Sleeeep.

This one turned out to be a lot more full of surprise than I was prepared for.
When I realized that some client send file in two-steps, it really threw a wrench to my plans...
But it turned out to an amazing learning opportunity.
I really hope that you are learning as much as I am.
On the bright side, this forced me to put together all the tools that I know I will need for the next post.
Next, I want to look into streaming in more details and build some middlewares, starting with a logger.
From there, I am sure that I can tackle building a nice little router which will wrap this up pretty nicely.

All of the code is available on Github, if you have a question do no hesitate to ask...
Oh speaking of that, I launched a Discord server, if you want to join.

🐙 You will find the code for this episode here: https://github.com/i-y-land/HTTP/tree/episode/03

💬 You can join the I-Y community on Discord: https://discord.gg/eQfhqybmSc

At any rate, if this article was useful to you, hit the like button, leave a comment to let me know or best of all,
follow if you haven't already!

Ok bye now...

Building a HTTP Server from scratch: Understanding Request & Response

Sebastien Filion — Mon, 05 Jul 2021 17:18:09 +0000

Oh, hey there!

I'm glad you made it to this second post of the "Build the System: HTTP server" series.
This post is dedicated to decoding HTTP requests and encoding the response. I will also, offer a reliable way to test
our code for a more resilient project.
If you haven't read the first post of the series yet, I think you might want to. Just click here to read it.
I'll wait patiently for your return.

This article is a transcript of a Youtube video I made.

Alright, now that I know we're all on the same page, let's write some code.
For this project, I will use JavaScript and Deno, but the concepts don't change no matter what language or runtime you
are using.
Also one last disclaimer: this project first aim is to educate it will in no way be complete or the most performant!
I will discuss specifically the improvements we can bring to make it more performant and I will go through various
iteration with that in mind. At the end of the project, if there are part worth salvaging, I will replace the essential
parts.
All that to say, just enjoy the ride.

The first thing that I need to do is to announce listening on a port.
The incoming connection will be represented by a Readable/Writable resource.
First, I will need to read from the resource a specific amount of bytes. For this example, I will read around a KB.
The variable xs is a Uint8Array. I already wrote an article about this but long story short, a Typed Array is an array
that can only hold a specific amount of bit per item. In this case we need 8 bits (or one byte) array because you need 8 bits
to encode a single UTF-8 character.

🐙 You will find the code for this post here: https://github.com/i-y-land/HTTP/tree/episode/02

As a convenience, I will decode the bytes to a string and log the result to the console.
Finally, I will encode a response and write it to the resource.

// scratch.js
for await (const connection of Deno.listen({ port: 8080 })) {
  const xs = new Uint8Array(1024);
  await Deno.read(connection.rid, xs);

  console.log(new TextDecoder().decode(xs));

  await Deno.write(
    connection.rid,
    new TextEncoder().encode(
      `HTTP/1.1 200 OK\r\nContent-Length: 12\r\nContent-Type: text/plain\r\n\r\nHello, World`
    )
  );
}

Now, I will run the code:

deno run --allow-net="0.0.0.0:8080" scratch.js

On a different terminal session I can use curl to send an HTTP request.

curl localhost:8080

On the server's terminal, we can see the request, and on the client's terminal we can see the response's body:
"Hello, World"

Great!

To get this started on the right foot, I will refactor the code into a function named serve in a file called
server.js. This function will take a listener and a function that takes a Uint8Array and returns a Promise of a
Uint8Array!

// library/server.js
export const serve = async (listener, f) => {
  for await (const connection of listener) {
    const xs = new Uint8Array(1024);
    const n = await Deno.read(connection.rid, xs);

    const ys = await f(xs.subarray(0, n));
    await Deno.write(connection.rid, ys);
  }
};

Notice that the read function returns the number of byte that was read. So we can use the subarray method to pass
a lense on the appropriate sequence to the function.

// cli.js
import { serve } from "./server.js";

const $decoder = new TextDecoder();
const decode = $decoder.decode.bind($decoder);
const $encoder = new TextEncoder();
const encode = $encoder.encode.bind($encoder);

if (import.meta.main) {
  const port = Number(Deno.args[0]) || 8080;
  serve(
    Deno.listen({ port }),
    (xs) => {
      const request = decode(xs);
      const [requestLine, ...lines] = request.split("\r\n");
      const [method, path] = requestLine.split(" ");
      const separatorIndex = lines.findIndex((l) => l === "");
      const headers = lines
        .slice(0, separatorIndex)
        .map((l) => l.split(": "))
        .reduce(
          (hs, [key, value]) =>
            Object.defineProperty(
              hs,
              key.toLowerCase(),
              { enumerable: true, value, writable: false },
            ),
          {},
        );

      if (method === "GET" && path === "/") {
        if (
          headers.accept.includes("*/*") ||
          headers.accept.includes("plain/text")
        ) {
          return encode(
            `HTTP/1.1 200 OK\r\nContent-Length: 12\r\nContent-Type: text/plain\r\n\r\nHello, World`,
          );
        } else {
          return encode(
            `HTTP/1.1 204 No Content\r\nContent-Length: 0\r\n\r\n`,
          );
        }
      }

      return encode(
        `HTTP/1.1 404 Not Found\r\nContent-Length: 0\r\n\r\n`,
      );
    },
  )
    .catch((e) => console.error(e));
}

Now that I have an way to parse the headers, I think it's a good opportunity to officialize all of this and write a new
utility function and the appropriate tests.

// library/utilities.js

export const parseRequest = (xs) => {
  const request = decode(xs);
  const [h, body] = request.split("\r\n\r\n");
  const [requestLine, ...ls] = h.split("\r\n");
  const [method, path] = requestLine.split(" ");
  const headers = ls
    .map((l) => l.split(": "))
    .reduce(
      (hs, [key, value]) =>
        Object.defineProperty(
          hs,
          key.toLowerCase(),
          { enumerable: true, value, writable: false },
        ),
      {},
    );

  return { method, path, headers, body };
};

// library/utilities_test.js

Deno.test(
  "parseRequest",
  () => {
    const request = parseRequest(
      encode(`GET / HTTP/1.1\r\nHost: localhost:8080\r\nAccept: */*\r\n\r\n`),
    );

    assertEquals(request.method, "GET");
    assertEquals(request.path, "/");
    assertEquals(request.headers.host, "localhost:8080");
    assertEquals(request.headers.accept, "*/*");
  },
);

Deno.test(
  "parseRequest: with body",
  () => {
    const request = parseRequest(
      encode(
        `POST /users HTTP/1.1\r\nHost: localhost:8080\r\nAccept: */*\r\n\r\n{"fullName":"John Doe"}`,
      ),
    );

    assertEquals(request.method, "POST");
    assertEquals(request.path, "/users");
    assertEquals(request.headers.host, "localhost:8080");
    assertEquals(request.headers.accept, "*/*");
    assertEquals(request.body, `{"fullName":"John Doe"}`);
  },
);

Now that I have a parseRequest function, logically I need a new function to stringify the response...

// library/utilities.js

import { statusCodes } from "./status-codes.js";

export const normalizeHeaderKey = (key) =>
  key.replaceAll(/(?<=^|-)[a-z]/g, (x) => x.toUpperCase());

export const stringifyHeaders = (headers = {}) =>
  Object.entries(headers)
    .reduce(
      (hs, [key, value]) => `${hs}\r\n${normalizeHeaderKey(key)}: ${value}`,
      "",
    );

export const stringifyResponse = (response) =>
  `HTTP/1.1 ${statusCodes[response.statusCode]}${
    stringifyHeaders(response.headers)
  }\r\n\r\n${response.body || ""}`;

// library/utilities_test.js

Deno.test(
  "normalizeHeaderKey",
  () => {
    assertEquals(normalizeHeaderKey("link"), "Link");
    assertEquals(normalizeHeaderKey("Location"), "Location");
    assertEquals(normalizeHeaderKey("content-type"), "Content-Type");
    assertEquals(normalizeHeaderKey("cache-Control"), "Cache-Control");
  },
);

Deno.test(
  "stringifyResponse",
  () => {
    const body = JSON.stringify({ fullName: "John Doe" });
    const response = {
      body,
      headers: {
        ["content-type"]: "application/json",
        ["content-length"]: body.length,
      },
      statusCode: 200,
    };
    const r = stringifyResponse(response);

    assertEquals(
      r,
      `HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nContent-Length: 23\r\n\r\n{"fullName":"John Doe"}`,
    );
  },
);

So now, we have everything we need to refactor our handler function and make it more concise and declarative.

import { serve } from "./library/server.js";
import {
  encode,
  parseRequest,
  stringifyResponse,
} from "./library/utilities.js";

if (import.meta.main) {
  const port = Number(Deno.args[0]) || 8080;
  serve(
    Deno.listen({ port }),
    (xs) => {
      const request = parseRequest(xs);

      if (request.method === "GET" && request.path === "/") {
        if (
          request.headers.accept.includes("*/*") ||
          request.headers.accept.includes("plain/text")
        ) {
          return Promise.resolve(
            encode(
              stringifyResponse({
                body: "Hello, World",
                headers: {
                  "content-length": 12,
                  "content-type": "text/plain",
                },
                statusCode: 200,
              }),
            ),
          );
        } else {
          return Promise.resolve(
            encode(stringifyResponse({ statusCode: 204 })),
          );
        }
      }

      return Promise.resolve(
        encode(
          stringifyResponse({
            headers: {
              "content-length": 0,
            },
            statusCode: 404,
          }),
        ),
      );
    },
  )
    .catch((e) => console.error(e));
}

So at this we can deal with any simple request effectively. To wrap this up and prepare the project for future iteration,
I will add a test for the serve function. Obviously, this function is impossible to keep pure and to test without
complex integration tests -- which I keep for later.
An actual connection is a bit figety so I thought I could mock it using a file as the resource since files are
readable/wriatable.
The first thing I did is to write a function to factorize an async iterator and purposely make it break after the first
iteration.
After that, I create a file with read/write permissions. With that, I can write the HTTP request, then move the cursor
back to the beginning of the file for the serve function to read back. Within the handler function, I make some
assertions on the request for sanity's sake, then flush the content and move the cursor back to the beginning before
writing a response.
Finally, I can move the cursor back to the beginning one last time, to read the response, make one last assertion then
cleanup to complete the test.

// library/server_test.js

import { assertEquals } from "https://deno.land/std@0.97.0/testing/asserts.ts";

import { decode, encode, parseRequest } from "./utilities.js";
import { serve } from "./server.js";

const factorizeConnectionMock = (p) => {
  let i = 0;

  return {
    p,
    rid: p.rid,
    [Symbol.asyncIterator]() {
      return {
        next() {
          if (i > 0) {
            return Promise.resolve({ done: true });
          }
          i++;
          return Promise.resolve({ value: p, done: false });
        },
        values: null,
      };
    },
  };
};

Deno.test(
  "serve",
  async () => {
    const r = await Deno.open(`${Deno.cwd()}/.buffer`, {
      create: true,
      read: true,
      write: true,
    });

    const xs = encode(`GET /users/1 HTTP/1.1\r\nAccept: */*\r\n\r\n`);

    await Deno.write(r.rid, xs);

    await Deno.seek(r.rid, 0, Deno.SeekMode.Start);

    const connectionMock = await factorizeConnectionMock(r);

    await serve(
      connectionMock,
      async (ys) => {
        const request = parseRequest(ys);

        assertEquals(
          request.method,
          "GET",
          `The request method was expected to be \`GET\`. Got \`${request.method}\``,
        );
        assertEquals(
          request.path,
          "/users/1",
          `The request path was expected to be \`/users/1\`. Got \`${request.path}\``,
        );
        assertEquals(
          request.headers.accept,
          "*/*",
        );

        await Deno.ftruncate(r.rid, 0);
        await Deno.seek(r.rid, 0, Deno.SeekMode.Start);

        const body = JSON.stringify({ "fullName": "John Doe" });

        return encode(
          `HTTP/1.1 200 OK\r\nContent-Type: image/png\r\nContent-Length: ${body.length}\r\n\r\n${body}`,
        );
      },
    );

    await Deno.seek(r.rid, 0, Deno.SeekMode.Start);

    const zs = new Uint8Array(1024);
    const n = await Deno.read(r.rid, zs);

    assertEquals(
      decode(zs.subarray(0, n)),
      `HTTP/1.1 200 OK\r\nContent-Type: image/png\r\nContent-Length: 23\r\n\r\n{"fullName":"John Doe"}`,
    );

    Deno.remove(`${Deno.cwd()}/.buffer`);
    Deno.close(r.rid);
  },
);

At this point we have a good base to work from. Unfortunately our server is a bit limitted, for example, if a request
is larger than a KB, we'd be missing part of the message, that means no upload or download of medium size files.
That's what I plan to cover on the next post. This will force us to be a little bit more familiar with
manipulation of binary bytes.

At any rate, if this article was useful to you, hit the like button, leave a comment to let me know or best of all,
follow if you haven't already!

Ok bye now...

Understanding the HTTP Protocol

Sebastien Filion — Mon, 28 Jun 2021 16:06:25 +0000

Oh, hey there!

This is the first article of a new series of a series called "Build the System".
On "Build the System", or BTS for short, I want to explore how the technologies we use as web developers
works. I will explain the theory and then I will build the system from the ground up.
I think this will be a great opportunity to also discuss software design, documentation and testing.
To launch this new project I chose to dissect the HTTP protocol.
We use HTTP every day and it's kinda important in web development, wouldn't you say so?

This article is a transcript of a Youtube video I made.

I said that HTTP was a protocol, more specifically a network protocol. A protocol can be defined as:

[...] a set of rules, conventions, and data structures that dictate how devices exchange data
across networks. In other words, network protocols can be equated to languages that two devices must understand
for seamless communication of information, regardless of their infrastructure and design disparities.

It's similar to when you cross a co-worker in the morning and they ask you "How are you?" and you answer

"Fine, and you?" even though you're not "fine" at all...

We know that HTTP is a protocol because that's what the P stands for. The other letters stands for HyperText Transfer...
Now we know that HTTP is a protocol used to Transfer HyperText... 🤷

The word HyperText kinda means that it's text beyond text... Or text that overcame the limitation or constraints of
text...
The word was first used to refer to text that could contain hyperlinks; to link various documents together.
Today, HTTP can be used for a lot more than text -- so the guy who coined the words hypertext and hyperlink, also came
up with "hypermedia"... But, I guess HMTP didn't sound right?

So yeah...

The Hypertext Transfer Protocol (HTTP) is an application layer protocol for distributed, collaborative, hypermedia
information systems.

Alright, let's just jump into it. Here I have two terminal sessions open. One will be the server, and the second
will be the client. The client will make a request and the server will "respond".

The first terminal will be used to listen to a TCP connection and write a response. The second terminal will be used
to make a HTTP request.

By the way, TCP is:

[...] one of the main protocols of the Internet protocol suite. ... a connection between client and
server is established before data can be sent.

I might cover TCP/IP further in another article, but all you need to know is that it provides reliable, ordered, and
error-checked delivery of a stream of bytes.

To handle the server-side of things, I will use netcat which is a tool to make or listen to arbitrary TCP and UDP
connections.
On the server's teminal I will type nc -k -l 8080 to listen to incoming connections to the port 8080 of my computer.
On the client's terminal I will initiate a HTTP request using the curl command.

$ curl localhost:8080

On the server side, you'll see our first request.

GET / HTTP/1.1
Host: localhost:8080
User-Agent: curl/7.64.1
Accept: */*

Let's analyze it a little.

The first line is usually refered to as the "request-line".
The first part is called the method. It defines what action to perform on the resource. GET is often the default.
The second part is the request-uri or path. It identifies the resource that is being requested. / being the root. "What
the root is" is open to interpretation.
Finally, the last part defines the version of the protocol that is used. For now we'll focus on version 1.1 Which is
slowly being phased out in favour of version 2 or 3. But they build on top of one another.
I'll cover HTTP/2 or 3 if there is interest.

Don't hesitate to like, comment and subscribe to let me know you want to learn more about HTTP!

Anyway... Every line of the message are delimited by a CRLF sequence or end-of-line (EOL) marker.
CRLF stands for "Carriage Return and Line Feed". They are usually represented by ASCII 13 and 10 or \r\n.

After the "request-line", the request may or may not come with headers. Headers can be seen as metadata on the request.
They are represented by a field name and value separated by colon and, usually, a space.

In this example, the headers tells us that:

- the request was made to `localhost` which can be useful to know if the server acts as a gateway or proxy;
- the request was sent by the `curl` program;
- the client would accept anything as a response;

Finally, an empty line is sent either ending the request or beginning the body stream.

In this example, there is no body -- the client is waiting for a response.

I will type-in the response...

HTTP/1.1 204 No Content

The response starts with the status-line.
The first part confirms the protocol and the version.
The second part is the status code. I'm sure you know a few already, like 200 or the infamous 404.
Then finally, the reason phrase.
Like for the request, the status-line may be followed by headers and then it must send an empty line and potentially a
body stream.

With this response, we're telling the client that the request was successfully received, understood and accepted but
there will be no content.
This completes the request and the client terminates the connection.

Let's do two more examples...

On the client's terminal I will send a new request:

$ curl localhost:8080/ping

On the server side we can read the GET request to /ping. I will simply respond with a short message but this time,
I will include some headers.

HTTP/1.1 200 OK
Content-Length: 4
Content-Type: text/plain

pong

The Content-Length header will warn the client to expect 4 bytes of data; the Content-Type headers will inform that
the message is just some plain old text. Then following an empty line I can type my message "pong".
This completes the request and the client terminates the connection.

Okay, let's do one last example.

This time, the client will send a body message.

curl localhost:8080/users -H "Content-Length: 23" -H "Content-Type: application/json" -d "{\"fullName\":\"John Doe\"}"

POST /users HTTP/1.1
Host: localhost:8080
User-Agent: curl/7.64.1
Accept: */*
Content-Type: application/json
Content-Length: 23

{"fullName":"John Doe"}

Server side, we can read the request-line, we have a POST request to the /users resource. The Content-Type is
application/json, and the server should expect 23 bytes. After an empty line, we can read the JSON.

HTTP/1.1 201 Created
Location: localhost:8080/users/1
Content-Length: 0

The server respond with a status code of 201 which means that it created the resource.
Then the server tells the client where it can find the new resource with the Location header. Finally, the response
states that there will be no incoming message.
This completes the request and the client terminates the connection.

I think this is a good place to end this introduction to the HTTP protocol.
I hope I was successful on shedding some light on the mysteries of HTTP. Or at least make you curious to learn more...
On the next article, we will start implementing a HTTP server.
We will learn how to decode a streams of bytes into an actionable request and then respond appropriately.
I will also try to offer a reliable way to test our code.
On the following article we will prepare our server to receive and send static files to and from the file system.
Then, on the next article we will implement data streams which is necessary to deal with larger files or to upgrade to
HTTP/2.
Finally, if all goes to plan, the fifth article will focus on building a programmatic router, think Express, Koa or Oak.

At any rate, if this article was useful to you, hit the like button, leave a comment to let me know or best of all,
follow if you haven't already!

Ok bye now...

Colorize text in the terminal with ANSI Sequence

Sebastien Filion — Mon, 07 Jun 2021 12:11:42 +0000

Oh, hey there!

If you are like me, you find a certain satisfaction using a well crafted CLI or terminal tool...
Especially when they use colour or redraw the terminal.

Something like that: https://github.com/erroneousboat/slack-term
Or, this: https://github.com/fdehau/tui-rs
Or, that: https://github.com/wtfutil/wtf

"how do we make on of those?" You are asking yourself? (hopefully) -- This must be your lucky day because this
is the subject of this video! > ANSI Sequence <

This article is a transcript of a Youtube video I made.

An ANSI sequence is a standard signaling to control everything from the cursor position, to styling to colours.
In other words, it's a sequence of characters that makes commands to the terminal.

I made this interactive script to show you the core concepts. It uses JavaScript and Deno, but the concept
don't change no matter what language or runtime you are using.

I will run this script that will simply redirect raw input from my keyboard to the terminal's output.

deno run --unstable echo.js

I can start typing anything...

hello

Now, I will type the first ANSI sequence to render all incoming input red.

ESC + [ + 3 + 1 + m

world

If I want to render the text in blue, I just need to type in a new sequence with the appropriate number for blue, 34.

ESC + [ + 3 + 4 + m

Now everything is blue ~~

I can also set the background color.

ESC + [ + 4 + 3 + m

It's like IKEA

Next, I can just reset the colors.

ESC + [ + 0 + m

Back to basics.

Within the boundary of my terminal I can move the cursor.

ESC + [ + H

I'm typing here!

ESC + [ + 2 + 0 + ; + 2 + 0 + H

Now, I'm typing here.

If I move my cursor back to the top-left corner, I can clear the rest of the screen, giving me a blank slate...

ESC + [ + 0 + H + ESC + [ + J

I did promise you some drawing, let me type away.

ESC[45m
ESC[2;2H
space
space
space
ESC[2;6H
space
space
space
ESC[3;3H
space
space
space
space
space
ESC[4;5H
space

Oh, look... a heart! Which reminds me that if you're following me, I love you!

Before we go any further, I think it's important to mention that not all terminal client can interpret ANSI sequence, so you might
want to test yours before getting too excited. That being said, it probably does...
If you want to confirm, go to your terminal and type echo $TERM... If the value contains one of those words, you're good to go...
It'll probably say "xterm" though...

xterm
rxvt
eterm
screen
tmux
vt100
vt102
vt220
vt320
ansi
scoansi
cygwin
linux
konsole
bvterm

Now if you're running PowerShell, I think that it supports ANSI sequences, but the feature is turned off by default...

While we're on the terminal, another thing you might want to play with is a command called tput -- the command is useful when you need to query
the state and features of your terminal.

The commands that might interest you are the followings.

To output the number of columns

tput cols

To output the number of rows/ or lines

tput lines

And, finally to output the number of colours

tput colors

So now we know that my terminal has 80 columns, 25 rows and is capable of displaying 256 colours. Awesome!

So, this video is marketed as "JavaScript Fundamentals", so I guess I better start writing some JavaScript.

So first and foremost, the easiest way to type an ANSI sequence in JavaScript is to start typing \x1b ~ which is the hexadecimal number for the Escape key.

So we can do something like this:

console.log("\x1b[31mSomething went wrong.\x1b[0m");

And, this works for all sort of sequences...

console.log("\x1b[H\x1b[J\x1b[36mAh! so clean\x1b[0m");

It's already obvious that using ANSI sequences this way can become very noisy. So let's wrap the commands we want to use into function.
For this demo, I will use TypedArray instead of strings. If you are not quite familiar with TypedArray, I wrote an article explaining all about it. I'll wait.

At any rate, here's a refresher: A Typed Array is an array that can only hold a specific amount of bit per item.
When dealing with character, it's typical to use an unsigned 8 bits integers. If you don't know why, again, read the article.

I'll explain this one briefly, you can see the final result here: https://github.com/sebastienfilion/i-y/blob/main/fundamentals/ANSI/generate.js
And get yourself a copy of the artwork here: https://raw.githubusercontent.com/sebastienfilion/i-y/main/fundamentals/ANSI/logo.js

You can also watch the video to get more explanations and the time-lapse.

First I create a few utility functions like this one to facilitate the process:

const clear = () => new Uint8Array([ 27, 91, 72, 27, 91, 74 ]);

The function returns a TypedArray of the following character \x1b, [, H, \x1b, [, J -- \x1b being the hexadecimal number for the "escape" key.

Once I have all of my utility functions ready, all returning a TypedArray, I create a buffer of 1KB.

const xs = new Uint8Array(1024);

Then I reduce all of the commands into the buffer:

[
  clear(),
  inverse(),
  fill(10)
]
  .reduce(
    (i, ys) => xs.set(ys, i) || i + ys.byteLength,
    0
  );

To finally display my art running:

await Deno.write(Deno.stdout.rid, xs);

ANSI sequence are a powerful way to make any CLI or terminal tool interactive and engaging.
I suggest that you look up the Wikipedia page to get a reference for all the various sequence and their effects.
If you're curious and want to learn more about TypedArray, Readable/Writable-Streams and ANSI sequence, you should subscribe to this channel.
I will use these concepts in an incoming video series.
Otherwise, don't forget to hit the "like" button, share or leave a comment if you've learned something.
Okay bye now...

https://en.wikipedia.org/wiki/ANSI_escape_code

Manipulating binary data with TypedArray

Sebastien Filion — Mon, 31 May 2021 18:00:38 +0000

Oh, hey there!

If you played with Readable, Writable-Stream or, Text-Encoder, Decoder you might have encountered a Typed Array.
They are odd looking Arrays that can only hold numbers for some reason...

This article is a transcript of a Youtube video I made.

A Typed Array is an array that can only hold a specific amount of bit per item.
By the way, a bit is like a binary; 0 or 1 while a byte is typically 8 bits.
8 bits can represent a positive integer up to 255, while 16 bits can hold a positive integer up to 510? Well no,
it's 65,536 or 2e16!

00000000 -> 0
00000001 -> 1
01000001 -> 65

There are many Typed Array, from signed 8 bits integers, from -128 to 127, to unsigned 8 bits integers, from 0 to 255
and, all the way to unsigned 64 bits integers which is like 18 quintillion.

The Typed Array that can hold 8 bit positive integers is called a Uint8Array.
8 bits happens to be the perfect amount of memory to hold any English character...
This character encoding standard is called ASCII. It's one of the earliest and, most famous character table that is
still in use today.
The table encodes every character that you may find on an American keyboard plus some special character like null.
In the late 80', early 90' the International Organization for Standardization, ISO, came up with a new encoding table to
standardize international character set; from East-European, to Greek, to Arabic, to Japanese. This table is known as UTF-8.
Today it encodes 154 languages and all the emojis. The UTF-8 encoding is used on 97% of all web pages.

So back to Uint8Array. The Web API specify a pair called TextEncoder and TextDecoder.
They are used to convert a string to a Uint8Array of UTF-8 encoded text and vice-versa.
So for example, if type new TextEncoder().encode("A"), we'll get a Uint8Array of 1 byte represented as 65. So the
code 65 is the capital letter "A".
If you tried to encode letters from other character sets, for example the greek letter lambda
it would return a Uint8Array of two bytes, while the Chinese character for "love" requires
3 bytes.

> new TextEncoder().encode("A")
Uint8Array(2) [ 65 ]
// 01000001

> new TextEncoder().encode("λ")
Uint8Array(2) [ 206, 187 ]
// 11001110 10111011

> new TextEncoder().encode("爱")
Uint8Array(3) [ 231, 136, 177 ]
// 11100111 10001000 10110001

> new TextEncoder().encode("愛")
Uint8Array(3) [ 230, 132, 155 ]
// 11100110 10000100 10011011

Speaking of love...
I love you if you are following me!

Let's take a moment to play with the Text Encoder to make some sense of it. As I've mentioned earlier, capital
letter "A" is represented by the number 65. Logically B is 66 and C is 67.

new TextEncoder().encode("ABC");
Uint8Array(2) [ 65, 66, 67 ]

Now, not so intuitively, lower case is "a" is 97 not 91 🤷. 91 is the left square bracket.

new TextEncoder().encode("abc");
Uint8Array(2) [ 97, 98, 99 ]

Finally, 0 isn't 0 but 48. 0 is null. The first 31 characters are meta character -- they won't show on screen. 27 is
escape, 10 is a line feed and 7 will make your terminal "ding"!

new TextEncoder().encode("012");
Uint8Array(3) [ 48, 49, 50 ]

The TextDecoder constructor can be passed a string, to define the encoder to use. The default being utf-8.

new TextDecoder().decode(Uint8Array.from([ 65, 66, 67 ]));
"ABC"

If the character can't be decoded, it will return what's called a replacement character.

new TextDecoder().decode(Uint8Array.from([ 255 ]))
"�"

You can force the decoder to "throw" in this kind of situation.

new TextDecoder("utf-8", { fatal: true }).decode(Uint8Array.from([ 255 ]))
// Uncaught TypeError: Decoder error.

The Typed Array is mostly interoperable with Arrays as they share many of the same methods.

One of the major difference with an Array, is that a Typed Array can't be extended after being initialized.

const xs = new Uint8Array(12);
xs.set([ 72, 101, 108, 108, 111, 44,  32,  87, 111, 114, 108, 100 ], 0);
// Hello, World
xs.set([ 68, 101, 110, 111 ], 7);
// Hello, Denod
const ys = xs.subarray(0, 11);
// Hello, Deno
ys.map(x => x >= 65 && x <= 90 ? x + 32 : x);
// hello, deno

Although this is often abstracted away, let's use fetch to find a Uint8Array in the wild.

fetch("https://randomuser.me/api/")
  .then(response => response.body.getReader().read())
  .then(({ value }) => {
    console.log(JSON.parse(new TextDecoder().decode(value)));
  });

If you want to learn about the Readable/Writable-Stream in more details, let me know in the comments.
At any rate I intend to cover it on a project-based series sometime soon. So follow if you want to be notified when
I will release this new series

If you are running Deno, we can experiment further with Deno.write to write the unparsed JSON to the terminal.

fetch("https://randomuser.me/api/")
  .then(response => response.body.getReader().read())
  .then(({ value }) => {

    return Deno.write(Deno.stdout.rid, value);
  });

We could also write the body to a file and read it back.

fetch("https://randomuser.me/api/")
  .then(response => response.body.getReader().read())
  .then(({ value }) => {

    return Deno.writeFile(`${Deno.cwd()}/user.json`, value);
  })
  .then(() => {

    return Deno.readFile(`${Deno.cwd()}/user.json`);
  })
  .then((b) => {
    console.log(JSON.parse(new TextDecoder().decode(b)));
  });

A Typed Array is a very memory efficient way to read and write raw binary data.
When you receive data as a Typed Array and, you decode it to a string for example, there is a performance cost.
In JavaScript, the String manipulation methods are hyper optimized -- but if you have a lot of data to decode and re-encode; it might be worth learning to modify the data stream directly.
I have plans to cover this in more details in a future article.
If that's something that sounds interesting to you, it's probably a good idea that you follow. You can also hit "like", share or comment to let me know that this was useful to you.

Running stuff on Deno

Sebastien Filion — Mon, 17 May 2021 12:45:31 +0000

Oh. Hey there!

I'm happy you came back for this third post. The subject today is: "Running stuff on Deno".

This post is a transcript of a Youtube video I made.

I brushed over this on the previous post because I wanted to cover it in detail.
One of the thing that I truly love about Deno is that it is chuck full of tools -- out-of-the-box; with all
of that, I can be productive in seconds without any setup.

The REPL

The first tool that I think we should explore is the REPL. It is a terminal-based interactive runtime.
It is especially useful when you need to test bits of code without having to create a file or bootstrap
a project.

To bring up the REPL, all you need to do is to execute the deno command and, you're ready to go.
The first thing you'll see, is the current version and instruction on how to exit.
On an empty line just hit ctrl+d or type close().
Here we can type-in any valid JavaScript expression.

const message = "hello world".split("").reverse().join("");

Using the tab key, you get autocompletion. Deno.writeTe [tab] and poof:

Deno.writeTextFile(`${Deno.cwd()}/scratch`, message);

Here we can just read the file back to confirm that it was written properly.

Deno.readTextFile(`${Deno.cwd()}/scratch`).then(message => console.log(message));

Since all the IO method returns a Promise, this is a perfect segway into "top-level await" -- For this
example, I will assume that you are familiar with the async/await. Deno allows the use of await within
the global scope although it is usually reserved for specifically flagged functions. We can leverage this
feature when using the REPL.

await Deno.readTextFile(`${Deno.cwd()}/scratch`);

In the REPL usual import syntax isn't available, we can leverage top-level await and the import
function available in the runtime to import modules.

const { getUser } = await import("https://raw.githubusercontent.com/sebastienfilion/i-y/main/02/users.js");

await getUser();

The REPL is full of shortcuts. Here's a few that I enjoy!

ctrl+R, up/down arrows to search for something.

> "he
const message = "hello world".split("").reverse().join("");

CTRL+U, CTRL+Y to cut or paste a line, useful when you need to remove a line quickly.

There's also a special character _ that always refers to the last evaluated value.

const multiply = (x) => x * 2;

[ 42, 24, 12 ].map(multiply);

_.map(multiply);

> [ 168, 96, 48 ]

In the same vein, _error refers to the last error that was thrown.

[ 42, 24 12 ]
> SyntaxError: Unexpected number
_error

It's important to note that the REPL might be executed with the --unstable flag, if you need access unstable APIs.

Deno.consoleSize(Deno.stdout.rid);

Finally, you can pipe files or expression into the REPL using the --.

echo "2 + 2" | deno --

You can also add the --unstable flag.

echo "Deno.consoleSize(Deno.stdout.rid);" | deno --unstable --

Be careful though cause running code like this doesn't execute in a sandbox. So you may be giving open access to your computer to some stranger... This is a perfect segue into permissions...

echo "await Deno.readTextFile(\"./scratch\")" | deno --

Running with permissions

All the code for this demo is available on Github.

So that's for the REPL. In my experience, it is one of the most complete and friendlier REPL out there.
Now let's talk about the run subcommand in detail. As I mentioned earlier, I brushed over it during the
previous videos because I wanted to cover it in detail. I also want to explore the permission API as
it is one of Deno main selling-point.

Take this code as an example. It uses the fetch function to access a given movie's data over HTTP.

// movies.js
export function getMovieByTitle (APIKey, title) {

  return fetch(`http://www.omdbapi.com/?apikey=65ea1e8b&t=${encodeURIComponent(title)}`)
    .then(response => response.json());
}

To run this code, we'll import it into a file and pass the OMDB_API_KEY environment variable.

// scratch.js
import { getMovieByTitle } from "./movies.js";

getMovieByTitle(Deno.env.get("OMDB_API_KEY"), "Tenet")
  .then(movie => console.log(movie));

So, now we use the --allow-net and --allow-env flags to grant the right permissions when running the file.

OMDB_API_KEY=████████ deno run --allow-net="www.omdbapi.com" --allow-env="OMDB_API_KEY" scratch.js

Ok, so now let's say that we want to write to a file the title and the description of our favourite movies; we can create a CLI that will take the title of the movie, fetch it and write it to the File System.

// cli.js
const [ title ] = Deno.args;

getMovieByTitle(Deno.env.get("OMDB_API_KEY"), title)
  .then(
    movie => Deno.writeTextFile(
      `${Deno.cwd()}/movies`,
      `${movie.Title}: ${movie.Plot}\r\n`,
      { append: true }
    )
  )
  .then(() => console.log("...done"));

To run this file, we'll need to grand "write" permission with --allow-write.

OMDB_API_KEY=████████ deno run --allow-net="www.omdbapi.com" --allow-env="OMDB_API_KEY" --allow-read=$(pwd) --allow-write=$(pwd) cli.js "The Imitation Game"

Another way to grant permission is with --prompt. This option will prompt the user for every permission not granted already when the runtime reaches the code.

OMDB_API_KEY=████████  deno run --prompt cli.js "Tron"

From this, I just want to take a quick detour to explore the runtime's Permission API.

console.log(await Deno.permissions.query({ name: "write", path: import.meta.url }));

await Deno.permissions.request({ name: "write", path: import.meta.url })

console.log(await Deno.permissions.query({ name: "write", path: import.meta.url }));

await Deno.permissions.revoke({ name: "write", path: import.meta.url })

console.log(await Deno.permissions.query({ name: "write", path: import.meta.url }));

The object part is called a "permission descriptor" -- they all have a "name" property but, the other property might be
different.
For example... to read and write it's "path"...

const readDescriptor = { name: "read", path: import.meta.url };
const writeDescriptor = { name: "write", path: import.meta.url };

const environmentDescriptor = { name: "env", variable: "OMDB_API_KEY" };
const netDescriptor = { name: "net", command: "www.omdbapi.com" };

const runDescriptor = { name: "run", command: "git" };

Okay, we're back on track now. Now that we can add movies to our file, I think it would be useful for our tool to read them back to us. I wrote a small utility to display the file while creating an opportunity to use an unstable API.

import { getMovieByTitle } from "./movies.js";
import { prepareForViewport } from "https://raw.githubusercontent.com/sebastienfilion/i-y/main/deno/03/utilities.js";

function displayMovies (data) {
  const { columns, rows } = Deno.consoleSize(Deno.stdout.rid);

  return Deno.write(
    Deno.stdout.rid,
    prepareForViewport(data, { columns, rows, title: "My movie collection" })
  );
}

if (import.meta.main) {
  const [ action, title ] = Deno.args;

  if (action === "fetch") getMovieByTitle(Deno.env.get("OMDB_API_KEY"), title)
    .then(
      movie => Deno.writeTextFile(
        `${Deno.cwd()}/movies`,
        `${movie.Title}: ${movie.Plot}\r\n`,
        { append: true }
      )
    )
    .then(() => console.log("...done"));
  else if (action === "display") Deno.readFile(`${Deno.cwd()}/movies`)
    .then(displayMovies);
  else console.error(`There are no action "${action}"`);
}

So this time, because we use Deno.consoleSize which is marked as unstable, we need to add the --unstable flag. Also, because we are reading from our movie file, we need to grand read permission with --allow-read.

OMDB_API_KEY=████████ deno run --allow-net="www.omdbapi.com" --allow-env="OMDB_API_KEY" --allow-read=$(pwd) --allow-write=$(pwd) cli.js fetch "WarGames"

If you were to download the code and run it with --watch, you'd be able to play with the options of prepareForViewport.

You can change the title, or the ratio for a number between 0 and 1, the default is 0.8.

OMDB_API_KEY=65ea1e8b deno run --allow-env=OMDB_API_KEY --allow-net="www.omdbapi.com" --allow-read=$(pwd) --allow-write=$(pwd) --unstable --watch cli.js display

Before closing on this chapter, I want to talk about one more permission flag, --allow-run. This flag allows the code
to run a command, for example ls, git, etc...
The command will not be executed in the same sandbox as Deno.
Meaning that a malicious developer could do the following... which would output all the structure of your current
working directory.

const process = Deno.run({ cmd: [ "ls", "." ] });
await process.status();

Giving permission to a process to run any command could be a huge security risk.
Always use --allow-run along with the commands that you know will be used. For example --allow-run=git... to allow a process to use Git on the current working directory.
I will do a full video on the Deno.run API later down the line.

Um, I've avoided using it up until now; there's also a --allow--all flag or -A... To grand all the
permissions...
It's safe to use while you're developing -- but don't be lazy use the appropriate flag when running code you find on the
Internet.

When you'll get bored of always typing the run command with all of it's permissions, you may want to consider simply
creating an executable.

echo "OMDB_API_KEY=65ea1e8b deno run --allow-env=OMDB_API_KEY --allow-net="www.omdbapi.com" --allow-read=$(pwd) --allow-write=$(pwd) --unstable --watch cli.js display" | ilm
chmod +x ilm
./ilm

That was a long one...
In this post we saw how to run stuff with Deno and more importantly how to run stuff safely using the Permission flags.
On the next post, we will resume our Deno-journey and explore all the tools that can help us write better code...
Like the linter, the formatter, the test runner and the documentation generator!

Promises aren't just a way to deal with async operations...

Sebastien Filion — Tue, 11 May 2021 13:57:39 +0000

Oh, hey there!

So you think you understand Promises, huh?

In JavaScript, Promises are both a way to handle asynchronous operations and a data structure.

This article is a transcript of a Youtube video I made.

Whether you are making a HTTP request, querying a database or writing to the console, I/O operations can
be very slow. Because JavaScript is single-threaded by design -- can only do one thing at a time -- asynchronous
or async operations are very common.
Let me give you an example, say that when a user of a web app clicks on a button that trigger a HTTP request to
an API -- the JavaScript runtime had to wait for the request to resolve before handling any other operation,
it would make for a pretty sluggish experience.
Instead the engine makes the request, put it aside and gets ready to handle any other operations. Every now and then,
the process will look at the request -- and be like "are you done yet?". When the request finally resolves, the engine
will execute a function defined by the developer to handle the response.
You might know those as "callback functions".

A good example of this is setTimeout. It's a function that takes another function as argument which will be
executed asynchronously later.

console.log("Before...");

setTimeout(() => console.log("...One second later"), 1000);

console.log("...After");

Callbacks works just fine in many cases but starts becoming especially difficult to deal with when multiple
inter-dependant async operations are needed.

retrieveCurrentUser((error, user) => {
  if (error) return handleError(error);

  setCurrentUserStatus(user.ID, "active", (error) => {
    if (error) return handleError(error);

    retriveActiveThreadsForUser(user.ID, 10, (error, threads) => {
      if (error) return handleError(error);

      threads.forEach(thread => subscribeToThread(thread.ID, user.ID, error => handleError(error)));
    });
  });
});

So let's talk about Promises. I mentioned earlier that a Promise is both a way to deal with async operations and a data
structure. Here's what I meant by that.

Imagine you have the number 42 that you assign to x. From this point on, x refers to the number 42 and can be
used as such. Imagine a function called f that simply multiplies by 2 any number. Now, if we were to pass x to the
function f, it would produce a new number, 84, that we can assign to the variable y. From then on, y is 84.

const f = x => x * 2;
const x = 42;
const y = f(x);

A Promise represents a value that may or may not exists yet. If we assign p as the Promise of 42, you can also, say
that p refers to the number 42 and be used as such. The difference is that because p may or may not be 42 just
yet -- remember async operations -- so; the value, 42, can't be accessed directly.
We use the .then method to access and transform the value.
Similar to our previous example, if we have a function f that multiplies any number by 2, and we apply it to our
Promise p, it would produce a new Promise of the value 84 which we can assign to the variable q. From then on, q
is a Promise of the number 84. It is important to note that p is still a Promise of 42.

const f = x => x * 2;
const p = Promise.resolve(42);
const q = p.then(f);

So now, what if we have a function called g that takes any number, multiplies it by 2, but returns a Promise of the
result? After we apply the function g to our Promise p -- which is still 42, we still end up with a Promise of
84.

const g = x => Promise.resolve(x * 2);
const r = p.then(g);

The rule is that if a function returns a value that is not a Promise, the value will be wrapped in a new Promise. But
if the value is already a Promise, we don't need to wrap it again!

A Promise represents a value that may or may not exists yet. But it also represents the status of the async operation.
A Promise can either be resolved or rejected. The .then method actually accepts two functions as argument. The first
one, for the happy-path, if the operation has resolved. The second one to handle any error that may have occured.

mysteriousAsyncOperation()
  .then(
    handleSuccess,
    handleError
  );

Because Promises are often chained together, there is also a .catch method that accepts a function to handle the first
error that occurs, breaking the chain.

mysteriousAsyncOperation()
  .then(secondMysteriousAsyncOperation)
  .catch(handleError);

A rejected Promise that has been "caught" always returns a resolved Promise.

mysteriousAsyncOperation()
  .then(secondMysteriousAsyncOperation)
  .catch(error => alert("¯\_(ツ)_/¯‍"))
  .then(() => alert("Everything is fine actually."));

Now, let's go back to our previous example with multiple inter-dependant async operations...

const $user = retrieveCurrentUser();
const $threads = userPromise.then(
  user => setCurrentUserStatus(user.ID, "active")
    .then(() => retriveActiveThreadsForUser(user.ID, 10))
);

Promise.all([ $user, $threads ])
  .then(([ user, threads ]) => Promise.all(threads.map(thread => subscribeToThread(thread.ID, user.ID))))
  .catch(error => alert("Something went wrong."));

And from then on, $user and $threads still represent the initial values and can be used again and again without any
unnecessary nesting.

$threads.then(threads => threads.forEach(thread => {
  const e = document.createElement("iy-thread");
  e.value = thread;
  document.body.appendChild(e);
}));

Through my examples, you might have noticed that you can factorize a resolved Promise with the Promise.resolve
function. You can deduct that there's also a Promise.reject function. These functions are useful when you need a quick
way to get a Promise.
But, if you want to create a Promise from an async operation you will need the Promise constructor.

function wait (d) {

  return new Promise(resolve => setTimeout(resolve), d);
}

wait(1000)
  .then(() => alert("Waited one second..."));

The Promise constructor handler function also passes a reject function as the second argument.

function waitOrThrow (d) {

  return new Promise((resolve, reject) => {
    if (Math.random() > 0.5) reject(new Error("Better change next time."));
    else setTimeout(resolve, d);
  });
}

waitOrThrow(1000)
  .then(
    handleSuccess,
    handleError
  );

A Promise is a data structure that represents any type of value which may or may not exists yet.
The Promise protects the value to be accessed directly.
A handler function can be defined to access and transform the value.
When the handler function returns a value, it creates a new Promise for this value.
In modern JavaScript, understanding and mastering Promises is a very important skill!
They look at lot more scary than they are. But I swear, Promises are your friend.

Deno and modules

Sebastien Filion — Mon, 03 May 2021 16:44:07 +0000

In this second article of the Deno series, I want to talk about modules.

This article is a transcript of a Youtube video I made.

If you used Node before -- or any server-side language for that matter -- you will know the gist about modules.
Splitting your code into modules can help you keep your project organized or enable library makers to
share their work.

Browser-side, modules have been available for about four years now.
Before, developers would add their namespace to the global object hoping there wouldn't be any collision.
If you are more of a front-end-type-person, you probably started using modules through babel or webpack.

Deno implements the ES modules specifications, which means we're importing stuff, not requiring stuff.
Working with ES modules is great because it means that Deno code can be ported to modern browser -- it isn't the case with Node code.

Alright -- let's fire up our editor and write some modules.

In this demo I will cover how to export and import modules and, how to ensure the integrity of your dependencies.
During this session, I will assume that you have a basic understanding of JavaScript and it's ecosystem. If you have any
question, don't hesitate to write them in the comment section bellow.

Take this code where we export a string and function.

// users.js
export const targetURL = "https://randomuser.me/api/";

export function getUser () {

  return fetch(`${targetURL}?inc=name,email,login,id`)
    .then(response => response.json())
    .then(({ results: [ user ] }) => user);
}

We can import it like this using the relative path.

// scratch.js

import * as users from "./users.js";

users.getUser()
  .then(user => {
    console.log(`Welcome to ${user.name.title}. ${user.name.last}`);
  });

You can use the Deno run subcommand with the appropriate flag.

deno run --allow-net=randomuser.me ./sratch.js

You can also use the remote URL to the file when using a third-party dependency.

// scratch.js

import * as users from "https://raw.githubusercontent.com/sebastienfilion/i-y/main/deno/02/users.js";

If the resource is on a private server, Deno has an environment variable DENO_AUTH_TOKENS that can be used as bearer token when making the request to the server.

// scratch.js

import * as users from "https://raw.githubusercontent.com/sebastienfilion/i-y-private/main/02/users.js";

DENO_AUTH_TOKENS=████████████████████████████████████████@raw.githubusercontent.com deno run mod.js

In the Deno community, there are some conventions when naming files. The first one is to have a mod.js file to export all public files -- it's a good practice when developing libraries.

// mod.js

export * from "./users.js";

The second one is to have a deps.js file to import and expose third-party dependencies.

// deps.js

export * from "https://raw.githubusercontent.com/sebastienfilion/i-y/main/deno/02/mod.js";

Alternatively, you can create a JSON file to use as import-map for the dependencies. It's useful when you want to import dependencies by name instead of by URL.

{
  "imports": {
    "i-y": "https://raw.githubusercontent.com/sebastienfilion/i-y/main/02/mod.js"
  }
}

// scratch.js

import * as users from "i-y";

deno run --allow-net=randomuser.me --import-map="map.json" ./scratch.js

Warning: import-map can't be composed; so if you are a library maker, try to avoid them.

When there's a new version of your dependencies available, you may need to refresh your cache.

// users.js
export const targetURL = "https://randomuser.me/api/";

export function getUser () {

  return fetch(`${targetURL}?inc=name,email,login,id`)
    .then(response => response.json())
    .then(({ results: [ { name: { title: "title, last: surname } } ] }) => ({ title, surname }));"
}

// scratch.js

import * as users from "https://raw.githubusercontent.com/sebastienfilion/i-y/main/02/users.js";

users.getUser()
  .then(user => {
    console.log(`Welcome to ${user.title}. ${user.surname}`);
  });

You can force the runtime to reload all dependencies with the --reload flag.

deno run --allow-net=randomuser.me --reload ./sratch.js

Or you can be more granular and specify the dependencies you want to reload.

deno run --allow-net=randomuser.me --reload=https://raw.githubusercontent.com/sebastienfilion/i-y ./sratch.js

Also, you can use the cache subcommand.

deno cache --reload
deno run --allow-net=randomuser.me ./sratch.js

Finally, you can ensure integrity of your dependencies by writing a lock file.

deno cache --lock=lock.json --lock-write ./scratch.js

If something happens to the dependencies, you'd get an error warning you of the changes...

deno run --lock=lock.json --allow-net=randomuser.me --reload ./scratch.js

[error]

Okay. Up until now, we worked with modules, importing local files and remote files.
We played with fire by not versioning our dependencies and relied on the integrity check to save us from ourselves.
The Deno community also maintains module registries, similar to NPM. These registries enforce SemVer on all modules -- this is an essential when dealing with multiple dependencies.

We are going to visit two registries, but there are more.
First up is deno/x -- the Deno core team maintains this registry. It is simple to use you can leverage Github Webhook to publish a module.

Next up is Nest -- Nest publishes modules to the permaweb, meaning that they are permanent and immutable.
They do this through blockchain technology. Unlike deno/x, Nest uses a CLI tool to publish modules.

Using a module from one of these registries, you will notice that the path includes a version number that ensures that your import code is immutable.
It also means that you can -- in theory -- import different versions from
the same source.

In this episode, we experimented with Deno modules.
We also did an overview of the many ways you can publish a module within the Deno-verse.

On the next episode, I want to cover the many tools that come out of the box when installing Deno.

You have a linter, a formatter, a test runner, a bundler, an installer and, so much more; I might have to make it a three-parter.

What is Deno?

Sebastien Filion — Tue, 20 Apr 2021 20:26:26 +0000

In this new -- ongoing series -- I want to talk about Deeno. Or Dehno. Or Déno.
Eh, *Dunno, Take your pick.

This article is a transcript of a Youtube video I made.

Deno is a JavaScript runtime, just like Node. Actually, the creator of Node, Ryan Dahl, initialized the Deno project. Maybe you
noticed that Deno is an anagram for Node. Clever.

For the past ten years or so, ECMA International -- the people responsible for standardizing JavaScript -- and browser
manufacturers have been hard at work improving the JavaScript ecosystem.

In retrospect, the design decisions made by the Node creators are feeling more and more foreign.
That's what Ryan said during a presentation in 2018.

So TL;DR Deno is a secure JavaScript runtime that feels more in-line with the modern JavaScript ecosystem. The project bundles more of
the techs that the community loves; things like TypeScript, a linter, testing tools and WASM.

Okay, so let's go over what makes Deno different than Node.

ES Modules

So, the first thing -- which might be shocking if you're coming from Node -- is ECMAScript modules. That is "importing" files instead of "requiring" files.

I know you're going to say "Node has ES modules too," which is stable since version 15 -- But -- Node doesn't accept full URL as the target; every script must be downloaded locally, usually via NPM.

I think it's good that Deno implemented module fetching like the browsers do. It makes for a more consistent code base and, it's the first step toward portable code.

With this feature you can decentralize module registries; You can either import from local files, host them yourself or use a third-party registry like deno/x or nest.land.

import { serve } from "https://deno.land/std@0.88.0/http/server.ts";

Web Platform API

Next up is the "Web Platform API" -- when it makes sense, the Deno team implements APIs using the Whataburger (WHATWG) specs.

That means that things like fetch, WebSocket and Workers are available out-of-the-box.

These kinds of efforts help bridge the gap between "browser code" and "server code," -- making it possible to write code that works everywhere.

fetch("https://randomuser.me/api")
  .then(response => response.json());

TypeScript

Next on the list is "A TypeScript compiler" [confettis]. You heard that right. Deno is a polyglot.

In the same project, you can have JavaScript and TypeScript files without any extra step.
No Babel, no Webpack, no nothing else.

Considering the ceiling shattering rise of TypeScript in recent years, having TypeScript as a first-citizen format is a game-changer that surely will make a lot of developers life easier.

Permissions

Have you ever worried that a Node module might be spying on you, uploading your browser history or stealing your bitcoins for the 4th time? Enter the Permission API.

Straight from the CLI, you can specify if a program can read or write to local files or make network requests.

The rules can be pretty granular, which should give you peace of mind when running code you found on the Internet. But also, it lets your users know that they can trust your module.

deno run --allow-net=randomuser.me

WebAssembly

Remember when I told you that Deno was a polyglot?

Well Deno can also execute WebAssembly modules - which means that, through WASM, Deno can execute code from any language that can be compiled down to the binary byte code format.

Support for this is fairly common in other languages like C, C++ and Rust.

That feature makes it possible to port existing libraries with close to native performance.

Community

Last but not least, is the community. The Deno community is very active and very helpful.

Most of the action happens on the official Discord server.

If you're curious to know more about Deno and haven't yet, I highly recommend that you join the server.

Alright, so with all of this covered, I think it would be great to show you Deno in action!

First we can define a function to experiment with. Notice that we can use TypeScript without any extra-step. Our getUser function will fetch a random user and return a Promise.

// users.ts 
export const targetURL = "https://randomuser.me/api/";

interface Users {
  name: string;
  email: string;
  login: {
    uuid: string;
  };
}

export function getUser (): Promise<Users> {

  return fetch(`${targetURL}?inc=name,email,login,id`)
    .then(response => response.json())
    .then(({ results: [ user ] }) => user);
}

Next we could create a new file to run some tests on our function. The Deno namespace comes with a test function.

// users_test.ts
import { assert } from "https://deno.land/std@0.90.0/testing/asserts.ts";
import { getUser } from "./users.ts";


Deno.test("getUser", async () => {
  const user = await getUser();

  assert(typeof user === "object", "The user is not an object.");
  assert(user.name, "The user does not have a name.");
  assert(user.email, "The user does not have an email.");
});

To run our test, we can use the deno test sub-command, with the appropriate permissions.

deno test --allow-net=randomuser.me users_test.ts

The Deno test runner will detect any resource leak and throw an error. This is a great way to find bugs early.

// users_test.ts

...

Deno.test("getUser: leaking", async () => {
  getUser();
});

deno test --allow-net=randomuser.me users_test.ts

[error]

Finally, we can create a new file to be used as CLI that will write a JSON file to the File System using our previous getUser function. Notice that most I/O functions are available under the Deno namespace.

// cli.ts
import { getUser } from "./users.ts";

export async function writeUserFile () {
  const user = await getUser();

  return Deno.writeFile(`./${user.login.uuid}.json`, new TextEncoder().encode(JSON.stringify(user, null, 2)));
}

if (import.meta.meta) {
  console.log("%cGenerating a new user...", "color: blue");
  await writeUserFile();
  console.log("%cDone!", "color: green");
}

We can run our new tool with the appropriate permissions.

deno run --allow-write=$(pwd) --allow-net=randomuser.me cli.ts

Cherry on top, we can use the deno install sub-command to create a system named command. This is a great way to share utility CLI to colleague and friends.

deno install --allow-read=$(pwd) --allow-write=$(pwd) --allow-net=randomuser.me -n generate-users https://raw.githubusercontent.com/sebastienfilion/i-y/main/01/cli.ts

Now, everything you need a random user, you can just run this nifty little command.

generate-users

That's all folks.

In this article, we talked about Deno, what makes it unique, and we went through an example of Deno in action.

I hope this made you curious to learn more about this fantastic project.

In the next article, we will cover the tools that come out of the box with Deno and how the community deals with modules -- no more package.json.