DEV Community: Tim

Scoped Branch Protection With GitHub And Monorepos

Tim — Thu, 21 Mar 2024 01:23:08 +0000

Background

Before we dive in, here is a brief overview of the technologies we will be working with.

GitHub Branch Protection

GitHub has an awesome feature called branch protection, which allows repository owners
"to disable force pushing, prevent branches from being deleted, and optionally require status checks before merging."

This post discusses the last part of that blurb: the ability to require status checks to pass before code can be merged.

Branch protection rules are only available in public or paid repositories. They are not available in free-tier private repositories.

Monorepos

The monorepo pattern of code organization has really taken off in the past few years. Having all your applications and shared packages live in one repo has many benefits. Tools like Nx and Turborepo make things even better by adding features like caching and parallel task execution.

Typically, applications live in separate directories within a monorepo like this



.
|-- apps
|   |-- api
|   {% raw %}`-- react-app
`-- packages
    `-- shared-package
```

>For more about monorepos, see [this video](https://www.youtube.com/watch?v=9iU_IE6vnJ8)

## Monorepos + Required Checks 
Protected branches with required status checks do not play very nicely with monorepos. There are a few ways things can go wrong if you're not careful. Let me explain.

Let's say you have a monorepo with an API and React application, each with its own test suite. You enable branch protection on the master branch and add a required status check to ensure tests pass before code can be merged.
```yml
name: PR Check
on:
  pull_request:
    types: [opened, synchronize]
    branches:
      - "master"
jobs:
  pr-check:
  # Run tests
```

![Adding the required check](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3tjus7b9eyekcok6a5oa.png)

>The action needs to run before it will appear as an option in this section.

You have a feature branch that you want to merge into the master branch. It only changes code inside the `app/api` directory. You open a PR on GitHub, and the check begins to run.

Do you see the problem here? You'd be forgiven for missing it. I sure did.

You made changes to the API code, so it makes sense for the API tests to run. However, the check runs *all* the tests, no matter what. That means you not only have to wait for the API tests to run before you can merge, but you also need to wait for the React app tests to run. If the API tests finish first, you're stuck waiting for tests checking code you didn't change.

![Waiting for the required check](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/huu71hau4913utiinrk5.png)


This might not seem like a big deal, but if you have a large team opening many PRs across many apps and packages with many status checks, such as linting, testing, or formatting, this time can start to add up fast.

But you are a smart engineer, so you try to optimize the workflow by splitting the check into two: one for the API tests and the other for the React tests. You configure them to be triggered if a branch changes the code of the respective app. That way, when you only change the code inside the `app/api` directory, only the API tests will run, and vice versa with the React app.

```yml
name: API PR Check
on:
  pull_request:
    types: [opened, synchronize]
    branches:
      - "master"
    paths:
      - "apps/api/**"
jobs:
  api-check:
  # Run API tests
```

```yml
name: React PR Check
on:
  pull_request:
    types: [opened, synchronize]
    branches:
      - "master"
    paths:
      - "apps/react/**"
jobs:
  react-check:
  # Run React tests
```

![Updating the required check](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/62girg8aqvwn1tahobih.png)

You update your branch protection rule with the two new checks and save. Mission accomplished.

The next day, you open a new PR with changes in `apps/api` directory. The API tests run and pass—fantastic! The React tests don't run—amazing! So much more efficient! Except... you can't merge. That tantalizing merge button is disabled. But why?

![Unable to merge](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/oddjdsjnu4vku9y6e57g.png)

The rule is not satisfied until *all* the required checks *pass*. To pass, a check needs to *run*. Since your changes only impact the `apps/api` directory, the React check did not run. Since the React check is required to pass to merge, the rule is not satisfied, and the merge is blocked.

This issue has caused me some grief since adopting monorepos, but no more! 

Enter [merge gatekeeper](https://github.com/upsidr/merge-gatekeeper). What does it do? From the README
>_By placing Merge Gatekeeper to run for all PRs, it can check all other CI jobs that get kicked off, and ensure all the jobs are completed successfully. If there is any job that has failed, Merge Gatekeeper will fail as well. This allows merge protection based on Merge Gatekeeper, which can effectively ensure any CI failure will block merge. All you need is the Merge Gatekeeper as one of the PR based GitHub Action_

In other words, it solves this exact problem. Let's see how to use it.

Create a new action that looks like this
```yml
name: Merge Protection
on:
  pull_request:
    types: [opened, synchronize]
    branches:
      - "master"
jobs:
  merge-gatekeeper:
    runs-on: ubuntu-latest
    permissions:
      checks: read
      statuses: read
    steps:
      - name: Run Merge Gatekeeper
        uses: upsidr/merge-gatekeeper@v1
        with:
          token: ${{ secrets.GITHUB_TOKEN }}

```
>Notice the action is not scoped to changes in a specific directory.

Next, update your branch protection rule so that this new check is the **only** one required.

![Adding merge gatekeeper](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/a0od6gdd4ac04x8ttnyn.png)

Now, when you open a PR that changes only code in the `apps/api` directory, just the API test will run, and if they pass, you will be able to merge!

![Checks pass](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/y5p43bhjngwmoweuihlu.png)

>The same will be true if you make a change that impacts just the `/react` directory, or both.

And with that, you now have a protected branch with scoped status checks!

Adding Sign In With Google To A Remix App From Scratch

Tim — Wed, 28 Feb 2024 19:40:20 +0000

While several libraries provide Google authentication for Remix, such as remix-auth-google, they may not meet all use cases or be cumbersome to adapt to your existing authentication patterns. This was the case when I looked into implementing Google authentication into my project. So, let's look into how we can do it ourselves!

Prerequisites

You will need the client secret, client ID, and redirect URL values from Google. Below is a link about how to get them. Save them in a dotenv file in the root of your directory.

# .env
GOOGLE_CLIENT_ID='...'
GOOGLE_CLIENT_SECRET='...'
GOOGLE_REDIRECT_URI='...'

https://developers.google.com/identity/protocols/oauth2/web-server#creatingcred

Make sure the redirect URL maps to a route in your remix app. In this example, we create a route at /google-callback.

Setting Up The Google SDK

We will use google-auth-library to handle the dirty details of the OAuth process for us. Let's start by setting that up.
Run npm i google-auth-library.

// app/lib/oauth-providers/google.ts
import { OAuth2Client } from "google-auth-library";

// Make sure the environment variables are set
if (
  !process.env.GOOGLE_CLIENT_ID ||
  !process.env.GOOGLE_CLIENT_SECRET ||
  !process.env.GOOGLE_REDIRECT_URI
) {
  throw new Error(
    "Missing GOOGLE_CLIENT_ID, GOOGLE_CLIENT_SECRET, or GOOGLE_REDIRECT_URI"
  );
}

const oauthClient = new OAuth2Client({
  clientId: process.env.GOOGLE_CLIENT_ID,
  clientSecret: process.env.GOOGLE_CLIENT_SECRET,
  redirectUri: process.env.GOOGLE_REDIRECT_URI
});

export const generateAuthUrl = (state: string) => {
  return oauthClient.generateAuthUrl({
    access_type: "online",
    scope: ["https://www.googleapis.com/auth/userinfo.email"],
    state
  });
};

We create an OAuth client and a helper function to generate the Google sign in URL we will send our users to. The scope represents what Google related access we want from the user. In this case, we want to see what their email is. You can see a full list of scopes here. The state is any data we want to be carried through the OAuth process and returned to us once completed. That will make more sense when we use it later.

The google-auth-library SDK has great inline documentation. Hover over any function or argument you want to know more about for more information.

Creating The Register Page

We will start by creating the route the user will use to register. The route will have a loader function and link.

// app/routes/register.tsx
import { type LoaderFunctionArgs, json } from "@remix-run/node";
import { useLoaderData } from "@remix-run/react";

import { generateAuthUrl } from "../lib/oauth-providers/google";

export async function loader({ request }: LoaderFunctionArgs) {
  // Put "register" in the state so we know where the user is 
  // coming from when they are sent back to us from Google.
  return json({ googleAuthUrl: generateAuthUrl("register") });
}

export default function Register() {
  const { googleAuthUrl } = useLoaderData<typeof loader>();

  return (
    <div>
      <a href={googleAuthUrl}>Continue with Google</a>
    </div>
  );
}

We want to ensure we call generateAuthUrl in a server context so we don't expose our Google secret to the client.

When the user clicks on the link, they will be sent to a Google sign in page. After completing it, they will be sent to the redirect URL set when creating credentials in Google.

Handling the Response From Google

When the user is sent back to us from Google, there will be a few pieces of data in the query parameters of the URL. The two we are interested in are code and state. The code can be exchanged for access and identity tokens from Google. The state is the string we passed to generateAuthUrl earlier. We can access this data and process it accordingly in the loader function of the route.

// app/routes/google-callback.tsx
import { type LoaderFunctionArgs } from "@remix-run/node";

export async function loader({ request }: LoaderFunctionArgs) {
  const searchParams = new URL(request.url).searchParams
  const code = searchParams.get("code");
  const state = searchParams.get("state");

  if (!state || !code) {
    // Throw an error or redirect back to the register page
  }
}

// This component should never get rendered, so it can be anything
export default function GoogleCallback() {
  return (
    <div>
      <h1>GoogleCallback</h1>
    </div>
  );
}

Back in the oauth-providers/google.ts file, let's add another function.

// app/lib/oauth-providers/google.ts
...

export const getTokenFromCode = (code: string) => {
  const { tokens } = await oauthClient.getToken(code);
  if (!tokens.id_token) {
    throw new Error("Something went wrong. Please try again.");
  }

  const payload = await oauthClient.verifyIdToken({
    idToken: tokens.id_token,
    audience: process.env.GOOGLE_CLIENT_ID
  });

  const idTokenBody = payload.getPayload();

  if (!idTokenBody) {
    throw new Error("Something went wrong. Please try again.");
  }

  return idTokenBody
}

This function exchanges the code for an identity token and then verifies the token. It is important to verify the token to prevent malicious users from abusing the system.

The identity token carries basic information about the user and looks something like this.

{
  "iss": 'https://accounts.google.com',
  "azp": '...',
  "aud": '...',
  "sub": '...',
  "email": '...',
  "email_verified": true,
  "at_hash": '...',
  "iat": 1709138090,
  "exp": 1709140490
}

This is a good breakdown of what is in an identity token. What we are interested in is the sub attribute, which represents the unique ID of the user from Google's perspective. We can use it to associate the user in our database with their Google account.

Registering A User

Returning to our loader, we can use our new function to register a user.

// app/routes/google-callback.tsx
import { type LoaderFunctionArgs, redirect } from "@remix-run/node";

import { getTokenFromCode } from "../lib/oauth-providers/google";

export async function loader({ request }: LoaderFunctionArgs) {
  const searchParams = new URL(request.url).searchParams
  const code = searchParams.get("code");
  const state = searchParams.get("state");

  if (!state || !code) {
    // Throw an error or redirect back to the register page
  }

  const idToken = await getTokenFromCode(code)

  // Replace with your database code
  const user = await db.users.find.where({ google_sub: idToken.sub })

  if (state === "register" && user) {
    // The Google account is already associated with a user
    // Throw an error or redirect to the sign in page
    return redirect("/sign-in")
  }

  if (state === "register") {
    // Replace with your database code
    const newUser = await db.insert.users({ google_sub: idToken.sub })
    // Insert whatever you need here to create a session
    await createSession(newUser)
    return redirect("/dashboard")
  }
}

User Sign In

After a user is registered, they will need to be able to sign in using their Google account later on. The process for this is not very different from the register flow. First, we create the route and get an auth URL from Google. This time, we will set our state to "sign-in."

// app/routes/sign-in.tsx
import { type LoaderFunctionArgs } from "@remix-run/node";
import { useLoaderData } from "@remix-run/react";

import { generateAuthUrl } from "../lib/oauth-providers/google";

export async function loader({ request }: LoaderFunctionArgs) {
  return json({ googleAuthUrl: generateAuthUrl("sign-in") });
}

export default function SignIn() {
  const { googleAuthUrl } = useLoaderData<typeof loader>();

  return (
    <div>
      <a href={googleAuthUrl}>Continue with Google</a>
    </div>
  );
}

The user will go through the same Google sign in flow and be sent to the /google-callback route after. We can update the loader to handle the sign in scenario.

// app/routes/google-callback.tsx
import { type LoaderFunctionArgs, redirect } from "@remix-run/node";

import { getTokenFromCode } from "../lib/oauth-providers/google";

export async function loader({ request }: LoaderFunctionArgs) {
  const searchParams = new URL(request.url).searchParams
  const code = searchParams.get("code");
  const state = searchParams.get("state");

  if (!state || !code) {
    // Throw an error or redirect back to the register page
  }
  const idToken = await getTokenFromCode(code)

  // Replace with your database code
  const user = await db.users.find.where({ google_sub: idToken.sub })

  ...

  if (state === "sign-in" && !user) {
    // The Google account is not associated with a user
    // Throw an error or redirect back to the register page
    return redirect("/register")
  }

  if (state === "sign-in") {
    await createSession(user)
    return redirect('/dashboard')
  }

  // If somehow a user ends up here (they should not be able to)
  // send them to the home page 
  return redirect('/')
}

And there you have it! A simple user sign in/register flow using Google OAuth. Use this as a starting point for your application. Be sure to add styles and error handling for a nice user experience.

// app/lib/oauth-providers/google.ts
import { OAuth2Client } from "google-auth-library";

if (
  !process.env.GOOGLE_CLIENT_ID ||
  !process.env.GOOGLE_CLIENT_SECRET ||
  !process.env.GOOGLE_REDIRECT_URI
) {
  throw new Error(
    "Missing GOOGLE_CLIENT_ID, GOOGLE_CLIENT_SECRET, or GOOGLE_REDIRECT_URI"
  );
}

const oauthClient = new OAuth2Client({
  clientId: process.env.GOOGLE_CLIENT_ID,
  clientSecret: process.env.GOOGLE_CLIENT_SECRET,
  redirectUri: process.env.GOOGLE_REDIRECT_URI
});

export const generateAuthUrl = (state: string) => {
  return oauthClient.generateAuthUrl({
    access_type: "online",
    scope: ["https://www.googleapis.com/auth/userinfo.email"],
    state
  });
};

export const getTokenFromCode = (code: string) => {
  const { tokens } = await oauthClient.getToken(code);
  if (!tokens.id_token) {
    throw new Error("Something went wrong. Please try again.");
  }

  const payload = await oauthClient.verifyIdToken({
    idToken: tokens.id_token,
    audience: process.env.GOOGLE_CLIENT_ID
  });

  const idTokenBody = payload.getPayload();

  if (!idTokenBody) {
    throw new Error("Something went wrong. Please try again.");
  }

  return idTokenBody
}

// app/routes/register.tsx
import { type LoaderFunctionArgs, json } from "@remix-run/node";
import { useLoaderData } from "@remix-run/react";

import { generateAuthUrl } from "../lib/oauth-providers/google";

export async function loader({ request }: LoaderFunctionArgs) {
  return json({ googleAuthUrl: generateAuthUrl("register") });
}

export default function Register() {
  const { googleAuthUrl } = useLoaderData<typeof loader>();

  return (
    <div>
      <a href={googleAuthUrl}>Continue with Google</a>
    </div>
  );
}

// app/routes/sign-in.tsx
import { type LoaderFunctionArgs } from "@remix-run/node";
import { useLoaderData } from "@remix-run/react";

import { generateAuthUrl } from "../lib/oauth-providers/google";

export async function loader({ request }: LoaderFunctionArgs) {
  return json({ googleAuthUrl: generateAuthUrl("sign-in") });
}

export default function SignIn() {
  const { googleAuthUrl } = useLoaderData<typeof loader>();

  return (
    <div>
      <a href={googleAuthUrl}>Continue with Google</a>
    </div>
  );
}

// app/routes/google-callback.tsx
import { type LoaderFunctionArgs, redirect } from "@remix-run/node";

import { getTokenFromCode } from "../lib/oauth-providers/google";

export async function loader({ request }: LoaderFunctionArgs) {
  const searchParams = new URL(request.url).searchParams
  const code = searchParams.get("code");
  const state = searchParams.get("state");

  if (!state || !code) {
    return redirect("/")
  }
  const idToken = await getTokenFromCode(code)

  const user = await db.users.find.where({ google_sub: idToken.sub })

  if (state === "register" && user) {
    return redirect("/sign-in")
  }

  if (state === "register") {
    const newUser = await db.insert.users({ google_sub: idToken.sub })
    await createSession(newUser)
    return redirect('/dashboard')
  }

  if (state === "sign-in" && !user) {
    return redirect("/register")
  }

  if (state === "sign-in") {
    await createSession(user)
    return redirect('/dashboard')
  }

  return redirect('/')
}

export default function GoogleCallback() {
  return (
    <div>
      <h1>GoogleCallback</h1>
    </div>
  );
}

Make JSON file validator in just 5 lines 👀

Tim — Sun, 16 Jul 2023 01:42:56 +0000

In my previous post, I reviewed the different web APIs for working with files. In this post, we will put that knowledge to use and make a simple JSON file validator web app.

A user can go to our app, upload their JSON file, and it will tell them if it is valid or not, all from the browser. No dependencies, no call to external APIs. Just pure JavaScript and HTML.

First the boilerplate HTML, CSS, and element selectors.

<div class="container">
  <h1><code>JSON</code> validator</h1>
  <input name="file" accept=".json" type="file" id="file-input" />
  <div id="message" />
</div>

The attribute type="file" tells the browser to render a file selector instead of a normal text input. accept=".json" tells the browser to only allow the user to select JSON files.

<style>
  .container {
    padding-top: 10px;
    max-width: 500px;
    margin: auto;
    display: flex;
    flex-direction: column;
  }
  h1 {
    font-size: 30px;
  }
  code {
    border-radius: 3px;
    background-color: gainsboro;
    padding: 4px;
    font-size: 0.9em;
  }
  #message {
    padding-top: 20px;
    font-size: 20px;
  }
</style>

<script type="module">
  const input = document.getElementById("file-input");
  const message = document.getElementById("message");
</script>

To start, add an event listener to the input's change event. The callback will receive an event with the <input> as its target. The input the user selects is accessible via the files attribute, which is a list of Files. Since our input only takes one file, we will take the file at index 0 of the list.

addEventListener("change", (e) => {
  const file = e.target.files[0];
});

Next, we need to get the text in the file. This is where the magic of web APIs comes in. A File is a wrapper around another web API, Blob (funny name, I know. It actually is shorthand for binary large object). So anything a Blob can do, a File can do, which is handy because Blob has a method, .text(), which returns the file content as text. Perfect.

.text() returns a promise, so we will make our function async and await the result.

addEventListener("change", async (e) => {
  const file = e.target.files[0];
  const text = await file.text();
});

With the text of the file in hand, we can do our JSON validation with yet another web API available in all browsers, JSON.parse(). This method takes a string as its argument and converts it to a JavaScript object if it is valid JSON or it will throw an error if it's not valid JSON. So all the hard work of parsing and checking the string is done for us by the browser!

Since there is the potential of an error being thrown, we need to wrap the function call in a try...catch block.

input.addEventListener("change", async (e) => {
  const file = e.target.files[0];
  const text = await file.text();
  try {
    JSON.parse(text);
    // Do something if it works
  } catch (e) {
      // Do something if it fails
  }
});

If the JSON file is invalid, no code below JSON.parse(text) and above catch will run. Instead, it will jump directly to the code inside the catch block.

This means we can put the code to handle valid JSON under JSON.parse(text) and the code to handle invalid JSON under catch.

To alert the user if their JSON file is valid, we will append HTML inside of the <div id="message" />.

The error thrown by .parse() should look something like JSON.parse: (some generic error message) at line 2 column 3 of the JSON data. By splitting the error message string on the word at we can get the location of the error in the JSON file to display to the user.

input.addEventListener("change", async (e) => {
  const file = e.target.files[0];
  const text = await file.text();
  try {
    JSON.parse(text);
    message.innerHTML = `<div>Valid <code>JSON</code> ✅<div/>`;
  } catch (e) {
     message.innerHTML = `<div>Invalid <code>JSON</code> ❌ at ${e.message.split(" at ")[1]}<div/>`;
  }
});

And there you have it. A fully functional JSON validator. Here is the full HTML file you can copy and play around with!

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta http-equiv="X-UA-Compatible" content="IE=edge" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>JSON Validator</title>
    <script type="module">
      const input = document.getElementById("file-input");
      const message = document.getElementById("message");

      input.addEventListener("change", async (e) => {
        const file = e.target.files[0];
        const text = await file.text();
        try {
          JSON.parse(text);
          message.innerHTML = `<div>Valid <code>JSON</code> ✅<div/>`;
        } catch (e) {
          message.innerHTML = `<div>Invalid <code>JSON</code> ❌ at ${e.message.split(" at ")[1]}<div/>`;
        }
      });
    </script>
    <style>
      body {
        font-size: 16px;
      }
      .container {
        padding-top: 10px;
        max-width: 500px;
        margin: auto;
        display: flex;
        flex-direction: column;
      }
      h1 {
        font-size: 30px;
      }
      code {
        border-radius: 3px;
        background-color: gainsboro;
        padding: 4px;
        font-size: 0.9em;
      }
      #message {
        padding-top: 20px;
        font-size: 20px;
        font-weight: 500;
      }
    </style>
  </head>
  <body>
    <div class="container">
      <h1><code>JSON</code> validator</h1>
      <input name="file" accept=".json" type="file" id="file-input" />
      <div id="message" class="message" />
    </div>
  </body>
</html>

Web File API deep dive 🤿 (2024)

Tim — Wed, 12 Jul 2023 16:17:03 +0000

Perquisites

Before continuing, you should skim over the MDN docs for file inputs if you have never worked with them before. This post assumes basic familiarity with the <input> and <form> elements. I'll be covering the various interfaces that make up the web File API.

I am using React in this post, but you could adapt the code for use with vanilla JavaScript or a different frontend library without many changes.

Getting the files: The `<input type="file">`

export default function FileUploader() {
  return (
    <form>
      <input
        name="fileInput"
        type="file"
        accept="image/*"
        multiple
      />
    </form>
  )
}

Here we have a React component that renders a form containing an input of type file. Each browser renders a file input differently, but its behavior will be the same across them all. I am using Firefox.

Clicking on Browse... (or whatever your browser renders) opens a file selection dialog.

The multiple attribute is a boolean that controls if the user can upload 1 or multiple files. It is false by default.

For the remainder of this post, we will have it set to true, so the code needs to be able to handle multiple files.

The accept attribute is a string that determines what types of files to allow the user to select in the dialog. accept="image/*" is a special value that allows all image type files. See here for more special values and here for common MIME types that can also be used.

This is what the dialog looks like on MacOS. Notice the text file cannot be selected.

If we remove the accept attribute, the text file can be selected.

Even with the accept attribute set, YOU ALWAYS NEED TO VALIDATE USER-PROVIDED FILES MANUALLY. A user could open the dev tools and remove accept or find some other workaround. Never trust user input, especially when it comes to files that could take up a lot of bandwidth and storage and hike up a cloud bill. At the very minimum do a client-side check, but also on the server if possible. Later on in the post, I will show you a way to do client-side validation.

Working with files: The `FileList` and `File` interfaces

Let's add an onChange handler to the input and see what kind of juicy data we get from it.

<input
  onChange={(e) => console.log(e.target.files)}
  ...    
/>

Every input has a files property that is a FileList object. So it's no surprise we see FileList [ File, File ] in the console.

input.files will ways be a FileList, even if multiple=false is set in the input.

FileList [ File, File ] looks like an array and it behaves like an array when you interact with it in the console. Based on that, you would think FileList is an array and you can call all your favorite array methods on it. However, if you try this you will (like me when I made such a bold assumption) be left with immeasurable disappointment and a ruined day. FileList is an unmodifiable list, a special data type (and a rare find these days) with a whopping 1 method, .item(index), which returns the file at the given index. No .map() or .forEach() here.

FileList is iterable, so we can use a for...of loop to traverse it.

Ok, so what about the Files in FileList? What's their story? A glance at the MDN docs tells us File is a special kind of Blob, which is itself just a generic way to represent data. File has a few extra properties Blob does not. The most useful one is .name, which is the name of the file from the user's device. Like FileList, both File and Blob are read-only interfaces. We will see how we can manipulate its data despite this shortly.

The take-home point here is that File inherits all the methods and properties of Blob and has a few of its own. So anything we can do with a Blob, we can do with a File

Doing something with files: The `FileReader`, `Blob`, and `objectUrl` interfaces

To read the content of a file, we can use the aptly named FileReader API. There are only two use cases where it makes sense to use FileReader. The first is getting the file content as a binary string. The other is getting it as a data URL. Most other use cases should be covered by Blob instance methods or objectURL.

Here is an example of reading the files as binary strings with FileReader and .readAsBinaryString(). The .readAsDataURL() method works the same way.

<input
  onChange={(e) => {
    const files = e.target.files || []
    for (const file of files) {
      const reader = new FileReader()
      reader.onload = () => {
        console.log(reader.result)
      }
      reader.readAsBinaryString(file)
    }
  }}
  ...    
/>

Here is the result of that. Don't you just love some raw binary gibberish?

There is a synchronous version of FileReader, FileReaderSync, but it is only accessible in Web Workers to prevent blocking the main thread.

If you want to do any kind of manipulation of the content of a file (remember, File and Blob are read-only), use the .arrayBuffer() Blob method to clone the file data into an ArrayBuffer.

<input
  onChange={async (e) => {
    const files = e.target.files || []
    for (const file of files) {
      const arrayBuffer = await file.arrayBuffer()
      const typedArray = new Uint8Array(arrayBuffer)
      // Do something with the data here
    }
  }}
  ...    
/>

What if we want to display the selected files in the component? We could use a FileReader and .readAsDataURL() to get a data URL to use as an image src, and that should work, but there is a better, non-event-based way to do it.

Enter URL.createObjectURL(). We pass a File or Blob to this static method and it returns a URL we can use as an image src.

URL.createObjectURL() is not just for images. It can create a URL representation for any File or Blob.

Putting all this together, we can update the component to something like this.

import { useState } from "react"

export default function FileUploader() {
  const [files, setFiles] = useState<File[]>([])
  return (
    <form>
      <input
        name="fileInput"
        type="file"
        accept="image/*"
        multiple
        onChange={(e) => {
          setFiles(Array.from(e.target.files || []))
        }}
      />
      {files.map((file) => (
        <img
          key={file.name}
          src={URL.createObjectURL(file)}
          alt={file.name}
          width={200}
          height={200}
        />
      ))}
    </form>
  )
}

Array.from(e.target.files) enables files.map() further down. Remember that input.files is a FileList without array methods. So we create an array that will be easier to work with.

Here is what it looks like when I select two images.

If we open the inspector and look at the src of the <img> we will see something like blob:http://localhost:3000/0467eeb6-697d-4737-98db-fdef50e78637. This URL is a reference to the file created within the context of the current document. This means the URL will continue to work until the page it was generated on is refreshed or navigated away from, at which point the URL is revoked and stops working.

If you need to revoke a URL programmatically, you can use URL.revokeObjectURL('blob:http....'). A situation that would require this is client-side navigation. For example, when using the Next.js Link and router, the object URL is not revoked because the document is not unloaded. To ensure memory safety, add a mechanism to manually revoke the URL. This could be done with the onLoad handler of an <img>.

{files.map((file) => {
  const objectUrl = URL.createObjectURL(file)
  return (
    <img
      key={file.name}
      src={objectUrl}
      alt={file.name}
      width={200}
      height={200}
      onLoad={() => URL.revokeObjectURL(objectUrl)}
    />
  )
})}

Fun fact: Blobs and URL.createObjectURL() are what power the thumbnails that appear when you hover over the video progress bar on Youtube or Netflix. Here is a Netflix engineer discussing this

Sending the file somewhere else

Now that we have played around with the files on the client, it's time to send them off to a server somewhere. This can be done by adding an onSubmit handler to the form.

For inputs that allow multiple files, access them within the submit handler using the .getAll() FormData method. The reason to use this method is when an input allows multiple files, they are all stored as form values under the same key, in this example, the key is fileInput (from the name attribute of the <input>). Using the more common .get()method will return only the first value at that key, whereas .getAll() returns all of the values stored at the key.

<form
  onSubmit={async (e) => {
    e.preventDefault()
    const files = new FormData(e.target as HTMLFormElement).getAll('fileInput')
  }}
>

Another reason to use .getAll() is it will return the files in an array and not a FileList, making our lives easier.

Next, run the files through some validation. What this looks like will depend on the use case. Here is an example checking the type and size Blob properties.

onSubmit={async (e) =>{
  ...
  files.forEach((file) => {
    // `.type` is the file MIME type
    if (file.type !== "desired/type") {
      // File type error logic here
    }
    // `.size` is the file size in bytes
    if (file.size > 10_000_000) {
      // File size error logic here
    }
  })
  ...
}}

With the files validated, it's time to send them to the server. You will need to determine how your server expects to receive files. Here are two patterns.

The first is the server can handle all of the files stored under the same FormData key. Loop through the files, append them to FormData, and send them on their way.

onSubmit={async (e) =>{
  ...
  const formData = new FormData()
  files.forEach((file) => formData.append("my_files", file))
  const resp = await fetch("...", { method: "POST", body: formData })
  // Do something with the response
  ...
}}

Next is if the server needs the files uploaded individually. Loop through the files, create a request with its own FormData for each, and then await them all.

onSubmit={async (e) =>{
  ...
  const requests = files.map((file) => {
    const formData = new FormData()
    formData.append("my_file", file)
    return fetch("...", { method: "POST", body: formData })
  })
  const responses = await Promise.all(requests)
  // Do something with the responses
  ...
}}

Putting it all together

Combining everything covered, the component might look something like this.

import { useState } from "react"

export default function FileUploader() {
  const [files, setFiles] = useState<File[]>([])

  return (
    <form
            onSubmit={async (e) => {
                e.preventDefault()

                const files = new FormData(e.target as HTMLFormElement).getAll('fileInput')

                files.forEach((file) => {
                    if (file.type !== "desired/type") {
                        throw new Error("Wrong file type")
                    }
                    if (file.size > 10_000_000) {
                        throw new Error("File size is too big")
                    }
                })


                const requests = files.map((file) => {
              const formData = new FormData()
              formData.append("my_file", file)
              return fetch("...", { method: "POST", body: formData })
            })
            await Promise.all(requests)

            }}
    >
      <input
        name="fileInput"
        type="file"
        accept="image/*"
        multiple
        onChange={(e) => {
          setFiles(Array.from(e.target.files || []))
        }}
      />
      <button type="submit">
        Submit
      </button>
      {files.map((file) => {
        const objectUrl = URL.createObjectURL(file)
        return (
          <img
            key={file.name}
            src={objectUrl}
            alt={file.name}
            width={200}
            height={200}
            onLoad={() => URL.revokeObjectURL(objectUrl)}
          />
        )
      })}
    </form>
  )
}

Once you have the functionality down, be sure to add style and accessibility features! Here is a good place to start if you need a guide.

If you want more on the topic of files and the web, MDN has this great post with lots of more information that is worth checking out.