DEV Community: Alex

I Analyzed Dozens of AI Agent Rules Files. Most Are Making Your Agent Worse.

Alex — Sat, 14 Mar 2026 20:02:03 +0000

An ETH Zurich study from February 2026 tested something most of us assumed was helping: giving AI coding agents a rules file.

The results were not what anyone expected.

LLM-generated rules files: -3% task success rate, +20% cost
Human-written rules files: +4% success rate, but +19% cost
No rules file at all was the baseline

The researchers' conclusion: "Omit LLM-generated context files entirely and limit human-written instructions to non-inferable details."

In other words — the rules file most developers copy-paste from cursor.directory or generate with ChatGPT is actively making their AI agent slower, more expensive, and less accurate.

I wanted to understand why, so I crawled thousands of rules files from GitHub and analyzed them. Here's what I found.

The four types of rules (and why only one matters)

Every rule in your rules file falls into one of four tiers:

Essential

The agent cannot discover this from your code or config files. Remove it and the agent will make wrong decisions.

- Use pnpm, never npm or yarn
- Auth uses custom middleware chain, not Express standard
- All API responses must use the envelope format from lib/api-response.ts
- Bugs are tracked in Linear project "BACKEND" — reference ticket IDs in commits

The litmus test: run the agent without this rule. Does it make a mistake that no project file could have prevented? If yes — it's essential.

Helpful

The agent could figure this out by reading multiple files, but the rule saves significant exploration time.

- Prefer server actions for mutations, not API routes
- Use the cn() utility (clsx + twMerge) for conditional class merging
- Test files go in __tests__/ directories, not alongside source

These are worth keeping. They save the agent from reading 10 files to infer what one line could tell it.

Redundant

The agent detects this automatically from standard project files. These waste context window tokens.

- Use TypeScript                    ← tsconfig.json exists
- This project uses React           ← package.json says so
- Use ESLint for linting             ← .eslintrc.json exists
- The project uses 2-space indent    ← .prettierrc says so

Every one of these burns tokens that could hold conversation history or code context instead. Delete them.

Improve Codebase

This shouldn't be an AI instruction at all. It should be a linter rule, a config flag, or a CI check.

- Don't use var                      ← Add ESLint no-var rule
- Use strict mode                    ← Set strict: true in tsconfig.json
- Always run tests before commits    ← Set up husky pre-commit hook
- Use meaningful variable names      ← Add ESLint naming-convention rule

The key distinction: detection vs. enforcement. "This project uses pnpm" is redundant (the lockfile reveals it). "Always use pnpm, never npm" is essential — it enforces a preference the agent can't infer from files alone.

Real example: before and after

Here's a typical rules file I found on GitHub (anonymized, but representative of hundreds I analyzed):

Before (28 rules)

# Project Rules

## General
- Use TypeScript
- Use React with Next.js
- Write clean, maintainable code
- Follow best practices
- Use functional components

## Code Style
- Use 2-space indentation
- Use single quotes
- Use semicolons
- Use camelCase for variables
- Use PascalCase for components
- Prefer const over let, never use var

## Architecture
- Use the App Router
- Use server actions for mutations, not API routes
- Separate business logic from API routes
- Use the repository pattern for data access

## Error Handling
- Always handle errors properly
- Use try-catch blocks
- Never silently swallow errors

## Testing
- Write tests for all new features
- Use Vitest for unit tests
- Mock external dependencies

## Performance
- Use React.memo for expensive components
- Lazy load routes
- Optimize images with next/image

How does this break down?

Essential: 3 rules (server actions for mutations, repository pattern, separate business logic)
Helpful: 7 rules (Vitest, mock externals, React.memo, lazy load, next/image, camelCase, PascalCase)
Redundant: 8 rules (Use TypeScript, Use React, Use Next.js, App Router, functional components, write clean code, follow best practices, write tests)
Improve Codebase: 7 rules (2-space indent, single quotes, semicolons, no var, handle errors, try-catch, never swallow errors)

Over half the file is noise. 15 out of 28 rules either duplicate what the agent already knows or belong in tooling config.

After (11 rules)

# Project Rules

- Use pnpm, never npm or yarn
- Use server actions for mutations, not API routes
- Separate business logic from API routes into /lib/services/
- Use the repository pattern — all DB access goes through /lib/repositories/
- API responses use the envelope format: { success, data, error, meta }
- Error handling: use neverthrow Result types, not try-catch
- All new endpoints need Zod schema validation in /lib/validations.ts
- Test with Vitest — mock external deps, never mock the database
- Use next/image for all images, enforce WebP format
- Commits follow conventional format: feat|fix|refactor: description
- Never import from @prisma/client directly — use the db wrapper from /lib/db

Every rule here is something the agent cannot infer from project files, or would waste significant time discovering. The file is 60% smaller and every line changes agent behavior.

The context window problem

Why does this matter so much? Because of how LLM context windows work.

Your rules file, conversation history, code context, and the AI's own reasoning all share the same fixed-size window. A 200-line rules file with 15 essential lines and 185 lines of noise means:

Token waste — you're paying for the AI to read instructions it doesn't need
Attention dilution — research shows LLMs pay less attention to content in the middle of long contexts (the "lost in the middle" effect). Your essential rules get buried between redundant ones.
Reduced code context — every token spent on "Use TypeScript" is a token that can't hold actual code the agent needs to understand

The ETH Zurich numbers make sense when you think about it this way. A bloated rules file doesn't just fail to help — it actively competes with useful context for the agent's attention.

How to audit your own rules file

Take your current .cursorrules, CLAUDE.md, or AGENTS.md and ask three questions about each rule:

1. Can the agent see this in a config file?

If tsconfig.json, package.json, .eslintrc, biome.json, .prettierrc, or any other standard config already expresses this — delete the rule.

2. Is this enforceable by tooling?

If a linter rule, formatter config, git hook, or CI check can enforce this deterministically — move it there. Don't ask an AI to enforce what a tool can guarantee.

3. Would removing this rule cause the agent to make a wrong decision?

If the answer is "no, it would just take longer" — it's helpful, keep it but don't stress about the wording. If the answer is "yes, it would produce incorrect code" — that's essential, make it precise and specific.

What actually changes AI behavior

Addy Osmani's research found that specific tool mentions in context files increase agent usage from 0.01 to 1.6 times per task. The pattern is clear: specificity drives behavior change.

Rules that work:

- Use server actions for mutations, not API routes
- All dates stored as UTC in the database, converted to local only in the UI layer
- Never import from @prisma/client directly — use the typed wrapper from /lib/db

Rules that don't:

- Write clean code
- Follow best practices
- Handle errors properly

The difference is degrees of freedom. "Handle errors properly" gives the agent infinite valid interpretations. "Use neverthrow Result types, never try-catch in service functions" gives it exactly one path.

I built a tool to automate this

After classifying rules by hand, I got tired and built Agent Rules Builder — a free tool that does this automatically.

The Analyzer is the feature I use most. Paste your existing rules file, and the AI classifies every single rule into the four tiers — with a quality score, reasoning, and concrete suggestions. For "Improve Codebase" rules, it tells you exactly what linter rule or config flag to add instead.

It also catches cross-cutting issues: near-duplicate rules, contradictions, vague rules, and security problems like accidentally exposed API keys or internal paths.

The Builder is for when you're starting from scratch or setting up a new project. Pick your language (67 supported), select your framework, and browse a library of 10,000+ rules organized by category — Architecture, Testing, Security, Performance, Error Handling, Agent Workflow, and more.

What makes it different from template directories: every rule has (or can have) community votes. Other developers upvote or downvote individual rules — not whole files, each rule separately. Bad rules sink, good ones surface on the trending page.

And each rule gets an AI quality assessment — classified as essential, helpful, redundant, or improve-codebase with a score from 1-10 — so you can see at a glance whether a rule justifies its context window cost before you add it.

You can edit rules directly, add your own, choose between three verbosity levels, and export to any format.

The takeaway

Most rules files hurt more than they help because they're full of instructions the AI doesn't need. The fix is simple:

Delete anything the agent can read from config files
Move anything enforceable to linter/formatter/CI
Keep only what the agent genuinely can't infer
Make every remaining rule specific enough to have one interpretation

A 10-line file of essential rules beats a 200-line file of copy-pasted advice every time.

Just for Fun: Create a Battery Indicator with React (Works Only in Chrome)

Alex — Tue, 19 Mar 2024 02:44:24 +0000

In this article, we’re going to build a fun project — a battery indicator component using React. This component will show you the current battery level and whether your device is charging.

However, our battery indicator will rely on the Navigator.getBattery() method to retrieve battery information which is not supported by Mozilla and Safari browsers. And also it’s available only in secure contexts (HTTPS).

This means that our component will only work on Chrome and other Chromium-based browsers like Edge.

So, that’s why it’s more of a fun project than a practical one.

Let’s get started!

1. Create a new React project using Vite and the React template.

Run the following commands to create a new React project using Vite and the React template.

npm create vite@latest battery-indicator - - template react
cd battery-indicator
npm install

2. Install Tailwind CSS.

We’ll use Tailwind CSS to style our battery indicator component. Run the following commands to install Tailwind CSS and its dependencies.

npm install -D tailwindcss postcss autoprefixer
npx tailwindcss init -p

Update the tailwind.config.js file to include the following:

/** @type {import('tailwindcss').Config} */
export default {
  content: ['./index.html', './src/**/*.{js,ts,jsx}'],
  theme: {
    extend: {},
  },
  plugins: [],
}

Then add @tailwind directives on top of the src/index.css file.

@tailwind base;
@tailwind components;
@tailwind utilities;

3. Create the battery indicator component.

Create a new file called BatteryIndicator.jsx in the src/components folder. This file will contain the battery indicator component.

Let’s start with the battery info state and a function to update the battery info.

// Initial battery info: battery level: 0-100, charging status: true/false, supported by the browser: true/false 
const initialBatteryInfo = { level: 0, charging: false, supported: true };

// Battery info state
const [batteryInfo, setBatteryInfo] = useState(initialBatteryInfo);

// Update the battery info
const updateBatteryInfo = (battery) => {
  setBatteryInfo({ level: battery.level * 100, charging: battery.charging, supported: true });
};

We have to be sure that the browser supports the Battery Status API before we can use it. To do this, we can check if the getBattery method is available in the navigator object by navigator.getBattery. If it’s available, we can use it to get the battery status. If it’s not, we set the supported property of the battery info state to false.

Then, we use the useEffect hook to check if the browser supports the Battery Status API and setup the event listeners for the battery status changes.

The function that provides the battery status is navigator.getBattery(). It’s an async function so we need to use await to resolve it.

useEffect(() => {
  // Check if the browser supports the Battery Status API and setup the event listeners
  const checkBatteryAPIAndSetup = async () => {
    if (navigator.getBattery) {
      try {
        // Get the battery status
        const battery = await navigator.getBattery();
        updateBatteryInfo(battery);

        // Setup the event listeners for the battery status changes
        battery.addEventListener('chargingchange', () => updateBatteryInfo(battery));
        battery.addEventListener('levelchange', () => updateBatteryInfo(battery));
      } catch (error) {
        console.error('Battery status is not supported.');
        setBatteryInfo((prev) => ({ ...prev, supported: false }));
      }
    } else {
      console.error('Battery status is not supported.');
      setBatteryInfo((prev) => ({ ...prev, supported: false }));
    }
  };

  checkBatteryAPIAndSetup();
}, []);

Now, let’s create the UI for the battery indicator component. It will consist of the battery level bar, the battery level text, and the charging icon.

// Component to display the battery info
const BatteryInfo = ({ batteryInfo }) => (
    <div className={`w-32 h-14 border-2 ${batteryInfo.charging ? 'border-green-500' : 'border-gray-500'} rounded-lg flex items-center justify-start overflow-hidden relative`}>
      {/* Battery level bar */}
      <div
        className={`${batteryInfo.level > 20 ? 'bg-green-300' : 'bg-red-500'} h-full`}
        style={{ width: `${batteryInfo.level}%` }}
      ></div>
      {/* Battery level text */}
      <div className="absolute w-full h-full flex items-center justify-center text-lg font-semibold">
        {batteryInfo.level.toFixed(0)}%
      </div>
      {batteryInfo.charging && <ChargingIcon />}
    </div>
);

The battery level bar has width that corresponds to the battery level. We use the style attribute to set the width of the bar based on the battery level. We use the toFixed method to round the battery level to the nearest whole number.

Simple logic is used to determine the color of the battery level bar based on the battery level. If the battery level is greater than 20%, the bar will be green, otherwise, it will be red.

We also show the charging icon if the device is charging:

const ChargingIcon = () => (
    <svg className="absolute right-0 mr-[-6px] w-6 h-6 text-green-500" fill="none" viewBox="0 0 24 24" stroke="currentColor">
    <path strokeLinecap="round" strokeLinejoin="round" strokeWidth={2} d="M11 7L6 12h4v8l5-5h-4v-8z" />
  </svg>
);

Now, let’s put it all together in the BatteryIndicator component:

import { useState, useEffect } from 'react';

// Initial battery info: battery level: 0-100, charging status: true/false, supported by the browser: true/false 
const initialBatteryInfo = { level: 12, charging: true, supported: true };

export function BatteryIndicator() {
  const [batteryInfo, setBatteryInfo] = useState(initialBatteryInfo);

  // Update the battery info
  const updateBatteryInfo = (battery) => {
    setBatteryInfo({ level: battery.level * 100, charging: battery.charging, supported: true });
  };

  useEffect(() => {
    // Check if the browser supports the Battery Status API and setup the event listeners
    const checkBatteryAPIAndSetup = async () => {
      if (navigator.getBattery) {
        try {
          // Get the battery status
          const battery = await navigator.getBattery();
          updateBatteryInfo(battery);

          // Setup the event listeners for the battery status changes
          battery.addEventListener('chargingchange', () => updateBatteryInfo(battery));
          battery.addEventListener('levelchange', () => updateBatteryInfo(battery));
        } catch (error) {
          console.error('Battery status is not supported.');
          setBatteryInfo((prev) => ({ ...prev, supported: false }));
        }
      } else {
        console.error('Battery status is not supported.');
        setBatteryInfo((prev) => ({ ...prev, supported: false }));
      }
    };

    checkBatteryAPIAndSetup();
  }, []);

  return (
    <div className="flex items-center justify-center h-screen">
    <div className="text-center">
      {batteryInfo.supported ? (
        <div className="flex flex-col items-center justify-center space-y-2">
          <BatteryInfo batteryInfo={batteryInfo} />
        </div>
      ) : (
        <div className="p-4 rounded-md bg-gray-200 text-gray-700">
          Battery status is not supported in this browser.
        </div>
      )}
    </div>
  </div>
  );
}

So, our component is ready. Now, let’s import it into the App component, remove anything else, and see it in action.

import { BatteryIndicator } from './components/BatteryIndicator';

function App() {
  return (
    <div>
      <BatteryIndicator />
    </div>
  );
}

export default App;

Now, run the following command to start the development server and open the app in your browser.

npm run dev

Open localhost:5173 in your browser. You should see the battery indicator component showing the current battery level and whether your device is charging.

If your device is charging you should see a charging icon on the right side of the battery level bar.

If the battery level is less than 20% the battery level bar will be red:

If the battery status is not supported in your browser, you should see a message saying “Battery status is not supported in this browser.”

That’s it! You’ve created a battery indicator component using React. It’s a fun project that works only in Chrome and other Chromium-based browsers like Edge. It’s not practical, but it’s a fun way to learn about the Battery Status API and how to use it in a React app.

References

GitHub repository of the code from this article
Live demo of the app from this article
Originally posted on Medium

I’m always looking to meet new people in tech. Feel free to connect with me on LinkedIn!

TypeScript Index Signatures: 4 Examples Type-Safe Dynamic Objects

Alex — Wed, 13 Mar 2024 01:25:07 +0000

In TypeScript, index signatures are a powerful feature that allows you to type objects with unknown structures.

They are especially useful when you want to define a type for objects with unknown properties or when you want to create a dictionary-like data structure.

Additionally, index signatures are often used to create complex utility types that can be used to manipulate and transform other types.

What are Index Signatures?

An index signature defines a type for the values that an object can have at a particular index (key). It specifies a contract between the keys and the values of an object.

interface MyInterface {
  [key: string]: number;
}

In this example, the index signature [key: string] defines that any key of type string will have a corresponding value of type number.

This means that any object implementing the MyInterface interface can have any number of string keys, and the values associated with those keys must be numbers.

Example 1: String-to-String Dictionary

Let’s say you want to create a simple dictionary that maps language codes (like “en”, “fr”, “es”) to their full English names (“English”, “French”, “Spanish”). With an index signature, you can define a type for this dictionary that allows for any number of language codes as keys, but ensures that all values are strings.

Define the Dictionary interface:

interface LanguageDictionary {
  [key: string]: string;
}

Now we are sure that any string can be used as a key and all values are also strings:

const languages: LanguageDictionary = {
  en: "English",
  fr: "French",
  es: "Spanish",
  // You can add more languages without changing the type
};

// Adding a new language to the dictionary
languages.de = "German";

// Retrieving a language name
console.log(languages.en); // Output: "English"

Example 2: Product Inventory Object

Suppose you are building an e-commerce application and you want to represent a product inventory object that has a fixed set of properties (name, price) and a dynamic set of properties (stock for different sizes).

You can use an index signature to define a type for this object that allows for a fixed set of properties and a dynamic set of properties.

To maintain strict type safety without mixing specific properties with index signatures when their types differ, it’s better to separate the concerns.

// Bad example of mixing properties with index signatures
type BadProduct = {
  name: string;
  price: number;
  // Error - Property 'name' of type 'string' is not assignable to 'string' index type 'number'
  [size: string]: number; 
}


// Better solution - It's immediately clear which parts of the object are fixed and which parts are dynamic.
type Product = {
  name: string;
  price: number;
  stock: {
    [size: string]: number;
  };
}

In this example the name and price fields maintain their strict types, separate from the dynamically typed stock object.

const tShirt: Product = {
  name: 'T-Shirt',
  price: 20,
  stock: {
    'S': 10,
    'M': 15,
    'L': 5,
  },
};

// Accessing the 'M' stock value using nested object with bracket notation
console.log(product1.stock['M']); // Output: 15

This pattern is scalable and can be extended to include other dynamic properties if needed, by adding more nested objects or arrays with their specific types.

Example 3: Creating a Custom Utility Type

Index signatures are essential in TypeScript for creating complex utility types because they allow for flexible and dynamic data structures while maintaining type safety.

Suppose you have a type representing a user with several properties, and you want to create a new type where all of these properties are optional.

There is built-in utility type Partial that does exactly that. Earlier, I wrote about TypeScript utility types.

However, for understanding of index signatures, let’s create a custom utility type Optional that does the same thing.

type User = {
  id: number;
  name: string;
  age: number;
  email: string;
};

type Optional<T> = {
  [K in keyof T]?: T[K];
};

type OptionalUser = Optional<User>;

// OptionalUser is equivalent to:
// type OptionalUser = {
//   id?: number | undefined;
//   name?: string | undefined;
//   age?: number | undefined;
//   email?: string | undefined;
// }

In the above example, we created a custom utility type Optional that takes a type T and returns a new type where all properties of T are optional.

Here we use index signatures to iterate over the keys of T and make each property optional by adding a ? after the property name.

This allows us to create a new type OptionalUser that has all the properties of User but with each property being optional.

Example 4: API response with dynamic keys

When working with APIs, you often receive data with a fixed set of properties and a dynamic set of properties. Index signatures are useful for defining types for such data.

Suppose you have an API that returns a response with a fixed set of properties (status, message) and a dynamic set of properties (data for different resources).

You can use an index signature to define a type for this response that allows for a fixed set of properties and a dynamic set of properties.

type ApiResponse = {
  status: string;
  message: string;
  [resource: string]: any;
};

const response: ApiResponse = {
  status: "success",
  message: "Data fetched successfully",
  users: [{ id: 1, name: "John" }, { id: 2, name: "Doe" }],
  products: [{ id: 1, name: "Laptop", price: 1000 }],
  // You can add more resources without changing the type
};

In this example, the ApiResponse type has a fixed set of properties (status, message) and an index signature [resource: string] that allows for any number of dynamic properties with any value type.

Conclusion

Index signatures are a powerful feature in TypeScript that allows you to define types for objects with unknown structures. They are especially useful when you want to create dictionary-like data structures or when you want to define complex utility types.

I hope this article has given you a good understanding of how to use index signatures in TypeScript and how they can be used to create complex and flexible types. If you have any questions or feedback, feel free to leave a comment below.

Check out my other TypeScript articles:

This article was originally posted on Medium.

Making React Components More Flexible with TypeScript Generics: 3 Examples

Alex — Tue, 12 Mar 2024 13:00:00 +0000

Generic React components allow you to create a component that can be used with any data type. This is a great way to create reusable components that can be used in multiple places in your application.

There are a lot of articles and tutorials on generics in TypeScript, so I will focus on how to use generics in React components to make them more flexible and reusable.

Example 1: Simple Generic React Component (easy to understand)

In this example, we will create a generic React component that can be used with any data type.

To clarify generics' benefits, we will create a component with data of type T and a render function.

Create a Generic React Component

// Define a generic type for the props
type Props<T> = {
  data: T
  render: (data: T) => React.ReactNode
}

// Create a generic React component
export function GenericComponent<T,>({ data, render }: Props<T>) {
  return <div>{render(data)}</div>
}

Here, GenericComponent accepts prop render, which is a function that takes data of type T and returns a React.ReactNode. This pattern, often called “render props,” gives you more control over how the data is rendered.

Next, we use GenericComponent with different types of data, showing how TypeScript ensures that the types are used consistently.

Use the GenericComponent with a string

import GenericComponent from './GenericComponent'

function RenderString() {
  return <GenericComponent<string> 
    data='Hello, world!' 
    render={(data) => <span>{data.toUpperCase()}</span>} 
  />
}

Here, GenericComponent is explicitly told to expect a string. The render function converts the string to uppercase, which is a valid operation on strings. TypeScript ensures that the operations you perform in the render prop are appropriate for the type of data passed.

Use the GenericComponent with a custom type

type Person = {
  name: string
  age: number
}

import GenericComponent from './GenericComponent'

function RenderPerson() {
  return (
    <GenericComponent<Person>
      data={{ name: 'Alice', age: 30 }}
      render={(data) => (
        <span>
          {data.name} is {data.age} years old
        </span>
      )}
    />
  )
}

In this case, TypeScript ensures that the data prop matches the type expected by the render function.

Let’s move on to a more complex example. We will create a generic component that can be used with a list of items.

Use the GenericComponent with a List of Todos


import { useState } from 'react'
import GenericComponent from './GenericComponent'

interface TodoItem {
  id: number
  title: string
  completed: boolean
}

export function TodoList() {
  // Initial list of todos
  const initialTodos: TodoItem[] = [
    { id: 1, title: 'Learn React', completed: false },
    { id: 2, title: 'Write blog post', completed: true },
    { id: 3, title: 'Study TypeScript', completed: false },
  ]

  // State to track todos
  const [todos, setTodos] = useState<TodoItem[]>(initialTodos)

  // Function to toggle the completion status of a todo item
  function toggleTodoCompletion(id: number) {
    const updatedTodos = todos.map((todo) => 
        (todo.id === id ? { ...todo, completed: !todo.completed } : todo))
    setTodos(updatedTodos)
  }

  // Complex render function for a list of TodoItems
  function renderTodos(todos: TodoItem[]) {
    return (
      <ul>
        {todos.map((todo) => (
          <li
            key={todo.id}
            style={{ textDecoration: todo.completed ? 'line-through' : 'none', cursor: 'pointer' }}
            onClick={() => toggleTodoCompletion(todo.id)}
          >
            {todo.title}
          </li>
        ))}
      </ul>
    )
  }

  return <GenericComponent<TodoItem[]>  
    data={todos} 
    render={renderTodos} 
  />
}

In this example, we create a TodoList component that uses GenericComponent to render a list of TodoItems. The render function is more complex, as it needs to handle a list of items. TypeScript ensures that the data prop matches the type expected by the render function.

In the next example, the benefits of generics will be more clear. We will create a generic component that fetches data from an API and displays it.

Example 2: Generic React Component to Fetch and Display Data

In this example, we will create a generic React component that fetches data from an API and displays it. This is so often used in real-world applications that it’s a great example of how generics can make your components more flexible and reusable.

Create a Generic React Component to Fetch Data


import { useEffect, useState } from 'react'

// Define a generic type for the props
type Props<T> = {
  url: string
  render: (data: T) => React.ReactNode
}

// Create a generic React component
export function FetchAndDisplay<T,>({ url, render }: Props<T>) {
  const [data, setData] = useState<T | null>(null)
  const [loading, setLoading] = useState(true)
  const [error, setError] = useState<Error | null>(null)

  useEffect(() => {
    const fetchData = async () => {
      try {
        const response = await fetch(url)
        if (!response.ok) {
          throw new Error('Failed to fetch data')
        }
        const json = await response.json()
        setData(json)
      } catch (error) {
        setError(error)
      } finally {
        setLoading(false)
      }
    }
    fetchData()
  }, [url])

  if (loading) {
    return <div>Loading...</div>
  }

  if (error) {
    return <div>Error: {error.message}</div>
  }

  if (data) {
    return <div>{render(data)}</div>
  }

  return null
}

In this example, FetchAndDisplay is a generic component that accepts a URL and a render function. It uses the fetchto make a request to the URL and then calls the render function with the data. The component also handles loading and error states.

Using the FetchAndDisplay component to Fetch and Display Posts

import { FetchAndDisplay } from './FetchAndDisplay'

type Post = {
  userId: number
  id: number
  title: string
  body: string
}

function RenderPosts(posts: Post[]) {
  return (
    <ul>
      {posts.map((post) => (
        <li key={post.id}>
          <h3>{post.title}</h3>
          <p>{post.body}</p>
        </li>
      ))}
    </ul>
  )
}

export function PostList() {
  return <FetchAndDisplay<Post[]> 
    url='https://jsonplaceholder.typicode.com/posts'  
    render={RenderPosts} 
  />
}

In this example, we use the FetchAndDisplay component to fetch a list of posts from the JSONPlaceholder API and then render them using the RenderPosts function.

Having the generic component FetchAndDisplay allows us to fetch and display data from any API and render it in any way we want.

For example, we can use the same component to fetch and display a list of users, comments, or any other type of data.

Using the FetchAndDisplay component to Fetch and Display Users

import { FetchAndDisplay } from './FetchAndDisplay'

type User = {
  id: number
  name: string
  email: string
}

function RenderUsers(users: User[]) {
  return (
    <ul>
      {users.map((user) => (
        <li key={user.id}>
          <h3>{user.name}</h3>
          <p>{user.email}</p>
        </li>
      ))}
    </ul>
  )
}

export function UserList() {
  return <FetchAndDisplay<User[]> 
    url='https://jsonplaceholder.typicode.com/users' 
    render={RenderUsers} 
  />
}

Here, we use the same FetchAndDisplay component to fetch a list of users from the JSONPlaceholder API and then render them using the RenderUsers function. This is the power of generics in React components.

Example 3: Generic React Component for Form Input

This example demonstrates how to create a generic form component that can be used with different types of form fields.

In real-world applications, I would use a library like Formik or react-hook-form for form handling, but to demonstrate generics, we will create a simple form component from scratch.

First, define TypeScript types to specify the structure of your form fields and the props your generic form component will accept. This ensures type safety and helps manage the form’s state and behavior.


type FormField = {
  name: string
  label: string
  type: 'text' | 'email' | 'password' // Extend as needed
}

type GenericFormProps<T> = {
  fields: FormField[]
  initialValues: T
  onSubmit: (values: T) => void
}

Next, create a functional component that takes in fields, initial values, and an onSubmit function.

import { useState } from 'react'

export function GenericForm<T>({ fields, initialValues, onSubmit }: GenericFormProps<T>) {
  const [values, setValues] = useState<T>(initialValues)

  function handleChange(e: React.ChangeEvent<HTMLInputElement>) {
    const { name, value } = e.target
    setValues((prevValues) => ({ ...prevValues, [name]: value }))
  }

  function handleSubmit(e: React.FormEvent<HTMLFormElement>) {
    e.preventDefault()
    onSubmit(values)
  }

  return (
    <form onSubmit={handleSubmit}>
      {fields.map((field) => (
        <div key={field.name}>
          <label htmlFor={field.name}>{field.label}</label>
          <input type={field.type} name={field.name} 
                 value={(values as any)[field.name]} 
                 onChange={handleChange} />
        </div>
      ))}
      <button type='submit'>Submit</button>
    </form>
  )
}

Finally, use the GenericForm component with specific form fields and initial values.


type UserFormValues = {
  name: string
  email: string
  password: string
}

const userFormFields: FormField[] = [
  { name: 'name', label: 'Name', type: 'text' },
  { name: 'email', label: 'Email', type: 'email' },
  { name: 'password', label: 'Password', type: 'password' },
]

const initialValues: UserFormValues = {
  name: '',
  email: '',
  password: '',
}

function App() {
  const handleSubmit = (values: UserFormValues) => {
    console.log('Form Submitted', values)
  }

  return (
    <div>
      <h1>User Registration</h1>
      <GenericForm 
        fields={userFormFields} 
        initialValues={initialValues} 
        onSubmit={handleSubmit} 
      />
    </div>
  )
}

In this example, without using generics, you would have to create a separate form component for each type of form. With generics, you can create a single form component that can be used with any type of form fields.

Bonus Example: Table Component with Generics

This example is taken from the video:

In the example, the table component is made generic by adding a type parameter called TRow. This TRow parameter represents the type of data that will be in each row of the table.

The Table component takes in an array of rows and a renderRow function. The renderRow function is called for each row in the array and is responsible for rendering the row.


const Table = <TRow extends Record<string, any>>(props: {
  rows: TRow[];
  renderRow: React.FC<TRow>;
}) => {
  return (
    <table>
      <tbody>
        {props.rows.map((row) => (
          <props.renderRow {...row} />
        ))}
      </tbody>
    </table>
  )
}

In this example, the Table component takes in an array of rows and a renderRow function. The renderRow function is called for each row in the array and is responsible for rendering the row.

<Table
  rows={[
    { name: 'Alice', age: 30 },
    { name: 'Bob', age: 40 },
  ]}
  renderRow={({ name, age }) => (
    <tr>
      <td>{name}</td>
      <td>{age}</td>
    </tr>
  )}
/>

In this example, the Table component is used to render a table with two rows, each containing a name and an age. The TRow type parameter ensures that the rows array and the renderRow function are used consistently.

Conclusion

Generics are a powerful feature of TypeScript that can make your React components more flexible and reusable. They allow you to create components that can be used with any data type, which is especially useful in real-world applications where you often need to work with different types of data.

I hope this article has given you a good understanding of how to use generics in React components and how they can make your components more flexible and reusable. If you have any questions or feedback, please feel free to leave a comment below.

Helpful Resources

Check out my other articles on TypeScript:

This article was originally posted on Medium.

TypeScript Enums: 5 Real-World Use Cases

Alex — Tue, 12 Mar 2024 01:29:40 +0000

If you’re looking to make your TypeScript code more organized, enums are a powerful tool. They group related values together, giving your code better structure and readability. Let’s dive in and explore how to use them!

In TypeScript, enums are declared using the enum keyword. For example, you could create an enum to represent the different fruit prices as follows:

Example 1: Basic Enum Structure — Key Movements

Demonstrates a common use case for enums: making choices within a limited set clearer.

enum Movement {
  Up = "UP",
  Down = "DOWN",
  Left = "LEFT",
  Right = "RIGHT"
}

function handlePlayerInput(key: string) {
  switch (key) {
    case Movement.Up:
     // Move player character up
      break;
    case Movement.Down:
      // Move player character down
      break;
    // ... other cases
  }
}

This demonstrates the fundamental syntax for defining an enum. The Movement enum has four members representing directions.

Example 2: Enums with Values

Enums can have associated values (numbers, strings, etc.). Here, StatusCode associates HTTP status codes, ensuring type-safety and preventing the use of arbitrary numbers.

enum StatusCode {
  OK = 200,
  BadRequest = 400,
  NotFound = 404
}

function handleResponse(code: StatusCode) {
  if (code === StatusCode.OK) {
    // Handle successful response
  } else if (code === StatusCode.NotFound) {
    // Handle resource not found
  } 
  // ... other cases
}

Image source: https://restfulapi.net/http-status-codes

Example 3: Enum from Redux Toolkit

Redux Toolkit is a popular library for state management in React applications. It makes heavy use of TypeScript for type safety.

This enum defines distinct states for asynchronous actions (like fetching data). This is common in state management libraries.

enum PayloadActionLoadingState {
  Idle = "idle",
  Loading = "loading",
  Failed = "failed",
  Success = "success" 
}

Example 4: Enum as a Discriminated Union

This enum defines two possible shapes: Circle and Rectangle. It acts as a foundation for ensuring type safety when working with different shapes.

Each shape type (Circle, Rectangle) is represented as a member of the ShapeType enum.

The Shape interface has a type property that must be a member of the ShapeType enum.

enum ShapeType {
    Circle = "Circle",
    Rectangle = "Rectangle"
  }

  interface Shape {
    type: ShapeType; 
  }

  interface Circle extends Shape {
    type: ShapeType.Circle;
    radius: number;
  }

  interface Rectangle extends Shape {
    type: ShapeType.Rectangle;
    width: number;
    height: number;
  }

  function calculateArea(shape: Shape): number {
    switch (shape.type) {
      case ShapeType.Circle:
        const circle = shape as Circle; // Type assertion to Circle
        return Math.PI * circle.radius * circle.radius;
      case ShapeType.Rectangle:
        const rectangle = shape as Rectangle; // Type assertion to Rectangle
        return rectangle.width * rectangle.height;
      default:
        throw new Error("Invalid shape type");
    }
  }

Specific shape interfaces (Circle, Rectangle) extend the base Shape interface and must have their type property set to the corresponding enum value.

This lets the calculateArea function use the type property as a discriminator to determine the appropriate calculation.

Example 5: Enums as Data Structures

This TypeScript code defines a simple model for representing playing cards, focusing on the suits, ranks, and the color of the card, which is derived from its suit.

It consists of two enums, a function to get a card’s numerical value, an interface to describe a card’s structure, and a function to create a card.

enum Suit {
  Hearts = "♥", // Red suit
  Diamonds = "♦", // Red suit
  Clubs = "♣", // Black suit
  Spades = "♠" // Black suit
}

enum Rank {
  Ace = 1,
  Two, Three, Four, Five, Six, Seven, Eight, Nine, Ten,
  Jack, Queen, King 
}

function getCardValue(rank: Rank): number {
  if (rank <= Rank.Ten) {
    return rank;
  } else {
    return 10;
  }
}

interface Card {
  suit: Suit;
  rank: Rank;
  color: string; // Derived property based on suit
}

function createCard(suit: Suit, rank: Rank): Card {
  return {
    suit,
    rank,
    color: suit === Suit.Hearts || suit === Suit.Diamonds ? 'Red' : 'Black'
  }
}

// Usage
let card1 = createCard(Suit.Hearts, Rank.Ace);
console.log(`The Ace of Hearts is red: ${card1.color}`); // Output: The Ace of Hearts is red: Red

let card2 = createCard(Suit.Spades, Rank.Queen);
console.log(`The Queen of Spades is black: ${card2.color}`); // Output: The Queen of Spades is black: Black

Function getCardValue takes a Rank as an argument and returns a number. For ranks Ace through Ten (numerically 1 to 10), it returns the rank's numeric value. For face cards (Jack, Queen, King), which have rank values greater than 10, it returns 10.

Two card objects are created using the createCard function: card1 as the Ace of Hearts (which is red) and card2 as the Queen of Spades (which is black).

This code is a straightforward way to model the basic properties of playing cards in a type-safe manner using TypeScript’s features like enums, interfaces, and type inference.

Conclusion

Enums provide a solid foundation for clean and structured TypeScript code. As your projects grow in complexity, their benefits will continue to shine.

TypeScript enums are a powerful tool to elevate your code. They enhance readability, maintainability, and type safety.

Check out my other TypeScript articles:

This article was originally posted on Medium.

5 Resources Each TypeScript Developer Should Know About

Alex — Sun, 10 Mar 2024 21:03:47 +0000

Want to become a TypeScript pro? Master advanced TypeScript skills with these resources, from type definitions and challenging type puzzles to practical utility libraries and API development tools.

1. DefinitelyTyped — A collection of type definitions

DefinitelyTyped serves as a community-driven collection of high-quality TypeScript type definitions.

DefinitelyTyped / DefinitelyTyped

The repository for high quality TypeScript type definitions.

Definitely Typed

The repository for high quality TypeScript type definitions.

You can also read this README in Español, 한국어, Русский, 简体中文, Português, Italiano, 日本語 and Français!

Link to Admin manual

!!! Important! This repo has recently changed layout! !!!

Definitely Typed has recently changed to a proper pnpm monorepo; you may want to reread this document for changes to the layout of packages in this repo.

At the very least, you may want to git clean -fdx the repo (or node ./scripts/clean-node-modules.js on Windows) to clean up node_modules and run pnpm install --filter . to install the workspace root. See further sections for more info on pnpm install.

Current status

This section tracks the health of the repository and publishing process. It may be helpful for contributors experiencing any issues with their PRs and packages.

Most recent build type-checked/linted cleanly:
All packages…

View on GitHub

In simple terms, it provides TypeScript interfaces and type information for JavaScript libraries that don’t have them built-in.

When you’re using a JavaScript library in a TypeScript project, TypeScript needs to know the shapes and types of the library’s exports to type-check your code correctly.

It’s highly popular and has over 100 million weekly downloads!

This repo allows developers to use existing JavaScript libraries within their TypeScript projects seamlessly, ensuring type safety.

If the library doesn’t provide its own types, you can likely find them in DefinitelyTyped. These type definitions are then used by the TypeScript compiler to understand the library’s structure, offering auto-completion, type checking, and other IDE features for a smoother development experience.

Example (Lodash)

Let’s say you’re working on a TypeScript project and decide to use Lodash, a popular utility library. Lodash itself is written in JavaScript and doesn’t include TypeScript definitions.

Here's how you can use Lodash with types in your project with the help of DefinitelyTyped:

First, install Lodash:

npm install lodash

Then, install the type definitions for Lodash from DefinitelyTyped:

npm install @types/lodash --save-dev

Now, you can use Lodash in your TypeScript file with full type support:

import _ from 'lodash';

// Example usage with full type support
let numArray: number[] = [1, 2, 3, 4, 5];
let sum: number = _.sum(numArray); // Lodash's sum function

console.log(sum); // Output will be 15

By using DefinitelyTyped's type definitions, you can maintain type safety and take full advantage of TypeScript's features, even when using JavaScript libraries.

2. Type Challenges — A collection of TypeScript puzzles

The repo provides a collection of TypeScript puzzles similar to LeetCode problems.

Each challenge focuses on a specific aspect of TypeScript’s type system, from basic concepts to complex type manipulation. You’ll often need to use generic types or apply advanced features like conditional types and mapped types to solve them.

type-challenges / type-challenges

Collection of TypeScript type challenges with online judge

Collection of TypeScript type challenges

English | 简体中文 | 日本語 | 한국어 | Português

Intro

~~by the power of TypeScript's well-known Turing Completed type system~~

High-quality types can help improve projects' maintainability while avoiding potential bugs.

This project is aimed at helping you better understand how the type system works, writing your own utilities, or just having fun with the challenges. We are also trying to form a community where you can ask questions and get answers you have faced in the real world - they may become part of the challenges!

You can also use some existing type utility libraries such as type-fest. There are also some stale packages (not actively maintained) that you could reference, like utility-types, ts-toolbelt, SimplyTyped.

Challenges

Click the following badges to see details of the challenges.

Note: Challenges work in the strict mode.

By Tags

By Plain Text…

View on GitHub

These exercises will help you to test and improve your understanding of TypeScript’s type system.

Example: First of Array

Let’s look at the “Easy” level challenge. The task is to construct a generic type First that takes an array and returns its first element:

type arr1 = ['a', 'b', 'c']
type arr2 = [3, 2, 1]

type head1 = First<arr1> // expected to be 'a'
type head2 = First<arr2> // expected to be 3

To solve the task you can use the Typescript playground

The solution might be something like this:

type First<T extends any[]> = T extends [infer First, ...infer Rest] ? First : never

T extends any[]: We constrain the input T to be an array.
T extends [infer First, ...infer Rest]: Pattern matching to extract the first element as First and the rest of the array as Rest.
First : never: Conditional type for handling empty arrays.

The Type Challenges repository provides a structured and engaging way to level up your TypeScript mastery.

3. Utility types — A collection of pre-written utility types

This collection of pre-written utility types saves you time and effort when working with different data types in TypeScript.

This repository offers a collection of utility types that can be applied in various TypeScript projects.

piotrwitek / utility-types

Collection of utility types, complementing TypeScript built-in mapped types and aliases (think "lodash" for static types).

utility-types

Collection of utility types, complementing TypeScript built-in mapped types and aliases (think "lodash" for static types).

Found it useful? Want more updates?

Show your support by giving a ⭐

What's new?

🎉 Added new utilities 🎉

Features

Providing a set of Common Types for TypeScript projects that are idiomatic and complementary to existing TypeScript Mapped Types so you don't need to copy them between the projects.
Providing a set of Additional Types compatible with Flow's Utility Types to allow much easier migration to TypeScript.

Goals

Quality - thoroughly tested for type correctness with type-testing library dts-jest
Secure and minimal - no third-party dependencies
No runtime cost - it's type-level only

Installation

# NPM
npm install utility-types

# YARN
yarn add utility-types

Compatibility Notes

TypeScript support

v3.x.x - TypeScript v3.1+
v2.x.x - TypeScript v2.8.1+
v1.x.x - TypeScript v2.7.2+

Funding Issues

Utility-Types is an open-source project created by people investing…

View on GitHub

These types exist solely at compile time, leaving no runtime cost in your final JavaScript code.

Example: TypeScript Typeguard isPrimitive

First, install utility-types:

npm install utility-types

Let's use isPrimitive Typeguard example - a TypeScript Typeguard for the Primitive type
This can be useful to control the type of a parameter as the program flows.

import { Primitive, isPrimitive } from 'utility-types';

const consumer = (param: Primitive[] | Primitive): string => {
    if (isPrimitive(param)) {
        // typeof param === Primitive
        return String(param) + ' was Primitive';
    }
    // typeof param === Primitive[]
    const resultArray = param
        .map(consumer)
        .map(rootString => '\n\t' + rootString);
    return resultArray.reduce((comm, newV) => comm + newV, 'this was nested:');
};

Primitive: This type represents the basic building blocks of JavaScript and TypeScript values: strings, numbers, booleans, etc.
isPrimitive: This type guard function lets you dynamically check if a given variable is a primitive type. This is especially valuable when working with data that could have varying structures.

Benefits of using utility-types:

Cleaner code: The isPrimitive type guard avoids manual typeof checks and potential branching.
Type safety: It ensures that we're only manipulating primitive values within the appropriate code block.

4. Typescript book — Open-source e-book

Free and open-source e-book that dives deeply into TypeScript's features. Perfect if you prefer a traditional book-like format.

You can freely access, read, and even contribute to the book’s content.

basarat / typescript-book

📚 The definitive guide to TypeScript and possibly the best TypeScript book 📖. Free and Open Source 🌹

TypeScript Deep Dive

Learn Professional TypeScript. I've been looking at the issues that turn up commonly when people start using TypeScript. This is based on the lessons from Stack Overflow / DefinitelyTyped and general engagement with the TypeScript community. You can follow for updates and don't forget to ★ on GitHub 🌹

Reviews

Thanks for the wonderful book. Learned a lot from it. (link)
Its probably the Best TypeScript book out there. Good Job (link)
Love how precise and clear the examples and explanations are! (link)
For the low, low price of free, you get pages of pure awesomeness. Chock full of source code examples and clear, concise explanations, TypeScript Deep Dive will help you learn TypeScript development. (link)
Just a big thank you! Best TypeScript 2 detailed explanation! (link)
This gitbook got my project going pronto. Fluent easy…

View on GitHub

The book is known for its easy-to-understand explanations and illustrative examples. This makes it suitable for both beginners and experienced programmers who want to improve their TypeScript knowledge.

The book describes various aspects of TypeScript, from its core concepts and syntax to advanced topics like generics, decorators, and metaprogramming.

5. tRPC.io — End-to-end typesafe API

tRPC offers a solution for building modern APIs with a focus on type safety and developer experience. This open-source project provides tools and libraries needed to construct type-safe APIs.

basarat / typescript-book

📚 The definitive guide to TypeScript and possibly the best TypeScript book 📖. Free and Open Source 🌹

TypeScript Deep Dive

Reviews

Thanks for the wonderful book. Learned a lot from it. (link)
Its probably the Best TypeScript book out there. Good Job (link)
Love how precise and clear the examples and explanations are! (link)
For the low, low price of free, you get pages of pure awesomeness. Chock full of source code examples and clear, concise explanations, TypeScript Deep Dive will help you learn TypeScript development. (link)
Just a big thank you! Best TypeScript 2 detailed explanation! (link)
This gitbook got my project going pronto. Fluent easy…

View on GitHub

tRPC integrates seamlessly with popular web frameworks such as React, Next.js and Express.js.

Type-safe APIs have several advantages:

Reduced Errors: Static type checking helps catch potential errors at development time, preventing runtime issues that can be difficult to debug later.
Improved Maintainability: A type-safe API provides a clear understanding of the data structures and interactions involved.
Enhanced Developer Experience: Autocompletion and other IDE features powered by static types can significantly improve development speed and overall developer satisfaction.

Becoming a true TypeScript expert takes time and practice. These resources will help you build a strong start. Keep learning, keep trying new things, and connect with other TypeScript developers to take your skills to the next level.

Check out my other articles on TypeScript:

This article was originally posted on Medium.

Building a File Storage With Next.js, PostgreSQL, and Minio S3

Alex — Fri, 02 Feb 2024 20:57:51 +0000

It is the second part of a series of articles about building file storage with Next.js, PostgreSQL, and Minio S3. In the first part, we have set up the development environment using Docker Compose.

In this part, we will build a full-stack application using Next.js, PostgreSQL, and Minio S3. We will build something like this:

Introduction

In the early days of web development, files like images and documents were stored on the web server along with the application code. However, with increasing user traffic and the need to store large files, cloud storage services like Amazon S3 have become the preferred way to store files.

Separating the storage of files from the web server provides several benefits, including:

Scalability and performance
Large file support (up to 5TB)
Cost efficiency
Separation of concerns

In this article, we will build an example of a file storage application using Next.js, PostgreSQL, and Minio S3. There are two main ways to upload files to S3 from Next.js:

Using API routes to upload and download files.
This is a simpler approach, but it has a limitation of 4MB, if you try to upload file more than 4MB, you will get a Next.js error "API Routes Response Size Limited to 4MB" Error in Next.js".
Using presigned URLs to get temporary access to upload files and then upload files directly from frontend to S3.
This approach is a little bit more complex, but it does not use resources on the Next.js server with file uploads.

Source Code

You can find the full source code for this tutorial on GitHub

Shared code for both approaches

Some code will be shared between the two approaches like UI components, database models, utility functions, and types.

Database schema

To save information about the uploaded files, we will create a File model in the database. In the schema.prisma file, add the following model:

model File {
    id           String   @id @default(uuid())
    bucket       String
    fileName     String   @unique
    originalName String
    createdAt    DateTime @default(now())
    size         Int
}

id - the unique identifier of the file in the database generated by the uuid() function.
bucket - the name of the bucket in S3 where the file is stored, in our case it will be the same for all files.
fileName - the name of the file in S3, it will be unique for each file. If users upload files with the same name, the new file will overwrite the old one.
originalName - the original name of the file that the user uploaded. We will use it to display the file name to the user when downloading the file.
createdAt - the date and time when the file was uploaded.
size - the size of the file in bytes.

After creating the model, we need to apply the changes to the database. We can do this using db push or db migrate command, the difference between the two commands is that db push will drop the database and recreate it, while db migrate will only apply the changes to the database. More information about the commands can be found in Prisma docs. In our case it doesn't matter which command we use, so we will use db push command.

Environment variables

If we use Docker Compose to run the application for testing and development, we can store environment variables in the compose file because there is no need to keep them secret. However, in production, we should store environment variables in a .env.
Here is an example of the .env file for AWS S3 and PostgreSQL. Replace the values with your own.

DATABASE_URL="postgresql://postgres:postgres@REMOTESERVERHOST:5432/myapp-db?schema=public"

S3_ENDPOINT="s3.amazonaws.com"
S3_ACCESS_KEY="AKIAIOSFODNN7EXAMPLE"
S3_SECRET_KEY="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
S3_BUCKET_NAME="my-bucket"

For using Google Cloud Storage, the .env file will look like this:

S3_ENDPOINT="storage.googleapis.com"
...

Utility functions and types

Since we are using Minio S3 as the storage service, we need to install the Minio library to interact with S3. This library is compatible with any S3-compatible storage service, including Amazon S3, Google Cloud Storage, and others.

Install the Minio library using the following command:

npm install minio

Then, let's create a utility functions to upload files to Minio S3. Personally, I prefer to create a separate file for utility functions, so I will create a file s3-file-management.ts in the utils folder. Here we use included in T3 stack library env-nextjs to validate environment variables.

import * as Minio from 'minio'
import type internal from 'stream'
import { env } from '~/env.js'

// Create a new Minio client with the S3 endpoint, access key, and secret key
export const s3Client = new Minio.Client({
  endPoint: env.S3_ENDPOINT,
  port: env.S3_PORT ? Number(env.S3_PORT) : undefined,
  accessKey: env.S3_ACCESS_KEY,
  secretKey: env.S3_SECRET_KEY,
  useSSL: env.S3_USE_SSL === 'true',
})

export async function createBucketIfNotExists(bucketName: string) {
  const bucketExists = await s3Client.bucketExists(bucketName)
  if (!bucketExists) {
    await s3Client.makeBucket(bucketName)
  }
}

Main page with file list

The main page will contain the upload form and the list of uploaded files.

To get the list of files from the database, we will create a function fetchFiles that sends a GET request to the API route to get the list of files from the database.

And here is the full code of the main page:

import Head from 'next/head'
import { UploadFilesS3PresignedUrl } from '~/components/UploadFilesForm/UploadFilesS3PresignedUrl'
import { FilesContainer } from '~/components/FilesContainer'
import { useState, useEffect } from 'react'
import { type FileProps } from '~/utils/types'
import { UploadFilesRoute } from '~/components/UploadFilesForm/UploadFilesRoute'

export type fileUploadMode = 's3PresignedUrl' | 'NextjsAPIEndpoint'

export default function Home() {
  const [files, setFiles] = useState<FileProps[]>([])
  const [uploadMode, setUploadMode] = useState<fileUploadMode>('s3PresignedUrl')

  // Fetch files from the database
  const fetchFiles = async () => {
    const response = await fetch('/api/files')
    const body = (await response.json()) as FileProps[]
    // set isDeleting to false for all files after fetching
    setFiles(body.map((file) => ({ ...file, isDeleting: false })))
  }

  // fetch files on the first render
  useEffect(() => {
    fetchFiles().catch(console.error)
  }, [])

  // determine if we should download using presigned url or Nextjs API endpoint
  const downloadUsingPresignedUrl = uploadMode === 's3PresignedUrl'
  // handle mode change between s3PresignedUrl and NextjsAPIEndpoint
  const handleModeChange = (event: React.ChangeEvent<HTMLSelectElement>) => {
    setUploadMode(event.target.value as fileUploadMode)
  }

  return (
    <>
      <Head>
        <title>File Uploads with Next.js, Prisma, and PostgreSQL</title>
        <meta name='description' content='File Uploads with Next.js, Prisma, and PostgreSQL ' />
        <link rel='icon' href='/favicon.ico' />
      </Head>
      <main className='flex min-h-screen items-center justify-center gap-5 font-mono'>
        <div className='container flex flex-col gap-5 px-3'>
          <ModeSwitchMenu uploadMode={uploadMode} handleModeChange={handleModeChange} />
          {uploadMode === 's3PresignedUrl' ? (
            <UploadFilesS3PresignedUrl onUploadSuccess={fetchFiles} />
          ) : (
            <UploadFilesRoute onUploadSuccess={fetchFiles} />
          )}
          <FilesContainer
            files={files}
            fetchFiles={fetchFiles}
            setFiles={setFiles}
            downloadUsingPresignedUrl={downloadUsingPresignedUrl}
          />
        </div>
      </main>
    </>
  )
}

API route to get list of files from the database

To make request to the database, we need to create an API route. Create a file index.ts in the pages/api/files folder. This file will return the list of files from the database. For simplicity, we will not use pagination, we just get the 10 latest files from the database.
You can implement it using skip and take. More information about pagination can be found in the Prisma docs.

import type { NextApiRequest, NextApiResponse } from 'next'
import type { FileProps } from '~/utils/types'
import { db } from '~/server/db'

const LIMIT_FILES = 10

const handler = async (req: NextApiRequest, res: NextApiResponse) => {
  // Get the 10 latest files from the database
  const files = await db.file.findMany({
    take: LIMIT_FILES,
    orderBy: {
      createdAt: 'desc',
    },
    select: {
      id: true,
      originalName: true,
      size: true,
    },
  })
  // The database type is a bit different from the frontend type
  // Make the array of files compatible with the frontend type FileProps
  const filesWithProps: FileProps[] = files.map((file) => ({
    id: file.id,
    originalFileName: file.originalName,
    fileSize: file.size,
  }))

  return res.status(200).json(filesWithProps)
}

export default handler

Mode switch menu

The mode switch menu will allow the user to switch between the two approaches for uploading files.

export type ModeSwitchMenuProps = {
  uploadMode: fileUploadMode
  handleModeChange: (event: React.ChangeEvent<HTMLSelectElement>) => void
}
function ModeSwitchMenu({ uploadMode, handleModeChange }: ModeSwitchMenuProps) {
  return (
    <ul className='flex items-center justify-center gap-2'>
      <li>
        <label htmlFor='uploadMode'>Upload Mode:</label>
      </li>
      <li>
        <select
          className='rounded-md border-2 border-gray-300'
          id='uploadMode'
          value={uploadMode}
          onChange={handleModeChange}
        >
          <option value='s3PresignedUrl'>S3 Presigned Url</option>
          <option value='NextjsAPIEndpoint'>Next.js API Endpoint</option>
        </select>
      </li>
    </ul>
  )
}

File item UI

To display the files, we will create a component FileItem.tsx that will display the file name, size, and a delete button. Here is a simplified version of the file without functions to download and delete files. These functions will be added later.

import { type FileProps } from '~/utils/types'
import { LoadSpinner } from './LoadSpinner'
import { formatBytes } from '~/utils/fileUploadHelpers'

type FileItemProps = {
  file: FileProps
  fetchFiles: () => Promise<void>
  setFiles: (files: FileProps[] | ((files: FileProps[]) => FileProps[])) => void
  downloadUsingPresignedUrl: boolean
}

export function FileItem({ file, fetchFiles, setFiles, downloadUsingPresignedUrl }: FileItemProps) {
  return (
    <li className='relative flex items-center justify-between gap-2 border-b py-2 text-sm'>
      <button
        className='truncate text-blue-500 hover:text-blue-600 hover:underline  '
        onClick={() => downloadFile(file)}
      >
        {file.originalFileName}
      </button>

      <div className=' flex items-center gap-2'>
        <span className='w-32 '>{formatBytes(file.fileSize)}</span>

        <button
          className='flex w-full flex-1 cursor-pointer items-center justify-center
           rounded-md bg-red-500 px-4 py-2 text-white hover:bg-red-600
           disabled:cursor-not-allowed disabled:opacity-50'
          onClick={() => deleteFile(file.id)}
          disabled={file.isDeleting}
        >
          Delete
        </button>
      </div>

      {file.isDeleting && (
        <div className='absolute inset-0 flex items-center justify-center rounded-md bg-gray-900 bg-opacity-20'>
          <LoadSpinner size='small' />
        </div>
      )}
    </li>
  )
}

File container UI

To display the files, we will create a component FileContainer.tsx that will display the list of files using the FileItem component.

import { type FilesListProps } from '~/utils/types'
import { FileItem } from './FileItem'

export function FilesContainer({ files, fetchFiles, setFiles, downloadUsingPresignedUrl }: FilesListProps) {
  if (files.length === 0) {
    return (
      <div className='flex h-96 flex-col items-center justify-center '>
        <p className='text-xl'>No files uploaded yet</p>
      </div>
    )
  }

  return (
    <div className='h-96'>
      <h1 className='text-xl '>
        Last {files.length} uploaded file{files.length > 1 ? 's' : ''}
      </h1>
      <ul className='h-80 overflow-auto'>
        {files.map((file) => (
          <FileItem
            key={file.id}
            file={file}
            fetchFiles={fetchFiles}
            setFiles={setFiles}
            downloadUsingPresignedUrl={downloadUsingPresignedUrl}
          />
        ))}
      </ul>
    </div>
  )
}

Upload form UI

To upload files, we will create a form with a file input field. UploadFilesFormUI.tsx will contain the UI for the upload form which will be used in both approaches. Here is a simplified version of the file:

import Link from 'next/link'
import { LoadSpinner } from '../LoadSpinner'
import { type UploadFilesFormUIProps } from '~/utils/types'

export function UploadFilesFormUI({ isLoading, fileInputRef, uploadToServer, maxFileSize }: UploadFilesFormUIProps) {
  return (
    <form className='flex flex-col items-center justify-center gap-3' onSubmit={uploadToServer}>
      <h1 className='text-2xl'>File upload example using Next.js, MinIO S3, Prisma and PostgreSQL</h1>
      {isLoading ? (
        <LoadSpinner />
      ) : (
        <div className='flex h-16 gap-5'>
          <input
            id='file'
            type='file'
            multiple
            className='rounded-md border bg-gray-100 p-2 py-5'
            required
            ref={fileInputRef}
          />
          <button
            disabled={isLoading}
            className='m-2 rounded-md bg-blue-500 px-5 py-2 text-white
                hover:bg-blue-600  disabled:cursor-not-allowed disabled:bg-gray-400'
          >
            Upload
          </button>
        </div>
      )}
    </form>
  )
}

1.1 Upload files using Next.js API routes (4MB limit)

The diagram above shows the steps involved in uploading and downloading files using Next.js API routes.

To upload files:

User sends a POST request to the API route with the file to upload.
The API route uploads the file to S3 and returns the file name.
The file name is saved in the database.

Frontend - Upload form logic for API routes

First, we will create a UploadFilesRoute.tsx file with the logic for the upload form.

The algorithm for uploading files to the server is as follows:

The user selects files to upload, and the fileInputRef is updated with the selected files.
Form data is created from the selected files using the createFormData function and FormData API.
The form data is sent to the server using POST request to the /api/files/upload/smallFiles route.
The server uploads the files to S3 and returns status and message in the response.

It's usually a good idea to extract the logic of the UI component into a separate file. One way is to create hooks for the logic and use the hooks in the UI component, however, for simplicity, we will create a separate file for the logic "fileUploadHelpers.ts" and use it in the "UploadFilesRoute" component.

/**
 * Create form data from files
 * @param files files to upload
 * @returns form data
 */
export function createFormData(files: File[]): FormData {
  const formData = new FormData()
  files.forEach((file) => {
    formData.append('file', file)
  })
  return formData
}

Here is a simplified version, without validation, loading state and error handling:

import { useState, useRef } from 'react'
import { validateFiles, createFormData } from '~/utils/fileUploadHelpers'
import { MAX_FILE_SIZE_NEXTJS_ROUTE } from '~/utils/fileUploadHelpers'
import { UploadFilesFormUI } from './UploadFilesFormUI'

type UploadFilesFormProps = {
  onUploadSuccess: () => void
}

export function UploadFilesRoute({ onUploadSuccess }: UploadFilesFormProps) {
  const fileInputRef = useRef<HTMLInputElement | null>(null)

  const uploadToServer = async (event: React.FormEvent<HTMLFormElement>) => {
    event.preventDefault()

    const files = Object.values(fileInputRef.current?.files)

    const formData = createFormData(files)
    const response = await fetch('/api/files/upload/smallFiles', {
      method: 'POST',
      body: formData,
    })
    const body = (await response.json()) as {
      status: 'ok' | 'fail'
      message: string
    }
  }

  return (
    <UploadFilesFormUI
      isLoading={isLoading}
      fileInputRef={fileInputRef}
      uploadToServer={uploadToServer}
      maxFileSize={MAX_FILE_SIZE_NEXTJS_ROUTE}
    />
  )
}

Check the full code in the GitHub repository

Backend - Upload files using Next.js API routes

1. Create utility functions to upload files using Minio S3

To upload files to S3, we will create a utility function saveFileInBucket that uses the putObject method of the Minio client to upload the file to the S3 bucket. The function createBucketIfNotExists creates a bucket if it doesn't exist.

/**
 * Save file in S3 bucket
 * @param bucketName name of the bucket
 * @param fileName name of the file
 * @param file file to save
 */
export async function saveFileInBucket({
  bucketName,
  fileName,
  file,
}: {
  bucketName: string
  fileName: string
  file: Buffer | internal.Readable
}) {
  // Create bucket if it doesn't exist
  await createBucketIfNotExists(bucketName)

  // check if file exists - optional.
  // Without this check, the file will be overwritten if it exists
  const fileExists = await checkFileExistsInBucket({
    bucketName,
    fileName,
  })

  if (fileExists) {
    throw new Error('File already exists')
  }

  // Upload image to S3 bucket
  await s3Client.putObject(bucketName, fileName, file)
}

/**
 * Check if file exists in bucket
 * @param bucketName name of the bucket
 * @param fileName name of the file
 * @returns true if file exists, false if not
 */
export async function checkFileExistsInBucket({ bucketName, fileName }: { bucketName: string; fileName: string }) {
  try {
    await s3Client.statObject(bucketName, fileName)
  } catch (error) {
    return false
  }
  return true
}

2. Create an API route to upload files

Next, we will create an API route to handle file uploads. Create a file smallFiles.ts in the pages/api/files/upload folder. This file will do both the file upload and save the file name in the database.

To parse the incoming request, we will use the formidable library. Formidable is a Node.js module for parsing form data, especially file uploads.

The algorithm for uploading files to the server

Get files from the request using formidable.

Then, for each file:

Read the file from the file path using fs.createReadStream.
Generate a unique file name using the nanoid library.
Save the file to S3 using the saveFileInBucket function that invokes the putObject method of the Minio client.
Save the file info to the database using Prisma file.create method.
Return the status and message in the response to the client.

The file upload and saving the file info to the database will be done concurrently using Promise.all. Also consider using Promise.allSettled to handle errors in the file upload and saving the file info to the database.

If an error occurs during the file upload or saving the file info to the database, we will set the status to 500 and return an error message.

import type { NextApiRequest, NextApiResponse } from 'next'
import fs from 'fs'
import { IncomingForm, type File } from 'formidable'
import { env } from '~/env'
import { saveFileInBucket } from '~/utils/s3-file-management'
import { nanoid } from 'nanoid'
import { db } from '~/server/db'

const bucketName = env.S3_BUCKET_NAME

type ProcessedFiles = Array<[string, File]>

const handler = async (req: NextApiRequest, res: NextApiResponse) => {
  let status = 200,
    resultBody = { status: 'ok', message: 'Files were uploaded successfully' }

  // Get files from request using formidable
  const files = await new Promise<ProcessedFiles | undefined>((resolve, reject) => {
    const form = new IncomingForm()
    const files: ProcessedFiles = []
    form.on('file', function (field, file) {
      files.push([field, file])
    })
    form.on('end', () => resolve(files))
    form.on('error', (err) => reject(err))
    form.parse(req, () => {
      //
    })
  }).catch(() => {
    ;({ status, resultBody } = setErrorStatus(status, resultBody))
    return undefined
  })

  if (files?.length) {
    // Upload files to S3 bucket
    try {
      await Promise.all(
        files.map(async ([_, fileObject]) => {
          const file = fs.createReadStream(fileObject?.filepath)
          // generate unique file name
          const fileName = `${nanoid(5)}-${fileObject?.originalFilename}`
          // Save file to S3 bucket and save file info to database concurrently
          await saveFileInBucket({
            bucketName,
            fileName,
            file,
          })
          // save file info to database
          await db.file.create({
            data: {
              bucket: bucketName,
              fileName,
              originalName: fileObject?.originalFilename ?? fileName,
              size: fileObject?.size ?? 0,
            },
          })
        })
      )
    } catch (e) {
      console.error(e)
      ;({ status, resultBody } = setErrorStatus(status, resultBody))
    }
  }

  res.status(status).json(resultBody)
}

// Set error status and result body if error occurs
export function setErrorStatus(status: number, resultBody: { status: string; message: string }) {
  status = 500
  resultBody = {
    status: 'fail',
    message: 'Upload error',
  }
  return { status, resultBody }
}

// Disable body parser built-in to Next.js to allow formidable to work
export const config = {
  api: {
    bodyParser: false,
  },
}

export default handler

Remember to include export const config, this prevents built-in body parser of Next.js from parsing the request body, which allows formidable to work.

1.2 Download files using Next.js API routes (4MB limit)

To download files:

User sends a GET request to the API route with the file id to download.
The API route requests the file name from the database.
The API route downloads the file from S3.
The file is piped to the response object and returned to the client.

Frontend - Download files using Next.js API routes

To download files, we will create a function downloadFile inside of the FileItem component. The function sends a GET request to the API route to download the file from S3. The file is returned to the user from the API route.

const downloadFile = async (file: FileProps) => {
  window.open(`/api/files/download/smallFiles/${file.id}`, '_blank')
}

Backend - Download files using Next.js API routes

1. Create a utility function to download files from S3

To download files from S3, we will create a utility function getFileFromBucket that uses the getObject method of the Minio client to download the file from the S3 bucket.

/**
 * Get file from S3 bucket
 * @param bucketName name of the bucket
 * @param fileName name of the file
 * @returns file from S3
 */
export async function getFileFromBucket({ bucketName, fileName }: { bucketName: string; fileName: string }) {
  try {
    await s3Client.statObject(bucketName, fileName)
  } catch (error) {
    console.error(error)
    return null
  }
  return await s3Client.getObject(bucketName, fileName)
}

2. Create an API route to download files

To download files, we will create an API route to handle file downloads. Create a file [id].ts in the pages/api/files/download/ folder. This file will download the file from S3 and return it to the user.

Here we use a dynamic route of Next.js with [id] to get the file id from the request query. More information about dynamic routes can be found in the Next.js docs.

The algorithm for downloading files from the server is as follows:

Get the file name and original name from the database using Prisma file.findUnique method.
Get the file from the S3 bucket using the getFileFromBucket function.
Set the header for downloading the file.
Pipe the file to the response object.

import { type NextApiRequest, type NextApiResponse } from 'next'
import { getFileFromBucket } from '~/utils/s3-file-management'
import { env } from '~/env'
import { db } from '~/server/db'

async function handler(req: NextApiRequest, res: NextApiResponse) {
  const { id } = req.query
  if (typeof id !== 'string') return res.status(400).json({ message: 'Invalid request' })

  // get the file name and original name from the database
  const fileObject = await db.file.findUnique({
    where: {
      id,
    },
    select: {
      fileName: true,
      originalName: true,
    },
  })
  if (!fileObject) {
    return res.status(404).json({ message: 'Item not found' })
  }
  // get the file from the bucket and pipe it to the response object
  const data = await getFileFromBucket({
    bucketName: env.S3_BUCKET_NAME,
    fileName: fileObject?.fileName,
  })

  if (!data) {
    return res.status(404).json({ message: 'Item not found' })
  }
  // set header for download file
  res.setHeader('content-disposition', `attachment; filename="${fileObject?.originalName}"`)

  // pipe the data to the res object
  data.pipe(res)
}

export default handler

2.1 Upload files using presigned URLs

In the diagram above, we can see the steps involved in uploading and downloading files using presigned URLs. It is a more complex approach, but it does not use resources on the Next.js server with file uploads. The presigned URL is generated on the server and sent to the client. The client uses the presigned URL to upload the file directly to S3.

To upload files:

The user sends a POST request to the API route with the file info to upload.
The API route sends requests to S3 to generate presigned URLs for each file.
The S3 returns the presigned URLs to the API route.
The API route sends the presigned URLs to the client.
The client uploads the files directly to S3 using the presigned URLs and PUT requests.
The client sends the file info to the API route to save the file info.
The API route saves the file info to the database.

Frontend - Upload form logic for presigned URLs

1. Create function to send request to Next.js API route to get presigned URLs

/**
 * Gets presigned urls for uploading files to S3
 * @param formData form data with files to upload
 * @returns
 */
export const getPresignedUrls = async (files: ShortFileProp[]) => {
  const response = await fetch('/api/files/upload/presignedUrl', {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
    },
    body: JSON.stringify(files),
  })
  return (await response.json()) as PresignedUrlProp[]
}

2. Create function to upload files using PUT request to S3 with presigned URL

The function uploadToS3 sends a PUT request to the presigned URL to upload the file to S3.

/**
 * Uploads file to S3 directly using presigned url
 * @param presignedUrl presigned url for uploading
 * @param file  file to upload
 * @returns  response from S3
 */
export const uploadToS3 = async (presignedUrl: PresignedUrlProp, file: File) => {
  const response = await fetch(presignedUrl.url, {
    method: 'PUT',
    body: file,
    headers: {
      'Content-Type': file.type,
      'Access-Control-Allow-Origin': '*',
    },
  })
  return response
}

3. Create function to save file info in the database

The function saveFileInfoInDB sends a POST request to the API route to save the file info in the database.

/**
 * Saves file info in DB
 * @param presignedUrls presigned urls for uploading
 * @returns
 */
export const saveFileInfoInDB = async (presignedUrls: PresignedUrlProp[]) => {
  return await fetch('/api/files/upload/saveFileInfo', {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
    },
    body: JSON.stringify(presignedUrls),
  })
}

4. Create a form to upload files using presigned URLs

/**
 * Uploads files to S3 and saves file info in DB
 * @param files files to upload
 * @param presignedUrls  presigned urls for uploading
 * @param onUploadSuccess callback to execute after successful upload
 * @returns
 */
export const handleUpload = async (files: File[], presignedUrls: PresignedUrlProp[], onUploadSuccess: () => void) => {
  const uploadToS3Response = await Promise.all(
    presignedUrls.map((presignedUrl) => {
      const file = files.find(
        (file) => file.name === presignedUrl.originalFileName && file.size === presignedUrl.fileSize
      )
      if (!file) {
        throw new Error('File not found')
      }
      return uploadToS3(presignedUrl, file)
    })
  )

  if (uploadToS3Response.some((res) => res.status !== 200)) {
    alert('Upload failed')
    return
  }

  await saveFileInfoInDB(presignedUrls)
  onUploadSuccess()
}

5. Create a form to upload files using presigned URLs using functions from the previous steps

Here I show a simplified version of the file, without validation, loading state, and error handling.

import { useState, useRef } from 'react'
import { validateFiles, MAX_FILE_SIZE_S3_ENDPOINT, handleUpload, getPresignedUrls } from '~/utils/fileUploadHelpers'
import { UploadFilesFormUI } from './UploadFilesFormUI'
import { type ShortFileProp } from '~/utils/types'

type UploadFilesFormProps = {
  onUploadSuccess: () => void
}

export function UploadFilesS3PresignedUrl({ onUploadSuccess }: UploadFilesFormProps) {
  const fileInputRef = useRef<HTMLInputElement | null>(null)
  const [isLoading, setIsLoading] = useState(false)

  const uploadToServer = async (event: React.FormEvent<HTMLFormElement>) => {
    event.preventDefault()
    // get File[] from FileList
    const files = Object.values(fileInputRef.current.files)
    // validate files
    const filesInfo: ShortFileProp[] = files.map((file) => ({
      originalFileName: file.name,
      fileSize: file.size,
    }))

    const presignedUrls = await getPresignedUrls(filesInfo)

    // upload files to s3 endpoint directly and save file info to db
    await handleUpload(files, presignedUrls, onUploadSuccess)

    setIsLoading(false)
  }

  return (
    <UploadFilesFormUI
      isLoading={isLoading}
      fileInputRef={fileInputRef}
      uploadToServer={uploadToServer}
      maxFileSize={MAX_FILE_SIZE_S3_ENDPOINT}
    />
  )
}

Backend - Upload files using presigned URLs

1. Create helper function to generate presigned URLs for uploading files to S3

/**
 * Generate presigned urls for uploading files to S3
 * @param files files to upload
 * @returns promise with array of presigned urls
 */
export async function createPresignedUrlToUpload({
  bucketName,
  fileName,
  expiry = 60 * 60, // 1 hour
}: {
  bucketName: string
  fileName: string
  expiry?: number
}) {
  // Create bucket if it doesn't exist
  await createBucketIfNotExists(bucketName)

  return await s3Client.presignedPutObject(bucketName, fileName, expiry)
}

2. Create an API route to send requests to S3 to generate presigned URLs for each file

In this approach, we do not use formidable to parse the incoming request, so we do not need to disable the built-in body parser of Next.js. We can use the default body parser.

The algorithm for generating presigned URLs for uploading files to S3:

Get the files info from the request body.
Check if there are files to upload.
Create an empty array to store the presigned URLs.

For each file:

Generate a unique file name using the nanoid library.
Get the presigned URL using the createPresignedUrlToUpload function.
Add the presigned URL to the array.

Then return the array of presigned URLs in the response to the client.

import type { NextApiRequest, NextApiResponse } from 'next'
import type { ShortFileProp, PresignedUrlProp } from '~/utils/types'
import { createPresignedUrlToUpload } from '~/utils/s3-file-management'
import { env } from '~/env'
import { nanoid } from 'nanoid'

const bucketName = env.S3_BUCKET_NAME
const expiry = 60 * 60 // 24 hours

export default async function handler(req: NextApiRequest, res: NextApiResponse) {
  if (req.method !== 'POST') {
    res.status(405).json({ message: 'Only POST requests are allowed' })
    return
  }
  // get the files from the request body
  const files = req.body as ShortFileProp[]

  if (!files?.length) {
    res.status(400).json({ message: 'No files to upload' })
    return
  }

  const presignedUrls = [] as PresignedUrlProp[]

  if (files?.length) {
    // use Promise.all to get all the presigned urls in parallel
    await Promise.all(
      // loop through the files
      files.map(async (file) => {
        const fileName = `${nanoid(5)}-${file?.originalFileName}`

        // get presigned url using s3 sdk
        const url = await createPresignedUrlToUpload({
          bucketName,
          fileName,
          expiry,
        })
        // add presigned url to the list
        presignedUrls.push({
          fileNameInBucket: fileName,
          originalFileName: file.originalFileName,
          fileSize: file.fileSize,
          url,
        })
      })
    )
  }

  res.status(200).json(presignedUrls)
}

3. Create an API route to save file info in the database with Prisma

The algorithm for saving file info in the database:

Get the file info from the request body.
Save the file info to the database using the Prisma file.create method.
Return the status and message in the response to the client.

import type { NextApiRequest, NextApiResponse } from 'next'
import { env } from '~/env'
import { db } from '~/server/db'
import type { PresignedUrlProp, FileInDBProp } from '~/utils/types'

export default async function handler(req: NextApiRequest, res: NextApiResponse) {
  if (req.method !== 'POST') {
    res.status(405).json({ message: 'Only POST requests are allowed' })
    return
  }

  const presignedUrls = req.body as PresignedUrlProp[]

  // Get the file name in bucket from the database
  const saveFilesInfo = await db.file.createMany({
    data: presignedUrls.map((file: FileInDBProp) => ({
      bucket: env.S3_BUCKET_NAME,
      fileName: file.fileNameInBucket,
      originalName: file.originalFileName,
      size: file.fileSize,
    })),
  })

  if (saveFilesInfo) {
    res.status(200).json({ message: 'Files saved successfully' })
  } else {
    res.status(404).json({ message: 'Files not found' })
  }
}

2.2 Download files using presigned URLs

To download files:

The user sends a GET request with file id to the API route to get file.
The API route sends a request to the database to get the file name and receives the file name.
The API route sends a request to S3 to generate a presigned URL for the file and receives the presigned URL.
The API route sends the presigned URL to the client.
The client downloads the file directly from S3 using the presigned URL.

Frontend - Download files using presigned URLs

To download files, we will create a function downloadFile inside of the FileItem component. The function sends a GET request to the API route to get the presigned URL for the file from S3. The file is returned to the user from the API route.

async function getPresignedUrl(file: FileProps) {
  const response = await fetch(`/api/files/download/presignedUrl/${file.id}`)
  return (await response.json()) as string
}

const downloadFile = async (file: FileProps) => {
  const presignedUrl = await getPresignedUrl(file)
  window.open(presignedUrl, '_blank')
}

Backend - Download files using presigned URLs

1. Create helper function to generate presigned URLs for downloading files from S3

To download files from S3, we will create a utility function createPresignedUrlToDownload that uses the presignedGetObject method of the Minio client to generate a presigned URL for the file.

export async function createPresignedUrlToDownload({
  bucketName,
  fileName,
  expiry = 60 * 60, // 1 hour
}: {
  bucketName: string
  fileName: string
  expiry?: number
}) {
  return await s3Client.presignedGetObject(bucketName, fileName, expiry)
}

2. Create an API route to send requests to S3 to generate presigned URLs for each file

The algorithm for generating presigned URLs for downloading files from S3:

Get the file name from the database using the file id.
Get the presigned URL using the createPresignedUrlToDownload function.
Return the presigned URL in the response to the client.

import type { NextApiRequest, NextApiResponse } from 'next'
import { createPresignedUrlToDownload } from '~/utils/s3-file-management'
import { db } from '~/server/db'
import { env } from '~/env'

/**
 * This route is used to get presigned url for downloading file from S3
 */
export default async function handler(req: NextApiRequest, res: NextApiResponse) {
  if (req.method !== 'GET') {
    res.status(405).json({ message: 'Only GET requests are allowed' })
  }

  const { id } = req.query

  if (!id || typeof id !== 'string') {
    return res.status(400).json({ message: 'Missing or invalid id' })
  }

  // Get the file name in bucket from the database
  const fileObject = await db.file.findUnique({
    where: {
      id,
    },
    select: {
      fileName: true,
    },
  })

  if (!fileObject) {
    return res.status(404).json({ message: 'Item not found' })
  }

  // Get presigned url from s3 storage
  const presignedUrl = await createPresignedUrlToDownload({
    bucketName: env.S3_BUCKET_NAME,
    fileName: fileObject?.fileName,
  })

  res.status(200).json(presignedUrl)
}

3. Delete files from S3

Frontend - Delete files from S3

File deletion can be done using a DELETE request to the API route. I created a delete function in FileItem component, which sends a DELETE request to the API route to delete the file from the S3 bucket and the database.

The algorithm for deleting files from S3:

Remove the file from the list of files on the client immediately.
Send a DELETE request to the API route to delete the file from the S3 bucket and the database.
Fetch the files after deleting.

Here is an example of the delete function in the FileItem component:

async function deleteFile(id: string) {
  // remove file from the list of files on the client
  setFiles((files: FileProps[]) =>
    files.map((file: FileProps) => (file.id === id ? { ...file, isDeleting: true } : file))
  )
  try {
    // delete file request to the server
    await fetch(`/api/files/delete/${id}`, {
      method: 'DELETE',
    })
    // fetch files after deleting
    await fetchFiles()
  } catch (error) {
    console.error(error)
    alert('Failed to delete file')
  } finally {
    // remove isDeleting flag from the file
    setFiles((files: FileProps[]) =>
      files.map((file: FileProps) => (file.id === id ? { ...file, isDeleting: false } : file))
    )
  }
}

Backend - Delete files from S3

1. Create a utility function to delete files from S3

To delete files from S3, we will create a utility function deleteFileFromBucket that uses the removeObject method of the Minio client to delete the file from the S3 bucket.

/**
 * Delete file from S3 bucket
 * @param bucketName name of the bucket
 * @param fileName name of the file
 * @returns true if file was deleted, false if not
 */
export async function deleteFileFromBucket({ bucketName, fileName }: { bucketName: string; fileName: string }) {
  try {
    await s3Client.removeObject(bucketName, fileName)
  } catch (error) {
    console.error(error)
    return false
  }
  return true
}

2. Create an API route to delete files from S3

Here is an example of an API route to delete files from S3. Create a file delete/[id].ts in the pages/api/files/delete folder. This file will delete the file from the S3 bucket and the database.

The algorithm for deleting files from S3:

Get the file name in the bucket from the database using the file id.
Check if the file exists in the database.
Delete the file from the S3 bucket using the deleteFileFromBucket function.
Delete the file from the database using the Prisma file.delete method.
Return the status and message in the response to the client.

import type { NextApiRequest, NextApiResponse } from 'next'
import { deleteFileFromBucket } from '~/utils/s3-file-management'
import { db } from '~/server/db'
import { env } from '~/env'

export default async function handler(req: NextApiRequest, res: NextApiResponse) {
  if (req.method !== 'DELETE') {
    res.status(405).json({ message: 'Only DELETE requests are allowed' })
  }
  const { id } = req.query

  if (!id || typeof id !== 'string') {
    return res.status(400).json({ message: 'Missing or invalid id' })
  }

  // Get the file name in bucket from the database
  const fileObject = await db.file.findUnique({
    where: {
      id,
    },
    select: {
      fileName: true,
    },
  })

  if (!fileObject) {
    return res.status(404).json({ message: 'Item not found' })
  }
  // Delete the file from the bucket
  await deleteFileFromBucket({
    bucketName: env.S3_BUCKET_NAME,
    fileName: fileObject?.fileName,
  })
  // Delete the file from the database
  const deletedItem = await db.file.delete({
    where: {
      id,
    },
  })

  if (deletedItem) {
    res.status(200).json({ message: 'Item deleted successfully' })
  } else {
    res.status(404).json({ message: 'Item not found' })
  }
}

Deploy locally using Docker Compose

To deploy the application locally, we will use Docker Compose to run the Next.js app, PostgreSQL, and MinIO S3. I explained how to set up the Docker Compose file in the previous article.

Here I want to mention some changes to the Docker Compose file we need to make because we need to use presigned URLs.

The key point is - when presigned url is created inside the docker container, it will have the same host as we set in the docker compose file. For example, if we have environment: - S3_ENDPOINT=minio in the docker compose file, the presigned url will have the host minio. All presigned urls will be like http://minio:9000/bucket-name/file-name.
These urls will not work on the client side (if we do not add minio to the hosts file). Here we cannot use localhost either, because localhost will be the host of the container, not the host of the client.

The solution is to use the kubernetes.docker.internal as the Minio S3 endpoint. This is a special DNS name that resolves to the host machine from inside a Docker container. It is available on Docker for Mac and Docker for Windows.

Also make sure that kubernetes.docker.internal is in the hosts file (it should be there by default). Then the presigned urls will be like http://kubernetes.docker.internal:9000/bucket-name/file-name and will work on the client side.

Here is the full Docker Compose file:

version: '3.9'
name: nextjs-postgres-s3minio
services:
  web:
    container_name: nextjs
    build:
      context: ../
      dockerfile: compose/web.Dockerfile
      args:
        NEXT_PUBLIC_CLIENTVAR: 'clientvar'
    ports:
      - 3000:3000
    volumes:
      - ../:/app
    environment:
      - DATABASE_URL=postgresql://postgres:postgres@db:5432/myapp-db?schema=public
      - S3_ENDPOINT=kubernetes.docker.internal
      - S3_PORT=9000
      - S3_ACCESS_KEY=minio
      - S3_SECRET_KEY=miniosecret
      - S3_BUCKET_NAME=s3bucket
    depends_on:
      - db
      - minio
    # Optional, if you want to apply db schema from prisma to postgres
    command: sh ./compose/db-push-and-start.sh
  db:
    image: postgres:15.3
    container_name: postgres
    ports:
      - 5432:5432
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
      POSTGRES_DB: myapp-db
    volumes:
      - postgres-data:/var/lib/postgresql/data
    restart: unless-stopped
  minio:
    container_name: s3minio
    image: bitnami/minio:latest
    ports:
      - '9000:9000'
      - '9001:9001'
    volumes:
      - minio_storage:/data
volumes:
  postgres-data:
  minio_storage:

To run the application, use the following command:

docker-compose -f compose/docker-compose.yml --env-file .env up

After you run the application, you can access it at http://localhost:3000.
The Minio S3 will be available at http://localhost:9000. You can use the access key minio and the secret key miniosecret to log in.

The next step would be to deploy the application to a cloud provider. I will cover this in the next article.

You can check the live demo of the application here. It is shared with the public, so all files are visible to everyone. You can upload, download, and delete any files. Please do not upload any sensitive information.

Conclusion

In this article, we learned how to upload and download files using Next.js, PostgreSQL, and Minio S3. We also learned how to use presigned URLs to upload and download files directly from the client to S3. We created an application to upload, download, and delete files from S3 using presigned URLs. We also learned how to deploy the application locally using Docker Compose.

Hope you found this article helpful. If you have any questions or suggestions, feel free to leave a comment.

References and Further Reading

I'm always open to making new connections! Feel free to connect with me on LinkedIn

Building a Local Development Environment: Running a Next.js Full-Stack App with PostgreSQL and Minio S3 Using Docker

Alex — Sat, 06 Jan 2024 22:37:13 +0000

Introduction
Prerequisites
Building a local development environment
- 1. Create a Next.js application
- 2. Configure Next.js to work with Docker
- 3. Add .dockerignore file
- 4. Configure Prisma to work with Docker
- 5. Create a Dockerfile for the Next.js application
- 6. Create a docker-compose.yml file
Run the application
Conclusion
References and further reading

Introduction

As a developer working on a full-stack application, you need to have a local development environment that is as close as possible to the production environment. It will allow you to test and debug your application locally before deploying it to production.

Almost every full-stack application needs a database and a file storage so let's build a basic full-stack application that can save and retrieve data from a database and upload and download files from a file storage.

You can run your own PostgreSQL and Minio S3 server locally, or even use a cloud service like AWS RDS and S3. But it will take some time to set up and configure. Using docker-compose will make it super easy to set up a local development environment for your full-stack application. After you test it locally, all you need to switch to the production environment is to change the environment variables.

Additionally, you can test your application end-to-end (for example, using Cypress) in a local environment that is as close as possible to the production environment. Having pre-configured docker-compose file will make it easy to set up a CI/CD pipeline like GitHub Actions or GitLab CI.

Once you have a docker-compose file, you and your team can use it to set up the same local development environment on any machine in a few minutes with just one command. Overall, it will save you a lot of time and make your life easier.

In this article, we will look at how to build a local development environment for a full-stack Next.js application with Prisma ORM connected to PostgreSQL as a database and Minio S3 as a file storage using Docker-Compose.

You can find the full source code for this tutorial on GitHub

When you are ready to deploy your application to production, you can use any type of database:

Supabase (I love this one)
Vercel Postgres
AWS RDS
Google Cloud SQL
Azure Database for PostgreSQL
Heroku Postgres
DigitalOcean Managed Databases
ScaleGrid
...

The same goes for the file storage, you need one that is compatible with the S3 protocol:

AWS S3
Google Cloud Storage (I tested it, it works)
Wasabi (One of the cheapest options)
Backblaze B2
DigitalOcean Spaces
...

Prerequisites

To follow this tutorial, you need to have Docker and Docker-Compose installed on your machine. You can find the instructions on how to install Docker and Docker-Compose on the official Docker website.

When I firstly faced the task of setting up a local development environment for Next.js, Prisma and PostgreSQL, I tried to use T3 Docker tutorial but it didn't work for me. However, I use it as a starting point for this tutorial.

Building a local development environment

1. Create a Next.js application

Let's start by creating a Next.js application. We will use the T3 stack (TypeScript, TailwindCSS, and Prisma ORM) for this tutorial to skip installing and configuring all the dependencies which is out of the scope of this article. You can find more information about the T3 stack.

Run the following command to create a new Next.js:

npm create t3-app@latest

After you run the command, you will be asked several questions:

   ___ ___ ___   __ _____ ___   _____ ____    __   ___ ___
  / __| _ \ __| /  \_   _| __| |_   _|__ /   /  \ | _ \ _ \
 | (__|   / _| / /\ \| | | _|    | |  |_ \  / /\ \|  _/  _/
  \___|_|_\___|_/‾‾\_\_| |___|   |_| |___/ /_/‾‾\_\_| |_|

◇  What will your project be called?
│  local-nextjs-postgres-s3
│
◇  Will you be using TypeScript or JavaScript?
│  TypeScript
│
◇  Will you be using Tailwind CSS for styling?
│  Yes
│
◇  Would you like to use tRPC?
│  No
│
◇  What authentication provider would you like to use?
│  None
│
◇  What database ORM would you like to use?
│  Prisma
│
◇   EXPERIMENTAL  Would you like to use Next.js App Router?
│  No
│
◇  Should we initialize a Git repository and stage the changes?
│  Yes
│
◇  Should we run 'npm install' for you?
│  Yes
│
◇  What import alias would you like to use?
│  ~/

2. Configure Next.js to work with Docker

According to the Next.js documentation

Next.js can automatically create a standalone folder that copies only the necessary files for a production deployment including select files in node_modules.

To reduce image size we need to add output: "standalone" in the next.config.js file.
The next.config.js file should look like this:

const config = {
  reactStrictMode: true,
  output: "standalone",
  // ...
};

3. Add .dockerignore file

I prefer having a separate folder for the docker files, so I created a folder called compose in the root of the project. We need to add a .dockerignore file to this folder to exclude unnecessary files from the Docker image. The .dockerignore file should look like this:

.env
Dockerfile
.dockerignore
.next
.git
.gitignore
node_modules
npm-debug.log
README.md

4. Configure Prisma to work with Docker

In the prisma/schema.prisma file, we need to

Change the provider from sqlite to postgresql:
Add binaryTargets to the generator block. It will allow us to use the Prisma CLI inside the Docker container. You need to your binaryTargets specific to your OS and architecture. For example, for M1 Mac, I use "linux-musl-arm64-openssl-3.0.x". To support other OS and architectures, you need to add them to the binaryTargets array. More about binaryTargets in the Prisma documentation.

I want to use the same schema.prisma file for local development on machines with different OS and architectures and also for CI/CD pipelines on GitHub Actions. So in my case, the prisma/schema.prisma file looks like this:

generator client {
    provider      = "prisma-client-js"
    binaryTargets = ["native", "linux-musl-arm64-openssl-3.0.x", "linux-musl-openssl-3.0.x", "rhel-openssl-1.0.x"]
}

datasource db {
  provider = "postgresql"
  url      = env("DATABASE_URL")
}

5. Create a Dockerfile for the Next.js application

Inside the compose folder, create a file called web.Dockerfile with the following content:

FROM node:18-alpine

RUN mkdir app
COPY ../prisma  ./app
COPY ../package.json ../package-lock.json ./app
WORKDIR /app

RUN npm ci

CMD ["npm", "run", "dev"]

We will use the node:18-alpine image as a base image. It is a lightweight image that contains Node.js 18 and npm. We will copy the /prisma folder, package.json and package-lock.json files to the /app folder and run npm ci to install the dependencies.

Using 'clean install' (npm ci) instead of 'install' (npm i) is a good practice for Docker images. It will ensure that the dependencies are installed from the package-lock.json file and not from the node_modules cache. This is faster than 'install', which is especially important for CI/CD pipelines where you want to keep the build time as short as possible.

6. Create a docker compose file

Docker compose file is used to define and run multi-container Docker applications with a single command docker-compose up.

Here we will not go into details about docker-compose files. In general our docker-compose file creates 3 services: web (Next.js application built with our Dockerfile), db (PostgreSQL database), and minio (Minio S3 file storage). Remember to add volumes for the database and file storage services. Otherwise, the data will be lost when you stop the containers.

It is generally not recommended to store environment variables in the docker-compose file. However, in this particular scenario, for educational purposes and given that we are exclusively using it for local development and testing, it looks acceptable.
If you do not want to store secrets in the docker-compose file, you should use a .env file and use ${VARIABLE_NAME} syntax to reference the variables. More about environment variables in docker-compose files Docker compose file reference.

Inside the compose folder, create a file called docker-compose.yml with the following content:

version: "3.9"
name: nextjs-postgres-s3minio
services:
  web:
    container_name: nextjs
    build:
      context: ../
      dockerfile: compose/web.Dockerfile
      args:
        NEXT_PUBLIC_CLIENTVAR: "clientvar"
    ports:
      - 3000:3000
    volumes:
      - ../:/app
    environment:
      - DATABASE_URL=postgresql://postgres:postgres@db:5432/myapp-db?schema=public
      - S3_ENDPOINT=minio
      - S3_PORT=9000
      - S3_ACCESS_KEY=minio
      - S3_SECRET_KEY=miniosecret
      - S3_BUCKET_NAME=s3bucket
    depends_on:
      - db
      - minio
  db:
    image: postgres:15.3
    container_name: postgres
    ports:
      - 5432:5432
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
      POSTGRES_DB: myapp-db
    volumes:
      - postgres-data:/var/lib/postgresql/data
    restart: unless-stopped
  minio:
    container_name: s3minio
    image: bitnami/minio:latest
    ports:
      - "9000:9000"
      - "9001:9001"
    volumes:
      - minio_storage:/data
volumes:
  postgres-data:
  minio_storage:

Run the application

Depending on where you have your docker-compose file located and what options you want to use,

you need to run the following command:

# If you have docker files in the root of the project
docker-compose up

# In our case, we have dockerfile and docker-compose file in the `compose` folder, so we need to run:
docker-compose -f compose/docker-compose.yml up

# --- Optional ---
# For running the application with secrets ${VARIABLE_NAME} stored in the .env file, we would need to run:
docker-compose -f compose/docker-compose.yml --env-file .env up

# If you want to run the application in the background, you can use the -d flag:
docker-compose -f compose/docker-compose.yml up -d

It will run build the Next.js application and run it on port 3000. It will also download the PostgreSQL and Minio S3 docker images and run them on ports 5432 and 9000 respectively.

After the application is built and images are downloaded, you will see the following output:

You can access the application at http://localhost:3000, PostgreSQL database at http://localhost:5432 (login: postgres, password: postgres), and Minio S3 at http://localhost:9000 (login: minio, password: miniosecret).
Credentials for the database and file storage are stored in the docker-compose file. You can change them if you want.

Conclusion

In this article, we looked at how to build a local development environment for a full-stack Next.js application with Prisma ORM connected to PostgreSQL as a database and Minio S3 as a file storage using Docker-Compose.

References and further reading:

I hope you found this article useful. If you have any questions or comments, please let me know in the comments below.

In the the next article, we add a file upload functionality and database integration to our application.

I'm always open to making new connections! Feel free to connect with me on LinkedIn

Implementing Debounce in React for Efficient Delayed Search Queries

Alex — Fri, 05 Jan 2024 01:21:56 +0000

In this article, we will look at how to implement debounce in React for efficient delayed search queries. The problem we are trying to solve is that we want to make an API call to search for a user after the user has stopped typing for 1s (in real life, it would be more like 300ms). It will make the API calls more efficient and reduce the load on the server.

We will use the debounce function from my previous article Let’s implement a Debounce function in Javascript. But we will need to adjust it a bit to use it in a React component.

You can check how it works in the CodeSandbox below:

The main problem to use the debounce function in React is that we need to store the timer ID between renders. If we just use a useState hook, the timer ID will be reset on every render. So for this, we will use the useRef hook as it is recommended by the React team.

Let's adjust the debounce function to use the useRef hook:

const debounceTimeout = useRef<ReturnType<typeof setTimeout> | null>(null);

function debounce(func, delay: number) {
  return function (...args: []) {
    if (debounceTimeout.current) {
     clearTimeout(debounceTimeout.current);
    } 
   debounceTimeout.current = setTimeout(() => {
      func(...args);
      debounceTimeout.current = null;
    }, delay);
  };
}

We are going to search through a list of countries. We will use the useState hook to store the search term and the search result. We will use the debounce function to create a debounced search function that will be called after the wait time has passed:

const [searchTerm, setSearchTerm] = useState("");
const [searchResult, setSearchResult] = useState<string[]>([]);

const performSearch = (query: string) => {
  if (query.length === 0) {
    setSearchResult([]);
    return;
  }
  const result = countries
    .filter((country) => country.toLowerCase().includes(query.toLowerCase()))
    .sort();
  setSearchResult(result);
};

So the debounced search function will look like this:

const debouncedSearch = debounce(performSearch, DELAY);

We will use the debounced search function in the input change handler:

const handleInputChange = (event: React.ChangeEvent<HTMLInputElement>) => {
  const { value } = event.target;
  setSearchTerm(value);
  debouncedSearch(value);
};

I prefer using Tailwind CSS for styling, so I added some styling to the component.

The full code of the component

import { useState, useRef } from 'react'
import { countries } from './countries'
import './App.css'

// set delay for debounce function in milliseconds
const DELAY = 1000

export default function App() {
  const [searchTerm, setSearchTerm] = useState('')
  const [searchResult, setSearchResult] = useState<string[]>([])
  const debounceTimeout = useRef<ReturnType<typeof setTimeout> | null>(null)

  // Search function
  const performSearch = (query: string) => {
    if (query.length === 0) {
      setSearchResult([])
      return
    }
    const result = countries.filter((country) => country.toLowerCase().includes(query.toLowerCase())).sort()
    setSearchResult(result)
  }

  // Debounce function
  function debounce(func, delay: number) {
    return function (...args: []) {
      if (debounceTimeout.current) clearTimeout(debounceTimeout.current)
      debounceTimeout.current = setTimeout(() => {
        func(...args)
        debounceTimeout.current = null
      }, delay)
    }
  }

  // Debounced search function
  const debouncedSearch = debounce(performSearch, DELAY)

  // Event handler for input changes
  const handleInputChange = (event: React.ChangeEvent<HTMLInputElement>) => {
    const { value } = event.target
    setSearchTerm(value)
    debouncedSearch(value)
  }
  const delayMessage = (
    <>
      <p>Delay in effect while typing... </p> <p>Search will run after 1 second of inactivity.</p>
    </>
  )
  const searchResultEmpty = searchResult?.length === 0
  // get first 20 results
  const searchResultList = searchResult.slice(0, 10).map((c) => <li key={c}>{c}</li>)
  return (
    <div className='flex flex-col gap-5'>
      <div className='h-12'>{debounceTimeout.current && delayMessage}</div>
      <div className='flex gap-5 items-center h-24'>
        <label htmlFor='search-input' className='text-xl'>
          Search:
        </label>
        <input
          className='border-2 border-gray-300 rounded-md px-3 py-2 text-lg'
          type='text'
          id='search-input'
          value={searchTerm}
          onChange={handleInputChange}
          placeholder='Type to search...'
        />
        <div className='w-12'>{debounceTimeout.current && <LoadingCircle />}</div>
      </div>
      <div className='h-64'>
        {{ searchResultEmpty } && <ul className='text-left list-disc list-inside text-lg'>{searchResultList}</ul>}
        {searchResultEmpty && searchTerm.length > 0 && !debounceTimeout.current && (
          <div className='text-lg animate-pulse'>No results found</div>
        )}
      </div>
    </div>
  )
}

function LoadingCircle() {
  return (
    <svg
      className='animate-spin h-12 w-12 text-gray-700'
      xmlns='http://www.w3.org/2000/svg'
      fill='none'
      viewBox='0 0 24 24'
    >
      <circle className='opacity-50' cx='12' cy='12' r='10' stroke='currentColor' strokeWidth='4'></circle>
      <path fill='currentColor' d='M4 12a8 8 0 018-8V0C5.373 0 0 5.373 0 12h4z'></path>
    </svg>
  )
}

You also can check my GitHub repository debounce-react for the full code.

I hope you found this article useful. If you have any questions or comments, please let me know in the comments below.

I'm always open to making new connections! Feel free to connect with me on LinkedIn

Exploring 4 Ways to Compare Objects in JavaScript with Performance Analysis

Alex — Thu, 04 Jan 2024 00:35:35 +0000

Introduction
Ways to compare objects and arrays in Javascript:
- 1. Fast-deep-equal and other similar libraries
- 2. Node.js assert.deepEqual() method and Node.js util.isDeepStrictEqual() method
- 3. JSON.stringify() method
- 4. Custom object and array comparison function
Performance comparison

Introduction

Deep equality check is a common problem in Javascript. Unlike the regular equality operator (== or ===), which only checks for shallow equality, deep equal traverses through the entire structure of the objects or arrays to validate their equality.

The problem with the regular equality operator is that it only checks references of the objects or arrays. If two objects or arrays have the same values but different references, they will not be considered equal.

There are several common ways to compare objects and arrays in Javascript:

1. Fast-deep-equal and other similar libraries

This library provides a function called equal() which can be used to compare objects and arrays. It is a very popular library, it has 20+ million weekly downloads on npm.

This library wasn't in the first version of this article, it was suggested by Aleksei Mikhailov

The main advantage of this library is that it is very fast. It is 10-100 times faster than other libraries like Lodash's isEqual() method according to the author's benchmark tests:

Library	Ops/sec
fast-deep-equal	261,950
fast-equals	230,957
fast-deep-equal/es6	212,991
nano-equal	187,995
shallow-equal-fuzzy	138,302
underscore.isEqual	74,423
util.isDeepStrictEqual	46,440
lodash.isEqual	36,637
deep-eql	35,312
ramda.equals	12,054
deep-equal	2,310
assert.deepStrictEqual	456

To use it, you need to install the library using npm or yarn.

npm install fast-deep-equal

Then you can import it in your code and use it.

import equal from "fast-deep-equal";

const obj1 = { a: 1, b: 2 };
const obj2 = { a: 1, b: 2 };

equal(obj1, obj2); // true

Other libraries:

2. Node.js assert.deepEqual() method and Node.js util.isDeepStrictEqual() method

This is a part of the Node.js assert module.

The main downside of this method is that it can only be used in Node.js but not in the browser.

Another issue is this function throws an error if the objects are not equal. This is not ideal if you want to use it in a conditional statement. You can use try/catch to handle the error, but it is not ideal.

util.isDeepStrictEqual() is similar to assert.deepEqual() but it does not throw an error if the objects are not equal. It returns true or false depending on whether the objects are equal or not. This is more suitable for conditional statements.

const assert = require("assert").strict;

const obj1 = { a: 1, b: 2 };
const obj2 = { a: 1, b: 2 };

const obj3 = { a: 1, b: 3 };

assert.deepEqual(obj1, obj2); // true

assert.deepEqual(obj1, obj3); // AssertionError [ERR_ASSERTION]: Expected values to be strictly deep-equal:

3. JSON.stringify() method

This is a built-in method in Javascript. It converts a Javascript object or array into a JSON string. The JSON string can be compared using the regular equality operator. This is the most used method to compare objects and arrays in Javascript.

But it has some downsides, the main issue with this method is that order of the keys in the object matters. If the order of the keys is different, the JSON string will be different even if the objects are equal. To handle this, you can sort the keys before comparing them.

Another problem is when one of the objects contains an undefined value. The JSON.stringify() method will convert the undefined value to null. This will cause the comparison to fail even if the objects are equal. To handle this, you can use a custom replacer function to convert undefined values to null.

One more limitation with JSON.stringify() is that it does not work if the object or array contains functions or circular references. For example, if the object contains a function, it will be converted to null. If the object contains a circular reference, it will throw an error. There are other issues with converting objects to JSON strings, you can read more about them here.

const obj1 = { a: 1, b: 2 };
const obj2 = { a: 1, b: 2 };

JSON.stringify(obj1) === JSON.stringify(obj2); // true

// Order of the keys matters
const obj3 = { b: 2, a: 1 };

JSON.stringify(obj1) === JSON.stringify(obj3); // false

// Undefined values are converted to null
const obj4 = { a: 1, b: undefined };
JSON.stringify(obj4); // "{"a":1}"

const obj5 = { a: 1 };
JSON.stringify(obj5); // "{"a":1}"

JSON.stringify(obj4) === JSON.stringify(obj5); // true

// Functions are converted to null

const obj6 = {
  a: 1,
  b: function () {
    console.log("hello");
  },
};

JSON.stringify(obj6); // "{"a":1}"

4. Custom object and array comparison function

Let's implement a custom object and array comparison function. We will use recursion to implement the comparison function.

I wrote this function as an Exercise "2628. JSON Deep Equal" from LeetCode

/**
 * @param {null|boolean|number|string|Array|Object} o1
 * @param {null|boolean|number|string|Array|Object} o2
 * @return {boolean}
 */
function areDeeplyEqual(o1, o2) {
  // 1. Primitive values - if strictly equal, return true
  if (o1 === o2) return true;
  // 2. Null values - if either object is null, they are not equal
  if (o1 === null || o2 === null) return false;
  // 4. If not object, compare directly
  if (typeof o1 !== "object") {
    return o1 === o2;
  }

  // 5. Arrays - if one is array and other is not, they are not equal
  if (Array.isArray(o1) || Array.isArray(o2)) {
    if (!Array.isArray(o1) || !Array.isArray(o2)) return false;
    // If arrays have different lengths, they are not equal
    if (o1.length !== o2.length) return false;
    // Compare each element of the arrays
    for (let i = 0; i < o1.length; i++) {
      if (!areDeeplyEqual(o1[i], o2[i])) return false;
    }
  }

  // 6. Objects - if objects have different number of keys, they are not equal
  if (Object.keys(o1).length !== Object.keys(o2).length) return false;
  // Compare each key-value pair of the objects
  for (const [key, value] of Object.entries(o1)) {
    if (!o2.hasOwnProperty(key)) return false;
    if (!areDeeplyEqual(value, o2[key])) return false;
  }

  return true;
}

This function is not perfect, it has some limitations.
But it works for most cases. It was a good exercise to implement this function because it helped me understand how deep equality works in Javascript.

Performance comparison

I was curious to see how these methods compare in terms of performance. So I wrote a simple function for generating random objects with huge number of keys and values.

/**
 * @param {number} size
 * @return {Object}
 */
function generateHugeObject(size) {
  const obj = {};
  for (let i = 0; i < size; i++) {
    // toString(36) converts the number to base 36 (0-9a-z)
    // substring(2, 12) removes the first two characters (0.)
    const key = Math.random().toString(36).substring(2, 12);
    obj[key] = Math.random().toString(36).substring(2, 12);
  }
  return obj;
}

Then I wrote a simple function to compare two objects using each of the methods. I used the performance.now() method to measure the time taken by each method.
The results of this method are not very accurate and may vary from machine to machine, but it is good enough for our purpose.

let startFastDeepEqual = performance.now();
equal(obj1, obj2); // true
let timeTakenFastDeepEqual = performance.now() - startFastDeepEqual;

let startLodash = performance.now();
isEqual(obj1, obj2); // true
let timeTakenLodash = performance.now() - startLodash;

let startNodeAssert = performance.now();
assert.deepStrictEqual(obj1, obj2); // true
let timeTakenNodeAssert = performance.now() - startNodeAssert;

let startJSONStringify = performance.now();
JSON.stringify(obj1) === JSON.stringify(obj2); // true
let timeTakenJSONStringify = performance.now() - startJSONStringify;

let startCustom = performance.now();
areDeeplyEqual(obj1, obj2); // true
let timeTakenCustom = performance.now() - startCustom;

console.log(`FastDeepEqual: ${timeTakenFastDeepEqual}`);
console.log(`Lodash: ${timeTakenLodash}`);
console.log(`Node Assert: ${timeTakenNodeAssert}`);
console.log(`JSON.stringify: ${timeTakenJSONStringify}`);
console.log(`Custom: ${timeTakenCustom}`);

I ran this code with different sizes of objects and got the following results:

Size of object	FastDeepEqual	Lodash	Node Assert	JSON.stringify	Custom
100	0.0731	0.4316	0.0167	0.0202	0.2763
1000	0.2522	0.5901	0.3884	0.1584	0.5741
10000	2.3789	3.0968	3.0775	2.7993	7.3304
100000	30.8043	52.8975	37.6808	33.9532	56.0962
500000	173.8129	200.1934	219.5169	311.2552	448.4530
1000000	398.3462	455.5247	531.1270	736.3083	1085.5547

Graphical representation of the results:

Obviously, my implementation is not optimized. Does anyone know how to improve it? Please let me know if you have any suggestions.
Anyway, it was an interesting experiment, I learned how to use the performance.now() method and how to compare objects and arrays in Javascript.

I hope this article was helpful to you. Thanks for reading!

I'm always open to making new connections! Feel free to connect with me on LinkedIn

Let’s implement a Debounce function in Javascript

Alex — Mon, 01 Jan 2024 22:21:57 +0000

A debounce function is used to limit the rate at which a function can fire.

Previously I used Lodash’s debounce function, so I didn’t know how it worked and decided to implement it myself. I think it would be a good exercise and maybe help someone else come up with a better understanding of how it works.

Check how debounce works:

Real-life examples:

Doing an API call on every keypress in an input. If the user types fast, we will fire a lot of API calls, which is not good. We can use debounce to limit the rate at which the API calls are made.
A button click is handled by a function that will only run once in a given period, such as the Save button. If the user clicks the button multiple times, we will only run the function once.
When a user resizes a window or scrolls a page, which can trigger dozens or hundreds of calls in a small period of time.

Implementing a debounce function is not that hard. Two key concepts are used: closures and setTimeout.

export function debounce(callback: () => void, interval: number) {
  let timeout: number | null = null;
  return function (...args: unknown[]) {
    if (timeout) clearTimeout(timeout);
    timeout = setTimeout(() => callback(...args), interval);
  };
}

Closure is a function that has access to the parent scope, even after the parent function has closed. In our case, the parent function is the debounce function and the child function is the function that will be called after the wait time has passed. The child function will have access to the parent function’s scope, even after the parent function has closed.

SetTimeout is a function that will call a function after a given time has passed. SetTimeout returns a timer ID that can be used to cancel the timer by calling the clearTimeout function. We will use this to reset the timer if the function is called again before the wait time has passed.

I'm always open to making new connections! Feel free to connect with me on LinkedIn