DEV Community: Joan Roucoux

Easily Sync Your Subtitles with React, RHF and shadcn/ui

Joan Roucoux — Wed, 22 Jan 2025 22:51:42 +0000

Introduction

As a big fan of movies and TV shows, I often run into subtitle syncing issues that can be frustrating. This usually happens when the .srt file comes from a different source, causing delays with the actual dialogue. Adjusting the subtitles manually on a TV isn’t always an option, as the settings might not allow precise changes or may be outside of range.

To solve this problem, let's create a simple tool that allows you to shift the timecodes of your subtitles to perfectly sync them with your video.

We'll use React with Vite, shadcn/ui (a collection of reusable components I have been wanting to try for a while), React Hook Form for form management, and Zod for schema-based form validation.

Here is what our application will look like:

So let's jump into it 🚀

1. Setting Up the Project

We'll start by setting up a new React project with TypeScript preconfigured powered by Vite. Run the following to create our sub-shifter-app:

pnpm create vite@latest sub-shifter-app --template react-ts # Create the app
cd ./sub-shifter-app # Navigate to the project directory
pnpm install # Install dependencies

Next, to install shadcn/ui and its configuration, simply follow the installation steps here from step 2 to step 7, which are very well explained. ⚠️ But be careful ⚠️.

🚨 TailwindCSS 4.0.0 will not work with shadcn/ui if you follow the documentation (as it has not been updated yet - 22/01/2025).

To avoid compatibility issues, please install TailwindCSS version 3.4.17 by running the following commands:

pnpm install -D tailwindcss@3.4.17 postcss autoprefixer
pnpm dlx tailwindcss@3.4.17 init -p

This ensures everything works properly with the current setup.

...

After that, install the UI components needed for building our form interface:

pnpm dlx shadcn@latest add button form input label

When installing the form component using the command above, it will automatically add the required dependencies react-hook-form and zod.

About theming and colors, feel free to explore the hand-picked themes available here, which you can easily copy and paste into the application. For this tutorial, I chose the red theme (only copy/paste the .dark class in src/index.css like below):

@tailwind base;
@tailwind components;
@tailwind utilities;

@layer base {
  .dark {
    --background: 0 0% 3.9%;
    --foreground: 0 0% 98%;
    --card: 0 0% 3.9%;
    --card-foreground: 0 0% 98%;
    --popover: 0 0% 3.9%;
    --popover-foreground: 0 0% 98%;
    --primary: 0 72.2% 50.6%;
    --primary-foreground: 0 85.7% 97.3%;
    --secondary: 0 0% 14.9%;
    --secondary-foreground: 0 0% 98%;
    --muted: 0 0% 14.9%;
    --muted-foreground: 0 0% 63.9%;
    --accent: 0 0% 14.9%;
    --accent-foreground: 0 0% 98%;
    --destructive: 0 62.8% 30.6%;
    --destructive-foreground: 0 0% 98%;
    --border: 0 0% 14.9%;
    --input: 0 0% 14.9%;
    --ring: 0 72.2% 50.6%;
    --chart-1: 220 70% 50%;
    --chart-2: 160 60% 45%;
    --chart-3: 30 80% 55%;
    --chart-4: 280 65% 60%;
    --chart-5: 340 75% 55%;
  }
}

@layer base {
  * {
    @apply border-border;
  }
  body {
    @apply bg-background text-foreground;
  }
}

Lastly, let’s enable the dark mode (because, as viewers, we would prefer to avoid being blinded by a bright background 😆). Update the code in index.html as follows:

...
<body class="dark">
...

Great, we can now move on and start building the form of our application.

2. Building the SRT Shifter Form

In this section, we'll create the SRTShifterForm component, allowing users to upload an SRT subtitle file and apply a time offset to resync the subtitles like below.

We'll divide this into two parts: designing the UI and implementing the underlying logic, ensuring a clear separation of concerns.

2.1 Designing the UI

Below is the complete code for the component. We'll break it down in detail to understand each part:

import { useForm } from 'react-hook-form';
import { Button } from '@/components/ui/button';
import { Input } from '@/components/ui/input';
import { zodResolver } from '@hookform/resolvers/zod';
import {
  Form,
  FormField,
  FormItem,
  FormLabel,
  FormControl,
  FormMessage
} from './ui/form';
import {
  type FormSchemaType,
  FORM_INPUT_NAMES,
  ACCEPTED_FILE_TYPE,
  FormSchema,
  defaultValues,
  handleFormSubmit,
} from '../config/SRTShifterFormConfig';

const SRTShifterForm = () => {
  const form = useForm<FormSchemaType>({
    resolver: zodResolver(FormSchema),
    defaultValues,
  });

  const onSubmit = async (data: FormSchemaType): Promise<void> =>
    await handleFormSubmit(data, form);

  return (
    <>
      <section className="flex-1">
        <h2 className="text-2xl font-semibold mb-4">
          How to sync an SRT file with a video?
        </h2>
        <p className="leading-7 [&:not(:first-child)]:mt-4">
          Choose the SRT file you want to sync.
        </p>
        <p className="leading-7 [&:not(:first-child)]:mt-4">
          Enter the offset you want to apply to your subtitles, starting with "+" to add time or "-" to subtract it.
          For example, <strong>+1.20</strong> adds 1 second and 200 milliseconds to each timecode.
        </p>
        <p className="leading-7 [&:not(:first-child)]:mt-4">
          Click <strong>"Shift SRT Subtitles"</strong> to apply the offset and sync your subtitles.
        </p>
        <p className="leading-7 [&:not(:first-child)]:mt-4">
          Your resynced file will be automatically downloaded once ready 🍿
        </p>
      </section>
      <Form {...form}>
        <form
          onSubmit={form.handleSubmit(onSubmit)}
          className="flex-1 space-y-6 rounded-xl border bg-card text-card-foreground shadow p-6"
        >
          <FormField
            control={form.control}
            name={FORM_INPUT_NAMES.FILE}
            render={({ field: { onChange } }) => (
              <FormItem>
                <FormLabel>
                  Upload SRT File (.srt extension)
                </FormLabel>
                <FormControl>
                  <Input
                    id={FORM_INPUT_NAMES.FILE}
                    type="file"
                    accept={ACCEPTED_FILE_TYPE}
                    onChange={async (event) =>
                      onChange(event.target.files && event.target.files[0])}
                  />
                </FormControl>
                <FormMessage />
              </FormItem>
            )}
          />
          <FormField
            control={form.control}
            name={FORM_INPUT_NAMES.OFFSET}
            render={({ field }) => (
              <FormItem>
                <FormLabel>
                  Time Offset
                </FormLabel>
                <FormControl>
                  <Input
                    id={FORM_INPUT_NAMES.OFFSET}
                    type="text"
                    placeholder="+1.20"
                    {...field}
                  />
                </FormControl>
                <FormMessage />
              </FormItem>
            )}
          />
          <Button type="submit" className="w-full">
            Shift SRT Subtitles
          </Button>
        </form>
      </Form>
    </>
  );
};

export default SRTShifterForm;

You might have noticed that we are importing several methods from SRTShifterFormConfig.ts. We'll dive deeper into these in the next part.

So let's break it down!

First we used the custom hook useForm from React Hook Form for managing the form and incorporate Zod for validation:

const form = useForm<FormSchemaType>({
  resolver: zodResolver(FormSchema),
  defaultValues,
});

Then, we defined an onSubmit handler to process the form data when the user clicks the submit button:

const onSubmit = async (data: FormSchemaType): Promise<void> =>
  await handleFormSubmit(data, form);

We added a section in the render to provide clear instructions on how to use the application:

<section className="flex-1">
  <h2 className="text-2xl font-semibold mb-4">
    How to sync an SRT file with a video?
  </h2>
  ...
</section>

We added a <Form /> with two fields:

A file input which allows users to upload their .srt file:

<FormField
  control={form.control}
  name={FORM_INPUT_NAMES.FILE}
  render={({ field: { onChange } }) => (
    <FormItem>
      <FormLabel>
        Upload SRT File (.srt extension)
      </FormLabel>
      <FormControl>
        <Input
          id={FORM_INPUT_NAMES.FILE}
          type="file"
          accept={ACCEPTED_FILE_TYPE}
          onChange={async (event) =>
            onChange(event.target.files && event.target.files[0])}
        />
      </FormControl>
      <FormMessage />
    </FormItem>
  )}
/>

A text input where users specify the offset in the format +value or -value (e.g. +1.20):

<FormField
  control={form.control}
  name={FORM_INPUT_NAMES.OFFSET}
  render={({ field }) => (
    <FormItem>
      <FormLabel>
        Time Offset
      </FormLabel>
      <FormControl>
        <Input
          id={FORM_INPUT_NAMES.OFFSET}
          type="text"
          placeholder="+1.20"
          {...field}
        />
      </FormControl>
      <FormMessage />
    </FormItem>
  )}
/>

And finally, we defined the submit button, which triggers the onSubmit handler mentioned above:

<Button type="submit" className="w-full">
  Shift SRT Subtitles
</Button>

Great, we can proceed with implementing the underlying logic.

2.2 Implementing the underlying logic

In this section, we'll set up the logic for the SRTShifterForm component, including form validation, file handling, and timestamp resynchronization.

As mentioned earlier, I like to separate the logic into a different file to maintain a clean and simple component (which should primarily handle UI rendering and event management). This also makes it easier to test the methods defined in the config file.

To begin, let's create a new file: src/config/SRTShifterFormConfig.ts.

We'll start by defining constants to store the names of form inputs and error messages, ensuring consistent use across our form schema and error handling:

export const FORM_INPUT_NAMES = {
  FILE: 'file',
  OFFSET: 'offset',
} as const;

const FORM_ERROR_MESSAGES = {
  EMPTY_FILE_CONTENT: 'File content is empty.',
  WRONG_FILE_TYPE: 'Please upload a valid .srt file.',
  WRONG_FILE_SIZE: 'File size must be less than 2MB.',
  EMPTY_OFFSET: 'Offset is required.',
  WRONG_OFFSET_FORMAT: 'Offset must be a number, starting with \'+\' or \'-\'.',
  OFFSET_OUT_OF_RANGE: 'Offset must be between -3600 and +3600 seconds.',
} as const;

export const ACCEPTED_FILE_TYPE = '.srt';

const MAX_FILE_SIZE = 2 * 1024 * 1024; // 2MB
const OFFSET_REGEX = /^[+-][0-9]+\.?[0-9]*$/; // e.g. +1.20
const MAX_OFFSET_SIZE = 3600; // 1 hour in seconds

Then, we can define the form schema using Zod and the constants we just created, which handles validation for both the SRT file upload and the offset input:

import { z } from 'zod';
...
export const FormSchema = z.object({
  [FORM_INPUT_NAMES.FILE]: z
    .instanceof(File)
    .refine((file) => file.name.endsWith(ACCEPTED_FILE_TYPE), {
      message: FORM_ERROR_MESSAGES.WRONG_FILE_TYPE,
    })
    .refine((file) => file.size <= MAX_FILE_SIZE, {
      message: FORM_ERROR_MESSAGES.WRONG_FILE_SIZE,
    }),
  [FORM_INPUT_NAMES.OFFSET]: z
    .string()
    .min(1, {
      message: FORM_ERROR_MESSAGES.EMPTY_OFFSET,
    })
    .regex(OFFSET_REGEX, {
      message: FORM_ERROR_MESSAGES.WRONG_OFFSET_FORMAT,
    })
    .refine(
      (offset) => {
        const value = parseFloat(offset);
        return (
          !isNaN(value) && value >= -MAX_OFFSET_SIZE && value <= MAX_OFFSET_SIZE
        );
      },
      {
        message: FORM_ERROR_MESSAGES.OFFSET_OUT_OF_RANGE,
      }
    ),
});

The Zod schema validates the following:

The file must be of the valid type (.srt) and not exceed 2MB.
The offset must be a string that matches the specified format and is within a valid range (between -3600 and 3600 seconds).

Next, let's add the form's default values and specify the type based on the schema:

export type FormSchemaType = z.infer<typeof FormSchema>;

export const defaultValues: FormSchemaType = {
  file: new File([], ''),
  offset: '',
};

Now, to manage the uploaded file and handle the resync process, we can define three helper methods:

downloadResyncedFile: This method creates a Blob from the resynced content and triggers the file download.

export const downloadResyncedFile = (resyncedContent: string, fileName: string): void => {
  // Create a Blob from the resynced content
  const blob = new Blob([resyncedContent], { type: 'text/plain' });
  const url = URL.createObjectURL(blob);

  // Create an anchor element and trigger the download
  const link = document.createElement('a');
  link.href = url;
  link.download = fileName.replace(/\.srt$/, '-resynced.srt');
  link.click();

  // Clean up the object URL
  URL.revokeObjectURL(url);
};

adjustTimestamp: This method adjusts a given timestamp by adding or subtracting the offset in seconds.

const adjustTimestamp = (timestamp: string, offset: number): string => {
  const [hours, minutes, seconds] = timestamp.split(':');
  const [secs, millis] = seconds.split(',');

  // Convert the timestamp to milliseconds
  let totalMillis =
    parseInt(hours) * 3600000 +
    parseInt(minutes) * 60000 +
    parseInt(secs) * 1000 +
    parseInt(millis);

  // Apply the offset (convert seconds to milliseconds)
  totalMillis += offset * 1000;

  // Handle negative timestamps by clamping to zero
  if (totalMillis < 0) {
    totalMillis = 0;
  }

  // Convert back to "hh:mm:ss,SSS" format
  const newHours = Math.floor(totalMillis / 3600000);
  const newMinutes = Math.floor((totalMillis % 3600000) / 60000);
  const newSeconds = Math.floor((totalMillis % 60000) / 1000);
  const newMillis = totalMillis % 1000;

  // Return new timestamp
  return `${String(newHours).padStart(2, '0')}:${String(newMinutes).padStart(
    2,
    '0'
  )}:${String(newSeconds).padStart(2, '0')},${String(newMillis).padStart(
    3,
    '0'
  )}`;
};

resyncTimestamps: This method adjusts the timestamps of the subtitle file by applying the specified offset.

const resyncTimestamps = (content: string, offset: string): string => {
  // Parse the offset to a number
  const parsedOffset = parseFloat(offset);

  // Process each line of the file
  const lines = content.split('\n');
  const resyncedLines = lines.map((line) => {
    const timecodeMatch = line.match(
      /(\d{2}:\d{2}:\d{2},\d{3}) --> (\d{2}:\d{2}:\d{2},\d{3})/
    );

    // Resync timestamps
    if (timecodeMatch) {
      const [_, startTime, endTime] = timecodeMatch;
      const adjustedStartTime = adjustTimestamp(startTime, parsedOffset);
      const adjustedEndTime = adjustTimestamp(endTime, parsedOffset);
      return `${adjustedStartTime} --> ${adjustedEndTime}`;
    }

    return line;
  });

  // Join the adjusted lines back into a single string
  return resyncedLines.join('\n');
};

Finally, we can create the handleFormSubmit method to tie everything together: read the file content, process it with the specified offset, and trigger the download of the resynced file.

import { UseFormReturn, } from 'react-hook-form';
...
export const handleFormSubmit = async (
  data: FormSchemaType,
  form: UseFormReturn<FormSchemaType>
): Promise<void> => {
  const { file, offset } = data;

  // Read the file content
  const reader = new FileReader();
  reader.onload = (event) => {
    // Get the file content
    const fileContent = event?.target?.result as string;
    if (!fileContent) {
      form.setError(FORM_INPUT_NAMES.FILE, {
        type: 'manual',
        message: FORM_ERROR_MESSAGES.EMPTY_FILE_CONTENT,
      });
      return;
    }

    // Resync timestamps
    const resyncedContent = resyncTimestamps(fileContent, offset);

    // Download the resynced file
    downloadResyncedFile(resyncedContent, file.name);
  };

  reader.readAsText(file);
};

Now with the form's UI and logic in place, we can move on and build the rest of the UI.

3. Building the Rest of the UI

In this section, we'll create a simple Header component and update our App component to integrate the SRTShifterForm we just built.

Our Header component is very basic and includes a title and a subtitle. Add the following code in src/components/Header.tsx:

const Header = () => (
  <header className="text-center mb-8">
    <h1 className="text-4xl font-bold leading-tight tracking-tighter">
      <span className="text-primary">SRT</span> Sub Shifter
    </h1>
    <p className="text-base text-muted-foreground">
      Sync the SRT file with your video by shifting the timecodes of your subtitles.
    </p>
  </header>
);

export default Header;

Then, change the src/App.tsx with:

import Header from './components/Header';
import SRTShifterForm from './components/SRTShifterForm';

const App = () => (
  <div className="container mx-auto p-8">
    <Header />
    <div className="flex flex-col lg:flex-row gap-8">
      <SRTShifterForm />
    </div>
  </div>
);

export default App;

We are now done and ready to test our application!

4. Testing

Simply run pnpm dev and navigate to http://localhost:5173/ to see the result. Feel free to upload an SRT file and add an offset to see if the timestamps change 🙂 You’ll see in the code repo that I’ve added a resources folder with a few SRT files for you to try.

You can also check if the form is blocking you when submitting it without any data:

Or with an empty file and an incorrect offset format:

And that's how we wrap up this tutorial!

Conclusion

With this very simple tool, you can easily sync your subtitles by adjusting the timecodes, ensuring they match the dialogue perfectly. I hope you enjoyed!

You can find the code here.

Happy watching 🍿

Building a Scalable URL Shortener with Node.js (Part 2/2)

Joan Roucoux — Fri, 08 Nov 2024 17:05:08 +0000

Introduction

In the first part of this tutorial, we built the backend services for our URL shortener application using using Node.js, Redis, MongoDB, Apache ZooKeeper. Now, in this second part, we will focus on developing the frontend, making it easy for users to shorten and access URLs. We will use the following stack:

React: A JavaScript library for building the user interface.
RTK Query: A data-fetching and state management tool for making API requests to our backend.
Material UI: A component library for creating our responsive, modern UI.

Here is a preview of what our application will look like:

So let's jump into it 🚀

Setting up React and RTK

From the root directory url-shortener-demo-app, run the following command to initialize the client using the official Redux Toolkit + TS template for Vite:

npx degit reduxjs/redux-templates/packages/vite-template-redux client
cd client

Then, install all the required dependencies:

npm install react-router-dom @mui/material @emotion/react @emotion/styled @mui/icons-material @mui/lab

The packages you installed above include:

react-router-dom: Enables client side routing.
@mui/material @emotion/react @emotion/styled: Adds Material UI to our project.
@mui/icons-material: Adds Icon components.
@mui/lab: Components that are not yet available in the MUI core.

Then, because Material UI uses the Roboto font by default, add the following code inside the <head /> tag in index.html to install the font with the Google Fonts CDN:

<link rel="preconnect" href="https://fonts.googleapis.com" />
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin />
<link
  rel="stylesheet"
  href="https://fonts.googleapis.com/css2?family=Roboto:wght@300;400;500;700&display=swap"
/>

Now, to make sure our application works with Docker, we need to modify vite.config.ts and add host:true like below:

...
server: {
  open: true,
  host:true,
},
...

And finally, remove some of the generated files to clean up the src folder before adding our pages and components:

Remove features folder.
Remove App.css, App.test.tsx, App.tsx, index.css, logo.svg files.

Setting Up the Theme

We will custom the default theme used by Material UI using createTheme() to enable the dark mode and change some of the properties like colors, shape and typography.

In a src/theme/theme.ts file, add the following code:

import { createTheme } from '@mui/material/styles';
import type { Theme } from '@emotion/react';

const theme: Theme = createTheme({
  palette: {
    mode: 'dark',
    primary: {
      main: '#6200EE',
      dark: '#3700B3',
    },
    secondary: {
      main: '#03DAC6',
      dark: '#018786',
    },
  },
  shape: {
    borderRadius: 8,
  },
  typography: {
    h1: {
      fontWeight: 600,
      letterSpacing: '-0.24px',
    },
    h4: {
      fontWeight: 600,
      letterSpacing: '-0.06px',
    },
  },
});

export default theme;

Then, create a layout component that will wrap up our pages in src/components/Layout.tsx:

import { Outlet } from 'react-router-dom';
import { ThemeProvider } from '@mui/material/styles';
import { Container, Paper } from '@mui/material';
import CssBaseline from '@mui/material/CssBaseline';
import theme from '../theme/theme';

const Layout = (): JSX.Element => (
  <ThemeProvider theme={theme}>
    <CssBaseline />
    <Container component="main" maxWidth="sm" sx={{ mb: 4 }}>
      <Paper
        variant="outlined"
        sx={{ my: { xs: 3, md: 6 }, p: { xs: 2, md: 3 } }}
      >
        <Outlet />
      </Paper>
    </Container>
  </ThemeProvider>
);

export default Layout;

We will use this layout later, in the "Setting Up the Router" section.

Setting Up the URLs API Slice

Next, we will set up our URL API slice using RTK Query to interact with our servers through Nginx, implementing the following query operations:

getUrl: Retrieves a specific URL based on its shortened key. It will redirect the user to the original URL with window.location.replace(originalUrl);.
submitUrl: Submits an original URL and returns its shortened key.

In src/slices/urlsApiSlice.ts, add the following code:

import { createApi, fetchBaseQuery } from '@reduxjs/toolkit/query/react';

interface SubmitUrlBodyProps {
  originalUrl: string;
}

export const urlsApiSlice = createApi({
  reducerPath: 'urlsApi',
  baseQuery: fetchBaseQuery({
    baseUrl: import.meta.env.VITE_API_BASE_URL,
  }),
  endpoints: (builder) => ({
    getUrl: builder.query<void, string>({
      query: (shortenUrlKey: string) => ({
        url: `/urls/${encodeURIComponent(shortenUrlKey)}`,
        responseHandler: async (response: Response) => {
          if (!response.ok) {
            const errorText = await response.text();
            throw new Error(errorText);
          }
          const originalUrl = await response.text();
          window.location.replace(originalUrl);
        },
      }),
    }),
    submitUrl: builder.mutation<string, SubmitUrlBodyProps>({
      query: (body: SubmitUrlBodyProps) => ({
        url: '/urls',
        method: 'POST',
        body,
        responseHandler: 'text',
      }),
    }),
  }),
});

export const { useGetUrlQuery, useSubmitUrlMutation } = urlsApiSlice;

We're loading our baseUrl value from an environment variable VITE_API_BASE_URL, so make sure to add it to the existing .env file as well as some service configuration:

VITE_APP_LOCAL_PORT=5173
VITE_APP_DOCKER_PORT=5173
VITE_API_BASE_URL=http://${NGINX_HOST}:${NGINX_DOCKER_PORT}/api

Finally, we need to make one more change in src/app/store.ts to register our URL API slice in the Redux store:

import type { Action, ThunkAction } from '@reduxjs/toolkit';
import { combineSlices, configureStore } from '@reduxjs/toolkit';
import { setupListeners } from '@reduxjs/toolkit/query';
import { urlsApiSlice } from '../slices/urlsApiSlice';

const rootReducer = combineSlices(urlsApiSlice);
export type RootState = ReturnType<typeof rootReducer>;

export const makeStore = (preloadedState?: Partial<RootState>) => {
  const store = configureStore({
    reducer: rootReducer,
    middleware: (getDefaultMiddleware) => {
      return getDefaultMiddleware().concat(urlsApiSlice.middleware);
    },
    preloadedState,
  });
  setupListeners(store.dispatch);
  return store;
};

export const store = makeStore();

export type AppStore = typeof store;
export type AppDispatch = AppStore['dispatch'];
export type AppThunk<ThunkReturnType = void> = ThunkAction<
  ThunkReturnType,
  RootState,
  unknown,
  Action
>;

Setting Up the Home Page

Let's create our home page, where users can enter a URL to shorten and easily copy the generated link. This page will feature a simple form component and a data component to manage the request success and error states.

But first, just like we did in the first part of this tutorial, we need to implement a function to validate user input and a function to copy the result to the clipboard.

In src/utils/index.ts, add both methods:

export const isValidUrl = (value: string): boolean => {
  const pattern: RegExp = new RegExp(
    '^https?:\\/\\/' + // Protocol (http or https)
      '(?:www\\.)?' + // Optional www.
      '[-a-zA-Z0-9@:%._\\+~#=]{1,256}' + // Domain name characters
      '\\.[a-zA-Z0-9()]{1,6}\\b' + // Top-level domain
      '(?:[-a-zA-Z0-9()@:%_\\+.~#?&\\/=]*)$', // Optional query string
    'i' // Case-insensitive flag
  );

  return pattern.test(value);
};

export const copyToClipboard = (text: string): void => {
  navigator.clipboard.writeText(text);
};

Next, let's create the form component:

It will contain an input field and a button, both wrapped in a <form /> tag.
On submit, it will validate the inserted URL. If the URL is invalid, an error message will be displayed; otherwise it will call the trigger method with the provided input.

In src/components/UrlForm.tsx, add the following code:

import { type ChangeEvent, useEffect, useState } from 'react';
import { Grid2, TextField } from '@mui/material';
import { LoadingButton } from '@mui/lab';
import { Box } from '@mui/system';
import { isValidUrl } from '../utils/Utils';

interface UrlFormType {
  isSuccess: boolean;
  isLoading: boolean;
  trigger: (originalUrl: string) => void;
}

const UrlForm = ({
  isSuccess,
  isLoading,
  trigger,
}: UrlFormType): JSX.Element => {
  const [originalUrl, setOriginalUrl] = useState<string>('');
  const [onError, setOnError] = useState<boolean>(false);

  useEffect(() => {
    if (isSuccess) {
      setOriginalUrl('');
    }
  }, [isSuccess]);

  const handleChange = (event: ChangeEvent<HTMLInputElement>): void => {
    setOriginalUrl(event.target?.value);
    setOnError(false);
  };

  const handleSubmit = (event: ChangeEvent<HTMLFormElement>): void => {
    event.preventDefault();
    if (isValidUrl(originalUrl)) {
      trigger(originalUrl);
    } else {
      setOnError(true);
    }
  };

  return (
    <Box component="form" noValidate autoComplete="off" onSubmit={handleSubmit}>
      <Grid2 container spacing={2} justifyContent="center" alignItems="top">
        <Grid2 size={{ xs: 12, md: 8 }}>
          <TextField
            id="outlined-basic"
            placeholder="Enter a link to shorten it"
            variant="outlined"
            error={onError}
            helperText={onError ? 'Please enter a valid link' : ''}
            fullWidth
            required
            onChange={handleChange}
            value={originalUrl}
          />
        </Grid2>
        <Grid2 size={{ xs: 12, md: 4 }}>
          <LoadingButton
            type="submit"
            variant="contained"
            loading={isLoading}
            loadingIndicator="Loading..."
            fullWidth
            sx={{ height: '56px' }}
          >
            Shorten URL
          </LoadingButton>
        </Grid2>
      </Grid2>
    </Box>
  );
};

export default UrlForm;

Then, let's create the data component to display both success and error messages once the request is completed.

In src/components/UrlData.tsx, add the following code:

import { useEffect, useState } from 'react';
import { Link as RouterLink } from 'react-router-dom';
import { Alert, IconButton, Link } from '@mui/material';
import ContentCopyIcon from '@mui/icons-material/ContentCopy';
import DoneIcon from '@mui/icons-material/Done';
import { copyToClipboard } from '../utils/Utils';

interface UrlDataType {
  isSuccess: boolean;
  isError: boolean;
  shortenUrlKey: string | undefined;
}

const UrlData = ({
  isSuccess,
  isError,
  shortenUrlKey,
}: UrlDataType): JSX.Element => {
  const [isCopied, setIsCopied] = useState<boolean>(false);

  const shortenUrl = `${window.location.origin}/${shortenUrlKey}`;

  const handleOnClick = (): void => {
    copyToClipboard(shortenUrl);
    setIsCopied(true);
  };

  useEffect(() => {
    const timer = setTimeout(() => setIsCopied(false), 1300);
    return () => {
      clearTimeout(timer);
    };
  }, [isCopied]);

  return (
    <>
      {(isSuccess && (
        <Alert
          variant="outlined"
          severity="success"
          sx={{
            mt: 2,
          }}
          action={
            <IconButton
              aria-label="copy"
              color="inherit"
              onClick={handleOnClick}
            >
              {isCopied ? <DoneIcon /> : <ContentCopyIcon />}
            </IconButton>
          }
        >
          <Link
            target='_blank'
            color="inherit"
            component={RouterLink}
            to={`/${shortenUrlKey}`}
          >
            {shortenUrl}
          </Link>
        </Alert>
      )) ||
        (isError && (
          <Alert
            severity="error"
            sx={{
              mt: 2,
            }}
          >
            Oops, something went wrong.. Please try again.
          </Alert>
        ))}
    </>
  );
};

export default UrlData;

Finally, let’s create the home page, using useSubmitUrlMutation() to handle the POST request and integrating both the form and data components we just built.

In src/page/HomePage.tsx, add the following code:

import { Typography } from '@mui/material';
import { useSubmitUrlMutation } from '../slices/urlsApiSlice';
import UrlData from '../components/UrlData';
import UrlForm from '../components/UrlForm';

const HomePage = (): JSX.Element => {
  const [
    trigger,
    {
      data,
      isLoading,
      isSuccess,
      isError,
    },
  ] = useSubmitUrlMutation();

  return (
    <>
      <Typography
        component="h1"
        variant="h4"
        color="primary"
        align="center"
        gutterBottom
      >
        Create Short URLs
      </Typography>
      <Typography variant="subtitle1" align="center" sx={{ mb: 4 }}>
        This application makes long links look cleaner and easier to share!
      </Typography>
      <UrlForm
        isSuccess={isSuccess}
        isLoading={isLoading}
        trigger={(originalUrl: string) => trigger({ originalUrl })}
      />
      <UrlData
        isSuccess={isSuccess}
        isError={isError}
        shortenUrlKey={data}
      />
    </>
  );
};

export default HomePage;

Great, our home page is now done!

Setting Up the Url Page

Next, let's set up the URL page, which will handle requests when a user visits with a shortened key. As mentioned earlier, the redirect is handled directly in the API slice. Here, we only need to manage the loading and error states after making the GET request to retrieve the original URL.

In src/page/UrlPage.tsx, add the following code:

import { Link, useParams } from 'react-router-dom';
import { Box, Button, CircularProgress, Typography } from '@mui/material';
import { useGetUrlQuery } from '../slices/urlsApiSlice';

const UrlPage = (): JSX.Element => {
  const { shortenUrlKey } = useParams();

  const {
    isError,
  } = useGetUrlQuery(shortenUrlKey || '');

  return (
    <>
      {(isError && (
        <>
          <Typography
            component="h1"
            variant="h4"
            color="primary"
            align="center"
            gutterBottom
          >
            Oops, something went wrong
          </Typography>
          <Typography variant="subtitle1" align="center" sx={{ mb: 4 }}>
            The link has expired or is no longer available.
          </Typography>
          <Box textAlign='center'>
            <Button variant="contained" component={Link} to={'/'}>
              Back home
            </Button>
          </Box>
        </>
      )) || (
          <Box textAlign='center'>
            <CircularProgress />
          </Box>
        )}
    </>
  );
};

export default UrlPage;

Awesome, both pages are now ready to use!

Setting Up the Router

We will initialize the routing for our application using react-router-dom, with lazy-loaded pages for better performance. The setup will include:

A base route / that wraps the pages inside the Layout component, rendering the HomePage by default.
A dynamic route :shortenUrlKey that displays the UrlPage.
All other routes will be redirected to the index.

In src/router.tsx, add the code below:

import { lazy } from 'react';
import { Navigate, createBrowserRouter } from 'react-router-dom';
import Layout from './components/Layout';

const HomePage = lazy(() => import('./pages/HomePage'));
const UrlPage = lazy(() => import('./pages/UrlPage'));

const router = createBrowserRouter([
  {
    path: '/',
    element: <Layout />,
    children: [
      { index: true, element: <HomePage /> },
      { path: ':shortenUrlKey', element: <UrlPage /> },
    ],
  },
  {
    path: '*',
    element: <Navigate to="/" />,
  },
]);

export default router;

And finally, change the application entry point in src/main.tsx to implement our router:

import React, { Suspense } from 'react';
import { createRoot } from 'react-dom/client';
import { Provider } from 'react-redux';
import { RouterProvider, } from 'react-router-dom';
import { CircularProgress } from '@mui/material';
import { store } from './app/store';
import router from './router';

const container = document.getElementById("root")

if (container) {
  const root = createRoot(container)

  root.render(
    <React.StrictMode>
      <Provider store={store}>
        <Suspense fallback={<CircularProgress />}>
          <RouterProvider router={router} />
        </Suspense>
      </Provider>
    </React.StrictMode>,
  )
} else {
  throw new Error(
    "Root element with ID 'root' was not found in the document. Ensure there is a corresponding HTML element with the ID 'root' in your HTML file.",
  )
}

That's it, we now have our application (almost) ready!

Setting Up Nginx

We used Nginx in the first part as a load balancer and reverse proxy to distribute traffic across server instances, and we can also use it to serve the frontend, so the user doesn't need to know the port on which the client is running.

Let's add in the Nginx configuration nginx/nginx.conf file:

...
# Serve frontend
location / {
  proxy_pass http://client:$VITE_APP_DOCKER_PORT;
  proxy_set_header Host $host;
  proxy_set_header X-Real-IP $remote_addr;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header X-Forwarded-Proto $scheme;
  proxy_set_header X-Forwarded-Port $server_port;
}

# Serve backend
...

Don't forget to add the key VITE_APP_DOCKER_PORT=${VITE_APP_DOCKER_PORT} in the environment section of the service nginx in the docker-compose.yml file.

Containerization with Docker

Just like in the first part of this tutorial, we will use Docker to containerize our client application. Create a Dockerfile in the /client folder that sets up the Node environment, installs dependencies, and starts the app like below:

# Use an official Node runtime as the base image
FROM node:22.11.0

# Set the working directory
WORKDIR /usr/src/client

# Copy package.json and package-lock.json to the container
COPY package*.json ./

# Install application dependencies
RUN npm install

# Copy the rest of the application code
COPY . .

# Run the application
CMD [ "npm", "run", "start" ]

Then update the existing docker-compose.yml file at the project root to build the client using its Dockerfile:

...
client:
    depends_on:
      - nginx
    environment:
      - VITE_API_BASE_URL=${VITE_API_BASE_URL}
    build:
      context: ./client
      dockerfile: Dockerfile
    volumes:
      - ./client:/usr/src/client
      - /usr/src/client/node_modules
    ports:
      - ${VITE_APP_LOCAL_PORT}:${VITE_APP_DOCKER_PORT}

Testing and Deployment

Run docker compose up -d --build to rebuild the images and start all the containers again.

Then simply go to http://localhost/, you should the see the client running:

That's it! You can now interact with the UI, create short URLs, and even test accessing an invalid URL like http://localhost/wrongToken to see the result.

Conclusion

You have reached the end this tutorial! I hope you enjoyed and learned a few things along the way 😄

In this second part of this tutorial, we learned how to build the frontend of our URL shortener application using React and RTK Query and serving it with Nginx. You can find the complete code for this project here.

By combining the backend services from the first part with the frontend, we've created a fully functional and scalable URL shortener application, allowing users to easily shorten URLs.

As mentioned before, if you're interested in going further, check out my other repository here. In this version, I've added extra features like a visit counter and a purge system to clean all expired URLs, all managed through a task queue service.

Building a Scalable URL Shortener with Node.js (Part 1/2)

Joan Roucoux — Fri, 08 Nov 2024 17:05:03 +0000

Introduction

I'm sure you're familiar with URL shortener tools like TinyURL and Bitly, as they are widely used online. It simply takes a long URL and creates a shorter, unique alias that redirects to the original link.

For example, a URL like https://www.example.com/movies/avengers-infinity-war/casting/chris-hemsworth could be shortened to something like https://short.url/abc123, making it much easier to read and share. And then, when the user accesses the shortened link, the browser requests the server, which looks up the alias in its database and redirects the user to the original URL.

This might look simple when you first think about it, but it's actually tricky to build which is why they are commonly used in system design interviews. So creating one yourself is a great way to practice and see the challenges involved (handling high traffic, concurrency and race conditions in key generation, data storage and scaling...).

In the first part of this tutorial, we will focus on developing the backend of our URL shortener application using the following stack:

Node.js: A JavaScript runtime to power the server instances.
MongoDB: A NoSQL database for storing original URLs.
Redis: In-memory data store for caching frequently accessed URLs.
Apache ZooKeeper: A centralized service to generate unique IDs and prevent race conditions between instances.
Nginx: A load balancer and reverse proxy to distribute traffic across server instances.
Docker: A containerization tool to manage all the services.

Here's the high-level architecture of the application:

So let's jump into it 🚀

Initializing the Project

First, let's see what our file tree will look like:

url-shortener-demo-app
├─ .env
├─ docker-compose.yml
├─ client - Part 2 of this tutorial
├─ nginx
│  ├─ Dockerfile
│  └─ nginx.conf
└─ server
   ├─ src
   │  ├─ config
   │  │  ├─ mongoose.ts
   │  │  ├─ redis.ts
   │  │  └─ zookeeper.ts
   │  ├─ controllers
   │  │  └─ urlsController.ts
   │  ├─ models
   │  │  └─ Url.ts
   │  ├─ repositories
   │  │  └─ urlsRepository.ts
   │  ├─ routes
   │  │  └─ urlsRoutes.ts
   │  ├─ services
   │  │  └─ urlsService.ts
   │  ├─ utils
   │  │  └─ index.ts
   │  └─ index.ts
   ├─ .dockerignore
   ├─ Dockerfile
   ├─ package-lock.json
   ├─ package.json
   └─ tsconfig.json

Our project app url-shortener-demo-app will include a client folder for the React app that we will build in the second part of this tutorial, a nginx folder for Nginx configuration and a server folder for managing URL-related operations. It's a good thing to separate concepts like routers, controllers and services to enhance code organization and improve maintainability.

We will first start with the server code. Create a new directory for the project and initialize a new application:

mkdir url-shortener-demo-app
cd url-shortener-demo-app
mkdir server
cd server
npm init -y

Then, install all the required dependencies:

npm install fastify @fastify/cors mongoose ioredis zookeeper
npm install --save-dev typescript @types/node ts-node

The packages you installed above include:

fastify: A simple web framework for handling routes and middleware.
@fastify/cors: Middleware for enabling Cross-Origin Resource Sharing (CORS).
zookeeper: A client for interacting with Apache ZooKeeper.
mongoose: An Object Data Modeling (ODM) library for MongoDB.
ioredis: A client for interacting with Redis.
And some devDependencies for Typescript support.

Next, open the package.json file and a in the scripts section a new script to run the application:

...
"scripts": {
  "start": "ts-node src/index.ts"
},
...

And finally, create a tsconfig.json file for Typescript configuration by running:

tsc --init

Great, we can now move on and add the configuration files for MongoDB, Redis and ZooKeeper.

Setting Up MongoDB

Using mongoose in our application makes it easier to create and manage data within MongoDB, providing a more convenient and structured approach.

In src/config/mongoose.ts, add the following code to configure our MongoDB connection:

import { connect } from 'mongoose';

const {
  MONGODB_USER,
  MONGODB_PASSWORD,
  MONGODB_DATABASE,
  MONGODB_HOST,
  MONGODB_DOCKER_PORT,
} = process.env;

const MONGO_URI = `mongodb://${MONGODB_USER}:${MONGODB_PASSWORD}@${MONGODB_HOST}:${MONGODB_DOCKER_PORT}/${MONGODB_DATABASE}?authSource=admin`;

// Connect to MongoDB
export const connectToMongoDB = async (): Promise<void> => {
  try {
    await connect(MONGO_URI);
    console.log('Successfully connected to MongoDB');
  } catch (error) {
    console.error('Error connecting to MongoDB:', error);
    throw error;
  }
};

We're loading some environment variables from the .env file in the root directory, so be sure to create it and add the following keys:

MONGODB_USER=user
MONGODB_PASSWORD=pass
MONGODB_DATABASE=urls
MONGODB_HOST=mongo
MONGODB_LOCAL_PORT=27017
MONGODB_DOCKER_PORT=27017

Then, create the URL data model, which will define the structure of our documents in the database. This model will include the following attributes:

originalUrl: The original URL that users want to shorten.
shortenUrlKey: A unique identifier generated for the shortened URL.
createdAt: A timestamp indicating when the URL was created.
expiresAt: A timestamp indicating when the shortened URL will expire.

In src/models/Url.ts, add the following code:

import { Document, Schema, model } from 'mongoose';

export interface IUrl extends Document {
  originalUrl: string;
  shortenUrlKey: string;
  createdAt: Date;
  expiresAt: Date;
}

const schema = new Schema<IUrl>({
  originalUrl: {
    type: String,
    required: true,
    unique: true,
  },
  shortenUrlKey: {
    type: String,
    required: true,
    unique: true,
  },
  createdAt: {
    type: Date,
    default: new Date(),
  },
  expiresAt: {
    type: Date,
    default: new Date(new Date().setMinutes(new Date().getMinutes() + 10)), // default is 10 minutes, for demonstration only
  },
});

export default model<IUrl>('url', schema);

Setting Up Redis

In the same way that we use mongoose to interact with our MongoDB database, we will use ioredis client to interact with Redis and manage frequently accessed URLs by leveraging caching mechanisms for faster retrieval.

We will override some of the initial Redis methods to:

Add a new entry in Redis cache with set().
Retrieve a Redis cache entry with get().
Extend TTL of an existing entry with extendTTL().

In src/config/redis.ts, add the following code:

import { Redis } from 'ioredis';

const { REDIS_HOST, REDIS_DOCKER_PORT } = process.env;

export enum RedisExpirationMode {
  EX = 'EX', // Expire in seconds
}

let client: Redis | null;

// Get Redis client
const getRedisClient = (): Redis => {
  if (!client) {
    const config = {
      host: REDIS_HOST,
      port: Number(REDIS_DOCKER_PORT),
      maxRetriesPerRequest: null,
    };

    client = new Redis(config);
  }

  return client;
};

// Connect to Redis
export const connectToRedis = async (): Promise<void> => {
  const client = getRedisClient();

  client
    .on('connect', () => {
      console.log('Successfully connected to Redis');
    })
    .on('error', (error) => {
      console.error('Error on Redis:', error.message);
    });
};

// Set a key/value pair
export const set = async (
  key: string,
  value: string,
  expirationMode: RedisExpirationMode,
  seconds: number
): Promise<void> => {
  try {
    await getRedisClient().set(key, value, expirationMode, seconds);
    console.info(`Key ${key} created in Redis cache`);
  } catch (error) {
    console.error(`Failed to create key in Redis cache: ${error}`);
  }
};

// Get a value from a key
export const get = async (key: string): Promise<string | null> => {
  try {
    const value = await getRedisClient().get(key);
    console.info(`Value with key ${key} retrieved from Redis cache`);
    return value;
  } catch (error) {
    console.error(
      `Failed to retrieve value with key ${key} in Redis cache: ${error}`
    );
    return null;
  }
};

// Extend TTL of a key
export const extendTTL = async (
  key: string,
  additionalTimeInSeconds: number
) => {
  // Get the current TTL of the key
  const currentTTL = await getRedisClient().ttl(key);

  if (currentTTL > 0) {
    // Calculate the new TTL
    const newTTL = currentTTL + additionalTimeInSeconds;

    // Set the new TTL
    await getRedisClient().expire(key, newTTL);
    console.info(`TTL for key ${key} extended to ${newTTL} in Redis cache`);
  } else {
    console.error(`Failed to extend TTL of key ${key} in Redis cache`);
  }
};

Finally, add the following keys to the .env file to configure Redis service:

REDIS_HOST=redis
REDIS_LOCAL_PORT=6379
REDIS_DOCKER_PORT=6379

Setting Up ZooKeeper

Apache ZooKeeper will help us avoid race conditions by making sure that only one node generates a token at a time, maintaining data integrity.

One approach could be to assign each server registered to the ZooKeeper service a specific token range and generate a token within that range. However, I chose a simpler solution: checking if a token already exists under the /tokens path. If the token is not found, it will attempt to generate a new token repeatedly until it finds one that is available.

For instance, if /tokens/existingToken already exists, it will try again and register /tokens/newToken if available. In our example, we will use a token that is 6 characters long, which gives us around 69 billion possibilities (64^6). This should provide a comfortable buffer before we encounter any collisions in our demo app.

First, add a method to generate a base64 token in src/utils/index.ts:

import { randomBytes } from 'crypto';

export const generateBase64Token = (length: number): string => {
  const buffer = randomBytes(Math.ceil((length * 3) / 4)); // Generate enough random bytes
  return buffer
    .toString('base64') // Convert to Base64
    .replace(/\+/g, '-') // URL-safe: replace + with -
    .replace(/\//g, '_') // URL-safe: replace / with _
    .replace(/=+$/, '') // Remove padding
    .slice(0, length); // Ensure fixed length
};

Next, in src/config/zookeeper.ts, add the following code to:

Connect the client to ZooKeeper.
Create the /tokens node if it doesn't exist.
Generate the unique token using the generateBase64Token() method we just created.

import ZooKeeper from 'zookeeper';
import { generateBase64Token } from '../utils';

const { ZOOKEEPER_HOST, ZOOKEEPER_DOCKER_PORT } = process.env;

const host = `${ZOOKEEPER_HOST}:${ZOOKEEPER_DOCKER_PORT}`;

let client: ZooKeeper | null;

const TOKENS_NODE_PATH = '/tokens';

const MAX_RETRIES = 3;
const MAX_TOKEN_SIZE = 6;

// Get ZooKeeper client
const getZookeeperClient = (): ZooKeeper => {
  if (!client) {
    const config = {
      connect: host,
      timeout: 5000,
      debug_level: ZooKeeper.constants.ZOO_LOG_LEVEL_WARN,
      host_order_deterministic: false,
    };

    client = new ZooKeeper(config);
  }

  return client;
};

// Connect to ZooKeeper
export const connectToZookeeper = async (): Promise<void> => {
  const client = getZookeeperClient();

  await new Promise<void>((resolve, reject) => {
    client.connect(client.config, async (error) => {
      if (error) {
        console.error('Error connecting to ZooKeeper:', error);
        reject();
      }
      console.log('Successfully connected to ZooKeeper');
      await createTokensNode();
      resolve();
    });
  });
};

// Create '/tokens' node if it doesn't exist
const createTokensNode = async (): Promise<void> => {
  const client = getZookeeperClient();
  const doesTokensNodeExist = await client.pathExists(TOKENS_NODE_PATH, false);

  // If it does, do nothing
  if (doesTokensNodeExist) {
    console.info(`Tokens node ${TOKENS_NODE_PATH} already exists`);
    return;
  }

  // If it doesn't exist, create the root path
  await new Promise<void>((resolve, reject) => {
    client.mkdirp(TOKENS_NODE_PATH, (error) => {
      if (error) {
        console.error(`Failed to create tokens node: ${error}`);
        reject();
      }
      console.info(`Tokens node ${TOKENS_NODE_PATH} created`);
      resolve();
    });
  });
};

// Create a node
const createNode = async (path: string, data: Buffer): Promise<void> => {
  try {
    await getZookeeperClient().create(
      path,
      data,
      ZooKeeper.constants.ZOO_EPHEMERAL
    );
    console.info(`Node ${path} created`);
  } catch (error) {
    console.error(`Failed to create node: ${error}`);
    throw error;
  }
};

// Generate a unique token with retries for collision detection
export const generateUniqueToken = async (retryCount = 0): Promise<string> => {
  const client = getZookeeperClient();
  const token = generateBase64Token(MAX_TOKEN_SIZE);
  const uniqueTokenPath = `${TOKENS_NODE_PATH}/${token}`;

  // Create a child node with the generated token
  try {
    // Check if the unique token node already exists
    const doesUniqueTokenNodeExist = await client.pathExists(
      uniqueTokenPath,
      false
    );

    // If it does, retry
    if (doesUniqueTokenNodeExist) {
      if (retryCount < MAX_RETRIES) {
        console.log(
          `Token collision detected for path: ${uniqueTokenPath}. Retrying... Attempt ${
            retryCount + 1
          } of ${MAX_RETRIES}`
        );
        return await generateUniqueToken(retryCount + 1);
      } else {
        throw new Error(
          `Failed to generate a unique token after ${MAX_RETRIES} attempts due to collisions.`
        );
      }
    }

    // If it doesn't exist, create the node
    await createNode(uniqueTokenPath, Buffer.from(token));

    return token; // Return the unique token on success
  } catch (error) {
    console.error(`Error generating the unique token node: ${error}`);
    throw error;
  }
};

And finally, add the following keys to the .env file to configure the service:

ZOOKEEPER_HOST=zookeeper
ZOOKEEPER_LOCAL_PORT=2181
ZOOKEEPER_DOCKER_PORT=2181

Great, we have all of our configuration files now ready! We can move on and start implementing the actual URL logic.

Implementing the URL Shortening Logic

In this section, we will walk through the process of implementing the logic needed to shorten a URL and manage the redirection when users access the shortened link.

Implementing URL repository

The URL repository will handle all database operations for managing shortened URLs:

Add a new URL to the database with create().
Retrieve all saved URLs with findAll().
Fetch a specific URL based on given parameters with findOne().

Add the following code in src/repositories/urlRepository.ts:

import Url, { IUrl } from '../models/Url';

interface ICreateParams {
  shortenUrlKey: string;
  originalUrl: string;
}

interface IFindOneParams {
  shortenUrlKey?: string;
  originalUrl?: string;
}

// Create a shortened URL
const create = async (params: ICreateParams): Promise<IUrl> => {
  console.log(`Creating URL with params: ${JSON.stringify(params)}`);
  const result: IUrl = await Url.create({ ...params });
  console.log(`Created URL: ${JSON.stringify(result)}`);
  return result;
};

// Find all URLs
const findAll = async (): Promise<IUrl[]> => {
  console.log('Finding all URLs');
  const result: IUrl[] = await Url.find();
  console.log(`Found URLs: ${result?.length || 0}`);
  return result;
};

// Find a specific URL
const findOne = async (params: IFindOneParams): Promise<IUrl | null> => {
  console.log(`Finding one URL with params: ${JSON.stringify(params)}`);
  const result: IUrl | null = await Url.findOne({ ...params });
  console.log(`Found URL: ${JSON.stringify(result)}`);
  return result;
};

export { create, findAll, findOne };

Implementing URL validation

Before shortening a URL, it's essential to verify that the input provided by the user is valid. We will use a simple Regex found online for it (you can find plenty of other patterns as well depending on your needs).

Add the following method to src/utils/index.ts:

...
export const isValidUrl = (value: string): boolean => {
  const pattern: RegExp = new RegExp(
    '^https?:\\/\\/' + // Protocol (http or https)
      '(?:www\\.)?' + // Optional www.
      '[-a-zA-Z0-9@:%._\\+~#=]{1,256}' + // Domain name characters
      '\\.[a-zA-Z0-9()]{1,6}\\b' + // Top-level domain
      '(?:[-a-zA-Z0-9()@:%_\\+.~#?&\\/=]*)$', // Optional query string
    'i' // Case-insensitive flag
  );

  return pattern.test(value);
};

Implementing Service Logic

The service layer is responsible for orchestrating the business logic of the URL shortener application, acting as an intermediary between the controller and the repository we just created.

As mentioned in the Redis section, we will leverage Redis for caching frequently accessed URLs to improve performance by reducing database queries.

In src/services/urlService.ts, add the following code:

import { generateUniqueToken } from '../config/zookeeper';
import { get, set, extendTTL, RedisExpirationMode } from '../config/redis';
import { IUrl } from '../models/Url';
import { isValidUrl } from '../utils';
import { create, findAll, findOne } from '../repositories/urlsRepository';

const ONE_MINUTE_IN_SECONDS = 60;

// Get all shortened URLs
export const getAllUrls = async (): Promise<IUrl[]> => await findAll();

// Get a specific shortened URL by its key
export const getUrlByShortenUrlKey = async (
  shortenUrlKey: string
): Promise<string | null> => {
  // Try to get the original URL from Redis cache
  const cachedOriginalUrl = await get(shortenUrlKey);
  if (cachedOriginalUrl) {
    // Extend TTL
    await extendTTL(shortenUrlKey, ONE_MINUTE_IN_SECONDS);

    return cachedOriginalUrl; // Return the cached original URL
  }

  // If not in cache, retrieve from database
  const savedUrl = await findOne({ shortenUrlKey });
  if (savedUrl) {
    // Cache the original URL created by its shorten URL key
    await set(
      savedUrl.shortenUrlKey,
      savedUrl.originalUrl,
      RedisExpirationMode.EX,
      ONE_MINUTE_IN_SECONDS
    );

    return savedUrl.originalUrl; // Return the saved original URL
  }

  return null; // Return null if nothing found
};

// Create a new shortened URL
export const createShortenedUrl = async (
  originalUrl: string
): Promise<string | null> => {
  // Check if URL is valid
  if (!isValidUrl(originalUrl)) {
    return null;
  }

  // Retrieve from database
  const savedUrl = await findOne({ originalUrl });
  if (savedUrl) {
    return savedUrl.shortenUrlKey; // Return the saved shortened URL key
  }

  // If not in database, generate a new shortened URL key and save it
  const shortenUrlKey = await generateUniqueToken();
  if (shortenUrlKey) {
    const newUrl = await create({
      originalUrl,
      shortenUrlKey,
    });

    // Cache the original URL created by its shorten URL key
    await set(
      newUrl.shortenUrlKey,
      newUrl.originalUrl,
      RedisExpirationMode.EX,
      ONE_MINUTE_IN_SECONDS
    );

    return newUrl.shortenUrlKey; // Return shortened URL key
  }

  return null; // Return null if token generation failed
};

Implementing Controller Logic

Next, let's implement the URL controller methods to manage the HTTP status codes and return appropriate messages for our operations. As you might have seen, I chose to return a 200 status code along with the original URL (and not a 301 redirect) in the getUrl() method to prevent any CORS issues between the client and the requested URLs later on.

In src/controllers/urlController.ts, add the following code:

import { FastifyReply, FastifyRequest } from 'fastify';
import {
  createShortenedUrl,
  getAllUrls,
  getUrlByShortenUrlKey,
} from '../services/urlsService';

// Get all shortened URLs
export const getUrls = async (
  _request: FastifyRequest,
  reply: FastifyReply
): Promise<void> => {
  try {
    const urls = await getAllUrls();

    return reply.code(200).send(urls);
  } catch (error) {
    return reply
      .code(500)
      .send('Failed to retrieve the list of URLs. Please try again later');
  }
};

// Get a specific URL by its key
export const getUrl = async (
  request: FastifyRequest<{
    Params: {
      shortenUrlKey: string;
    };
  }>,
  reply: FastifyReply
): Promise<void> => {
  try {
    const { shortenUrlKey } = request.params;
    const originalUrl = await getUrlByShortenUrlKey(shortenUrlKey);

    if (!originalUrl) {
      return reply
        .code(404)
        .send('The requested shortened URL could not be found');
    }

    return reply.code(200).send(originalUrl);
  } catch (error) {
    return reply.code(500).send('Unable to retrieve the specified URL');
  }
};

// Create a new shortened URL
export const postUrl = async (
  request: FastifyRequest<{
    Body: {
      originalUrl: string;
    };
  }>,
  reply: FastifyReply
): Promise<void> => {
  try {
    const { originalUrl } = request.body;
    const shortenUrlKey = await createShortenedUrl(originalUrl);

    if (!shortenUrlKey) {
      return reply.code(400).send('The provided URL is invalid');
    }

    return reply.code(201).send(shortenUrlKey);
  } catch (error) {
    return reply.code(500).send('Failed to create a shortened URL');
  }
};

Implementing Routing Logic

Now, let's register the routes under the /urls prefix.

In src/routes/urls.ts, add the following code:

import { FastifyInstance } from 'fastify';
import { postUrl, getUrls, getUrl } from '../controllers/urlsController';

export const urlsRoutes = async (fastify: FastifyInstance) => {
  fastify.register(
    async (router: FastifyInstance) => {
      // Get all shortened URLs
      router.get('/', getUrls);

      // Get a specific URL by its key
      router.get('/:shortenUrlKey', getUrl);

      // Create a new shortened URL
      router.post('/', postUrl);
    },
    { prefix: '/urls' }
  );
};

Setting Up The Server

And finally, let's create a src/index.ts to set up the Fastify server and:

Configure CORS.
Register the URL router under the /api prefix.
Connect MongoDB, Redis, and ZooKeeper.
Start the server.

import Fastify, { FastifyInstance } from 'fastify';
import fastifyCors from '@fastify/cors';
import { connectToMongoDB } from './config/mongoose';
import { connectToRedis } from './config/redis';
import { connectToZookeeper } from './config/zookeeper';
import { urlsRoutes } from './routes/urlsRoutes';

// Fastify server instance
const fastify = Fastify();

// Configure server
fastify
  .register(fastifyCors) // Register CORS
  .register(
    async (fastify: FastifyInstance) => {
      fastify.register(urlsRoutes); // Register URL routes
    },
    { prefix: '/api' }
  );

// Start the server
const start = async () => {
  try {
    // Connect to MongoDB, Redis and ZooKeeper
    await connectToMongoDB();
    await connectToRedis();
    await connectToZookeeper();

    // Start Fastify server
    await fastify.listen({
      port: Number(process.env.NODE_SERVER_LOCAL_PORT),
      host: process.env.NODE_SERVER_HOST,
    });
    console.log('Server is now listening');
  } catch (error) {
    console.error('Failed to start server:', error);
    process.exit(1);
  }
};

start();

Don't forget to add the following keys to the .env file:

NODE_SERVER_HOST=0.0.0.0
NODE_SERVER_LOCAL_PORT=3000

Awesome, our server is now ready! We can move on and configure Nginx.

Setting Up Nginx

As mentioned in the introduction, we will use Nginx as a load balancer and reverse proxy to distribute traffic across server instances and improve response times. We will use the default round-robin algorithm, which is ideal for distributing requests evenly and making our application more resilient.

In a nginx/nginx.conf file, add the following configuration:

Create an upstream node_servers block to define the group of servers listening on local port 3000. You will see later in the docker-compose setup that we did not define a specific port for each server instance, allowing Docker to dynamically assign ports and manage load balancing.
Add a server block which listens on port 80 for incoming requests.
Include a location block to forward requests made on the /api/ path to the node_servers group using proxy_pass. And add proxy_set_header directives to ensure that client request details are forwarded to the servers.

upstream node_servers {
  server server:$NODE_SERVER_LOCAL_PORT;
}

server {
  listen $NGINX_DOCKER_PORT;

  # Serve backend
  location /api/ {
    proxy_pass http://node_servers/api/;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Forwarded-Port $server_port;
  }
}

Also add the following keys to the .env file to configure the Nginx service:

NGINX_HOST=localhost
NGINX_LOCAL_PORT=80
NGINX_DOCKER_PORT=80

Containerization with Docker

With everything set up, we will now use Docker to containerize all of our services.

Let's first dockerize the server application. Create a Dockerfile in the /server folder that sets up the Node environment, installs dependencies, and starts the app like below:

# Use an official Node runtime as the base image
FROM node:22.11.0

# Set the working directory
WORKDIR /usr/src/server

# Copy package.json and package-lock.json to the container
COPY package*.json ./

# Install application dependencies
RUN npm install

# Copy the rest of the application code
COPY . .

# Run the application
CMD [ "npm", "run", "start" ]

Also add a .dockerignore file with node_modules inside because some libraries like zookeeper can cause some issues when compiled on different OS.

Next, create a Dockerfile in the nginx folder to configure Nginx by using the official Nginx image, copying our custom nginx.conf file to the container, and running the service:

# Use an official Nginx as the base image
FROM nginx:stable-alpine

# Copy nginx.conf to the container
COPY nginx.conf /etc/nginx/templates/default.conf.template

# Run the server
CMD ["nginx", "-g", "daemon off;"]

Finally, create a docker-compose.yml file at the root of your project setting up the services (MongoDB, Redis and ZooKeeper), while building the server and Nginx from their respective Dockerfiles:

services:
  mongo:
    image: mongo:latest
    environment:
      - MONGO_INITDB_ROOT_USERNAME=${MONGODB_USER}
      - MONGO_INITDB_ROOT_PASSWORD=${MONGODB_PASSWORD}
    ports:
      - ${MONGODB_LOCAL_PORT}:${MONGODB_DOCKER_PORT}
    volumes:
      - ./mongodb:/data/db

  redis:
    image: redis:latest
    ports:
      - ${REDIS_LOCAL_PORT}:${REDIS_DOCKER_PORT}

  zookeeper:
    image: zookeeper:latest
    ports:
      - ${ZOOKEEPER_LOCAL_PORT}:${ZOOKEEPER_DOCKER_PORT}

  server:
    depends_on:
      - mongo
      - redis
      - zookeeper
    environment:
      - MONGODB_USER=${MONGODB_USER}
      - MONGODB_PASSWORD=${MONGODB_PASSWORD}
      - MONGODB_DATABASE=${MONGODB_DATABASE}
      - MONGODB_HOST=${MONGODB_HOST}
      - MONGODB_DOCKER_PORT=${MONGODB_DOCKER_PORT}
      - REDIS_HOST=${REDIS_HOST}
      - REDIS_DOCKER_PORT=${REDIS_DOCKER_PORT}
      - ZOOKEEPER_HOST=${ZOOKEEPER_HOST}
      - ZOOKEEPER_DOCKER_PORT=${ZOOKEEPER_DOCKER_PORT}
      - NODE_SERVER_HOST=${NODE_SERVER_HOST}
      - NODE_SERVER_LOCAL_PORT=${NODE_SERVER_LOCAL_PORT}
    build:
      context: ./server
      dockerfile: Dockerfile
    volumes:
      - ./server:/usr/src/server
      - /usr/src/server/node_modules
    deploy:
      mode: replicated
      replicas: 3

  nginx:
    depends_on:
      - server
    environment:
      - NODE_SERVER_LOCAL_PORT=${NODE_SERVER_LOCAL_PORT}
      - NGINX_DOCKER_PORT=${NGINX_DOCKER_PORT}
    build:
      context: ./nginx
      dockerfile: Dockerfile
    ports:
      - ${NGINX_LOCAL_PORT}:${NGINX_DOCKER_PORT}

As you can see, we are passing down all the environment variables defined in the .env file, keeping all configurations centralized in one place, which makes adjustments simple.

Testing and Deployment

Run docker compose up -d to start all containers in detached mode. Once they're running, you can use cURL or any other tool to test the API routes:

Save a new URL (you can also test an incorrect URL format to check if the service returns a 400 status code):

curl --location 'http://localhost/api/urls' \
--header 'Content-Type: application/json' \
--data '{
    "originalUrl": "URL_HERE"
}'

Get all URLs:

curl --location 'http://localhost/api/urls'

Retrieve the original URL:

curl --location 'http://localhost/api/urls/SHORTENED_TOKEN_HERE'

Or use Postman which is more friendly:

You can also check the logs in Docker to monitor the activity across the containers (I'm using Docker Desktop here):

Conclusion

You have reached the end of the first part of this tutorial! I hope you enjoyed 😄

We learned how to build a scalable URL shortener application from scratch using Node.js, Redis, MongoDB, Apache ZooKeeper, Nginx and Docker. You can find the complete code for this project here.

In the second part, we will focus on developing the frontend of our application using React and RTK Query, allowing users to interact with our servers through a minimal UI.

And if you're interested in going further, check out my other repository here. In this version, I've added extra features like a visit counter and a purge system to clean all expired URLs, all managed through a task queue service.

Update README with Latest Articles Using GitHub Actions and Deno 2

Joan Roucoux — Sun, 15 Sep 2024 19:47:18 +0000

Introduction

I wanted to give my GitHub profile a fresh new look, and since I’ve started writing articles on Dev.to, I thought it would be a great idea to automatically showcase the most recent ones on my profile.

You can find pre-made GitHub Actions workflows which periodically fetches the latest content and updates your README. However, I wanted to create a very basic, more flexible solution from scratch using Deno 2.0 (with Typescript).

That’s why I’m writing this article, to share the steps with you, which you can easily adapt whether you’re fetching data from Dev.to, YouTube videos, RSS feeds, Medium, or any other source!

I’m assuming you already have a GitHub profile repository set up with a README.md file. If not, you can get started here.

1. Creating the GitHub Action Workflow

We can start by setting up the GitHub Action Workflow:

Create a .github/workflows/update-readme-articles.yml file in your profile repository.
Add a schedule to run the job every month at 0:00 UTC (or more often if you are a fast writer, which is not my case 😅). You can also include the workflow_dispatch event trigger, enabling the workflow to be run manually when needed.
Add in the jobs section the following steps:
1. Check out the codebase with actions/checkout@v4.
2. Set up Deno with denoland/setup-deno@v2.
3. Run the update-readme-articles.ts script to update the README.
4. Commit and push only changed files to the repository.

name: Update readme articles

on:
  # Add a schedule to run the job every month at 0:00 UTC
  schedule:
    - cron: '0 0 1 * *'

  # Allow running this workflow manually
  workflow_dispatch:

jobs:
  update-readme-articles:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout sources
        uses: actions/checkout@v4

      - name: Setup Deno environment
        uses: denoland/setup-deno@v2
        with:
          deno-version: v2.x

      - name: Run script
        run: |
          deno run -A scripts/update-readme-articles.ts

      - name: Commit and push changes
        env:
          # This is necessary in order to push a commit to the main branch
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          git config --local user.email "github-actions[bot]@users.noreply.github.com"
          git config --local user.name "github-actions[bot]"
          git add .
          if [[ -n "$(git status --porcelain)" ]]; then
            git commit -m ":wrench: Update readme articles"
            git push origin main
          fi

2. Updating the README

2.1 Adding markers

To update the README with the latest articles from Dev.to, start by adding markers to your README.md file as shown below:

# Hi 👋, I'm Joan Roucoux

## About me 💬

...

## My latest articles 📝

<!-- ARTICLES:START -->
<!-- ARTICLES:END -->

🚀 This section is updated by GitHub Actions Workflows ❤️

...

2.2 Initializing the Deno Project

Run the following command to initialize a Deno project (make sure Deno is installed locally. If not, follow this):

deno init

2.3 Creating the script logic

Next, you’ll need to create a script that follows these steps:

Create the required files:
scripts/types.ts: define the types for your articles.
scripts/update-readme-articles.ts: main logic to fetch and process articles.
Read the original file content with fs.readFile().
Fetch latest articles using the /articles operation from the Dev.to API with your name. For instance with mine, it gives https://dev.to/api/articles?username=joanroucoux&page=1&per_page=5. Feel free to adapt the source of your content here as mentioned in the introduction.
Generate new markdown with generateArticlesContent() and replace the content between markers with replaceContentBetweenMarkers().
Save the updated file with fs.writeFile().

The updated markdown code will look like this:

...
<!-- ARTICLES:START -->
- [Update README with Latest Articles Using GitHub Actions](https://dev.to/joanroucoux/update-readme-with-latest-articles-using-github-actions-41m3)
- [Building a Marvel Search Application with Qwik](https://dev.to/joanroucoux/building-a-marvel-search-application-with-qwik-ll7)
<!-- ARTICLES:END -->
...

Which will be rendered as follows:

Here are the .ts files, which performs the above steps:

// types.ts
interface User {
  name: string;
  username: string;
  twitter_username: string | null;
  github_username: string;
  user_id: number;
  website_url: string;
  profile_image: string;
  profile_image_90: string;
}

export interface Article {
  type_of: string;
  id: number;
  title: string;
  description: string;
  readable_publish_date: string;
  slug: string;
  path: string;
  url: string;
  comments_count: number;
  public_reactions_count: number;
  collection_id: number | null;
  published_timestamp: string;
  positive_reactions_count: number;
  cover_image: string | null;
  social_image: string;
  canonical_url: string;
  created_at: string;
  edited_at: string;
  crossposted_at: string | null;
  published_at: string;
  last_comment_at: string;
  reading_time_minutes: number;
  tag_list: string[];
  tags: string;
  user: User;
}

export interface ArticlePreview {
  title: string;
  url: string;
}

// update-readme-articles.ts
import * as fs from 'node:fs/promises';
import path from 'node:path';
import { Article, ArticlePreview } from './types.ts';

const DEV_TO_API_BASE_URL = 'https://dev.to/api';
const USERNAME = 'joanroucoux';

const main = async (): Promise<void> => {
  // Read the original file content
  const filePath = '../README.md';
  const markdown = await readFile(filePath);

  // Proceed only if the file was read successfully
  if (markdown) {
    // Fetch latest articles
    const articles = await fetchArticles(USERNAME);

    // Generate new content
    const newContent = generateArticlesContent(articles);

    // Replace content between markers
    const START_MARKER = '<!-- ARTICLES:START -->';
    const END_MARKER = '<!-- ARTICLES:END -->';
    const updatedMarkdown = replaceContentBetweenMarkers(
      markdown,
      START_MARKER,
      END_MARKER,
      newContent
    );

    // Save the updated file
    await saveFile(filePath, updatedMarkdown);
  }
};

// Fetch latest articles
const fetchArticles = async (
  username: string,
  page: number = 1,
  perPage: number = 5
): Promise<ArticlePreview[]> => {
  const response = await fetch(
    `${DEV_TO_API_BASE_URL}/articles?username=${username}&page=${page}&per_page=${perPage}`
  );
  const data: Article[] = await response.json();
  return data?.map((article: Article) => ({
    title: article.title,
    url: article.url,
  }));
};

// Generate markdown from articles
const generateArticlesContent = (articles: ArticlePreview[]): string => {
  let markdown = '';

  articles?.forEach((article) => {
    markdown += `- [${article.title}](${article.url})\n`;
  });

  return markdown;
};

// Read file
const readFile = async (filePath: string): Promise<string | null> => {
  try {
    const absolutePath = path.resolve(import.meta.dirname, filePath);
    console.log('Reading file from:', absolutePath);
    return await fs.readFile(absolutePath, 'utf8');
  } catch (err) {
    console.error('Error reading file:', err);
    return null;
  }
};

// Generate updated markdown
const replaceContentBetweenMarkers = (
  markdown: string,
  startMarker: string,
  endMarker: string,
  newContent: string
): string => {
  const regex = new RegExp(`(${startMarker})([\\s\\S]*?)(${endMarker})`, 'g');
  return markdown.replace(regex, `$1\n${newContent}$3`);
};

// Save file
const saveFile = async (filePath: string, content: string): Promise<void> => {
  try {
    const absolutePath = path.resolve(import.meta.dirname, filePath);
    await fs.writeFile(absolutePath, content, 'utf8');
    console.log('File has been saved successfully!');
  } catch (err) {
    console.error('Error saving file:', err);
  }
};

main();

3. Pushing and testing

That’s it! After pushing all the files to your repository, you can manually trigger the workflow from the "Actions" tab or wait for it to run based on the scheduled trigger. You can also test the script locally by running deno run -A --watch scripts/update-readme-articles.ts before pushing. Once completed, check the logs for any errors and verify that your README.md has been updated correctly.

4. Conclusion

By automating the process of updating your README with GitHub Actions, you make sure the file stays up-to-date with your latest content. It's also a flexible solution that you can easily tweak to integrate different sources of data!

I hope you found this tutorial helpful! You can check out my repository here 🚀

Resources:

Building a Marvel Search Application with Qwik

Joan Roucoux — Mon, 22 Jul 2024 11:18:12 +0000

Introduction

In this tutorial, we will see how to create an application that lets users search for Marvel characters using Qwik, Tailwind CSS and the Marvel API.

Qwik has been available for a while now and is an excellent choice for building web applications that are fast, scalable, and maintainable.

Before we start, you will need an API key to access Marvel’s resources:

Go to https://developer.marvel.com/ and click on the “Get Started” button to create your account.
Next, go to “/account” to retrieve your API public and private keys.
Don't forget to add *.* for localhost support to your authorised domains like below.

We can now begin and build our Qwik app 👇

Step 1: Initialise the project

To get started with Qwik locally, you need the following:

Node.js v18.17 or higher
Your favorite IDE (I use VSCode personally)

Use the Qwik CLI command npm create qwik@latest to generate a blank starter application:

npm create qwik@latest
- Where would you like to create your new project? => ./qwik-marvel-demo-app
- Select a starter => Empty App (Qwik City + Qwik)
- Would you like to install npm dependencies? => Yes
- Initialise a new git repository? => As you want
- Finishing the install. Wanna hear a joke? => As you want (but they are pretty good 😆)

Next, move to the new app folder and run the application to make sure everything is okay:

- cd ./qwik-marvel-demo-app
- npm start

You should see the default app running in your browser:

Step 2: Configure Tailwind CSS and daisyUI

We will use Tailwind CSS and daisyUI to quickly build the UI for our components.

Run npm run qwik add tailwind to set up Tailwind:

npm run qwik add tailwind
- Ready to apply the tailwind updates to your app? => Yes

Next, install daisyUI as a Tailwind CSS plugin by running npm i -D daisyui@latest.

Then add to tailwind.config.js the following configuration:

export default {
  //...
  plugins: [require("daisyui")],
  daisyui: {
    themes: [
      {
        dark: {
          ...require("daisyui/src/theming/themes")["dark"],
          primary: "#e62429",
          "primary-content": "#e0e0e0",
          "base-100": "#191919",
          "base-200": "#0c0c0c",
          "base-300": "#000000",
          "base-content": "#e0e0e0",
          "neutral-content": "#e0e0e0",
        },
      },
    ],
  },
};

As you can see, we are only enabling the daisy “dark” theme and overriding some of the base attribute colors.

Step 3: Configure the Marvel API

I like to start my projects by setting up the services, which makes building the UI more efficient later on.

So let’s jump into it, let’s create a fetchMarvelAPI() method in src/services/marvel.ts: this method will call the Marvel API in a generic way, dynamically adding the necessary parameters to keep the code DRY (Don't Repeat Yourself). By doing this, it will be easier for you to create more services later if you want to add new functionalities.

import type { RequestEventBase } from "@builder.io/qwik-city";
import { Md5 } from "ts-md5";
import { buildSearchParams } from "~/utils";

export const getMarvelContext = (requestEvent: RequestEventBase) => {
  const baseURL = requestEvent.env.get("VITE_MARVEL_PUBLIC_BASE_URL");
  const publicApiKey = requestEvent.env.get("VITE_MARVEL_PUBLIC_API_KEY");
  const privateApiKey = requestEvent.env.get("VITE_MARVEL_PRIVATE_API_KEY");
  const ts = Date.now().toString();
  const hash = Md5.hashStr(ts + privateApiKey + publicApiKey);

  return {
    publicApiKey,
    privateApiKey,
    baseURL,
    ts,
    hash,
  };
};

type MarvelContext = ReturnType<typeof getMarvelContext>;

type FetchMarvelAPIArgs = {
  context: MarvelContext;
  path: string;
  query?: Partial<Record<string, string>>;
};

const fetchMarvelAPI = async <T = unknown>({
  context,
  path,
  query,
}: FetchMarvelAPIArgs): Promise<T> => {
  const params = buildSearchParams({
    apikey: context.publicApiKey,
    ts: context.ts,
    hash: context.hash,
    ...query,
  });

  const url = `${context.baseURL}/${path}?${params}`;
  const response = await fetch(url);

  if (!response.ok) {
    // eslint-disable-next-line no-console
    console.error(url);
    throw new Error(`[fetchMarvelAPI] An error occurred: ${response.statusText}`);
  }

  return response.json();
};

Quick note: in order to call the API, we will have to generate a hash using MD5, so make sure to install the ts-md5 package by running npm i ts-md5.

As you can see, we are also importing our API keys as server-side variables that can only be accessed in resources that expose the RequestEvent object.
So make sure to create at the root of the application a .env.local file with both your keys as well as the Marvel public base URL:

VITE_MARVEL_PUBLIC_BASE_URL=https://gateway.marvel.com/v1/public
VITE_MARVEL_PUBLIC_API_KEY=YOUR_PUBLIC_KEY_HERE
VITE_MARVEL_PRIVATE_API_KEY=YOUR_PRIVATE_KEY_HERE

What about this buildSearchParams() import? This method builds a URLSearchParams object from a key/value pairs object, so that query params can be easily added to our service call.

In a new file src/utils/index.ts, add the following method:

export const buildSearchParams = (
  query?: Record<string, unknown>,
): URLSearchParams => {
  const entries = Object.entries(query || {});
  const pairs = entries.flatMap(([key, value]) =>
    value !== undefined && value !== null ? [[key, `${value}`]] : [],
  );
  return new URLSearchParams(pairs);
};

As mentioned earlier, our users will be able to search for Marvel characters. So let's create on top of the existing fetchMarvelAPI() a new method getCharacters() to fetch the Marvel API resource “/characters".

// src/services/marvel.ts

import type { DataWrapper, Character } from "~/types";
import { buildSearchParams, getOffset } from "~/utils";
//...
type GetCharactersArgs = {
  context: MarvelContext;
  page: number;
  limit: number;
  startsWith?: string;
};

export const getCharacters = async ({
  context,
  page,
  limit,
  startsWith,
}: GetCharactersArgs) =>
  await fetchMarvelAPI<DataWrapper<Character>>({
    context,
    path: "/characters",
    query: {
      offset: getOffset(page, limit),
      limit: String(limit),
      nameStartsWith: startsWith,
    },
  });

Again, we are referring to a new getOffset() method from the utils folder in the query object. It's perfect for skipping a specified number of results when fetching the API, which is useful when creating a pagination system (something we will implement later in this tutorial).

So in src/utils/index.ts, add the following method:

export const getOffset = (page: number, limit: number): string =>
  String(limit * (page - 1));

Don't forget to add all the required types in a new src/types/index.ts:

export type Nullable<T> = T | null;

export type DataWrapper<T> = {
  code?: number;
  status?: Nullable<string>;
  copyright?: Nullable<string>;
  attributionTextisplay?: Nullable<string>;
  attributionHTML?: Nullable<string>;
  data?: DataContainer<T>;
  etag?: Nullable<string>;
};

export type DataContainer<T> = {
  offset?: number;
  limit?: number;
  total?: number;
  count?: number;
  results?: T[];
};

export type Url = {
  type?: Nullable<string>;
  url?: Nullable<string>;
};

export type Image = {
  path?: Nullable<string>;
  extension?: Nullable<string>;
};

export type ResourceList = {
  available?: number;
  returned?: number;
  collectionURI?: Nullable<string>;
  items?: ResourceSummary[];
};

export type ResourceSummary = {
  resourceURI?: Nullable<string>;
  name?: Nullable<string>;
  type?: Nullable<string>;
  role?: Nullable<string>;
};

export type Character = {
  id: number;
  name?: Nullable<string>;
  description?: Nullable<string>;
  mediaType?: Nullable<string>;
  modified?: Nullable<string>;
  resourceURI?: Nullable<string>;
  urls?: Url[];
  thumbnail?: Image;
  comics?: ResourceList;
  stories?: ResourceList;
  events?: ResourceList;
  series?: ResourceList;
};

Great, we are now done with this step! We can now focus on building the UI of our application.

Step 4: Build the layout

Even if our application is just one page, we can use the power of layouts to add shared UI using the daisyUI header/footer components. This is helpful if you want to add other pages later and avoid copying/pasting code in each page component.

So let’s customise the existing base layout in src/routes/layout.tsx:

export default component$(() => (
  <div class="flex flex-col min-h-screen bg-base-300">
    <div class="navbar sticky top-0 justify-center bg-primary text-primary-content">
      <p class="text-xl">Marvel Search App</p>
    </div>
    <main class="grow max-w-[81rem] w-full self-center px-8">
      <Slot />
    </main>
    <footer class="footer footer-center sticky bottom-0 p-4">
      <aside>
        <p>Data provided by Marvel. © {new Date().getFullYear()} Marvel</p>
      </aside>
    </footer>
  </div>
));

Because we must attribute Marvel as the source of data whenever we display any results from the Marvel API, the footer is the perfect place to add the required text.

Now if you refresh the application, you should see this:

Step 5: Build the search component

Let's proceed and create our first component: the search form. It will include a simple text input and a submit button.

Create a new file in src/components/search-form/SearchForm.tsx with the following:

import { component$ } from "@builder.io/qwik";
import type { Nullable } from "~/types";

type Props = {
  searchTerm: Nullable<string>;
};

export const SearchForm = component$((props: Props) => (
  <section class="bg-base-200 mt-8 rounded">
    <div class="flex flex-col items-center gap-4 p-8">
      <h1 class="text-3xl uppercase">Explore</h1>
      <p>Search your favorite Marvel characters!</p>
      <form class="flex flex-col md:flex-row justify-center items-center max-w-lg w-full gap-4">
        <input
          type="text"
          placeholder="Spider-Man, Thor, Avengers..."
          name="search"
          class="input w-full md:w-3/6"
          value={props.searchTerm}
        />
        <button type="submit" class="btn btn-primary w-full md:w-1/6">
          Search
        </button>
      </form>
    </div>
  </section>
));

Next, import your component in src/routes/index.tsx:

//...
import { SearchForm } from "~/components/search-form/SearchForm";

export default component$(() => {
  return (
    <>
      <SearchForm />
    </>
  );
});

Perfect, your application should look like this:

We can now move on and start fetching actual data from the Marvel API.

Step 6: Fetch data

To fetch data on the server so it becomes available to use in our page component when the page loads, we are going to use the routeLoader$() capacity from Qwik.

Let’s add a useSearchLoader() in src/routes/index.tsx, which will invoke getCharacters() that we previously defined based on a search term extracted from the URL parameters.

import { routeLoader$, type DocumentHead } from "@builder.io/qwik-city";
import { getMarvelContext, getCharacters } from "~/services/marvel";

export const useSearchLoader = routeLoader$(async (requestEvent) => {
  const searchTerm = requestEvent.url.searchParams.get('search');

  if (!searchTerm) {
    return null;
  }

  const params = {
    context: getMarvelContext(requestEvent),
    limit: 12,
    page: 1,
    startsWith: searchTerm,
  };

  return await getCharacters(params);
});

//...

But why use routeLoader$() over routeAction$() in our case? Because routeLoader$() allows us to rely exclusively on URL search parameters to retrieve our inputs before loading data.

Explanation: when the user clicks the form submit button, the URL is automatically appended with the form data (e.g. ?search=spiderman). The page then reloads, triggers again our loader, extracts the search input from the URL, loads the data on the server, and gives back the control to the component. One major advantage of this approach is that it makes sharing the URL easy.

We can now access our useSearchLoader() data in our component and see if the call was a success by showing the number of results for example:

// src/routes/index.tsx

import { routeLoader$, useLocation, type DocumentHead } from "@builder.io/qwik-city";
//...
export default component$(() => {
  const location = useLocation();
  const searchTerm = location.url.searchParams.get('search');

  const resource = useSearchLoader();

  return (
    <>
      <SearchForm
        searchTerm={searchTerm}
      />
      {resource.value?.data?.results && (
        <p>Total results: {resource.value.data.total}</p>
      )}
    </>
  );
});

We also used useLocation() here to retrieve the input from the RouteLocation object, making sure that the user's search is preserved when the page reloads.

So now if you search for the term "Thor", you should see this:

Great, we now successfully receive data from the Marvel API!

Step 7: Build the results component

Let's build the UI to display our characters. We will first create a grid component that automatically adjusts its columns based on screen size with Tailwind CSS.

Create a new file in src/components/character-grid/CharacterGrid.tsx:

import { component$ } from "@builder.io/qwik";
import type { Character } from "~/types";
import { CharacterCard } from "../character-card/CharacterCard";

type Props = {
  collection: Character[];
  total?: number;
};

export const CharacterGrid = component$((props: Props) => (
  <section class="my-8">
    <div class="flex flex-col md:flex-row items-center gap-2 mb-4">
      <p class="text-xl">
        Total results
      </p>
      <div class="badge badge-lg">{props.total}</div>
    </div>
    <div class="grid grid-cols-[repeat(auto-fill,minmax(12rem,1fr))] justify-items-center gap-4">
      {props.collection.map((character) => (
        <CharacterCard key={character.id} character={character} />
      ))}
    </div>
  </section>
));

Next, add the UI for a single character by creating a new file in src/components/character-card/CharacterCard.tsx with the following content:

import { component$ } from "@builder.io/qwik";
import type { Character } from "~/types";
import { getThumbnail } from "~/utils";

type Props = {
  character: Character;
};

export const CharacterCard = component$((props: Props) => (
  <div class="bg-base-200 w-48 overflow-hidden rounded">
    <div class="hover:scale-105 transition duration-300">
      <img
        alt={props.character.name || ""}
        width={192}
        height={288}
        class="object-cover object-top w-48 h-72"
        src={getThumbnail(props.character.thumbnail)}
      />
    </div>
    <p class="py-4 px-2 font-bold">{props.character.name}</p>
  </div>
));

This component is simple, it displays only a few information like the name of the character or the image. Speaking of the image, let’s add a new getThumbnail() in src/utils/index.ts to build the full path from the Image object:

import type { Image } from "~/types";
//...
export const getThumbnail = (thumbnail: Image | undefined): string => {
  if (!thumbnail) {
    return "";
  }

  return `${thumbnail.path}.${thumbnail.extension}`;
};

Finally, import the CharacterGrid component in src/routes/index.tsx to see the result:

import { CharacterGrid } from "~/components/character-grid/CharacterGrid";
//...

export default component$(() => {
  //...

  return (
    <>
      <SearchForm
        searchTerm={searchTerm}
      />
      {resource.value?.data?.results && (
        <CharacterGrid
          collection={resource.value.data.results}
          total={resource.value.data.total}
        />
      )}
    </>
  );
});

Now if you refresh your browser, you should see the following:

It looks great! We are almost done but we still have one more thing to do before we wrap up this tutorial.

Step 8: Add a load more button

As you may have noticed earlier, we have limited our useSearchLoader() to 12 items. So if you search for “s”, you will have a lot more results but only 12 characters displayed on your screen.

So how to display all of them? With a load more button! It is very useful for breaking up large datasets into smaller, more manageable chunks.

First, let’s bring some changes to our CharacterGrid component to display a “Show more” button at the bottom:

import { component$, type QRL } from "@builder.io/qwik";
//...
type Props = {
  //...
  currentPage: number;
  onMore$?: QRL<() => void>;
  totalPages: number;
};
export const CharacterGrid = component$((props: Props) => (
  <section class="my-8">
    //...
    {props.currentPage < props.totalPages && (
      <div class="flex justify-center mt-8">
        <button
          type="button"
          class="btn btn-primary text-base-content"
          onClick$={props.onMore$}
        >
          Show more
        </button>
      </div>
    )}
  </section>
));

This button will be displayed only if the total number of pages is superior to the current page.

Next, prepare a new loader to fetch our data. Since we cannot reuse useSearchLoader(), which only triggers when the page loads, we will implement a new function using Qwik's server$() capability. This function allows us to define a server-exclusive function, making it ideal for loading our data when the user clicks on the button.

In src/routes/index.tsx:

import { routeLoader$, server$, useLocation, type DocumentHead } from "@builder.io/qwik-city";
//...
export const getMore = server$(async function (page, searchTerm) {
  const params = {
    context: getMarvelContext(this),
    limit: 12,
    page,
    startsWith: searchTerm,
  };

  return await getCharacters(params);
});

Now, to call our newly created getMore() method, we need to modify our component to implement the callback along with two state objects. The first one, currentPage, will hold the current page number which will increment every time the user clicks on the button, and the second one, collection, will store all the results from our requests made to the API.

// src/routes/index.tsx

import { $, component$, useSignal } from "@builder.io/qwik";
import type { Character } from "~/types";
import { getTotalPages } from "~/utils";

//...
export default component$(() => {
  //...

  const currentPage = useSignal<number>(1);
  const collection = useSignal<Character[]>(
    resource.value?.data?.results || []
  );

  const handleMore = $(async () => {
    const newData = await getMore(currentPage.value + 1, searchTerm);
    const newResults = newData.data?.results || [];
    collection.value = [...collection.value, ...newResults];
    currentPage.value += 1;
  });

  return (
    <>
      //...
      {resource.value?.data?.results && (
        <CharacterGrid
          collection={collection.value}
          currentPage={currentPage.value}
          onMore$={handleMore}
          total={resource.value.data.total}
          totalPages={getTotalPages(resource.value.data.total, 12)}
        />
      )}
    </>
  );
});

Finally, to determine the total number of pages, add the following method in src/utils/index.ts:

export const getTotalPages = (
  total: number | undefined,
  limit: number,
): number => (total ? Math.ceil(total / limit) : 1);

Awesome, we are now done with our load more button. You can click the "Show more" button and see the additional search results:

Conclusion

If you are reading this, it means you have reached the end of this tutorial. Thank you for following along, I hope you enjoyed!
Your search page is now complete, feel free to modify anything you want; the goal was to create a playground for you to customise in the future.

For more details, check out my demo repository here.

And if you want to go further, we can also explore a more complete version of the application I created here.

Resources: